# LIST: Learning Implicitly from Spatial Transformers
for Single-View 3D Reconstruction

Mohammad Samiul Arshad and William J. Beksi
Department of Computer Science and Engineering
The University of Texas at Arlington, Arlington, TX, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Accurate reconstruction of both the geometric and topological details of a 3D
object from a single 2D image embodies a fundamental challenge in computer
vision. Existing explicit/implicit solutions to this problem struggle to
recover self-occluded geometry and/or faithfully reconstruct topological shape
structures. To resolve this dilemma, we introduce LIST, a novel neural
architecture that leverages local and global image features to accurately
reconstruct the geometric and topological structure of a 3D object from a
single image. We utilize global 2D features to predict a coarse shape of the
target object and then use it as a base for higher-resolution reconstruction.
By leveraging both local 2D features from the image and 3D features from the
coarse prediction, we can predict the signed distance between an arbitrary
point and the target surface via an implicit predictor with great accuracy.
Furthermore, our model does not require camera estimation or pixel alignment.
It provides an uninfluenced reconstruction from the input-view direction.
Through qualitative and quantitative analysis, we show the superiority of our
model in reconstructing 3D objects from both synthetic and real-world images
against the state of the art. Our source code is publicly available to the
research community [15].

##  1 Introduction

Constructing a truthful portrayal of the 3D world from a single 2D image is a
basic problem for many applications including robot manipulation and
navigation, scene understanding, view synthesis, virtual reality, and more.
Following the work of Erwin Kruppa [13] in camera motion estimation and the
recovery of 3D points, researchers have attempted to solve the 3D
reconstruction issue using structure from motion [36, 18, 31], and visual
simultaneous localization and mapping [8, 30]. However, the main limitation of
such approaches is that they require multiple observations of the desired
object or scene from distinct viewpoints with shared features. Such a multi-
view formulation allows for integrating information from numerous images to
compensate for occluded geometry.

Fig. 1: Five unique views of objects reconstructed by LIST from a single RGB
image. Not only does our model accurately recover occluded geometry, but also
the reconstructed surfaces are not influenced by the input-view direction.

Reconstructing a 3D object from a single image is a more difficult task since
a sole image does not contain the whole topology of the target shape due to
self-occlusions. Researchers have tried both explicit and implicit techniques
to reconstruct a target object with self-occluded parts. Explicit methods
attempt to infer the target shape directly from the input image. Nevertheless,
a major drawback of such approaches is that the output resolution needs to be
defined in advance, which constrains these techniques from achieving high-
quality results. Recent advances in implicit learning offer a solution to
reconstruct the target shape in an arbitrary resolution by indirectly
inferring the desired surface through a distance/occupancy field. Then, the
target surface is reconstructed by extracting a zero level set from the
distance/occupancy field.

Implicit 3D reconstruction from a single view is an active area of research
where one faction of techniques [20, 3] encode global image features into a
latent representation and learn an implicit function to reconstruct the
target. Yet, these approaches can be easily outperformed by simple retrieval
baselines [35]. Therefore, global features alone are not sufficient for a
faithful reconstruction. Another faction leverages both local and global
features to learn the target implicit field from pixel-aligned query points.
However, such methods rely on ground-truth/estimated camera parameters for
training/inference [38, 14], or they assume weak perspective projection [28,
10].

To address these shortcomings we propose LIST, a novel deep learning framework
that can reliably reconstruct the topological and geometric structure of a 3D
object from a single RGB image. Our method does not depend on weak perspective
projection, nor does it require any camera parameters during training or
inference. Moreover, we leverage both local and global image features to
generate highly-accurate topological and geometric details. To recover self-
occluded geometry and aid the implicit learning process, we first predict a
coarse shape of the target object from the global image features. Then, we
utilize the local image features and the predicted coarse shape to learn a
signed distance function (SDF).

Due to the scarcity of real-world 2D-3D pairs, we train our model on synthetic
data. However, we use both synthetic and-real world images to test the
reconstruction ability of LIST. Through qualitative analysis we highlight our
model’s superiority in reconstructing high-fidelity geometric and topological
structure. Via a quantitative analysis using traditional evaluation metrics,
we show that the reconstruction quality of LIST surpasses existing works.
Furthermore, we design a new metric to investigate the reconstruction quality
of self-occluded geometry. Finally, we provide an ablation study to validate
the design choices of LIST in achieving high-quality single-view 3D
reconstruction.

##  2 Related Work

In this section we summarize pertinent work on the reconstruction of 3D
objects from a single RGB image via implicit learning. Interested readers are
encouraged to consult [7] for a comprehensive survey on 3D reconstruction from
2D images. Contrary to explicit representations, implicit ones allow for the
recovery of the target shape at an arbitrary resolution. This benefit has
attracted interest among researchers to develop novel implicit techniques for
different applications. Dai _et al_. [5] used a voxel-based implicit
representation for shape completion. DeepSDF, introduced by Park _et al_.
[25], is an auto-decoder that learns to estimate signed distance fields.
However, DeepSDF requires test-time optimization, which limits its efficiency
and capability.

To further improve 3D object reconstruction quality, Littwin and Wolf [16]
utilized encoded image features as the network weights of a multilayer
perceptron. Wu _et al_. [37] explored sequential part assembly by predicting
the SDFs for structural parts separately and then combining them together. For
self-supervised learning, Liu _et al_. [17] proposed a ray-based field probing
technique to render the implicit surfaces as 2D silhouettes. Niemeyer _et al_.
[23] used supervision from RGB, depth, and normal images to reconstruct rich
geometry and texture. Chen and Zhang [3] proposed generative models for
implicit representations and leveraged global image features for single-view
reconstruction. For multiple 3D vision tasks, Mescheder _et al_. [20]
developed OccNet, a network that learns to predict the probability of a
volumetric grid cell being occupied.

Pixel-aligned approaches [28, 29, 10, 1] have employed local query feature
extraction from image pixels to improve 3D human reconstruction. Xu _et al_.
[38] incorporated similar ideas for 3D object reconstruction. To enhance the
reconstruction quality of surface details, Li and Zhang [14] utilized normal
images and a Laplacian loss in addition to aligned features. Zhao _et al_.
[40] exploited coarse prediction and unsigned distance fields to reconstruct
garments from a single view. Duggal and Pathak [6] proposed category specific
reconstruction by learning a topology aware deformation field. Mittal
$\emph{et al}.\hbox{}$ introduced AutoSDF [21], a model that encodes local
shape regions separately via patch-wise encoding. However, these prior works
rely on weak perspective projection and the rendering of metadata to align
query points to image pixels. In contrast, LIST does not require any alignment
or rendering data, and it recovers more accurate topological structure and
geometric details.

##  3 Implicit Function Learning from Unaligned Pixel Features

Fig. 2: To reconstruct the target object from a single RGB image, LIST first
predicts the coarse topology from the global image features. Simultaneously,
local image features are used to extract local geometry at the given query
locations. Finally, an SDF predictor ($\Psi$) estimates the signed distance
field ($\sigma$) to reconstruct the target shape. Note that images and colors
are for visualization purposes only.

Given a single RGB image of an object, our goal is to reconstruct the object
in 3D with highly-accurate topological structure and self-occluded geometry.
We model the target shape as an SDF and extract the underlying surface from
the zero level set of the SDF during inference. To train our model we employ
an image and query point pair ($x_{i},Q_{i}$), where $Q_{i}$ is a set of 3D
coordinates (query points) in close vicinity to the surface of the object with
a measured signed distance and $x_{i}$ is a rendering of the object from a
random viewpoint. An overview of the our framework is presented in Fig. 2. The
details of each component are provided in the following subsections.

###  3.1 Query Features From Coarse Predictions

Consider an RGB image $x_{i}\subset X\in\mathbb{R}^{H\times W\times 3}$ of
height $H$ and width $W$. We propose a convolutional neural encoder-decoder
$\Omega_{\omega}$, parameterized by weights $\omega$, to extract latent
features from the image and predict a coarse estimation $\dot{y}_{i}^{x_{i}}$
of the target object. Concretely,

$\Omega_{\omega}(x_{i})\coloneqq\dot{y}_{i}^{x_{i}}~{}|~{}\mathbb{R}^{H\times
W\times 3}\rightarrow\mathbb{R}^{N\times 3},$ (1)

where $\dot{y}_{i}^{x_{i}}$ is a point cloud representation of the target and
$N$ is the resolution of the point cloud. Note that the subscript $i$
indicates $i$-th sample and the superscript $x_{i}$ designates the source
variable. For high-performance point cloud generation, we utilize tree
structured graph convolutions (TreeGCN) [32] to decode the image features.

We use the coarse prediction $\dot{y}_{i}$ as a guideline for the topological
structure of the target shape in a canonical space. To extract query features
from this coarse prediction, first we discretize the point cloud in an
occupancy grid $\dot{u}_{i}^{\dot{y}_{i}}\in 1^{M\times M\times M}$ of
resolution $M$. However, the coarse prediction may contain gaps and noisy
points that may impair the reconstruction quality. To resolve this, we employ
a shallow convolutional network $\Gamma_{\ddot{o}}$ parameterized by weights
$\ddot{o}$ to generate a probabilistic occupancy grid from
$\dot{u}_{i}^{\dot{y}_{i}}$,

$\dot{v}_{i}^{\dot{u}_{i}}\coloneqq\Gamma_{\ddot{o}}(\dot{u}_{i}^{\dot{y}_{i}})\colon
1^{M\times M\times M}\rightarrow[0,1]^{M\times M\times M}.$ (2)

Specifically, our aim is to find the neighboring points of $\dot{y}_{i}$ with
a high chance of being a surface point of the target shape.

Although it is possible to regress the voxel representation directly from the
global image features [4, 33, 10], learning a high-resolution voxel occupancy
prediction requires a significant amount of computational resources [10].
Moreover, we empirically found that point cloud prediction followed by voxel
discretization achieves better accuracy on diverse shapes rather than
predicting the voxels directly.

Next, a neural network $\Xi_{\xi}$, parameterized by weights $\xi$, maps the
probabilistic occupancy grid (2) to a high-dimensional latent matrix through
convolutional operations. Then, our multi-scale trilinear interpolation scheme
$\mathit{I}$ extracts relevant query features $f_{C}$ at each query location
$q_{i}$ from the mapped features. More formally,

$f_{C}\coloneqq\mathit{I}(\Xi_{\xi}(\dot{v}_{i}^{\dot{u}_{i}}),Q_{i}).$ (3)

In addition to $q_{i}$, we also consider the neighboring points at a distance
$d$ from $q_{i}$ along the Cartesian axes to capture rich 3D features, i.e.,

$q_{j}=q_{j}+k\cdot\hat{n}_{j}\cdot d,$ (4)

where $k\in\\{1,0,-1\\}$, $j\in\\{1,2,3\\}$, and
$\hat{n}_{j}\in\mathbb{R}^{3}$ is the $j$-th Cartesian axis unit vector.

###  3.2 Localized Query Features

The coarse prediction and query features $f_{C}$ can aid the recovery of the
topological structure of the target shape. Nevertheless, relevant local
features are also required to recover fine geometric details. To achieve this,
prior arts assume weak perspective projection [28, 10] or align the query
points to the image pixel locations through the ground-truth/estimated camera
parameters [38, 14]. Predicting the camera parameters is analogous to
predicting the object pose from a single image, which is itself a hard problem
in computer vision. It involves a high chance of error and a computationally
expensive training procedure. Furthermore, the error in the pose/camera
estimation may lead to the loss of geometric details in the reconstruction.

To overcome these limitations, we obtain insight from spatial transformers
[11] and leverage the spatial relationship between the input image and the
coarse prediction. Via the coarse prediction, which portrays an object from a
standard viewpoint and the query points that delineate the coarse predictions,
it is possible to localize the query points to the local image features. This
is done by predicting a spatial transformation with the aid of global features
from the input image and the coarse prediction as follows.

First, we define a convolutional neural encoder $\Pi_{\pi}$, parameterized by
weights $\pi$, to encode the input image into local $(l^{x_{i}}_{\pi})$ and
global $(z^{x_{i}}_{\pi})$ features. Concretely,

$\Pi_{\pi}(x_{i})\coloneqq\\{l^{x_{i}}_{\pi},z^{x_{i}}_{\pi}\\}.$ (5)

Concurrently, a neural module $K_{\kappa}$ encodes the coarse prediction
$\dot{y}_{i}^{x_{i}}$ into global point features. Using global features from
both the image and the coarse prediction, the spatial transformer $\Theta$
estimates a transformation to localize the query points in the image feature
space. Then, localized query points $\tilde{Q_{i}}$ are generated by applying
the predicted transformation to $Q_{i}$,

$\Theta_{\theta}(z_{\pi}^{x_{i}},K_{\kappa}(\dot{y}_{i}^{x_{i}}),Q_{i})\coloneqq\tilde{Q_{i}}~{}|~{}\mathbb{R}^{N\times
3}\rightarrow\mathbb{R}^{N\times 2}.$ (6)

Finally, a bi-linear interpolation scheme $\mathcal{B}$ extracts the local
query features $f_{L}$ from the local image features $l^{x_{i}}_{\pi}$,

$\mathit{f}_{L}\coloneqq\mathcal{B}(l_{\pi}^{x_{i}},\tilde{Q_{i}}).$ (7)

Note that the point encoder $K_{\kappa}$ and the localization network $\Theta$
are designated to ensure an accurate SDF prediction. Therefore, we do not use
any camera parameters during training and we optimize these neural modules
directly with the SDF prediction objective. This has the following benefits:
(i) additional modules or training to predict the projection matrix and object
pose from a single image are not required; (ii) reconstructions are free from
any pose estimation error, which boosts reconstruction accuracy.

###  3.3 Signed Distance Function Prediction

To estimate the final signed distance $\Delta_{i}$, we combine the coarse
features $f_{C}$ with the localized query features $f_{L}$ and utilize a
multilayer neural function defined as

$\Psi_{\psi}(f_{C},f_{L})\coloneqq\begin{cases}\mathbb{R}^{-},&\text{if
$q_{i}$ is inside the target surface}\\\
\mathbb{R}^{+},&\text{otherwise}.\end{cases}$ (8)

###  3.4 Loss Functions

We incorporate the chamfer distance (CD) loss and optimize the weights
$\omega$ to accurately estimate the coarse shape of the target. More
specifically,

$\mathcal{L}_{\text{CD}}(y_{i},\dot{y}_{i})=\sum_{a\in\dot{y}_{i}}\min_{b\in
y_{i}}{||a-b||}^{2}+\sum_{b\in y_{i}}\min_{b\in\dot{y}_{i}}{||b-a||}^{2},$ (9)

where $y_{i}\in\mathbb{R}^{N\times 3}$ is a set of 3D coordinates collected
from the surface of the object and $\dot{y}_{i}\in\mathbb{R}^{N\times 3}$ is
the estimated coarse shape. To supervise the probabilistic occupancy grid
prediction, we discretize $y_{i}$ to generate the ground-truth occupancy
$v_{i}^{y_{i}}\in 1^{M\times M\times M}$. The neural weight $\ddot{o}$ is then
optimized by the binary cross-entropy loss,

$\mathcal{L}_{V}(v_{i},\dot{v}_{i})=-\frac{1}{|v_{i}|}\Sigma(\gamma
v_{i}\log{\dot{v}_{i}}+(1-\gamma)(1-v_{i})\log(1-{\dot{v}_{i}})),$ (10)

where $\gamma$ is a hyperparameter to control the influence of the
occupied/non-occupied grid points. To optimize the SDF prediction, we collect
a set of query points $Q_{i}$ within distance $\delta$ of the target surface
and measure their signed distance $\sigma_{i}$. The estimated signed distance
is then guided by optimizing the neural weights $\xi$, $\pi$, $\theta$, and
$\psi$ through

$\mathcal{L}_{\text{SDF}}=\frac{1}{|Q_{i}|}\Sigma(\sigma_{i}-\Delta_{i})^{2}.$
(11)

###  3.5 Training Details

We incorporate a two-stage procedure to train LIST. In the first stage, we
only focus on the coarse prediction from the input image $x_{i}$ and optimize
the weights $\omega$ through $\mathcal{L}_{\text{CD}}$. Then, we freeze
$\omega$ after convergence to a minimum validation accuracy and start the
second stage for the SDF prediction. During the second stage, we jointly
optimize $\ddot{o}$, $\xi$, $\pi$, $\kappa$, $\theta$, and $\psi$ through the
combined loss $\mathcal{L}=\mathcal{L}_{V}+\mathcal{L}_{\text{SDF}}$. LIST can
also be trained end-to-end by jointly minimizing $\mathcal{L}_{CD}$ with
$\mathcal{L}_{V}$ and $\mathcal{L}_{\text{SDF}}$. However, we found the two-
stage training procedure easier to evaluate and quicker to converge during
experimental evaluation. To reconstruct an object at test time, we first
densely sample a fixed 3D grid of query points and predict the signed distance
for each point. Then, we use the marching cubes [19] algorithm to extract the
target surface from the grid.

##  4 Experimental Evaluation

In this section, we describe the details of our experimental setup and
results. Additional information, including implementation details, can be
found in the supplementary material.

###  4.1 Datasets

Similar to [14] and [21], we utilized the 13-class subset of the ShapeNet [2]
dataset to train LIST. The renderings and processed meshes from [38] were used
as the input view and target shape. We trained a single model on all 13
categories. Additionally, we employed the Pix3D [34] dataset to test LIST on
real-world scenarios. The train/test split from [39] was used to evaluate on
all 9 categories of Pix3D. Following [39], we preprocessed the Pix3D target
shapes to be watertight for training.

To prepare the ground-truth data, we first normalized the meshes to a unit
cube and then sampled 50 k points from the surface of each object. Next, we
displaced the sampled points with a Normal distribution of zero mean and
varying standard deviation. Lastly, we calculated the signed distance for
every point. To supervise the coarse prediction and probabilistic occupancy
grid estimation, we sub-sampled 4 k points from the surface via farthest point
sampling. Further details regarding the data preparation strategy can be found
in the supplementary material.

###  4.2 Baseline Models

For single-view reconstruction via synthetic images, we compared against the
following prior arts: IMNET [3], and D2IM-Net [14]. IMNET does not require
pose estimation. However, the reconstruction only unitizes global features
from an image. D2IM-Net extracts local features by aligning the query points
to image pixels through rendering metadata and it uses a pose estimation
module during inference.

For single-view reconstruction from real-world images, we evaluated against
TMN [24], MGN [22], and IM3D [39]. TMN deforms a template mesh to reconstruct
the target object. MGN and IM3D perform reconstruction through the following
steps: (i) identify objects in a scene, (ii) estimate their poses, and (iii)
reconstruct each object separately.

###  4.3 Metrics

We computed commonly used metrics (e.g., CD, intersection over union (IoU),
and F-score), to evaluate the performance of LIST. The definitions of these
metrics can be found in the supplementary material. Nonetheless, these
traditional metrics do not differentiate between visible/occluded surfaces
since they evaluate the reconstruction as a whole. To investigate the
reconstruction quality of occluded surfaces, we propose to isolate
visible/occluded surfaces based on the viewpoint of the camera and evaluate
them separately using the traditional metrics. A visual depiction of this new
strategy is presented in Fig. 3.

To measure the reconstruction quality of occluded surfaces, we first align the
predicted/ground-truth meshes to their projection in the input image using the
rendering metadata. Then, we assume the camera location as a single source of
light and cast rays onto the mesh surface by ray casting [27]. Next, we
identify the visible/occluded faces through the ray-mesh intersection and
subdivide the identified faces to separate them. Note that the rendering
metadata is only used to evaluate the predictions. Finally, we sample 100 k
points from the separated occluded faces to compute the
$\text{CD}_{\text{os}}$, and voxelize the sampled points to compute the
$\text{IoU}_{\text{os}}$ and $\text{F-Score}_{\text{os}}$.

In our implementation, we set the canvas resolution to $4096\times 4096$
pixels and generated one ray per pixel from the camera location. It is
important to note that ray casting and computing ray-mesh intersections are
computationally demanding tasks. Therefore, to manage time and resources, we
chose five sub-classes (chair, car, plane, sofa, table) to evaluate occluded
surface reconstruction.

(a)

(b)

(c)

(d)

(e)

Fig. 3: To evaluate the reconstruction quality of occluded surfaces, we first
align the reconstructed shape (b) with the input image (a) and cast rays onto
the surface (c). Next, we identify the (red) faces that intersect with the
rays via ray-mesh intersection and separate the reconstructed mesh into (d)
visible and (e) occluded areas.

###  4.4 Single-View 3D Reconstruction Evaluation

Fig. 4: A qualitative comparison between LIST and the baseline models using the ShapeNet [2] dataset. Our model recovers significantly better topological and geometric structure, and the reconstruction is not tainted by the input-view direction. GT denotes the ground-truth objects. |  | plane | bench | cabinet | car | chair | display | lamp | speaker | rifle | sofa | table | phone | boat | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CD$\downarrow$ | IMNET | 18.95 | 17.34 | 15.17 | 10.86 | 14.72 | 16.77 | 83.64 | 33.41 | 10.33 | 13.35 | 19.32 | 9.16 | 15.24 | 21.40
D2IM-Net | 13.25 | 12.51 | 9.47 | 7.83 | 11.31 | 15.33 | 34.08 | 17.62 | 8.55 | 12.34 | 14.26 | 8.11 | 15.73 | 13.87
LIST | 12.13 | 13.49 | 7.45 | 1.04 | 9.20 | 13.65 | 47.31 | 16.75 | 7.32 | 9.92 | 11.14 | 7.91 | 15.78 | 13.31
IoU$\uparrow$ | IMNET | 39.43 | 44.65 | 49.25 | 55.75 | 51.22 | 53.34 | 29.26 | 50.66 | 46.43 | 51.12 | 41.63 | 52.79 | 49.61 | 47.31
D2IM-Net | 45.44 | 48.45 | 48.60 | 53.58 | 53.13 | 52.72 | 32.45 | 51.75 | 50.76 | 53.35 | 45.17 | 53.06 | 52.89 | 49.33
LIST | 49.03 | 47.57 | 56.29 | 65.57 | 52.70 | 57.34 | 24.80 | 55.34 | 52.42 | 56.79 | 47.90 | 58.98 | 54.35 | 52.23
F-score$\uparrow$ | IMNET | 48.87 | 31.78 | 44.34 | 48.78 | 41.45 | 48.32 | 21.23 | 48.29 | 52.92 | 44.12 | 45.21 | 51.52 | 52.31 | 44.54
D2IM-Net | 51.37 | 36.76 | 43.49 | 51.77 | 45.56 | 50.82 | 29.57 | 51.93 | 56.25 | 48.34 | 47.23 | 54.84 | 52.73 | 47.74
LIST | 52.46 | 36.39 | 42.51 | 53.12 | 46.62 | 51.78 | 22.88 | 52.67 | 58.24 | 50.52 | 49.62 | 56.89 | 53.58 | 48.25
Table 1: Quantitative results using the ShapeNet [2] dataset for various models. The metrics reported are the following: chamfer distance (CD), intersection over union (IoU), and F-score. The CD values are scaled by $10^{-3}$. Fig. 5: A qualitative comparison between LIST and the baseline models using distinct views of the same object. Not only can our model both maintain better topological structure and geometric details, but it also provides a reconstruction that is stable across different views of the object. |  | plane | car | chair | sofa | table | Mean
---|---|---|---|---|---|---|---
CD${}_{\text{os}}\downarrow$ | IMNET | 24.11 | 13.34 | 15.47 | 24.34 | 26.86 | 20.82
D2IM-Net | 26.23 | 13.44 | 13.59 | 20.45 | 23.45 | 19.43
LIST | 18.93 | 6.57 | 12.66 | 18.44 | 21.76 | 15.67
IoU${}_{\text{os}}\uparrow$ | IMNET | 45.63 | 46.87 | 38.32 | 45.87 | 39.02 | 43.14
D2IM-Net | 48.44 | 50.33 | 49.43 | 50.32 | 42.22 | 48.14
LIST | 53.15 | 55.37 | 51.25 | 55.22 | 43.17 | 51.63
F${}_{\text{os}}$-score$\uparrow$ | IMNET | 40.93 | 46.94 | 44.43 | 46.84 | 45.64 | 44.95
D2IM-Net | 47.21 | 50.73 | 48.89 | 49.15 | 47.72 | 48.73
LIST | 50.33 | 52.55 | 49.34 | 51.02 | 48.11 | 50.27
Table 2: A quantitative evaluation of the occluded surfaces of reconstructed
synthetic objects via our evaluation strategy. The metrics reported are the
following: chamfer distance (CD${}_{\text{os}}$), intersection over union
(IoU${}_{\text{os}}$), and F${}_{\text{os}}$-score. The CD${}_{\text{os}}$
values are scaled by $10^{-3}$.

####  4.4.1 Single-View 3D Reconstruction from Renderings of Synthetic Objects

In this experiment we performed single-view 3D reconstruction on the test set
of the ShapeNet dataset. The qualitative and quantitative results are
displayed in Fig. 4 and Table 1, respectively. In comparison to the baselines,
the topological structure and occluded geometry recovered by LIST are
considerably better. For example, in row 3 all of the baselines struggle to
reconstruct the tail of the airplane and they fail to estimate the full length
of the wings. In row 5, none of the baselines were able to recover the
occluded part of the table. In contrast, LIST not only recovers the structure,
but it also maintains the gap in between. Moreover, notice that in row 2 D2IM-
Net fails to resolve the directional view ambiguity and imprints an arm shaped
silhouette on the seat rather than reconstructing the arm. This indicates a
strong influence of the input-view direction in the reconstructed surface.
Conversely, LIST can resolve view-directional ambiguity and provide a
reconstruction that is uninfluenced by the input-view direction. As shown in
Table 1, LIST outperforms all the other baseline models.

We also evaluated LIST against the baselines on occluded surface recovery by
partitioning the reconstructions using our proposed metric. The results are
recorded in Table 2. LIST outperformed all the baselines hence showcasing the
superiority of our approach in reconstructing occluded geometry. Furthermore,
LIST provides a stable reconstruction across different views of the same
object as shown in Fig. 5. However, the use of ground-truth rendering data
instead of the estimated data improved the reconstruction quality. This
indicates the source of the problem to be the sub-optimal prediction of the
camera pose. Nonetheless, LIST is free from any such complication as our
framework does not require any explicit pose estimation.

####  4.4.2 Single-View 3D Reconstruction from Real Images

|  | bed | bookcase | chair | desk | sofa | table | tool | wardrobe | misc | Mean
---|---|---|---|---|---|---|---|---|---|---|---
CD$\downarrow$ | TMN | 7.78 | 5.93 | 6.86 | 7.08 | 4.25 | 17.42 | 4.13 | 4.09 | 23.68 | 9.03
MGN | 5.99 | 6.56 | 5.32 | 5.93 | 3.36 | 14.19 | 3.12 | 3.83 | 26.93 | 8.36
IM3D | 4.11 | 3.96 | 5.45 | 7.85 | 5.61 | 11.73 | 2.39 | 4.31 | 24.65 | 6.72
LIST | 5.81 | 1.74 | 6.11 | 3.87 | 2.08 | 1.68 | 1.99 | 0.80 | 5.16 | 4.36
IoU$\uparrow$ | LIST | 45.61 | 39.54 | 41.15 | 59.68 | 67.34 | 49.12 | 27.82 | 43.87 | 34.72 | 46.77
F-score$\uparrow$ | LIST | 58.18 | 67.22 | 60.01 | 78.34 | 70.14 | 69.19 | 46.48 | 75.70 | 39.14 | 65.66
Table 3: A quantitative evaluation of the occluded surfaces of reconstructed
real-world objects using our evaluation strategy. The metrics reported are the
following: chamfer distance (CD${}_{\text{os}}$), intersection over union
(IoU${}_{\text{os}}$), and F${}_{\text{os}}$-score. The CD${}_{\text{os}}$
values are scaled by $10^{-3}$.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p) Input

(q) MGN

(r) IM3D

(s) LIST

(t) GT

Fig. 6: Single-view reconstruction using real-world images from the Pix3D [34]
test set (best viewed zoomed in).

In this experiment we evaluated single-view 3D reconstruction on the test set
of the Pix3D dataset. The qualitative and quantitative results are provided in
Fig. 6 and Table 3, respectively. The baseline results were obtained from the
respective papers. Compared to other methods our approach generates the most
precise 3D shapes, which results in the lowest average CD and F-score. Notice
that in Fig. 6, rows 3 and 4, only LIST can accurately recover the back and
legs of the chair. Additionally, LIST reconstructions provide a smooth
surface, precise topology, and fine geometric details.

###  4.5 Ablation Study

Fig. 7: Qualitative results obtained from the ablation study using different network settings. | Base | OL |  $1$E |  $2$D | EC | Final
---|---|---|---|---|---|---
CD$\downarrow$ | 11.35 | 9.64 | 10.72 | 8.48 | 7.89 | 7.32
IoU$\uparrow$ | 51.34 | 53.95 | 51.40 | 55.23 | 55.10 | 56.83
F-score$\uparrow$ | 43.11 | 48.06 | 45.92 | 51.37 | 51.33 | 52.75
Table 4: Quantitative results obtained from the ablation study using different
network settings.

####  4.5.1 Setup

To investigate the impact of each individual component in our single-view 3D
reconstruction model, we performed an ablation study with the following
network options.

  * •

Base: A version of LIST that predicts the signed distance utilizing only
global image features and coarse predictions.

  * •

OL: An improved Base version that uses the probabilistic occupancy from the
coarse prediction and occupancy loss.

  * •

1E: A version of LIST where local and global image features from the same
encoder are used for both coarse prediction and localized query feature
extraction.

  * •

2D: LIST with two separate decoders to estimate the signed distance from local
and global query features. The final prediction is obtained by adding both
estimations.

  * •

EC: We train LIST without the localization module and use a separate pose
estimation module similar to [14] to predict the camera parameters. The
estimated camera parameters were used to transform the query points during
inference.

To maximize limited computational resources, we focused on the most diverse
five sub-classes (chair, car, plane, sofa, table) of the ShapeNet dataset for
this ablation study. The qualitative and quantitative results of the
experiments are recorded in Fig. 7 and Table 4 respectively.

####  4.5.2 Discussion

In the ablation experiments the Base version was able to recover global
topology, but it lacked local geometry. As shown in Fig 7, the probabilistic
occupancy and optimization loss helped recover some details in the OL version.
Conversely, the performance decreased slightly after the inclusion of local
details in the single-encoder version (1E). We hypothesize that the task of
query point localization, while estimating the coarse prediction, overloads
the encoder and hinders meaningful feature extraction for the signed distance
prediction. To overcome this issue, we used a separate encoder for the coarse
prediction and query point localization. The dual-decoder version (2D),
performed similar to the final model. Nonetheless, we found that the geometric
details had a thicker reconstruction than the target during qualitative
evaluation. This motivated the fusion of features rather than predictions in
the final version.

We also ablated the localization module using estimated camera parameters
during training and inference. As shown in Table 4, the final version of LIST
outscores the version employing estimated camera (EC) parameters. This
indicates that our localization module with an SDF prediction objective is
more suitable for single-view reconstruction compared to a camera pose
estimation sub-module. More importantly, this removes the requirement for
pixel-wise alignment through camera parameters for local feature extraction.
Note that the EC reconstruction appears qualitatively similar to the others
and was therefore omitted in Fig. 7.

###  4.6 Limitations and Future Directions

Although LIST achieves state-of-the-art performance on single-view 3D
reconstruction, there are some limitations. For example, the model may
struggle with very small structures. We speculate that this is due to the
coarse predictor failing to provide a good estimation of such structures.
Please see the supplementary material for examples of failed reconstruction
results. Another shortcoming is the need for a clear image background. LIST
can reconstruct targets from real-world images, yet it requires an uncluttered
background to do this. In the future, we will work towards resolving these
issues.

##  5 Conclusion

In this paper we introduced LIST, a network that implicitly learns how to
reconstruct a 3D object from a single image. Our approach does not assume weak
perspective projection, nor does it require pose estimation or rendering data.
We achieved state-of-the-art performance on single-view reconstruction from
renderings of synthetic objects. Furthermore, we demonstrated domain
transferability of our model by recovering 3D surfaces from images of real-
world objects. We believe our approach could be beneficial for other problems
such as object pose estimation and novel view synthesis.

## Acknowledgments

The authors acknowledge the Texas Advanced Computing Center (TACC) at The
University of Texas at Austin for providing software, computational, and
storage resources that have contributed to the research results reported
within this paper.

## References

  * [1] Yukang Cao, Guanying Chen, Kai Han, Wenqi Yang, and Kwan-Yee K Wong.  Jiff: Jointly-aligned implicit face function for high quality single view clothed human reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2729–2739, 2022.
  * [2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [3] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  * [4] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In Proceedings of the European Conference on Computer Vision, pages 628–644. Springer, 2016.
  * [5] Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner.  Shape completion using 3d-encoder-predictor cnns and shape synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5868–5877, 2017.
  * [6] Shivam Duggal and Deepak Pathak.  Topologically-aware deformation fields for single-view 3d reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1536–1546, 2022.
  * [7] Kui Fu, Jiansheng Peng, Qiwen He, and Hanxiao Zhang.  Single image 3d object reconstruction based on deep learning: A review.  Multimedia Tools and Applications, 80(1):463–498, 2021.
  * [8] Jorge Fuentes-Pacheco, José Ruiz-Ascencio, and Juan Manuel Rendón-Mancha.  Visual simultaneous localization and mapping: a survey.  Artificial Intelligence Review, 43(1):55–81, 2015.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [10] Tong He, John Collomosse, Hailin Jin, and Stefano Soatto.  Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction.  In Proceedings of the Advances in Neural Information Processing Systems, volume 33, pages 9276–9287, 2020.
  * [11] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  In Proceedings of the Advances in Neural Information Processing Systems, volume 28, 2015.
  * [12] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [13] Erwin Kruppa.  Zur Ermittlung eines Objektes aus zwei Perspektiven mit innerer Orientierung.  Hölder, 1913.
  * [14] Manyi Li and Hao Zhang.  D2im-net: Learning detail disentangled implicit fields from single images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10246–10255, 2021.
  * [15] https://github.com/robotic-vision-lab/Learning-Implicitly-From-Spatial-Transformers-Network.
  * [16] Gidi Littwin and Lior Wolf.  Deep meta functionals for shape representation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1824–1833, 2019.
  * [17] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li.  Learning to infer implicit surfaces without 3d supervision.  In Proceedings of the Advances in Neural Information Processing Systems, volume 32, 2019.
  * [18] H Christopher Longuet-Higgins.  A computer algorithm for reconstructing a scene from two projections.  Nature, 293(5828):133–135, 1981.
  * [19] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  ACM Siggraph Computer Graphics, 21(4):163–169, 1987.
  * [20] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  * [21] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani.  Autosdf: Shape priors for 3d completion, reconstruction and generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 306–315, 2022.
  * [22] Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang.  Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 55–64, 2020.
  * [23] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504–3515, 2020.
  * [24] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia.  Deep mesh reconstruction from single rgb images via topology modification networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9964–9973, 2019.
  * [25] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165–174, 2019.
  * [26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Proceedings of the Advances in Neural Information Processing Systems, volume 32, pages 8024–8035, 2019.
  * [27] Scott D Roth.  Ray casting for modeling solids.  Computer Graphics and Image Processing, 18(2):109–144, 1982.
  * [28] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2304–2314, 2019.
  * [29] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 84–93, 2020.
  * [30] Muhamad Risqi U Saputra, Andrew Markham, and Niki Trigoni.  Visual slam and structure from motion in dynamic environments: A survey.  ACM Computing Surveys, 51(2):1–36, 2018.
  * [31] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4104–4113, 2016.
  * [32] Dong Wook Shu, Sung Woo Park, and Junseok Kwon.  3d point cloud generative adversarial network based on tree structured graph convolutions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3859–3868, 2019.
  * [33] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437–2446, 2019.
  * [34] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman.  Pix3d: Dataset and methods for single-image 3d shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2974–2983, 2018.
  * [35] Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3405–3414, 2019.
  * [36] Shimon Ullman.  The interpretation of structure from motion.  Proceedings of the Royal Society of London. Series B. Biological Sciences, 203(1153):405–426, 1979.
  * [37] Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen.  Pq-net: A generative part seq2seq network for 3d shapes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 829–838, 2020.
  * [38] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann.  Disn: Deep implicit surface network for high-quality single-view 3d reconstruction.  In Proceedings of the Advances in Neural Information Processing Systems, volume 32, 2019.
  * [39] Cheng Zhang, Zhaopeng Cui, Yinda Zhang, Bing Zeng, Marc Pollefeys, and Shuaicheng Liu.  Holistic 3d scene understanding from a single image with implicit representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8833–8842, 2021.
  * [40] Fang Zhao, Wenhao Wang, Shengcai Liao, and Ling Shao.  Learning anchored unsigned distance functions with gradient direction alignment for single-view garment reconstruction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12674–12683, 2021.

## Supplementary Material

In Fig. 8, we show a qualitative comparison of occluded surface
reconstruction. Examples of failed reconstructions are displayed in Fig. 9.
More qualitative comparisons between LIST and the baseline models using the
ShapeNet dataset are highlighted in Fig. 10. The results of LIST
reconstructions using distinct views of the same object are provided in Fig.
11, Fig. 12, and Fig. 13.

Fig. 8: A qualitative comparison between LIST and the baseline models on
occluded surface reconstruction using the ShapeNet dataset. GT denotes the
ground-truth objects. Fig. 9: Examples of failed LIST reconstructions.

##  1 Evaluation Metrics

Chamfer Distance (CD): The chamfer distance (CD) between two meshes is defined
as

$\text{CD}(y_{\text{GT}},y_{\text{pred}})=\sum_{a\in y_{pred}}\min_{b\in
y_{gt}}{||a-b||}+\sum_{b\in y_{gt}}\min_{b\in y_{pred}}{||b-a||},$ (12)

where, $y_{GT}$ and $y_{pred}$ are two point clouds extracted from the surface
of the ground-truth and reconstructed object, respectively.

Intersection over Union (IoU): The volumetric intersection over union (IoU) is
defined as the quotient of the volume of the intersection of two meshes and
the volume of their union,

$\text{IoU}(\mathcal{M}_{\text{pred}},\mathcal{M}_{\text{GT}})=\frac{|\mathcal{M}_{\text{pred}}\cap\mathcal{M}_{\text{GT}}|}{|\mathcal{M}_{\text{pred}}\cup\mathcal{M}_{\text{GT}}|}.$
(13)

F-score: The F-score, proposed in [35] as a comprehensive scoring metric for
single-view reconstruction, combines precision and recall to quantify the
overall reconstruction quality. Concretely, the F-score at a distance
threshold $d$ is given by

$F(d)=\frac{2\cdot P(d)\cdot R(d)}{P(d)+R(d)},$

where $P(\cdot)$ and $R(\cdot)$ represents the precision and recall,
respectively. Precision quantifies the accuracy while recall assesses the
completeness of the reconstruction. For the ground-truth $y_{gt}$ and
reconstructed point cloud $y_{pred}$, the precision of an outcome at $d$ can
be calculated as

$P(d)=\sum_{i\in y_{\text{pred}}}[\min_{j\in y_{\text{GT}}}||i-j||<d].$

Similarly, the recall for a given $d$ may be computed as

$R(d)=\sum_{j\in y_{\text{GT}}}[\min_{i\in y_{\text{pred}}}||j-i||<d].$

To evaluate the reconstructions between LIST and the baselines we used
$d=1\%$.

##  2 Data Preparation

To prepare the ground truth, first the target shape was normalized into a unit
cube and 50k points were sampled from the surface of the object. The query
points were prepared by adding random Gaussian noise ($n$) to the surface
points. Specifically,

$Q_{j}=Q_{S}+n~{}|~{}n\in\mathcal{N}(0,P),$ (14)

where $Q_{S}$ are the sampled points and $P\in\mathbb{R}^{3\times 3}$ is a
diagonal covariance matrix with entries $P_{i,i}=\rho$. We empirically found
that $45\%$ of the points at $\rho=0.003$, $44\%$ of the points at
$\rho=0.01$, and $10\%$ of the points at $\rho=0.07$ achieved the best
results.

##  3 Implementation, Training, and Inference Details

###  3.1 Implementation Overview

LIST was implemented using the PyTorch [26] library. To optimize the model,
the Adam [12] optimizer was used with coefficients $(0.9,0.99)$, learning rate
$10^{-4}$, and weight decay $10^{-5}$. A pretrained ResNet [9] was employed as
the image encoder in $\Omega$ and $\Pi$. We closely followed the generator in
[32] to implement the coarse predictor in $\Omega$ with tree-structured
convolutions. However, we empirically found that the degree values
$(2,2,2,2,2,2,64)$ provided a better coarse estimation in our settings. We set
the coarse point cloud density to $N=4000$, and the occupancy grid resolution
to $M=128$. To generate a probabilistic occupancy with the same grid, we
utilized a shallow convolutional network $\Gamma$.

We define $\Xi$ as a convolutional neural network to map the probabilistic
occupancy grid into a high-dimensional latent space. To extract the global
query features and localize the query points, we used a fully-connected neural
network $\Theta$. The global image features are fused with the global query
features on the 3rd layer of $\Theta$. During training, we augment the images
with random color jitter, and normalize the values to $[0,1]$. To improve the
estimation accuracy, we scale the ground-truth and predicted SDF values by
$10.0$. Following [3], we disentangled the query points by scaling with $2.0$
and swapping the 1st and 3rd axis to extract query features from the coarse
prediction. At test time, we extract the query points from a grid in the range
$[-0.5,0.5]$ with resolution $128^{3}$.

##  4 Training and Inference Time

To train LIST it takes $\approx$ 1 s to make a forward pass on an Intel i7
machine with an NVIDIA GeForce GTX 1080Ti GPU. To fully pass through the Pix3D
and ShapeNet datasets, it takes approximately 35 and 50 min, respectively. Our
training process involved using 4 1080Ti GPUs for 100 epochs with a batch size
of 8. To reconstruct the mesh of a single object from a corresponding RGB
image, it takes $\approx$ 7 s on average at a grid resolution of $128^{3}$.

Fig. 10: A qualitative comparison between LIST and the baseline models using
the ShapeNet dataset. Our model recovers significantly better topological and
geometric structure, and the reconstruction is not tainted by the input-view
direction. GT denotes the ground-truth objects. Fig. 11: Qualitative results
of LIST reconstructions using distinct views of the same object. Odd rows
represent the input and even rows represent the reconstructions. Fig. 12:
Qualitative results of LIST reconstructions using distinct views of the same
object. Odd rows represent the input and even rows represent the
reconstructions. Fig. 13: Qualitative results of LIST reconstructions using
distinct views of the same object. Odd rows represent the input and even rows
represent the reconstructions.


# Binary Classification: Counterbalancing Class Imbalance by Applying
Regression Models in Combination with One-Sided Label Shifts

Peter Bellmann, Heinke Hihn  , Daniel Braun  and Friedhelm Schwenker
Institute of Neural Information Processing, Ulm University, James-Franck-Ring,
89081 Ulm, Germany
{peter.bellmann, heinke.hihn, daniel.braun<EMAIL_ADDRESS>
https://orcid.org/0000-0002-3244-3661 https://orcid.org/0000-0002-8637-6652
https://orcid.org/0000-0001-5118-0812

###### Abstract

In many real-world pattern recognition scenarios, such as in medical
applications, the corresponding classification tasks can be of an imbalanced
nature. In the current study, we focus on binary, imbalanced classification
tasks, i.e. binary classification tasks in which one of the two classes is
under-represented (minority class) in comparison to the other class (majority
class). In the literature, many different approaches have been proposed, such
as under- or oversampling, to counter class imbalance. In the current work, we
introduce a novel method, which addresses the issues of class imbalance. To
this end, we first transfer the binary classification task to an equivalent
regression task. Subsequently, we generate a set of negative and positive
target labels, such that the corresponding regression task becomes balanced,
with respect to the redefined target label set. We evaluate our approach on a
number of publicly available data sets in combination with Support Vector
Machines. Moreover, we compare our proposed method to one of the most popular
oversampling techniques (SMOTE). Based on the detailed discussion of the
presented outcomes of our experimental evaluation, we provide promising ideas
for future research directions.

##  1 INTRODUCTION

Imbalanced data sets, i.e. data sets including classes that are not
represented approximately equal [Chawla, 2005], occur in many areas of machine
learning research, such as in fraud detection [Jurgovsky et al., 2018],
outlier detection [Aggarwal, 2015], and medical applications [Fakoor et al.,
2013]. Imbalanced data sets can lead to a significant loss of performance in
most standard classifier learning algorithms, as they assume a balanced class
distribution.
In the past, a great effort has been made to address this issue [Sun et al.,
2009]. One can broadly identify the following general solution paradigms
specific to class imbalance [Sun et al., 2009]: 1) data-level approaches
include different forms of resampling [Xiaolong et al., 2019] and synthetic
data generation [Chawla et al., 2002] to represent the classes equally; 2)
algorithm-level approaches, where the idea is to choose an appropriate
inductive bias, e.g., adaptive penalties [Lin et al., 2002] and to adjust the
decision boundary [Wu and Chang, 2003] of Support Vector Machines [Vapnik,
2013]; 3) cost-level approaches adapt the misclassification costs to reflect
the imbalance; and 4) boosting approaches [Galar et al., 2011], which are
common in multi-classifier systems [Hihn and Braun, 2020, Bellmann et al.,
2018, Kuncheva, 2014]. A recently proposed example for an algorithm-level
approach is the Pattern-Based Classifier for Class Imbalance Problems
(PBC4cip) introduced by Loyola-González et al. [Loyola-González et al., 2017].
A pattern is a set of relational statements that describe objects. A contrast
pattern is a pattern that appears significantly more often in a class than it
does in the remaining classes. The idea of the PBC4cip approach is to weight
the sum of support for the patterns in each class by taking the class
imbalance level into account.
In this work, we focus on Support Vector Machines (SVMs) and propose a novel
approach, which best fits to the algorithm-level category, to tackle the
difficulties introduced by imbalanced data sets. Note that classification SVMs
can be reformulated as regression SVMs. Based on this property, we derive a
label transformation technique that maps binary labels, $\\{-1,+1\\}$, to
_random_ unique positive and negative regression targets, respectively. We
bound the target interval to $[-|X^{-}|,|X^{+}|]$, where $|X^{-}|$ denotes the
number of minority class samples and $|X^{+}|$ the number of majority class
samples. Moreover, we scale the targets such that the corresponding interval
becomes symmetric. In effect, this symmetric target range alleviates some of
the problems caused by the data sets’ class imbalance, as we will show
empirically.
This work is structured as follows. In Sec. 2, we shortly discuss some related
work. We introduce our approach in Sec. 3. In Sec. 4, we briefly describe the
data sets. We present and discuss the results in Sec. 5.

##  2 RELATED WORK

Most likely, data-level techniques constitute the most popular approaches to
deal with class imbalance, which can be further sub-categorised in
undersampling and oversampling. The general undersampling approach is a
straightforward approach for ensemble-based classification models, i.e. for
multi-classifier systems (MCSs) which consist of a set of so-called base
classifiers. In an MCS, each base classifier is trained on a subset of the
initial training set. Therefore, to overcome the issue of class imbalance, in
combination with an MCS, one can train each ensemble member based on a
balanced subset of the training set, by undersampling the corresponding
majority class of the current classification task.
In contrast, one can also use some kind of oversampling techniques, which lead
to a balanced class distribution. Thereby, one straightforward approach is the
generation of additional artificial data of the minority class, for example,
by simply adding noise to the initial minority class samples. Depending on the
classification task, one can also apply other data augmentation techniques,
such as rotations or shifts, for instance, in image-based classification.
One popular oversampling approach is the Synthetic Minority Over-sampling
Technique (SMOTE) proposed by Chawla et al. [Chawla et al., 2002]. The SMOTE
method can be briefly summarised as follows. Assume that we want to generate a
new minority class data point based on the data sample
$x\in\mathbb{R}^{d},d\in\mathbb{N}$. In the first step, one has to determine
the $k$ nearest neighbours to $x$, from the set of minority class samples,
based on a predefined value for $k\in\mathbb{N}$, as well as on a predefined
distance function. Second, one of the $k$ neighbours to $x$ is chosen
randomly, for combination with $x$. Let the randomly chosen data sample from
the set of nearest neighbours be denoted by $y\in\mathbb{R}^{d}$. Then, a
synthetically generated new data point, $\tilde{x}\in\mathbb{R}^{d}$, is
defined as follows,

$\tilde{x}_{i}=x_{i}+r_{i}\cdot(x_{i}-y_{i}),\quad\forall i=1,\ldots,d,$

whereby $r_{i}\in[0,1]$ is a randomly generated number.

##  3 METHODOLOGY

In the current section, we first provide the main notations. Subsequently, we
briefly describe the functionality of Support Vector Machines, followed by the
introduction of our proposed approach. Finally, we list the performance
measures that will be used in our experimental validation.

###  3.1 Formalisation

Let $X\subset\mathbb{R}^{d}$, $d\in\mathbb{N}$, be a $d$-dimensional data set
which constitutes a binary classification task. Further, let the set of labels
be denoted by $\Omega=\\{-1,+1\\}$. In the current work, we assume that the
classification task $(X,\Omega)$ is imbalanced. Without loss of generality, we
assume that the data samples from the minority and the majority classes are
associated with the class labels $-1$ and $+1$, respectively. Moreover, by
$X^{-}$ and $X^{+}$, we respectively define the set of the minority class and
majority class samples, i.e.

$\displaystyle X^{-}$ $\displaystyle:=\\{x\in X:l(x)=-1\\},$ $\displaystyle
X^{+}$ $\displaystyle:=\\{x\in X:l(x)=+1\\},$

whereby $l(x)$ denotes the label of $x$.

###  3.2 Support Vector Machines

Initially, Support Vector Machines (SVMs) were introduced by Vapnik [Vapnik,
2013] as a classification model that is trained to separate the classes of a
binary classification task. An SVM provides a unique decision boundary, which
is obtained during the training phase by combining the following two
objectives. First, an SVM tries to separate the provided two sample sets.
Second, an SVM maximises the so-called margin. The margin is defined as the
width between the two sample sets, based on the computed hyperplane. The data
samples, which are near to the opposing class, and which are used to define
the corresponding hyperplane are denoted as support vectors. For two linearly
inseparable classes, additional slack variables are introduced, to penalise
the misclassification of training samples, based on their distance to the
current hyperplane.
Moreover, SVM models were modified to learn different regression tasks, e.g.
[Abe, 2005]. In the current work, we will denote classification SVMs and
regression SVMs by cSVM and rSVM, respectively.

###  3.3 Binary Regression Approach

The main idea of our approach is based on the transformation of binary
classification tasks to a specific type of regression tasks, as discussed
below.

####  3.3.1 Basic Idea

As briefly discussed above, SVMs can be used as both classification and
regression models. Therefore, for binary classification tasks, equivalently,
one can train an rSVM in combination with the label set $\Omega=\\{-1,+1\\}$.
Thus, to classify a (test) sample, $z\in\mathbb{R}^{d}$, one simply has to
compute $sgn(rSVM(z))$, with $sgn(x)$ denoting the signum of data sample $x$.

####  3.3.2 Labels Generation

In our experiments, in Section 5, we will show that one can replace the label
set $\Omega=\\{-1,+1\\}$ by a set of random negative ($x\in X^{-}$) and
positive ($x\in X^{+}$) numbers, in combination with rSVM models. Therefore,
during the training phase of rSVM models, we will define the set of target
labels, $\tilde{\Omega}$, as follows,

$\tilde{\Omega}:=\\{-|X^{-}|,\ldots,-1,+1,\ldots,+|X^{+}|\\},$ (1)

whereby $|\cdot|$ denotes the number of elements in the corresponding set.
Thereby, we randomly assign each $x\in X^{-}$ to one of the values from the
label subset $\\{-|X^{-}|,\ldots,-1\\}$, and each $x\in X^{+}$ to one of the
labels from the set $\\{1,\ldots,|X^{+}|\\}$. Note that each element of
$\tilde{\Omega}$ is assigned to exactly one data sample $x\in X$.

####  3.3.3 One-Sided Labels Shift

To overcome the issues caused by the imbalance of the given class
distribution, we can further modify our proposed approach with focus on the
set $\tilde{\Omega}$, and apply a one-sided labels shift as follows.
The training of an rSVM model, in combination with the set $\tilde{\Omega}$,
leads to a specific function, $t:\mathbb{R}^{d}\to\mathbb{R}$. However, during
the training, the target function is bounded to the set $\tilde{\Omega}$, i.e.
to the interval $I:=\left[-|X^{-}|,+|X^{+}|\right]$, with $|X^{-}|<|X^{+}|$.
Therefore, due to the skewness of $I$, which originates from the corresponding
class imbalance, $sgn(t)$ is more likely to lead to the value $+1$. To
overcome this issue, we propose to balance the initial classification task by
balancing the interval $I$, i.e. by shifting the redefined target labels
specific to the minority class to the value $-|X^{+}|$. More precisely, with
$\Delta:=|X^{+}|-|X^{-}|$, by $\tilde{\Omega}_{s}$, we denote the set of
shifted target labels, which we define as follows,

$\displaystyle\tilde{\Omega}_{s}:$
$\displaystyle=\\{-(|X^{-}|+\Delta),\ldots,-(1+\Delta),+1,\ldots,+|X^{+}|\\}$
$\displaystyle=\\{-|X^{+}|,\ldots,-(1+\Delta),+1,\ldots,+|X^{+}|\\}.$ (2)

####  3.3.4 Example & Shift Generalisation

Let us assume that the data set
$X=\\{x_{1},\ldots,x_{15}\\}\subset\mathbb{R}^{d}$ consists of 15 data
samples, with $X^{-}=\\{x_{1},\ldots,x_{5}\\}$ and
$X^{+}=\\{x_{6},\ldots,x_{15}\\}$, i.e. $l(x_{i})=-1,l(x_{j})=+1$, $\forall
i=1,\ldots,5$, $j=6,\ldots,15$. Then, for the training of an rSVM model, we
define the set of labels as

$\tilde{\Omega}:=\\{-5,\ldots,-1,+1,\ldots,+10\\},$

since it holds, $|X^{-}|=5$ and $|X^{+}|=10$. Note that the data samples
specific to the set $X^{-}$ ($X^{+}$) are randomly associated to the negative
(positive) numbers from the set $\tilde{\Omega}$, such that each $x\in X$ has
a unique label, i.e. $l(x_{i})\neq l(x_{j})\;\forall i\neq j$. Moreover,
applying the proposed labels shift leads to the following set of target
labels,

$\tilde{\Omega}_{s}:=\\{-10,\ldots,-6,+1,\ldots,+10\\}.$

In our experiments, we will focus on the recognition of the minority classes.
Therefore, we will analyse the performance of cSVM and rSVM models in
combination with the sets $\tilde{\Omega}$ and $\tilde{\Omega}_{s}$. Moreover,
we will analyse the effects of larger shifts. More precisely, the definition
of the set $\tilde{\Omega}_{s}$ can be generalised by introducing the
parameter $\Delta_{m}$ which we define as follows,

$\Delta_{m}:=m\cdot|X^{+}|-|X^{-}|,$ (3)

with a multiplier $m$, which we will bound to the interval $[1,1.5]$, in our
experiments. We denote the corresponding set of target labels by
$\tilde{\Omega}_{s}^{(m)}$, i.e.

$\tilde{\Omega}_{s}^{(m)}:=\\{-(|X^{-}|+\Delta_{m}),\ldots,-(1+\Delta_{m}),1,\ldots,|X^{+}|\\}.$

The definition of $\tilde{\Omega}_{s}^{(m)}$ constitutes a generalisation of
the set $\tilde{\Omega}_{s}$ $-$ and hence of the set $\tilde{\Omega}$ $-$ for
all $m\geq|X^{-}|/|X^{+}|$. Specifically, it holds,
$\tilde{\Omega}_{s}=\tilde{\Omega}_{s}^{(1)}$. Moreover, for
$m=|X^{-}|/|X^{+}|$, it follows, $\Delta_{m}=0$, and therefore it holds,
$\tilde{\Omega}=\tilde{\Omega}_{s}^{(m)}$.
Let us recall the above example from the current section, i.e. $|X^{-}|=5$,
and $|X^{+}|=10$. We now set the parameter $m$ to the value $2$. Then, it
follows, $\Delta_{2}=2\cdot 10-5=15$. Hence, the corresponding shifted set of
target labels is redefined as follows,

$\tilde{\Omega}_{s}^{(2)}=\\{-20,\ldots,-16,+1,\ldots,+10\\}.$

###  3.4 Performance Measures

Table 1: General confusion table. Rows denote true class labels. Columns denote predicted class labels. $T^{-}$: True minority. $F^{+}$: False majority. $F^{-}$: False minority. $T^{+}$: True majority. Min.: Minority. Maj.: Majority. $\downarrow$ true, $\rightarrow$ predicted | Min. Class | Maj. Class
---|---|---
Minority Class | $T^{-}$ | $F^{+}$
Majority Class | $F^{-}$ | $T^{+}$

There exist different performance measures for imbalanced classification tasks
that are based on the evaluation of so-called confusion matrices, which we
depict in Table 1. Using the entries of the confusion matrix, one can compute
the measures specificity ($spe$), precision ($pre$), and recall ($rec$), as
defined in Table 2. Note that the measure recall is also denoted as
sensitivity. Using the definitions from Tables 1 and 2, in our experiments, we
will compute the geometric mean (G-mean) and the F1-score, which are two
popular performance measures for imbalanced classification tasks, and which
are defined as follows,

$\textnormal{G-mean}=\sqrt{rec\cdot
spe},\quad\textnormal{F1-score}=\frac{2\cdot pre\cdot rec}{pre+rec},$ (4)

whereby $spe$, $pre$, and $rec$ are defined in Table 2.

Table 2: Definition of different measures, based on Table 1. Specificity ($spe$) | Precision ($pre$) | Recall ($rec$)
---|---|---
$\frac{T^{+}}{F^{-}+T^{+}}$ | $\frac{T^{-}}{T^{-}+F^{-}}$ | $\frac{T^{-}}{T^{-}+F^{+}}$

Note that our proposed approach does not fit completely to one of the four
main counterbalancing categories from Section 1. There is no additional
generation of synthetic data points, i.e. the size of the training set is left
unchanged with the initial imbalance, specific to the number of minority and
majority class samples (data-level). Moreover, there is no explicit adaptation
of the cost function (cost-level). Obviously, there is no correlation to any
of the existing boosting-based ensembles (boosting approaches). Therefore, our
proposed method fits best to the algorithm-level category. However, there is
no explicit shift of the decision boundaries obtained by the corresponding SVM
models. Thus, our proposed method seems to represent an additional meta-level
or target-level category, or a specific algorithm-level subcategory.

##  4 DATA SETS

In the current work, we will analyse the following four data sets, which are
all publicly available on the UCI Machine Learning Repository111Hyperlink:
http://archive.ics.uci.edu/ml [Dua and Graff, 2017].
The Arrhythmia data set constitutes a task for the distinction between present
and absent cardiac arrhythmia. Initially, this data set was defined as a
$16$-class classification task, with one normal class and $15$ classes defined
as abnormal, i.e. referring to some kind of cardiac arrhythmia. However, eight
of the $15$ classes include less than ten samples each. Therefore, this data
set is mostly used in a binary classification setting. From the initial $279$
features, we removed five features that included missing values. The provided
features were extracted from different domains, which are denoted by DI, DII,
DIII, AVR, AVL, AVF, as well as V1,$\ldots$,V6. Moreover, the data set also
includes nominal features, such as the age, sex, and height, amongst others.
The UCI repository includes several data sets covering breast cancer
classification tasks. In the current study, we analyse the data set that is
denoted as Breast Cancer Wisconsin (Original), and which we will simply denote
by Breast Cancer. This data set consists of nine ordinal-scaled features,
ranging from $1$ to $10$ each. The features provided are, e.g., the clump
thickness, the uniformity of cell size, and the uniformity of cell shape,
amongst others.
The Heart Disease data set constitutes a binary classification task for the
distinction between the presence and absence of heart diseases, based on
different patients. As explained on the corresponding UCI repository database
download page, initially, this data set consisted of five classes and $76$
features. However, only $13$ of the provided features are publicly available.
The provided features include the patients’ age, sex, as well as resting blood
pressure, amongst others.
The Ionosphere data set is a binary classification task which includes radar
returns from the ionosphere. The classes are denoted by good and bad. The
first class is composed of samples that were showing evidence of some type of
structure in the ionosphere. The latter class is composed of the remaining
samples. All of the provided features are continuous. Technical details
specific to the data set are provided in [Sigillito et al., 1989].
Table 3 summarises the properties of all data sets described above. Note that
we include the Arrhythmia and Heart Disease data sets to additionally evaluate
the effectiveness of our proposed method in combination with slightly
imbalanced classification tasks.

Table 3: Data set properties. #S: Number of samples. #F: Number of features. ($P$%): $P$ percent of the data belong to the minority class. Data Set | #S | #F | Class Distribution
---|---|---|---
Arrhythmia | 452 | 274 | 207 : 245 (46%)
Breast Cancer | 683 | 9 | 239 : 444 (35%)
Heart Disease | 270 | 13 | 120 : 150 (44%)
Ionosphere | 351 | 34 | 126 : 225 (36%)

##  5 RESULTS & DISCUSSION

In the current section, we first provide our experimental settings, followed
by the presentation of our results. Finally, we discuss the obtained outcomes,
and provide some ideas for future research directions.

Table 4: Averaged G-mean and F1-score results in % based on rSVMs. AR: Arrhythmia. BC: Breast Cancer. HD: Heart Disease. IO: Ionosphere. $\Omega$/$\tilde{\Omega}$/$\tilde{\Omega}_{s}^{(1.0)}$: rSVM in combination with the initial, $\Omega=\\{-1,+1\\}$, redefined, $\tilde{\Omega}$, redefined and shifted, $\tilde{\Omega}_{s}^{(1.0)}$, label sets (see Section 3, for the definitions). The best performing method is depicted in bold. The improvement in performance is statistically significant, according to the two-sided Wilcoxon signed-rank test, at a significance level of 5%, with respect to $\Omega$ and $\tilde{\Omega}$, for each data set, based on both performance measures. Standard deviation values are denoted by $\pm$. | AR (100$\times$20 Folds) | BC (100$\times$20 Folds) | HD (100$\times$10 Folds) | IO (100$\times$10 Folds)
---|---|---|---|---
| G-mean | F1-score | G-mean | F1-score | G-mean | F1-score | G-mean | F1-score
$\Omega$ | 62.9 | 58.6 | 94.8 | 93.9 | 79.1 | 76.7 | 76.7 | 73.7
| $\pm 2.6$ | $\pm 3.1$ | $\pm 0.9$ | $\pm 1.0$ | $\pm 2.9$ | $\pm 3.3$ | $\pm 2.8$ | $\pm 3.4$
$\tilde{\Omega}$ | 66.5 | 61.8 | 91.6 | 90.7 | 81.5 | 79.3 | 73.1 | 69.3
| $\pm 2.7$ | $\pm 3.3$ | $\pm 1.2$ | $\pm 1.3$ | $\pm 2.4$ | $\pm 2.8$ | $\pm 2.5$ | $\pm 3.1$
$\tilde{\Omega}_{s}^{(1.0)}$ | $\bf{68.1}$ | $\bf{64.4}$ | $\bf{96.5}$ | $\bf{95.5}$ | $\bf{82.0}$ | $\bf{80.1}$ | $\bf{82.3}$ | $\bf{78.4}$
| $\bf{\pm 2.3}$ | $\bf{\pm 2.8}$ | $\bf{\pm 0.8}$ | $\bf{\pm 1.0}$ | $\bf{\pm 2.2}$ | $\bf{\pm 2.5}$ | $\bf{\pm 1.9}$ | $\bf{\pm 2.4}$
Table 5: Averaged G-mean and F1-score results in %. AR: Arrhythmia. BC: Breast Cancer. HD: Heart Disease. IO: Ionosphere. SMOTE: cSVM based on $\Omega=\\{-1,+1\\}$, in combination with an exactly balanced class distribution (see Section 2, for details). $\tilde{\Omega}_{s}^{(m)}$: rSVM in combination with the redefined and shifted label sets $\tilde{\Omega}_{s}^{(m)}$ (see Section 3, for the definitions). The best performing methods are depicted in bold. An asterisk (∗) indicates a statistically significant difference between the best performing rSVM approach and the cSVM SMOTE method, according to the two-sided Wilcoxon signed-rank test, at a significance level of $5\%$. Standard deviation values are denoted by $\pm$. | AR (100$\times$20 Folds) | BC (100$\times$20 Folds) | HD (100$\times$10 Folds) | IO (100$\times$10 Folds)
---|---|---|---|---
| G-mean | F1-score | G-mean | F1-score | G-mean | F1-score | G-mean | F1-score
SMOTE | 66.5 | 63.8 | 97.1 | 95.8 | ${}^{*}\bf{83.0}$ | ${}^{*}\bf{81.2}$ | ${}^{*}\bf{83.0}$ | ${}^{*}\bf{79.2}$
| $\pm 2.7$ | $\pm 3.0$ | $\pm 0.7$ | $\pm 0.9$ | $\bf{\pm 2.0}$ | $\bf{\pm 2.3}$ | $\bf{\pm 2.2}$ | $\bf{\pm 2.7}$
$\tilde{\Omega}_{s}^{(1.0)}$ | 68.1 | 64.4 | 96.5 | 95.5 | 82.0 | 80.1 | 82.3 | 78.4
| $\pm 2.3$ | $\pm 2.8$ | $\pm 0.8$ | $\pm 1.0$ | $\pm 2.2$ | $\pm 2.5$ | $\pm 1.9$ | $\pm 2.4$
$\tilde{\Omega}_{s}^{(1.1)}$ | 68.6 | 65.5 | 97.1 | 96.0 | 81.9 | 80.1 | 81.6 | 77.2
| $\pm 2.0$ | $\pm 2.3$ | $\pm 0.7$ | $\pm 0.9$ | $\pm 2.3$ | $\pm 2.5$ | $\pm 1.9$ | $\pm 2.4$
$\tilde{\Omega}_{s}^{(1.2)}$ | 68.7 | 66.3 | 97.6 | ${}^{*}\bf{96.4}$ | 81.7 | 80.2 | 81.2 | 76.4
| $\pm 1.9$ | $\pm 2.1$ | $\pm 0.6$ | $\bf{\pm 0.8}$ | $\pm 2.5$ | $\pm 2.6$ | $\pm 2.0$ | $\pm 2.4$
$\tilde{\Omega}_{s}^{(1.3)}$ | ${}^{*}\bf{68.8}$ | 67.0 | ${}^{*}\bf{97.7}$ | $96.4$ | 81.1 | 79.8 | 81.0 | 76.0
| $\bf{\pm 2.2}$ | $\pm 2.3$ | $\bf{\pm 0.6}$ | $\pm 0.9$ | $\pm 2.6$ | $\pm 2.5$ | $\pm 2.0$ | $\pm 2.5$
$\tilde{\Omega}_{s}^{(1.4)}$ | ${}^{*}\bf{68.8}$ | 67.5 | ${}^{*}\bf{97.7}$ | 96.2 | 80.4 | 79.3 | 80.8 | 75.7
| $\bf{\pm 2.2}$ | $\pm 2.2$ | $\bf{\pm 0.6}$ | $\pm 0.9$ | $\pm 2.4$ | $\pm 2.3$ | $\pm 1.9$ | $\pm 2.4$
$\tilde{\Omega}_{s}^{(1.5)}$ | 68.7 | ${}^{*}\bf{67.9}$ | ${}^{*}\bf{97.7}$ | 96.1 | 79.7 | 78.9 | 80.7 | 75.4
| $\pm 2.2$ | $\bf{\pm 2.1}$ | $\bf{\pm 0.6}$ | $\pm 0.9$ | $\pm 2.4$ | $\pm 2.2$ | $\pm 1.9$ | $\pm 2.4$
Table 6: Averaged confusion tables. SMOTE: cSVM based on $\Omega=\\{-1,+1\\}$, in combination with an exactly balanced class distribution. $\tilde{\Omega}_{s}^{(m)}$: rSVM in combination with the redefined and shifted label sets $\tilde{\Omega}_{s}^{(m)}$ (see Section 3, for the definitions). Min: Minority class. Maj: Majority class. Rows denote true class labels. Columns denote predicted class labels. $\downarrow\;\;\,$ true | Arrhythmia | Breast Cancer | Heart Disease | Ionosphere
---|---|---|---|---
$\rightarrow$ predicted | Min | Maj | Min | Maj | Min | Maj | Min | Maj
SMOTE | 132 | 75 | 232 | 7 | 97 | 23 | 95 | 31
74 | 171 | 13 | 431 | 22 | 128 | 18 | 207
$\tilde{\Omega}_{s}^{(1.0)}$ | 126 | 81 | 228 | 11 | 96 | 24 | 92 | 34
56 | 189 | 11 | 433 | 24 | 126 | 17 | 208
$\tilde{\Omega}_{s}^{(1.1)}$ | 132 | 75 | 232 | 7 | 98 | 22 | 93 | 33
63 | 182 | 12 | 432 | 27 | 123 | 23 | 202

###  5.1 Experimental Settings

We are using the MATLAB222https://www.mathworks.com/products/matlab.html
software (version R2019b), in combination with the default parameters for
linear SVM models, i.e. cSVMs and rSVMs with linear kernels. As evaluation
approaches, we apply a $100\times 20$-fold cross validation (CV) for the data
sets Arrhythmia and Breast Cancer. In combination with the data sets Heart
Disease and Ionosphere, we apply a $100\times 10$-fold CV, due to the low
number of data points. Thereby, the class distribution of each fold reflects
the initial class distribution, for each of the data sets, i.e. we apply
stratified CVs. Note that by setting the number of evaluation folds
respectively to $10$ and $20$, we ensure that each of the test folds includes
at least $10$ data samples from the minority class. Moreover, by applying
$100$ iterations for each of the CV evaluations, we ensure a fair comparison
across all analysed models. To test for statistically significant differences
between the implemented models, we will apply the two-sided Wilcoxon signed-
rank test [Wilcoxon, 1945], at a significance level of $5\%$.
Note that in the current study, we focus on SVM models, since they are popular
machine learning tools which are widely used in binary classification task
scenarios. Moreover, we apply the default parameter settings to focus on the
effectiveness of label shifts in particular.
As we will discuss later in detail, in this work, we provide the basic form of
our proposed balancing technique (see Section 5.5). Therefore, for the
comparison to the state-of-the-art, we focus on the SMOTE method, which is a
very intuitive and easily interpretable, yet quite challenging approach.

###  5.2 Initial Regression Experiments

First, we focus on determining the best regression approach, based on the
following three label set variants. We evaluate the rSVM performances based on
the initial label set, $\Omega=\\{-1,+1\\}$, the modified label set,
$\tilde{\Omega}=\\{-|X^{-}|,+|X^{+}|\\}$, as well as the modified and
symmetric label set,
$\tilde{\Omega}^{(1.0)}_{s}=\\{-|X^{+}|,\ldots,-(|X^{+}|-|X^{-}|+1),1,\ldots,|X^{+}|\\}$,
as discussed in Section 3. Table 4 states the corresponding performance values
for all of the four data sets.
From Table 4, we can make the following observations. Based on the Arrhythmia
and Heart Disease data sets, training rSVM models specific to the redefined
label set, $\tilde{\Omega}$, instead of the initial label set, $\Omega$, leads
to an improvement, in combination with both performance measures, G-mean and
F1-score. In contrast, based on the Breast Cancer and Ionosphere data sets,
training rSVM models specific to the redefined label set, $\tilde{\Omega}$,
instead of the initial label set, $\Omega$, leads to a decrease of both
performance measures. However, the training of rSVMs in combination with the
redefined and symmetric label set, $\tilde{\Omega}^{(1.0)}_{s}$, leads to the
best performance values, based on all data sets and both measures. The
improvement in performance was always statistically significant, according to
the two-sided Wilcoxon signed-rank test, at a $5\%$ significance level, with
respect to both approaches, $\Omega$ and $\tilde{\Omega}$.
However, in general, binary classification tasks are not transferred to
regression tasks. Therefore, in the next step, we will compare our proposed
approach to common classification SVM models.

###  5.3 Classification vs. Regression

In the current section, we will analyse the following two research questions.
First, we will evaluate the effect of further shifting the minority class
labels, such that the redefined label sets become imbalanced again, however
over-representing the minority class. Second, we will compare our proposed
method to a regular cSVM model, which is trained in combination with the SMOTE
method. Thereby, we apply the standard SMOTE approach in combination with the
Euclidean distance, and set the number of nearest neighbours to $k=10$ (see
Section 2). Note that we apply the SMOTE method such that the current training
set is always equally distributed, i.e.
$|X^{-}_{\textnormal{train}}|=|X^{+}_{\textnormal{train}}|$, for each
validation fold, and each data set. Table 5 states the obtained results,
including the performance values for $\tilde{\Omega}^{(m)}_{s}$, with
$m=1.1,1.2,1.3,1.4,1.5$.
From Table 5, we can make the following observations. Additionally shifting
the labels, with respect to $\tilde{\Omega}^{(1.0)}_{s}$, increased the
performance specific to both measures, G-mean and F1-score, based on the
Arrhythmia and Breast Cancer data sets. In contrast, based on the Heart
Disease and Ionosphere data sets, further labels shifts, with respect to
$\tilde{\Omega}^{(1.0)}_{s}$, decreased the averaged performance in general,
in combination with both measures. Moreover, our proposed method outperformed
the common cSVM approach, in combination with SMOTE (and the initial binary
label set $\Omega$), based on the Arrhythmia, as well as Breast Cancer data
set, with respect to both performance measures. On the other hand, our
proposed method was outperformed by the cSVM SMOTE approach, for the other two
data sets, also with respect to both performance measures.
Table 6 illustrates the effects of increasing the labels shift, based on the
rounded confusion tables. The tables are averaged with respect to the $100$
repetitions of the $10$-fold and $20$-fold cross validations, specific to the
different data sets, respectively. Comparing the cSVM SMOTE method to the
$\tilde{\Omega}^{(1.0)}_{s}$ based approach, leads to the following
observation, with respect to the recognition of the minority class. The cSVM
SMOTE method slightly outperforms the $\tilde{\Omega}^{(1.0)}_{s}$ based
approach on all of the four data sets. The effect of shifting the minority
class labels is also clearly depicted in Table 6. By increasing $m$ from $1.0$
to $1.1$, we can observe that the recognition of the minority class improved,
while simultaneously the recognition of the majority class decreased, based on
all four data sets. The recognition of the minority class, specific to the
approaches cSVM SMOTE and $\tilde{\Omega}^{(1.1)}_{s}$, is approximately the
same.

###  5.4 Discussion

In the first part of our experiments, we showed that shifting the labels to a
symmetric target interval improves the performance of rSVM models, with
respect to the initial class label set $\Omega$, as well as to the redefined
and asymmetric target label set $\tilde{\Omega}$. Moreover, we showed that
additional one-sided shifts of the labels improve the recognition of the
minority class, while simultaneously decreasing the recognition rate specific
to the majority class. Moreover, the performance based on our proposed
approach is comparable to the performance of cSVM SMOTE models. One main
advantage of our proposed method is that there is no additional generation of
synthetic data. This property is especially beneficial in classification tasks
where the corresponding class imbalance is relatively huge, and the feature
dimension very high. In general, the operational cost, for determining a small
set of nearest neighbours, has to be taken into account in tasks including
high-dimensional data. Moreover, it is also necessary to store the
additionally generated data. Applying our proposed approach avoids both of the
aforementioned issues.
Similar to other counterbalancing methods, it is possible to apply our
proposed method also in multi-class settings, in a straightforward manner.
Therefore, one can simply choose one of the existing divide-and-conquer
methods, such the error-correcting output codes [Dietterich and Bakiri, 1991],
including, e.g., the one-versus-one and one-versus-all approaches. Moreover,
one can also apply our proposed method in form of cascaded classification
architectures [Frank and Hall, 2001], if it is possible to detect an ordinal
class structure in the current classification task, as recently proposed, for
instance, in [Bellmann and Schwenker, 2020, Lausser et al., 2020].

###  5.5 Future Research Directions

Note that in the current study, we introduced the basic form of our proposed
method. Therefore, in the following, we want to provide some promising ideas
for future research directions.
First, one could include the parameter $m$ in the process of classifier
design. As we showed in the previous section, the parameter $m$ has a strong
influence on the recognition rate of the minority class. Depending on the
current classification task and performance measure, one could apply some kind
of optimisation techniques, e.g. grid search, to determine optimal values for
$m$.
Second, note that in the current study, we proposed to assign the labels
randomly, between the new set of labels $\tilde{\Omega}^{(m)}_{s}$ and the
corresponding data samples from the majority (only positive labels) and the
minority (only negative labels) classes. Therefore, we have to further analyse
whether the classification performance can be improved by applying specific
target label assigning approaches. To this end, one could define some kind of
ordering function $f$, $f:\mathbb{R}^{d}\to\mathbb{R}$, and apply $f$
separately to the sets $X^{-}$ and $X^{+}$. Subsequently, one can assign the
target labels according to the sorted values of $f$. More precisely, $x\in
X^{+}$ is assigned to the target label $|X^{+}|$, if $f(x)>f(y)$, $\forall
y\in X^{+}$. Similarly, $x\in X^{+}$ is assigned to the target label $1$, if
$f(x)<f(y)$, $\forall y\in X^{+}$. Analogously, based on the set $X^{-}$, the
data sample specific to the lowest/highest value of $f$ is assigned to the
lowest/highest negative value of the current label set
$\tilde{\Omega}^{(m)}_{s}$. We assume that assigning the target labels
appropriately can lead to a smooth regression target function, which could
lead to a better classification performance. In future work, we want to
evaluate different examples for the ordering function $f$, for instance,

$f(x):=\sum_{i=1}^{d}|x_{i}|,\quad f(x):=\sum_{i=1}^{d}x_{i}^{n},$

e.g., with $n\in\\{1,2\\}$.
Third, our proposed method is not based on any specific property of SVM
models. Similar to SVMs, for instance, decision trees [Breiman et al., 1984]
can also be implemented as both, classification and regression models.
Therefore, we aim to analyse our proposed method in combination with
additional classification - or even plain regression - models. Moreover, we
aim to adapt our approach to deep models, for instance, by scaling the target
labels to the interval $[-1,1]$, to avoid the exploding gradient problem.
Fourth, we aim to analyse our proposed method in combination with ensemble-
based classification systems. Therefore, for instance, one could design an
ensemble in which each base classifier/regressor is trained on 1) different
target label intervals, and/or 2) different assignments between training
samples and target labels. Moreover, one could include different
classification/regression models, or combine our proposed method with some
approaches from the different class imbalance solution categories, which we
briefly discussed in Section 1.
Finally, after adapting our method specific to some or all of the
modifications proposed above, we aim to provide a detailed comparison to
latest state-of-the-art techniques including highly imbalanced data sets.

## ACKNOWLEDGEMENTS

The work of Friedhelm Schwenker and Peter Bellmann is supported by the project
Multimodal recognition of affect over the course of a tutorial learning
experiment (SCHW623/7-1) funded by the German Research Foundation (DFG). The
work of Daniel Braun and Heinke Hihn is supported by the European Research
Council, grant number ERC-StG-2015-ERC, Project ID: 678082, BRISC: Bounded
Rationality in Sensorimotor Coordination. We gratefully acknowledge the
support of NVIDIA Corporation with the donation of the Tesla K40 GPU used for
this research.

## REFERENCES

  * [Abe, 2005] Abe, S. (2005).  Support Vector Machines for Pattern Classification.  Advances in Pattern Recognition. Springer, London, U.K.
  * [Aggarwal, 2015] Aggarwal, C. C. (2015).  Outlier analysis.  In Data mining, pages 237–263. Springer.
  * [Bellmann and Schwenker, 2020] Bellmann, P. and Schwenker, F. (2020).  Ordinal classification: Working definition and detection of ordinal structures.  IEEE Access, 8:164380–164391.
  * [Bellmann et al., 2018] Bellmann, P., Thiam, P., and Schwenker, F. (2018).  Multi-classifier-Systems: Architectures, Algorithms and Applications, pages 83–113.  Springer International Publishing, Cham.
  * [Breiman et al., 1984] Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. (1984).  Classification and Regression Trees.  Wadsworth.
  * [Chawla, 2005] Chawla, N. V. (2005).  Data mining for imbalanced datasets: An overview.  In The Data Mining and Knowledge Discovery Handbook, pages 853–867. Springer.
  * [Chawla et al., 2002] Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. (2002).  Smote: Synthetic minority over-sampling technique.  Journal of Artificial Intelligence Research, pages 321–357.
  * [Dietterich and Bakiri, 1991] Dietterich, T. G. and Bakiri, G. (1991).  Error-correcting output codes: A general method for improving multiclass inductive learning programs.  In AAAI, pages 572–577. AAAI Press / The MIT Press.
  * [Dua and Graff, 2017] Dua, D. and Graff, C. (2017).  UCI machine learning repository.
  * [Fakoor et al., 2013] Fakoor, R., Ladhak, F., Nazi, A., and Huber, M. (2013).  Using deep learning to enhance cancer diagnosis and classification.  In Proceedings of the international conference on machine learning, volume 28. ACM New York, USA.
  * [Frank and Hall, 2001] Frank, E. and Hall, M. A. (2001).  A simple approach to ordinal classification.  In ECML, volume 2167 of Lecture Notes in Computer Science, pages 145–156. Springer.
  * [Galar et al., 2011] Galar, M., Fernandez, A., Barrenechea, E., Bustince, H., and Herrera, F. (2011).  A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches.  IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(4):463–484.
  * [Hihn and Braun, 2020] Hihn, H. and Braun, D. A. (2020).  Specialization in hierarchical learning systems.  Neural Processing Letters, 52:2319–2352.
  * [Jurgovsky et al., 2018] Jurgovsky, J., Granitzer, M., Ziegler, K., Calabretto, S., Portier, P.-E., He-Guelton, L., and Caelen, O. (2018).  Sequence classification for credit-card fraud detection.  Expert Systems with Applications, 100:234–245.
  * [Kuncheva, 2014] Kuncheva, L. I. (2014).  Combining Pattern Classifiers: Methods and Algorithms.  John Wiley & Sons.
  * [Lausser et al., 2020] Lausser, L., Schäfer, L. M., Kühlwein, S. D., Kestler, A. M. R., and Kestler, H. A. (2020).  Detecting ordinal subcascades.  Neural Process Lett, 52:2583–2605.
  * [Lin et al., 2002] Lin, Y., Lee, Y., and Wahba, G. (2002).  Support vector machines for classification in nonstandard situations.  Machine learning, 46(1-3):191–202.
  * [Loyola-González et al., 2017] Loyola-González, O., Medina-Pérez, M. A., Martínez-Trinidad, J. F., Carrasco-Ochoa, J. A., Monroy, R., and García-Borroto, M. (2017).  Pbc4cip: A new contrast pattern-based classifier for class imbalance problems.  Knowl. Based Syst., 115:100–109.
  * [Sigillito et al., 1989] Sigillito, V. G., Wing, S. P., Hutton, L. V., and Baker, K. B. (1989).  Classification of radar returns from the ionosphere using neural networks.  Johns Hopkins APL Technical Digest, 10(3):262–266.
  * [Sun et al., 2009] Sun, Y., Wong, A. K., and Kamel, M. S. (2009).  Classification of imbalanced data: A review.  International journal of pattern recognition and artificial intelligence, 23(04):687–719.
  * [Vapnik, 2013] Vapnik, V. (2013).  The nature of statistical learning theory.  Springer science & business media.
  * [Wilcoxon, 1945] Wilcoxon, F. (1945).  Individual comparisons by ranking methods.  Biometrics Bulletin, 1(6):80–83.
  * [Wu and Chang, 2003] Wu, G. and Chang, E. Y. (2003).  Class-boundary alignment for imbalanced dataset learning.  In ICML 2003 workshop on learning from imbalanced data sets II, Washington, DC, pages 49–56.
  * [Xiaolong et al., 2019] Xiaolong, X., Wen, C., and Yanfei, S. (2019).  Over-sampling algorithm for imbalanced data classification.  Journal of Systems Engineering and Electronics, 30(6):1182–1191.