# ACL-SPC: Adaptive Closed-Loop system for
Self-Supervised Point Cloud Completion

Sangmin Hong1 Mohsen Yavartanoo2††footnotemark:  Reyhaneh Neshatavar2 Kyoung
Mu Lee1,2
1IPAI, 2Dept. of ECE & ASRI, Seoul National University, Seoul, Korea
<EMAIL_ADDRESS>equal contribution

###### Abstract

Point cloud completion addresses filling in the missing parts of a partial
point cloud obtained from depth sensors and generating a complete point cloud.
Although there has been steep progress in the supervised methods on the
synthetic point cloud completion task, it is hardly applicable in real-world
scenarios due to the domain gap between the synthetic and real-world datasets
or the requirement of prior information. To overcome these limitations, we
propose a novel self-supervised framework ACL-SPC for point cloud completion
to train and test on the same data. ACL-SPC takes a single partial input and
attempts to output the complete point cloud using an adaptive closed-loop
(ACL) system that enforces the output same for the variation of an input. We
evaluate our ACL-SPC on various datasets to prove that it can successfully
learn to complete a partial point cloud as the first self-supervised scheme.
Results show that our method is comparable with unsupervised methods and
achieves superior performance on the real-world dataset compared to the
supervised methods trained on the synthetic dataset. Extensive experiments
justify the necessity of self-supervised learning and the effectiveness of our
proposed method for the real-world point cloud completion task. The code is
publicly available from this link.

##  1 Introduction

Figure 1:  Overview of our proposed pipeline. We first generate $C_{0}$ using
the initial partial point cloud. Then, multiple synthetic point clouds $P_{v}$
are generated from the random views of $C_{0}$. We input the generated $P_{v}$
to the network and make predicted complete point clouds. We take the loss
between $C_{0}$ and $C_{v}$ to optimize the parameters of the network
$f_{\theta}$.

Along with the development of autonomous driving cars and robotics, the usage
of depth sensors such as LiDARs has increased. These sensors can collect
numerous points in the 3D space, and the combination of these points forms a
3D representation called a point cloud. Point cloud representation has been
widely used in many applications as it is highly convertible to other 3D data
representations, _e.g_., voxel and mesh, and accessible for obtaining
information from the real world. However, point clouds obtained from a real-
world sensor, _e.g_., a LiDAR, are often incomplete and sparse due to
occlusion, limitations of sensor resolution, and viewing angle [49] leading to
loss of some geometric information and difficulty in proceeding with further
applications _e.g_., object detection [26] and object segmentation [7]. We
define such point clouds as partial point clouds. Therefore, point cloud
completion is a crucial task that infers completing geometric 3D shapes by
using such partial point cloud observations.

With the advent of deep learning, previous data-driven works [49, 40, 43] have
been able to solve this task using complete point cloud ground-truths. Even
though such methods have achieved decent performance, they are not applicable
in real-world scenarios where the ground-truth point clouds are not easy to
obtain. For these reasons, researchers have recently attempted to overcome the
lack of high-quality and large-scale paired training data using multiple views
of the point cloud in unsupervised and weakly-supervised manners. Especially,
recent methods [15, 21] leverage multi-view consistency of the desired object,
which shows effectiveness in supervising 3D shape prediction. PointPnCNet [21]
claims that its method is based on self-supervised learning. However,
combining multi-view consistency enables reconstructing a complete 3D point
cloud and can be weak supervision. Moreover, collecting multiple partial views
of an object in real-world scenarios is difficult as gathering ground-truth
point clouds. Therefore, the necessity for multi-view consistency prevents
this method from being fully self-supervised. Meanwhile, other methods [6, 50,
41, 13] exploit unpaired partial and complete point clouds [6, 41] or pre-
trained models [50, 13] on synthetic data to overcome the difficulty of
collecting ground-truth. However, the need for unpaired data limits the
methods’ applicability to a few categories.

To overcome the challenges mentioned above, we propose a novel and the first
self-supervised method called ACL-SPC for point cloud completion using only a
single partial point cloud. We develop an adaptive closed-loop (ACL) [2]
system as shown in Figure 1 to design our self-supervised point cloud
completion framework ACL-SPC. In ACL-SPC, an encoder adaptively reacts to the
variance in the input by adjusting its parameters to generate the same output.
Using our developed ACL, our method tries to generate a complete point cloud
from a single partial input captured from an unknown viewpoint without any
prior information or multi-view consistency and also simulates several
synthetic partial point clouds from the reconstructed point cloud. Under our
defined novel loss function, our ACL-SPC can learn to generate the same
complete point cloud from all such synthetic point clouds and the initial
partial point cloud without any supervision. In the experiments, we
demonstrate the ability of our method to restore a complete point cloud and
the effect of our designed loss functions on saving fine details and improving
quantitative performance. We also evaluate our method with various datasets,
including real-world scenarios, and verify that our method can be applied in
practice. Evaluation results show that our method is comparable to other
unsupervised methods and performs better than the supervised method trained on
a synthetic dataset.

Our main contributions can be summarized as follows:

  * •

We propose ACL-SPC by developing an adaptive control-loop ACL framework to
solve the point cloud completion problem in a self-supervised manner.

  * •

We also design an effective self-supervised loss function to train our method
without requiring any other information and using only a single partial point
cloud taken from an unknown viewpoint.

  * •

Our method achieves superior performance in real-world scenarios compared to
methods trained on synthetic datasets and comparative performance among other
unsupervised methods.

##  2 Related Works

###  2.1 Supervised point cloud completion

Point cloud completion is the task of reconstructing a complete geometry of a
shape from partial point clouds. Before the advancement of the deep neural
network, some traditional geometric-based methods [22, 32, 9] have been
attempted to complete shapes using the geometric priors from a partial input
without any external data. Other methods [20, 29, 30, 36, 38] have been
proposed to handle the point cloud completion task by utilizing the symmetry
property of the object to complete the incomplete parts.

With the development of deep learning, some learning-based methods [9, 23, 34]
have shown up to solve complete the partial point cloud using a large amount
of data. However, these methods have converted a partial point cloud into
voxels to apply convolutional neural networks (CNNs), which leads to
computational complexity and losing some geometrical information of point
clouds. PCN [49] as the first data-driven approach learned a completion
network directly from point clouds rather than converting to other
representations. Further, various works have proposed developed architectures
using a novel rooted tree structure [37], 3D grids as the intermediate
representation [43], feedback refinement module [46], and transformers [48,
52] to improve the performance. However, these methods deviate from real-world
scenarios, as gathering ground truth point clouds is cost-inefficient and not
practical.

Figure 2:  The framework of ACL-SPC. Our framework consists of an encoder-
decoder style network where the parameters are shared between the objects. The
network adopts the PolyNet [47] as the encoder and three fully connected (FC)
layers as the decoder. Our network first takes the input partial point cloud
and generates an estimated complete point cloud. Using this point cloud, we
make multiple synthesized partial point clouds as new inputs. Again, the
network outputs estimated complete point clouds from the synthesized partial
point clouds. We apply consistency loss between multiple estimated complete
point clouds and optimize the parameters of the network.

###  2.2 Unsupervised point cloud completion

Due to the aforementioned limitations of supervised methods for point cloud
completion, existing unsupervised approaches [15, 21, 16] have been proposed
to handle point cloud completion tasks where ground truth data are
unavailable. Meanwhile, weakly supervised methods [15, 16] have attempted to
predict the complete point cloud using multiple partial views, which are not
always available in real-world scenarios. Later, PointPnCNet [21] introduced
an inpainting framework with geometric consistency to overcome the above issue
and claim that the method is the first self-supervised work for this task.
Nevertheless, this method has exploited geometric consistency between multi-
views of an object and showed that without this supervision, they could not
complete the partial point cloud. Moreover, there have been attempts to
utilize unpaired complete point clouds from synthetic datasets to solve the
difficulty of accessing ground-truths [6, 50, 41, 13]. As a domain gap exists
between unpaired complete point clouds and partial point clouds, these methods
design architectures that can transform from one domain to the other and
eventually solve the issue. However, these methods can only be suitable for
categories available in synthetic datasets.

###  2.3 Self-supervised Learning

Self-supervised learning has attracted increasing attention in computer vision
due to its practicality and ability to avoid the need for expensive annotated
datasets. Following the advances in CNNs, recent self-supervised learning
methods have incorporated generative [14, 54, 17, 39, 31, 18] and contrastive
approaches [44, 1, 10], to learn the features from unlabeled data where an
input itself provides supervision. Furthermore, researchers [33, 35, 11, 51,
53, 45, 27, 19] have started to apply self-supervised methods on point clouds
to overcome the cumbersome task of annotating. These works have successfully
shown great performance on feature learning to handle tasks such as
classification [33, 35, 11, 53, 45, 27, 19], segmentation [33, 35, 11, 53, 45,
27, 19] or upsampling [51]. In this way, we propose the first self-supervised
method for point cloud completion using only a partial point cloud as input
without any prior information.

##  3 Method

In control theory [24, 25], closed-loop systems have many applications in
various areas such as aerospace, electronics, and biomedical. Especially
adaptive closed-loop (ACL) is a system where a controller automatically gives
a compensated signal for the variation in the system so that the overall
result remains the same [2, 24, 25]. In an ACL system, a controller outputs an
appropriate signal after receiving feedback from the error between the desired
output and the generated one. Meanwhile, obtaining a complete point cloud
generator invariant to the view of captured partial point clouds is essential
for point cloud completion tasks. We believe that the aforementioned attribute
of the ACL system can be utilized for this task because it is appropriate for
constructing the same complete point cloud, whatever a partial point cloud of
an object comes in as an input. Therefore, we develop the concept of ACL for
point cloud completion and introduce a novel self-supervised partial point-
cloud completion framework (ACL-SPC).

Supervision | Method |  | Airplane |  | Car |  | Chair |  | Average |
---|---|---|---|---|---|---|---|---|---|---
|  |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$ |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$ |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$ |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$ |
|  DPC [16] |  |  - |  - |  3.91 |  |  - |  - |  3.47 |  |  - |  - |  4.30 |  |  - |  - |  3.89 |
Unsupervised |  Gu  _et al_. [15] |  |  0.91 |  1.05 |  1.95 |  |  1.27 |  1.41 |  2.68 |  |  1.69 |  1.64 |  3.33 |  |  1.29 |  1.36 |  2.65 |
|  PointPnCNet [21] |  |  1.58 |  1.74 |  3.32 |  |  1.98 |  2.98 |  4.96 |  |  2.72 |  2.68 |  5.40 |  |  1.75 |  2.46 |  4.56 |
Self-supervised | Ours |  |  1.20 |  0.80 |  2.01 |  |  1.65 |  1.28 |  2.93 |  |  2.25 |  1.46 |  3.71 |  |  1.70 |  1.18 |  2.88 |
Table 1:  Quantitative results on three categories airplane, car and chair. We
also calculate average values among the categories. P, C, and CD refers to
precision, coverage, and Chamfer distance, respectively. All the values are
multiplied by $100$.

(a) Input

(b) GT

(c) Multi-View

(d) GRNet[43]

(e) Gu  _et al_. [15]

(f) Ours

Figure 3:  Qualitative comparison on the ShapeNet dataset. We visualize a) the
partial input, b) completed ground truth point cloud, c) multi-view point
cloud, results on d) GRNet, e) Gu  _et al_., and f) ours. The multi-view point
cloud is the concatenation of five random partial views of an object. Our
result show that our method can recover most of the missing parts from the
partial input.

Supervision | Method |  P |  C |  CD
---|---|---|---|---
Supervised |  GRNet [43] |  4.63 |  6.90 |  11.53
SFNet [42] |  14.12 |  12.64 |  26.76
pcn [49] |  9.83 |  17.96 |  27.79
Unsupervised |  Gu [15] |  8.70 |  10.70 |  19.40
PointPnCNet [21] |  9.00 |  10.00 |  19.00
Self-supervised | Ours |  11.67 |  5.63 |  17.30
(a)

---
Input GT GRNet[43] SFNet [42] pcn [49] Ours
(b)

Figure 4: Evaluation on the SemanticKITTI [3] dataset. We compare our results
with various supervised and unsupervised methods by a) evaluating our
quantitative results in terms of precision (P), coverage (C), and Chamfer
distance (CD) and b) visualizing their outputs. For supervised methods, we use
their pretrained models on the synthetic PCN [49] dataset to evaluate on the
SemanticKITTI.

###  3.1 ACL-SPC

Using a conventional ACL as a point cloud completion system requires the
target complete the point cloud and several partial point cloud observations
to optimize the system. However, accessing the target complete point cloud and
several partial observations in real-world scenarios is not always possible.
Therefore, we develop the ACL system such that it generates the complete point
cloud using only a single partial observation and without requiring the target
complete point cloud for optimizing the system, as shown in Figure 2. To
achieve this goal, we employ a learnable model $f_{\theta}$ on an input
partial point cloud observation $P_{0}\in\mathbb{R}^{N_{\text{p}}\times 3}$ as
follows:

$C_{0}=f_{\theta}(P_{0}),$ (1)

where $C_{0}\in\mathbb{R}^{N_{\text{c}}\times 3}$ is the generated complete
point cloud and $N_{\text{p}}$ and $N_{\text{c}}$ refers to the number of
points in the input and output point cloud, respectively. Then we apply a
partial point cloud generator $g_{v}$ to generate a set of partial point
clouds $P_{v}$ from the generated point cloud $C_{0}$ as follows:

$\forall v\in\\{v_{i}\\}_{i=1}^{N_{\text{s}}},\;P_{v}=g_{v}(C_{0}),$ (2)

where $v_{i}$ is a random parameter to generate $N_{\text{s}}$ number of
different partial point clouds. We again employ the same model $f_{\theta}$ on
the generated synthetic partial point clouds $P_{v}$ to generate the same
point cloud $C_{0}$ as follows:

$\forall
v\in\\{v_{i}\\}_{i=1}^{N_{\text{s}}},\;C_{v}=f_{\theta}(P_{v})=f_{\theta}(g_{v}(C_{0})),$
(3)

where $C_{v}$ is the predicted complete point cloud for the generated point
cloud $C_{0}$. Then, we optimize the system by a loss function between the
predicted complete point clouds $C_{v}$ and the generated initial complete
point cloud $C_{0}$. Accordingly, our ACL-SPC learns to generate the same
complete point cloud for different partial point cloud observations $P_{v}$
synthesized from the generated $C_{0}$. Since the learnable model $f_{\theta}$
is optimized to map any partial point cloud $P_{v}$ to its corresponding
target complete point cloud $C_{0}$, the generated point cloud $C_{0}$ as the
output of $f_{\theta}$ on the input partial point cloud $P_{0}$ must predict
the target complete point cloud.

###  3.2 Loss functions

To train our network $f_{\theta}$, we use two self-supervised loss functions.
First, to optimize the ACL-SPC and guarantee to generate the same predicted
complete point clouds, we design the consistency loss function
$\mathcal{L}^{\text{cons}}$ between the predicted complete point clouds
$C_{v}$ and $C_{0}$ as follows:

$\mathcal{L}^{\text{cons}}=\frac{1}{N_{\text{c}}\times
N_{\text{s}}}\sum_{v\in\\{v_{i}\\}_{i=1}^{N_{s}}}||C_{v}-C_{0}||^{2}_{2},$ (4)

where $||.||_{2}$ represents the $L_{2}$ norm. We further utilize the weighted
Chamfer distance [21] loss $\mathcal{L}^{\text{wcd}}$ between the predicted
complete point cloud $C_{0}$ and the input partial point cloud $P_{0}$. The
weighted Chamfer distance is invariant to the permutation of the order of
points which is composed of two parts with the corresponding weights as
follows:

$\small\mathcal{L}^{\text{wcd}}=\frac{\alpha}{N_{\text{c}}}\sum_{p\in
C_{0}}\min_{q\in P_{0}}||p-q||_{2}+\frac{\beta}{{N_{\text{p}}}}\sum_{q\in
P_{0}}\min_{p\in C_{0}}||q-p||_{2}.$ (5)

The first term measures the mean distance for each point in the source point
cloud $C_{0}$ to the closest point in the target point cloud $P_{0}$, while
the second term measures the mean distance from each point in the target point
cloud $P_{0}$ to its nearest point in the source point cloud $C_{0}$.
Therefore, the second term leads the predicted point cloud $C_{0}$ to cover
the points in the target point cloud $P_{0}$ while the first term performs as
a regularizer. We set $\alpha=0.1$ and $\beta=0.9$ to enforce the points to
cover the non-missing parts of the point cloud and let the remaining points be
flexible to fill in the missing parts. The total loss
$\mathcal{L}^{\text{total}}$ is the weighted summation of two aforementioned
loss functions as follows:

$\mathcal{L}^{\text{total}}=\lambda_{\text{cons}}\mathcal{L}^{\text{cons}}+\mathcal{L}^{\text{wcd}}.$
(6)

###  3.3 Training details

Our model $f_{\theta}$ includes an encoder $\mathcal{E}$ which learns the
local and global features from the partial input point clouds and a decoder
$\mathcal{D}$ to generate the points of the complete point clouds as shown in
Figure 2. We use PolyNet [47], a powerful spatial graph CNN, as the encoder
$\mathcal{E}$ which consists of four squeezed PolyConv layers with the sizes
of $64$, $128$, $256$, and $512$, respectively. We apply a random down-
sampling followed by max-pooling after the first three PolyConv layers to
reduce the point size to $512$, $128$, and $32$, respectively. We employ a
global average pooling after the last PolyConv layer to eliminate the spatial
dependency and obtain $512$ invariant features to various partial observations
and point permutations. We use three fully connected (FC) layers as the
decoder $\mathcal{D}$ with the sizes of $1024$, $1024$, and
${N_{\text{c}}}\times 3$, respectively, where the ReLU non-linear activation
function is applied on the outputs of the first and the second FC layers. In
fact, $g_{v}$ is a non-learnable function that generates the synthetic partial
point clouds by projecting the generated complete point cloud to a depth map
at a random view $v$ from azimuth [$0\degree$, $360\degree$] and elevation
[$-20\degree$, $40\degree$]. We then back-project the depth map into 3D. To
avoid double backpropagation and optimize $f_{\theta}$ only once, we use the
detach operator [28] as shown in Figure 2. We exploit the Adam optimizer and
optimize the model for every $32$ different input partial point clouds and
their $N_{\text{s}}$ synthetic partial point clouds and set $N_{s}=8$ and
$\lambda_{\text{cons}}=10$ in our baseline. For the inference, we feed the
input partial point cloud to the trained model $f_{\theta}$ to directly
generate the corresponding complete point cloud, which takes $12ms$ on average
for each sample with a NVIDIA RTX 2080Ti.

##  4 Experiments

###  4.1 Datasets and metrics

In this section, we discuss the training and evaluation datasets and the
metrics used to compare our proposed method ACL-SPC with the related methods.

Supervision |  |  Method |  |  | ScanNet |  | MatterPort3D |  | KITTI
---|---|---|---|---|---|---|---|---|---
|  |  | Chair | Table |  | Chair | Table |  | Car
|  |  |  UCD$\downarrow$ |  UHD$\downarrow$ |  UCD$\downarrow$ |  UHD$\downarrow$ |  |  UCD$\downarrow$ |  UHD$\downarrow$ |  UCD$\downarrow$ |  UHD$\downarrow$ |  |  UCD$\downarrow$ |  UHD$\downarrow$
Unsupervised |  |  pcl2pcl [6] |  |  |  17.3 |  10.1 |  9.1 |  11.8 |  |  15.9 |  10.5 |  6.0 |  11.8 |  |  9.2 |  14.1
|  ShapeInversion [50] |  |  |  3.2 |  10.1 |  3.3 |  11.9 |  |  3.6 |  10.0 |  3.1 |  11.8 |  |  2.9 |  13.8
|  $+$UHD [50] |  |  |  4.0 |  9.3 |  6.6 |  11.0 |  |  4.5 |  9.5 |  5.7 |  10.7 |  |  5.3 |  12.5
|  Cycle4Comp. [41] |  |  |  5.1 |  6.4 |  3.6 |  5.9 |  |  8.0 |  8.4 |  4.2 |  6.8 |  |  3.3 |  5.8
|  DE [13] |  |  |  2.8 |  5.4 |  2.5 |  5.2 |  |  3.8 |  6.1 |  2.5 |  5.4 |  |  1.8 |  3.5
|  OptDE [13] |  |  |  2.6 |  5.5 |  1.9 |  4.6 |  |  3.0 |  5.5 |  1.9 |  5.3 |  |  1.6 |  3.5
Self-supervised |  |  Ours |  |  |  1.4 |  4.7 |  1.8 |  5.1 |  |  1.8 |  4.8 |  2.1 |  4.9 |  |  2.0 |  4.9
Table 2:  Quantitative results on the real-world datasets [8, 4, 12] in the
categories of chair, table, and car. We evaluate the method in terms of UCD
and UHD where the values are multiplied by $10^{2}$ and $10^{4}$ respectively.

Input

ShapeInv. [50]

OptDE [13]

Ours

Input

ShapeInv. [50]

OptDE [13]

Ours

Figure 5:  Qualitative results on the real-world datasets [8, 4, 12] in the
categories of chair, table, and car.

Synthetic Datasets: ShapeNet [5] is a large-scaled dataset including curated
3D shapes represented by CAD models, which consists of $55$ categories. Among
them, we focus on three categories, airplanes, cars, and chairs, to
maleaaintain the same setup as the previous works [15, 21, 16]. Followed by
these works, we capture RGB-D data from five random views for each object,
transfer them to 3D, and resample $3096$ points of them to generate a set of
partial point clouds. The ground truth complete point clouds with the fixed
$8192$ number of points are used for evaluation.

Real-World Datasets: Similar to previous works [6, 50, 13], we evaluate our
method using three sources of real scans: ScanNet [8] (chairs and tables),
MatterPort3D [4] (chairs and tables), and KITTI [12] (car). The ScanNet and
MatterPort3D datasets are richly annotated 3D reconstructions of indoor
environments, whereas the KITTI dataset is of outdoor scenes. We resample the
points to 2048 points to match the previous work’s settings [6, 50, 13].
SemanticKITTI dataset [3] is derived from the KITTI dataset [12], including
only the car objects which are captured in multi-views from sequence $00$ to
$10$ when parked. We take out the sequence $08$ for testing and exploit the
other sequences for training. Note that the input points are resampled to
$1024$ for convenience. As the SemanticKITTI dataset has no ground truth
complete point cloud, we follow the same steps as in the previous work [15] to
generate them by aggregating partial point clouds.
Metrics. We utilize the Chamfer distance (CD) between the reconstructed point
cloud and the ground truth to evaluate the performance of our ACL-SPC method.
Chamfer distance is the average distance between each point in a point cloud
and the nearest point in the other as follows:

$\begin{split}\mathcal{CD}(C_{0},\text{GT})={1\over{N_{c}}}\sum_{p\in
C_{0}}\min_{q\in GT}||p-q||_{2}\\\ +{1\over{N_{g}}}\sum_{q\in GT}\min_{p\in
C_{0}}||q-p||_{2},\end{split}$ (7)

where GT is defined as the ground truth complete point cloud with $N_{g}$
points. The first and second term refers to precision and coverage,
respectively. Precision infers how much the generated points are distributed
well compared to the ground-truth data, while coverage refers to how much the
missing parts of the partial point cloud are filled in. Accordingly, the
coverage is an important metric for point cloud completion tasks, reflecting
the effectiveness of the methods to fill the missing parts. Additionally, we
utilize two metrics called Unidirectional Chamfer Distance (UCD) and
Unidirectional Hasudorff Distance (UHD) for the real-world datasets [8, 4, 12]
in the same way as previous works [6, 50, 13]. To calculate UCD, we obtain the
first term of $\mathcal{CD}(P_{0},C_{0})$ between the partial input point
cloud $P_{0}$ and the predicted complete point cloud $C_{0}$ using equation 7.
Similarly, we measure the UHD with the single side of Hausdorff distance as
follows:

$\mathcal{UHD}(P_{0},C_{0})=\max_{p\in P_{0}}\min_{q\in C_{0}}||p-q||_{2}$ (8)

Although the two metrics do not reflect the completeness of shape, they can
give a fair comparison where ground-truth data is unavailable.

###  4.2 Evaluation on synthetic dataset

In this section, we qualitatively and quantitatively evaluate our ACL-SPC
method on the ShapeNet dataset and compare the results with the related
methods [15, 21, 16]. We train our network for each category separately with
$1000$ epochs by a learning rate of $0.001$, which is decayed by $0.5$ for
every $200$ epochs to generate $N_{c}=8192$ points. We visualize and compare
the results of our method with the supervised [43] and unsupervised [15]
methods in Figure 3. We note that GRNet [43] and Gu  _et al_. [15] utilize the
GT and the multi-view information as their supervision, respectively. Using
the multi-view information leads to achieving a high-quality appearance for Gu
_et al_. [15] because the concatenated point cloud from five random partial
point clouds is almost as the GT as shown in Figure 3(c). Even without this
information, our ACL-SPC method shows comparable results in completing the
missing parts of the input in a fully self-supervised manner. Moreover, our
quantitative results in Table 1 show that our method can outperform the
unsupervised methods DPC [16] and PointPnCNet [21] with a large gap $1.01$ and
$1.68$ with the CD on average while performing only $0.23$ lower performance
compared to Gu  _et al_. method. Therefore, our method can learn even better
without any prior information compared to some of the unsupervised methods
that have leveraged multiple partial views. Moreover, our method outperforms
all the unsupervised methods by the coverage metric, which shows its
superiority in covering the missing parts.

###  4.3 Evaluation on real-world dataset

We evaluate our self-supervised method ACL-SPC on SemanticKITTI [3] dataset
and compare the results with both supervised and unsupervised methods. We
train our network for $500$ epochs with a learning rate of $0.001$, and it is
decayed by $0.5$ for every $200$ epochs to output $N_{c}=8192$ number of
points. We exploit pretrained models of the supervised methods [43, 42] on the
synthetic PCN [49] dataset to test on the real-world SemanticKITTI dataset. As
shown in Figure LABEL:fig:QuantKITTI, our method outperforms the unsupervised
methods Gu  _et al_. [15] and PointPnCNet [21] in terms of coverage and CD. It
also achieves better coverage compared to the supervised method GRNet [43],
while it shows superior performance in all metrics compared to the supervised
method SFNet [42]. Moreover, we can see through Figure LABEL:tab:QualiKITTI
that the supervised methods perform poorly on the real-world dataset compared
to our method due to the domain gap with the synthetic dataset, which
emphasizes the generalizability of our self-supervised method in real-world
scenarios. Additionally, we can validate that the coverage is more important
than other metrics as our method shows better qualitative results than GRNet
[43] even though our method shows larger precision and CD values.

Furthermore, we quantitatively and qualitatively evaluate our method on
ScanNet [8], MatterPort3d [4], and KITTI [12] dataset as shown in Table 2 and
Figure 5, respectively. In contrast to the unsupervised methods [6, 50, 41,
13] that require synthetic datasets in addition to the real-world datasets for
training because they either need unpaired ground truth [6, 41] or pretrained
model [41, 13], our method is trained only on real-world datasets. Except for
some metrics in the table categories, our method generally performs better
than the state-of-the-art [13] in ScanNet and MatterPort3D datasets as shown
in Table 2. However, in the KITTI dataset, our method is slightly behind the
state-of-the-art by $0.4$ and $1.4$ on UCD and UHD, respectively. We also
qualitatively compare the results with unsupervised methods [50, 13] as shown
in Figure 5. The ShapeInversion [50] generally fails to generate the missing
parts in some cases, such as the chair class of MatterPort3D datasets.
Meanwhile, OptDE [13] generates lots of noise in most samples, especially in
the categories of the ScanNet [8] dataset. In contrast, our method generates
plausible points in the missing parts of the input in all samples. One
drawback of our results is that the output point cloud is not uniformly
distributed, being sparser in the input regions where points were absent.
Thus, even without requiring any synthetic datasets, we illustrate that our
method is competitive compared to other unsupervised methods.

###  4.4 Ablation study

In this section, we further analyze our novel ACL-SPC method by extensive
ablations studies on test-time adaptation, the effect of defined loss
functions, the number of synthesized data, training on the multi-class
dataset, and training on a dataset including only one view of objects.

####  4.4.1 Test-time adaptation

Supervision |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$
---|---|---|---
Supervised |  17.29 |  8.57 |  25.86
Self-supervised |  11.67 |  5.63 |  17.30
Test-time adapt. |  9.62 |  7.09 |  16.71
Table 3:  Evaluation on Test-time adaptation. We train the network in three
modes: supervised, self-supervised, and test-time adaptation. The values are
multiplied by 100.

Loss | Airplane |  | Car |  | Chair
---|---|---|---|---|---
P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$ |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$ |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$
$-\mathcal{L}^{wcd}$ |  1.15 |  18.76 |  19.91 |  |  4.66 |  26.51 |  31.17 |  |  3.91 |  26.41 |  30.32
$-\mathcal{L}^{cons}$ |  1.10 |  1.22 |  2.32 |  |  1.45 |  2.11 |  3.56 |  |  1.99 |  1.74 |  3.73
$\mathcal{L}^{total}$ |  1.20 |  0.81 |  2.01 |  |  1.65 |  1.28 |  2.93 |  |  2.25 |  1.46 |  3.71
(a)

---
Input GT $-\mathcal{L}^{\text{cons}}$ $\mathcal{L}^{\text{total}}$
(b)

Figure 6: Evaluation on the effects of loss functions. We show a) precision (P) and coverage (C) values multiplied by 100 for each experiment with different losses, and b) qualitative results. Ablation | Setup |  | Airplane |  | Car |  | Chair |  | Average
---|---|---|---|---|---|---|---|---|---
|  |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$ |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$ |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$ |  |  P$\downarrow$ |  C$\downarrow$ |  CD$\downarrow$
|  1 |  |  |  1.19 |  0.85 |  2.04 |  |  1.63 |  1.27 |  2.90 |  |  2.37 |  1.45 |  3.82 |  |  1.73 |  1.19 |  2.92
Num Syns |  4 |  |  |  1.23 |  0.82 |  2.05 |  |  1.62 |  1.30 |  2.92 |  |  2.55 |  1.43 |  3.97 |  |  1.80 |  1.18 |  2.98
|  8 |  |  |  1.20 |  0.81 |  2.01 |  |  1.65 |  1.28 |  2.93 |  |  2.25 |  1.46 |  3.71 |  |  1.70 |  1.18 |  2.89
Class |  Single |  |  |  1.20 |  0.81 |  2.01 |  |  1.65 |  1.28 |  2.93 |  |  2.25 |  1.46 |  3.71 |  |  1.70 |  1.18 |  2.89
Multi |  |  |  1.40 |  0.79 |  2.19 |  |  1.66 |  1.25 |  2.91 |  |  2.35 |  1.42 |  3.76 |  |  1.80 |  1.15 |  2.96
Views |  1 |  |  |  1.23 |  0.89 |  2.12 |  |  1.63 |  1.27 |  2.90 |  |  2.15 |  1.54 |  3.69 |  |  1.67 |  1.23 |  2.90
5 |  |  |  1.20 |  0.81 |  2.01 |  |  1.65 |  1.28 |  2.93 |  |  2.25 |  1.46 |  3.71 |  |  1.70 |  1.18 |  2.89
Table 4:  Quantitative effects of the number of synthetic partial views,
single-/multi-class training, and single-/multi-view training. We present the
values of precision (P), coverage (C), and Chamfer distance (CD) multiplied by
100.

Similar to previous works [6, 50, 41, 13], we show that our method can also be
effective for test-time adaptation. We train and test our network in three
different schemes, as shown in Table 3. First, we train the network in the
supervised setting on a synthetic dataset and then evaluate it on a real-world
dataset. Second, we train our network in a self-supervised manner, without any
pretraining, and train and test it on the real-world dataset. Finally, on the
test-adaptation setting, it goes through the pretraining stage and then moves
on to the adaptation stage with our ACL-SPC framework. Table 3 displays the
precision, coverage, and CD of each experiment setting. The results show the
suitability of our ACL-SPC method not only for self-supervised learning but
also for test-time adaption.

####  4.4.2 Effect of each loss

We evaluate the effects of each loss by taking out each loss at a time for
each experiment. We report the quantitative results for the experiments
without $\mathcal{L}^{\text{wcd}}$, without $\mathcal{L}^{\text{cons}}$, and
with the total loss $\mathcal{L}^{\text{total}}$ in Figure 6(a). Taking out
$\mathcal{L}^{wcd}$ affects critically in the results as there is no guarantee
to cover points in the input. On the other hand, excluding
$\mathcal{L}^{\text{cons}}$ results in a worse coverage value which proves the
importance of our proposed ACL-SPC to fill the missing parts of partial input
point clouds. To visualize this effect, we qualitatively compare our results
with and without $\mathcal{L}^{\text{cons}}$ in Figure
LABEL:tab:qual_effect_loss. Without $\mathcal{L}^{\text{cons}}$, only the
input is covered while the missing parts are still uncovered. We note that
since without $\mathcal{L}^{\text{wcd}}$ there is no constraint to generate
points in the location of the input, the method produce all points in the same
position.

####  4.4.3 Number of synthesized data

We analyze how the number of synthesized data can influence point cloud
completion results. Table 4 presents the precision, coverage, and CD values
when $N_{s}$ is set to $1$, $4$, and $8$. According to the results, we achieve
the best performance by $N_{s}=8$ on average among other setups, while there
is no dramatic difference as it shows only $0.03$, $0.01$, and $0.03$
difference compared to having one number of synthesized data. Consequently,
having more synthesized data slightly enhances the performance.

####  4.4.4 Training on multi-class

In our main experiment, we train our model on a specific class and present the
results in section 4.2. However, in real-world scenarios where the classes of
objects are not identified, it is necessary to train the network on multi-
class objects. Table 4 shows the quantitative results of the ShapeNet dataset
when trained with multi-class objects. The results demonstrate not much
difference in the performance as the precision and CD values were $0.10$ and
$0.07$ worse, while the coverage was $0.03$ better in the multi-class
training. Thus, we believe that our network can learn to understand the
appropriate features of a particular object even when there are various
classes in the training set.

####  4.4.5 Single-view training

As mentioned in section 2, recent works [15, 21, 16] leverage multi-partial
views for supervision. Even though our method does not take the advantage of
multi-view supervision, we validate the power of our method to be trained on
only a single view per object. As the training set includes multiple partial
views for the same object, we take out these views and leave only one partial
view to prove that it does not significantly affect our method’s performance.
According to Table 4, our method shows only $0.01$ CD difference with the
model trained with only the single partial view included in the training set.
Through the results, we confirm that our method can still perform as expected
even without multi-views of an object in the training set.

##  5 Conclusion

In this paper, we propose ACL-SPC, the first self-supervised point cloud
completion method from only a single input partial point cloud. Our method
learns to complete partial point clouds by adaptively controlling the output
in a closed-loop system. We also introduce a consistency loss to generate the
same complete point cloud and learn the geometric features of the object. Our
extensive experiments demonstrate that our method can be more useful in real-
world scenarios without performance degradation than other methods. In most
cases, our method shows better performance in the coverage than precision
showing the excellent performance of filling in the missing parts.

Limitations and future works. One remaining limitation of our method is that
there is no constraint to not generate redundant points, which results in high
precision values. To improve the precision and reduce noise, we will apply
more constraints for future works. We will further find applications of our
self-supervised framework in other point cloud restoration tasks such as
denoising and upsampling.

Acknowledgement. This work was supported in part by the IITP grants
[No.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul
National University), No.2022-0-00156, No. 2021-0-02068, and No.2022-0-00156],
and the NRF grant [No. 2021M3A9E4080782] funded by the Korea government
(MSIT).

## References

  * [1] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo.  Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding.  In CVPR2022, 2022.
  * [2] Karl J Åström and Björn Wittenmark.  Adaptive control.  Courier Corporation, 2013.
  * [3] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall.  Semantickitti: A dataset for semantic scene understanding of lidar sequences.  In ICCV, 2019.
  * [4] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  3DV, 2017.
  * [5] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  Shapenet: An information-rich 3d model repository.  CoRR.
  * [6] Xuelin Chen, Baoquan Chen, and Niloy J Mitra.  Unpaired point cloud completion on real scans using adversarial training.  In ICLR, 2020.
  * [7] Xieyuanli Chen, Shijie Li, Benedikt Mersch, Louis Wiesmann, Jürgen Gall, Jens Behley, and Cyrill Stachniss.  Moving object segmentation in 3d lidar data: A learning-based approach exploiting sequential data.  RA-L, 2021.
  * [8] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In CVPR, 2017.
  * [9] Angela Dai, Charles Ruizhongtai Qi, and Matthias Niessner.  Shape completion using 3d-encoder-predictor cnns and shape synthesis.  In CVPR, 2017.
  * [10] Bi’an Du, Xiang Gao, Wei Hu, and Xin Li.  Self-contrastive learning with hard negative sampling for self-supervised point cloud learning.  In Proceedings of the 29th ACM International Conference on Multimedia, 2021.
  * [11] Benjamin Eckart, Wentao Yuan, Chao Liu, and Jan Kautz.  Self-supervised learning on 3d point clouds by learning discrete generative models.  In CVPR.
  * [12] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, 2012.
  * [13] Jingyu Gong, Fengqi Liu, Jiachen Xu, Min Wang, Xin Tan, Zhizhong Zhang, Ran Yi, Haichuan Song, Yuan Xie, and Lizhuang Ma.  Optimization over disentangled encoding: Unsupervised cross-domain point cloud completion via occlusion factor manipulation.  In ECCV, 2022.
  * [14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  NeurIPS, 2014.
  * [15] Jiayuan Gu, Wei-Chiu Ma, Sivabalan Manivasagam, Wenyuan Zeng, Zihao Wang, Yuwen Xiong, Hao Su, and Raquel Urtasun.  Weakly-supervised 3d shape completion in the wild.  In ECCV, 2020.
  * [16] Eldar Insafutdinov and Alexey Dosovitskiy.  Unsupervised learning of shape and pose with differentiable point clouds.  In NeurIPS, 2018.
  * [17] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [18] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim.  Learning to discover cross-domain relations with generative adversarial networks.  In ICML, 2017.
  * [19] Haotian Liu, Mu Cai, and Yong Jae Lee.  Masked discrimination for self-supervised learning on point clouds.  In ECCV, 2022.
  * [20] Niloy J. Mitra, Leonidas J. Guibas, and Mark Pauly.  Partial and approximate symmetry detection for 3d geometry.  ACM, 2006.
  * [21] Himangi Mittal, Brian Okorn, Arpit Jangid, and David Held.  Self-supervised point cloud completion via inpainting.  In BMVC, 2021.
  * [22] Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa.  Laplacian mesh optimization.  ACM.
  * [23] Duc Thanh Nguyen, Binh-Son Hua, Khoi Tran, Quang-Hieu Pham, and Sai-Kit Yeung.  A field model for repairing 3d shapes.  In CVPR, 2016.
  * [24] Norman S. Nise.  Control Systems Engineering.  John Wiley & Sons, Inc., USA, 3rd edition, 2000.
  * [25] Katsuhiko Ogata et al.  Modern control engineering, volume 5.  Prentice hall Upper Saddle River, NJ, 2010.
  * [26] Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin.  Libra r-cnn: Towards balanced learning for object detection.  In CVPR, 2019.
  * [27] Yatian Pang, Wenxiao Wang, Francis E. H. Tay, Wei Liu, Yonghong Tian, and Li Yuan.  Masked autoencoders for point cloud self-supervised learning.  In ECCV, 2022.
  * [28] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * [29] Mark Pauly, Niloy J. Mitra, Johannes Wallner, Helmut Pottmann, and Leonidas J. Guibas.  Discovering structural regularity in 3d geometry.  2008\.
  * [30] Joshua Podolak, Philip Shilane, Aleksey Golovinskiy, Szymon Rusinkiewicz, and Thomas Funkhouser.  A planar-reflective symmetry transform for 3d shapes.  2006\.
  * [31] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In ICML, 2016.
  * [32] Kripasindhu Sarkar, Kiran Varanasi, and Didier Stricker.  Learning quadrangulated patches for 3d shape parameterization and completion.  CoRR, 2017.
  * [33] Jonathan Sauder and Bjarne Sievers.  Self-supervised deep learning on point clouds by reconstructing space.  In NeurIPS.
  * [34] Abhishek Sharma, Oliver Grau, and Mario Fritz.  Vconv-dae: Deep volumetric shape learning without object labels.  In ECCV, 2016.
  * [35] Charu Sharma and Manohar Kaul.  Self-supervised few-shot learning on point clouds.  In NeurIPS.
  * [36] Minhyuk Sung, Vladimir G. Kim, Roland Angst, and Leonidas Guibas.  Data-driven structural priors for shape completion.  2015\.
  * [37] Lyne P. Tchapmi, Vineet Kosaraju, Hamid Rezatofighi, Ian Reid, and Silvio Savarese.  Topnet: Structural point cloud decoder.  In CVPR, 2019.
  * [38] S. Thrun and B. Wegbreit.  Shape from symmetry.  In ICCV, 2005.
  * [39] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks.  In ICML, 2016.
  * [40] Xiaogang Wang, Marcelo H Ang Jr, and Gim Hee Lee.  Cascaded refinement network for point cloud completion.  In CVPR, 2020.
  * [41] Xin Wen, Zhizhong Han, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Yu-Shen Liu.  Cycle4completion: Unpaired point cloud completion using cycle transformation with missing region coding.  In CVPR, 2021.
  * [42] Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han.  Snowflakenet: Point cloud completion by snowflake point deconvolution with skip-transformer.  In ICCV, 2021.
  * [43] Haozhe Xie, Hongxun Yao, Shangchen Zhou, Jiageng Mao, Shengping Zhang, and Wenxiu Sun.  Grnet: Gridding residual network for dense point cloud completion.  In ECCV, 2020.
  * [44] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany.  Pointcontrast: Unsupervised pre-training for 3d point cloud understanding.  In ECCV, 2020.
  * [45] Siming Yan, Zhenpei Yang, Haoxiang Li, Li Guan, Hao Kang, Gang Hua, and Qixing Huang.  Implicit autoencoder for point cloud self-supervised representation learning, 2022.
  * [46] Xuejun Yan, Hongyu Yan, Jingjing Wang, Hang Du, Zhihong Wu, Di Xie, Shiliang Pu, and Li Lu.  Fbnet: Feedback network for point cloud completion.  In ECCV, 2022.
  * [47] Mohsen Yavartanoo, Shih-Hsuan Hung, Reyhaneh Neshatavar, Yue Zhang, and Kyoung Mu Lee.  Polynet: Polynomial neural network for 3d shape recognition with polyshape representation.  In 3DV, 2021.
  * [48] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou.  Pointr: Diverse point cloud completion with geometry-aware transformers.  In ICCV, 2021.
  * [49] Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert.  Pcn: Point completion network.  In 3DV, 2018.
  * [50] Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy.  Unsupervised 3d shape completion through gan inversion.  In CVPR, 2021.
  * [51] Wenbo Zhao, Xianming Liu, Zhiwei Zhong, Junjun Jiang, Wei Gao, Ge Li, and Xiangyang Ji.  Self-supervised arbitrary-scale point clouds upsampling via implicit neural representation.
  * [52] Haoran Zhou, Yun Cao, Wenqing Chu, Junwei Zhu, Tong Lu, Ying Tai, and Chengjie Wang.  Seedformer: Patch seeds based point cloud completion with upsample transformer.  In ECCV, 2022.
  * [53] Junsheng Zhou, Xin Wen, Yu-Shen Liu, Yi Fang, and Zhizhong Han.  Self-supervised point cloud representation learning with occlusion auto-encoder.
  * [54] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In ICCV, 2017.


11institutetext:  CV:HCI@KIT, Karlsruhe Institute of Technology, Germany
https://cvhci.anthropomatik.kit.edu/ 22institutetext: ACCESS@KIT, Karlsruhe
Institute of Technology, Germany
%\email{\\{name,surname\\}@kit.edu}https://www.access.kit.edu/
33institutetext: Computer Vision & Biometrics Lab, Indian Institute of
Information Technology Allahabad, India
https://cvbl.iiita.ac.in/ 44institutetext: University of Trier, Germany

# Alt4Blind: A User Interface to Simplify Charts Alt-Text Creation

Omar Moured Both authors contributed equally to this work.1122 Shahid Ali
Farooqui* 33 Karin Müller 22 Sharifeh Fadaeijouybari 44 Thorsten Schwarz 22
Mohammed Javed 33 Rainer Stiefelhagen 1122

###### Abstract

Alternative Texts (Alt-Text) for chart images are essential for making
graphics accessible to people with blindness and visual impairments.
Traditionally, Alt-Text is manually written by authors but often encounters
issues such as oversimplification or complication. Recent trends have seen the
use of AI for Alt-Text generation. However, existing models are susceptible to
producing inaccurate or misleading information. We address this challenge by
retrieving high-quality alt-texts from similar chart images, serving as a
reference for the user when creating alt-texts. Our three contributions are as
follows: (1) we introduce a new benchmark comprising 5,000 real images with
semantically labeled high-quality Alt-Texts, collected from Human Computer
Interaction venues. (2) We developed a deep learning-based model to rank and
retrieve similar chart images that share the same visual and textual
semantics. (3) We designed a user interface (UI) to facilitate the alt-text
creation process. Our preliminary interviews and investigations highlight the
usability of our UI. For the dataset and further details, please refer to our
project page: https://moured.github.io/alt4blind/.

###### Keywords:

Alt-Text Image Retrieval CLIP Model.

##  1 Introduction

Making charts accessible to a broader audience is essential to ensure equal
access. Part of this audience includes people with blindness and visual
impairments individuals who access data through different modalities: tactile
and auditory. The choice between these modalities often depends on the
available technologies and the preferences of people with blindness and visual
impairments. Although technologies like sonification and tactile materials are
available, Alt-Text remains the primary method for BVI individuals to acquire
information about images through screen readers [4]. In accordance by the WCAG
111https://www.w3.org/WAI/alt/, an alt text is required for a graphic to be
accessible. These summaries are embedded as hidden tags within documents or
web pages. A study [2] revealed that over 80% of web pages often neglect to
include alt text. Furthermore, even when available, it often doesn’t comply
with recommended guidelines (e.g. WCAG) and standards (e.g. W3C).

Chart alt-text can originate from two primary sources: manually created by
humans or (semi-)automatically generated by AI systems. Human-generated
descriptions are often accurate but typically require expertise. Recent
studies have highlighted a significant Alt-text deficiency in publications
[1]. On the other hand, automatically generated captions are quicker to
obtain, and require no expertise but may suffer from information inaccuracy.
Thus, crafting a high-quality chart alt-text is not a trivial task [6].

Instead, this work investigates the role of using AI in aiding both experts
and non-expert users to write high-quality alt-text by presenting high-
quality, similar charts with alt-text as references for them. To achieve this,
we built a comprehensive dataset composing 5,000 chart images sourced from
venues providing alt-text with their publications. We then trained an AI model
to categorize images based on their visual and textual similarities.
Subsequently, the most similar images are retrieved and presented to the user
through our Alt4Blind interface (Figure 1).

Figure 1: Alt4Blind UI: (1) Menu bar offering access to guidelines and a
tutorial. (2) Space for uploaded images featuring a function bar (zoom, move,
fit). (3) Text field for user input, accompanied by a button to update the
retrieved image. (4) Retrieved charts based on the uploaded image, can be
further enhanced with text query.

##  2 Related Work

In this section, we first examine existing datasets in the field of chart
analysis, followed by a discussion on the available models and tools for
creating textual descriptions of charts.

###  2.1 Datasets

Chart-2-Text [5] is a popular dataset in computer vision. It comprises a large
number of chart images of various types, with each image accompanied by
summary and bounding box annotations. However, it is not tailored for people
with blindness and visual impairments, the summaries are not structured and do
not adhere to the accessibility guidelines. In contrast, Vistext [10]
comprises 8822 synthesized images, each manually assigned alt-texts,
representing a significant effort. However, it features synthetic charts,
restricted to line and bar charts, and the alt-texts were not authored by the
original image creators, potentially omitting crucial context. Another
dataset, HCI Alt-Text [1], collected chart figures from HCI conferences and
aims to analyse the quality of provided alt-texts. However, it consists of 511
chart images, making it less suitable for deep-learning models.

###  2.2 Alt-Text Creation Models

Chart Analysis is an emerging field in computer vision, featuring recent works
such as ChartAssistant [8] and ChartLLama [3]. These models are trained for
various chart-related tasks, including description generation, chart2table,
and chart2code, among others. Despite being trained on a vast chart corpus,
these models fall short of addressing accessibility concerns.

###  2.3 Tools

K. Mack [7] analyzed various interface designs to facilitate the creation of
high-quality alt-texts. Their results concluded that interfaces assisting
authors in deciding what to include in the alt-text were well-received and led
to higher-quality descriptions. However, there are very few tools that adopt
this approach. WATAA [4] is one such tool, which presents users with AI-
generated descriptions as a reference. Nevertheless, AI descriptions are prone
to hallucinations.

##### In this paper, we focus on addressing both limitations: (i) creating a
database with high-quality alt-texts, and (ii) developing an intelligent tool
that aids people with and without accessibility knowledge to author high-
quality alt-text.

##  3 Exploratory Interviews and Findings

With the help of the Center for Digital Accessibility and Assistive
Technologies in Karlsruhe Institute of Technologies 222www.access.kit.edu, we
recruited six participants with varying levels of accessibility expertise,
from none to advanced (working in the field of accessibility). We conducted
exploratory interviews with these participants (experts (P1-P3), non-experts
(P4-P6)) to identify necessary features for the development of an intuitive
user interface, the Alt4Blind app. Participants were not compensated.

###  3.1 Procedure

The interviews took place in a one-hour face-to-face session following the
steps:

  1. 1.

Initially, we introduced a basic line chart featuring three distinct lines,
see the illustration in the upper left of Figure 1-2.

  2. 2.

Participants were then instructed to write a description for a blind person.

  3. 3.

Following this, we inquired about the challenges faced and their perspective
on essential features for a web tool designed for this task.

No compensation was offered for participation in this study. Furthermore, the
user study received approval from the KIT Internal Ethical Committee.

###  3.2 Qualitative Results

During the interview, we took notes that we analyzed after the sessions. The
analysis revealed the following observations: P1-P3, maintained a coherent
reading order, starting with visual semantics (e.g., chart type, title, and
axis labels) followed by contextual information (e.g., line trends, and
comparisons). In response to the follow-up questions, P1 noted that they
usually find Multivariate and Panel charts challenging. P4-P6 discussed how
they were unfamiliar with alt-texts as it’s often hidden in tags, and
suggested the need for an initial guideline before start writing. Based on
these insights, we identified as a feature that an initial suggestion for
user-uploaded images is necessary. Thus, we chose to (1) display a real
initial suggestion, i.e. displaying real chart images with human-authored
high-quality alt-texts. (2) the images should be visually or contextually
relevant to the user chart.

Figure 2: Our retrieval system leverages both the text and image encoder
modules of the fine-tuned CLIP model. This ensures similarity at both visual
and contextual levels.

##  4 Alt4Blind

In this section, we describe (i) our dataset, (ii) the AI-based model for
ranking and retrieving similar charts with alt-texts, and (iii) the creation
of the Alt4Blind User Interface.

###  4.1 Creation of our Dataset

We first crawled 25,000 images with alt-text from publicly available HCI-
related conferences spanning the last 10 years to expand on the HCI Alt-Text
dataset [1]. In the next step, we filtered out images that are not charts,
have a short alt-text or lack semantics, so we end up with 5,000 chart images,
marking a tenfold increase from the previous dataset.

Our dataset contains a wide variety of chart types, from familiar forms like
Line, Bar, Area, Pie, and Scatter charts to more unique visualizations not
present in previous works such as multi-variate and panel charts. We used the
4-level semantic model by Lundgard et al. [6] to rate alt-text quality.
Initial scoring was conducted using ChatGPT-4 by assessing the number of
semantics and levels present in each alt-text. The richest samples were
subsequently manually reviewed.

###  4.2 Chart Image Retrieval

Image retrieval involves searching for images similar to a given query image
or text, with a focus on ensuring the top results include similar chart
semantics. For this purpose, we employed CLIP ViT-B/32 [9] as our baseline
model. The CLIP architecture encompasses both a text and an image encoder,
each producing a 512-dimensional feature vector. The CLIP model uses the
unsupervised contrastive pre-training approach to cluster the samples in the
latent space. Hence, we fine-tuned the model on our dataset, with a batch size
of 16 for 10 epochs. Next, during the inference phase, Figure 2, we input the
image, extract the encoders’ representations and compare it to the samples
from the datasets. The top three candidates with the highest cosine similarity
scores are then retrieved. Our model achieved 92% in P@3 and 85% in R@3,
demonstrating high capabilities in Precision (P@3) and Recall (R@3) for
displaying similar chart images within the top three results, respectively.

###  4.3 User Interface

For our prototype, we utilized React JS333https://react.dev/, a JavaScript
library ideal for building user interfaces. It includes a landing page for
users to drag-and-drop chart images and a tutorial for first-time use. Once an
image is uploaded, the backend model display the top three similar candidates
on the main page (see Figure 1). Users can enlarge to view the full alt-text.
The "Get Similar" button allows users to refresh the candidates when they type
their summary in the text field.

##  5 Preliminary Evaluation

We conducted tests with the same participants, P1-P6, to assess our user
interface. Participants were presented with Panel and Multivariate charts,
which were unfamiliar to all. The samples are depicted in Figure 1, and
participants were instructed to:

  1. 1.

Review the guidelines and participate in the tutorial chart session first.

  2. 2.

Use Alt4Blind to upload the chart and create alternative text for the image
provided.

  3. 3.

Describe their experience in detail.

During the session, we tracked mouse movements and recorded user interaction
behaviors using pen and paper. All participants produced detailed
descriptions. P4-P6 especially benefited from the feature that allows copying
sentences from various similar charts to craft their descriptions. Expert
users appreciated this feature and recommended adding chart captions to
further complement the alternative text. P1 suggested increasing access to
more than three similar charts, whereas P3 proposed a feature to replace
irrelevant images among the selected charts.

##  6 Discussion and Limitations

In this section, we shortly discuss our results and the limitations of this
work. Intelligent Features: Our tool has demonstrated efficiency in enabling
both inexperienced and experienced users to author high-quality alt-texts.
However, it could further benefit from additional intelligent functionalities,
such as LLM-based feedback and descriptions, which are currently under
development.
Captions: While the current implementation assists users in creating alt-text,
we believe that integrating captions into the tool is essential, as captions
and alt-texts often complement each other
Interactions: Current UI does not allow users to have control over the similar
chart section. Future iterations should offer users control, allowing them to
omit, replace, or view more charts.
User Study: Our initial investigations, though encouraging, were conducted
with a limited number of users and lacked comprehensive control measures.
Future studies should engage a larger and more diverse group of participants
and should involve people with blindness and visual impairments.

##  7 Conclusion

We presented an online, open-source tool to enhance alt-text writing by using
AI image retrieval, making it more engaging. Our shared dataset invites
further development by the NLP and Computer Vision communities to advance
chart summarization tasks with a focus on people with blindness and visual
impairments. Currently, while our work primarily provides examples of
alternative texts, it lays the groundwork for future research incorporating
Vision-Language models to address accessibility concerns.

####  7.0.1 Acknowledgements

The authors would like to acknowledge the help of P. Venkatesh for his support
in developing the UI. We would also like to thank the HoreKa computing cluster
at KIT for the computing resources used to conduct this research.

####  7.0.2

This research was funded by the European Union’s Horizon $2020$ research and
innovation program under the Marie Sklodowska-Curie grant agreements no.
$861166$.

## References

  * [1] Chintalapati, S.S., Bragg, J., Wang, L.L.: A dataset of alt texts from hci publications: Analyses and uses towards producing more descriptive alt texts of data visualizations in scientific papers. In: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. pp. 1–12 (2022)
  * [2] Commission, D.R.: The web: Access and inclusion for disabled people; a formal investigation. The Stationery Office (2004)
  * [3] Han, Y., Zhang, C., Chen, X., Yang, X., Wang, Z., Yu, G., Fu, B., Zhang, H.: Chartllama: A multimodal llm for chart understanding and generation (2023)
  * [4] Jeong, H., Chun, M., Lee, H., Oh, S.Y., Jung, H.: Wataa: Web alternative text authoring assistant for improving web content accessibility. In: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces. pp. 41–45 (2023)
  * [5] Kantharaj, S., Leong, R.T.K., Lin, X., Masry, A., Thakkar, M., Hoque, E., Joty, S.: Chart-to-text: A large-scale benchmark for chart summarization. arXiv preprint arXiv:2203.06486 (2022)
  * [6] Lundgard, A., Satyanarayan, A.: Accessible visualization via natural language descriptions: A four-level model of semantic content. IEEE transactions on visualization and computer graphics 28(1), 1073–1083 (2021)
  * [7] Mack, K., Cutrell, E., Lee, B., Morris, M.R.: Designing tools for high-quality alt text authoring. In: Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility. pp. 1–14 (2021)
  * [8] Meng, F., Shao, W., Lu, Q., Gao, P., Zhang, K., Qiao, Y., Luo, P.: Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning (2024)
  * [9] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [10] Tang, B.J., Boggust, A., Satyanarayan, A.: Vistext: A benchmark for semantically rich chart captioning. arXiv preprint arXiv:2307.05356 (2023)


operator ${\mathbf{H}}_{\mathfrak{a}}$ the eigenvalue asymptotics is valid and
the coefficient in the asymptotics is given by the limit in the formula (1.6),
using Lemma 2.5. ∎

##  Appendix A Potential theory approach

We consider here the Poincaré-Steklov problem for the Laplacian, first, in
dimension $d+1\geq 3.$ Consider the single and double layer potential
operators on the Lipschitz boundary $\Sigma,$

$\sc\mbox{S}\hskip 1.0pt:L_{2}(\Sigma)\to L_{2}(\Sigma),\,\sc\mbox{S}\hskip
1.0pt:f(x)\mapsto\int_{\Sigma}R(x-y)f(y)d\mu_{\Sigma}(y),$ (A.1)

and

$\sc\mbox{D}\hskip 1.0pt:L_{2}(\Sigma)\to L_{2}(\Sigma),\,\sc\mbox{D}\hskip
1.0pt:f(x)\mapsto\int_{\Sigma}\partial_{\nu(y)}R(x-y)d\mu_{\Sigma}(y),$ (A.2)

where $R$ is the fundamental solution for the Laplacian in
${\mathbb{R}}^{d+1}.$ The N-to-D operator $N\\!\\!\\!D$ is expressed via these
operators as

$N\\!\\!\\!D=\sc\mbox{S}\hskip 1.0pt(\frac{1}{2}+\sc\mbox{D}\hskip
1.0pt)^{-1},$ (A.3)

see, e.g., [4], [5], [21]. To be more exact, the $N\\!\\!\\!D$ operator is
considered here on functions, orthogonal in $L_{2}(\Sigma)$ to constants. For
a smooth surface $\Sigma,$ both potential operators are order $-1$
pseudodifferential operators on $\Sigma.$ Being considered on functions
orthogonal to constants, the operator $(\frac{1}{2}+\sc\mbox{D}\hskip 1.0pt)$
is invertible and

$(\frac{1}{2}+\sc\mbox{D}\hskip 1.0pt)^{-1}=2-2\sc\mbox{D}\hskip
1.0pt(\frac{1}{2}+\sc\mbox{D}\hskip 1.0pt)^{-1}.$ (A.4)

This means that

$N\\!\\!\\!D=2\sc\mbox{S}\hskip 1.0pt-2\sc\mbox{S}\hskip
1.0pt\sc\mbox{D}\hskip 1.0pt(\frac{1}{2}+\sc\mbox{D}\hskip 1.0pt)^{-1},$ (A.5)

Therefore the difference between the N-to-D operator $N\\!\\!\\!D$ and twice
the single layer potential $2\sc\mbox{S}\hskip 1.0pt$ is an order $-2$
pseudodifferential operator, and, by standard perturbational arguments, the
eigenvalue asymptotics of $N\\!\\!\\!D$ is the same in the leading term as the
asymptotics for the single layer potential $2\sc\mbox{S}\hskip 1.0pt$; finding
the latter is a simple exercise for a smooth surface.

If $\Sigma$ is not infinitely smooth, for example, belongs to $C^{1+\alpha}$,
$\alpha>0$ (such surfaces are often called Lyapunov ones), the above scheme
still works. First of all, discarding the pseudodifferential approach, one
should return to considering the operators S and D as integral operators with
kernel having singularity at the diagonal. Here, on the one hand, the
eigenvalue asymptotics of the operator S is known since the results of M.Sh.
Birman and M.Z. Solomyak on general weakly polar integral operators, see [12],
where such eigenvalue asymptotics was found under rather weak regularity
conditions. At the same time, the operator D is now a singular integral
operator, but for a surface of class $C^{1+\alpha}$, it is still compact.
Thus, the operator $N\\!\\!\\!D$ becomes, by (A.5), a relatively compactly
perturbed single layer operator, and, again, by standard methods (say, using
the Keldysh perturbation theorem) we arrive at the Weyl asymptotics for the
$N\\!\\!\\!D$ operator.

So, it looks like this is not that long way remains to Lipschitz surfaces.
However some serious obstacles appear. First of all, for a Lipschitz surface,
the double layer integral operator D is not obviously bounded. In fact, it
required a hard work to prove that it is bounded in $L_{2}(\Sigma)$ and,
moreover, that the operator $\frac{1}{2}+\sc\mbox{D}\hskip 1.0pt$ is
invertible. Further on, it turned out that D is not necessarily compact; even
in the two-dimensional case, in the presence of a corner point of a piece-wise
smooth boundary, it has a fragment of essential spectrum near zero. As for the
single layer potential $\sc\mbox{S}\hskip 1.0pt,$ the results of [12] are, by
themselves, not sufficient to establish eigenvalue asymptotics. However, in
[6], M.S.Agranovich and B.A.Amosov succeeded in proving order sharp two-sided
eigenvalue estimates for potential type operators on Lipschitz surfaces,
which, in their turn, produce two-sided eigenvalue estimates for
$N\\!\\!\\!D.$ Moreover, a localization technic was developed in [6]. It
turned out that if the non-smooth singularities of the surface $\Sigma$ are
localized in the sense that the surface is smooth outside a closed set of zero
measure, then the contribution of these singularities of $\Sigma$ both to the
behavior of eigenvalues of S and to the non-compactness of D is negligible,
and the Weyl asymptotic formula for the P-S problem was proved in this
setting.

In that period, after M.S.Agranovich started popularizing the problem on
Steklov eigenvalue asymptotics for Lipschitz surfaces without additional
restrictions, Prof. Grigory Tashchiyan and the author of this paper managed in
2006 to dispose of one of complications. Namely, in [37], we used the
approximation of Lipschitz surfaces by smooth ones, in the sense of a certain
integral norm, to prove that the corresponding potential type integral
operators, properly relocated, so that they act on one and the same surface,
can be compared, their difference becomes small in the spectral sense (see
Lemma 6.3), and the passage to limit in formulas for spectral asymptotics
becomes possible. Although this result did not solve the Lipschitz Steklov
eigenvalue asymptotics problem, it solved a closely related one. Namely, in
the _interface_ spectral problem, where the spectral parameter is placed, for
solutions of the (Laplace, in the simplest case) equation at the _jump_ of the
normal derivative across the Lipschitz surface $S\subset\Omega:$

$\displaystyle\Omega=\Omega_{1}\cup\Omega_{2}\cup
S,\,S=\overline{\Omega_{1}}\cap\overline{\Omega_{2}},\,\Delta
u=0,\,\mbox{in}\,\Omega_{1}\cup\Omega_{2},$ (A.6)
$\displaystyle\lambda\left[\frac{\partial
u}{\partial_{\nu(x)}}\right]_{x}=u(x),\,x\in S,$

the solution involves only the single layer potential operator S and the Weyl
asymptotics for the Lipschitz surface is justified. Rather optimistically, we
hoped that the Steklov problem will succumb as well.

One of the approaches we tried to explore was the following. Although we know
that the double layer operator D is not expected to be compact, probably, the
_composition_ D S can be shown to be _weaker_ than S itself. The notion
’weaker’ should imply that the singular numbers of D S decay faster than the
ones of S ,

$s_{n}(\sc\mbox{D}\hskip 1.0pt\sc\mbox{S}\hskip
1.0pt)=o(s_{n}(\sc\mbox{S}\hskip 1.0pt))=o(n^{-\frac{1}{d}}).$ (A.7)

In the eigenvalue studies, there are a lot of occasions when the
multiplication by a non-compact operator nevertheless improves the rate of
decay of the eigenvalues of the product.

Hypothesis. Consider the composition D S of single and double layer
potentials. Recall that S is the integral operator with kernel $C|x-y|^{1-d}$
over a Lipschitz surface of dimension $d>1.$ Let ${\mathcal{T}}(x,y)$ be the
integral kernel of the composition S D ,

${\mathcal{T}}(x,y)=\int_{\Sigma}|x-z|^{1-d}\partial_{\nu(z)}|z-y|^{1-d}dz.$
(A.8)

Suppose, hopefully, that

${\mathcal{T}}(x,y)|x-y|^{d-1}=o(1)\,\mbox{as }\,y\to x,\,\mbox{uniformly
}\,\mbox{in}\,x\in\Sigma.$ (A.9)

######  Proposition A.1.

Suppose that $d\geq 3$ and (A.9) is satisfied. Then
$\lambda_{j}(\sc\mbox{S}\hskip 1.0pt\sc\mbox{D}\hskip
1.0pt)=o(\lambda_{j}(\sc\mbox{S}\hskip 1.0pt)).$

###### Proof.

For a fixed $\epsilon$, we find a neighborhood
$\mathfrak{U}_{\epsilon}\subset\Sigma\times\Sigma$ of the diagonal $x=y$ such
that $|{\mathcal{T}}(x,y)|<\epsilon|x-y|^{1-d}$ for
$(x,y)\in\mathfrak{U}_{\epsilon}$. Take a cut-off function
$\chi_{\epsilon}(x,y)$ which equals zero outside $\mathfrak{U}_{\epsilon}$ and
equals $1$ inside, this means, in a neighborhood of the diagonal $x=y.$ The
operator $\sc\mbox{T}\hskip 1.0pt=\sc\mbox{S}\hskip 1.0pt\sc\mbox{D}\hskip
1.0pt$ splits into the sum of two operators, $\sc\mbox{T}\hskip
1.0pt_{\epsilon},$ with kernel ${\mathcal{T}}(x,y)\chi_{\epsilon}(x,y)$ and
$\sc\mbox{T}\hskip 1.0pt_{\epsilon}^{\prime}$ with kernel
${\mathcal{T}}(x,y)(1-\chi_{\epsilon}(x,y)).$ The second operator has a
bounded kernel, therefore, in particular, belongs to the Hilbert-Schmidt
class, and its singular values decay at least as
$\lambda_{j}(\sc\mbox{T}\hskip
1.0pt_{\epsilon}^{\prime})=O(j^{-\frac{1}{2}}),$ i.e. faster than the
eigenvalues of $\sc\mbox{S}\hskip 1.0pt.$ As for the first operator,
$\sc\mbox{T}\hskip 1.0pt_{\epsilon},$ we can apply the result by G.Kostometov,
[28], Theorem 1 (see also, [6], Theorem 4.1.) According to this theorem, for
an integral operator T in a bounded domain in ${\mathbb{R}}^{d}$ with kernel
having form ${\mathcal{T}}(x,y)=\phi(x,y)|x-y|^{-{\mathbf{k}}},$
$d/2<{\mathbf{k}}<d,$ with a bounded function $\phi$, the eigenvalue estimate
holds

${\mathbf{n}}^{\sup}(\theta,\sc\mbox{T}\hskip 1.0pt)\leq
C\|\phi\|_{L_{\infty}}^{\theta},\theta=\frac{d}{d-{\mathbf{k}}.}$ (A.10)

In our case, in local co-ordinates, for
${\mathcal{T}}_{\epsilon}(x,y)={\mathcal{T}}(x,y)\chi_{\epsilon}(x,y),$
${\mathbf{k}}=d-1$ and $\phi(x,y)={\mathcal{T}}(x,y)\chi(x,y)|x-y|^{1-d}$,
$\|\phi\|_{L_{\infty}}\leq\epsilon$. So, (A.10) gives
${\mathbf{n}}^{\sup}(\theta,\sc\mbox{T}\hskip 1.0pt_{\epsilon})\leq
C\epsilon^{\theta},$ and by the arbitrariness of $\epsilon,$ we arrive at the
required estimate. ∎

This estimate leads to the justification of the Weyl asymptotics, provided
(A.9).

Using somewhat finer estimates, one covers the case of $d=2$ (excluded in the
estimates in [28], [6], since here the condition $d/2<{\mathbf{k}}$ is not
met). The hypothesis itself, meanwhile, stays unresolved, although for some
particular nonregular surfaces we were able to justify it.

##  Appendix B Relaxing the smoothness conditions.

Another possible approach consists in finding the conditions on the surface
$\Sigma$, just a little bit more restrictive than Lipschitz ones, but still
granting the compactness of the double layer operator D , restricted to the
complement of some small set, containing major singularities.

Here, the results of [24] proved to be useful. Let
$\Omega\subset{\mathbb{R}}^{d+1}.$ The domain $\Omega$ is called
$\mbox{VMO}_{1}$-domain if (3.2) holds and the almost everywhere existing
normal vector field $\boldsymbol{\nu}(x)$ belongs to the space
$\mbox{VMO}(\Sigma)$, the closure of the space of continuous vector fields on
$\Sigma$ in the VMO metric. Of course, the class of $\mbox{VMO}_{1}$ domains
contains all $C^{1}$ domains, but does not contain _all_ Lipschitz domains. By
Theorem 4.35 in [24], singular integral operators of the type of double layer
potential $\sc\mbox{D}\hskip 1.0pt,$ are compact in $L_{2}(\Sigma).$ By a
localization of this property, as it was done, e.g., in (see, e.g., [6], [5],
[2]) we arrive at the following result.

######  Proposition B.1.

Let the $\Omega$ be a Lipschitz domain and let the boundary $\Sigma$ belong to
$\mbox{VMO}_{1}$ outside a closed set $E$ of the surface measure zero. Then
for the $N\\!\\!\\!D$ operator the Weyl asymptotic formula holds.

The proof, leading, again, to (A.7), consists in splitting the Steklov problem
$\partial_{\nu}u=\lambda^{-1}u$ on $\Sigma$ into two weighted ones,

$\partial_{\nu}u=\lambda\rho_{\epsilon}u,\,\mbox{and}\,\partial_{\nu}u=\lambda(1-\rho_{\epsilon})u,$
(B.1)

where $\rho_{e}$ is the characteristic function of a small neighborhood of the
set $E$, having surface measure less than $\epsilon$. This leads to the
corresponding splitting of the operator ${\mathbf{H}}$ in (4.5) into
${\mathbf{H}}={\mathbf{H}}_{\rho_{\epsilon}}+{\mathbf{H}}_{1-\rho_{\epsilon}}.$
For the first of these operators, the smallness for
${\mathbf{n}}^{\sup}(d,{\mathbf{H}}_{\rho_{\epsilon}})$ can be proved on the
base of Theorem 2.1. For the remaining operator,
${\mathbf{H}}_{1-\rho_{\epsilon}},$ it follows from the $VMO_{1}$ compactness
property for the double layer potential that the
${\mathbf{n}}^{\sup}(d,{\mathbf{H}}_{\rho_{\epsilon}})=0.$

One may hope that the approximation of a Lipschitz surface by $VMO_{1}$ ones
may lead to a more straightforward proof of the Weyl asymptotics for the P-S
eigenvalues.

## References

  * [1] R. Adams, J. Fournier. Sobolev Spaces. Pure and Applied Mathematics, vol. 140. Elsevier/Academic Press, Amsterdam, 2003.
  * [2] M.S. Agranovich. _On a mixed Poincaré–Steklov type spectral problem in a Lipschitz domain._ Russian Journal of Mathematical Physics, 13, (3) (2006), 239–244.
  * [3] M.S. Agranovich. _Potential type operators and transmission problems for strongly elliptic second order systems in Lipschitz domains._ Funct. Anal. and Appl. 43 (2009) 165–183.
  * [4] M.S. Agranovich. _Spectral problems for second–order strongly elliptic systems in smooth and non–smooth domains_ , Russian Math. Surveys, 57 (5) (2002), 847–920.
  * [5] M.S. Agranovich. _Spectral properties of potential type operators for a certain class of strongly elliptic systems on smooth and Lipschitz surfaces._ Trans. Mosc. Math. Soc. 62, 1–47 (2001).
  * [6] M.S. Agranovich, B.A. Amosov._Estimates of _s_ –numbers and spectral asymptotics for integral operators of potential type on nonsmooth surfaces._ Funct. Anal. Appl. 30 (2) (1996) 75–89.
  * [7] M. Agranovich, B. Katsenelenbaum, A. Sivov, N. Voitovich. Generalized Method of Eigenoscillations in Diffraction Theory. WILEY-VCH Verlag Berlin GmbH, Berlin, 1999.
  * [8] W. Arendt, A.F.M. ter Elst, _The Dirichlet-to-Neumann operator on rough domains_ , Journ. of Diff. Equat., 251 (2011), 2100–2124.
  * [9] W. Arendt, R. Mazzeo. _Spectral properties of the Dirichlet-to-Neumann operator on Lipschitz domains._ Ulmer Seminare, Heft 12 (2007), 28–38.
  * [10] J. Behrndt, A.F.M. ter Elst,_Dirichlet–to–Neumann maps on bounded Lipschitz domains,_ Journ. Diff. Equat., 259 (2015), 5903–5926.
  * [11] M.Sh. Birman, M.Z. Solomyak. Quantitative analysis in Sobolev imbedding theorems and applications to spectral theory. (Russian). Tenth Mathematical School (Russian),Inst. Mat. Akah. Nauk Ukrain. SSR, Kiev, 1974. pp. 5–189. English translation in: American Mathematical Society Translations, Ser. 2. 114 (1980).
  * [12] M.Sh. Birman, M.Z. Solomyak, _Asymptotics of the spectrum of weakly polar integral operators._ (Russian) Izv. Akad. Nauk SSSR Ser. Mat. 34, (5) (1970), 1142–1158. English translation in: Math. USSR, Izvestija 4, (1971) 1151–1168.
  * [13] M.Sh. Birman, M.Z. Solomyak. _Spectral asymptotics of nonsmooth elliptic operators._ I, Trans. Moscow math. Soc. 27 (1972); English: 27, (1975) 1–52.
  * [14] M.Sh. Birman, M.Z. Solomyak,_Asymptotic behavior of the spectrum of pseudodifferential operators with anisotropically homogeneous symbols._ I. (Russian) Vestnik Leningrad. Univ. 1977, no. 13, Mat. Meh. Astronom, 13–21. (English) Vestnik Leningr. Univ. Math. 10 (1982) 237–247. II. (Russian) Vestnik Leningrad. Univ. no. 13, Mat. Mekh. Astronom. (1979), 5–10. (English) Vestn. Leningr. Univ., Math. 12 (1980), 155–161.
  * [15] M.Sh. Birman, M.Z. Solomyak. _Asymptotic behavior of the spectrum of variational problems on solutions of elliptic equations._ (Russian) Sibirsk. Mat. Zh. 20, (1) (1979), 3–22. English translation in: Sib. Math. J. 20 (1979), 1–15.
  * [16] A.P. Calderón. _Boundary value problems for the Laplace equation in Lipschitzian domains._ Recent progress in Fourier analysis (El Escorial, 1983), 33–48, North-Holland Math. Stud., 111, North-Holland, Amsterdam, 1985.
  * [17] B. Colbois, A. Girouard, C. Gordon, D. Sher. _Some recent developments on the Steklov eigenvalue problem._ Arxiv:2212.12528
  * [18] A.F.M. ter Elst, E.M. Ouhabaz,_Dirichlet-to-Neumann and elliptic operators on $C^{1+k}$-domains: Poisson and Gaussian bounds_. Journ. Differ. Equat. 267 (2019), (7) 4224–4273.
  * [19] A. Girouard, M. Karpukhin, M. Levitin, I. Polterovich. _The Dirichlet–to–Neumann map, the boundary Laplacian, and Hörmander’s rediscovered manuscript._ J. Spectr. Theory 12 (2022), 1, 195–225.
  * [20] A. Girouard, I. Polterovich, _Spectral geometry of the Steklov problem,_ J. Spectr. Theory, 7:2 (2017), 321–359.
  * [21] G. Grubb. Distributions and Operators. Graduate Texts in Mathematics, 252, Springer-Verlag, New York, 2009.
  * [22] D. Hilbert. Grundzüge einer allgemeinen Theorie der linearen Integralgleichungen. Teubner. Leipzig und Berlin. 1912.
  * [23] S. Hoffman, M. Mitrea, M. Taylor. _Geometric and transformational properties of Lipschitz domains, Semmes-Kenig-Toro domains, and other classes of finite perimeter domains._ Journ. of Geom. Analysis, 17, (4) (2007), 593–647.
  * [24] S. Hofmann, M. Mitrea, M. Taylor, _Singular integrals and elliptic boundary problems on regular Semmes-Kenig-Toro domains_ , IMRN (2010), 14, 2567–2865.
  * [25] P. Jones. _Quasiconformal mappings and extendability of functions in Sobolev spaces._ Acta Math. 147 (1981) 71–88.
  * [26] M. Karpukhin, J. Lagacé, I. Polterovich. _Weyl’s law for the Steklov problem on surfaces with rough boundary_ arXiv:2204.05294
  * [27] T. Kato, Perturbation Theory for Linear Operators. Springer, 1984.
  * [28] G. Kostometov, _Asymptotic behavior of the spectrum of integral operators with a singularity on the diagonal._ Math. USSR-Sb., 23:3 (1974), 417–424; Russian: Matematichesky Sborn. 94 (136):3(7) (1974), 444–-451.
  * [29] M. Levitin, D. Mangouri, I. Polterovich. Topics in Spectral Geometry. (Book in preparation) https://michaellevitin.net/Book
  * [30] M. Levitin, L. Parnovski, I. Polterovich, and D. A. Sher, _Sloshing, Steklov and corners: Asymptotics of sloshing eigenvalues_ , J. d’Anal. Math. 146 (2022), 65–125.
  * [31] M. Mitrea, M. Taylor,_Boundary layer methods for Lipschitz domains in Riemannian manifolds,_ J. Funct. Anal. 163 (1999), 181–251.
  * [32] H. Poincaré._Sur l’équilibre et les mouvements des mers._ Journ. de Math. (5) 2 (1896) 57-102. (see: H.Poincaré, Oevres, vol.VIII, 1950, P. 198–236)
  * [33] R. Ponge. _Connes’ integration and Weyl’s laws._ J. Noncommut. Geom. 17 (2023), 2, 719–767, Arxiv:2107.01242.
  * [34] L. Provenzano, J. Stubbe,_Weyl-type bounds for Steklov eigenvalues_ , J. Spectr. Theory 9, no. 1 (2019), 349–377.
  * [35] G. Rozenblum, _On the asymptotics of the eigenvalues of certain two-dimensional spectral problems,_ Sel. Math. Sov. 5:3 (1986), 233–244.
  * [36] G. Rozenblum, E. Shargorodsky _Eigenvalue estimates and asymptotics for weighted pseudodifferential operators with singular measures in the critical case._ Partial Differential Equations, Spectral Theory, and Mathematical Physics -— the Ari Laptev Anniversary Volume, 331–354, EMS Ser. Congr. Rep., EMS Press, Berlin, 2021.
  * [37] G. Rozenblum, G. Tashchiyan, _Eigenvalue asymptotics for potential type operators on Lipschitz surfaces._ Russian J. Math. Phys. 13 (3) (2006) 326–339.
  * [38] G. Rozenblum, G. Tashchiyan. _Eigenvalues of the Birman–Schwinger operator for singular measures: the noncritical case._ J. Funct. Anal. 283 (2022), (12) Paper No. 109704.
  * [39] L. Sandgren, _A Vibration Problem,_ Medd. Lunds Univ., Math. Sem., (1955) 1–84.
  * [40] V.A. Steklov._Sur les problèmes fondamentaux de la physique matematique._ Annales scientifiques de l’ È.N.S. 3e série, 19 (1902), 191–259; 19 (1902), p. 455–490.
  * [41] T. Suslina. _Spectral asymptotics of variational problems with elliptic constraints in domains with piecewise smooth boundary._ Russ. J. Math. Phys. 6 (2) (1999), 214–234.
  * [42] T. Suslina. _Asymptotics of the spectrum of variational problems arising in the theory of fluid oscillations._ (Russian) Sovrem. Mat. Fundam. Napravl. 67 (2021), no. 2, 363–407.
  * [43] M. Taylor. The Dirichlet-to-Neumann Map And Fractal Variants. Course notes, Univ. North Carolina, 2021. (https://www.ams.org/open-math-notes/files/course-material/OMN-202110-111310-1-Course-notes-v2.pdf, https://mtaylor.web.unc.edu/wp-content/uploads/sites/16915/2018/04/dton.pdf)
  * [44] M. Taylor. Partial Differential Equations II. Qualitative Studies of Linear Equations. Springer, 2011.
  * [45] G. Verchota. _Layer potentials and regularity for the Dirichlet problem for Laplace’s equation in Lipschitz domains._ J. Funct. Anal. 59 (1984), 3, 572–611.
  * [46] I. Vulis, M. Solomjak, _Spectral asymptotic analysis for degenerate second order elliptic operators._ (English) Math. USSR, Izv. 8(1974), 1343–1371. (Russian) Izv. Akah. Nauk SSSR Ser. Mat. 38 (1974), 1362–1392.
  * [47] I. Vulis, _Spectral asymptotic behavior of elliptic operators of arbitrary order with a strong degeneracy on the boundary of the domain._ (Russian) Problems in mathematical physics, No. 8 (Russian), pp. 56–69, 173. Izdat. Leningrah. Univ., Leningrad, 1976.
  * [48] W.P. Ziemer. Weakly Differentiable Functions. Springer-Verlag, NY, 2012.