# A Skeleton-based Approach For Rock Crack Detection Towards A Climbing Robot
Application

Josselin Somerville Roberts  Department of
Computer Science
Stanford University
<EMAIL_ADDRESS>Paul-Emile Giacomelli  Department of
Aeronautics and Astronautics
Stanford University
<EMAIL_ADDRESS>Yoni Gozlan  Institute for Computational
and Mathematical Engineering
Stanford University
<EMAIL_ADDRESS>Julia Di  Department of
Mechanical Engineering
Stanford University
<EMAIL_ADDRESS>
###### Abstract

Conventional wheeled robots are unable to traverse scientifically interesting,
but dangerous, cave environments. Multi-limbed climbing robot designs, such as
ReachBot, are able to grasp irregular surface features and execute climbing
motions to overcome obstacles, given suitable grasp locations. To support
grasp site identification, we present a method for detecting rock cracks and
edges, the SKeleton Intersection Loss (SKIL). SKIL is a loss designed for thin
object segmentation that leverages the skeleton of the label. A dataset of
rock face images was collected, manually annotated, and augmented with
generated data. A new group of metrics, LineAcc, has been proposed for thin
object segmentation such that the impact of the object width on the score is
minimized. In addition, the metric is less sensitive to translation which can
often lead to a score of zero when computing classical metrics such as Dice on
thin objects. Our fine-tuned models outperform previous methods on similar
thin object segmentation tasks such as blood vessel segmentation and show
promise for integration onto a robotic system.

###### Index Terms:

computer vision, segmentation, thin object, crack detection, blood vessel
detection, climbing robotics

##  I Introduction

Figure 1: Workflow of the ViT-B based model discussed in this work (Section
IV-A) with crop size of $512$ and patch size of $16$.

In exploration tasks, robots offer versatility and robustness for navigating
novel environments. However, challenging terrain, like caverns and steep
slopes which are of high astrobiological and geological interest, have often
hampered traditional approaches such as wheeled rovers. Aerial vehicles
meanwhile may be suited for rough terrain exploration, but have limited flight
time and ability to traverse caves. To address this mobility gap, ReachBot is
a novel robot concept that uses microspine grippers at the end of extendable
booms to achieve a large workspace and wrench capability. This results in
higher navigation capability across the rocky scrambles that one may find in
lava tubes and caves on the Moon and Mars [1, 2, 3].

Of crucial importance is a perception system that identifies a good grasp site
on a realistic rock surface. While one may identify grasp sites through basic
primitives, such as trial and error, we note that in climbing robot
applications, misplaced holds may lead to mission failure and robot damage.

Furthermore, ReachBot’s large workspace encourages a grasp site identification
method capable from a distance. As an example, we consider cracks and edges as
promising rock features for microspine grippers, because microspines may
engage well with the asperities of a crack or edge leading to a strong grasp
[4]. Yet identifying rock cracks from a distance can prove complex because
cracks are typically long and thin features that are susceptible to lighting
conditions and the angle of the camera with respect to the rock wall.

Statement of Contributions: This work focuses on improving rock crack
detection towards a climbing robot application, with the following
contributions:

  * •

We present a new segmentation loss, SKIL (SKeleton Intersection Loss), to
identify thin features such as cracks on rock walls, blood vessels, or other
thin objects.

  * •

We propose novel metrics for evaluating the detection results of fine and
longitudinal objects, LineAcc. These metrics are based on the position
(LineAcc-pos) and sizes (LineAcc-width/length) of the cracks, and are more
robust to ground truth labeling errors than classical metrics.

  * •

We manually collected and generated a new dataset of rock wall images, which
is open-sourced to support the development of other rock climbing-related
applications. All code, commands, and datasets are included in this Github
repo as well as instructions to recreate results.

To assess the added value of SKIL and our new metric, we evaluated and
compared performance on blood vessel segmentation benchmark datasets, an
existing and related task in the medical imaging community.

Paper Organization: The rest of the paper is organized as follows. Section II
provides an overview of related works for the task. Section III discusses the
robot design and important considerations for crack detection. Section IV
explains the proposed SKIL loss and new metrics. Section V describes the
experiments and results. Finally, Section VI summarizes our contributions and
explores future work.

##  II Related Works

###  II-A Existing Datasets

We examined the possibility of leveraging existing datasets for this work.
Rock segmentation for indoor climbing or climbing robots generally feature
artificially colored rocks with indoor lighting, which are not representative
of the outdoor volcanic rock walls that ReachBot will encounter on a potential
mission to a Martian lava tube [5, 6, 7]. Rock classification datasets are
collected outdoors and can be specified by sediment type (including volcanic),
but rock type alone does not necessarily correspond to whether a rock is
graspable, and many datasets are only of standalone rocks and not natural rock
faces as one would encounter in a cave or lava tube [8, 9, 10, 11]. Datasets
on infrastructural cracks are also potential candidates for crack detection,
but most man-made structures like concrete walls and bridges are
macroscopically smooth with very fine cracks, and do not represent realistic
surfaces for this robot mission [12, 13].

###  II-B Existing Methods

Because ReachBot does not have an anthropomorphic gripper, existing grasp site
identification approaches developed for human climbing do not fit the
kinematics of the gripper [7]. Alternatively, there are models for the
microspine grippers on a variety of rock-climbing robots [14, 4, 15], and past
analyses have developed a stochastic wrench limit surface, which depends on
both the gripper design and surface topography [16, 17, 18]. While these
model-based approaches work for analyzing microspine grasps, they often
require detailed ground truth topographical knowledge which limits them from
deployment on a real robot.

Therefore, we are interested in learning to detect rock features for grasping,
such as cracks and edges. As a first step, crack detection methods exist in
other related applications such as infrastructural monitoring, but these tend
to be extremely fine cracks [19, 20]. A similar task can be found in the
medical imaging space with blood vessel segmentation [21, 22]. Several losses
have been proposed for blood vessels segmentation task (e.g. imbalanced[23],
connected[24] and thin[25]) and some state-of-the-art methods for vessel
segmentation, such as U-Net [26, 27, 28], appear promising for rock crack
detection. We further discuss in Section V their performance for this use
case, and investigate a modified skeleton-based approach for detecting crack
and edge features for a robotics application, and benchmark performance with
related blood vessel datasets.

###  II-C Existing Metrics

For image segmentation, IoU and Dice coefficient are commonly used metrics
[29]. These metrics measure the pixel-wise agreement between prediction and
ground truth, which penalizes any small misalignments between the predicted
crack pixels and the true crack (which may often occur with manual
annotation). Further, adjustments to the Dice score to make the prediction
less sensitive to small position misalignments, such as a diffused prediction,
would still not help us understand model performance on graspable cracks.

Many alternative metrics have been proposed for image segmentation [30]. In
the context of crack detection, spatial distance-based metrics may be relevant
such as the Hausdorff distance [31]. However, they are sensitive to the
thickness of the thin objects, which is uncertain in our case. One proposed
metric is to only compute the Dice score around thin zones of the ground truth
[32]. This metric does not reflect performance on larger zones of ground truth
and does not mitigate the translation sensitivity.

##  III Gripper Discussion

The ReachBot design concept allows for navigation through cave-like
environments with the use of microspine grippers on the end of extendable
booms, as shown in Fig. 2.

Figure 2: The ReachBot concept uses microspine grippers at the end of
extendable booms to reach a large workspace inside a Martian lava tube. Inset
is a microspine gripper prototype grasping real lava rock in a Mojave Lava
Tube.

ReachBot’s microspine gripper, inset in Fig. 2, grasps onto rocks successfully
when a sufficient number of microspines are engaged on the surface asperities
(such as those around the rock cracks). Because microspine grasping is
inherently stochastic, there is no formal guarantee for grasp success.
However, several parameters influence the envelope of graspable features,
including the distribution of millimeter-scale asperities of the surface, the
geometry of the surface based on the gripper design, and the sharpness of the
microspines (which dull after too much continuous use) [33].

Because obtaining micrometer-level surface characterization of the asperities
is not practical on the robot, we instead investigate macroscale rock features
for grasping. Akin to how a trained rock climber may look for certain rock
shapes to fit their hand, previous work has already investigated rock geometry
analysis for a ReachBot-specific gripper [33]. Classical shape-fitting
techniques were used, which work best at identifying symmetrically convex
shapes but miss potential irregular grasp sites such as cracks, ledges, and
edges. It is important to identify as many grasp sites as possible because
ReachBot’s unique design enables navigation even when grasp sites are sparse,
but only if the robot knows they exist.

##  IV Proposed Approach

###  IV-A Model Architecture

All the experiments were run with a ViT-B[34] backbone with a crop size of
$512\times 512$ and a patch size of $16\times 16$ using the UPerNet[35]
method. This model was chosen qualitatively as it offered good performances
after a few hundred epochs. Other models considered included SAM[36],
ResNet[37], and DeiT[38], but were deemed too complex for a resource-
constrained hardware context. As the goal is to compare losses and eventually
run on a field robot in a cave, we wanted to keep the model as simple as
possible.

###  IV-B LineAcc Segmentation Metrics

We propose several metrics, grouped under the name of LineAcc, to measure
different aspects of the segmentation that classic metrics did not capture as
discussed earlier.

####  IV-B1 Center line position

The first metric is the center line position, which is referred to as
$\text{LineAcc}_{\text{pos}}$. The goal is to measure if the cracks predicted
are well positioned regardless of the width or length of the crack.

Given a prediction $\mathcal{P}$ and label $\mathcal{L}$, we introduce their
skeletons $\mathcal{S_{P}}$ and $\mathcal{S_{L}}$. The metric
$\text{LineAcc}_{\text{pos}}$ is defined as:

$\begin{split}\text{LineAcc}_{\text{pos}}(\mathcal{P},\mathcal{L})&=\frac{\sum\mathcal{S_{P}}\times
e^{-d(\mathcal{S_{L}})^{2}/2\sigma^{2}}}{\sum\mathcal{S_{P}}}\\\
&\times\frac{\sum\mathcal{S_{L}}\times
e^{-d(\mathcal{S_{P}})^{2}/2\sigma^{2}}}{\sum\mathcal{S_{L}}}\end{split}$ (1)

With $d$ a distance function such that for $I\in\mathbb{R}^{H\times W}$,

$d(I)_{x,y}=\min_{u,v,I(u,v)=1}|x-u|+|y-v|$ (2)

This metric returns a score illustrating how close the two skeletons are (with
a Gaussian decrease).

####  IV-B2 Width and length ratios

We further introduce two ratio-based metrics: the width and length ratios.
These two metrics illustrate how close the width and length of a predicted
crack is to the ground truth, independently of the position of the crack.

$\text{LineAcc}_{\text{length}}(\mathcal{P},\mathcal{L})=\exp\left(-\left|\frac{\sum\mathcal{S_{P}}+\epsilon}{\sum\mathcal{S_{L}}+\epsilon}-1\right|\right)$
(3)
$\text{LineAcc}_{\text{width}}(\mathcal{P},\mathcal{L})=\exp\left(-\left|\frac{\left(\sum\mathcal{L}\right)\left(\sum\mathcal{S_{P}}\right)+\epsilon}{\left(\sum\mathcal{P}\right)\left(\sum\mathcal{S_{L}}\right)+\epsilon}-1\right|\right)$
(4)

These two metrics are also not correlated as the length ratio
$\text{LineAcc}_{\text{length}}$ relies on the skeletons (independent to
width) and the width ratio $\text{LineAcc}_{\text{width}}$ uses the inverse of
the length ratio to balance its effect. A constant $\epsilon=10^{-3}$ is added
to avoid dividing by zero.

###  IV-C SKIL: a new loss

We investigated new losses that optimize for the new metrics. Although CL-Dice
[24] loss appeared promising (computes the Dice score between the skeleton of
the prediction and the ground truth label), the skeleton intersection is
dependent on the width of the labels and the Dice score does not optimize the
center line position well. As we plan to use a combination of Dice and a new
loss, this was not desirable.

####  IV-C1 SKIL-Dice, a loss adapted to the new metrics

The core idea of SKIL-Dice is a skeleton-based approach that abstracts the
width of the crack. Another improvement that was necessary was to integrate
the distance from the two center lines. As an example, with CL-Dice, if the
ground truth is a thin vertical rectangle of width $w$, then whether the
prediction is shifted by a distance $D$ of 1 pixel or $w/2-1$, the score would
be the same. However, the $\text{LineAcc}_{\text{pos}}$ score would be
proportional to $e^{-D^{2}/2\sigma^{2}}$.

In order to solve this issue, we propose to first compute the soft skeleton
(first introduced [24]), then apply a smooth diffusion and finally compute the
Dice score between the two diffused skeletons.

Given a binary image $I$, we compute with our smooth diffusion the diffused
image $\text{Dif}(I)$ such that the pixels in $I$ equal to 1 are still equal
to 1 in $\text{Dif}(I)$, and the pixels equal to 0 in $I$ are now equal in
$\text{Dif}(I)$ to a value that depends on the distance to the closest pixel
equal to 1 in $I$. One could think of using the distance function $d$ defined
in Eq. 2, but this is not differentiable. We propose the following algorithm:

Algorithm 1 Smooth binary mask diffusion

procedure smooth_dilate($I,s_{\text{border}},n_{\text{iter}}^{\text{max}},f$)

$s_{\text{dilate}}\leftarrow\max\left(1,s_{\text{border}}//n_{\text{iter}}^{\text{max}}\right)$

$I_{\text{enlarged}}\leftarrow\text{copy}(I)$

for $d\in\\{1,...,\min\left(s_{\text{border}},s_{\text{dilate}}\right)\\}$ do

$I_{\text{enlarged}}^{\text{new}}\leftarrow\textbf{soft\\_dilate}(I_{\text{enlarged}},s_{\text{dilate}})$

$I_{\text{enlarged}}\leftarrow
f*I_{\text{enlarged}}+(1-f)*I_{\text{enlarged}}^{\text{new}}$

end for

return $I_{\text{enlarged}}$

end procedure

where $\textbf{soft\\_dilate}(I,d)$ is a maxpool with a kernel of size
$(1+2d)\times(1+2d)$ with the corners removed.

This introduces 3 parameters of the loss: $s_{\text{border}}$ the size of the
diffused skeleton, $n_{\text{iter}}^{\text{max}}$ the maximum number of
iterations for the diffusion (more iterations take more memory), and $f$ the
factor that controls the decrease. Finally, SKIL-Dice is defined as (for the
sake of simplicity parameters are omitted):

$\text{SKIL}^{\text{Dice}}(P,L)=\text{Dice}(\text{Dif}(S_{\tilde{P}}),\text{Dif}(S_{L}))$
(5)

with $\tilde{P}=T(P,s)$ the prediction threshold. We use a smooth threshold
$T(I,s)$ with $s$ the sharpness:

$T(P,s)=\text{sigmoid}\left(s(P-0.5)\right)$ (6)

####  IV-C2 SKIL-Product

We also propose another variant, SKIL-Product. This version is as similar as
possible to the CL-Dice loss. The predictions and labels are skeletonized,
then diffused and finally the product of the predicted skeleton and the
diffused skeleton of the label is computed. This is done instead of using the
Dice score in order to have a normalization factor of 1.

$g(A,B)=\frac{\sum\left(S_{A}*\text{Dif}(S_{B})\right)+\epsilon}{\sum
S_{A}+\epsilon}$ (7)
$\text{SKIL}^{\text{Prod}}(P,L)=1-\sqrt{g(\tilde{P},L)*g(L,\tilde{P})}$ (8)

##  V Results

We discuss the following questions:

  * •

Are our new metrics visually compelling? For example, given two predictions of
identical Dice scores, does LineAcc capture if one is visually better than the
other?

  * •

Does SKIL improve performance and if yes which variant is better: SKIL-Dice or
SKIL-Product?

  * •

Is SKIL useful in similar segmentation tasks?

  * •

Does SKIL help on poorly annotated images?

###  V-A Dataset collection

The Cracks Reachbot dataset has been open-sourced and is available here.

####  V-A1 Real data

We have collected an initial dataset of 234 rock wall photographs at Pinnacles
National Park, Monument Valley, and Castle Rock using an Intel RealSense D435
camera. From these 234 images, we selected and annotated 100 images that
presented interesting cracks and edges. Figure 3 illustrates some of the
images.

Figure 3: Crack dataset images. The first row shows raw images of cracks, and
the second row shows annotated images.

Figure 4: Examples of images generated using DALL-E 2[39].

####  V-A2 Generated data

In order to add more diverse data, we also generated some images using DALL-E
2 [39]. The prompt used was, ”A natural rock face with no apparent edges,
cracks or holds.”. We asked for no visible edges to have more natural-looking
rock faces (Figure 4).

We generated and annotated an additional 136 images, which we combined into a
dataset of real and generated images composed of 50 real images for evaluation
and 186 images for training, including 50 real and 136 generated ones. In the
rest of this paper, we only consider this augmented dataset as our dataset of
100 real images was too small to evaluate.

###  V-B Metrics Evaluation

For comparison purposes, we define the combined metric,
$\text{LineAcc}_{\text{comb.}}$ (combined):

$\begin{split}\text{LineAcc}_{\text{comb.}}&=2\text{LineAcc}_{\text{pos.}}+0.5\text{LineAcc}_{\text{width}}\\\
&+0.5\text{LineAcc}_{\text{length}}+0.5\text{Dice}+0.5\text{IOU}\end{split}$
(9)

The coefficients were chosen manually to prioritize detecting cracks in the
correct locations, but not necessarily with very precise width or length.

(a) Image

(b) Label

(c) Pred. 1

(d) Pred. 2

Figure 5: Qualitative results: We show two generated predictions with the same
Dice score (absolute error of 1%) but significantly different performance on
$\text{LineAcc}_{\text{comb.}}$. From top to bottom, the Dice scores are 0.6,
0.4, and 0.2. The second prediction has a $\text{LineAcc}_{\text{comb.}}$
score that is $0.23$, $0.29$, and $0.22$ higher than the first, respectively.

To evaluate our new metrics we generated two predictions with the same Dice
score but different $\text{LineAcc}_{\text{comb.}}$ to check which one was
visually better. The predictions were generated using random deformations as
described in Section V-D1. Figure 5 shows that $\text{LineAcc}_{\text{comb.}}$
is less sensitive to the prediction width (first row), to small translations
(second row), and rewards predictions with the correct line length (last row).

###  V-C Loss performance

To answer the question: Does SKIL improve training, we trained a ViT-B network
on the crack dataset and a combined dataset of blood vessels (Section V-A).
The training was run on an NVIDIA RTX A6000 and averaged on 20 runs each for
better precision. We used a linear learning scheduler that reaches the maximum
at 25% of the training. The model was trained for 100 epochs with a maximum
learning rate of $1\times 10^{-5}$ for the cracks dataset while it was trained
for 200 epochs with a maximum learning rate of $5\times 10^{-5}$ for the blood
vessels dataset. All models were fine-tuned with the Adam[40] optimizer from a
pre-trained model. For all experiments, the parameters used for the SKIL loss
(both Dice and Product) were $s_{\text{border}}=20$,
$n_{\text{iter}}^{\text{max}}=50$, $f=0.82$ and $s=10$. These parameters were
qualitatively chosen and manually tuned to offer the behavior we wanted, and
parameter search optimization will be part of future work.

####  V-C1 Quantitative results

For each training, we used a mixture of the Dice loss with a coefficient of 3
and the studied loss with a coefficient of 1 (For the Dice-only training we,
therefore, had a coefficient of 4). In the tables below we will only reference
the studied loss even though the Dice loss was also used for all of them.

| Classical |  | LineAcc
---|---|---|---
| IOU | Dice |  | Pos. | Width | Length |  | Combined
ab Dice | 0.255 | 0.396 ± 0.001 |  | 0.555 | 0.608 | 0.726 |  | 0.548 ± 0.001
ab CL-Dice | 0.253 | 0.394 ± 0.001 |  | 0.548 | 0.560 | 0.712 |  | 0.536 ± 0.001
ab SKIL-Dice | 0.261 | 0.404 ± 0.001 |  | 0.570 | 0.648 | 0.764 |  | 0.569 ± 0.001
ab SKIL-Prod. | 0.261 | 0.403 ± 0.001 |  | 0.560 | 0.627 | 0.721 |  | 0.553 ± 0.001

TABLE I: Quantitative results: Comparison of different metrics on the cracks
dataset (averaged over 20 runs) after training for 100 epochs with 4 losses:
Dice, CL-Dice, SKIL-Dice (This Work) and SKIL-Product (This Work). The
standard error is reported for the Dice and $\text{LineAcc}_{\text{comb.}}$
score.

| Classical |  | LineAcc
---|---|---|---
| IOU | Dice |  | Pos. | Width | Length |  | Combined
ab Dice | 0.264 | 0.400 ± 0.001 |  | 0.746 | 0.631 | 0.643 |  | 0.618 ± 0.001
ab CL-Dice | 0.270 | 0.410 ± 0.001 |  | 0.745 | 0.584 | 0.657 |  | 0.618 ± 0.001
ab SKIL-Dice | 0.257 | 0.389 ± 0.001 |  | 0.823 | 0.685 | 0.746 |  | 0.679 ± 0.001
ab SKIL (Prod) | 0.259 | 0.391 ± 0.001 |  | 0.803 | 0.665 | 0.711 |  | 0.661 ± 0.001

TABLE II: Quantitative results: Comparison of different metrics on the vessels
datasets (aggregation of DRIVE, HRF, STARE, CHASE-DB1) (averaged over 20 runs)
after training for 200 epochs with 4 losses: Dice, CL-Dice, SKIL-Dice (This
Work) and SKIL-Product (This Work). The standard error is reported for the
Dice and $\text{LineAcc}_{\text{comb.}}$ score.

Table I shows that SKIL-Dice and SKIL-Product both improve the Dice score and
$\text{LineAcc}_{\text{comb.}}$ score. The results of both SKIL variants are
close, but SKIL-Dice offers slightly better performances, so in the rest of
this paper we will only consider SKIL-Dice and refer to it as SKIL.

Table II shows that for the blood vessels dataset, SKIL slightly decreases the
Dice score but the Combined score is substantially increased. We report visual
predictions in the qualitative results later, Section V-C3.

####  V-C2 Qualitative results on cracks

Figure 6 shows the qualitative results obtained on different crack images.
From the top, images (1) – (3) show that SKIL detects more details, with fewer
gaps in predictions, and image (4) shows a more accurate predicted width.
Image (5) is trained after 10 epochs and illustrates the sample efficiency
with SKIL.

(a) Image

(b) Label

(c) Dice

(d) CL-Dice

(e) SKIL

Figure 6: Qualitative results: From the top, we show results on 4 crack images
(after 100 epochs) and one result on the last crack (after 10 epochs). With
SKIL, the predicted cracks are more connected and the network is able to learn
much faster (see last row). Visual improvements are marked by red arrows.

####  V-C3 Qualitative results on blood vessels

Figure 7 illustrates how much better the model performs with SKIL (SKIL-Dice).
The first two rows show that when trained with SKIL, the network is able to
detect thin vessels that are not detected when trained with just Dice or CL-
Dice. In addition, the third row also shows that even for large vessels, SKIL
does a better job at predicting the correct width. For example, in the third
row from the top, the middle diagonal vessel is actually two parallel vessels
which are mostly fused when trained with Dice or CL-Dice, but cleanly
separated when trained with SKIL. Finally, the last row illustrates the fact
that SKIL enables the network to converge faster. After only 10% of its
training, the network identifies more vessels with SKIL.

(a) Image

(b) Label

(c) Dice

(d) CL Dice

(e) SKIL

Figure 7: Qualitative results: From the top, we show results from HRF, STARE,
CHASE-DB1 (after 200 epochs), and finally CHASE-DB1 (after 20 epochs). The
predictions show that with SKIL (SKILL-Dice), the vessels are thinner and more
detailed (Dice and CL-Dice tend to miss small vessels). Visual improvements
are marked by red arrows.

####  V-C4 SKIL with U-Net

The quantitative results in Table II are not state of the art, so we further
investigate the choice of the ViT-B[41] architecture for our crack detection
problem given that the U-Net[26] architecture performs better for blood vessel
segmentation (See Table IV). We show that the U-Net architecture does not
perform as well on the crack segmentation task across the four losses. We
believe this is because while rock crack detection and blood vessel detection
are similar tasks for benchmarking, they still have nuanced differences, such
as blood vessels having a more radial configuration as they emerge from a
central optic nerve [42]. Figure 8 and Table III report U-Net results.

Figure 8: Predictions (bottom row) from U-Net versus ground truth (top row) on
4 images. From left, in images (1) and (2) the model detects all the lines
including lines that are not cracks; in (3) predicts patches rather than
cracks; and in (4) predicts correct position but incorrect crack width.

For completeness, a comparison of the losses is reported in Table III and IV.
The networks are trained from scratch with a maximum learning rate of $1\times
10^{-2}$ for $40000$ steps (approximately $2000$ epochs depending on the size
of the dataset) using the Adam[40] optimizer. While the results slightly favor
SKIL, it is not possible to conclude statistical significance given the
relative error.

| Classical |  | LineAcc
---|---|---|---
| IOU | Dice |  | Pos. | Width | Length |  | Combined
ab Dice | 0.093 | 0.166 ± 0.003 |  | 0.397 | 0.218 | 0.640 |  | 0.372 ± 0.003
ab CL-Dice | 0.116 | 0.203 ± 0.003 |  | 0.393 | 0.374 | 0.625 |  | 0.391 ± 0.003
ab SKIL-Dice | 0.109 | 0.192 ± 0.002 |  | 0.400 | 0.249 | 0.610 |  | 0.374 ± 0.004
ab SKIL-Prod. | 0.106 | 0.188 ± 0.003 |  | 0.403 | 0.227 | 0.620 |  | 0.375 ± 0.004

TABLE III: Metrics when training a U-Net architecture (averaged over 10 runs)
on the cracks dataset with 4 losses: Dice, CL-Dice, SKIL-Dice (This Work) and
SKIL-Product (This Work). The standard error is reported for the Dice and
$\text{LineAcc}_{\text{comb.}}$ score. U-Net performs worse on the cracks
dataset across all 4 losses compared to Table I.

| STARE |  | DRIVE
---|---|---|---
| Dice | LineAcc-Comb. |  | Dice | LineAcc-Comb.
ab Cross Entropy | 0.8200 ± 0.0008 | 0.8932 ± 0.0013 |  | 0.7684 ± 0.0009 | 0.8680 ± 0.0009
ab CL-Dice | 0.8204 ± 0.0009 | 0.8860 ± 0.0009 |  | 0.7750 ± 0.0002 | 0.8639 ± 0.0008
ab SKIL-Dice | 0.8175 ± 0.0009 | 0.9024 ± 0.0014 |  | 0.7679 ± 0.0009 | 0.8868 ± 0.0008
ab SKIL-Product | 0.8205 ± 0.0008 | 0.9027 ± 0.0011 |  | 0.7718 ± 0.0005 | 0.8850 ± 0.0008

TABLE IV: Comparison of different loss scores trained on a U-Net architecture
(averaged over 10 runs) and standard errors when trained separately on STARE
and DRIVE (HRF and CHASE DB1 have been omitted for readability). The loss used
consisted of the Dice loss (coefficient of 3) and the loss in this Table
(coefficient of 1).

###  V-D Generalization on poor quality annotations

One of the motivations of SKIL and $\text{LineAcc}_{\text{pos}}$ was to
abstract the width of the cracks from the training and evaluation. This was
important as the true width of a graspable crack is subjective (to the point
that it was not always consistent between labelers). More generally, given the
stochasticity of grasping with microspine grippers, crack detection for
robotic grasping may always be poorly annotated. To test SKIL’s performance on
poorly annotated data, we created some random deformations to the blood vessel
labels, trained on them and evaluated them on clean labels.

####  V-D1 Generation of poorly annotated blood vessels

We created 3 deformations (Figure 9): a simple shift, a random width change
that changes the width of the vessels but does not create or remove any
vessels, and a random branch cutter which removes vessels with a probability
inversely proportional to the thickness. Below we explain the latter two.

To change the width of the vessels, given a binary image $I$, we first compute
the skeleton $\mathcal{S}_{I}$ as well as the distance to the closest pixel
equal to 0 at each pixel, noted $D_{I}^{0}$ (Analog to $d$ in Section IV-B1).
Then we multiply $D_{I}^{0}$ by some Perlin noise [43] $\mathcal{N}^{P}$ to
finally apply a decreasing dilation which consists of iteratively applying
some maxpool of size $3\times 3$ and subtract 1 each time to the pixels that
were equal to 0 before the pooling. This way, a pixel of value $x$ will
generate a diamond of width $x$.

$\mathcal{D}_{\text{width}}(I)=\left(\textbf{decrease\\_dilate}(D_{I}^{0}*\mathcal{S}_{I}*\mathcal{N}^{P})\right)>0$
(10)

The parameters chosen for the Perlin noise were: amplitude between $0.1$ and
$2.0$, resolution of $3$, $4$ octaves with a persistence of $0.5$, and a
lacunarity of $3$.

The random branch cutter also relies on the same idea, however instead of
multiplying the distance values $D_{I}^{0}$ by some Perlin noise, the values
are changed to 0 based on a probability $\mathcal{P}$ computed as follows:

$\mathcal{P}(I)=\mathcal{N}^{P}*\left(\frac{\sum\mathcal{S}_{I}*D_{I}^{0}}{\sum\mathcal{S}_{I}}*\frac{1}{\epsilon+D_{I}^{0}}\right)^{\alpha}$

where $\epsilon=1\times 10^{-3}$ is a constant to avoid dividing by zero and
$\alpha$ is the selectiveness, set to $0.2$ in our experiments. A pixel is
then set to 0 if $\mathcal{P}(I)>1-p$ with $p=0.35$. All the parameters for
the Perlin noise are identical.

(a) Shift

(b) Random Width

(c) Branch Crop

(d) Combination

Figure 9: Deformed annotations on the DRIVE dataset. The first row shows the
original annotation; following rows show deformation examples (ordered by
column). The last column is a combination of the shift, random width, and
branch crop deformations, with a probability of 0.5 for each deformation.
Examples of visual changes are marked by red arrows. (The last column is
unmarked as the changes are visible enough.)

####  V-D2 Quantitative results

Table V summarizes the results obtained with the four different deformations
and shows the performance of each loss when the model is trained on the
deformed data, and Table VI helps to interpret these results by reporting
relative performance between the network trained on the clean data and
deformed data. We see that SKIL substantially improves performance on shifted
labels deformations due to its soft dilation. The results for the random width
deformation show that even with the clean data the network does not perform
well regarding the width prediction as the deformation does not deteriorate
this metric at all. The cropped branches deformation seems to indicate that
SKIL slightly performs better when some annotations are missing but the
difference is too small to generalize. With combined deformations, SKIL
outperforms CL-Dice and Dice so we conclude that SKIL does adapt to poorly
annotated data.

| Classical |  | LineAcc
---|---|---|---
| IOU | Dice |  | Pos. | Width | Length |  | Combined
Shifted labels |  |  |  |
ab Dice | 0.215 | 0.349 ± 0.002 |  | 0.485 | 0.490 | 0.504 |  | 0.444 ± 0.002
ab CL-Dice | 0.221 | 0.359 ± 0.002 |  | 0.420 | 0.441 | 0.481 |  | 0.407 ± 0.002
ab SKIL-Dice | 0.210 | 0.340 ± 0.003 |  | 0.751 | 0.564 | 0.700 |  | 0.618 ± 0.002
Random width |  |  |  |
ab Dice | 0.276 | 0.419 ± 0.001 |  | 0.778 | 0.574 | 0.678 |  | 0.638 ± 0.001
ab CL-Dice | 0.274 | 0.418 ± 0.001 |  | 0.776 | 0.541 | 0.688 |  | 0.635 ± 0.001
ab SKIL-Dice | 0.268 | 0.406 ± 0.001 |  | 0.831 | 0.616 | 0.783 |  | 0.687 ± 0.001
Cropped branches |  |  |  |
ab Dice | 0.267 | 0.407 ± 0.001 |  | 0.682 | 0.581 | 0.599 |  | 0.576 ± 0.001
ab CL-Dice | 0.274 | 0.417 ± 0.001 |  | 0.671 | 0.549 | 0.605 |  | 0.571 ± 0.001
ab SKIL-Dice | 0.264 | 0.401 ± 0.001 |  | 0.795 | 0.637 | 0.689 |  | 0.651 ± 0.001
Combined deformations |  |  |  |
ab Dice | 0.271 | 0.419 ± 0.001 |  | 0.653 | 0.507 | 0.577 |  | 0.551 ± 0.001
ab CL-Dice | 0.269 | 0.418 ± 0.001 |  | 0.623 | 0.475 | 0.568 |  | 0.532 ± 0.001
ab SKIL-Dice | 0.266 | 0.410 ± 0.001 |  | 0.798 | 0.558 | 0.718 |  | 0.651 ± 0.001

TABLE V: Different metrics on the vessels datasets (aggregation of DRIVE, HRF,
STARE, CHASE-DB1) (averaged over 10 runs) when training for 200 epochs with 3
losses: Dice, CL-Dice and SKIL (This Work). Standard error is reported. The
experiments were run on 4 types of deformations. Combined deformation is a
combination of shift, random width, and branch crop with a probability of 0.75
for each deformation.

| Classical |  | LineAcc
---|---|---|---
| IOU | Dice |  | Pos. | Width | Length |  | Combined
Shifted labels |  |  |  |
ab Dice | 81.4% | 87.3 ± 0.7% |  | 65.0% | 77.7% | 78.4% |  | 71.8 ± 0.3%
ab CL-Dice | 81.9% | 87.6 ± 0.4% |  | 56.4% | 75.5% | 73.2% |  | 65.9 ± 0.3%
ab SKIL-Dice | 81.7% | 87.4 ± 0.7% |  | 91.25% | 82.3% | 93.8% |  | 91.0 ± 0.2%
Random width |  |  |  |
ab Dice | 104.5% | 104.8 ± 0.2% |  | 104.3% | 91.0% | 105.4% |  | 103.2 ± 0.1%
ab CL-Dice | 101.5% | 102.0 ± 0.1% |  | 104.2% | 92.6% | 104.7% |  | 102.8% ± 0.1%
ab SKIL-Dice | 104.3% | 104.4 ± 0.1% |  | 101.0% | 89.9% | 105.0% |  | 101.2 ± 0.1%
Cropped branches |  |  |  |
ab Dice | 101.1% | 101.8 ± 0.1% |  | 91.4% | 92.1% | 93.2% |  | 93.2 ± 0.2%
ab CL-Dice | 101.5% | 101.7 ± 0.1% |  | 90.1% | 94.0% | 92.1% |  | 92.4 ± 0.1%
ab SKIL-Dice | 102.7% | 103.1 ± 0.1% |  | 96.6% | 93.0% | 92.4% |  | 95.9 ± 0.1%
Combined derformations |  |  |  |
ab Dice | 102.7% | 104.8 ± 0.2% |  | 87.5% | 80.3% | 89.7% |  | 89.2 ± 0.2%
ab CL-Dice | 99.3% | 102.0 ± 0.2% |  | 83.6% | 81.3% | 86.5% |  | 86.1 ± 0.2%
ab SKIL-Dice | 103.5% | 105.4 ± 0.2% |  | 97.0% | 81.5% | 96.2% |  | 95.9 ± 0.1%

TABLE VI: Relative performance of Table V. For a given method and metric, the
score trained on the deformed data is divided by the score trained on the
clean data.

##  VI Conclusion

In this work, we present SKIL, a skeleton-based segmentation loss to identify
cracks on rock walls, and evaluate its performance on both rock cracks and a
similar task, blood vessel detection. We also present three novel metrics
(LineAcc) for evaluating this method. Finally, we publish a new dataset
containing 236 images of rocks with graspable edges and cracks annotated.
Experiments show that SKIL and the LineAcc metrics perform well on our dataset
of rock cracks and on several existing datasets for blood vessel detection.

In this work, we take the first step to developing an accurate learning-based
approach for visual rock crack detection to enable a climbing robot. Next
steps include parameter finetuning, deployment on hardware for field testing
with a partial robot system, and interfacing with motion planning systems to
enable full autonomy.

## Acknowledgements

Support for this work was provided by NASA under the Innovative Advanced
Concepts (NIAC) program.

## References

  * [1] S. Schneider, A. Bylard, T. G. Chen, P. Wang, M. R. Cutkosky, and M. Pavone, “ReachBot: A small robot for large mobile manipulation tasks,” 2022.
  * [2] S. Newdick, T. G. Chen, B. Hockman, E. Schmerling, M. R. Cutkosky, and M. Pavone, “Designing reachbot: System design process with a case study of a martian lava tube mission,” 2023.
  * [3] S. Newdick, N. Ongole, T. G. Chen, E. Schmerling, M. Cutkosky, and M. Pavone, “Motion planning for a climbing robot with stochastic grasps,” 2023.
  * [4] A. T. Asbeck, S. Kim, M. R. Cutkosky, W. R. Provancher, and M. Lanzetta, “Scaling hard vertical surfaces with compliant microspine arrays,” _The International Journal of Robotics Research_ , vol. 25, no. 12, pp. 1165–1179, 2006.
  * [5] S. Ekaireb, M. Ali, P. Pathuri, P. Haresh Batia, R. Sharma, and N. Manjunath-Murkal, “Computer vision based indoor rock climbing analysis,” 2022.
  * [6] M. Richardson, K. Petrini, and M. Proulx, “Climb-o-vision: A computer vision driven sensory substitution device for rock climbing,” Apr. 2022.
  * [7] D. Pandurevic, P. Draga, A. Sutor, and K. Hochradel, “Analysis of competition and training videos of speed climbing athletes using feature and human body keypoint detection algorithms,” _Sensors_ , vol. 22, no. 6, p. 2251, 2022.
  * [8] X. Sang, “Rock classiﬁcation from field image patches analyzed using a deep convolutional neural network,” _Mathematics_ , 2019.
  * [9] W. Chen, L. Su, X. Chen, and Z. Huang, “Rock image classiﬁcation using deep residual neural network with transfer learning,” _Front. Earth Sci. 10:1079447_ , 2023.
  * [10] X. Ran, L. Xue, Y. Zhang, Z. Liu, X. Sang, and J. He, “Rock classification from field image patches analyzed using a deep convolutional neural network,” _Mathematics_ , vol. 7, no. 8, p. 755, 2019.
  * [11] J. P. Lacassie, J. R. Del Solar, B. Roser, and F. Hervé, “Visualization of volcanic rock geochemical data and classification with artificial neural networks,” _Mathematical Geology_ , vol. 38, pp. 697–710, 2006.
  * [12] R. Ali, J. H. Chuah, M. S. A. Talip, N. Mokhtar, and M. A. Shoaib, “Structural crack detection using deep convolutional neural networks,” _Automation in Construction_ , vol. 133, p. 103989, 2022.
  * [13] D. Ai, G. Jiang, S.-K. Lam, P. He, and C. Li, “Computer vision framework for crack detection of civil infrastructure—a review,” _Engineering Applications of Artificial Intelligence_ , vol. 117, p. 105478, 2023.
  * [14] A. Parness, N. Abcouwer, C. Fuller, N. Wiltsie, J. Nash, and B. Kennedy, “Lemur 3: A limbed climbing robot for extreme terrain mobility in space,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 5467–5473.
  * [15] W. Ruotolo, F. S. Roig, and M. R. Cutkosky, “Load-sharing in soft and spiny paws for a large climbing robot,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1439–1446, 2019.
  * [16] S. Wang, H. Jiang, T. Myung Huh, D. Sun, W. Ruotolo, M. Miller, W. R. Roderick, H. S. Stuart, and M. R. Cutkosky, “Spinyhand: Contact load sharing for a human-scale climbing robot,” _Journal of Mechanisms and Robotics_ , vol. 11, no. 3, p. 031009, 2019.
  * [17] H. Jiang, S. Wang, and M. R. Cutkosky, “Stochastic models of compliant spine arrays for rough surface grasping,” _The International Journal of Robotics Research_ , vol. 37, no. 7, pp. 669–687, 2018.
  * [18] S. Iacoponi, M. Calisti, and C. Laschi, “Simulation and analysis of microspines interlocking behavior on rocky surfaces: an in-depth study of the isolated spine,” _Journal of Mechanisms and Robotics_ , vol. 12, no. 6, p. 061016, 2020.
  * [19] S. Zhou, C. Canchila, and W. Song, “Deep learning-based crack segmentation for civil infrastructure: data types, architectures, and benchmarked performance,” _Automation in Construction_ , vol. 146, p. 104678, 2023.
  * [20] L. Minh Dang, H. Wang, Y. Li, L. Q. Nguyen, T. N. Nguyen, H.-K. Song, and H. Moon, “Deep learning-based masonry crack segmentation and real-life crack length measurement,” _Construction and Building Materials_ , vol. 359, p. 129438, 2022.
  * [21] P. M. Samuel and T. Veeramalai, “Vssc net: Vessel specific skip chain convolutional network for blood vessel segmentation,” _Computer Methods and Programs in Biomedicine_ , vol. 198, p. 105769, 2021.
  * [22] M. K. Kar, D. R. Neog, and M. K. Nath, “Retinal vessel segmentation using multi-scale residual convolutional neural network (msr-net) combined with generative adversarial networks,” _Circuits, Systems, and Signal Processing_ , vol. 42, no. 2, pp. 1206–1235, Feb 2023.
  * [23] Y. Zhang, M. He, Z. Chen, K. Hu, X. Li, and X. Gao, “Bridge-net: Context-involved u-net with patch-based loss weight mapping for retinal blood vessel segmentation,” _Expert Systems with Applications_ , vol. 195, p. 116526, 2022.
  * [24] S. Shit, J. C. Paetzold, A. Sekuboyina, A. Zhylka, I. Ezhov, A. Unger, J. P. W. Pluim, G. Tetteh, and B. H. Menze, “cldice - a topology-preserving loss function for tubular structure segmentation,” _CoRR_ , vol. abs/2003.07311, 2020.
  * [25] G.-X. Xu and C.-X. Ren, “Spnet: A novel deep neural network for retinal vessel segmentation based on shared decoder and pyramid-like loss,” _Neurocomputing_ , vol. 523, pp. 199–212, 2023.
  * [26] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [27] Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4_. Springer, 2018, pp. 3–11.
  * [28] H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y.-W. Chen, and J. Wu, “Unet 3+: A full-scale connected unet for medical image segmentation,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 1055–1059.
  * [29] M. H. Hesamian, W. Jia, X. He, and P. Kennedy, “Deep learning techniques for medical image segmentation: achievements and challenges,” _Journal of digital imaging_ , vol. 32, pp. 582–596, 2019.
  * [30] A. A. Taha and A. Hanbury, “Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool,” _BMC Med. Imaging_ , vol. 15, no. 1, p. 29, Aug. 2015.
  * [31] O. U. Aydin, A. A. Taha, A. Hilbert, A. A. Khalil, I. Galinovic, J. B. Fiebach, D. Frey, and V. I. Madai, “On the usage of average hausdorff distance for segmentation performance assessment: hidden error when used for ranking,” _Eur. Radiol. Exp._ , vol. 5, no. 1, p. 4, Jan. 2021.
  * [32] J. H. Liew, S. Cohen, B. Price, L. Mai, and J. Feng, “Deep interactive thin object selection,” in _Winter Conference on Applications of Computer Vision (WACV)_ , 2021.
  * [33] T. G. Chen, S. Newdick, J. Di, C. Bosio, N. Ongole, M. Lapôtre, M. Pavone, and M. Cutkosky, “Locomotion as manipulation: Grasp analysis for reachbot,” _Science Robotics_ , 2023.
  * [34] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: transformers for image recognition at scale,” _arXiv:2010.11929v2_ , 2021.
  * [35] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun, “Unified perceptual parsing for scene understanding,” _CoRR_ , vol. abs/1807.10221, 2018.
  * [36] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” _arXiv:2304.02643_ , 2023.
  * [37] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015.
  * [38] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou, “Training data-efficient image transformers distillation through attention,” in _International Conference on Machine Learning_ , vol. 139, July 2021, pp. 10 347–10 357.
  * [39] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” 2022.
  * [40] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017.
  * [41] A. DosoViTskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [42] H. Kolb, _Simple Anatomy of the Retina_. University of Utah Health Sciences Center, Salt Lake City (UT), 1995.
  * [43] K. Perlin, “Improving noise,” in _Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques_ , ser. SIGGRAPH ’02. New York, NY, USA: Association for Computing Machinery, 2002, p. 681–682.