# SemiCurv: Semi-Supervised Curvilinear Structure Segmentation

Xun Xu, Manh Cuong Nguyen, Yasin Yazici, Kangkang Lu, Hlaing Min, Chuan-Sheng
Foo X. Xu, C. Nguyen, Y. Yazici, K. Lu, H. Min, and C.S. Foo are with the
Institute for Infocomm Research (I2R), A-STAR, Singapore. e-mail:
<EMAIL_ADDRESS>
###### Abstract

Recent work on curvilinear structure segmentation has mostly focused on
backbone network design and loss engineering. The challenge of collecting
labelled data, an expensive and labor intensive process, has been overlooked.
While labelled data is expensive to obtain, unlabelled data is often readily
available. In this work, we propose SemiCurv, a semi-supervised learning (SSL)
framework for curvilinear structure segmentation that is able to utilize such
unlabelled data to reduce the labelling burden. Our framework addresses two
key challenges in formulating curvilinear segmentation in a semi-supervised
manner. First, to fully exploit the power of consistency based SSL, we
introduce a geometric transformation as strong data augmentation and then
align segmentation predictions via a differentiable inverse transformation to
enable the computation of pixel-wise consistency. Second, the traditional mean
square error (MSE) on unlabelled data is prone to collapsed predictions and
this issue exacerbates with severe class imbalance (significantly more
background pixels). We propose a N-pair consistency loss to avoid trivial
predictions on unlabelled data. We evaluate SemiCurv on six curvilinear
segmentation datasets, and find that with no more than $5\%$ of the labelled
data, it achieves close to $95\%$ of the performance relative to its fully
supervised counterpart. The demo code is available at the project website
111https://github.com/alex-xun-xu/SemiCurv.

###### Index Terms:

semi-supervised learning, semantic segmentation

##  I Introduction

Curvilinear structure segmentation aims to extract thin, curvilinear
structures from images. It has wide applications, including road crack
segmentation, road segmentation from satellite images and biomedical image
segmentation (e.g. segmenting blood vessels and cells). Existing research into
curvilinear structure segmentation has focused on designing better network
architectures [1, 2] or loss functions (e.g. topology loss [3, 4]) to better
account for the specific nature of curvilinear structures like connectivity
and loop formation.

A major challenge in curvilinear structure segmentation that has received less
attention is the need for a large amount of labelled data to obtain good
performance. In many cases, acquiring such labelled data for segmentation
tasks is non-trivial and sometimes requires expert labelling. Moreover, the
fine detail present in curvilinear structures demands increased annotation
effort in comparison to the coarse-grained polygon-based annotation that is
sufficient for general (object-based) segmentation tasks. At the same time, we
often have access to unlabelled data at very low cost. For example, road
scanning vehicles are running everyday to collect road surface photographs,
satellites are collecting aerial images on a daily basis, and medical images
are abundant from daily diagnoses. This availability of unlablled data
suggests the use of semi-supervised learning (SSL) approaches that are able to
exploit the unlabelled data to reduce the amount of labelled data required.

Much work in SSL has focused on semi-supervised classification [5], and far
fewer works have studied the use of SSL for image segmentation, much less
curvilinear structure segmentation. Existing SSL methods for semantic
segmentation adopt one of two approaches: 1) utilizing a Generative
Adversarial Network (GAN) to enforce that predicted segmentation maps are
indistinguishable from real segmentation maps [6, 7] or 2) enforcing
consistency of predicted segmentation maps on unlabelled data between two
augmented samples [8, 9, 10]. The first strategy still requires a considerable
amount of labelled data to train the GAN. In contrast, consistency-based
approaches are able to work with much less labelled data and are more
promising in the low-label regime. Therefore, we build upon the consistency-
based approach towards semi-supervised segmentation in this work.

Even though previous work has developed methods for semi-supervised
segmentation, we argue that direct application of these methods to the
curvilinear segmentation task is sub-optimal for the following two reasons.
First, the types of data augmentation used by state-of-the-art SSL methods for
generic segmentation tasks has thus far been limited to pixel-wise
perturbations, including a mix-up of multiple images [9, 10]. Pixel-wise
perturbations do not create more diverse geometric views of image data (e.g.
vertical cracks will always remain vertical under pixel-wise perturbations),
and thus may not fully exploit the power of consistency based SSL. Moreover,
mixing up images could shift the data away from the real data manifold (i.e.
the mixed-up image may not look real), potentially harming SSL as demonstrated
in our experiments. Secondly, curvilinear structures typically occupy a small
fraction (often under 10% and as little as 1% in the datasets we considered)
of the image compared to the background, leading to a severely class-
imbalanced segmentation target; this is unlike the more general semantic
segmentation datasets [11] considered by these existing methods. As we will
show, the class imbalance and sparsity leads to a “collapsing” issue with the
consistency loss used by SSL methods, resulting in models predicting constant
outputs.

We next describe the key improvements we make to consistency-based SSL to
address these challenges associated specifically with curvilinear segmentation
tasks:

_1) Differentiable affine transformation._ First, we propose affine
transformations as data augmentation for curvilinear images. This augmentation
is effective because of two reasons. First, images with curvilinear structures
are often captured from planar surfaces; this is characteristic of images of
road surfaces, satellite images, and images of cells taken under a microscope,
to give a few examples. Hence, synthesizing novel viewpoints as augmentation
by affine transformation is valid [12]. Second, segmentation of curvilinear
structures is expected to be affine equivariant: for instance, the notion of a
crack on a road surface should remain the same regardless of rotation,
translation or resizing of the image, unlike in semantic segmentation tasks
where the pose of an semantic object is often fixed or subject to small
variation.

One challenge with using affine transformations as data augmentation is that
despite being more diverse, they prohibit computing consistency loss at the
pixel level between two randomly augmented images because pixel-wise
correspondence is not preserved during affine transformation. As illustrated
in Fig. 1, supposing $\mathbf{\hat{I}}$ and $\mathbf{\bar{I}}$ are randomly
augmented from $\mathbf{I}$, there is no explicit pixel-wise correspondence
between $\mathbf{\hat{I}}$ and $\mathbf{\bar{I}}$ with which we can enforce
the predictions to be consistent. To address this alignment issue, we propose
to employ an inverse affine transformation following the networks’ predictions
as illustrated in Fig. 2. With both forward and inverse transformations, the
pixel-to-pixel correspondence is restored, enabling the computation of the
pixel-wise consistency loss between student and teacher models. We note that
incorporating the inverse transformation into the networks requires it to be
differentiable for gradients to backpropagate; as affine transformation is
differentiable, the inverse transformation allows end-to-end learning as part
of the networks.

Figure 1: Illustration of aligning augmented images. Two random augmentations
are applied to original image $\mathbf{I}$, resulting in $\mathbf{\hat{I}}$
and $\mathbf{\tilde{I}}$. As pixel-to-pixel correspondence is no longer
available between $\mathbf{\hat{I}}$ and $\mathbf{\tilde{I}}$, we apply
additional inverse transformation to align two augmented images.

_2) N-pair consistency._ As noted above, we observe that the widely adopted
consistency loss, mean square error (MSE), is prone to a “collapsing” issue.
Trivial solutions to minimize the MSE on unlabelled data exist where both
student and teacher networks predict constant outputs. The introduction of a
non-learnable teacher model may partially address this issue, but cannot
guarantee in principle that a trivial solution is not learned. Moreover, we
notice that there are often far fewer positive (foreground) pixels than
negative ones (background) in curvilinear structure datasets, suggesting the
prior distribution is highly biased towards background. This imbalance further
encourages the trivial solution where the learned model predicts all
background on the unlabelled data. To overcome this challenge, we propose to
use an N-pair loss [13] to avoid trivial predictions on unlabelled data,
inspired by the recent success of contrastive learning [14, 15].

_3) Spatial coordinate encoding._ Finally, there is clear spatial connectivity
and correlation in curvilinear structure patterns, in that positive pixels are
spatially adjacent. Although topology was studied to improve connectivity [4,
3], they require fully labelled ground-truth to provide topological
supervision. To exploit the spatial correlation prior, we propose to add
spatial coordinate encoding as features to all layers of the backbone
segmentation network. To avoid the potential overfitting to absolute spatial
coordinates [16], we encode spatial adjacency using a sinusoid function, which
preserves spatial adjacency information while reducing risk of overfitting.

In summary, our work makes the following contributions:

  * •

We introduce a differentiable affine transformation to fully exploit the
strength of consistency based semi-supervised learning for curvilinear image
segmentation.

  * •

We provide insight into a “collapsing” issue with MSE consistency loss defined
on unlabelled data and propose to use N-pair loss to mitigate this issue.

  * •

To further capture the spatial correlation, we incorporate sinusoid spatial
encoding as additional features.

  * •

To the best of our knowledge, this is the first attempt to investigate into
curvilinear structure segmentation in a semi-supervised fashion. We
extensively benchmarked state-of-the-art semi-supervised methods on six
curvilinear segmentation datasets.

##  II Related Work

###  II-A Curvilinear Structure Segmentation

We briefly review previous work on curvilinear structure segmentation in the
context of road segmentation from satellite images, medical image segmentation
and crack detection in pavements. Early work in road segmentation [17] used
fully connected networks, but were soon followed by CNNs [18]. Recent work has
built on the CNN approach, focusing on designing deeper and wider networks and
considering information at multiple scales [19, 20]. In the medical imaging
community, efforts have focused on further improving the popular UNet backbone
[21, 22]. These fully supervised methods have shown to be competitive in
delineating linear structures like membrane segmentation [23]. Progress in the
area of crack segmentation proceeded in a similar trajectory, where recent
progress is due to improved backbone network design [24, 1]. Orthogonal to the
above, in this work, we address the semi-supervised learning setting to reduce
the amount of labelled data needed to train high performance models by
exploiting unlabelled data.

To model spatial correlation and connectivity in curvilinear structures, a few
works have incorporated topological constraints when training the segmentation
model. One study utilized intermediate features from a VGG network to capture
topological structure [4], while another work used concepts from persistent
homology to provide more principled topological features; this idea was
implemented by defining a loss between ground-truth and predicted persistent
diagrams [3], which assumes access to ground-truth labels. In this work, we
adopt a simpler approach by using positional encodings to capture spatial
correlation.

###  II-B Semi-Supervised Learning

We briefly review the two genres of semi-supervised learning (SSL) methods,
namely the consistency-based SSL and adversarial training based SSL; for a
more complete review, please refer to [5]. Consistency-based SSL methods [25,
26, 27] utilize the unlabelled data to constrain learning by enforcing the
model’s predictions on the unlabelled data to be consistent with a specified
target. Consistency regularization was firstly introduced in the context of
deep learning by [28]. To better exploit the power of ensemble learning [25]
used temporal ensembling of models’ predictions as a target for consistency.
[26] further proposed to ensemble the model parameters over time to obtain
more stable consistency targets. Different from the previously mentioned
approaches, the VAT method [27] proposed to enforce consistency of a model’s
predictions with that on an adversarially perturbed input [29]. Orthogonal to
the above, the MixMatch method [30] proposed to interpolate between labelled
and unlabelled samples to diversify the training set. Due to the high accuracy
of temporal ensembling and low memory footprint, we adopted Mean Teacher [26]
as the backbone framework. Another line of works adapt generative adversarial
training to semi-supervised learning. Specifically, in the seminal work [31],
a discriminator was introduced to distinguish generated samples from real ones
and at the same time classify real samples into respective categories.

###  II-C Semi-Supervised Segmentation

Semi-Supervised learning has been applied to semantic image segmentation to
alleviate the expensive image annotation task. Following the adversarial
training approach [31], [32] first employed a discriminator to distinguish
generated fake images from real ones to improve semi-supervised learning. A
conditional generative adversarial network (GAN) was further employed by
ensuring that a discriminator cannot distinguish predicted segmentation masks
on unlabelled images from ground-truth segmentation masks (on the labelled
images) [6]; [7] further imposes cycle consistency (building on CycleGAN) to
constrain the learning on unlabelled data. However, under low labeled data
training a GAN is prone to instability and may not generalize well to
unlabeled data. Following the consistency-based SSL approaches, [9] proposed
to enforce consistency between the segmentation predictions of images subject
to a MixUp augmentation, i.e. random crops of the two images are superimposed
and consistency is enforced on the segmentation predictions of respective
patches. In the context of medical image segmentation, a framework similar to
Mean Teacher has been applied to skin lesion and CT-scan segmentation [33].
All these existing methods only consider data augmentations that are either
pixel-wise perturbations [9] or rotations that are multiples of $90^{\circ}$
[33] with pairwise consistency; such augmentations are not strong enough to
fully exploit the power of consistency regularization. These works also do not
address the issue where the MSE loss used to enforce consistency is prone to
collapse.

Figure 2: Overview of SemiCurv, a semi-supervised learning framework for
curvilinear structure segmentation. We adopt a mean-teacher like architecture
for semi-supervised learning. i) We first propose a geometric transformation
and the inverse transformation to allow stronger data augmentation. ii) To
account for the severe class-imbalance issue, we employ Dice loss and N-Pair
loss for labelled and unlabelled data respectively. iii) Spatial encoding is
further incorporated to capture spatial correlations.

##  III Methodology

###  III-A Overview of SemiCurv

We first provide an overview of our SemiCurv approach, then describe the
individual components in greater detail. Our SemiCurv approach, illustrated in
Fig. 2, is built upon the consistency-based SSL framework (Sect. III-B). Both
labelled and unlabelled images are first augmented by a differentiable
geometric transformation (Sect. III-C) and then fed into student and teacher
networks. The teacher network is a parameter-wise temporal ensemble
(exponential moving average) of the student network and generates a target
(pseudo-label) to encourage consistency on the unlabelled data. Sinusoid
positional encoding is appended to each convolution layer in the encoder to
explicitly capture spatial correlations (Sect. III-E). The predicted
segmentation maps (posterior probabilities) from both networks are transformed
back to the original pose via the differentiable inverse transform, and the
dice loss (Sect. III-F) and N-pair loss (Sect. III-D) are used on the labelled
and unlabelled data respectively for training; in particular, we show how the
N-pair loss resolves a “collapsing” issue associated with the MSE loss
commonly used for consistency on the unlabelled data.

###  III-B Consistency-based Semi-supervised Learning

We denote a training set of $N_{l}$ labelled images and their ground-truth
segmentation maps as
$\mathcal{D}_{l}=\\{(\mathbf{X}^{l}_{i},\mathbf{Y}^{l}_{i})\\}_{i=1}^{N_{l}}$,
and denote the set of $N_{u}$ unlabelled images as
$\mathcal{D}_{u}=\\{\mathbf{X}^{u}_{j}\\}_{j=1}^{N_{u}}$. The loss function
optimized by semi-supervised learning methods can be generally written as Eq.
(1), where $\mathit{l}_{l}$, $\mathit{l}_{u}$, and $\gamma$ are supervised
loss on labelled images, unsupervised loss on unlabelled images and balancing
hyperparameter, respectively.

$L(\mathcal{D}_{l},\mathcal{D}_{u})=\frac{1}{N_{l}}\sum_{i=1}^{N_{l}}\mathit{l}_{l}(\mathbf{X}^{l}_{i},\mathbf{Y}^{l}_{i})+\gamma\frac{1}{N_{u}}\sum_{j=1}^{N_{u}}l_{u}(\mathbf{X}^{u}_{j})$
(1)

We note that the loss functions of multiple SSL methods [26, 25, 27] can be
written in this form. Most existing works adopt the cross-entropy loss or its
variants as the supervised loss $l_{l}$ for classification tasks. In this
work, we demonstrate in Sect. III-F that for segmentation with severe class
imbalance, Dice loss is more appropriate. For the unsupervised loss, the $\Pi$
model [25], mean teacher model (MT) [26], VAT [27], MixMatch [30] and
variants, all apply mean square error (MSE) consistency between the prediction
posteriors of an unlabelled image under two different augmentations
$t_{1}(\cdot)$ and $t_{2}(\cdot)$, and two different segmentation networks
$f_{1}(\cdot)$ and $f_{2}(\cdot)$ as in Eq. 2. We show in Sect. III-D that
under severe class imbalance the N-pair loss is more effective than MSE in
avoiding trivial solutions.

$l_{u}(\mathbf{X})=||f_{1}(t_{1}(\mathbf{X}))-f_{2}(t_{2}(\mathbf{X}))||_{F}^{2}$
(2)

SemiCurv is based on the mean teacher (MT) method [26] for semi-supervised
classification, which has also been previously applied to the segmentation
task. Briefly, the MT method maintains two networks: one fully trainable
network called the student network, and another non-trainable network called
the teacher network. The teacher network provides pseudo-labels to supervised
the student network to learn. To improve stability of training, the parameters
of teacher are the exponential moving average (EMA) of the student network’s
parameters. In the rest of the paper, we denote the student network as
$f(\cdot)$ and teacher network as $\hat{f}(\cdot)$.

(a)  (b)

Figure 3: (a) Illustration of differentiable transformation. Red solid box and
yellow dashed box indicate transformed and original views. The final yellow
solid box indicates the prediction after differentiable inverse transformation
where consistency loss is computed. (b) Loss surface for MSE consistency loss.
When both student and teacher produce exactly the same result (may not be
correct), as indicated by the diagonal, the MSE loss is always zero.

###  III-C Differentiable Geometric Transformation for Augmentation

Strong data augmentation is key to the success of consistency-based semi-
supervised learning [34]. As discussed, existing semi-supervised segmentation
methods often consider a limited set of augmentations. To further increase the
variation of poses to improve feature learning we introduce affine
transformations for stronger data augmentation. We denote the transformation
applied to input image $\mathbf{X}$ as $t(\mathbf{X})$. To enable computing
pixel-wise consistency loss, we further apply the inverse transformation to
the output of both student and teacher networks as $t^{-1}(f(t(\mathbf{X})))$.
With the inverse transformation, predictions on two arbitrarily augmented
images are directly comparable at the pixel level.

Concretely, an affine transformation involves an arbitrary combination of
scaling, rotation, shearing and translation, and can be formulated as a matrix
multiply with the following transformation matrix,

$\mathbf{H}=\begin{bmatrix}a_{11}&a_{12}&d_{x}\\\ a_{21}&a_{22}&d_{y}\\\
0&0&1\end{bmatrix}$ (3)

For every pixel at $(\hat{w},\hat{h})$ in the image after affine
transformation, we find its coordinate in the original image by
$[w,h,1]^{\top}=\mathbf{H}^{-1}[\hat{w},\hat{h},1]^{\top}$. We employ bilinear
interpolation to produce the pixel intensity at every pixel after
transformation. Concretely, we can find the 4 neighbouring pixels before the
transform as $\\{x_{\left\lfloor w\right\rfloor,\left\lfloor
h\right\rfloor},x_{\left\lceil w\right\rceil,\left\lfloor
h\right\rfloor},x_{\left\lfloor w\right\rfloor,\left\lceil
h\right\rceil},x_{\left\lceil w\right\rceil,\left\lceil h\right\rceil}\\}$.
The pixel intensity value after transformation $\hat{x}_{\hat{w},\hat{h}}$ can
be determined by bilinear interpolation as,

$\hat{x}_{\hat{w},\hat{h}}=[\left\lceil w\right\rceil-w,w-\left\lfloor
w\right\rfloor]\begin{bmatrix}x_{\left\lfloor w\right\rfloor,\left\lfloor
h\right\rfloor}&x_{\left\lceil w\right\rceil,\left\lfloor h\right\rfloor}\\\
x_{\left\lfloor w\right\rfloor,\left\lceil h\right\rceil}&x_{\left\lceil
w\right\rceil,\left\lceil
h\right\rceil}\end{bmatrix}\begin{bmatrix}\left\lceil h\right\rceil-h\\\
h-\left\lfloor h\right\rfloor\end{bmatrix}$ (4)

Because of the bilinear transformation $\hat{x}_{\hat{w},\hat{h}}$ is
obviously differentiable w.r.t. $\\{x_{w,h}|w\in\\{0,W-1\\},h\in\\{0,H-1\\}$
where $H$ and $W$ are the height and width of image respectively, i.e.
$t(\mathbf{X})$ is differentiable w.r.t. $\mathbf{X}$. For the inverse
transformation, we simply apply $\mathbf{H}^{-1}$ for affine transformation
and thus $t^{-1}(f(t(\mathbf{X})))$ is still differentiable w.r.t.
$\mathbf{X}$.

We also notice that to generate diverse and realistic augmented image samples,
we extrapolate images by mirroring the image over the edges. To give an
example, the result of a 1D sequence “$abcd$” after mirror flipping is
“$dcb|abcd|cba$”. This is achieved by clipping the pixel location before
transformation with $x=(x//W\%2)*(W-x\%W)+(1-x//W\%2)*(x\%W)$ where $//$ and
$\%$ are floor and modulo divisions respectively. An illustration of
augmentation for image segmentation is given in Fig. 3 (a). With the
differentiable transformation, we are able to randomly generate two augmented
images $t_{1}(\mathbf{X})$ and $t_{2}(\mathbf{X})$ that are respectively fed
into the student and teacher networks $f$ and $\hat{f}$. The two prediction
posteriors can be then aligned via the differentiable inverse transformation
$t_{1}^{-1}(f(t_{1}(\mathbf{X})))$ and
$t_{2}^{-1}(\hat{f}(t_{2}(\mathbf{X})))$. For the rest of the paper, we denote
prediction posteriors after alignment as
$g_{i1}=t_{1}^{-1}(f(t_{1}(\mathbf{X}_{i})))$ (student) and
$\hat{g}_{j2}=t_{2}^{-1}(\hat{f}(t_{2}(\mathbf{X}_{j})))$ (teacher).

###  III-D Avoiding Collapsed Predictions on Unlabelled Data

Consistency-based SSL methods commonly use mean square error (MSE) for the
consistency loss on unlabelled data. In this section, we first point out that
MSE loss allows collapse of model predictions on unlabelled data to the
majority class as this is a trivial solution of the MSE consistency loss. We
then introduce the N-pair loss as a way to mitigate this issue thus enabling
the unlabelled data to better regularize the model.

Without loss of generality, suppose we have two scalar predictions (pixel-wise
predictions) $g_{1}$ and $g_{2}$. We visualize the loss surface for MSE
pairwise consistency in Fig. 3 (b), and observe that the MSE loss is flat
along the diagonal because MSE loss is minimized when the predictions from the
two networks match. When the class distribution is highly imbalanced as in the
case of curvilinear segmentation, where there are far more background pixels
than foreground pixels, the two networks can achieve zero MSE consistency loss
just by assigning every single pixel to the majority class. In this case every
pixel will be predicted as background. We term this all majority class
prediction as a collapsed prediction. An instance of this behaviour is shown
in Fig. 8 (a) where training IoU gets closer to $1$, yet validation IoU
collapses to $0$. Avoiding this often requires very careful selection of the
EMA hyperparameter and consistency weights, which requires expensive tuning
runs.

In this work, inspired by the recent progress in contrastive learning [14], we
propose to use an N-pair loss on unlabelled data [13] to avoid collapsed
predictions. The N-pair loss simultaneously exploits all unlabelled samples in
a training mini-batch to construct one positive pair to encourage similarity
and multiple negative pairs to encourage diversity in predictions. By
enforcing diversity with the negative pairs, the N-pair loss effectively
penalizes models that give collapsed predictions on all unlabelled data. We
illustrate the construction of the N-pair loss for curvilinear structure
segmentation in Fig. 4: in a mini-batch of $N_{B}$ unlabelled images the
positive pair is chosen as the predictions from student and teacher on an
anchor image $\mathbf{X}_{i}$, $\\{(g_{i1},\hat{g}_{i2})|i=1\cdots N_{B}\\}$,
while negative pairs are chosen as the predictions between the student’s
predictions on the same anchor image and teacher’s predictions on other images
in the mini-batch, $\\{(g_{i1},\hat{g}_{j2})|i\neq j;i,j=1\cdots N_{B}\\}$.
Formally, the N-pair loss is given by,

$l_{\textrm{N-pair}}=-\frac{1}{N_{B}}\sum_{i=1}^{N_{B}}\log\frac{\exp(sim(g_{i1},\hat{g}_{i2})/\tau)}{\sum_{j=1}^{N_{B}}\exp(sim(g_{i1},\hat{g}_{j2})/\tau)}$
(5)

where $N_{B}$, $\tau$ and $sim$ are the mini-batch size, temperature parameter
and similarity metric, respectively. We use cosine similarity defined in Eq.
(6) for the similarity metric $sim$, where $\mathbf{vec}$ vectorizes a matrix
into a vector and $\delta$ is a small value to avoid division by zero.

$sim({g}_{1},{g}_{2})=\frac{\mathbf{vec}({g}_{1}+\delta)^{\top}\mathbf{vec}({g}_{2}+\delta)}{||\mathbf{vec}({g}_{1}+\delta)||_{2}\cdot||\mathbf{vec}({g}_{2}+\delta)||_{2}}$

(6)

We now formally show how N-pair loss prevents collapsed predictions that can
occur when MSE is used. Specifically, we show that the N-pair loss will be
lower for correct predictions than collapsed predictions, unlike MSE loss.
First consider the case when predictions all collapse to a single class, often
the background resulting in all zero predictions, i.e. $g_{1}=\mathbf{0}$ and
$\hat{g}_{2}=\mathbf{0}$. Then, the cosine similarity for all positive and
negative pairs are 1; the same result holds when predictions are all
foreground, i.e. $g_{1}=\mathbf{1}$ and $\hat{g}_{2}=\mathbf{1}$. Then, the
N-pair loss for this collapsed prediction is

$\tilde{l}_{\textrm{N-pair}}=-\frac{1}{N_{B}}*N_{B}\log 1/N_{B}=\log N_{B}.$
(7)

We note that in this case, the MSE loss is $\tilde{l}_{mse}=0$. When the
predictions are all correct, the positive pair similarity is $1$, and for
negative pairs this is approximately $\epsilon<<1$, because the segmentation
masks for arbitrary two images are unlikely to significantly match, and the
N-pair loss becomes

$\begin{split}l_{\textrm{N-pair}}&=-\frac{1}{N_{B}}*N_{B}(\log\frac{\exp
1/\tau}{\exp 1/\tau+(N_{B}-1)*\exp\epsilon/\tau})\\\
&=\log(1+(N_{B}-1)*\exp\frac{\epsilon-1}{\tau})\end{split}$ (8)

while the MSE loss is still $l_{mse}=0$. Since
$\exp\frac{\epsilon-1}{\tau}<1$, it is easy to verify that
$l_{\textrm{N-pair}}<\tilde{l}_{\textrm{N-pair}}$ when $N_{B}$ is larger than
1 and the ratio $l_{\textrm{N-pair}}/\tilde{l}_{\textrm{N-pair}}$ is getting
smaller with increased batchsize $N_{B}$. This suggests the gap between N-pair
loss under collapsed prediction and correct prediction is more significant
with larger batchsize. We therefore conclude that the N-pair loss can easily
distinguish good predictions from collapsed ones as opposed to MSE loss, for
which the loss is $0$ in both cases.

Figure 4: Illustration of N-pair loss for unlabelled images. Positive pairs
are constructed between the predictions of teacher and student networks on the
same input image. Negative pairs are constructed between the predictions of
teacher and student networks on different input images.

###  III-E Modelling Spatial Correlation with Positional Encoding

The nature of curvilinear structure segmentation requires the network to
encode spatial correlation. For example, cracks, roads, and blood vessels are
spatially continuous thus pixels adjacent to other positive (foreground)
pixels are likely to be positive as well. Though it is difficult to handcraft
features that capture this correlation, encoding spatial locations has been
shown to be effective [16]. One straightforward way to encode spatial
locations is by appending linear coordinates, normalized to between 0 and 1,
to the intermediate feature layers as additional feature channels. However,
learning directly on this absolute positional encoding risks overfitting to
specific locations, especially when training with very few labelled samples.
Inspired by the positional encoding of [35], we apply a sinusoid coordinate
encoding as in Eq. (9) where $S_{x},S_{y}$ are the linear positional encoding
normalized to between 0 and 1 and $K$ is a period parameter. Such periodic
positional encoding allows the network to be aware of relative location while
reducing the risk of overfitting to absolute locations. The positional
encodings are channel-wise concatenated to each intermediate activation map of
the encoder network as shown in Fig. 5.

$\hat{S}_{x}=\sin K\pi S_{x},\quad\hat{S}_{y}=\cos K\pi S_{y}$ (9)

Figure 5: Sinusoid coordinate encoding for capturing spatial correlation.
Encodings are concatenated to feature maps in channel-wise fashion.

###  III-F Dice Loss for Class Imbalance

It is well-known that class imbalance can affect the performance of machine
learning models [36, 37, 38]. The cross-entropy (CE) loss commonly used for
supervised segmentation tasks averages the pixel-wise CE loss over all pixels
in the image; CE loss biases learning towards the majority class when there is
an imbalanced class distribution. In curvilinear structure segmentation tasks,
this imbalance is particularly severe due to a low positive class ratio (see
Table I). As a result, models trained with CE may have high per-pixel accuracy
but low intersection over union (IoU), which is the most common evaluation
metric for segmentation tasks. In order to mitigate this issue, Dice loss [39]
(Eq. 10) was proposed as an alternative for segmentation; it focuses on the
intersection of predictions with ground truth only for the positive class.

$l_{dice}=1-\frac{2\mathbf{vec}(g_{1}+\delta)^{\top}\mathbf{vec}(\mathbf{Y}+\delta)}{||\mathbf{vec}(g_{1}+\delta)||_{1}+||\mathbf{vec}(\mathbf{Y}+\delta)||_{1}}$
(10)

###  III-G Implementation Details

The final loss function is the weighted combination of supervised loss and
unlabelled loss: $l_{total}=l_{dice}+\gamma l_{\textrm{N-pair}}$. As commonly
done in consistency-based SSL, we do not apply the unlabelled loss
$l_{\textrm{N-pair}}$ at the beginning of training but instead gradually
increase the strength of $\gamma$ following a sigmoid function
$\gamma=\exp(-10(\frac{min(T,T_{rp})}{T_{rp}}-1)^{2})$ where $T$ and $T_{rp}$
are epoch number and hyperparameter, respectively. We set the mean teacher EMA
hyperparameter $\alpha$ to 0.999. For all experiments, we use the SGD
optimizer, fixing the total number of training epochs to 1000 and
$T_{rp}=500$, the initial learning rate at $0.001$ and continuously decay by
half every 500 epochs. Each epoch is defined as cycling all labeled data once.
The sinusoid spatial encoding parameter was set as $K=4$. The overall
algorithm is summarized in Algorithm 1.

Input : Labelled Images $\\{(\mathbf{X}_{i}^{l},\mathbf{Y}_{i}^{l})\\}$ and
Unlabelled Images $\\{\mathbf{X}_{j}^{u}\\}$

Output : Segmentation Model Parameters $\Theta$

for  _$T\leftarrow 1$ to $1000$_ do

for  _$\\{\mathbf{X}^{l}_{i},\mathbf{Y}^{l}_{i},\mathbf{X}^{u}_{j}\\}$ in
mini-batch_ do

// Random Augmentation

Compute
$t(\mathbf{X}^{l}_{i}),t_{1}(\mathbf{X}^{u}_{i}),t_{2}(\mathbf{X}^{u}_{i})$;

// Forward Pass

Compute
$f(t_{1}(\mathbf{X}_{i}^{l}))$,$f(t_{1}(\mathbf{X}_{i}^{u}))$,$\hat{f}(t_{2}(\mathbf{X}_{i}^{u}))$

// Supervised Loss

Compute $l_{dice}$ according to Eq. (10);

// Unsupervised Loss

Compute $l_{\textrm{N-pair}}$ according to Eq. (5);

// Unlabelled Loss Weight

$\gamma=\exp(-10(\frac{min(T,T_{rp})}{T_{rp}}-1)^{2})$;

// Train One Minibatch

$\Theta=\Theta-\alpha\nabla_{\Theta}l_{total}$;

Algorithm 1 Semi-Supervised Curvilinear Structure Segmentation: Training
Algorithm

##  IV Experiments

We validate the efficacy of SemiCurv on 6 different curvilinear structure
datasets spanning 3 domains: road crack segmentation, road segmentation from
satellite images, and biomedical image segmentation.

###  IV-A Datasets

CrackForest [40] was proposed for crack segmentation from paved road images.
It consists of 118 labelled images in total. We created a datasplit for
evaluating semi-supervised segmentation by randomly splitting the whole
dataset into $80\%$ for training, $10\%$ for validation, and $10\%$ for
testing. Among the training images, we further assume $5\%$ or $1\%$ of the
images are labelled. Crack500 [1] is a more comprehensive dataset consisting
of 1896/348/1124 labelled road crack images for training, validation, and
testing respectively. We follow the standard datasplit proposed in [1] and
assume the training set is partially labelled. Gaps384 [1] was created from a
large road crack detection dataset GAPS [41] by manually selecting 384 images
and cropping out smaller regions with cracks, totalling 508 annotated images.
We follow the standard data split for evaluation. MIT Road [18] was proposed
for automatically extracting road from satellite images. As we notice there
are many blank regions in the RGB images, we preprocess the images to manually
crop non-blank regions. In total, we obtain 6880/1215/440 labelled images for
training, validation, and testing respectively. EM128 [23] was created for
evaluating segmentation on cell membranes. It consists of 30 labelled images
and we follow the practice in [42] to reserve 15 images for training and the
rest for testing. To allow evaluation of semi-supervised learning, we divide
each image into 16 regions each covering $128\times 128$ pixels. In total we
have 240/240 for training and testing respectively and we name the derived
dataset as EM128. DRIVE128 The DRIVE dataset [43] was developed for evaluating
segmentation on retinal blood vessels. It consists of 20 labelled images which
we split into 10 for training and 10 for testing. Non-overlapping patches of
size $128\times 128$ pixels are cropped in a similar way to EM128 and we
denote this derived dataset as DRIVE128. The overall number of training,
validation, and test samples are shown in Table II. The number of labelled
training data at different levels of supervision are provided as well. We note
that the level of class imbalance is severe on most of these datasets, as can
be seen from the low percentage of positive (foreground) pixels given in Table
I. The high class imbalance potentially makes the model more likely to
collapse to predicting all pixels as background on the unlabelled data.

TABLE I: Percentage ($\%$) of positive pixels in datasets.

Dataset | CrackForest | Crack500 | Gaps384 | MITRoad | EM128 | DRIVE128
---|---|---|---|---|---|---
Pos. pixel $\%$ | $2.5\%$ | $6.0\%$ | $1.2\%$ | $5.1\%$ | $22.0\%$ | $8.6\%$

TABLE II: Semi-supervised learning data splits for all 6 datasets.

Split |  | CrackForest | Crack500 | Gaps384 | DRIVE128 | MITRoad | EM128
---|---|---|---|---|---|---|---
Train | Labelled Percent. | 5% | 1% | 5% | 1% | 5% | 1% | 5% | 1% | 1% | 0.1% | 10% | 5%
#Labeled | 5 | 1 | 95 | 19 | 23 | 5 | 8 | 2 | 69 | 7 | 24 | 12
#Unlabeled | 91 | 95 | 1801 | 1877 | 442 | 460 | 152 | 158 | 6811 | 6873 | 216 | 228
Test |  | 11 | 1124 | 39 | 160 | 440 | 240

###  IV-B Evaluation Metric

To evaluate the quality of segmentation predictions, we compute Intersection
over Union (IoU) using a threshold of 0.5 on prediction posteriors for all
test images and report the mean IoU. We also evaluate the F1 measure of
precision and recall at threshold 0.5. This metric treats segmentation as a
binary classification task. For all 6 datasets, we set the background pixel
label as 0 and foreground as 1.

###  IV-C Competing Methods

We extensively compare SemiCurv with existing fully supervised methods (with
different backbone networks) and state-of-the-art generic semi-supervised
learning methods, as described in the following.

Fully Supervised Methods: We first investigate the state-of-the-art fully
supervised curvilinear structure segmentation, edge detection, and semantic
segmentation methods. This provides context of what is achievable for
curvilinear segmentation tasks. FPHBN [1] is the state-of-the-art method for
crack segmentation; we used the reported results on Crack500 in this paper for
comparison. HED [44] was proposed for detecting edges in images. Due to the
similar nature of edge and linear structures defined in this work, we evaluate
this backbone as a baseline. DeepLab v3+ [45] is the state-of-the-art backbone
network for semantic segmentation tasks. We evaluate this network here to show
that generic semantic segmentation networks do not generalize well to
curvilinear structure segmentation. UNet [21] was originally proposed for
medical image segmentation. We use a variant that adds residual connections in
each convolution block as the backbone network in this work. Details of the
backbone network is given in the Supplementary Material.

Semi-Supervised Methods: We compare against state-of-the-art semi-supervised
semantic segmentation, medical image segmentation, and generic SSL methods
adapted to curvilinear segmentation. CutMix [9] proposed to generate random
mask to mix two images and consistency is applied to between the predictions
of teacher model and student model over the masked regions. TCSM [33] adapts
the mean teacher framework to learn from unlabelled data. The augmentation is
limited to scaling and rotation in multiples of $90^{\circ}$. VAT [27]
proposed to learn the optimal augmentation by maximizing the consistency on
unlabelled data. The final objective aims to minimize the pairwise consistency
as well as the entropy on unlabelled data. cGAN [46] is another line of semi-
supervised semantic segmentation approach. The conditional GAN (cGAN) based
method introduced a discriminator network to differentiate ground-truth
segmentation mask from predicted ones and the predictions on unlabeled data
can be constrained by the discriminator. Mean Teacher (Baseline) [26] proposed
to use a temporal ensemble, as teacher, of a learnable student model to
provide pseudo labels to train the student model. MSE consistency is used
between teacher and student outputs. We use this as our baseline method.
SemiCurv is our proposed model incorporating differentiable geometric
transformations, N-pair objective, and sinusoid positional encoding.

###  IV-D Quantitative Results

We present the comparison of both fully supervised and semi-supervised methods
with $5\%$ and $1\%$ labelled data in Table III; for MITRoad and EM128 we
consider different labeling budgets due to the size of dataset. In the fully
supervised block, UNet ($*95\%$) and UNet ($*90\%$) indicate the relative
performance $95\%*m$ and $90\%*m$ where $m$ is the performance achieved by the
fully supervised method trained with $100\%$ of the labelled data. From the
extensive comparison, we make the following observations. First, the adapted
UNet is a very strong backbone network for curvilinear structure segmentation.
It outperforms the state-of-the-art semantic segmentation backbone DeepLabV3+,
edge detection network HED, and network specifically designed for crack
segmentation FPHBN on all datasets. Moreover, under the semi-supervised
setting, with only $5\%$ and $1\%$ labelled data SemiCurv can match and even
outperform the $95\%$ and $90\%$ relative performance of fully supervised
counterparts in terms of IoU and F1, respectively. By comparing to the state-
of-the-art semi-supervised methods for segmentation, we still observe very
significant advantages for SemiCurv. In particular, the improvement from
baseline is more significant in the lower label regime. For example, on
CrackForest IoU improved $17\%$ and $2.2\%$ from the UNet baseline trained
using $1\%$ and $5\%$ labelled data respectively. The improvement on all other
datasets also exhibits similar patterns with large improvements over the UNet
baseline. We also observe CutMix to be a very competitive method, in
particular on the MIT Road dataset. We speculate that this is because the MIT
Road dataset resembles generic semantic segmentation problems, such that it
benefits from cutmix style augmentation. Finally, TCSM produces much worse
results compared to both MT and SemiCurv. We speculate that this is due to
weak augmentation adopted in TCSM hampering semi-supervised performance.

TABLE III: Quantitative evaluations of semi-supervised curvilinear segmentation. All numbers are in $\%$. (*95/90%) indicates multiplied by 95/90%. |  | CrackForest | Crack500 | Gaps384 | DRIVE128 | MIT Road | EM128
---|---|---|---|---|---|---|---
Fully Supervised | Model | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1
Labelled Data | 100% | 100% | 100% | 100% | 100% | 100%
DeepLabV3 | 51.2 | 67.1 | 38.8 | 52.5 | 37.3 | 53.1 | 30.5 | 33.6 | 33.2 | 48.3 | 46.0 | 61.8
HED | 54.7 | 70.2 | 39.5 | 53.9 | 35.2 | 49.0 | 33.6 | 37.6 | 40.1 | 55.0 | 46.6 | 63.0
FPHBN | NA | NA | 48.9 | NA | NA | NA | NA | NA | NA | NA | NA | NA
Unet 100% | 70.7 | 84.7 | 49.6 | 62.9 | 39.0 | 52.9 | 59.2 | 60.4 | 58.4 | 71.3 | 67.5 | 80.2
Unet 95% | 67.1 | 80.4 | 47.1 | 59.7 | 37.1 | 50.3 | 56.2 | 57.3 | 55.5 | 67.7 | 64.1 | 76.2
Unet 90% | 63.6 | 76.4 | 44.6 | 56.6 | 35.1 | 47.8 | 53.3 | 54.5 | 52.6 | 64.3 | 60.7 | 72.4
| Labelled Data | 5% | 5% | 5% | 5% | 1% | 10%
Semi-Supervised | Unet | 60.1 | 74.3 | 46.5 | 60.5 | 34.9 | 48.9 | 49.1 | 56.5 | 53.8 | 67.3 | 63.4 | 77.1
CutMix | 66.0 | 79.3 | 44.4 | 57.5 | 33.5 | 46.9 | 50.3 | 49.5 | 55.8 | 69.0 | 62.2 | 76.3
VAT | 53.9 | 69.1 | 48.1 | 62.5 | 29.6 | 42.3 | 47.8 | 56.3 | 48.3 | 62.6 | 57.9 | 72.6
TCSM | 49.9 | 66.2 | 36.8 | 50.7 | 37.8 | 52.2 | 44.8 | 51.0 | 32.5 | 47.0 | 33.0 | 48.8
MT | 66.9 | 79.1 | 48.0 | 61.9 | 34.3 | 48.2 | 54.2 | 60.9 | 55.2 | 68.7 | 65.3 | 78.6
cGAN | 67.0 | 79.2 | 48.1 | 62.0 | 33.7 | 47.3 | 55.5 | 62.4 | 53.5 | 66.6 | 62.1 | 74.8
Ours | 68.0 | 80.9 | 49.2 | 62.4 | 38.0 | 53.4 | 56.0 | 61.3 | 56.0 | 69.1 | 65.7 | 79.1
Labelled Data | 1% | 1% | 1% | 1% | 0.1% | 5%
Unet | 36.6 | 48.8 | 44.4 | 58.3 | 28.3 | 41.2 | 34.1 | 45.5 | 35.9 | 50.5 | 57.0 | 71.5
CutMix | 56.5 | 71.2 | 40.2 | 54.6 | 30.1 | 43.6 | 43.0 | 50.8 | 44.9 | 59.2 | 63.9 | 77.6
VAT | 29.3 | 41.5 | 43.3 | 58.4 | 18.8 | 17.3 | 31.4 | 41.9 | 25.4 | 38.6 | 57.1 | 72.2
TCSM | 30.5 | 43.9 | 41.9 | 56.2 | 28.6 | 41.7 | 28.8 | 34.6 | 31.6 | 45.4 | 28.6 | 43.6
MT | 61.9 | 75.9 | 45.6 | 59.8 | 31.1 | 44.3 | 37.7 | 47.3 | 46.4 | 61.1 | 63.2 | 76.9
cGAN | 54.6 | 66.9 | 45.4 | 59.5 | 30.1 | 42.8 | 42.6 | 53.4 | 43.6 | 57.4 | 60.3 | 73.4
Ours | 64.4 | 78.1 | 46.2 | 60.1 | 33.7 | 48.4 | 44.3 | 51.5 | 47.7 | 62.2 | 64.1 | 77.9

###  IV-E Ablation Study

Here we carry out ablation studies to investigate the effectiveness of each
component and present the results in Table V.

Supervised Loss: We first compare the supervised loss, SupLoss, adopted for
labelled images. UNet with weighted binary cross entropy loss, WBCE, is
consistently worse than UNet with dice loss, Dice. This suggests that
optimizing dice loss is able to better generalize under severe class
imbalance.

Semi-Supervised Learning: We next compare the fully supervised baseline with
the mean teacher model, MT. With mean square error (MSE) consistency loss, we
observe significant improvement of MT over the baseline. We further evaluate
MT with and without the differentiable geometric transformation, GeoTform. The
significant difference in performance between the two settings indicates that
strong augmentation is vital to the success of consistency-based SSL.

MSE vs N-pair Consistency Loss: We further compare MT with mean square error
loss, MSE, and N-pair loss. The improvement demonstrates the superiority of
considering both positive and negative pairs simultaneously with N-pair. In
addition to the numerical improvement, we also show in Fig. 8 (a) that N-pair
is more robust than MSE and avoids collapsed predictions. As we discussed in
Sect. III-D, the pairwise mean square error, MSE, consistency loss is prone to
a collapsing issue. This becomes severe when the data is highly imbalanced and
very few labelled samples are available for training. We present examples for
both MSE consistency and N-pair consistency on CrackForest with only $1\%$
labelled data in Fig. 8 (a). MSE produces unstable validation performance, and
it eventually collapses to all zeros around $800$ epochs. In contrast, with
the proposed N-pair loss the validation performance keeps stable throughout
the training. Overfitting, due to too few labelled data, is still observed but
it avoids collapsing to all zero predictions.

Positional Encodings: We analyze the effect of including various forms of
spatial encoding. First, when including linear spatial encoding, Linear Spt.
Enc., we sometimes observe that performance degrades. This can be attributed
to the potential overfitting to absolute locations; using sinusoid spatial
encoding instead, Sinusoid Spt. Enc., we observe clear and consistent
improvements on all datasets. We investigate the impact of constant $K$ in Eq.
9 which controls the period of sinusoid positional encoding. The final
SemiCurv is evaluated with $K=1,2,4,8$ with results shown in Fig. 8 (b). We
observe an optimal range of $K$ around 4, and overall the results are robust
to $K$ from $2$ to $8$. This shows sinusoid positional encoding is a robust
component of our approach.

For alternative positional encodings, we evaluate two additional methods on
CrackForest segmentation task. The results are presented in Tab. IV. We
observe a small drop of performance when swapping out our proposed sinusoid
encoding with learned position embedding [35] and rotary position embedding
[47]. Overall, the proposed sinusoid position encoding is still the better
option.

TABLE IV: Comparison of position embeddings. CrackForest | Sinusoid Enc. (ours) | Learned [35] | Rotary [47]
---|---|---|---
1% | 64.41 | 62.93 | 62.96
5% | 68.02 | 67.49 | 67.56

Alternative Similarity Metric We further explore a different option for the
similarity metric used in the N-pair loss. L2 distance is often used in N-pair
loss for metric learning and we therefore evaluated the induced RBF kernel for
comparison against cosine similarity. Formally, the L2 distance induced
similarity is given by

$l_{\textrm{N-pair}}=-\frac{1}{N_{B}}\sum_{i=1}^{N_{B}}\log\frac{\exp(-||g_{i1}-\hat{g}_{i2}||_{2})}{\sum_{j=1}^{N_{B}}\exp(-||g_{i1}-\hat{g}_{j2}||_{2})}$
(11)

The quantitative comparison between using cosine similarity and L2 distance
induced similarity is shown in Table V under the similarity metric (Sim.)
column. For both similarity metrics, we use the SemiCurv framework and keep
the training protocols unchanged. We observe from the comparison that cosine
similarity outperforms L2 distance consistently on all datasets. Also,
stability of training suffers with L2 distance compared to SemiCurv with
cosine distance. This can be attributed to the normalizing effect of cosine
similarity, while L2 distance is affected by the absolute number of positive
pixel predictions.

TABLE V: Ablation study for all 6 curvilinear datasets.

|  |  |  |  |  | CrackForest 1% | Crack500 1% | Gaps384 1% | MIT Road 1% | EM128 5% | DRIVE128 5%
---|---|---|---|---|---|---|---|---|---|---|---
SSL | SupLoss | GeoTform | Consist. | Spt.Enc. | Sim. | IoU | IoU | IoU | IoU | IoU | IoU
- | WBCE | ✓ | - | - | - | 21.1 | 38.6 | 24.3 | 41.7 | 40.9 | 34.9
- | Dice | ✓ | - | - | - | 36.6 | 44.4 | 28.3 | 51.6 | 57.0 | 49.1
MT | Dice | - | MSE | - | - | 50.5 | 43.2 | 19.2 | 51.4 | 53.8 | 43.9
MT | Dice | ✓ | MSE | - | - | 61.9 | 45.6 | 31.1 | 55.2 | 63.2 | 54.2
MT | Dice | ✓ | N-pair | - | Cosine | 63.7 | 45.9 | 31.6 | 55.2 | 63.9 | 55.9
MT | Dice | ✓ | N-pair | Linear | Cosine | 62.8 | 45.9 | 34.0 | 54.5 | 63.7 | 44.9
MT | Dice | ✓ | N-pair | Sinusoid | Cosine | 64.4 | 46.2 | 33.7 | 56.0 | 64.1 | 56.0
MT | Dice | ✓ | N-pair | Sinusoid | L2 | 58.4 | 45.2 | 29.5 | 53.0 | 55.9 | 43.0

Time Complexity We analyze the time complexity of proposed semi-supervised
learning method. With more unlabeled data, it takes more time to allow the
network to iterate unlabeled data. The training time is roughly linearly
proportional to the ratio of unlabeled and labeled data. We further
empirically evaluated the training time for fully supervised learning method
and SemiCurv in Tab. VI. Despite more training time required, our SemiCurv
performs substantially better at inference stage and there is no additional
cost at inference stage.

TABLE VI: Empirical evaluation of time complexity for proposed method. CrackForest | FullSup | SemiCurv
---|---|---
1% | 20m | 1h21m
5% | 100m | 5h30m

Exponential Moving Averaging Hyperparameter We choose the EMA hyperparameter
for teacher model according to the validation performance. A lower $\alpha$
will allow faster update of teacher network while a higher $\alpha$ slows down
the update of teacher network. This hyperparameter is particularly sensitive
for MSE consistency loss according because MSE loss is more likely to produce
collapsed prediction, i.e. predicting all pixels as background. In this
section we further provide empirical evidence for the chosen hyperparameter
$\alpha$ by varying $\alpha$ from 0.9 to 0.9999 and compare the performance on
CrackForest dataset. The results in Fig. 6 suggest choosing $\alpha=0.999$ is
the optimal choice for both mean teacher and our SemiCurv models.

Figure 6: Evaluation of EMA hyperparameter.

###  IV-F Qualitative Results

We further present qualitative comparisons of UNet, CutMix, MT, and SemiCurv
on 4 datasets in Fig. 7. We make the following observations: first, for both
MT and SemiCurv, the visual quality of segmentation results are more appealing
and better mimicks the ground-truth compared to the fully supervised baseline
UNet. In particular, our method is able to generate more well-connected
predictions for $037$ (1st row) thanks to the contribution of appending
positional encoding. We further notice that SemiCurv produces, in general,
cleaner outputs for $037$ (1st row), $022$ (2nd row), $066\\_1$ (3rd row). We
also observe, in $0192\\_541\\_1$ (4th row), the proposed approach captures
very subtle cracks on the top. The segmentation results on DRIVE dataset
(5th/6th rows) also suggest the superiority of SemiCurv. It produces
relatively high fidelity and well-delineated blood vessel segmentation maps.
In comparison, UNet, CutMix, and MT tend to mingle different blood vessels
together which is less common in SemiCurv’s predictions. For satellite image
segmentation, the visualization covers industrial areas (row 7) and rural
areas (rows 8-9). We observe consistent improvements of SemiCurv compared to
both UNet and state-of-the-art semi-supervised image segmentation CutMix.
Interestingly, the SemiCurv predictions sometimes contain valid road branches
that are missing in the ground-truth annotation (last row).

Figure 7: Qualitative comparisons for curvilinear structure segmentation.

(a)  (b)

Figure 8: (a) Comparing MSE loss v.s. N-pair loss over collapsed predictions.
(b) Performance v.s. sinusoid parameter $K$.

##  V Conclusion

In this work we addressed curvilinear structure segmentation in a semi-
supervised learning setting to exploit “freely” available and abundant amounts
of unlabelled data with our proposed SemiCurv framework. In particular, we
introduced stronger augmentation involving affine geometric transformations
which we showed to be essential to the success of SSL. We further identified
implications of the severe class imbalance in curvilinear segmentation tasks
on the widely used MSE consistency loss, and showed that N-pair loss should be
used instead to mitigate these issues. Finally, we found that sinusoid
positional encoding is effective in further improving segmentation
performance. Our extensive experiments on 6 curvilinear structure segmentation
datasets from 3 different domains demonstrates the effectiveness of SemiCurv,
and detailed ablation studies validated the importance of each of our proposed
components. Our SemiCurv framework and the extensive results presented in this
work provide a foundation for future work in SSL for curvilinear segmentation.

## Acknowledgment

This research is supported by the Agency for Science, Technology and Research
(A*STAR) under its AME Programmatic Funds (Grant No. A20H6b0151).

## References

  * [1] F. Yang, L. Zhang, S. Yu, D. Prokhorov, X. Mei, and H. Ling, “Feature pyramid and hierarchical boosting network for pavement crack detection,” _IEEE Transactions on Intelligent Transportation Systems_ , 2019.
  * [2] F. Wang, Y. Gu, W. Liu, Y. Yu, S. He, and J. Pan, “Context-aware spatio-recurrent curvilinear structure segmentation,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [3] X. Hu, F. Li, D. Samaras, and C. Chen, “Topology-preserving deep image segmentation,” in _Advances in Neural Information Processing Systems_ , 2019\.
  * [4] A. Mosinska, P. Marquez-Neila, M. Koziński, and P. Fua, “Beyond the pixel-wise loss for topology-aware delineation,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [5] J. E. Van Engelen and H. H. Hoos, “A survey on semi-supervised learning,” _Machine Learning_ , vol. 109, no. 2, pp. 373–440, 2020.
  * [6] W. C. Hung, Y. H. Tsai, Y. T. Liou, Y. Y. Lin, and M. H. Yang, “Adversarial learning for semi-supervised semantic segmentation,” in _British Machine Vision Conference_ , 2018.
  * [7] A. K. Mondal, A. Agarwal, J. Dolz, and C. Desrosiers, “Revisiting cyclegan for semi-supervised segmentation,” _arXiv preprint arXiv:1908.11569_ , 2019.
  * [8] Y. Ouali, C. Hudelot, and M. Tami, “Semi-supervised semantic segmentation with cross-consistency training,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [9] G. French, T. Aila, S. Laine, M. Mackiewicz, and G. Finlayson, “Semi-supervised semantic segmentation needs strong, high-dimensional perturbations,” in _British Machine Vision Conference_ , 2020.
  * [10] V. Olsson, W. Tranheden, J. Pinto, and L. Svensson, “Classmix: Segmentation-based data augmentation for semi-supervised learning,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021.
  * [11] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” _International journal of computer vision_ , 2015.
  * [12] A. M. Andrew, “Multiple view geometry in computer vision,” _Kybernetes_ , 2001\.
  * [13] K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” in _Advances in neural information processing systems_ , 2016\.
  * [14] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International Conference on Machine Learning_ , 2020.
  * [15] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [16] R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski, “An intriguing failing of convolutional neural networks and the coordconv solution,” in _Advances in Neural Information Processing Systems_ , 2018.
  * [17] V. Mnih and G. E. Hinton, “Learning to detect roads in high-resolution aerial images,” in _European Conference on Computer Vision_ , 2010.
  * [18] V. Mnih, “Machine learning for aerial image labeling,” Ph.D. dissertation, University of Toronto, 2013.
  * [19] C. Henry, S. M. Azimi, and N. Merkle, “Road segmentation in sar satellite images with deep fully convolutional neural networks,” _IEEE Geoscience and Remote Sensing Letters_ , 2018.
  * [20] Y. Liu, J. Yao, X. Lu, M. Xia, X. Wang, and Y. Liu, “Roadnet: Learning to comprehensively analyze road networks in complex urban scenes from high-resolution remotely sensed images,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2018.
  * [21] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015.
  * [22] F. Isensee, J. Petersen, A. Klein, D. Zimmerer, P. F. Jaeger, S. Kohl, J. Wasserthal, G. Koehler, T. Norajitra, S. Wirkert _et al._ , “nnu-net: Self-adapting framework for u-net-based medical image segmentation,” in _Bildverarbeitung für die Medizin 2019_. Springer, 2019.
  * [23] I. Arganda-Carreras, S. C. Turaga, D. R. Berger, D. Cireşan, A. Giusti, L. M. Gambardella, J. Schmidhuber, D. Laptev, S. Dwivedi, J. M. Buhmann _et al._ , “Crowdsourcing the creation of image segmentation algorithms for connectomics,” _Frontiers in neuroanatomy_ , 2015.
  * [24] Q. Zou, Z. Zhang, Q. Li, X. Qi, Q. Wang, and S. Wang, “Deepcrack: Learning hierarchical convolutional features for crack detection,” _IEEE Transactions on Image Processing_ , 2018.
  * [25] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” _International Conference on Learning Representations_ , 2017.
  * [26] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in _Advances in neural information processing systems_ , 2017.
  * [27] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,” _IEEE transactions on pattern analysis and machine intelligence_ , 2018.
  * [28] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, “Semi-supervised learning with ladder networks,” in _Advances in neural information processing systems_ , 2015.
  * [29] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [30] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [31] A. Odena, “Semi-supervised learning with generative adversarial networks,” _arXiv preprint arXiv:1606.01583_ , 2016.
  * [32] N. Souly, C. Spampinato, and M. Shah, “Semi supervised semantic segmentation using generative adversarial network,” in _IEEE International Conference on Computer Vision_ , 2017.
  * [33] X. Li, L. Yu, H. Chen, C.-W. Fu, L. Xing, and P.-A. Heng, “Transformation-consistent self-ensembling model for semisupervised medical image segmentation,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2020.
  * [34] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, “Unsupervised data augmentation for consistency training,” in _Advances in Neural Information Processing Systems_ , 2020, pp. 6256–6268.
  * [35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [36] X. Guo, Y. Yin, C. Dong, G. Yang, and G. Zhou, “On the class imbalance problem,” in _2008 Fourth International Conference on Natural Computation_ , 2008.
  * [37] J. M. Johnson and T. M. Khoshgoftaar, “Survey on deep learning with class imbalance,” _Journal of Big Data_ , 2019.
  * [38] H. Guo, Y. Li, J. Shang, G. Mingyun, H. Yuanyue, and G. Bing, “Learning from class-imbalanced data: Review of methods and applications,” _Expert Syst. Appl._ , pp. 220–239, 2017.
  * [39] C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. J. Cardoso, “Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations,” in _Deep learning in medical image analysis and multimodal learning for clinical decision support_. Springer, 2017.
  * [40] Y. Shi, L. Cui, Z. Qi, F. Meng, and Z. Chen, “Automatic road crack detection using random structured forests,” _IEEE Transactions on Intelligent Transportation Systems_ , 2016.
  * [41] M. Eisenbach, R. Stricker, D. Seichter, K. Amende, K. Debes, M. Sesselmann, D. Ebersbach, U. Stoeckert, and H.-M. Gross, “How to get pavement distress detection ready for deep learning? a systematic approach.” in _International Joint Conference on Neural Networks_ , 2017.
  * [42] M. Seyedhosseini, M. Sajjadi, and T. Tasdizen, “Image segmentation with cascaded hierarchical models and logistic disjunctive normal networks,” in _IEEE International Conference on Computer Vision_ , 2013.
  * [43] J. Staal, M. D. Abràmoff, M. Niemeijer, M. A. Viergever, and B. Van Ginneken, “Ridge-based vessel segmentation in color images of the retina,” _IEEE transactions on medical imaging_ , 2004.
  * [44] S. Xie and Z. Tu, “Holistically-nested edge detection,” in _IEEE international conference on computer vision_ , 2015.
  * [45] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _European Conference on Computer Vision_ , 2018.
  * [46] W.-C. Hung, Y.-H. Tsai, Y.-T. Liou, Y.-Y. Lin, and M.-H. Yang, “Adversarial learning for semi-supervised semantic segmentation,” in _British Machine Vision Conference_ , 2018.
  * [47] J. Su, Y. Lu, S. Pan, B. Wen, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” _arXiv preprint arXiv:2104.09864_ , 2021.


XX XX 8

# A Portal for High-Precision Atomic Data and Computation: Design and Best
Practices

This work has been submitted to the IEEE for possible publication. Copyright
may be transferred without notice, after which this version may no longer be
accessible. Parinaz Barakhshan  University of Delaware, Department of
Electrical and Computer Engineering  Akshay Bhosale  University of Delaware,
Department of Electrical and Computer Engineering  Amani Kiruga  University of
Delaware, Department of Computer and Information Sciences  Rudolf Eigenmann
University of Delaware, Department of Electrical and Computer Engineering
Marianna S. Safronova  University of Delaware, Department of Physics and
Astronomy  Bindiya Arora  Perimeter Institute for Theoretical Physics and Guru
Nanak Dev University

###### Abstract

The atom portal [1], udel.edu/atom, provides the scientific community with
easily accessible high-quality data about properties of atoms and ions, such
as energies, transition matrix elements, transition rates, radiative
lifetimes, branching ratios, polarizabilities, and hyperfine constants. The
data are calculated using a high-precision state-of-the-art linearized
coupled-cluster method, high-precision experimental values are used where
available. All values include estimated uncertainties. Where available,
experimental results are provided with references. This paper provides an
overview of the portal and describes the design as well as applied software
engineering practices.

††journal:

Introduction In a number of present applications, ranging from studies of
fundamental interactions to the development of future technologies, accurate
atomic theory is indispensable to the design and interpretation of
experiments, with direct experimental measurement of relevant parameters being
impossible or infeasible. These data are also in high demand by broader
atomic, plasma, astrophysics, and nuclear physics communities. The need for
high-precision atomic modeling has increased significantly in recent years
with the development of atomic-based quantum technologies for a wide range of
fundamental and practical applications. Further rapid advances will require
accurate knowledge of basic atomic properties, most of which remain highly
uncertain and difficult to measure experimentally. The lack of a comprehensive
atomic database for modern research applications created a bottleneck in
experimental design and analysis as well as industry applications which
motivated the creation of the portal for high-precision atomic data and
computation.

This paper reports the release of Version 2.0 of an online portal that
provides high-precision atomic data derived from computationally intensive
calculations performed on a growing number of atoms and ions. This project
fills the need for high-quality atomic data and software in several scientific
communities. The latest version of the portal, Version 2.0, provides over half
a dozen properties of atoms and ions, including highly charged ions.

As of November 2022, over 2600 users have accessed the website since it was
published in April 2021. Users are connecting from 93 countries. The top ten
countries that have accessed the website were the United States (36%), Germany
(11%), India (10%), United Kingdom (6%), China (5%), Canada (3%), Switzerland
(3%), Singapore (3%), France (2%), and Russia (2%). There have been 5,567
downloads and prints from the website.

The latest version of the portal has added new properties and elements as well
as enhanced the user interface based on community feedback. The majority of
web pages are generated in an automated manner from the data supplied by the
physicists. In this way, human error and repetitive work for different
elements and properties can be reduced and the creation of pages for newly
introduced elements is made easy for non-experts.

The project is an interdisciplinary effort where physicists have produced the
data and computer scientists have developed the website for presenting the
data. Software engineering approaches have been applied to facilitate this
joint portal development, resulting in savings of time and effort.

The remainder of the paper is organized as follows. The portal overview
section provides a brief overview of the portal. A description of how the data
is generated is provided in the generation of the Physics data section. The
section on portal design challenges describes the challenges we encountered
during the development of the portal. The practices employed in this project
are described in the section entitled applied software engineering practices,
followed by conclusions.

## Portal Overview

The current version of the portal provides users with pre-computed data on
atomic properties through an interactive interface, allowing them to view and
download the provided information. This information includes the properties –
transition matrix elements, transition rates, radiative lifetimes, branching
ratios, hyperfine constants, quadrupole moments, scalar and dynamic
polarizabilities, and energies of atoms and ions, including Li, Be+, Na, Mg+,
K, Ca+, Rb, Sr+, Cs, Ba+, Fr, and Ra+. Furthermore, 13 highly charged ions
have been added to this version, including Cs6+, Ba7+, Ce9+, Pr10+, Nd11+,
Nd12+, Nd13+, Sm13+, Sm14+, Sm15+, Eu14+, Cf15+, and Cf17+.

Figure 1 illustrates the home page of the Atom portal, which provides instant
access to information about atoms and ions based on pre-computed properties.

Figure 1: Home page of the atom portal at udel.edu/atom

An example property calculated for these atoms is their polarizability, which
describes their response to an external electric field. Figure 2 illustrates
the graphical interface designed for easy interaction by the user.

Figure 2: Display of polarizability for element Na

Data for 12 elements are calculated using a state-of-the-art linearized
coupled-cluster method that will be discussed in more detail in the next
section. High-precision experimental values are used where available with
references. Data for highly-charged ions is taken from the literature and
references to the original source. The goal of the portal is to provide a
recommended value that users should use for any given property, i.e. this is
the most accurate value given the state-of-the-art knowledge in atomic
physics.

## Generation of the Physics Data

This section explains how physicists generate data. Data provided by the Atom
portal for Li, Be+, Na, Mg+, K, Ca+, Rb, Sr+, Cs, Ba+, Fr, and Ra+ are
computed and evaluated for accuracy based on a long history of Physics
research in developing accurate models for the computation of atomic
properties and corresponding computational codes using a linearized coupled-
cluster method reviewed in [2]. The polarizability computations are described
in [3]. The computations of the quadrupole moments are described in [4].

This method is also referred to as the all-order method in literature as it
involves summing series of dominant many-body perturbation terms to all
orders. In the single-double (SD) all-order approach, single and double
excitations of the Dirac-Fock orbitals are included. The single, double,
particle triple (SDpT) all-order approach also includes classes of the triple
excitations. Omitted higher excitations are estimated by the scaling procedure
which can be started from either SD or SDpT approximations. We carry out four
all-order computations for each of the electric-dipole matrix elements, ab
initio SD and SDpT and scaled SD and SDpT. Either SD or SD scaled data are
taken as final values based on the comparison of different contributions to
the matrix elements.

An algorithm is used to determine the uncertainties of the electric-dipole
matrix elements based on the spread of the four results, size of the
correlation correction, and comparison of different contributions to the
matrix elements [5]. A detailed description of the SD and SDpT all-order
approaches with all formulas is given in the same document.

The computational codes have been made efficient through parallel computation
using OpenMP pragmas embedded in the Fortran 90 code and running on multi-core
machines. Due to the high resource demand for the code runs, computations are
performed on compute nodes of the Caviness computational cluster [7] at the
University of Delaware (UD). These code runs calculate the broad range of
atomic properties mentioned in the introduction.

Energies provide information about the electronic energy level relative to the
ground state energy for each of the elements. The wavelength refers to photon
wavelength for the transition between two electronic energy levels. Transition
matrix elements quantify the strength of electronic transition between two
levels. Transition rates give the probability of an electron making the
transition. Radiative lifetimes are the average times that an excited atom can
remain in that state before emitting a photon. Branching ratios describe the
relative probabilities of all possible transitions. Hyperfine constants
describe the interactions between the nucleus and the electrons and are used
to quantify the resulting splitting of energy levels due to this effect.
Quadrupole moments give information about the shape of an atoms’ electron
cloud. The scalar and dynamic polarizabilities describe the atoms’ response to
an external electric field; dynamic polarizability is used to understand the
response of an atom to a laser light of a given frequency.

The data for Cs6+, Ba7+, Ce9+, Pr10+, Nd11+, Nd12+, Nd13+, Sm13+, Sm14+,
Sm15+, Eu14+, Cf15+, and Cf17+ highly-charged ions have been computed by the
University of Delaware team and collaborators using various methods, including
the all-order method described above, and other methods that combine
configuration interaction and coupled cluster approaches [6]. The use of more
complicated methods is required due to the complicated electronic structures
of some of the ions evaluated. As these computations are recent and represent
the state-of-the-art, we use the generated data for the portal and provide
references to original papers.

## Portal Design Challenges

This section describes the challenges faced in creating a portal that is easy
to populate with data for physicists and easy to navigate for the user
community.

### Automated portal generation from physics data

The portal interface provides functions that allow the user to select the
data, present it in a user-oriented way, and print/download the viewed
information. Users can select a particular atom or ion, choose one or several
electronic states, and obtain data for a particular property they want to
view. The original data underlying the portal is provided by physicists in
well-formatted files that are placed in a source directory. The names and
format of all data files are standardized for automated input.

One design goal was to avoid manual portal changes when physicists provide or
update new data. Data updates could stem from improvements of the
computational codes or from high-precision physics experiments conducted by
any researcher in the community. Another goal was to create the user interface
functionality without the need for dynamic web content support, making the
website portable to a large range of potential web servers. We addressed these
challenges through the development of a portal-generation algorithm,
implemented through Python scripts. The algorithm produces static web pages
with the physics data directly embedded, and with JavaScript methods providing
the needed user functionality.

### Data interface for physics developers

We aimed to make it easy for physicists, both those involved in the project
and external researchers providing experimental data, to supply their
information without needing to understand web technology. To this end, we
defined appropriate data formats for expressing elements, ions, and their
properties. The data is in part generated automatically by the computational
codes. Manual assistance is needed for such tasks as ensuring data validity,
providing data from physics experiments, and uploading the data to the
portal’s source directory. The chosen format uses CSV (comma-separated
values), which can also be generated from an excel spreadsheet.

### Integrating data from different sources

The provision of data from two sources – the computational codes and from
external experimental measurements – poses challenges for both the user
interface and the implementation. For the user interface, we chose a display
that marks data from sources other than our computational codes with a  tag.
Clicking on this tag opens a dialog box providing a reference and a link to
the paper describing the experiment. The presence of experimental data is
signaled to the portal-generation algorithm by the presence of a corresponding
CSV file. The algorithm reads this file and replaces the computed properties
by the experimentally provided values in the display, adding the tags.

### Presentation of complex state designations

Physicists use complex notations for electron states, which involve
superscripts, subscripts, and fractions. An example of such a complex notation
is $5f6p^{2}\;{}^{2}F_{5/2}$. In general, web page designers can use HTML
notation for displaying such expressions. Alternatively, LaTeX libraries
support the same. The state designations also pose challenges for the formats
of physics input and download files, which use plain text CSV. While there are
Unicode representations of superscripts and subscripts for plain text, many
viewing tools only support ASCII code; they display special Unicode characters
as “unknown”, such as question marks or blank squares. We discarded this
solution, as we do not control the users’ viewer tools. The variance in the
first part of the designation requires formatting decisions (consider $6s6p$,
$6s^{2}6p$, $6s^{2}6p^{2}$, and $5d^{3}6s6p$ all of which appear for some
atoms). Our solution is to employ a plain-text, ASCII representation of the
states. We refer to this representation as the dot notation. For example, in
dot notation, the example state mentioned above is represented as 5f.6p2
2F5/2, which indicates there is one 5f electron and two 6f electrons in this
electronic state designation, allowing for automated input and conversion to
the HTML. The second part of the designation always has a format of a
superscript, an upper case letter, and a subscript, simplifying automation.
Such formatting allows to describe complicated state designations and is
already in use by the physics community [11]. An alternative would be the use
of a LaTeX math notation, which is also an ASCII. The LaTeX representation of
the above example would be `5f6p^2\;^2F^0_{5/2}`.

We preferred dot notation for its simplicity and the fact that it is already
in use by physicists. Furthermore, for internal processing, the states are
enumerated. A search function, for example, can look up the state’s number,
instead of performing a string search operation.

### Eliminated redundancy in menu navigation

In our initial version of the portal, it was possible to navigate to certain
information pages on more than one path. For example, to see the transition
rates for atom K, one could choose the Transition Rate tab on the front page
and then select the atom. Alternatively, the user could choose K in the
“Transition Rates” menu, which was available on all pages. While convenient,
this form of redundant path navigation was confusing to some users. We
eliminated the redundancy in Version 2.

### Links to related external websites

The information provided by the Atom portal is complimentary to the data
provided by other websites. In atomic structure, the most commonly used
database is NIST’s ASD (Atomic Spectra Database) [11]. It has the largest
collection of measured atomic energy levels and transition rates. However,
while the accuracy of experimental energies is higher, the accuracy of
experimental transition rates is generally low, with a few exceptions for most
commonly studied elements. The database only focuses on these properties. To
avoid redundancy, we use ASD energy data where available, creating references
to the ASD website instead of duplicating ASD data. While redundant data could
add convenience of having all data in the same page, for example – users would
like to see energy levels next to transition rates – pointing the user to the
ASD website better acknowledges their contribution and improves
maintainability, in case NIST updates ASD information. The transition rates on
our website are expected to be more accurate than on the ASD website, and we
do not use ASD data. We also provide energy levels that are omitted on the ASD
website due to lack of experimental data.

### Data consistency checks

Ensuring that the data shown on the Atom portal is always correct is a non-
trivial task. Currently, substantial manual work is involved in verifying
correctness, while certain checks are automated already. For example, the
display format for uncertainty is provided in the physics input files and also
checked by a method in our portal-generation algorithm. To eliminate the need
for manual intervention, it is vital to automate the data consistency checks,
and is being addressed in our ongoing work.

### User feedback function

Websites can only reach maturity with substantial user feedback. A prominent
user-feedback button serves to solicit suggestions for improvements from
portal users. The same function allows users to see suggestions made by
others, including the implementation status. Users can also view the features
that are forthcoming in the immediate and future releases.

### Tracking Portal Use

The Google Analytics tool allows website administrators to track and analyze
users who visit their websites. A Google Analytics script has been embedded in
each page to collect data, such as the number of users, session statistics,
approximate geographical location, browser type, device information, and the
number of downloads and prints. This information allows us to understand the
portal’s use and make data-driven design decisions.

## Applied Software Engineering Practices

Careful attention to software engineering principles not only increases
maintainability and sustainability; it is also a requirement of our project
sponsor, the National Science Foundation (NSF). We describe the practices, as
defined in an effort to develop best practices for multidisciplinary projects,
in which research software engineering teams support, and collaborate with,
domain science groups [8]. We briefly explain each of the recommended
practices, followed by the specific use in our project and illustrating
examples. The recommendations identify two categories: (i) best practices for
project collaborations and (ii) best practices for software development. For
reference, we use the same titles of the practices; we also use the term
Xperts for computer and data experts supporting domain scientists (physicists,
in our project).

### Diversity of Xpert Backgrounds

Software development teams often include experts with diverse backgrounds. As
a result, training should be planned so that less-familiar practices can be
acquired as needed [9].

In the Atom project, developers from different backgrounds joined the
development team at various stages of the development process. To be able to
contribute to the project, each participant had to undergo a learning process.

For example, to automate the creation of web pages, developers from both
computer science and physics departments were added to the project for a
limited time. The on-boarding process included steps, such as becoming
familiar with the code, gaining the skills needed to make modifications, and
asking questions.

### Breadth of Xpert Skills Needed

Computational and Data-intensive (CDI) applications today incorporate a broad
range of technologies, including computing paradigms, programming languages,
architectures, and algorithms.

The technologies to be used in the project and the skills required for the
development team were determined from the project requirements. We then
engaged project members with the necessary expertise.

### Collaborative Assistance Between Xperts and Domain scientists

A recommended form of interaction between Xperts and domain scientists is
through collaborative assistance, where Xperts work side by side (physically
or virtually) with the domain scientists whose projects they support.
Throughout the joint work, the collaborators tend to pick up sufficient
knowledge of each others’ skills and terminology. Supporting and facilitating
collaboration between the members of the project is crucial. Joint, periodic
reviews of milestones will reduce issues that may impede progress.

The project engaged both physicists and computer scientists in close
collaboration. The collaboration involved weekly meetings to discuss the
progress, provide updates, possibly revise decisions, assign new development
tasks, and discuss next steps.

### Overcoming the Terminology Gap

Computer science and domain science use different jargon. This issue may
complicate the understanding of the problem domain and project requirements
since domain scientists tend to use rich vocabulary. The key to successful
collaboration is keeping the vocabulary to the essentials and investing time
in explaining new terms. Using many examples and frequent feedback from both
sides will help bridge this gap.

At first, in this project, it was difficult to understand the domain
requirements, due to terminology gaps. While many terms were familiar to
domain scientists, getting a clear understanding of their exact meaning was
challenging for computer scientists. As a result, there were numerous
questions asked by both parties so as to understand the terminologies.
Detailed written explanation of physics terminology with examples were
provided by physicists and discussed during the meetings, so they can be used
for future references as well as by new team members.

Additionally, the development team requested that the domain scientists
provide multiple examples of the functionalities expected from the portal.
Prototypes were used to arrive at a common understanding.

### Understanding the Domain Problem and Developing a Project Plan

Xperts must devote substantial time to understand the domain problem, turning
possibly vague ideas into a feasible approach, and developing a clear project
plan. At the start of the development of Version 2, a detailed document
describing new functionalities with examples was developed by the combined
team and thoroughly discussed to guide future work. Having a clear
understanding of the requirements for each version of the project is crucial
and impacts many of the decisions that should be made along the way. Project
requirements need to be well thought out, balanced, and clearly understood by
all involved. It is important that the identified requirements are not dropped
or compromised halfway through the project.

In this project, the requirements of different versions of the project were
identified, documented, discussed, and sometimes prototyped to make sure they
were feasible to be implemented and reflected what the team had envisioned and
then implemented. The implementation was subjected to multiple revisions to
ensure that it met the requirements of not only the physicists but also of the
user community. Many examples were provided by both parties to ensure that the
requirements were intelligible to team members. Prototyping helped team
members visualize how the final pages would look like. It also helped in
highlighting unanticipated design or technical challenges. By giving everyone
a clear vision of the project, the prototype made everyone more involved in
the process of building the pages.

### Prioritize Functional Requirements

A dilemma that is often encountered in the process of planning any project is
the existence of many desirable features but only a short project duration.
Essential features should be differentiated from desirable requirements and
should be strictly prioritized. Keep in mind that the requirements can change
as new insights into a research project emerge, so re-assessments and re-
prioritizing the requirements are necessary.

As part of this project, there was a discussion on how users can interact with
the portal. One option was to display a periodic table to the user, with
elements for which data were available being active and others greyed out. We
decided, however, to prioritize elements and ions for which data was ready. We
began with a simpler visual design of those elements only and created pages
for each of their available properties.

### Source Code Management and Version Control

Source code management and version control systems are used to track the
evolution of an application. Source code management tools greatly facilitate
collaborative software development. Such tools enable software roll-back to a
previous, well-defined state and help developers start a new branch of the
software. The latter can be important if two collaborators want to add their
own separate feature sets.

Because we only wanted team members to have access to the repository, we used
GitHub’s private repository as the version control tool.

### Documentation

There are many advantages to a well-documented application. Here, we will
enlist those advantages that apply specifically to our project. Well-
documented programs are easier to maintain and extend. Documentation makes it
easier for new developers to reuse and extend code, which is a great concern
for research teams where students graduate, leave, and are replaced by new
students. Furthermore, documentation substantially enhances reproducibility.

For this purpose, we published documentation to be used exclusively by the
project team on GitHub Pages. Along with Github repositories of project files,
the use of GitHub pages to share instructions for generating the web pages
greatly improved collaboration between team members. To evaluate the
usefulness of the documentation to new users, testers from departments other
than computer science were added to the project and were asked to follow
instructions provided in the documentation to accomplish specific tasks. Their
feedback helped us improve the documentation.

###  Maintainability & Sustainability

Sustainability [10] ensures that the software will continue to be available in
the future, on new platforms, meeting new needs. A requirement of this project
was maintaining the portal for a long period of time without significant
additional web development, even if funding were to be discontinued. It is
important for physicists to have the ability to add new elements and
properties to the portal without extending the code.

To fulfill this requirement, the process of creating web pages had to be
automated. To this end, we standardized the file naming conventions and file
content formats that were provided by the physics collaborators. A script to
create templates for different properties was developed. Using these
templates, HTML files were generated based on the contents of specific
property files. A Python script can process and modify the files that are read
in and generate HTML files or even graphical representations.

The script has the ability to run on all elements and property files
associated with those elements. This feature is added in case the template
itself is changed, and we want all web pages created for that property to have
the same look. In addition, the script allows the user to pass the name of an
individual element for which the property pages should be created. This option
is used to add a new element or to update a specific data file.

Creating web pages is now as simple as dragging and dropping new data files
into the repository and then running the script in one of its two available
forms in order to generate the relevant web pages.

### User Community Engagement and Exchange

It is vital to engage the user community in the design and development of the
user interface. Prototypes help obtain better feedback from the user
community.

Giving presentations and talks on initial versions of this portal was a way to
collect feedback. Moving forward, we will hold workshops to further engage the
user community.

## Conclusion

The Atom Portal provides the user community with high-quality atomic data. The
current version offers access to the properties of 25 atoms and ions.

We have described the design challenges in creating the portal as well as the
practices that have been applied in the development process. The practices
have led to considerable time savings in generating and updating the Atom
Portal as well as in enhancing maintainability and extensibility.

Future releases will cover up to 100 atoms and ions. The implementation will
be capable of retrieving data from a database as opposed to the current
implementation of embedding the data directly into the web pages. Using
dynamic web development, the creation of portal pages will be fully automated.
Additionally, future updates will include the release of atomic computation
codes, allowing researchers to reproduce some of the data currently provided
through the portal.

## ACKNOWLEDGMENT

This work was supported in part by the National Science Foundation under
Awards No. OAC-1931339 and OAC-2209639. Research at Perimeter Institute is
supported in part by the Government of Canada through the Department of
Innovation, Science and Economic Development and by the Province of Ontario
through the Ministry of Colleges and Universities. We thank Adams Marrs for
work on Version 1 of the portal.

## References

  * [1] P. Barakhshan, A. Marrs, A. Bhosale, B. Arora, R. Eigenmann, and M. S. Safronova, “Portal for high-precision atomic data and computation (version 2.0),” 2022. [Online]. Available: https://www.udel.edu/atom
  * [2] M. S. Safronova and W. R. Johnson, “All-Order Methods for Relativistic Atomic Structure Calculations,” _Advances in Atomic Molecular and Optical Physics_ , vol. 55, pp. 191–233, Jan. 2008. Available: https://doi.org/10.1016/S1049-250X(07)55004-4.
  * [3] B. Arora, M. S. Safronova, and C. W. Clark, “Magic wavelengths for the np-ns transitions in alkali-metal atoms,” _Phys. Rev. A_ , vol. 76, no. 5, p. 052509, Nov. 2007. Available: https://doi.org/10.1103/PhysRevA.76.052509.
  * [4] D. Jiang, B. Arora, and M. S. Safronova, “Electric quadrupole moments of metastable states of Ca+, Sr+ , and Ba+,” _Phys. Rev. A_ , vol. 78, no. 2, p. 022514, Aug. 2008. Available: https://doi.org/10.1103/PhysRevA.78.022514
  * [5] M. S. Safronova, _High-precision calculations of atomic properties and parity nonconservation in systems with one valence electron_. University of Notre Dame, 2001.
  * [6] S. G. Porsev, U. I. Safronova, M. S. Safronova, P. O. Schmidt, A. I. Bondarev, M. G. Kozlov, I. I. Tupitsyn, _Optical clocks based on the Cf 15+ and Cf17+ ions_, _Phys. Rev. A_ , vol. 102, p. 012802, July 2020. Available: https://doi.org/10.1103/PhysRevA.102.012802
  * [7] “CAVINESS, Supporting Researchers At University Of Delaware ,” 2022. [Online]. Available: https://sites.udel.edu/it-rci/compute/community-cluster-program/caviness/
  * [8] Barakhshan, P. & Eigenmann, R. Exchanging Best Practices for Supporting Computational and Data-Intensive Research, The Xpert Network. Practice And Experience In Advanced Research Computing. (2022), https://doi.org/10.1145/3491418.3530293
  * [9] R. Eigenmann and P. Barakhshan, “The Xpert Network, workshop on best practices and tools for computational and data-intensive research,” University Of Delaware, Report P-29, 2019. [Online]. Available: https://cpb-us-w2.wpmucdn.com/sites.udel.edu/dist/6/8980/files/2019/08/ICS-workshop-report-RE4.pdf
  * [10] D. S. Katz, “Defining software sustainability,” 2016. [Online]. Available: https://danielskatzblog.wordpress.com/2016/09/13/defining-software-sustainability/
  * [11] A. Kramida, Yu. Ralchenko, J. Reader and NIST ASD Team, “NIST Atomic Spectra Database (ver. 5.8),” 2021. [Online]. Available: https://physics.nist.gov/asd

Parinaz Barakhshan is a Ph.D. candidate and Research Assistant at the
University of Delaware, Department of Electrical and Computer Engineering. Her
research interests include optimizing compilers and performance evaluation for
High-Performance Computing systems. Contact her at<EMAIL_ADDRESS>
Akshay Bhosale is a Ph.D. candidate and Research Assistant in the Department
of Electrical and Computer Engineering, University of Delaware. His research
interests include Compilers, Optimization techniques and Automatic
Parallelization. Contact him at<EMAIL_ADDRESS>
Amani Kiruga is an undergraduate researcher at the University of Delaware,
Department of Computer and Information Sciences. His research interests
include machine learning and computer vision. Contact him at<EMAIL_ADDRESS>
Rudolf Eigenmann is a Professor at the Department of Electrical and Computer
Engineering, University of Delaware, USA. His research interests include
optimizing compilers, programming methodologies, tools, and performance
evaluation for high-performance computing, as well as the design of
cyberinfrastructure. Contact him at<EMAIL_ADDRESS>
Marianna Safronova is a Professor at the Department of Physics and Astronomy,
University of Delaware, USA. Dr. Safronova’s diverse research interests
include atomic theory method and code development, quantum clocks, searches
for new physics with quantum sensors, and others. Contact her at
<EMAIL_ADDRESS>
Bindiya Arora is currently a PSI fellow at the Perimeter Institute for
Theoretical Physics, Waterloo Canada. She is working as an Assistant Professor
in the Department of Physics, Guru Nanak Dev University, India. Dr. Arora’s
research interests include high-precision atomic structure calculations for
their applications in future technological developments. Contact her at
<EMAIL_ADDRESS>