# SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-
Supervised Medical Image Segmentation

Chenyu You  Yuan Zhou  Ruihan Zhao  Lawrence Staib  James S. Duncan  Life
Fellow, IEEE C. You is with the Department of Electrical Engineering, Yale
University, New Haven, CT 06510 USA (e-mail:chenyu.you@yale.edu).Y. Zhou is
with the Department of Radiology & Biomedical Imaging, Yale University, New
Haven, CT 06510 USA.R. Zhao is with the Department of Electrical and Computer
Engineering, The University of Texas at Austin, TX 78712 USA.L. H. Staib and
J. S. Duncan are with Departments of Radiology & Biomedical Imaging,
Biomedical Engineering, and Electrical Engineering, Yale University, New
Haven, CT 06511 USA (e-mail:james.duncan@yale.edu).

###### Abstract

Automated segmentation in medical image analysis is a challenging task that
requires a large amount of manually labeled data. However, most existing
learning-based approaches usually suffer from limited manually annotated
medical data, which poses a major practical problem for accurate and robust
medical image segmentation. In addition, most existing semi-supervised
approaches are usually not robust compared with the supervised counterparts,
and also lack explicit modeling of geometric structure and semantic
information, both of which limit the segmentation accuracy. In this work, we
present SimCVD, a simple contrastive distillation framework that significantly
advances state-of-the-art voxel-wise representation learning. We first
describe an unsupervised training strategy, which takes two views of an input
volume and predicts their signed distance maps of object boundaries in a
contrastive objective, with only two independent dropout as mask. This simple
approach works surprisingly well, performing on the same level as previous
fully supervised methods with much less labeled data. We hypothesize that
dropout can be viewed as a minimal form of data augmentation and makes the
network robust to representation collapse. Then, we propose to perform
structural distillation by distilling pair-wise similarities. We evaluate
SimCVD on two popular datasets: the Left Atrial Segmentation Challenge (LA)
and the NIH pancreas CT dataset. The results on the LA dataset demonstrate
that, in two types of labeled ratios (i.e., 20% and 10%), SimCVD achieves an
average Dice score of 90.85% and 89.03% respectively, a 0.91% and 2.22%
improvement compared to previous best results. Our method can be trained in an
end-to-end fashion, showing the promise of utilizing SimCVD as a general
framework for downstream tasks, such as medical image synthesis, enhancement,
and registration.

Medical image segmentation, contrastive learning, knowledge distillation,
geometric constraints.

##  1 Introduction

Figure 1: Overview of our SimCVD in training. Given a 3D input volume, our
SimCVD jointly predicts the 3D probability maps and the SDMs of the object
using a student network and a teacher network. The student network is trained
by stochastic gradient descent using two supervised losses
($\mathcal{L}_{\mathrm{seg}}$, $\mathcal{L}_{\mathrm{sdm}}$) and three
unsupervised terms ($\mathcal{L}_{\mathrm{contrast}}$,
$\mathcal{L}_{\mathrm{pd}}$, $\mathcal{L}_{\mathrm{con}}$). Specifically,
$\mathcal{L}_{\mathrm{contrast}}$ is designed to distill “boundary-aware”
knowledge by contrastive learning in the shared latent space, and
$\mathcal{L}_{\mathrm{pd}}$ is designed to exploit structural relationships
among location-paired voxel representations from the encoders. The teacher
network’s weights are updated with a “momentum update” (exponential moving
average) of the student network’s weights.

Medical image segmentation is a popular task in both machine learning and
medical imaging communities [1, 2, 3, 4, 5]. Compared to traditional
segmentation approaches, deep neural network based segmentation methods have
achieved much stronger performance in recent years with huge advances in
representation learning [6, 7, 8, 9, 10, 11, 12, 13]. However, previous state-
of-the-art approaches are mostly trained with a large amount of labeled data,
which pose significant practical challenges in many medical segmentation tasks
where there is a scarcity of labeled data due to the heavy burden of
annotating images.

In recent years, a wide variety of semi-supervised methods [14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26] have been designed to tackle these issues,
which learn from limited labeled data along with a large amount of unlabeled
data, achieving significant improvements in accuracy and greatly reducing the
labeling cost. The common paradigms include adversarial learning, knowledge
distillation, and self-supervised learning. Contrastive learning, a sub-area
of self-supervised learning, has recently been noted as a promising direction
since it has shown great promise in learning useful representations with
limited human supervision [27, 28, 29, 25, 30]. This is often best understood
as pulling together semantically similar (positive) samples and pushing apart
non-similar (negative) samples in a shared latent space. The representations
uncovered by these contrastive objectives are capable of boosting the
performance of any vision system especially in scenarios where the amount of
annotated data available for the downstream tasks is extremely low, which is
well suited for medical image analysis.

Despite advances in semi-supervised learning benchmarks, previous methods
still face several major challenges: (1) Suboptimal performance: although
prior works have achieved promising segmentation accuracy in the setting of
limited annotations, semi-supervised models are usually not robust due to some
information loss, compared with fully-supervised counterparts; (2) Geometric
information loss: previous segmentation networks are poor at characterizing
geometry, i.e., leveraging the intrinsic geometric structure of the images,
such as the object boundary. As a consequence, it is often hard to accurately
recognize object contours; and (3) Generalization ability: considering the
limited amount of training data, training deep models is usually deficient due
to over-fitting and co-adapting [31, 32].

In this work, we address the question: can we advance state-of-the-art voxel-
wise representation learning in a more extreme few-annotation-setting for
medical image segmentation? To this end, we present SimCVD, a simple
contrastive voxel-wise representation distillation framework, which can be
utilized to produce superior voxel-wise representations from unlabeled data
for improving network performance. Our proposed SimCVD, built upon the mean-
teacher framework [33], can address the above-mentioned challenges as follows.
First, SimCVD predicts the output geometric representations with only two
different dropout [34] masks (Figure 1). In other words, we pass two views of
the geometric representations to the mean-teacher model, obtain two
representations as “positive pairs”, by applying two independent dropout
masks, and learn effective representations by efficiently associating
positives and disassociating negatives in the shared latent space. Though this
unsupervised learning strategy is simple, we find this approach is strikingly
effective compared to other common data augmentation techniques (e.g.,
inpainting and local shuffle pixel). More importantly, as we will show, it
achieves comparable performance to previous fully-supervised approaches.
Through a series of thorough analyses, we find that dropout can be viewed as
minimal data augmentation for performance improvement, and it can effectively
regularize the training of deep neural networks, avoid representation collapse
and enhance model generalization.

Second, we attribute the cause of the geometric information loss to the need
for geometric shape constraints. We address this challenge by performing
multi-task learning that jointly predicts a segmentation map along with a
signed distance map (SDM) [35, 36, 37, 12, 22]. The SDM calculates the signed
distance function of the object, i.e., the distance of a voxel from the
boundary of the object, with the sign determined by whether the voxel is
within the object. Thus, it can be viewed as a global shape constraint on the
labeled data. Considering that the SDM can provide a more flexible geometric
measure of the object boundary, we move beyond the supervised learning scheme
and exploit the regularity in geometric shapes among different object classes
through distilling “boundary-aware” knowledge via a contrastive objective
among the unlabeled data. This enables the model to learn boundary-aware
features more effectively by encouraging the networks to produce segmentation
maps with similar distance map distributions on the entire dataset.

Third, it is challenging to train the segmentation model on small training
sets since deep neural networks trained on a limited amount of data are prone
to over-fitting. To this end, we propose to use knowledge distillation (KD),
which has been shown to be effective in segmentation and classification tasks
[38, 39, 40]. The key idea of KD is that a teacher model is first trained, and
then used to guide the training of the student model for improving
generalization ability. In the medical domain, most existing KD methods [41,
42] simply consider the segmentation problem as a pixel/voxel-level
classification problem. In contrast, considering that medical image semantic
segmentation is a structured prediction problem, we present a novel structured
knowledge pair-wise distillation, which further use the structural knowledge
from the mean-teacher model, while avoiding co-adapting and over-fitting.

Our contributions are summarized as follows. First, we propose a novel
contrastive distillation model termed SimCVD featured by (i) boundary-aware
representations that incorporate rich information of the object shape, (ii) a
distillation objective which contrasts different distance map distributions
jointly in the shared latent space, and (iii) a pair-wise distillation
objective to further distill pair-wise structural knowledge. Second, we
demonstrate that, in the setting of very limited annotation, simply using
dropout can deliver more robust end-to-end segmentation performance compared
to heavily relying on a large amount of labeled data. Third, we conduct
experiments on two popular benchmark datasets to evaluate SimCVD. The results
demonstrate that SimCVD significantly outperforms other state-of-the-art semi-
supervised approaches, while achieving competitive performance compared to
fully-supervised counterparts.

##  2 Related Work

Semi-Supervised Medical Image Segmentation In recent years, substantial
efforts [15, 16, 17, 43, 44, 45, 14, 21, 46, 47, 48, 49, 50, 51, 52, 50, 53,
54, 55, 56, 12, 19, 57, 58, 59, 60, 61, 62, 63, 64] have been devoted to
incorporating unlabeled data to improve network performance due to limited
annotations. Yu et al.[21] investigated an uncertainty map based on the mean-
teacher framework [33] to guide the student network to capture better
features. Li et al.[22] proposed to use signed distance fields for boundary
prediction to improve the performance. Also, Luo et al.[24] proposed a dual-
task-consistency (DTC) model for semi-supervised medical image segmentation by
jointly predicting the pixel-wise segmentation maps and the global-level level
set representations on the unlabeled data. Our method aims at a more practical
and challenging scenario: we train our model in a more extreme few-annotation
setting that relies only on a small number of annotations, while achieving
superior segmentation accuracy.

Contrastive Learning Self-supervised learning (SSL) [65, 66, 67, 62] has
provided robust benefits to vision tasks by learning effective visual
representations from unlabeled data in an unsupervised setting. It is based on
a commonly-held belief that superior performance gains can be achieved through
improved representation learning. Recently, contrastive learning, a type of
self-supervised learning, has received a lot of interests [65, 68, 69, 70, 27,
71, 25, 30, 72, 73]. The key idea of contrastive learning is to learn powerful
representations that optimize similarity constraints to discriminate similar
pairs (positive) and dissimilar pairs (negative) within a dataset. The primary
stream of subsequent work focuses on the choice of dissimilar pairs, which is
critical to the quality of learned representations. The loss function used to
quantify the contrast is chosen from several options, such as InfoNCE [74],
Triplet [75], and so on. Recent studies [68, 70] introduced memory bank or
momentum contrast to use more negative samples for contrast computation. In
the context of medical imaging, Chaitanya et al.[25] extended a contrastive
learning framework to extract global and local cues in a stage-wise way, which
requires human intervention and extensive training time. In contrast to
Chaitanya et al.[25], our unified work focuses on explicit modeling of the
intrinsic geometric structure of the semantic objects in an end-to-end manner,
and hence is able to recognize object boundaries more effectively and
efficiently.

Knowledge Distillation The idea of knowledge distillation is to minimize the
KL-divergence between the output distributions of the teacher model and the
student model, and thus avoid over-fitting. KD has been applied to a variety
of tasks [76, 77, 78, 79, 80, 81], including image classification [38, 82, 83,
84] and semantic segmentation [85, 40]. Recent works [83, 82] found that the
student model can outperform the teacher model when they share the same
network architecture. Zhang et al.[86] proposed to collaboratively train
multiple student models with co-distillation, which improves performance of
those individual models. At the same time, in the context of medical imaging,
among those existing state-of-the-art KD methods, the self-ensemble mean-
teacher framework [33] is widely explored for image segmentation. Different
from the existing methods that separately exploit class probabilities for each
voxel, we consider knowledge distillation as a structured prediction problem
by matching the relational similarity among all pairs of voxels from the
encoded feature maps of the mean-teacher model. We have found that our
approach significantly improves learning better voxel-wise representations.

##  3 Method

In this section, we introduce SimCVD, a semi-supervised segmentation network,
which is built from scratch by effectively leveraging scarce labeled data and
ample unlabeled data for improving end-to-end voxel-wise representation
learning (See Figure 1). We first overview our proposed SimCVD and then
describe the task formulation of SimCVD. Finally, we detail each component of
SimCVD in the following subsections.

###  3.1 Overview

We aim to construct an end-to-end voxel-wise contrastive distillation
algorithm to learn boundary-aware representations in the setting of extremely
few annotations for volumetric medical imaging segmentation. Although the
accuracy of supervised models is usually higher than that of semi-supervised
models, the former requires much more labeled data than the latter. In many
clinical situations, we only have few annotated data but a large amount of
unlabeled data. This situation necessitates a semi-supervised segmentation
algorithm that can utilize the unlabeled data to improve the segmentation
performance.

To this end, we propose a novel contrastive distillation framework to advance
state-of-the-art voxel-wise representation learning. In particular, our base
multi-task segmentation network tackles two tasks simultaneously:
classification and regression. Specifically, the segmentation network takes
the input volume batch and jointly predicts the probability maps
(classification) and the SDMs of the object (regression). To obtain better
representations, we propose to perform structured distillation in the latent
feature space, followed by contrasting the boundary-aware features in the
prediction space, to learn more effective boundary-aware representations from
3D unlabeled data by regularizing the embedding space and exploring the
geometric and spatial context of training voxels. At test time, we remove the
mean teacher and two projection heads, and only the student network is
deployed for the medical segmentation tasks.

###  3.2 Task Formulation

In this work, we consider a set of training data (3D images) including $N$
labeled data and $M$ unlabeled data, where $N\ll M$. For simplicity, we denote
the small set of labeled data as
$\mathcal{D}_{l}=\left\\{\left(\mathbf{X}_{i},\mathbf{Y}_{i},\mathbf{Y}^{\mathrm{sdm}}_{i}\right)\right\\}_{i=1}^{N}$,
and abundant unlabeled data as
$\mathcal{D}_{u}=\\{\mathbf{X}_{i}\\}_{i=N+1}^{N+M}$, where
$\mathbf{X}_{i}\in\mathbb{R}^{H\times W\times D}$ is the volume input,
$\mathbf{Y}_{i}\in\\{0,1\\}^{H\times W\times D}$ is the ground-truth label,
and $\mathbf{Y}^{\mathrm{sdm}}_{i}\in\mathbb{R}^{H\times W\times D}$ is the
computed ground truth SDMs from $\mathbf{Y}_{i}$, which measures the distance
from each voxel to the object boundary. Every 3D image $\mathbf{X}_{i}$
consists of a set of 2D image slices
$\mathbf{X}_{i}=[\mathbf{x}_{i,1},...,\mathbf{x}_{i,D}]$ where
$\mathbf{x}_{i,j}\in\mathbb{R}^{H\times W}$.

Our proposed SimCVD framework consists of a mean-teacher network,
$\mathcal{F}_{t}(\mathbf{X};\theta_{t})$, and a student network,
$\mathcal{F}_{s}(\mathbf{X};\theta_{s})$. Inspired by recent work [14, 33],
the optimization of these two networks can be achieved with an exponential
moving average (EMA) which uses a weighted combination of the parameters of
the student network and the parameters of the teacher network to update the
latter. This strategy has been widely shown to improve training stability and
the model’s final performance. Motivated by this idea, our training strategy
is divided into two steps. At each iteration, we first optimize the student
network $\mathcal{F}_{s}$ by stochastic gradient descent. Then we update the
teacher weights $\theta_{t}$ using an exponential moving average of the
student weights $\theta_{s}$.

The inputs to the two networks are perturbed versions of the same image. That
is, given a volume input $\mathbf{X}_{i}$, we first add different
perturbations (i.e., affine transformation and random crop) to generate two
different images $\mathbf{X}_{i}^{t}$ and $\mathbf{X}_{i}^{s}$. We then feed
$\mathcal{F}_{t}$ and $\mathcal{F}_{s}$ with these two corresponding augmented
images to obtain two confidence score (probability) maps $\mathbf{Q}^{t}_{i}$
and $\mathbf{Q}^{s}_{i}$.

Before we present our proposed SimCVD in detail, we first describe our base
architecture below.

###  3.3 Base Architecture

Our base architecture adopts V-Net [21] as the network backbone, which
consists of an encoder network $e_{t}:\mathbb{R}^{H\times W\times
D}\rightarrow\mathbb{R}^{H^{\prime}\times W^{\prime}\times D^{\prime}\times
D_{e}}$ and a decoder network $d_{t}:\mathbb{R}^{H^{\prime}\times
W^{\prime}\times D^{\prime}\times D_{e}}\rightarrow[0,1]^{H\times W\times
D}\times[-1,1]^{H\times W\times D}$ for the teacher network, and similarly
$e_{s}$, $d_{s}$ for the student network, i.e., $\mathcal{F}_{t}=d_{t}\circ
e_{t}$ and $\mathcal{F}_{s}=d_{s}\circ e_{s}$.
$H^{\prime},W^{\prime},D^{\prime}$ are the size of the hidden pattern and
$D_{e}$ is the encoded feature dimension. Inspired by previous work on medical
imaging segmentation [12, 22], we incorporate multi-task learning into
$\mathcal{F}$ to jointly perform both classification and regression tasks.

Given input $\mathbf{X}_{i}$, the classification branch is designed to
generate the probability map $\mathbf{Q}_{i}^{s}\in[0,1]^{H\times W\times D}$,
and the regression branch is designed to predict the SDM
$\mathbf{Q}_{i}^{s,\mathrm{sdm}}\in[-1,1]^{H\times W\times D}$. The design of
the regression branch is simple yet effective, only including the hyperbolic
tangent function. This design brings two clear benefits: (1) we can eventually
encode rich geometric structure information to improve segmentation accuracy,
and (2) we can implicitly enforce continuity and smoothness terms for better
segmentation maps. Similarly, we have outputs $\mathbf{Q}_{i}^{t}$ and
$\mathbf{Q}_{i}^{t,\mathrm{sdm}}$ from the teacher network.

Supervised Loss $\mathcal{L}_{\mathrm{sup}}$ For training on labeled data, we
define the supervised loss as:

$\displaystyle\mathcal{L}_{\mathrm{sup}}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_{\mathrm{seg}}(\mathbf{Q}_{i}^{s},\mathbf{Y}_{i})+\frac{\alpha}{N}\sum_{i=1}^{N}\mathcal{L}_{\mathrm{mse}}(\mathbf{Q}^{s,\mathrm{sdm}}_{i},\mathbf{Y}_{i}^{\mathrm{sdm}}),$
(1)

where $\mathcal{L}_{\mathrm{seg}}$ denotes the segmentation loss (Dice and
Cross-entropy) [21], and $\mathcal{L}_{\mathrm{mse}}$ is the mean squared
error loss. $\alpha$ is a hyperparameter. Note that the SDM loss [12] is
imposed as geometric constraints in training.

Table 1: Quantitative segmentation results on the LA dataset. The backbone
network of all evaluated methods are V-Net.

Method |  # scans used | Metrics
---|---|---
Labeled | Unlabeled | Dice[%] | Jaccard[%] | ASD[voxel] | 95HD[voxel]
V-Net [7] | 80 | 0 | 91.14 | 83.82 | 1.52 | 5.75
V-Net | 16 | 0 | 86.03 | 76.06 | 3.51 | 14.26
DAN [15] | 16 | 64 | 87.52 | 78.29 | 2.42 | 9.01
CPS [87] | 16 | 64 | 87.87 | 78.61 | 2.16 | 12.87
MT [33] | 16 | 64 | 88.42 | 79.45 | 2.73 | 13.07
Entropy Mini [88] | 16 | 64 | 88.45 | 79.51 | 3.72 | 14.14
UA-MT [21] | 16 | 64 | 88.88 | 80.21 | 2.26 | 7.32
ICT [89] | 16 | 64 | 89.02 | 80.34 | 1.97 | 10.38
SASSNet [22] | 16 | 64 | 89.27 | 80.82 | 3.13 | 8.83
DTC [24] | 16 | 64 | 89.42 | 80.98 | 2.10 | 7.32
Chaitanya et al.[25] | 16 | 64 | 89.94 | 81.82 | 2.66 | 7.23
SimCVD (ours) | 16 | 64 | 90.85 | 83.80 | 1.86 | 6.03
V-Net [7] | 8 | 0 | 79.99 | 68.12 | 5.48 | 21.11
DAN [15] | 8 | 72 | 75.11 | 63.47 | 3.57 | 19.04
CPS [87] | 8 | 72 | 84.09 | 73.17 | 2.41 | 22.55
MT [33] | 8 | 72 | 84.24 | 73.26 | 2.71 | 19.41
Entropy Mini [88] | 8 | 72 | 85.90 | 75.60 | 2.74 | 18.65
UA-MT [21] | 8 | 72 | 84.25 | 73.48 | 3.36 | 13.84
ICT [89] | 8 | 72 | 85.39 | 74.84 | 2.88 | 17.45
SASSNet [22] | 8 | 72 | 86.81 | 76.92 | 3.94 | 12.54
DTC [24] | 8 | 72 | 87.49 | 78.03 | 2.37 | 9.06
Chaitanya et al.[25] | 8 | 72 | 84.95 | 74.77 | 3.70 | 10.68
SimCVD (ours) | 8 | 72 | 89.03 | 80.34 | 2.59 | 8.34

Figure 2: Visual comparisons with other methods on LA dataset. As observed,
SimCVD achieves superior performance with more accurate borders and shapes. We
train all the evaluated methods in the setting of 8 annotated images. Red and
blue denote the predictions and ground truths, respectively.

###  3.4 Boundary-aware Contrastive Distillation

Many prior methods distill knowledge merely in the shared prediction space by
delivering the student network that matches the accuracy of the teacher
network. However, this strategy is not robust for the following reasons: (1)
the learned voxel-wise representations from the mean-teacher model are usually
not robust due to the lack of geometric information; (2) the segmentation
model still suffers from generalization issues; and (3) the network
performance needs to be further improved. Therefore, we propose to perform
boundary-aware contrastive distillation to train our model for better
segmentation accuracy.

Our method differs from previous state-of-the-art methods in three aspects:
(1) SimCVD imposes the global consistency in object boundary contours to
capture more effective geometric information; (2) previous methods follow the
standard setting in considering the relations of local patches, while SimCVD
aims to exploit correlations among all pairs of voxels to improve robustness;
and (3) due to the computational cost, SimCVD does not use a large memory
bank. SimCVD trains the contrastive objective as an auxiliary loss during the
volume batch updates. To specify our voxel-wise contrastive distillation
algorithm on unlabeled sets, we define two discrimination terms: boundary-
aware contrastive loss and pair-wise distillation loss.

Boundary-aware Contrastive Loss $\mathcal{L}_{\mathrm{contrast}}$ We describe
our unsupervised boundary-aware contrastive objective as follows. Our key idea
is to make use of “boundary-aware” knowledge by a contrastive learning
objective that enforces the consistency of the predicted SDM outputs on the
unlabeled set during training. The key ingredient to working with two views of
input images is to apply dropout as mask. Specifically, given the collection
of an input volume $\mathbf{X}_{i}$, the student SDM
$\mathbf{Q}_{i}^{s,\mathrm{sdm}}$, the teacher SDM
$\mathbf{Q}_{i}^{t,\mathrm{sdm}}$, we first directly add them up to build two
boundary-aware features:
$\mathbf{Q}_{i}^{s,\mathrm{ba}}=\mathbf{X}_{i}+\mathbf{Q}_{i}^{s,\mathrm{sdm}}$
and
$\mathbf{Q}_{i}^{t,\mathrm{ba}}=\mathbf{X}_{i}+\mathbf{Q}_{i}^{t,\mathrm{sdm}}$.
Then, we feed them into the projection heads with two independent dropout
masks $z_{i}^{s},z_{i}^{t}$, and contrast positives and negatives by using the
InfoNCE loss. We denote the same slice from the two boundary-aware features as
positive, and slices at different locations or from different inputs as
negative.

The boundary-aware features are created by adding the original 3D volume to
the SDM because we want to fuse both the distance and the intensity
information. Another way to achieve this is concatenation – adding another
dimension to the feature tensor – requires a more complex projection head
which is more prone to over-fitting. Thus, the projection head
${\mathcal{H}}\\!:\\!\mathbb{R}^{H\times W\times
D}\\!\\!\rightarrow\\!\mathbb{R}^{D_{h}\times D}$ encodes each 2D slice to a
$D_{h}$-dimensional feature vector. The implementation is simple, which
includes an alpha dropout [90], an adaptive average pooling, and a 3-layer
multilayer perceptron (MLP). Here the MLP is designed to convert each 2D slice
to a vector.

Denoting the output of the projection head as
$\mathbf{H}_{i}^{s}=\mathcal{H}(\mathbf{Q}_{i}^{s,\mathrm{ba}};z_{i}^{s})$,
$\mathbf{H}_{i}^{t}=\mathcal{H}(\mathbf{Q}_{i}^{t,\mathrm{ba}};z_{i}^{t})$,
and the $jth$ row of $\mathbf{H}_{i}$ as $\mathbf{h}_{i,j}$, the InfoNCE loss
[74] is defined by:

$\mathcal{L}(\mathbf{h}_{i,j}^{t},\mathbf{h}_{i,j}^{s})=-\log\frac{\exp(\mathbf{h}_{i,j}^{t}\\!\cdot\\!\mathbf{h}_{i,j}^{s}/\tau)}{\sum_{k,l}\exp({\mathbf{h}_{i,j}^{t}\\!\cdot\\!\mathbf{h}_{k,l}^{s}/\tau})},\\!\\!\vspace{-2pt}$
(2)

where $\tau$ is a temperature hyperparameter. The indices $k$ and $l$ in the
denominator are randomly sampled from a mini-batch of images such that $B$ 2D
slices are sampled in total. $i$ and $j$ denote the 3D image index and slice
index, respectively. The $\mathbf{h}_{k,l}^{s}$’s in the denominator that are
not $\mathbf{h}_{i,j}^{s}$ are called negative samples. Inspired by the recent
success [25], our boundary-aware contrastive loss is defined as:

$\mathcal{L}_{\mathrm{contrast}}=\frac{1}{|\mathcal{N}^{+}|}\sum_{\forall(i,j)\in\mathcal{N}^{+}}[\mathcal{L}(\mathbf{h}_{i,j}^{t},\mathbf{h}_{i,j}^{s})+\mathcal{L}(\mathbf{h}_{i,j}^{s},\mathbf{h}_{i,j}^{t})],\vspace{-1pt}$
(3)

where $\mathcal{N}^{+}=\\{(i,j):i=N+1,\dots,N+M,j=1,\dots,D\\}$ denotes a
collection of the positive 2D slice pairs. Note that the index $i$ in
$\mathcal{N}^{+}$ is over all the unlabeled data, hence these data affect the
training of $\mathcal{F}_{s}$ and $\mathcal{F}_{t}$.

Table 2: Quantitative segmentation results on the pancreas dataset. The
backbone network of all evaluated methods are V-Net.

Method |  # scans used | Metrics
---|---|---
Labeled | Unlabeled | Dice[%] | Jaccard[%] | ASD[voxel] | 95HD[voxel]
V-Net [7] | 62 | 0 | 77.84 | 64.78 | 3.73 | 8.92
V-Net | 12 | 0 | 62.42 | 48.06 | 4.77 | 22.34
MT [33] | 12 | 50 | 71.29 | 56.69 | 2.82 | 16.31
DAN [15] | 12 | 50 | 68.67 | 53.97 | 3.07 | 15.78
CPS [87] | 12 | 50 | 69.28 | 54.02 | 3.07 | 15.34
Entropy Mini [88] | 12 | 50 | 69.33 | 54.32 | 2.92 | 15.29
UA-MT [21] | 12 | 50 | 72.43 | 57.91 | 4.25 | 11.01
ICT [89] | 12 | 50 | 70.06 | 55.66 | 2.98 | 13.05
SASSNet [22] | 12 | 50 | 70.47 | 55.74 | 4.26 | 10.95
DTC [24] | 12 | 50 | 74.07 | 60.17 | 2.61 | 10.35
Chaitanya et al.[25] | 12 | 50 | 70.79 | 55.76 | 6.08 | 15.35
SimCVD (ours) | 12 | 50 | 75.39 | 61.56 | 2.33 | 9.84

Figure 3: Visual comparisons with other methods on the pancreas dataset. We
train all the evaluated methods in the setting of 12 annotated images. Red and
blue denotes the predictions and ground truths, respectively.

Pair-wise Distillation Loss $\mathcal{L}_{\mathrm{pd}}$ On one hand, boundary-
aware contrastive objectives uncover distinctive global boundary-aware
representations that benefit the training of downstream tasks, e.g. object
classification, when limited labeled data is available. On the other hand,
dense predictive tasks, e.g. semantic segmentation, may require more
discriminative spatial representations. As complementary to boundary-aware
contrastive objectives, a promising local pair-wise strategy is vital for the
medical image segmentation tasks. With this insight, we propose to perform
voxel-to-voxel pair-wise distillation to explicitly explore structural
relationships between voxel samples to improve spatial labeling consistency.

In our implementation, we enforce such a constraint on the hidden patterns
from the encoders $e_{t}$ and $e_{s}$. Specifically, let
$\mathbf{V}_{i}^{t}\in\mathbb{R}^{H^{\prime}W^{\prime}D^{\prime}\times D_{e}}$
and $\mathbf{V}_{i}^{s}\in\mathbb{R}^{H^{\prime}W^{\prime}D^{\prime}\times
D_{e}}$ be the first-3-dimension-flattened hidden patterns of
$e_{t}(\mathbf{X}_{i})$ and $e_{s}(\mathbf{X}_{i})$ respectively, and
$\mathbf{v}_{i,j}$ be the $jth$ row of $\mathbf{V}_{i}$. The pair-wise
distillation loss is defined as:

$\mathcal{L}_{\mathrm{pd}}=-\frac{1}{M}\sum_{i=N+1}^{N+M}\sum_{j=1}^{H^{\prime}W^{\prime}D^{\prime}}\log\frac{\exp(s(\mathbf{v}_{i,j}^{s},\mathbf{v}_{i,j}^{t}))}{\sum_{k}\exp(s(\mathbf{v}_{i,j}^{s},\mathbf{v}_{i,k}^{t}))},\vspace{-3pt}$
(4)

where
$s(\mathbf{v}_{1},\mathbf{v}_{2})=\frac{\mathbf{v}_{1}\cdot\mathbf{v}_{2}}{\|\mathbf{v}_{1}\|\|\mathbf{v}_{2}\|}$
measures the cosine of the angle between two $\mathbf{v}$’s as their
similarity. Note again that this loss also involves all the unlabeled data.

Consistency Loss $\mathcal{L}_{\mathrm{con}}$ Inspired by recent work [14,
33], consistency is designed to further encourage training stability and
performance improvements on the unlabeled set. In our implementations, we
first perform different perturbation operations on the unlabeled input volume
$\mathbf{X}_{i}$, i.e., adding noise $\eta_{i}$, and then define the
consistency loss as:

$\mathcal{L}_{\mathrm{con}}=\frac{1}{M}\sum_{i=N+1}^{N+M}\mathcal{L}_{\mathrm{mse}}\left(\mathcal{F}_{s}\left(\mathbf{X}_{i}^{s}+\eta_{i}^{s}\right),\mathcal{F}_{t}\left(\mathbf{X}_{i}^{t}+\eta_{i}^{t}\right)\right).\vspace{-3pt}$
(5)

Overall Training Objective SimCVD is a general semi-supervised framework for
combining contrastive distillation with geometric constraints. In our
experiments, we train SimCVD with two objective functions — a supervised
objective and an unsupervised objective. For the labeled data, we define the
supervised loss in Section 3.3. For the unlabeled data, the unsupervised
training objective consists of the boundary-aware contrastive loss, pair-wise
distillation loss, and consistency loss in Section 3.4. The overall loss
function is:

$\mathcal{L}=\mathcal{L}_{\mathrm{sup}}+\lambda\mathcal{L}_{\mathrm{contrast}}+\beta\mathcal{L}_{\mathrm{pd}}+\gamma\mathcal{L}_{\mathrm{con}},\vspace{-3pt}$
(6)

where $\lambda,\beta,\gamma$ are hyperparameters that balance each term.

##  4 Experimental Setup

###  4.1 Dataset and Pre-processing

We evaluated our approach on two popular benchmark datasets: the Left Atrium
(LA) MR dataset from the Atrial Segmentation
Challenge111http://atriaseg2018.cardiacatlas.org/, and the NIH pancreas CT
dataset [91]. For the Left Atrium dataset, it comprises 100 3D gadolinium-
enhanced MR imaging scans (GE-MRIs) with expert annotations, with an isotropic
resolution of $0.625\times 0.625\times 0.625\text{mm}^{3}$. Following the
experimental setting in [21], we use 80 scans for training, and 20 scans for
evaluation. We employ the same pre-processing methods by cropping all the
scans at the heart region and normalizing the intensities to zero mean and
unit variance. All the training sub-volumes are augmented by random cropping
to $112\times 112\times 80\text{mm}^{3}$. For the pancreas dataset, it
contains 82 contrast-enhanced abdominal CT scans. Following the experimental
settings in [24], we randomly select 62 scans for training, and 20 scans for
evaluation. In the pre-processing, we first truncate the intensities of the CT
images into the window [$-125$, $275$] HU [92], and then resample all the data
into a fixed isotropic resolution of $1.0\times 1.0\times 1.0\text{mm}^{3}$.
Finally, we crop all the scans centering at the pancreas region, and normalize
the intensities to zero mean and unit variance. All the training sub-volumes
are augmented by random cropping to $96\times 96\times 96\text{mm}^{3}$. In
this study, we compare all the methods on LA and the pancreas dataset with
respect to 20% labeled ratio. To emphasize the effectiveness of SimCVD, we
further validate all the methods with respect to 10% labeled ratio on LA
dataset.

###  4.2 Implementation Details

In this study, all evaluated methods are implemented in PyTorch, and trained
for $6000$ iterations on an NVIDIA 1080Ti GPU with a batch size of $4$. For
data augmentation, we use standard data augmentation techniques (i.e., random
rotation, flipping, and cropping). We set the hyper-parameters $\alpha$,
$\lambda$, $\beta$, $\gamma$, $\tau$ as $0.1$, $0.5$, $0.1$, $0.1$, $0.5$,
respectively. For the projection head, we set $p=0.1$ in the AlphaDropout
layer, and output size $128\times 128$ for AdaptiveAvgPool2d. We use SGD
optimizer with a momentum of $0.9$ and a weight decay of $0.0005$ to
optimize network parameters. The initial learning rate is set as $0.01$ and
divided by $10$ every $3000$ iterations. For EMA updates, we follow the
experimental setting in [21], where the EMA decay rate $\alpha$ is set to
$0.999$. We use the time-dependent Gaussian warming-up function
$\Psi_{\text{con}}(t)\\!=\\!\exp{\left(-5\left(1-t/t_{\max}\right)^{2}\right)}$
to ramp up parameters, where $t$ and $t_{\max}$ denote the current and the
maximum training step, respectively. For fairness, we do not adopt any post-
processing step.

In the testing stage, we adopt four metrics to evaluate the segmentation
performance: Dice coefficient (Dice), Jaccard Index (Jaccard), 95% Hausdorff
Distance (95HD), and Average Symmetric Surface Distance (ASD). Following [21,
24, 93], we adopt a sliding window strategy, which uses a stride with
$18\times 18\times 4$ for the LA and $16\times 16\times 16$ for the pancreas.

##  5 Results

###  5.1 Experiments: Left Atrium

We compare SimCVD with published results from previous state-of-the-art semi-
supervised segmentation methods, including V-Net [7], MT [33], DAN [15], CPS
[87], Entropy Mini [88], UA-MT [21], ICT [89], SASSNet [22], DCT [24], and
Chaitanya et al.[25] on the LA dataset in two labeled ratio settings (i.e.,
10% and 20%).

The quantitative results on the LA dataset are shown in Table 1. SimCVD
substantially improved the segmentation accuracy in both 10% and 20% labeled
cases. The results are visualized in Fig 2. Specifically, in the setting of
20% labeled ratio, our proposed SimCVD raises the previous best average
results from 89.94% to 90.85% and from 81.82% to 83.80% in terms of Dice and
Jaccard, even achieving comparable performance to the fully supervised
baseline. Using the 10% labeled ratio, SimCVD further advances the state-of-
the-art results from 87.49% to 89.03% in Dice. The gains in Jaccard, ASD, and
95HD are also substantial, achieving 80.34%, 2.59, and 8.34, respectively.
This suggests that: (1) taking voxel samples with a contrastive objective
yields better voxel embeddings; (2) incorporating pair-wise spatial labeling
consistency can boost the performance by accessing more structural knowledge;
and (3) utilizing a geometric constraint (i.e., SDM) is capable of helping
identify more accurate boundaries. Leveraging all these aspects, we can
observe consistent performance gains.

###  5.2 Experiments: Pancreas

To further evaluate the effectiveness of SimCVD, we compare our model on the
pancreas CT dataset. Experimental results on the pancreas CT dataset are
summarized in Table 2. We observe that our model consistently outperforms all
previous methods, achieving up to 6.72% absolute improvements in Dice. As
shown in Figs. 2 and 3, our method is capable of predicting high-quality
object segmentation, considering the fact that the improvement in such a
setting is difficult. This demonstrates: (1) the necessity of comprehensively
considering both boundary-aware contrast and pair-wise distillation; and (2)
the efficacy of global shape information. Compared to the previous strong
models, our approach achieves large improvements on all the datasets,
demonstrating its effectiveness.

Table 3: Ablation on (a) model component: SimCVD w/o SDM; SimCVD w/o adaptive
max pooling; (b) loss formulation: SimCVD w/o
$\mathcal{L}_{\mathrm{contrast}}$; SimCVD w/o $\mathcal{L}_{\mathrm{pd}}$;
SimCVD w/o $\mathcal{L}_{\mathrm{sdm}}$, compared to the baseline and our
proposed SimCVD.

| Method | Metrics | p-value (vs. SimCVD, [%])
---|---|---|---
| Dice[%] | Jaccard[%] | ASD[voxel] | 95HD[voxel]
| Baseline (UA-MT) | 84.24 | 73.26 | 2.71 | 19.41 | 0.019
(a) | SimCVD w/o SDM | 88.24 | 79.07 | 4.19 | 11.43 | 2.47
SimCVD w/ Adaptive Max Pooling | 88.32 | 79.26 | 3.79 | 14.69 | 0.33
(b) | SimCVD w/o $\mathcal{L}_{\mathrm{contrast}}+\mathcal{L}_{\mathrm{pd}}$ | 84.97 | 74.49 | 6.13 | 19.98 | 0.018
SimCVD w/o $\mathcal{L}_{\mathrm{contrast}}$ | 85.13 | 74.57 | 5.97 | 16.61 | 2.9e-6
SimCVD w/o $\mathcal{L}_{\mathrm{pd}}$ | 88.11 | 78.89 | 2.89 | 12.58 | 2.02
| SimCVD w/o $\mathcal{L}_{\mathrm{sdm}}$ | 88.85 | 80.03 | 2.71 | 9.02 | 5.14
| SimCVD | 89.03 | 80.34 | 2.59 | 8.34 | -

##  6 Ablation Study

In this section, we conduct extensive studies to better understand SimCVD. We
justify the inner working of SimCVD from two perspectives: (1) boundary-aware
contrastive distillation (Section 6.1), and (2) the projection head (Section
6.2). In these studies, we evaluate our proposed method on the LA dataset with
10% labeled ratio (8 labeled and 72 unlabeled).

###  6.1 Analysis on Boundary-aware Contrastive Distillation

Ablation on Model Component In the model formulation, our motivation is to
advance state-of-the-art voxel-wise representations by capturing the geometric
and semantic information in 3D space. Rather than transferring knowledge
across confidence score maps directly, our SimCVD distills “boundary-aware”
knowledge from the teacher network. To validate the idea of boundary-aware
contrastive distillation, we compare SimCVD to an ablative baseline (i.e.,
SimCVD w/o SDM). Table 3 (a) compares each component of SimCVD in the 10%
labeled setting. First, we observe that removing the SDMs in training hurts
the segmentation performance by $\\!-0.79\%$, $\\!-1.27\%$, $\\!-1.6$, and
$\\!-3.09$ absolute differences in terms of Dice, Jaccard, ASD, and 95HD. This
confirms our intuition that the learned boundary-aware representations provide
a good prior for improving segmentation accuracy. We also find that using
adaptive max pooling strategy (i.e., SimCVD w/ adaptive max pooling) largely
degrades the segmentation performance. Our segmentation results demonstrate
that SimCVD is an effective approach, outperforming the best previous method
with $\\!+0.71\%$, $\\!+1.08\%$, $\\!+1.20$, and $\\!+6.35$ absolute
differences in terms of Dice, Jaccard, ASD, and 95HD. We hypothesize that it
is because “w/ adaptive max pooling” leads to information loss during
training.

Table 4: Ablation on dropout rates $p$ and pooling size.

| Method | Metrics
---|---|---
| Dice[%] | Jaccard[%] | ASD[voxel] | 95HD[voxel]
Dropout | $p=0.0$ | 87.69 | 78.23 | 2.49 | 11.03
$p=0.01$ | 87.98 | 78.69 | 3.08 | 11.17
$p=0.02$ | 87.99 | 78.70 | 2.60 | 9.03
$p=0.05$ | 88.01 | 78.71 | 2.78 | 10.60
$p=0.1$ | 89.03 | 80.34 | 2.59 | 8.34
| $p=0.2$ | 88.10 | 78.86 | 3.28 | 12.70
| $p=0.5$ | 86.67 | 76.59 | 4.20 | 14.89
Pooling Size | $16\times 16$ | 87.04 | 77.22 | 3.69 | 14.28
$32\times 32$ | 87.53 | 77.98 | 3.13 | 11.56
$64\times 64$ | 88.37 | 79.27 | 2.75 | 8.84
$128\times 128$ | 89.03 | 80.34 | 2.59 | 8.34
$256\times 256$ | 87.99 | 78.71 | 2.82 | 9.97

Ablation on Loss Formulation In the loss formulation, our main idea is to pull
closer similar (positive) pairs upon the same threshold, while pushing apart
dissimilar (negative) pairs. Our learning objective is designed to jointly
exploit effective correlations in the prediction and feature space in an
informative way. To evaluate the effectiveness of each objective term, we
conduct ablation studies by removing each term separately. As shown in Table 3
(b), we observe that SimCVD outperforms the ablative baseline “SimCVD w/o
$\mathcal{L}_{\mathrm{contrast}}$” by a large margin and achieves a
$\\!+3.90\%\\!$ increase in Dice. It clearly demonstrates it can effectively
capture global context structure and local cues of 3D shapes. Next, we study
whether enforcing similarity constraints in the feature space is effective
enough to exploit structured knowledge in practice. We find that, under the
10% labeled setting, removing $\mathcal{L}_{\mathrm{pd}}$ leads to a
performance drop, and the accuracy measures decreases by $\\!-0.92\%\\!$ and
$\\!-1.45\%\\!$ in Dice and Jaccard, respectively. Our qualitative results
confirm that pair-wise distillation is effective for SimCVD in improving the
network performance. When we remove $\mathcal{L}_{\mathrm{sdm}}$, the network
performance does not drop significantly. We speculate that our boundary-aware
contrastive distillation framework is capable of eliciting “boundary-aware”
knowledge from the teacher model with high accuracy. This further highlights
the effectiveness of our proposed SimCVD. To further verify the robustness of
SimCVD, we perform pair-wise t-test between SimCVD and the other methods,
using the per-case test Dice score as samples. The null hypothesis states that
the test scores come from the same distribution, and thus the methods do not
differ; whereas the alternative hypothesis is that SimCVD yields higher Dice
score. For all the alternative methods in the Table below, the p-values are
close to or less than $0.05$. Small p-values indicates that we are confident
in rejecting the null hypothesis and conclude that SimCVD indeed outperforms
the baseline models.

###  6.2 Analysis on Projection Head

To further understand how different aspects of our projection head contribute
to the superior model performance, we conduct extensive experiments and
discuss our findings below.

How to Interpret Dropout? Our experimental results have shown that SimCVD is
an effective approach. In the following, we aim to answer two questions.
First, how can we interpret SimCVD’s dropout training strategy? Can we view
dropout as a form of data augmentation? Second, is it capable of exploiting
additional informative cues in practice?

First, we examine whether removing dropout during training can achieve
comparable performance. Table 4 shows the ablation result of our dropout on
LA. As shown in Table 4, we observe that using dropout achieves a much better
result on LA dataset. Compared to the setting $p\\!=\\!0.1$, we find that “no
dropout” ($p\\!=\\!0$) leads to a dramatic performance degradation by
$\\!-1.34\%$, $\\!-2.11\%$, $\\!-1.71$, $\\!-2.69$ absolute differences in
terms of Dice, Jaccard, ASD, and 95HD, respectively. While in the case of
$p\\!=\\!0.5$, it also significantly hurts the network performance. On the
other hand, we observe slight improvements on the other $p$ settings, compared
to “no dropout”, but eventually underperform SimCVD. This clearly demonstrates
the superiority of our dropout strategy to learn better representations with
respect to different pairs of augmented images. We speculate that adding
dropout can be interpreted as a minimal form of data augmentation, in which
the positive pair takes two views of the same images, and their
representations make a clear difference in dropout masks.

Effect of Augmentation Techniques To further examine our hypothesis, we
compare common data augmentation techniques (i.e., local shuffle pixel, non-
linear transformation, in-painting, out-painting) in Table 5. As is shown, the
quantitative results reveal interesting behavior of different data
augmentation: adding more data augmentation does not further contribute to the
good model performance. We note that, somewhat surprisingly, it hurts the
final prediction performance, and none of them outperforms the basic dropout
mask. This suggests that by including these data augmentation techniques, it
is possible to introduce additional noise during training, which leads to the
representation collapse.

Table 5:  Ablation on different data augmentations on the LA dataset. All of
them include the dropout masks ($p=0.1$).

Method | Metrics
---|---
Dice[%] | Jaccard[%] | ASD[voxel] | 95HD[voxel]
SimCVD | 89.03 | 80.34 | 2.59 | 8.34
\+ Local Shuffle Pixel | 88.15 | 78.97 | 1.96 | 8.66
\+ Non-linear Intensity Transformation | 88.02 | 78.80 | 2.68 | 10.29
\+ In-painting | 88.37 | 79.26 | 2.84 | 10.97
\+ Out-painting | 88.24 | 79.07 | 2.58 | 10.62

Effect of Pooling Size In Table 3, we demonstrate the network improvements
from using adaptive mean pooling instead of adaptive max pooling. We
investigate the effects of different pooling sizes in Table 4. Empirically, we
observe that using a larger pooling size clearly improves performance
consistently. However, we find that the results can not be improved further by
increasing the pooling size to $256$. In our implementation, we set the
pooling size as $128$.

##  7 Conclusion

In this work, we propose SimCVD, a simple contrastive distillation learning
framework, which largely advances state-of-the-art voxel-wise representation
learning on medical segmentation tasks. Specifically, we present an
unsupervised training strategy, which takes two views of an input volume and
predicts their signed distance maps of their object boundaries in a
contrastive objective, with only two different dropout masks. We further
conduct extensive analyses to understand the state-of-the-art performance of
our approach, and demonstrate the importance of learning distinct boundary-
aware representations and using dropout as the minimal data augmentation
technique. We also propose to perform structural distillation by distilling
pair-wise similarities, which achieves good performance improvements. Our
experimental results show that SimCVD obtained new state-of-the-art results on
two benchmarks in an extreme few-annotation setting.

We believe that our unsupervised training framework provides a new perspective
on data augmentation along with unlabeled 3D medical data. We also plan to
extend our method to solve multi-class medical image segmentation tasks.

## References

  * [1] L. H. Staib and J. S. Duncan, “Model-based deformable surface finding for medical images,” IEEE Transactions on Medical Imaging, vol. 15, no. 5, pp. 720–731, 1996.
  * [2] J. Yang, L. H. Staib, and J. S. Duncan, “Neighbor-constrained segmentation with level set based 3-d deformable models,” IEEE Transactions on Medical Imaging, vol. 23, no. 8, pp. 940–948, 2004.
  * [3] J. Yang and J. S. Duncan, “3D image segmentation of deformable objects with joint shape-intensity prior models using level sets,” Medical Image Analysis, vol. 8, no. 3, pp. 285–294, 2004.
  * [4] A. Chakraborty, L. H. Staib, and J. S. Duncan, “Deformable boundary finding in medical images by integrating gradient and region information,” IEEE Transactions on Medical Imaging, vol. 15, no. 6, pp. 859–870, 1996.
  * [5] L. H. Staib and J. S. Duncan, “Boundary finding with parametrically deformable models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 11, pp. 1061–1075, 1992.
  * [6] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in MICCAI, pp. 234–241, Springer, 2015\.
  * [7] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in 3DV, pp. 565–571, IEEE, 2016.
  * [8] W. Bai, O. Oktay, M. Sinclair, H. Suzuki, M. Rajchl, G. Tarroni, B. Glocker, A. King, P. M. Matthews, and D. Rueckert, “Semi-supervised learning for network-based cardiac MR image segmentation,” in MICCAI, pp. 253–260, Springer, 2017.
  * [9] P.-A. Ganaye, M. Sdika, and H. Benoit-Cattin, “Semi-supervised learning for segmentation under semantic constraint,” in MICCAI, pp. 595–602, Springer, 2018.
  * [10] C. You, J. Yang, J. Chapiro, and J. S. Duncan, “Unsupervised wasserstein distance guided domain adaptation for 3D multi-domain liver segmentation,” in Interpretable and Annotation-Efficient Learning for Medical Image Computing, pp. 155–163, Springer, 2020.
  * [11] Y. Wang, X. Wei, F. Liu, J. Chen, Y. Zhou, W. Shen, E. K. Fishman, and A. L. Yuille, “Deep distance transform for tubular structure segmentation in ct scans,” in CVPR, pp. 3833–3842, 2020.
  * [12] Y. Xue, H. Tang, Z. Qiao, G. Gong, Y. Yin, Z. Qian, C. Huang, W. Fan, and X. Huang, “Shape-aware organ segmentation by predicting signed distance maps,” in AAAI, 2020.
  * [13] X. Li, L. Yu, H. Chen, C.-W. Fu, L. Xing, and P.-A. Heng, “Transformation-consistent self-ensembling model for semisupervised medical image segmentation,” IEEE Transactions on Neural Networks and Learning Systems, 2020.
  * [14] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” arXiv preprint arXiv:1610.02242, 2016.
  * [15] Y. Zhang, L. Yang, J. Chen, M. Fredericksen, D. P. Hughes, and D. Z. Chen, “Deep adversarial networks for biomedical image segmentation utilizing unannotated images,” in MICCAI, pp. 408–416, Springer, 2017.
  * [16] X. Li, L. Yu, H. Chen, C.-W. Fu, and P.-A. Heng, “Semi-supervised skin lesion segmentation via transformation consistent self-ensembling model,” arXiv preprint arXiv:1808.03887, 2018.
  * [17] D. Nie, Y. Gao, L. Wang, and D. Shen, “Asdnet: Attention based semi-supervised deep networks for medical image segmentation,” in MICCAI, pp. 370–378, Springer, 2018.
  * [18] S. Qiao, W. Shen, Z. Zhang, B. Wang, and A. Yuille, “Deep co-training for semi-supervised image recognition,” in ECCV, pp. 135–152, 2018.
  * [19] G. Bortsova, F. Dubost, L. Hogeweg, I. Katramados, and M. de Bruijne, “Semi-supervised medical image segmentation via learning consistency under transformations,” in MICCAI, pp. 810–818, Springer, 2019.
  * [20] K. Li, W. Zhou, H. Li, and M. A. Anastasio, “Assessing the impact of deep neural network-based image denoising on binary signal detection tasks,” IEEE transactions on medical imaging, vol. 40, no. 9, pp. 2295–2305, 2021.
  * [21] L. Yu, S. Wang, X. Li, C.-W. Fu, and P.-A. Heng, “Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation,” in MICCAI, pp. 605–613, Springer, 2019.
  * [22] S. Li, C. Zhang, and X. He, “Shape-aware semi-supervised 3D semantic segmentation for medical images,” in MICCAI, pp. 552–561, Springer, 2020\.
  * [23] J. Zhu, Y. Li, Y. Hu, K. Ma, S. K. Zhou, and Y. Zheng, “Rubik’s cube+: A self-supervised feature learning framework for 3D medical image analysis,” Medical Image Analysis, p. 101746, 2020.
  * [24] X. Luo, J. Chen, T. Song, and G. Wang, “Semi-supervised medical image segmentation through dual-task consistency,” in AAAI, 2020.
  * [25] K. Chaitanya, E. Erdil, N. Karani, and E. Konukoglu, “Contrastive learning of global and local features for medical image segmentation with limited annotations,” in NeurIPS, 2020.
  * [26] C. You, R. Zhao, L. Staib, and J. S. Duncan, “Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation,” arXiv preprint arXiv:2105.07059, 2021.
  * [27] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in ICML, 2020.
  * [28] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” in ICLR, 2019.
  * [29] W. Bai, C. Chen, G. Tarroni, J. Duan, F. Guitton, S. E. Petersen, Y. Guo, P. M. Matthews, and D. Rueckert, “Self-supervised learning for cardiac MR image segmentation by anatomical position prediction,” in MICCAI, pp. 541–549, Springer, 2019.
  * [30] J. Peng, P. Wang, C. Desrosiers, and M. Pedersoli, “Self-paced contrastive learning for semi-supervised medical image segmentation with meta-labels,” in NeurIPS, 2021.
  * [31] C. Yang, L. Xie, C. Su, and A. L. Yuille, “Snapshot distillation: Teacher-student optimization in one generation,” in CVPR, pp. 2859–2868, 2019.
  * [32] J. Zhuang, J. Cai, R. Wang, J. Zhang, and W.-S. Zheng, “Deep knn for medical image classification,” in MICCAI, pp. 127–136, Springer, 2020.
  * [33] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in NeurIPS, pp. 1195–1204, 2017.
  * [34] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014\.
  * [35] S. Perera, N. Barnes, X. He, S. Izadi, P. Kohli, and B. Glocker, “Motion segmentation of truncated signed distance function based volumetric surfaces,” in WACV, pp. 1046–1053, IEEE, 2015.
  * [36] S. Dangi, C. A. Linte, and Z. Yaniv, “A distance map regularized cnn for cardiac cine MR image segmentation,” Medical Physics, vol. 46, no. 12, pp. 5637–5651, 2019.
  * [37] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in CVPR, pp. 165–174, 2019.
  * [38] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
  * [39] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,” arXiv preprint arXiv:1412.6550, 2014\.
  * [40] Y. Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, and J. Wang, “Structured knowledge distillation for semantic segmentation,” in CVPR, 2019.
  * [41] F. Yu, J. Zhao, Y. Gong, Z. Wang, Y. Li, F. Yang, B. Dong, Q. Li, and L. Zhang, “Annotation-free cardiac vessel segmentation via knowledge transfer from retinal images,” in MICCAI, 2019.
  * [42] K. Li, S. Wang, L. Yu, and P.-A. Heng, “Dual-teacher++: Exploiting intra-domain and inter-domain knowledge with reliable transfer for cardiac segmentation,” IEEE Transactions on Medical Imaging, 2020.
  * [43] Y. He, G. Yang, Y. Chen, Y. Kong, J. Wu, L. Tang, X. Zhu, J.-L. Dillenseger, P. Shao, S. Zhang, et al., “Dpa-densebiasnet: Semi-supervised 3D fine renal artery segmentation with dense biased network and deep priori anatomy,” in MICCAI, pp. 139–147, Springer, 2019.
  * [44] Z. Zhou, V. Sodha, M. M. R. Siddiquee, R. Feng, N. Tajbakhsh, M. B. Gotway, and J. Liang, “Models genesis: Generic autodidactic models for 3D medical image analysis,” in MICCAI, pp. 384–393, Springer, 2019.
  * [45] L. Yang, R. P. Ghosh, J. M. Franklin, S. Chen, C. You, R. R. Narayan, M. L. Melcher, and J. T. Liphardt, “Nuset: A deep learning tool for reliably separating and analyzing crowded cells,” PLoS computational biology, 2020\.
  * [46] X. Chen, S. Sun, N. Bai, K. Han, Q. Liu, S. Yao, H. Tang, C. Zhang, Z. Lu, Q. Huang, et al., “A deep learning-based auto-segmentation system for organs-at-risk on whole-body computed tomography images for radiation therapy,” Radiotherapy and Oncology, 2021.
  * [47] S. Shanlin, H. Kun, K. Deying, T. Hao, Y. Xiangyi, and X. Xiaohui, “Topology-preserving shape reconstruction and registration via neural diffeomorphic flow,” in CVPR, 2022.
  * [48] H. Tang, X. Chen, Y. Liu, Z. Lu, J. You, M. Yang, S. Yao, G. Zhao, Y. Xu, T. Chen, et al., “Clinically applicable deep learning framework for organs at risk delineation in ct images,” Nature Machine Intelligence, 2019\.
  * [49] H. Tang, X. Liu, K. Han, X. Xie, X. Chen, H. Qian, Y. Liu, S. Sun, and N. Bai, “Spatial context-aware self-attention model for multi-organ segmentation,” in WACV, 2021.
  * [50] X. Zhang, D. G. Martin, M. Noga, and K. Punithakumar, “Fully automated left atrial segmentation from mr image sequences using deep convolutional neural network and unscented kalman filter,” in International Conference on Bioinformatics and Biomedicine, IEEE, 2018.
  * [51] X. Zhang, M. Noga, D. G. Martin, and K. Punithakumar, “Fully automated left atrium segmentation from anatomical cine long-axis mri sequences using deep convolutional neural network with unscented kalman filter,” Medical Image Analysis, 2021.
  * [52] H. Tang, C. Zhang, and X. Xie, “Nodulenet: Decoupled false positive reduction for pulmonary nodule detection and segmentation,” in MICCAI, 2019.
  * [53] X. Zhang, M. Noga, and K. Punithakumar, “Fully automated deep learning based segmentation of normal, infarcted and edema regions from multiple cardiac mri sequences,” in Myocardial Pathology Segmentation Combining Multi-Sequence CMR Challenge, pp. 82–91, Springer, 2020.
  * [54] S. Sun, Y. Liu, N. Bai, H. Tang, X. Chen, Q. Huang, Y. Liu, and X. Xie, “Attentionanatomy: A unified framework for whole-body organs at risk segmentation using multiple partially annotated datasets,” in ISBI, IEEE, 2020.
  * [55] X. Yan, H. Tang, S. Sun, H. Ma, D. Kong, and X. Xie, “After-unet: Axial fusion transformer unet for medical image segmentation,” in WACV, 2022.
  * [56] H. Tang, X. Liu, S. Sun, X. Yan, and X. Xie, “Recurrent mask refinement for few-shot medical image segmentation,” in ICCV, 2021.
  * [57] J. Ma, Z. Wei, Y. Zhang, Y. Wang, R. Lv, C. Zhu, C. Gaoxiang, J. Liu, C. Peng, L. Wang, et al., “How distance transform maps boost segmentation cnns: an empirical study,” in Medical Imaging with Deep Learning, PMLR, 2020\.
  * [58] J. Castillo-Navarro, B. Le Saux, A. Boulch, and S. Lefèvre, “On auxiliary losses for semi-supervised semantic segmentation,” in ECML PKDD, 2020.
  * [59] B. Murugesan, K. Sarveswaran, S. M. Shankaranarayana, K. Ram, J. Joseph, and M. Sivaprakasam, “Psi-net: Shape and boundary aware joint multi-task deep network for medical image segmentation,” in EMBC, IEEE, 2019.
  * [60] Y. Xia, D. Yang, Z. Yu, F. Liu, J. Cai, L. Yu, Z. Zhu, D. Xu, A. Yuille, and H. Roth, “Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation,” Medical Image Analysis, 2020.
  * [61] H. Zheng, L. Lin, H. Hu, Q. Zhang, Q. Chen, Y. Iwamoto, X. Han, Y.-W. Chen, R. Tong, and J. Wu, “Semi-supervised segmentation of liver using adversarial learning with deep atlas prior,” in MICCAI, pp. 148–156, Springer, 2019\.
  * [62] X. Zhuang, Y. Li, Y. Hu, K. Ma, Y. Yang, and Y. Zheng, “Self-supervised feature learning for 3D medical images by playing a rubik’s cube,” in MICCAI, pp. 420–428, Springer, 2019.
  * [63] A. Taleb, W. Loetzsch, N. Danz, J. Severin, T. Gaertner, B. Bergner, and C. Lippert, “3D self-supervised methods for medical imaging,” in NeurIPS, pp. 18158–18172, 2020.
  * [64] C. You, R. Zhao, F. Liu, S. Chinchali, U. Topcu, L. Staib, and J. S. Duncan, “Class-aware generative adversarial transformers for medical image segmentation,” arXiv preprint arXiv:2201.10737, 2022.
  * [65] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in CVPR, 2006.
  * [66] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in ICCV, pp. 1422–1430, 2015.
  * [67] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in ECCV, pp. 69–84, Springer, 2016.
  * [68] Z. Wu, Y. Xiong, X. Y. Stella, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in CVPR, 2018.
  * [69] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” arXiv preprint arXiv:1906.05849, 2019.
  * [70] I. Misra and L. v. d. Maaten, “Self-supervised learning of pretext-invariant representations,” in CVPR, pp. 6707–6717, 2020.
  * [71] M. Federici, A. Dutta, P. Forré, N. Kushman, and Z. Akata, “Learning robust representations via multi-view information bottleneck,” in ICLR, 2020.
  * [72] N. Chen, C. You, and Y. Zou, “Self-supervised dialogue learning for spoken conversational question answering,” in INTERSPEECH, 2021.
  * [73] C. You, N. Chen, and Y. Zou, “Self-supervised contrastive cross-modality representation learning for spoken question answering,” arXiv preprint arXiv:2109.03381, 2021.
  * [74] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
  * [75] X. Wang and A. Gupta, “Unsupervised learning of visual representations using videos,” in ICCV, pp. 2794–2802, 2015.
  * [76] C. You, N. Chen, F. Liu, D. Yang, and Y. Zou, “Towards data distillation for end-to-end spoken conversational question answering,” arXiv preprint arXiv:2010.08923, 2020.
  * [77] C. You, N. Chen, and Y. Zou, “Contextualized attention-based knowledge transfer for spoken conversational question answering,” in INTERSPEECH, 2021.
  * [78] C. You, N. Chen, and Y. Zou, “MRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question Answering,” in IJCAI, 2021.
  * [79] C. You, N. Chen, and Y. Zou, “Knowledge distillation for improved accuracy in spoken question answering,” in ICASSP, 2021.
  * [80] H. Ma, T. Chen, T.-K. Hu, C. You, X. Xie, and Z. Wang, “Undistillable: Making a nasty teacher that cannot teach students,” in ICLR, 2020.
  * [81] H. Ma, T. Chen, T.-K. Hu, C. You, X. Xie, and Z. Wang, “Good students play big lottery better,” arXiv preprint arXiv:2101.03255, 2021.
  * [82] T. Furlanello, Z. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, “Born again neural networks,” in ICML, 2018.
  * [83] C. Yang, L. Xie, S. Qiao, and A. Yuille, “Knowledge distillation in generations: More tolerant teachers educate better students,” arXiv preprint arXiv:1805.05551, 2018.
  * [84] Y. Li, J. Yang, Y. Song, L. Cao, J. Luo, and L.-J. Li, “Learning from noisy labels with distillation,” in ICCV, pp. 1910–1918, 2017.
  * [85] J. Xie, B. Shuai, J.-F. Hu, J. Lin, and W.-S. Zheng, “Improving fast segmentation with teacher-student learning,” arXiv preprint arXiv:1810.08476, 2018.
  * [86] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu, “Deep mutual learning,” in CVPR, pp. 4320–4328, 2018.
  * [87] X. Chen, Y. Yuan, G. Zeng, and J. Wang, “Semi-supervised semantic segmentation with cross pseudo supervision,” in CVPR, 2021.
  * [88] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez, “Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation,” in CVPR, pp. 2517–2526, 2019.
  * [89] V. Verma, K. Kawaguchi, A. Lamb, J. Kannala, Y. Bengio, and D. Lopez-Paz, “Interpolation consistency training for semi-supervised learning,” in IJCAI, 2019.
  * [90] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self-normalizing neural networks,” in NeurIPS, pp. 972–981, 2017.
  * [91] H. R. Roth, A. Farag, E. Turkbey, L. Lu, J. Liu, and R. M. Summers, “Data from pancreas-ct. the cancer imaging archive,” 2016.
  * [92] Y. Zhou, Z. Li, S. Bai, C. Wang, X. Chen, M. Han, E. Fishman, and A. L. Yuille, “Prior-aware neural network for partially-supervised multi-organ segmentation,” in ICCV, pp. 10672–10681, 2019.
  * [93] X. Luo, “SSL4MIS.” https://github.com/HiLab-git/SSL4MIS, 2020.


# Discovery of a nova super-remnant cavity surrounding RS Ophiuchi

M. W. Healy-Kalesh,1 M. J. Darnley,1 É. J. Harvey,1,2 and A. M. Newsam1
1Astrophysics Research Institute, Liverpool John Moores University, Liverpool,
L3 5RF, UK
2UK Astronomy Technology Centre, STFC, Royal Observatory, Edinburgh, EH9 3HJ,
UK
E-mail<EMAIL_ADDRESS>
(Accepted 2024 February 07. Received 2024 February 06; in original form 2023
September 05)

###### Abstract

The prototypical nova super-remnant (NSR) was uncovered around the most
rapidly recurring nova (RN), M31N 2008-12a. Simulations of the growth of NSRs
revealed that these large structures should exist around all novae, whether
classical or recurrent. NSRs consist of large shell-like structures
surrounding excavated cavities. Predictions, informed by these simulations,
led to the discovery of an extended cavity coincident with the Galactic RN, RS
Ophiuchi, in far-infrared archival IRAS images. We propose that this cavity is
associated with RS Oph and is therefore evidence of another NSR to be
uncovered.

###### keywords:

stars: individual (RS Oph) – novae, cataclysmic variables – hydrodynamics –
ISM: general

††pubyear: 2024††pagerange: Discovery of a nova super-remnant cavity
surrounding RS Ophiuchi–References

##  1 Introduction

Recurrent novae (RNe) are a sub-type of cataclysmic variables that host an
accreting white dwarf (WD) within a close binary system (Walker, 1954). RNe
exhibit non-terminal thermonuclear driven eruptions (Starrfield et al., 1972,
2020) where the inter-eruption time-scale is less than a century. During a
nova eruption, much of the accreted material is ejected from the surface of
the WD (Starrfield et al., 2020) at velocities of hundreds to thousands of
kilometers per second (O’Brien et al., 2001). This ejected material can lead
to the formation of sub-parsec nova shells, (see, e.g., Bode et al., 2004;
Shara et al., 2007, 2012; Harvey et al., 2016). A proportion of the accreted
material may remain to grow the WD (Yaron et al., 2005; Hachisu et al., 2007;
Kato et al., 2015; Hillman et al., 2015, 2016; Starrfield et al., 2021) toward
the Chandrasekhar limit: potentially leading to a type Ia supernova (Whelan &
Iben, 1973; Hachisu et al., 1999a; Hachisu et al., 1999b; Hillebrandt &
Niemeyer, 2000).

A nova super-remnant (NSR) is a vast extended shell surrounding a RN formed by
the cumulative effect of eruptions sweeping up local interstellar medium (ISM)
over the lifetime of the system (Darnley et al., 2019). The first NSR found is
located around the annually erupting RN, M 31N 2008-12a (12a; see Darnley et
al., 2016; Darnley & Henze, 2020; Darnley, 2021); 12a is predicted to reach
the Chandrasekhar limit in less than 20 kyr (Darnley et al., 2017). Initially
observed in 1992 within a H$\alpha$ survey (Walterbos & Braun, 1992), this NSR
was ‘rediscovered’ and associated with 12a in 2014 (Darnley et al., 2015). The
12a NSR is substantially larger than any other nova shell, with semi-minor and
-major axes extending to 90 and 134 parsecs, respectively (Darnley et al.,
2019), rivalling the sizes of the largest known supernova remnants (Stil &
Irwin, 2001).

A preliminary study with one-dimensional hydrodynamical simulations (using
Morpheus; Vaytet et al., 2007) demonstrated the viability of NSR formation and
persistence over the life-time of a nova system. Healy-Kalesh et al. (2023a)
presented an extensive suite of 1D hydrodynamical simulations exploring the
impact of accretion rate, WD temperature and initial mass, and ISM density on
NSR evolution. The Healy-Kalesh et al. (2023a) study included the evolution of
WD mass and the subsequent impact on ejecta properties, and incorporated
radiative cooling. All Healy-Kalesh et al. (2023a) NSRs consisted of a very
low density excavated cavity surrounded by a hot ejecta pile-up region all
contained within a thin, high-density shell of swept up ISM.

Darnley et al. (2019) and Healy-Kalesh et al. (2023a) demonstrated that
dynamic NSRs should form around all nova systems, including classical novae
(Darnley & Henze, 2020; Darnley, 2021). Furthermore, NSRs can exist around
novae with growing or eroding WDs, indicating that NSRs should surround old
novae with low-mass WDs as well as RNe (Healy-Kalesh et al., 2023a). Recently,
a second NSR has been discovered around the Galactic RN, KT Eridani, through
detection of its H$\alpha$ shell (Shara et al., 2023; Healy-Kalesh et al.,
2023b). A survey of M 31 and LMC RNe did not find evidence for other NSRs
(Healy-Kalesh et al., 2024). Although such a dearth of NSRs may be due to the
lower intrinsic luminosity of NSRs around systems with longer recurrence
periods.

RS Ophiuchi is a Galactic symbiotic RN at a distance of $1.4^{+0.6}_{-0.2}$
kpc (Barry et al., 2008) or $2.4^{+0.3}_{-0.2}$ kpc (Bailer-Jones et al.,
2021), however this latter Gaia determination may be biased by the long-period
RS Oph binary (Schaefer, 2018). RS Oph hosts a WD with mass
$1.2-1.4\,\text{M}_{\odot}$ (Mikołajewska & Shara, 2017) and a giant donor
(M0/2 III; Dobrzycka & Kenyon, 1994; Anupama & Mikołajewska, 1999). RS Oph has
a substantial accretion rate to drive eruptions in 1898, 1933, 1958, 1967,
1985, 2006, and 2021 (Page et al., 2022); exhibiting a recurrence period of
${\sim}15$ years (Darnley, 2021). Eruptions in 1907 (Schaefer, 2004) and 1945
(Oppenheimer & Mattei, 1993) may have been missed due to Sun constraints
(Mikołajewska & Shara, 2017).

Very Long Baseline Interferometry (VLBI) radio observations of RS Oph during
its 1985 eruption revealed a bipolar structure at a position angle of
$84^{\circ}$ (Porcas et al., 1987; Taylor et al., 1989). This lack of
spherical symmetry was also seen in VLBI and Multi-Element Radio-Linked
Interferometer Network (MERLIN) radio images following the 2006 eruption with
collimated bipolar jet-like flows expanding toward the East-West directions
(O’Brien et al., 2006; Sokoloski et al., 2008; Rupen et al., 2008; Mohamed et
al., 2013). Expanding lobes with the same East-West orientation were observed
in late-time Hubble Space Telescope and Chandra X-ray imaging after the 2006
eruption (Bode et al., 2007; Ribeiro et al., 2009; Montez et al., 2022).
However, NIR and X-ray observations shortly after the 2006 eruption showed
ejecta expanding in a different direction to the bipolar radio structure (Lane
et al., 2007; Luna et al., 2009).

As RS Oph has a short recurrence period, a massive WD, and a high accretion
rate, it should be surrounded by a dynamic NSR grown over the life-time of the
system (Darnley et al., 2019; Darnley & Henze, 2020; Darnley, 2021; Healy-
Kalesh et al., 2023a). In this Letter we present evidence for an excavated NSR
cavity surrounding RS Oph. In Section 2 we predict the size of an RS Oph NSR;
in Section 3 we describe the discovery of the RS Oph NSR cavity and its likely
association with the nova system; the implications of uncovering another NSR
are discussed in Section 4, before we present our conclusions in Section 5.

##  2 RS Ophiuchi NSR modelling

###  2.1 RS Ophiuchi NSR dynamics

Healy-Kalesh et al. (2023a) presented a grid of NSRs from novae with a range
of accretion rates within a range of ISM densities. Here we utilise that grid
to determine the dynamics of the NSR around an RS Oph-like system with a
recurrence period of 15 years.

We select models where a WD is grown from $1\,\text{M}_{\odot}$ with an
accretion rate of $1\times 10^{-7}\,\text{M}_{\odot}\,\text{yr}^{-1}$ within
ISM ranging from $1.67\times 10^{-25}\,\text{g}\,\text{cm}^{-3}$ ($n=0.1$) to
$1.67\times 10^{-22}\,\text{g}\,\text{cm}^{-3}$ ($n=100$; Runs 1–7 from Healy-
Kalesh et al., 2023a). These models predict that a $1\,\text{M}_{\odot}$ WD
takes ${\sim}25$ Myr ($\sim$650,000 eruptions) to grow to
$1.27\,\text{M}_{\odot}$, i.e., a recurrence period of 15 years. We show the
density, pressure, velocity and temperature characteristics of the NSR at this
stage in Figure 1 for a range of ISM densities.

Figure 1: Simulated dynamics of the NSR surrounding RS Oph from Healy-Kalesh
et al. (2023a) for different ISM densities.

As shown in Figure 1, NSR radial size decreases with increasing ISM density.
Specifically, the outer edges of the NSR shells (and corresponding shell
thicknesses) extend to 102 (5%), 79 (3%), 62 (2%), 48 (2%), 37 (3%), 29 (4%)
and 22 pc (7%) for
$n=10^{-1},10^{-0.5},10^{0},10^{0.5},10^{1},10^{1.5},10^{2}$, respectively.
The velocity of the outer shell is $\lesssim 3\,\text{km}\,\text{s}^{-1}$ for
all NSRs and the temperature at the outer edges range from 130 K ($n=0.1$) to
2 K ($n=100$). The size of the NSR cavity, the inner-most region, also
decreases with increasing ISM density. The largest simulated cavity,
${\sim}$16 pc, occurs for $n=0.1$. The cavity density is predicted to be up to
four orders of magnitude lower than the surrounding ISM.

###  2.2 ISM density and NSR cavity prediction

To estimate the ISM density in the region around RS Oph we took two
approaches. The first utilises the Besançon model of the
Galaxy111http://cdsarc.u-strasbg.fr/viz-bin/qcat?J/A+A/453/635 (see, for e.g.,
Marshall et al., 2006), which provides the distribution of $K$-band extinction
along different lines of sight. The second method uses the three-dimensional
dusts maps222http://argonaut.skymaps.info from Green et al. (2019). In both
cases, we used the Galactic coordinates and distance to RS Oph to find the
relevant grid cell containing the system. Optical extinction333$K$-band
extinction from the Besançon model was converted to $V$-band extinction using
Cardelli et al. (1989). was converted to hydrogen column density using the
relation in Güver & Özel (2009), assuming Solar abundances ($X=0.739$; Basu &
Antia, 2004). The two methods yielded $n_{1}=0.032\pm 0.001$ and
$n_{2}=0.066\pm 0.003$, respectively. While statistically inconsistent,
neither method is favoured, therefore we assume an ISM density around RS Oph
of $n=0.05\pm 0.02$.

As the Healy-Kalesh et al. (2023a) grid doesn’t include ISM as sparse as
$n=0.05$, we select the $n=0.1$ model as the most suitable for predicting the
current size of the RS Oph NSR shell and cavity (see Figure 1). As such, we
predict that the radius of any NSR shell surrounding RS Oph should be $\gtrsim
100$ pc, with the radius of the inner evacuated cavity extending to $\gtrsim
15$ pc. With RS Oph at a distance of $1.4^{+0.6}_{-0.2}$ kpc, this equates to
an angular size of $\gtrsim 250^{\prime}$ and $\gtrsim 40^{\prime}$ for the
diameters of the NSR shell and cavity, respectively. Following the methodology
outlined in Healy-Kalesh et al. (2023a), we predict the total X-ray luminosity
and H$\alpha$ luminosity from the RS Oph NSR to be ${\lesssim}10^{25}$ erg s-1
and ${\lesssim}4\times 10^{31}$ erg s-1, respectively.

##  3 The search for a cavity around RS Ophiuchi

###  3.1 Structure in IRIS data

Given the approximate size of a potential RS Oph NSR, a search for a structure
of similar dimensions was conducted in the vicinity. We concluded that the
signature of a NSR cavity might be particular evident in the far-IR due to the
‘missing’ cool gas. The search therefore included the new generation Infrared
Astronomical Satellite (IRAS) catalogue, known as IRIS (Improved Reprocessing
of the IRAS Survey; Miville-Deschênes & Lagache, 2005).

A cavity-like structure was promptly found within a $3^{\circ}\times
3^{\circ}$ IRIS 100 $\mu$m image containing RS Oph (see Figure 2). The cavity
has semi-major and -minor axes of approximately ${\sim}40^{\prime}$ and
${\sim}12^{\prime}$, with a position angle of ${\sim}50^{\circ}$. At a
distance of ${\sim}1.4$ kpc (Barry et al., 2008), this equates to a physical
size of ${\sim}16\times 5$ pc. While only ${\sim}6\%$ of the size of the
predicted overall NSR, this is a close match to the predicted extent of NSR
cavities simulated by Healy-Kalesh et al. (2023a). The sheer scale of this
cavity might suggest a reason why this structure around RS Oph has not be
associated with the RN until now.

Figure 2: Cavity surrounding RS Oph. The white circles are the current
location of RS Oph. Left: A $3^{\circ}\times 3^{\circ}$ IRIS image of the
surroundings of RS Oph in the far infrared (100 $\mu$m). Right: A
$1.3^{\circ}\times 1.3^{\circ}$ region from the left image (see cyan box).

###  3.2 Determining association between cavity and RS Oph

To confidentially associate this far-IR cavity with RS Oph, we must exclude a
coincidental alignment between two unrelated phenomena. Here we define a
cavity to be any elliptical region where the flux per unit sky area is less
than the flux per unit sky area within a surrounding elliptical annulus. We
denote the ratio between the fluxes of the annulus and the cavity to be
$\mathscr{R}$; $\mathscr{R}>1$ defines a cavity.

####  3.2.1 Reference ellipse

For a reference ellipse, the ellipticity and position angle are free
parameters; the position is fixed to the location of RS Oph and the semi-major
axis is fixed at ${\sim}40^{\prime}$ (a projected size of ${\sim}16$ pc), to
match the prediction of the cavity size from simulations. The inner and outer
semi-major-axes of the annulus are defined as 1.0 and 1.5 times the semi-major
axis of the cavity.

With this reference ellipse cavity fixed at the position of RS Oph, we find
that an ellipticity of 0.4 and a position angle of 45∘ produces the largest
ratio $\mathscr{R}=9.74$. This best fitting cavity is illustrated in Figure 3
(red ellipse) and thus defines our border for the cavity around RS Oph. We
refer to this best-fitting ellipse as $\mathscr{E}_{\text{cavity}}$.

Figure 3: Left: An inverted $9^{\circ}\times 9^{\circ}$ IRIS 100 $\mu$m
centred on RS Oph (red cross). The best fitting elliptical cavity
$\mathscr{E}_{\text{cavity}}$ (see text) is illustrated by the red ellipse.
Right: Histogram illustrating the distribution of cavity ratios $\mathscr{R}$
computed at 361,201 locations within the IRIS data (left). The horizontal red
line denotes $\mathscr{E}_{\text{cavity}}$.

####  3.2.2 Monte Carlo analysis

A Monte Carlo analysis utilised a $9^{\circ}\times 9^{\circ}$ background
subtracted IRIS image centred on RS Oph. We sampled $\sim$360,000 ellipses,
each with a randomly selected semi-major axis, ellipticity and position angle
on a uniformly distributed spatial grid across the image and computed
$\mathscr{R}$ at each position. The semi-major axis is taken from a range of
radial sizes (${\sim}16-22$ pc) derived from linearly interpolating between
$n=0.05\pm 0.02$ to match estimates of ISM density around RS Oph (see Section
2.2); ellipticity is taken from a range derived from the system having an
inclination between $0-70^{\circ}$, and the position angle is taken from a
range between $0-180^{\circ}$. The resultant distribution of the computed
$\mathscr{R}$ for all the ellipses is shown in the right panel of Figure 3 and
as a heat map in Figure 4.

Figure 4: Heat map of ellipse ratio values $\mathscr{R}$ for the ellipse
$\mathscr{E}_{\text{cavity}}$ as a function of position within the IRIS data
(see Figure 3). The elliptical cavity $\mathscr{E}_{\text{cavity}}$ is shown
centred on the position of RS Oph.

We find that 97.64% of the sample of ellipses (without any fixed parameters)
have a ratio $\mathscr{R}<9.74$. As such, the probability of a chance
alignment of RS Oph with a cavity that has less far-IR emission than the
$\mathscr{E}_{\text{cavity}}$ cavity is $2.36\%$. We note that Figure 4
demonstrated that many positions with $\mathscr{R}>9.74$ are in a small region
near to the position of RS Oph; which further strengthens the association
between the cavity and RS Oph. As $\mathscr{R}$ is determined by both the
cavity and the surrounding emission, the heat map may not be a reliable source
of the position of the cavity centre.

##  4 Discussion

Having identified a large structure centred on the location of the RN RS Oph
where we expect to find a vast NSR, we have explored the possibility of
coincidental alignment. We found that the probability of finding a cavity-like
structure such as this in the same vicinity ($9^{\circ}\times 9^{\circ}$) of
the sky is $<2.36\%$ and so the location of this cavity is significant.

We would expect there to be various structures within the Galactic medium with
similar properties to the proposed NSR cavity surrounding RS Oph. For example,
superbubbles are cavities created within the ISM by the winds from OB
associations (Tomisaka & Ikeuchi, 1986; McCray, 1988; Wright, 2020). From
inspection of the most well-studied OB associations provided in Wright (2020),
we find that there is no superbubble spatially coincident with RS Oph (the
closest being Ser OB2 with an angular radius of 31 pc; Tetzlaff et al., 2010;
Mel’nik & Dambis, 2017) so we rule out this as a possibility.

Other structures that can span similar areas on the sky are supernova remnants
(SNRs). Cross-matching the large sample of 294 SNRs in Green (2019) with the
location of our discovered RS Oph cavity reveals zero overlapping remnants as
they are all confined to the Galactic plane, as do the 132 SNRs provided in
the Chandra Supernova Remnant Catalog444https://hea-
www.cfa.harvard.edu/ChandraSNR/snrcat_gal.html (a proportion of these SNRs are
the same as those in Green, 2019). A search for other sources within an
approximately 20 arcminute radius of RS Oph using the SIMBAD database (Wenger
et al., 2000) displays a number of X-ray sources, including a compact object
(Lin et al., 2012), and one radio source, however none of the identified
sources could be responsible for the found cavity.

We can also consider the orientation of RS Oph which has a large impact on the
shaping and geometry of the shells within its immediate vicinity. The position
angle of the found cavity (${\sim}45^{\circ}$) does not match the value
derived from the resolved shell around RS Oph created during the 2006 eruption
($85^{\circ}$; Ribeiro et al., 2009). Yet, the cavity does resemble the
general shape of a much more extended and evolved bipolar structure (O’Brien
et al., 2006; Ribeiro et al., 2009) that we would expect to grow over the
lifetime of RS Oph as a RN if it were consistently producing ‘dumbbell-shaped’
remnants (Ribeiro et al., 2009). Furthermore, the likely inhomogeneous
material surrounding RS Oph at much greater distances than the immediate
vicinity may contribute to the discrepancy between position angle of the
cavity and the shell grown from a single eruption.

Though we have outlined (and subsequently ruled out) a number of alternative
explanations for the cavity, the most compelling finding we present for its
association with RS Oph is the elliptical cavity we have identified is
situated with its major axis running through the location of RS Oph, with a
very low probability of a coincident alignment. In fact, the cavity structure
possibly associated with RS Oph can be seen in a $30^{\circ}\times 30^{\circ}$
far-infrared image as shown in Figure 5. We note that there also appears to be
a ring-like structure to the northwest of the elliptical cavity which may have
also been shaped by RS Oph. However, we can not speculate further on this.

Figure 5: Cavity surrounding RS Ophiuchi can be identified in a
$30^{\circ}\times 30^{\circ}$ IRAS image. The cyan cross indicates the
location of RS Oph. The white dashed-line box is a $3^{\circ}\times 3^{\circ}$
region which we show in the inset.

Uncovering a further example of a NSR through the detection of its cavity
component around another RN, RS Oph, reinforces the hypothesis that these
structures are created from many prior nova eruptions sweeping up the local
ISM (Darnley et al., 2019; Healy-Kalesh et al., 2023a). Additionally, finding
this likely NSR cavity within far-IR data encourages broadening our search for
more NSRs to infrared wavelengths alongside H$\alpha$ and X-ray emission.

##  5 Conclusions

The RN M31N 2008-12a hosts the prototypical nova super-remnant: an immense
dynamically-grown shell with a major axis greater than 100 parsecs formed from
the continual sweeping of local ISM. Similar NSRs are predicted to surround
other novae as it is the eruptions emanating from the central WD that drives
their growth.

In far-infrared IRAS imaging, we have uncovered the cavity of such a NSR
coincident with the Galactic RN RS Oph and we find that it is very likely that
they are associated. Narrow-band imaging of the surroundings of RS Oph akin to
those which were employed to detect the NSR shell associated with KT Eri
(Shara et al., 2023; Healy-Kalesh et al., 2023b), would cement this
connection. Ultimately, if this cavity is part of a more extended NSR, then
this finding strengthens the hypothesis that all RNe should be surrounded by a
NSR and further indicates that the 12a NSR is not unique.

## Acknowledgements

The authors would like to gratefully thank our anonymous referee for their
time spent reviewing our manuscript and for their constructive feedback that
improved our study. MWH-K, MJD, and ÉJH recieved funding from the UK Science
and Technology Facilities Council grant ST/S505559/1. The authors thank Prof.
Steven Longmore for assistance estimating local ISM density. This work made
use of the high performance computing facilities at Liverpool John Moores
University, partly funded by LJMU’s Faculty of Engineering and Technology and
by the Royal Society. This research has made use of the SIMBAD database,
operated at CDS, Strasbourg, France (see Wenger et al., 2000). This research
has made use of the VizieR catalogue access tool, CDS, Strasbourg, France. The
original description of the VizieR service was published in Ochsenbein et al.
(2000).

## Data Availability

Data in this study can be shared on reasonable request to the corresponding
author. The analysis in this work made use of the Python libraries: Numpy
(Harris et al., 2020) and Matplotlib (Hunter, 2007).

## References

  * Anupama & Mikołajewska (1999) Anupama G. C., Mikołajewska J., 1999, A&A, 344, 177
  * Bailer-Jones et al. (2021) Bailer-Jones C. A. L., Rybizki J., Fouesneau M., Demleitner M., Andrae R., 2021, AJ, 161, 147
  * Barry et al. (2008) Barry R. K., Mukai K., Sokoloski J. L., Danchi W. C., Hachisu I., Evans A., Gehrz R., Mikolajewska J., 2008, in Evans A., Bode M. F., O’Brien T. J., Darnley M. J., eds, Astronomical Society of the Pacific Conference Series Vol. 401, RS Ophiuchi (2006) and the Recurrent Nova Phenomenon. p. 52
  * Basu & Antia (2004) Basu S., Antia H. M., 2004, ApJ, 606, L85
  * Bode et al. (2004) Bode M. F., O’Brien T. J., Simpson M., 2004, ApJ, 600, L63
  * Bode et al. (2007) Bode M. F., Harman D. J., O’Brien T. J., Bond H. E., Starrfield S., Darnley M. J., Evans A., Eyres S. P. S., 2007, ApJ, 665, L63
  * Cardelli et al. (1989) Cardelli J. A., Clayton G. C., Mathis J. S., 1989, ApJ, 345, 245
  * Darnley (2021) Darnley M. J., 2021, in The Golden Age of Cataclysmic Variables and Related Objects V. p. 44 (arXiv:1912.13209)
  * Darnley & Henze (2020) Darnley M. J., Henze M., 2020, Advances in Space Research, 66, 1147
  * Darnley et al. (2015) Darnley M. J., et al., 2015, A&A, 580, A45
  * Darnley et al. (2016) Darnley M. J., et al., 2016, ApJ, 833, 149
  * Darnley et al. (2017) Darnley M. J., et al., 2017, ApJ, 849, 96
  * Darnley et al. (2019) Darnley M. J., et al., 2019, Nature, 565, 460
  * Dobrzycka & Kenyon (1994) Dobrzycka D., Kenyon S. J., 1994, AJ, 108, 2259
  * Green (2019) Green D. A., 2019, Journal of Astrophysics and Astronomy, 40, 36
  * Green et al. (2019) Green G. M., Schlafly E., Zucker C., Speagle J. S., Finkbeiner D., 2019, ApJ, 887, 93
  * Güver & Özel (2009) Güver T., Özel F., 2009, MNRAS, 400, 2050
  * Hachisu et al. (1999a) Hachisu I., Kato M., Nomoto K., Umeda H., 1999a, ApJ, 519, 314
  * Hachisu et al. (1999b) Hachisu I., Kato M., Nomoto K., 1999b, ApJ, 522, 487
  * Hachisu et al. (2007) Hachisu I., Kato M., Luna G. J. M., 2007, ApJ, 659, L153
  * Harris et al. (2020) Harris C. R., et al., 2020, Nature, 585, 357
  * Harvey et al. (2016) Harvey E., Redman M. P., Boumis P., Akras S., 2016, A&A, 595, A64
  * Healy-Kalesh et al. (2023a) Healy-Kalesh M. W., Darnley M. J., Harvey É. J., Copperwheat C. M., James P. A., Andersson T., Henze M., O’Brien T. J., 2023a, MNRAS, 521, 3004
  * Healy-Kalesh et al. (2023b) Healy-Kalesh M. W., Darnley M. J., Shara M. M., Lanzetta K. M., Garland J. T., Gromoll S., 2023b, arXiv e-prints, , arXiv:2310.17258
  * Healy-Kalesh et al. (2024) Healy-Kalesh M. W., Darnley M. J., Shara M. M., 2024, arXiv e-prints, p. arXiv:2401.04583
  * Hillebrandt & Niemeyer (2000) Hillebrandt W., Niemeyer J. C., 2000, ARA&A, 38, 191
  * Hillman et al. (2015) Hillman Y., Prialnik D., Kovetz A., Shara M. M., 2015, MNRAS, 446, 1924
  * Hillman et al. (2016) Hillman Y., Prialnik D., Kovetz A., Shara M. M., 2016, ApJ, 819, 168
  * Hunter (2007) Hunter J. D., 2007, Computing in Science & Engineering, 9, 90
  * Kato et al. (2015) Kato M., Saio H., Hachisu I., 2015, ApJ, 808, 52
  * Lane et al. (2007) Lane B. F., et al., 2007, ApJ, 658, 520
  * Lin et al. (2012) Lin D., Webb N. A., Barret D., 2012, ApJ, 756, 27
  * Luna et al. (2009) Luna G. J. M., Montez R., Sokoloski J. L., Mukai K., Kastner J. H., 2009, ApJ, 707, 1168
  * Marshall et al. (2006) Marshall D. J., Robin A. C., Reylé C., Schultheis M., Picaud S., 2006, A&A, 453, 635
  * McCray (1988) McCray R., 1988, in Roger R. S., Landecker T. L., eds, IAU Colloq. 101: Supernova Remnants and the Interstellar Medium. p. 447
  * Mel’nik & Dambis (2017) Mel’nik A. M., Dambis A. K., 2017, MNRAS, 472, 3887
  * Mikołajewska & Shara (2017) Mikołajewska J., Shara M. M., 2017, ApJ, 847, 99
  * Miville-Deschênes & Lagache (2005) Miville-Deschênes M.-A., Lagache G., 2005, ApJS, 157, 302
  * Mohamed et al. (2013) Mohamed S., Booth R., Podsiadlowski P., 2013, in Di Stefano R., Orio M., Moe M., eds, Proceedings IAU Symposium Vol. 281, Binary Paths to Type Ia Supernovae Explosions. pp 195–198, doi:10.1017/S1743921312014998
  * Montez et al. (2022) Montez R., Luna G. J. M., Mukai K., Sokoloski J. L., Kastner J. H., 2022, ApJ, 926, 100
  * O’Brien et al. (2001) O’Brien T. J., Davis R. J., Bode M. F., Eyres S. P. S., Porter J. M., 2001, in Schilizzi R. T., ed., International Astronomical Union Vol. 205, Galaxies and their Constituents at the Highest Angular Resolutions. p. 260
  * O’Brien et al. (2006) O’Brien T. J., et al., 2006, Nature, 442, 279
  * Ochsenbein et al. (2000) Ochsenbein F., Bauer P., Marcout J., 2000, A&AS, 143, 23
  * Oppenheimer & Mattei (1993) Oppenheimer B., Mattei J. A., 1993, in American Astronomical Society Meeting Abstracts. p. 55.03
  * Page et al. (2022) Page K. L., et al., 2022, MNRAS, 514, 1557
  * Porcas et al. (1987) Porcas R. W., Davis R. J., Graham D. A., 1987, in Bode M. F., ed., RS Ophiuchi (1985) and the Recurrent Nova Phenomenon. p. 203
  * Ribeiro et al. (2009) Ribeiro V. A. R. M., et al., 2009, ApJ, 703, 1955
  * Rupen et al. (2008) Rupen M. P., Mioduszewski A. J., Sokoloski J. L., 2008, ApJ, 688, 559
  * Schaefer (2004) Schaefer B. E., 2004, IAU Circ., 8396, 2
  * Schaefer (2018) Schaefer B. E., 2018, MNRAS, 481, 3033
  * Shara et al. (2007) Shara M. M., et al., 2007, Nature, 446, 159
  * Shara et al. (2012) Shara M. M., Mizusawa T., Wehinger P., Zurek D., Martin C. D., Neill J. D., Forster K., Seibert M., 2012, ApJ, 758, 121
  * Shara et al. (2023) Shara M. M., et al., 2023, arXiv e-prints, p. arXiv:2310.17055
  * Sokoloski et al. (2008) Sokoloski J. L., Rupen M. P., Mioduszewski A. J., 2008, ApJ, 685, L137
  * Starrfield et al. (1972) Starrfield S., Truran J. W., Sparks W. M., Kutter G. S., 1972, ApJ, 176, 169
  * Starrfield et al. (2020) Starrfield S., Bose M., Iliadis C., Hix W. R., Woodward C. E., Wagner R. M., 2020, ApJ, 895, 70
  * Starrfield et al. (2021) Starrfield S., Bose M., Iliadis C., Hix W. R., Woodward C. E., Wagner R. M., 2021, in The Golden Age of Cataclysmic Variables and Related Objects V. p. 30 (arXiv:2006.01827)
  * Stil & Irwin (2001) Stil J. M., Irwin J. A., 2001, ApJ, 563, 816
  * Taylor et al. (1989) Taylor A. R., Davis R. J., Porcas R. W., Bode M. F., 1989, MNRAS, 237, 81
  * Tetzlaff et al. (2010) Tetzlaff N., Neuhäuser R., Hohle M. M., Maciejewski G., 2010, MNRAS, 402, 2369
  * Tomisaka & Ikeuchi (1986) Tomisaka K., Ikeuchi S., 1986, PASJ, 38, 697
  * Vaytet et al. (2007) Vaytet N. M. H., O’Brien T. J., Bode M. F., 2007, ApJ, 665, 654
  * Walker (1954) Walker M. F., 1954, PASP, 66, 230
  * Walterbos & Braun (1992) Walterbos R. A. M., Braun R., 1992, A&AS, 92, 625
  * Wenger et al. (2000) Wenger M., et al., 2000, A&AS, 143, 9
  * Whelan & Iben (1973) Whelan J., Iben Icko J., 1973, ApJ, 186, 1007
  * Wright (2020) Wright N. J., 2020, New Astron. Rev., 90, 101549
  * Yaron et al. (2005) Yaron O., Prialnik D., Shara M. M., Kovetz A., 2005, ApJ, 623, 398