# Vulnerability Analysis of Time Synchronization in Automotive Ethernet

Rishikesh Kakade∗1, Joey Chou∗2, Shannon Torcato∗3 ∗Department of Electrical
and Computer Engineering, University of Waterloo
{1rkakade, 2t6chou<EMAIL_ADDRESS>

###### Abstract

The operation of many network communication protocols require accurate time
synchronization between nodes. In the automotive space, IEEE 802.3bw (commonly
referred to as automotive ethernet) is quickly becoming the most popular in-
vehicle communication protocol between electronic control units (ECUs). The
rapid advance of autonomous vehicles is predicated on a high throughput of
multiple HD video streams, LIDAR readings, ML inferences, and vehicular
control signals propagating on the same bus. To ensure reliability, security,
and safety, for millions of passengers and bystanders, it is essential that
the stack is robust at even the lowest level. The time synchronization
protocol in this standard is IEEE 802.1 AS which was initially developed for
virtual and virtual bridge local area networks (LANs). This paper presents an
analysis of failure modes affecting the ability of in-vehicle automotive
ethernet networks to synchronize clocks between ECUs. To demonstrate this, we
use the OMNet++ discrete event simulator to showcase normal operation,
failover to the hot-standby clock, and a black hole attack on one network
switch.

##  I Introduction

In the early 1980s, automotive manufacturers began including electronic
control units inside their cars. These tiny devices were the first step away
from manual, mechanical control of cars by users, toward increased automation.
in 1983, Bosch introduced the Controller Area Network protocol that arbitrated
communication between ECUs. The protocol synchronizes between nodes using a
synchronization byte that is transmitted at the start of each data packet
transmitted. In the present day, this progression has led to an average car
containing over 100 discrete ECUs, each responsible for controlling the
millions of tiny electromechanical adjustments that make driving as effortless
as it is today.

The next evolution of this rapid progress is the the autonomous vehicle space.
The amount of data a vehicle generates and processes has increased drastically
due to an increase in the number of sensors present in the vehicle, some of
which are LiDAR, cameras, and radar. These sensors are used to scan the
surroundings of the vehicle to detect other vehicles and road conditions in
real time. Due to this increase in data throughput required inside a vehicle,
car manufacturers have reached the limits of the measly 2-wire, 500kbps, CAN
protocol.

The leading candidate to succeed the CAN protocol is ethernet. Ethernet has
data throughput rates of 100 Mbps — representing a 200x improvement in
transmission rate. It is a natural choice because the concept has been well
tested and proven as the interface that connects the the internet. The large
body of knowledge already available enables the industry to quickly adopt it
and integrate into existing systems.

A foundational, yet often overlooked component of the ethernet standard in
this application is its method of synchronizing time across ECUs. It leverages
Generic Precision Time Protocol (gPTP). The protocol synchronizes by comparing
the difference in time between 2 master clocks. All other slave nodes
synchronize to the master nodes by listening periodically for synchronization
pulses transmitted from the master clock to many slaves clocks. As a
redundancy measure, the protocol allows slaves to synchronize to more than one
master clock. This acts as a fail safe in case one master clock loses its link
to the rest of the network.

This method of synchronization is robust enough for most internet
applications. However in the automotive space, every component in the car is
subject to an incredible amount of scrutiny because of the enormous potential
for loss of life. The time synchronization layer provides room for a number of
failure modes that could cause critical loss of function and possibly, of
life. Additional risk is introduced with the fact that modern vehicles are
exposed to the outside world over the internet. From any layer of the stack,
external adversaries could carry out an attack that exploits a vulnerability.

TSN which stands for Time-Sensitive Networking is a protocol defined by IEEE
802 standards that by default makes Ethernet deterministic. In particular, TSN
addresses the transmission which are bounded by low latency and which has high
availability. Some of the real-world applications are automotive vehicles or
industrial control facilities that use real-time control streams.

Attacks could include jamming, routing infrastructure attacks, and byzantine
attacks. With this large attack surface, it is especially important that
designers keep robustness in mind. By simulating this time synchronization
protocol we are able to perform experimentation at a lower cost. When it comes
to a global time clock, there is no one standard time that is followed. Each
node has its own internal clock that it uses to keep a track of events. Over
time the clocks in the network diverge due to finite clock accuracy.

This paper will investigate a number of failure modes Key metrics of interest
are global convergence time, clock offset, clock drift, and the maximum number
of adversarial faults that can be tolerated.

##  II Literature Review

In [9], the authors propose a new gateway framework based on the controller
area network (CAN) for in-vehicle networks (IVNs). The proposed system is easy
to reuse and verify this helps reduce the development costs and time. This
system introduces new functionalities such as parallel programming, diagnostic
routing, network management (NM), dynamic routing update, multiple routing
configuration, and security.

In [12], the authors describe why Ethernet is more efficient than other legacy
technologies such as CAN and MOST. Ethernet is preferred due to its increased
bandwidth. The sensors all around the vehicle help build new and exciting
features for safety such as collision avoidance, lane departure detection,
traffic sign classification, blind spot detection, driver intent detection,
pedestrian detection, automatic cruise control, and many others as described
in [7].

The authors of [14] compare different clock synchronization algorithms against
traffic types with individual timing requirements. The authors were able to
focus on strategies for scheduling flows and factors leading to delay. They
were able to find that the bandwidth of the flow is reserved based demand of
the actual bandwidth, this helps introduce a more stable transmission. When
adopted alone SPQ ruler can lead to high-priority data with large flow bursts
which would lead to low priority flow to have high delays.

In the paper [10], the authors present a novel approach in PTP systems to
reduce the clock time drifting significantly called timing fault recovery
(TFR). In this approach, the fault occurrence is detected by TFR and within a
short duration, it is recovered using a handshake mechanism. With increased
reliability and availability of PTP systems, the TFR approach also provides
constancy and clock stability.

The authors of [13] proposed and designed an IEEE 1588 protocol for a clock
synchronization system using FPGA. This system is very accurate and can detect
time precision at the microsecond level in the synchronization of the master
clock and slave clock. This design is an improvement to the current time
synchronization precision, it also helped reduce the complex hardware circuit
design, and helped reduce the development cost and difficulty, this provides a
benchmark value to research done in the future.

In [8], the authors discuss the performance of an autonomous vehicle. The data
measured from various sources are integrated and processed, depending on
whether time information is available from the sensor measurement the data
will vary. Temporal distortion of the controller in which missing patches of
information are patched up at different instants so the situation can be
perceived. On the other hand, time information-based sensor measurements are
aware of the time differences among the values this allows the autonomous
driving controller to have a better understanding of the situation by
compensating for the time differences. The authors conclude that to increase
autonomous vehicle performance, time synchronization-based research should be
looked into thoroughly sensors and controllers connected via communication
networks.

[1] presents a very similar analysis of the weaknesses of gPTP as is presented
in this paper. Man-in-the-middle (MIM) attacks operate in a number of ways
such as by intercepting and removing valid synchronization messages, by
manipulating messages, and by delaying legitimate messages. Denial of Service
(DoS) attacks often work by flooding the network with messages. Time source
attacks involve corrupting the source of truth the master clocks rely on (GPS
fraud) one of the master clocks being compromised. Master selection attacks
operate by having an attacking node send the same messages a master node
would, fooling slave nodes to synchronize to the wrong clock. The report also
suggests counter measures against these attacks.

In [11], the authors created a simulation model of the IEEE 802.11AS
generalized Precision Time Protocol(gPTP) standard on OMNET++ for TSN specific
synchronization. The authors utilize measurements of propagation delay and
time synchronization between systems that are time-aware. The model described
in this papers utilizes clocks with constant drift and end stations. The
results observed that every one second propagation delay is measured whereas
it was every 125 ms or 62.5 ms for synchronization message sent from GM. As a
result of this there is a difference in clock drifts. The time difference of
all the nodes after synchronization to GM is zero as expected from the gPTP
simulation model.

Black Hole Attack In this type of attack, the adversary stops propagating
forward the information or data but is still an active node in the routing
protocol. The adversarial node when part of a routing protocol path prevents
further communication to take place on that part. This type of attack affects
secure and insecure routing protocols due to their nature of rendering the
normal methods of route maintenance useless

Flood Rushing Attack In this attack, the flood duplicate suppression technique
of the node is exploited. The adversarial flood tries to be the first to
affect the nodes over the actual flood during propagation. When a node
receives a flood packet from an adversary then the nodes will ignore the
legitimate version and will forward propagate the adversarial version. Due to
this, it is almost impossible for the node network to be free from adversarial
propagation.

Byzantine Wormhole Attack In this kind of attack, the compromised nodes
communicate with each other in order to gain an advantage. By communicating
with each other they are able to create a shortcut in the network. By sending
out a route request an adversary can discover a route in the network, to
execute the attack, the tunnel packets through the non-adversarial nodes. In
order to increase the probability of being selected as the route path, the
adversaries use the low cost appearance of the wormhole links. They then drop
all of the data packets to disrupt the network. By having as minimum as two
compromised nodes this particular attack can be performed.

Byzantine Overlay Network Wormhole Attack This kind of attack is similar to
the wormhole attack but in this case, several nodes are compromised and they
form a network on top of the existing network. The adversaries tunnel packets
through the overlay network and make it seem like they are all neighbours to
the routing protocol. This helps increase the chances of being selected on
routes [1].

##  III Methodology

To build our model we use an OMNet++ simulation with the INET 4.4 library [3].

We can combine the gPTP model with the INET library along with the other
standard network protocols that are present in the library. For the simulation
model, we do not consider the best master clock algorithm of the gPTP
protocol. Therefore, we predefine the state of the node as master or slave. We
are more interested in clock synchronization and measurement of propagation
delay. We introduce a clock drift so that we can implement a time
synchronization protocol.

In order to meet gPTP requirements, we implement time synchronization and
propagation delay measurements using a class from the OMNeT library [4]. We
use the in-built EthernetIIFrame available in the INET library which has
encapsulated gPTP messages.

In INET, network nodes and interfaces do not have local time. Instead, the
global time is defined as the simulation time. Network clock submodules are
required by network nodes to simulate local time-based network nodes and to
demonstrate effects such as clock drift. The oscillator submodules are used to
keep a track of time. The basic working is that the oscillator periodically
produces ticks which are counted by the clock modules. Inaccuracies in
oscillators can lead to clock drift.

There can be drift in clocks of the time kept between different network
devices. When two clocks are synchronized, the time difference between them
has an upper bound. We can achieve this between clocks by periodically
synchronizing them. In gPTP, there is a time synchronization of the slave
clocks to the time of the master clocks within a time domain of the gPTP.
Multiple time domains of gPTP can be obtained by a network. Multiple clocks
can be synchronized to multiple master clocks in case there is a failure in
the master clock or a link break causes the system to go offline. The general
structure is that there exists a single master clock followed by a number of
slave clocks. Synchronization by the protocol is achieved by sending sync
messages to the slave clocks from the master clock. By having multiple gPTP
submodules, a node can have multiple gPTP time domains. Depending on their
location, a gPTP can follow one out of three possible roles: master, bridge,
or slave. By having multiple gPTP submodules, a node can have multiple gPTP
time domains. Depending on their location, a gPTP can follow one out of three
possible roles: master, bridge, or slave. Master gPTP modules of nodes contain
the time domain master clock.

Figure 1: Arrival of messages due to impact of clock drift on gPTP network.

The fundamental device underlying the simulation the clock seen in Figure 2.
It is modelled using basic clock classes from the INET library [3]. The basic
models include the ability to represent clock drift. Clock drift arises due to
inaccuracies in the low-cost crystal oscillators often used in automotive
ECUs. In the simulation, we have modelled the drift as a random walk, with
steps varying by $\pm 100\mu$s every second. In the real world, this variance
is generally caused by process variations, thermal expansion, and device
aging. The change in clock drift over time is ignored because it changes very
slowly relative to the time frames on which automotive ethernet networks
operate. For this short simulation, clock frequency is assumed to be constant.

Figure 2: TSN Clock model taken from iNet class library.  Figure 3: TSN Switch
model taken from iNet class library. Figure 4: Wheel motor controller network
whose switches are in a ring topology. Contains double redundant clocks
representing central car computer and central motor body controller.

We emulate 4 ECUs controlling the 4 DC motors of an electric car. The ECUs are
synchronized to a central body controller and the main computer as a
redundancy measure. In the simulation the network is set up with two master
clocks; each disseminating two time domains. Each node in the network keeps
track of 4 total times. The switching network is shaped as a ring to further
emphasize the benefit of having multiple time domains propagating in different
directions as a redundancy measure.

To summarize the way the network in Figure 4 works [3];

  1. 1.

Primary clock has one clock, two master time domains. The domains are
propagated in both clockwise and counterclockwise directions along the ring

  2. 2.

Secondary clock has 4 time domains. Two are slaves to the primary clock’s
domains while the others are redundant domains.

  3. 3.

all other nodes in the network are slaves to the four total time domains
propagating in the network

This paper presents three scenarios. The first is operation of the network
under normal conditions. Under normal circumstances, the only disturbance that
needs to be corrected for is clock drift as described in Figure 1. It shows
that between successive synchronization pulses from the master nodes to each
slave, the relative clock drift causes the delta between each local time to be
non-constant. In the second scenario, we simulate a failure of the central
body controller. This could be caused by a number of failure modes, such as
power loss, bus message overloads, and firmware failures. We expect that the
slaves in the network should be able to switch over to the redundant secondary
master in the car’s central computer. In the third, we simulate a black hole
attack on one of the network switches. Here, we make the switch unable to
forward synchronization messages to the slave ECU it is connected to. It is
expected that the device should be unable to synchronize with the rest of the
network.

The relevant simulation files are made open source on Github [5].

##  IV Results

###  IV-A Normal operation

Figure 5 shows the difference between the clocks from ’true’ simulation time
of the network under normal conditions. The clock in the front right wheel has
the most dramatic clock drift which explains why peak to peak difference is
higher than in others. Despite this, the network is still able to synchronize
all components to within about $\pm 100$ns of one another. We include
simulation of the other, redundant time domain disseminated by the main
computer at this time. We find that all nodes in the network are able to
synchronize to this as well, even though it has a different clock drift. When
the end-to-end delay is constant there is no clock drift as the generated
packets are in sync with the source.

Figure 5: Difference between device time and true simulation time under normal
conditions of the domain disseminated by the primary clock in the body
controller. Figure 6: Difference between device time and true simulation time
under normal conditions disseminated by the secondary clock in the main
computer. Observe that the drift rate is different between this and Figure 5
because it originates from a different clock.

###  IV-B Failure of primary clock

Figure 7 shows the difference between the clocks from ’true’ simulation time
of the network when the primary clock in the body controller fails. Unlike in
the normal mode simulation, we find that as soon as the primary clock fails,
all time domains disseminated by it cease to exist. We find that the simulator
does not simulate past this event where a domain stops existing. However, the
overall system is able to switch over to time domain 2 which is disseminated
by the secondary clock in the main computer. The primary master clock in this
configuration has a link failure as a result there is a stoppage in the two
time domains’ time synchronization of the primary master clock. After our
master clock fails, the active clocks differ in their time stamp.

Figure 7: Difference between device time and true simulation time when the
central body controller (primary clock) fails for domain 2 originating in the
central computer (secondary). Observe that the system is able to switch over
to the main computer clock after the body controller fails. Figure 8:
Difference between device time and true simulation time when the central body
controller (primary clock) fails for domain 0 originating in primary clock.
Observe that the simulation stops at the moment the primary node fails.

###  IV-C Black hole network switch

Figure 9 shows the difference between the clocks from ’true’ simulation time
of the network when a black hole attack is performed on the front-left network
switch which forces the front-left motor ECU to be left nonoperational. We see
that this is the case by the thin blue line in Figure 9. The clock drift of
this ECU was preset to be the same as the master clock to create a simpler
graph.

Figure 9: Difference between device time and true simulation time when the
front-left switch refuses to forward synchronization messages to the front-
left motor ECU. Observe that the ECU is unable to synchronize with the rest of
the network and just ticks at its own clock drift.

##  V Conclusion

In this paper, we have proposed a gPTP simulation model containing the main
operations: time synchronization and propagation delay measurements. We
demonstrated the failure of different modes affecting the ability of
synchronization of ethernet networks present in automotive vehicles between
ECUs. We used OMNet++ as the simulation tool to showcase different scenarios
such as normal operation, fail-over to the hot-standby clock, and a black hole
attack on one network switch. We also showed how a black hole attack can
effect the time synchronization between master clock and slave clocks. Through
our experimentation, we demonstrated the vulnerabilities that exist in the
present ethernet networks present in automotive vehicles. Further research can
be done to overcome this vulnerability and create a more secure ethernet
network system by implementing additional redundancy checks that each ECU node
is synchronized with the rest of the network.

## Acknowledgment

The authors would like to thank Prof. Otman Basir for giving the time and
resources to choose and develop our ideas into this report.

## References

  * [1] [Online]. Available: https://standards.ieee.org/wp-content/uploads/import/documents/other/d1-10_jesse_making_gptp_capable_for_secure_time_synchronization.pdf
  * [2] “802.1as - timing and synchronization.” [Online]. Available: https://www.ieee802.org/1/pages/802.1as.html
  * [3] “Effects of time synchronization on time-aware shaping.” [Online]. Available: https://inet.omnetpp.org/docs/showcases/tsn/combiningfeatures/ gptpandtas/doc/index.html
  * [4] “Using gptp¶.” [Online]. Available: https://inet.omnetpp.org/docs/showcases/tsn/timesynchronization/gptp/ doc/index.html
  * [5] “Vulnerability-analysis-of-time-synchronization-in-automotive-ethernet.” [Online]. Available: https://github.com/RishiKakade/Vulnerability-Analysis-of-Time-Synchronization-in-Automotive-Ethernet
  * [6] B. Bob and L. Lo, “The case for ethernet in automotive communications,” _SIGBED Rev._ , vol. 8, no. 4, p. 7–15, dec 2011. [Online]. Available: https://doi.org/10.1145/2095256.2095257
  * [7] A. Doshi, B. T. Morris, and M. M. Trivedi, “On-road prediction of driver’s intent with multimodal sensory cues,” _IEEE Pervasive Computer_ , vol. 10, p. 22–34, 2011.
  * [8] H. J. Kim, U. Lee, M. Kim, and S. Lee, “Time-synchronization method for can–ethernet networks with gateways,” _Applied Sciences_ , vol. 10, no. 24, p. 8873, Dec 2020. [Online]. Available: http://dx.doi.org/10.3390/app10248873
  * [9] J. Kim, S.-H. Seo, N. T. Hai, B. M. Cheon, Y. S. Lee, and J. W. Jeon, “Gateway framework for in-vehicle networks based on can, flexray, and ethernet,” _IEEE Transactions on Vehicular Technology_ , vol. 64, pp. 4472–4486, 2015\.
  * [10] A. A. Omar and J. M. Rhee, “Tfr: A novel approach for clock synchronization fault recovery in precision time protocol (ptp) networks,” _Applied Sciences_ , vol. 8, no. 1, 2018. [Online]. Available: https://www.mdpi.com/2076-3417/8/1/21
  * [11] H. Puttnies, P. Danielis, E. Janchivnyambuu, and D. Timmermann, “A simulation model of ieee 802.1as gptp for clock synchronization in omnet++,” 2018.
  * [12] S. Tuohy, M. Glavin, C. Hughes, E. Jones, M. Trivedi, and L. Kilmartin, “Intra-vehicle networks: A review,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 2, pp. 534–554, 2015.
  * [13] B. Zhao and N. Wang, “The implementation of ieee 1588 clock synchronization system based on fpga,” pp. 216–220, 2014.
  * [14] L. Zhao, F. He, E. Li, and J. Lu, “Comparison of time sensitive networking (tsn) and ttethernet,” _2018 IEEE/AIAA 37th Digital Avionics Systems Conference (DASC)_ , pp. 1–7, 2018.


11institutetext:  Computer Vision Laboratory, ETH Zurich, Switzerland 11email:
<EMAIL_ADDRESS>22institutetext: School of Computer Science and Technology, Beijing Institute
of Technology, China
22email<EMAIL_ADDRESS>
https://github.com/0liliulei/Mem3D

# Quality-Aware Memory Network for
Interactive Volumetric Image Segmentation

Tianfei Zhou The first two authors contribute equally to this work.11 Liulei
Li* 22 Gustav Bredell 11 Jianwu Li 22 Ender Konukoglu 11

###### Abstract

Despite recent progress of automatic medical image segmentation techniques,
fully automatic results usually fail to meet the clinical use and typically
require further refinement. In this work, we propose a quality-aware memory
network for interactive segmentation of 3D medical images. Provided by user
guidance on an arbitrary slice, an interaction network is firstly employed to
obtain an initial 2D segmentation. The quality-aware memory network
subsequently propagates the initial segmentation estimation bidirectionally
over the entire volume. Subsequent refinement based on additional user
guidance on other slices can be incorporated in the same manner. To further
facilitate interactive segmentation, a quality assessment module is introduced
to suggest the next slice to segment based on the current segmentation quality
of each slice. The proposed network has two appealing characteristics: 1) The
memory-augmented network offers the ability to quickly encode past
segmentation information, which will be retrieved for the segmentation of
other slices; 2) The quality assessment module enables the model to directly
estimate the qualities of segmentation predictions, which allows an active
learning paradigm where users preferentially label the lowest-quality slice
for multi-round refinement. The proposed network leads to a robust interactive
segmentation engine, which can generalize well to various types of user
annotations (e.g., scribbles, boxes). Experimental results on various medical
datasets demonstrate the superiority of our approach in comparison with
existing techniques.

###### Keywords:

Interactive Segmentation Memory-Augmented Network

##  1 Introduction

Accurate segmentation of organs/lesions from medical imaging data holds the
promise of significant improvement of clinical treatment, by allowing the
extraction of accurate models for visualization, quantification or simulation.
Although recent deep learning based automatic segmentation engines [21, 34,
29, 16] have achieved impressive performance, they still struggle to achieve
sufficiently accurate and robust results for clinical practice, especially in
the presence of poor image quality (e.g., noise, low contrast) or highly
variable shapes (e.g., anatomical structures). Consequently, interactive
segmentation [18, 32, 33, 2, 28, 27] garners research interests of the
medical image analysis community, and recently became the choice in many real-
life medical applications.

In interactive segmentation, the user is factored in to play a crucial role in
guiding the segmentation process and in correcting errors as they occur (often
in an iteratively-refined manner). Classical approaches employ Graph Cuts
[1], GeoS [4] or Random Walker [6, 5] to incorporate scribbles for
segmentation. Yet, these methods require a large amount of input from users to
segment targets with low contrast and ambiguous boundaries. With the advent of
deep learning, there has been a dramatically increasing interest in learning
from user interactions. Recent methods demonstrate higher segmentation
accuracy with fewer user interactions than classical approaches. Despite this,
many approaches [11, 26, 22] only focus on 2D medical images, which are
infeasible to process ubiquitous 3D data. Moreover, 2D segmentation does not
allow the integration of prior knowledge regarding the 3D structure, and
slice-by-slice interactive segmentation will impose extremely high annotation
cost to users. To address this, many works [3, 20, 13, 28, 27] carefully
design 3D networks to segment voxels at a time. While these methods enjoy
superior ability of learning high-order, volumetric features, they require
significantly more parameters and computations in comparison with the 2D
counterparts. This necessitates compromises in the 3D network design to fit
into a given memory or computation budget.

To address these issues, we take a novel perspective to explore memory-
augmented neural networks [25, 12, 23, 14] for 3D medical image segmentation.
Memory networks augment neural networks with an external memory component,
which allows the network to explicitly access the past experiences. They have
been shown effective in few-shot learning [23], contrastive learning [7,
29], and also been explored to solve reasoning problems in visual dialog [25,
12]. The basic idea is to retrieve the relevant information from the external
memory to answer a question at hand by using trainable memory modules. We take
inspiration from these efforts to cast volumetric segmentation as a memory-
based reasoning problem. Fundamental to our model is an external memory, which
enables the model to online store segmented slices in the memory and later
mine useful representations from the memory for segmenting other slices. In
this way, our model makes full use of context within 3D data, and at the same
time, avoids computationally expensive 3D operations. During segmentation, we
dynamically update the memory to maintain shape or appearance variations of
the target. This facilitates easy model updating without extensive parameter
optimization. Based on the memory network, we propose a novel interactive
segmentation engine with three basic processes: 1) Initialization: an
interaction network is employed to respond to user guidance on an arbitrary
slice to obtain an initial 2D segmentation of a target. 2) Propagation: the
memory network propagates the initial mask to the entire volume. 3)
Refinement: the physician could provide extra guidance on low-quality slices
for iterative refinement if the segmentation results are unsatisfactory.

Our contributions are three-fold: First, we propose a memory-augmented network
for volumetric interactive segmentation. It is able to incorporate rich 3D
contextual information, while avoiding expensive 3D operations. Second, we
equip the memory network with a quality assessment module to assess the
quality of each segmentation. This facilitates automatic selection of
appropriate slices for iterative correction via human-in-the-loop. Third, our
approach outperforms previous methods by a significant margin on two public
datasets, while being able to handle various forms of interactions (e.g.,
scribbles, bounding boxes).

$f_{\text{In}}$$f_{\text{Mem}}$$f_{\text{Mem}}$$f_{\text{Enc}}^{M}$$f_{\text{Enc}}^{M}$$f_{\text{Enc}}^{M}$$f_{\text{Enc}}^{Q}$$f_{\text{Dec}}$$\mathbf{S}_{k}$$I_{k}$$0.87$$h_{k}\\!=$(a)(b)$\mathbf{F}_{k}$

Figure 1: Illustration of the proposed 3D interactive segmentation engine. (a)
Simplified schematization of our engine that solves the task with an
interaction network ($f_{\text{In}}$) and a quality-aware memory network
($f_{\text{Mem}}$). (b) Detailed network architecture of $f_{\text{Mem}}$. ©
denotes the concatenation operation. Zoom in for details.

##  2 Methodology

Let $V\in\mathbb{R}^{h\times w\times c}$ be a volumetric image to be
segmented, which has a spatial size of $h\times w$ and $c$ slices. Our
approach aims to obtain a 3D binary mask $\mathbf{S}\in\\{0,1\\}^{h\times
w\times c}$ for a specified target by utilizing user guidance. As shown in
Fig. 1 (a), the physician is asked to provide an initial input on an
arbitrary slice ${I}_{i}\in\mathbb{R}^{h\times w}$, where ${I}_{i}$ denotes
the $i$-th slice of $V$. Then, an interaction network ($f_{\text{In}}$, §2.1)
is employed to obtain a coarse 2D segmentation
$\mathbf{S}_{i}\in[0,1]^{h\times w}$ for ${I}_{i}$. Subsequently,
$\mathbf{S}_{i}$ is propagated to all other slices with a quality-aware memory
network ($f_{\text{Mem}}$, §2.2) to obtain $\mathbf{S}$. Our approach also
takes into account iterative refinement so that segmentation performance can
be progressively improved with multi-round inference. To aid the refinement,
the memory network has a module that estimates the segmentation performance on
each slice and suggests the user to place guidance on the slice with the worst
segmentation quality.

###  2.1 Interaction Network

The interaction network takes the user annotation at an interactive slice
${I}_{i}$ to segment the specified target (or refine the previous result). At
the $t^{th}$ round, its input consists of three images: the original gray-
scale image ${I}_{i}$, the segmentation mask from the previous round
$\mathbf{S}_{i}^{t-1}$, and a binary image
$\mathbf{M}_{i}\\!\in\\!\\{0,1\\}^{h\times w}$ that encodes user guidance.
Note that in the first round (i.e., $t\\!=\\!0$), the segmentation mask
$\mathbf{S}_{i}^{-1}$ is initialized as a neutral mask with $0.5$ for all
pixels. These inputs are concatenated along the channel dimension to form an
input tensor $\mathbf{X}_{i}^{t}\\!\in\\!\mathbb{R}^{h\times w\times 3}$. The
interaction network $f_{\text{IN}}$ conducts the segmentation for $I_{i}$ as
follows:

$\displaystyle\mathbf{S}_{i}^{t}=f_{\text{In}}(\mathbf{X}_{i}^{t})\in\mathbb{R}^{h\times
w}.$ (1)

####  2.1.1 Region-of-interest (ROI).

To further enhance performance and avoid mistakes in case of small targets or
low-contrast tissues, we propose to crop the image according to the rough
bounding-box estimation of user input, and apply $f_{\text{IN}}$ only to the
ROI. We extend the bounding box by 10% along sides to preserve more context.
Each ROI region is resized into a fixed size for network input. After
segmentation, the mask made within the ROI is inversely warped and pasted back
to the original location.

###  2.2 Quality-Aware Memory Network

Given the initial segmentation $\mathbf{S}_{i}^{t}$, our memory network learns
from the interactive slice ${I}_{i}$ and segments the specified target in
other slices. It stores previously segmented slices in an external memory, and
takes advantage of the stored 3D image and corresponding segmentation to
improve the segmentation of each 2D query image. The network architecture is
shown in Fig. 1(b). In the following paragraphs, the superscript ‘$t$’ is
omitted for conciseness unless necessary.

####  2.2.1 Key-Value Embedding.

Given a query slice ${I}_{k}$, the network mines useful information from
memory $\mathcal{M}$ for segmentation. Here, each memory cell
$\mathcal{M}_{j}\in\mathcal{M}$ consists of a slice ${I}_{m_{j}}$ and its
segmentation mask $\mathbf{S}_{m_{j}}$, where $m_{j}$ indicates the index of
the slice in the original volume. As shown in Fig. 1(b), we first encode the
query ${I}_{k}$ as well as each memory cell
$\mathcal{M}_{j}\\!=\\!\\{{I}_{m_{j}},\\!~{}\mathbf{S}_{m_{j}}\\}$ into pairs
of key and value using dedicated encoders (i.e., query $f_{\text{Enc}}^{Q}$
and memory encoder $f_{\text{Enc}}^{M}$):

$\displaystyle\mathbf{K}_{k}^{Q},\mathbf{V}_{k}^{Q}$
$\displaystyle=f_{\text{Enc}}^{Q}({I}_{k}),$ (2)
$\displaystyle\mathbf{K}_{m_{j}}^{M},\mathbf{V}_{m_{j}}^{M}$
$\displaystyle=f_{\text{Enc}}^{M}({I}_{m_{j}},\mathbf{S}_{m_{j}}).$ (3)

Here, $\mathbf{K}_{k}^{Q}\\!\in\\!\mathbb{R}^{H\times W\times C/8}$ and
$\mathbf{V}_{k}^{Q}\\!\in\\!\mathbb{R}^{H\times W\times C/2}$ indicate key and
value embedding of the query $I_{k}$, respectively, whereas
$\mathbf{K}_{v_{j}}^{M}$ and $\mathbf{V}_{v_{j}}^{M}$ correspond to the key
and value of the memory cell $\mathcal{M}_{j}$. $H$, $W$ and $C$ denote the
height, width and channel dimension of the feature map from the backbone
network, respectively. Note that for each memory cell, we apply Eq. (3) to
obtain pairs of key and value embedding. Subsequently, all memory embedding
are stacked together to build a pair of 4D key and value features
(i.e.,$\mathbf{K}^{M}\\!\in\\!\mathbb{R}^{N\times H\times W\times{C}/{8}}$ and
$\mathbf{V}^{M}\\!\in\\!\mathbb{R}^{N\times H\times W\times C/2}$), where
$N=|\mathcal{M}|$ denotes memory size.

####  2.2.2 Memory Reading.

The memory read controller retrieves relevant information from the memory
based on the current query. Following the key-value retrieval mechanism in
[12, 25], we first compute the similarity between every 3D location
$p\in\mathbb{R}^{3}$ in $\mathbf{K}^{M}$ with each spatial location
$q\in\mathbb{R}^{2}$ in $\mathbf{K}_{k}^{Q}$ with dot product:

$s_{k}(p,q)=\frac{\mathbf{K}^{M}(p)\cdot\mathbf{K}_{k}^{Q}(q)}{\|\mathbf{K}^{M}(p)\|\|\mathbf{K}_{k}^{Q}(q)\|}\in[-1,1],$
(4)

where $\mathbf{K}^{M}(p)\\!\in\\!\mathbb{R}^{C/8}$ and
$\mathbf{K}_{k}^{Q}(q)\\!\in\\!\mathbb{R}^{C/8}$ denote the features at the
$p^{th}$ and $q^{th}$ position of $\mathbf{K}^{M}$ and $\mathbf{K}_{k}^{Q}$,
respectively. Next, we compute the read weight $w_{k}$ by softmax
normalization:

$w_{k}(p,q)=\exp(s_{k}(p,q))/\sum\nolimits_{o}\exp(s_{k}(o,q))\in[0,1].$ (5)

Here, $w_{k}(p,q)$ measures the matching probability between $p$ and $q$. The
memory summarization is then obtained using the weight to combine the memory
value:

$\mathbf{H}_{k}(q)=\sum\nolimits_{p}w_{k}(p,q)\mathbf{V}^{M}(p)\in\mathbb{R}^{C/2}.$
(6)

Here, $\mathbf{V}^{M}(p)\\!\in\\!\mathbb{R}^{C/2}$ denotes the feature of the
$p^{th}$ 3D position in $\mathbf{V}^{M}$. $\mathbf{H}_{k}(q)$ indicates the
summarized representation of location $q$. For all $H\times W$ locations in
$\mathbf{K}_{k}^{Q}$, we independently apply Eq. (6) and obtain the feature
map $\mathbf{H}_{k}\in\mathbb{R}^{H\times W\times C/2}$. To achieve a more
comprehensive representation, the feature map is concatenated with query value
$\mathbf{V}_{k}^{Q}$ to compute a final representation
$\mathbf{F}_{k}=\texttt{cat}(\mathbf{H}_{k},\mathbf{V}_{k}^{Q})\in\mathbb{R}^{H\times
W\times C}$.

####  2.2.3 Final Segmentation Readout.

$\mathbf{F}_{k}$ is leveraged by a decoder network $f_{\text{Dec}}$ to predict
the final segmentation probability map for the query slice ${I}_{k}$:

$\mathbf{S}_{k}=f_{\text{Dec}}(\mathbf{F}_{k})\in[0,1]^{h\times w}.$ (7)

####  2.2.4 Quality Assessment Module.

While the memory network provides a compelling way to produce 3D segmentation,
it does not support human-in-the-loop scenarios. To this end, we equip the
memory network with a lightweight quality assessment head, which computes a
quality score for each segmentation mask. In particular, we consider mean
intersection-over-union (mIoU) as the basic index for quality measurement. For
each query ${I}_{k}$, we take the feature $\mathbf{F}_{k}$ and the
corresponding segmentation $\mathbf{S}_{k}$ together to regress a mIoU score
$h_{k}$:

$h_{k}=f_{\text{QA}}(\mathbf{F}_{k},\mathbf{S}_{k})\in[0,1],$ (8)

where $\mathbf{S}_{k}$ is firstly resized to a size of $H\times W$ and then
concatenated with $\mathbf{F}_{k}$ for decoding. The slice with the lowest
score is curated for next-round interaction.

###  2.3 Detailed Network Architecture

We follow [31] to implement the interaction network $f_{\text{In}}(\cdot)$ as
a coarse-to-fine segmentation network, however, other network architectures
(e.g., U-Net [21]) can also be used here instead. The network is trained
using the cross-entropy loss. For the quality-aware memory network, we utilize
ResNet-50 [8] as the backbone network for both $f_{\text{Enc}}^{Q}$ (Eq. (2))
and $f_{\text{Enc}}^{M}$ (Eq. (3)). The res4 feature map of ResNet-50 is taken
for computing the key and value embedding. For $f_{\text{Dec}}(\cdot)$, we
first apply Atrous Spatial Pyramid Pooling module after the memory read
operation to enlarge the receptive field. We use three parallel dilated
convolution layers with dilation rates 2, 4 and 8. Then, the learned feature
is decoded with a residual refinement module proposed in [19]. The quality-
aware module, $f_{\text{QA}}(\cdot)$, consists of three $3\\!\times\\!3$
convolutional layers and three fully connected layers.

##  3 Experiment

####  3.0.1 Experimental Setup.

Our experiments are conducted on two public datasets: MSD [24] includes ten
subsets with different anatomy of interests, with a total of 2,633 3D volumes.
In our experiments, we study the most challenging lung ($64/32$ for train/val)
and colon ($126/64$ for train/val) subsets. KiTS19 [9] contains $300$
arterial phase abdominal CT scans with annotations of kidney and tumor. We use
the released $210$ scans ($168/42$ for train/val) for experiments.

For comparison, we build a baseline model, named Interactive 3D nnU-Net, by
adapting nnU-Net [10] into an interactive version. Specifically, we use the
interaction network (§2.1) to obtain an initial segmentation, and this segment
is then concatenated with the volume as the input of 3D nnU-Net. The quality-
aware iterative refinement is also applied. In addition, we compare with a
state-of-the-art method DeepIGeoS [28]. Several non-interactive methods are
also included.

####  3.0.2 Interaction Simulation.

Our approach can support various types of user interactions, which facilitates
use in clinical routine. We study three common interactions: Scribbles provide
sparse labels to describe the targets and rough outreach, Bounding Boxes
outline the sizes and locations of targets, whereas Extreme Points [15]
outline a more compact area of a target by labeling its leftmost, rightmost,
top, bottom pixels. To simulate scribbles, we manually label the data in
KiTS19 and MSD, resulting in $3,\\!585$ slices. Bounding boxes and extreme
points can be easily simulated from ground-truths with relaxations. We train
an independent $f_{\text{In}}$ for each of these interaction types. In the
first interaction round, we compute a rough ROI according to user input. Then
we treat all the pixels out of the enlarged ROI as the background, and the
pixels specified by scribbles, or in regions of bounding boxes and extreme
points as foreground. We encode user guidance as a binary image $\mathbf{M}$
(§2.1) for input.

Table 1: Quantitative results (DSC %) on (left) MSD [24] and (right) KiTS19
[9] val.

method | lung cancer | colon cancer
---|---|---
non-interactive methods:
C2FNAS [30] | 70.4 | 58.9
3D nnU-Net [10] | 66.9 | 56.0
interactive methods:
Interactive 3D nnU-Net [10]
scribbles | 73.9 | 68.1
bounding boxes | 74.7 | 68.5
extreme points | 75.1 | 69.8
DeepIGeoS [28]
scribbles | 76.6 | 72.3
bounding boxes | 77.2 | 73.0
extreme points | 77.5 | 73.2
Ours
scribbles | 80.9 | 79.7
bounding boxes | 81.5 | 79.3
extreme points | 82.0 | 80.4

method | kidney (organ) | kidney (tumor)
---|---|---
non-interactive methods
Mu et al. [17] | 97.4 | 78.9
3D nnU-Net [10] | 96.9 | 85.7
interactive methods
Interactive 3D nnU-Net [10]
scribbles | 94.5 | 86.3
bounding boxes | 95.3 | 86.8
extreme points | 95.6 | 87.6
DeepIGeoS [28]
scribbles | 95.7 | 87.6
bounding boxes | 96.4 | 88.5
extreme points | 96.7 | 88.9
Ours
scribbles | 96.9 | 88.2
bounding boxes | 97.0 | 88.4
extreme points | 97.0 | 89.1

Figure 2: Qualitative results of our approach v.s. Interactive 3D nnU-Net on
two samples in MSD-Lung (row #1) and KITS19 (row #2), using scribbles and
extreme points as supervision, respectively. (a) Interactive slices; (b)
Results of interactive slices using the interaction network; (c)-(e) Results
of other slices. Zoom in for details.

####  3.0.3 Training and Testing Details.

Our engine is implemented in PyTorch. We use the same settings as [31] to
train $f_{\text{In}}$ (§2.1). The quality-aware memory network
$f_{\text{Mem}}$ (§2.2) is trained using Adam with learning rate 1e-5 and
batch size $8$ for $120$ epochs. To make a training sample, we randomly sample
$5$ temporally ordered slices from a 3D image. During training, the memory is
dynamically updated by adding the slice and mask at the previous step to the
memory for the next slice.

During inference, simulated user hints are provided to $f_{\text{IN}}$ for an
initial segmentation of the interactive slice. Then, for each query slice, we
put this interactive slice and the previous slice with corresponding
segmentation mask into the memory as the most important reference information.
In addition, we save a new memory item every $N$ slices, where $N$ is
empirically set to $5$. We do not add all slices and corresponding masks into
memory to avoid large storage and computational costs. In this way, our memory
network achieves the effect of online learning and adaption without additional
training.

####  3.0.4 Quantitative and Qualitative Results.

Table 1 (left) reports segmentation results of various methods on MSD val. For
interactive methods, we report results at the $6^{th}$ round which well
balances accuracy and efficiency. It can be seen that our method leads to
consistent performance gains over the baselines. Specifically, our approach
significantly outperforms Interactive 3D nnU-Net by more than 7% for lung
cancer and 10% for colon cancer, and outperforms DeepIGeoS [28] by more than
4% and 7%, respectively. Moreover, for different types of interaction, our
method produces very similar performance, revealing its high robustness to
user input. Table 1 (right) presents performance comparisons on KiTS19 val.
The results demonstrate that, for kidney tumor segmentation, our engine
generally outperforms the baseline models. The improvements are lower than
seen for the MSD dataset due to the fact that the initial segmentation is
already of high quality resulting in smaller dice score gains for adjustments.

Fig. 2 depicts qualitative comparisons of our approach against Interactive 3D
nnU-Net on representative examples from MSD and KITS19. As seen, our approach
produces more accurate segmentation results than the competitor.

Table 2: Ablation study of the quality assessment module in terms of DSC (%).

variant | MSD (lung) | MSD (colon) | KiTS19 (tumor)
---|---|---|---
oracle | 81.4 | 80.4 | 89.1
random | 80.1 | 77.5 | 86.8
quality assess. | 81.3 | 79.7 | 88.6

Figure 3: The impact of number of interactions on MSD Lung (left) and Colon
(right).

####  3.0.5 Efficacy of Quality Assessment Module.

The quality assessment module empowers the engine to automatically select
informative slices for iterative correction. To prove its efficacy, we design
two baselines: ‘oracle’ selects the worst segmented slice by comparing the
masks with corresponding ground-truths, while ‘random’ conducts random
selection. As reported in Table 2, our method (i.e., quality assessment
module) significantly outperforms ‘random’ across three sets, and is
comparable to ‘oracle’, proving its effectiveness.

####  3.0.6 Impact of Multi-Round Refinement.

Fig. 3 shows DSC scores with growing number of interactions on lung and colon
subsets of MSD. We observe that multi-round refinement is crucial for
achieving higher segmentation performance, and the performance becomes almost
marginal at the $16^{th}$ round.

####  3.0.7 Runtime Analysis.

For a 3D volume with size $512\\!\times\\!512\\!\times\\!100$, our method
needs 5.13 s on average for one-round segmentation on a NVIDIA 2080Ti GPU,
whereas Interactive 3D nnU-Net needs 200 s. Hence our engine enables a
significant increase in inference speed.

##  4 Conclusion

This work presents a novel interactive segmentation engine for 3D medical
volumes. The key component is a memory-augmented neural network, which employs
an external memory for accurate and efficient 3D segmentation. Moreover, the
quality-aware module empowers the engine to automatically select informative
slices for user feedback, which we believe is an important added value of the
memory network. Experiments on two public datasets show that our engine
outperforms other alternatives while having a much faster inference speed.

## References

  * [1] Boykov, Y.Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & region segmentation of objects in nd images. In: ICCV. pp. 105–112 (2001)
  * [2] Bredell, G., Tanner, C., Konukoglu, E.: Iterative interaction training for segmentation editing networks. In: International Workshop on Machine Learning in Medical Imaging. pp. 363–370 (2018)
  * [3] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: MICCAI. pp. 424–432 (2016)
  * [4] Criminisi, A., Sharp, T., Blake, A.: Geos: Geodesic image segmentation. In: ECCV. pp. 99–112 (2008)
  * [5] Grady, L.: Random walks for image segmentation. IEEE TPAMI 28(11), 1768–1783 (2006)
  * [6] Grady, L., Schiwietz, T., Aharon, S., Westermann, R.: Random walks for interactive organ segmentation in two and three dimensions: Implementation and validation. In: MICCAI. pp. 773–780 (2005)
  * [7] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: CVPR. pp. 9729–9738 (2020)
  * [8] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
  * [9] Heller, N., Sathianathen, N., Kalapara, A., Walczak, E., Moore, K., Kaluzniak, H., Rosenberg, J., Blake, P., Rengel, Z., Oestreich, M., et al.: The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes. arXiv preprint arXiv:1904.00445 (2019)
  * [10] Isensee, F., Petersen, J., Klein, A., Zimmerer, D., Jaeger, P.F., Kohl, S., Wasserthal, J., Koehler, G., Norajitra, T., Wirkert, S., et al.: nnu-net: Self-adapting framework for u-net-based medical image segmentation. arXiv preprint arXiv:1809.10486 (2018)
  * [11] Kitrungrotsakul, T., Yutaro, I., Lin, L., Tong, R., Li, J., Chen, Y.W.: Interactive deep refinement network for medical image segmentation. arXiv preprint arXiv:2006.15320 (2020)
  * [12] Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., Socher, R.: Ask me anything: Dynamic memory networks for natural language processing. In: ICML. pp. 1378–1387 (2016)
  * [13] Liao, X., Li, W., Xu, Q., Wang, X., Jin, B., Zhang, X., Wang, Y., Zhang, Y.: Iteratively-refined interactive 3d medical image segmentation with multi-agent reinforcement learning. In: CVPR. pp. 9394–9402 (2020)
  * [14] Lu, X., Wang, W., Danelljan, M., Zhou, T., Shen, J., Van Gool, L.: Video object segmentation with episodic graph memory networks. In: ECCV (2020)
  * [15] Maninis, K.K., Caelles, S., Pont-Tuset, J., Van Gool, L.: Deep extreme cut: From extreme points to object segmentation. In: CVPR. pp. 616–625 (2018)
  * [16] Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3DV. pp. 565–571 (2016)
  * [17] Mu, G., Lin, Z., Han, M., Yao, G., Gao, Y.: Segmentation of kidney tumor by multi-resolution vb-nets (2019)
  * [18] Olabarriaga, S.D., Smeulders, A.W.: Interaction in the segmentation of medical images: A survey. MedIA 5(2), 127–142 (2001)
  * [19] Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., Jagersand, M.: Basnet: Boundary-aware salient object detection. In: CVPR. pp. 7479–7489 (2019)
  * [20] Rajchl, M., Lee, M.C., Oktay, O., Kamnitsas, K., Passerat-Palmbach, J., Bai, W., Damodaram, M., Rutherford, M.A., Hajnal, J.V., Kainz, B., et al.: Deepcut: Object segmentation from bounding box annotations using convolutional neural networks. IEEE TMI 36(2), 674–683 (2016)
  * [21] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241 (2015)
  * [22] Sakinis, T., Milletari, F., Roth, H., Korfiatis, P., Kostandy, P., Philbrick, K., Akkus, Z., Xu, Z., Xu, D., Erickson, B.J.: Interactive segmentation of medical images through fully convolutional neural networks. arXiv preprint arXiv:1903.08205 (2019)
  * [23] Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., Lillicrap, T.: Meta-learning with memory-augmented neural networks. In: ICML. pp. 1842–1850 (2016)
  * [24] Simpson, A.L., Antonelli, M., Bakas, S., Bilello, M., Farahani, K., Van Ginneken, B., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., et al.: A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv:1902.09063 (2019)
  * [25] Sukhbaatar, S., Szlam, A., Weston, J., Fergus, R.: End-to-end memory networks. arXiv preprint arXiv:1503.08895 (2015)
  * [26] Sun, J., Shi, Y., Gao, Y., Wang, L., Zhou, L., Yang, W., Shen, D.: Interactive medical image segmentation via point-based interaction and sequential patch learning. arXiv preprint arXiv:1804.10481 (2018)
  * [27] Wang, G., Li, W., Zuluaga, M.A., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., et al.: Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE TMI 37(7), 1562–1573 (2018)
  * [28] Wang, G., Zuluaga, M.A., Li, W., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., et al.: Deepigeos: a deep interactive geodesic framework for medical image segmentation. IEEE TPAMI 41(7), 1559–1572 (2018)
  * [29] Wang, W., Zhou, T., Yu, F., Dai, J., Konukoglu, E., Van Gool, L.: Exploring cross-image pixel contrast for semantic segmentation. arXiv preprint arXiv:2101.11939 (2021)
  * [30] Yu, Q., Yang, D., Roth, H., Bai, Y., Zhang, Y., Yuille, A.L., Xu, D.: C2fnas: Coarse-to-fine neural architecture search for 3d medical image segmentation. In: CVPR (2020)
  * [31] Zhang, S., Liew, J.H., Wei, Y., Wei, S., Zhao, Y.: Interactive object segmentation with inside-outside guidance. In: CVPR. pp. 12234–12244 (2020)
  * [32] Zhao, F., Xie, X.: An overview of interactive medical image segmentation. Annals of the BMVA 2013(7), 1–22 (2013)
  * [33] Zhou, Y., Xie, L., Shen, W., Wang, Y., Fishman, E.K., Yuille, A.L.: A fixed-point model for pancreas segmentation in abdominal ct scans. In: MICCAI. pp. 693–701 (2017)
  * [34] Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., Liang, J.: Unet++: A nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 3–11 (2018)