# CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted
Interventions

Tom Vercauteren, Mathias Unberath, Nicolas Padoy, and Nassir Navab T.
Vercauteren is with the School of Biomedical Engineering & Imaging Sciences,
King’s College London.M. Unberath is with the Department of Computer Science,
Johns Hopkins University.N. Padoy is with the ICube institute, University of
Strasbourg, CNRS, IHU Strasbourg, FranceN. Navab is with the Fakultät für
Informatik, Technische Universität München.

###### Abstract

Data-driven computational approaches have evolved to enable extraction of
information from medical images with a reliability, accuracy and speed which
is already transforming their interpretation and exploitation in clinical
practice. While similar benefits are longed for in the field of interventional
imaging, this ambition is challenged by a much higher heterogeneity. Clinical
workflows within interventional suites and operating theatres are extremely
complex and typically rely on poorly integrated intra-operative devices,
sensors, and support infrastructures. Taking stock of some of the most
exciting developments in machine learning and artificial intelligence for
computer assisted interventions, we highlight the crucial need to take context
and human factors into account in order to address these challenges.
Contextual artificial intelligence for computer assisted intervention, or
CAI4CAI, arises as an emerging opportunity feeding into the broader field of
surgical data science. Central challenges being addressed in CAI4CAI include
how to integrate the ensemble of prior knowledge and instantaneous sensory
information from experts, sensors and actuators; how to create and communicate
a faithful and actionable shared representation of the surgery among a mixed
human-AI actor team; how to design interventional systems and associated
cognitive shared control schemes for online uncertainty-aware collaborative
decision making ultimately producing more precise and reliable interventions.

###### Index Terms:

Artificial intelligence, computer assisted interventions, interventional
workflow, intra-operative imaging, surgical planning, data fusion, surgical
scene understanding, context-aware user interface, machine and deep learning,
surgical data science

##  I Introduction

Contemporary progresses in machine learning and artificial intelligence have
permitted the development of tools that can assist clinicians in exploiting
and quantifying clinical data including images, textual reports and genetic
information. State-of-the-art algorithms are becoming mature enough to provide
automated analysis when applied to well-controlled clinical studies and trials
[1, 2], but adapting these tools for patient-specific management remains an
active research area, with the bulk of the research community having focused
on fully automated machine learning tools. These considerations become
especially critical in the highly heterogeneous context of surgery and
interventional procedures which require patient- and team-specific decision
support tools able to draw information from non-standardised interventional
devices integrated in diverse interventional suites. Compared to computational
tasks in radiology, the domain of computer-assisted intervention further
creates unique methodological challenges, such as imposing stringent time
constraints in the interventional suite, requiring knowledge of procedural
data, and needing methods that deal with dynamic environments.

In this paper, keeping a focus on imaging data, we review existing work, and
share insights on future developments, of machine learning strategies that
decipher, support, augment and integrate in various surgical and
interventional workflows while providing the flexibility required by clinical
management. Flexibility is for example mandated to be able to deal with
missing input sources, react to real-time user feedback, adapt to the patient
risk aversion and preferences, handle uncertain or contradictory information,
learn from potentially small and heterogeneous data, etc. All of which are
common in computer assisted interventions. Imaging sources of particular
interest for surgery and intervention include a wide range of well-known
interventional modalities such as surgical microscopy, video endoscopy, X-ray
fluoroscopy and ultrasound; more emerging biophotonics imaging modalities such
as hyperspectral imaging, endomicroscopy and photoacoustic imaging; but also
span classical radiology modalities such as MRI and CT which remain the main
sources of imaging data for pre-operative intervention planning and post-
operative assessment. We argue that the stringent need to consider context
when analysing surgical and interventional data coupled with the heterogeneity
of information sources and domain knowledge in computer assisted intervention
applications calls for the development of novel domain-specific contextual
artificial intelligence solutions, a domain we coin as CAI4CAI. Feeding into
the broader field of Surgical Data Science [3, 4, 5], CAI4CAI will focus on
the underpinning machine learning methodology exploiting contextual
information and human interaction to enable the required responsiveness to
deliver clinical impact in surgery and interventional sciences.

To support our claim, we highlight some of the transformative machine learning
research results and methodologies currently being developed across the
spectrum of tasks in computer assisted interventions. The impact of machine
learning in intervention planning is discussed in Section II; intra-operative
data fusion in Section III; intelligent intra-operative imaging in Section IV;
surgical and endoscopic vision in Section V; and clinical workflow monitoring
and support in Section VI. In these sections, we will highlight how flexible
deep-learning based tools are becoming critical for the design of effective
and efficient intervention planning solutions. During surgery, navigation
solutions are often used to map preoperative information in the context of the
intervention. However, navigation does not account for intra-operative
changes. Learning how to co-register images is now leading to intra-operative
registration solutions that are able to cope with the highly challenging task
of aligning pre-operative to intra-operative images coming from different
imaging modalities. Concurrently, AI methodology is advancing to go beyond
traditional navigation-based data fusion and image overlay to exploit
information coming from complex or synergistic data sources. This is giving
rise to what we refer to as intelligent intra-operative imaging. Data-driven
modelling strategies coming from the computer vision community are acting as
instrumental starting points to achieve semantic information extraction from
interventional data sources including endoscopic videos, with applications
ranging from automated polyp detection to surgical activity recognition. To
deliver improved clinical outcomes through AI, all these building blocks are
increasingly being integrated at the level of the complete surgical workflow
with applications spanning the full breath of surgical data science. In this
area, starting from data-driven mapping of clinical workflow and skills
assessment, AI is now helping make contextual decision support tools and
conditionally autonomous intervention a reality. Finally, closing thoughts are
provided and further budding applications of CAI4CAI are discussed in Section
VII.

##  II Intervention Planning

###  II-A Clinical Adoption of Intervention Planning Tools

Once a decision is made for a patient to undergo an interventional procedure,
for any non-trivial operation, patient-specific planning of the intervention
is required. The steps involved usually necessitate acquisition of reference
pre-operative imaging data, semantic segmentation of anatomical structures in
these images, determination of the surgical approach and elaboration of an
intra-operative plan leading to optimal outcomes for the patient. Such a plan
might encompass establishing a surgical path and target, designing or
selecting a patient-specific implant or assistive adjunct tool such as a drill
or saw guide [6]. In the vast majority of cases, such intervention planning is
performed by a team of healthcare professionals, each with their own
expertise, known as the multidisciplinary team (MDT). Relatively little
computer assistance is currently available for interventional planning in the
clinic. Notable exceptions can be found in the field of neurosurgery, oral and
maxillofacial surgery and orthopedic surgery. What these specialties share is
a relatively static surgical scene thanks to the proximity of rigid bone
structures. Computed tomography (CT) provides a rich source of 3D imaging
information in this context. Indeed, thanks to the quantitative nature of CT
images and the good contrast of bone, automated segmentation of bone has
proven to be clinically reliable. Since the seminal work of the Retrospective
Registration Evaluation Project (RREP) [7], it is also clear that preoperative
rigid registration of different imaging modalities such as MR and CT provides
a robust means of fusing soft tissue contrast information with accurate bone
delineation for neurosurgical planning. Such technical advances have supported
the adoption of stereotactic surgery as a means of accurately targeting and
guiding instrument towards deep seated brain structures for procedures such as
brain biopsies for tumour grading and electrode implantation for the treatment
of movement disorder or the localisation of epileptic seizure onset zones.
While computer assisted surgical planning and subsequent surgical navigation
have become standard of care in neurosurgery and a few other disciplines, even
in these fields, there is major scope to make the workflow more efficient
through the development of further machine learning enabled computer
assistance.

Figure 1: Interactive algorithms are required to deliver context-aware
artificial intelligence. In this example, using the algorithm presented in
[8], brain tumour segmentation is initially performed automatically using a
pre-trained algorithm. As part of the surgical planning, the user may want to
refine the segmentation by providing scribbles to denote areas that should be
excluded (green) or included (pink) irrespective of the initial segmentation.
The algorithm then adapts its output to respect the user input.

###  II-B Machine Learning in Interventional Planning

Commercial surgical planning products are still limited in the automation they
support, with many of the most advanced ones essentially relying on classical
image analysis methods such as atlas-based segmentation [9] to delineate soft-
tissue structures of interests for patient showing no gross pathological brain
changes. Clinicians are often left with manual or generic interactive methods
to delineate other structures of interest and define their surgical plan. When
interventional planning only relies on the clinician getting a volumetric
representation of the patient anatomy from pre-operative data, advanced
visualisation techniques such as cinematic rendering [10] can be considered as
alternatives to explicit segmentation of structures. These may produce results
that are less sensitive to noise and data variability but do not enable more
quantitative planning. Developments of deep machine learning segmentation
algorithms dedicated to medical imaging [11, 12] is rapidly changing to level
of accuracy at which automated segmentation of structures of interest can be
done in population of patients even in the presence of gross pathological
changes [13]. Yet many challenges remain for these tools to become of
practical use for intervention planning purposes. Poor generalisation when
faced with slight domain changes is a recognised problem in the entire medical
imaging community including on the diagnostic side. Expanding the size of the
datasets on which deep learning algorithms are trained would certainly
mitigate generalisation issues by providing a much larger variety of training
cases. Collaborative efforts within the community are notably focusing on
providing open-access large annotated datasets for machine learning training
purposes in some specific use cases [1]. Yet, collecting task-specific large
annotated databases for medical imaging purposes faces its own challenges
given the time and expertise required to provide detailed annotations as well
as the legal, privacy and storage questions pertaining to sharing large
patient datasets across multiple sites. Federated learning for multi-
institutional collaboration in medical imaging [14, 15] provides a potential
technical solution to this problem. Implementing such solutions at scale will
require concerted efforts reaching far beyond the methodological research
community. Furthermore, changes such as device upgrades or challenges posed by
new clinical indications will not be captured by increasing the pool of
retrospective training data. Active research to address such inevitable but
unpredictable domain gaps is rooted in domain adaptation techniques [16].
These advances are necessary for automated machine learning tools to make an
impact in the clinical setting. Prospective randomised clinical trials (RCT)
are widely seen as the only source of trustworthy clinical evidence, yet
studies implementing RCTs with systems relying on deep learning tools for
medical imaging currently remain noteworthy exceptions [17].

###  II-C Importance of Flexible Contextual Machine Learning

What distinguishes segmentation in surgical planning from segmentation in
diagnostic imaging is nonetheless that the objective is not necessarily always
that of reaching the best performance in getting the structures delineated
with sub-voxel accuracy. Surgical planning needs to respect patient-specific
needs and preferences of the surgeon. This requires putting the clinical team
at the centre and promoting flexible tools that integrate into the surgical
workflow. Interactive deep learning methodologies are emerging to combine rich
prior knowledge embedded in retrospective data from previous patients with as-
sparse-as-possible annotations provided by clinicians [18, 8]. As illustrated
in Figure 1, deep interactive segmentation allows the clinical expert to
refine the results from an initial automated step and most importantly to
adapt the inferred results on the fly based on contextual information.
Furthermore, given the heterogeneity and evolving nature of surgical practice,
additional flexibility is required to handle potentially missing input
modalities. Recent work in deep machine learning are focusing on dealing with
such dynamic hetero-modal context while exploiting heterogeneous sources of
data for the training process [19, 20]. Bringing flexible machine learning
tools to maturity will certainly play an important role in supporting the
clinical adaption of AI in surgery.

As highlighted above, segmentation of structures from pre-operative images is
often the foundation of computer assisted surgical planning and this currently
remains the state-of-the-art in many commercial solutions. Such static
segmentation when combined with intra-operative registration already provides
useful surgical navigation information for relatively static surgical scenes
as is the case in neurosurgery. Nevertheless, computer assistance for
intervention planning has the potential to provide impact much beyond the
ability to automate the creation of 3D anatomical models and overlay of
functional data. Patient-specific simulation of given surgical plans has for
example been introduced in orthopedic surgery with a long history in
acetabular fracture surgery [21]. State-of-the-art orthopedic surgery planning
systems allow to design patient-specific implants and patient-specific
surgical guides by enabling the simulation of the effect of different implants
and implantation strategy on key outcome-related parameters such as the range
of motion of an articulation or the limb length [22]. Yet, these tools often
ignore the effect of soft-tissue in the simulation process and still require
very labour intensive work for the surgical team to design patient-specific
plans. Experts systems capable of automatically optimising the surgical plan
for a given orthopedic surgery are now being developed [23] and promise to
make surgical planning more efficient [24]. In the context of deep brain
insertion of instruments, machine learning approaches capable of automatically
planning trajectories of multiple instruments, to maximise the efficacy of the
surgery while minimising intra-operative risks and avoiding collisions between
instruments have demonstrated a significant reduction in planning time for the
implantation of stereo-electroencephalography electrodes for epilepsy
treatment [25] and for laser interstitial thermal therapy [26]. Contextual and
flexible machine learning for surgical planning promises to push the
boundaries of interventional planning by exploiting data-driven approaches and
real-time user feedback to efficiently plan for complex situations. An
instrument bending model was for example trained in [27] to predict the
deviation between an original surgical plan assuming rigid electrodes and the
actual electrode paths as measured on a post-operative CT. Provided reliable
uncertainty estimates on the prediction can be achieved, embedding such
deflection models in the trajectory planning is expected to improve the safety
and accuracy of stereo-electroencephalography electrode implantation planning.

Effectively, planning is moving away from extraction of information captured
in existing data and representative of a given (pre-operative) time point.
Context-aware learning methods are now being developed to also predict
therapy-related changes and better inform interventional planning. By
exploiting computationally complex noninvasive cardiac electrophysiology
modelling coupled with transfer learning approaches, [28] notably achieved
online personalized predictions of electrophysiology cardiac resynchronization
therapy responses, thereby paving the way for better patient selection and
patient-specific therapy optimisation. In non-quasi-static environments,
surgical planning is currently further limited by our capabilities to predict
intra-operative anatomical changes. In abdominal surgery for example,
segmentation of structures from preoperative images may inform the clinician
about the relative spatial organisation of lesions and vascular structures.
However, at the onset of a minimally invasive procedure, gas insufflation is
typically performed to create the surgical workspace. This has a serious
impact on the geometry of the anatomy and challenges any attempt of intra-
operative use of a 3D model of the anatomy generated from pre-insufflation
images. Current approaches typically rely on focusing on smaller regions where
rigidity assumptions between pre- and intra-operative data may still hold [29]
thereby limiting the scope of the surgical planning. Data-driven prediction of
anatomical changes relating to gas insufflation in laparoscopic surgery was
proposed in [30]. Still in the context of liver surgery, a system able to take
into account non-imaging patient data and factual knowledge gathered from
quotable sources such as clinical guidelines was proposed to support
individualised treatment planning [31]. While relying on handcrafted features
and exploiting models with limited expressiveness, this study paved the way
for more holistic interventional planning. It is expected that context-aware
interventional planning will be informed by refined prediction models to
suggest therapeutic plans cognizant of clinical experience as well as
potential intra-operative changes and associated risks but also flexible
enough to take into account any further input from the interventional team
interacting with a responsive planning system.

##  III Intra-operative Data fusion

###  III-A Navigation and Image Registration Challenges

No matter how refined and capable interventional planning becomes, its full
value for procedural guidance and intra-operative decision making support
remains contingent on appropriate geometric alignment with intra-operatively
acquired data. This alignment is achieved using registration methods that
either rely on dedicated external hardware, such as optical or electromagnetic
tracking systems [32], or operate directly on intra-operative images [33].

Image-based registration in the interventional context has received
substantial academic attention [34, 35]. This is because external navigation,
while improving surgical accuracy, is associated with increased procedural
time, and complex and manual intra-operative calibration procedures that may
lead to a high level of surgeon frustration [36]. It is widely believed that
image-based registration will better integrate with procedural workflow,
mitigating many negative aspects of external tracking approaches while
providing similar accuracy. Further, since no additional hardware is required,
there is great potential for widespread adoption and deployment of these
purely software-driven methods. This suggests that navigated surgery may also
become available in remote and rural hospitals that could not afford dedicated
equipment otherwise.

Despite clear opportunity, image-based registration is not yet widely used in
interventional clinical practice. This is because, depending on the clinical
context, several challenges of image-based registration have not yet been
solved reliably. During surgery, the anatomy undergoes highly complex
deformations including the loss of mass or topological changes during
resections. Accurately recovering bio-mechanically plausible transformations
that represent anatomical change from pre- to intra-operative state that are
measured with different imaging modalities is the subject of ongoing research.
Here, we will focus on two of the associated challenges: 1) Modeling image
similarity between images of the same anatomy but acquired with different
modalities, and 2) estimating initial transformation parameters that are good
enough for registration algorithms to succeed.

On a high level, image registration seeks to find a transformation that, when
applied to the moving image, aligns it with the target image such that
locations in both images are in correspondence. Quantifying _correspondence_
is achieved using image similarity metrics that, usually, operate on the image
intensity values. Straightforward comparison of intensity values, e. g., using
a simple sum of squared differences, is generally unrewarding since the
underlying assumption on image formation are prohibitively strong, even when
moving and target image are acquired with the same imaging modality. For
interventional image fusion, the problem is more challenging since images of
different modalities must be aligned. In this case, the additive Gaussian
noise assumption underpinning the sum of squared differences is certainly
violated. Even worse, due to the different physical processes that govern
image formation, there is no guarantee that the same anatomical structures are
visible in both images, thereby challenging the adequacy of co-occurrence-
based similarity metrics, including correlation and mutual information.
Nonetheless, despite these limitations, model-based image similarity criteria
currently remain state-of-the-art performers in many interventional image-
registration tasks including ultrasound to MRI registration for neurosurgical
guidance [37, 38].

###  III-B Contextual Learning for Image Registration

Using deep learning to go past some of the limitation of classical image
registration is an active area of research. However, due to the fundamental
challenge of gathering ground-truth data for image registration, many of the
most successful learning-based registration methods for diagnostic images
exploit unsupervised learning and optimise a classical image similarity metric
based loss [39, 40]. This approach remain unsuitable for most interventional
purposes where more flexible solutions are required. A prominent example
highlighting the need to take the interventional context into account is
transrectal ultrasound (TRUS)-guided prostate biopsy. Conventionally, the
biopsy target is segmented on pre-operative 3D MR images and this must then be
registered to intra-operative 3D TRUS volumes. Since MR and TRUS images
exhibit substantially different image appearance, contrast, and artifact
level, this suggests that no good mathematical model exists to describe image
similarity between these two modalities. Data-driven approaches that do not
explicitly model intensity correlations to test for image correspondence but
optimize a surrogate measure thereof now achieve state-of-the-art performance.
One candidate surrogate measure can be defined by enforcing segmentations of
the same structures to exhibit maximal overlap after registration [41].
Remarkably, learning to optimise for such losses does not require access to
ground-truth for the spatial transformation and leverages application-specific
annotations that are considered as weak annotations. Further contextual
information can be captured by learning data-driven spatial transformation
models or regularisation terms [42]. Related physics-based deformation models
have been trained to predict shape changes in segmented organs from sparse
annotations which could be used for augmented reality purposes [43, 44].
Taking account of the interventional context one step further, [45] noticed
that in many cases including MR-TRUS guided biopsy, the main purpose of
interventional data fusion is to propagate a patient-specific target defined
on a pre-operative image to its interventional counterpart and proposed to
replace the registration step by a conditional segmentation one.

Even in scenarios where data-driven similarity metrics may be learned, finding
the transformation that optimally aligns a pair of images can remain non-
trivial. This is because image similarity is well defined, i. e. informative,
only in a narrowly circumscribed vicinity around the true transformation,
emphasizing the need for appropriate initialization, such that the initial
mismatch falls within the _capture range_ of the image similarity metric and
optimization algorithm [46]. While adequate initialization is challenging in
all registration scenarios, it is considered to be most detrimental in slice-
to-volume applications. Such applications are common in image-guided
interventions, with the most prominent examples being the bijective alignment
of 2D B-mode ultrasound to 3D MR or CT volumes or the projective registration
of pre-operative 3D MR or CT volumes, or CAD models to intra-operative 2D
X-ray or endoscopy images.

In cases where the 3D imaging protocol context is well defined, i. e. one is
guaranteed to observe the same extent of anatomy, direct approaches to
initialization are possible. These methods only accept the 2D image as input
and directly estimate its initial pose relative to a 3D canonical atlas
coordinate system that is implicitly defined by the choice of 3D image
database [47, 48] or tool model [49]. These approaches are attractive, mainly
due to two reasons: First, run times are short since only 2D images must be
processed; and second, they lend themselves well for scenarios where 2D slices
are acquired successively to reconstruct a full 3D volume. However, due to the
complexity of the problem and canonical atlas assumption, their performance is
often limited in practice.

When a canonical space cannot be defined, alternative approaches typically
mimic the external tracking workflow where relative poses are inferred
analytically. While external tracking devices require attachment or
implantation of artificial fiducial markers to get position information
readouts, AI-based approaches seek to establish correspondence directly from
the images or from sparse but corresponding image locations. In [50], by
learning from a dataset of tracked ultrasound, the authors demonstrated that,
without inference-time reliance on the tracker, deep learning approaches can
estimate the 3D motion occurring in between consecutive 2D ultrasound images
with an accuracy far exceeding that of conventional speckle decorrelation
techniques and matching that of the external tracker. This is allowing for
sensorless 3D freehand ultrasound and creates new opportunities in computer
assisted interventions. Another complementary powerful concept for tracker-
less image alignment is the detection and identification of anatomical
landmarks. These are particularly appealing since they carry semantic meaning,
and consequently, define point correspondence across modalities and domains.
Reliably detecting anatomical landmarks is complicated because of changing
appearance based on viewpoints, but has recently become possible due to
powerful convolutional neural network-based image analysis for anatomical
landmarks as shown in the pelvis [51, 46] and knees [52]. The same concept of
point correspondence naturally extends to tools and implants where, rather
than relying on anatomical landmarks, keypoints on the CAD model are used [53,
54, 55]. The aforementioned approaches aim at discovering well defined points,
however, finding the same arbitrary point in multiple images is equally
appropriate to establish correspondence. In this formulation of the problem,
an AI-based algorithm is trained to produce a pose invariant latent
representation of point appearance. Then, query points can be randomly sampled
in one image that are then re-discovered in the target image [56], thereby
establishing correspondence. This approach is appealing since it does not
impose any prior on the imaged object, however, learning a pose invariant
latent representation so far has only been demonstrated for comparably small
pose differences.

Figure 2: Realistic simulation of X-ray image formation from pre-operative CT
is one possibility to create large quantities of well annotated images.
Pipeline represents the simulation approach described in [57].

##  IV Intelligent Intra-operative Imaging

###  IV-A From Data Fusion to Intelligent Imaging

Intelligent intra-operative imaging refers to augmenting the value of intra-
operative images for clinical decision making by providing additional
information that is tailored to the context of the intervention. In
increasingly granular order, context here describes the interventional
requirements specific to a certain procedure, step in the surgical workflow,
decision, or even surgeon’s preferences. So far, efforts in this direction are
dominated by data fusion methods that seek to enrich intra-operative images
with procedural planning information that exists from pre-operative data.
While this approach, even when relying on classical CAI tools, has been
deployed successfully for several types of procedures [33], it is
fundamentally limited in its capabilities of fully leveraging all acquired
data. This is because the value of intra-operative images is reduced to a
proxy to support, e. g., image-based registration or as a means for overlay,
while all _intelligent information_ that really augments the decision making
is propagated solely from pre-operative images. In addition to under-
exploiting intra-operative images, this strategy only allows for displaying
information derived from pre-operative data that becomes outdated as surgery
progresses. This calls for the development of intelligent intra-operative
imaging that fully leverages the information contained in interventionally
acquired data in real-time. Augmenting decision making in this way offers
clear opportunities by: 1) automating quantitative measurements required for
precision medicine; and 2) extracting information that is otherwise not easily
accessible which may allow development of new surgical techniques. Still,
contextual and intelligent interventional image analysis is not yet mainstream
technology because, compared to diagnostic image analysis, the environment for
developing AI solutions is even more hostile. From our experience working with
clinical collaborators across different sites and specialties, we believe that
this is primarily due to three reasons. First, while hundreds of images are
acquired for procedural guidance, only very few, if any, are archived [58, 59,
60], thereby suggesting a severe lack of meaningful data for researchers to
work with. Second, learning targets beyond segmentation are not well
established or defined. Third, images of the anatomy are acquired from
multiple viewpoints, the exact poses of which are not reproduced nor known.
Finally, the overall variability in the data is further amplified by surgical
modification of anatomy and presence of tools. Overall, the accessible data is
heavily unstructured and exhibits enormous variation, which challenges
meaningful data augmentation strategies. As a consequence, in order to train
AI algorithms on interventional images, solutions to the dataset curation and
annotation problem must be found first. Overcoming these hurdles seems
challenging and is reflected in the observation that only very little work has
considered learning in this context. It is worth mentioning that the lack of
annotated and/or paired data equally affects other methods presented in this
manuscript.

###  IV-B Simulation-based Training

First steps in addressing the data problem have been taken, serving as a
stepping stone for the transformative technology that is _intelligent
imaging_. While large scale acquisition of highly structured data is tractable
for some interventional applications, particularly ultrasound [61, 62], most
other approaches rely on synthetic data generation from physical models of the
scene. This paradigm is attractive because all quantities of interest are
precisely known by design, however, if simulation is performed naïvely, AI
models trained on synthetic data will not generalize to clinically acquired
images because of the large domain mismatch paired with poor generalizability
of today’s models [57]. Three complementary ways have recently been shown to
mitigate this problem. First, if clinically acquired data is available in
addition to the well annotated synthetic data, style transfer algorithms can
be trained that alter the appearance of real data to close the domain gap, as
shown for ophthalmic surgical microscopy [63, 64]. Using such enhanced
simulated data for training of more complex tasks has been applied
successfully to endoscopy [65] and X-ray imaging [66]. Second, if too little
clinical data is available, learning a style transfer algorithm is impossible.
In these cases, a powerful alternative is increasing the realism of
synthetically generated images in a model-based approach. Doing so requires
accurate models of all physical principles that govern image formation,
however, approximations are usually required to reduce simulation time to
acceptable levels. Realistic simulation works well for X-ray-based modalities
as illustrated in Figure 2) and demonstrated in [67, 57]. It has also been
proposed in endoscopic imaging [68]. However, the level of required realism
likely depends on the application and learning target, since it has been shown
that even less realistic simulations could be adequate, e. g., in some
ultrasound applications [69]. The aforementioned approaches aim at reproducing
real data appearance which is very complicated in practice. If closely
matching real data appearance is found to be impossible, domain randomization
can be used to improve the robustness of the trained model to partially unseen
data. Rather than perfectly matching real data characteristics, the goal of
domain randomization is to generate multiple versions of the same sample with
all but the important characteristics randomized. When training AI algorithms
on such datasets, the models are assumed to become robust to these types of
domain changes. Domain randomization can be seen as image formation-based data
augmentation and has recently been applied to X-ray imaging [70] as well as
colonoscopy [68], where achieving realistic image appearance is very
complicated due to fine texture and specular reflectance of the tissue. It is
worth mentioning that all the above techniques for synthetic data usage are
similar in that AI algorithms never process real data during training. This
characteristic is associated with a notable drop in performance when applied
to real data due to residual domain mismatch. Consequently, assessing
algorithmic performance only on a synthetic test set will severely
overestimate the AI models accuracy during deployment and quantitative
experiments on clinical data are required. Ultimately, training the AI
directly on real data is preferable, highlighting the need for further
research on un- and self-supervised learning to leverage large quantities of
unlabeled data.

###  IV-C Intelligent Imaging in Interventional Biophotonics

Although conventional interventional imaging, such as X-ray fluoroscopy,
surgical microscopy, endoscopy and ultrasound will benefit from being
augmented by contextual AI, another interesting area in which the intelligent
imaging paradigm is expected to make an important impact is that of
interventional biophotonics imaging. The initial focus in biophotonics has
been on developing optimal, task-specific, contrast agents that would be
merely be directly visualised, e.g. in tumour-specific fluorescence imaging.
The biophotonics community has however faced stringent challenges in
identifying versatile contrast agents suitable for use in patients and
realised that tissue differentiation would remains challenging with such an
approach. Advanced high-dimensional optical imaging techniques are currently
seen as promising solutions for intraoperative tissue characterisation, with
the advantages of being non-contact, non-ionising and non- or minimally-
invasive. However, because of the high-dimensional nature of the generated
data, direct visualisation by the clinical team becomes impractical. This
calls for automated learning-based information extraction before display. As
in the previous examples of intelligent imaging, many of the most advanced AI-
supported interventional biophotonics imaging devices currently exploit model-
based learning or unsupervised learning. Point-based measurement devices able
to measure Raman scattering have recently been translated into commercial
products [71] with support from supervised classification [72] or usupervised
dimensionality reduction [73]. Addressing the lack of wide-field information
in point-based systems, the community has looked into modalities such as
hyperspectral imaging [74] with an increasing use of machine learning to solve
some of the intrinsic challenges of high-dimensional data. Indeed, while
bearing rich information, the raw 2D-space+wavelength+time data that
hyperspectral imaging produces is difficult to interpret for clinicians as it
generates a temporal flow of three-dimensional information which cannot be
simply displayed in an intuitive fashion. Innovative use of Invertible Neural
Networks in combination with model-driven simulation has been used to train
neural network based regressors which are capable of real-time operation and
can provide uncertainty estimates for oxygen saturation measurement from
hyperspectral data [75]. Unsupervised deep manifold embedding for
hyperspectral imaging was proposed in [76] and deep learning was used for
reconstruction from sparse hyperspectral data [77]. Intelligent imaging
concept with simulation- or model-based training are also being progressed
with other emerging biophotonics imaging modalities such as for super-
resolution in endomicroscopy [78, 79], and artefact suppression in
photoacoustic imaging [80].

###  IV-D Towards Prospectively Planned Intelligent Imaging

With the availability of training data, either via dedicated data collection
or synthetic generation, AI algorithms can be developed to analyze intra-
operative images in near real-time and supply contextual information to
improve decision making. Omitting applications to endoscopic video sources
which are discussed in depth in Section V, and focusing first on
interventional X-ray imaging, benefits of real-time machine learning range
from segmentation of tools [81, 82, 53], anatomical landmark detection [51,
52], anatomy localization [83], and denoising [84, 85], to surgical phase
recognition [81]. Corresponding developments can be found for ultrasound
imaging [86, 87, 88].

While the above lists of applications merely hints at the potential that AI-
based analysis of internventional images has to offer, there is an interesting
observation: The majority of _intelligent imaging_ algorithms, including all
aforementioned methods, try to provide richer information by automated
analysis of traditionally acquired images, with little or no knowledge of the
image acquisition workflow. This raises an interesting question: If it is
known what information is desired or desirable at any given point during
surgery, is it possible to prospectively acquire an image that is most
informative in that particular context? First steps in this direction have
recently been reported, exploiting ultrasound image formation to suppress
scatter [89] or beamforming a B-mode image [90, 91] together with producing
its segmentation [69]. Zaech et al. [92] use an AI-based algorithm to
recommend task-optimal and patient specific C-arm X-ray trajectories during
cone-beam CT of spinal fusion surgery, and similar ideas arise for ultrasound
transducer positioning [93].

The domain of real-time interventional image analysis is fairly untapped as of
yet but offers great opportunities for workflow analysis, surgical progress
monitoring including anticipation and adverse event detection, and supplying
rich information for human-in-the-loop decision making. Additionally, task-
aware and autonomous imaging modalities may benefit interventional imaging
already one step before the image is analyzed and may thus give rise to
disruptive technology and novel surgical approaches.

##  V Surgical and Endoscopic Vision

###  V-A Recognizing Endoscopic Activity

Standard endoscopic imaging is certainly the modality most closely relating to
natural images. It should therefore not be surprising that machine learning
tools for interventional images have developed most rapidly in this field. As
a proxy for the eyes of the surgeon inside the patient, the endoscopic camera
is the privileged source of digital information to understand the activities
performed during endoscopic procedures. Endoscopic videos usually capture most
of the activities performed within the patient. Recognizing and understanding
these activities is essential to develop novel assistance systems that are
reactive to the context, e. g., that can provide timely instructions to OR
staff, enforce safety checkpoints or log automatically relevant information
within the surgical report. Surgical activity recognition from endoscopic
videos is however a highly challenging task due to the variability existing
across patients, surgical treatments and surgical teams.

In the recent years, a large body of work has focused on recognizing the
surgical steps of a procedure directly from the videos [94, 95, 96, 97, 98,
99]. This has notably been the case in cholecystectomy, a common procedure
consisting in removing the gallbladder, which is frequently used in research
due to its high frequency of occurrence and well-standardized protocol [100].
There, the steps include for instance “calot triangle dissection, cystic duct
and artery clipping and cutting, gallbladder dissection and gallbladder
packaging”. Recognition of these steps allows for the automated understanding
of the progress of the surgery. To perform recognition, models of the
underlying workflow of the procedure are learnt from datasets of exemplary
videos, annotated manually with the different steps. In [97], the model
consists for example of a visual feature extractor relying on a deep neural
network that feeds a temporal recognition model, like a hierarchical hidden
Markov model or an LSTM model. Several types of procedures have been
successfully studied for step recognition besides cholecystectomy. Examples
are cataract surgery [95, 96] and laparoscopic sleeve gastrectomy [98]. As the
current recognition methods show very promising results and real-time
capabilities, they can potentially be directly embedded in the endoscopic
tower to deliver contextual support. Other interesting prediction tasks have
been tackled with success using deep learning methods. In [101, 102], the
remaining duration of the procedure is predicted in real-time using deep
recurrent models trained directly from video data. In [97, 103, 104], the
presence of the instruments in the surgical scene is automatically detected.
Additional applications include bleeding and smoke detection [105, 106], as
well as surgery type identification at the beginning of the procedure [107].

Beyond the recognition of the surgical steps indicating the progress of the
surgery and the recognition of events such as bleeding, many potential
applications, like safety monitoring and human-robot cooperation, require a
finer level of understanding of the surgical activities. Future research
therefore needs to demonstrate accurate recognition of the detailed
interactions between the tools and the anatomy. To have impact beyond a single
operating room, recognition methods will also need to scale up to different
types of surgeries, operating rooms and hospitals without requiring the manual
annotations of large datasets for each situation. Recent methods exploiting
non-annotated videos through self-supervision or weak-supervision [108, 109,
110, 111, 104] or exploiting synthetically generated surgeries [64] may prove
very useful to train the next generation of surgical recognition systems.

Figure 3: Endoscopic video (top), monocular depth estimate (middle), and
rendering of a photorealistic reconstruction (bottom). Results were achieved
using the self-supervised method described in [112].

###  V-B Understanding Image Semantics

Understanding the surgical scene from the endoscopic images is fundamental for
context-aware intelligent computer-aided assistance. During augmented reality
visualization, precise pixel-based segmentation of the tools is necessary for
handling occlusions and providing the user with the correct perception.
Implementing safety warnings, such as no-go zones, requires the detection of
the critical anatomy. When another imaging modality is used, its registration
to the endoscopic video may require the localization of anatomical landmarks
[113]. Similarly, implementing degrees of autonomy during robotic surgery
requires the localisation and recognition of the neighboring tools and
anatomy.

Recently, a large body of work has targeted the detection and segmentation of
surgical instruments [114]. Deep learning methods have been proposed for both
bounding box or articulated tool detection [115, 116, 117] and for pixel-based
tool segmentation [118, 119]. Their superiority has been confirmed on
laparoscopic and surgical microscopy datasets in two international challenges
organized in 2015 and 2017 at the MICCAI conferences [120, 121]. Still, the
datasets used for evaluation are limited in size and variability. They are far
from representing the diversity of surgical scenes, which can indeed be very
challenging due to the presence of occlusions, smoke, bleeding, specularity,
motion blur, and deformation. Furthermore the aforementioned approaches are
fully-supervised and therefore impose important burden on the collection of
representative training datasets. New approaches are needed that can
generalize easily to various types of procedures and be trained using weaker
information for training, such as image-level tool presence [104], point
annotation [122] or scribbles [123].

Far less work has addressed the much needed anatomy detection and
segmentation, certainly due to the lack of available public datasets. The
community is however putting large efforts in this direction, as illustrated
by the recent generation of the CaDIS dataset [124], which contains pixel-
level annotations for 36 semantic classes in cataract surgery videos. Progress
has also been achieved in specific areas, such as liver segmentation [125],
lesion detection and characterisation during gastroscopy [126] or polyp
detection during colonoscopy [127, 17]. Here again, deep learning is the
state-of-the art, as demonstrated for polyp detection in a challenge organized
at MICCAI 2015 [128]. Thanks to the real-time capabilities of deep learning
approaches, the intra-operative benefits of such systems already start to be
evaluated in randomised clinical trials [17].

###  V-C Reconstructing Anatomic Geometry

Endoscopy mimics the surgeon’s eyes within the body, but due to the monocular
construction of endoscopes it lacks one important visual cue: Depth. This
shortcoming has implications: It has recently been shown that the availability
of 3D anatomic geometry benefits several clinical tasks, including the
detection of critical anatomy such as polyps [129] and the registration of
pre-operative 3D data to endoscopy video to enable navigation [130]. In
addition, analyzing 3D representations of anatomy would allow for the
introduction of quantitative measurements, enabling the standardization of
clinical reporting across sites. Recovering anatomic 3D geometry, e. g. to
augment endoscopic video with depth cues or to provide dense 3D
reconstruction, has gained considerable traction and is now an emerging
discipline with developments often orthogonal to those for complementary tasks
e. g. segmentation. This is because deep learning-based algorithms are able to
exploit image-level features to provide dense depth estimates even from
monocular video, complementing traditional optical endoscopy with depth
sensing as ”pseudo modality”. However, training depth estimation algorithms on
endoscopic sequences is complicated in practice because no paired depth
measurements exist naturally. While paired data can be generated in silico via
simulation from CT [65, 68, 131], the resulting trained models will need to
overcome the domain mismatch to real clinical data with methods similar to
that presented in Section IV. Recently, self-supervised training paradigms
that rely on traditional multi-view stereo approaches have received increasing
attention as they can be trained directly and solely from endoscopic video.
Multi-view stereo algorithms including structure from motion [130, 112] and
simultaneous localization and mapping [132] can be adapted to work with
endoscopic video, but they cannot provide dense 3D reconstructions due to the
lack of photometric constancy in endoscopic video and texture scarceness that
complicate feature matching across frames. These algorithms do, however,
provide a few reconstructed 3D points, and more importantly, relative camera
poses that can be used to supervise monocular depth estimation [132, 112]. A
representative photorealistic reconstruction achieved using a structure from
motion supervised depth estimation method is shown in Figure 3. These methods
achieve state-of-the-art performance with good generalization ability,
however, the resulting reconstructions are only up to scale. Among the biggest
premises of video-based reconstruction is the possibility of monitoring
anatomical change during surgery. This would require methods to robustly
handle various sorts of uncontrollable variation, including bleeding, smoke,
or tool presence. Solutions to these problems are currently unknown. Even in
more controlled scenarios, widespread adoption of learning-based
reconstruction from endoscopic video is hindered by the lack of publicly
available datasets, making it unclear how well today’s algorithms perform on
clinical data. This challenge is further aggravated by the lack of direct
evaluation targets: When applied to real clinical data, current reconstruction
or dense estimation algorithms can only be evaluated via surrogate tasks, such
as video-CT registration [112, 133] or polyp classification [129].

##  VI Clinical Workflow Monitoring and Support

Figure 4: Capturing the 3D context of the operating room is necessary for
providing AI-based decision support and monitoring risk. In this example, the
staff radiation exposure during a X-ray based procedure is computed in-situ
via simulation and displayed with augmented reality in a training scenario
[134].

###  VI-A The Notion of Surgical Control Tower

While imaging alone provides valuable information, modern procedures rely
increasingly on a variety of complex devices and intricate workflows. This
limits the knowledge extraction that AI systems can do based on imaging alone,
and makes it difficult for humans to properly analyse in real-time the wealth
of available data. Furthermore, even though the quality of care has generally
improved with the introduction of new surgical techniques and devices, adverse
events still occur, a large part of which are preventable [135, 136]. Humans
are prone to fatigue, teams to miscommunications, devices can fail, and for
all roles, surgical tasks require an ever increasing level of specialization.
The increased use of digital equipment in the OR however opens up new
opportunities for support and monitoring, at the level of the whole room, by
providing artificial intelligence systems with real-time data that capture a
faithful representation of the processes taking place during the surgery.
Indeed, most of the activities happening in the room can be captured digitally
either through interactions with equipment, such as information systems, room
control interfaces, imaging devices and instruments, or though the use of
sensors, such as ceiling-mounted cameras, which are now becoming widespread
and increasingly used for documentation, teaching and augmented reality
assistance. Consequently, it is highly likely that in the near future
assistance systems will be fully integrated in a digital OR that will monitor
surgical processes though AI, akin to a _surgical control tower_ [137, 138],
that can analyze the whole digital information in real-time to provide
context-aware support and information within and outside the OR. Applications
for such a control tower are for instance the transmission of live information
about the OR status, the adaptation of user-interfaces to the surrounding
context, the display of instructions within the OR, the creation of an
automated report, the recording of the activities for archiving and legal
purposes, the enforcement of safety checklists, the detection of anomalies
with respect to past workflows, and improved scheduling for staff and
patients. To perform these tasks, the control tower will have access to and
crunch masses of multi-modal digital data coming from hundreds of past
surgeries.

###  VI-B An Endeavour Rooted in Surgical Data Science

An essential component of the control tower is the data-driven modeling and
understanding of the clinical activities, an undertaking that taps into the
emerging research field of Surgical Data Science [3, 4]. Machine learning has
been key to generate models of procedural interventions from data [139, 140]
and ontologies have also been developed to standardize the resulting models
[141]. Implementations of such AI-based applications start to emerge in
various institutions, besides the ones focusing on analysing endoscopic videos
already mentioned in Section V. As video data remains one of the main source
of information, they highly rely on deep learning. Videos captured by cameras
mounted in the room provide indeed a rich source of information about the
activities without disrupting the workflow. For instance, a patient and staff
radiation exposure monitoring system for hybrid procedures illustrated in
Figure 4 was proposed in [134]. It relies on several RGB-D cameras to estimate
the 3D pose of the persons and room layout, which can then be used to simulate
and visualize in situ X-ray propagation around the patient table. [142]
develops a system to monitor hand-hygiene in hospital corridors in order to
analyse and reduce hospital acquired infection. The approach uses a large set
of depth cameras installed to observe the hand-soap dispensers. For the
intensive care unit, [143, 144] present methods based on color or depth video
data for the detection of patient mobilization activities. Key building blocks
to the success of these applications are the estimation of clinician and staff
poses [145, 146, 147], as well as the recognition of their activities [148,
149, 150, 151]. As for traditional visual data, deep learning based approaches
are currently the best performing methods for these tasks, though it should be
noted that they do not necessarily perform as well on clinical data yet. This
is due to the specificity of clinical videos, where staff wear gowns and
masks, colors are often similar, and cameras observe the room from restricted
positions, but also from the fact that there is no clinical COCO or Imagenet
dataset yet. [152] evaluates state-of-the-art human pose estimation approaches
and [153] state-of-the-art face detection approaches on clinical data. Both
studies show a large margin for improvement. Since the development of large
annotated datasets of clinical videos may be difficult due to the expertise
required and the restrictions on data, other approaches need to be developed,
for instance using non annotated data for transfer learning [153].

This will also help deploy the surgical control tower in new clinical
environments, as the variability in room layout, camera configuration and
workflow can be high. Retraining the assistance systems using only non-
annotated data from the novel environment or a tiny subset of annotated data
will be crucial for the adoption of these technologies. As even the collection
of non-annotated video data can be challenging due to data and privacy
regulations, it may also be required to implemented federated learning
approaches or develop methods able to cope with privacy-preserving data, such
as depth-only videos [142] or even low-resolution depth videos [154]. In
[154], it is shown that 2D human pose estimation can be achieved with
reasonably high accuracy on depth images downsampled by ten to the resolution
64x48. By using other information, such as system events [155] or speech
analysis [156], the analysis of clinical activities will be further improved.

##  VII Discussion and Conclusion

While AI is starting to impact CAI, as described in this paper, there are a
number of challenges that are specific to surgery and intervention to overcome
to deliver clinical impact. Leveraging context within learning paradigms will
be crucial to address those in a clinically meaningful way. The emerging field
of CAI4CAI offers researchers a large set of open problems to tackle. These
notably stem from the heterogeneity of surgical procedures and their
particular requirements for intra-operative imaging [157]; the difficulties in
data acquisition; the complexity in modeling and inferring decision making
processes; and the intricacy of the execution of surgical tasks. Over the
years, the CAI community has defined increasingly powerful Surgical Process
Models [158] to gain actionable understanding of surgical procedures while
describing interventions as a sequence of tasks and activities at different
granularity levels. At the finest level, mapping what should be the _Language
of Surgery_ [159], researchers currently break down surgical gestures into
semantically relevant motion units called _surgemes_ that are further composed
of sequences of motion primitives named _dexemes_ [160, 161, 162]. Yet, this
taxonomy mostly focused on the surgical action and in particular on surgical
tool manipulation and could thus rather be considered as mapping the _Language
of Surgical Dexterity_. This is already a laudable achievement and led to
scientists and engineers being able to, e. g., quantify the success of a
training program for executing different surgical actions [163, 164]. As
suggested by the study conducted by Birkmeyer et al. for bariatric surgery
[165], surgical skills can be highly correlated to surgical outcome for
certain procedures. AI systems have been shown capable of evaluating technical
skills using data from either training scenarios [166] or real procedures
[167]. Yet, by severely under-utilising the rich information contained in
other data sources, the _Language of Surgical Dexterity_ is still not
capturing the most complex aspects of surgical decision making. To address the
need to capture, understand and support all the cognitive interactions and
processes taking place in the operating room, the Surgical Data Science
community will need to drive the deployment of real-time multi-modal data
acquisition systems that will be used routinely. At the same time, it will
foster the development of new standards and regulations aiming at increasing
the interoperability of data, devices and models. This will directly benefit
CAI4CAI by simplifying the implementation and training of learning algorithms
involving databases from multiple institutions while maintaining privacy, e.g.
through federated learning. CAI4CAI in combination with Surgical Data Science
and Surgical Process Modelling could thus aim at defining and understanding
the ultimate _Language of Surgery_ based on a large number of heterogeneous
data sources used continuously by surgeons and interventional teams to
guarantee the best outcomes for a given procedure. As the field blossoms,
CAI4CAI researchers will address some of the most rewarding questions in
computer assisted intervention. Could CAI4CAI allow us to learn how decisions
are made, or missed, throughout surgical procedures? Could CAI4CAI support
such decision makings? Instead of going through the traditional path of
segmentation, registration, navigation and visualization, could contextual
machine learning allow us to optimize these steps for each given objective and
allow for real-time computation and feedback based on large amount of
heterogeneous data including pre- and intra-operative imaging, patient
characteristics and surgeon preferences?

With more capable and flexible learning paradigms, synergistic collaboration
is expected to happen between humans and AI-powered actors. The field is
already seeing exciting attempts to bring the user and the user experience at
the centre of our research questions. For example, novel spatially-aware
visualisation beyond traditional user interfaces is explored in [134, 168].
The challenge of improving human situational awareness in operating rooms with
solutions beyond visualisation is addressed in [169] with the use of context-
specific soundtracks. Introduction of novel multimodal interaction paradigms
and technologies within operating rooms will require extensive use of machine
learning to optimize the user interfaces and to provide maximally relevant
information and support, while preventing inattentional blindness [170]. By
developing systems able to learn from previous surgeries performed by experts
how to provide context-aware support and instructions directly in the OR, in
the manner of a virtual coach as in [171], AI could have a strong impact in
improving patient care. This is another aspect of CAI4CAI which needs
particular focus from the scientific community and requires multidisciplinary
teams including clinicians, user experience experts and machine learning
scientists to work together and come up with intelligent end-to-end CAI
solutions.

Finally, in this paper we did not have a particular focus on robotics. However
both surgical robotics and robotic imaging will play increasingly crucial
roles in the years to come. Machine learning is demonstrating convincing
results in real-time tool tracking [172, 173, 174, 118]. This for example
enables automatic positioning of intra-operative OCT imaging planes within
surgical microscopy for ophthalmic surgery [119, 175]. Integration of robotics
within surgical suites would require them to act intelligently,
synergistically with the human team and to be fully context-aware at all
moments. The wish to have real-time multi-modal imaging requires full
intelligence and automation. It also requires direct communication and
collaboration between surgical robots, imaging robots, surgeons and surgical
teams. CAI4CAI will have the challenge of enabling such ultimate intelligence,
which requires many years of research and development in many disciplines
while remembering past experience with the first generation of context-aware
computing [176]. Not only does CAI4CAI offer numerous exciting research
directions but it also promises to revolutionize surgery and therefore the
future of healthcare at a global scale.

## Acknowledgment

Tom Vercauteren is supported by a Medtronic / Royal Academy of Engineering
Research Chair [RCSRF1819\7\34], by the Wellcome Trust [203148/Z/16/Z;
WT101957] and by the Engineering and Physical Sciences Research Council
(EPSRC) [NS/A000049/1; NS/A000027/1]. Mathias Unberath is supported by an
NIH/NIBIB R01 EB0223939, Johns Hopkins University internal funding sources,
and a Fellowship of the Malone Center for Engineering in Healthcare at Johns
Hopkins University. Nicolas Padoy is supported by French state funds managed
by the ANR within the Investissements d’Avenir program under references
ANR-16-CE33-0009 (DeepSurg), ANR-18-CE45-0011-03 (OptimiX) and
ANR-11-LABX-0004 (Labex CAMI) and by BPI France (project CONDOR).

## References

  * [1] A. L. Simpson, M. Antonelli, S. Bakas, M. Bilello, K. Farahani, B. van Ginneken, A. Kopp-Schneider, B. A. Landman, G. Litjens _et al._ , “A large annotated medical image dataset for the development and evaluation of segmentation algorithms,” arXiv:1902.09063, 2019.
  * [2] E. Gibson, W. Li, C. Sudre, L. Fidon, D. I. Shakir, G. Wang, Z. Eaton-Rosen, R. Gray, T. Doel _et al._ , “NiftyNet: a deep-learning platform for medical imaging,” _Comput. Methods Programs Biomed._ , vol. 158, pp. 113–122, 2018.
  * [3] L. Maier-Hein, S. S. Vedula, S. Speidel, N. Navab, R. Kikinis, A. Park, M. Eisenmann, H. Feussner, G. Forestier _et al._ , “Surgical data science for next-generation interventions,” _Nat. Biomed. Eng._ , vol. 1, no. 9, pp. 691–696, 2017.
  * [4] S. S. Vedula and G. D. Hager, “Surgical data science: The new knowledge domain,” _Innovative Surgical Sciences_ , vol. 2, no. 3, pp. 109–121, 2007\.
  * [5] L. Maier-Hein, M. Eisenmann, C. Feldmann, H. Feussner, G. Forestier, S. Giannarou, B. Gibaud, G. D. Hager, M. Hashizume _et al._ , “Surgical data science: A consensus perspective,” arXiv:1806.03184, 2018.
  * [6] T. Peters and K. Cleary, _Image-guided interventions: technology and applications_. Springer Science & Business Media, 2008.
  * [7] J. West, J. M. Fitzpatrick, M. Y. Wang, B. M. Dawant, C. R. Maurer Jr, R. M. Kessler, R. J. Maciunas, C. Barillot, D. Lemoine _et al._ , “Comparison and evaluation of retrospective intermodality brain image registration techniques,” _J. Comput. Assist. Tomogr._ , vol. 21, no. 4, pp. 554–568, 1997.
  * [8] G. Wang, M. A. Zuluaga, W. Li, R. Pratt, P. A. Patel, M. Aertsen, T. Doel, A. L. David, J. Deprest _et al._ , “DeepIGeoS: A deep interactive geodesic framework for medical image segmentation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 41, no. 7, pp. 1559–1572, Jul. 2019.
  * [9] J. E. Iglesias and M. R. Sabuncu, “Multi-atlas segmentation of biomedical images: A survey,” _Med. Image Anal._ , vol. 24, no. 1, pp. 205–219, 2015\.
  * [10] D. Comaniciu, K. Engel, B. Georgescu, and T. Mansi, “Shaping the future through innovations: From medical imaging to precision medicine,” _Med. Image Anal._ , vol. 33, pp. 19–26, 2016.
  * [11] W. Li, G. Wang, L. Fidon, S. Ourselin, M. J. Cardoso, and T. Vercauteren, “On the compactness, efficiency, and representation of 3D convolutional networks: Brain parcellation as a pretext task,” in _Proc. IPMI’17_ , ser. LNCS, vol. 10265. Springer-Verlag, Jun. 2017, pp. 348–360.
  * [12] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” _Med. Image Anal._ , vol. 42, pp. 60–88, 2017.
  * [13] S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, R. T. Shinohara, C. Berger, S. M. Ha _et al._ , “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BraTS challenge,” arXiv:1811.02629, 2018.
  * [14] M. J. Sheller, G. A. Reina, B. Edwards, J. Martin, and S. Bakas, “Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation,” in _Proc. BrainLes Challenge of MICCAI’18_ , 2018, pp. 92–104.
  * [15] A. G. Roy, S. Siddiqui, S. Pölsterl, N. Navab, and C. Wachinger, “Braintorrent: A peer-to-peer environment for decentralized federated learning,” arXiv:1905.06731, 2019.
  * [16] K. Kamnitsas, C. Baumgartner, C. Ledig, V. Newcombe, J. Simpson, A. Kane, D. Menon, A. Nori, A. Criminisi _et al._ , “Unsupervised domain adaptation in brain lesion segmentation with adversarial networks,” in _Proc. IPMI’17_ , 2017, pp. 597–609.
  * [17] P. Wang, T. M. Berzin, J. R. Glissen Brown, S. Bharadwaj, A. Becq, X. Xiao, P. Liu, L. Li, Y. Song _et al._ , “Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: A prospective randomised controlled study,” _Gut_ , 2019, epub ahead of print.
  * [18] G. Wang, W. Li, M. A. Zuluaga, R. Pratt, P. Patel, M. Aertsen, T. Doel, A. L. David, J. Deprest _et al._ , “Interactive medical image segmentation using deep learning with image-specific fine-tuning,” _IEEE Trans. Med. Imag._ , vol. 37, no. 7, pp. 1562–1573, Jul. 2018.
  * [19] R. Dorent, W. Li, J. Ekanayake, S. Ourselin, and T. Vercauteren, “Learning joint lesion and tissue segmentation from task-specific hetero-modal datasets,” in _Proc. MIDL’19_ , ser. Proc. Mach. Learn. Res., M. J. Cardoso, A. Feragen, B. Glocker, E. Konukoglu, I. Oguz, G. Unal, and T. Vercauteren, Eds., vol. 102, 2019, pp. 164–174.
  * [20] R. Dorent, S. Joutard, M. Modat, S. Ourselin, and T. Vercauteren, “Hetero-modal variational encoder-decoder for joint modality completion and segmentation,” in _Proc. MICCAI’19_ , 2019.
  * [21] A. M. Digioia III, B. Jaramaz, C. Nikou, R. S. Labarca, J. E. Moody, and B. D. Colgan, “Surgical navigation for total hip replacement with the use of hipnav,” _Oper. Tech. Orthop._ , vol. 10, no. 1, pp. 3–8, 2000.
  * [22] M. Boudissa, A. Courvoisier, M. Chabanas, and J. Tonetti, “Computer assisted surgery in preoperative planning of acetabular fracture surgery: state of the art,” _Expert Rev. Med. Devices_ , vol. 15, no. 1, pp. 81–89, 2018.
  * [23] P. Kulyk, L. Vlachopoulos, P. Fürnstahl, and G. Zheng, “Fully automatic planning of total shoulder arthroplasty without segmentation: A deep learning based approach,” in _Computational Methods and Clinical Applications in Musculoskeletal Imaging_ , T. Vrtovec, J. Yao, G. Zheng, and J. M. Pozo, Eds. Springer International Publishing, 2019, pp. 22–34.
  * [24] F. Carrillo, L. Vlachopoulos, A. Schweizer, L. Nagy, J. Snedeker, and P. Fürnstahl, “A time saver: Optimization approach for the fully automatic 3D planning of forearm osteotomies,” in _Proc. MICCAI’17_ , M. Descoteaux, L. Maier-Hein, A. Franz, P. Jannin, D. L. Collins, and S. Duchesne, Eds. Springer International Publishing, 2017, pp. 488–496.
  * [25] R. Sparks, G. Zombori, R. Rodionov, M. Nowell, S. B. Vos, M. A. Zuluaga, B. Diehl, T. Wehner, A. Miserocchi _et al._ , “Automated multiple trajectory planning algorithm for the placement of stereo-electroencephalography (SEEG) electrodes in epilepsy treatment,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 12, no. 1, pp. 123–136, Jan. 2017.
  * [26] V. N. Vakharia, R. Sparks, K. Li, A. G. O’Keeffe, A. Miserocchi, A. W. McEvoy, M. R. Sperling, A. Sharan, S. Ourselin _et al._ , “Automated trajectory planning for laser interstitial thermal therapy in mesial temporal lobe epilepsy,” _Epilepsia_ , vol. 59, no. 4, pp. 814–824, 2018.
  * [27] A. Granados, M. Mancini, S. B. Vos, O. Lucena, V. Vakharia, R. Rodionov, A. Miserocchi, A. W. McEvoy, J. S. Duncan _et al._ , “A machine learning approach to predict instrument bending in stereotactic neurosurgery,” in _Proc. MICCAI’18_ , A. F. Frangi, J. A. Schnabel, C. Davatzikos, C. Alberola-López, and G. Fichtinger, Eds. Cham: Springer International Publishing, 2018, pp. 238–246.
  * [28] S. Giffard-Roisin, H. Delingette, T. Jackson, J. Webb, L. Fovargue, J. Lee, C. A. Rinaldi, R. Razavi, N. Ayache, and M. Sermesant, “Transfer learning from simulations on a reference anatomy for ECGI in personalized cardiac resynchronization therapy,” _IEEE Trans. Biomed. Eng._ , vol. 66, no. 2, pp. 343–353, 2018.
  * [29] J. Ramalhinho, M. R. Robu, S. Thompson, K. Gurusamy, B. Davidson, D. Hawkes, D. Barratt, and M. J. Clarkson, “A pre-operative planning framework for global registration of laparoscopic ultrasound to CT images,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 13, no. 8, pp. 1177–1186, Aug. 2018\.
  * [30] S. F. Johnsen, S. Thompson, M. J. Clarkson, M. Modat, Y. Song, J. Totz, K. Gurusamy, B. Davidson, Z. A. Taylor _et al._ , “Database-based estimation of liver deformation under pneumoperitoneum for surgical image-guidance and simulation,” in _Proc. MICCAI’15_ , N. Navab, J. Hornegger, W. M. Wells, and A. Frangi, Eds. Springer International Publishing, 2015, pp. 450–458.
  * [31] K. März, M. Hafezi, T. Weller, A. Saffari, M. Nolden, N. Fard, A. Majlesara, S. Zelzer, M. Maleshkova _et al._ , “Toward knowledge-based liver surgery: holistic information processing for surgical decision support,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 10, no. 6, pp. 749–759, Jun. 2015.
  * [32] T. Koivukangas, J. P. Katisko, and J. P. Koivukangas, “Technical accuracy of optical and the electromagnetic tracking systems,” _SpringerPlus_ , vol. 2, no. 1, p. 90, 2013.
  * [33] F. A. Jolesz, _Intraoperative imaging and image-guided therapy_. Springer Science & Business Media, 2014.
  * [34] J. P. Pluim and J. M. Fitzpatrick, “Image registration,” _IEEE Trans. Med. Imag._ , vol. 22, no. 11, pp. 1341–1343, 2003.
  * [35] R. Liao, L. Zhang, Y. Sun, S. Miao, and C. Chefd’Hotel, “A review of recent advances in registration techniques applied to minimally invasive therapy,” _IEEE Trans. Multimedia_ , vol. 15, no. 5, pp. 983–1000, 2013.
  * [36] L. Joskowicz and E. J. Hazan, “Computer aided orthopaedic surgery: incremental shift or paradigm change?” 2016.
  * [37] B. Fuerst, W. Wein, M. Müller, and N. Navab, “Automatic ultrasound–MRI registration for neurosurgery using the 2D and 3D LC2 metric,” _Med. Image Anal._ , vol. 18, no. 8, pp. 1312–1319, 2014.
  * [38] W. Wein, “Brain-shift correction with image-based registration and landmark accuracy evaluation,” in _Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation_. Springer International Publishing, 2018, pp. 146–151.
  * [39] B. D. de Vos, F. F. Berendsen, M. A. Viergever, H. Sokooti, M. Staring, and I. Išgum, “A deep learning framework for unsupervised affine and deformable image registration,” _Med. Image Anal._ , vol. 52, pp. 128–143, 2019.
  * [40] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. V. Dalca, “VoxelMorph: a learning framework for deformable medical image registration,” _IEEE Trans. Med. Imag._ , 2019.
  * [41] Y. Hu, M. Modat, E. Gibson, W. Li, N. Ghavami, E. Bonmati, G. Wang, S. Bandula, C. M. Moore _et al._ , “Weakly-supervised convolutional neural networks for multimodal image registration,” _Med. Image Anal._ , vol. 49, pp. 1–13, Oct. 2018.
  * [42] Y. Hu, E. Gibson, N. Ghavami, E. Bonmati, C. M. Moore, M. Emberton, T. Vercauteren, J. A. Noble, and D. C. Barratt, “Adversarial deformation regularization for training image registration neural networks,” in _Proc. MICCAI’18_ , ser. LNCS, vol. 11070, 2018, pp. 774–782.
  * [43] M. Pfeiffer, C. Riediger, J. Weitz, and S. Speidel, “Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 14, no. 7, pp. 1147–1155, Jul. 2019.
  * [44] J.-N. Brunet, A. Mendizabal, A. Petit, N. Golse, E. Vibert, and S. Cotin, “Physics-based deep neural network for augmented reality during liver surgery,” in _Proc. MICCAI’19_ , ser. LNCS, 2019.
  * [45] Y. Hu, E. Gibson, D. C. Barratt, M. Emberton, J. A. Noble, and T. Vercauteren, “Conditional segmentation in lieu of image registration,” in _Proc. MICCAI’19_ , 2019.
  * [46] J. Esteban, M. Grimm, M. Unberath, G. Zahnd, and N. Navab, “Towards fully automatic X-ray to CT registration,” in _Proc. MICCAI’19_. Springer, 2019.
  * [47] B. Hou, A. Alansary, S. McDonagh, A. Davidson, M. Rutherford, J. V. Hajnal, D. Rueckert, B. Glocker, and B. Kainz, “Predicting slice-to-volume transformation in presence of arbitrary subject motion,” in _Proc. MICCAI’17_. Springer, 2017, pp. 296–304.
  * [48] M. Bui, S. Albarqouni, M. Schrapp, N. Navab, and S. Ilic, “X-Ray PoseNet: 6 DoF pose estimation for mobile X-Ray devices,” in _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2017, pp. 1036–1044.
  * [49] S. Miao, Z. J. Wang, and R. Liao, “A CNN regression approach for real-time 2D/3D registration,” _IEEE Trans. Med. Imag._ , vol. 35, no. 5, pp. 1352–1363, 2016.
  * [50] R. Prevost, M. Salehi, S. Jagoda, N. Kumar, J. Sprung, A. Ladikos, R. Bauer, O. Zettinig, and W. Wein, “3D freehand ultrasound without external tracking using deep learning,” _Med. Image Anal._ , 2018.
  * [51] B. Bier, M. Unberath, J.-N. Zaech, J. Fotouhi, M. Armand, G. Osgood, N. Navab, and A. Maier, “X-ray-transform invariant anatomical landmark detection for pelvic trauma surgery,” in _Proc. MICCAI’18_. Springer, 2018, pp. 55–63.
  * [52] B. Bier, K. Aschoff, C. Syben, M. Unberath, M. Levenston, G. Gold, R. Fahrig, and A. Maier, “Detecting anatomical landmarks for motion estimation in weight-bearing imaging of knees,” in _International Workshop on Machine Learning for Medical Image Reconstruction_. Springer, 2018, pp. 83–90.
  * [53] C. Gao, M. Unberath, R. Taylor, and M. Armand, “Localizing dexterous surgical tools in x-ray for image-based navigation,” in _Proc. IPCAI’19_. Springer, 2019, pp. 1–4.
  * [54] D. Kügler, A. Stefanov, and A. Mukhopadhyay, “i3PosNet: Instrument pose estimation from X-ray,” arXiv:1802.09575, 2018.
  * [55] H. Esfandiari, R. Newell, C. Anglin, J. Street, and A. J. Hodgson, “A deep learning framework for segmentation and pose estimation of pedicle screw implants based on C-arm fluoroscopy,” _Int. J. Comput. Assisted Radiol. Surg._ , pp. 1–14, 2018.
  * [56] H. Liao, W.-A. Lin, J. Zhang, J. Zhang, J. Luo, and S. K. Zhou, “Multiview 2d/3d rigid registration via a point-of-interest network for tracking and triangulation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 638–12 647.
  * [57] M. Unberath, J.-N. Zaech, C. Gao, B. Bier, F. Goldmann, S. C. Lee, J. Fotouhi, R. Taylor, M. Armand, and N. Navab, “Enabling machine learning in x-ray-based procedures via realistic simulation of image formation,” _Int. J. Comput. Assisted Radiol. Surg._ , pp. 1–12, 2019.
  * [58] D. A. Clunie, D. K. Dennison, D. Cram, K. R. Persons, M. D. Bronkalla _et al._ , “Technical challenges of enterprise imaging: HIMSS-SIIM collaborative white paper,” _J. Digit. Imaging_ , vol. 29, no. 5, pp. 583–614, 2016.
  * [59] F. M. Murad, S. Banerjee, B. A. Barth, Y. M. Bhat, S. S. Chauhan, K. T. Gottlieb, V. Konda, J. T. Maple, P. Pfau _et al._ , “Image management systems,” _Gastrointest. Endosc._ , vol. 79, no. 1, pp. 15–22, 2014.
  * [60] A. Deganello, M. E. Sellars, G. T. Yusuf, and P. S. Sidhu, “How much should I record during a CEUS examination? Practical aspects of the ”real-time” feature of a contrast ultrasound study,” _Ultraschall Med._ , vol. 39, no. 05, pp. 484–486, 2018.
  * [61] F. Milletari, V. Birodkar, and M. Sofka, “Straight to the point: reinforcement learning for user guidance in ultrasound,” arXiv:1903.00586, 2019.
  * [62] Y. Hu, E. Gibson, L.-L. Lee, W. Xie, D. C. Barratt, T. Vercauteren, and J. A. Noble, “Freehand ultrasound image simulation with spatially-conditioned generative adversarial networks,” in _Proc. RAMBO Workshop of MICCAI’17_ , ser. LNCS, vol. 10555. Springer-Verlag, Sep. 2017, pp. 105–115.
  * [63] S. Engelhardt, R. De Simone, P. M. Full, M. Karck, and I. Wolf, “Improving surgical training phantoms by hyperrealism: Deep unpaired image-to-image translation from real surgeries,” in _Proc. MICCAI’18_ , 2018, pp. 747–755.
  * [64] I. Luengo, E. Flouty, P. Giataganas, P. Wisanuvej, J. Nehme, and D. Stoyanov, “SurReal: Enhancing surgical simulation realism using style transfer,” in _British Machine Vision Conference 2018, BMVC 2018, Northumbria University, Newcastle, UK, September 3-6, 2018_ , 2018, p. 116.
  * [65] F. Mahmood, R. Chen, and N. J. Durr, “Unsupervised reverse domain adaptation for synthetic medical images via adversarial training,” _IEEE Trans. Med. Imag._ , vol. 37, no. 12, pp. 2572–2581, 2018.
  * [66] Y. Zhang, S. Miao, T. Mansi, and R. Liao, “Task driven generative modeling for unsupervised domain adaptation: Application to x-ray image segmentation,” in _Proc. MICCAI’18_. Springer, 2018\.
  * [67] M. Unberath, J.-N. Zaech, S. C. Lee, B. Bier, J. Fotouhi, M. Armand, and N. Navab, “DeepDRR–a catalyst for machine learning in fluoroscopy-guided procedures,” in _Proc. MICCAI’18_. Springer, 2018.
  * [68] F. Mahmood, R. Chen, S. Sudarsky, D. Yu, and N. J. Durr, “Deep learning with cinematic rendering-fine-tuning deep neural networks using photorealistic medical images,” arXiv:1805.08400, 2018.
  * [69] A. A. Nair, T. D. Tran, A. Reiter, and M. A. L. Bell, “A deep learning based alternative to beamforming ultrasound images,” in _Proc. ICASSP’18_. IEEE, 2018, pp. 3359–3363.
  * [70] D. Toth, S. Cimen, P. Ceccaldi, T. Kurzendorfer, K. Rhode, and P. Mountney, “Training deep networks on domain randomized synthetic x-ray data for cardiac interventions,” in _Proc. MIDL’19_ , ser. Proc. Mach. Learn. Res., M. J. Cardoso, A. Feragen, B. Glocker, E. Konukoglu, I. Oguz, G. Unal, and T. Vercauteren, Eds., vol. 102, 2019, pp. 468–482.
  * [71] M. Jermyn, K. Mok, J. Mercier, J. Desroches, J. Pichette, K. Saint-Arnaud, L. Bernstein, M.-C. Guiot, K. Petrecca, and F. Leblond, “Intraoperative brain cancer detection with Raman spectroscopy in humans,” _Science translational medicine_ , vol. 7, no. 274, pp. 274ra19–274ra19, 2015.
  * [72] M. Jermyn, J. Desroches, J. Mercier, M.-A. Tremblay, K. St-Arnaud, M.-C. Guiot, K. Petrecca, and F. Leblond, “Neural networks improve brain cancer detection with Raman spectroscopy in the presence of operating room light artifacts,” _J. Biomed. Opt._ , vol. 21, no. 9, p. 094002, 2016.
  * [73] C. Banbury, R. Mason, I. Styles, N. Eisenstein, M. Clancy, A. Belli, A. Logan, and P. G. Oppenheimer, “Development of the self optimising Kohonen Index Network (sKiNet) for Raman spectroscopy based detection of anatomical eye tissue,” _Sci. Rep._ , vol. 9, no. 1, p. 10812, 2019.
  * [74] J. Shapey, Y. Xie, E. Nabavi, R. Bradford, S. R. Saeed, S. Ourselin, and T. Vercauteren, “Intraoperative multispectral and hyperspectral label-free imaging: a systematic review of in vivo clinical studies,” _J. Biophotonics_ , p. e201800455, 2019, in press.
  * [75] T. J. Adler, L. Ardizzone, A. Vemuri, L. Ayala, J. Gröhl, T. Kirchner, S. Wirkert, J. Kruse, C. Rother _et al._ , “Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks,” _Int. J. Comput. Assisted Radiol. Surg._ , pp. 1–11, 2019.
  * [76] D. Ravì, H. Fabelo, G. M. Callic, and G.-Z. Yang, “Manifold embedding and semantic segmentation for intraoperative guidance with hyperspectral brain imaging,” _IEEE Trans. Med. Imag._ , vol. 36, no. 9, pp. 1845–1857, 2017\.
  * [77] J. Lin, N. T. Clancy, J. Qi, Y. Hu, T. Tatla, D. Stoyanov, L. Maier-Hein, and D. S. Elson, “Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks,” _Med. Image Anal._ , vol. 48, pp. 162–176, 2018.
  * [78] D. Ravì, A. B. Szczotka, , S. P. Pereira, and T. Vercauteren, “Adversarial training with cycle consistency for unsupervised super-resolution in endomicroscopy,” _Med. Image Anal._ , 2019, in press.
  * [79] A. B. Szczotka, D. Ravì, D. I. Shakir, S. P. Pereira, and T. Vercauteren, “Effective deep learning training for single-image super-resolution in endomicroscopy exploiting video-registration-based reconstruction,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 13, no. 6, pp. 917–924, Jun. 2018.
  * [80] D. Allman, F. Assis, J. Chrispin, and M. A. L. Bell, “Deep neural networks to remove photoacoustic reflection artifacts in ex vivo and in vivo tissue,” in _2018 IEEE International Ultrasonics Symposium (IUS)_. IEEE, 2018, pp. 1–4.
  * [81] P. Ambrosini, D. Ruijters, W. J. Niessen, A. Moelker, and T. van Walsum, “Fully automatic and real-time catheter segmentation in X-ray fluoroscopy,” in _Proc. MICCAI’17_. Springer, 2017, pp. 577–585.
  * [82] K. Breininger, T. Würfl, T. Kurzendorfer, S. Albarqouni, M. Pfister, M. Kowarschik, N. Navab, and A. Maier, “Multiple device segmentation for fluoroscopic imaging using multi-task learning,” in _Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis_. Springer, 2018, pp. 19–27.
  * [83] R. Sa, W. Owens, R. Wiegand, M. Studin, D. Capoferri, K. Barooha, A. Greaux, R. Rattray, A. Hutton _et al._ , “Intervertebral disc detection in X-ray images using faster R-CNN,” in _Engineering in Medicine and Biology Society (EMBC), 2017 39th Annual International Conference of the IEEE_. IEEE, 2017, pp. 564–567.
  * [84] Y. Matviychuk, B. Mailhé, X. Chen, Q. Wang, A. Kiraly, N. Strobel, and M. Nadar, “Learning a multiscale patch-based representation for image denoising in x-ray fluoroscopy,” in _2016 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2016, pp. 2330–2334.
  * [85] S. G. Hariharan, N. Strobel, C. Kaethner, M. Kowarschik, S. Albarqouni, R. Fahrig, and N. Navab, “Learning-based x-ray image denoising utilizing model-based image simulations,” in _Proc. MICCAI’19_. Springer, 2019.
  * [86] E. M. A. Anas, P. Mousavi, and P. Abolmaesumi, “A deep learning approach for real time prostate segmentation in freehand ultrasound guided biopsy,” _Med. Image Anal._ , 2018.
  * [87] C. Mwikirize, J. L. Nosher, and I. Hacihaliloglu, “Convolution neural networks for real-time needle detection and localization in 2D ultrasound,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 13, no. 5, pp. 647–657, 2018\.
  * [88] M. Pesteie, V. Lessoway, P. Abolmaesumi, and R. N. Rohling, “Automatic localization of the needle target for ultrasound-guided epidural injections,” _IEEE Trans. Med. Imag._ , vol. 37, no. 1, pp. 81–92, 2017\.
  * [89] A. C. Luchies and B. C. Byram, “Deep neural networks for ultrasound beamforming,” _IEEE Trans. Med. Imag._ , vol. 37, no. 9, pp. 2010–2021, 2018.
  * [90] W. Simson, R. Göbl, M. Paschali, M. Krönke, K. Scheidhauer, W. Weber, and N. Navab, “End-to-end learning-based ultrasound reconstruction,” arXiv:1904.04696, 2019.
  * [91] W. Simson, M. Paschali, N. Navab, and G. Zahnd, “Deep learning beamforming for sub-sampled ultrasound data,” in _2018 IEEE International Ultrasonics Symposium (IUS)_. IEEE, 2018, pp. 1–4.
  * [92] J.-N. Zaech, C. Gao, B. Bier, R. Taylor, A. Maier, N. Navab, and M. Unberath, “Learning to avoid poor images: Towards task-aware C-arm cone-beam CT trajectories,” in _Proc. MICCAI’19_. Springer, 2019.
  * [93] F. Milletari, N. Rieke, M. Baust, M. Esposito, and N. Navab, “CFCM: Segmentation via coarse to fine context memory,” arXiv:1806.01413, 2018.
  * [94] T. Blum, N. Padoy, H. Feußner, and N. Navab, “Modeling and online recognition of surgical phases using hidden markov models,” in _Proc. MICCAI’08_ , 2008, pp. 627–635.
  * [95] F. Lalys, L. Riffaud, D. Bouget, and P. Jannin, “A framework for the recognition of high-level surgical tasks from video images for cataract surgeries.” _IEEE Trans. Biomed. Eng._ , vol. 59, no. 4, pp. 966–976, 2012\.
  * [96] G. Quellec, M. Lamard, B. Cochener, and G. Cazuguel, “Real-time segmentation and recognition of surgical tasks in cataract surgery videos,” _IEEE Trans. Med. Imag._ , vol. 33, no. 12, pp. 2352–2360, Dec 2014.
  * [97] A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. de Mathelin, and N. Padoy, “EndoNet: A deep architecture for recognition tasks on laparoscopic videos,” _IEEE Trans. Med. Imag._ , vol. 36, no. 1, pp. 86–97, 2017.
  * [98] M. Volkov, D. A. Hashimoto, G. Rosman, O. R. Meireles, and D. Rus, “Machine learning and coresets for automated real-time video segmentation of laparoscopic and robot-assisted surgery,” in _Proc. ICRA’17_ , 2017, pp. 754–759.
  * [99] S. Bodenstedt, D. Rivoir, A. Jenke, M. Wagner, M. Breucha, B. P. Müller-Stich, S. T. Mees, J. Weitz, and S. Speidel, “Active learning using deep bayesian networks for surgical workflow analysis,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 14, no. 6, pp. 1079–1087, 2019.
  * [100] N. Padoy, T. Blum, S.-A. Ahmadi, H. Feussner, M.-O. Berger, and N. Navab, “Statistical modeling and recognition of surgical workflow,” _Med. Image Anal._ , vol. 16, no. 3, pp. 632–641, 2012.
  * [101] A. P. Twinanda, G. Yengera, D. Mutter, J. Marescaux, and N. Padoy, “RSDNet: Learning to predict remaining surgery duration from laparoscopic videos without manual annotations,” _IEEE Trans. Med. Imag._ , vol. (to appear), 2019.
  * [102] S. Bodenstedt, M. Wagner, L. Mündermann, H. Kenngott, B. P. Müller-Stich, M. Breucha, S. T. Mees, J. Weitz, and S. Speidel, “Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 14, no. 6, pp. 1089–1095, 2019.
  * [103] H. A. Hajj, M. Lamard, P. Conze, B. Cochener, and G. Quellec, “Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks,” _Med. Image Anal._ , vol. 47, pp. 203–218, 2018.
  * [104] C. I. Nwoye, D. Mutter, J. Marescaux, and N. Padoy, “Weakly supervised convolutional lstm approach for tool tracking in laparoscopic videos,” _Int. J. Comput. Assisted Radiol. Surg._ , 2019.
  * [105] A. Leibetseder, M. J. Primus, and K. Schoeffmann, “Automatic smoke classification in endoscopic video,” in _MultiMedia Modeling - 24th International Conference, MMM 2018, Bangkok, Thailand, February 5-7, 2018, Proceedings, Part II_ , 2018, pp. 362–366.
  * [106] T. Okamoto, T. Ohnishi, H. Kawahira, O. Dergachyava, P. Jannin, and H. Haneishi, “Real-time identification of blood regions for hemostasis support in laparoscopic surgery,” _Signal, Image and Video Processing_ , vol. 13, no. 2, pp. 405–412, 2019.
  * [107] S. Kannan, G. Yengera, D. Mutter, J. Marescaux, and N. Padoy, “Future-state predicting LSTM for early surgery type recognition,” _IEEE Trans. Med. Imag._ , 2019.
  * [108] I. Funke, A. Jenke, S. T. Mees, J. Weitz, S. Speidel, and S. Bodenstedt, “Temporal coherence-based self-supervised learning for laparoscopic workflow analysis,” in _OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, - and - Skin Image Analysis - First International Workshop, OR 2.0 2018, 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, Third International Workshop, ISIC 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16 and 20, 2018, Proceedings_ , 2018, pp. 85–93.
  * [109] G. Yengera, D. Mutter, J. Marescaux, and N. Padoy, “Less is more: Surgical phase recognition with less annotations through self-supervised pre-training of CNN-LSTM networks,” arXiv:1805.08569, 2018.
  * [110] T. Roß, D. Zimmerer, A. S. Vemuri, F. Isensee, M. Wiesenfarth, S. Bodenstedt, F. Both, P. Kessler, M. Wagner _et al._ , “Exploiting the potential of unlabeled endoscopic video data with self-supervised learning,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 13, no. 6, pp. 925–933, 2018\.
  * [111] T. Yu, D. Mutter, J. Marescaux, and N. Padoy, “Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition,” in _Proc. IPCAI 2019_ , 2019.
  * [112] X. Liu, A. Sinha, M. Unberath, M. Ishii, G. D. Hager, R. H. Taylor, and A. Reiter, “Self-supervised learning for dense depth estimation in monocular endoscopy,” in _OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis_. Springer, 2018, pp. 128–138.
  * [113] D. Ntourakis, R. Memeo, L. Soler, J. Marescaux, D. Mutter, and P. Pessaux, “Augmented reality guidance for the resection of missing colorectal liver metastases: An initial experience,” _World journal of surgery_ , vol. 40, 08 2015.
  * [114] D. Bouget, M. Allan, D. Stoyanov, and P. Jannin, “Vision-based and marker-less surgical tool detection and tracking: a review of the literatureeasy,” _Med. Image Anal._ , vol. 35, pp. 633–654, 2017.
  * [115] D. Sarikaya, J. J. Corso, and K. A. Guru, “Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection,” _IEEE Trans. Med. Imag._ , vol. 36, no. 7, pp. 1542–1549, 2017.
  * [116] T. Kurmann, P. Márquez-Neila, X. Du, P. Fua, D. Stoyanov, S. Wolf, and R. Sznitman, “Simultaneous recognition and pose estimation of instruments in minimally invasive surgery,” in _Proc. MICCAI’17_ , 2017, pp. 505–513.
  * [117] A. Jin, S. Yeung, J. Jopling, J. Krause, D. Azagury, A. Milstein, and L. Fei-Fei, “Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks,” in _2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, NV, USA, March 12-15, 2018_ , 2018, pp. 691–699.
  * [118] L. C. Garcia-Peraza-Herrera, W. Li, L. Fidon, C. Gruijthuijsen, A. Devreker, G. Attilakos, J. Deprest, E. Vander Poorten, D. Stoyanov _et al._ , “ToolNet: Holistically-nested real-time segmentation of robotic surgical tools,” in _Proc. IROS’17_ , 2017.
  * [119] I. Laina, N. Rieke, C. Rupprecht, J. P. Vizcaíno, A. Eslami, F. Tombari, and N. Navab, “Concurrent segmentation and localization for tracking of surgical instruments,” in _Proc. MICCAI’17_. Springer, 2017, pp. 664–672.
  * [120] S. Bodenstedt, M. Allan, A. Agustinos, X. Du, L. Garcia-Peraza-Herrera, H. Kenngott, T. Kurmann, B. Müller-Stich, S. Ourselin _et al._ , “Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery,” arXiv:1805.02475, 2018.
  * [121] M. Allan, A. Shvets, T. Kurmann, Z. Zhang, R. Duggal, Y. Su, N. Rieke, I. Laina, N. Kalavakonda _et al._ , “2017 robotic instrument segmentation challenge,” arXiv:1902.06426, 2019.
  * [122] L. Lejeune, J. Grossrieder, and R. Sznitman, “Iterative multi-path tracking for video and volume segmentation with sparse point supervision,” _Med. Image Anal._ , vol. 50, pp. 65–81, 2018.
  * [123] F. Fuentes-Hurtado, A. Kadkhodamohammadi, E. Flouty, S. Barbarisi, I. Luengo, and D. Stoyanov, “Easylabels: weak labels for scene segmentation in laparoscopic videos,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 14, no. 7, pp. 1247–1257, 2019.
  * [124] E. Flouty, A. Kadkhodamohammadi, I. Luengo, F. Fuentes-Hurtado, H. Taleb, S. Barbarisi, G. Quellec, and D. Stoyanov, “Cadis: Cataract dataset for image segmentation,” arXiv:1906.11586, 2019.
  * [125] E. Gibson, M. R. Robu, S. Thompson, P. E. Edwards, C. Schneider, K. Gurusamy, B. Davidson, D. J. Hawkes, D. C. Barratt, and M. J. Clarkson, “Deep residual networks for automatic segmentation of laparoscopic videos of the liver,” in _Medical Imaging 2017: Image-Guided Procedures, Robotic Interventions, and Modeling_ , vol. 10135. International Society for Optics and Photonics, 2017, p. 101351M.
  * [126] M. A. Everson, L. C. Garcia Peraza Herrera, W. Li, I. Muntion Luengo, O. Ahmad, M. R. Banks, C. Magee, D. Alzoubaidi, H. Hsu _et al._ , “Artificial intelligence for the real-time classification of intrapapillary capillary loop patterns in the endoscopic diagnosis of early oesophageal squamous cell carcinoma: A proof-of-concept study,” _United European Gastroenterology J._ , p. 2050640618821800, 2019.
  * [127] M. Misawa, S. ei Kudo, Y. Mori, T. Cho, S. Kataoka, A. Yamauchi, Y. Ogawa, Y. Maeda, K. Takeda _et al._ , “Artificial intelligence-assisted polyp detection for colonoscopy: Initial experience,” _Gastroenterology_ , vol. 154, no. 8, pp. 2027 – 2029.e3, 2018.
  * [128] J. Bernal, N. Tajkbaksh, F. J. Sánchez, B. J. Matuszewski, H. Chen, L. Yu, Q. Angermann, O. Romain, B. Rustad _et al._ , “Comparative validation of polyp detection methods in video colonoscopy: results from the MICCAI 2015 endoscopic vision challenge,” _IEEE Trans. Med. Imag._ , vol. 36, no. 6, pp. 1231–1249, 2017.
  * [129] F. Mahmood, Z. Yang, R. Chen, D. Borders, W. Xu, and N. J. Durr, “Polyp segmentation and classification using predicted depth from monocular endoscopy,” in _Medical Imaging 2019: Computer-Aided Diagnosis_ , vol. 10950\. International Society for Optics and Photonics, 2019, p. 1095011.
  * [130] S. Leonard, A. Sinha, A. Reiter, M. Ishii, G. L. Gallia, R. H. Taylor, and G. D. Hager, “Evaluation and stability analysis of video-based navigation system for functional endoscopic sinus surgery on in vivo clinical data,” _IEEE Trans. Med. Imag._ , vol. 37, no. 10, pp. 2185–2195, 2018.
  * [131] M. Visentini-Scarzanella, T. Sugiura, T. Kaneko, and S. Koto, “Deep monocular 3D reconstruction for assisted navigation in bronchoscopy,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 12, no. 7, pp. 1089–1099, 2017.
  * [132] J. Wang, S. Song, H. Ren, C. M. Lim, and M. Q.-H. Meng, “Surgical instrument tracking by multiple monocular modules and a sensor fusion approach,” _IEEE Trans. Autom. Sci. Eng._ , vol. 16, no. 2, pp. 629–639, 2018.
  * [133] X. Liu, A. Sinha, M. Ishii, G. D. Hager, A. Reiter, R. H. Taylor, and M. Unberath, “Self-supervised learning for dense depth estimation in monocular endoscopy,” _arXiv preprint arXiv:1902.07766_ , 2019.
  * [134] N. L. Rodas, F. Barrera, and N. Padoy, “See it with your own eyes: Markerless mobile augmented reality for radiation awareness in the hybrid room,” _IEEE Trans. Biomed. Eng._ , vol. 64, no. 2, pp. 429–440, 2017.
  * [135] J. T. James, “A new, evidence-based estimate of patient harms associated with hospital care,” _Journal of Patient Safety_ , vol. 9, no. 3, p. 122–128, 2013.
  * [136] J. W. Suliburk, Q. M. Buck, C. J. Pirko, N. N. Massarweh, N. R. Barshes, H. Singh, and T. K. Rosengart, “Analysis of Human Performance Deficiencies Associated With Surgical Adverse EventsAnalysis of Human Performance Deficiencies Associated With Surgical Adverse EventsAnalysis of Human Performance Deficiencies Associated With Surgical Adverse Events,” _JAMA Network Open_ , vol. 2, no. 7, pp. e198 067–e198 067, 07 2019.
  * [137] N. Padoy, “Vers une tour de contrôle des blocs opératoires ?” in _Santé et intelligence artificielle_ , B. N. Cédric Villani, Ed. CNRS Editions, 2018.
  * [138] ——, “Machine and deep learning for workflow recognition during surgery,” _Minimally Invasive Therapy & Allied Technologies (MITAT)_, 2019.
  * [139] P. Jannin and X. Morandi, “Surgical models for computer-assisted neurosurgery,” _Neuroimage_ , vol. 37, no. 3, pp. 783–791, 2007.
  * [140] T. Blum, N. Padoy, H. Feußner, and N. Navab, “Workflow mining for visualization and analysis of surgeries,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 3, no. 5, pp. 379–386, 2008.
  * [141] B. Gibaud, G. Forestier, C. Feldmann, G. Ferrigno, P. J. S. Gonçalves, T. Haidegger, C. Julliard, D. Katic, H. Kenngott _et al._ , “Toward a standard ontology of surgical process models,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 13, no. 9, pp. 1397–1408, 2018.
  * [142] A. Haque, M. Guo, A. Alahi, S. Yeung, Z. Luo, A. Rege, J. Jopling, N. L. Downing, W. Beninati _et al._ , “Towards vision-based smart hospitals: A system for tracking and monitoring hand hygiene compliance,” in _Machine Learning in Healthcare Conference (MLHC), Proceedings of Machine Learning Research_ , 2017.
  * [143] A. J. Ma, N. Rawat, A. Reiter, C. Shrock, A. Zhan, A. Stone, A. Rabiee, S. Griffin, D. M. Needham, and S. Saria, “Measuring patient mobility in the ICU using a novel noninvasive sensor,” _Critical care medicine_ , vol. 45, no. 4, pp. 630–636, 04 2017.
  * [144] S. Yeung, F. Rinaldo, J. Jopling, B. Liu, R. Mehra, N. L. Downing, M. Guo, G. M. Bianconi, A. Alahi _et al._ , “A computer vision system for deep learning-based detection of patient mobilization activities in the icu,” _npj Digital Medicine_ , vol. 2, no. 1, p. 11, 2019.
  * [145] A. Haque, B. Peng, Z. Luo, A. Alahi, S. Yeung, and F. Li, “Viewpoint invariant 3D human pose estimation with recurrent error feedback,” in _Proc. ECCV_ , 2016, pp. 160–177.
  * [146] V. Belagiannis, X. Wang, H. B. Shitrit, K. Hashimoto, R. Stauder, Y. Aoki, M. Kranzfelder, A. Schneider, P. Fua _et al._ , “Parsing human skeletons in an operating room,” _Mach. Vis. Appl._ , vol. 27, no. 7, pp. 1035–1046, 2016.
  * [147] A. Kadkhodamohammadi, A. Gangi, M. de Mathelin, and N. Padoy, “A multi-view RGB-D approach for human pose estimation in operating rooms,” in _Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2017.
  * [148] I. Chakraborty, A. Elgammal, and R. S. Burd, “Video based activity recognition in trauma resuscitation,” in _10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , 2013, pp. 1–8.
  * [149] C. Lea, J. C. Facker, G. D. Hager, R. H. Taylor, and S. Saria, “3D sensing algorithms towards building an intelligent intensive care unit,” in _AMIA Summits on Translational Science_ , 2013.
  * [150] A. P. Twinanda, E. O. Alkan, A. Gangi, M. de Mathelin, and N. Padoy, “Data-driven spatio-temporal RGBD feature encoding for action recognition in operating rooms,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 10, no. 6, pp. 737–747, 2015.
  * [151] A. P. Twinanda, P. Winata, A. Gangi, M. D. Mathelin, and N. Padoy, “Multi-stream deep architecture for surgical phase recognition on multi-view rgbd videos,” in _M2CAI Workshop at MICCAI_ , 2016.
  * [152] V. Srivastav, T. Issenhuth, K. Abdolrahim, M. de Mathelin, A. Gangi, and N. Padoy, “MVOR: A multi-view RGB-D operating room dataset for 2D and 3D human pose estimation,” in _MICCAI-LABELS_ , 2018.
  * [153] T. Issenhuth, V. Srivastav, A. Gangi, and N. Padoy, “Face detection in the operating room: Comparison of state-of-the-art methods and a self-supervised approach,” _Int. J. Comput. Assisted Radiol. Surg._ , 2019.
  * [154] V. Srivastav, A. Gangi, and N. Padoy, “Privacy-preserving human pose estimation on low-resolution depth images,” in _MICCAI_ , 2019.
  * [155] A. Malpani, C. Lea, C. C. G. Chen, and G. D. Hager, “System events: Readily accessible features for surgical phase detection,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 11, no. 6, pp. 1201–1209, 2016.
  * [156] Y. Gu, X. Li, S. Chen, H. Li, R. A. Farneth, I. Marsic, and R. S. Burd, “Language-based process phase detection in the trauma resuscitation,” in _IEEE ICHI_ , 2017.
  * [157] N. Navab, C. Hennersperger, B. Frisch, and B. Fuerst, “Personalized, relevance-based multimodal robotic imaging and augmented reality for computer assisted interventions,” in _Med. Image Anal._ , 06 2016.
  * [158] F. Lalys and P. Jannin, “Surgical process modelling: a review,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 9, no. 3, pp. 495–511, 2014.
  * [159] C. E. Reiley and G. D. Hager, “Using robots to train the surgeons of tomorrow,” _IEEE Spectrum_ , 2011.
  * [160] H. C. Lin, I. Shafran, T. E. Murphy, A. M. Okamura, D. D. Yuh, and G. D. Hager, “Automatic detection and segmentation of robot-assisted surgical motions,” in _Proc. MICCAI’05_ , 2005, pp. 802–810.
  * [161] H. C. Lin, I. Shafran, D. Yuh, and G. D. Hager, “Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions,” _Computer Aided Surgery_ , vol. 11, no. 5, pp. 220–230, 2006.
  * [162] F. Despinoy, D. Bouget, G. Forestier, C. Penet, N. Zemiti, P. Poignet, and P. Jannin, “Unsupervised trajectory segmentation for surgical gesture recognition in robotic training,” _IEEE Trans. Biomed. Eng._ , vol. 63, no. 6, pp. 1280–1291, 2015.
  * [163] B. Varadarajan, C. E. Reiley, H. Lin, S. Khudanpur, and G. D. Hager, “Data-derived models for segmentation with application to surgical assessment and training,” in _Proc. MICCAI’09_ , 2009, pp. 426–434.
  * [164] N. Padoy and G. D. Hager, “Human-machine collaborative surgery using learned models,” in _IEEE International Conference on Robotics and Automation, ICRA 2011, Shanghai, China, 9-13 May 2011_ , 2011, pp. 5285–5292.
  * [165] J. D. Birkmeyer, J. F. Finks, A. O’Reilly, M. Oerline, A. M. Carlin, A. R. Nunn, J. Dimick, M. Banerjee, and N. J. Birkmeyer, “Surgical skill and complication rates after bariatric surgery,” _N. Engl. J. Med._ , vol. 369, no. 15, pp. 1434–1442, 2013.
  * [166] A. Zia, Y. Sharma, V. Bettadapura, E. L. Sarin, and I. A. Essa, “Video and accelerometer-based motion analysis for automated surgical skills assessment,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 13, no. 3, pp. 443–455, 2018.
  * [167] T. S. Kim, M. O’Brien, S. Zafar, G. D. Hager, S. Sikder, and S. S. Vedula, “Objective assessment of intraoperative technical skill in capsulorhexis using videos of cataract surgery,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 14, no. 6, pp. 1097–1105, 2019.
  * [168] J. Fotouhi, M. Unberath, T. Song, W. Gu, A. Johnson, G. Osgood, M. Armand, and N. Navab, “Interactive flying frustums (IFFs): spatially aware surgical data visualization,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 14, no. 6, pp. 913–922, 2019.
  * [169] S. Matinfar, M. A. Nasseri, U. Eck, M. Kowalsky, H. Roodaki, N. Navab, C. P. Lohmann, M. Maier, and N. Navab, “Surgical soundtracks: automatic acoustic augmentation of surgical procedures,” _Int. J. Comput. Assisted Radiol. Surg._ , vol. 13, no. 9, pp. 1345–1355, Sep. 2018.
  * [170] O. Pauly, B. Diotte, P. Fallavollita, S. Weidert, E. Euler, and N. Navab, “Machine learning-based augmented reality for improved surgical scene understanding,” _Comput. Med. Imaging Graph._ , vol. 41, pp. 55–60, 2015\.
  * [171] A. Malpani, S. S. Vedula, H. C. Lin, G. D. Hager, and R. H. Taylor, “Real-time teaching cues for automated surgical coaching,” arXiv:1704.07436, 2017.
  * [172] A. Reiter, P. K. Allen, and T. Zhao, “Feature classification for tracking articulated surgical tools,” in _Proc. MICCAI’12_. Springer, 2012, pp. 592–600.
  * [173] X. Du, T. Kurmann, P. Chang, M. Allan, S. Ourselin, R. Sznitman, J. D. Kelly, and D. Stoyanov, “Articulated multi-instrument 2-d pose estimation using fully convolutional networks,” _IEEE Trans. Med. Imag._ , vol. 37, no. 5, pp. 1276–1287, 2018.
  * [174] D. Pakhomov, V. Premachandran, M. Allan, M. Azizian, and N. Navab, “Deep residual learning for instrument segmentation in robotic surgery,” arXiv:1703.08580, 2017.
  * [175] N. Rieke, D. J. Tan, C. A. di San Filippo, F. Tombari, M. Alsheakhali, V. Belagiannis, A. Eslami, and N. Navab, “Real-time localization of articulated surgical instruments in retinal microsurgery,” _Med. Image Anal._ , vol. 34, pp. 82–100, 2016.
  * [176] T. Erickson, “Some problems with the notion of context-aware computing,” _Commun. ACM_ , vol. 45, no. 2, pp. 102–104, Feb. 2002.

|  Tom Vercauteren Tom Vercauteren is Professor of Interventional Image
Computing at King’s College London since 2018 where he holds the Medtronic /
Royal Academy of Engineering Research Chair in Machine Learning for Computer-
assisted Neurosurgery. From 2014 to 2018, he was Associate Professor at UCL
where he acted as Deputy Director for the Wellcome / EPSRC Centre for
Interventional and Surgical Sciences (2017-18). From 2004 to 2014, he worked
for Mauna Kea Technologies, Paris where he led the research and development
team designing image computing solutions for the company’s CE-marked and FDA-
cleared optical biopsy device. He is a Columbia University and Ecole
Polytechnique graduate and obtained his PhD from Inria in 2008. Tom
Vercauteren’s research focuses on translational medical image computing,
machine learning and interventional imaging devices with a specific interest
in their development for surgery and interventional sciences.
---|---
|  Mathias Unberath is an Assistant Research Professor in the Department of
Computer Science at Johns Hopkins University, and is affiliated with the
Laboratory for Computational Sensing and Robotics and the Malone Center for
Engineering in Healthcare. Mathias first joined Hopkins as a postdoctoral
fellow after graduating summa cum laude from the Friedrich-Alexander-
Universität Erlangen-Nürnberg with a BSc in Physics, a MSc in Optical
Technologies, and a PhD in Computer Science. He was an ERASMUS scholar at the
University of Eastern Finland and DAAD fellow at Stanford University. Mathias’
research at the intersection of computer vision including augmented reality,
machine learning, and medical physics has been recognized with multiple
national and international awards, and aims at pushing the boundaries of
computer assistance in medical imaging and image-guided interventions.
---|---
|  Nicolas Padoy is a full Professor of Computer Science at the University of
Strasbourg, where he began as an Assistant Professor on a Chair of Excellence
in 2012. He created and is currently leading the research group CAMMA on
Computational Analysis and Modeling of Medical Activities, which focuses on
computer vision, activity recognition, artificial intelligence and the
applications thereof to surgical workflow analysis and human-machine
cooperation during surgery. From 2009 to 2011, he was a postdoctoral
researcher and later an Assistant Research Professor in the Laboratory for
Computational Interactions and Robotics at the Johns Hopkins University. He
completed his PhD jointly between the Chair for Computer Aided Medical
Procedures at Technische Universität München (TUM) and the INRIA group MAGRIT
in Nancy. He graduated with a Maîtrise in Computer Science from the Ecole
Normale Supérieure de Lyon in 2003 and with a Diploma in Computer Science from
TUM in 2005.
---|---
|  Nassir Navab is a full Professor and Director of the Laboratory for
Computer Aided Medical Procedures, Technical University of Munich and Johns
Hopkins University. He completed his PhD at INRIA and University of Paris XI,
France, and enjoyed two years of a post-doctoral fellowship at MIT Media
Laboratory before joining Siemens Corporate Research (1994-2003). He received
the Siemens Inventor of the Year Award in 2001, the SMIT Society Technology
award in 2010 and the ‘10 years lasting impact award’ of IEEE ISMAR in 2015.
In 2012, he was elected as a Fellow of the MICCAI Society. He has acted as a
member of the board of directors of the MICCAI Society, 2007-2012 and
2014-2017, and serves on the Steering committee of the IEEE Symposium on Mixed
and Augmented Reality (ISMAR) and Information Processing in Computer Assisted
Interventions (IPCAI). He is the inventor of 47 granted US patents and more
than 50 International ones. His current research interests include multimodal
imaging, medical augmented reality, computer assisted surgery, medical
robotics, and machine learning.
---|---