Nema et al

*Shubhangi Nema, Ph.D. Research Scholar, Systems and Control Group, Indian Institute of Technology Bombay, Mumbai, 470001, India.

Funding
Financial support for this study was provided by Prime Minister’s Research
Fellows (PMRF) scheme, India (PMRF Id no. 1300229 dated May 2019) for pursuing
research in higher educational institutions in India. https://www.pmrf.in/.

#  Plug-in for visualizing 3D tool tracking from videos of Minimally Invasive
surgeries

Shubhangi Nema  Abhishek Mathur  Prof. Leena Vachhani  Shubhangi and Prof.
Leena are with Systems and Control Group, Indian Institute of Technology
Bombay, Maharashtra, India (e-mail: Shubhangi Nema<EMAIL_ADDRESS>Leena Vachhani<EMAIL_ADDRESS> Abhishek is with Mechanical
Engineering, Indian Institute of Technology Goa, Goa, India (e-mail: Abhishek
Mathur<EMAIL_ADDRESS><EMAIL_ADDRESS>
(<day> <Month>, <year>; <day> <Month>, <year>; <day> <Month>, <year>)

###### Abstract

[Abstract]

Background
This paper tackles instrument tracking and 3D visualization challenges in
minimally invasive surgery (MIS), crucial for computer-assisted interventions.
Conventional and robot-assisted MIS encounter issues with limited 2D camera
projections and minimal hardware integration.

Method
The objective is to track and visualize the entire surgical instrument,
including shaft and metallic clasper, enabling safe navigation within the
surgical environment. The proposed method involves 2D tracking based on
segmentation maps, facilitating creation of labeled dataset without extensive
ground-truth knowledge. Geometric changes in 2D intervals express motion, and
kinematics based algorithms process results into 3D tracking information.

Result
Synthesized and experimental results in 2D and 3D motion estimates demonstrate
negligible errors, validating the method for labeling and motion tracking of
instruments in MIS videos.

Conclusion
The conclusion underscores the proposed 2D segmentation technique’s simplicity
and computational efficiency, emphasizing its potential as direct plug-in for
3D visualization in instrument tracking and MIS practices.

###### keywords:

Minimally Invasive Surgery, Surgical instruments, 3D tracking, Geometric cues

††articletype: Original Article

##  1 Introduction

Advances in robotics, computer graphics, and virtual reality (VR) have been
increasingly applied to surgical robotics 1. In the cutting-edge
interdisciplinary research field of information and medical sciences, research
on virtual surgery simulation systems has significant application value for
reducing surgery risks, cutting training costs, and protecting human health 2,
3. With the help of virtual surgery training platform, trainee surgeons can
skillfully master the operations of surgical instruments, the general
procedure of surgery, and the anatomy of a diseased region or organ. The
accurate 3-dimensional (3D) movement tracking of surgical instruments is a
vital part of the VR-based surgery training simulation system 4, 5. Assessment
of surgical instrument motion can be implemented to objectively classify skill
levels. In robotic surgery, surgeons can view the surgical scene on a screen
in the form of 2D projections as captured by the camera 6. Segmentation of
surgical instruments used during surgery 7, 8, 9 is a first step to tracking
their motions for understanding the surgical procedure. We aim to develop
techniques for 3D motion tracking of MIS instruments from raw surgical videos
available currently. The multi-class segmentation problem 9 has been
formulated to distinguish between different instruments or different parts of
an instrument (connected sets of a surgical instrument) from the background. A
deep residual learning-based technique 7 incorporates dilated convolutions for
multi-class segmentation by advancing binary-segmentation performance.
Advancements in multi-class segmentation of robotics and rigid instruments 9
using unpaired training approach conveys feasibility of good accuracy
segmentation on unseen data as well. The segmentation maps are readily
available from these methods. We investigate the segmented maps to track the
2D movements of the surgical instruments used to carry out any interventions.
These movements can not be directly translated into real-world scenarios;
thus, a 3D tracking tool is necessary to have better scene perception and
carry out smooth movements of instruments by the surgeons.

The 3D motion of surgical instrument has been attempted by either making
physical modifications to have good image contrast or having some knowledge of
the instrument or environment in captured raw surgical videos.

Methods based on physical modification: To track instrument motion, the
modifications are done to improve the appearance of instrument in the captured
video. Typically markers have been placed on instruments and tracking of
markers renders the 3D motion of instruments. Work by Zhang et al. 10 uses the
intensity information of the image for the surgical instrument marker
localization. The evaluation consists of several experiments such as camera
calibration, endoscope image distortion correction, endoscope holder
simulation, and visual tracking using pcBird. There has also been the use of
3D trackers in virtual reality. According to their physical properties, they
are roughly classified into five subcategories: mechanical tracker 11,
magnetic tracker 12, ultrasonic tracker 13, 14, optical tracker 15 and hybrid
tracker 16. Some of them can provide high positioning accuracy 17, and have
been used in medical applications. A 3D surgical instrument tracking and
positioning method with a high performance-price ratio has been highly
desirable for computer-based virtual surgery simulation systems 18. An
iterative approach 19 based on the projection-Procrustes technique determines
the three-dimensional positions and orientations of known sparse objects from
a single, perspective projection. A 3D tracking and positioning method for
surgical instruments 20 based on stereoscopic vision combines force sensor and
embedded acquisition device is proposedin order to measure soft tissues
parameters. An algorithm for real-time tracking of laparoscopy instruments 21
attached with colored tapes (markers) uses the video cues of a standard
physical laparoscopy training box with a single fisheye camera. An intelligent
tracking system for surgical instruments is proposed based on data fusion of
multicamera modules 22. The method evaluated for static positioning and
dynamic trajectory tracking. However, these existing devices do not provide
cost-effective solutions limiting their usage by medical centers and research
institutes. Moreover, there are practical difficulties in adopting the use of
external markers during surgery. The use of markers increases the procedural
time during surgery and needs an ethics board’s approval. These are
accompanied by the risk of infection and occlusion of markers affecting
tracking performance.

Methods using raw surgical videos: As markers or contrasting colored surgical
instruments require changing procedures, the investigations have attempted to
directly use raw surgical videos without making any procedural changes. A 2D
tooltip location tracking algorithm in laparoscopic training videos analyses
hand & eye coordination 23. The tool-tip detection algorithm is based on
background subtraction and thresholding foreground showing promising outcomes
in occlusion-free and well-contrasting surgical scenarios. The geometric and
photometric invariants common to standard Fundamentals of Laparoscopic Surgery
(FLS) training boxes have been utilized 24 for tracking the spatial motion of
standard laparoscopic instruments from videos.

Early image-based methods predominately estimate instrument poses based on
image processing techniques that utilize hand-crafted visual features
(sometimes apriori known) and more complex learned discriminative models.
Tracking of an instrument is performed independently on each frame, and the
methods are typically fast and robust, and handling complex and fast motion
becomes easier. A 2D tracker based on a Generalized Hough Transform using SIFT
features is developed 25 for tracking surgical instruments for minimally
invasive and robotic-assisted surgery. The technique initializes the 3D
tracker at each frame which recovers the 3D instrument pose over long
sequences. A few direct 3D tracking based on deep learning methods have also
been investigated. A fully convolutional network (FCN) with optical flow for
tracking 26 and TernausNet-16, a deep convolutional neural network-based
instrument tracking 27; an extension of U-Net 28 constructed on VGG-16 29 are
two classic examples. However, these methods are computationally demanding and
necessitate a large volume of labeled datasets.

Our approach is to disintegrate the problem of 3D tracking through 2D rendered
images by identifying the 2D movement of the surgical instrument and then
converting it to the 3D motion using geometric dynamics. Moreover, this work
aims to track each part of the instrument for avoiding collision with the
sensitive tissues/vessels. The objective is to use lower dimensional results
to track instruments in higher dimensions with simple geometric computations
for visualization of 3D instrument motion and thus facilitating creation of
labeled dataset (in terms of motion labels for position and rotation angles
with respect to time).

A surgical instrument has mainly two parts, the shaft, and the metallic
clasper/end-effector. The metallic clasper attached to the shaft performs
critical interventions such as grasping, cutting, etc. We use interval
arithmetic approach 30to consider the complete shape of the instrument in
terms of intervals to avoid collision of any part of the instrument with
critical organs.

The summary of contributions of the proposed work building up the structure of
the rest of this paper is as follows:

  * •

A novel approach that combines a unique methodology and problem formulation
for 2D tracking in surgical instruments is presented next. It employs
kinematics to process the 2D tracking results and generate corresponding 3D
movements. The approach considers the instrument’s shape through connected
intervals, analyzes geometric changes within these intervals to estimate
motion, and results in an algorithm that accommodates various shapes while
enhancing tool dexterity.

  * •

The proposed algorithms for 2D and 3D tracking of a surgical instrument for a
complete surgery are presented in Section 3. The algorithm uses a sequence of
simple functions from an existing open-source vision package and computes the
3D action of instrument parts (shaft and clasper).

  * •

Section 4 presents simulations to visualize the 3D motion of surgical
instruments. The results from these simulations provide evidence of the
precise and accurate movements achieved by the instruments. The error analysis
in 2D and 3D is performed to assess the accuracy claims, that eliminates the
necessity to rely on ground truth information. The quantitative experimental
results for 3D tracking are compared to state-of-the-art methods, showing a
significant reduction in error. Moreover, the average error obtained from the
proposed method using monocular vision is comparable to that of stereo vision-
based 3D tracking. The proposed approach produces tracking results without any
physical intervention and can be used for autonomous tracking of surgical
instruments during MIS. The future scope of this work to implement the
strategy in an experimental setup for providing real-time training or
assistance to the surgeons is discussed in Section 5, along with conclusions.

##  2 Proposed work

Figure 1: Surgical instrument representation (zoomed view) with two links (k=
{1,2}) for nth frame (a) Parameters of $R_{k}^{n}$ (b) Parameters of
$[B_{k}]^{n}$

In an image frame of endoscopic video (frame size of (720 $\times$ 526
$pixel^{2}$) in the dataset6), we propose to represent the surgical instrument
as a combination of approximated rectangular intervals. Let the length of the
video be $N$ image frames and the number of instrument parts is $K$. The
approximated rectangular interval corresponding to each instrument part
($k=1,\ldots,K$) in each $n^{th}$ image frame ($I^{n}$, $n\in[1,N]$) is
$[B_{k}]^{n}:=\\{x_{k},y_{k},l_{k},w_{k},\theta_{k}\\}$, with center
($x_{k},y_{k}$), length ($l_{k}$), width ($w_{k}$), angle ($\theta_{k}$)
(refer Fig. 1 (a)). The angle $\theta_{k}$ is with respect to the axis aligned
with respect to the $k-1$ interval in the 2D projected image of the
instrument. The instrument configuration comprises horizontal , vertical
positions and rotation on the image plane. This rotation represents the
rotation of the projected instrument (on image plane) separately for each part
of the instrument. We consider the image plane as $X-Y$ plane, with $Z-$ axis
orthogonal and projecting out of the image plane. A surgical instrument in an
image projection in the proposed work has two connected intervals defined
using bounding boxes for the metallic clasper ($[B_{1}]^{n},k=1$) and the
shaft ($[B_{2}]^{n},k=2$), an illustration of the same is in Fig. 1(b).

The overlapping region between two connected intervals is considered in this
work to incorporate actual movement between the shaft and the clasper through
a linkage. The interval coordinates or bounding boxes ($[B_{k}]^{n}$ with
center ($x_{k},y_{k}$), length($l_{k}$), width($w_{k}$), angle($\theta_{k}$)
surrounding the metallic clasper ($k=1$) and the shaft ($k=2$) along with
their corresponding corner points are required to track 3D motion for surgical
instruments. The 2D segmented images ($I^{n}|n=1,\ldots,N$) of the surgical
instruments are taken as inputs in order to generate these bounding boxes
using Algorithm 1. Each bounding box in an image frame $n$ has four vertices
or corners $\\{(x_{k,i}^{n},y_{k,i}^{n})|_{i=\\{1,2,3,4\\}}\\}$ which serve to
be the primary input that can be processed using Algorithm 2 The corner points
for a bounding box are obtained using the description of $[B_{k}]^{n}$ as
follows:

$\displaystyle(x_{k,1}^{n},y_{k,1}^{n})=(x_{k}-(l_{k}/2)cos\theta_{k},y_{k}+(w_{k}/2)sin\theta_{k})$
$\displaystyle(x_{k,2}^{n},y_{k,2}^{n})=(x_{k}+(l_{k}/2)cos\theta_{k},y_{k}+(w_{k}/2)sin\theta_{k})$
(1)
$\displaystyle(x_{k,3}^{n},y_{k,3}^{n})=(x_{k}-(l_{k}/2)cos\theta_{k},y_{k}-(w_{k}/2)sin\theta_{k})$
$\displaystyle(x_{k,4}^{n},y_{k,4}^{n})=(x_{k}+(l_{k}/2)cos\theta_{k},y_{k}-(w_{k}/2)sin\theta_{k}).$

The 3D movements for the instrument’s clasper and the shaft are generated in
the form of translation matrix ($\mathcal{T}_{k}$) and rotation matrix
($\mathcal{R}_{k}$) using the following steps: Let $\Delta x_{k}$, $\Delta
y_{k}$, and $\Delta z_{k}$ abe the movement of the instrument’s $k^{th}$ part
($k=1$ for clasper and $k=2$ for shaft) in $x$, $y$ and $z$ direction
respectively. The proposed method, as its first step, tracks the motion of the
centroid of the bounding box. In particular, change in the $x-$coordinate of
the centroid ($C_{k,x}$) gives $\Delta x_{k}$ and change in the $y-$coordinate
of the centroid ($C_{k,y}$) gives y ($\Delta y_{k}$).

The centroids ($C_{k,x}^{n}$ and $C_{k,y}^{n}$) of the bounding box are given
by

$C_{k,x}^{n}=\sum_{i=1}^{4}x_{k,i}^{n}/4,C_{k,y}^{n}=\sum_{i=1}^{4}y_{k,i}^{n}/4$
(2)

Now to compute the movement in $z-$ direction ($\Delta z_{k}$), we capture the
geometric changes in the bounding box $[B_{k}]^{n}$. The area of a bounding
box is given by 111% is the modulo operator giving the remainder of the
division operation.

$A_{k}^{n}=1/2*(\sum_{i=1}^{3}(x_{k,i}^{n}*y_{k,(i\%4+1))}^{n}-\sum_{i=1}^{3}(x_{k,(i\%4+1)}^{n}*y_{k,i}^{n}))$
(3)

This is required to compute the change in scale ($S_{k}^{n}$) in each frame
which is evaluated by taking the ratio of areas of bounding boxes in the
current ($n$) and previous frame ($n-1$) respectively. Thus, the $z-$
coordinates for a bounding box is evaluated using the scale change
($S_{k}^{n}$), area ($A_{k}^{n}$), and the given focal length $f$ of the
endoscope as follows:

$z_{k}^{n}\approx sqrt(f^{2}.(S_{k}^{n})^{2}/A_{k}^{n})$ (4)

Now, using $\Delta z_{k}=z_{k}^{n}-z_{k}^{n-1}$, the translation matrix
$\mathcal{T}_{k}$ is given by

$\mathcal{T}_{k}\approx T_{Z}(\Delta z_{k})T_{Y}(\Delta y_{k})T_{X}(\Delta
x_{k})$ (5)

where,

$T_{Z}(\Delta z_{k})\approx\begin{bmatrix}1&0&0\\\ 0&1&0\\\ 0&0&\Delta
z_{k}\end{bmatrix}$ $T_{Y}(\Delta y_{k})=\begin{bmatrix}1&0&0\\\ 0&1&\Delta
y_{k}\\\ 0&0&1\end{bmatrix}$

$T_{X}(\Delta x_{k})=\begin{bmatrix}1&0&\Delta x_{k}\\\ 0&1&0\\\
0&0&1\end{bmatrix}$

For computing the rotation matrix $\mathcal{R}_{k}$, the net change in roll,
pitch, and yaw orientations of each instrument part, $(\Delta\phi_{k})$,
$(\Delta\theta_{k})$, and $\Delta\psi_{k})$ respectively is computed using the
changes in internal angle of each edge of the bounding box. The slope of each
edge of the bounding box is given by

$m_{k,i}^{n}=(y_{k,i\%4+1}^{n}-y_{k,i}^{n})/(x_{k,i\%4+1}^{n}-x_{k,i}^{n}),$
(6)

and each internal angle is obtained by $a_{k,i}^{n}$ is defined as,

$a_{k,i}^{n}=tan^{-1}((m_{k,{i\%4+1}}^{n}-m_{k,i}^{n})/(1+m_{k,i\%4+1}^{n}*m_{k,i}^{n}))$
(7)

The change in roll-pitch-yaw are computed by capturing the following geometric
changes heuristically:

  * •

$\Delta\phi_{k}$ is the rotation about the $x-$axis. The projection of the
bounding box on the image plane changes the internal angle of the bounding
box. Hence, the change in roll is captured as the difference between the
individual sums of $\angle 1$ ($a_{k,1}^{n}$) $\&$ $\angle 2$ ($a_{k,2}^{n}$)
and $\angle 3$ ($a_{k,3}^{n}$) $\&$ $\angle 4$ ($a_{k,4}^{n}$). The change in
angle is zero when the instrument part is parallel to the image plane, but it
develops a non-zero value directly proportional to the roll of the
corresponding instrument part.

  * •

$\Delta\theta_{k}$ is the rotation about the y axis and the change in pitch is
captured as the difference between the individual sums of $\angle 1$
($a_{k,1}^{n}$) $\&$ $\angle 4$ ($a_{k,4}^{n}$) and $\angle 3$ ($a_{k,3}^{n}$)
$\&$ $\angle 2$ ($a_{k,2}^{n}$).

  * •

In the case of $\Delta\psi_{k}$, the axis is orthogonally out of the image
plane. Hence, we find the initial inclination (defined using $m_{k,i}^{n}$) of
the sides of the bounding box and subtract the average from $\pi$ radians.

Thus, the rotation matrix $\mathcal{R}_{k}$ matrix is computed using

$\mathcal{R}_{k}\approx
R_{Z}(\Delta\psi_{k})R_{Y}(\Delta\theta_{k})R_{X}(\Delta\phi_{k})$ (8)

where,

$R_{Z}(\Delta\psi_{k})=\begin{bmatrix}cos(\Delta\psi_{k})&-sin(\Delta\psi_{k})&0\\\
sin(\Delta\psi_{k})&cos(\Delta\psi_{k})&0\\\ 0&0&1\end{bmatrix}$
$R_{Y}(\Delta\theta_{k})=\begin{bmatrix}cos(\Delta\theta_{k})&0&sin(\Delta\theta_{k})\\\
0&1&0\\\ -sin(\Delta\theta_{k})&0&cos(\Delta\theta_{k})\end{bmatrix}$

$R_{X}(\Delta\phi_{k})=\begin{bmatrix}1&0&0\\\
0&cos(\Delta\phi_{k})&-sin(\Delta\phi_{k})\\\
0&sin(\Delta\phi_{k})&cos(\Delta\phi_{k})\end{bmatrix}$

A novel 3D motion tracking algorithm to robustly estimate the full 3D pose of
surgical instruments in minimally invasive surgery using 2D image projections
is presented next.

Result:$\\{x_{k,i}^{n},y_{k,i}^{n}\\}|$ k=1,2 and i=1,2,3,4

Input: $I^{n}$

1:for $k=\\{1,2\\}$: do

2: $M_{k}^{n}=mask(I^{n},\mbox{intensity value range for box }k)$;

3: $O_{k}^{n}$ = $I^{n}$ & $M_{k}^{n}$;

4: $G_{k}^{n}=gray(O_{k}^{n})$;

5: $H_{k}^{n}=contour$( $G_{k}^{n}$, contour retrieval Mode, technique);

6: $R_{k}^{n}$ = $rect\\_bound(O_{k}^{n},H_{k}^{n})$;

7: $[B_{k}^{n}]=minAreaRect(R_{k}^{n})$;

8: $x_{k,i}^{n},y_{k,i}^{n}=boxPoints(B_{k}^{n})$;

9:end for

Algorithm 1 Bounding boxes for a surgical instrument in an image frame:
$Bound\\_box$ ($I_{n}$)

##  3 Algorithms

This section demonstrates the Algorithm 1 and 2 based on geometric changes in
intervals and kinematics.

The $n^{th}$ image frame ($I^{n}$) of the video sequence, where $n=1,\ldots,N$
contains three regions, the metallic clasper, shaft and the background
represented using different pixel values.

Algorithm 1 generates the bounding boxes for the clasper ($[B_{1}^{n}],k=1$)
and the shaft ($[B_{2}^{n}],k=2$) and their respective vertices or bounding
box points ($(x_{k,i}^{n},y_{k,i}^{n})$ from the image frame $I^{n}$. The
bounding box interval and the corresponding internal angles changes with each
image frame during $z$ movement of the individual instrument part. The
metallic clasper and the shaft are connected through a linkage, resulting in
an overlapping region on the image plane. This arrangement allows for
appropriate movements and rotations. When generating the corresponding
bounding box intervals, the algorithm takes into account all the pixels
belonging to either the shaft or the metallic clasper, regardless of their
presence in the overlapping region ensuring the connectivity between the
intervals.

Figure 2: Pictorial representation bounding box intervals formation for shaft
and the metallic clasper

Result: $\mathcal{T}_{k},\mathcal{R}_{k}$ Input: $I^{n}|n=1,2...N$

Initialize: ${n=1}$

1:while $n<=N$ do ($x_{k,i}^{n},y_{k,i}^{n}$) = $Bound\\_box(I^{n})$

2: for i = $\\{1,2,3,4\\}$ do

3: for k = $\\{1,2\\}$ do

4: Compute: $A_{k}^{n}$ using Eq. (3)

5: $S_{k}^{n}$ = $A_{k}^{n}/A_{k}^{(n-1)}$

6: Compute: $z_{k}^{n}$ using Eq. (4)

7: Compute: $C_{k,x}^{n},C_{k,y}^{n},m_{k,i}^{n},a_{k,i}^{n}$ using Eq. (2, 6,
7)

8: $\Delta x_{k}^{n}$ = $C_{k,x}^{n}-C_{k,x}^{(n-1)}$

9: $\Delta y_{k}^{n}$ = $C_{k,y}^{n}-C_{k,y}^{(n-1)}$

10: $\Delta z_{k}^{n}$ = $z_{k}^{n}-z_{k}^{(n-1)}$

11: $e_{k,r}^{n}$ = $a_{k,1}^{n}+a_{k,4}^{n}-a_{k,2}^{n}-a_{k,3}^{n}$

12: $\Delta\phi_{k}$ = $e_{k,r}^{n}-e_{k,r}^{(n-1)}$

13: $e_{k,p}^{n}$ = $a_{k,3}^{n}+a_{k,4}^{n}-a_{k,1}^{n}-a_{k,2}^{n}$

14: $\Delta\theta_{k}$ = $e_{k,p}^{n}-e_{k,p}^{(n-1)}$

15: $e_{k,y}^{n}$ = $(180-\sum(tan^{(-1)}(m_{k,i}^{n})))/4$

16: $\Delta\psi_{k}$ = $e_{k,y}^{n}-e_{k,y}^{(n-1)}$

17: Compute $\mathcal{T}_{k}$ using Eq. (5)

18: Compute $\mathcal{R}_{k}$ using Eq. (8)

19: end for

20: end for$n++$

21:end while

Algorithm 2 3D movement using 2D boxPoints of clasper and shaft

Following functions are used in the Algorithm 1:

  * •

$mask()$: The function detects an object in the segmented image based on the
range of pixel intensity values for each $k$. It computes mask ($M_{k}^{n}$)
in image $I^{n}$ with pixel intensity values in the specified range.

  * •

$gray()$: This function converts RGB image $O_{k}^{n}$ into binary gray scale
image $G_{k}^{n}$. The image $G_{k}^{n}$ has only binary pixel values i.e.,
either the threshold value or 255.

  * •

$contour()$: The function computes a curve ($H_{k}^{n}$) joining all the
continuous points (along the boundary), having same color or intensity. The
contour is formed around the shaft or the clasper based on the corresponding
specified pixel intensity range. For better accuracy, we applied this function
on the binary gray image $G_{k}^{n}$. The input parameters are the source
image ($G_{k}^{n}$), the contour retrieval mode (a mode that returns only
extreme outer contours), the contour approximation method (we used chain
approximation that stores the pixel coordinates of the boundary of a shape).

  * •

$rect\\_bound$(): This function highlights the region of interest as a
quadrilateral aligned with the $X-Y$ axis of the image after obtaining
contours from an image. The function uses contour list ($H_{k}^{n}$) and draws
an approximate rectangle around the masked output image ($O_{k}^{n}$) with top
left corners ($x_{k}^{\prime},y_{k}^{\prime}$), length $L_{k}^{\prime}$ and
width $W_{k}^{\prime}$ as shown in Fig. 2

  * •

$minAreaRect()$: The function forms a bounding box with minimum area, that
also considers the rotation of the detected instrument parts (shaft or the
metallic clasper). It returns a Box2D structure ($[B_{k}]^{n}$) for given $k$
and $n$ which contains following details - ( center ($x_{k},y_{k}$),
(length($l_{k}$), width($w_{k}$), angle($\theta_{k}$) for each contour
($H_{k}$).

  * •

$boxPoints()$: The function will give the four corners
($x_{k,i}^{n},y_{k,i}^{n}$) of the rotated bounding box for clasper $(k=1)$
and for the shaft $(k=2)$.

The image frames ($I^{n}|n=1,2...N$) in a surgical video with a surgical
instrument serve as the input to Algorithm 2. The Algorithm 2 is the main
algorithm that uses the output from the Algorithm 1. The sequence of
operations in Algorithm 2 is applied separately for each bounding box points
($(x_{k,i}^{n},y_{k,i}^{n})$ ( $k=1$for clasper & $k=2$ for shaft ) for an
image frame $I^{n}$ obtained using Algorithm 1 to obtain their 3D motion
($\mathcal{T}_{k},\mathcal{R}_{k}$). For each corner box point $i$ and each
instrument part $k$, the area ($A_{k}^{n}$) is calculated using Eq. 3 and
stored in a variable which then computes the scale of change ($S_{k}^{n}$).
The $z$-coordinates ($z_{k}^{n}$) for bounding boxes in each frame is
evaluated using Eq.(4) for which the centroids ($C_{k,x}^{n}$ and
$C_{k,y}^{n}$) are computed using Eq. (2). The slope ($m_{k,i}^{n}$) between
the corner points and bounding box angles ($a_{k,i}^{n}$ ) are evaluated using
Eq. (6) and Eq. (7) respectively. Next, the $\Delta x_{k}^{n}$, $\Delta
y_{k}^{n}$, $\Delta z_{k}^{n}$ and $\Delta\phi_{k}$, $\Delta\theta_{k}$,
$\Delta\psi_{k}$ are obtained for eventually computing the translation matrix
$\mathcal{T}_{k}$ and rotation matrix $\mathcal{R}_{k}$. The transformations
obtained for the shaft and the metallic clasper from frame to frame give a
complete 3D track for the surgical instrument.

It is also essential to note, that the moving camera frame only occurs when
the surgical view needs to be changed to access a particular area or tissue
and this is an intermediate procedure between the main surgical processes.
During the surgical procedure, there is minimal to no movement of the camera
frame. For incorporating the change of pose due to moving camera frame in
practise, change in camera motion can be easily detected by a large change in
the tool area/position in subsequent images. This information can be used to
initialize the algorithm.

##  4 Simulations and results analysis

This section illustrates the simulation technique for visualizing results of
the proposed approach followed by results to support claims of estimated
movements made by the surgical instruments.

Figure 3:  (a) 3D Visualization and (b) 2D projected frames

###  4.1 3D movement visualization

The outputs from Algorithm 1 are generated in the form of corner points of the
bounding boxes enclosing the shaft and the metallic clasper. The 2D bounding
boxes are simulated using the CV2 python library for a raw surgical video. The
simulation results in Fig. 2 show that the bounding boxes for the clasper and
the shaft are connected via linkage in order to incorporate proper movements
and rotations. The output of Algorithm 2 is the individual movements in the
$x$, $y$, and $z$ directions for the clasper and the shaft, respectively,
along with their individual roll, pitch, and yaw movements captured through
subsequent frames. The approach is easily extended to multiple instruments
present in the environment based on the range of pixel intensity values of
each part (shaft and the metallic clasper) in each instrument. Furthermore,
sequential processing of the bounding boxes preserves the connectivity between
different parts belonging to a particular instrument. Thus, the individual
part movement in 3D is obtained by processing bounding boxes corresponding to
each instrument similar to a single instrument. The objective is to use this
output to generate a 3D simulation (refer Fig. 3(a)) that successfully
captures and visualizes the robotic arm’s movement during endoscopic surgery,
hence gives a direct plug-in method for 3D tracking. A 3D visualization module
of Python, Visual Python or vpython (available at glowscript.org), is utilized
to simulate the 3D movements of the surgical instrument.

The clasper and the shaft are simulated as cylindrical objects pertaining to
their sizes and orientations. The length of each cylinder is the same as the
length of the respective bounding box and the radius of each cylinder is half
the width of the respective bounding box while 3D rendering. The base point of
the clasper is always attached to the end of the shaft so that both the
cylinders representing the shaft and clasper are always connected in the 3D
simulation. Using the motion appending function, the $x$, $y$, and $z$
positions of each cylinder’s base are periodically appended by the
corresponding output values that are obtained from the algorithm. Moreover,
built-in functions in vpython library change the cylinder axes according to
the individual roll, pitch, and yaw movements generated. The 3D rendering
sampling rate is the same as the frame rate and velocity data. The 3D
visualization of the instrument motion is simulated through an interactive
screen to view from multiple camera angles for any further analysis.

###  4.2 Error Analysis in 2D

The proposed algorithm is suitable where ground truths for 3D tracking is not
available. In order to validate our algorithm, we performed error analysis in
2D as well as in 3D.

To substantiate the proposed 3D visualization, a video is recorded from a
fixed camera angle to emulate the raw surgical video. The corresponding 2D
projected image (obtained from a fixed camera angle) of the simulated
instrument, a pair of cylinders is shown in Fig. 3 (b). The 2D projection of
the 3D visualization is on the X-Y plane which was assumed to be the viewing
plane of the original raw surgical video captured using the endoscope. This
data is then compared to the preexisting data, and the error plot is analyzed.

Raw surgical videos captured using the endoscope are the only inputs available
or rendered in order to perform any intervention. We aimed for the 3D
movements of surgical instruments from 2D images to give the surgeons a better
prospect. Thus, to improve the efficacy of the 3D simulated results, the 2D
raw surgical videos are bench-marked with respect to the emulated 2D video.
Error in the algorithm is found by reapplying it on the simulation and
comparing the obtained trajectory with the original data by measuring the
differences in each movement, including the rotation angles.

Table 1: Average error in position and rotation between corresponding variables in 2D raw image and 2D synthetic image Tool Component | x (in pixels) | y (in pixels) |  $\theta$ (in degree)
---|---|---|---
Shaft | 6.472 $\pm$ 2.918 | 8.506 $\pm$ 2.263 | 1.248 $\pm$ 0.734
Metallic-clasper | 4.422 $\pm$ 2.856 | 8.456 $\pm$ 4.877 | 1.851 $\pm$ 0.781

The average error (obtained from dividing the sum of all absolute errors by
the number of measurements/frames) in pixels for $x$ and $y$ movements and in
degrees for rotation angle ($\theta$) are obtained between the emulated 2D
video corresponding to the 3D movements and the captured video. A tabular
representation is given in Table. 1. The results provide evidence of the
precise and accurate movements achieved by the instruments with negligible
errors.

###  4.3 Error Analysis in 3D: Experimental validation

The proposed algorithms are validated using an experiment conducted for a
scaled-up object (a rectangular box with approximate dimensions) executing all
the required movements in all 3 dimensions. The ground truth for 3D movements
is established using a motion capture system that provides motion feedback
with very fine resolution. We captured 10 videos (=10 runs) using a webcam and
simultaneously record their ground truth on Vicon software used in motion
capture system. The proposed algorithms for 2D tracking followed by 3D
tracking are then applied to these videos and the 3D movements for the scaled
object (with comparable changes in the image pixels for the corresponding
motion between the object and the surgical instrument from EndoVis dataset)
are generated in the form of $x$ , $y$ , $z$ movements and roll, pitch and yaw
in order to give a complete 3D track for the scaled object. The relative
motion errors in 3D are estimated by comparing the obtained results with the
ground truth motion.

Figure 4: Experimental results from 1 run Table 2: Average Error (from 10 runs) in position and rotation between corresponding variables in 3D ground truth tracking data collected from motion capture system and 3D tracking obtained using proposed algorithm Error variable for the scaled object | Average Error
---|---
x (in mm) | 1.05 $\pm$ 0.05
y (in mm) | 0.615 $\pm$ 0.0132
z (in mm) | 0.854 $\pm$ 0.0275
Roll (in rad) | 1.57 $\times$ $10^{-4}$ $\pm$ 0.35 $\times$ $10^{-4}$
Pitch (in rad) | 2.25 $\times$ $10^{-4}$ $\pm$ 0.41 $\times$ $10^{-4}$
Yaw (in rad) | 1.39 $\times$ $10^{-4}$ $\pm$ 0.65 $\times$ $10^{-4}$

A pictorial representation of frames extracted from recorded video and
bounding box formation for 2D tracking is shown in left side of Figure 4 along
with the error estimation in 3D movement is demonstrated in right side of
Figure 4. Average Error (from 10 runs) in position and rotation between
corresponding variables in 3D ground truth tracking data collected from motion
capture system and 3D tracking obtained using proposed algorithm is
demonstrated in Table 2.

Table 3: Comparative study on average errors in translational motion Method | Average Error
---|---
x (in mm) | y (in mm) | z (in mm) | Translational motion
|  |  | Mean | S.D.
Marker intensity information 10 | Visual tracking | -2.132 | -12.817 | -2.999 | - | -
pc Bird | -20.82 | 22.452 | -22.257 | - | -
Color tape markers (single fisheye camera) 21 | 1.5 | 4 | 3 | - | -
Monocular module 22 | - | - | - | 2.54 | 0.22
Multicamera module 22 | - | - | - | 2.01 | 0.19
| Generalized hough transform using SIFT 25
---
- | - | - | 3.70 | 2.28
Stereoscopic vision based 20 | 1.4 | 0.7 | 0.25 | 0.31 | -
Proposed Method | 1.05 | 0.615 | 0.854 | 0.3989 | 0.303

###  4.4 Comparative study

In previous works on tracking algorithms, average errors for $x$, $y$ and $z$
movements (translational motion) have been reported using different setups
mimicking surgical instrument. Reporting of the quantitative rotational motion
error are missing to the best of our knowledge. We present a comparative study
of the proposed work with minimum error reported in literature (in Table 3).
The method utilizing intensity information of the image for the marker
localization 10 gives huge errors for 3D tracking using pc Bird, that reduces
to a significant amount using visual tracking. Our proposed method reduces
this error further by 50.8%, 94.99% and 71.52% for $x$, $y$ and $z$ movement
respectively computed from data reported in Table 3. The percentage reduction
in errors by our method for x,y and z movements are 30%, 84.63% and 71.53% for
a similar method that uses color tape markers to capture 3D movements using a
single fisheye camera 21. The average translational error is reduced by more
than 80% using the proposed method as compared to methods that uses monocular
module 22, multicamera module and a combined 2D3D method in which generalized
hough transform is utilized for SIFT feature extraction 25. Thus, it is
observed that the proposed method reduces the individual and combined
translational average error by a decent margin as compared to existing state-
of-the-art methods and produces the results comparable to the stereoscopic
vision based method 20.

Moreover, the tracking algorithms run on a computing machine with i7
compatible with CPU/GPU. The execution time for processing 1270 image frames
is $\approx$ 20 ms for the complete 3D tracking with segmented maps available
a-priori and consumes only 46$\%$ of CPU memory that makes the method
computationally inexpensive.

##  5 Conclusion and Future work

In this paper, we present a novel 2D tracker for the segmentation maps of
surgical instruments, which combines with a 3D tracking algorithm to robustly
estimate the full 3D pose of instruments in minimally invasive surgery. The
proposed plug-in approach produces tracking results for any number of
segmented instruments present in the surgical environment without any physical
intervention, making it likely to succeed in market dissemination. The
proposed method considers changing intervals of the surgical instruments (for
the connected set of the shaft and the metallic clasper) while moving in a
surgical environment cluttered with obstacles (sensitive tissues, veins) and
uses lower dimensional results to track surgical instruments in higher
dimensions. Synthesized and experimental results in 2D and 3D motion estimates
show that errors are negligible for the method to be used for dataset
labelling and motion tracking of instrument for futuristic applications. Due
to the use of simple computations, future developments in this method will
focus on the real-time implementation of a 3D tracking system. It is
interesting to note that connected parts angular motion is independent of each
other, and a methodology to extract 3D independent motions of each part from
their 2D motion in image frames has multiple solutions giving an opportunity
for future investigations. Moreover, we will focus on developing algorithms
that effectively track the dynamic reference frame’s motion and consistently
transform tool poses to the desired fixed reference frame, ensuring robust and
reliable results.

## Acknowledgments

Financial support for this study was provided by Prime Minister’s Research
Fellows (PMRF) scheme, India (PMRF Id no. 1300229 dated May 2019) for pursuing
research in higher educational institutions in India. https://www.pmrf.in/.
Thanks are also due to Indian Institute of Technology, Bombay, India.

## Statements and Declarations

### Funding:

This work was supported by Prime Minister’s Research Fellows (PMRF) scheme.
PMRF Id no. 1300229 awarded to Shubhangi Nema.

### Conflict of interest disclosure:

The authors declare that they have no conflict of interest.

### Ethical approval:

This article does not contain any studies with human participants or animals
performed by any of the authors.

### Authors’ contributions:

All authors contributed to the study conception and design. Finding research
gap, idea generation, data collection, experiments, analysis, and
interpretation of results were performed by Shubhangi Nema and Abhishek
Mathur. Leena Vachhani advised on the findings of this work, reviewed the
methods, and agreed on the novelty of the article before submission. The first
draft of the manuscript was written by Shubhangi Nema and all authors
commented on previous versions of the manuscript. All authors read and
approved the final manuscript.

### Data availability:

The author confirms that all data generated or analysed during this study are
included in this published article.

## Bibliography

  * 1 Galuret S, Vallée N, Tronchot A, Thomazeau H, Jannin P, Huaulmé A. Gaze behavior is related to objective technical skills assessment during virtual reality simulator-based surgical training: a proof of concept. International Journal of Computer Assisted Radiology and Surgery 2023: 1–9.
  * 2 Basdogan C, Sedef M, Harders M, Wesarg S. VR-based simulators for training in minimally invasive surgery. IEEE Computer Graphics and Applications 2007; 27(2): 54–66.
  * 3 Suh I, Mukherjee M, Oleynikov D, Siu KC. Training program for fundamental surgical skill in robotic laparoscopic surgery. The International Journal of Medical Robotics and Computer Assisted Surgery 2011; 7(3): 327–333.
  * 4 Beulens AJ, Namba HF, Brinkman WM, et al. Analysis of the video motion tracking system “Kinovea” to assess surgical movements during robot-assisted radical prostatectomy. The International Journal of Medical Robotics and Computer Assisted Surgery 2020; 16(2): e2090.
  * 5 Nema S, Vachhani L. Surgical instrument detection and tracking technologies: Automating dataset labeling for surgical skill assessment. Frontiers in Robotics and AI 2010: 308.
  * 6 Speidel S. Endovis sub-challenge: Instrument segmentation and tracking.. https://endovissub-instrument.grand-challenge.org/Data/; 2012-22.
  * 7 Pakhomov D, Premachandran V, Allan M, Azizian M, Navab N. Deep residual learning for instrument segmentation in robotic surgery. In: International Workshop on Machine Learning in Medical ImagingSpringer. ; 2019: 566–573.
  * 8 Shvets AA, Rakhlin A, Kalinin AA, Iglovikov VI. Automatic instrument segmentation in robot-assisted surgery using deep learning. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA)IEEE. ; 2018: 624–628.
  * 9 Nema S, Vachhani L. Unpaired deep adversarial learning for multi-class segmentation of instruments in robot-assisted surgical videos. The International Journal of Medical Robotics and Computer Assisted Surgery 2010: e2514.
  * 10 Zhang X, Payandeh S. Application of visual tracking for robot-assisted laparoscopic surgery. Journal of Robotic systems 2002; 19(7): 315–328.
  * 11 Schulze F, Bühler K, Neubauer A, Kanitsar A, Holton L, Wolfsberger S. Intra-operative virtual endoscopy for image guided endonasal transsphenoidal pituitary surgery. International journal of computer assisted radiology and surgery 2010; 5(2): 143–154.
  * 12 Choi C, Kim J, Han H, Ahn B, Kim J. Graphic and haptic modelling of the oesophagus for VR-based medical simulation. The International Journal of Medical Robotics and Computer Assisted Surgery 2009; 5(3): 257–266.
  * 13 Zhang L, Ye M, Chan PL, Yang GZ. Real-time surgical tool tracking and pose estimation using a hybrid cylindrical marker. International journal of computer assisted radiology and surgery 2017; 12: 921–930.
  * 14 Krieg J. Motion tracking: Polhemus technology. Virtual Reality Systems 1993; 1(1): 32–36.
  * 15 Trejos AL, Patel RV, Naish MD, Schlachta CM. Design of a sensorized instrument for skills assessment and training in minimally invasive surgery. In: 2008 2nd IEEE RAS & EMBS International Conference on Biomedical Robotics and BiomechatronicsIEEE. ; 2008: 965–970.
  * 16 Yamaguchi S, Yoshida D, Kenmotsu H, et al. Objective assessment of laparoscopic suturing skills using a motion-tracking system. Surgical endoscopy 2011; 25(3): 771–775.
  * 17 Stoll J, Novotny P, Howe R, Dupont P. Real-time 3D ultrasound-based servoing of a surgical instrument. In: Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006.IEEE. ; 2006: 613–618.
  * 18 Halic T, Kockara S, Bayrak C, Rowe R. Mixed reality simulation of rasping procedure in artificial cervical disc replacement (ACDR) surgery. In: BMC bioinformatics. 11. Springer. ; 2010: 1–17.
  * 19 Seslija P, Habets DF, Peters TM, Holdsworth DW. Feasibility of 3D tracking of surgical tools using 2D single plane X-ray projections. In: Medical Imaging 2008: Visualization, Image-Guided Procedures, and Modeling. 6918. SPIE. ; 2008: 212–221.
  * 20 Duan Z, Yuan Z, Liao X, Si W, Zhao J. 3D tracking and positioning of surgical instruments in virtual surgery simulation.. Journal of multimedia 2011; 6(6).
  * 21 Gautier B, Tugal H, Tang B, Nabi G, Erden MS. Real-time 3D tracking of laparoscopy training instruments for assessment and feedback. Frontiers in Robotics and AI 2021; 8: 751741.
  * 22 Chen L, Ma L, Zhang F, Yang X, Sun L. An intelligent tracking system for surgical instruments in complex surgical environment. Expert Systems with Applications 2023: 120743.
  * 23 Jiang X, Zheng B, Atkins MS. Video processing to locate the tooltip position in surgical eye–hand coordination tasks. Surgical innovation 2015; 22(3): 285–293.
  * 24 Allen BF, Kasper F, Nataneli G, Dutson E, Faloutsos P. Visual tracking of laparoscopic instruments in standard training environments. In: IOS Press. 2011 (pp. 11–17).
  * 25 Du X, Allan M, Dore A, et al. Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery. International journal of computer assisted radiology and surgery 2016; 11(6): 1109–1119.
  * 26 García-Peraza-Herrera LC, Li W, Gruijthuijsen C, et al. Real-time segmentation of non-rigid surgical tools based on deep learning and tracking. In: International Workshop on Computer-Assisted and Robotic EndoscopySpringer. ; 2016: 84–95.
  * 27 Cheng T, Li W, Ng WY, et al. Deep learning assisted robotic magnetic anchored and guided endoscope for real-time instrument tracking. IEEE Robotics and Automation Letters 2021; 6(2): 3979–3986.
  * 28 Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted interventionSpringer. ; 2015: 234–241.
  * 29 Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014\.
  * 30 Nema S, Vachhani L. Safe and Fast Path Planner for Minimally Invasive Surgery. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)IEEE. ; 2021: 2549–2554.