# Extraction of Key-frames of Endoscopic Videos by using Depth Information

Pradipta Sasmal, Avinash Paul, M.K. Bhuyan,, and Yuji Iwahori P. Sasmal,
Avinash Paul and M.K. Bhuyan are with the Department of Electronics and
Electrical Engineering, Indian Institute of Technology Guwahati, India.Yuji
Iwahori is with Department of Computer Science, Chubu University, Kasugai,
Japan. *Author 1 and Author 2 have equal contribution. E-mails: (s.pradipta,
paul18, and<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A deep learning-based monocular depth estimation (MDE) technique is proposed
for selection of most informative frames (key frames) of an endoscopic video.
In most of the cases, ground truth depth maps of polyps are not readily
available and that is why the transfer learning approach is adopted in our
method. An endoscopic modalities generally capture thousands of frames. In
this scenario, it is quite important to discard low-quality and clinically
irrelevant frames of an endoscopic video while the most informative frames
should be retained for clinical diagnosis. In this view, a key-frame selection
strategy is proposed by utilizing the depth information of polyps. In our
method, image moment, edge magnitude, and key-points are considered for
adaptively selecting the key frames. One important application of our proposed
method could be the 3D reconstruction of polyps with the help of extracted key
frames. Also, polyps are localized with the help of extracted depth maps.

###### Index Terms:

Polyps, Monocular depth, Key-frames.

##  I Introduction

Wireless Capsule Endoscopy (WCE) is a non-invasive modality to monitor the
conditions of the internal viscera of a human body. WCE moves along the
gastro-intestinal (GI) tract to capture images. It is extensively used to
detect polyps in colon regions, which become cancerous if left untreated.
Colorectal cancer is the third most prevalent cancer today [1]. The capsule
moves under the peristalsis movement, and it is very difficult to control the
motion and orientation of the camera. Thus, redundant and clinically non-
significant frames are generally obtained in a video sequence. WCE takes
nearly 8 hours, capturing close to 50000 frames. A large part of the data is
clinically not significant and needs to be removed [2].

Several methods have been proposed for detection and localization of polyps in
endoscopy frame [3][4]. A recent work focusing on video summarization instead
of anomalies detection like bleeding or ulceration is proposed by Li et al.
[5]. Iakovidis et al. [6] used clustering-based methods for video
summarization. Researchers are working on visual attention models, like
saliency maps for finding key-frames of videos [7]. Malignant polyps usually
have a convex shape and are more textured compared to benign polyps. Getting a
3D view of the polyp surface can greatly help in resection [8]. A good 3D
reconstruction of an object in an image entails dense depth estimation. The 3D
view gives shape and size information of a polyp. Depth estimation of
endoscopic images is a challenging task as the endoscopic images are
monocular.

Eigen et al., [9] introduced a multi-scale information approach which takes
care of both global scene structure and local neighboring pixel information. A
scale-invariant loss is used for MDE. Similarly, Xu et al. [10] formulated MDE
as a continuous random field problem (CRF). They fused the multi-scale
estimation computed from the inner semantic layers of a CNN with a CRF
framework. Instead of finding continuous depth maps, Fu et al. [11] estimated
depth using an ordinal regression approach.

Depth is generally obtained using sensors like LIDAR, Kinect, or by using
stereo cameras. Sensors are expensive and stereo cameras are not generally
used in endoscopy due to several restrictions. Obtaining ground-truth training
data for depth estimation is very difficult in endoscopic imaging, and so,
supervised methods are not feasible for endoscopic image classification.
Finding correspondence between two images for 3D reconstruction is also
difficult in endoscopy videos. It is quite difficult to find corresponding
features across the frames.

Hence, unsupervised and semi-supervised methods are employed for MDE. Garg et
al. [12] used binocular stereo image pairs for the training of CNNs and then
minimized a loss function formed by the wrapping of the left view image into
its right of the stereo pair. Godard et al. [13] improved this method by using
the left-right consistency criterion. They trained CNNs on stereo images but
used a single image for inference. They introduced a new CNN architecture that
computes end-to-end MDE. The network was trained with an efficient
reconstruction loss function. The state-of-the-art unsupervised MDE method,
i.e., Monodepth [13] model has limited application in in-vivo images like
endoscopic images. This is due to the fact that most models leverage outdoor
scenes [14] and a few indoor scenes [15] for training, and they use high-end
sensors or stereo cameras, while the WCE method only captures monocular
images. Hence, it is important to devise a method that can perform MDE in
medical imaging datasets which generally do not have ground truth depth
information. That is why, a transfer learning approach is adopted in our
method for estimating depth. Transfer learning refers to a learning method
where what has been learned in one setting is exploited to improve
generalization in another setting [16]. Zero-shot learning is the extreme case
of transfer learning where no labeled examples are present. In our method, a
zero-shot learning approach for MDE [17] is employed.

The proposed method consists of two main steps. The first step focuses on
depth estimation, and the second step extracts key-frames. As mentioned above,
a zero-shot learning approach is adopted for depth estimation in endoscopic
videos. We propose a framework to select the most informative frames of an
endoscopic video sequence. Our method employs a three criteria approach to
identify the key-frames. Subsequently, these key-frames can be used for 3D
reconstruction. Our method is unique in a sense that it considers depth
information to find key-frames. Finally, any of the selected key-frames can
then be used for 3D reconstruction using a GUI. Experimental results clearly
demonstrate the effectiveness of our method in selecting the key-frames, and
subsequent polyp visualization.

##  II Proposed method

###  II-A Depth estimation

Due to insufficient ground truth depth data in endoscopy video datasets, a
transfer learning approach is adopted for MDE in our proposed method. Lasinger
et al. [17] proposed a zero-shot learning for depth estimation. We used a pre-
trained model trained on diverse datasets by Lasinger et al. [17] in our work.
The model was trained for depth maps obtained in three different ways. First,
the dataset contains depth maps obtained using LIDAR sensors. This method
gives depth maps of high quality. Second, the Structure from Motion (SfM)
approach is employed to estimate the depth. The third method of getting depth
information from stereo images of 3D movies dataset. It uses optical flow to
find motion vectors from each of the stereo images. Then, the left-right image
disparity is used to find a depth map.

Figure 1: Proposed method of finding key-frames

Pre-trained network architecture. A ResNet- based architecture as proposed by
Xian et al. [18] is used for depth estimation. Adam optimizer is used with a
learning rate of $10^{-4}$ for layers which are randomly initiated and
$10^{-5}$ for layers initialized with pre-trained weights. Decay rates for the
optimizer are set at $\beta_{1}=.9$ and $\beta_{2}=.999$, training uses a
batch size of 8. Due to different image aspect ratios, images are cropped and
augmented for training.

Loss function. A shift and scale invariant loss function is chosen to address
the problems pertaining to training on three different datasets. Let
$\mathbf{d}$ $\in$ $\mathbb{R}^{\mathit{N}}$ be the computed inverse depth and
$\mathbf{d^{\prime}}$ $\in$ $\mathbb{R}^{\mathit{N}}$ be ground truth inverse
depth, where $\mathit{N}$ is the number of pixels in a frame. Here $s$ and $t$
represent scale and shift, respectively and they are positive real numbers.
This can be represented in a vector form by taking
$\vec{\mathbf{d}}_{i}$=$(\mathbf{d}_{i},1)^{\intercal}$ and
$\mathbf{p}$=$(s,t)^{\intercal}$ and thus the loss function becomes:

$\displaystyle\mathcal{L}(\mathbf{d}_{i},\mathbf{d^{\prime}}_{i})=\underset{\mathbf{p}}{\text{min}}$
$\displaystyle\dfrac{1}{2\mathit{N}}\sum_{i=1}^{\mathit{N}}({\vec{\mathbf{d}}_{i}}^{\intercal}\mathbf{p}-\mathbf{d^{\prime}}_{i})^{2}$
(1)

The closed-form solution is given as:

$\displaystyle\mathbf{p}^{opt}=(\sum_{i=1}^{\mathit{N}}\vec{\mathbf{d}}_{i}{\vec{\mathbf{d}}_{i}}^{\intercal})^{-1}(\sum_{i=1}^{\mathit{N}}\vec{\mathbf{d}}_{i}\mathbf{d^{\prime}}_{i})$
(2)

Substituting $\mathbf{p}^{opt}$ into (1) we get:

$\displaystyle\mathcal{L}(\mathbf{d}_{i},\mathbf{d^{\prime}}_{i})=\underset{\mathbf{p}}{\text{min}}$
$\displaystyle\dfrac{1}{2\mathit{N}}\sum_{i=1}^{\mathit{N}}({\vec{\mathbf{d}}_{i}}^{\intercal}\mathbf{p}^{opt}-\mathbf{d^{\prime}}_{i})^{2}$
(3)

Regularization term. A multi-scale scale-invariant regularization term is used
which does gradient matching to the depth inverse space. This biases
discontinuities to be sharp and coincide with ground truth discontinuities.
The regularization term can be defined as,

$\mathcal{L}_{r}(\mathbf{d}_{i},\mathbf{d^{\prime}}_{i})=\dfrac{1}{\mathit{N}}\sum_{j=1}^{k}\sum_{i=1}^{\mathit{N}}(|\Delta_{x}{Q_{i}}^{k}|+|\Delta_{y}{Q_{i}}^{k}|)$
(4)

where,

$Q_{i}={\vec{\mathbf{d}}_{i}}^{\intercal}\mathbf{p}^{opt}-\mathbf{d^{\prime}}_{i}$
(5)

Here $Q^{k}$ gives the difference of inverse depth maps at a scale $k$. Also,
the scale is applied before finding $x$ and $y$ gradients.

Modified loss function. The final loss function for a training set of size
$M$, taking into consideration of the regularization term becomes:

$\mathcal{L}_{final}=\dfrac{1}{M}\sum_{i=1}^{M}\mathcal{L}(\mathbf{d}^{i},(\mathbf{d^{\prime}})^{i})+\alpha\mathcal{L}_{r}(\mathbf{d}^{i},(\mathbf{d^{\prime}})^{i})$
(6)

Here $\alpha$ is taken as 0.5.

###  II-B Selection of key-frames

During the colonoscopy, not all the captured frames are clinically
significant. Most of the frames may have redundant information, or may not be
useful from a diagnostic perspective. Such frames need to be discarded and the
clinically informative frames need to be retained. It is also strenuous and
computationally intensive for a physician to investigate each frame of a video
sequence. Thus, we propose a key-frame selection technique. Subsequently, 3D
reconstruction is done to perform further analysis of the polyps. The key-
frame selection method is given in Fig. 1.

Colour space conversion. Our dataset contains images which are in RGB color
space. Taking clues from the human visual system which works on saliency, we
changed the color space from RGB to COC which gives a better perception in the
medical imaging [19].

The image is subsequently used to find key-frames. A frame should satisfy
three criteria before being selected as a key-frame. Firstly, it should be
significantly different from neighboring frames. Second, the key-frame should
give significant depth information of a polyp. Third, the polyp should not be
occluded in the key-frame. We ensured that the above requirements are met, and
they are formulated as follows:

Image moment: Image moments give the information of the shape of a region
along with its boundaries and texture. Hu moments [20] are considered as they
are invariant to affine transformation, and moment distances of consecutive
frames are used to identify the redundant frames of a video. The frames with a
higher moment distance will be considered as a key frame. The moment distance
$d$ between two images is calculated as:

$d=\sum_{i=1}^{i=7}{(I_{i}-I^{\prime}_{i})^{2}}$ (7)

Figure 2: Plot of Moment distance, Edge density, Number of key-points and the
total fused score vs frame number.

Edge density: In our proposed method, the key-frames which have significant
depth information are only considered for 3D reconstruction of a polyp. It is
observed that the polyp images having more edges have more depth information.
The edge information can be obtained with the help of the gradient magnitude
of an image. Before finding the gradients, images were smoothed using a
Gaussian kernel.

Horizontal and vertical gradients are obtained using Sobel operators $S_{x}$
and $S_{y}$ and then the gradient magnitude $\Delta S$ is calculated as
follows:

$\Delta S=\sqrt{(S_{x})^{2}+(S_{y})^{2}}$ (8)

Key-point detection: The proposed moment-based key-frame detection method may
capture some occluded frames. So, the objective is to select non-occluded key-
frames from a group of key-frames which were extracted by our proposed image
moment and edge density-based criteria. For this, a key-point detection based
technique is used.

For key-point detection and extraction, we used ORB (Oriented FAST and Rotated
BRIEF). ORB operates on an image to obtain a scale pyramid. It also takes into
account the orientation of the image patch around the key point. Moreover, ORB
is computationally faster and robust to noises in endoscopic images. The
frames containing a lesser number of ORB points correspond to occluded polyps.

Adaptive key-frame selection. After finding the moment distance (d), edge
magnitude (s), and the number of ORB points (p), we normalize these scores
using min-max normalization.

The variable having greater variance is given more weight-age. Here, $w_{i}$
is the weight of the normalized score. To consider intra-variable changes, we
used the sum of the magnitude of difference between consecutive frame scores
as a measure to find weights. We then normalized this score to be used as
weights for finding a fused score. The weights are given by:

$\displaystyle
d_{1}=\sum_{i=1}^{n}{|d_{i}-d^{\prime}_{i}|},s_{1}=\sum_{i=1}^{n}{|s_{i}-s^{\prime}_{i}|},p_{1}=\sum_{i=1}^{n}{|p_{i}-p^{\prime}_{i}|}$
(9) $\displaystyle
w_{1}=\dfrac{d_{1}}{d_{1}+s_{1}+p_{1}},w_{2}=\dfrac{s_{1}}{d_{1}+s_{1}+p_{1}},w_{3}=\dfrac{p_{1}}{d_{1}+s_{1}+p_{1}}$
(10) $f=w_{1}d_{1}+w_{2}s_{1}+w_{3}p_{1}$ (11)

Here, $d_{1}$, $s_{1}$, $p_{1}$ are the sum of magnitudes of difference
between consecutive frame scores and $f$ is the fused score obtained by
adaptively weighting the three frame scores. The frames with the highest fused
scores are selected according to a threshold value. The variance of each
criterion with frame number is shown in Fig. 2.

##  III Experimental Results

The proposed method is evaluated on the publicly available dataset. This
dataset contains colonoscopic video sequences from three classes, namely
adenoma, serrated and hyperplasic. The adenoma class contains 40 sequences,
serrated contains 15, while hyperplasic contains 21 sequences [21]. In this
work, we consider only the frames from the adenoma (malignant) class because
this class needs the maximum attention of the physician.

Figure 3: Key-frames obtained by our method and their corresponding depth
maps. The polyp is visible from different viewing angles in these selected
frames.

Our method performs better than the state-of-the-art MDE methods. The depth
estimation results are shown in Fig. 4, where the first column represents the
input images, while the second and the third column show the comparative
results between monodepth model [13] and zero-shot cross-dataset transfer pre-
trained model [17]. This clearly shows that monodepth performs well in outdoor
environments than our method. However, the Zero-shot learning method is more
accurate in predicting depth in endoscopic images.

Input image

Monodepth [13]

Zero-shot [17]

Figure 4: Comparison of MDE on two input images, one outdoor and the other one
is an endoscopy image. The depth map by Monodepth [13] performs well for
outdoor environment while giving unsatisfactory results for the endoscopy
image . However, the zero-shot learning method [17] clearly performs well for
medical images but cannot accurately estimate the depth in outdoor scenes.
Figure 5: Polyp boundary detection using depth map; Column 1: Original
endoscopic image, Column 2: Generated depth maps, Column 3: Detected polyp
boundary using canny edge detection algorithm, Column 4: Edge refinement using
connected component analysis. First three rows of image samples are taken from
CVC clinic database [22], the last two rows of images are frames taken from a
video sequence of the publicly available dataset [21].

Our method is the first-of-its-kind in which key-frames are extracted from an
endoscopic video using depth maps. Also, it is robust to occlusions. As
redundant frames are discarded in our method, it is more convenient for
physicians to analyze important frames of a video sequence. As explained
earlier, the moment distance criterion between consecutive frames is used to
ensure that redundant frames are identified, and then discarded. The edge
magnitude criterion leverages the depth images data to select the best frames.
Frames with fewer ORB points have occluded polyps and these frames are
redundant. Adaptive thresholding is used to apply three criteria to obtain
essential frames for 3D reconstruction.

The selected key-frames are finally used to reconstruct the 3D surface of the
polyp. We have used Facebook’s 3D image GUI to view the reconstructed polyp
surface, the link to the video is shown here: $\href
https://youtu.be/PJKfk0Mqu2I$. 3D visualization of a polyp helps in surgeries
involving the removal of the polyp from its root. This gives better
visualization of polyps for diagnosis. Fig. 3 shows some of the results of
key-frame extraction and the corresponding depth maps. No publicly available
datasets or methods using them that predict depth maps from endoscopic frames
exist. Thus, a comparison between different methods for predicting depth from
endoscopic images couldn’t be performed.

Another application of our proposed method could be automatic segmentation of
polyps in endoscopic images. The depth maps generated by our proposed method
can further be used for polyp localization. The canny edge detector is used
over the depth maps and subsequently, polyp boundary is determined by using
connected component analysis. Fig. 5 shows localized polyps in some of the
endoscopic image samples. The segmentation performance on some of the
sequences of the CVC clinic database is shown in Table I. We defined mIoU as
the mean intersection over the union of the segmented polyp masks to the
ground truth masks. In polyp segmentation, an IoU score of $\geq$ 0.5 is
generally considered good [23].

TABLE I: Key frame selection and segmentation performance using our method on some of the sequences of CVC clinic database (Sequences with only the elevated polyps are considered) Sequence | #Key frames | mIoU $>$ 0.5?
---|---|---
104-126 | 7 | Yes
127-151 | 11 | Yes
298-317 | 2 | Yes
343-363 | 7 | Yes
384-408 | 13 | Yes
409-428 | 8 | Yes
479-503 | 20 | Yes
504-528 | 6 | Yes
572-591 | 4 | Yes
592-612 | 5 | Yes

##  IV Conclusion

Our proposed method can determine depth maps using a zero-shot learning
approach. The essential frames are picked out from WCE videos with the help of
depth information and the proposed three criteria selection strategy. The
selection of a threshold value for the final fused score must be empirically
set to extract the key-frames. Experimental results show the efficacy of the
proposed method in selecting key frames from endoscopic videos and subsequent
segmentation of detected polyps in the key frames with the help of extracted
depth maps. Also, the 3D model could be used in clinical diagnosis and
surgeries. One possible extension of this work could be the visualization of
polyps in detected key frames in an augmented reality framework.

## Acknowledgement

The work of Yuji Iwahori was supported by the JSPS Grant-in-Aid Scientific
Research (C) under Grant 20K11873 for the endoscope and other medical image
researches

## References

  * [1] R. L. Siegel, K. D. Miller, S. A. Fedewa, D. J. Ahnen, R. G. Meester, A. Barzi, and A. Jemal, “Colorectal cancer statistics, 2017,” _CA: a cancer journal for clinicians_ , vol. 67, no. 3, pp. 177–193, 2017.
  * [2] H.-G. Lee, M.-K. Choi, B.-S. Shin, and S.-C. Lee, “Reducing redundancy in wireless capsule endoscopy videos,” _Computers in biology and medicine_ , vol. 43, no. 6, pp. 670–682, 2013.
  * [3] B.-P. Li and M. Q.-H. Meng, “Comparison of several texture features for tumor detection in ce images,” _Journal of medical systems_ , vol. 36, no. 4, pp. 2463–2469, 2012.
  * [4] M. Tjoa, S. Krishnan, and R. Doraiswami, “Automated diagnosis for segmentation of colonoscopic images using chromatic features,” in _IEEE CCECE2002. Canadian Conference on Electrical and Computer Engineering. Conference Proceedings (Cat. No. 02CH37373)_ , vol. 2. IEEE, 2002, pp. 1177–1180.
  * [5] B. Li, M. Q.-H. Meng, and Q. Zhao, “Wireless capsule endoscopy video summary,” in _2010 IEEE International Conference on Robotics and Biomimetics_. IEEE, 2010, pp. 454–459.
  * [6] D. K. Iakovidis, S. Tsevas, and A. Polydorou, “Reduction of capsule endoscopy reading times by unsupervised image mining,” _Computerized Medical Imaging and Graphics_ , vol. 34, no. 6, pp. 471–478, 2010.
  * [7] X.-s. Hua, L. Lu, H.-j. Zhang, and H. District, “A generic framework of user attention model and its application in video summarization,” _IEEE Transaction on multimedia_ , vol. 7, no. 5, pp. 907–919, 2005.
  * [8] R. Law, A. Das, D. Gregory, S. Komanduri, R. Muthusamy, A. Rastogi, J. Vargo, M. B. Wallace, G. S. Raju, R. Mounzer _et al._ , “Endoscopic resection is cost-effective compared with laparoscopic resection in the management of complex colon polyps: an economic analysis,” _Gastrointestinal endoscopy_ , vol. 83, no. 6, pp. 1248–1257, 2016.
  * [9] D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in _Advances in neural information processing systems_ , 2014, pp. 2366–2374.
  * [10] E. Ricci, W. Ouyang, X. Wang, N. Sebe _et al._ , “Monocular depth estimation using multi-scale continuous crfs as sequential deep networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 6, pp. 1426–1440, 2018.
  * [11] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2002–2011.
  * [12] R. Garg, V. K. BG, G. Carneiro, and I. Reid, “Unsupervised cnn for single view depth estimation: Geometry to the rescue,” in _European Conference on Computer Vision_. Springer, 2016, pp. 740–756.
  * [13] C. Godard, O. Mac Aodha, and G. J. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 270–279.
  * [14] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving,” in _Proc. CVPR_ , pp. 3354–3361.
  * [15] A. Saxena, M. Sun, and A. Y. Ng, “Learning 3-d scene structure from a single still image,” in _2007 IEEE 11th International Conference on Computer Vision_. IEEE, 2007, pp. 1–8.
  * [16] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [17] K. Lasinger, R. Ranftl, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” _arXiv preprint arXiv:1907.01341_ , 2019.
  * [18] K. Xian, C. Shen, Z. Cao, H. Lu, Y. Xiao, R. Li, and Z. Luo, “Monocular relative depth perception with web stereo data supervision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 311–320.
  * [19] S. Engel, X. Zhang, and B. Wandell, “Colour tuning in human visual cortex measured with functional magnetic resonance imaging,” _Nature_ , vol. 388, no. 6637, p. 68, 1997.
  * [20] M.-K. Hu, “Visual pattern recognition by moment invariants,” _IRE transactions on information theory_ , vol. 8, no. 2, pp. 179–187, 1962.
  * [21] P. Mesejo, D. Pizarro, A. Abergel, O. Rouquette, S. Beorchia, L. Poincloux, and A. Bartoli, “Computer-aided classification of gastrointestinal lesions in regular colonoscopy,” _IEEE transactions on medical imaging_ , vol. 35, no. 9, pp. 2051–2063, 2016.
  * [22] J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, D. Gil, C. Rodríguez, and F. Vilariño, “Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians,” _Computerized Medical Imaging and Graphics_ , vol. 43, pp. 99–111, 2015.
  * [23] M. Yamada, Y. Saito, H. Imaoka, M. Saiko, S. Yamada, H. Kondo, H. Takamaru, T. Sakamoto, J. Sese, A. Kuchiba _et al._ , “Development of a real-time endoscopic image diagnosis support system using deep learning technology in colonoscopy,” _Scientific reports_ , vol. 9, no. 1, pp. 1–9, 2019.


# AutoMRISimQA: an automated system for daily quality control of a 3T MRI
simulator

Aitang Xing  Liverpool & Macarthur Cancer Therapy Centre, Liverpool, NSW 1871,
Australia  Ingham Institute for Applied Medical Research, Liverpool, NSW 1871,
Australia  South West Sydney Clinical Campuses, University of New South Wales,
Sydney, Australia  Gary Goozee  Liverpool & Macarthur Cancer Therapy Centre,
Liverpool, NSW 1871, Australia  Ingham Institute for Applied Medical Research,
Liverpool, NSW 1871, Australia  South West Sydney Clinical Campuses,
University of New South Wales, Sydney, Australia  Gary Liney  Liverpool &
Macarthur Cancer Therapy Centre, Liverpool, NSW 1871, Australia  Ingham
Institute for Applied Medical Research, Liverpool, NSW 1871, Australia  Sankar
Arumugam  Liverpool & Macarthur Cancer Therapy Centre, Liverpool, NSW 1871,
Australia  Ingham Institute for Applied Medical Research, Liverpool, NSW 1871,
Australia  South West Sydney Clinical Campuses, University of New South Wales,
Sydney, Australia  Shrikant Deshpande  Liverpool & Macarthur Cancer Therapy
Centre, Liverpool, NSW 1871, Australia  Ingham Institute for Applied Medical
Research, Liverpool, NSW 1871, Australia  South West Sydney Clinical Campuses,
University of New South Wales, Sydney, Australia  Anthony Espinoza  Liverpool
& Macarthur Cancer Therapy Centre, Liverpool, NSW 1871, Australia  Ingham
Institute for Applied Medical Research, Liverpool, NSW 1871, Australia  South
West Sydney Clinical Campuses, University of New South Wales, Sydney,
Australia  Alison Gray  Liverpool & Macarthur Cancer Therapy Centre,
Liverpool, NSW 1871, Australia  Ingham Institute for Applied Medical Research,
Liverpool, NSW 1871, Australia  South West Sydney Clinical Campuses,
University of New South Wales, Sydney, Australia  Vasilis Kondilis  Liverpool
& Macarthur Cancer Therapy Centre, Liverpool, NSW 1871, Australia  Doaa
Elwadia  Liverpool & Macarthur Cancer Therapy Centre, Liverpool, NSW 1871,
Australia  Ingham Institute for Applied Medical Research, Liverpool, NSW 1871,
Australia  Robba Rai  Liverpool & Macarthur Cancer Therapy Centre, Liverpool,
NSW 1871, Australia  Ingham Institute for Applied Medical Research, Liverpool,
NSW 1871, Australia  Lois Holloway  Liverpool & Macarthur Cancer Therapy
Centre, Liverpool, NSW 1871, Australia  Ingham Institute for Applied Medical
Research, Liverpool, NSW 1871, Australia  South West Sydney Clinical Campuses,
University of New South Wales, Sydney, Australia

Abstract A software system named AutoMRISimQA was developed to monitor the
daily performance of a wide-bore 3T scanner (MRI) which was designed and
dedicated to radiotherapy simulation. AutoMRISimQA can monitor the performance
of the MRI simulator not only by using image quality indices such as signal-
to-noise ratio (SNR), uniformity, ghosting, and contrast but also performing a
quick check of geometric accuracy as well as the external lasers
quantitatively. It was implemented into the daily clinical workflow in 2013
and has been used for more than 10 years. It was also seamlessly integrated
with QATrack+, allowing continuous monitoring of the consistency of the MRI
simulator’s performance.
---

##  1 Introduction

MRI simulator (MRISim) is a magnetic resonant imaging (MRI) scanner designed
and dedicated for radiotherapy planning, dose calculation and treatment
response [1, 2]. The use of MRI in radiotherapy requires the scanner to be
more stable and reproducible than the one used for diagnostic purposes. A
recently published TG 284 report recommended a list of tests to be done daily,
but how to perform these checks depends on the individual centre [3].

MRISim was commissioned and integrated into the routine workflow in 2013 in
our centre [4, 5]. A fully automated daily quality control (QC) program was
developed and has been running for more than 10 years. The purpose of this
paper is to report how the system was designed, implemented and integrated
into a radiation oncology workflow.

##  2 Materials and Methods

###  2.1 MRI Simulator and phantom

MRI simulator is a wide-bore 3T MRI scanner (Siemens 3T Skyra, Erlangen,
Germany) equipped with an external simulation laser and a flat couch [5]. The
phantom is the AQUARIUS phantom (MRI-0005, S/N 0010260016, LAP of America,
Laser application, USA), which was originally designed for checking the
alignment of simulation lasers [4]. The phantom is cylindrical, and inside the
phantom, there are four long tubes filled with CuSO4 solution and the rest is
filled with water. As shown in Figure 1, the phantom was set on the flat table
aligned with external lasers and then scanned using a T1 weighted spin echo
sequence (echo time(TE) =30ms and repetition time(TR)=3410ms) daily. The MRI
images were transferred to a shared network location on a server. An in-house
system, named AutoMRISimQA was developed to monitor the MRI simulator’s daily
performance using phantom images.

Figure 1: Siemens Skyra 3T MRI Simulator with a flat couch top. The AQUARIUS
phantom was set up on the couch and aligned with external lasers.

###  2.2 Design of AutoMRISimQA system

The system was designed and divided into six modules. Figure LABEL:fig:module
showes the overall structure of the system. The first module is used to detect
if a new image series arrives. If the new images were found, the second module
will be triggered to sort image series, according to the acquisition date and
time. The image pixel data were also extracted. The second module uses these
data to calculate image quality indices such as signal-to-noise ratio(SNR),
which characterize the performance of the MRI scanner. The calculated QC
parameters are passed to the third module, whose role is to make a logical
judgement. The parameters are compared with their corresponding reference
values and tolerance. If any of the QC parameters are out of tolerance, an
alert e-mail will be automatically sent to the responsible physicists. A PDF
report was also generated at the same time to assist physicists in quickly
diagnosing where the problem is. At our centre, all QC data for radiotherapy
equipment were managed using QATrack+ [6], an open-source web-based system
designed for radiotherapy quality control. The QATrack module is used to
automatically update the database of QATrack+.

###  2.3 Implementation of the system

Matlab, Java and Python were used for system implementation. As QATrack+ was
implemented in pure Python, the module for updating the QATrack+ database was
written using Python. The access to the database was through the Python
representational state transfer(REST)framework [7]. In this way, the risk of
accidental damage to the database was minimized. Except for the report module,
the rest of the system was implemented using Matlab. A rich of imaging process
functions provided by Matlab’s image processing toolbox is the major
consideration of using Matlab as the main programming language. Matlab also
can seamlessly integrate Java applications into the Matlab program. The report
module was written in Java using the iText package [8]and integrated into
Matlab.

The core component of the system is the one used for calculating the QC
parameters from the phantom images. The image slice shown in Figure
LABEL:fig:phantom (A) was used to calculate the uniformity and SNR using the
four $1cm^{2}$ red circle regions. The percentage intensity uniformity (PIU)
is defined as

$PIU=100\times(1-\frac{S_{max}-S_{min}}{S_{max}+S_{min}})$ (1)

Where $S_{max}$ and $S_{min}$ represent the maximum and minimum signal
averaged over four red circles.

The SNR is calculated as the ratio of the mean signal averaged over four areas
defined by the red circles to the standard deviation over the same ROI.
Another parameter, termed output (MRI signal), is defined as the ratio of the
standard deviation to the mean of the pixel values over the red circle ROI at
the centre of the images.

The ghosting effect was calculated using the image slice and eight elliptical
regions as indicated in Figure LABEL:fig:phantom(B). The ghosting ratio (GR)
was calculated as:

$GR=\frac{\left|\left(\left(S_{top}+S_{btm}\right)-\left(S_{top}+S_{btm}\right)\right)\right|}{2S_{mean}}$
(2)

Where $S_{top}$ , $S_{btm}$ , $S_{left}$ and $S_{right}$ are the signals
averaged large eclipses at the top, bottom, right and left of the images,
respectively. $S_{mean}$ is the mean signal averaged over four small
ellipsoids at the center of the image.

The contrast was calculated as the ratio of the mean signal value over areas
defined by the four green circles in high contrast regions to the mean signal
value determined over the four red circle regions. The image was segmented to
isolate the four high signal regions and then the centroids of these regions
were found. Two diagonal distances between these centers were calculated as
shown in Figure LABEL:fig:phantom(C). The forward and backward diagonal
distance was recorded as d45 and d135 for reporting purposes. The laser
coordinates Y is the slice position, where laser coordinates X and Z are the
center of the central image slice, which were calculated using centroid
coordinates of four high contrast regions.

Figure 4: Updated daily QC results displayed through the QATrack+ web
interface. Figure 5: Configurability of the system using a simple text file.
Figure 6: Continuous monitoring of MRI simulator using autoMRISimQA and
QATrack+ web interface. SNR (green curve) and MRI signal (red curve) were
generated by QATrack+.

##  3 Results

The system has been implemented into the MRI-based radiotherapy workflow since
2013. As a start of a daily routine, a radiographer or radiation therapist
(RT) scans the laser phantom. Then the images are sent to a location sitting
on a shared network drive. The system automatically processes the images and
updates the QATrack+ database. The daily QC results can be viewed on any
computer connected to the hospital’s private network using the web-based
interface of QATrack+ or the PDF report.

As an example, Figure 4 showed one QC result generated by autoMRISimQA
results. The traffic lights were used to indicate the acceptability of the
results along with the tolerance values.

The main setting for the system can also be configured via a text file as
shown in Figure 5. The use of configuration files makes the system more
flexible and maintainable.

The statistical analysis based on the data acquired over a certain period
provided useful information regarding the MRI’s performance and a tool for
continuous monitoring of MRI scanner performance. As an example, Figure 6
shows the trend of SNR and MRI signals over a few months, which were generated
by QATrack+.

##  4 Discussion

AutoMRISimQA achieved an automated constancy check of MRISim’s performance by
monitoring MRI imaging system in one go using multiple image quality indices,
which are directly related to the use of MRI in radiotherapy planning and
recommended by TG 284 [3], but also doing a quick check of geometric accuracy.
In addition to the visual check of simulation laser alignment, the laser
positions were also quantitatively monitored.

The work presented here demonstrates a feasible solution for developing an
automated daily QC program for an MRI simulator, which can be used and
implemented by other centres. The source code of the system is available on
the request and will be on GitHub soon. Although the system implemented is
based on the laser phantom, other commercial phantoms or custom-made phantoms
can be used.

Several authors developed software tools for quality assurance of a diagnostic
MRI scanner using ACR MRI phantom [9, 10]. An open-source image processing
tool, IQWorks, was also used for MRI quality assurance purposes [11]. In
comparison with these tools, autoMRISimQA has several unique features: (1) the
system was fully automated once the phantom images were transferred to the
network location from the MRI control console; (2) It was seamlessly
integrated with an open-source QC data management system. The daily QA results
and the trend of the system’s performance can be viewed on any computer
connected to the hospital network; (3) Automatic email notification allows
physicists monitor the MRISim and take action only when it is required.

The clinical benefit of implementing the AutoMRISimQA system into the clinical
workflow is efficiency, especially for a busy clinic with a large number of
patients going through the scanner. Before the use of this system, the RTs
needed to do two QC tests daily: a customer QC using a uniform phantom and the
check of the alignment of simulation lasers using the laser phantom. The
average time spent on morning daily QC was half an hour. Now it was reduced to
four minutes, the scanning time of laser phantom. Furthermore, the continuous
monitoring of MRISim’s performance ensures the short, medium and, especially,
long-term follow-up of technical characteristics and their possible drift with
time.

##  5 Conclusion

A software system was designed, and implemented for daily quality controls of
an MRI simulator. As an open-source tool, the system provides a prototype
solution for monitoring the performance of a dedicated MRI scanner used in a
radiation oncology department.

## References

  * [1] RL Christiansen, HR Jensen and C Brink  “Magnetic Resonance–Only Workflow and Validation of Magnetic Resonance–Based Dose Calculations for Radiation Therapy of Prostate Cancer”  In _International Journal of Radiation Oncology, Biology, Physics_ 96.2 Elsevier, 2016, pp. E645
  * [2] Ann D King and Harriet C Thoeny  “Functional MRI for the prediction of treatment response in head and neck squamous cell carcinoma: potential and limitations”  In _Cancer imaging_ 16 Springer, 2016, pp. 1–8
  * [3] Carri K Glide-Hurst et al.  “Task group 284 report: magnetic resonance imaging simulation in radiotherapy: considerations for clinical implementation, optimization, and quality assurance”  In _Medical physics_ 48.7 Wiley Online Library, 2021, pp. e636–e670
  * [4] Aitang Xing et al.  “Commissioning and quality control of a dedicated wide bore 3T MRI simulator for radiotherapy planning”  In _Int J Cancer Ther Oncol_ 4.2, 2016, pp. 1–10
  * [5] Aitang Xing et al.  “Quality assurance of the radiotherapy workflow integrating a dedicated wide-bore 3T MRI simulator”  In _World Congress on Medical Physics and Biomedical Engineering, June 7-12, 2015, Toronto, Canada_ , 2015, pp. 539–542  Springer
  * [6] R Studinski et al.  “SU-E-T-103: Development and Implementation of Web Based Quality Control Software”  In _Medical Physics_ 41.6Part12 Wiley Online Library, 2014, pp. 246–246
  * [7] “Python REST API Framework” Accessed: 2024-1-1, http://python-eve.org/
  * [8] “Java iText package” Accessed: 2024-1-1, https://itextpdf.com/
  * [9] Jidi Sun et al.  “An open source automatic quality assurance (OSAQA) tool for the ACR MRI phantom”  In _Australasian physical & engineering sciences in medicine_ 38 Springer, 2015, pp. 39–46
  * [10] J Yung, W Stefan, D Reeve and RJ Stafford  “TU-F-CAMPUS-I-05: Semi-Automated, Open Source MRI Quality Assurance and Quality Control Program for Multi-Unit Institution”  In _Medical Physics_ 42.6Part36 Wiley Online Library, 2015, pp. 3647–3647
  * [11] Jason Fazakerley et al.  “The application of IQWorks image analysis software to quality assurance in MR imaging”  In _Biomedical Physics & Engineering Express_ 2.2 IOP Publishing, 2016, pp. 025011