# From Semantic Retrieval to Pairwise Ranking: Applying Deep Learning in
E-commerce Search

Rui Li, Yunjiang Jiang, Wenyun Yang, Guoyu Tang, Songlin Wang, Chaoyi Ma, Wei
He, Xi Xiong, Yun Xiao, Eric Yihong Zhao  JD.com, Mountain View, CA {rui.li,
yunjiang.jiang, wenyun.yang, tangguoyu, wangsonglin3, machaoyi, hewei92,
xiongxi, xiaoyun1<EMAIL_ADDRESS>
(2019)

###### Abstract.

We introduce deep learning models to the two most important stages in product
search at JD.com, one of the largest e-commerce platforms in the world.
Specifically, we outline the design of a deep learning system that retrieves
semantically relevant items to a query within milliseconds, and a pairwise
deep re-ranking system, which learns subtle user preferences. Compared to
traditional search systems, the proposed approaches are better at semantic
retrieval and personalized ranking, achieving significant improvements.

Neural networks; Semantic Search; Personalized Ranking

††journalyear: 2019††copyright: acmcopyright††conference: Proceedings of the
42nd International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 21–25, 2019; Paris, France††booktitle: Proceedings
of the 42nd International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR ’19), July 21–25, 2019, Paris, France††price:
15.00††doi: 10.1145/3331184.3331434††isbn: 978-1-4503-6172-9/19/07††ccs:
Information systems Information retrieval

##  1\. Introduction

Over past decades, online shopping platforms have become ubiquitous in
people’s daily life. In a typical e-commerce search engine, two major stages
are involved to answer a user query, namely Candidate Retrieval, which uses
inverted indexes to efficiently retrieve candidates based on term matching,
and Candidate Ranking, which orders those candidates based on factors, such as
relevance and predicted conversion ratio. Here we share our experiences and
promising results of applying deep learning for both stages in e-commerce
search at JD.com.

For candidate retrieval, we introduce the Deep Semantic Retrieval (DSR)
system, in addition to traditional term matching based retrieval systems. DSR
encodes queries and items into a semantic space based on click logs and human
supervision data, and uses an efficient K-nearest neighbor search algorithm to
retrieve relevant items from billions of candidates within milliseconds. For
candidate ranking, we introduce the Deep Pairwise Ranking (DPR) system to
leverage both hand-crafted numerical features and raw sparse features. DPR
uses a Siamese architecture to effectively learn preference between a pair of
items under the same query from billions of search log data and efficiently
score each item individually (without enumerating all the pairs) online.

##  2\. Deep Semantic Retrieval

The most critical challenge for this stage is enabling semantic retrieval
beyond exact term matches. Based on our analysis, this problem impacts around
$20\%$ of our search traffic.

Figure 1. Overview of deep semantic retrieval system.

System Overview

Neural network models naturally address this problem via semantic term
embeddings. While many deep neural network models (e.g., DSSM (Huang et al.,
2013)) have been proposed for search relevance ranking, our DSR system
explores the rare territory of candidate retrieval. As Figure 1 shows, DSR has
three parts.

Offline Training

process learns a query, item two tower model. Each tower consists of 1) an
input layer that concatenates input features of a query or an item, 2) an
encoding ReLU multi-layer, and 3) a normalization layer to ensure L2
normalized output embeddings.

Offline Indexing

module computes all the item embeddings from the item corpus, then builds an
embedding index offline for efficient online embedding retrieval.

Online Serving

module loads the query part of the two tower model and computes an embedding
for any query, which then retrieves the $K$ nearest/most similar items from
the item embedding index.

Model Overview

Now, we give a overview of our proposed model. Given a training set
$\mathcal{D}=\\{(q_{i},s^{+}_{j},s^{-}_{k})\;|\;i,j,k\\}$, where $q_{i}$
stands for a query, $s^{+}_{j}$ stands for a positive item that is relevant to
$q_{i}$, and $s^{-}_{k}$ stands for a negative item that is irrelevant to
$q_{i}$, we learn the query and item towers jointly by minimizing the
following loss function

$L(\mathcal{D})=\sum_{(q_{i},s_{j}^{+},s_{k}^{-})\in\mathcal{D}}\max\left(0,\delta-\left(Q(q_{i})^{T}S(s_{j}^{+})-Q(q_{i})^{T}S(s_{k}^{-})\right)\right)$

where $\delta\geq 0$ is the margin used in hinge loss, $Q(\cdot)$ denotes the
query tower, $S(\cdot)$ denotes the item tower and the superscript $\cdot^{T}$
denotes transpose operation. Intuitively, this function employs the hinge loss
to encourage the dot product between a query $q_{i}$ and a positive item
$s_{j}^{+}$ to be larger than the product between the query $q_{i}$ and a
negative item $s_{k}^{-}$ by a certain margin $\delta$.

Experimental Results

Table 1 shows relative improvement of DSR in online A/B tests in terms of four
core business metrics, including user click through rate (UCTR), user
conversion rate (UCVR), and gross merchandise value (GMV), as well as query
rewrite rate (QRR), which is believed to be a good indicator of search
satisfaction. Table 2 reports the time consumed by DSR for indexing and
searching $120$ million items with Nvidia Tesla P40 GPU and Intel $64$-core
CPU: GPU achieves ~10x speed up. Table 3 shows a few sample queries to better
illustrate the power of DSR at bridging semantic gaps between queries and
relevant items.

| UCTR | UCVR | GMV | QRR
---|---|---|---|---
overall | $+1.61\%$ | $+0.98\%$ | $+0.54\%$ | $-4.34\%$
long tail | $+9.7\%$ | $+10.03\%$ | $+7.5\%$ | $-9.99\%$
Table 1. Online A/B test improvements. | indexing (sec.) | search (ms) | QPS
---|---|---|---
CPU | $3453$ | $9.92$ | $100$
GPU | $499$ | $0.74$ | $1422$
Table 2. Efficiency for indexing and serving. query | retrieved item
---|---
奶粉 大童 | 美赞臣 安儿健A+ 4段
(milk powder big kid) | $\left(\begin{minipage}{108.405pt}Enfamil A+ level-4 for 3-6 yr old \end{minipage}\right)$
学习自由泳器材 | 英发/yingfa 划臂
$\left(\begin{minipage}{72.26999pt} learn free-style swimming equipment \end{minipage}\right)$ | (yingfa hand paddle )
Table 3. Good cases from DSR

##  3\. Deep Pairwise Ranking

Traditional e-commerce search systems apply gradient boosted trees (xgboost)
models (Chen and Guestrin, 2016) to rank products based on predicted
conversion rates. Relying mainly on numeric features, those models cannot
effectively explore sparse features, such as user’s click/purchase history.
Deep learning(DL) naturally address this problem as it can leverage those raw
features via embeddings. However, to apply DL in ranking, we need to address
1) big computation overhead and 2) sufficient amount of good training data
required by DL.

Overview

To address those challenges, we first introduce a _re-ranking_ stage, which
only scores the top K (100) items selected from earlier phases with a complex
DL model, to reduce the required computation, as we observe that 90% of
clicks/orders are within top 75 positions in existing results. Second, we
choose to train a pairwise ranking model with a large amount of noisy logs.
While the top 100 results are usually all reasonably relevant, the focus is
personalized preference, which is hard to quantify. Logs, though abundant, are
only good for approximating theses subtle preferences among presented items.
Thus, instead of learning conversion rate, as most e-commerce ranking models
do, we learn user preference between a pair of items for a given query.

Figure 2. Simplified Siamese ranking model.

Model Details

In order to effectively learn preference between a pair of items under the
same query offline and efficiently score each item individually (without
enumerating all the pairs) online, we design a pairwise Siamese model, as
shown in Figure 2. The training data consists of triples $(q,a,b)$ where $a,b$
are two items co-occurring under the same session with query $q$. The Siamese
structure has two identical towers (sharing the same parameters) for item $a$
and item $b$. Each tower takes all the input features from user, query, and
item, goes through 3 ReLU layers, and outputs a logit. The difference of the
two logits is then combined with the binary label in a cross-entropy loss
function. During online serving, only one of the two identical towers is used
to score each item.

The model uses the following features: 1) hand picked numeric features, such
as user’s purchasing power and past CTR/CVR/sale volume, 2) raw text features
for queries and items, tokenized into unigrams and bigrams, and 3) raw user
action features, such as historical click streams. The latter two use sum-
pooled embeddings.

Experimental Results

Our top-line validation metric is order-based average session AUC. Our DPR
model, which uses not only all the Xgboost features (about 130 total) but also
sparse features, performs significantly better than Xgboost methods (Table 4).
Table 5 shows results of online A/B tests. DPR significantly improves all the
core business metrics, including UCTR, UCVR, GMV, and average order position
(AOP). As DPR uses orders as positive examples during our training, it
achieves the most gain on UCVR. However, similar gains on UCTR/GMV are also
observed.

| session AUC | order NDCG@5
---|---|---
Xgboost | 0.852 | 0.522
Siamese DL | 0.870 | 0.573
Table 4. Offline validation metrics. | UCTR | UCVR | GMV | AOP
---|---|---|---|---
overall | $+1.16\%$ | $+1.99\%$ | $+1.04\%$ | $-2.10\%$
Table 5. Online A/B test improvements.

## References

  * (1)
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _KDD_. ACM, 785–794.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data. In _CIKM_. 2333–2338.


# Object Detection During Newborn Resuscitation Activities

Øyvind Meinich-Bache, Kjersti Engan, Ivar Austvoll, Trygve Eftestøl, Helge
Myklebust, Ladislaus Blacy Yarrot, Hussein Kidanto and Hege Ersdal

###### Abstract

Objective: Birth asphyxia is a major newborn mortality problem in low-resource
countries. International guideline provides treatment recommendations;
however, the importance and effect of the different treatments are not fully
explored. The available data is collected in Tanzania, during newborn
resuscitation, for analysis of the resuscitation activities and the response
of the newborn. An important step in the analysis is to create activity
timelines of the episodes, where activities include ventilation, suction,
stimulation etc. Methods: The available recordings are noisy real-world videos
with large variations. We propose a two-step process in order to detect
activities possibly overlapping in time. The first step is to detect and track
the relevant objects, like bag-mask resuscitator, heart rate sensors etc., and
the second step is to use this information to recognize the resuscitation
activities. The topic of this paper is the first step, and the object
detection and tracking are based on convolutional neural networks followed by
post processing. Results: The performance of the object detection during
activities were 96.97 % (ventilations), 100 % (attaching/removing heart rate
sensor) and 75 % (suction) on a test set of 20 videos. The system also
estimate the number of health care providers present with a performance of
71.16 %. Conclusion: The proposed object detection and tracking system
provides promising results in noisy newborn resuscitation videos.
Significance: This is the first step in a thorough analysis of newborn
resuscitation episodes, which could provide important insight about the
importance and effect of different newborn resuscitation activities.

###### Index Terms:

Newborn Resuscitation, Automatic Video Analysis, Object Detection,
Convolutional Neural Networks

††This work is part of the Safer Births project which has received funding
from: Laerdal Global Health, Laerdal Medical, University of Stavanger, Helse
Stavanger HF, Haydom Lutheran Hospital, Laerdal Foundation for Acute Medicine,
University in Oslo, University in Bergen, University of Dublin – Trinity
College, Weill Cornell Medicine and Muhimbili National Hospital. The work was
partly supported by the Research Council of Norway through the Global Health
and Vaccination Programme (GLOBVAC) project number 228203.†† Ø, Meinich-Bache,
K, Engan, I, Autsvoll and T, Eftestøl is with the Dep. of Electrical
Engineering and Computer Science, University of Stavanger, Norway. H,
Myklebust with Laerdal Medical, Norway. L. B, Yarrot with the Research
Institute, Haydom Lutheran Hospital, Manyara, Tanzania. H, Kidanto with the
School of Medicine, Aga Khan University, Dar es Salaam, Tanzania. H, Ersdal
with the Faculty of Health Sciences, University of Stavanger, Norway, and the
Department of Anesthesiology and Intensive Care, Stavanger University
Hospital, Norway.

##  I Introduction

Globally, one million newborns die within the first 24 hours of life each
year. Most of these deaths are caused by complications during birth and birth
asphyxia, and the mortality rates are highest in low-income countries [1]. As
many as 10-20 % of newborns require assistance to begin breathing and
recognition of birth asphyxia and initiation of newborn resuscitation is
crucial for survival [1, 2, 3]. International guidelines on newborn
resuscitation exists, however, the importance and effect of the different
treatments and therapeutic activities are not fully explored.

Safer Births111www.saferbirths.com is a research project to establish new
knowledge on how to save lives at birth, and the project has, among other
things, collected data during newborn resuscitation episodes at Haydom
Lutheran Hospital in Tanzania since 2013. The collected data contains video
recordings, ECG and accelerometer measurements from a heart rate sensor (HRS)
attached to the newborn, and measurements of pressure, flow and expired CO2
from a bag-mask resuscitator (BMR). A thorough analysis of the collected data
could provide important insight about different effects of the resuscitation
activities. To be able to study such effects it is necessary to quantify the
series of performed activities, in addition to measuring the condition of the
newborn during resuscitation and knowing the outcome. A timeline documenting
activities like ventilation, stimulation and suction would be of immense
value. From such a timeline it would be possible to extract parameters like
the amount of both total and continuous time used, the number of starts and
stops for different activities etc. The generation of the timelines should
preferably be done automatically by using the collected signals and/or video,
thus allowing large amounts of data to be analyzed. The value of such
timelines would clearly be i) for research and increased knowledge on the
effects of newborn resuscitation activities. A future implementation of a
complete system would also be useful on-site: ii) as a debriefing tool,
summarizing the activities with no need to study video recordings and iii) as
a real-time feedback system.

Previously, in Huyen et.al [4], our research group proposed an activity
detector based on the HRS signals and the detector discriminated the
activities stimulation, chest compressions and other with a accuracy of 78.7
%. Stimulation and chest compressions are therapeutic activities, whereas
other would include moving and drying the baby, touching the HRS etc. These
activities would result in movement in the HRS, and thus be visible in both
the ECG and the accelerometer signals, but are not considered therapeutic
activities or treatment of the newborn. Using automatic video analysis of the
video recordings during the resuscitation episodes could potentially improve
the performance achieved using the HRS signals. Furthermore, video analysis
could possibly detect activities and information that are difficult or
impossible to detect from the ECG and accelerometer signals, like; is the HRS
attached to the newborn or not, and how many health care providers (HCPs) are
present.

The importance of video analysis of newborn resuscitation episodes has been
well documented for both evaluation and training purposes [5, 6, 7, 8, 9].
However, manual inspection and annotation is very time consuming, and limits
the amount of data that can be analyzed. In addition, a manual inspection
entails privacy issues. Thus, there is a need for automatic video analysis of
these episodes. Conventional image and pattern recognition methods, e.g
segmentation and tracking, has been applied in automatic video analysis for
decades [10], but in recent years Deep Neural Networks (DNNs) has shown it‘s
superior strength in the field [11, 12, 13, 14]. In the topic of object and
activity detection in resuscitation in general, others have propose the usage
of passive radio-frequency identification (RFID) tags on the objects for
object motion and interaction detection [15, 16, 17]. Chakraborty et.al [18]
proposed an object and activity detector for trauma resuscitation video
recordings based on object segmentation and a Markov Logic Network model. In
the area of newborn resuscitation Guo et.al [19] proposed an activity
detection system for newborn resuscitation videos based on DNN and linear
Support-Vector Machines (SVMs). Their dataset included 17 videos recorded with
a frame rate of 25 frames per second (FPS) at a hospital in Nepal, and the
group aimed to detect the activities stimulation, suction, ventilation and
crying. The pre-trained Faster RCNN network and the object class People were
used to propose areas involving the newborn, and motion salient areas were
further used as input to two pre-trained Convolutional Neural Networks (CNN)
from [11] designed to extract motion and spatial features. Further, the
features was combined and used as input to linear SVMs, trained on their own
dataset, to detect the activities.

All though there are similarities between the dataset from [19] and our
dataset, they are both noisy real-world videos with large variations, there
are some specific tasks an challenges that differs between the studies. First,
we aim to detect activities that are not newborn location dependent or
movement dependent, like, the number of HCP present, and is the HRS attached
or not. Second, in our dataset the newborns are wrapped in blankets most of
the time, even before being placed at the resuscitation table, and the image
examples from [19], which shows fully uncovered newborns, are more infrequent
in our dataset. Thus, using a pre-trained Person detection network as
suggested in [19] would most likely not be the best approach. In addition, our
videos are recorded with variable frame rate, which in some case are very low
and causes motion blurred images of poor quality, resulting in larger per
frame motion variations than for images recorded with fixed frame rates.
Considering all this, we believe that using an object detection and tracking
approach to localize the relevant activity detection areas would be a more
robust first step in activity detection. Further, using the areas around each
objects would simplify the detection problem to a binary classification
problem for the specific activities; is the object being used in resuscitation
or not. The topic of this paper is the first step and the object detection and
tracking is based on CNNs followed by post processing. Neural networks for
object detection requires a lot of training data, so in addition to using
image frames from the videos, we use histogram matching [20] for augmentation
and also a synthetic dataset. The object detection is performed on each video
frame and here we use the well known YOLOv3 [21] network, used in various
object detection applications [22, 23, 24]. Post processing is used to fill in
missing detections and track the area around the objects during the episodes.

Figure 1: Block scheme of the activity detection system. The red dotted line
encircles the steps proposed in this paper. 1: Generated dataset is input to
YOLOv3 object detection network. 2: Detected objects. 3: Detected object area
after post processing. 4: Sequence of images from areas are used as input to
sequential neural networks. 5: Activity time lines is the final output.

##  II Data material

The dataset is collected using Laerdal Newborn Resuscitation Monitors (LNRM)
[25] and with cameras mounted over the resuscitation tables. The dataset
contains almost 500 videos with corresponding LNRM data. The LNRM records the
signals measured by the green HRS and the BMR, both shown at the top of Figure
3 C.

The video recordings were initiated to provide additional support in cases and
research objectives where the other collected signal or observed data were
difficult to interpret. However, the videos are of variable quality and camera
and scene settings are not standardized for the different resuscitation tables
included in the dataset. The variations are caused by different camera types,
camera angles, video resolutions (1024$\times$1280, 720$\times$1280, and
1200$\times$1600), camera distances from resuscitation tables, variable frame
rates (2-30 frames per second), unfocused cameras and light settings. All
these variations, especially the variable frame rate, make automatic video
analysis more challenging. In some cases the frame rate is as low as two
frames per second, resulting in motion blurred image frames of poor quality.
In Figure 2 some of these challenges are depicted; A) Motion blurring, B) far
away camera position, C) occlusion due to camera angle and D) poor lighting
conditions. In addition, the videos also have variations like HCPs using
different colored rubber gloves, HCPs that do not wear rubber gloves,
different colored HCP uniforms and clothing, and colorful and patterned
blankets brought by the mothers to wraps the newborn in. The activity
timelines that are relevant to generate are:

  * •

1) Bag-mask ventilations: Respiratory support.

  * •

2) Suction: Removal of fluids from nasal and oral cavities using a device
called suction penguin (SP).

  * •

3) HRS attached to newborn or not.

  * •

4) Stimulation: Warming, drying, and rubbing the newborns‘s back.

  * •

5) Chest compressions. Keep oxygenated blood flowing to the brain and other
vital organs.

  * •

6) Number of HCPs present.

  * •

7) Newborn wrapped in blanket or not.

Activity 1), 2), 3), 4) and 5) can be detected by tracking the objects BMR,
SP, HRS and HCPs hands (HCPH), and by analyzing their surrounding areas, 6) by
counting the number of detected HCPH, and 7) by analyzing an area around the
newborn, found from motion analysis and the location of the detected objects.

Figure 2: A: Motion blurring due to low frame rate, 1024x1280. B: Camera far
away, 1200 x 1600. C: Occlusion (ventilating newborn behind health care
provider), 1024x1280. D: Poor lighting, 720 x 1280.

##  III Methods

A block scheme of the planned activity detection system is shown in Figure 1.
The steps proposed in this paper is encircled with a red dotted line. These
include dataset generation using the collected videos, augmentation of images
from the collected videos, generation of a synthetic dataset, object detection
using YOLOv3 [21], post processing to select the areas surrounding the
relevant objects and an estimation of the number of HCPs involved in the
resuscitation at each moment in time.

###  III-A Data Generation

A dataset, VideoD, of 3093 images for object detection training is created by
selecting evenly spread image frames from 21 randomly selected videos. The
objects are manually labelled using the Image Labeler [26].

####  III-A1 Augmentation dataset

VideoD is further augmented to a new dataset, HistD, by using histogram
matching [20]. A frame from 10 randomly selected videos are used as histogram
reference frames, and each of the images in VideoD are augmented with each of
the reference frames creating in total 34 023 images. 6 of 10 examples of the
histogram match augmentation is shown for one of the frames in Figure 3 B.

Figure 3: A: Example of a frame used in VideoD. B: Examples of histogram match
augmented images from HistD. C: Scene for recording objects to be used in the
generation of synthetic dataset, masked objects and an example of a generated
frame in SynthD.

####  III-A2 Synthetic dataset

A synthetic dataset, SynthD, is created in an attempt of generating example
images with the variation found in the original dataset. Because of the
colourful and patterned blankets used in the resuscitation videos, the objects
we want to detect can appear on all kinds of backgrounds, thus over 6000
different backgrounds, both natural images and texture images are used. First,
hands with different coloured gloves and no gloves, two types of BMR that both
appear in the collected resuscitation videos, the HRS and the SP were video
recorded in front of a blue screen in all possible angles. Object masks are
created using video frames, $I(x,y)_{i}$, where $x,y$ denote the pixel
coordinates and $i$ the frame number, from the recorded object videos by:

$\mathit{OM(x,y)_{i,c}}=I_{B}(x,y)_{i,c}-I_{L}(x,y)_{i,c}<T_{CK,c}$ (1)

where $c$ denote the object class, $I_{B}$ the blue channel, $I_{L}$ the RGB
luminance value $(0.3I_{R}+0.59I_{G}+0.11I_{B})$ and $T_{CK}$ the chroma key
thresholds for each $c$. Around 6300 masks per class are created in average.

Next, a background is randomly drawn from the 6482 examples and objects and
masks are cast at random positions onto the background. One example of each
object, except from HCPH where we use a number between one and three examples,
is used. The objects are randomly scaled with the object‘s typical size
relative to the size of the image frame - found from VideoD, and hue,
saturation and lightness is also randomly chosen between 60-100 % of the
original object images.

In order to make the object appear as realistic as possible, the final
synthetic images are filtered with a small motion blur where the length,
$len$, and angle, $\theta$, of the motion are randomly chosen. The scene for
recording objects, masked objects and an example of a generated synthetic
image is shown in Figure, 3 C.

####  III-A3 Split image dataset

In an attempt of better utilizing the resolution in the video frames and to be
able to predict the smallest objects, the images in $HistD$ are split into
five equally sized sub images generating a new dataset, $SplitD$. The four
first images are generated from splitting the image into four parts, and the
fifth is extracted at the center of the original image frame. This fifth sub
image would typically contain more objects than the rest, and become an
overlap of the other four sub images. The bounding box annotation is also
split and the resulting bounding boxes is removed if they are $<$ 40 % of the
size of another box representing the same object in another sub image. This
step ensures that all the resulting bounding boxes contain a significant part
of the objects, making the resulting images good training examples.

####  III-A4 Dataset for testing

A dataset, $TestD$, of 1000 images is created by selecting 50 evenly spread
image frames from 20 randomly selected videos, not previously used for
training, where the mean duration per video is around 7 minutes. The test
images are labelled using Image Labeler [26]. A split version,
$TestD_{split}$, of $TestD$ is also created with the same procedure as
explained in section III-A3.

###  III-B Object detection

The proposed system uses the well known YOLOv3 [21] in the object detection
step. YOLOv3 is comparable to the state of the art models on the mAP50 metric
[21], and is chosen for the following reasons: 1) Speed - YOLOv3 can perform
predictions on video streams in real time - which could be useful in a future
application for our proposed system, 2) YOLOv3 is state-of-the-art at
predicting the correct class, rather than focusing on accurate bounding box
predictions - which suits the problem at hand well. 3) It predicts small
objects with better precision than medium and large objects [21] \- which also
suits the problem at hand well, and finally, 4) due to the limited size of
labelled training data, using transfer learning with a-state-of-the-art model
as YOLOv3 as the starting point will most likely outperform any training from
scratch.

####  III-B1 Network structure (YOLOv3)

YOLOv3 [21] is a fully convolutional network, meaning no fully-connected
layers are used. It consist of 75 convolutional layers in total and performs
downsampling by using convolutional layers with a stride of two instead of
using pooling layers. The network also includes residual blocks [27] and
performs detection on three different scales in order to detect objects of
different size. The detections on the different scales utilize feature maps
from deeper layers in a similar concept to feature pyramid networks [28] and
the features go through convolutional layers before outputting 3D tensors with
dimension:

$N\times N\times[3\times(4+1+C)]$ (2)

where $N$ is the number of grids at that scale (13, 26 and 52 if image size is
$416\times 416$), 3 the number of bounding boxes for each grid, 4 the box
coordinates and size, 1 the objectness prediction, $oP$, and C the number of
object classes. The YOLO algorithm further performs non-maximum suppression:
Removing predicted object with an objectness score below a threshold, $T_{o}$,
and by removing predictions of same class where the bounding box overlap more
than threshold $T_{IoU}$.

####  III-B2 Post processing object detection

Post processing is performed on the detection of BMR, SP and HRS to fill in
missing detections in frames and to create areas surrounding the object
throughout the video. Since we can have multiple true occurrences of HCPH in
the same frame, HCPH do not undergo these steps. Denote $obj\in{1:4}$ to be
the object classes where $1=$ BMR, $2=$ SP, $3=$ HRS and $4=$ HCPH, and
$N_{E,i}$ to represent the number of detections in image, i, of episode, E.
For $obj_{p}\in\\{1,2,3\\}\subset obj$ we estimate the most likely object
position in each $i$ by; first, creating blank images,
$IB(x,y,obj_{p})_{E,i}$. Second, for each pixel areas,
$pA_{E,i,obj_{p,n}}=\\{x^{E,i,obj_{p}}_{n},y^{E,i,obj_{P}}_{n}\\}$,
representing all pixel coordinates of a detected object, $obj(n)_{E,i}$, in an
image we add the detection‘s $oP$ score, $oP(n)_{E,i,obj_{P}}$, to the
matching coordinates in $IB(x,y,obj_{p})_{E,i}$.

_For_ $n=1:N_{E,i}$ _do_ :

$\mathit{IB(x,y,obj_{p})_{E,i}}=\left\\{\begin{array}[]{l
l}\mathit{IB(\cdot)}+oP(n)_{E,i,obj_{p}},\\\ \quad\quad\forall\\{x,y\\}\in
pA_{E,i,obj_{p,n}(n)}\\\ \quad\quad\text{if}\quad obj(n)_{E,i}=obj_{p}\\\
\mathit{IB(\cdot)},\quad\text{otherwise}\end{array}\right.$ (3)

Further the centroid coordinates,
$(x_{c}^{{E,i,obj_{p}}},y_{c}^{E,i,obj_{p}})$, of the most likely object
position is found from:

$(x_{c}^{(\cdot)},y_{c}^{(\cdot)})=cent(max(IB(x,y,obj_{p})_{E,i}>T_{obj_{p}})$
(4)

where $T_{obj_{p}}$ defines thresholds for the different object classes.
Denote $d\in{X,Y}$. Each $x_{c}^{(\cdot)}$ and $y_{c}^{(\cdot)}$ are stored in
location vectors, $L(i)_{E,d,obj_{p}}$, representing timelines of the center
position of each object as a function of the video frames.
$L(i)_{E,d,obj_{p}}$ further undergoes the three post processing steps
illustrated with an example in Figure 4, listed as follows:
1) Filling detection gaps by choosing the previous detected value
$\rightarrow\quad Lf(i)_{E,d,obj_{p}}$.
2) Short peak removal. If $||Lf(i)_{(\cdot)}-Lf(i-1)_{(\cdot)}||>T_{peak}$, we
check if it is an actual large change in object position, or if it returns to
a value where
$Lf(i+1:i+10)_{(\cdot)}-Lf(i-1)_{(\cdot)}||<T_{stable}$. This step filters out
short false detections of the objects, and outputs the peak removed signal,
$Lpr(i)_{E,d,obj_{p}}$.
3) Signal smoothing by applying a moving average filter of length $N_{f1}$:

$\mathit{Ls(i)_{E,d,obj_{p}}}=\frac{1}{N_{f1}}\sum_{l=-N_{f1}/2}^{N_{f1}/2}{\mathit{Lpr(l)_{E,d,obj_{p}}}}$
(5)

Finally, object area tracking throughout sequences is performed by adding a
$500\times 500$ bounding box, $BB_{track,E,obj_{p}}$, around each
$\mathit{Ls(i)_{E,d,obj_{p}}}$ onto the original videos. The size of
$BB_{track,E,obj_{p}}$ ensure that it is possible to detect what activities
are performed in the area, and thus discriminate the activities from movement
and noise. An example of the tracking results is shown in step 3 of Figure 1.

Figure 4: Example of post processing the centroid X-coordinate of the detected
bag-mask resuscitator (BMR). Horizontal axis is the image frame in the video
and vertical axis the pixel position in the frame.

###  III-C Estimation of number of health care providers present

Timelines of the number of HCPs present in the resuscitation videos are
generated from the number of detected hands in the image frames,
$\mathit{nH(i)_{E}}$.

_For_ $n=1:N_{E,i}$ _do_ :

$\mathit{nH(i)_{E}}=\left\\{\begin{array}[]{l l}nH(i)_{E}+1,\quad\text{if
}obj(n)_{E,i}=4\\\ \quad\quad\quad\text{and }oP(n)_{E,i}>T_{\mathit{HCPH}}\\\
nH(i)_{E},\hskip 22.76228pt\text{otherwise}\end{array}\right.$ (6)

where $T_{\mathit{HCPH}}$ is a threshold for detection of HCPHs. To remove
noise, $nH(i)_{E}$ is further smoothed by a moving average filter:

$\mathit{\overline{nH}(i)_{E}}=\frac{1}{N_{f2}}\sum_{l=-N_{f2}/2}^{N_{f2}/2}{\mathit{nH(l)_{E}}}$
(7)

where $N_{f2}$ is the filter size. Finally, $\mathit{\overline{nH}(i)_{E}}$ is
converted to the detected number of HCPs, $nHCP(i)_{E}$, by:

$\mathit{nHCP(i)_{E}}=\begin{cases}0&\text{if
}\quad\mathit{\overline{nH}(i)_{E}}\leq T_{zero}\\\ 1&\text{if }\quad
T_{zero}<\mathit{\overline{nH}(i)_{E}}\leq T_{one}\\\ 2&\text{if }\quad
T_{one}<\mathit{\overline{nH}(i)_{E}}\leq T_{two}\\\ 3&\text{if
}\quad\mathit{\overline{nH}(i)_{E}}>T_{two}\end{cases}$ (8)

##  IV Experiments

We used the original pretrained weights for YOLOv3, darknet53, and trained
different models by further training the weights with four different sets of
training data, $VideoD$, $HistD$, $HistD+SynthD$ and $SplitD+SynthD$. An
initialization stage is used to get a stable loss by first freezing all layers
except the top 3 layers. In the next and final stage all layers are further
trained with learning rate decay and early stopping. The batch size was set to
16. The mean Average Precision (mAP) criterion defined in the PASCAL VOC 2012
competition222http://host.robots.ox.ac.uk/pascal/VOC/voc2012/ was used to
compare single-image object detection results from the models trained on the
four different mixtures of the datasets. mAP is a function of precision,
recall and the Intersection over Unions (IoU), the overlap between predicted
and true bounding box. The threshold for IoU was set to 0.5.

The best models were further used in detection of the objects and the post
processing steps to evaluate the performance of the proposed object regions.
The proposed regions were added to the original video and the detection
results were manually evaluated by annotating timelines using the video
annotation tool ELAN333https://tla.mpi.nl/tools/tla-tools/elan/. The annotated
timelines for each $E$ are:

  * •

The number of HCPs: $\mathit{nHCP_{ref,E}(i)}$,

  * •

activities - ventilations, attaching or removing HRS, and suction,
$A_{obj_{p},E}(i)$,

  * •

is the object visible: $V_{obj_{p},E}(i)$ and

  * •

is the object detected: $D_{obj_{p},E}(i)$ ($>$ half the object is included in
$BB_{track,E,obj_{p}}$)

The main task of the object detection and tracking is to find approximate
regions around the objects that can be used for further activity recognition.
The aim is not to propose very accurate regions that centers the object
perfectly, but more importantly to propose smoothly updated regions that
surround the object over time. Thus, we classify a tracking result as correct
if the object is at least 50 % included in the proposed region.

Since our aim is to track a single object of each of the classes SP, HRS and
BMR throughout the whole video, we can evaluate the objects individually. The
established metric Multiple Object Tracking Accuracy (MOTA) can be seen in the
context of single-object short-term tracking and be simplified to the
percentage of correctly tracked frames [29]. Thus, the performance, P, is
evaluated for each object class and each episode, E, by the general equation

$P=(\frac{1}{N_{s}}\sum_{i=1}^{N_{s}}{I_{f}(i)})*100$ (9)

where $N_{s}$ is the number of frames in the episode and $I_{f}(i)$ an
indicator function defined as 1 if $|detection(i)_{E}-reference(i)_{E}|=0$ and
0 otherwise. The average performance, $\overline{P}$, of the post processed
object detection are estimated using Eq. 9 with $D_{obj_{p},E}(i)$ as
_detection_ $V_{obj_{p},E}(i)$) as _reference_ , and by averaging over the
episodes.

Further, we evaluate the performance of the object detection during the
relevant resuscitation activities, ventilation (BMR), Attaching or removing
HRS and suction (SP). From $A_{obj_{p},E}(i)$ we locate the activity sequences
and use them as _reference_ in Eq. 9. Their corresponding sequences in time in
$D_{obj_{p},E}(i)$ is here used as _detection_ and an activity is classified
as detected if the detection overlap with the reference data $>$ 80 % of the
time.

The timelines $\mathit{nHCP(i)_{E}}$ is found as explained in Section III-C
and the average performance, $\overline{P}$, of the prediction of number of
HCPs is estimated using Eq. 9 with $\mathit{nHCP_{ref,E}(i)}$ as _reference_
and $\mathit{nHCP(i)_{E}}$ as _detection_. In addition, the average prediction
error, $\overline{E}$, of $||\mathit{nHCP_{ref,E}(i)}-\mathit{nHCP(i)_{E}}||$
is estimated over the episodes. The total performance, $P$, of the classes no
HCP, one HCP, two HCP and three (or more) HCP is also estimated using Eq. 9,
where the class-relevant sequences in $\mathit{nHCP_{ref,E}(i)}$ is the
_reference_ and the corresponding sequences in time in $\mathit{nHCP(i)_{E}}$
is the _detection_.

When the results are averaged over results from individual episodes, quartile
measurements, $Q$, are also provided.

The experiments are done using Python444https://www.python.org/ and a
Keras555https://keras.io/ implementation of YOLOv3 developed by user
qqwwee666https://github.com/qqwweee/keras-yolo3 with minor modifications.
Since the objects often are occluded in the videos and the camera distance
varies, the objects‘s size and form have large variations. Therefore, we have
chosen to use the YOLOv3 anchor boxes determined using k-means clustering on
the large COCO dataset [21] instead of estimating anchor boxes from our
limited truth data.

The threshold and parameter values used in the experiments are:
$T_{CK,c}\in\\{80,180\\}$, $len=3-7$, $\theta=3-10$, $T_{o}=0.05$,
$T_{IoU}=0.45$ , $T_{obj}=[0.1,0.05,0.1]$ for BMR, SP and HRS,
$T_{\mathit{HCPH}}=0.1$ $T_{peak}=200$, $T_{stable}=50$, $T_{zero}=0.2$,
$T_{one}=2$, $T_{two}=4$, $N_{f1}=5$ and $N_{f2}=40$.

##  V Results

The mean average precision, mAP, results are listed in Table I for the object
detection using models trained on the datasets $VideoD$, $HistD$,
$HistD+SynthD$ and $SplitD+SynthD$. For the objects HCPH, BMR and HRS using a
combination of $HistD$ and $SynthD$ and image size $416\times 416$ provided
the best results. There was no significant improvement by increasing the image
input size to $608\times 608$. For detection of SP we achieved the best result
by using a model trained on $SplitD$ and $SynthD$, and an image size of
$608\times 608$. This model also provided the best overall mAP.

Table I: Object detection results, measured with mAP50, for models trained with different datasets. HCPH = health care provider hand, BMR = bag-mask resuscitator, HRS = heart rate sensor and SP = suction penguin. |  | $VideoD$
---
$416\times 416$
| $HistD$
---
$416\times 416$
| $HistD+$
---
$SynthD$
$416\times 416$
| $SplitD+$
---
$SynthD$
$608\times 608$
HCPH | 63.91 | 68.49 | 70.07 | 68.55
BMR | 57.45 | 57.54 | 62.07 | 59.77
HRS | 62.79 | 71.61 | 79.38 | 73.49
SP | 25.92 | 18.86 | 19.25 | 42.02
Total | 52.52 | 54.12 | 57.69 | 60.96
Table II: Performance results. Top section: Object detection (using a 500x500 area) after post processing. Middle: object tracking when relevant activities occurs (# detected / # true). Bottom: Prediction of the number of health care providers. | Object detection
---
(post processed)
| $\overline{P}$
---
Q (25,50,75)
BMR | 96.66 % |  | 96.23, 100, 100 (%)
---
HRS | 97.88 % | 100, 100, 100 (%)
SP | 76.86 % |  | 70.99, 81.67, 92.82 (%)
---
|  |
| Object detection
---
during activity
$P$ | Activities
BMR | 96.97 % (64/66) | Ventilation
| HRS
---
100 % (43/43) | Attach/remove HRS
SP | 75.00 % (45/60) | Suction
|  |
| HCP detection
---
$P$ |
No HCP | 90.70 % |
One HCP | 90.48 % |
Two HCPs | 53.31 % |
| Three (or more)
---
HCPs
6.88 % |
| $\overline{P}$ | Q (25,50,75)
| HCP correct pred.
---
71.16 % |  | 50.72, 78.56, 89.45 (%)
---
| $\overline{E}$ |
| HCP pred. error
---
0.32 | 0.11 0.22 0.54

The detection results from models trained on $HistD+SynthD$ and
$SplitD+SynthD$ were combined and used in the post processing steps explained
in Section III-B2 to achieve the results listed in Table II. The proposed
tracking area surround more than half the object in close to 100 % of the time
for VB and HRS, and almost 77 % for the SP.

During the activities Ventilations (BMR), Attach/remove HRS (HRS) and Suction
(SP) the tracking area surrounds the object during the activities in 97, 100
and 75 % of the occurrences respectively.

Table II also shows the results of HCP detection and the first four results
listed are estimated over all samples and episodes, and the last two results
are estimated per episode. The performance of the detection of number of HCPs
is above 90 % when there are zero or one HCP present. However, for two and
more than two HCPs the performance is 53 and 6 % respectively. The mean
prediction error is here 0.32, in other words, when the number of estimated
HCP is incorrect, it is usually underestimated by one.

Figure 5 shows the distribution of the sub groups FPS $\leq 8$ and FPS $>8$ in
the groups detected and undetected SP during suction. For the group undetected
we list the most likely reason for why the SP were undetected. The group
others represent the sequences where no large challenges was observed during
the activity.

##  VI Discussion

The proposed system shows promising results for object detection and tracking
in noisy real-world videos of a newborn resuscitation scene. As proposed in
Figure 1 the areas around the objects will be used as input to sequential
neural networks trained to recognize the different activities by analyzing the
areas for short time sequences. Other relevant areas like the area around the
newborn, which could be found from the detected hand movements, and around the
detected HCPHs can also be used as inputs to the sequential analysis.

Due to the suction penguins transparency and small size, the system struggles
with detecting it in some of the episodes. Especially in videos with low frame
rate and motion blurred images it could be very difficult to detect a SP held
in the hand of a health care provider. In additon, the system also has problem
detecting the SP in unfocused video sequences and in activity sequences with
large occlusions. Using the sub-image approach and the SplitD model improved
the detections of the SP. This suggests that it could be possible to further
improve the results by experimenting with the size and cropping of training
examples. In addition, we could experiment with the generation of the
synthetic data to see if it is possible to generate more realistic examples.

In future recordings the problem with detection of SP could be solved by using
fixed camera settings, focus, frame rate and distance from resuscitation
tables and by using two camera angles to avoid occlusion.

The performance of detected number of HCPs present in the video is very good
for zero and one HCP present, but the system struggles to detect the number of
HCPs when there are more than one HCPs present. Instead, in cases of false
detection, these are mostly being mislabeled as one HCPs less than the
reference data shows. The cause for this is a mixture of variations in the
dataset and of camera angles. The system performs worse when the HCPs are not
wearing rubber gloves, suggesting the need for more training examples from
similar episodes. The cameras are also often placed in a side-position where
the HCPs occludes other HCPs and hands. Training the network to discriminate
between left and right hands could also improve the performance of the
detected number of HCPs present in the videos.

Figure 5: Object detection during suction. Detected and undetected sequences
with the subgroups low and medium frames per second (FPS) rate.

##  VII Conclusion and future work

The proposed system shows promising object detection and tracking results in
noisy real-world videos. The object detection performance during activities
was 97 % on ventilation, 100 % on attaching or removing heart rate sensor and
75 % on suction. The system also estimate the number of health care providers
(HCP) present with an accuracy of 71 %.

In future work we will investigate the possibility of discriminating between
left and right HCP hands and implementing hand tracking to improve the
performance of the estimated number of HCP. We will also experiment with
different network structures and training data to try to improve the detection
of the suction device, in addition to increasing the amount of training data
in general to get a better overall detection performance. Further, we will
continue with step two of the planned system: inputting the proposed object
areas to sequential neural networks to detect the resuscitation activities.
This will produce timelines useful for quantifying the use of different
resuscitation activities, which could further provide new knowledge on the
effects of activities on newborn resuscitation outcome. In the future, such a
system could also be implemented on-site as a post-resuscitation debriefing
tool, and/or for real-time feedback and decision support during newborn
resuscitation. The latter would require a very high-performance system.

##  VIII Acknowledgement

###  VIII-A Funding

Our research is part of the Safer Births project which has received funding
from: Laerdal Global Health, Laerdal Medical, University of Stavanger, Helse
Stavanger HF, Haydom Lutheran Hospital, Laerdal Foundation for Acute Medicine,
University in Oslo, University in Bergen, University of Dublin – Trinity
College, Weill Cornell Medicine and Muhimbili National Hospital. The work was
partly supported by the Research Council of Norway through the Global Health
and Vaccination Programme (GLOBVAC) project number 228203.

For the specific study of this paper; Laerdal Medical provided the video
equipment. Laerdal Global Health funded data collection in Tanzania and IT
infrastructure. The University of Stavanger funded the interpretation of the
data.

###  VIII-B Ethical approval

This study was approved by the National Institute of Medical Research (NIMR)
in Tanzania (NIMR/HQ/R.8a/Vol. IX/1434) and the Regional Committee for Medical
and Health Research Ethics (REK), Norway (2013/110/REK vest). Parental
informed verbal consent was obtained for all resuscitated newborns.

###  VIII-C Conflict of interests

Myklebust is employed by Laerdal Medical. He contributed to study design and
critical revision of the manuscript, but not in the analysis and
interpretation of the data.

## References

  * [1] S. Wright, K. Mathieson, L. Brearley, S. Jacobs, L. Holly, R. Wickremasinghe, and A. Renton, “Ending newborn deaths: ensuring every baby survives.” 2014.
  * [2] A. C. Lee, S. Cousens, S. N. Wall, S. Niermeyer, G. L. Darmstadt, W. A. Carlo, W. J. Keenan, Z. A. Bhutta, C. Gill, and J. E. Lawn, “Neonatal resuscitation and immediate newborn assessment and stimulation for the prevention of neonatal deaths: a systematic review, meta-analysis and delphi estimation of mortality effect,” _BMC public health_ , vol. 11, no. 3, p. S12, 2011.
  * [3] H. L. Ersdal, E. Mduma, E. Svensen, and J. M. Perlman, “Early initiation of basic resuscitation interventions including face mask ventilation may reduce birth asphyxia related mortality in low-income countries: a prospective descriptive observational study,” _Resuscitation_ , vol. 83, no. 7, pp. 869–873, 2012.
  * [4] H. Vu, K. Engan, T. Eftestøl, A. Katsaggelos, S. Jatosh, S. Kusulla, E. Mduma, H. Kidanto, and H. Ersdal, “Automatic classification of resuscitation activities on birth-asphyxiated newborns using acceleration and ecg signals,” _Biomedical Signal Processing and Control_ , vol. 36, pp. 20–26, 2017.
  * [5] C. Skåre, A. M. Boldingh, J. Kramer-Johansen, T. E. Calisch, B. Nakstad, V. Nadkarni, T. M. Olasveengen, and D. E. Niles, “Video performance-debriefings and ventilation-refreshers improve quality of neonatal resuscitation,” _Resuscitation_ , vol. 132, pp. 140–146, 2018.
  * [6] B. Gelbart, R. Hiscock, and C. Barfield, “Assessment of neonatal resuscitation performance using video recording in a perinatal centre,” _Journal of paediatrics and child health_ , vol. 46, no. 7-8, pp. 378–383, 2010.
  * [7] S. Maya-Enero, F. Botet-Mussons, J. Figueras-Aloy, M. Izquierdo-Renau, M. Thió, and M. Iriondo-Sanz, “Adherence to the neonatal resuscitation algorithm for preterm infants in a tertiary hospital in spain,” _BMC pediatrics_ , vol. 18, no. 1, p. 319, 2018.
  * [8] I. Nadler, P. M. Sanderson, C. R. Van Dyken, P. G. Davis, and H. G. Liley, “Presenting video recordings of newborn resuscitations in debriefings for teamwork training,” _BMJ quality & safety_, vol. 20, no. 2, pp. 163–169, 2011.
  * [9] K. Schilleman, M. L. Siew, E. Lopriore, C. J. Morley, F. J. Walther, and A. B. te Pas, “Auditing resuscitation of preterm infants at birth by recording video and physiological parameters,” _Resuscitation_ , vol. 83, no. 9, pp. 1135–1139, 2012.
  * [10] J. D. Courtney, “Automatic video indexing via object motion analysis,” _Pattern Recognition_ , vol. 30, no. 4, pp. 607–625, 1997.
  * [11] G. Gkioxari and J. Malik, “Finding action tubes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 759–768.
  * [12] S. Ma, L. Sigal, and S. Sclaroff, “Learning activity progression in lstms for activity detection and early detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1942–1950.
  * [13] A. Montes, A. Salvador, S. Pascual, and X. Giro-i Nieto, “Temporal activity detection in untrimmed videos with recurrent neural networks,” _arXiv preprint arXiv:1608.08128_ , 2016.
  * [14] B. Singh, T. K. Marks, M. Jones, O. Tuzel, and M. Shao, “A multi-stream bi-directional recurrent neural network for fine-grained action detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1961–1970.
  * [15] S. Parlak and I. Marsic, “Detecting object motion using passive rfid: A trauma resuscitation case study,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 62, no. 9, pp. 2430–2437, 2013.
  * [16] S. Parlak, A. Sarcevic, I. Marsic, and R. S. Burd, “Introducing rfid technology in dynamic and time-critical medical settings: Requirements and challenges,” _Journal of biomedical informatics_ , vol. 45, no. 5, pp. 958–974, 2012.
  * [17] S. Parlak, I. Marsic, A. Sarcevic, W. U. Bajwa, L. J. Waterhouse, and R. S. Burd, “Passive rfid for object and use detection during trauma resuscitation,” _IEEE Transactions on Mobile Computing_ , vol. 15, no. 4, pp. 924–937, 2016.
  * [18] I. Chakraborty, A. Elgammal, and R. S. Burd, “Video based activity recognition in trauma resuscitation,” in _2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_. IEEE, 2013, pp. 1–8.
  * [19] Y. Guo, J. Wrammert, K. Singh, K. Ashish, K. Bradford, and A. Krishnamurthy, “Automatic analysis of neonatal video data to evaluate resuscitation performance,” in _Computational Advances in Bio and Medical Sciences (ICCABS), 2016 IEEE 6th International Conference on_. IEEE, 2016, pp. 1–6.
  * [20] R. C. Gonzalez and R. E. Woods, _Digital Image Processing_ , third edition ed. Pearson, 2008.
  * [21] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv_ , 2018\.
  * [22] W. Xu and S. Matzner, “Underwater fish detection using deep learning for water power applications,” _arXiv preprint arXiv:1811.01494_ , 2018.
  * [23] L. Heng, B. Choi, Z. Cui, M. Geppert, S. Hu, B. Kuan, P. Liu, R. Nguyen, Y. C. Yeo, A. Geiger _et al._ , “Project autovision: Localization and 3d scene perception for an autonomous vehicle with a multi-camera system,” _arXiv preprint arXiv:1809.05477_ , 2018.
  * [24] I. Sárándi, T. Linder, K. O. Arras, and B. Leibe, “Synthetic occlusion augmentation with volumetric heatmaps for the 2018 eccv posetrack challenge on 3d human pose estimation,” _arXiv preprint arXiv:1809.04987_ , 2018.
  * [25] H. Vu, T. Eftestøl, K. Engan, J. Eilevstjønn, L. B. Yarrot, J. E. Linde, and H. L. Ersdal, “Automatic detection and parameterization of manual bag-mask ventilation on newborns,” _IEEE journal of biomedical and health informatics_ , vol. 21, no. 2, pp. 527–538, 2017.
  * [26] MATLAB and I. L. Computer Vision System Toolbox, “The mathworks, inc., natick, massachusetts, united states.”
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [28] T.-Y. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie, “Feature pyramid networks for object detection.” in _CVPR_ , vol. 1, no. 2, 2017, p. 4.
  * [29] L. Čehovin, A. Leonardis, and M. Kristan, “Visual object tracking performance measures revisited,” _IEEE Transactions on Image Processing_ , vol. 25, no. 3, pp. 1261–1274, 2016.