# Reinforcement Learning Based Optimal Camera Placement for Depth Observation
of Indoor Scenes

Yichuan Chen1, Manabu Tsukada1, and Hiroshi Esaki1  1 Graduate School of
Information Science and Technology, The University of Tokyo, Tokyo, Japan
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Exploring the most task-friendly camera setting—optimal camera placement (OCP)
problem—in tasks that use multiple cameras is of great importance. However,
few existing OCP solutions specialize in depth observation of indoor scenes,
and most versatile solutions work offline. To this problem, an OCP online
solution to depth observation of indoor scenes based on reinforcement learning
is proposed in this paper. The proposed solution comprises a simulation
environment that implements scene observation and reward estimation using
shadow maps and an agent network containing a soft actor-critic (SAC)-based
reinforcement learning backbone and a feature extractor to extract features
from the observed point cloud layer-by-layer. Comparative experiments with two
state-of-the-art optimization-based offline methods are conducted. The
experimental results indicate that the proposed system outperforms seven out
of ten test scenes in obtaining lower depth observation error. The total error
in all test scenes is also less than 90% of the baseline ones. Therefore, the
proposed system is more competent for depth camera placement in scenarios
where there is no prior knowledge of the scenes or where a lower depth
observation error is the main objective.

###### Index Terms:

Camera placement, Deep reinforcement learning, Point cloud data, Shadow map,
Soft actor-critic

##  I Introduction

Industrial automation using cyber-physical systems is being researched and
developed for the next industrial revolution (Industry 4.0). These systems
often require the use of automated sensors to complete the information
gathering in the workplace. Depth cameras are a kind of such automated
sensors. In many application scenarios of depth cameras such as surveillance
or 3D reconstruction, depth information obtained by a single depth camera is
insufficient for the task. Therefore, it is necessary to introduce multiple
depth cameras to conduct a comprehensive observation of the observed object to
overcome this limitation; however, it leads to another problem: what
arrangement should be employed for these depth cameras to achieve this goal
more effectively. This problem is referred to as the optimal camera placement
(OCP) problem. The solutions for the OCP problem can be widely applied to the
automatic planning of various multicamera application scenarios in Industry
4.0, which can help improve the adaptability and resource utilization of
relevant systems in the working environment.

One of the main ideas for general OCP is to determine the possible solutions
in the candidate camera placement pool. Each element of the candidate pool
combines a set of camera parameters such as camera positions or orientations,
which are obtained by discretizing the possible range of parameters. [1] uses
an idea based on particle swarm optimization (PSO), which simulates the
information-sharing behavior of migrating animals to search for the optimum.
Many inertial particles move and search in parallel in the camera parameter
candidate pool with a tendency to the current global optimum. Since the camera
parameter candidate pool is a discrete solution space, it uses a variant of
the PSO algorithm called binary PSO. Another two similar research studies
focused on [2] artificial bee colonies and [3] differential evolution. Other
studies that do not discretize camera parameters use greedy search algorithms
to find the OCP. [4] proposed trans-dimensional simulated annealing (TDSA)
that uses a probabilistic decision model to enable the increase or decrease in
the number of cameras to be two extra search directions, i.e., the dimension
of the model becomes the model variable to be optimized.

In contrast to exploring general solutions, other studies focus on developing
dedicated OCP solutions for specific problems. One such example is the most
researched specific issue of OCP, i.e., surveillance. In contrast to the
research on applying general solutions to surveillance problems, many studies
address specific aspects of the surveillance problem [5, 6, 7, 8]. After the
development of the unmanned aerial vehicle (UAV), a considerable amount of
research has focused on the placement of UAVs [9] to obtain a solution for
maximizing the area covered by UAVs while considering energy consumption,
which is a critical metric to UAVs. Some other researchers studied tasks
similar to depth cameras placement. [10] provides an approach to the minimal
geometry reconstruction error for an object whereas [11] mentions the method
to optimize the camera placement for object recognition. However, they are not
suitable for larger-scale complex objects (such as rooms) and require the
premeasurement of the target object.

Although many OCP solutions in various scenarios have been listed above, they
are unsuitable for our task because of their specialization. We expect to
apply OCP to the depth observation of indoor scenes. The working principles of
different types of depth cameras are different; however, all depth cameras
require the high-precision perception of the observation target, which limits
their effective range. Depth cameras designed for indoor scenes can only
guarantee the completeness and accuracy of the results within a few meters.
This limitation makes depth cameras more dependent on suitable camera
placement than traditional optical cameras. Further, the use of existing OCP
frameworks to plan the placement of depth cameras may be insufficient because
only a few of the existing OCP frameworks are designed to ensure the quality
of the depth observations. Besides, most existing OCP algorithms require prior
knowledge of scene information as input, such as floor plans or pre-measured
three-dimensional data, which prevents them from online usage.

The main contributions of this study are summarized as follows:

  * •

Proposed a reinforcement learning (RL) based camera placement system
specialized for depth observation in indoor scenes.

  * •

Designed a sufficiently fast simulation environment required by training.

  * •

Designed an agent network for learning the placement of depth cameras; this
network can deal with unstructured point cloud data.

  * •

Conducted comparative experiments with traditional offline approaches, which
shows that the proposed system as an online algorithm can achieve more
accurate depth observations than offline algorithms.

The rest of this paper is organized as follows. Section II provides the
detailed definition of the research problem. Section III presents an overview
and the detailed design of the proposed system from two parts of the system:
the simulation environment and agent network. Subsequently, Section IV
describes the conducted experiments and the composition of the used dataset.
Finally, the paper is concluded in Section V.

##  II Problem Definition

For simplicity, we assume that all cameras are omnidirectional, and therefore,
we use only camera positions as variable camera parameters. Omnidirectional
depth cameras are already widely used in depth observation research, such as
[12, 13, 14]. Section III-B1 shows that our proposed system can be easily
transferred to regular non-omnidirectional depth cameras. Another assumption
considered for the sake of simplicity is that the number of cameras is fixed
during the optimization process.

Based on these assumptions, the camera settings to be optimized include the
camera positions. Therefore, the target of the depth camera placement for
depth observation in indoor scenes can be given as finding the camera
positions $\mathbf{p}$ as

$\operatorname*{arg\,min}_{\mathbf{p}\in\mathbb{S}}D(\mathbf{p}\,|\,\mathbf{C}),$
(1)

where $\mathbb{S}$ denotes the valid value range of $\mathbf{p}$, i.e., the
whole indoor space; $\mathbf{C}$ is the point cloud data that represents the
scene; $D(\mathbf{p}\,|\,\mathbf{C})$ is the difference between the ground
truth and depth images observed by cameras placed as $\mathbf{p}$, and the
research target is to minimize this difference.

The difference $D_{\mathbf{j}}(\mathbf{p}\,|\,\mathbf{C})$ between the depth
camera observation and the ground truth at a given viewpoint $\mathbf{j}$ can
be intuitively defined as the sum of pixel-wise differences on the depth image
as

$D_{\mathbf{j}}(\mathbf{p}\,|\,\mathbf{C})=\sum_{pixel\in\mathcal{I}_{\mathbf{j}}}||\,d_{pixel}(\mathbf{p}\,|\,\mathbf{C})-d^{{}^{\prime}}_{pixel}(\mathbf{C})\,||,$
(2)

where $\mathcal{I}_{\mathbf{j}}$ denotes the depth image observed at
$\mathbf{j}$; $d_{pixel}(\mathbf{p}\,|\,\mathbf{C})$ represents the pixel
value on the depth image calculated with the partial point cloud observed by
cameras at $\mathbf{p}$, and $d^{{}^{\prime}}_{pixel}(\mathbf{C})$ represents
the pixel value on the depth image calculated with the entire point cloud,
i.e., the ground truth.

Ideally, $D(\mathbf{p}\,|\,\mathbf{C})$ should be defined as the sum of
$D_{\mathbf{j}}(\mathbf{p}\,|\,\mathbf{C})$ for all viewpoints $\mathbf{j}$ in
$\mathbb{S}$. However, this definition is difficult to calculate in practice.
Therefore, the appropriate approach is to consider some sampling points in
$\mathbb{S}$, and define $D(\mathbf{p}\,|\,\mathbf{C})$ as the sum of
$D_{\mathbf{j}}(\mathbf{p}\,|\,\mathbf{C})$ on these sampling points. With
this definition, the target can be formulated as

$D(\mathbf{p}\,|\,\mathbf{C})=\sum_{\mathbf{j}\in\mathbf{V}}D_{\mathbf{j}}(\mathbf{p}\,|\,\mathbf{C}),$
(3)

where $\mathbf{V}$ represents the set of sampling points.

##  III Reinforcement Learning System

###  III-A System Overview

The overview of the training routine of the proposed system is illustrated in
Figure 1. Similar to other reinforcement learning tasks, the training process
of this system is conducted through an iterative interaction between the agent
and the environment.

Figure 1: Overview of training routine.

The soft actor critic (SAC) [15] algorithm is adopted as the agent backbone.
The agent network comprises three parts: an actor network representing the
strategy responsible for generating actions from the extracted features and
estimated Q-value; a critic network responsible for the estimation of Q-value
from the extracted features and action; and a feature extractor that processes
point cloud data. The actor network and critic network use the implementation
provided by the stable baselines3 [16], and therefore, they are not described
in detail in this paper.

The actions produced by the agent network are designed to be camera movements
in each turn. The agent generates actions for all camera globally. Given the
actions output by the agent, the simulation environment becomes responsible
for outputting the corresponding observation and reward. For our research, the
output is the partial point cloud observed until the current turn and the
reward of the movements (including the difference of space coverage and depth
observation error compared to the last turn).

The evaluation routines are similar to the training routines that involve
repeated interactions between the agent and the environment. However, the
estimation of the Q-value and computation of the rewards on which it depends
are unnecessary because there is no need to consider backpropagation. The
modules related to them can be discarded from Figure 1. Moreover, when used in
real-world applications, the entire simulated environment is replaced by the
natural environment and only the agent network is required.

The camera movements generated by the agent gradually converge to zero after a
certain number of interaction cycles. Thus, the environment is set with the
maximum number of interactions. The loop is stopped after reaching this
maximum value. The camera positions at the end are used as the final positions
to evaluate the performance of the depth camera placement using (3).

###  III-B Simulation Environment

For our research, the required simulation environment must perform the
following functions while considering both speed and accuracy:

  * •

Scene point cloud observation

  * •

Space coverage estimation

  * •

Depth observation error estimation

These functions correspond to the three dark-colored modules in the simulation
environment in Figure 1. The proposed system implements these functions based
on shadow maps, which correspond to the light-colored module in the simulated
environment in Figure 1.

####  III-B1 Shadow Maps Calculator

Basic shadow maps are similar to depth images because the depth information
along different directions is stored pixel-by-pixel. The name shadow maps is
based on its application in real-time rendering. Shadow maps can be used to
store distances from the nearest obstacle to the light source. Further, shadow
maps can be used to evaluate whether the light source illuminates a specific
fragment by comparing their distance with the shadow map value.

Figure 2: Shadow maps for omnidirectional cameras. The left part represents
the shadow map, and the right represents the corresponding coverage volume in
the three-dimensional space of the painted one element in the left. $\alpha$
and $\beta$ are the indices of the elements in the shadow map and their
corresponding spherical coordinate values. $\mathbf{P}$ is the camera
position. The radius of the ball on the right is the effective range of the
depth camera. $\mathbf{V}_{\alpha,\,\beta}$ is a quadrangular pyramid-like
body surrounded by four radii (corresponding to the four vertices of the
painted cell in the left image) and the spherical surface. The value of the
painted cell is the minimum distance from all points in
$\mathbf{V}_{\alpha,\,\beta}$ to the center of the sphere.

Shadow maps can be calculated directly from the point cloud data. Further,
shadow maps can be defined in a spherical coordinate system with the camera as
the center and independent of the radial distance because users are assumed to
use omnidirectional cameras. The definition and calculation of the shadow map
are shown in Figure 2 and (4).

$\textbf{SM}_{i}[\alpha,\,\beta]=\min_{\mathbf{point}\in\mathbf{V}_{i,\,\alpha,\,\beta}}||\,\mathbf{point}-\mathbf{P_{i}}\,||_{2}$
(4)

The above shadow maps are defined for omnidirectional depth cameras; however,
they can still be used for normal (non-omnidirectional) depth cameras. The
calculation is also easy for normal scenarios; only slight modifications (4)
as required.

$\textbf{SM}_{i}[\alpha,\,\beta]=\min_{\mathbf{point}\in\mathbf{V}_{i,\,\alpha,\,\beta}\cap\mathbf{F_{i}}}||\,\mathbf{point}-\mathbf{P_{i}}\,||_{2},$
(5)

where $\mathbf{F_{i}}$ denotes the viewing cone of the i-th camera. This
change is essentially as an additional judgment. The shadow map is updated
only when the point falls within the viewing cone of the camera.

Each point is considered a rectangle when calculating the shadow maps. Not
only the cell in which the point falls but also the surrounding cells are
updated. The rectangles are centered on the cell where the point is located;
however, their size is not fixed. Each point is considered a cube whose side
length is twice the distance from the point to the nearest neighbor in the
point cloud. Without this amendment, the gap between the points may uncover
points that should not be visible behind them.

####  III-B2 Visibility Test in Following Modules

It is simple to conduct a visibility analysis to determine if a point is
visible to a camera once the shadow map is calculated. The approach is to
identify the cell where the point falls in the shadow map of the camera and to
determine if the distance to that camera is less than the value stored in the
cell. A small compensation is added to the shadow map value. Thus, the visible
area is slightly expanded, and it covers visible points that would have been
erroneously eliminated because of the sampling accuracy.

####  III-B3 Scene Point Cloud Observer

One step to obtain the partial point cloud observed in the current turn is
checking the visibility of all cameras for each point in the point cloud. If a
point is visible by any camera, it is added to the list of observed points.
Finally, this list is output as the simulation of the observation results. The
observation flag is set to false each time the environment is reset. Once the
flag of some point is set to true, it will always be true, which indicates
that this point will be included in the observation.

####  III-B4 Space Coverage Calculator

The space is discretized to estimate the space coverage. The bounding box of
the scene point cloud is considered as the entire space, and it is then
uniformly divided into small cells. Each cell in the grid determines whether
it is in the visible area by checking the visibility of its center point. The
number of cells that pass the visibility check is then multiplied by the
volume of an individual cell to obtain the space coverage.

####  III-B5 Depth Error Calculator

Calculating the depth error at a single point is straightforward. Owing to the
similarity of shadow maps and depth images, they can be directly considered
the depth information observed at a certain point. Therefore, to compare the
depth observation obtained by placed cameras and the ground truth, shadow maps
are calculated with the partial point cloud visible to placed cameras and the
entire point cloud, respectively. Then, the depth error is calculated by
accumulating the differences of the depth values pixel-by-pixel.

Nevertheless, it is difficult to compute the depth error for the entire space.
The depth errors at some sampling points are used to represent the overall
depth error. Then, the summation of the depth errors at all sampling points
can estimate the overall depth error. Sampling points are placed in the center
and four corners ($\frac{2}{3}$ of the position from the center point to the
vertex of the bounding box) of the bounding box of the scene. The height of
the sampling points is half of the height of the scene bounding box.

###  III-C Agent Network

The critical parts of the agent network include a feature extractor module and
a reward mapping module, excluding the actor and critic networks, which are
not our contributions. The feature extractor module extracts latent features
from the input observed point clouds. The reward mapping module preprocesses
the rewards from the environment to guide the training direction better.

####  III-C1 Feature Extractor

Inspired by other networks that extract features from point clouds [17, 18],
the proposed feature extractor extracts point cloud features level-by-level
through sampling and grouping. The core part of this module is the extraction
unit, which extracts a set of key points and features from the input point
cloud. The extraction unit is repeated three times in the feature extractor to
extract the features of the different levels. The last layer is set to have
only one key point, which extracts the global features of the point cloud. The
features extracted from the point cloud are concatenated with manually
extracted features (bounding box of observed point cloud and current camera
positions) as the input of the subsequent actor and critic network.

Figure 3: Forward data flow diagram of the extraction unit in feature
extractor module.

The specific structure of the extraction unit is shown in Figure 3. The input
point cloud is first sampled to obtain $\mathbf{K}$ key points, and then, the
point clouds are grouped with these key points as the center points. Each
group has $\mathbf{G}$ points. Sampling and grouping operations are the same
as in [18]. These operations select some points from the point cloud based on
indices, which are constant for each point cloud. Therefore, these indices are
precomputed in the observation cache module and passed into the extraction
unit when extracting features.

Then, the grouped points are passed through a multilayer perceptron (MLP) with
shared weights to extract latent features. The MLP comprises three layers with
the same output channel number $\mathbf{C^{\prime}}$. The following maximum
pooling layer filters out the maximum components of each group in the output
of the MLP. The filtered components are concatenated with the center point
position of each group. Therefore, the final output of the feature extraction
unit is similar to a point cloud containing $\mathbf{K}$ points, and each
point has 3 position channels and $\mathbf{C^{\prime}}$ feature channels.

####  III-C2 Reward Mapping

The simulation environment provides two reward outputs: space coverage SC and
depth observation error DOE. Further, a penalty term P for preventing the
actions of moving the camera outside the room is added. This term is a Boolean
variable, and it is calculated whenever the rewards are calculated. When any
camera position is outside the bounding box of the scene, P is set to 1.
Otherwise, it is set to 0. These three reward terms are combined into a reward
scalar Rew as

$\begin{split}\textit{Rew}\,&(\,\textit{SC},\,\textit{DOE},\,\textit{P})=\\\
&(K_{\textit{SC}}\times\Delta\textit{SC}+K_{\textit{DOE}}\times\Delta\textit{DOE})\times(1-\textit{P})\\\
&+K_{\textit{P}}\times\textit{P},\end{split}$ (6)

where $\Delta\textit{SC}$ and $\Delta\textit{DOE}$ represent the difference in
space coverage and depth observation error compared to the previous turn.
$K_{SC}$, $K_{DOE}$, and $K_{P}$ denote the weights of space coverage, depth
observation error, and penalty terms, respectively.

When using (6) directly for the loss calculation of the Q-network, the agent
rapidly expands the coverage area of the camera at the beginning, and this is
accompanied by a significant decrease in the depth error. Then, the agent may
appear sluggish. The agent keeps the cameras immobile even though more
suitable (lower depth error) positions may exist near the cameras.

For this problem, we propose a solution that involves mapping rewards by
functions with sufficiently large positive first-order derivatives and
positive second-order derivatives. In the case of already large rewards,
sufficiently large positive first-order derivatives allow even small
increments of rewards to produce significant increments of mapped rewards.
Furthermore, the positive second-order derivative makes this effect
increasingly enhanced as the base reward becomes larger. In this specific
implementation, we use the third power function as the mapping function.

##  IV Experiments

The experiments described in this paper are performed on an Ubuntu 18.04
server with two AMD Ryzen Threadripper 3970X CPUs, 128GB RAM, and one NVIDIA
TITAN RTX graphics card.

###  IV-A Dataset

Our training and evaluating datasets are based on the S3DIS dataset [19],
which is a large-scale indoor scene dataset obtained by scanning the real-
world campus rooms by depth cameras, including 3 buildings, 6 areas, and a
total of 272 scenes. The point cloud data of scenes from area 1 to area 5 are
used to construct the training dataset, and data from area 6 are used to
construct the evaluation dataset.

However, not all scenes in the S3DIS dataset are suitable for the experiments.
10 scenes from each area are selected based on the following rules to form the
experiment datasets.

  * •

Scene shape completeness: The selected scene must have closed walls and
ceilings. An open space may interfere with the measurement of depth
observation errors and evaluation of whether the camera is inside the scene.
This rule only applies to the training dataset. In order not to introduce bias
to the evaluation set, the evaluation set still contains open scenes.

  * •

Scene size: The selected scene must be of the appropriate size to be roughly
covered by the chosen number of cameras without too much omission or overlap.
If the scene is too small, then the camera can easily cover the whole scene.
If the scene is too large, a good depth observation cannot be obtained no
matter how the cameras are placed because the number of cameras is fixed.
Optimizing camera placement in both cases is not meaningful, and therefore,
they cannot be used for our experiments.

  * •

Flat room: We restrict camera positions in experiments on a plane parallel to
the ground to simplify the calculation. This restriction makes the simplified
model handle flat rooms. Therefore, we excluded non-flat rooms such as
stairwells from the training and evaluation sets.

Further, segmented scenes in the S3DIS dataset are all axis-aligned. To avoid
this property from introducing bias into datasets, two additional scenes for
each scene are added by rotating the scene by 60° and 120° around the center
of its bounding box. Thus, the final dataset used for the experiments contains
$60\times 3=180$ scenes, 150 of which are used for training and 30 for
evaluation.

###  IV-B Comparative Experiment

####  IV-B1 Baselines

Two versatile non-deep-learning OCP solutions, TDSA [4] and BPSO [1] are used
as baselines to compare with our proposed system.

####  IV-B2 Metrics

The primary metric of the comparative experiment is the depth observation
error, i.e., (3), which is calculated in the simulation environment.
Therefore, we can directly use the depth observation error output from the
simulation environment as the evaluation metric.

####  IV-B3 Results

Figure 4: Depth observation error of each approach in each evaluation scene. TABLE I: Depth error sum of all evaluation scenes Approach | Ours | TDSA | BPSO
---|---|---|---
Depth Error Sum | 48.4 | 56.6 | 54.3
TABLE II: Computation time for each approach Approach | Ours (Training) | TDSA | BPSO
---|---|---|---
Time |  $\sim$ 9 h |  $\sim$ 6 h |  $\sim$ 13 h

Figure 4 shows the depth observation error obtained by each approach in each
evaluation scene. The error of each scene is the average of the depth
observation errors obtained by itself and those in the two derived scenes.
TABLE I shows the sum of errors for all scenes. The proposed system performs
better in 7/10 scenes, and the total error is the smallest. This result shows
that our system is better than general algorithms on camera placement for
depth observation tasks. TABLE II summarizes the computation times for each
approach. The time required by the baselines is the computation time for all
30 evaluation scenes. The results indicate that even if our approach is
considered an offline algorithm, its time consumption remains acceptable.

###  IV-C Training Statistics

(a) Performance

(b) Losses

Figure 5: Performance and losses of agent measured during training.

Figure 5 shows the additional details of the training process. Figure 5 shows
that after one million interactions, the performance of the agent has almost
stabilized, and the losses of modules gradually converged.

###  IV-D Example Visualization

(a)

(b)

Figure 6: Examples of camera placement results and trajectories output by the
proposed system.

Figure 6 shows some examples of camera placement results and trajectories
output by the proposed system in the test scenes. The red balls, white lines,
and semi-transparent white balls represent the final camera positions output
by the system, trajectories of the camera movement during the repeated
interactions with the environment, and influential ranges of the cameras,
respectively.

##  V Conclusion

In this study, a reinforcement learning-based OCP system for depth observation
tasks in indoor scenes was proposed. As an online algorithm, it behaves better
in obtaining more accurate depth observations compared to offline baselines.
Therefore, it is more competent for depth camera placement in scenarios where
there is no prior knowledge of the scenes or where a lower depth observation
error is the main objective.

One of the main limitations of the proposed system is that it can be overly
constrained to some corners of the scene in some scenes, thus neglecting other
regions. This is the main factor why the system did not beat the baselines in
all test scenes. Another major limitation is that it is restricted to using a
fixed number of depth cameras. Therefore, future work may include introducing
collaborative learning and exploring a way to encode the movements of
variable-length cameras.

## References

  * [1] Y. Morsly, N. Aouf, M. S. Djouadi, and M. Richardson, “Particle swarm optimization inspired probability algorithm for optimal camera network placement,” _IEEE Sensors Journal_ , vol. 12, no. 5, pp. 1402–1412, 2011\.
  * [2] D. Chrysostomou and A. Gasteratos, “Optimum multi-camera arrangement using a bee colony algorithm,” in _IEEE International Conference on Imaging Systems and Techniques Proceedings_. IEEE, 2012, pp. 387–392.
  * [3] B. Zhang, X. Zhang, X. Chen, and Y. Fang, “A differential evolution approach for coverage optimization of visual sensor networks with parallel occlusion detection,” in _IEEE International Conference on Advanced Intelligent Mechatronics (AIM)_. IEEE, 2016, pp. 1246–1251.
  * [4] J. Liu, S. Sridharan, C. Fookes, and T. Wark, “Optimal camera planning under versatile user constraints in multi-camera image processing systems,” _IEEE Transactions on Image Processing_ , vol. 23, no. 1, pp. 171–184, 2013\.
  * [5] K. Yabuta and H. Kitazawa, “Optimum camera placement considering camera specification for security monitoring,” in _IEEE International Symposium on Circuits and Systems_. IEEE, 2008, pp. 2114–2117.
  * [6] A. T. Murray, K. Kim, J. W. Davis, R. Machiraju, and R. Parent, “Coverage optimization to support security monitoring,” _Computers, Environment and Urban Systems_ , vol. 31, no. 2, pp. 133–147, 2007.
  * [7] Y. Yao, C.-H. Chen, B. Abidi, D. Page, A. Koschan, and M. Abidi, “Can you see me now? sensor positioning for automated and persistent surveillance,” _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , vol. 40, no. 1, pp. 101–115, 2009.
  * [8] K. R. Konda and N. Conci, “Optimal configuration of ptz camera networks based on visual quality assessment and coverage maximization,” in _Seventh International Conference on Distributed Smart Cameras (ICDSC)_. IEEE, 2013, pp. 1–8.
  * [9] D. Zorbas, L. D. P. Pugliese, T. Razafindralambo, and F. Guerriero, “Optimal drone placement and cost-efficient target coverage,” _Journal of Network and Computer Applications_ , vol. 75, pp. 16–31, 2016.
  * [10] G. Olague and R. Mohr, “Optimal camera placement for accurate reconstruction,” _Pattern recognition_ , vol. 35, no. 4, pp. 927–944, 2002\.
  * [11] S. D. Roy, S. Chaudhury, and S. Banerjee, “Active recognition through next view planning: a survey,” _Pattern Recognition_ , vol. 37, no. 3, pp. 429–446, 2004.
  * [12] J. Huang, Z. Chen, D. Ceylan, and H. Jin, “6-dof vr videos with a single 360-camera,” in _IEEE Virtual Reality (VR)_. IEEE, 2017, pp. 37–44.
  * [13] B. Luo, F. Xu, C. Richardt, and J.-H. Yong, “Parallax360: Stereoscopic 360 scene representation for head-motion parallax,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 24, no. 4, pp. 1545–1553, 2018.
  * [14] A. Serrano, I. Kim, Z. Chen, S. DiVerdi, D. Gutierrez, A. Hertzmann, and B. Masia, “Motion parallax for 360 rgbd video,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 25, no. 5, pp. 1817–1827, 2019.
  * [15] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1861–1870.
  * [16] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann, “Stable baselines3,” https://github.com/DLR-RM/stable-baselines3, 2019\.
  * [17] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 652–660.
  * [18] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” _arXiv preprint arXiv:1706.02413_ , 2017.
  * [19] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese, “3d semantic parsing of large-scale indoor spaces,” in _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition_ , 2016.


11institutetext:  CVIT, International Institute of Information Technology,
Hyderabad, India
email<EMAIL_ADDRESS>
# New Performance Measures for Object Tracking under Complex Environments

Ajoy Mondal

###### Abstract

Various performance measures based on the ground truth and without ground
truth exist to evaluate the quality of a developed tracking algorithm. The
existing popular measures - average center location error (acle) and average
tracking accuracy (ata) based on ground truth, may sometimes create confusion
to quantify the quality of a developed algorithm for tracking an object under
some complex environments (e.g., scaled or oriented or both scaled and
oriented object). In this article, we propose three new auxiliary performance
measures based on ground truth information to evaluate the quality of a
developed tracking algorithm under such complex environments. Moreover, one
performance measure is developed by combining both two existing measures (acle
and ata) and three new proposed measures for better quantifying the developed
tracking algorithm under such complex conditions. Some examples and
experimental results conclude that the proposed measure is better than
existing measures to quantify one developed algorithm for tracking objects
under such complex environments.

###### Keywords:

Tracking, evaluation, performance measure, scale, orientation and complex
environment.

##  1 Introduction

Object tracking in a video sequence has been an active research area in
computer vision during the last few decades due to its several real-life
application yilmaz_survey_2006 ; survey_2013 ; benchmark_wu2013 ;
smeulders2014visual ; li2018deep ; fiaz2019handcrafted . Every year, large
number of tracking algorithms have been developed and corresponding papers are
published in journal and conferences. The quality of the developed algorithm
is compared with state-of-the-art techniques based on some performance
measures. When the developed algorithm is compared with state-of-the-art
techniques, several questions may be arises. Are there standard sequences that
can be used for evaluation purposes? Is there a standard evaluation protocol?
What kind of performance measure should be used? Unfortunately, there are
currently no definite answers to all these questions.

However, several performance measures are proposed to quantify the quality of
a newly developed tracking algorithm as compared to state-of-the-art
techniques performance_evaluation_2008 ; performance_evaluation_2009 ;
vcehovin2016visual ; yin2007performance . Center location error (cle) and area
overlap (aol) are basically considered to evaluate the performance of a
tracking algorithm. These two measures are useful to compare performances of
different tracking algorithms. Other measures like precision plot (pp) based
on center location error and success plot (sp) based on area overlap are also
considered to evaluate the performance of the tracking algorithm throughout
the complete sequence. All these measures may sometimes be unable or create
confusion to quantify the quality of an algorithm for tracking single object
under complex environments (like object with shape deformation, scaled object,
oriented object, both scaled and oriented object). In this article, we propose
three new complementary measures: error in estimated height of optimal tracker
($E_{h}^{t}$), error in estimated width of optimal tracker ($E_{w}^{t}$) and
error in estimated orientation angle of optimal tracker ($E_{\theta}^{t}$);
and another composite measure based on three proposed and two existing
measures to evaluate the performance of a tracking algorithm under such
complex environments. Various examples and experimental results conclude that
the proposed composite measure is better to quantify the quality of an
algorithm than the existing measures.

##  2 Existing Performance Measures based on Ground Truth Information

Two performance measures namely center location error ($E_{C}^{t}$) and area
overlap ($E_{A}^{t}$) are popularly considered to quantify a tracking
algorithm. They can be calculated as

$\displaystyle\begin{array}[]{l}E_{C}^{t}=\|C_{g}^{t}-C^{t}\|;\\\ \\\
E_{A}^{t}=\frac{T_{g}^{t}\cap T^{t}}{T_{g}^{t}\cup T^{t}},\end{array}$ (4)

where $T_{g}^{t}$ is a ground truth rectangular tracker with center
$C_{g}^{t}$ at $t^{th}$ frame and $T^{t}$ is a rectangular tracker with center
$C^{t}$ at $t^{th}$ frame generated by a tracking algorithm. Similarly average
center location error ($E_{C}^{avg}$) and average area overlap or average
tracking accuracy ($E_{A}^{avg}$) over all the frames can be calculated as

$\displaystyle\begin{array}[]{l}E_{C}^{avg}=\frac{1}{N}\sum_{t=1}^{N}\|C_{g}^{t}-C^{t}\|;\\\
\\\ E_{A}^{avg}=\frac{1}{N}\sum_{t=1}^{N}\frac{T_{g}^{t}\cap
T^{t}}{T_{g}^{t}\cup T^{t}},\end{array}$ (8)

if video contains $N$ number of frames. For a good tracking algorithm,
$E_{C}^{t}$ and $E_{C}^{avg}$ should be $0$ whereas $E_{A}^{t}$ and
$E_{A}^{avg}$ should be $1$. Some plots (e.g., precision plot and success
plot) based on center location error and area overlap, respectively are
considered to evaluate performance of the tracking algorithms.

#### Precision Plot:

It is considered to measure the overall tracking performance Babenko2010 ;
henriques2015high ; benchmark_wu2013 . It shows that the percentage of frames
where the estimated location of the object is within the given threshold
distance with respect to the ground truth.

#### Success Plot:

To measure the performance of an algorithm on a sequence of frames, we count
the number of successful frames whose overlap $E_{A}^{t}$ is larger than a
given threshold value. The success plot shows the ratios of successful frames
at different thresholds varied from $0$ to $1$. Success rate value at a
specific threshold (e.g., $0.5$) may not always be a good representative for
evaluating the tracker’s performance benchmark_wu2013 . Instead of that, the
area under the curve (auc) of each success plot is considered to evaluate the
performance of the tracking algorithm benchmark_wu2013 .

Though these measures are popularly considered to evaluate the performance of
a tracking algorithm, they may sometimes unable or create confusion to
quantify the quality of a developed algorithm for tracking an object under
some complex environments (e.g., scaled or oriented or both scaled and
oriented object). The following section presents some examples where the
existing performance measures fail or create confusion to evaluate the
performance of a tracking algorithm under such complex environments.

##  3 Problems of Existing Evaluation Measures for Scaled Object Tracking

(a) (b)

Figure 1: Example 1 - area overlap between ground truth tracker and trackers
obtained by (a) Alg1 and (b) Alg2.

###  3.1 Example1:

Here, we consider a simple example (see Figure 1). Suppose $T_{g}^{t}$ (green
colored rectangle) with center $C_{g}^{t}$, height $h_{g}^{t}$ and width
$w_{g}^{t}$ be the ground truth tracker. Again let $T_{1}^{t}$ (red colored
rectangle) with center $C_{1}^{t}$, height $h_{1}^{t}$ and width $w_{1}^{t}$
and $T_{2}^{t}$ (pink colored rectangle) with center $C_{2}^{t}$, height
$h_{2}^{t}$ and width $w_{2}^{t}$ be two trackers generated by two different
(scaled object) tracking algorithms Alg$1$ and Alg$2$. Here, we also assume
that the trackers $T_{g}^{t}$ and $T_{1}^{t}$ have same center locations but
different heights and widths (see Figure 1(a)). Therefore, we have
$C_{g}^{t}=C_{1}^{t}$, $h_{g}^{t}<h_{1}^{t}$ and $w_{g}^{t}<w_{1}^{t}$. Now
let $w_{1}^{t}=w_{g}^{t}+w^{t}$ and $h_{1}^{t}=h_{g}^{t}+h^{t}$. We also
assume that the trackers $T_{g}^{t}$ and $T_{2}^{t}$ have same heights but
different center locations and widths (see Figure 1(b)). Therefore, we have
$C_{g}^{t}<C_{2}^{t}$, $h_{g}^{t}=h_{2}^{t}$ and $w_{g}^{t}<w_{2}^{t}$. Now
let $C_{2}^{t}=C_{g}^{t}+\alpha^{t}$ and $w_{2}^{t}=w_{g}^{t}+w^{t}$.

Now area overlap ($E_{A_{1}}^{t}$) between ground truth tracker $T_{g}^{t}$
and tracker $T_{1}^{t}$ generated by Alg$1$ at $t^{th}$ frame, is calculated
as

$\displaystyle\begin{array}[]{l}E_{A_{1}}^{t}=\frac{{T_{g}^{t}\cap
T_{1}^{t}}}{{T_{g}^{t}\cup
T_{1}^{t}}}=\frac{{w_{g}^{t}h_{g}^{t}}}{{\left({w_{g}^{t}+w^{t}}\right)\left({h_{g}^{t}+h^{t}}\right)}}\\\
=\frac{{w_{g}^{t}h_{g}^{t}}}{{w_{g}^{t}h_{g}^{t}+h^{t}w_{g}^{t}+w^{t}h_{g}^{t}+w^{t}h^{t}}}=\frac{{w_{g}^{t}h_{g}^{t}}}{{s^{t}+h^{t}w_{g}^{t}+w^{t}h^{t}}},\end{array}$
(11)

where $s^{t}=w_{g}^{t}h_{g}^{t}+w^{t}h_{g}^{t}$.

Again area overlap ($E_{A_{2}}^{t}$) between ground truth tracker $T_{g}^{t}$
and tracker $T_{2}^{t}$ generated by Alg$2$ at $t^{th}$ frame, is calculated
as

$\displaystyle\begin{array}[]{l}E_{A_{2}}^{t}=\frac{{T_{g}^{t}\cap
T_{2}^{t}}}{{T_{g}^{t}\cup
T_{2}^{t}}}=\frac{{w_{g}^{t}h_{g}^{t}}}{{w_{2}^{t}h_{2}^{t}}}=\frac{{w_{g}^{t}h_{g}^{t}}}{{\left({w_{g}^{t}+w^{t}}\right)\left({h_{g}^{t}}\right)}}=\frac{{w_{g}^{t}h_{g}^{t}}}{{s^{t}}},\end{array}$
(13)

since $s^{t}=w_{g}^{t}h_{g}^{t}+w^{t}h_{g}^{t}$.

Now center location errors $E_{C_{1}}^{t}$ and $E_{C_{2}}^{t}$ of trackers
$T_{1}^{t}$ and $T_{2}^{t}$ are calculated as

$\displaystyle\begin{array}[]{l}E_{C_{1}}^{t}=\left\|{C_{g}^{t}-C_{1}^{t}}\right\|=0,\end{array}$
(15)

since $C_{g}^{t}=C_{1}^{t}$.

$\displaystyle\begin{array}[]{l}E_{C_{2}}^{t}=\left\|{C_{g}^{t}-C_{2}^{t}}\right\|=\left\|{C_{g}^{t}-(C_{g}^{t}+\alpha^{t})}\right\|=\alpha^{t}.\end{array}$
(17)

Eqs. (11) and (13) conclude that $E_{A_{2}}^{t}>E_{A_{1}}^{t}$, i.e., tracker
$T_{2}^{t}$ has more area overlap with ground truth tracker $T_{g}^{t}$ than
tracker $T_{1}^{t}$ at $t^{th}$ frame. On the other hand, Eqs. (15) and (17)
conclude that $E_{c_{1}}^{t}<E_{C_{2}}^{t}$, i.e., tracker $T_{1}^{t}$ has
lesser center location error than tracker $T_{2}^{t}$ at $t^{th}$ frame. If we
consider center location error as a performance measure then tracker
$T_{1}^{t}$ is better than tracker $T_{2}^{t}$ at $t^{th}$ frame. If we
consider area overlap as a performance measure then tracker $T_{2}^{t}$ is
better than tracker $T_{1}^{t}$ at $t^{th}$ frame. If we consider both these
measures no decision can be made. However, tracker $T_{1}^{t}$ has lesser
center location error with lesser area overlap and tracker $T_{2}^{t}$ has
higher center location error with higher area overlap at $t^{th}$ frame.
Therefore, minimum center location error indicates maximum area overlap for
the fixed sized tracker is not true for the scaled tracker and performance
measures: center location error and area pverlap individually can make
ambiguous decision for the scaled tracker.

###  3.2 Example2:

a b

Figure 2: Example 2 - area overlap between ground truth tracker and tracker
obtained by (a) Alg$1$ (b) Alg$2$.

Here, we consider another simple example (see Figure 2). Suppose $T_{g}^{t}$
(green colored rectangle) with center $C_{g}^{t}$, height $h_{g}^{t}$ and
width $w_{g}^{t}$ be the ground truth tracker at $t^{th}$ frame. Again let
$T_{1}^{t}$ (red colored rectangle) with center $C_{1}^{t}$, height
$h_{1}^{t}$ and width $w_{1}^{t}$ and $T_{2}^{t}$ (pink colored rectangle)
with center $C_{2}^{t}$, height $h_{2}^{t}$ and width $w_{2}^{t}$ be two
trackers generated by two different (scaled object) tracking algorithms Alg$1$
and Alg$2$, respectively. Here, we also assume that the trackers $T_{g}^{t}$
and $T_{1}^{t}$ have same center location and width but different height (see
Figure 2(a)). Therefore, we have $C_{g}^{t}=C_{1}^{t}$, $w_{g}^{t}=w_{1}^{t}$
and $h_{g}^{t}>h_{1}^{t}$. Now let $h_{g}^{t}=h_{1}^{t}+h^{t}$. We also assume
that the trackers $T_{g}^{t}$ and $T_{2}^{t}$ have same center location and
height but different width (see Figure 2(b)). Therefore, we have
$C_{g}^{t}=C_{2}^{t}$, $h_{g}^{t}=h_{2}^{t}$ and $w_{g}^{t}>w_{2}^{t}$. Now
let $w_{g}^{t}=w_{2}^{t}+w^{t}$.

Now area overlap ($E_{A_{1}}^{t}$) between ground truth tracker $T_{g}^{t}$
and tracker $T_{1}^{t}$ generated by Alg$1$ at $t^{th}$ frame, is calculated
as

$\displaystyle\begin{array}[]{l}E_{A_{1}}^{t}=\frac{{T_{g}^{t}\cap
T_{1}^{t}}}{{T_{g}^{t}\cup
T_{1}^{t}}}=\frac{{h_{1}^{t}w_{1}^{t}}}{{h_{g}^{t}w_{g}^{t}}}=\frac{{(h_{g}^{t}-h^{t})w_{g}^{t}}}{{h_{g}^{t}w_{g}^{t}}}=1-\frac{h^{t}}{{h_{g}^{t}}}.\end{array}$
(19)

Again area overlap ($E_{A_{2}}^{t}$) between the ground truth tracker
$T_{g}^{t}$ and tracker $T_{2}^{t}$ generated by Alg$2$ at $t^{th}$ frame, is
calculated as

$\displaystyle\begin{array}[]{l}E_{A_{2}}^{t}=\frac{{T_{g}^{t}\cap
T_{2}^{t}}}{{T_{g}^{t}\cup
T_{2}^{t}}}=\frac{{w_{2}^{t}h_{2}^{t}}}{{w_{g}^{t}h_{g}^{t}}}=\frac{{(w_{g}^{t}-w^{t})h_{g}^{t}}}{{w_{g}^{t}h_{g}^{t}}}=1-\frac{w^{t}}{{w_{g}^{t}}}.\end{array}$
(21)

Now center location errors $E_{C_{1}}^{t}$ and $E_{C_{2}}^{t}$ of the trackers
$T_{1}^{t}$ and $T_{2}^{t}$ at $t^{th}$ frame, respectively are calculated as

$\displaystyle\begin{array}[]{l}E_{C_{1}}^{t}=\left\|{C_{g}^{t}-C_{1}^{t}}\right\|=0,\end{array}$
(23)

since $C_{g}^{t}=C_{1}^{t}$.

$\displaystyle\begin{array}[]{l}E_{C_{2}}^{t}=\left\|{C_{g}^{t}-C_{2}^{t}}\right\|=0,\end{array}$
(25)

since $C_{g}^{t}=C_{2}^{t}$.

Now if,

$\displaystyle\begin{array}[]{l}1-\frac{h^{t}}{h_{g}^{t}}=1-\frac{w^{t}}{w_{g}^{t}}\Rightarrow\frac{h^{t}}{w^{t}}=\frac{h_{g}^{t}}{w_{g}^{t}},\end{array}$
(27)

then $E_{C_{1}}^{t}=E_{C_{2}}^{t}$. Therefore, this example shows that both
these algorithms Alg$1$ and Alg$2$ are equally good.

##  4 Problems of Existing Evaluation Measures for Oriented Object Tracking

###  4.1 Example1:

a

b c

Figure 3: Example 1 - area overlap between ground truth oriented tracker and
fixed trackers obtained by (a) Alg1 (b) Alg2.

Here, we consider one example where the object is changing its orientation
from one frame to another, but the trackers $T_{1}$ and $T_{2}$ generated by
two different tracking algorithms Alg$1$ and Alg$2$, respectively are fixed.
We assume that ground truth tracker ($T_{g}$) is not fixed i.e., it changes
its orientation based on object orientation. Suppose, in $t^{th}$ frame, the
ground truth tracker $T_{g}^{t}$ (green colored rectangle) with center
$C_{g}^{t}$, height $h_{g}^{t}$ and width $w_{g}^{t}$, is look like as
depicted in Figure 3(a). After few number of frames, say in $r^{th}$ ($t<r$)
frame, the object changes its orientation (assume with $90^{o}$). And the
corresponding ground truth tracker $T_{g}^{r}$ (green colored rectangle) with
center $C_{g}^{r}$, height $h_{g}^{r}$ and width $w_{g}^{r}$ in the $r^{th}$
frame can be look like as in Figures 3(b) and (c). Two different tracking
algorithms (Alg$1$ and Alg$2$) generate two fixed sized trackers $T_{1}^{r}$
(dotted red colored rectangle) with center $C_{1}^{r}$, height $h_{1}^{r}$ &
width $w_{1}^{r}$ and $T_{2}^{r}$ (dotted pink colored rectangle) with center
$C_{2}^{r}$, height $h_{2}^{r}$ & width $w_{2}^{r}$ in the $r^{th}$ frame are
displayed in Figures 3(b) and (c), respectively. We also assume that
$C_{g}^{r}=C_{1}^{r}$, $C_{g}^{r}<C_{2}^{r}$, $h_{1}^{r}=h_{2}^{r}=w_{g}^{2}$,
$w_{1}^{r}=w_{2}^{r}=h_{g}^{r}$ and $C_{2}^{r}=C_{g}^{r}+\alpha^{r}$. We also
assume that $w_{1}^{r}=w_{g}^{r}+w^{r}$, $h_{g}^{r}=h_{1}^{g}+h^{r}$,
$w_{2}^{r}=w_{g}^{r}+w^{r}$ and $h_{g}^{r}=h_{2}^{g}+h^{r}$.

Now area overlap ($E_{A_{1}}^{r}$) between ground truth tracker $T_{g}^{r}$
and tracker $T_{1}^{r}$ generated by Alg$1$ at the $r^{th}$ frame, is
calculated as

$\displaystyle\begin{array}[]{l}E_{A_{1}}^{r}=\frac{{T_{g}^{r}\cap
T_{1}^{r}}}{{T_{g}^{r}\cup T_{1}^{r}}}\\\
=\frac{{h_{1}^{r}w_{g}^{r}}}{{h_{1}^{r}w_{g}^{r}+h_{1}^{r}w^{r}+h^{r}w_{g}^{r}}}.\end{array}$
(30)

Again area overlap ($E_{A_{2}}^{r}$) between the ground truth tracker
$T_{g}^{r}$ and tracker $T_{2}^{r}$ generated by Alg$2$ at the $r^{th}$ frame,
is calculated as

$\displaystyle\begin{array}[]{l}E_{A_{2}}^{t}=\frac{{T_{g}^{r}\cap
T_{2}^{r}}}{{T_{g}^{r}\cup T_{2}^{r}}}\\\
=\frac{{h_{2}^{r}w_{g}^{r}}}{{h_{2}^{r}w_{g}^{r}+h_{2}^{r}w^{r}+h^{r}w_{g}^{r}}}\\\
=\frac{{h_{1}^{r}w_{g}^{r}}}{{h_{1}^{r}w_{g}^{r}+h_{1}^{r}w^{r}+h^{r}w_{g}^{r}}},\end{array}$
(34)

since $h_{1}^{r}=h_{2}^{r}$ and $w_{1}^{r}=w_{2}^{r}$.

Now center location errors $E_{C_{1}}^{r}$ and $E_{C_{2}}^{r}$ of the trackers
$T_{1}^{r}$ and $T_{2}^{r}$ at the $r^{th}$ frame, respectively are calculated
as

$\displaystyle\begin{array}[]{l}E_{C_{1}}^{r}=\left\|{C_{g}^{r}-C_{1}^{r}}\right\|\\\
=0,\end{array}$ (37)

since $C_{g}^{r}=C_{1}^{r}$.

$\displaystyle\begin{array}[]{l}E_{C_{2}}^{r}=\left\|{C_{g}^{r}-C_{2}^{r}}\right\|\\\
=\left\|{C_{g}^{r}-(C_{g}^{r}+\alpha^{r})}\right\|\\\ =\alpha^{r},\end{array}$
(41)

since $C_{2}^{r}=C_{g}^{r}+\alpha^{r}$.

Eqs. (30) and (34) conclude that $E_{A_{2}}^{r}=E_{A_{1}}^{r}$, i.e., both
trackers $T_{2}^{r}$ and $T_{1}^{r}$ at $t^{th}$ frame are equally good. On
the other hand, Eqs. (37) and (41) conclude that
$E_{C_{1}}^{r}<E_{C_{2}}^{r}$, i.e., the tracker $T_{1}^{r}$ has lesser center
location error than the tracker $T_{2}^{r}$ at the $r^{th}$ frame. If we
consider center location error as a performance measure then the tracker
$T_{1}^{r}$ is better than the tracker $T_{2}^{r}$ at the $r^{th}$ frame. If
we consider area overlap as a performance measure then both the trackers
$T_{1}^{r}$ and $T_{2}^{r}$ are equally good at the $t^{th}$ frame. If we
consider both these measures no decision can be made. Therefore, minimum
center location error indicates maximum area overlap for the fixed sized
tracker is not true for the oriented object and performance measures
individually can make ambiguous decision for the oriented object.

###  4.2 Example2:

a

b c

Figure 4: Example 2 - area overlap between ground truth oriented tracker and
oriented trackers obtained by (a) $Alg_{1}$ (b) $Alg_{2}$.

Here, we consider another simple example (see Figure 4). Suppose $T_{g}^{t}$
(green colored rectangle) with center $C_{g}^{t}$, height $h_{g}^{t}$, width
$w_{g}^{t}$ and $\theta_{r}^{t}=45^{o}$ be the ground truth tracker at the
$t^{th}$ frame is displayed in Figure 4(a). After few number of frames, the
object changes its orientation and the corresponding ground truth tracker is
also oriented based on object orientation. Let $T_{g}^{r}$ (green colored
rectangle) with center $C_{g}^{r}$, height $h_{g}^{r}$, width $w_{g}^{r}$ and
$\theta^{r}=0^{o}$ be ground truth tracker at the $r^{th}$ frame. Again let
$T_{1}^{r}$ (red colored rectangle) with center $C_{1}^{r}$, height
$h_{1}^{r}$, width $w_{1}^{r}$ & orientation $\theta_{1}^{r}=135^{o}$ and
$T_{2}^{r}$ (pink colored rectangle) with center $C_{2}^{r}$, height
$h_{2}^{r}$, width $w_{2}^{r}$ & orientation $\theta_{2}^{r}=45^{o}$ be two
trackers generated by two different (oriented object) tracking algorithms
Alg$1$ and Alg$2$ and are displayed in Figures 4(b) and (c), respectively.

Here, we also assume that the trackers $T_{g}^{r}$ and $T_{1}^{r}$ have same
center location, width, height but different orientation (see Figure 4(b)).
Therefore, we have $C_{g}^{r}=C_{1}^{r}$, $w_{g}^{r}=w_{1}^{r}$,
$h_{g}^{r}=h_{1}^{r}$ and $\theta_{g}^{r}\neq\theta_{1}^{r}$. Now let
$\theta_{1}^{r}=\theta_{g}^{r}+135^{o}$. We also assume that the trackers
$T_{g}^{r}$ and $T_{2}^{r}$ have same center location, height, width but
different orientation (see Figure 4(c)). Therefore, we have
$C_{g}^{r}=C_{2}^{r}$, $h_{g}^{r}=h_{2}^{r}$, $w_{g}^{r}=w_{2}^{r}$ and
$\theta_{g}^{r}\neq\theta_{2}^{r}$. Now let
$\theta_{2}^{r}=\theta_{g}^{r}+45^{o}$.

This example concludes that two different oriented trackers have the same
values for area overlap and center location error. If we consider the existing
performance measures: area overlap and center location error, then both these
Alg$1$ and Alg$2$ are equally good. But for oriented object tracking, both
these algorithms Alg$1$ and Alg$2$ provides different interpretations.

##  5 Problems of Existing Evaluation Measure for Scaled and Oriented Object
Tracking

###  5.1 Example1:

a

b c

Figure 5: Example 1- area overlap between ground truth scaled & oriented
tracker and scaled & oriented trackers obtained by (a) Alg1 and (b) Alg2.

Here, we consider one example where object is changing its scale and
orientation together from one frame to another and the trackers $T_{1}$ and
$T_{2}$ generated by two different tracking algorithms Alg$1$ and Alg$2$,
respectively are not fixed (i.e., both are changing their scale and
orientation together based on object’s scale and orientation). We assume that
ground truth tracker ($T_{g}$) is not fixed i.e., it changes its scale and
orientation based on object’s scale and orientation. We assume that at
$t^{th}$ frame, the ground truth tracker (green colored rectangle) $T_{g}^{t}$
with center $C_{g}^{t}$, height $h_{g}^{t}$, width $w_{g}^{t}$ and orientation
$\theta{g}^{t}=0^{o}$ is presented as in Figure 5(a). In the $r^{th}$ ($r>t$)
frame, the object changes its scale and orientation jointly and the
corresponding ground truth tracker (green colored rectangle) $T_{g}^{r}$ with
center $C_{g}^{r}$, height ($h_{g}^{r}$), width ($w_{g}^{r}$) and
$\theta_{g}^{r}=90^{o}$ is shown in both Figures 5(b) and (c). Here, it is
also assume that $h_{g}^{r}>h_{g}^{t}$ and $w_{g}^{r}>w_{g}^{t}$. Therefore,
we have $h_{g}^{r}=h_{g}^{t}+h^{r}$, $w_{g}^{r}=w_{g}^{t}+w^{r}$ and
$\theta_{g}^{r}=\theta_{g}^{t}+90^{o}$.

We assume that two different tracking algorithms generates two trackers
$T_{1}^{r}$ (dotted red colored rectangle with center $C_{1}^{r}$, height
$h_{1}^{r}$, width $w_{1}^{r}$ & orientation $\theta_{1}^{r}=0^{o}$) and
$T_{2}^{r}$ (dotted pink colored rectangle with center $C_{2}^{r}$, height
$h_{2}^{r}$, width $w_{2}^{r}$ and orientation $\theta_{2}^{r}=90^{o}$) in
Figure 5(b) and (c), respectively. Here, we also assume that
$w_{1}^{r}=w_{g}^{r}$, $h_{1}^{r}=h_{g}^{r}-h^{r}$, $h_{2}^{r}=h_{g}^{r}$,
$w_{2}^{r}=w_{g}^{r}-w^{r}$, $C_{1}^{r}=C_{g}^{r}$,
$C_{2}^{r}=C_{g}^{r}+w^{r}$ and $h^{r}>w^{r}$.

Area overlap for both the trackers $T_{1}^{r}$ and $T_{2}^{r}$ with ground
truth tracker $T_{g}^{r}$ are calculated as

$\displaystyle\begin{array}[]{l}E_{A_{1}}^{r}=\frac{{T_{g}^{r}\cap
T_{1}^{r}}}{{T_{g}^{r}\cup T_{1}^{r}}}\\\
=\frac{{w_{1}^{r}h_{1}^{r}}}{{w_{g}^{r}h_{g}^{r}}}=\frac{{w_{g}^{r}\left({h_{g}^{r}-h^{r}}\right)}}{{w_{g}^{r}h_{g}^{r}}}=1-\frac{h^{r}}{{h_{g}^{r}}}\end{array}$
(44)

and

$\displaystyle\begin{array}[]{l}E_{A_{2}}^{r}=\frac{{T_{g}^{r}\cap
T_{2}^{r}}}{{T_{g}^{r}\cup T_{2}^{r}}}\\\
=\frac{{w_{2}^{r}h_{2}^{r}}}{{w_{g}^{r}h_{g}^{r}}}=\frac{{h_{g}^{r}\left({w_{g}^{r}-w^{r}}\right)}}{{w_{g}^{r}h_{g}^{r}}}=1-\frac{w^{r}}{{w_{g}^{r}}}.\end{array}$
(47)

Now center location errors $E_{{C}_{1}}^{r}$ and $E_{{C}_{2}}^{r}$ for both
the trackers $T_{1}^{r}$ and $T_{2}^{r}$, respectively are calculated as

$\displaystyle\begin{array}[]{l}E_{{C}_{1}}^{r}=\|C_{1}^{r}-C_{g}^{r}\|=0\end{array}$
(49)

and

$\displaystyle\begin{array}[]{l}E_{{C}_{2}}^{r}=\|C_{2}^{r}-C_{g}^{r}\|=\|C_{g}^{r}+w^{r}-C_{g}^{r}\|=w^{r}.\end{array}$
(51)

Now if $\frac{h^{r}}{h_{g}^{r}}=\frac{w^{r}}{w_{g}^{r}}$, then
$E_{A_{1}}^{r}=E_{A_{2}}^{r}$. This example highlights that two different
trackers have different center location error but same amount of area overlap
if $\frac{h^{r}}{h_{g}^{r}}=\frac{w^{r}}{w_{g}^{r}}$. Therefore, in this
example also, the existing performance measures fail to evaluate quality of
scaled and oriented object tracking algorithms. In the next section, we have
developed three new auxiliary measures to quantify tracking algorithms in such
complex environments.

##  6 Proposed Performance Measures

Three new auxiliary evaluation measures namely error in estimated height of
optimal tracker ($E_{h}^{t}$), error in estimated width of optimal
tracker($E_{w}^{t}$) and error in estimated orientation angle of optimal
tracker ($E_{\theta}^{t}$), respectively have been proposed to validate the
tracking algorithms. For the $t^{th}$ frame, these are calculated as

$\displaystyle\begin{array}[]{l}E_{h}^{t}=\|h_{g}^{t}-h^{t}\|,\\\
E_{w}^{t}=\|w_{g}^{t}-w^{t}\|,\\\
E_{\theta}^{t}=\|\theta_{g}^{t}-\theta^{t}\|,\end{array}$ (55)

where $h_{g}^{t}$, $w_{g}^{t}$ and $\theta_{g}^{t}$ are the actual height,
width and orientation angle, respectively, and $h^{t}$, $w^{t}$ and
$\theta^{t}$ denote the estimated height, width and orientation angle for the
$t^{th}$ frame, respectively. $\|.\|$ is the Euclidean norm. For a good
tracking system, these three measures should be equal to zero.

Now, we define average error in estimated height of optimal tracker
($E_{h}^{avg}$), average error in estimated width of optimal tracker
($E_{w}^{avg}$) and average error in estimated orientation angle of optimal
tracker ($E_{\theta}^{avg}$), to measure an overall performances of tracking
algorithms. These are calculated as

$\displaystyle\begin{array}[]{l}E_{h}^{avg}=\frac{1}{N_{frames}}\sum_{t=1}^{N_{frames}}E_{h}^{t},\\\
E_{w}^{avg}=\frac{1}{N_{frames}}\sum_{t=1}^{N_{frames}}E_{w}^{t},\\\
E_{\theta}^{avg}=\frac{1}{N_{frames}}\sum_{t=1}^{N_{frames}}E_{\theta}^{t},\end{array}$
(59)

where $N_{frames}$ is the total number of frames of a video sequence. For a
good tracking system, $E_{h}^{avg}$, $E_{w}^{avg}$ and $E_{\theta}^{avg}$
should be equal to zero.

Another composite measure called as matching score ($E_{ms}^{t}$) at the
$t^{th}$ frame between ground truth tracker ($T_{g}^{t}$) and tracker
($T^{t}$) generated by any tracking algorithm is proposed to evaluate
performance of tracking algorithm under such complex environment. Three new
proposed measures and two existing measures are combined to calculate matching
score as

$\displaystyle\begin{array}[]{l}E_{ms}^{t}=\frac{1}{5}\left[f_{c}^{t}+f_{h}^{t}+f_{w}^{t}+f_{\theta}^{t}+f_{A}^{t}\right],\end{array}$
(61)

where

$\displaystyle\begin{array}[]{l}f_{c}^{t}=\frac{1}{1+E_{c}^{t}}\end{array}$
(63)

$\displaystyle\begin{array}[]{l}f_{h}^{t}=\frac{1}{1+E_{h}^{t}}\end{array}$
(65)

$\displaystyle\begin{array}[]{l}f_{w}^{t}=\frac{1}{1+E_{w}^{t}}\end{array}$
(67)

$\displaystyle\begin{array}[]{l}f_{\theta}^{t}=\frac{1}{1+E_{\theta}^{t}}\end{array}$
(69)

$\displaystyle\begin{array}[]{l}f_{A}^{t}=E_{A}^{t}\end{array}$ (71)

with $E_{C}^{t}$, $E_{h}^{t}$, $E_{w}^{t}$, $E_{\theta}^{t}$ and $E_{A}^{t}$
are center location, height, width, orientation errors and area overlap at
$t^{th}$ frame, respectively.

Now we define average matching score ($E_{ms}^{avg}$) to measure overall
performance of tracking algorithm, is calculated as

$\displaystyle\begin{array}[]{l}E_{ms}^{avg}=\frac{1}{N_{frames}}\sum_{t=1}^{N_{frames}}E_{ms}^{t},\end{array}$
(73)

where $N_{frames}$ is the total number of frames of a video sequence. For a
good tracking system, $E_{ms}^{avg}$ should be equal to $1$.

Sequence | Area Under The Curve (AUC): Success Plot$\uparrow$
---|---
| ANT | ASMS | KCF | LADCF | MEEM | RCO | SA-Siam-P | SA-Siam-R | SiamRPN | UPDT
Bag | 0.666 | 0.473 | 0.565 | 0.693 | 0.326 | 0.679 | 0.682 | 0.679 | 0.729 | 0.674
Basketball | 0.591 | 0.629 | 0.448 | 0.686 | 0.623 | 0.682 | 0.584 | 0.633 | 0.711 | 0.659
Blanket | 0.605 | 0.682 | 0.637 | 0.604 | 0.549 | 0.693 | 0.604 | 0.535 | 0.484 | 0.558
Bmx | 0.203 | 0.376 | 0.489 | 0.308 | 0.352 | 0.526 | 0.422 | 0.451 | 0.333 | 0.406
Book | 0.304 | 0.426 | 0.339 | 0.541 | 0.419 | 0.602 | 0.457 | 0.553 | 0.574 | 0.509
Butterfly | 0.303 | 0.601 | 0.428 | 0.498 | 0.356 | 0.568 | 0.473 | 0.534 | 0.565 | 0.526
Crabs1 | 0.414 | 0.411 | 0.298 | 0.377 | 0.386 | 0.440 | 0.369 | 0.442 | 0.552 | 0.485
Dinosaur | 0.619 | 0.399 | 0.352 | 0.438 | 0.383 | 0.452 | 0.634 | 0.537 | 0.588 | 0.608
Fernando | 0.428 | 0.424 | 0.388 | 0.445 | 0.454 | 0.457 | 0.482 | 0.442 | 0.573 | 0.449
Fish2 | 0.399 | 0.483 | 0.327 | 0.396 | 0.343 | 0.435 | 0.475 | 0.568 | 0.523 | 0.428
Girl | 0.476 | 0.689 | 0.506 | 0.589 | 0.468 | 0.513 | 0.598 | 0.621 | 0.626 | 0.396
gymnastics2 | 0.377 | 0.452 | 0.529 | 0.436 | 0.426 | 0.512 | 0.450 | 0.472 | 0.565 | 0.471
hand | 0.456 | 0.471 | 0.489 | 0.406 | 0.404 | 0.490 | 0.527 | 0.481 | 0.583 | 0.485
helicopter | 0.631 | 0.611 | 0.552 | 0.440 | 0.420 | 0.538 | 0.665 | 0.515 | 0.691 | 0.573
Iceskater2 | 0.468 | 0.572 | 0.419 | 0.543 | 0.538 | 0.569 | 0.511 | 0.576 | 0.630 | 0.597
Matrix | 0.259 | 0.463 | 0.457 | 0.538 | 0.490 | 0.674 | 0.453 | 0.575 | 0.521 | 0.415
Motocross2 | 0.345 | 0.593 | 0.300 | 0.488 | 0.209 | 0.603 | 0.603 | 0.489 | 0.390 | 0.525
Shaking | 0.238 | 0.509 | 0.707 | 0.688 | 0.553 | 0.716 | 0.727 | 0.677 | 0.782 | 0.707
Soccer1 | 0.367 | 0.443 | 0.648 | 0.627 | 0.518 | 0.694 | 0.431 | 0.533 | 0.618 | 0.676
Traffic | 0.630 | 0.458 | 0.766 | 0.658 | 0.727 | 0.739 | 0.713 | 0.713 | 0.767 | 0.733
Average over |  |  |  |  |  |  |  |  |  |
60 sequences | 0.439 | 0.508 | 0.482 | 0.520 | 0.447 | 0.579 | 0.543 | 0.551 | 0.590 | 0.544
Table 1: Shows auc of the success plots of the existing measures. The bold value indicates the best tracker.  Sequence | Area Under The Curve (AUC): Success Plot of Matching Score $\uparrow$
---|---
| ANT | ASMS | KCF | LADCF | MEEM | RCO | SA-Siam-P | SA-Siam-R | SiamRPN | UPDT
Bag | 0.273 | 0.186 | 0.204 | 0.297 | 0.142 | 0.295 | 0.294 | 0.277 | 0.376 | 0.295
Basketball | 0.269 | 0.315 | 0.178 | 0.329 | 0.308 | 0.329 | 0.261 | 0.307 | 0.374 | 0.327
Blanket | 0.358 | 0.406 | 0.328 | 0.337 | 0.273 | 0.374 | 0.309 | 0.277 | 0.244 | 0.318
Bmx | 0.071 | 0.138 | 0.176 | 0.100 | 0.125 | 0.173 | 0.154 | 0.151 | 0.110 | 0.144
Book | 0.144 | 0.234 | 0.184 | 0.296 | 0.220 | 0.318 | 0.246 | 0.326 | 0.322 | 0.287
Butterfly | 0.124 | 0.241 | 0.172 | 0.193 | 0.148 | 0.220 | 0.188 | 0.209 | 0.241 | 0.209
Crabs1 | 0.232 | 0.198 | 0.169 | 0.178 | 0.190 | 0.192 | 0.159 | 0.209 | 0.263 | 0.209
Dinosaur | 0.254 | 0.173 | 0.182 | 0.190 | 0.183 | 0.217 | 0.292 | 0.239 | 0.261 | 0.275
Fernando | 0.177 | 0.156 | 0.149 | 0.166 | 0.193 | 0.184 | 0.171 | 0.169 | 0.219 | 0.162
Fish2 | 0.206 | 0.243 | 0.195 | 0.236 | 0.236 | 0.239 | 0.233 | 0.299 | 0.299 | 0.210
Girl | 0.211 | 0.323 | 0.216 | 0.250 | 0.210 | 0.214 | 0.247 | 0.258 | 0.284 | 0.195
gymnastics2 | 0.160 | 0.236 | 0.302 | 0.222 | 0.231 | 0.286 | 0.244 | 0.258 | 0.259 | 0.282
hand | 0.251 | 0.275 | 0.276 | 0.199 | 0.212 | 0.250 | 0.288 | 0.259 | 0.338 | 0.279
helicopter | 0.328 | 0.302 | 0.254 | 0.222 | 0.197 | 0.277 | 0.389 | 0.235 | 0.369 | 0.319
Iceskater2 | 0.166 | 0.221 | 0.176 | 0.217 | 0.236 | 0.247 | 0.211 | 0.242 | 0.274 | 0.260
Matrix | 0.161 | 0.251 | 0.254 | 0.285 | 0.251 | 0.353 | 0.224 | 0.295 | 0.268 | 0.209
Motocross2 | 0.103 | 0.188 | 0.101 | 0.146 | 0.091 | 0.189 | 0.203 | 0.157 | 0.116 | 0.163
Shaking | 0.096 | 0.301 | 0.374 | 0.357 | 0.220 | 0.321 | 0.364 | 0.293 | 0.417 | 0.360
Soccer1 | 0.186 | 0.229 | 0.349 | 0.308 | 0.236 | 0.369 | 0.209 | 0.265 | 0.314 | 0.340
Traffic | 0.392 | 0.296 | 0.508 | 0.389 | 0.434 | 0.471 | 0.469 | 0.467 | 0.519 | 0.504
Average over |  |  |  |  |  |  |  |  |  |
60 sequences | 0.208 | 0.246 | 0.237 | 0.246 | 0.217 | 0.276 | 0.258 | 0.259 | 0.294 | 0.268
Table 2: Shows auc of the success plots of new measure - matching score. The
bold value indicates the best tracker.

To measure the performance of an algorithm on a sequence of frames, we count
the number of successful frames whose matching score ($E_{ms}^{t}$) is larger
than a given threshold value. The matching score plot shows the ratios of
successful frames at the thresholds varied from $0$ to $1$. The matching rate
value at a specific threshold (e.g., $0.5$) may not always be a good
representative for evaluating the trackers’ performance. Instead of that, the
area under the curve (auc) of each matching score plot is considered to
evaluate the performance of the tracking algorithms.

##  7 Experiments

###  7.1 Dataset

We use entire vot-2018111https://www.votchallenge.net/vot2018/dataset.html
kristan2018sixth  dataset for our experiments. vot-2018 dataset contains 60
video sequences. The target in the sequences is annotated by a rotated
bounding box and all sequences are per-frame annotated by the following visual
attributes: (i) occlusion, (ii) illumination change, (iii) motion change, (iv)
size change, and (v) camera motion. Frames that do not correspond to any of
the five attributes are denoted as (vi) unassigned. All these complex factors
make object tracking difficult on this dataset. If a tracker fails to track
object in a frame, we reset the tracker to 0 (as oppose to vot protocol). We
only consider one-pass evaluation (ope) to draw the plots to compare
performances of all the trackers with respect to both existing and the
proposed measures.

###  7.2 Baselines

We use various state-of-the-art trackers on vot-2018 dataset such as ant1 [9],
kcf222https://github.com/vojirt/kcf henriques2014high ,
asms333https://github.com/vojirt/asms vojir2014robust , updt bhat2018unveiling
, ladcf444https://github.com/XU-TIANYANG/LADCF.git xu2019learning ,
meem555http://www.cs.bu.edu/groups/ivc/software/MEEM/ zhang2014meem , rco1
[9], sa-siam-r1 [9], sa-siam-p1 [9], and siamrpn li2018high  for our
experiments. Among them, sa-siam-r, sa-siam-p, and siamrpn apply siamese
networks, ladcf, rco, kcf, and updt are developed based on discriminative
correlation filters, asms applies mean shift, ant uses optical flow, and meem
is based on support vector machine.

###  7.3 Quantitative Results Analysis

Sequence | Precision Value at Threshold 20 Pixels $\uparrow$
---|---
| ANT | ASMS | KCF | LADCF | MEEM | RCO | SA-Siam-P | SA-Siam-R | SiamRPN | UPDT
Bag | 0.607 | 0.474 | 0.526 | 0.995 | 0.036 | 0.959 | 0.995 | 0.995 | 0.995 | 0.995
Basketball | 0.934 | 0.941 | 0.538 | 0.989 | 0.804 | 0.992 | 0.895 | 0.989 | 0.934 | 0.990
Blanket | 0.933 | 0.978 | 0.987 | 0.996 | 0.987 | 0.987 | 0.991 | 0.987 | 0.973 | 0.889
Bmx | 0.263 | 0.237 | 0.316 | 0.237 | 0.263 | 0.263 | 0.145 | 0.211 | 0.197 | 0.158
Book | 0.360 | 0.600 | 0.383 | 0.909 | 0.611 | 0.909 | 0.646 | 0.783 | 0.817 | 0.806
Butterfly | 0.291 | 0.907 | 0.318 | 0.523 | 0.325 | 0.649 | 0.358 | 0.589 | 0.709 | 0.669
Crabs1 | 0.381 | 0.713 | 0.250 | 0.363 | 0.350 | 0.456 | 0.588 | 0.594 | 0.719 | 0.806
Dinosaur | 0.715 | 0.248 | 0.267 | 0.684 | 0.340 | 0.494 | 0.809 | 0.638 | 0.813 | 0.985
Fernando | 0.507 | 0.223 | 0.562 | 0.627 | 0.647 | 0.678 | 0.619 | 0.363 | 0.634 | 0.606
Fish2 | 0.571 | 0.845 | 0.258 | 0.761 | 0.371 | 0.803 | 0.800 | 0.874 | 0.819 | 0.855
Girl | 0.916 | 0.921 | 0.313 | 0.729 | 0.849 | 0.378 | 0.281 | 0.335 | 0.595 | 0.176
gymnastics2 | 0.329 | 0.683 | 0.736 | 0.688 | 0.729 | 0.738 | 0.692 | 0.696 | 0.671 | 0.725
hand | 0.978 | 0.719 | 0.697 | 0.873 | 0.655 | 0.742 | 0.828 | 0.757 | 0.865 | 0.933
helicopter | 0.637 | 0.863 | 0.595 | 0.472 | 0.301 | 0.623 | 0.643 | 0.836 | 0.879 | 0.675
Iceskater2 | 0.402 | 0.454 | 0.266 | 0.521 | 0.646 | 0.897 | 0.636 | 0.761 | 0.714 | 0.829
Matrix | 0.480 | 0.770 | 0.650 | 0.850 | 0.850 | 0.990 | 0.730 | 0.830 | 0.760 | 0.910
Motocross2 | 0.098 | 0.148 | 0.00 | 0.066 | 0.033 | 0.098 | 0.098 | 0.098 | 0.213 | 0.197
Shaking | 0.660 | 0.488 | 0.858 | 0.939 | 0.923 | 0.997 | 0.995 | 0.997 | 0.997 | 0.995
Soccer1 | 0.513 | 0.515 | 0.865 | 0.918 | 0.837 | 0.954 | 0.546 | 0.819 | 0.859 | 0.959
Traffic | 0.953 | 0.613 | 0.993 | 0.995 | 0.995 | 0.995 | 0.995 | 0.995 | 0.995 | 0.995
Average over |  |  |  |  |  |  |  |  |  |
60 sequences | 0.576 | 0.617 | 0.519 | 0.707 | 0.578 | 0.730 | 0.664 | 0.707 | 0.758 | 0.758
Table 3: Shows precision value at threshold (20 pixels). The bold value indicates the best tracker.  Sequence | Success Rate at Threshold 0.5 $\uparrow$
---|---
| ANT | ASMS | KCF | LADCF | MEEM | RCO | SA-Siam-P | SA-Siam-R | SiamRPN | UPDT
Bag | 0.888 | 0.444 | 0.796 | 0.949 | 0.077 | 0.888 | 0.868 | 0.893 | 0.995 | 0.949
Basketball | 0.805 | 0.849 | 0.366 | 0.934 | 0.906 | 0.966 | 0.775 | 0.847 | 0.942 | 0.899
Blanket | 0.644 | 0.867 | 0.920 | 0.787 | 0.760 | 0.942 | 0.827 | 0.667 | 0.418 | 0.778
Bmx | 0.118 | 0.224 | 0.487 | 0.197 | 0.263 | 0.579 | 0.355 | 0.316 | 0.197 | 0.342
Book | 0.251 | 0.400 | 0.320 | 0.697 | 0.463 | 0.789 | 0.568 | 0.703 | 0.731 | 0.669
Butterfly | 0.159 | 0.795 | 0.258 | 0.490 | 0.166 | 0.755 | 0.424 | 0.702 | 0.609 | 0.616
Crabs1 | 0.325 | 0.313 | 0.150 | 0.200 | 0.286 | 0.363 | 0.150 | 0.344 | 0.706 | 0.544
Dinosaur | 0.899 | 0.328 | 0.288 | 0.420 | 0.316 | 0.515 | 0.776 | 0.571 | 0.782 | 0.822
Fernando | 0.318 | 0.277 | 0.329 | 0.449 | 0.445 | 0.455 | 0.466 | 0.390 | 0.688 | 0.404
Fish2 | 0.419 | 0.555 | 0.216 | 0.239 | 0.323 | 0.281 | 0.481 | 0.665 | 0.623 | 0.365
Girl | 0.432 | 0.895 | 0.510 | 0.844 | 0.382 | 0.560 | 0.867 | 0.911 | 0.811 | 0.153
gymnastics2 | 0.336 | 0.438 | 0.596 | 0.421 | 0.404 | 0.471 | 0.429 | 0.429 | 0.629 | 0.433
hand | 0.367 | 0.494 | 0.543 | 0.255 | 0.337 | 0.566 | 0.581 | 0.487 | 0.764 | 0.487
helicopter | 0.794 | 0.794 | 0.524 | 0.329 | 0.463 | 0.523 | 0.773 | 0.486 | 0.870 | 0.605
Iceskater2 | 0.414 | 0.743 | 0.393 | 0.648 | 0.648 | 0.685 | 0.569 | 0.704 | 0.785 | 0.765
Matrix | 0.150 | 0.520 | 0.580 | 0.620 | 0.560 | 0.880 | 0.500 | 0.740 | 0.580 | 0.240
Motocross2 | 0.098 | 0.836 | 0.246 | 0.459 | 0.0819 | 0.853 | 0.869 | 0.574 | 0.164 | 0.623
Shaking | 0.022 | 0.551 | 0.858 | 0.997 | 0.699 | 0.997 | 0.984 | 0.981 | 0.975 | 0.981
Soccer1 | 0.253 | 0.457 | 0.801 | 0.816 | 0.625 | 0.883 | 0.431 | 0.561 | 0.793 | 0.875
Traffic | 0.895 | 0.576 | 0.974 | 0.921 | 0.989 | 0.974 | 0.958 | 0.927 | 0.969 | 0.974
Average over |  |  |  |  |  |  |  |  |  |
60 sequences | 0.429 | 0.568 | 0.508 | 0.584 | 0.459 | 0.696 | 0.632 | 0.645 | 0.702 | 0.626
Table 4: Shows success rate at threshold (0.5) in the success plot of the existing measure. The bold value indicates the best tracker.  Sequence | New Success Rate at Threshold 0.5 $\uparrow$
---|---
| ANT | ASMS | KCF | LADCF | MEEM | RCO | SA-Siam-P | SA-Siam-R | SiamRPN | UPDT
Bag | 0.066 | 0.015 | 0.005 | 0.077 | 0.005 | 0.102 | 0.097 | 0.03 | 0.229 | 0.066
Basketball | 0.052 | 0.095 | 0.023 | 0.123 | 0.066 | 0.126 | 0.047 | 0.081 | 0.215 | 0.105
Blanket | 0.280 | 0.284 | 0.080 | 0.142 | 0.031 | 0.156 | 0.093 | 0.084 | 0.084 | 0.129
Bmx | 0.000 | 0.026 | 0.013 | 0.000 | 0.013 | 0.000 | 0.039 | 0.026 | 0.000 | 0.000
Book | 0.017 | 0.057 | 0.034 | 0.128 | 0.069 | 0.137 | 0.103 | 0.240 | 0.189 | 0.120
Butterfly | 0.007 | 0.013 | 0.007 | 0.007 | 0.000 | 0.019 | 0.000 | 0.007 | 0.053 | 0.007
Crabs1 | 0.069 | 0.019 | 0.031 | 0.013 | 0.025 | 0.000 | 0.006 | 0.075 | 0.081 | 0.025
Dinosaur | 0.043 | 0.034 | 0.065 | 0.015 | 0.046 | 0.028 | 0.095 | 0.064 | 0.046 | 0.039
Fernando | 0.051 | 0.000 | 0.003 | 0.000 | 0.014 | 0.000 | 0.000 | 0.010 | 0.024 | 0.000
Fish2 | 0.081 | 0.048 | 0.100 | 0.100 | 0.145 | 0.061 | 0.065 | 0.139 | 0.171 | 0.029
Girl | 0.028 | 0.125 | 0.035 | 0.027 | 0.053 | 0.013 | 0.033 | 0.039 | 0.097 | 0.021
gymnastics2 | 0.000 | 0.113 | 0.213 | 0.104 | 0.121 | 0.158 | 0.138 | 0.150 | 0.100 | 0.171
hand | 0.059 | 0.105 | 0.154 | 0.034 | 0.059 | 0.086 | 0.112 | 0.112 | 0.184 | 0.075
helicopter | 0.189 | 0.138 | 0.095 | 0.041 | 0.102 | 0.079 | 0.312 | 0.105 | 0.259 | 0.162
Iceskater2 | 0.002 | 0.013 | 0.007 | 0.013 | 0.044 | 0.044 | 0.035 | 0.029 | 0.076 | 0.037
Matrix | 0.020 | 0.120 | 0.150 | 0.160 | 0.110 | 0.190 | 0.020 | 0.160 | 0.120 | 0.040
Motocross2 | 0.000 | 0.000 | 0.016 | 0.000 | 0.016 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
Shaking | 0.000 | 0.148 | 0.230 | 0.167 | 0.025 | 0.093 | 0.184 | 0.060 | 0.326 | 0.205
Soccer1 | 0.038 | 0.089 | 0.176 | 0.117 | 0.051 | 0.229 | 0.048 | 0.079 | 0.133 | 0.153
Traffic | 0.230 | 0.126 | 0.439 | 0.209 | 0.304 | 0.408 | 0.387 | 0.408 | 0.492 | 0.419
Average over |  |  |  |  |  |  |  |  |  |
60 sequences | 0.062 | 0.078 | 0.094 | 0.074 | 0.065 | 0.097 | 0.091 | 0.095 | 0.144 | 0.090
Table 5: Shows the success rate at threshold (0.5) in the success plots of the
proposed measure — matching score. The bold value indicates the best tracker.

Tables 1 and 2 show auc of success plots of the existing measure and the
proposed measure, respectively. The bold value indicates the best tracker
among the considered state-of-the-art trackers. Tables 3, 4, and 5 present
precision value at threshold 20 pixels, success rate at threshold 0.5 , and
new success rate at threshold 0.5. Bold values in the tables indicate the best
trackers. According to auc of success plots of existing measures, SiamRPM
performs the best on 10 sequences among randomly selected 20 sequences. In
case of average of 60 sequences, SiamRPM also performs the best. SiamRPM
obtains the best results for 11 sequences with respect to auc of success plots
of the proposed measure — matching score. SiamRPM also performs the best with
respect to auc in the average success plot of 60 sequences.

###  7.4 Qualitative Results Analysis

Bag sequence

Basketball sequence

Blanket sequence

Bmx sequence

Book sequence

Butterfly sequence

Figure 6: Illustrates the tracking results of ten state-of-the-art trackers on
Bag, Basketball, Blanket, Bmx, Book, and Butterfly sequences. Each colored
rectangle corresponds to each tracker’s output.

Crabs1 sequence

Dinosaur sequence

Fernando sequence

Fish2 sequence

Girl sequence

Gymnastics2 sequence

Figure 7: Illustrates the tracking results of ten state-of-the-art trackers on
Crabs1, Dinosaur, Fernando, Fish2, Girl, and Gymnastics2 sequences. Each
colored rectangle corresponds to each tracker’s output.

Hand sequence

Helicopter sequence

Iceskater2 sequence

Matrix sequence

Motocross2 sequence

Shaking sequence

Figure 8: Illustrates the tracking results of ten state-of-the-art trackers on
Hand, Helicopter, Iceskater2, Matrix, Motocross2, and Shaking sequences. Each
colored rectangle corresponds to each tracker’s output.

Soccer1 sequence

Traffic sequence

Figure 9: Illustrates the tracking results of ten state-of-the-art trackers on
Soccer1, and Traffic sequences. Each colored rectangle corresponds to each
tracker’s output.

Figures 6, 7, 8, and 9 show the tracking results (selected few frames) of
vot-2018 dataset obtained using the considered baselines — ant, asms, kcf,
ladcf, meem, rco, sa-siam-p, sa-siam-r, siamrpn, and updt. Object in Bag, Bmx,
Book, Butterfly, Dinosaur, Fernando, Fish2, Gymnastics2, Hand, Iceskater2, and
Motocross2 changes scale and orientation. Basketball, Fish2, Girl, and Soccer1
sequences contain occlusions. Matrix and Shaking sequences under go
illumination variation. Object changes its scale in Blanket, Crabs1, and
Helicopter sequences. For scaled and oriented object in the sequences such as
Butterfly, Dinosaur, Fernando, Fish2, Gymnastics2, Hand, Iceskater2, and
Motocross2, the trackers — sa-siam-p, sa-siam-r, siamrpn, and updt performs
better than other trackers. siamrpn, ant, rco, and sa-siam-r obtain robust
tracking results on Basketball, Girl, Soccer1, and Fish2 sequences,
respectively.

###  7.5 Analysis of Precision and Success Plots

The success plot shows the ratio of successful frames at threshold varied from
0 to 1. Sometimes, the success plots and the precision plots make a different
conclusion because they measure the tracker’s other characteristics based on
various metrics. However, auc score of the success plot interprets the overall
performance, which is more accurate than the center location error at the
precision threshold value plot benchmark_wu2013 . We analyze the tracker
performance using auc score of success plot, precision plot, and new success
plot. The tracker has a maximum auc score of the success plot indicates the
best tracker. Since vot-2018 dataset contains 60 video sequences, it is
challenging to analyze the state-of-the-art trackers’ performances on every
sequence. For this, we randomly select few (20) video sequences to analyze the
existing trackers’ performances based on auc of success, precision, and new
success plots. It is also challenging to analyze the trackers’ average
performances over vot-2018 dataset, using three plots to correspond to each
video sequence of this dataset. For this reason, we use average success,
precision, and new success plots over all sequences of this dataset. We
analyze the performances of the trackers based on average auc corresponds to
these three plots.

Precision Plot Success Plot New Success Plot

Bag sequence

Basketball sequence

Blanket sequence

Bmx sequence

Book sequence

Butterfly sequence

Figure 10: Illustrates the precision and success plots popularly used for
evaluating a tracking algorithm and the success plot based on the proposed new
measure - matching score of ten different methods on Bag, Basketball, Blanket,
Bmx, Book, and Butterfly sequences. Each plot corresponds to one-pass
evaluation (ope).

Precision Plot Success Plot New Success Plot

Crabs1 sequence

Dinosaur sequence

Fernando sequence

Fish2 sequence

Girl sequence

Gymnastics2 sequence

Figure 11: Illustrates the precision and success plots popularly used for
evaluating a tracking algorithm and the success plot based on the proposed new
measure - matching score of ten different methods on Crabs1, Dinosaur,
Fernando, Fish2, Girl, and Gymnastics2 sequences. Each plot corresponds to
one-pass evaluation (ope).

Precision Plot Success Plot New Success Plot

Hand sequence

Helicopter sequence

Iceskater2 sequence

Matrix sequence

Motocross2 sequence

Shaking sequence

Figure 12: Illustrates the precision and success plots popularly used for
evaluating a tracking algorithm and the success plot based on the proposed new
measure - matching score of ten different methods on Hand, Helicopter,
Iceskater2, Matrix, Motocross2, and Shaking sequences. Each plot corresponds
to one-pass evaluation (ope).

Precision Plot Success Plot New Success Plot

Soccer1 sequence

Traffic sequence

Traffic sequence

Figure 13: Illustrates the precision and success plots popularly used for
evaluating a tracking algorithm and the success plot based on the proposed new
measure - matching score of ten different methods on Soccer1 and Traffic
sequences. Each plot corresponds to one-pass evaluation (ope).

Figures 10, 11, 12, and 13 present precision, success, and new success plots
correspond to randomly selected video sequences — Bag, Basketball, Blanket,
Bmx, Book, Butterfly, Crabs1, Dinosaur, Fish2, Fernando, Girl, Gymnastics2,
Hand, Helicopter, Iceskater2, Matrix, Motocross2, Shaking, Soccer1, and
Traffic of vot-2018 dataset. For these plots, we use auc score to summarize
the performances of the trackers. Tables 1 and 2 highlight auc of the success
plots of the existing measure and the proposed measure — matching score. These
tables also highlight the auc of average success plots over 60 sequences of
vot-2018 dataset. Tables 3, 4, and 5 highlight the precision at threshold 20
pixels, success rate at threshold value 0.5. The precision and success plots
make different conclusions for most of the sequences of vot-2018 dataset. We
make unique conclusion using new success plots of the proposed measure -
matching score.

##  8 Conclusions

In this paper, we propose a new measure - matching score to better quantify a
tracker for tracking scaled and oriented objects. The proposed measure -
matching score is a combination of two existing measures and three new
measures. For some sequences containing scaled and oriented objects, two
existing measures make two different remarks. It is very difficult to make a
conclusion from these remarks. On the other hand, the proposed measure makes a
concrete remark which highlights the best tracker. The sufficient examples and
results conclude that the proposed measure is more accurate than the existing
one to quantify a tracker for tracking scaled and oriented objects.

##  9 Compliance with Ethical Standards

###  9.1 Conflict of Interest

All authors declare that they have no conflicts of interest.

###  9.2 Ethical Approval

This article does no contain any studies with human participants or animals
performed by any of the authors.

## References

  * (1) Babenko, B., Yang, M., Belongie, S.: Robust object tracking with online multiple instance learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence 33(8), 1619–1632 (2011)
  * (2) Baumann, A., Boltz, M., Ebling, J., Koenig, M., Loos, H.S., Merkel, M., Niem, W., Warzelhan, J.K., Yu, J.: A review and comparison of measures for automatic video surveillance systems.  EURASIP Journal on Image and Video Processing 2008(824726), 1–30 (2008)
  * (3) Bhat, G., Johnander, J., Danelljan, M., Shahbaz Khan, F., Felsberg, M.: Unveiling the power of deep tracking.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 483–498 (2018)
  * (4) Čehovin, L., Leonardis, A., Kristan, M.: Visual object tracking performance measures revisited.  IEEE Transactions on Image Processing 25(3), 1261–1274 (2016)
  * (5) Fiaz, M., Mahmood, A., Javed, S., Jung, S.K.: Handcrafted and deep trackers: Recent visual object tracking approaches and trends.  ACM Computing Surveys (CSUR) 52(2), 43 (2019)
  * (6) Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters.  IEEE transactions on pattern analysis and machine intelligence 37(3), 583–596 (2014)
  * (7) Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters.  IEEE Transactions on Pattern Analysis and Machine Intelligence 37(3), 583–596 (2015)
  * (8) Kasturi, R., Goldgof, D., Soundararajan, P., Manohar, V., Garofolo, J., Bowers, R., Boonstra, M., Korzhova, V., Zhang, J.: Framework for performance evaluation of face, text, and vehicle detection and tracking in video: Data, metrics and protocol.  IEEE Transactions on Pattern Analysis and Machine Intelligence 31(2), 319–336 (2009)
  * (9) Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Cehovin Zajc, L., Vojir, T., Bhat, G., Lukezic, A., Eldesokey, A., et al.: The sixth visual object tracking vot2018 challenge results.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 0–0. Springer (2018)
  * (10) Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High performance visual tracking with siamese region proposal network.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8971–8980 (2018)
  * (11) Li, P., Wang, D., Wang, L., Lu, H.: Deep visual tracking: Review and experimental comparison.  Pattern Recognition 76, 323–338 (2018)
  * (12) Li, X., Hu, W., Shen, C., Zhang, Z., Dick, A., Hengel, A.V.D.: A survey of appearance models in visual object tracking.  ACM Transactions on Intelligent Systems and Technology 4(4), 1–58 (2013)
  * (13) Smeulders, A.W., Chu, D.M., Cucchiara, R., Calderara, S., Dehghan, A., Shah, M.: Visual tracking: An experimental survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence 36(7), 1442–1468 (2014)
  * (14) Vojir, T., Noskova, J., Matas, J.: Robust scale-adaptive mean-shift for tracking.  Pattern Recognition Letters 49, 250–258 (2014)
  * (15) Wu, Y., Lim, J., Yang, M.H.: Online object tracking: A benchmark.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2411–2418. IEEE (2013)
  * (16) Xu, T., Feng, Z.H., Wu, X.J., Kittler, J.: Learning adaptive discriminative correlation filters via temporal consistency preserving spatial feature selection for robust visual object tracking.  IEEE Transactions on Image Processing 28(11), 5596–5609 (2019)
  * (17) Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey.  ACM Computing Surveys 38(4), 1264–1291 (2006)
  * (18) Yin, F., Makris, D., Velastin, S.A.: Performance evaluation of object tracking algorithms.  In: IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, Rio De Janeiro, Brazil, pp. 1–25. Citeseer (2007)
  * (19) Zhang, J., Ma, S., Sclaroff, S.: MEEM: robust tracking via multiple experts using entropy minimization.  In: European conference on computer vision, pp. 188–203. Springer (2014)