# Large-Scale Person Detection and Localization using Overhead Fisheye Cameras

Lu Yang1111The first two authors contribute equally to this work. , Liulei
Li2111The first two authors contribute equally to this work. , Xueshi Xin1 ,
Yifan Sun3 , Qing Song1 , Wenguan Wang4222Corresponding author: Wenguan Wang.
1 Beijing University of Posts and Telecommunications 2 ReLER, AAII, University
of Technology Sydney
3 Baidu 4 ReLER, CCAI, Zhejiang University
https://LOAFisheye.github.io/

###### Abstract

Location determination finds wide applications in daily life. Instead of
existing efforts devoted to localizing tourist photos${}_{\\!}$
captured${}_{\\!}$ by${}_{\\!}$ perspective${}_{\\!}$ cameras, in this
article,${}_{\\!}$ we focus${}_{\\!}$ on${}_{\\!}$ devising${}_{\\!}$
person${}_{\\!}$ positioning${}_{\\!}$ solutions${}_{\\!}$ using${}_{\\!}$
over- head fisheye cameras. Such solutions are advantageous in large field of
view (FOV), low cost, anti-occlusion, and un- aggressive${}_{\\!}$
work${}_{\\!}$ mode${}_{\\!}$ (without${}_{\\!}$ the${}_{\\!}$
necessity${}_{\\!}$ of${}_{\\!}$ cameras${}_{\\!}$ car-${}_{\\!}$ ried
by${}_{\\!}$ persons). However, related studies are quite scarce, due to the
paucity of data. To stimulate research in this exciting area, we present LOAF,
the first large-scale overhead fisheye dataset for person detection and
localization. LOAF is built with many essential features, _e.g_., i) the data
cover abundant diversities in scenes, human pose, density, and location; ii)
it contains currently the largest number of annotated pedestrian, _i.e_., 457K
bounding boxes with ground-truth location information; iii) the body-boxes are
labeled as radius-aligned so as to fully address the positioning challenge. To
approach localization, we build a fisheye person detection network, which
exploits the fisheye distortions by a rotation-equivariant training strategy
and predict radius-aligned human boxes end-to-end. Then, the actual locations
of the detected persons are calculated by a numerical solution on the fisheye
model and camera altitude data. Extensive experiments on LOAF validate the
superiority of our fisheye detector w.r.t. previous methods, and show that our
whole fisheye positioning solution is able to locate all persons in FOV with
an accuracy of 0.5 m, within 0.1 s.

##  1 Introduction

Accurate position finding of persons attracts growing interest from both
research and industrial communities, since it${}_{\\!}$ plays${}_{\\!}$
a${}_{\\!}$ crucial${}_{\\!}$ role${}_{\\!}$ in${}_{\\!}$ numerous${}_{\\!}$
location-sensitive${}_{\\!}$ applica- tion${}_{\\!}$ scenarios${}_{\\!}$
(_e.g_.,${}_{\\!}$ surveillance,${}_{\\!}$ smart${}_{\\!}$ home,${}_{\\!}$
public${}_{\\!}$ health). Nevertheless, due to the line-of-sight (LOS) issue,
GPS is unreliable${}_{\\!}$ in${}_{\\!}$ interior${}_{\\!}$ spaces${}_{\\!}$
and${}_{\\!}$ urban${}_{\\!}$ canyon.${}_{\\!}$ To${}_{\\!}$ overcome
such${}_{\\!}$ limitation,${}_{\\!}$ various${}_{\\!}$ alternative${}_{\\!}$
solutions${}_{\\!}$ are${}_{\\!}$ investigated.${}_{\\!}$ Signal${}_{\\!}$
based${}_{\\!}$ solutions,${}_{\\!}$ including${}_{\\!}$ Bluetooth${}_{\\!}$
[13]${}_{\\!}$ and${}_{\\!}$ Wi-Fi [74], are popular, but they are easily
interfered by changing environments and nearby human bodies [73]. A
complementary stream of work is vision based; they typically make use of
traditional cameras, RGBD cameras, or in-built smartphone cameras, and enjoy
the advantage of reliable services. To get location information, visual
positioning solutions usually refer to a pre-acquired 3D map or a geo-tagged
database as the scene representation [63], or directly utilize the captured
image to estimate the camera pose [38].

Figure 1: ${}_{\\!}$Person${}_{\\!}$ positioning${}_{\\!}$ using${}_{\\!}$
overhead${}_{\\!}$ fisheye${}_{\\!}$ camera:${}_{\\!}$ We detect humans on
omnidirectional images and then project detects onto the real world
coordinates to obtain physical locations. Compared with using perspective
cameras, our fisheye camera based solution is favored in low cost, high
accuracy, and fast speed.

Although visual localization has been a hotspot issue for many years, existing
efforts are mainly dedicated to urban place recognition or indoor camera
localization, based on perspective cameras [63, 38, 62, 42]. None of them
addresses person positioning by using overhead fisheye cameras, even though
fisheye cameras are widely used in visual surveillance applications. One
possible reason is the lack of accessible datasets, compounded by considerable
costs involved in data collection. In this article, we provide a large-scale
overhead fisheye dataset, LOAF, for person detection and localization in both
indoor and outdoor scenes.

${}_{\\!}$Compared${}_{\\!}$ with${}_{\\!}$ perspective${}_{\\!}$ cameras,
overhead${}_{\\!}$ (top-view)${}_{\\!}$ fisheye${}_{\\!}$ cameras${}_{\\!}$
are promoted due to less occlusion among people${}_{\\!}$ and${}_{\\!}$
larger${}_{\\!}$ field${}_{\\!}$ of${}_{\\!}$ view${}_{\\!}$ (FOV) –
allowing${}_{\\!}$ the${}_{\\!}$ coverage${}_{\\!}$ of${}_{\\!}$ a${}_{\\!}$
large${}_{\\!}$ space${}_{\\!}$ using${}_{\\!}$ a${}_{\\!}$ single, low-
cost${}_{\\!}$ camera. Only${}_{\\!}$ few public datasets [21, 41, 24] provide
top-view fisheye data. Unfortunately, they only cover quite few scenes and
their people-box annotations do not adequately address the positioning
challenge, negatively affecting location approximation (see §3.2 for detailed
analysis). Differently, LOAF specifically targets at person localization in
surveillance applications, and has the following appealing characteristics:

  * •

Large-scale: LOAF is the largest in the filed, to our best knowledge. It
consists of over 70 videos, with more than 43K frame images, 457K person-
detection annotations as well as corresponding location information.

  * •

High diversity: LOAF contains a wide variety of surveillance scenarios;
it${}_{\\!}$ includes${}_{\\!}$ a${}_{\\!}$ total${}_{\\!}$ of${}_{\\!}$
11${}_{\\!}$ indoor${}_{\\!}$ and${}_{\\!}$ 40${}_{\\!}$ out- door scenes, and
the data are captured at different times of day and cover different
illumination conditions.

  * •

Positioning-aware person-box annotation: LOAF offers radius-aligned human-box
annotations. Compared${}_{\\!}$ with${}_{\\!}$ other${}_{\\!}$
person${}_{\\!}$ representations, _i.e_., head${}_{\\!}$ center${}_{\\!}$
[21], axis- aligned${}_{\\!}$ [60] or${}_{\\!}$ body-aligned${}_{\\!}$
box${}_{\\!}$ [41, 24], used in previous fisheye datasets, the radius-aligned
box is promoted, as it fits well radially-oriented bodies [41], and, more
importantly, it is better aware of the positioning problem.${}_{\\!\\!\\!}$

${}_{\\!}$Moreover,${}_{\\!}$ we${}_{\\!}$ devise${}_{\\!}$ a${}_{\\!}$
person${}_{\\!}$ positioning${}_{\\!}$ system${}_{\\!}$ that${}_{\\!}$ first
detects persons from raw fisheye images, and then calcula- tes physical
locations based on fisheye visual model and al- titude${}_{\\!}$
information${}_{\\!}$ (see${}_{\\!}$ Fig. 1).${}_{\\!}$ Clearly,${}_{\\!}$
high-quality${}_{\\!}$ person${}_{\\!}$ detection is the crucial premise for
precise localization using overhead fisheye cameras. Fisheye lenses provide
large FOV, at the cost of strong radial distortion. This makes pe- destrian
detection in top-view fisheye images a much harder task, compared with using
perspective cameras. Studies on overhead fisheye pedestrian detection are very
scarce [60, 41, 50, 24, 64]; they rarely concern the link with person
positioning, and many of them [60, 41] are even not trainable due to the lack
of fisheye data. With our LOAF dataset, we develop${}_{\\!}$ a${}_{\\!}$
novel${}_{\\!}$ query${}_{\\!}$ based${}_{\\!}$ fisheye${}_{\\!}$
human${}_{\\!}$ detector.${}_{\\!}$ It${}_{\\!}$ expli- citly exploits fisheye
geometry by accommodating rotation equivariance into the matching between
queries and human instances during network training. The insight here is
intui- tive:${}_{\\!}$ for${}_{\\!}$ a${}_{\\!}$ robust${}_{\\!}$
fisheye${}_{\\!}$ detector,${}_{\\!}$ rotation${}_{\\!}$ of${}_{\\!}$
a${}_{\\!}$ fisheye${}_{\\!}$ image should result in correspondingly rotated
detections. In addition,${}_{\\!}$ our${}_{\\!}$ detection${}_{\\!}$
algorithm${}_{\\!}$ learns${}_{\\!}$ to${}_{\\!}$ predict radius-aligned
person boxes, facilitating localization estimation.

We test our fisheye person detection algorithm as well as our whole
positioning system over LOAF. We find that our detector significantly
outperforms previous methods, and our full system delivers precise
localization results. We also empirically show that our algorithm generalizes
well on previous top-view fisheye person detection datasets [24, 65].

##  2 Related Work

Accurate${}_{\\!}$ Positioning.${}_{\\!}$ GPS${}_{\\!}$ is${}_{\\!}$
the${}_{\\!}$ most${}_{\\!}$ popular${}_{\\!}$ system${}_{\\!}$ for outdoor
localization. As it requires LOS between the satellites and the handset,
GPS${}_{\\!}$ does${}_{\\!}$ not${}_{\\!}$ function well in urban canyons,
indoors${}_{\\!}$ and${}_{\\!}$ basements [30, 26].${}_{\\!}$ This${}_{\\!}$
triggered${}_{\\!}$ the development of alternative positioning solutions,
following a${}_{\\!}$ multi-disciplinary${}_{\\!}$ approach.${}_{\\!}$
Concretely,${}_{\\!}$ there${}_{\\!}$ are${}_{\\!}$ two${}_{\\!}$
main${}_{\\!}$ schemes of the alternatives: signal-based and vision-based.

Signal-based${}_{\\!}$ positioning${}_{\\!}$ systems${}_{\\!}$
typically${}_{\\!}$ lean${}_{\\!}$ on sound wave${}_{\\!}$ [49],
geomagnetism${}_{\\!}$ [31], radio${}_{\\!}$ frequency${}_{\\!}$
(RF)${}_{\\!}$ [26, 74, 54, 13, 1], and infrared radiation (IR)${}_{\\!}$
[71], as well as different location determination techniques, such as TOA
(time of arrival)${}_{\\!}$ [30] and RSS (received signal strength)${}_{\\!}$
[26]. The main challenge to signal-based systems is the sensitivity to
environment${}_{\\!}$ changes,${}_{\\!}$ such${}_{\\!}$ as${}_{\\!}$
object${}_{\\!}$ moving,${}_{\\!}$ diffraction${}_{\\!}$ and reflection, which
affect signal propagation [26].

Visual${}_{\\!}$ data${}_{\\!}$ is${}_{\\!}$ another${}_{\\!}$
potential${}_{\\!}$ information${}_{\\!}$ source${}_{\\!}$ for${}_{\\!}$ pre-
cise${}_{\\!}$ localization.${}_{\\!}$ Since${}_{\\!}$ put${}_{\\!}$
forward${}_{\\!}$ [61],${}_{\\!}$ visual${}_{\\!}$ positioning${}_{\\!}$
has${}_{\\!}$ became${}_{\\!}$ a${}_{\\!}$ hot${}_{\\!}$ topic${}_{\\!}$
in${}_{\\!}$ robotics${}_{\\!}$ and${}_{\\!}$ computer${}_{\\!}$
vision.${}_{\\!}$ Some methods${}_{\\!}$ adopt${}_{\\!}$ a${}_{\\!}$ pre-
built${}_{\\!}$ geo-tagged${}_{\\!}$ database${}_{\\!}$ or${}_{\\!}$
a${}_{\\!}$ 3D${}_{\\!}$ scene model,${}_{\\!}$ as${}_{\\!}$ the${}_{\\!}$
reference${}_{\\!}$ for${}_{\\!}$ camera${}_{\\!}$ location${}_{\\!}$
estimation${}_{\\!}$ [32, 77, 66, 78, 76].${}_{\\!}$ Some${}_{\\!}$
others${}_{\\!}$ rely${}_{\\!}$ on${}_{\\!}$ recognizing${}_{\\!}$
some${}_{\\!}$ dep- loyed${}_{\\!}$ coded${}_{\\!}$ targets${}_{\\!}$ [27, 59,
20, 35],${}_{\\!}$ _e.g_.,${}_{\\!}$ concentric rings, barcodes,${}_{\\!}$
colored${}_{\\!}$ dots,${}_{\\!}$ _etc_.${}_{\\!}$ Some${}_{\\!}$
recent${}_{\\!}$ ones${}_{\\!}$ utilize${}_{\\!}$ deep learning techniques to
replace some components (_e.g_., image retrieval, descriptor matching) in
traditional systems [4, 16], or regress the camera pose directly [38, 37,
7]. Though promising, existing visual positioning systems are mostly founded
on perspective cameras.

Due to the problem complexity, there is no persuasive solution for precise
positioning yet, and hybrid schemes are often applied in practice. Fisheye
cameras have advantage of providing wide FOV with low cost and reduced
occlusion, while the research landscape is sparse for fisheye camera based
localization [79, 21]. To foster research in this direction and facilitate
practical deployment, we contribute the first overhead fisheye dataset, to our
knowledge, that allows to conduct the task of person localization at large
scale.

Deep Learning based Visual (Camera) Localization. Re- garding${}_{\\!}$
application${}_{\\!}$ scenarios,${}_{\\!}$ scholars${}_{\\!}$ in${}_{\\!}$
computer${}_{\\!}$ vision community are mainly aware of city-scale location
recogni- tion [2, 72, 58, 57] and indoor camera localization [5, 67]. Popular
visual localization approaches can be divided${}_{\\!}$ as${}_{\\!}$
retrieval-based,${}_{\\!}$ regression-based,${}_{\\!}$ and structure-based,
according to the camera pose estimation strategy. Retrieval-based methods [58,
3, 23, 75, 56] represent a scene as a database of geo-tagged photos and use
geo-tag of the most relevant database photo as an approximation to the camera
position [58]. Regression-based methods learn to encode the scene into a deep
network and directly regress a 6DOF camera pose [38, 67, 37] from a captured
image. Structure-based methods${}_{\\!}$ [6, 55, 29] pose the
localization${}_{\\!}$ problem${}_{\\!}$ as${}_{\\!}$ a${}_{\\!}$
camera${}_{\\!}$ resectioning${}_{\\!}$ task${}_{\\!}$ [58].${}_{\\!}$
They${}_{\\!}$ first represent scenes via 3D models and establish a set of
2D-3D matches, then recover the full camera pose by employing a PnP solver
[33] inside a RANSAC [28] loop.

These visual positioning techniques seek to localize images captured by
handheld devices [42, 11] or vehicle cameras [47] and demand pre-created geo-
tagged databases or 3D maps. And they work on an active mode – users need to
carry the cameras. Differently, we address person localization in surveillance
scenarios, by using stationary, overhead fisheye cameras. This yields a
passive, low-cost scheme, where the localization is completed by a numerical
solution on the fisheye model fused with altitude information. Thus our scheme
faces a different challenge, _i.e_., conducting robust person detection from
non-rectilinear fisheye images. Overall, our scheme fills the gap left by
conventional studies and is complementary to existing positioning systems.

Person${}_{\\!}$ Detection${}_{\\!}$ in${}_{\\!}$ Overhead${}_{\\!}$
Fisheye${}_{\\!}$ Images. As${}_{\\!}$ a${}_{\\!}$ ca- nonical${}_{\\!}$
subproblem of object detection, pedestrian detection has long received great
interest owing to its broad applications such as intelligent surveillance and
autonomous vehicles [12, 8]. By contrast, people detection in overhead
fisheye images has been studied much less, due to the absence of such
datasets. Moreover, adapting standard pedestrian detectors to top-view fisheye
cameras is difficult [15]: First, the appearance of people in fisheye images
is arbitrary-oriented, while in perspective images, people typically appear
upright. Second, people suffer from severe geometric distortions, particularly
in the fisheye image’s periphery.

Faced with the above challenges, early attempts make use of handcrafted
features (_e.g_., HOG, LBP) as well as standard pedestrian detectors with
slight modifications to account${}_{\\!}$ for${}_{\\!}$ fisheye${}_{\\!}$
geometry [39, 14, 17, 69]. The${}_{\\!}$ common${}_{\\!}$ approach is
dewarping fisheye images [14] or features [39] so as to approximate normal
people’ appearances from the deformed ones. Though this simplifies the
classification of features, approximation errors are inevitable, causing
performance degradation [64]. Later, a few CNN-based fisheye detectors were
developed. Some of them are training-free. For example, [60] runs standard
YOLOv2 [51] on dewarped versions of overlapped windows extracted from a
fisheye image.${}_{\\!}$ Li _et al_.  [41]${}_{\\!}$ rotate${}_{\\!}$
a${}_{\\!}$ fisheye${}_{\\!}$ image${}_{\\!}$ in${}_{\\!}$ 15∘${}_{\\!}$
increments, and apply off-the-shelf YOLOv3 [52] only to the top-center region
of each rotated image, where people usually appear upright. Some more recent
methods are trainable: [64] trains YOLOv2 with rotated perspective images so
as to handle omnidirectional images without test-time transformation; [24]
trains YOLOv3 with human-aligned boxes.

Although many prior arts${}_{\\!}$ [60, 41, 64] allowing to detect
persons${}_{\\!}$ without${}_{\\!}$ any${}_{\\!}$ fisheye${}_{\\!}$
training${}_{\\!}$ data,${}_{\\!}$ they${}_{\\!}$ require${}_{\\!}$
a${}_{\\!}$ cer- tain${}_{\\!}$ amount${}_{\\!}$ of${}_{\\!}$
computation${}_{\\!}$ time${}_{\\!}$ for post-processing:${}_{\\!}$
[60]${}_{\\!}$ carries${}_{\\!}$ out${}_{\\!}$ detection${}_{\\!}$
in${}_{\\!}$ multiple${}_{\\!}$ perspective${}_{\\!}$ images${}_{\\!}$
dewarped from one omnidirectional image; [41]${}_{\\!}$ applies${}_{\\!}$
YOLOv3${}_{\\!}$ 24${}_{\\!}$ times${}_{\\!}$ to${}_{\\!}$ each${}_{\\!}$
fisheye${}_{\\!}$ image;${}_{\\!}$ [64]${}_{\\!}$ needs${}_{\\!}$ a${}_{\\!}$
grouping${}_{\\!}$ process${}_{\\!}$ to${}_{\\!}$ eliminate${}_{\\!}$
numerous${}_{\\!}$ redundant${}_{\\!}$ results,${}_{\\!}$ caused${}_{\\!}$
by${}_{\\!}$ a${}_{\\!}$ rotation-invariant${}_{\\!}$ training${}_{\\!}$
strategy.${}_{\\!}$ Hence,${}_{\\!}$ the${}_{\\!}$ utility${}_{\\!}$
of${}_{\\!}$ previous${}_{\\!}$ me- thods${}_{\\!}$ is${}_{\\!}$
limited${}_{\\!}$ in${}_{\\!}$ the${}_{\\!}$ context${}_{\\!}$ of${}_{\\!}$
visual${}_{\\!}$ positioning.${}_{\\!}$ Our${}_{\\!}$ ana- lysis${}_{\\!}$
in${}_{\\!}$ §3.2${}_{\\!}$ sheds${}_{\\!}$ light${}_{\\!}$ on${}_{\\!}$
the${}_{\\!}$ weaknesses${}_{\\!}$ of${}_{\\!}$ human representations adopted
by existing${}_{\\!}$ fisheye${}_{\\!}$ detectors${}_{\\!}$ (_e.g_.,
head${}_{\\!}$ center${}_{\\!}$ [21], horizontal${}_{\\!}$ or${}_{\\!}$ body-
aligned${}_{\\!}$ box${}_{\\!}$ [60, 41, 24])${}_{\\!}$ in regard${}_{\\!}$
to${}_{\\!}$ localization.${}_{\\!}$ Thus${}_{\\!}$ we${}_{\\!}$
supply${}_{\\!}$ our${}_{\\!}$ large-scale${}_{\\!}$ dataset with${}_{\\!}$
positioning-aware${}_{\\!}$ person-detection${}_{\\!}$ annotation.${}_{\\!}$
As${}_{\\!}$ a${}_{\\!}$ re- sult,${}_{\\!}$ our${}_{\\!}$ human${}_{\\!}$
detector${}_{\\!}$ can${}_{\\!}$ benefit${}_{\\!}$ from${}_{\\!}$ end-to-
end,${}_{\\!}$ fisheye${}_{\\!}$ visual${}_{\\!}$ pattern${}_{\\!}$
learning${}_{\\!}$ and${}_{\\!}$ output${}_{\\!}$ radius-aligned${}_{\\!}$
body-boxes${}_{\\!}$ for${}_{\\!}$ precise${}_{\\!}$ positioning.${}_{\\!}$
Further,${}_{\\!}$ by${}_{\\!}$ regularizing${}_{\\!}$ the${}_{\\!}$
learning${}_{\\!}$ of${}_{\\!}$ image${}_{\\!}$ representations${}_{\\!}$
and${}_{\\!}$ instance${}_{\\!}$ queries${}_{\\!}$ with${}_{\\!}$
rotation${}_{\\!}$ equivariance,${}_{\\!}$ our${}_{\\!}$ algorithm${}_{\\!}$
naturally${}_{\\!}$ addresses${}_{\\!}$ the${}_{\\!}$ omnidi-
rectional${}_{\\!}$ nature${}_{\\!}$ of${}_{\\!}$ fisheye${}_{\\!}$
images,${}_{\\!}$ yet${}_{\\!}$ using${}_{\\!}$ standard${}_{\\!}$ detec-
tion${}_{\\!}$ network${}_{\\!}$ architecture${}_{\\!}$ designed${}_{\\!}$
for${}_{\\!}$ perspective${}_{\\!}$ images.

##  3 LOAF Dataset

###  3.1 Dataset Acquisition

We first describe how our fisheye images are collected.

Apparatus and Technical Specifications. A TL-IPC59AE fisheye camera with 1.1mm
focal length is adopted${}_{\\!}$ for data recording.${}_{\\!}$ It${}_{\\!}$
has${}_{\\!}$ a${}_{\\!}$ wide${}_{\\!}$ FOV, reaching${}_{\\!}$ the${}_{\\!}$
full${}_{\\!}$ circle${}_{\\!}$ in${}_{\\!}$ the${}_{\\!}$
horizontal${}_{\\!}$ plane${}_{\\!}$ and${}_{\\!}$ 180∘ in${}_{\\!}$
the${}_{\\!}$ vertical${}_{\\!}$ plane.${}_{\\!}$ This${}_{\\!}$ offers a
clear advantage in reducing deployment cost – installing just one fisheye
camera instead of multiple conventional cameras to monitor the same region.
However, the severe geometric distortions introduced prevent the use of
standard detectors, which are designed for conventional cameras [8].

Data Capturing. The fisheye camera is mounted on the ceiling (indoors) or
poles (outdoors), 2.5$\sim$4.0 m from the ground with 200$\sim$300 m2 FOV.
Considering the factors${}_{\\!}$ such${}_{\\!}$ as${}_{\\!}$
scenario${}_{\\!}$ diversity, pedestrians${}_{\\!}$ density${}_{\\!}$
and${}_{\\!}$ weather condition, we capture 110 fisheye image sequences in 80
realistic scenarios as the raw data pool. The recorded sequences span 14 hrs;
the image resolution is 2952$\times$2952 pixels, and the frame rate is
10$\sim$20 fps. Eventually, 42,942 images, sampled at 1 fps, are collected to
construct our LOAF dataset.

Figure 2: Human annotation comparison for visual positioning (§3.2). Previous
methods typically use the detected human head [21] () or the center () of
human-aligned body-box () [41, 24] to determine the physical location $P$,
bringing positioning errors (_i.e_., $PP_{1}$, $PP_{2}$). We instead leverage
the radius-aligned box () as human representation. The midpoint () of side
$AB$, _i.e_., the closest point to the image center on the radius-aligned box,
better corresponds to the human actual position.

###  3.2 Dataset Annotation

We next describe how our fisheye images are annotated.

Dataset | #Scene | #Video | #Image | #People | Max Resolution | FOV (m2) | Annotation | FPS
---|---|---|---|---|---|---|---|---
Indoor | Outdoor | Total | Avg. | Max | People Detection | Location | Attribute
PIROPO [21] | 2 | 0 | 27 | - | - | - | - | 800$\times$600 |  $<$50 | Head Center |  |  | 10
HABBOF [41] | 2 | 0 | 4 | 5,837 | 20,466 | 3.5 | 5 | 2,048$\times$2,048 |  $<$36 | Human-aligned box |  |  | 12$\sim$30
MW-R [24] | 6 | 0 | 19 | 8,752 | 22,825 | 2.6 | 6 | 1,488$\times$1,488 |  $<$36 | Human-aligned box |  |  | 15
CEPDOF [24] | 5 | 0 | 8 | 25,504 | 173,073 | 6.8 | 13 | 2,048$\times$2,048 |  $<$36 | Human-aligned box |  |  | 1$\sim$10
WEPDTOF [65] | 14 | 0 | 16 | 10,544 | 93,363 | 8.9 | 35 | 2,592$\times$1,944 |  $<$36 | Human-aligned box |  |  | 1$\sim$10
LOAF | 11 | 39 | 74 | 42,942 | 457,762 | 10.5 | 65 | 2,952$\times$2,952 | 200$\sim$300 | Radius-aligned box | ✓ | ✓ | 10$\sim$20

Table 1: Overhead${}_{\\!}$ fisheye${}_{\\!}$ datasets${}_{\\!}$
comparison${}_{\\!}$ (§3.4). LOAF${}_{\\!}$ is the largest in terms of the
total number of pedestrian and scene categories.

Person Detection. As the performance of localization relies critically on the
quality of person detections, significant effort should be spent on the
fisheye person-detection annotation. Some human-detection representations were
explored in previous datasets: [21] labels the center point of each human
head; [41, 24] opt for human-aligned person-boxes. However, these
representations have several shortcomings, especially with regards to the
localization task. First, they suffer from some inherent limitations. The
point-based representation is not applicable in other analysis tasks (_e.g_.,
re-identification, attribute recognition); the human-aligned box has ambiguity
[10, 68] – the ground-truths of human-aligned boxes are not uniquely
determined [24]. Second, and most important, these human-detection
representations cannot meet the need of precise localization. Clearly, based
on the fisheye camera model, one can project 2D detections on the 3D world for
localization. However, the center of human head or of human-aligned box is not
the exact placement occupied by human on the image plane, causing
errors${}_{\\!}$ to${}_{\\!}$ estimations${}_{\\!}$ of${}_{\\!}$
human${}_{\\!}$ physical${}_{\\!}$ position${}_{\\!}$ (see${}_{\\!}$
Fig.${}_{\\!}$ 2).${}_{\\!}$

We instead label each person through a radius-aligned rectangular${}_{\\!}$
box.${}_{\\!}$ Such${}_{\\!}$ representation${}_{\\!}$ is${}_{\\!}$
favored${}_{\\!}$ as${}_{\\!}$ it:${}_{\\!}$ i)${}_{\\!}$ allows unique
groundtruth box assignment; ii) fits well radially-oriented human bodies
presented in fisheye images; and iii) better corresponds to the actual
position of human on the image plane, facilitating physical localization.
Notably, although the radially-oriented box constraint was explored in a prior
fisheye detector [41], there is no previous datasets provide such kind of
annotation, neither is there any literature points out the advantage of such
representation in localization. Finally, around 457K human box annotations are
obtained. High-quality annotation is ensured via a rigorous quality check,
conducted by highly skilled reviewers.

Figure 3: Calibration for groundtruth person location annotation. (a) Ground
makers (with 0.05m precision). (b) Top-view fisheye image capture for
calibration. See §3.2 for details.

Person Localization. For each scene, a 10 m ruler (with 0.05 m
accuracy)${}_{\\!}$ is${}_{\\!}$ placed${}_{\\!}$ on${}_{\\!}$ the${}_{\\!}$
ground,${}_{\\!}$ and${}_{\\!}$ one${}_{\\!}$ end${}_{\\!}$ of${}_{\\!}$
the${}_{\\!}$ ruler is directly under the fisheye lens (cf. Fig. 3 (a)). We
take a fisheye picture and use this marked picture as the calibration for all
the dataset images recorded in this scene (cf. Fig. 3 (b)). Finally, for each
annotated human, the${}_{\\!}$ physical${}_{\\!}$ location,${}_{\\!}$
at${}_{\\!}$ sub-decimeter${}_{\\!}$ precision,${}_{\\!}$ is${}_{\\!}$
provided (cf. Fig. 4).

(a) Our proposed LOAF(b) PIROPO [21](c) HABBOF [41](d) CEPDOF [24]

Figure 4: Example images from different datasets (§3.2-§3.4). Prior datasets
[21, 41, 24] are restricted to few indoor scenes with person detection
annotation only, _i.e_., head center () or human-aligned box (). In contrast,
LOAF covers challenging indoor and outdoor scenes with human detection,
localization and scene attribute annotations. The person-detection annotation
is given as the radius-aligned box (), which is more suitable for
localization. For better visualization, we only present location ground-truths
for some of persons.

Scene Attribute. To enable in-depth analysis, each image is annotated with
multiple attributes, including day/night, outdoor/indoor, and
sunny/rain/foggy/snow.

###  3.3 Dataset Design

We then list several key aspects of our dataset design.

Privacy Protection. To protect personal information, we apply the gaussian
filter to blur all the visible facial regions in our dataset, and conduct
experiments on the blurred data.

Dataset Splits. LOAF contains 29,569 training, 4,600 validation, and 8,773
testing images (approximately 7 train, 1 val, and 2 test). Moreover, to better
evaluate models’ generalization ability, LOAF is split into five sets: train
(7/28 indoor/outdoor scenes, 29,569 images), val seen (1/2 indoor/outdoor
scenes, 1,700 images), val unseen (2/3 indoor/outdoor scenes, 2,900 images),
test seen (2/3 indoor/outdoor scenes, 2,774 images), test unseen (2/8
indoor/outdoor scenes, 5,999 images). There are no overlapping scenes between
unseen and train sets.

Dataset${}_{\\!}$ Accessibility.${}_{\\!}$ Only${}_{\\!}$ the${}_{\\!}$
desensitized${}_{\\!}$ version${}_{\\!}$ of${}_{\\!}$ our${}_{\\!}$
dataset${}_{\\!}$ will${}_{\\!}$ be${}_{\\!}$ released online, under${}_{\\!}$
the${}_{\\!}$ Creative Commons Attribution-NonCommercial-ShareAlike 4.0
License [19].

0306090120150180210240270300330 ° ° ° ° ° ° ° ° ° ° ° ° 4803602401200 |  05k10k15k20k25k30kCenter Distance (centimeter)0100200 Number of Boxes Number of Person Boxes010203040506001k2k Number of Images
---|---

(a) Person scale (b) Person locations (c) Person density

Figure 5: Dataset Statistics (§3.4): We summarize LOAF with the distribution
of (a) person scale (area in pixel), (b) person locations (horizontal distance
between human and the fisheye lens in the real world), and (c) person density
(number of person per image).

###  3.4 Dataset Features and Analysis

Finally,${}_{\\!~{}}$ we${}_{\\!~{}}$ present${}_{\\!~{}}$
statistic${}_{\\!~{}}$ analysis${}_{\\!~{}}$ of${}_{\\!~{}}$ LOAF${}_{\\!~{}}$
in${}_{\\!~{}}$ com-

parison with existing overhead fisheye datasets [21, 41, 24]. LOAF
distinguishes itself from three aspects (cf. Table 1):

Large-scale: LOAF has 42,942 fisheye images with more than 457K person boxes.
Moreover, LOAF data are captured by an advanced fisheye camera, which is
capable of covering a larger area (200$\sim$300 m2) with higher pedestrian
density (2$\sim$65 persons per scene, 10.5 in average). This makes LOAF the
largest overhead fisheye dataset in terms of the total number of pedestrian
and scene categories.

High${}_{\\!}$ Diversity: Existing${}_{\\!}$ datasets${}_{\\!}$
limit${}_{\\!}$ in${}_{\\!}$ data${}_{\\!}$ diversities, _i.e_.,
only${}_{\\!}$ containing${}_{\\!}$ very${}_{\\!}$ few${}_{\\!}$
indoor${}_{\\!}$ scenes${}_{\\!}$ (2$\sim$14)${}_{\\!}$ and${}_{\\!}$
completely${}_{\\!}$ missing${}_{\\!}$ outdoor${}_{\\!}$ scenarios${}_{\\!}$
(cf. Fig. 4). In contrast, LOAF${}_{\\!}$ involves${}_{\\!}$ 51${}_{\\!}$
realistic${}_{\\!}$ scenes, including${}_{\\!}$ 11${}_{\\!}$ indoor${}_{\\!}$
scenes (_e.g_., lab, office, library, classroom) and 40 outdoor scenes
(_e.g_., street, playground, parking lot, square). The recorded data cover
four seasons under different illumination (_e.g_., morning, noon, afternoon)
and weather (_e.g_., sunny, rain, snow) conditions, and involve vast variance
of human pose (_e.g_., walking, standing, and sitting), scale, location, and
density (cf. Fig. 5). Thus our dataset better reflects the distribution in
real-world surveillance scenarios.

Rich and Positioning-aware Annotation: LOAF is provided with rich ground-
truths for detection, localization, and scene attribute, which lays a solid
foundation for fisheye camera based human-centric analysis. Hence, as
demonstr- ated in §3.2, the radius-aligned human-box representation is adopted
during our annotation. Compared with human-head center based point annotation
[21] and human-aligned person-boxes [41, 24] used in previous datasets,
radius-aligned human-boxes are more suitable for the position task.

##  4 Our Approach

Our overhead fisheye camera based person localization solution comprises two
parts. The former computes 2D detections that locate the people on the image
plane (§4.1). The latter converts the 2D detections to 3D-world coordinates,
obtaining the physical location of the people (§4.2).

###  4.1 Overhead Fisheye Person Detection

Core${}_{\\!}$ Idea:${}_{\\!}$ Rotation${}_{\\!}$ Equivariance.${}_{\\!}$ One
of the reasons for the tremendous success of CNNs is their equivariance to
horizontal and vertical shifts and the resulting invariance to local
deformations [40, 25]. This${}_{\\!}$ stimulates${}_{\\!}$ a${}_{\\!}$
line${}_{\\!}$ of${}_{\\!}$ efforts${}_{\\!}$ to${}_{\\!}$ learn${}_{\\!}$
robust${}_{\\!}$ representations${}_{\\!}$ equivariant to generic types of
transformations${}_{\\!}$ [34, 18, 70]. Formally,${}_{\\!}$ a${}_{\\!}$
representation${}_{\\!}$ $f_{\\!}$ is${}_{\\!}$ said${}_{\\!}$ to${}_{\\!}$
be${}_{\\!}$ equivariant${}_{\\!}$ with${}_{\\!}$ a${}_{\\!}$
geometric${}_{\\!}$ transformation${}_{\\!}$ $g$${}_{\\!}$ (_e.g_.,${}_{\\!}$
cropping, flipping) for an input (image) $I$ if:

$\displaystyle f(g({I}))\approx g(f(I)).$ (1)

That is to say, the output representation $f(I)$ is changed in the${}_{\\!}$
same${}_{\\!}$ way${}_{\\!}$ as${}_{\\!}$ the${}_{\\!}$
transformation${}_{\\!}$ $g_{\\!}$ imposed${}_{\\!}$ to${}_{\\!}$
the${}_{\\!}$ input${}_{\\!}$ $I$.${}_{\\!}$

In${}_{\\!}$ this${}_{\\!}$ work,${}_{\\!}$ we${}_{\\!}$ devise${}_{\\!}$
a${}_{\\!}$ query-based${}_{\\!}$ fisheye${}_{\\!}$ person${}_{\\!}$ de-
tector${}_{\\!}$ that${}_{\\!}$ exhibits${}_{\\!}$ 360∘-rotational${}_{\\!}$
equivariance${}_{\\!}$ through${}_{\\!}$ an elaborately${}_{\\!}$
designed${}_{\\!}$ training${}_{\\!}$ protocol.${}_{\\!}$ Our key insight is
derived from the omnidirectional nature of this task: if one rotates the input
fisheye image by an arbitrary angle, then the outputs of a robust fisheye
detector should change accord- ingly.${}_{\\!}$ We${}_{\\!}$ thus${}_{\\!}$
design${}_{\\!}$ a${}_{\\!}$ rotation${}_{\\!}$ equivariant${}_{\\!}$
training${}_{\\!}$ strategy, which forces the matching between object queries
and image representations to be equivariant against 360∘-rotations.
In${}_{\\!}$ this${}_{\\!}$ way,${}_{\\!}$ the${}_{\\!}$ rotational${}_{\\!}$
symmetry${}_{\\!}$ of${}_{\\!}$ omnidirectional${}_{\\!}$ im- ages${}_{\\!}$
is${}_{\\!}$ explicitly${}_{\\!}$ addressed,${}_{\\!}$ without${}_{\\!}$
architectural${}_{\\!}$ modification of standard detectors developed for
perspective images.

Rotation${}_{\\!}$ Equivariant${}_{\\!}$ Training${}_{\\!}$ for${}_{\\!}$
Query-based${}_{\\!}$ Fisheye${}_{\\!}$ Detection.${}_{\\!}$ As${}_{\\!}$
shown${}_{\\!}$ in${}_{\\!}$ Fig.${}_{\\!}$ 6,${}_{\\!}$ our${}_{\\!}$
fisheye${}_{\\!}$ detector${}_{\\!}$ is${}_{\\!}$ built${}_{\\!}$
upon${}_{\\!}$ DETR${}_{\\!}$ [9],${}_{\\!}$ promoted${}_{\\!}$ by${}_{\\!}$
a${}_{\\!}$ rotation${}_{\\!}$ equivariant${}_{\\!}$ training${}_{\\!}$
strategy.${}_{\\!~{}}$ Basically,${}_{\\!~{}}$ DETR${}_{\\!~{}}$
conducts${}_{\\!~{}}$ detection${}_{\\!~{}}$ in${}_{\\!~{}}$ a${}_{\\!~{}}$
query-

based fashion.${}_{\\!}$ Denote${}_{\\!}$ $\mathcal{F}_{\\!}$ as${}_{\\!}$
a${}_{\\!}$ feature${}_{\\!}$ encoder${}_{\\!}$ ${}_{\\!}$ that${}_{\\!}$
extracts representation${}_{\\!}$ ${}_{\\!}$ $\bm{I}_{\\!}$ of${}_{\\!}$
image${}_{\\!}$ $I$,${}_{\\!}$ and${}_{\\!}$ $\kappa_{\\!}$ as${}_{\\!}$
a${}_{\\!}$ query${}_{\\!}$ creator${}_{\\!}$ that${}_{\\!}$ outputs${}_{\\!}$
a${}_{\\!}$ set${}_{\\!}$ of${}_{\\!}$ $N_{\\!}$ object-aware${}_{\\!}$
descriptors${}_{\\!}$ ${}_{\\!}$ $\\{\bm{q}_{n}\\}_{n=1}^{N}$ from $\bm{I}$.
DETR employs $\\{\bm{q}_{n}\\}_{n=1}^{N}$ as queries to retrieve target
objects from $\bm{I}$:

$\displaystyle\\{\hat{\bm{b}}_{n}\\}_{n=1}^{{N}}=\mathcal{D}(\bm{I},\\{\bm{q}_{n}\\}_{n=1}^{N}),$
(2)

where${}_{\\!}$ $\bm{I}\\!=\\!\mathcal{F}(I)$,${}_{\\!}$
$\\{\bm{q}_{n}\\}_{n=1\\!}^{N}\\!=\\!\kappa(\bm{I})$,${}_{\\!}$ and${}_{\\!}$
$\mathcal{D}_{\\!}$ refers${}_{\\!}$ to${}_{\\!}$ the${}_{\\!}$ Trans-
former${}_{\\!}$ decoder${}_{\\!}$ ${}_{\\!}$ which${}_{\\!}$
formulates${}_{\\!}$ the${}_{\\!}$ object-query${}_{\\!}$ match- ing${}_{\\!}$
process${}_{\\!}$ via${}_{\\!}$ neural${}_{\\!}$ cross-attention${}_{\\!}$
computation.${}_{\\!}$ $\\{\hat{\bm{b}}_{n}\\}_{n=1}^{{N}}$ are${}_{\\!}$
the${}_{\\!}$ set${}_{\\!}$ of${}_{\\!}$ predicted${}_{\\!}$
parameterized${}_{\\!}$ object${}_{\\!}$ bounding${}_{\\!}$ boxes.${}_{\\!}$

Our rotation equivariant training further encourages robust object-query
matching which is equivariant against ro- tations.${}_{\\!}$ First,${}_{\\!}$
given${}_{\\!}$ an${}_{\\!}$ input${}_{\\!}$ rotation${}_{\\!}$
transformation${}_{\\!}$ $g^{r}$,${}_{\\!}$ a${}_{\\!}$ ro- bust fisheye
detector is desired to be able to extract rotation-equivariant representation.
Analogous to Eq. 1, we have:

$\displaystyle\mathcal{F}(g^{r}({I}))\approx g(\mathcal{F}(I))=g^{r}(\bm{I}).$
(3)

$g^{r\\!}({I})$$I$$\kappa$$\mathcal{F}$$\mathcal{F}$$\kappa$$\\{\bm{q}_{n}^{g^{r}\\!}\\}_{n=1}^{N}$$g^{r}(\bm{I})$$\bm{I}$$\\{\bm{q}_{n}\\}_{n=1}^{N}$$\mathcal{D}$$\mathcal{D}$$\mathcal{L}_{\text{det}}$

$\mathcal{L}_{\text{rotat-equi}}$

$\\{\bm{b}_{\ell(n)}\\}_{n=1}^{{N}}$$\\{\bm{b}^{g^{r}}_{\ell(n)}\\}_{n=1}^{{N}}$

Figure 6: Our rotation equivariant training strategy for query-based${}_{\\!}$
fisheye${}_{\\!}$ person${}_{\\!}$ detection${}_{\\!}$ (§4.1).${}_{\\!}$
For${}_{\\!}$ equivariant${}_{\\!}$ training,${}_{\\!}$ object queries
$\\{\bm{q}_{n}^{g^{r}\\!}\\}_{n=1}^{N}$ created for rotated image
$g^{r\\!}(I)$ and the rotated representation $g^{r\\!}(\bm{I})$ are fed into
the decoder $\mathcal{D}$; the rotated ground-truths
$\\{\bm{b}^{g^{r}}_{\ell(n)}\\}_{n=1}^{{N}}$ are set as the training targets.

Also, it is reasonable to assume that the final output of a robust fisheye
detector should be changed in the same way to the rotation $g^{r}$ applied to
the input fisheye image $I$:

$\displaystyle\\{\bm{b}^{g^{r}}_{\ell(n)}\\}_{n=1}^{{N}}\approx\mathcal{D}(\bm{I}^{g^{r}},\\{\bm{q}_{n}^{g^{r}}\\}_{n=1}^{N}),$
(4)

where $\bm{I}^{g^{r}\\!\\!}$ indicates${}_{\\!}$ the${}_{\\!}$
feature${}_{\\!}$ of${}_{\\!}$ rotated${}_{\\!}$ image${}_{\\!}$
$g^{r}(I)$,${}_{\\!}$  _i.e_.,
$\bm{I}^{g^{r}\\!\\!\\!}\\!=_{\\!}\\!\mathcal{F}(g^{r\\!}({I}))$.${}_{\\!}$
Similarly,${}_{\\!}$ $\\{\bm{q}_{n}^{g^{r}\\!}\\}_{n=1\\!\\!}^{N}$
denote${}_{\\!}$ the${}_{\\!}$ object${}_{\\!}$ queries${}_{\\!}$ ${}_{\\!}$
derived${}_{\\!}$ from${}_{\\!}$ $g^{r\\!}(I)$,${}_{\\!}$ _i.e_.,${}_{\\!}$
$\\{\bm{q}_{n}^{g^{r}\\!}\\}_{n=1\\!}^{N}\\!=\\!\\!\kappa(\bm{I}^{g^{r}\\!})$.${}_{\\!}$
$\\{\bm{b}^{g^{r}}_{\ell(n)}\\}_{n=1}^{{N}}$ are the rotated groundtruth
bounding boxes, where $\ell(n)$ returns the groundtruth index for $n$-th
query.

Considering Eq. 3 and Eq. 4, we can have:
$\\{\bm{b}^{g^{r}}_{\ell(n)}\\}_{n=1}^{{N}}\approx\mathcal{D}(g^{r\\!}(\bm{I}),\\{\bm{q}_{n}^{g^{r}\\!}\\}_{n=1}^{N})$.${}_{\\!}$
Hence${}_{\\!}$ our${}_{\\!}$ rotation${}_{\\!}$ equivariant${}_{\\!}$
training${}_{\\!}$ objective${}_{\\!}$ is given as:

$\displaystyle\mathcal{L}_{\text{rotat-
equi}}=\mathcal{L}_{\text{det}}\big{(}\\{\bm{b}^{g^{r}}_{\ell(n)}\\}_{n=1}^{{N}},\mathcal{D}(g^{r\\!}(\bm{I}),\\{\bm{q}_{n}^{g^{r}\\!}\\}_{n=1}^{N})\big{)}.$
(5)

Here${}_{\\!}$ $\mathcal{L}_{\text{det\\!}}$ is${}_{\\!}$ the${}_{\\!}$
standard${}_{\\!}$ detection${}_{\\!}$ loss${}_{\\!}$ in${}_{\\!}$
DETR${}_{\\!}$ [9].${}_{\\!}$ As${}_{\\!}$ such,${}_{\\!}$ the rotation
equivariance properties for both fisheye representation $\mathcal{F}$ (cf.
Eq. 3) and the query-based detection prediction
$\mathcal{D}(\bm{I},\\{\bm{q}_{n}\\}_{n=1}^{N})$ (cf. Eq. 4) are sought in a
single training target. This also allows us to effortlessly adapt standard
DETR${}_{\\!}$ to${}_{\\!}$ our${}_{\\!}$ omnidirectional${}_{\\!}$
detection${}_{\\!}$ setting.${}_{\\!}$ Note${}_{\\!}$ that${}_{\\!}$ our
rotation${}_{\\!}$ equivariant${}_{\\!}$ training${}_{\\!}$ differs${}_{\\!}$
from${}_{\\!}$ rotational${}_{\\!}$ data${}_{\\!}$ aug- mentation${}_{\\!}$
technique${}_{\\!}$ which${}_{\\!}$ only${}_{\\!}$ views${}_{\\!}$
rotated${}_{\\!}$ images${}_{\\!}$ as${}_{\\!}$ in- dividual${}_{\\!}$
training${}_{\\!}$ samples${}_{\\!}$ (see${}_{\\!}$ §5.4${}_{\\!}$
for${}_{\\!}$ detailed${}_{\\!}$ experiments).${}_{\\!}$

###  4.2 2D-3D Projection based Person Localization

General Fisheye Camera Model. The perspective projection of a normal pinhole
camera can be written as $r=f\tan\theta$, where $r$ indicates the projection
distance between the principal point and the image point, $f$ is the focal
length, and $\theta$ is the angle between the incident ray and the camera’s
optical axis. However, fisheye lens does not follow this perspective
projection model [48], as the FOV equals to 180∘ (cf. Fig. 7). Fortunately,
the image formation of different kinds of fisheye lenses can be approximated
by a general polynomial projection model [36], _i.e_.,

$\displaystyle
r(\theta)=\sum^{n}\nolimits_{i=1}k_{i}\theta^{2i-1},~{}~{}~{}~{}n=1,2,3,4,\cdots$
(6)

High distortions can be handled well when $n=5$ [36]. The coefficients $k$s
can be obtained from camera calibration.

| val | test |
---|---|---|---
Method | mAP $\uparrow$ | AP50$\uparrow$ | AP75$\uparrow$ | APn$\uparrow$ | APm$\uparrow$ | APf$\uparrow$ | APseen$\uparrow$ | APunseen$\uparrow$ | mAP $\uparrow$ | AP50$\uparrow$ | AP75$\uparrow$ | APn$\uparrow$ | APm$\uparrow$ | APf$\uparrow$ | APseen$\uparrow$ | APunseen$\uparrow$ | FPS $\uparrow$
Seide _et al_. [60] | 21.8 | 59.8 | 7.6 | 32.8 | 28.5 | 2.3 | 23.2 | 19.5 | 20.2 | 58.2 | 7.1 | 32.2 | 28.3 | 3.9 | 22.4 | 18.9 | 10.2
Li _et al_. [41] | 28.5 | 63.3 | 20.1 | 46.8 | 24.2 | 1.3 | 33.8 | 29.3 | 27.2 | 65.2 | 21.3 | 47.2 | 24.8 | 1.3 | 31.8 | 27.6 | 0.6
Tamura _et al_. [64] | 34.8 | 72.1 | 27.7 | 51.7 | 38.8 | 8.7 | 38.8 | 32.9 | 34.2 | 72.8 | 28.7 | 53.7 | 37.3 | 6.5 | 39.5 | 33.2 | 10.2
RAPiD [24] | 40.3 | 77.9 | 34.8 | 55.3 | 41.9 | 9.2 | 44.7 | 37.6 | 39.2 | 77.9 | 35.4 | 54.8 | 40.1 | 7.9 | 44.2 | 37.3 | 8.4
Ours | 47.2 | 82.3 | 48.2 | 63.8 | 54.1 | 14.0 | 50.6 | 45.5 | 46.2 | 81.1 | 47.3 | 66.1 | 53.5 | 12.6 | 49.3 | 44.9 | 12.1

Table 2: Person detection results on val and test sets of LOAF (§5.2).

$\phi$

Figure 7: ${}_{\\!}$Our fisheye camera model${}_{\\!}$ based
person${}_{\\!\\!}$ localization (§4.2).${}_{\\!}$ Here${}_{\\!}$
$\bm{p}_{\\!}$ and${}_{\\!}$ $\bm{p}^{\prime}_{\\!}$ denote${}_{\\!}$
the${}_{\\!}$ projection${}_{\\!}$ point${}_{\\!}$ of${}_{\\!}$ the
distortion${}_{\\!}$ point${}_{\\!}$ $\bm{P}_{\\!}$ by${}_{\\!}$
fisheye${}_{\\!}$ lens and${}_{\\!}$ normal${}_{\\!}$ perspective${}_{\\!}$
lens, respectively.

2D-3D Projection. Given a detected human location point
$\bm{p}\\!=\\!(u,v)^{\top}$ in the fisheye image pixel coordinate system, our
target is to calculate its 3D location $\bm{P}=(X,Y,Z)^{\top}$ in the camera
coordinate system, where $Z$ is the altitude of the fisheye camera and priorly
known. To do so, one can first calculate $r$ as:
$r\\!=\\!\sqrt{(x^{2}+y^{2})}$, where $x\\!=\\!(u\\!-\\!u_{0})/f$ and
$y\\!=\\!(v\\!-\\!v_{0})/f$, and $(u_{0},v_{0})^{\top}$ are the coordinates of
the principal point in the fisheye image. Then $\theta$ can be obtained by
solving Eq. 6 using a numerical means [36]. With the altitude $Z$ of the
camera, the actual location $(X,Y)^{\top\\!}$ is given as:

$\displaystyle\binom{X}{Y}=Z\binom{\tan\theta\cos\phi}{\tan\theta\sin\phi},$
(7)

where $\phi=\arctan(y,x)$ refers to the polar angle, which is shared by both
$\bm{p}$ and $\bm{P}$.

It can be seen that precise determination of the person location point
$\bm{p}$ on the image plane is vital for estimating the corresponding physical
position $\bm{P}$. However, most previous approaches, restricted to axis-
aligned [79, 60] or person-aligned [24] human-box representation, use the
center of the detection box to approximate $\bm{p}$. Few exceptions [21],
built on point-based human representation, treat the center of the detected
human head as $\bm{p}$. However, they suffer from a similar issue as the human
head center is even often far from the exact position $\bm{p}$ human stand on,
w.r.t. the image plane. Instead, with our radius-aligned human-box
representation, the closest point to the principal point on the bounding box
better corresponds to $\bm{p}$.

###  4.3 Implementation Details

Network Architecture. Our fisheye person detector is built upon DAB-
DETR${}_{\\!}$ [44], a prevalent variant of DETR${}_{\\!}$ [9] but converges
much faster. Swin-T [45] is${}_{\\!}$ utilized${}_{\\!}$ as${}_{\\!}$
the${}_{\\!}$ backbone. For our LOAF with radius-aligned person boxes, the
output of our detector is a 4D vector $\hat{\bm{b}}\\!\in\\![0,1]^{4}$ that
parameterizes 2D center coordinates, height, and width. Note that, for
traditional fisheye detection datasets like [24, 65] with arbitrary-oriented
person box annotations, an extra output dimension is needed for rotation angle
regression.

Training Objective. Our fisheye person detector is end-to-
end${}_{\\!}$ trained${}_{\\!}$ by${}_{\\!}$ jointly${}_{\\!}$
optimizing${}_{\\!}$ the${}_{\\!}$ vanilla${}_{\\!}$ detection${}_{\\!}$ loss
used in DETR [9, 44] (referred as $\mathcal{L}_{\text{Det}}$) and our proposed
rotation-equivariant constraint (_i.e_., $\mathcal{L}_{\text{rotat-equi}}$ in
Eq. 5):

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{det}}+\lambda\mathcal{L}_{\text{rotat-
equi}},$ (8)

where the coefficient $\lambda$ is empirically set to 0.5.

Method | mAP $\uparrow$ | AP50 $\uparrow$ | AP75 $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | F-Score $\uparrow$
---|---|---|---|---|---|---
Seide _et al_. [60] | 20.9 | 50.6 | 10.2 | 80.6 | 39.5 | 53.0
Li _et al_. [41] | 34.2 | 75.7 | 28.6 | 86.3 | 65.4 | 74.4
Tamura _et al_. [64] | 29.3 | 61.0 | 23.4 | 88.8 | 51.2 | 65.0
RAPiD [24] | 39.3 | 85.4 | 26.0 | 89.2 | 78.7 | 83.6
Ours | 46.8 | 88.1 | 36.8 | 90.2 | 87.4 | 88.6

Table 3: Person detection results on CEPDOF [24] (§5.2).

Training.${}_{\\!}$ Our${}_{\\!}$ fisheye${}_{\\!}$ detector${}_{\\!}$
is${}_{\\!}$ trained${}_{\\!}$ with${}_{\\!}$ a${}_{\\!}$ batch${}_{\\!}$
size${}_{\\!}$ of${}_{\\!}$ 8${}_{\\!}$ for${}_{\\!}$ 50${}_{\\!}$
epochs,${}_{\\!}$ where${}_{\\!}$ the${}_{\\!}$ AdamW [46]${}_{\\!}$
optimizer${}_{\\!}$ is${}_{\\!}$ employed${}_{\\!}$ with${}_{\\!}$ base
learning${}_{\\!}$ rate${}_{\\!}$ 2e-4${}_{\\!}$ and${}_{\\!}$
decayed${}_{\\!}$ by${}_{\\!}$ 0.1${}_{\\!}$ at${}_{\\!}$ epoch${}_{\\!}$ 40\.
The remaining hyper-parameters are determined${}_{\\!}$ following${}_{\\!}$
[64, 24].${}_{\\!}$ Specifically,${}_{\\!}$ we${}_{\\!}$ initialize${}_{\\!}$
backbones${}_{\\!}$ with${}_{\\!}$ Image- Net [22]${}_{\\!}$ pre-trained
weights and adopt standard data augmentation techniques, _i.e_., color jitter,
horizontal flip, and random scaling, with a base training size of${}_{\\!}$
608$\times$608.${}_{\\!}$ For${}_{\\!}$ CEPDOF${}_{\\!}$ [24]${}_{\\!}$
and${}_{\\!}$ WEPDTOF${}_{\\!}$ [65],${}_{\\!}$ we${}_{\\!}$ use${}_{\\!}$
a${}_{\\!}$ batch${}_{\\!}$ size${}_{\\!}$ of${}_{\\!}$ 128${}_{\\!}$ and pre-
train the detector on COCO [43] for 50 epochs to prevent over-fitting, as
in${}_{\\!}$ [24, 65]. For the computation of our rotation-equivariant loss
$\mathcal{L}_{\text{rotat-equi}}$, the training images are rotated by a degree
randomly sampled from 0 to 360. Our${}_{\\!}$ detector${}_{\\!}$ is${}_{\\!}$
implemented${}_{\\!}$ in${}_{\\!}$ PyTorch${}_{\\!}$ and${}_{\\!}$
trained${}_{\\!}$ on${}_{\\!}$ eight

NVIDIA Tesla V100 GPUs with a 32GB memory per-card.

Inference.${}_{\\!}$ Once${}_{\\!}$ trained,${}_{\\!}$ our${}_{\\!}$
fisheye${}_{\\!}$ detector${}_{\\!}$ can${}_{\\!}$ be${}_{\\!}$ directly
applied for locating persons on the omnidirectional image plane. After that,
the physical locations of the detected persons can be easily obtained through
the numerical solution described in §4.2. For fisheye person detection, we
follow prior work [60, 41, 64, 24] to use 1024$\times$1024 input resolution
without any test-time augmentation or post-processing. Testing is conducted on
a single NVIDIA V100 GPU with 16 GB memory.

##  5 Experiment

###  5.1 Experimental Setup

Evaluation Protocol. On the top of LOAF, we conduct ex- periments for fisheye
based person detection and localization. Following the dataset splitting (cf.
§3.3), performance are reported on seen and unseen scenes, respectively, for
both val and test sets. This allows us to assess the generalization ability
over different surveillance scenarios. For comprehensive study, we further
report person detection performance on two existing fisheye datasets, CEPDOF
[24] and WEPDTOF [65]. Note that localization cannot be test since${}_{\\!}$
[24, 65]${}_{\\!}$ only${}_{\\!}$ provide${}_{\\!}$ person${}_{\\!}$
bounding${}_{\\!}$ box${}_{\\!}$ annotations.

Method | mAP $\uparrow$ | AP50 $\uparrow$ | AP75 $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | F-Score $\uparrow$
---|---|---|---|---|---|---
Seide _et al_. [60] | 16.1 | 39.4 | 9.0 | 70.9 | 38.6 | 50.0
Li _et al_. [41] | 25.2 | 69.9 | 30.2 | 81.4 | 64.5 | 72.0
Tamura _et al_. [64] | 28.8 | 59.8 | 24.2 | 77.0 | 52.4 | 62.4
RAPiD [24] | 37.7 | 72.0 | 26.8 | 73.3 | 67.8 | 70.4
Ours | 45.4 | 85.1 | 36.2 | 84.7 | 74.4 | 79.5

Table 4: Person detection results on WEPDTOF [65] (§5.2).

| val | test
---|---|---
Method | mPE $\downarrow$ | PEn $\downarrow$ | PEm $\downarrow$ | PEf $\downarrow$ | PEseen $\downarrow$ | PEunseen $\downarrow$ | mPE $\downarrow$ | PEn $\downarrow$ | PEm $\downarrow$ | PEf $\downarrow$ | PEseen $\downarrow$ | PEunseen $\downarrow$
Seide _et al_. [60] | 1.298 | 0.561 | 1.332 | 3.109 | 1.206 | 1.382 | 1.321 | 0.706 | 1.309 | 3.482 | 1.306 | 1.386
Li _et al_. [41] | 0.898 | 0.502 | 0.871 | 2.650 | 0.832 | 0.962 | 0.913 | 0.543 | 0.884 | 2.780 | 0.904 | 0.998
Tamura _et al_. [64] | 0.755 | 0.429 | 0.736 | 1.862 | 0.709 | 0.821 | 0.778 | 0.471 | 0.826 | 2.160 | 0.724 | 0.836
RAPiD [24] | 0.674 | 0.426 | 0.623 | 1.403 | 0.625 | 0.757 | 0.682 | 0.461 | 0.664 | 1.445 | 0.638 | 0.776
Ours | 0.387 | 0.164 | 0.382 | 0.786 | 0.332 | 0.419 | 0.392 | 0.171 | 0.391 | 0.825 | 0.343 | 0.413

Table 5: Person localization results on val and test sets of LOAF (§5.3).

Evaluation Metrics. For fisheye person detection, we follow COCO [43] to
report the mean average precision (mAP) for IoU $\in$ [0.5: 0.05: 0.95].
We also employ AP50 and AP75 for${}_{\\!}$ further${}_{\\!}$
analysis.${}_{\\!}$ For${}_{\\!}$ fisheye${}_{\\!}$ person${}_{\\!}$
localization,${}_{\\!}$ we mea- sure positional error (PE) in meters, _i.e_.,
Euclidean distance of calculated position and ground-truth position, as the
conventions in visual localization [42, 58, 73, 79]. For detailed evaluation,
on our LOAF, we report performance w.r.t. horizontal distance between human
and the fisheye lens, _i.e_., near (0$\sim$10 m), middle${}_{\\!}$ (10$\sim$20
m),${}_{\\!}$ and${}_{\\!}$ far (larger${}_{\\!}$ than${}_{\\!}$ 20 m). Hence
we have AP{n,m,f} and PE{n,m,f} accordingly.

###  5.2 Performance on Person Detection

LOAF. We first report the person detection performance on LOAF.${}_{\\!}$
Specifically,${}_{\\!}$ four${}_{\\!}$ recent${}_{\\!}$ deep${}_{\\!}$
learning${}_{\\!}$ based${}_{\\!}$ fisheye person detectors [60, 41, 64, 24]
are involved for compari- son.${}_{\\!}$ Among${}_{\\!}$ them,${}_{\\!}$ [60,
41]${}_{\\!}$ are${}_{\\!}$ training-free,${}_{\\!}$ thus${}_{\\!}$
their${}_{\\!}$ scores are obtained by directly running the algorithms on the
val and test data. As [64, 24] are trainable methods, we first train them on
the train set of LOAF following their offi- cial setups, and then report the
scores on the val and test sets. Quantitative comparison results are presented
in Table 2. Some essential observations are as follows:

  * •

Our fisheye person detector significantly outperforms existing methods across
all the metrics. For example, our detector provides a considerable performance
gain in mAP, _i.e_., 6.9% and 7.0% higher than the second best, RAPiD [24], on
the val and test sets, respectively.

  * •

Our detector not only handles nearby persons but also approaches distant
targets well. Specifically, for the targets at 10$\sim$20 m away from the
fisheye lens, our performance gain over other methods is more significant than
the improvement reported with the persons at 0$\sim$10 m distance.

  * •

Our detector also yields small performance gap between the seen and unseen
scenes (_e.g_., 49.3 $\rightarrow$ 44.9 in terms of mAP), validating our good
generalizability.

  * •

For those persons at very far distance ($>$20 m), our algorithm, though still
giving higher scores than other competitors, suffers from great performance
degradation. This sheds light on the direction of our future efforts.

CEPDOF${}_{\\!}$ [24].${}_{\\!}$ CEPDOF${}_{\\!}$ contains${}_{\\!}$
eight${}_{\\!}$ videos${}_{\\!}$ with${}_{\\!}$ 25,504 frames${}_{\\!}$
in${}_{\\!}$ total.${}_{\\!}$ Following${}_{\\!}$ the${}_{\\!}$
official${}_{\\!}$ setup,${}_{\\!}$ we${}_{\\!}$ train${}_{\\!}$ our
fisheye${}_{\\!}$ person${}_{\\!}$ detector${}_{\\!}$ on${}_{\\!}$
HABBOF${}_{\\!}$ [41]${}_{\\!}$ and${}_{\\!}$ MW-R${}_{\\!}$ [24]
datasets,${}_{\\!}$ and${}_{\\!}$ report${}_{\\!}$ performance${}_{\\!}$
on${}_{\\!}$ CEPDOF.${}_{\\!}$ As${}_{\\!}$ shown${}_{\\!}$ in${}_{\\!}$
Table${}_{\\!}$ 3, our fisheye detector delivers state-of-the-art performance:
it significantly outperforms RAPiD [24], the current top-leading algorithm,
by 7.5% in terms of mAP.

WEPDTOF${}_{\\!}$ [65].${}_{\\!}$ WEPDTOF has 16 videos with 10,544 frames in
total. Following the official setup, we use HABBOF [41], MW-R [24], and
CEPDOF${}_{\\!}$ [24] for training and WEPDTOF for testing. Table 4 summarizes
the results. Impressively, our detector greatly suppresses all the other
competitors across all the evaluation metrics.

###  5.3 Performance on Person Localization

Then we study the person localization performance on LOAF. None of previous
fisheye person detectors [60, 41, 64, 24] are aware of the task of person
localization. We therefore follow the common practice [79, 21] in the field of
visual localization: for [60, 41, 64, 24], we project centers of their
detected human boxes on the 3D world, based on the same strategy of ours (cf.
§4.2), for physical position estimation. As shown in Table 5, our system
produces much small localization errors in comparison with [60, 41, 64, 24],
_i.e_., 0.392 m vs 0.682 m [24] and 0.773 m of [64] on the test set of LOAF.
In addition, for all the methods, the localization performance is declined as
target distance increases, but our system suffers from the smallest drop.
Moreover, the results suggest there is still large room for improvement, thus
we hope that our dataset could encourage continuous efforts in this
challenging task.

|  |  val seen |  val unseen
---|---|---|---
Method | mAP $\uparrow$ | mAP $\uparrow$ | AP50 $\uparrow$ | AP75 $\uparrow$ | mAP $\uparrow$ | AP50 $\uparrow$ | AP75 $\uparrow$
Baseline [44] | 43.1 | 46.4 | 81.2 | 48.4 | 40.9 | 76.2 | 38.9
\+ Rotation Aug. | 45.6 | 49.2 | 82.8 | 52.8 | 43.2 | 78.9 | 43.9
\+ Rotation Equ. | 47.2 | 50.6 | 83.7 | 54.6 | 45.5 | 80.4 | 47.1

Table 6: Diagnostic results on val set of LOAF (§5.4).

###  5.4 Diagnostic Study

Table 6 studies the efficacy of our rotation equivariant training strategy
(§4.1), on the val set of LOAF. Our baseline model [44] (row #1) gains 43.1%
mAP. After adopting rotational data${}_{\\!}$ augmentation${}_{\\!}$ (row #2),
the${}_{\\!}$ performance${}_{\\!}$ boosts${}_{\\!}$ by 2.5%${}_{\\!}$ mAP. By
contrast, our rotation equivariant training brings much larger improvements
over the baseline, _e.g_., 4.1% mAP gain. It is remarkable that, training with
standard rotational data augmentation technique can be viewed as a specific
case of our equivariant training – only learning rotation-equivariant object-
querying (cf. Eq. 4).

##  6 Conclusion

We presented LOAF – the first top-view fisheye dataset that supports large-
scale study for person localization in realistic surveillance scenarios. With
radius-aligned person-box annotations and precise location ground-truths, LOAF
closes a crucial gap in the literature as these cases are not covered by
previous datasets and annotation protocols. We further proposed an efficient
fisheye person detection model that is equipped with a rotation-equivariant
training strategy. The physical locations of detected persons are formulated
based on the fisheye model and the altitude of the camera. We empirically
verified the effectiveness and promising performance of our algorithm.

| # Video | # Image | # People | Scene | Season | Time
---|---|---|---|---|---|---
| Total | Avg. | Max | Total | Indoor | Outdoor | Spring | Summer | Autumn | Morning | Noon | Afternoon
train | 51 | 29,569 | 315,262 | 10.6 | 65 | 35 | 7 | 28 | 13 | 30 | 8 | 10 | 15 | 26
val | seen | 3 | 1,700 | 18,460 | 10.8 | 29 | 3 | 1 | 2 | 1 | 2 | 0 | 1 | 1 | 1
unseen | 5 | 2,900 | 29,381 | 10.1 | 44 | 5 | 2 | 3 | 1 | 3 | 1 | 1 | 2 | 2
total | 8 | 4,600 | 47,841 | 10.4 | 44 | 8 | 3 | 5 | 2 | 5 | 1 | 2 | 3 | 3
test | seen | 5 | 2,774 | 28,666 | 10.3 | 41 | 5 | 2 | 3 | 1 | 3 | 1 | 1 | 2 | 2
unseen | 10 | 5,999 | 65,993 | 10.0 | 44 | 10 | 2 | 8 | 3 | 5 | 2 | 3 | 3 | 4
total | 15 | 8,773 | 94,659 | 10.1 | 44 | 15 | 4 | 11 | 4 | 8 | 3 | 4 | 5 | 6
Total | 74 | 42,942 | 457,762 | 10.5 | 65 | 50 | 11 | 39 | 17 | 38 | 11 | 14 | 20 | 32

Table S1: Detailed statistics of LOAF. # indicates the number of elements.

This document provides additional materials to supplement our main manuscript.
We first present more statistics about LOAF in §A, and then give extra
implementation details of our method in §B. More qualitative results on the
test set of LOAF are summarized in §C. Next, we state the ethical conducts in
§D. Finally, we provide the pseudo of our proposed rotation equivariant
training strategy in §E.

##  Appendix A Additional Dataset Analysis

More Statistics. LOAF is captured from multiple indoor/ outdoor scenes
(_e.g_., library, classroom, street, parking lot) across three seasons, we
summarize the detailed statistics in Table S1, including the number of boxes,
video sequences, _etc_. As seen, the majority of videos are collected from
outdoor environments characterized by increased complexity, larger fields of
view, and a higher number of human targets when compared to the indoor ones.
These videos are divided into train, val, and test sets in the ratio of 7:1:2
respectively, while ensuring an roughly even distribution of attributes
(_e.g_., season, time) across these sets.

##  Appendix B More Implementation Details

Training${}_{\\!}$ Objective.${}_{\\!}$ We${}_{\\!}$ extend${}_{\\!}$
the${}_{\\!}$ Generalized${}_{\\!}$ IoU (GIoU) loss [53] utilized in vanilla
DETR [9] for bounding box regression to the rotated setup. Concretely, Brute-
force search is leveraged to compute the minimum enclosing box between two
rotated bounding boxes. It is implemented in a fully differentiable manner and
adapted for parallel processing on GPU, which merely defers the training speed
by around 5% when compared to the axis-aligned setup.

##  Appendix C Qualitative Evaluation

Visual Comparison. Fig. S1-S5 compare our method with existing work
qualitatively. It is obvious that our proposed method consistently presents
more accurate detection and localization results, regardless of the category
of scenes. Notably, it is much more effective than existing work for targets
that are relatively small or densely arranged.

Diversity. To render a more intuitive understanding of the diversity of LOAF,
a collage constituted from various scenes characterized by distinct attributes
is given in Fig. S6.

##  Appendix D Ethical Conducts

To protect the privacy of individuals and groups, we utilize Gaussian filters
to blur all visible facial regions in LOAF. The proprietary data can only be
accessed for non-commercial purposes to prevent inappropriate usage.

##  Appendix E Pseudo Code

We offer the pseudo code for our proposed query-based rotation equivariant
training strategy in Algorithm S1.

Algorithm S1 Pseudo-code for our proposed rotation equivariant training
strategy.

⬇

"""

I:␣input␣image

gt:␣ground␣truth

angle:␣degree␣of␣clockwise␣rotation

$\lambda$:␣the␣balance␣factor

"""

def rotat_equi_training(I, gt):

# $\mathcal{F}(I)$

m1 = Encoder(I)

angle = randint(0, 360)

# $\mathcal{F}(g^{r}(I))$

m2 = Encoder(rotate(I, angle))

# $\\{\bm{q}_{n}\\}_{n=1\\!}^{N}\\!=\\!\kappa(\bm{I})$

query1 = gen_proposal(m1)

#
$\\{\bm{q}_{n}^{g^{r}\\!}\\}_{n=1\\!}^{N}\\!=\\!\\!\kappa(\bm{I}^{g^{r}\\!})$

query2 = gen_proposal(m2)

# $\mathcal{D}(\bm{I},\\{\bm{q}_{n}\\}_{n=1}^{N})$

det1 = Decoder(m1, query1)

# $\mathcal{D}(g^{r\\!}(\bm{I}),\\{\bm{q}_{n}^{g^{r}\\!}\\}_{n=1}^{N})$

det2 = Decoder(rotate(m1, angle), query2)

#
$\mathcal{L}_{\text{det}}\big{(}\\{\bm{b}_{\ell(n)}\\}_{n=1}^{{N}},\mathcal{D}(\bm{I},\\{\bm{q}_{n}\\}_{n=1}^{N})\big{)}$

loss1 = det_loss(det1, gt)

#
$\mathcal{L}_{\text{det}}\big{(}\\{\bm{b}^{g^{r}}_{\ell(n)}\\}_{n=1}^{{N}},\mathcal{D}(g^{r\\!}(\bm{I}),\\{\bm{q}_{n}^{g^{r}\\!}\\}_{n=1}^{N})\big{)}$

loss2 = det_loss(det2, label_rotate(gt, angle)

return loss1 + $\lambda$*loss2

Seide _et al_. [60]

Li _et al_. [41]

Tamura _et al_. [64]

RAPiD [24]

Ours

Figure S1: Visual comparison of detection results on the test set of LOAF.
indicates targets missed by our method.

Seide _et al_. [60]

Li _et al_. [41]

Tamura _et al_. [64]

RAPiD [24]

Ours

Figure S2: Visual comparison of detection results on the test set of LOAF.
indicates targets missed by our method.

Seide _et al_. [60]

Li _et al_. [41]

Tamura _et al_. [64]

RAPiD [24]

Ours

Figure S3: Visual comparison of detection results on the test set of LOAF.
indicates targets missed by our method.

Seide _et al_. [60]

Li _et al_. [41]

Tamura _et al_. [64]

RAPiD [24]

Ours

Figure S4: Visual comparison of detection results on the test set of LOAF.
indicates targets missed by our method.

Seide _et al_. [60]

Li _et al_. [41]

Tamura _et al_. [64]

RAPiD [24]

Ours

Ground truth

Figure S5: Visual comparison of localization results on the test set of LOAF.
We selected three targets per frame for clear visualization. Figure S6: A
collage constituted from various scenes characterized by distinct attributes.

## References

  * [1] Abdulrahman Alarifi, AbdulMalik Al-Salman, Mansour Alsaleh, Ahmad Alnafessah, Suheer Al-Hadhrami, Mai A Al-Ammar, and Hend S Al-Khalifa.  Ultra wideband indoor positioning technologies: Analysis and recent advances.  Sensors, 16(5):707, 2016.
  * [2] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In CVPR, 2016.
  * [3] Vassileios Balntas, Shuda Li, and Victor Prisacariu.  Relocnet: Continuous metric learning relocalisation using neural nets.  In ECCV, 2018.
  * [4] Federico Boniardi, Abhinav Valada, Rohit Mohan, Tim Caselitz, and Wolfram Burgard.  Robot localization in floor plans using a room layout edge extraction network.  In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2019.
  * [5] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother.  Dsac-differentiable ransac for camera localization.  In CVPR, 2017.
  * [6] Eric Brachmann and Carsten Rother.  Expert sample consensus applied to camera re-localization.  In ICCV, 2019.
  * [7] Guillaume Bresson, Li Yu, Cyril Joly, and Fabien Moutarde.  Urban localization with street views using a convolutional neural network for end-to-end camera pose regression.  In IEEE Intelligent Vehicles Symposium, 2019.
  * [8] Jiale Cao, Yanwei Pang, Jin Xie, Fahad Shahbaz Khan, and Ling Shao.  From handcrafted to deep features for pedestrian detection: a survey.  IEEE TPAMI, 2021.
  * [9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [10] Bao Xin Chen and John K Tsotsos.  Fast visual object tracking with rotated bounding boxes.  arXiv preprint arXiv:1907.03892, 2019.
  * [11] David M Chen, Georges Baatz, Kevin Köser, Sam S Tsai, Ramakrishna Vedantham, Timo Pylvänäinen, Kimmo Roimela, Xin Chen, Jeff Bach, Marc Pollefeys, et al.  City-scale landmark identification on mobile devices.  In CVPR, 2011.
  * [12] Long Chen, Shaobo Lin, Xiankai Lu, Dongpu Cao, Hangbin Wu, Chi Guo, Chun Liu, and Fei-Yue Wang.  Deep neural network based vehicle and pedestrian detection for autonomous driving: A survey.  IEEE TITS, 2021.
  * [13] Liang Chen, Ling Pei, Heidi Kuusniemi, Yuwei Chen, Tuomo Kröger, and Ruizhi Chen.  Bayesian fusion for indoor positioning using bluetooth fingerprints.  Wireless Personal Communications, 2013.
  * [14] An-Ti Chiang and Yao Wang.  Human detection in fish-eye images using hog-based detectors over rotated windows.  In ICME Workshop, 2014.
  * [15] Sheng-Ho Chiang, Tsaipei Wang, and Yi-Fu Chen.  Efficient pedestrian detection in top-view fisheye images using compositions of perspective view patches.  Image and Vision Computing, 2021.
  * [16] Shin-Fang Ch’ng, Naoya Sogi, Pulak Purkait, Tat-Jun Chin, and Kazuhiro Fukui.  Resolving marker pose ambiguity by robust rotation averaging with clique constraints.  In ICRA, 2020.
  * [17] Ibrahim Cinaroglu and Yalin Bastanlar.  A direct approach for human detection with catadioptric omnidirectional cameras.  In Signal Processing and Communications Applications Conference, 2014.
  * [18] Taco Cohen and Max Welling.  Group equivariant convolutional networks.  In ICLR, 2016.
  * [19] Creative Commons.  Attribution-noncommercial-sharealike 4.0 international.  https://creativecommons.org/licenses/by-nc-sa/4.0/.
  * [20] Joseph DeGol, Timothy Bretl, and Derek Hoiem.  Chromatag: A colored marker and fast detection algorithm.  In ICCV, 2017.
  * [21] Carlos R del Blanco, Pablo Carballeira, Fernando Jaureguizar, and Narciso García.  Robust people indoor localization with omnidirectional cameras using a grid of spatial-aware classifiers.  Signal Processing: Image Communication, 2021.
  * [22] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [23] Mingyu Ding, Zhe Wang, Jiankai Sun, Jianping Shi, and Ping Luo.  Camnet: Coarse-to-fine retrieval for camera re-localization.  In ICCV, 2019.
  * [24] Zhihao Duan, Ozan Tezcan, Hayato Nakamura, Prakash Ishwar, and Janusz Konrad.  Rapid: rotation-aware people detection in overhead fisheye images.  In CVPR Workshop, 2020.
  * [25] Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis.  Learning SO(3) equivariant representations with spherical cnns.  In ECCV, 2018.
  * [26] Zahid Farid, Rosdiadee Nordin, and Mahamod Ismail.  Recent advances in wireless indoor localization techniques and system.  Journal of Computer Networks and Communications, 2013.
  * [27] Mark Fiala.  Artag, a fiducial marker system using digital techniques.  In CVPR, 2005.
  * [28] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [29] Hugo Germain, Vincent Lepetit, and Guillaume Bourmaud.  Neural reprojection error: Merging feature learning and camera pose estimation.  In CVPR, 2021.
  * [30] Yanying Gu, Anthony Lo, and Ignas Niemegeers.  A survey of indoor positioning systems for wireless personal networks.  IEEE Communications surveys & tutorials, 2009.
  * [31] Janne Haverinen and Anssi Kemppainen.  Global indoor self-localization based on the ambient magnetic field.  Robotics and Autonomous Systems, 2009.
  * [32] James Hays and Alexei A Efros.  Im2gps: estimating geographic information from a single image.  In CVPR, 2008.
  * [33] Joel A Hesch and Stergios I Roumeliotis.  A direct least-squares (dls) method for pnp.  In ICCV, 2011.
  * [34] Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang.  Transforming auto-encoders.  In ICANN, 2011.
  * [35] Danying Hu, Daniel DeTone, and Tomasz Malisiewicz.  Deep charuco: Dark charuco marker pose estimation.  In CVPR, 2019.
  * [36] Juho Kannala and Sami S Brandt.  A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses.  IEEE TPAMI, 28(8):1335–1340, 2006.
  * [37] Alex Kendall and Roberto Cipolla.  Geometric loss functions for camera pose regression with deep learning.  In CVPR, 2017.
  * [38] Alex Kendall, Matthew Grimes, and Roberto Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In ICCV, 2015.
  * [39] Oded Krams and Nahum Kiryati.  People detection in top-view fisheye imaging.  In IEEE International Conference on Advanced Video and Signal based Surveillance, 2017.
  * [40] Karel Lenc and Andrea Vedaldi.  Understanding image representations by measuring their equivariance and equivalence.  In CVPR, 2015.
  * [41] Shengye Li, M Ozan Tezcan, Prakash Ishwar, and Janusz Konrad.  Supervised people counting using an overhead fisheye camera.  In IEEE International Conference on Advanced Video and Signal Based Surveillance, 2019.
  * [42] Yunpeng Li, Noah Snavely, Dan Huttenlocher, and Pascal Fua.  Worldwide pose estimation using 3d point clouds.  In ECCV, 2012.
  * [43] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [44] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  Dab-detr: Dynamic anchor boxes are better queries for detr.  In ICLR, 2022.
  * [45] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [46] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [47] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman.  1 year, 1000 km: The oxford robotcar dataset.  The International Journal of Robotics Research, 36(1):3–15, 2017\.
  * [48] Kenro Miyamoto.  Fish eye lens.  JOSA, 54(8):1060–1061, 1964.
  * [49] Jun Qi and Guo-Ping Liu.  A robust high-accuracy ultrasound indoor positioning system based on a wireless sensor network.  Sensors, 17(11):2554, 2017.
  * [50] Hazem Rashed, Eslam Mohamed, Ganesh Sistu, Varun Ravi Kumar, Ciaran Eising, Ahmad El-Sallab, and Senthil Yogamani.  Generalized object detection on fisheye cameras for autonomous driving: Dataset, representations and baseline.  In WACV, 2021.
  * [51] Joseph Redmon and Ali Farhadi.  Yolo9000: better, faster, stronger.  In CVPR, 2017.
  * [52] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [53] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union.  2019\.
  * [54] Samer S Saab and Zahi S Nakad.  A standalone rfid indoor positioning system using passive tags.  IEEE Transactions on Industrial Electronics, 2010.
  * [55] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In CVPR, 2019.
  * [56] Paul-Edouard Sarlin, Ajaykumar Unagar, Mans Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, et al.  Back to the feature: Learning robust camera localization from pixels to pose.  In CVPR, 2021.
  * [57] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al.  Benchmarking 6dof outdoor visual localization in changing conditions.  In CVPR, 2018.
  * [58] Torsten Sattler, Akihiko Torii, Josef Sivic, Marc Pollefeys, Hajime Taira, Masatoshi Okutomi, and Tomas Pajdla.  Are large-scale 3d models really necessary for accurate visual localization?  In CVPR, 2017.
  * [59] Gerald Schweighofer and Axel Pinz.  Robust pose estimation from a planar target.  IEEE TPAMI, 28(12):2024–2030, 2006.
  * [60] Roman Seidel, André Apitzsch, and Gangolf Hirtz.  Improved person detection on omnidirectional images with non-maxima suppression.  In International Conference on Computer Vision Theory and Applications, 2019.
  * [61] Steve Shafer, John Krumm, Barry Brumitt, Brian Meyers, Mary Czerwinski, and Daniel Robbins.  The new easyliving project at microsoft research.  In Proceedings of the 1998 DARPA/NIST Smart Spaces Workshop, 1998\.
  * [62] Xun Sun, Yuanfan Xie, Pei Luo, and Liang Wang.  A dataset for benchmarking image-based localization.  In CVPR, 2017.
  * [63] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  In CVPR, 2018.
  * [64] Masato Tamura, Shota Horiguchi, and Tomokazu Murakami.  Omnidirectional pedestrian detection by rotation invariant training.  In WACV, 2019.
  * [65] Ozan Tezcan, Zhihao Duan, Mertcan Cokbas, Prakash Ishwar, and Janusz Konrad.  Wepdtof: A dataset and benchmark algorithms for in-the-wild people detection and tracking from overhead fisheye cameras.  In WACV, 2022.
  * [66] Gonzalo Vaca-Castano, Amir Roshan Zamir, and Mubarak Shah.  City scale geo-spatial trajectory estimation of a moving camera.  In CVPR, 2012.
  * [67] Florian Walch, Caner Hazirbas, Laura Leal-Taixe, Torsten Sattler, Sebastian Hilsenbeck, and Daniel Cremers.  Image-based localization using lstms for structured feature correlation.  In ICCV, 2017.
  * [68] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr.  Fast online object tracking and segmentation: A unifying approach.  In CVPR, 2019.
  * [69] Tsaipei Wang, Chia-Wei Chang, and Yu-Shan Wu.  Template-based people detection using a single downward-viewing fisheye camera.  In International Symposium on Intelligent Signal Processing and Communication Systems, 2017.
  * [70] Wenguan Wang, James Chenhao Liang, and Dongfang Liu.  Learning equivariant segmentation with instance-unique querying.  In NeurIPS, 2013.
  * [71] Roy Want, Andy Hopper, Veronica Falcao, and Jonathan Gibbons.  The active badge location system.  ACM Transactions on Information Systems, 10(1):91–102, 1992.
  * [72] Tobias Weyand, Ilya Kostrikov, and James Philbin.  Planet-photo geolocation with convolutional neural networks.  In ECCV, 2016.
  * [73] Aoran Xiao, Ruizhi Chen, Deren Li, Yujin Chen, and Dewen Wu.  An indoor positioning system based on static objects in large indoor scenes by using smartphone cameras.  Sensors, 2018.
  * [74] Chouchang Yang and Huai-Rong Shao.  Wifi-based indoor positioning.  IEEE Communications Magazine, 2015.
  * [75] Luwei Yang, Ziqian Bai, Chengzhou Tang, Honghua Li, Yasutaka Furukawa, and Ping Tan.  Sanet: Scene agnostic network for camera localization.  In ICCV, 2019.
  * [76] Khalid Yousif, Yuichi Taguchi, and Srikumar Ramalingam.  Monorgbd-slam: Simultaneous localization and mapping using both monocular and rgbd cameras.  In ICRA, 2017.
  * [77] Amir Roshan Zamir and Mubarak Shah.  Accurate image localization based on google maps street view.  In ECCV, 2010.
  * [78] Amir Roshan Zamir and Mubarak Shah.  Image geo-localization based on multiplenearest neighbor feature matching using generalized graphs.  IEEE TPAMI, 36(8):1546–1558, 2014.
  * [79] Jun Zhu, Jiangcheng Zhu, Xudong Wan, Chao Wu, and Chao Xu.  Object detection and localization in 3d environment by fusing raw fisheye image and attitude data.  Journal of Visual Communication and Image Representation, 59:128–139, 2019.