# CasCalib: Cascaded Calibration for Motion Capture from Sparse Unsynchronized
Cameras

Huibert Kwakernaak1 and Pradeep Misra2
1 Faculty of Electrical Engineering, Mathematics and Computer Science,
University of Twente, Enschede, The Netherlands 2 Department of Electrical
Engineering, Wright State University, Dayton, USA This work was not supported
by any organization James Tang1 Shashwat Suri1 Daniel Ajisafe1 Bastian Wandt2
Helge Rhodin1,3
1The University of British Columbia 2Linköping University 3Bielefeld
University

###### Abstract

It is now possible to estimate 3D human pose from monocular images with off-
the-shelf 3D pose estimators. However, many practical applications require
fine-grained absolute pose information for which multi-view cues and camera
calibration are necessary. Such multi-view recordings are laborious because
they require manual calibration, and are expensive when using dedicated
hardware. Our goal is full automation, which includes temporal
synchronization, as well as intrinsic and extrinsic camera calibration. This
is done by using persons in the scene as the calibration objects. Existing
methods either address only synchronization or calibration, assume one of the
former as input, or have significant limitations. A common limitation is that
they only consider single persons, which eases correspondence finding. We
attain this generality by partitioning the high-dimensional time and
calibration space into a cascade of subspaces and introduce tailored
algorithms to optimize each efficiently and robustly. The outcome is an easy-
to-use, flexible, and robust motion capture toolbox that we release to enable
scientific applications, which we demonstrate on diverse multi-view
benchmarks. Project website: https://github.com/jamestang1998/CasCalib.

##  I Introduction

Computer vision has now reached the mainstream, enabling detailed 3D
reconstructions from handheld video recordings with ubiquitous mobile phones.
However, when aiming for the reconstruction of dynamic human performances, it
still demands multiple cameras and manual calibration as single-view methods
suffer from occlusions and depth ambiguities. Hence, applications in visual
effects and medical studies, such as those in neuroscience studying motion
deficits, still rely on dedicated motion capture studios. However, their
manual calibration is cumbersome and error-prone, and their high cost renders
them entirely inaccessible to smaller companies and labs with a non-technical
background.

To combat these issues, we propose an end-to-end framework for camera
calibration and time-synchronization of a sparse-camera system. It directly
serves the before-mentioned human pose and shape estimation tasks. In
addition, once automatically estimated on full body pose, the calibration can
be applied for reconstructing other objects, including fine-grained face
reconstruction yielding more precise angle and shape estimates for embodied
virtual reality, expression estimation, and gesture recognition.

Figure 1: Cascaded calibration overview. From top to bottom, we show how we
break up the optimization problem into smaller subproblems by solving for a
subset of the parameters at a time, with subsequent steps refining the earlier
ones. The first step is the Single View Calibration step where we estimate the
normal vector $\mathbf{n}$ and the intrinsics $\mathbf{K}$. Then, we estimate
the time synchronization offset $\Delta t$. Finally, with the last three
steps, we estimate and refine the rotation matrix $\mathbf{R}$ and the
translation $\mathbf{T}$.

Many approaches towards automating 3D capture exist, but only for specialized
settings. Structure-from-motion techniques require either a continuous video
stream of a static [28] or slowly deforming object [2], or a dense array of
cameras that have largely overlapping fields of view. These are popular for
settings where there is only one camera or dozens of them, but nothing in
between. We refer to this intermediate setting as the sparse camera case. For
sparse camera setups, manual calibration with a checkerboard or other markers
arranged in a predetermined two-dimensional pattern is the most common [6].
However, the calibration object needs to be carefully placed and repositioned
by trained operators to ensure that the entire capture volume is covered, and
multiple cameras see the calibration object. Moreover, recalibration is
required when cameras move ever so slightly, and fabricating calibration
objects at very small or big scales, for example, meter scale for sports
events, is challenging.

To achieve fully automated calibration, a promising fully automatic direction
is to use humans or animals, which are usually present in the scene, as
calibration objects. Fei et al. [11] calibrate the focal length and ground
plane under the assumption that all people have roughly the same height and
the ground is flat, but do not address synchronization and the multi-view
case. Takahashi et al. [31] and Liu et al. [23] subsequently use 2D keypoint
detections in individual views to calibrate the extrinsic parameters of two or
more cameras using classical fundamental matrix estimation, which requires
seven or more correspondences across views. However, to establish the
correspondence, they assume that cameras are synchronized, and only a single
person is in view and visible from all cameras. In turn, Zhang et al. [40]
establish temporal synchronization but assume intrinsic and extrinsic
calibrated cameras, and still uses a single person. Xu et al. [36] rely on
reidentification, but synchronization is required and appearance matching is
challenging when humans look alike, such as when a sports team dresses in the
same uniform. To the best of our knowledge, there is no solution for
unsynchronized sparse camera setups with a variable number of persons that are
only partially visible.

We propose a cascaded calibration algorithm that breaks down the calibration
of high-dimensional parameter space into subspaces that can be searched or
optimized efficiently. For $N$ cameras, we solve for $N(4\times 6+1)$
parameters which correspond to $4$ intrinsics (focal lengths and focal
center), $6$ extrinsics (camera orientation and position), and $1$ temporal
shift parameters. The outcome is a sequential process with steps having the
cascading dependencies visualized in Figure 1, with subsequent steps starting
from preceding estimates that are further refined. In the first stage, camera
focal length (intrinsic) and their orientation with respect to the ground
(extrinsic) are estimated similarly to [11, 32], independently in each view so
that we don’t need temporal synchronization across cameras. In the second
stage, we estimate the temporal offset by taking the scalar distances of the
ankles from the center of the ground plane in order to reduce the dimensions
to one. Then we align the sequence temporally by searching for the frame
offset that results in the minimum distance between the sequences.

In the third stage, we use these to reduce the 6D extrinsics problem to
solving for 2D rotation and translation in the estimated ground planes, a 3D
space optimized by least squares, and a greedy yet efficient search of the
rotation on the ground plane. In the fourth stage, we refine the initialized
extrinsic parameters alongside the additional temporal offset using ICP ([1]
and [39]) in a 5D space. In the final stage, the entire 11D space is optimized
using bundle adjustment [33].

In summary, our main scientific contributions are:

  * •

The cascading decomposition of the calibration problem at an algorithmic
level.

  * •

Derivation of suitable objective functions and parameter representations for
each stage, and the empirical validation of their hyperparameters.

A further contribution for computer vision practitioners is:

  * •

Source code, documentation, and curated environments to aid future work and
deployment.

##  II Related Work

We categorize methods for multi-view calibration based on human poses by their
reconstruction methodology, such as optimization or deep learning; whether
they can handle a single person or multiple persons; the required prior
knowledge; and the output such as intrinsics, extrinsics, and temporal
synchronization. Table I compares the capabilities and requirements of
existing methods to ours.

Method | Framework | Finds Sync. | Finds Intrinsics | Finds Extrinsics | w/o GT Intrinsics | w/o GT Extrinsics | w/o GT Sync. | Multi-Person | Multi-view
---|---|---|---|---|---|---|---|---|---
Lee [20] | Deep | No | No | Yes | No | Yes | No | No | Yes
Qi Zhang [38] | Deep | Yes | No | No | No | No | Yes | Yes | Yes
Chaoning Zhang [37] | Deep | No | Yes | Yes | No | No | No | N/A | Yes
Grabner [14] | Deep | No | Yes | Yes | No | No | No | N/A | No
Jarved [18] | Optimization | No | No | No | Yes | Yes | No | Yes | Yes
Xu [36] | Optimization | No | No | Yes | No | Yes | No | Yes | Yes
Troung [34] | Optimization | No | No | Yes | No | Yes | No | Yes | Yes
Liu [23] | Optimization | No | Yes | Yes | Yes | Yes | No | No | Yes
Fei [11] | Optimization | No | Yes | Yes | Yes | Yes | Yes | Yes | No
Takahashi [31] | Optimization | Yes | No | Yes | No | Yes | Yes | Yes | Yes
Zhe Zhang [40] | Optimization | Yes | No | No | No | No | Yes | No | No
Ours | Optimization | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes

TABLE I: Comparison of related methods. We summarize the differences,
including which parameters they estimate and whether they require ground truth
input. Only ours is able to calibrate and synchronize multi-person sequences.

Optimization: These methods approach the calibration problem by obtaining
keypoints of the poses from each camera, and then geometrically solving for
the camera poses and optionally the 3D human pose. Liu et al. [23] performs
intrinsic and extrinsic calibration on videos containing a single person.
Since the videos contain a single person and are temporally synchronized, they
do not need to find multi-view correspondences between the cameras since the
person in one camera will always correspond to the person in the other camera.
First, they obtain 2D keypoints using OpenPose [5], then they use multi-view
correspondences of a single person to triangulate the 3D keypoints. Finally,
they reproject the 3D keypoints back to 2D image coordinates and they optimize
the reprojection error between the cameras. Similar to our last processing
step, the intrinsic and extrinsic parameters are further optimized using a
RANSAC loop [12], followed by bundle adjustment to optimize the camera poses.
Takahashi et al. [31] calibrate extrinsic and intrinsic parameters in a
similar way except they extend the method by using prior information about
human poses such as bone length constraint and smooth motion constraints.
However, neither Liu et al. [23] nor Takahashi et al. [31] solve the multi-
person case, which requires obtaining multi-view correspondences. In addition
to the requirement of multi-view correspondences, they also depend on temporal
synchronization.

Many methods try to solve the multi-person case using person reidentification.
Xu et al. [36] use matches bounding boxes between views. This method performs
relatively well on datasets where people wear distinct clothing, as such on
the Terrace and Basketball sequences [13], but performs comparatively worse on
their ConstructSite dataset where everyone is wearing the same uniform.
Another method of matching is to take reprojections between each camera and
match the closest pose in 3D to each other. This is used in several methods
([34, 18, 40]) and has the advantage that they do not depend on visual
features on the persons such as clothing, since those could have high
variation even on the same person, although combinatorically optimizing the
matches can be expensive even with efficient algorithms like the Hungarian
algorithm.

Deep Learning: These approaches solve the calibration problem by fully
regressing the camera parameters or predicting intermediate estimates through
training on a labeled dataset. Lee et al. [20] use a 3D pose estimator and
then use the predicted 3D poses as a 3D calibration object to optimize the
camera poses such that 3D poses match and re-project to 2D pose estimates.
This process alternates between optimizing the camera pose and intrinsics and
optimizing the human pose. Zhang et al. [37] perform camera calibration on pan
tilt zoom (PTZ) cameras that rotate and zoom but do not translate. Their
application requires an online estimation. They take in as input two images
from two views and use a three-part pipeline: a feature extractor based on a
Siamese network architecture [3], feature matching using correlation, and a
regression network, to automatically estimate the intrinsic and extrinsic
parameters, as well as the distortion parameters. Although this method can
solve a more challenging problem, in the case where the cameras are not
static, they require a large training set of over 100,000 image pairs. In
addition, they do not handle the case where cameras are unsynchronized.
Grabner et al. [14] approach the problem by extending the Faster/Mask R-CNN
framework [15], by augmenting it with a focal length predictor and refining it
using the reprojection error. However, despite the fact that their method gets
10 percent better focal lengths than the baseline, they use a very specific
dataset of common objects as references such as chairs, sofas, and cars
without occlusion. In general, these objects may not be present. Additionally,
all deep learning methods introduce the need for a training set and a training
procedure, which may not generalize to real-world scenes.

Synchronization: The previously described methods rely on the fact that the
cameras are temporally synchronized with each other, which is an assumption
that is often violated in practice. Besides methods such as hardware
synchronization, which requires a pre-recording step to do, human pose can be
used for temporal synchronization. In [40], Zhang et al. find the temporal
offset by minimizing error in epipolar lines from reprojecting 2D pose
detections. However, this method is in turn limited by the need for the
cameras to be calibrated. In [9], Eichler et al. perform both camera
synchronization and camera calibration using distances between joints from 3D
pose detectors for synchronization, followed by a multi-view reconstruction
for the extrinsic parameters. However, this method requires using a 3D pose
detector, as well as assuming intrinsics are known. Zhang et al. [38] perform
camera synchronization using a neural network architecture by warping one view
to another, which does not assume a constant frame rate but intrinsic and
extrinsic calibrations.

Summary: Although these methods have been shown to be effective, they all
require some sort of ground truth such as temporal synchronization, intrinsic,
and extrinsic which they use to solve for the unknowns. We seek to use only 2D
keypoint information without assuming calibration, synchronization, a training
dataset, or person re-identification.

##  III Method

Figure 2: System overview. A fine-grained view of the five stages in Figure 1,
including how detections of single persons in single views are treated
independently in Stage I and jointly subsequently. Variables are in the plate
notation, with $n$ the number of cameras and $m$ the number of people in the
scene.

To solve the camera calibration problem from multiple uncalibrated and
unsynchronized views, we propose to break down the problem into several lower-
dimensional problems. In a cascaded fashion, we start with a few variables
that are solved globally and subsequently add details while reducing the range
of the search space to stay practical.

We take in as input the 2D key point detections
$\mathbf{p}^{\text{img}}\in\mathbb{R}^{2}$ of the major human joints, such as
the head, neck, and ankles. Figure 1 shows how the single view calibration,
temporal search, rotation search, ICP, and bundle adjustment modules reduce
the dimensionality of the problem, and Figure 2 shows how the variables are
passed and refined between modules.

####  III-1 Single View Geometric Calibration

In the single view calibration case, our goal is to find the intrinsic camera
parameters $\mathbf{K}$ of the projection transformation

$\mathbf{p}^{\text{img}}=\mathbf{K}\mathbf{p}^{\text{cam}},\text{ where
}\mathbf{K}=\begin{pmatrix}f&0&o_{1}\\\ 0&f&o_{2}\\\ 0&0&1\end{pmatrix},$ (1)

mapping from 3D camera coordinates $\mathbf{p}^{\text{cam}}$ to 2D image
coordinates $\mathbf{p}^{\text{img}}$. We estimate $\mathbf{K}$, the ground
plane position $\mathbf{g}$, and orientation $\mathbf{n}$ relative to the
camera origin using a direct linear transform. To make the estimation feasible
with only $\mathbf{p}^{\text{img}}$ as input, we assume that persons are
standing up-right in some of the frames which makes them parallel to the
ground plane normal vector and have a constant metric height $h$. Furthermore,
following the cascaded, coarse-to-fine principle, we fix the focal point
$(o_{1},o_{2})$ to the image center and do not consider any relations across
cameras as synchronization is missing. Note that these strong assumptions are
lifted in later refinement stages.

##### Direct Linear Transform

Using homogeneous coordinates, the ankle and shoulder positions of three or
more people on a common ground plane are related by a linear system of
equations that, when solved for its null space, reveal the sought-after camera
parameters in closed form. The derivation is explained in the supplemental and
is analogous to that in [11]. In practice, we apply RANSAC [12] to filter out
outliers and we select the largest inlier set. In order to determine which
detections are inliers, we first reproject the ankle coordinates
$\mathbf{p}^{\text{img}}_{\text{ankle}}$ to 3D
$\mathbf{p}^{\text{cam}}_{\text{ankle}}$. Second, we add $h\mathbf{n}$ to
$\mathbf{p}^{\text{cam}}_{\text{ankle}}$ to get the predicted
$\mathbf{p}^{\text{cam}}_{\text{shoulder}}=\mathbf{p}_{\text{ankle}}+h\mathbf{n}$.
Finally, we reproject this point back to image coordinates using Eq. 7. We use
two metrics, shoulder pixel error and angle error to determine if it is an
inlier. The pixel error is computed as the pixel Euclidean distance between
the shoulder detection and the predicted shoulder. We normalize this by the 2D
pixel height of the person. The angle error is computed by computing the angle
between the vector from the $\mathbf{p}^{\text{img}}_{\text{ankle}}$ to the
$\mathbf{p}^{\text{img}}_{\text{shoulder}}$ and the vector from
$\mathbf{p}^{\text{img}}_{\text{ankle}}$ to the predicted
$\mathbf{p}^{\text{img}}_{\text{shoulder}}$. An ankle-shoulder pair is
considered an inlier if is less than both the angle and pixel threshold. For
our experiments, we use an angle threshold of 2.86 degrees and a pixel
threshold of 5 percent of the pixel height. Once we have the largest inlier
set, we run our DLT method on the entire inlier set to get the final focal
length and ground plane position and orientation. We show an example of our
single view calibration algorithm in Figure 3.

Figure 3: 2D Reconstruction. Visual results for the single view calibration
for Human3.6M Subject 1. The blue grid represents the ground plane predicted
by our method with a coordinate axis defined at the bottom of the image. The
green line from the ankle to the shoulders represents the ankle to shoulder
keypoints.

##### Filtering

Since we have an assumption that the persons are standing straight up, we must
filter out the non-standing poses in the scene. To determine this, for every
2D pose $\mathbf{p}$, we measure the 2D angles between the vectors from
shoulder to hip, hip to knee, and knee to ankle. We give the exact equations
in the supplemental.

Figure 4: Time synchronization. Time synchronization results between ref (red)
and sync (blue) sequences for subject 1 walking sequence in Human3.6M.

##### Relating multiple cameras through their ground planes

The relation between individually calibrated cameras is unknown, but the
estimated ground plane is shared. We select one camera to be the reference
camera, and use its plane coordinates as the world coordinate system. To
simplify subsequent steps, we compute for each other camera the homography
transformation from image coordinates $\mathbf{p}^{\text{img}}$ to the
estimated ground plane,

$\mathbf{p}^{\text{plane}}=[\mathbf{R}^{\text{cam}\rightarrow\text{plane}}|\bm{\tau^{\text{plane}}}]\mathbf{K}^{-1}\mathbf{p}^{\text{img}}.$
(2)

Figure 5 shows the resulting birds-eye view of the ground plane with estimated
person positions. The plane normal vector $\mathbf{n}$ is shared between all
cameras and defines one column of
$\mathbf{R}^{\text{cam}\rightarrow\text{plane}}$. For each camera, we derive
the other 2 basis vectors by the backprojection of the 2D horizontal line to
3D as the new x-axis, and finally the cross product of the normal vector with
the x-axis as the z-axis. The position $\bm{\tau}^{\text{plane}}$ is the back
projection of the image center to the ground plane. This construction is
intermittent. It remains to align the 2D position and orientation within the
ground plane to fully determine the camera extrinsics, as well as to estimate
the time shift.

Figure 5: Ground plane view. Bird-eye view of the ankles on the Terrace
sequence, with inliers in green and outliers in red.

####  III-2 1D Temporal Search

To support cameras starting or ending at different times, we model the time
relationship pairwise between cameras as
$t_{\text{ref}}=t_{\text{sync}}+\Delta t_{\text{sync}}$, a linear relationship
between reference camera sequence $t_{\text{ref}}$ and $t_{\text{sync}}$ of
the target camera. To find the optimal translation $\Delta t_{\text{sync}}$,
first, we project the detected ankle points onto the ground plane using
$\mathbf{R}^{\text{cam}\rightarrow\text{plane}}$ and shift them such that the
mean of the reference set is the same as the mean of sync set. In order to get
a signal that is time-sensitive but does not depend on the unknown camera
extrinsics, we compute the distance $d$ from the center for each point. Since
the search space is 1D, we can afford a brute-force search, with candidate
offsets ranging from 0 to one-third of the length of the sync sequence. Note
that if there is more than one person in the frame, then that time step has
more than one distance associated with it. We show an example of the temporal
alignment in Figure 4.

##### Search criteria

The alignment is scored by the absolute difference of $d_{\text{sync}}$ and
$d_{\text{ref}}$ within the same time step. If there is more than one person
in the frame, we compute an optimal matching using the Hungarian algorithm
[19]. While shifting the curves temporally, we continue the endpoints of the
curves by repeating the endpoint values. This helps prevent the curve from
shifting too much since larger shifts lead to smaller overlap and hence larger
uncertainty since we are not matching large amounts of the curve.

##### Filtering

Since noisy detections could cause outlier points to appear on the ground
plane, we remove outlier points on the ground plane using a density-based
spatial clustering of applications with noise (DBSCAN) [10] to find the
largest cluster of points on the ground plane, and then remove all the outlier
points.

####  III-3 2D Rotation Search

Once we match the videos temporally, we complete the extrinsic calibration
between the cameras. First, we shift the means of the ankle positions in 2D
plane coordinates from the _sync_ camera sequence to align with the reference
camera’s sequence. Then we search rotation angles from 0 to 360 degrees and
apply the rotation to each sequence. For each camera i, we compute our error
augmenting our detections $\mathbf{p}^{\text{plane}}_{i}$ with the time step,
$\hat{\mathbf{p}}^{\text{plane}}_{i}=(x,y,t)$, and computing the distance
between the closest points in the point cloud. Note that no closed-form
solution is possible, since correspondences between the two point clouds are
unknown. We elaborate further on the equations in the supplemental. We show an
example of the point cloud alignment process in Figure 6.

(a) Initial orientation

(b) Best rotation

Figure 6: 2D Rotation search. Visual results for 2D rotation search on
Human3.6M subject 1 with 2 cameras. Axes are in meters.

####  III-4 Iterative Closest Point

The previous section yields a first estimate of all camera extrinsics by
estimating the 2D plane rotation and position of cameras relative to each
other that remained unknown in Step 1. We refine that estimate using the
Iterative Closest Point (ICP). We find the closest points by utilizing the
previously estimated time synchronization to match the frames from the
reference view to the synchronization view and also the same Hungarian
matching process, when multiple persons are present. We then optimize the
rotation and translation by minimizing the Euclidean distance between the 2D
point clouds in the plane Iteratively, we re-associate the points and repeat
the process. Note that the initial 2D rotation search searches all angles
between 0 and 360, making adjustments from the ICP small.

####  III-5 Joint Camera Refinement (Bundle Adjustment)

Once we get the result from our ICP step, we pick the top $k$ poses with the
highest confidence to use in the final bundle adjustment [33] step that
refines all calibration parameters jointly, by using the association of
keypoints from the previous timesteps. As opposed to previous steps using
ankle and shoulder, we can now incorporate all body part detections. Figure 7
shows this triangulation. However, because the head and arm keypoints have a
larger range of motion and are often self-occluded, we exclude them during the
bundle adjustment. For camera pairs $i$ and $j$, we can define the
relationship from 2D to 3D as a line using $\ell(k)=m_{j}k+\bm{\tau}_{j}$,
with $k\in\mathbb{R}$ and

$m_{j}=\mathbf{R}^{\text{plane}\rightarrow\text{world}}_{j}(\mathbf{R}^{\text{cam}\rightarrow\text{plane}}_{j}\mathbf{K}_{i}^{-1}\textbf{P}^{\text{img}}_{j}-\bm{\tau}^{\text{cam}\rightarrow\text{plane}}_{j}).$
(3)

We then minimize the distance between pairs of lines by gradient descent.

Figure 7: Bundle adjustment. Visual 3D pose reconstruction for Human3.6M
subject 1 with 2 cameras where the red and blue poses represent the closest
points on each view ray

##  IV Experiments

For our experiments, we test different components of the pipeline on a variety
of datasets including Human3.6M [16], EPFL Terrace and Laboratory [13], and
vPTZ [26]. To aid comparisons with future work, we will make the code publicly
available. As in Fei et al., we adopt HRNet [29], as implemented by mmpose
[7], to obtain 2D keypoint detections and tracking. For steps that require the
results of previous steps, we run the previous parts of our pipeline to get
the intermediate results to use as input to the next part of our pipeline.

##### Metrics

We use different metrics to evaluate different components of the camera
calibration. Here is a brief breakdown of the metrics used:

  * •

$\mathbf{Percent}$ $\mathbf{Focal}$ $\mathbf{Error}$ is used to measure the
relative error in focal length and can be described as follows:

$\mathbf{\%f}=\mathbf{\frac{f_{predicted}-f_{gt}}{f_{gt}}}$ (4)

  * •

$\mathbf{Relative}$ $\mathbf{Rotation}$ $\mathbf{and}$ $\mathbf{Translation}$:
Since the coordinate system for the ground truth extrinsic is only uniquely
defined up to a global rotation and translation, to evaluate the camera pose,
we compute the relative rotation and translation from the reference camera to
all the other cameras for both the predicted and the ground truth camera
systems and then represent the error as follows

∘ $\displaystyle=\mathbf{|^{\circ}_{\;\;rel}-^{\circ}_{\;\;gt}|}$ (5)
$\displaystyle\mathbf{m}$ $\displaystyle=\mathbf{|m_{rel}-m_{gt}|}$ (6)

##### Datasets

We evaluate on the following datasets, each modeling a different setting in
terms of the number of people and cameras, and the scale.

  * •

$\mathbf{Human3.6M}$ [16] contains 4 cameras temporally synchronized and
calibrated cameras that are recorded on a variety of subjects and actions. For
our experiments, we use subjects 1,5,6,7,8,9,11 and use the walking action.
Each video only contains one subject at a time.

  * •

$\mathbf{Terrace}$ $\mathbf{and}$ $\mathbf{Laboratory}$ [13] from EPFL multi-
camera pedestrian videos contains 4 temporally synchronized and calibrated
cameras filming an outdoor scene with up to 7 subjects (Terrace) and another
indoor scene with up to 4 subjects (Laboratory). Although the cameras are
calibrated, the dataset has two versions of the calibrations. We show how we
process these files in the supplemental.

  * •

$\mathbf{vPTZ}$ [26] contains 3 outdoor scenes with numerous pedestrians with
4 cameras. Each video is filmed from a fairly high view and is representative
of outdoor security camera footage.

###  IV-A Single View Experiments

To test the single view calibration stage, we test our method on vPTZ and the
EPFL Terrace sequences. We compare against [11] single view calibration
method, and their results. In addition, we also compare against the other
methods that Fei et al. implemented for testing, including [21], [22], [4],
and [32]. We report our numerical results in Table II and visual results of
the ground plane and shoulder reprojections in Figure 8. We find that our
method is comparable to the related methods in the table, getting an average
error of 11 percent of the ground truth focal length. Although our method does
not outperform the other methods, we deemed it to be a good enough
reproduction to proceed with the main focus of this paper, as we did not have
the code with additional implementation details and hyperparameter choices.

In addition to these experiments, we also evaluate our single view calibration
on synthetic data, which is described in detail in the supplementary section.

(a) vPTZ. set1-cam-131

(b) vPTZ. set1-cam-132

(c) vPTZ. set2-cam-132

(d) Terrace. terrace1-cam0

Figure 8: Single view calibration results. Visual results of our single view
calibration algorithm from table II with the predicted ground plane
represented as a blue grid where every square is one meter. The green and red
lines represent reprojected ankle and head detections from the entire
sequence. Red represents outlier detections and green represents inlier
detections.

Method | set1-cam-131 $\downarrow$ | set1-cam-123 $\downarrow$ | set2-cam-132 $\downarrow$ | terrace1-cam0 $\downarrow$
---|---|---|---|---
[21] | 1.00 | 29.00 | N/A | N/A
[22] | 2.00 | 19.00 | N/A | N/A
[4] | N/A | 15.00 | 24.92 | 5.33
[32] | N/A | 10.14 | 12.07 | 1.43
[11] | 4.70 | 0.35 | 10.74 | 2.51
Ours | 11.31 | 0.66 | 18.36 | 13.18

TABLE II: Single View Calibration Results. Percent focal error on the vPTZ
sequences.

###  IV-B Temporal Synchronization Experiments

For comparing against existing synchronization methods, we test our method on
Human3.6M. We randomly cut a section of each video that is half the length of
the sequence for the Walking sequence for each subject. This is similar to the
experiments that Zhang et al. [40] do except they only test their method when
the true offset is 0, which is too simplistic. For our experiments, we shift
the sequence with offsets 0, 50, 100, 150, and 200. We run the experiments
using ground truth focal length and the focal length predicted by our method.
We show that our method has a mean and median prediction that is close to the
true offset, up to an error of 10 frames with a standard deviation of around
10 frames when we use the ground truth focal length. When we use our predicted
focal length, we get an error of up to about 11 frames and a standard
deviation of 15 frames for most of the experiments. We report our results in
Table III. We discuss one failure case happening for large shifts in the
limitations section.

For the multi-person case, we perform a similar experiment as with the single-
person case, except we shift the sequence with offsets 0, 25, and 50 on the
EPFL Terrace and Laboratory sequences using our predicted focal lengths. We
report our results in Tables VI and VII. For these sequences obtain an error
within 5 frames of the ground truth offset.

| Pred offset (gt f) | Pred offset (Pred f) |  [40]*
---|---|---|---
offset gt | mean | median | std | mean | median | std | mean
0 | 4.0 | 1.0 | 10.0 | 2.90 | 2.5 | 14.05 | 0.0
50 | 53.93 | 51 | 10.05 | 52.81 | 52.5 | 14.12 | N/A
100 | 103.88 | 101 | 10.17 | 102.67 | 102 | 14.22 | N/A
150 | 154.5 | 154 | 10.15 | 153.48 | 153 | 15.04 | N/A
200 | 204.76 | 205 | 10.082 | 189.26 | 201 | 51.70 | N/A

TABLE III: Temporal synchronization experiments. This table shows the
predicted offset given pairs of videos of Human3.6M with varying offsets as
well as their standard deviations. (gt f) means that we run our
synchronization using ground truth focal length, and (pred f) means we run our
synchronization using predicted focal length. ∗using GT calibration and not
tested on large offsets.

###  IV-C Synchronized Bundle Adjustment Experiments

To compare against existing methods requiring synchronized cameras, we test on
the EPFL Terrace sequence without introducing temporal shift. We compare
against Xu et al. [36] as well as the other methods that Xu et al. tested
including SIFT [24], BFM [17], SuperPoint [8], SuperGlue [27], and WxBS [25].
The oracle method in Table IV simply means that they used manual pose
annotations. Some methods, such as Sift + BFM, use the first method to extract
the keypoints, and the second method to match the keypoints. The Sift + BFM,
Superpoint + BFM, and Oracle methods use Xu et al. Geosolver after finding the
correspondences. We find that our method gives reasonable reconstructions
compared to the other methods, only being outperformed by Xu et al. and the
oracle. However, only by a small margin and they both utilize ground truth
intrinsics while we use our estimated intrinsic.

Method | mm $\downarrow$ | degree $\downarrow$
---|---|---
SIFT [24] \+ BFM [17] | 4599 | 55.03
SuperPoint [8] \+ [17] | 358 | 54.68
WxBS [25] | 1302 | 54.14
[8] \+ SuperGlue [27] | 9934 | 36.96
Oracle(Manual-pts) | 390 | 1.18
[36] (Manual-bbox) | 308 | 0.52
[36] (ReID-bbox) | 308 | 0.52
Ours | 138 | 1.82

TABLE IV: Synchronized bundle adjustment experiments. Camera pose error for
the Terrace sequence.

###  IV-D Multiview Offset Experiments

For multiview offset experiments, we test our method using a similar setting
as in the temporal experiments, except we also apply our multiview calibration
algorithm afterward in order to analyze the effects of synchronization
accuracy on the complete calibration pipeline. As a baseline, we also run our
multiview calibration method on the unsynchronized sequences. For these
experiments, we do not run our bundle adjustment algorithm since two views
provide insufficient constraints under noisy keypoint estimates. For the
single-person case, we test it on Human3.6M with 4 cameras for subjects
1,5,6,7,8,9, and 11. We report our results in Table V. We show that although
our synchronization method is not perfect, it performs much better than
running the multiview calibration algorithm without temporal synchronization.

For the multi-person case, we utilize the EPFL sequences Terrace and
Laboratory which contain up to 4 and 7 people respectively, and 4 cameras
each. We proceed with our experiments in a similar manner to the experiments
using pairwise cameras, however, we use timesteps 0, 25, and 50. We report our
results in Table VI and Table VII. Like in the single-person case, we show
that our synchronization method results in better performance than running the
multiview calibration algorithm without temporal synchronization.

offset (gt) | 0 | 100 | 150 | 200
---|---|---|---|---
offset (pred) | 5.36 | 105.21 | 156.21 | 191.96
No Sync | ° | 5.78 | 36.24 | 42.71 | 73.04
m | 0.057 | 0.26 | 0.46 | 0.82
%f | 15.06 | 15.06 | 15.06 | 15.06
Sync | ° | 10.67 | 10.58 | 10.99 | 16.89
m | 0.066 | 0.065 | 0.068 | 0.21
%f | 15.06 | 15.06 | 15.06 | 15.06
TABLE V: Multiview offset experiments for Human3.6M. We report the Camera pose error for unsynchronized sequences and synchronized. No sync means that the algorithm is run without running our synchronization step. Sync means we run our synchronization step. Offset pred means the predicted offset from our synchronization method. offset (gt) | 0 | 25 | 50
---|---|---|---
offset (pred) | 4 | 29 | 55
No Sync | ° | 2.73 | 6.32 | 13.07
m | 0.073 | 0.097 | 0.15
%f | 9.45 | 9.45 | 9.45
Sync | ° | 2.14 | 2.15 | 2.18
m | 0.070 | 0.069 | 0.069
%f | 9.45 | 9.45 | 9.45
TABLE VI: Multiview offset experiments for Terrace. Showing that the synchronization significantly improves camera position and angle. Focal length can be estimated from a single view and is not further refined in this experiment. offset (gt) | 0 | 25 | 50
---|---|---|---
offset (pred) | 1.67 | 29 | 55.33
No Sync | ° | 0.81 | 10.02 | 44.10
m | 1.15 | 1.16 | 1.15
%f | 15.63 | 15.63 | 15.63
Sync | ° | 0.51 | 0.61 | 0.81
m | 1.15 | 1.15 | 1.15
%f | 15.63 | 15.63 | 15.63
TABLE VII: Multiview offset experiments for the Laboratory sequence.
Improvements on this indoor sequence are consistent with the outdoor terrace
sequence in Tab. VI.

##  V Limitations and Discussion

From the results of our experiments, we demonstrate that our cascaded
calibration approach can produce comparable results to other methods while
requiring fewer inputs, only requiring a 2D detector instead of having to use
ground truth intrinsics, extrinsics, or temporal synchronization, or
additional tools such as tracking or reidentification. The results can serve
as a starting point for further research since we also publish our code and
hyper-parameters such that direct comparisons can be made.

Although we show that our method works well on most scenes, in certain extreme
configurations, some of the steps fail and subsequent steps cannot recover due
to the assumption that the initialization from the previous step is within
error bounds. In future work, we plan to detect such errors and to deploy
learning-based solutions to overcome them.

###  V-A Failure Case: Periodic Motion

In Human36m walking sequences, the people are walking in a circle. This causes
the distance curves for 2 views to have a periodic shape which means that for
every n frame, the curve repeats itself. This can be problematic for the time
synchronization since this would mean multiple offsets can give a similar
result.

Our algorithm’s time synchronization module fails on Subject 11 when we set
the offset to 200 because in Figure 9, we note that the distance curves have a
period of about 200 frames, which results in our error curve having two very
similar local minimums at around 0 and 200. However, such a large misalignment
paired with harmonic repetition is unusual in practice. In future work, the
local motion, such as the articulation of arms could be used to further
disambiguate frames. However, this is non-trivial as occlusions and different
viewing angles have to be considered.

(a) Sequences

(b) Error curve

Figure 9: Periodic motion. Due to the period of 200 frames, there are two
similar local minimums at 0 and 200.

##  VI Conclusion

We designed, implemented, and open-sourced a method to calibrate multiple
cameras, even when videos are out of sync. It enables new application domains
for 3D vision, where reconstructions require the precision of multi-view but
the ease of recording with consumer-grade cameras without hardware
synchronization capabilities. Our future works will focus on improving the
bundle adjustment step for sparse views in addition to substituting some of
our optimization-based steps with learning-based solutions.

##

## References

  * [1] P. Besl and N. McKay.  A method for registration of 3-d shapes, ieee t. pattern anal., 14, 239–256, 1992.
  * [2] C. Bregler, A. Hertzmann, and H. Biermann.  Recovering non-rigid 3d shape from image streams.  In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), volume 2, pages 690–696. IEEE, 2000.
  * [3] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah.  Signature verification using a" siamese" time delay neural network.  Advances in neural information processing systems, 6, 1993.
  * [4] G. Brouwers, M. Zwemer, R. Wijnhoven, and P. With.  Automatic calibration of stationary surveillance cameras in the wild.  In Computer Vision – ECCV 2016 Workshops, volume 9914, 10 2016.
  * [5] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields, 2019.
  * [6] T. Chavdarova, P. Baqué, S. Bouquet, A. Maksai, C. Jose, T. Bagautdinov, L. Lettry, P. Fua, L. Van Gool, and F. Fleuret.  Wildtrack: A multi-camera hd dataset for dense unscripted pedestrian detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5030–5039, 2018.
  * [7] M. Contributors.  Openmmlab pose estimation toolbox and benchmark.  https://github.com/open-mmlab/mmpose, 2020.
  * [8] D. DeTone, T. Malisiewicz, and A. Rabinovich.  Superpoint: Self-supervised interest point detection and description, 2018.
  * [9] N. Eichler, H. Hel-Or, and I. Shimshoni.  Spatio-temporal calibration of multiple kinect cameras using 3d human pose.  Sensors, 22(22), 2022.
  * [10] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu.  A density-based algorithm for discovering clusters in large spatial databases with noise.  In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD’96, page 226–231. AAAI Press, 1996.
  * [11] X. Fei, H. Wang, X. Zeng, L. L. Cheong, M. Wang, and J. Tighe.  Single view physical distance estimation using human pose, 2021.
  * [12] M. A. Fischler and R. C. Bolles.  Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.  Commun. ACM, 24(6):381–395, jun 1981.
  * [13] F. Fleuret, J. Berclaz, R. Lengagne, and P. Fua.  Multicamera people tracking with a probabilistic occupancy map.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):267–282, 2008.
  * [14] A. Grabner, P. M. Roth, and V. Lepetit.  Gp2c: Geometric projection parameter consensus for joint 3d pose and focal length estimation in the wild, 2019.
  * [15] K. He, G. Gkioxari, P. Dollar, and R. Girshick.  Mask r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [16] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, jul 2014.
  * [17] A. Jakubovic and J. Velagic.  Image feature matching and object detection using brute-force matchers.  pages 83–86, 09 2018.
  * [18] O. Javed, S. Khan, Z. Rasheed, and M. Shah.  Camera handoff: tracking in multiple uncalibrated stationary cameras.  In Proceedings Workshop on Human Motion, pages 113–118, 2000.
  * [19] H. W. Kuhn.  The hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [20] S.-E. Lee, K. Shibata, S. Nonaka, S. Nobuhara, and K. Nishino.  Extrinsic camera calibration from a moving person.  IEEE Robotics and Automation Letters, 7(4):10344–10351, 2022.
  * [21] J. Liu, R. Collins, and Y. Liu.  Surveillance camera autocalibration based on pedestrian height distributions.  In BMVC 2011 - Proceedings of the British Machine Vision Conference 2011, pages 117.1–117.11, 01 2011.
  * [22] J. Liu, R. Collins, and Y. Liu.  Robust autocalibration for a surveillance camera network.  In Proceedings of IEEE Workshop on Applications of Computer Vision, pages 433–440, 01 2013.
  * [23] K. Liu, L. Chen, L. Xie, J. Yin, S. Gan, Y. Yan, and E. Yin.  Auto calibration of multi-camera system for human pose estimation.  IET Computer Vision, 16(7):607–618, 2022.
  * [24] D. G. Lowe.  Distinctive image features from scale-invariant keypoints.  Int. J. Comput. Vision, 60(2):91–110, nov 2004.
  * [25] D. Mishkin, J. Matas, M. Perdoch, and K. Lenc.  Wxbs: Wide baseline stereo generalizations, 2015.
  * [26] H. Possegger, M. Rüther, S. Sternig, T. Mauthner, M. Klopschitz, P. M. Roth, and H. Bischof.  Unsupervised Calibration of Camera Networks and Virtual PTZ Cameras.  In Proc. Computer Vision Winter Workshop (CVWW), 2012.
  * [27] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich.  SuperGlue: Learning feature matching with graph neural networks.  In CVPR, 2020.
  * [28] J. L. Schonberger and J.-M. Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016.
  * [29] K. Sun, B. Xiao, D. Liu, and J. Wang.  Deep high-resolution representation learning for human pose estimation, 2019.
  * [30] I. E. Sutherland.  Three-dimensional data input by tablet.  Proceedings of the IEEE, 62(4):453–461, 1974.
  * [31] K. Takahashi, D. Mikami, M. Isogawa, and H. Kimata.  Human pose as calibration pattern: 3d human pose estimation with multiple unsynchronized and uncalibrated cameras.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1856–18567, 2018.
  * [32] Z. Tang, Y.-S. Lin, K.-H. Lee, J.-N. Hwang, and J.-H. Chuang.  Esther: Joint camera self-calibration and automatic radial distortion correction from tracking of walking humans.  IEEE Access, 7:10754–10766, 2019.
  * [33] B. Triggs, P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon.  Bundle adjustment - a modern synthesis.  In Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, ICCV ’99, page 298–372, Berlin, Heidelberg, 1999. Springer-Verlag.
  * [34] A. M. Truong, W. Philips, N. Deligiannis, L. Abrahamyan, and J. Guan.  Automatic multi-camera extrinsic parameter calibration based on pedestrian torsors †.  Sensors, 19(22), 2019.
  * [35] R. Tsai.  A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses.  IEEE Journal on Robotics and Automation, 3(4):323–344, 1987.
  * [36] Y. Xu, Y.-J. Li, X. Weng, and K. Kitani.  Wide-baseline multi-camera calibration using person re-identification, 2021.
  * [37] C. Zhang, F. Rameau, J. Kim, D. M. Argaw, J.-C. Bazin, and I. S. Kweon.  Deepptz: Deep self-calibration for ptz cameras.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
  * [38] Q. Zhang and A. B. Chan.  Single-frame based deep view synchronization for unsynchronized multi-camera surveillance, 2022.
  * [39] Z. Zhang.  Iterative point matching for registration of free-form curves and surfaces.  International journal of computer vision, 13(2):119–152, 1994.
  * [40] Z. Zhang, C. Wang, and W. Qin.  Semantically synchronizing multiple-camera systems with human pose estimation.  Sensors, 21(7), 2021.

Supplemental

##  VII Camera Calibration Derivation - Details

We follow the well-established direct linear transform (DLT) [30] method to
solve projective relations. We first write our constraints as a linear system
of equations that is solved using Singular Value Decomposition (SVD), up to
the unknown scale factor arising from the projection. To reach the form
$\mathbf{M}\mathbf{x}=0$, we take the cross product of

$\mathbf{p}^{\text{img}}=\mathbf{K}\mathbf{p}^{\text{cam}},\text{ where
}\mathbf{K}=\begin{pmatrix}f&0&o_{1}\\\ 0&f&o_{2}\\\ 0&0&1\end{pmatrix}$ (7)

and

$\mathbf{p}^{\text{cam}}_{\text{shoulder}}=\mathbf{p}_{\text{ankle}}+h\mathbf{n}$
(8)

and subtract the 2 results to derive

$\mathbf{p}_{\text{shoulder}}\times\mathbf{K}(\mathbf{p}_{\text{ankle}}+\mathbf{n}\cdot
h)-\mathbf{p}_{\text{ankle}}\times\mathbf{K}(\mathbf{p}_{\text{ankle}})=0$ (9)

with $h$ the person height, $\mathbf{n}$ the normal direction and
$\mathbf{p}_{\text{shoulder}}$ and $\mathbf{p}_{\text{ankle}}$ the shoulder
and ankle positions. In the following we subscript variables with an $x,y,z$
to indicate the x,y,z-coordinates and with a number $1,2,3$ to refer to
different person locations.

In matrix form, using
$\Delta\mathbf{p}_{x}=\mathbf{p}_{x}^{\text{shoulder}}-\mathbf{p}_{x}^{\text{ankle}}$,
$\Delta\mathbf{p}_{y}=\mathbf{p}_{y}^{\text{shoulder}}-\mathbf{p}_{y}^{\text{ankle}}$,
and $z$ to represent the unknown depth of the ankle, Eq. 9 can be expressed as

$\begin{pmatrix}0&-1&\mathbf{p}_{y}^{\text{shoulder}}&0&-1&\Delta\mathbf{p}_{y}\\\
1&0&-\mathbf{p}_{x}^{\text{shoulder}}&1&0&-\Delta\mathbf{p}_{x}\\\
\end{pmatrix}\begin{pmatrix}f\mathbf{n}_{x}\\\ f\mathbf{n}_{y}\\\
\mathbf{n}_{z}\\\ \mathbf{n}_{z}\mathbf{o}_{x}\\\
\mathbf{n}_{z}\mathbf{o}_{y}\\\ z/h\end{pmatrix}=0,$ (10)

where $f$ is the focal length and $\mathbf{o}$ the principal point of the
camera intrinsics $\mathbf{K}$. By using at least three 2D shoulder
$\mathbf{p}_{\text{shoulder}}$ and ankle $\mathbf{p}_{\text{ankle}}$
detections, we form the constraint matrix

$\textbf{D}=\begin{pmatrix}0&-1&\mathbf{p}_{y1}^{\text{shoulder}}&\Delta\mathbf{p}_{y1}&0&0\\\
1&0&-\mathbf{p}_{x1}^{\text{shoulder}}&-\Delta\mathbf{p}_{x1}&0&0\\\
0&-1&\mathbf{p}_{y2}^{\text{shoulder}}&0&\Delta\mathbf{p}_{y2}&0\\\
1&0&-\mathbf{p}_{x2}^{\text{shoulder}}&0&-\Delta\mathbf{p}_{x2}&0\\\
0&-1&\mathbf{p}_{y3}^{\text{shoulder}}&0&0&\Delta\mathbf{p}_{y3}\\\
1&0&-\mathbf{p}_{x3}^{\text{shoulder}}&0&0&-\Delta\mathbf{p}_{x3}\end{pmatrix}$
(11)

that gives the system of equations

$\textbf{D}\begin{pmatrix}f\mathbf{n}_{x}+\mathbf{n}_{z}\mathbf{o}_{x}\\\
f\mathbf{n}_{y}+\mathbf{n}_{z}\mathbf{o}_{y}\\\ \mathbf{n}_{z}\\\ z_{1}/h\\\
z_{2}/h\\\ z_{3}/h\end{pmatrix}=0.$ (12)

We solve Eq. 12 using SVD. Having more than three ankles and shoulders results
in an over-determined system, for which we can find a least-squares solution.

##### Ground normal extraction.

Since Eq. 12 is a $6\times 6$ system with rank five, any solution we find is
unique up to a scalar. In order to determine $\mathbf{n}$ from the SVD or
least-squares solution, we use the fact that the normal vector is
perpendicular to any vector formed by a pair of ankles. Using

$\begin{bmatrix}\bar{\mathbf{n}_{x}}\\\ \bar{\mathbf{n}_{y}}\\\
\bar{\mathbf{n}_{z}}\\\ \bar{z_{1}}\\\ \bar{z_{2}}\\\
\bar{z_{3}}\end{bmatrix}=\lambda\begin{bmatrix}\mathbf{n}_{x}+\mathbf{n}_{z}\mathbf{o}_{x}/f\\\
\mathbf{n}_{y}+\mathbf{n}_{z}\mathbf{o}_{y}/f\\\ \mathbf{n}_{z}/f\\\
z_{1}/(hf)\\\ z_{2}/(hf)\\\ z_{3}/(hf)\end{bmatrix},$ (13)

If we do not have a given focal length, we can derive the equation for the
focal length,

$f=\sqrt{\frac{(-(\bar{\mathbf{n}}_{x}-\bar{\mathbf{n}}_{z}\mathbf{o}_{x})\bar{\mathbf{p}}_{\text{x}}-(\bar{\mathbf{n}}_{y}-\bar{n}_{z}\mathbf{o}_{y})\bar{\mathbf{p}}_{\text{y}})}{(\bar{\mathbf{n}}_{z}(\bar{z}_{1}-\bar{z}_{2}))}},$
(14)

with

$\bar{\mathbf{p}}_{\text{x}}=((\mathbf{p}_{x1}^{\text{ankle}}-\mathbf{o}_{x})\bar{z}_{1}-(\mathbf{p}_{x2}^{\text{ankle}}-\mathbf{o}_{x})\bar{z}_{2})p$
(15)

and

$\bar{\mathbf{p}}_{\text{y}}=((\mathbf{p}_{y1}^{\text{ankle}}-\mathbf{o}_{y})\bar{z}_{1}-(\mathbf{p}_{y2}^{\text{ankle}}-\mathbf{o}_{y})\bar{z}_{2}).$
(16)

Either the estimated focal lengths or a given focal length enables us to
recover $\lambda\mathbf{n}$ and $\lambda$($z_{1}$, $z_{2}$, $z_{3}$). To
remove $\lambda$, we divide both vectors by the $L_{2}$ norm of
$\lambda\mathbf{n}$, giving us a unique $\mathbf{n}$ of length one and ankle
depths $z_{1}$, $z_{2}$, and $z_{3}$.

Using the normal vector $\mathbf{n}$ and the known depths $z_{1}$, $z_{2}$,
and $z_{3}$, we recover the orientation and position of the ground plane.

##  VIII Filtering

Since we require poses that are standing straight up for our single view
calibration method, we use 17 to determine if a pose is standing straight up
by measuring the angle of the knee keypoint and the angle of the hip keypoint.

$\begin{split}&\hat{\mathbf{p}}_{\text{right}_{1}}=\mathbf{p}_{\text{right
ankle}}^{\text{img}}-\mathbf{p}_{\text{right knee}}^{\text{img}}\\\
&\hat{\mathbf{p}}_{\text{right}_{2}}=\mathbf{p}_{\text{right
hip}}^{\text{img}}-\mathbf{p}_{\text{right knee}}^{\text{img}}\\\
&\hat{\mathbf{p}}_{\text{right}_{3}}=\mathbf{p}_{\text{right
shoulder}}^{\text{img}}-\mathbf{p}_{\text{right knee}}^{\text{img}}\\\
&\hat{\mathbf{p}}_{\text{left}_{1}}=\mathbf{p}_{\text{left
ankle}}^{\text{img}}-\mathbf{p}_{\text{left knee}}^{\text{img}}\\\
&\hat{\mathbf{p}}_{\text{left}_{2}}=\mathbf{p}_{\text{left
hip}}^{\text{img}}-\mathbf{p}_{\text{left knee}}^{\text{img}}\\\
&\hat{\mathbf{p}}_{\text{left}_{3}}=\mathbf{p}_{\text{left
shoulder}}^{\text{img}}-\mathbf{p}_{\text{left knee}}^{\text{img}}\\\
&L_{\text{right}}=|\frac{\hat{\mathbf{p}}_{\text{right}_{1}}\cdot\hat{\mathbf{p}}_{\text{right}_{2}}}{\|\hat{\mathbf{p}}_{\text{right}_{1}}\|\|\hat{\mathbf{p}}_{\text{right}_{2}}\|}-\pi|+|\frac{\hat{\mathbf{p}}_{\text{right}_{2}}\cdot\hat{\mathbf{p}}_{\text{right}_{3}}}{\|\hat{\mathbf{p}}_{\text{right}_{2}}\|\|\hat{\mathbf{p}}_{\text{right}_{3}}\|}-\pi|\\\
&L_{\text{left}}=|\frac{\hat{\mathbf{p}}_{\text{left}_{1}}\cdot\hat{\mathbf{p}}_{\text{left}_{2}}}{\|\hat{\mathbf{p}}_{\text{left}_{1}}\|\|\hat{\mathbf{p}}_{\text{left}_{2}}\|}-\pi|+|\frac{\hat{\mathbf{p}}_{\text{left}_{2}}\cdot\hat{\mathbf{p}}_{\text{left}_{3}}}{\|\hat{\mathbf{p}}_{\text{left}_{2}}\|\|\hat{\mathbf{p}}_{\text{left}_{3}}\|}-\pi|\\\
&\text{Filter}(p^{\text{img}})=\text{min}(L_{\text{right}},L_{\text{left}})\end{split}$
(17)

##  IX 2D Rotation Search - Details

We compute our error using Equation 18, where we augment our detections
$\mathbf{p}^{\text{plane}}_{i}$ with the time step, which we notate as
$\hat{\mathbf{p}}^{\text{plane}}_{i}=(x,y,t)$ we define $c_{1}$ and $c_{2}$ as
camera 1 and camera 2 or the ref camera and the sync camera. We also define
$\hat{F}$, $\hat{P}_{c,j}$, and $\hat{K}_{c,j,p}$ to represent the sets of
indices of the frames, poses for view $c$ and frame $i$, and keypoints for
view $c$, frame $i$, and pose $p$.

$\begin{split}&L_{\text{min}}(\text{c}_{1},\text{c}_{2},i,k_{1})=\min_{k_{2}\in\hat{P}_{c_{2},i}}\|\hat{\mathbf{p}}^{\text{plane}}_{\text{c}_{1},i,k_{1},\text{ankle}}-\hat{\mathbf{p}}^{\text{plane}}_{\text{c}_{2},i,k_{2},\text{ankle}}\|\\\
&L_{\text{t}}(\text{c}_{1},\text{c}_{2})=\sum_{i\in\hat{F},k_{1}\in\hat{P}_{c_{1},i}}L_{\text{min}}(\text{c}_{1},\text{c}_{2},i,k_{1})\\\
&L_{\text{rot}}(\text{c}_{1},\text{c}_{2})=L_{\text{t}}(\text{c}_{1},\text{c}_{2})+L_{\text{t}}(\text{c}_{2},\text{c}_{1})\end{split}$
(18)

##  X Bundle Adjustment - Details

The terms in our bundle adjustment equation are the intersection error
$L_{\text{3D}}$, left and right joint symmetry error $L_{\text{left,right}}$,
the height error $L_{h}$, and the constraint that the ankle is on the plane
$L_{p}$. The intersection error 19 is simply the distance of the closest
points from the reprojection rays. The left and right joint symmetry error 20
constrains the right and left joints to have the same length. We also
experimented with the height error 21 that constrains the joints connecting
from the ankle to the neck, to sum up to the assumed height $h$ and the 22
constrain the foot to be on the plane by making the z component of the ankle
coordinates 0.

$L_{\text{3D}}^{\text{c}_{1},\text{c}_{2}}=\frac{\sum_{j\in\hat{F},p\in\hat{P}_{j},k\in\hat{K}_{j,p}}\lVert\mathbf{p}^{\text{world}}_{\text{c}_{1},j,p,k}-\mathbf{p}^{\text{world}}_{\text{c}_{2},j,p,k}\rVert_{2}}{|\hat{F}||\hat{P}||\hat{K}|}$
(19)

$L_{\text{left,right}}^{\text{c}_{1},\text{c}_{2}}=\frac{\sum_{j\in\hat{F},p\in\hat{P}_{j},k\in\hat{K}_{j,p}}|\lVert
J^{\text{world}}_{\text{c}_{1},j,p,k}\rVert_{2}-\lVert
J^{\text{world}}_{\text{c}_{2},j,p,k}\rVert_{2}|}{|\hat{F}||\hat{P}||\hat{K}|}$
(20)

$L_{\text{h}}^{\text{c}}=\frac{\sum_{j\in\hat{F},p\in\hat{P}_{j},k\in\hat{K}_{j,p}}\lVert
J^{\text{world}}_{\text{c}_{1},j,p,k}\rVert_{2}}{|\hat{F}||\hat{P}||\hat{K}|}$
(21)

$L_{\text{p}}^{\text{cam}}=\frac{\sum_{j\in\hat{F},p\in\hat{P}_{j}}\lVert\mathbf{p}^{\text{world}}(z)_{\text{c},j,p,\text{ankle}}\rVert}{|\hat{F}||\hat{P}||\hat{K}|}$
(22)

##  XI Experiments

###  XI-A Hyperparameters

We use the EPFL terrace2 as a validation set for our hyperparameters.

###  XI-B Terrace Calibration

Although the terrace sequence cameras are calibrated for the terrace sequence,
the dataset has two versions of the calibrations. In one of them, instead of
the usual intrinsic and extrinsic matrices, the calibration files contain 2
homographies representing the transformation to head plane and ankle plane.
The other one contains intrinsic extrinsic calibrations using the Tsai
calibration method [35], however, the calibrations are based on a different-
sized image than the ones in the dataset. To rectify this, we use the
homographies to create a virtual checkboard on the ground plane as well as the
image dimension in the data and use opencv to compute the camera matrix.

###  XI-C Temporal Synchronization Experiments

In this section, we show our results for temporal synchronization experiments
on Human3.6M for each subject. We report our results in Table VIII.

Subject | Shift (gt) |  | Shift
---
(gt focal)
| Shift
---
(pred focal)
S1 | 0 | 11.24 | 11.82
S1 | 50 | 54.10 | 54.25
S1 | 100 | 104.10 | 104.25
S1 | 150 | 153.72 | 154.28
S1 | 200 | 204.18 | 204.28
S5 | 0 | 12.84 | 14.997
S5 | 50 | 55.49 | 55.68
S5 | 100 | 105.38 | 105.48
S5 | 150 | 155.38 | 155.28
S5 | 200 | 205.25 | 204.97
S6 | 0 | 13.94 | 21.75
S6 | 50 | 56.08 | 60.03
S6 | 100 | 106.14 | 110.18
S6 | 150 | 156.41 | 166.31
S6 | 200 | 206.51 | 215.91
S7 | 0 | 14.14 | 13.22
S7 | 50 | 56.77 | 54.34
S7 | 100 | 106.90 | 104.28
S7 | 150 | 157.24 | 154.8
S7 | 200 | 206.93 | 204.90
S8 | 0 | 15.71 | 14.46
S8 | 50 | 58.82 | 58.42
S8 | 100 | 108.58 | 106.79
S8 | 150 | 158.82 | 158.67
S8 | 200 | 208.92 | 208.67
S9 | 0 | 3.37 | 10.73
S9 | 50 | 52.63 | 52.15
S9 | 100 | 101.06 | 102.31
S9 | 150 | 150.39 | 152.98
S9 | 200 | 201.20 | 204.02
S11 | 0 | 9.22 | 12.71
S11 | 50 | 57.17 | 54.16
S11 | 100 | 107.58 | 104.02
S11 | 150 | 159.12 | 154.18
S11 | 200 | 209.28 | 70.31
TABLE VIII: Temporal synchronization experiments. We report our results for
each subject for Human3.6M. GT shift represents the ground truth offset, while
GT focal represents running our method with ground truth focal length, and
Pred focal represents running our method with the focal length predicted by
our method. Results are in frame offsets.

##  XII Simulations

We conduct simulation experiments the same as [11], using the same or similar
specifications that were given in their simulation study. In our simulated
experiments, we seek to test the effects of measurement noise, height
variations, and number of people on our single view calibration algorithm,
without the effects of detection noise present from pose detectors. This also
allows us to get ground truth heights of the people, which we would not have
access to in any of the datasets. We generate a scene, as shown in Fig. 10,
with a random ground plane and random shoulder and ankle center detections. We
specify image dimensions 1920.0 by 1080.0 with focal length $f_{X}=960$ and
$f_{y}=540$, which corresponds to a 90 degree FOV. We run each experiment for
5000 trials.

Metrics.

  * •

$\mathbf{Focal}$ $\mathbf{error}$: the percent focal error is simply computed
by $100\cdot\frac{|f_{\text{gt}-f_{\text{pred}}}|}{f_{\text{gt}}}$. This is
represented in the tables as $f_{x}\%$ and $f_{y}\%$

  * •

$\mathbf{Normal}$ $\mathbf{error}$: The degree normal error is the angle
difference between the ground truth and predicted normal vectors, represented
in the table as $N(°)$.

  * •

$\mathbf{Position}$ $\mathbf{error}$: The position error is computed by taking
the taking the predicted Euclidean distance from the camera to the ground
plane $\rho_{\text{pred}}$, and computing
$\frac{|\rho_{\text{pred}}-\rho_{\text{gt}}|}{\rho_{\text{gt}}}$. This is
represented by $\rho\%$.

  * •

$\mathbf{Reconstuction}$ $\mathbf{error}$: The reconstruction error is the
percent error between Euclidean distance between the predicted 3D points and
the ground truth 3D points divided by the ground truth distance to the camera.
This is represented in the table by $X\%$

  * •

$\mathbf{Failure}$ $\mathbf{rate}$: Since this could lead to an unsolvable
linear system, we record the failure rate which is the percentage of noisy
systems that are unsolvable. This is represented as $\text{fail}\%$

##### Measurement Noise Trials

For these experiments, we fix the height to be 1.6 meters and use 3 pairs of
shoulder and ankle center positions. We add a zero mean Gaussian with varying
standard deviations to these generated positions. We solve our DLT equations
with a height of 1.6 meters. We record our results in Table IX. For these
trials, adding zero error results in an error for all the metrics that were
virtually 0. As the noise standard deviation grows, the error for all metrics
increases as expected.

##### Height Variation Trials

For these experiments, we fix the detection noise to be sampled from a zero
mean Gaussian with a standard deviation of 0.5 and we sample heights from a
Gaussian centered at 1.7 meters with varying standard deviations. In our
algorithm, we solve the DLT equations with a height equal to 1.7m. We report
our results in Table XI. As expected, as the standard deviations of the
heights increase, the worse the results become since our algorithm uses a
fixed height of 1.7 meters.

##### Number of People Trials

For these experiments, we fix the detection noise as in the height variation
trails, but we sample the height from a Gaussian centered at 1.7 meters with a
standard deviation of 0.1 meters. Within our DLT equations, we fix the height
to be 1.7 meters like the Gaussian mean. We vary the number of pairs of
shoulder and ankle center detection and record our results in Table X. As the
number of people increases, the system becomes more and more over determined
which reduces the resulting error.

| Simulation measurement noise std. in pixels
---|---
Error | 0.1 | 0.2 | 0.5 | 1.0 | 2.0 | 5.0
$f_{x}\%$ | 1.85 | 3.45 | 6.88 | 9.70 | 16.83 | 27.03
$f_{y}\%$ | 1.08 | 1.81 | 3.66 | 6.01 | 10.13 | 17.75
N (°) | 0.15 | 0.26 | 0.60 | 1.09 | 2.22 | 4.30
$\rho\%$ | 1.58 | 0.71 | 1.77 | 3.23 | 7.22 | 22.52
$X\%$ | 1.06 | 1.74 | 3.57 | 5.92 | 10.08 | 17.64
fail$\%$ | 0.82 | 1.52 | 3.4 | 5.5 | 8.04 | 17.36
TABLE IX: Measurement noise. We show the error from our calibration for varying measurement noise standard deviations. | Number of people
---|---
Error | 5 | 10 | 20 | 50 | 100
$f_{x}\%$ | 27.33 | 27.68 | 26.29 | 18.21 | 21.98
$f_{y}\%$ | 32.40 | 25.64 | 25.25 | 14.94 | 14.17
N (°) | 3.74 | 3.33 | 2.62 | 1.76 | 1.13
$\rho\%$ | 21.46 | 21.66 | 18.92 | 21.64 | 24.09
$X\%$ | 20.96 | 21.24 | 20.65 | 30.07 | 42.82
fail$\%$ | 15.98 | 12.2 | 8.8 | 5.72 | 3.78
TABLE X: Number of people. We show the error from our calibration for varying numbers of people | Std. of height in meters
---|---
Error | 0.05 | 0.1 | 0.15 | 0.2 | 0.25
$f_{x}\%$ | 7.00 | 8.75 | 10.36 | 10.71 | 13.22
$f_{y}\%$ | 3.64 | 4.258 | 5.27 | 6.04 | 6.48
N (°) | 0.58 | 0.63 | 0.82 | 0.87 | 1.05
$\rho\%$ | 6.98 | 10.388 | 13.29 | 16.99 | 21.43
$X\%$ | 6.69 | 10.85 | 14.60 | 18.048 | 21.54
fail$\%$ | 2.62 | 3.5 | 4.18 | 4.7 | 5.98
TABLE XI: Height. We show the error from our calibration for varying random
heights that are drawn from a Gaussian distribution with varying standard
deviations. Figure 10: Simulation. This shows an example of a scene generated
by our simulation where the green dots in the distance represent ankle and
shoulder center detections, and the blue lines represent the ground plane.

##  XIII Noise propagation from stage to stage

We conduct experiments to investigate the effects of noise on our pipeline on
the EPFL terrace sequence [13] on cameras terrace1-c0_avi, terrace1-c1_avi,
and terrace1-c2_avi. We run 4 sets of experiments where we take in as input a
height parameter and 2D detections, and then we monitor how noise added to
different stages of the pipeline propagates to the intermediate and final
outputs. In separate experiments, we add noise to the 2D detections, the focal
length, the normal vector, and the time synchronization.

##### Detection noise

For these experiments, we add random noise to the 2D detections sampled from a
Gaussian with mean zero and standard deviation 0, 5, 10, 15, 20, 25, 30, and
35 pixels. We run the experiment 5 times for each standard deviation and take
the median results. We plot the results as error curves in Fig. 13. What we
find is that from 0 to 15 pixels the results for each component are relatively
close to each other. However, for standard deviations larger than 15, the
error for each component increases significantly since the initial steps of
finding the focal length and time synchronization start to fail, resulting in
the later stages having unrecoverable errors. We can see that in the zero
noise case in Figure 11(a), we have a much larger amount of detections that
can be used for the single view calibration whereas with the higher noise
case, in Figure 11(c) and 12(a), fewer detections are available for
calibration because the additional noise causes some poses to appear to be
non-standing, which gets filtered out before our RANSAC step. The available
ones are mostly outliers in the RANSAC loop. For time synchronization, we can
see from Figure 12(a) and Figure 12(b) that as the error standard deviation
increases, the time curves become less defined and thus cause the time
synchronization step to become less accurate.

##### Focal Length Noise

For these experiments, we sampled random noise from a Gaussian with mean zero
and standard deviation between 0 and 200 pixels to the ground truth focal
length while using the ground truth normal vector. We run the experiments five
times per standard deviation and take the median result. We find that larger
magnitudes of offsets result in worse rotation, translation, and temporal
synchronization results for the following steps. From Fig. 14 we see that the
errors do not increase significantly until we use standard deviations beyond
80 pixels. Since the average ground truth focal length is about 437 pixels, we
show that even if our predicted focal length is off by as much as 20 percent,
our system can still produce reasonable results for the other parameters
estimated by the subsequent steps.

##### Normal Noise

For these experiments, we add random noise to the ground truth normal vector
sampled from a Gaussian with mean zero and standard deviation between 0 and
0.5. After adding the noise, we normalize the vector again, run the test for
each standard deviation 5 times, and take the median result. Fig. 15 shows
that the higher the offset we add, the worse the rotation, translation, and
temporal synchronization results get for the following steps. We see that the
greatest increase in error occurs at 0.15 standard deviations.

##### Time Synchronization Noise

For these experiments, we add offsets to the temporal synchronization in a
range from -100 to 100 frames while using ground truth focal length and normal
vector. Fig. 16 shows the results. The higher the magnitude of offsets we add,
the worse the rotation, translation, and temporal synchronization results get
for the following steps. We report metrics for all the subsequent steps,
enabling a conclusion of how susceptible earlier and later modules are to
noise and how this noise propagates through the pipeline.

##### Rotation Noise

In order to test the effects of the rotation noise, we use Human3.6M to
compare the normalized mean per joint position error (NMPJPE) since Human3.6M
provides ground truth 3D keypoints. For these experiments, we add random noise
to the ground truth rotation matrices sampled from a Gaussian with mean zero
and standard deviation between 0 and 20 degrees. We run the test for each
standard deviation 5 times and take the median result. Fig. 18 shows that the
higher the standard deviation of the rotation noise, the worse the NMPJPE. We
see that the error curve is relatively flat from 1 to 3 but starts climbing
once we get past 5. However, our qualitative results on subject 1 on the
walking sequence are shown in 17, we see that the 0 noise case and the
5-degree standard deviation noise look almost identical.

(a) No noise

(b) Detection noise $\mu$=0 $\sigma$=15 pixels.

(c) Detection noise $\mu$=0 $\sigma$=35 pixels.

Figure 11: Terrace detection noise visualization. Experiments were conducted
on terrace1-c2 with noise sampled from a Gaussian distribution. Here, the
green lines represent inlier detections and the red lines represent the
outlier detections.

(a) Time synchronization in the 0 detection noise case.

(b) Time synchronization in the 10-pixel std detection noise case. In the zero
noise case, we see well-defined curves, however, with noise, outliers cause
them to be less well-defined.

Figure 12: Terrace detection noise level visualization.

(a) Percent focal length error.

(b) Temporal synchronization error

(c) Degree rotation angle error.

(d) Normalized position error.

Figure 13: Terrace detection noise error curves. We plot the error on all of
our metrics compared to the standard deviation of the noise added to the 2D
detections. Note that the errors are non-zero without added noise (std=0) due
to the original estimation noise.

(a) Temporal synchronization error.

(b) Degree rotation angle error.

(c) Normalized position error.

Figure 14: Terrace focal noise error curves. We plot the error on all of our
metrics compared to the offset added to the focal length.

(a) Temporal synchronization error.

(b) Degree rotation angle error.

(c) Normalized position error.

Figure 15: Terrace normal noise error curves. We plot the error on all of our
metrics compared to the standard deviation of the noise added to the normal
vector.

(a) Degree rotation angle error.

(b) Normalized position error.

Figure 16: Terrace temporal synchronization noise error curves. We plot the
error on all of our metrics compared to the offset added to the temporal
synchronization.

(a) 0 degree rotation noise.

(b) 5 degree rotation noise.

Figure 17: Reconstructed poses for rotational noise. Using Subject 1 from
Human3.6M on the walking sequence. The green pose represents the ground truth
3D and the red pose represents the reconstructed pose. Figure 18: NMPJPE error
on the Human3.6M dataset.