11institutetext:  Brown University, Division of Engineering, Providence RI
02912, USA. Draft date:
11email<EMAIL_ADDRESS>
# Multiview Differential Geometry of Curves

Ricardo Fabbri  Benjamin B. Kimia

###### Abstract

The field of multiple view geometry has seen tremendous progress in
reconstruction and calibration due to methods for extracting reliable point
features and key developments in projective geometry. Point features, however,
are not available in certain applications and result in unstructured point
cloud reconstructions. General image curves provide a complementary feature
when keypoints are scarce, and result in 3D curve geometry, but face
challenges not addressed by the usual projective geometry of points and
algebraic curves. We address these challenges by laying the theoretical
foundations of a framework based on the differential geometry of general
curves, including stationary curves, occluding contours, and non-rigid curves,
aiming at stereo correspondence, camera estimation (including calibration,
pose, and multiview epipolar geometry), and 3D reconstruction given measured
image curves. By gathering previous results into a cohesive theory, novel
results were made possible, yielding three contributions. First we derive the
differential geometry of an image curve (tangent, curvature, curvature
derivative) from that of the underlying space curve (tangent, curvature,
curvature derivative, torsion). Second, we derive the differential geometry of
a space curve from that of two corresponding image curves. Third, the
differential motion of an image curve is derived from camera motion and the
differential geometry and motion of the space curve. The availability of such
a theory enables novel curve-based multiview reconstruction and camera
estimation systems to augment existing point-based approaches. This theory has
been used to reconstruct a “3D curve sketch”, to determine camera pose from
local curve geometry, and tracking; other developments are underway.

###### Keywords:

Structure from Motion Multiview Stereo Torsion Non-Rigid Space Curves

††journal: International Journal of Computer Vision

##  1 Introduction

The automated estimation of camera parameters and the 3D reconstruction from
multiple views is a fundamental problem in computer vision. These tasks rely
on correspondence of image structure commonly in the form of keypoints,
although dense patches and curves have also been used.

Figure 1:  (a) Keypoint-based approaches give a sparse point cloud
reconstruction Agarwal et al (2009); Heinly et al (2015) which can be made
richer, sharper, and denser with curve structure. (b) Wide baseline views not
sharing interest points often share curve structure. (c,d) Not enough points
matching views of homogenous objects with sufficient curve structure. (e) Each
moving object or non-rigid object requires its own set of features, but may
not have enough texture.

Keypoint-based methods extract point features designed be stable with view
variations. They satisfy certain local conditions in the spatial and scale
dimensions Mikolajczyk and Schmid (2004); Harris and Stephens (1988); Moravec
(1977); Lowe (2004), and are attributed with a local description of the image
Mikolajczyk and Schmid (2005). While many of these points are not stable with
view changes in that they disappear/appear or vary abruptly, many are stable
enough for matching with ransac Fischler and Bolles (1981) to initialize
camera models for bundle adjustment Hartley and Zisserman (2000); Pollefeys et
al (2004); Agarwal et al (2009); Heinly et al (2015); Diskin and Asari (2015).

A major drawback of interest points is their sparsity compared to the curves
and surfaces composing the scene, producing a cloud of 3D points where
geometric structure is not explicit, Figure 1(a). This is less of a problem
for camera estimation Agarwal et al (2009), but in applications such as
architecture, industrial design, object recognition, and robotic manipulation,
explicit 3D geometry is required. Moreover, meshing unstructured point clouds
produces oversmoothing Kazhdan et al (2006); Furukawa and Ponce (2010). These
techniques are therefore inadequate for man-made environments Simoes et al
(2012) and objects such as cars Shinozuka and Saito (2014), non-Lambertian
surfaces such as that of the sea, appearance variation due to changing weather
Baatz et al (2012), and wide baseline Moreels and Perona (2007), Figure 1(b).
We claim that by using even _some_ curve information the 3D reconstruction can
be made more structually rich, sharper and less sparse.

The use of keypoints requires an abundance of features surviving the
variations between views. While this occurs for many scenes, in many others
this is not the case, such as $(i)$ homogeneous regions from man-made objects,
Figure 1(c,d); $(ii)$ moving objects require their own set of features which
are too few without sufficient texture, Figure 1(e); $(iii)$ non-rigid objects
require a rich set of features per local patch, Figure 1(f). While curve
features, like keypoints, may not be abundant the interior of highly
homogeneous regions or in nonrigid or small moving objects, _curves rely on
texture to a much lesser extent_ , as shown in Figure 1(c,d,e); the boundary
of homogeneous regions are good cues (often the _only_ cues), enabling
extracting information structure and motion. In all these situations, there
may be sufficient curvilinear structure, Figure 1, motivating augmenting the
use of interest points with curves.

Pixel-based multiview stereo relies on matching intensities across views,
resulting in a dense point cloud or a mesh reconstruction. This produces
detailed 3D reconstructions of objects imaged under controlled conditions by a
large number of precisely calibrated cameras Furukawa and Ponce (2007);
Habbecke and Kobbelt (2007); Hernández Esteban and Schmitt (2004); Goesele et
al (2007); Seitz et al (2006); Calakli et al (2012); Restrepo et al (2014).
However, there are a number of limitations: they typically assume that the
scene consists of a single object or that objects are of a specific type, such
as a building; they often require accurate camera calibration and operate
under controlled acquisition; and they need to be initialized by the visual
hull of the object or a bounded 3D voxel volume, compromising aplicability for
general scenery.

Curve-based multiview methods can be divided into three categories: $(i)$
convex hull construction, $(ii)$ occluding contour reconstruction, and $(iii)$
use of differential geometry in binocular and trinocular stereo. First, when
many views are available around an object, a visual hull has been constructed
from silhouette curves and then evolved to optimize photometric constraints
while constraining the projection to the silhouettes. The drawbacks are
similar to those of pixel-based multiview stereo. Second, the occluding
contours extracted from frames of a video have been used to reconstruct a
local surface model given the camera parameters for each frame. These methods
require highly controlled acquisition and image curves that are easy to
segment and track. In addition, since only silhouettes are used, internal
surface variations which may not map to apparent contours in any view will not
be captured, e.g., surface folds of a sculpture.

Third, some methods employ curve differential geometry in correlating
structure across views. Complete 3D reconstruction pipelines based on straight
lines Lebeda et al (2014); Zhang (2013); Fathi et al (2015), algebraic and
general curve features Teney and Piater (2012); Litvinov et al (2012); Fabbri
and Kimia (2010); Fabbri et al (2012); Pötsch and Pinz (2011) have been
proposed. The compact curve-based 3D representation that has found demand in
seveal tasks: fast recognition of general 3D scenery Pötsch and Pinz (2011),
efficient transmission of general 3D scenes, scene understanding and modeling
by reasoning at junctions Mattingly et al (2015), consistent non-
photorealistic rendering from video Chen and Klette (2014), modeling of
branching structures, to name a few Rao et al (2012); Kowdle et al (2012);
Wang et al (2014).

Related work: Differential geometry does not provide hard constraints for
matching in static binocular stereo, as known for tangents and curvatures
Robert and Faugeras (1991), and shown here for higher order. Heuristics have
been employed in short baseline to limit orientation difference Arnold and
Binford (1980); Grimson (1981); Sherman and Peleg (1990), to match appearance
via locally planar approximations Schmid and Zisserman (2000), or to require
3D curve reconstructions arising from two putative correspondence pairs to
have minumum torsion Li and Zucker (2003). When each stereo camera provides a
video and the scene (or stereo head) moves rigidly, differential geometry
provides a hard constraint Faugeras and Papadopoulo (1993); Papadopoulo
(1996); Faugeras and Papadopoulo (1992). Differential geometry is more
directly useful in trinocular and multiview stereo, as pioneered by Ayache and
Lustman (1987), due to the constraint that corresponding pairs of points and
tangents from two views uniquely determine a point and tangent in a third to
match line segments obtained from edge linking Ayache and Lustman (1987);
Spetsakis and Aloimonos (1991); Shashua (1994); Hartley (1995). Robert and
Faugeras (1991) extended this to include curvature: 3D curvature and normal
can be reconstructed from 2D curvatures at two views, determining the
curvature at a third. The use of curvature improved reconstruction precision
and density, with heuristics such as the ordering constraint Ohta and Kanade
(1985). Schmid and Zisserman (2000) derived multiview curvature transfer when
only the trifocal tensor is available, by a projective-geometric approach to
the osculating circle as a conic.

Curves have also been employed for camera estimation using the concept of
epipolar tangencies: corresponding epipolar lines are tangent to a curve at
corresponding points Cipolla and Giblin (1999); Astrom et al (1999); Astrom
and Kahl (1999); Kahl and Heyden (1998); Porrill and Pollard (1991); Kaminski
and Shashua (2004); Berthilsson et al (2001); Wong et al (2001); Mendonça et
al (2001); Wong and Cipolla (2004); Furukawa et al (2006); Hernandez et al
(2007); Cipolla et al (1995); Reyes and Bayro Corrochano (2005); Sinha et al
(2004). This is used to capture epipolar geometry or relative pose.

Curve-Based Multiview Geometry: What would be desirable is a generally
applicable framework, e.g., a handheld video acquiring images around objects
or a set of cameras monitoring a scene, where image curve structure can be
used to estimate camera parameters and reconstruct a 3D curve sketch on which
a surface can be tautly stretched like a tent on a metallic scaffold. This
paper provides the _mathematical foundation_ for this curve-based approach.
Image curve fragments are attractive because they have good localization, have
greater invariance than interest points to changes in illumination, are stable
over a greater range of baselines, and are denser than interest points.
Moreover, for the special case of occluding contours, dense 3D surface patch
reconstructions are available. The notion that image curves contain much of
the image information is supported by recent studies Koenderink et al (2013);
Zucker (2014); Kunsberg and Zucker (2014); Cole et al (2009).

This paper develops the theoretical foundations for using the differential
geometry of image curve structure as a complementary alternative to interest
points. This paper is organized along the lines of these questions: $(i)$ How
does the differential geometry of a space curve map to the differential
geometry of the image curve it projects to? $(ii)$ How can the differential
geometry of a space curve be reconstructed from that of two corresponding
image curves? $(iii)$ How does the differential geometry of an image curve
evolve under camera motion? Section 2 establishes notation for image and space
curves, camera projection and motion, and discusses the distinction between
_stationary_ and _non-stationary_ 3D contours. Section 3 relates the
differential geometry of image curves, i.e., tangent, curvature, and curvature
derivative from the differential geometry of the space curves they arise from,
i.e., tangent and normal, curvature, torsion, and curvature derivatives.
Section 4 derives the differential geometry of a space curve at a point from
that at two corresponding image curve points showing the key result that the
ratio of parametrization speeds is an intrinsic quantity. The key new result
is the reconstruction of torsion and curvature derivative, given corresponding
differential geometry in two views. Section 5 considers differential camera
motion and relates the differential geometry of a space curve to that of the
image and camera motion. Results are provided concerning image velocities and
accelerations with respect to time for different types of curves; in
particular, distinguishing apparent and stationary contours requires second-
order time derivatives Cipolla and Giblin (1999). We study the spatial
variation of the image velocity field along curves, which can be useful for
exploiting neighborhood consistency of velocity fields along curves. The main
new result generalizes a fundamental curve-based differential structure from
motion equation Papadopoulo and Faugeras (1996); Papadopoulo (1996) to
occluding contours.

_This paper integrates the above results under the umbrella of a unified
formulation and completes missing relationships._ As a generalized framework,
it is expected to serve as reference for research relating local properties of
general curves and surfaces to those of cameras and images. Much of this has
already been done, but a considerable amount has not, as mentioned earlier,
and most results are scattered in the literature. This theoretical paper has
been the foundation of practical work already reported on reconstruction and
camera estimation systems as follows. First, the pipeline for the
reconstruction of a 3D Curve Sketch from image fragments in numerous views
Fabbri and Kimia (2010); Usumezbas et al (2016) relies on the results on 3D
reconstruction and projection of differential geometry reported in this paper,
and a future extension of this pipeline would require most results in this
paper. Second, a recent practical algorithm for pose estimation based on
differential geometry of curves Fabbri et al (2012) relies on the theory
reported in this paper, treating the camera pose as unknowns and using
differential geometry to solve for them, cf. ensuing efforts by Kuang and
Åström (2013); Kuang et al (2014). Third, work on differential camera motion
estimation from families of curves based on the present work has been explored
by Jain (2009); Jain et al (2007b, a). These works are currently under intense
development in order to build a complete structure from motion pipeline based
on curves, which would use the majority of the results described in this
paper, including analogous results for multiview surface differential geometry
that are under development.

##  2 Notation and Formulation

###  2.1 Differential Geometry of Curves

For our purposes, a 3D space curve $\Gamma$ is a smooth map
$S\mapsto\boldsymbol{\Gamma}^{w}(S)$ of class $C^{\infty}$ from an interval of
$\mathbb{R}$ to $\mathbb{R}^{3}$, where $S$ is an arbitrary parameter,
$\tilde{S}$ is the arc-length parameter, and the superscript $w$ denotes the
world coordinates. The local Frenet frame of $\Gamma$ in world coordinates is
defined by the unit vectors tangent $\boldsymbol{T}^{w}$, normal
$\boldsymbol{N}^{w}$, binormal $\boldsymbol{B}^{w}$; $G$ is speed of
parametrization, curvature $K$, and torsion $\tau$. Similarly, a 2D curve
$\gamma$ is a map $s\mapsto\boldsymbol{\gamma}(s)$ of class $C^{\infty}$ from
an interval of $\mathbb{R}$ to $\mathbb{R}^{2}$, where $s$ is an arbitrary
parameter, $\tilde{s}$ is arc-length, $g$ is speed of parametrization,
$\boldsymbol{t}$ is (unit) tangent, $\boldsymbol{n}$ is (unit) normal,
$\kappa$ is curvature, and $\kappa^{\prime}$ is curvature derivative. We will
be concerned with regular curves, so that $G\neq 0$ and $g\neq 0$ unless
otherwise stated. By classical differential geometry do Carmo (1976), we have

$\left\\{\begin{aligned} G&=\|\boldsymbol{\Gamma}^{w^{\prime}}\|\\\
\boldsymbol{T}^{w}&=\frac{\boldsymbol{\Gamma}^{w^{\prime}}}{G}&\hskip
8.53581pt\boldsymbol{N}^{w}&=\frac{\boldsymbol{T}^{w^{\prime}}}{\|\boldsymbol{T}^{w^{\prime}}\|}&\hskip
8.53581pt\boldsymbol{B}^{w}&=\boldsymbol{T}^{w}\times\boldsymbol{N}^{w}\\\
K&=\frac{\|\boldsymbol{T}^{w^{\prime}}\|}{G}&\dot{K}&=\frac{K^{\prime}}{G}&\tau&=\frac{-\boldsymbol{B}^{w^{\prime}}\cdot\boldsymbol{N}^{w}}{G},\end{aligned}\right.\
\ \ \begin{bmatrix}\boldsymbol{T}^{w^{\prime}}\\\
\boldsymbol{N}^{w^{\prime}}\\\
\boldsymbol{B}^{w^{\prime}}\end{bmatrix}=G\begin{bmatrix}\hfill 0\,&\hfill
K&\,\,\,0\ \ \\\ -K&\hfill 0\,&\,\,\,\tau\ \ \\\ \hfill 0\,&-\tau\,&\,\,\,0\ \
\end{bmatrix}\begin{bmatrix}\boldsymbol{T}^{w}\\\ \boldsymbol{N}^{w}\\\
\boldsymbol{B}^{w}\end{bmatrix},$ (2.1)

and

$\displaystyle g$ $\displaystyle=\|\boldsymbol{\gamma}^{\prime}\|,$
$\displaystyle\hskip 19.91692pt\boldsymbol{t}$
$\displaystyle=\frac{\boldsymbol{\gamma}^{\prime}}{g},$ $\displaystyle\hskip
19.91692pt\boldsymbol{n}$ $\displaystyle=\boldsymbol{t}^{\perp},$
$\displaystyle\hskip 19.91692pt\kappa$
$\displaystyle=\frac{\boldsymbol{t}^{\prime}\cdot\boldsymbol{n}}{g},$
$\displaystyle\hskip 19.91692pt\dot{\kappa}$
$\displaystyle=\frac{\kappa^{\prime}}{g},$ (2.2)

where prime “$\,{}^{\prime}$” denotes differentiation with respect to an
arbitrary spatial parameter ($S$ or $s$). We use dot “$\dot{\ }$” to denote
differentiation with respect to arc-length ($\tilde{S}$ or $\tilde{s}$) only
when an entity clearly belongs to either a space or an image curve. The matrix
equations on the right of (2.1) are the Frenet equations. Note that both the
curvature derivatives $\dot{K}$ and $\dot{\kappa}$ are intrinsic quantities.

###  2.2 Perspective Projection

Figure 2: The perspective projection of a space curve in (a) one view, and (b)
$n$ views.

The projection of a 3D space curve $\Gamma$ into a 2D image curve $\gamma$ is
illustrated by Figure 2, where the world coordinate system is centered at $O$
with basis vectors
$\\{\boldsymbol{e}_{1}^{w},\boldsymbol{e}_{2}^{w},\boldsymbol{e}_{3}^{w}\\}$.
The _camera coordinate system_ is centered at $\boldsymbol{c}$ with basis
vectors $\\{\boldsymbol{e}_{1},\boldsymbol{e}_{2},\boldsymbol{e}_{3}\\}$. A
generic way of referring to individual coordinates is by means of the specific
subscripts $x,y$ and $z$ attached to a symbol, i.e.,
$\boldsymbol{v}=[v_{x},v_{y},v_{z}]^{\top}$ for any vector $\boldsymbol{v}$;
other subscripts denote partial differentiation. When describing coordinates
in the camera coordinate system we drop the $w$ superscript, e.g.,
$\boldsymbol{\Gamma}$ versus $\boldsymbol{\Gamma}^{w}$, which are related by

$\boldsymbol{\Gamma}=\boldsymbol{\mathcal{R}}(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})=\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}+\boldsymbol{\mathcal{T}},$
(2.3)

where $\boldsymbol{\mathcal{R}}$ is a rotation and
$\boldsymbol{\mathcal{T}}=-\boldsymbol{\mathcal{R}}\boldsymbol{c}$ denotes the
world coordinate origin in the camera coordinate system.

The projection of a 3D point $\boldsymbol{\Gamma}=[x,\,y,\,z]^{\top}$ onto the
image plane at $z=1$ is the point
$\boldsymbol{\gamma}=[\xi,\,\eta,\,1]^{\top}$ related by

$\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}\,\,\,\,\text{or}\,\,\,\,[x,\,y,\,z]^{\top}=[0pt\xi,\,0pt\eta,\,0pt]^{\top},$
(2.4)

where we say that $\boldsymbol{\gamma}$ is in normalized image coordinates
(focal distance is normalized to 1), and the depth is
$0pt=z=\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}$ from the third coordinate
equation. Observe that image points are treated as 3D points with $z=1$. Thus,
we can write

$\displaystyle\boldsymbol{\gamma}$
$\displaystyle=\frac{\boldsymbol{\Gamma}}{0pt}.$ (2.5)

We note that $\boldsymbol{e}_{3}^{\top}\boldsymbol{\gamma}^{(i)}=0$ and
$\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{(i)}=0pt^{(i)}$, where
$\boldsymbol{\gamma}^{(i)}$ is the $i^{th}$ derivative of
$\boldsymbol{\gamma}$ with respect to an arbitrary parameter, for any positive
integer $i$. Specifically,

$0pt=z,\qquad 0pt^{\prime}=GT_{z},\qquad
0pt^{\prime\prime}=G^{\prime}T_{z}+G^{2}KN_{z}.$ (2.6)

It is interesting to note that at near/far points of the curve, i.e.,
$0pt^{\prime}=0$, $T_{z}=0$.

In practice, normalized image coordinates
$\boldsymbol{\gamma}=[\xi,\,\eta,\,1]^{\top}$ are described in terms of image
pixel coordinates $\boldsymbol{\gamma}_{im}=[x_{im},\,y_{im},\,1]^{\top}$
through the intrinsic parameter matrix $\boldsymbol{\mathcal{K}}_{im}$
according to

$\boldsymbol{\gamma}_{im}=\boldsymbol{\mathcal{K}}_{im}\boldsymbol{\gamma},\ \
\ \ \ \
\boldsymbol{\mathcal{K}}_{im}=\begin{bmatrix}\alpha_{\xi}&\sigma&\xi_{o}\\\
0&\alpha_{\eta}&\eta_{o}\\\ 0&0&1\end{bmatrix},$ (2.7)

where as usual $\xi_{o}$ and $\eta_{o}$ are the principal points, $\sigma$ is
skew, and $\alpha_{\xi}$ and $\alpha_{\eta}$ are given by the focal length
divided by the width and height of a pixel in world units, respectively.

###  2.3 Discrete and Continuous Sets of Views

Two scenarios are considered. The first scenario consists of a _discrete set
of views_ where a set of $n$ pinhole cameras observe a scene as shown in
Figure 2, with the last subscript in the symbols indentifying the camera,
e.g., $\boldsymbol{\gamma}_{i}$ denotes an image point in the $i^{th}$ camera,
and $\boldsymbol{e}_{3,i}$ denotes $\boldsymbol{e}_{3}$ in the $i^{th}$ view.
The second scenario consists of a _continuous set of views_ from a
continuously moving camera observing a space curve which may itself be moving,
$\boldsymbol{\Gamma}^{w}(S,t)=[x^{w}(S,t),\,y^{w}(S,t),\,z^{w}(S,t)]^{\top}$,
where $S$ is the parameter along the curve and $t$ is time, described in the
camera coordinate system associated with time $t$ as
$\boldsymbol{\Gamma}(S,t)=\left[x(S,t),\,y(S,t),\,z(S,t)\right]^{\top}$,
Figure 3. For simplicity, we often omit the parameters $S$ or $t$.

Figure 3: Multiview formulation of continuous camera motion and a possibly
moving contour.

Let the camera position over time (_camera orbit_) be described by the space
curve $\boldsymbol{c}(t)$ and the camera orientation by a rotation matrix
$\boldsymbol{\mathcal{R}}(t)$. For simplicity, and without loss of generality,
we take the camera coordinate system at $t=0$ to be the world coordinate
system, i.e., $\boldsymbol{c}(0)=0$, $\boldsymbol{\mathcal{T}}(0)=0$, and
$\boldsymbol{\mathcal{R}}(0)=\text{\emph{I}}$, where I is the identity matrix.
Also, a stationary point can be modeled in this notation by making
$\boldsymbol{\Gamma}^{w}(t)=\boldsymbol{\Gamma}^{w}(0)=\boldsymbol{\Gamma}_{0}$.

A differential camera motion model using time derivatives of
$\boldsymbol{\mathcal{R}}(t)$ and $\boldsymbol{\mathcal{T}}(t)$ can be used to
relate frames in a small time interval. Since
$\boldsymbol{\mathcal{R}}\boldsymbol{\mathcal{R}}^{\top}=\text{\emph{I}}$,

$\frac{d\boldsymbol{\mathcal{R}}}{dt}\boldsymbol{\mathcal{R}}^{\top}+\boldsymbol{\mathcal{R}}\frac{d\boldsymbol{\mathcal{R}}}{dt}^{\top}=0,$
(2.8)

which implies that
${\mathbit{\Omega}}_{\times}\doteq\frac{d\boldsymbol{\mathcal{R}}}{dt}\boldsymbol{\mathcal{R}}^{\top}$
is a skew-symmetric matrix, explicitly written as

${\mathbit{\Omega}}_{\times}=\begin{bmatrix}0&-\Omega_{z}&\Omega_{y}\\\
\Omega_{z}&0&-\Omega_{x}\\\ -\Omega_{y}&\Omega_{x}&0\end{bmatrix},$ (2.9)

so that
$\frac{d\boldsymbol{\mathcal{R}}}{dt}={\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{R}}$.
Denote
$\mathbit{\Omega}=\left[\Omega_{x},\,\Omega_{y},\,\Omega_{z}\right]^{\top}$ as
a vector form characterization of ${\mathbit{\Omega}}_{\times}$. Similarly,
the second-derivative of $\boldsymbol{\mathcal{R}}(t)$ is represented by only
three additional numbers $\frac{d{\mathbit{\Omega}}_{\times}}{dt}$, so that

$\frac{d^{2}\boldsymbol{\mathcal{R}}}{dt^{2}}=\frac{d{\mathbit{\Omega}}_{\times}}{dt}\boldsymbol{\mathcal{R}}+{\mathbit{\Omega}}_{\times}\frac{d\boldsymbol{\mathcal{R}}}{dt}=\frac{d{\mathbit{\Omega}}_{\times}}{dt}\boldsymbol{\mathcal{R}}+{\mathbit{\Omega}}_{\times}^{2}\boldsymbol{\mathcal{R}}.$
(2.10)

Thus, a second-order Taylor approximation of the camera rotation matrix using
$R(0)=\text{\emph{I}}$ is

$\boldsymbol{\mathcal{R}}(t)\approx\text{\emph{I}}+{\mathbit{\Omega}}_{\times}(0)t+\frac{1}{2}\left[\frac{d{\mathbit{\Omega}}_{\times}}{dt}(0)+{\mathbit{\Omega}}_{\times}^{2}(0)\right]t^{2}.$
(2.11)

Similarly, the camera translation can be described by a differential model

$\mathbit{V}(t)\doteq\frac{d\boldsymbol{\mathcal{T}}}{dt}(t)=-{\mathbit{\Omega}}_{\times}(t)\boldsymbol{\mathcal{R}}(t)\boldsymbol{c}(t)-\boldsymbol{\mathcal{R}}(t)\frac{d\boldsymbol{c}}{dt}(t),\qquad\qquad\mathbit{V}(0)=-\frac{d\boldsymbol{c}}{dt}(0),$
(2.12)

and

$\frac{d\mathbit{V}}{dt}(t)=\frac{d^{2}\boldsymbol{\mathcal{T}}}{dt^{2}}(t)=-\frac{d^{2}\boldsymbol{\mathcal{R}}}{dt^{2}}(t)\boldsymbol{c}(t)-2\frac{d\boldsymbol{\mathcal{R}}}{dt}(t)\frac{d\boldsymbol{c}}{dt}(t)-\boldsymbol{\mathcal{R}}(t)\frac{d^{2}\boldsymbol{c}}{dt^{2}}(t),$
(2.13)

which at $t=0$ gives
$\frac{d\mathbit{V}}{dt}(0)=-2{\mathbit{\Omega}}_{\times}(0)\frac{d\boldsymbol{c}}{dt}(0)-\frac{d^{2}\boldsymbol{c}}{dt^{2}}(0)$.

The choice of whether to adopt the Taylor approximation of $\boldsymbol{c}(t)$
or $\boldsymbol{\mathcal{T}}(t)$ as primary is entirely dependent in which
domain the higher derivatives are expected to diminish, giving

$\displaystyle\boldsymbol{\mathcal{T}}(t)\approx\mathbit{V}(0)\,t+\frac{1}{2}\mathbit{V}_{t}(0)\,t^{2},\qquad\boldsymbol{c}(t)\approx-\mathbit{V}(0)t+\frac{1}{2}\left[-\mathbit{V}_{t}(0)+2{\mathbit{\Omega}}_{\times}(0)\mathbit{V}(0)\right]t^{2}.$
(2.14)

Symbol | Description | Symbol | Description
---|---|---|---
$\boldsymbol{\Gamma}^{w}$ | 3D point in the world coordinate system | $\boldsymbol{t}$ |  Image curve tangent $\boldsymbol{t}=\boldsymbol{\gamma}^{\prime}/g$
$\boldsymbol{\Gamma}$ |  3D point in the camera coord. syst. $\boldsymbol{\Gamma}=\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}+\boldsymbol{\mathcal{T}}$ | $\boldsymbol{n}$ |  Image curve normal $\boldsymbol{n}=\boldsymbol{t}^{\perp}$
$\boldsymbol{\mathcal{R}}$ | Rotation matrix: world to camera coordinates | $\kappa$ |  Curvature of the image curve $g\kappa\boldsymbol{n}=\boldsymbol{t}^{\prime}$
$\boldsymbol{\mathcal{T}}$ |  Translation vector: world to camera coord. $\boldsymbol{\mathcal{T}}=-R\boldsymbol{c}$ |  $S$, $\tilde{S}$ | Space curve arbitrary parameter & arclength, resp.
$\boldsymbol{c}$ | The camera center | $G$ |  Space curve speed of parametrization $G=\|\boldsymbol{\Gamma}^{\prime}\|$
${\mathbit{\Omega}}_{\times}$ | $\frac{d\boldsymbol{\mathcal{R}}}{dt}={\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{R}}$ |  $\boldsymbol{T}$, $\boldsymbol{T}^{w}$ | Space curve tangent camera & world coord., resp.
$\mathbit{\Omega}$ |  Vector form of the 3 entries of ${\mathbit{\Omega}}_{\times}$ |  $\boldsymbol{N}$, $\boldsymbol{N}^{w}$ | Space curve normal: camera & world coord., resp.
$\mathbit{V}$ |  $\mathbit{V}=\frac{d\boldsymbol{\mathcal{T}}}{dt}={\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}}-\boldsymbol{\mathcal{R}}\boldsymbol{c}_{t}$, also $\mathbit{V}=[V_{x},\,V_{y},\,V_{z}]^{\top}$ |  $\boldsymbol{B}$, $\boldsymbol{B}^{w}$ | Space curve binormal: camera & world coord., resp.
$0pt$ |  Depth of image point $\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}$ |  $\boldsymbol{e}_{1}$, $\boldsymbol{e}_{2}$, $\boldsymbol{e}_{3}$ | Basis vectors of the camera coordinate system
$\boldsymbol{\gamma}$ | 2D point in normalized image coordinates |  $\boldsymbol{e}_{1}^{w}$, $\boldsymbol{e}_{2}^{w}$, $\boldsymbol{e}_{3}^{w}$ | Basis vectors of the world coordinate system
$\boldsymbol{\gamma}_{im}$ | 2D point in pixel image coordinates | ′ |  Diff. with resp. to $S$ or $s$, depending on context
$s$, $\tilde{s}$ | Image curve arbitrary parameter & arclength, resp. | $\dot{\ }$ |  Diff. with resp. to arclength $\tilde{S}$ or $\tilde{s}$
$g$ |  Image curve speed of parametrization $g=\|\boldsymbol{\gamma}^{\prime}\|$ | $\theta$ |  The angle $\measuredangle(\boldsymbol{T},\boldsymbol{\gamma})$
$(u,v)$ |  Image velocities $\boldsymbol{\gamma}_{t}=[u,\,v,\,0]^{\top}$ |  |
Table 1: Notation.

###  2.4 Relating World and Camera-Centric Derivatives.

###### Proposition 1

The velocity of a 3D point $\boldsymbol{\Gamma}(t)$ in camera coordinates,
$\boldsymbol{\Gamma}_{t}(t)$, is related to its velocity in the world
coordinates $\boldsymbol{\Gamma}^{w}_{t}(t)$ by

$\displaystyle\boldsymbol{\Gamma}_{t}$
$\displaystyle={\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}+\mathbit{V}={\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}-\boldsymbol{\mathcal{R}}\boldsymbol{c}_{t},$
(2.15) $\displaystyle\boldsymbol{\Gamma}_{t}$
$\displaystyle={\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}_{0}+\mathbit{V}={\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}-\boldsymbol{\mathcal{R}}\boldsymbol{c}_{t},\
\ \ \ \text{for a fixed point,
$\boldsymbol{\Gamma}^{w}=\boldsymbol{\Gamma}_{0}$.}$ (2.16)

###### Proof

Differentiating Equation 2.3 with respect to time,

$\displaystyle\boldsymbol{\Gamma}_{t}$
$\displaystyle=\boldsymbol{\mathcal{R}}_{t}\boldsymbol{\Gamma}^{w}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}+\boldsymbol{\mathcal{T}}_{t}$
(2.17)
$\displaystyle={\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}+\mathbit{V}$
(2.18)
$\displaystyle={\mathbit{\Omega}}_{\times}(\boldsymbol{\Gamma}-\boldsymbol{\mathcal{T}})+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}+\mathbit{V}$
(2.19)
$\displaystyle={\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}+\mathbit{V}-{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}}.$
(2.20)

The result follows from using
$\boldsymbol{\mathcal{T}}=-\boldsymbol{\mathcal{R}}\boldsymbol{c}$,

$\mathbit{V}=\boldsymbol{\mathcal{T}}_{t}=-\boldsymbol{\mathcal{R}}_{t}\boldsymbol{c}-\boldsymbol{\mathcal{R}}\boldsymbol{c}_{t}=-{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{R}}\boldsymbol{c}-\boldsymbol{\mathcal{R}}\boldsymbol{c}_{t}={\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}}-\boldsymbol{\mathcal{R}}\boldsymbol{c}_{t}.$
(2.21)

###  2.5 Stationary and Non-Stationary Contours

It is important to differentiate between image contours arising from a space
curve that is changing at most with a rigid transform (stationary contours),
e.g., reflectance contours and sharp ridges, and image curves arising from
deforming space curves (non-stationary contours), e.g., occluding contours,
the _contour generators_ projecting to _apparent contours_. Stationary
contours are characterized by $\boldsymbol{\Gamma}_{t}^{w}=0$ while for
occluding contours the viewing direction $\boldsymbol{\Gamma}(S,t)$ is tangent
to the surface $\mathcal{M}$ with surface normal $\boldsymbol{N}$
($\boldsymbol{N}^{w}=\boldsymbol{\mathcal{R}}^{\top}\boldsymbol{N}$)

$\displaystyle\boldsymbol{\Gamma}^{\top}\boldsymbol{N}=0,\qquad\text{or}\qquad(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})^{\top}\boldsymbol{N}^{w}=0.$
(2.22)

For the image curve $\boldsymbol{\gamma}(s,t)$ arising from the occluding
contour, Figure 3, the normal $\boldsymbol{N}$ to $\mathcal{M}$ at an
occluding contour Cipolla and Giblin (1999) can be consistently taken as
$\boldsymbol{N}=\frac{\boldsymbol{\gamma}\times\boldsymbol{t}}{\|\boldsymbol{\gamma}\times\boldsymbol{t}\|}$.

Unless otherwise stated, we assume that the parametrization
$\boldsymbol{\Gamma}^{w}(S,t)$ of $\mathcal{M}$ is regular for occluding
contours, so that $\boldsymbol{\Gamma}^{w}_{S}(S,t)$ and
$\boldsymbol{\Gamma}^{w}_{t}(S,t)$ form the tangent plane to $\mathcal{M}$ at
$\boldsymbol{\Gamma}^{w}(S,t)$, and $t$ can be seen as a spatial parameter
Giblin and Weiss (1995). The correlation of the parametrization $S$ of
$\Gamma$ at time $t$ to that of nearby times is captured by
$\boldsymbol{\Gamma}_{t}^{w}(S,t)$, which is orthogonal to
$\boldsymbol{N}^{w}$ (since $\boldsymbol{N}^{w}$ is orthogonal to the tangent
plane), but is otherwise arbitrary as a one dimensional choice. It is common
to require that $\boldsymbol{\Gamma}^{w}_{t}(S,t)$ lay on the (infinitesimal)
epipolar plane, spanned by $\boldsymbol{\Gamma}^{w}(S,t)$,
$\boldsymbol{c}(t)$, and $\boldsymbol{c}_{t}(t)$, referred to as the _epipolar
parametrization_ Cipolla and Giblin (1999); Giblin and Weiss (1995),

$\boldsymbol{\Gamma}_{t}^{w}\times(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})=0,\qquad\text{or}\qquad\boldsymbol{\Gamma}_{t}^{w}=\lambda(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})\text{
for some $\lambda$.}$ (2.23)

##  3 Projecting Differential Geometry Onto a Single View

This section relates the intrinsic differential-geometric attributes of the
space curve and those of its perspective image curves. Specifically, the
derivatives $\boldsymbol{\Gamma}^{\prime}$,
$\boldsymbol{\Gamma}^{\prime\prime}$, and
$\boldsymbol{\Gamma}^{\prime\prime\prime}$ are first expressed in terms of the
differential geometry of $\Gamma$, namely
$\\{\boldsymbol{T},\boldsymbol{N},\boldsymbol{B},K,\dot{K},\tau\\}$, and
second, they are expressed in terms of the differential geometry of
$\boldsymbol{\gamma}$, namely
$\\{\boldsymbol{t},\boldsymbol{n},\kappa,\dot{\kappa}\\}$ using
$\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}$. Note that $\dot{K}$ and
$\dot{\kappa}$ are both intrinsic quantities. In equating these two
expressions, we relate
$\\{\boldsymbol{T},\boldsymbol{N},\boldsymbol{B},K,\dot{K},\tau\\}$ to
$\\{\boldsymbol{t},\boldsymbol{n},\kappa,\dot{\kappa}\\}$. Our purpose is to
eliminate the dependence on the parametrizations $(g,G)$, and depth $0pt$,
i.e., final expressions do not contain these unknowns nor their derivatives
$(g,g^{\prime},g^{\prime\prime})$, $(G,G^{\prime},G^{\prime\prime})$, or
unknown depth and its derivatives
$(0pt,0pt^{\prime},0pt^{\prime\prime},0pt^{\prime\prime\prime})$. Intrinsic
camera parameters are dealt with in Section 3.1.

###### Proposition 2

$\\{\boldsymbol{T},\,\boldsymbol{N},\,\boldsymbol{B},\,K,\,\dot{K},\,\tau,\,G,\,G^{\prime},\,G^{\prime\prime}\\}$
are related to
$\\{\boldsymbol{\gamma},\,\boldsymbol{t},\,\boldsymbol{n},\,\kappa,\,\dot{\kappa},\,g,\,g^{\prime},$
$g^{\prime\prime},\,0pt,\,0pt^{\prime},\,0pt^{\prime\prime},\,0pt^{\prime\prime\prime}\\}$
by

$\displaystyle G\boldsymbol{T}$
$\displaystyle=0pt^{\prime}\boldsymbol{\gamma}+0ptg\boldsymbol{t}$ (3.1)
$\displaystyle G^{\prime}\boldsymbol{T}+G^{2}K\boldsymbol{N}$
$\displaystyle=0pt^{\prime\prime}\boldsymbol{\gamma}+(20pt^{\prime}g+0ptg^{\prime})\boldsymbol{t}+0ptg^{2}\kappa\boldsymbol{n}$
(3.2)
$\displaystyle\begin{split}(G^{\prime\prime}-G^{3}K^{2})\boldsymbol{T}&+(3GG^{\prime}K+G^{3}\dot{K})\boldsymbol{N}+G^{3}K\tau\boldsymbol{B}=\\\
0pt^{\prime\prime\prime}\boldsymbol{\gamma}+[30pt^{\prime\prime}g+30pt^{\prime}g^{\prime}&+0pt(g^{\prime\prime}-g^{3}\kappa^{2})]\boldsymbol{t}+\,[30pt^{\prime}g^{2}\kappa+0pt(3gg^{\prime}\kappa+g^{3}\dot{\kappa})]\boldsymbol{n},\end{split}$
(3.3)

where $\Gamma$ and $\gamma$ are linked by a common parameter via projection
$\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}$.

###### Proof

First, writing $\boldsymbol{\Gamma}^{\prime}$,
$\boldsymbol{\Gamma}^{\prime\prime}$, and
$\boldsymbol{\Gamma}^{\prime\prime\prime}$ in the Frenet frame of
$\boldsymbol{\Gamma}$ as

$\displaystyle\boldsymbol{\Gamma}^{\prime}$ $\displaystyle=G\boldsymbol{T}$
(3.4) $\displaystyle\boldsymbol{\Gamma}^{\prime\prime}$
$\displaystyle=G^{\prime}\boldsymbol{T}+G^{2}K\boldsymbol{N}$ (3.5)
$\displaystyle\boldsymbol{\Gamma}^{\prime\prime\prime}$
$\displaystyle=(G^{\prime\prime}-G^{3}K^{2})\boldsymbol{T}+(3GG^{\prime}K+G^{2}K^{\prime})\boldsymbol{N}+G^{3}K\tau\boldsymbol{B}.$
(3.6)

Note that when expressed with respect to the arc-length of
$\boldsymbol{\Gamma}$, i.e., $G\equiv 1$, simple expressions result:

$\displaystyle\boldsymbol{\dot{\boldsymbol{\Gamma}}}$ $\displaystyle=T$ (3.7)
$\displaystyle\boldsymbol{\ddot{\boldsymbol{\Gamma}}}$
$\displaystyle=K\boldsymbol{N}$ (3.8)
$\displaystyle\boldsymbol{\dddot{\boldsymbol{\Gamma}}}$
$\displaystyle=-K^{2}\boldsymbol{T}+\dot{K}\boldsymbol{N}+K\tau\boldsymbol{B}.$
(3.9)

Second, differentiating $\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}$ gives

$\displaystyle\boldsymbol{\Gamma}^{\prime}$
$\displaystyle=0pt^{\prime}\boldsymbol{\gamma}+0pt\boldsymbol{\gamma}^{\prime}$
(3.10) $\displaystyle\boldsymbol{\Gamma}^{\prime\prime}$
$\displaystyle=0pt^{\prime\prime}\boldsymbol{\gamma}+20pt^{\prime}\boldsymbol{\gamma}^{\prime}+0pt\boldsymbol{\gamma}^{\prime\prime}$
(3.11) $\displaystyle\boldsymbol{\Gamma}^{\prime\prime\prime}$
$\displaystyle=0pt^{\prime\prime\prime}\boldsymbol{\gamma}+30pt^{\prime\prime}\boldsymbol{\gamma}^{\prime}+30pt^{\prime}\boldsymbol{\gamma}^{\prime\prime}+0pt\boldsymbol{\gamma}^{\prime\prime\prime}.$
(3.12)

This can be rewritten using expressions for the derivatives of
$\boldsymbol{\gamma}$, which are

$\displaystyle\boldsymbol{\gamma}^{\prime}$ $\displaystyle=g\boldsymbol{t}$
(3.13) $\displaystyle\boldsymbol{\gamma}^{\prime\prime}$
$\displaystyle=g^{\prime}\boldsymbol{t}+g^{2}\kappa\boldsymbol{n}$ (3.14)
$\displaystyle\boldsymbol{\gamma}^{\prime\prime\prime}$
$\displaystyle=(g^{\prime\prime}-g^{3}\kappa^{2})\boldsymbol{t}+(3gg^{\prime}\kappa+g^{2}\kappa^{\prime})\boldsymbol{n}.$
(3.15)

Thus, $\boldsymbol{\Gamma}^{\prime},\,\boldsymbol{\Gamma}^{\prime\prime}$, and
$\boldsymbol{\Gamma}^{\prime\prime\prime}$ can be written in terms of
$\boldsymbol{\gamma},\,\boldsymbol{t},\,\boldsymbol{n},\,\kappa,\,\dot{\kappa},$
$g$, $g^{\prime}$, $g^{\prime\prime}$,
$0pt,\,0pt^{\prime},\,0pt^{\prime\prime},\,0pt^{\prime\prime\prime}$ as

$\displaystyle\boldsymbol{\Gamma}^{\prime}$
$\displaystyle=0pt^{\prime}\boldsymbol{\gamma}$
$\displaystyle+0ptg\boldsymbol{t}$ (3.16)
$\displaystyle\boldsymbol{\Gamma}^{\prime\prime}$
$\displaystyle=0pt^{\prime\prime}\boldsymbol{\gamma}$
$\displaystyle+(20pt^{\prime}g+0ptg^{\prime})\boldsymbol{t}+0ptg^{2}\kappa\boldsymbol{n}$
(3.17) $\displaystyle\boldsymbol{\Gamma}^{\prime\prime\prime}$
$\displaystyle=0pt^{\prime\prime\prime}\boldsymbol{\gamma}$
$\displaystyle+[30pt^{\prime\prime}g+30pt^{\prime}g^{\prime}+0pt(g^{\prime\prime}-g^{3}\kappa^{2})]\boldsymbol{t}$
$\displaystyle+[30pt^{\prime}g^{2}\kappa+0pt(3gg^{\prime}\kappa+g^{3}\dot{\kappa})]\boldsymbol{n}.$
(3.18)

Equating (3.4-3.6) and (3.16-3.18) proves the proposition.

###### Corollary 1

Using the arc-length $\tilde{S}$ of the space curve as the common parameter,
i.e., when $G\equiv 1$, we have

$\displaystyle\boldsymbol{T}$ $\displaystyle=0pt^{\prime}\boldsymbol{\gamma}$
$\displaystyle+0ptg\boldsymbol{t}$ (3.19) $\displaystyle K\boldsymbol{N}$
$\displaystyle=0pt^{\prime\prime}\boldsymbol{\gamma}$
$\displaystyle+(20pt^{\prime}g+0ptg^{\prime})\boldsymbol{t}+0ptg^{2}\kappa\boldsymbol{n}$
(3.20)
$\displaystyle-K^{2}\boldsymbol{T}+\dot{K}\boldsymbol{N}+K\tau\boldsymbol{B}$
$\displaystyle=0pt^{\prime\prime\prime}\boldsymbol{\gamma}$
$\displaystyle+[30pt^{\prime\prime}g+30pt^{\prime}g^{\prime}+0pt(g^{\prime\prime}-g^{3}\kappa^{2})]\boldsymbol{t}$
$\displaystyle+[30pt^{\prime}g^{2}\kappa+0pt(3gg^{\prime}\kappa+g^{3}\dot{\kappa})]\boldsymbol{n}.$
(3.21)

First-Order Differential Geometry. We are now in a position to derive the
first-order differential attributes of the image curve $(g,\boldsymbol{t})$
from that of the space curve ($G$, $\boldsymbol{T}$). Note from (3.1) or
(3.19) that $\boldsymbol{T}$ lies on the plane spanned by $\boldsymbol{t}$ and
$\boldsymbol{\gamma}$, i.e., $\boldsymbol{T}$ is a linear combination of these
vectors. An exact relationship is expressed bellow.

###### Theorem 3.1

Given the tangent $\boldsymbol{T}$ at $\boldsymbol{\Gamma}$ when
$\boldsymbol{T}$ is not aligned with $\boldsymbol{\gamma}$, then the
corresponding tangent $\boldsymbol{t}$ and normal $\boldsymbol{n}$ at
$\boldsymbol{\gamma}$ are determined by

$\displaystyle\boldsymbol{t}=\frac{\mathbf{T}-T_{z}\boldsymbol{\gamma}}{\|\mathbf{T}-T_{z}\boldsymbol{\gamma}\|},\qquad\boldsymbol{n}=\boldsymbol{t}^{\perp}\doteq\boldsymbol{t}\times\boldsymbol{e}_{3}.$
(3.22)

###### Proof

Equation 3.1 states that $\boldsymbol{T}$, $\boldsymbol{t}$, and
$\boldsymbol{\gamma}$ are coplanar. Taking the dot product with
$\boldsymbol{e}_{3}$ and using
$\boldsymbol{e}_{3}^{\top}\boldsymbol{\gamma}=1$,
$\boldsymbol{e}_{3}^{\top}\boldsymbol{t}=0$, and $0pt^{\prime}=GT_{z}$
(Equation 2.6), isolate $\boldsymbol{t}$ in the original equation as

$\displaystyle\boldsymbol{t}=\frac{1}{0ptg}\left[G\boldsymbol{T}-0pt^{\prime}\boldsymbol{\gamma}\right]=\frac{G}{0ptg}[\boldsymbol{T}-T_{z}\boldsymbol{\gamma}]$
(3.23)

and the result follows by normalizing. The formula for the normal comes from
the fact that it lies in the image plane, therefore being orthogonal to both
$\boldsymbol{t}$ and $\boldsymbol{e}_{3}$.

Observe that the depth scale factor $0pt$ is not needed to find
$\boldsymbol{t}$ from $\boldsymbol{T}$. Moreover, when $\boldsymbol{\gamma}$
and $\boldsymbol{T}$ are aligned for a point on $\gamma$, Equation 3.23 still
holds, but implies that $g=0$ and $\boldsymbol{t}$ is undefined, i.e., that
the image curve will have stationary points and possibly corners or cusps.
Stationary points are in principle not detectable from the trace of
$\boldsymbol{\gamma}$ alone, but by the assumption of general position these
do not concern us.

A crucial quantity in relating differential geometry along the space curve to
that of the projected image curve is the _ratio of speeds of parametrizations_
$\frac{g}{G}(s)$. The following theorem derives the key result that this
quantity is _intrinsic_ in that it does not depend on the parametrization of
$\Gamma$ or of $\gamma$, thus allowing a relationship between the differential
geometry of the space and image curves.

###### Theorem 3.2

The ratio of speeds of the projected 2D curve $g$ and of the 3D curve $G$ at
corresponding points is an intrinsic quantity given by

$\frac{g}{G}=\frac{\|\boldsymbol{T}-T_{z}\boldsymbol{\gamma}\|}{z}\qquad\text{
or }\qquad g=\frac{\|G\boldsymbol{T}-0pt^{\prime}\boldsymbol{\gamma}\|}{0pt},$
(3.24)

i.e., it does not depend on the parametrization of $\boldsymbol{\Gamma}$ or of
$\boldsymbol{\gamma}$.

###### Proof

Follows from a dot product of Equation 3.23 with $\boldsymbol{t}$ and dividing
by $0pt=z$.

Second-Order Differential Geometry. The curvature of an image curve can be
derived from the curvature of the space curve, as shown by the next theorem.

###### Theorem 3.3

The curvature $\kappa$ of a projected image curve is given by

$\displaystyle\kappa=\left[\frac{\boldsymbol{N}-N_{z}\boldsymbol{\gamma}}{0ptg^{2}}\cdot\boldsymbol{n}\right]\,K,\text{
when $G\equiv
1$,}\qquad{or}\qquad\kappa=\left(\frac{G}{g}\right)^{2}\left[\frac{\boldsymbol{N}^{\top}(\boldsymbol{\gamma}\times\boldsymbol{t})}{0pt}\right]\,K,$
(3.25)

where $g$ and $\frac{G}{g}$ are given by Equation 3.24, and
$0pt=\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}$. The tangential
acceleration of a projected curve with respect to the arc length of the space
curve is given by

$\frac{dg}{d\tilde{S}}=\frac{[\boldsymbol{N}-N_{z}\boldsymbol{\gamma}]^{\top}\boldsymbol{t}\,K}{0pt}-2g\frac{T_{z}}{0pt},\qquad\text{or}\qquad\frac{dg}{d\tilde{S}}=-\frac{K\boldsymbol{N}^{\top}(\boldsymbol{\gamma}\times\boldsymbol{n})}{0pt}-2g\frac{T_{z}}{0pt}.$
(3.26)

###### Proof

Using Equation 2.6 in Equation 3.2 leads to

$\displaystyle G^{\prime}\boldsymbol{T}+G^{2}K\boldsymbol{N}$
$\displaystyle=(G^{\prime}T_{z}+G^{2}KN_{z})\boldsymbol{\gamma}+2GT_{z}g\boldsymbol{t}+0ptg^{\prime}\boldsymbol{t}+0ptg^{2}\kappa\boldsymbol{n}.$
(3.27)

First, in the case of $G\equiv 1$ curvature $\kappa$ can be isolated by taking
the dot product of the last equation with $\boldsymbol{n}$ which gives the
curvature projection formula (3.25). By instead taking the dot product with
$\boldsymbol{\gamma}\times\boldsymbol{t}$ we arrive at the alternative
formula, since the only remaining terms are those containing $\boldsymbol{N}$
or $\boldsymbol{n}$,

$G^{2}K\boldsymbol{N}^{\top}(\boldsymbol{\gamma}\times\boldsymbol{t})=0ptg^{2}\kappa\boldsymbol{n}^{\top}(\boldsymbol{\gamma}\times\boldsymbol{t}).$
(3.28)

Isolating $\kappa$ and using
$\boldsymbol{n}^{\top}(\boldsymbol{\gamma}\times\boldsymbol{t})=\boldsymbol{\gamma}^{\top}(\boldsymbol{t}\times\boldsymbol{n})=\boldsymbol{\gamma}^{\top}\boldsymbol{e}_{3}=1$
gives the desired equation. Second, the term $g^{\prime}$ can be isolated by
taking the dot product with $\boldsymbol{t}$ or with
$\boldsymbol{\gamma}\times\boldsymbol{n}$, giving the first and second
formulas, respectively, noting that
$\boldsymbol{t}^{\top}(\boldsymbol{\gamma}\times\boldsymbol{n})=-1$.

Note that formulas for the projection of 3D tangent and curvatures onto 2D
tangent and geodesic curvature appear in Cipolla and Zisserman (1992) and
(Cipolla and Giblin, 1999, pp. 73–75), but an actual image curvature was not
determined there. That the curvature of the space curve is related to the
curvature of the projected curve was derived in previous work Li and Zucker
(2003); Robert and Faugeras (1991), but our proof is much simpler and more
direct. Moreover, our proof methodology generalizes to relating higher order
derivatives such as curvature derivative and torsion, as shown below.

###### Theorem 3.4

The curvature derivative of a projected image curve $\boldsymbol{\gamma}$ is
derived from the local third-order differential geometry of the space curve as
follows

$\dot{\kappa}=\frac{[\dot{K}\boldsymbol{N}+K\tau\boldsymbol{B}]^{\top}(\boldsymbol{\gamma}\times\boldsymbol{t})}{0ptg^{3}}-3\kappa\left(\frac{T_{z}}{0ptg}+\frac{g^{\prime}}{g^{2}}\right),$
(3.29)

assuming $G\equiv 1$.

###### Proof

Taking the scalar product of Equation 3.21 with
$\boldsymbol{\gamma}\times\boldsymbol{t}$, and using
$\boldsymbol{T}^{\top}(\boldsymbol{\gamma}\times\boldsymbol{t})=0$ and
$\boldsymbol{n}^{\top}(\boldsymbol{\gamma}\times\boldsymbol{t})=\boldsymbol{\gamma}^{\top}(\boldsymbol{t}\times\boldsymbol{n})=\boldsymbol{\gamma}^{\top}\boldsymbol{e}_{3}=1$,

$\displaystyle[\dot{K}\boldsymbol{N}+K\tau\boldsymbol{B}]^{\top}(\boldsymbol{\gamma}\times\boldsymbol{t})$
$\displaystyle=30pt^{\prime}g^{2}\kappa+0pt(3gg^{\prime}\kappa+g^{3}\dot{\kappa}),$
(3.30) which using $0pt^{\prime}=T_{z}$ gives $\displaystyle
3T_{z}g^{2}\kappa+0pt(3gg^{\prime}\kappa+g^{3}\dot{\kappa})$
$\displaystyle=[\dot{K}\boldsymbol{N}+K\tau\boldsymbol{B}]^{\top}(\boldsymbol{\gamma}\times\boldsymbol{t}).$
(3.31)

Isolating $\dot{\kappa}$ gives the desired result. Since both $g$ and
$g^{\prime}$ are available from Equations 3.24 and 3.26, the theorem follows.

###  3.1 Intrinsic Camera Parameters and Differential Geometry

This section derives the relationship between the intrinsic differential
geometry $\\{\boldsymbol{t},\,\boldsymbol{n},\,\kappa,\,\dot{\kappa}\\}$ of
the curve in normalized image coordinates to those in image pixel coordinates,
$\\{\boldsymbol{t}_{im},\,\boldsymbol{n}_{im},\,\kappa_{im},\,\dot{\kappa}_{im}\\}$.
Using the intrinsic parameter matrix $\boldsymbol{\mathcal{K}}_{im}$ relating
$\boldsymbol{\gamma}_{im}=\boldsymbol{\mathcal{K}}_{im}\boldsymbol{\gamma}$,
by the linear Equation 2.7.

###### Theorem 3.5

The intrinsic quantities
$\\{\boldsymbol{t},\,\boldsymbol{n},\,\kappa,\,\dot{\kappa}\\}$ and
$\\{\boldsymbol{t}_{im},\,\boldsymbol{n}_{im},\,\kappa_{im},\,\dot{\kappa}_{im}\\}$
under linear transformation
$\boldsymbol{\gamma}_{im}=\boldsymbol{\mathcal{K}}_{im}\boldsymbol{\gamma}$
are related by

$\displaystyle g_{im}$
$\displaystyle=\|\boldsymbol{\mathcal{K}}_{im}\boldsymbol{t}\|,\qquad\boldsymbol{t}_{im}=\frac{\boldsymbol{\mathcal{K}}_{im}\boldsymbol{t}}{\|\boldsymbol{\mathcal{K}}_{im}\boldsymbol{t}\|},\qquad\boldsymbol{n}_{im}=\boldsymbol{t}_{im}\times\boldsymbol{e}_{3},$
(3.32) $\displaystyle g^{\prime}_{im}$
$\displaystyle=\frac{\kappa\boldsymbol{t}^{\top}\boldsymbol{\mathcal{K}}_{im}^{\top}\boldsymbol{\mathcal{K}}_{im}\boldsymbol{n}}{g_{im}},\qquad\kappa_{im}=\frac{\boldsymbol{n}_{im}^{\top}\boldsymbol{\mathcal{K}}_{im}\kappa\boldsymbol{n}}{g_{im}^{2}},$
(3.33) $\displaystyle\dot{\kappa}_{im}$
$\displaystyle=\frac{1}{g^{3}}\boldsymbol{n}_{im}^{\top}\boldsymbol{\mathcal{K}}_{im}(-\kappa^{2}\boldsymbol{t}+\dot{\kappa}\boldsymbol{n})-\frac{3g_{im}^{\prime}\kappa_{im}}{g_{im}^{2}}.$
(3.34)

where the speed $g_{im}$ is relative to unit speed at $\boldsymbol{\gamma}$.

###### Proof

Differentiating (2.7) with respect to the arc-length $\tilde{s}$ of
$\boldsymbol{\gamma}$ and using (3.13),
$\boldsymbol{\gamma}_{im}^{\prime}=\boldsymbol{\mathcal{K}}_{im}\dot{\boldsymbol{\gamma}}$
gives

$\displaystyle
g_{im}\boldsymbol{t}_{im}=\boldsymbol{\mathcal{K}}_{im}\boldsymbol{t}.$ (3.35)

Differentiating (2.7) a second time with respect to $\tilde{s}$, and using
Equation 3.14,

$g_{im}^{\prime}\boldsymbol{t}_{im}+g_{im}^{2}\kappa_{im}\boldsymbol{n}_{im}=\boldsymbol{\mathcal{K}}_{im}\kappa\boldsymbol{n}.$
(3.36)

Taking the dot product with $\boldsymbol{t}_{im}$ gives the formula for
$g_{im}^{\prime}$, and taking the dot product with $\boldsymbol{n}_{im}$ gives
the formula for $\kappa_{im}$. Differentiating (2.7) a third time with respect
to $\tilde{s}$, and using Equation 3.15, we have

$(g_{im}^{\prime\prime}-g_{im}^{3}\kappa_{im}^{2})\boldsymbol{t}_{im}+(3g_{im}g_{im}^{\prime}\kappa_{im}+g^{3}_{im}\dot{\kappa}_{im})\boldsymbol{n}_{im}=\boldsymbol{\mathcal{K}}_{im}(-\kappa^{2}\boldsymbol{t}+\dot{\kappa}\boldsymbol{n}).$
(3.37)

Taking the dot product with $\boldsymbol{n}_{im}$,

$3g_{im}g_{im}^{\prime}\kappa_{im}+g_{im}^{3}\dot{\kappa}_{im}=\boldsymbol{n}_{im}^{\top}\boldsymbol{\mathcal{K}}_{im}(-\kappa^{2}\boldsymbol{t}+\dot{\kappa}\boldsymbol{n}),$
(3.38)

and isolating $\kappa_{im}$, the last result follows.

The above theorem can also be used in its inverse form from
$\boldsymbol{\gamma}_{im}$ to $\boldsymbol{\gamma}$ by substituting
$\boldsymbol{\mathcal{K}}_{im}$ for $\boldsymbol{\mathcal{K}}_{im}^{-1}$, and
trivially exchanging the sub-indices. Moreover, the theorem is generally valid
for relating differential geometry under any linear transformation in place of
$\boldsymbol{\mathcal{K}}_{im}$.

##  4 Reconstructing Differential Geometry from Multiple Views

In the previous section, we derived the differential geometry of a projected
curve from a space curve. In this section, we derive the differential geometry
of a space curve $\Gamma$ from that of its projected image curves in multiple
views, namely $\boldsymbol{\gamma}_{i}$ for camera $i$, $i=1,\dots,N$. In
order to simplify the equations, in this section _all vectors are written in
the common world coordinate basis_ , including $\boldsymbol{\gamma}_{i}$.
Denote $\boldsymbol{\Gamma}_{i}:=\boldsymbol{\Gamma}^{w}-\boldsymbol{c}_{i}$,
namely $\boldsymbol{\Gamma}_{i}$ represents the vector from the $i^{th}$
camera center to the 3D point $\boldsymbol{\Gamma}^{w}$ in the world
coordinate system.

The reconstruction of a point on the space curve $\Gamma$ from two
corresponding image curve points $\boldsymbol{\gamma}_{1}$ and
$\boldsymbol{\gamma}_{2}$ can be obtained by equating the two expressions for
$\boldsymbol{\Gamma}^{w}$ given by Equation 2.4,

$\left\\{\begin{array}[]{c}\boldsymbol{\Gamma}^{w}-\mathbf{c}_{1}=0pt_{1}\boldsymbol{\gamma}_{1}\\\
\boldsymbol{\Gamma}^{w}-\mathbf{c}_{2}=0pt_{2}\boldsymbol{\gamma}_{2}\end{array}\right.\qquad\implies\qquad
0pt_{1}\boldsymbol{\gamma}_{1}-0pt_{2}\boldsymbol{\gamma}_{2}=\mathbf{c}_{2}-\mathbf{c}_{1}.$
(4.1)

Taking the dot product with $\boldsymbol{\gamma}_{1}$,
$\boldsymbol{\gamma}_{2}$, and
$\boldsymbol{\gamma}_{1}\times\boldsymbol{\gamma}_{2}$ gives

$\left\\{\begin{aligned}
0pt_{1}\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{1}-0pt_{2}\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{2}&=(\boldsymbol{c}_{2}-\boldsymbol{c}_{1})\cdot\boldsymbol{\gamma}_{1}\\\
0pt_{1}\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{2}-0pt_{2}\boldsymbol{\gamma}_{2}\cdot\boldsymbol{\gamma}_{2}&=(\boldsymbol{c}_{2}-\boldsymbol{c}_{1})\cdot\boldsymbol{\gamma}_{2}\\\
0&=(\boldsymbol{c}_{2}-\boldsymbol{c}_{1})\cdot(\boldsymbol{\gamma}_{1}\times\boldsymbol{\gamma}_{2}),\end{aligned}\right.$
(4.2)

which gives

$\left\\{\begin{aligned}
0pt_{1}&=\frac{(\boldsymbol{c}_{2}-\boldsymbol{c}_{1})\cdot\boldsymbol{\gamma}_{1}\,(\boldsymbol{\gamma}_{2}\cdot\boldsymbol{\gamma}_{2})-(\boldsymbol{c}_{2}-\boldsymbol{c}_{1})\cdot\boldsymbol{\gamma}_{2}\,(\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{2})}{(\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{1})(\boldsymbol{\gamma}_{2}\cdot\boldsymbol{\gamma}_{2})-(\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{2})^{2}}\\\
0pt_{2}&=\frac{(\boldsymbol{c}_{2}-\boldsymbol{c}_{1})\cdot\boldsymbol{\gamma}_{1}\,(\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{2})-(\boldsymbol{c}_{2}-\boldsymbol{c}_{1})\cdot\boldsymbol{\gamma}_{2}\,(\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{1})}{(\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{1})(\boldsymbol{\gamma}_{2}\cdot\boldsymbol{\gamma}_{2})-(\boldsymbol{\gamma}_{1}\cdot\boldsymbol{\gamma}_{2})^{2}},\end{aligned}\right.$
(4.3)

provided that
$(\boldsymbol{c}_{2}-\boldsymbol{c}_{1})\cdot(\boldsymbol{\gamma}_{1}\times\boldsymbol{\gamma}_{2})=0$.
This is precisely the well-known fact that this system of three equations in
two unknowns $0pt_{1}$ and $0pt_{2}$ can only be solved if the lines
$\boldsymbol{c}_{1}\boldsymbol{\gamma}_{1}$ and
$\boldsymbol{c}_{2}\boldsymbol{\gamma}_{2}$ intersect.

The crucial factor in relating the differential geometry of image curves in
distinct views is the relationship between their parametrization in each view,
given in the next theorem.

###### Proposition 3

The ratio of parametrization speeds in two views of a space curve at
corresponding points is given by

$\frac{g_{1}}{g_{2}}=\frac{0pt_{2}}{0pt_{1}}\frac{\|\boldsymbol{T}-(\boldsymbol{e}_{3,1}^{\top}\boldsymbol{T})\boldsymbol{\gamma}_{1}\|}{\|\boldsymbol{T}-(\boldsymbol{e}_{3,2}^{\top}\boldsymbol{T})\boldsymbol{\gamma}_{2}\|}.$
(4.4)

###### Proof

Follows by dividing expressions for $\frac{g_{1}}{G}$ and $\frac{g_{2}}{G}$
from Equation 3.24.

Next, note from Equation 3.1 that the unit vector $\boldsymbol{T}$ can be
written as

$\displaystyle\boldsymbol{T}=\frac{0pt^{\prime}}{G}\boldsymbol{\gamma}+0pt\frac{g}{G}\boldsymbol{t}.$
(4.5)

Since $\boldsymbol{T}$ is a unit vector, it can be written as

$\displaystyle\boldsymbol{T}$
$\displaystyle=\cos\theta\,\frac{\boldsymbol{\gamma}}{\|\boldsymbol{\gamma}\|}+\sin\theta\,\boldsymbol{t},\qquad\text{where
}\ \ \cos\theta=\frac{0pt^{\prime}}{G}\|\boldsymbol{\gamma}\|,\ \
\sin\theta=0pt\frac{g}{G}.$ (4.6)

Note that $0pt>0$ implies that $\sin\theta\geq 0$ or $\theta\in[0,\pi)$. Thus
the reconstruction of $\boldsymbol{T}$ from $\boldsymbol{t}$ requires the
discovery of the additional parameter $\theta$ which can be provided from
tangents at two corresponding points, as stated in the next result.

###### Theorem 4.1

Two tangent vectors at a corresponding pair of points, namely
$\boldsymbol{t}_{1}$ at $\boldsymbol{\gamma}_{1}$ and $\boldsymbol{t}_{2}$ at
$\boldsymbol{\gamma}_{2}$, reconstruct the corresponding space tangent
$\boldsymbol{T}$ at $\boldsymbol{\Gamma}$ as

$\displaystyle\boldsymbol{T}=\cos\theta_{1}\frac{\boldsymbol{\gamma}_{1}}{\|\boldsymbol{\gamma}_{1}\|}+\sin\theta_{1}\boldsymbol{t}_{1}=\cos\theta_{2}\frac{\boldsymbol{\gamma}_{2}}{\|\boldsymbol{\gamma}_{2}\|}+\sin\theta_{2}\boldsymbol{t}_{2},$
(4.7)

and

$\displaystyle 0pt_{1}g_{1}$ $\displaystyle=\sin\theta_{1}G,\qquad
0pt_{1}^{\prime}\|\boldsymbol{\gamma}_{1}\|=\cos\theta_{1}G,\qquad\frac{0pt_{1}^{\prime}}{0pt_{1}g_{1}}=-\frac{\boldsymbol{t}_{1}\cdot(\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2})}{\boldsymbol{\gamma}_{1}\cdot(\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2})}$
(4.8) $\displaystyle 0pt_{2}g_{2}$ $\displaystyle=\sin\theta_{2}G,\qquad
0pt_{2}^{\prime}\|\boldsymbol{\gamma}_{2}\|=\cos\theta_{2}G,\qquad\frac{0pt_{2}^{\prime}}{0pt_{2}g_{2}}=-\frac{\boldsymbol{t}_{2}\cdot(\boldsymbol{\gamma}_{1}\times\boldsymbol{t}_{1})}{\boldsymbol{\gamma}_{2}\cdot(\boldsymbol{\gamma}_{1}\times\boldsymbol{t}_{1})},$

where

$\displaystyle\tan\theta_{1}$
$\displaystyle=-\frac{1}{\|\boldsymbol{\gamma}_{1}\|}\frac{\boldsymbol{\gamma}_{1}\cdot(\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2})}{\boldsymbol{t}_{1}\cdot(\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2})},\qquad\theta_{1}\in[0,\pi)$
(4.9) $\displaystyle\tan\theta_{2}$
$\displaystyle=-\frac{1}{\|\boldsymbol{\gamma}_{2}\|}\frac{\boldsymbol{\gamma}_{2}\cdot(\boldsymbol{\gamma}_{1}\times\boldsymbol{t}_{1})}{\boldsymbol{t}_{2}\cdot(\boldsymbol{\gamma}_{1}\times\boldsymbol{t}_{1})},\qquad\theta_{2}\in[0,\pi).$

###### Proof

Equating the two expressions for $\boldsymbol{T}$ from Equation 4.6, one for
each view, gives Equation 4.7. Solving for $\theta_{1}$ by taking the dot
product with $\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2}$ gives

$\cos\theta_{1}\,\frac{\boldsymbol{\gamma}_{1}}{\|\boldsymbol{\gamma}_{1}\|}\cdot(\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2})+\sin\theta_{1}\,\boldsymbol{t}_{1}\cdot(\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2})=0,$
(4.10)

which leads to Equation 4.9 and similarly for $\theta_{2}$. Equation 4.8
follows from equating (4.10) and (4.5), then taking dot products with
$\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2}$.

Remark: Since $\boldsymbol{T}$ is orthogonal to both
$\boldsymbol{\gamma}_{1}\times\boldsymbol{t}_{1}$, and
$\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2}$ we have

$\displaystyle\varepsilon\boldsymbol{T}$
$\displaystyle=\frac{(\boldsymbol{t}_{1}\times\boldsymbol{\gamma}_{1})\times(\boldsymbol{t}_{2}\times\boldsymbol{\gamma}_{2})}{\|(\boldsymbol{t}_{1}\times\boldsymbol{\gamma}_{1})\times(\boldsymbol{t}_{2}\times\boldsymbol{\gamma}_{2})\|}$
$\displaystyle\varepsilon=\pm 1,$ (4.11)

where $\varepsilon$ is determined from

$\displaystyle\left\\{\begin{aligned}
\varepsilon\left[\boldsymbol{T}-(\boldsymbol{T}\cdot\boldsymbol{e}_{3,1})\boldsymbol{\gamma}_{1}\right]\cdot\boldsymbol{t}_{1}&>0\\\
\varepsilon\left[\boldsymbol{T}-(\boldsymbol{T}\cdot\boldsymbol{e}_{3,2})\boldsymbol{\gamma}_{2}\right]\cdot\boldsymbol{t}_{2}&>0.\end{aligned}\right.$
(4.12)

Figure 4: 3D Tangent reconstruction from two views as the intersection of two
planes.

Remark: This theorem implies that _any_ two tangents at corresponding points
can be consistent with at least one space tangent. Furthermore, the discovery
of $\boldsymbol{T}$ does not require the _a priori_ solution of $0pt_{1}$ or
$0pt_{2}$.

Remark: An analogous tangent reconstruction expression under continuous motion
may be derived, see Faugeras and Papadopoulo (1993).

###### Theorem 4.2

The normal vector $\boldsymbol{N}$ and curvature $K$ of a point on a space
curve $\Gamma$ with point-tangent-curvature at projections in two views
$(\boldsymbol{\gamma}_{1},\boldsymbol{t}_{1},\kappa_{1})$ and
$(\boldsymbol{\gamma}_{2},\boldsymbol{t}_{2},\kappa_{2})$ are given by solving
the system in the vector $\boldsymbol{N}K$

$\left\\{\begin{aligned}
G^{2}(\boldsymbol{\gamma}_{1}\times\boldsymbol{t}_{1})^{\top}\boldsymbol{N}K&=0pt_{1}g_{1}^{2}\,\,\kappa_{1}\\\
G^{2}(\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2})^{\top}\boldsymbol{N}K&=0pt_{2}g_{2}^{2}\,\,\kappa_{2}\\\
\boldsymbol{T}^{\top}\boldsymbol{N}K&=0,\end{aligned}\right.$ (4.13)

where $\boldsymbol{T}$ is given by Equation 4.5, $0pt_{1}$ and $0pt_{2}$ by
Equation 4.3, and $g_{1}$ and $g_{2}$ by Equation 3.24.

###### Proof

Taking the dot product of (3.27) with
$\boldsymbol{\gamma}\times\boldsymbol{t}$, for each view, we arrive at the
first two equations. The third equation imposes the solution $\boldsymbol{N}K$
to be normal to $\boldsymbol{T}$.

An analogous curvature reconstruction expression under continuous motion can
be derived, see Cipolla (1991). Theorem 4.2 is a variant form of a known
result Robert and Faugeras (1991); Li and Zucker (2003, 2006), using the
proposed unified formulation which enables a more condensed and generalizable
proof for the practical case of planar images. The next theorem leverages this
effective theoretical framework to achieve the reconstructuction of the
torsion and curvature derivative of a space curve from two image curves, which
is the central novel result of the present work.

###### Theorem 4.3

The torsion and curvature derivative at a point of a space curve can be
obtained from up to third order differential geometry
$\\{\kappa,\dot{\kappa}\\}$ at a pair of corresponding points in two views by
solving for an unknown vector $\mathbit{Z}$ in the system

$\left\\{\begin{aligned}
(\boldsymbol{\gamma}_{1}\times\boldsymbol{t}_{1})^{\top}\mathbit{Z}&=3g_{1}^{2}\kappa_{1}\boldsymbol{e}_{3,1}^{\top}\boldsymbol{T}+0pt_{1}(3g_{1}g_{1}^{\prime}\kappa_{1}+g_{1}^{3}\dot{\kappa}_{1})\\\
(\boldsymbol{\gamma}_{2}\times\boldsymbol{t}_{2})^{\top}\mathbit{Z}&=3g_{2}^{2}\kappa_{2}\boldsymbol{e}_{3,2}^{\top}\boldsymbol{T}+0pt_{2}(3g_{2}g_{2}^{\prime}\kappa_{2}+g_{2}^{3}\dot{\kappa}_{2})\\\
\boldsymbol{T}^{\top}\mathbit{Z}&=0,\end{aligned}\right.$ (4.14)

and by solving for the torsion $\tau$ and curvature derivative $\dot{K}$ from
$\mathbit{Z}=\dot{K}\boldsymbol{N}+K\tau\boldsymbol{B}$, i.e.,

$\displaystyle\tau$ $\displaystyle=\frac{\mathbit{Z}^{\top}\boldsymbol{B}}{K}$
(4.15) $\displaystyle\dot{K}$
$\displaystyle=\mathbit{Z}^{\top}\boldsymbol{N},$ (4.16)

with $\boldsymbol{T}$, $\boldsymbol{N}$, $\boldsymbol{B}$, $K$, $g_{1}$,
$g_{2}$, $g_{1}^{\prime}$, $g_{2}^{\prime}$, $0pt_{1}$, and $0pt_{2}$
determined from previous derivations, and assuming $G\equiv 1$.

###### Proof

Apply Equation (3.30) for two views, and let
$\mathbit{Z}:=\dot{K}\boldsymbol{N}+K\tau\boldsymbol{B}$ to get the first two
equations of (4.14). The last equation of (4.14) constrains $\mathbit{Z}$ to
be orthogonal to $\boldsymbol{T}$.

##  5 Projecting Differential Geometry Under Differential Motion

The goal of this section is to relate _differential observations_ in a series
of images from a continuous video sequence to the _differential geometry_ of
the space curve. As this relationship is governed by the _differential motion_
of the camera and its intrinsic parameters, we also aim to recover scene
geometry and camera motion/pose from these observations and equations. An
account of intrinsic camera calibration in this setting is left for future
work. We explore how differential scene properties are projected onto
differential image properties for points and curves, and expect future work to
apply this to surfaces.

Differential models of camera motion observing a _rigid scene_ were studied in
Longuet-Higgins and Prazdny (1980); Waxman and Ullman (1985); Maybank (1992);
Ma et al (2004); van den Hengel (2000); Triggs (1999); Astrom and Heyden
(1996); Heyden (2006); Baumela et al (2000); Dornaika and Sappa (2006); Kahl
and Heyden (2001); Vieville and Faugeras (1996); van den Hengel et al (2007);
Brodský et al (2000); Brodskỳ and Fermüller (2002). These papers studied how
the first and second-order motion of the image of fixed points relate to a
differential camera motion model. They also envisioned recovering local 3D
surface properties from the local behavior of the velocities of projected
surface points in an image neighborhood. Differential models for _nonrigid
curves_ observed in a monocular video sequence were studied in Faugeras and
Papadopoulo (1993); Papadopoulo and Faugeras (1996); Papadopoulo (1996);
Faugeras (1990), where it was established that multiple simultaneous video
sequences would be needed. This led to a practical work in the reconstruction
of nonrigid curves from multiview video Carceroni and Kutulakos (1999);
Carceroni (2001), exploiting temporal consistency within each video frame, as
well as consistency across the two video sequences. Differential models of
occluding contours were studied mainly in Cipolla (1991); Cipolla and Blake
(1992); Cipolla and Giblin (1999), relating the deformation of apparent
contours under differential camera motion to scene properties such as
occluding contours, and a differential-geometric model of the underlying 3D
surface.

###  5.1 Differential Relations for a Point

###### Theorem 5.1

(Moving 3D point) Let $\boldsymbol{\Gamma}^{w}(t)$ be a _moving_ point in
space, projected onto a moving camera as $\boldsymbol{\gamma}(t)$ with depth
$0pt(t)$. Let the differential velocity and rotation of the camera be
$\mathbit{V}$ and $\mathbit{\Omega}$, respectively, and let $\mathbit{V}_{t}$
and $\mathbit{\Omega}_{t}$ represent their derivative with respect to time
$t$, respectively. Then, the depth gradient and second derivative at $t=0$ are

$\displaystyle 0pt_{t}$
$\displaystyle=0pt\boldsymbol{e}_{3}^{\top}({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+\frac{1}{0pt}\boldsymbol{\Gamma}^{w}_{t}+\frac{1}{0pt}\mathbit{V})$
(5.1) $\displaystyle 0pt_{tt}$
$\displaystyle=0pt\,\boldsymbol{e}_{3}^{\top}({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+2\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}^{w}_{t}+\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{tt}+\boldsymbol{e}_{3}^{\top}\mathbit{V}_{t},$
(5.2)

and the velocity and acceleration of the projected point at $t=0$ are given by

$\displaystyle\boldsymbol{\gamma}_{t}$
$\displaystyle=\left({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}-(\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})\boldsymbol{\gamma}\right)+\frac{1}{0pt}(\boldsymbol{\Gamma}^{w}_{t}-\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t}\boldsymbol{\gamma})+\frac{1}{0pt}(\mathbit{V}-V_{z}\boldsymbol{\gamma})$
(5.3) $\displaystyle\boldsymbol{\gamma}_{tt}$
$\displaystyle=({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+\frac{2}{0pt}{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}^{w}_{t}+\frac{1}{0pt}\boldsymbol{\Gamma}^{w}_{tt}+\frac{1}{0pt}\mathbit{V}_{t}-\frac{20pt_{t}}{0pt}\boldsymbol{\gamma}_{t}-\frac{0pt_{tt}}{0pt}\boldsymbol{\gamma},$
(5.4)

which can be simplified as

$\displaystyle\boldsymbol{\gamma}_{tt}$
$\displaystyle=({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+\frac{2}{0pt}{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}^{w}_{t}+\frac{1}{0pt}\boldsymbol{\Gamma}^{w}_{tt}+\frac{1}{0pt}\mathbit{V}_{t}$
$\displaystyle-2\boldsymbol{e}_{3}^{\top}\left({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+\frac{\mathbit{V}}{0pt}+\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt}\right)\left(\frac{\mathbit{V}}{0pt}-\frac{V_{z}}{0pt}\boldsymbol{\gamma}+\mathbit{\Omega}_{\times}\boldsymbol{\gamma}-(\boldsymbol{e}_{3}^{\top}\mathbit{\Omega}_{\times}\boldsymbol{\gamma})\,\boldsymbol{\gamma}+\frac{1}{0pt}\boldsymbol{\Gamma}^{w}_{t}-\frac{\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t}}{0pt}\boldsymbol{\gamma}\right)$
$\displaystyle-\boldsymbol{e}_{3}^{\top}\left(({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+\frac{\mathbit{V}_{t}}{0pt}+2{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt}+\frac{\boldsymbol{\Gamma}^{w}_{tt}}{0pt}\right)\boldsymbol{\gamma}.$
(5.5)

###### Proof

The image velocity $\boldsymbol{\gamma}_{t}$ is dependent on the velocity
$\boldsymbol{\Gamma}_{t}$ of the 3D structure in camera coordinates, which
arises from both the motion of $\boldsymbol{\Gamma}^{w}$ and from the moving
camera. Differentiating
$\boldsymbol{\Gamma}=\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}+\boldsymbol{\mathcal{T}}$,
we get

$\boldsymbol{\Gamma}_{t}=\boldsymbol{\mathcal{R}}_{t}\boldsymbol{\Gamma}^{w}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}+\boldsymbol{\mathcal{T}}_{t}={\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}+\mathbit{V}.$
(5.6)

Differentiating $\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}$ we get

$\boldsymbol{\Gamma}_{t}=0pt\boldsymbol{\gamma}_{t}+0pt_{t}\boldsymbol{\gamma}.$
(5.7)

Equating these two expressions leads to

$0pt\boldsymbol{\gamma}_{t}+0pt_{t}\boldsymbol{\gamma}={\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}+\mathbit{V}\
\ \ \ \ \text{ for arbitrary $t$.}$ (5.8)

At $t=0$ we have
$\boldsymbol{\Gamma}^{w}=\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}$, leading
to

$0pt\boldsymbol{\gamma}_{t}+0pt_{t}\boldsymbol{\gamma}=0pt{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+\boldsymbol{\Gamma}^{w}_{t}+\mathbit{V}\
\ \ \ \ \text{ for $t=0$.}$ (5.9)

The depth gradient $0pt_{t}$ is then isolated by taking the dot product of
both sides of Equation 5.9 with $\boldsymbol{e}_{3}$, observing that
$\boldsymbol{e}_{3}^{\top}\boldsymbol{\gamma}=1$ and
$\boldsymbol{e}_{3}^{\top}\boldsymbol{\gamma}_{t}=0$, resulting in Equation
5.1. The expression for $0pt_{t}$ is then substituted into Equation 5.9 from
which $\boldsymbol{\gamma}_{t}$ can be isolated in the form of Equation 5.3.

The second order expressions $\boldsymbol{\gamma}_{tt}$ and $0pt_{tt}$ require
another time derivative of Equation 5.8,

$0pt\boldsymbol{\gamma}_{tt}+20pt_{t}\boldsymbol{\gamma}_{t}+0pt_{tt}\boldsymbol{\gamma}=({\mathbit{\Omega}}_{\times}^{2}\boldsymbol{\mathcal{R}}+{[\mathbit{\Omega}_{t}]}_{\times}\boldsymbol{\mathcal{R}})\boldsymbol{\Gamma}^{w}+{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}+\boldsymbol{\mathcal{R}}_{t}\boldsymbol{\Gamma}_{t}^{w}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}_{tt}^{w}+\mathbit{V}_{t}.$
(5.10)

Setting $t=0$ we have

$0pt\boldsymbol{\gamma}_{tt}+20pt_{t}\boldsymbol{\gamma}_{t}+0pt_{tt}\boldsymbol{\gamma}=({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})0pt\boldsymbol{\gamma}+2{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}_{t}^{w}+\boldsymbol{\Gamma}_{tt}^{w}+\mathbit{V}_{t}.$
(5.11)

Now the expression for $0pt_{tt}$ in the theorem can be obtained by dotting
with $\boldsymbol{e}_{3}$, giving Equation 5.2. Isolating
$\boldsymbol{\gamma}_{tt}$ we have

$\boldsymbol{\gamma}_{tt}=({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+\frac{1}{0pt}(2{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}_{t}^{w}+\boldsymbol{\Gamma}_{tt}^{w}+\mathbit{V}_{t}-20pt_{t}\boldsymbol{\gamma}_{t}-0pt_{tt}\boldsymbol{\gamma}).$
(5.12)

Substituting Equations 5.2 and 5.3 into the above, we obtain the final
expression for $\boldsymbol{\gamma}_{tt}$.

The Special Case of Fixed Points. The question of how the image of a fixed
point moves as the camera moves was studied by Longuet-Higgins and Prazdny
Longuet-Higgins and Prazdny (1980) and later by Waxman and Ullman Waxman and
Ullman (1985), giving the velocity $\boldsymbol{\gamma}_{t}$ for a fixed
point. This calculation also leads to the well-known epipolar constraint, the
notion of Essential matrix Longuet-Higgins (1981), and the _continuous
epipolar constraint_ Zhuang and Haralick (1984); Maybank (1992); Kanatani
(1993); Viéville and Faugeras (1995); Tian et al (1996); Brooks et al (1997);
Åström and Heyden (1998); Ponce and Genc (1998); Yi Ma (1998); Ma et al
(2004); Stewénius et al (2007); Lin et al (2009); Valgaerts et al (2012);
Schneevoigt et al (2014). Theorem 5.1 in the special case of a fixed point
gives interesting geometric insight into these classical results.
Specifically, setting $\boldsymbol{\Gamma}^{w}_{t}=0$ in first-order
computations of Equations 5.1 and 5.3 results in

$\displaystyle\frac{0pt_{t}}{0pt}$
$\displaystyle=\boldsymbol{e}_{3}^{\top}({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+\frac{\mathbit{V}}{0pt})$
(5.13) $\displaystyle\boldsymbol{\gamma}_{t}$
$\displaystyle=\mathbit{\Omega}_{\times}\boldsymbol{\gamma}-(\boldsymbol{e}_{3}^{\top}\mathbit{\Omega}_{\times}\boldsymbol{\gamma})\,\boldsymbol{\gamma}+\frac{\mathbit{V}}{0pt}-\frac{V_{z}}{0pt}\boldsymbol{\gamma}.$
(5.14)

Essential constraint. To derive the differential epipolar constraint,
eliminate $0pt$ from Equation 5.14 by first writing out the expression in
terms of $\xi$, $\eta$, $u$, and $v$, where
$\boldsymbol{\gamma}=[\xi,\,\eta,\,1]^{\top}$, and
$\boldsymbol{\gamma}_{t}=[u,\,v,\,0]$, and use
$\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}=-\Omega_{y}\xi+\Omega_{x}\eta$,
giving

$\displaystyle
u-\Omega_{y}\xi^{2}+\Omega_{x}\xi\eta+\Omega_{z}\eta-\Omega_{y}=\frac{1}{0pt}(V_{x}-V_{z}\xi)$
$\displaystyle
v+\Omega_{x}\eta^{2}-\Omega_{y}\xi\eta-\Omega_{z}\xi+\Omega_{x}=\frac{1}{0pt}(V_{y}-V_{z}\eta)$
(5.15)

and then eliminate $0pt$, giving

$\left(u-\Omega_{y}\xi^{2}+\Omega_{x}\xi\eta+\Omega_{z}\eta-\Omega_{y}\right)\left(V_{y}-\eta
V_{z}\right)=\left(v+\Omega_{x}\eta^{2}-\Omega_{y}\xi\eta-\Omega_{z}\xi+\Omega_{x}\right)\left(V_{x}-\xi
V_{z}\right),$

which is the epipolar constraint for differential motion. A more direct way of
deriving the epipolar constraint equation is to eliminate $0pt$ in Equation
5.9 with $\boldsymbol{\Gamma}^{w}_{t}=0$ by taking the dot-product with the
vector
$\mathbit{V}\times\boldsymbol{\gamma}={\mathbit{V}}_{\times}\boldsymbol{\gamma}$,
where ${\mathbit{V}}_{\times}$ is a skew-symmetric arrangement of
$\mathbit{V}$, and using
${\mathbit{\Omega}}_{\times}^{\top}=-{\mathbit{\Omega}}_{\times}$ gives

$\displaystyle
0pt\boldsymbol{\gamma}_{t}^{\top}{\mathbit{V}}_{\times}\boldsymbol{\gamma}$
$\displaystyle=-0pt\boldsymbol{\gamma}^{\top}{\mathbit{\Omega}}_{\times}{\mathbit{V}}_{\times}\boldsymbol{\gamma},$
resulting in the _differential epipolar constraint_
$\displaystyle\boldsymbol{\gamma}_{t}^{\top}{\mathbit{V}}_{\times}\boldsymbol{\gamma}$
$\displaystyle+\boldsymbol{\gamma}^{\top}{\mathbit{\Omega}}_{\times}{\mathbit{V}}_{\times}\boldsymbol{\gamma}=0.$
(5.16)

In comparison, the widely-known essential constraint for relating two views is
given by

$\boldsymbol{\gamma}_{2}^{\top}{\boldsymbol{\mathcal{T}}}_{\times}\boldsymbol{\mathcal{R}}\boldsymbol{\gamma}_{1}=0,$
(5.17)

where ${\boldsymbol{\mathcal{T}}}_{\times}\boldsymbol{\mathcal{R}}$, the
essential matrix, combines the effects of translation and rotation to relate
two points $\boldsymbol{\gamma}_{1}$ and $\boldsymbol{\gamma}_{2}$. In the
differential case, the two matrices ${\mathbit{V}}_{\times}$ and
${\mathbit{\Omega}}_{\times}{\mathbit{V}}_{\times}$ play a similar role to
${\boldsymbol{\mathcal{T}}}_{\times}\boldsymbol{\mathcal{R}}$ in the discrete
motion case to relate a point and its velocity.

###### Remark 1

Observe from Equation 5.14 that $\boldsymbol{\gamma}_{t}$ can also be written
as the sum of two components, one depending on $\mathbit{V}$, and the other on
$\mathbit{\Omega}$, i.e.,

$\boldsymbol{\gamma}_{t}=\frac{1}{0pt}A(\boldsymbol{\gamma})\mathbit{V}+B(\boldsymbol{\gamma})\mathbit{\Omega},\
\ \text{where}\ \ A(\boldsymbol{\gamma})=\begin{bmatrix}1&0&-\xi\\\
0&1&-\eta\\\ 0&0&0\end{bmatrix}\text{ and
}B(\boldsymbol{\gamma})=\begin{bmatrix}-\xi\eta&1+\xi^{2}&-\eta\\\
-(1+\eta)&\xi\eta&\xi\\\ 0&0&0\end{bmatrix}.$ (5.18)

That $\boldsymbol{\gamma}_{t}$ depends linearly on $\mathbit{V}$ and
$\mathbit{\Omega}$ (since $A$ and $B$ are only dependent on the position
$\boldsymbol{\gamma}$), with $0pt$ left in the equation, is the basis of
subspace methods in structure from motion Heeger and Jepson (1992).
Observations of image velocities
$\boldsymbol{\gamma}_{t,1},\boldsymbol{\gamma}_{t,2},\dots,\boldsymbol{\gamma}_{t,N}$
at points
$\boldsymbol{\gamma}_{1},\boldsymbol{\gamma}_{2},\dots,\boldsymbol{\gamma}_{N}$
provides $2N$ _linear_ equations in $\mathbit{V}$ and $\mathbit{\Omega}$,
given $0pt_{1},\dots,0pt_{N}$.

###  5.2 Differential Relations for a Curve

###### Theorem 5.2

(Deforming 3D curve) Consider a deforming 3D curve $\boldsymbol{\Gamma}(s,t)$
projecting to a family of 2D curves $\boldsymbol{\gamma}(s,t)$ with depth
$0pt(s,t)$, arising from camera motion with differential velocities of
translation and rotation $\mathbit{V}$ and $\mathbit{\Omega}$, respectively,
and let $\mathbit{V}_{t}$ and $\mathbit{\Omega}_{t}$ be their respective
derivatives in time. Then, the image velocity $\boldsymbol{\gamma}_{t}$ is
determined from
$\left\\{\mathbit{V},\,\mathbit{\Omega},\,\frac{\mathbit{V}}{0pt},\,\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt},\,\boldsymbol{t}\right\\}$,

$\displaystyle\alpha$
$\displaystyle=-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{n})-\left(\frac{\mathbit{V}}{0pt}-{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\mathcal{T}}}{0pt}+\boldsymbol{\mathcal{R}}\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt}\right)\cdot\boldsymbol{\gamma}\times\boldsymbol{n},$
(5.19) $\displaystyle\beta$
$\displaystyle=\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})+\left(\frac{\mathbit{V}}{0pt}-{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\mathcal{T}}}{0pt}+\boldsymbol{\mathcal{R}}\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt}\right)\cdot\boldsymbol{\gamma}\times\boldsymbol{t}.$
(5.20)

###### Proof

From Equation 2.15 and using
$-\boldsymbol{\mathcal{R}}\boldsymbol{c}_{t}=\mathbit{V}-{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}}$
from Equation 2.12,

$\boldsymbol{\Gamma}_{t}={\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}+\mathbit{V}-{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}.$
(5.21)

Using $\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}$ and
$\boldsymbol{\Gamma}_{t}=0pt_{t}\boldsymbol{\gamma}+0pt\boldsymbol{\gamma}_{t}$,

$0pt_{t}\boldsymbol{\gamma}+0pt\boldsymbol{\gamma}_{t}=0pt{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+\mathbit{V}-{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}}+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}.$
(5.22)

Taking the dot product with $\boldsymbol{\gamma}\times\boldsymbol{n}$ and
$\boldsymbol{\gamma}\times\boldsymbol{t}$,

$\left\\{\begin{aligned}
0pt\boldsymbol{\gamma}_{t}\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})&=0pt({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})+(\mathbit{V}-{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}})\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}\cdot(\boldsymbol{\gamma}\times\boldsymbol{n}),\\\
0pt\boldsymbol{\gamma}_{t}\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})&=0pt({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})+(\mathbit{V}-{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}})\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})+\boldsymbol{\mathcal{R}}\boldsymbol{\Gamma}^{w}_{t}\cdot(\boldsymbol{\gamma}\times\boldsymbol{t}).\end{aligned}\right.$
(5.23)

Now,

$\left\\{\begin{aligned}
\boldsymbol{\gamma}_{t}\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})&=(\alpha\boldsymbol{t}+\beta\boldsymbol{n})\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})=\alpha\boldsymbol{t}\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})=\alpha\boldsymbol{n}\times\boldsymbol{t}\cdot\boldsymbol{\gamma}=-\alpha\boldsymbol{e}_{3}^{\top}\boldsymbol{\gamma}=-\alpha\\\
\boldsymbol{\gamma}_{t}\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})&=(\alpha\boldsymbol{t}+\beta\boldsymbol{n})\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})=\beta\boldsymbol{n}\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})=\beta\boldsymbol{t}\times\boldsymbol{n}\cdot\boldsymbol{\gamma}=\beta\boldsymbol{e}_{3}^{\top}\boldsymbol{\gamma}=\beta.\end{aligned}\right.$
(5.24)

So that we can write

$\left\\{\begin{aligned}
\alpha&=-({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})-\left(\frac{\mathbit{V}}{0pt}-{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\mathcal{T}}}{0pt}+\boldsymbol{\mathcal{R}}\frac{\boldsymbol{\Gamma}_{t}^{w}}{0pt}\right)\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})\\\
\beta&=({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})+\left(\frac{\mathbit{V}}{0pt}-{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\mathcal{T}}}{0pt}+\boldsymbol{\mathcal{R}}\frac{\boldsymbol{\Gamma}_{t}^{w}}{0pt}\right)\cdot(\boldsymbol{\gamma}\times\boldsymbol{t}).\end{aligned}\right.$
(5.25)

Since we can switch the cross and dot products in a triple scalar product,
$\mathbit{\Omega}\times\boldsymbol{\gamma}\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})=\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{n})$
and
$\mathbit{\Omega}\times\boldsymbol{\gamma}\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})=\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})$,
giving the final result.

###### Corollary 2

The spatial variation of the velocity vector field $\boldsymbol{\gamma}_{t}$
along the curve and in time can be written as

$\displaystyle\boldsymbol{\gamma}_{st}$
$\displaystyle=\left(-\mathbit{V}+V_{z}\boldsymbol{\gamma}\right)\frac{0pt_{s}}{0pt^{2}}-\frac{V_{z}}{0pt}\boldsymbol{\gamma}_{s}+{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}_{s}-(\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}_{s})\boldsymbol{\gamma}-(\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})\boldsymbol{\gamma}_{s}$
$\displaystyle\frac{1}{0pt}(\boldsymbol{\Gamma}^{w}_{st}-\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{st}\boldsymbol{\gamma}-\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t}\boldsymbol{\gamma}_{s})-\frac{1}{0pt^{2}}(\boldsymbol{\Gamma}^{w}_{t}-\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t}\boldsymbol{\gamma})0pt_{s},$
(5.26)

and the time acceleration $\boldsymbol{\gamma}_{tt}$ is defined by

$\left\\{\begin{aligned}
\boldsymbol{t}^{\top}\boldsymbol{\gamma}_{tt}&=\boldsymbol{t}^{\top}({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+\frac{2}{0pt}\boldsymbol{t}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}^{w}_{t}+\frac{1}{0pt}\boldsymbol{t}^{\top}\boldsymbol{\Gamma}^{w}_{tt}+\frac{1}{0pt}\boldsymbol{t}^{\top}\mathbit{V}_{t}\\\
&-2\boldsymbol{e}_{3}^{\top}\left({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+\frac{\mathbit{V}}{0pt}+\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt}\right)\,\alpha-\boldsymbol{e}_{3}^{\top}\left(({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+\frac{\mathbit{V}_{t}}{0pt}+2{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt}+\frac{\boldsymbol{\Gamma}^{w}_{tt}}{0pt}\right)\boldsymbol{t}^{\top}\boldsymbol{\gamma},\\\
\boldsymbol{n}^{\top}\boldsymbol{\gamma}_{tt}&=\boldsymbol{n}^{\top}({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+\frac{2}{0pt}\boldsymbol{n}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}^{w}_{t}+\frac{1}{0pt}\boldsymbol{n}^{\top}\boldsymbol{\Gamma}^{w}_{tt}+\frac{1}{0pt}\boldsymbol{n}^{\top}\mathbit{V}_{t}\\\
&-2\boldsymbol{e}_{3}^{\top}\left({\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+\frac{\mathbit{V}}{0pt}+\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt}\right)\,\beta-\boldsymbol{e}_{3}^{\top}\left(({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+\frac{\mathbit{V}_{t}}{0pt}+2{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt}+\frac{\boldsymbol{\Gamma}^{w}_{tt}}{0pt}\right)\boldsymbol{n}^{\top}\boldsymbol{\gamma}.\end{aligned}\right.$
(5.27)

###### Proof

The $\boldsymbol{\gamma}_{st}$ expression in (2) is derived by differentiating
$\boldsymbol{\gamma}_{t}$ with respect to $s$ in Equation 5.3. Notice that
$\boldsymbol{\gamma}_{t}$ in the moving case decomposes into the same terms as
for the fixed case, Equation 5.14, plus terms dependent on
$\boldsymbol{\Gamma}^{w}_{t}$ given by
$\frac{1}{0pt}\left(\boldsymbol{\Gamma}^{w}_{t}-\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t}\boldsymbol{\gamma}\right)$.
Differentiating with respect to $s$ then gives a term equal to
$\boldsymbol{\gamma}_{st}$ for the fixed case plus terms dependent on
$\boldsymbol{\Gamma}^{w}_{t}$ and its spatial derivative, the latter being
obtained by differentiating the above expression with respect to $s$.

The expressions of $\boldsymbol{\gamma}_{tt}$ in the Frenet frame were
obtained by taking the dot product of (5.4) with $\boldsymbol{t}$ and
$\boldsymbol{n}$, noting that
$\boldsymbol{\gamma}_{t}\cdot\boldsymbol{t}=\alpha$ and
$\boldsymbol{\gamma}_{t}\cdot\boldsymbol{n}=\beta$. We then plug in
expressions (5.1) and (5.2) for $0pt_{t}$ and $0pt_{tt}$, respectively.

Special Case: Rigid Stationary Curve.

###### Corollary 3

(Rigid stationary 3D curve) Let $\boldsymbol{\Gamma}(\tilde{s})$ be a 3D curve
projecting to a family of 2D curves $\boldsymbol{\gamma}(\tilde{s},t)$ with
depth $0pt(\tilde{s},t)$, arising from camera motion with differential
velocity of translation and rotation $\mathbit{V}$ and $\mathbit{\Omega}$,
respectively. Let $\boldsymbol{t}$ denote the unit tangent to the image curve.
Then

$\boldsymbol{\gamma}_{\tilde{s}t}=\frac{-0pt_{\tilde{s}}}{0pt}\left(\frac{\mathbit{V}}{0pt}-\frac{V_{z}}{0pt}\boldsymbol{\gamma}\right)-\frac{V_{z}}{0pt}\boldsymbol{t}+{\mathbit{\Omega}}_{\times}\boldsymbol{t}-(\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{t})\boldsymbol{\gamma}-(\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})\boldsymbol{t}.$
(5.28)

###### Proof

Follows by setting $\boldsymbol{\Gamma}^{w}_{t}=0$ in Equation 2 and using the
spatial parameter as the arc-length of the image curve.

###### Corollary 4

The tangential and normal velocities of a rigid curve induced by a moving
camera are derived from
$\\{\boldsymbol{\gamma},\,\boldsymbol{t},\,\boldsymbol{n},\,\frac{\boldsymbol{\mathcal{T}}}{0pt},\,\mathbit{\Omega},\,\frac{\mathbit{V}}{0pt}\\}$
for any $t$ as

$\displaystyle\alpha$
$\displaystyle=-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{n})-\left(\frac{\mathbit{V}}{0pt}-{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\mathcal{T}}}{0pt}\right)\cdot\boldsymbol{\gamma}\times\boldsymbol{n}$
(5.29) $\displaystyle\beta$
$\displaystyle=\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})+\left(\frac{\mathbit{V}}{0pt}-{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\mathcal{T}}}{0pt}\right)\cdot\boldsymbol{\gamma}\times\boldsymbol{t}$
(5.30)

or

$\displaystyle\alpha$
$\displaystyle=-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{n})-\boldsymbol{\gamma}\times\boldsymbol{n}\cdot\frac{\mathbit{V}}{0pt}$
(5.31) $\displaystyle\beta$
$\displaystyle=\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})+\boldsymbol{\gamma}\times\boldsymbol{t}\cdot\frac{\mathbit{V}}{0pt}$
(5.32)

###### Proof

Follows directly from Theorem 5.2 and $\boldsymbol{\Gamma}^{w}_{t}=0$.

###### Corollary 5

The infinitesimal Essential constraint in the Frenet frame of the image of a
rigid curve is given by

$(\boldsymbol{\gamma}\times\boldsymbol{t})\cdot\mathbit{V}\left[\alpha+\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{n})\right]+(\boldsymbol{\gamma}\times\boldsymbol{n})\cdot\mathbit{V}\left[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})\right]=0.$
(5.33)

###### Proof

Eliminate $0pt$ from (5.32) and (5.31).

###### Corollary 6

(From Papadopoulo and Faugeras (1996); Papadopoulo (1996)) The tangential
velocity $\alpha$ can be fully determined from the normal velocity $\beta$ and
$\boldsymbol{\gamma}$, $\boldsymbol{t}$, $\boldsymbol{n}$, $\mathbit{\Omega}$,
and $\frac{\mathbit{V}}{0pt}$ without the explicit knowledge of $0pt$, as

$\alpha=-\left[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})\right]\frac{\mathbit{V}\cdot(\boldsymbol{\gamma}\times\boldsymbol{n})}{\mathbit{V}\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})}-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{n}).$
(5.34)

###### Proof

Follows by solving (5.33) for $\alpha$.

Special Case: Occluding Contours. A remarkable observation is derived below
that the first-order deformation of an apparent contour under epipolar
parametrization does not depend on the 3D surface geometry, since the
curvature-dependent terms cancel out for an occluding contour, cf. Cipolla and
Giblin (1999).

###### Theorem 5.3

(Occluding contours) Let $\boldsymbol{\Gamma}(s,t)$ be the contour generator
for apparent contours $\boldsymbol{\gamma}(s,t)$. Then the image velocity
$\boldsymbol{\gamma}_{t}$ at $t=0$ can be determined from
$\boldsymbol{\gamma}$ by $0pt$ and the infinitesimal motion parameters using
Equation 5.14, i.e., the same one used for a stationary contour.

###### Proof

Recall from Equation 2.23 that the velocity of an occluding contour under
epipolar parametrization statisfies
$\boldsymbol{\Gamma}^{w}_{t}=\lambda(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})$
for some $\lambda$, so that at $t=0$,

$\boldsymbol{\Gamma}^{w}_{t}=\lambda
0pt\boldsymbol{\gamma}\implies\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}_{t}^{w}=\lambda
0pt,$ (5.35)

so that
$\boldsymbol{\Gamma}^{w}_{t}=(\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t})\boldsymbol{\gamma}$
and the terms
$\boldsymbol{\Gamma}^{w}_{t}-(\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t})\boldsymbol{\gamma}=0$
so that all appearances of $\boldsymbol{\Gamma}^{w}_{t}$ cancel-out altogether
in Equation 5.3, giving exactly the same formula as for fixed contours,
Equation 5.14, when $\boldsymbol{\Gamma}^{w}_{t}=0$.

We now show exactly how the velocity of the 3D occluding contour,
$\boldsymbol{\Gamma}^{w}_{t}$, depends on the curvature of the occluding
surface Cipolla (1991); Cipolla and Blake (1992).

###### Theorem 5.4

The velocity of a 3D occluding contour under epipolar parametrization and
relative to a fixed world coordinate system (camera at $t=0$) is given by

$\displaystyle\boldsymbol{\Gamma}_{t}^{w}$
$\displaystyle=-\frac{\boldsymbol{c}_{t}^{\top}\boldsymbol{N}^{w}}{K^{t}}\cdot\frac{\boldsymbol{\Gamma}^{w}-\boldsymbol{c}}{\|\boldsymbol{\Gamma}^{w}-\boldsymbol{c}\|^{2}},$
for arbitrary $t$. (5.36) $\displaystyle\boldsymbol{\Gamma}_{t}^{w}$
$\displaystyle=-\frac{\boldsymbol{c}_{t}^{\top}\boldsymbol{N}}{K^{t}}\cdot\frac{\boldsymbol{\gamma}}{0pt\|\boldsymbol{\gamma}\|^{2}},$
for $t=0$, (5.37)

or, in terms of $\boldsymbol{\mathcal{T}}$ and $\boldsymbol{\mathcal{R}}$, and
image measurements,

$\boldsymbol{\Gamma}_{t}^{w}=\frac{1}{K^{t}}\left(\frac{\mathbit{V}^{\top}}{0pt}\,\frac{\boldsymbol{\gamma}\times\boldsymbol{t}}{\|\boldsymbol{\gamma}\times\boldsymbol{t}\|}\right)\frac{\boldsymbol{\gamma}}{\|\boldsymbol{\gamma}\|^{2}},\
\ \ \ \ \text{ for $t=0$,}$ (5.38)

where $K^{t}$ is the normal curvature of the occluding surface along the
visual direction.

###### Proof

The desired _formulae_ can be consistently derived by adapting variant Astrom
et al (1999) of the original result by Cipolla and Blake Cipolla (1991);
Cipolla and Blake (1992) to the proposed notation. This must be performed
carefully to establish correctness in a solid way. We thus provide an
alternative, clearer proof without using unit view spheres.

The normal curvature of the occluding surface along the visual direction is
given by classical differential geometry Cipolla and Giblin (1999) as

$K^{t}=-\frac{\boldsymbol{\Gamma}^{w\top}_{t}\boldsymbol{N}^{w}_{t}}{\boldsymbol{\Gamma}^{w\top}_{t}\boldsymbol{\Gamma}^{w}_{t}},$
(5.39)

using epipolar parametrization. Substituting the epipolar parametrization
condition of the second form of (2.23),

$K^{t}=-\frac{(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})^{\top}\boldsymbol{N}^{w}_{t}}{\lambda\|\boldsymbol{\Gamma}^{w}-\boldsymbol{c}\|^{2}}.$
(5.40)

Isolating $\lambda$ and plugging back into the epipolar parametrization
condition,

$\boldsymbol{\Gamma}_{t}^{w}=-\frac{(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})^{\top}\boldsymbol{N}^{w}_{t}}{K^{t}}\,\frac{\boldsymbol{\Gamma}^{w}-\boldsymbol{c}}{\|\boldsymbol{\Gamma}^{w}-\boldsymbol{c}\|^{2}}.$
(5.41)

We now show that
$(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})^{\top}\boldsymbol{N}^{w}_{t}=-\boldsymbol{c}_{t}^{\top}\boldsymbol{N}^{w}$,
thereby arriving at the desired expression for $\boldsymbol{\Gamma}_{t}^{w}$.
In fact, differentiating the occluding contour condition in the second form of
Equation 2.22 gives

$\displaystyle(\boldsymbol{\Gamma}_{t}^{w}-\boldsymbol{c}_{t})^{\top}\boldsymbol{N}^{w}+(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})^{\top}\boldsymbol{N}_{t}^{w}$
$\displaystyle=0,$ (5.42)
$\displaystyle-\boldsymbol{c}_{t}^{\top}\boldsymbol{N}^{w}+(\boldsymbol{\Gamma}^{w}-\boldsymbol{c})^{\top}\boldsymbol{N}_{t}^{w}$
$\displaystyle=0$ (5.43)

which, together with (5.41) produces the desired result

$\boldsymbol{\Gamma}_{t}^{w}=-\frac{\boldsymbol{c}_{t}^{\top}\boldsymbol{N}^{w}}{K^{t}}\,\frac{\boldsymbol{\Gamma}^{w}-\boldsymbol{c}}{\|\boldsymbol{\Gamma}^{w}-\boldsymbol{c}\|^{2}},\
\ \ \ \ \text{ for arbitrary $t$.}$ (5.44)

At $t=0$, we have $\boldsymbol{N}^{w}=\boldsymbol{N}$ and
$\boldsymbol{\Gamma}^{w}-\boldsymbol{c}=\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}$
(but note that
$\boldsymbol{\Gamma}_{t}(0)\neq\boldsymbol{\Gamma}^{w}_{t}(0)$), hence

$\displaystyle\boldsymbol{\Gamma}_{t}^{w}$
$\displaystyle=-\frac{\boldsymbol{c}_{t}^{\top}\boldsymbol{N}}{K^{t}}\,\frac{\boldsymbol{\gamma}}{0pt\|\boldsymbol{\gamma}\|^{2}},\
\ \ \ \ \text{ for $t=0$.}$ (5.45)

Using $\mathbit{V}=-\boldsymbol{c}_{t}$ from Equation 2.12 and
$\boldsymbol{N}=\frac{\boldsymbol{\gamma}\times\boldsymbol{t}}{\|\boldsymbol{\gamma}\times\boldsymbol{t}\|}$
gives the alternative form of this equation.

We now present a theorem relating observed quantities to camera motion, which
is key for calibrating 3D motion models from families of projected deforming
contours observed in video sequences with unknown camera motion, among other
applications. A form of this theorem appears in Papadopoulo and Faugeras
(1996); Papadopoulo (1996), Equation L1, but this is limited to rigid motion.
The following theorem generalizes the results to include occluding contours.
The fact that Equation LABEL:eq:papadopoulo:l1 in the theorem is also valid
for occluding contours is a new result, to the best of our knowledge. The term
$\boldsymbol{\Gamma}_{t}^{w}$ is zero for fixed contours, and is dependent on
surface curvature in the case of occluding contours. The equation is not valid
for arbitrary non-rigid contours because, in order to derive the normal flow
equation, we used
$\boldsymbol{\Gamma}^{w}_{t}\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})=0$,
which is only true for occluding and fixed contours.

###### Theorem 5.5

(A generalized form of the L1 equation of Papadopoulo and Faugeras (1996);
Papadopoulo (1996)) Given a 3D occluding contour or fixed curve, and the
family of projected curves $\boldsymbol{\gamma}(t)$ observed in a monocular
sequence of images from a moving camera, and given
$\boldsymbol{t},\kappa,\boldsymbol{n},\beta,\beta_{t}$ measurements at one
point, then the first and second order camera motion, $\mathbit{\Omega}$,
$\mathbit{V}$, $\mathbit{\Omega}_{t}$, $\mathbit{V}_{t}$ satisfy the
polynomial equation

$\displaystyle
V_{z}\,[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]^{2}+\mathbit{V}\cdot\boldsymbol{\gamma}\times\boldsymbol{t}\left(\beta_{t}-\mathbit{\Omega}_{t}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})-\mathbit{\Omega}\cdot\left[\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})\right]_{t}\right)$
(5.46)
$\displaystyle-\left[\mathbit{V}_{t}\cdot\boldsymbol{\gamma}\times\boldsymbol{t}+\mathbit{V}\cdot(\boldsymbol{\gamma}\times\boldsymbol{t})_{t}\right][\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]+\mathbit{V}\cdot\boldsymbol{\gamma}\times\boldsymbol{t}\,(\boldsymbol{e}_{3}\cdot{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]$
$\displaystyle+\boldsymbol{e}_{3}\cdot\boldsymbol{\Gamma}_{t}^{w}[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]^{2}+({\mathbit{\Omega}}_{\times}\mathbit{V})(\boldsymbol{\gamma}\times\boldsymbol{t})[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]=0.$

###### Proof

The normal velocity $\beta$ of an image contour follows Equation (5.30), which
holds for both stationary curves, Corollary 4, and for occluding contours,
Theorem 5.3. Differentiating it with respect to time,

$\displaystyle 0pt_{t}\beta+\beta_{t}0pt=$ $\displaystyle
0pt_{t}\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})+0pt\mathbit{\Omega}_{t}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})+0pt\mathbit{\Omega}[\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]_{t}$
(5.47)
$\displaystyle+(\boldsymbol{\gamma}\times\boldsymbol{t})_{t}(\mathbit{V}-{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}})+(\boldsymbol{\gamma}\times\boldsymbol{t})(\mathbit{V}_{t}-\mathbit{\Omega}_{t}\times\boldsymbol{\mathcal{T}}-{\mathbit{\Omega}}_{\times}\mathbit{V})$

Rearraging the terms,

$\displaystyle
0pt_{t}[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]+0pt[\beta_{t}-\mathbit{\Omega}_{t}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})-\mathbit{\Omega}\cdot[\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]_{t}]$
$\displaystyle=(\boldsymbol{\gamma}\times\boldsymbol{t})_{t}(\mathbit{V}-{\mathbit{\Omega}}_{\times}\boldsymbol{\mathcal{T}})+(\boldsymbol{\gamma}\times\boldsymbol{t})(\mathbit{V}_{t}-\mathbit{\Omega}_{t}\times\boldsymbol{\mathcal{T}}-{\mathbit{\Omega}}_{\times}\mathbit{V}).$
(5.48) Setting $t=0$, $\displaystyle
0pt_{t}[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]+0pt[\beta_{t}-\mathbit{\Omega}_{t}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})-\mathbit{\Omega}\cdot[\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]_{t}]$
$\displaystyle=(\boldsymbol{\gamma}\times\boldsymbol{t})_{t}\mathbit{V}+(\boldsymbol{\gamma}\times\boldsymbol{t})(\mathbit{V}_{t}-{\mathbit{\Omega}}_{\times}\mathbit{V})$
(5.49)

Now, from Equation 5.1, we can plug-in an expression for $0pt_{t}$ at $t=0$,

$\displaystyle(0pt\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+V_{z}+\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t})[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]+0pt(\beta_{t}-\mathbit{\Omega}_{t}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})-\mathbit{\Omega}\cdot[\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]_{t})$
$\displaystyle=(\boldsymbol{\gamma}\times\boldsymbol{t})_{t}\mathbit{V}+(\boldsymbol{\gamma}\times\boldsymbol{t})(\mathbit{V}_{t}-{\mathbit{\Omega}}_{\times}\mathbit{V}),$
(5.50)

which is analogous to Equation 7.28 of (Papadopoulo, 1996, p.167), but this
time with occluding contours also being included. Now, eliminating depth $0pt$
using Equation 5.32, e.g., by multiplying the above by
$[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]$,
we obtain

$\displaystyle[V_{z}(\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t}))+(\mathbit{V}\cdot\boldsymbol{\gamma}\times\boldsymbol{t})\,\boldsymbol{e}_{3}\cdot{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+$
(5.51)
$\displaystyle\boldsymbol{e}_{3}\cdot\boldsymbol{\Gamma}_{t}^{w}(\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t}))]\,[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]$
$\displaystyle+\mathbit{V}\cdot\boldsymbol{\gamma}\times\boldsymbol{t}\left[\beta_{t}-\mathbit{\Omega}_{t}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})-\mathbit{\Omega}\cdot[\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})]_{t}\right]=$
$\displaystyle[\mathbit{V}_{t}\cdot\boldsymbol{\gamma}\times\boldsymbol{t}+\mathbit{V}(\boldsymbol{\gamma}\times\boldsymbol{t})_{t}-({\mathbit{\Omega}}_{\times}\mathbit{V})(\boldsymbol{\gamma}\times\boldsymbol{t})]\,[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})].$

Rearranging the terms, we obtain the desired equation.

###### Remark 2

Note that previously reported results for the rigid case (Papadopoulo, 1996,
eq. 7.12) have an apparently missing term corresponding to the last term in
our Equation LABEL:eq:papadopoulo:l1,

$({\mathbit{\Omega}}_{\times}\mathbit{V})(\boldsymbol{\gamma}\times\boldsymbol{t})[\beta-\mathbit{\Omega}\cdot\boldsymbol{\gamma}\times(\boldsymbol{\gamma}\times\boldsymbol{t})].$

This is due to the fact that they used slightly different variables for the
translational component of the infinitesimal motion equations, but the results
are mathematically the same for the rigid case.

###### Theorem 5.6

The first spatial derivative of image apparent motion of both a fixed curve
and an occluding contour under epipolar correspondence is given by

$\boldsymbol{\gamma}_{st}=\left(-\frac{\mathbit{V}}{0pt}+\frac{V_{z}}{0pt}\boldsymbol{\gamma}\right)\frac{0pt_{s}}{0pt}-\frac{V_{z}}{0pt}\boldsymbol{\gamma}_{s}+{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}_{s}-(\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}_{s})\boldsymbol{\gamma}-(\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})\boldsymbol{\gamma}_{s}.$
(5.52)

Note that the derivative of depth $0pt_{s}$ can be expressed in terms of 3D
curve geometry as $0pt_{s}=\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}_{s}$.

###### Proof

Equation 5.52 follows by differentiating the fixed flow (5.14) with respect to
$s$, observing that only $0pt$ and $\boldsymbol{\gamma}$ depend on $s$. The
formula for $0pt_{s}$ is obtained from the observation that the dot product of
$\boldsymbol{\Gamma}=0pt\boldsymbol{\gamma}$ with $\boldsymbol{e}_{3}$ gives
$\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}=0pt$. Differentiating this with
respect to $s$ gives
$0pt_{s}=\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}_{s}$.

Theorem 5.1 gives an expression for the image acceleration of a moving 3D
point, which includes points lying on any type of contour (even non-rigid), in
terms of the evolution of the 3D curve. Since the latter is expressed in terms
of a fixed world coordinate system, the motion of the object and the motion of
the cameras are written down separately, even though they exert joint effects
on image velocity.

###### Theorem 5.7

The image acceleration of an occluding contour under epipolar parametrization
is given by

$\displaystyle\boldsymbol{\gamma}_{tt}$
$\displaystyle=({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}-[\boldsymbol{e}_{3}^{\top}({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}]\boldsymbol{\gamma}+2{\mathbit{\Omega}}_{\times}\frac{\boldsymbol{\Gamma}^{w}_{t}}{0pt}+\frac{\mathbit{V}_{t}}{0pt}-\frac{20pt_{t}}{0pt}\boldsymbol{\gamma}_{t}+\frac{\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t}}{0pt}\boldsymbol{\gamma}_{t}$
(5.53)
$\displaystyle-\frac{\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t}}{0pt}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}-\frac{\boldsymbol{e}_{3}^{\top}\mathbit{V}_{t}}{0pt}\boldsymbol{\gamma}-\frac{2\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}^{w}_{t}}{0pt}\boldsymbol{\gamma}+\frac{(e_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t})(\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})}{0pt}\boldsymbol{\gamma}\qquad\qquad\text{at
$t=0$,}$

where $\boldsymbol{\gamma}_{t}$ and $0pt_{t}$ are given by Equations 5.14 and
5.13, and $\boldsymbol{\Gamma}^{w}_{t}$ is dependent on curvature, Equation
5.38.

###### Proof

Substituting Equation 5.2 into Equation 5.12, we get

$\boldsymbol{\gamma}_{tt}=({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}+\frac{2{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}_{t}^{w}}{0pt}+\frac{\boldsymbol{\Gamma}^{w}_{tt}}{0pt}+\frac{\mathbit{V}_{t}}{0pt}-\frac{20pt_{t}\boldsymbol{\gamma}_{t}}{0pt}-[\boldsymbol{e}_{3}^{\top}({\mathbit{\Omega}}_{\times}^{2}+{[\mathbit{\Omega}_{t}]}_{\times})\boldsymbol{\gamma}]\boldsymbol{\gamma}-\frac{\boldsymbol{e}_{3}^{\top}\mathbit{V}_{t}}{0pt}\boldsymbol{\gamma}-\frac{2\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\Gamma}_{t}^{w}}{0pt}\boldsymbol{\gamma}-\frac{\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}_{tt}^{w}}{0pt}\boldsymbol{\gamma}.$
(5.54)

Now, let $\boldsymbol{v}$ be the viewing direction in world coordinates, so
that

$\boldsymbol{\gamma}=\boldsymbol{\mathcal{R}}\boldsymbol{v},$ (5.55)

and let $\boldsymbol{f}$ be the normal to the image plane in world
coordinates, so that

$\boldsymbol{e}_{3}=\boldsymbol{\mathcal{R}}\boldsymbol{f}.$ (5.56)

Thus,

$\boldsymbol{e}_{3}^{\top}\boldsymbol{\gamma}=\boldsymbol{f}^{\top}\boldsymbol{\mathcal{R}}^{\top}\boldsymbol{\mathcal{R}}\boldsymbol{v}=\boldsymbol{f}^{\top}\boldsymbol{v}=1.$
(5.57)

Note also that at $t=0$ we have $\boldsymbol{f}=\boldsymbol{e}_{3}$ and
$\boldsymbol{\gamma}=\boldsymbol{v}$. Now, the condition for epipolar
parametrization of an occluding contour, Equation 2.23, can be expressed as

$\boldsymbol{\Gamma}^{w}_{t}=\lambda\boldsymbol{v},$ (5.58)

for some scalar factor $\lambda$. Taking the dot product with $\boldsymbol{f}$
we have

$\displaystyle\lambda$
$\displaystyle=\boldsymbol{f}^{\top}\boldsymbol{\Gamma}^{w}_{t},$ (5.59)
$\displaystyle\boldsymbol{\Gamma}_{t}^{w}$
$\displaystyle=\boldsymbol{f}^{\top}\boldsymbol{\Gamma}_{t}^{w}\boldsymbol{v}.$
(5.60)

Differentiating (5.58) with respect to time and using (5.59) gives

$\displaystyle\boldsymbol{\Gamma}_{tt}^{w}=\lambda_{t}\boldsymbol{v}+\lambda\boldsymbol{v}_{t}$
$\displaystyle=\lambda_{t}\boldsymbol{v}+\boldsymbol{f}^{\top}\boldsymbol{\Gamma}_{t}^{w}\boldsymbol{v}_{t}.$
(5.61) Taking the dot product with $\boldsymbol{f}$,
$\displaystyle\boldsymbol{f}^{\top}\lambda_{t}\boldsymbol{v}+\boldsymbol{f}^{\top}(\boldsymbol{f}^{\top}\boldsymbol{\Gamma}_{t}^{w})\boldsymbol{v}_{t}$
$\displaystyle=\boldsymbol{f}^{\top}\boldsymbol{\Gamma}^{w}_{tt}$ (5.62)
$\displaystyle\lambda_{t}$
$\displaystyle=\boldsymbol{f}^{\top}\boldsymbol{\Gamma}^{w}_{tt}-(\boldsymbol{f}^{\top}\boldsymbol{\Gamma}_{t}^{w})\boldsymbol{f}^{\top}\boldsymbol{v}_{t}.$
(5.63) Thus,
$\displaystyle\boldsymbol{\Gamma}^{w}_{tt}=(\boldsymbol{f}^{\top}\boldsymbol{\Gamma}^{w}_{tt})\boldsymbol{v}-(\boldsymbol{f}^{\top}$
$\displaystyle\boldsymbol{\Gamma}^{w}_{t})(\boldsymbol{f}^{\top}\boldsymbol{v}_{t})\boldsymbol{v}+(\boldsymbol{f}^{\top}\boldsymbol{\Gamma}^{w}_{t})\boldsymbol{v}_{t},$
(5.64)
$\displaystyle\boldsymbol{\Gamma}^{w}_{tt}=\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{tt}\boldsymbol{\gamma}-(\boldsymbol{e}_{3}^{\top}$
$\displaystyle\boldsymbol{\Gamma}^{w}_{t})(\boldsymbol{e}_{3}^{\top}\boldsymbol{v}_{t})\boldsymbol{\gamma}+\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}_{t}^{w}\boldsymbol{v}_{t}$
$\displaystyle\text{at $t=0$}.$ (5.65)

In order to get $\boldsymbol{v}_{t}(0)$ in terms of $\boldsymbol{\gamma}$ we
write

$\displaystyle\boldsymbol{\gamma}_{t}$
$\displaystyle=\boldsymbol{\mathcal{R}}_{t}\boldsymbol{v}+\boldsymbol{\mathcal{R}}\boldsymbol{v}_{t}.$
(5.66) Thus $\displaystyle\boldsymbol{v}_{t}$
$\displaystyle=\boldsymbol{\gamma}_{t}-{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}\
\ \ \ \ \ \ \ \text{at $t=0$.}$ (5.67) Substituting back into (5.65),
$\displaystyle\boldsymbol{\Gamma}^{w}_{tt}=\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{tt}\boldsymbol{\gamma}+\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t}\boldsymbol{\gamma}_{t}$
$\displaystyle-\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}_{t}^{w}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma}+(\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{t})(\boldsymbol{e}_{3}^{\top}{\mathbit{\Omega}}_{\times}\boldsymbol{\gamma})\boldsymbol{\gamma}$
at $t=0$. (5.68)

Plugging this equation onto (5.54), the
$\boldsymbol{e}_{3}^{\top}\boldsymbol{\Gamma}^{w}_{tt}\boldsymbol{\gamma}/0pt$
terms cancel out, giving the final equation.

##  6 Mathematical Experiment

To illustrate and test the proposed theoretical framework, we have devised an
experiment around a synthetic dataset constructed for this research. This
dataset has already been used for validating a pose estimation system Fabbri
et al (2012). The dataset is composed of the following components:

  1. 1.

A variety of synthetically generated 3D curves (helices, parabolas, ellipses,
straight lines, and saddle curves) with well-known parametric equations, as
shown in Figure 5.

  2. 2.

Ground-truth camera models for a video sequence around the curves.

  3. 3.

Differential geometry of the space curves analytically computed up to third-
order (torsion and curvature derivative), using Maple when necessary. The
dataset together with C++ code implementing these expressions from Maple are
listed in the supplementary material Online Resource 1.

  4. 4.

The 3D curves are densely sampled, each 3D sample having attributed
differential geometry from the analytic computation (up to torsion and
curvature).

  5. 5.

A video sequence containing a family of 2D curve samples with attributed
differential geometry is rendered by projecting the 3D samples onto a
$500\times 400$ view using the ground truth cameras. These subpixel edgels
with attributed differential geometry simulate ideal aspects of what in
practice could be the output of high-quality subpixel edge detection and
grouping Tamrakar and Kimia (2007); Tamrakar (2008); Guo et al (2014).

  6. 6.

Correspondence between all samples obtained by keeping track of the underlying
3D points.

  7. 7.

Specific analytic expressions for 2D differential geometry were derived using
Maple since these are often too long due to perspective projection. These
expressions are also provided in the C++ code that synthesizes the dataset.

  8. 8.

C++ code implementing the formulas in this paper is also provided with the
dataset, and can be readily used in other projects.

For the present theoretical paper, the experiments consist of checking the
proposed expressions against the analytic expressions that are obtained by
differentiating each specific parametric equation. After projecting
differential geometry using our formulas applied to the 3D samples attributed
with differential geometry, we obtain corresponding 2D projected differential
geometry at each sample. We compare this to the differential geometry on the
curve projections analytically computed from the parametric equations,
observing a match. We then reconstruct these correspondences up to third-order
differential geometry using the proposed expressions, and observe that they
indeed match to the original analytic expressions from Maple. We have also
performed a similar experiment for the expressions involving occluding
contours, using a 3D ellipsoid and sphere.

We have observed a complete agreement between our code and the specific
analytic expressions, confirming that the formulas as presented in this
manuscript are correct. The source code of this illustrative experiment also
serves as an example of how to use the proposed framework in programming
practice, how to check for degenerate conditions stated in the theorems, among
others.

Figure 5:  Two views of the synthetic multiview curve differential geometry
dataset Fabbri et al (2012).

##  7 Conclusion

We presented a unified differential-geometric theory of projection and
reconstruction of general curves from multiple views. By gathering previously
scattered results using a coherent notation and proof methodology that scale
to expressing more sophisticated ideas, we were able to prove novel results
and to provide a comprehensive study on how the differential geometry of
curves behaves under perspective projection, including the effects of
intrinsic parameters. For instance, we derived how the tangent, curvature, and
curvature derivative of a space curve projects onto an image, and how the
motion of the camera and of the curve relates to the projections. This lead to
the _novel_ result that torsion – which characterizes the tri-dimensionality
of space curves – projects to curvature derivative in an image, and the
_novel_ result of how the parametrization of corresponding image curves are
linked across different views, up to third order to reflect the underlying
torsion. We also proved formulas for reconstructing differential geometry,
given differential geometry at corresponding points measured in at least two
views. In particular, this gives the _novel_ result of reconstructing space
curve torsion, given corresponding points, tangents, curvatures, and curvature
derivatives measured in two views. We determined that there are no
correspondence constraints in two views – any pair of points with attributed
tangents, curvatures, and curvature derivatives are possible matches, as long
as the basic point epipolar constraint is satisfied. There is, however, a
constraint in three or more views: from two views one can transfer
differential geometry onto other views and enforce measurements to match the
reprojections using local curve shape, avoiding clutter. This has been
demonstrated in a recent work in curve-based multiview stereo by the authors
Fabbri and Kimia (2010); Usumezbas et al (2016), namely for matching linked
curve fragments from a subpixel edge detector across many views. Experiments
clearly show that differential-geometric curve structure is essential for the
matching to be immune to edge clutter and linking instability, by enforcing
reprojections to match to image data in _local_ shape.

This paper is part of a greater effort of augmenting multiple view geometry to
model general curved structures Fabbri (2010). Work on camera pose estimation
based on curves using the formulas in this paper has been recently published
Fabbri et al (2012), and trifocal relative pose estimation using local curve
geometry to bootstrap the system is currently under investigation. In a
complete system, once a core set of three views are estimated and an initial
set of curves are reconstructed, more views can be iteratively registered
Fabbri et al (2012). These applications of the theory presented in this paper
and their ongoing extensions would enable a practical structure from motion
pipeline based on curve fragments, complementing interest point technology. We
have also been working on the multiview differential geometry of surfaces and
their shading.

## References

  * Agarwal et al (2009) Agarwal S, Snavely N, Simon I, Seitz SM, Szeliski R (2009) Building Rome in a day. In: Proceedings of the IEEE International Conference on Computer Vision, IEEE Computer Society
  * Arnold and Binford (1980) Arnold R, Binford T (1980) Geometric constraints in stereo vision. In: Proc. SPIE, San Diego, CA, vol 238–Image Processing for Missile Guidance, pp 281–292
  * Astrom and Heyden (1996) Astrom K, Heyden A (1996) Multilinear constraints in the infinitesimal-time case. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press, San Francisco, California, pp 833–838
  * Åström and Heyden (1998) Åström K, Heyden A (1998) Continuous time matching constraints for image streams. International Journal of Computer Vision 28(1):85–96, DOI 10.1023/A:1008006815607, URL http://dx.doi.org/10.1023/A:1008006815607
  * Astrom and Kahl (1999) Astrom K, Kahl F (1999) Motion estimation in image sequences using the deformation of apparent contours. IEEE Trans Pattern Anal Mach Intell 21(2):114–127, DOI http://dx.doi.org/10.1109/34.748821
  * Astrom et al (1999) Astrom K, Cipolla R, Giblin P (1999) Generalised epipolar constraints. Int J Comput Vision 33(1):51–72, DOI http://dx.doi.org/10.1023/A:1008113231241
  * Ayache and Lustman (1987) Ayache N, Lustman L (1987) Fast and reliable passive trinocular stereovision. In: $1^{st}$ International Conference on Computer Vision
  * Baatz et al (2012) Baatz G, Saurer O, Köser K, Pollefeys M (2012) Large scale visual geo-localization of images in mountainous terrain. In: Proceedings of the 12th European Conference on Computer Vision - Volume Part II, Springer-Verlag, Berlin, Heidelberg, ECCV’12, pp 517–530, DOI 10.1007/978-3-642-33709-3˙37, URL http://dx.doi.org/10.1007/978-3-642-33709-3_37
  * Baumela et al (2000) Baumela L, Agapito L, Bustos P, Reid I (2000) Motion estimation using the differential epipolar equation. Proceedings of the 15th International Conference on Pattern Recognition 3:848–851
  * Berthilsson et al (2001) Berthilsson R, Åström K, Heyden A (2001) Reconstruction of general curves, using factorization and bundle adjustment. International Journal of Computer Vision 41(3):171–182, DOI http://dx.doi.org/10.1023/A:1011104020586
  * Brodskỳ and Fermüller (2002) Brodskỳ T, Fermüller C (2002) Self-Calibration from Image Derivatives. International Journal of Computer Vision 48(2):91–114
  * Brodský et al (2000) Brodský T, Fermüller C, Aloimonos Y (2000) Structure from motion: Beyond the epipolar constraint. Int J Comput Vision 37(3):231–258, DOI http://dx.doi.org/10.1023/A:1008132107950
  * Brooks et al (1997) Brooks MJ, Chojnacki W, Baumela L (1997) Determining the egomotion of an uncalibrated camera from instantaneous optical flow. J Opt Soc Am A 14(10):2670–2677, DOI 10.1364/JOSAA.14.002670, URL http://josaa.osa.org/abstract.cfm?URI=josaa-14-10-2670
  * Calakli et al (2012) Calakli F, Ulusoy AO, Restrepo MI, Taubin G, Mundy JL (2012) High resolution surface reconstruction from multi-view aerial imagery. In: 3DIMPVT’12, IEEE, pp 25–32
  * Carceroni (2001) Carceroni R (2001) Recovering non-rigid 3D motion, shape and reflectance from multi-view image sequences: A differential-geometric approach. PhD thesis, University of Rochester
  * Carceroni and Kutulakos (1999) Carceroni R, Kutulakos K (1999) Toward recovering shape and motion of 3D curves from multi-view image sequences. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press, Fort Collins, Colorado, USA, pp 23–25
  * do Carmo (1976) do Carmo MP (1976) Differential Geometry of Curves and Surfaces. Prentice-Hall, New Jersey
  * Chen and Klette (2014) Chen TY, Klette R (2014) Animated non-photorealistic rendering in multiple styles. In: Image and Video Technology–PSIVT 2013 Workshops, Springer, pp 12–23
  * Cipolla (1991) Cipolla R (1991) Active visual inference of surface shape. Ph.D. dissertation, Univesity of Oxford
  * Cipolla and Blake (1992) Cipolla R, Blake A (1992) Surface shape from the deformation of apparent contours. International Journal of Computer Vision 9(2):83–112
  * Cipolla and Giblin (1999) Cipolla R, Giblin P (1999) Visual Motion of Curves and Surfaces. Cambridge University Press
  * Cipolla and Zisserman (1992) Cipolla R, Zisserman A (1992) Qualitative surface shape from deformation of image curves. International Journal of Computer Vision 8(1):53–69
  * Cipolla et al (1995) Cipolla R, Åström K, Giblin PJ (1995) Motion from the frontier of curved surfaces. In: Proceedings of the IEEE International Conference on Computer Vision, IEEE Computer Society Press, Boston, Massachusetts, pp 269–275
  * Cole et al (2009) Cole F, Sanik K, DeCarlo D, Finkelstein A, Funkhouser T, Rusinkiewicz S, Singh M (2009) How well do line drawings depict shape? In: ACM Transactions on Graphics (Proc. SIGGRAPH)
  * Diskin and Asari (2015) Diskin Y, Asari V (2015) Dense point-cloud representation of a scene using monocular vision. Journal of Electronic Imaging 24(2):023,003, DOI 10.1117/1.JEI.24.2.023003, URL http://dx.doi.org/10.1117/1.JEI.24.2.023003
  * Dornaika and Sappa (2006) Dornaika F, Sappa A (2006) 3D Motion from Image Derivatives Using the Least Trimmed Square Regression? LECTURE NOTES IN COMPUTER SCIENCE 4153:76
  * Fabbri (2010) Fabbri R (2010) Multiview differential geometry in application to computer vision. Ph.D. dissertation, Division Of Engineering, Brown University, Providence, RI, 02912
  * Fabbri and Kimia (2010) Fabbri R, Kimia BB (2010) 3D curve sketch: Flexible curve-based stereo reconstruction and calibration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press, San Francisco, CA, USA
  * Fabbri et al (2012) Fabbri R, Kimia BB, Giblin PJ (2012) Camera pose estimation using first-order curve differential geometry. In: Proceedings of European Conference on Computer Vision, Springer, Lecture Notes in Computer Science, pp 231–244
  * Fathi et al (2015) Fathi H, Dai F, Lourakis M (2015) Automated as-built 3D reconstruction of civil infrastructure using computer vision: Achievements, opportunities, and challenges. Advanced Engineering Informatics (0):–, DOI http://dx.doi.org/10.1016/j.aei.2015.01.012, URL http://www.sciencedirect.com/science/article/pii/S1474034615000245
  * Faugeras (1990) Faugeras O (1990) Computer Vision — ECCV 90: First European Conference on Computer Vision Antibes, France, April 23–27, 1990 Proceedings, Springer Berlin Heidelberg, Berlin, Heidelberg, chap On the motion of 3D curves and its relationship to optical flow, pp 105–117. DOI 10.1007/BFb0014856, URL http://dx.doi.org/10.1007/BFb0014856
  * Faugeras and Papadopoulo (1992) Faugeras O, Papadopoulo T (1992) Geometric invariance in computer vision. MIT Press, Cambridge, MA, USA, chap Disambiguating Stereo Matches with Spatio-temporal Surfaces, pp 310–331, URL http://dl.acm.org/citation.cfm?id=153634.153650
  * Faugeras and Papadopoulo (1993) Faugeras O, Papadopoulo T (1993) A theory of the motion fields of curves. International Journal of Computer Vision 10(2):125–156
  * Fischler and Bolles (1981) Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395, DOI http://doi.acm.org/10.1145/358669.358692
  * Furukawa and Ponce (2007) Furukawa Y, Ponce J (2007) Accurate, dense, and robust multi-view stereopsis. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society
  * Furukawa and Ponce (2010) Furukawa Y, Ponce J (2010) Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence 32:1362–1376, DOI http://doi.ieeecomputersociety.org/10.1109/TPAMI.2009.161
  * Furukawa et al (2006) Furukawa Y, Sethi A, Ponce J, Kriegman DJ (2006) Robust structure and motion from outlines of smooth curved surfaces. IEEE Trans Pattern Anal Mach Intell 28(2):302–315
  * Giblin and Weiss (1995) Giblin PJ, Weiss RS (1995) Epipolar curves on surfaces. Image and Vision Computing 13(1):33–44
  * Goesele et al (2007) Goesele M, Snavely N, Curless B, Hoppe H, Seitz S (2007) Multi-view stereo for community photo collections. In: ICCV2007 (2007), pp 1–8
  * Grimson (1981) Grimson WEL (1981) A Computer Implementation of a Theory of Human Stereo Vision. Royal Society of London Philosophical Transactions Series B 292:217–253
  * Guo et al (2014) Guo Y, Kumar N, Narayanan M, Kimia B (2014) A multi-stage approach to curve extraction. In: CVPR’14
  * Habbecke and Kobbelt (2007) Habbecke M, Kobbelt L (2007) A surface-growing approach to multi-view stereo reconstruction. Computer Vision and Pattern Recognition, IEEE Computer Society Conference on 0:1–8, DOI http://doi.ieeecomputersociety.org/10.1109/CVPR.2007.383195
  * Harris and Stephens (1988) Harris C, Stephens M (1988) A combined edge and corner detector. In: Alvey Vision Conference, pp 189–192
  * Hartley and Zisserman (2000) Hartley R, Zisserman A (2000) Multiple View Geometry in Computer Vision. Cambridge University Press
  * Hartley (1995) Hartley RI (1995) A linear method for reconstruction from lines and points. In: Proceedings of the IEEE International Conference on Computer Vision, IEEE Computer Society Press, Boston, Massachusetts, pp 882–887
  * Heeger and Jepson (1992) Heeger DJ, Jepson AD (1992) Subspace methods for recovering rigid motion i: algorithm and implementation. Int J Comput Vision 7(2):95–117, DOI http://dx.doi.org/10.1007/BF00128130
  * Heinly et al (2015) Heinly J, Schönberger JL, Dunn E, Frahm JM (2015) Reconstructing the world in six days. In: Computer Vision and Pattern Recognition
  * van den Hengel (2000) van den Hengel A (2000) Robust estimation of structure from motion in the uncalibrated case. PhD thesis, Adelaide University
  * van den Hengel et al (2007) van den Hengel A, Chojnacki W, Brooks M (2007) Determining the Translational Speed of a Camera from Time-Varying Optical Flow. LECTURE NOTES IN COMPUTER SCIENCE 3417:190
  * Hernandez et al (2007) Hernandez C, Schmitt F, Cipolla R (2007) Silhouette coherence for camera calibration under circular motion. IEEE Trans Pattern Anal Mach Intell 29(2):343–349
  * Hernández Esteban and Schmitt (2004) Hernández Esteban C, Schmitt F (2004) Silhouette and stereo fusion for 3D object modeling. Computer Vision and Image Understanding 96(3):367–392
  * Heyden (2006) Heyden A (2006) Differential-Algebraic Multiview Constraints. In: Proceedings of the 18th International Conference on Pattern Recognition (ICPR’06)-Volume 01, IEEE Computer Society Washington, DC, USA, pp 159–162
  * ICCV2007 (2007) ICCV2007 (2007) 11th IEEE International Conference on Computer Vision (ICCV 2007), 14-20 October 2007, Rio de Janeiro, Brazil, IEEE Computer Society
  * Jain (2009) Jain V (2009) Motion segmentation using differential geometry of curves and edges. Ph.D. dissertation, Division Of Engineering, Brown University, Providence, RI, 02912
  * Jain et al (2007a) Jain V, Kimia BB, Mundy JL (2007a) Background modeling based on subpixel edges. In: IEEE International Conference on Image Processing, IEEE, San Antonio, TX, USA, vol IV, pp 321–324
  * Jain et al (2007b) Jain V, Kimia BB, Mundy JL (2007b) Segregation of moving objects using elastic matching. Computer Vision and Image Understanding 108:230–242
  * Kahl and Heyden (1998) Kahl F, Heyden A (1998) Using conic correspondence in two images to estimate the epipolar geometry. In: Proceedings of the IEEE International Conference on Computer Vision, IEEE Computer Society Press, Bombay, India, p 761
  * Kahl and Heyden (2001) Kahl F, Heyden A (2001) Euclidean reconstruction and auto-calibration from continuous motion. In: Proceedings of the IEEE International Conference on Computer Vision, IEEE Computer Society Press, Vancouver, Canada, vol 2
  * Kaminski and Shashua (2004) Kaminski JY, Shashua A (2004) Multiple view geometry of general algebraic curves. International Journal of Computer Vision 56(3):195–219
  * Kanatani (1993) Kanatani K (1993) 3-d interpretation of optical flow by renormalization. International Journal of Computer Vision 11(3):267–282, DOI 10.1007/BF01469345, URL http://dx.doi.org/10.1007/BF01469345
  * Kazhdan et al (2006) Kazhdan M, Bolitho M, Hoppe H (2006) Poisson surface reconstruction. In: SGP ’06: Proceedings of the fourth Eurographics symposium on Geometry processing, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, pp 61–70
  * Koenderink et al (2013) Koenderink J, van Doorn A, Wagemans J (2013) SfS? not likely…. i-Perception 4(5):299