# Multi-task Learning for Camera Calibration

###### Abstract

For a number of tasks, such as 3D reconstruction, robotic interface,
autonomous driving, etc., camera calibration is essential. In this study, we
present a unique method for predicting intrinsic (principal point offset and
focal length) and extrinsic (baseline, pitch, and translation) properties from
a pair of images. We suggested a novel method where camera model equations are
represented as a neural network in a multi-task learning framework, in
contrast to existing methods, which build a comprehensive solution. By
reconstructing the 3D points using a camera model neural network and then
using the loss in reconstruction to obtain the camera specifications, this
innovative camera projection loss (CPL) method allows us that the desired
parameters should be estimated. As far as we are aware, our approach is the
first one that uses an approach to multi-task learning that includes
mathematical formulas in a framework for learning to estimate camera
parameters to predict both the extrinsic and intrinsic parameters jointly.
Additionally, we provided a new dataset named as CVGL Camera Calibration
Dataset [1] which has been collected using the CARLA Simulator [2]. Actually,
we show that our suggested strategy outperforms both conventional methods and
methods based on deep learning on 8 out of 10 parameters that were assessed
using both real and synthetic data. Our code and generated dataset are
available at https://github.com/thanif/Camera-Calibration-through-Camera-
Projection-Loss.

Index Terms—  Camera Projection Loss, Multi-task learning, Camera Calibration

##  1 Introduction

Camera calibration is the process of identifying the six extrinsic (rotation,
translation) and five intrinsic (focal length, image sensor format, and
principal point) characteristics of a certain camera.

Fig. 1: Using a trained version of the Inception-v3 [3] algorithm and the proposed Camera Projection Loss, our method calculates intrinsic (principal point offset) and extrinsic (baseline, pitch, and translation) parameters |
---|---
Fig. 2: Street maps of towns from CARLA Simulator (a) First Town. (b) Second
Town. Table 1: Overview of some recent configurations for different aspects of
Camera Calibration

Paper Input Tilt Roll Focal Length Radial Distortion Pan Rotation Translation
Fundamental Matrix Position Orientation [4] RGB Image $\times$ $\times$
$\times$ $\times$ $\times$ $\times$ [5] RGB Image, Projected Radar Data
$\times$ $\times$ $\times$ $\times$ $\times$ $\times$ $\times$ [6] RGB Image,
Raw LiDAR point cloud $\times$ $\times$ $\times$ $\times$ $\times$ $\times$
$\times$ $\times$ [7] Stereo Image pair $\times$ $\times$ $\times$ $\times$
$\times$ $\times$ $\times$ $\times$ $\times$ [8, 9, 10, 11] RGB Image pair
$\times$ $\times$ $\times$ $\times$ $\times$ $\times$ $\times$ $\times$ [12]
RGB Image $\times$ $\times$ $\times$ $\times$ $\times$ $\times$ $\times$
$\times$ [13, 14, 15, 16, 17] RGB Image $\times$ $\times$ $\times$ $\times$
$\times$ $\times$ $\times$ $\times$ [18] RGB Image $\times$ $\times$ $\times$
$\times$ $\times$ $\times$ $\times$ [19] RGB Image $\times$ $\times$ $\times$
$\times$ $\times$ $\times$ $\times$ $\times$ $\times$ [20] Head Detections,
Focal Length $\times$ $\times$ $\times$ $\times$ $\times$ $\times$ $\times$
$\times$ [21] RGB Image $\times$ $\times$ $\times$ $\times$ $\times$ $\times$
$\times$ [22] Putative matches $\times$ $\times$ $\times$ $\times$ $\times$
$\times$ $\times$ $\times$ $\times$

When the camera model can be fit by enough geometric constraints, relatively
exact calibration of cameras is possible thanks to the well-understood nature
of picture generation and extensive research in computer vision [23]. The best
results come from photographs that show man-made surroundings since they
provide strong indicators like straight lines and vanishing points that can be
used to determine the camera’s settings. Images taken outside of a lab can
also be processed using geometrical approaches. [24, 25]. Geometric-based
techniques, on the other hand, are less resistant to photographs shot in
unstructured locations, with subpar equipment, or under challenging lighting
circumstances since they rely on recognising and processing particular
indications like straight lines and vanishing points.

A perception system needs a variety of on-board sensors to function well.
Robots are using an increasing number of sensors in many modalities. A 3D
LiDAR, for instance, is used in conjunction with 2D cameras in an autonomous
vehicle since the latter’s rich colour information complements the former’s
sparse distance information. Several calibration methods have been put forth
recently [26, 27, 28, 29, 30]. However, the majority of these techniques use
checkerboards as fixed calibration targets, and demand a substantial amount of
manual labour [26, 27]. Therefore, automatic calibration approaches that can
considerably increase the flexibility and adaptability of these systems are
very necessary. There have been some published methods in this field in the
past [28, 29, 30] although the majority of these methods still rely on precise
calibration parameter initialization [28, 29], or need a lot more information
on ego-motion [30].

The majority of currently used techniques typically don’t take into account
the camera model’s underlying mathematical formulation, instead proposing an
all-encompassing framework to calculate the required parameters directly [19,
31, 18, 4, 32, 33, 12, 34, 35, 36]. Since they can only utilizing a single
image, determine the camera’s focal length, they are difficult to understand
for practical purposes. [33, 12, 34]. The following are the main benefits of
our efforts:

  * •

To the best of our knowledge, this research represents the first learning-
based approach to jointly estimate both intrinsic and extrinsic camera
parameters, such as camera baseline, disparity, pitch, translation, focal
length, and principal point offset.

  * •

The available learning-based techniques [33, 19, 34] due to a paucity of data,
have not been used to estimate all of the $10$ camera parameters. To resolve
this difficulty, we created a synthetic dataset using data from two CARLA
towns [2] featuring 49 distinct camera setups.

  * •

Instead of developing a whole system to anticipate the needed parameters,
unlike existing approaches [33], In a multi-task learning (MTL) framework, we
developed a new illustration that depicts the equations for the camera model
as a neural network.

  * •

We put out a brand-new camera projection loss (CPL) that integrates
mathematical equations into a framework for learning.

Fig. 3: Cyclists are divided into three classes: wide (red), intermediate
(blue) and narrow (brown). [37]

##  2 Related Work

Table 1 provides a summary of several recent settings for various camera
calibration aspects. Recent studies have suggested employing learning
techniques to estimate camera parameters, building on the success of
convolutional neural networks. In the past, various aspects of the calibration
utilising photographs problem have been researched. Workman _et al._ [19]
trained a CNN to do regression of a pinhole camera’s field of view,
concentrating on horizon line detection afterwards [35], which, provided the
focal length is known, serves as a substitute for the camera’s tilt and roll
angles. Rong  _et al._ [31] calibrate the single-parameter radial distortion
model using a classification method from Fitzgibbon [38]. Hold-Geoffroy  _et
al._ [18] extrinsic and intrinsic calibration were first coupled in a single
network, using a classification method to predict the pinhole camera’s tilt,
roll, and focal length. They utilised upright 360-degree imagery, a technique
that has already been used to provide training data, to artificially create
any size, focal length, and rotation of the images [4]. Methods based on
learning and tradition can coexist. In [32], The execution time and robustness
compared to entirely geometric approaches are accelerated by employing
learning methods to get a previous distribution on prospective camera settings
is used to refine them using conventional techniques. In our paper, we do not
adopt such a methodology. However, in these pipelines, the prediction made by
our technique can be used as a prior. More recently, Deep-Calib [12, 4] by
creating distorted images with the unified projection model, incorporate the
single picture self-calibration problem’s distortion parameter. [36].

In Machine Learning (ML), we often train a single model or a collection of
models to solve the problem in this way. Whether it’s a score based on a
benchmark or a key performance indicator (KPI) for a firm, we care about
optimising for a specific measure. After that, the models are adjusted and
modified until their performance reaches a plateau. This method usually yields
good results, but we miss out on information that could have helped us do
better on the metric that matters to us. These data were created using
training signals from related activity. By swapping representations between
tasks that are comparable to our starting work, we can increase the
generalisation of our model. Hard parameter sharing is the MTL strategy that
neural networks use the most frequently. In most cases, this is done by
retaining a small number of output layers that are task-specific while sharing
the hidden layers across all jobs. When hard parameters are shared,
overfitting is significantly decreased. This makes sense logically: the more
tasks we learn concurrently, the more work our model needs to undertake to
build a representation that covers all of them, and the lower the likelihood
we have of overfitting our starting task. On the other hand, soft parameter
sharing means that each work has its own model and unique set of parameters.
Then, to promote parameter similarity, the difference between the model’s
parameters is regularised. Hard parameter sharing has been used in this work
for learning.

Particularly, estimate of intrinsic or extrinsic parameters has been the focus
of most previous applications of calibration employing multiple inputs except
[7], when a pair of stereo pictures are used to directly estimate the
fundamental matrix. [5] to calculate rotation angles, combine radar projection
data with an RGB image, [9] focus on estimating the relative camera pose from
a pair of rgb photos. [6] The 2D camera and 3D LiDAR’s extrinsic calibration
parameters are estimated using a rgb image from a raw LiDAR point cloud and a
calibrated camera as input. [10] estimate the extrinsic parameters of two
cameras by feeding their model two input photos that capture a scene from
various viewpoints. [8] without the use of camera intrinsic or external
information, takes two photos as input and immediately infers the relative
positions. [39] works in a variety of real-world situations, including
calibrating camera intrinsic characteristics and (potentially considerable)
lens distortion employing numerous photos of a known pattern, countless
photographs of an unknown pattern, numerous photographs of a variety of
patterns, etc.

The loss function used to calibrate a convolutional neural network is a
composite loss that includes a loss component for each parameter. Typically,
this situation is referred to as multi-task learning [40]. The difficulty of
preparing a network to do a variety of jobs with varying losses is a problem
that multi-task learning must solve. The majority of these strategies use a
weighted average of the loss factors; however, they vary in how the weights
are determined during training: Kendall  _et al._ [41] To weight the various
loss components in accordance with a task-dependent uncertainty, use softmax
and gaussian likelihoods (for regression and classification, respectively).
Unlike these strategies based on uncertainty, Chen  _et al._ [42] by varying
the magnitudes of the gradients connected to each loss term, one can estimate
the weights’ value.

###  2.1 Handcrafted Features Based Methods

[21] outlines a way to determine camera parameters such as focal length,
position, and orientation from a single photograph of a scene rectangle with
an unknown aspect ratio and size. They start by resolving the unique situation
in which a scene rectangle’s centre is projected onto the image’s centre.
Coupled line cameras are used to formulate this problem, and they then give an
analytical solution. They then resolve the general situation without the
centering requirement by adding a quick preprocessing step. They also offer a
way to determine whether a quadrilateral image is a projection of a rectangle.
They use simulated and actual data to show how well the suggested strategy
works. [11] describe an approach for an online, continuous, per-frame stereo
extrinsic re-calibration that can only be used with sparse stereo
correspondences. They are able to model time-dependent variations because they
are able to retrieve the fixed baseline’s five degrees of freedom extrinsic
position for each frame. Minimizing epipolar errors yields the initial
extrinsic estimates, which are then filtered with a Kalman filter (KF). The
lower bound of the solution uncertainty calculated by Crámer-Rao is used to
calculate observation covariances. [16] explains how to estimate the rotation
and translation matrices of a camera from a series of photos. It is
presumptively known and fixed camera intrinsic matrix and distortion
coefficients throughout the entire sequence. [27] a web-based toolset for
fully automatic calibration of cameras and distances between cameras. Since
most calibrations just require a single image and range scan cases,in the
event of ambiguities, the suggested camera-to-range registration strategy is
capable of finding a variety of solutions., making it robust to a variety of
simple to use, fully automatic, and imaging conditions. The durability of
their system is demonstrated in tests utilising a number of sensors, including
as colour and grayscale cameras, the Kinect 3D sensor, and the Velodyne HDL-64
laser scanner, in both indoor and outdoor settings, and in various lighting
situations. [39] describe a technique for calibrating a camera’s inherent
properties and (potentially severe) lens distortion. Furthermore, this
approach does not depend on low-level features like edges or corners for
feature extraction. The approach makes use of cutting-edge tools for high-
dimensional convex optimization, especially those for recovering sparse
signals and matrix rank minimization. They demonstrate how Principal Component
Analysis might be extended to address the camera calibration problem and
resolved using analogous methods. [43] initiate a conversation about whether
image-based 3-D modelling methods could take the place of LIDAR equipment in
the collection of outside 3D data. In this context, the internal and exterior
camera calibration, as well as dense multi-view stereo, are the two key
challenges that need to be addressed. They have collected test results from
outdoor LIDAR and camera situations to evaluate both. [44] presents a method
for camera calibration that effectively directs utilising a common bundle
adjustment approach, the search for further image correspondences and greatly
improves camera calibration parameters. This method uses imprecise camera
parameter estimates and the results of a multi-view stereo system on resized
input photos provide top-down information. [45] proposes a unique technique
for predicting extrinsic camera parameters using sparse GPS position data and
feature points from an image sequence. Their approach, which is enhanced by
the use of GPS data to minimise accumulative estimating errors and is based on
a structure-from-motion technique. Additionally, mistracked features are also
removed using the position data. The suggested technique enables estimation of
extrinsic parameters from extremely lengthy image sequences without
accumulation of errors. [46] determines both the intrinsic and extrinsic
camera settings while simultaneously using numerous photos of multiple balls.
Conics or ellipses that resemble balls are projected onto the image. There are
5 separate constraints for each conic. 30 limitations are provided by 3 balls
in 2 photos. There are 12 parameters for the placements and radii of the three
balls, and there are 15 intrinsic and extrinsic parameters altogether.
Therefore, the bare minimum configuration to tackle this problem is 3 balls in
2 photos. The ball placements and radii, as well as any unknown camera
parameters, are best inferred from the boundary points. The algorithm’s
effectiveness and reliability are demonstrated by experimental results on real
photos. Calibrating network in [47] can identify the matrix of perspective,
projection, and transformation between relevant 2D picture pixels and the 3D
world points. The parameters of a camera model can be specified online that
fulfil the orthogonality restrictions on the rotational transformation
starting with random initial weights.

Fig. 4: Statistical Comparison of Camera Parameters from 3 datasets used. Blue
colour denotes CVGL Camera Calibration Dataset while Green colour represents
Cityscapes Dataset and Orange colour has been used for Tsinghua-Daimler
Cyclist Detection Benchmark. The plots follow the following order from left to
right: $f_{x},f_{y},u_{0},v_{0},b,d,\theta_{p},t_{x},t_{y},t_{z}$. Table 2:
Table showing statistics of the used datasets.

Dataset Value $f_{x}$ $f_{y}$ $u_{0}$ $v_{0}$ $b$ $d$ $t_{x}$ $t_{y}$ $t_{z}$
$\theta_{p}$ CVGL Min 15.005 15.005 56.0 56.0 1 -16.0 1 1 1 30 Mean 84.502
84.502 56.0 56.0 3.134 -5.636 3.134 3.284 2.853 84.780 Max 208.994 208.994
56.0 56.0 5 -3.531 5 5 5 150 Tsinghua-Daimler Min 2282.864 2281.794 1042.041
529.888 0.208 -6.753 2.0 0.125 1.23 0.022 Mean 2282.864 2281.794 1042.041
529.888 0.208 21.455 2.0 0.125 1.23 0.021 Max 2282.864 2281.794 1042.041
529.888 0.208 83.093 2.0 0.125 1.23 0.022 Cityscapes Min 2262.52 2225.540
1045.53 513.137 0.209 -4.675 1.7 -0.1 1.18 0.038 Mean 2264.256 2253.738
1081.188 514.988 0.213 15.539 1.699 0.037 1.230 0.041 Max 2268.36 2265.301
1096.98 519.277 0.222 57.339 1.7 0.1 1.3 0.05

###  2.2 CNN Based Methods

[4] describe a technique for extrapolating from a single image both inherent
(radial distortion and focal length) and extrinsic (tilt and roll) properties.
Instead of simply predicting the distortion parameters, they suggest a more
learner-friendly parameterization for radial distortion. Additionally,
predicting more heterogeneous factors makes the difficulty of loss balancing
worse. Preventing the need to reconcile heterogeneous loss terms, they suggest
a novel point projection-based loss function. They are the first to use a
single image to simultaneously determine the focal length, radial distortion,
tilt, roll, and other parameters. [5] put forth the first data-driven
technique for automatic rotating radar-camera calibration without specific
calibration objectives. A coarse and a fine convolutional neural network form
the foundation of their strategy. They use a training approach that takes
inspiration from boosting and trains the fine network using the residual error
of the coarse network. They gathered their own real-world data because there
were no public databases combining radar and camera observations. [6], in
order to automatically estimate the 6-DoF rigid body switch in real time
between a 2D camera and a 3D LiDAR, they propose a geometrically supervised
deep network. CalibNet eliminates the requirement for calibration targets,
which significantly reduces the amount of calibration work required. The
network only accepts the corresponding monocular image, the camera calibration
matrix K, and a LiDAR point cloud as inputs during training. They don’t
enforce direct control throughout train time (i.e., For instance, they do not
directly regress to the calibration parameters. Instead, they train the
network to predict calibration parameters that, from a geometric and
photometric perspective, maximise the consistency of the input point clouds
and images. Without retraining or domain adaption, CalibNet gains the ability
to solve the fundamental geometry problem iteratively while properly
predicting criteria for extrinsic calibration for various miscalibrations.
[7], in order to estimate fundamental matrices they propose new neural network
topologies that can be extended end-to-end without relying on point
correspondences. To maintain the fundamental matrix’s mathematical
characteristics as a seven-degree-of-freedom homogeneous rank-2 matrix, new
modules and layers are added. [8], without relying on camera intrinsic or
external information, the proposed RPNet network receives pairs of photos as
input and infers the relative positions directly. [10] suggests using a deep
learning network architecture for multi-view environment relative camera
position estimation. The suggested network uses an AlexNet variant
architecture as a regressor to anticipate the output of relative translation
and rotation. The suggested method is trained from scratch on a sizable data
set using two photos from the same scene as input. [22] creates a model for
the issue that divides it into a number of weighted homogeneous least-squares
issues, with robust weights calculated by deep networks. The proposed approach
performs feature extraction, matching, and model fitting directly on putative
correspondences and so integrates into common 3D vision workflows. The method
produces robust estimators that are computationally effective and can be
trained from beginning to end. [12] automatically calculates, from a single
input image, the focal length and distortion parameter of the camera. In order
to automatically create a massive dataset of millions of wide field-of-view
photos with ground truth intrinsic parameters, they use the abundance of
omnidirectional images that are readily available on the Internet to train the
CNN. [13] a method for scalable, automatic calibration of traffic cameras
called AutoCalib. In order to automatically produce a reliable estimate of the
camera calibration parameters from just hundreds of samples, AutoCalib
utilises deep learning to extract specific key-point features from video clips
of cars. The camera calibration parameters are output when a video segment has
been input. [14] uses a single RGB picture to predict translation and
rotation. [9] proposes a method for calculating the relative posture between
two cameras that is based on convolutional neural networks. The proposed
network immediately generates the relative rotation and translation as output
after receiving RGB images from both cameras as input. Transfer learning is
used to train the system from beginning to end using a sizable classification
dataset. The findings show that the proposed method clearly outperforms the
baseline when compared to popular local feature-based methods (SURF, ORB). A
variation of the suggested architecture with a spatial pyramid pooling (SPP)
layer is also tested and found to enhance performance even more. [18], a deep
convolutional neural network can directly extract camera calibration
parameters from a single image. This network outperforms previous techniques,
including new deep learning-based systems, in terms of standard L2 error and
is trained using autonomously generated samples from a sizable panorama
dataset. Users are asked to assess the realism of 3D objects composited with
and without ground truth camera calibration as part of a sizable human
perception study that is being conducted. They build a new perceptual measure
for camera calibration on the basis of this research, and they show that their
deep calibration network performs better on this measure than existing
approaches. Finally, they show how to use their calibration network for a
variety of tasks, such as picture retrieval, compositing, and the insertion of
virtual objects. [20] based on seeing people in the input video that was shot
with a static camera, provide a novel approach to estimating camera posture.
In particular, the suggested solution treats the used object detector as a
parameter rather than making any assumptions about it. Additionally, they just
train on synthetic data and do not require a sizable labelled collection of
actual data. [15], using several feature descriptor databases created for each
partitioned perspective and a feature descriptor that is almost invariant for
each keypoint, we present a unique approach to robust camera pose estimation.
Based on a set of training photos created for each perspective class, their
method uses deep learning to predict the viewpoint class for each incoming
image. For preparing these photos for deep learning and creating databases,
they provide two different methods. Images are created using a projection
matrix in the first technique to ensure reliable learning in a variety of
contexts with shifting backdrops. The second technique makes use of actual
visuals to teach a planar pattern in a specific setting. [19], to directly
estimate the focal length using just raw pixel intensities as input features,
investigate the application of a deep convolutional neural network trained on
natural photos acquired from online photo libraries. [17], rotation and
translation predictions are made using an RGB picture. [34] suggest a deep
learning-based method for automatically estimating the rotation angles,
distortion parameters, and focal lengths.

Fig. 5: Box Plots of All the Parameters in Cityscapes, CVGL and Tsinghua-
Daimler Dataset respectively. The plots follow the following order from left
to right: $f_{x},f_{y},u_{0},v_{0},b,d,\theta_{p},t_{x},t_{y},t_{z}$.

###  2.3 Deep-unfolding Based Methods

Deep-unfolding caters to incorporating mathematics in the architecture. [7],
to maintain the underlying matrix’s mathematical characteristics as a seven-
degree-of-freedom homogeneous rank-2 matrix, suggest new modules and layers.
They add a reconstruction layer and a normalising layer. The reconstruction
layer produces a fundamental matrix after receiving parameters as input. With
this method, rank two of the reconstructed matrix is assured. The F-matrix is
typically normalised by dividing it by the last entry. [48], to estimate
homography, introduce a contextual correlation layer (CCL). The CCL can be
employed in a variety of ways in a learning framework and is effective at
capturing the long-range correlation found in feature maps. [49] suggest using
the weighted average, one may create a homography flow representation. of the
eight previously established homography flow bases. They suggest a Low Rank
Representation (LRR) block that reduces the feature rank so that
characteristics associated with the dominating motions are kept and other
features are rejected. They take into account that 8 Degrees of Freedom (DOFs)
are present in a homography, which is much fewer than the rank of the network
features. They suggest that by performing a feature identity loss (FIL) to
verify that the learned image features are warp-equivariant—that is, that the
results should remain the same when the warp operation and feature extraction
are carried out in the opposite order—we may prevent this from happening.

When feasible, task-agnostic techniques can be replaced with domain knowledge
to balance loss components: Yin et al. [50], calibration of an 8-parameter
fisheye lens distortion model should be done. When attempting to minimise the
parameter directly, they point out the challenge of balancing loss components
of various natures and suggest a different approach based on the in accordance
with the photometric inaccuracy. In this study, we additionally investigate
the problem of balancing the loss components for camera calibration and
propose a speedier approach based on camera projections, similar to DeepCalib
[4] nonetheless, employing dependent proxy variables through a 2D to 3D
projection as opposed to using proxy variables (that can be seen in the image
directly) such as horizon line and the distance between the image’s horizon
line and its centre. An overview of our proposed approach is presented in Fig.
1

Fig. 6: Lambda layer representation of $x_{cam}$. Fig. 7: Lambda layer
representation of $y_{cam}$. Fig. 8: Lambda layer representation of $z_{cam}$.
Fig. 9: Deep Image Homography Estimation. HomographyNet, a Deep Convolutional
Neural Network, calculates the homography between two images directly. Corner
identification and homography estimation are not required as distinct phases
in the method, and all parameters are trained end-to-end using a sizable
dataset of labelled pictures. [33]

##  3 Proposed Method

In order to improve representation while learning, we suggest using multi-task
learning using an unique loss function built into a neural network to add
mathematical equations. In order to forecast the intrinsic and extrinsic
camera characteristics, a convolutional neural network is trained. As the
feature extractor, we employ dependent regressors with a comparable network
design to accomplish this. We use a Inception-v3 [3] pretrained on ImageNet
[51] as a feature extractor, the Lambda layers for loss computation are then
applied, using 13 regressors, of which 3 are related to the 3D point cloud and
3 are related to the camera settings. We use proxy variables, which are
dependent on one another but are not visible in the image, to forecast the
translation, baseline, pitch, primary point, focal length, instead of training
these regressors to do so. This enables us to connect our approach directly to
the multi-view geometry’s mathematical underpinnings [52] resulting in better
performance.

Camera Model The perspective projection model based on the pinhole camera is
the camera model that we take into consideration [53]. If $M$ has world
coordinates $(X,Y,Z)$ and projects onto a point $m$ that has pixel coordinates
$(u,v)$, the operation can be described, in homogeneous coordinates, by the
equation:

$\displaystyle S\begin{pmatrix}u\\\ v\\\
1\end{pmatrix}=\mathbf{P}\begin{pmatrix}X\\\ Y\\\ Z\\\ 1\end{pmatrix}$ (1)

where $S$ is a scaling factor and the matrix $\mathbf{P}$ is in the format

$\displaystyle P=\begin{pmatrix}p_{1}^{T}&p_{14}\\\ p_{2}^{T}&p_{24}\\\
p_{3}^{T}&p_{34}\end{pmatrix}$ (2)

The $3\times 4$ matrix $\mathbf{P}$ is commonly referred to as perspective
projection matrix and decomposed into two matrices:
$\mathbf{P}=\mathbf{A}\mathbf{D}$ where

$\displaystyle\mathbf{D}=\begin{pmatrix}\mathbf{R}&t\\\
0_{3}^{T}&1\end{pmatrix}\
\mathbf{A}=\begin{pmatrix}\alpha_{u}&-\alpha_{u}\cot\theta&u_{0}&0\\\
0&\frac{\alpha_{v}}{\sin\theta}&v_{0}&0\\\ 0&0&1&0\end{pmatrix}$

The $4\times 4$ matrix $D$ represents the mapping from world coordinates to
camera coordinates and accounts for six extrinsic parameters of the camera:
three for the rotation $\mathbf{R}$ which is normally specified by three
rotation (Euler) angles: $R_{x},\>R_{y},\>R_{z}$ and three for the translation
$t=(t_{x},\>t_{y},\>t_{z})^{T}.$ $0_{3}$ represents the null vector
$(0,\>0,\>0)^{T}.$ The $3\times 4$ matrix $\mathbf{A}$ represents the
intrinsic parameters of the camera: the scale factors
$\alpha_{u}\>{and}\>\alpha_{v}$, the coordinates $u_{0}\>{and}\>v_{0}$ of the
principal point, and the angle $\theta$ between the image axes.

In this work, we use 2D to 3D projection as a frame of reference, leaving $13$
free parameters: focal length ($f_{x}$, $f_{y}$), principal point ($u_{0}$,
$v_{0}$), disparity, baseline, pitch, translation ($t_{x}$, $t_{y}$, $t_{z}$)
and 3D coordinates $(X,Y,Z)$.

Consequently, the network’s required parameters are the focal length ($f_{x}$,
$f_{y}$), principal point ($u_{0}$, $v_{0}$), disparity, baseline, pitch and
translation ($t_{x}$, $t_{y}$, $t_{z}$).

Parameterization As revealed by previous work [19, 35, 18], the convergence
and final performance of the network can be considerably enhanced by a
suitable parameterization of the variables to forecast. It might be
challenging to deduce camera calibration characteristics from the image
content, such as the focal length or tilt angles. Instead, proxy parameters
that are easily visible in the image can serve as a more accurate
representation of them. As a stand-in for our parameters, we project 2D data
into 3D space.

$\mathbf{A}$ can also be written as:

$\displaystyle\begin{pmatrix}f_{x}&0&u_{0}\\\ 0&f_{y}&v_{0}\\\
0&0&1\end{pmatrix}$ (3)

A 2D point in the image coordinate system is projected to the camera
coordinate system, and subsequently to the world coordinate system (Eq. 1) and
(Eq. 2) as:

$\displaystyle\begin{pmatrix}u\\\ v\\\
1\end{pmatrix}\sim\begin{pmatrix}f_{x}&0&u_{0}\\\ 0&f_{y}&v_{0}\\\
0&0&1\end{pmatrix}\begin{pmatrix}r_{11}&r_{12}&r_{13}&t_{x}\\\
r_{21}&r_{22}&r_{23}&t_{y}\\\
r_{31}&r_{32}&r_{33}&t_{z}\end{pmatrix}\begin{pmatrix}X\\\ Y\\\ Z\\\
1\end{pmatrix}$ (4)

Combining (Eq. 1) , (Eq. 2) and $\mathbf{D}$ as:

$\displaystyle\begin{pmatrix}X\\\ Y\\\ Z\\\
1\end{pmatrix}\sim\begin{bmatrix}\begin{pmatrix}f_{x}&0&u_{0}\\\
0&f_{y}&v_{0}\\\ 0&0&1\end{pmatrix}\begin{pmatrix}\mathbf{R}&t\\\
0_{3}^{T}&1\end{pmatrix}\end{bmatrix}^{-1}\begin{pmatrix}u\\\ v\\\
1\end{pmatrix}$ (5)

$\displaystyle\begin{pmatrix}X\\\ Y\\\ Z\\\
1\end{pmatrix}\sim\begin{pmatrix}\mathbf{R}&t\\\
0_{3}^{T}&1\end{pmatrix}^{-1}\begin{pmatrix}f_{x}&0&u_{0}\\\ 0&f_{y}&v_{0}\\\
0&0&1\end{pmatrix}^{-1}\begin{pmatrix}u\\\ v\\\ 1\end{pmatrix}$ (6)

Image to Camera Transformation can be performed as follows:

$\displaystyle\mathbf{A}^{-1}=\begin{pmatrix}\frac{1}{f_{x}}&0&\frac{-u_{0}}{f_{x}}\\\
0&\frac{1}{f_{y}}&\frac{-v_{0}}{f_{y}}\\\ 0&0&1\end{pmatrix}$ (7)

Assuming skew = 0

$\displaystyle\begin{pmatrix}y_{cam}\\\ z_{cam}\\\
x_{cam}\end{pmatrix}\sim\begin{pmatrix}\frac{1}{f_{x}}&0&\frac{-u_{0}}{f_{x}}\\\
0&\frac{1}{f_{y}}&\frac{-v_{0}}{f_{y}}\\\
0&0&1\end{pmatrix}\begin{pmatrix}u\\\ v\\\ 1\end{pmatrix}$ (8)

$y_{cam}=\frac{u}{f_{x}}-\frac{u_{0}}{f_{x}}=\frac{u-u_{0}}{f_{x}}$ (9a)
$z_{cam}=\frac{v}{f_{y}}-\frac{v_{0}}{f_{y}}=\frac{v-v_{0}}{f_{y}}$ (9b)
$x_{cam}=1$ (9c)

For Camera to World Transformation:

$\displaystyle\begin{pmatrix}X\\\ Y\\\ Z\\\
1\end{pmatrix}\sim\begin{pmatrix}\mathbf{R}&t\\\
0_{3}^{T}&1\end{pmatrix}\begin{pmatrix}x_{cam}\\\ y_{cam}\\\ z_{cam}\\\
1\end{pmatrix}$ (10)

$\displaystyle\begin{pmatrix}X\\\ Y\\\
Z\end{pmatrix}\sim\begin{pmatrix}\cos\theta&0&\sin\theta\\\ 0&1&0\\\
-\sin\theta&0&\cos\theta\end{pmatrix}\begin{pmatrix}x_{cam}\\\ y_{cam}\\\
z_{cam}\end{pmatrix}+\begin{pmatrix}x\\\ y\\\ z\end{pmatrix}$ (11)

$X=x_{cam}*\cos\theta+z_{cam}*\sin\theta+x$ (12a) $Y=y_{cam}+y$ (12b)
$Z=-x_{cam}*\sin\theta+z_{cam}*\cos\theta+z$ (12c)

For Camera to Camera Transformation:

$xW=\frac{f_{x}*b}{disparity}$ (13a) $yW=y_{cam}*xW=-xW*\frac{u-u_{0}}{f_{x}}$
(13b) $zW=-xW*\frac{v-v_{0}}{f_{y}}$ (13c)

To project a point from image to camera coordinate:

$x_{cam}=f_{x}*b/disparity$ (14a) $y_{cam}=-(x_{cam}/f_{x})*(u-u_{0})$ (14b)
$z_{cam}=(x_{cam}/f_{y})*(v_{0}-v)$ (14c)

$x_{cam}$ works as a proxy for $f_{x}$, baseline and disparity while $y_{cam}$
works as a proxy for $f_{x}$, u and $u_{0}$ and $z_{cam}$ works as a proxy for
$f_{y}$, $v$ and $v_{0}$. Using $x_{cam}$, $y_{cam}$ and $z_{cam}$ from Eq.
14a, Eq. 14b and Eq. 14c respectively, points can be projected to world
coordinate system using:

$X=x_{cam}*\cos(\theta_{p})+z_{cam}*\sin(\theta_{p})+t_{x}$ (15a)
$Y=y_{cam}+t_{y}$ (15b)
$Z=-x_{cam}*\sin(\theta_{p})+z_{cam}*\cos(\theta_{p})+t_{z}$ (15c)

$X$ works as a proxy for pitch and $t_{x}$ while $Y$ works as a proxy for
$t_{y}$ and $Z$ works as a proxy for pitch and $t_{z}$.

Camera Projection Loss: It is especially important to weigh the loss
components when training a single architecture to predict parameters of
various magnitudes so that the learning process is not dominated by the
estimation of specific parameters. We observe that a single metric based on
the projection of points from 2D to 3D can be used for the calibration of
cameras in place of individually adjusting the camera settings. Lambda layer
representation of the loss can be seen in Fig. 11.

Given two images with known parameters $\omega=(f_{x},f_{y},u_{0},\\\
v_{0},b,d,\theta_{p},t_{x},t_{y},t_{z},X,Y,Z)$ and a prediction of such
parameters given by the network
$\hat{\omega}$=($f_{x}^{{}^{\prime}},f_{y}^{{}^{\prime}},u_{0}^{{}^{\prime}},v_{0}^{{}^{\prime}}$,
$b^{{}^{\prime}}$, $d^{{}^{\prime}}$, $\theta_{p}^{{}^{\prime}}$,
$t_{x}^{{}^{\prime}}$, $t_{y}^{{}^{\prime}}$, $t_{z}^{{}^{\prime}}$,
$X^{{}^{\prime}}$, $Y^{{}^{\prime}}$, $Z^{{}^{\prime}}$), we get the projected
point in world coordinate system through Eq. 14a \- Eq. 15c. Loss is computed
between actual $\omega$ and predicted $\hat{\omega}$ using:

$L({\omega},\hat{\omega})=(\frac{1}{n})\sum_{i=1}^{n}MAE({\omega},\hat{\omega})$
(16)

Separating sources of loss errors: By defining many errors in terms of a
single measure, the suggested loss resolves the task balance issue. A new
issue arises during learning when employing several camera characteristics to
anticipate the 3D points, as a point’s departure from its ideal projection can
be attributed to multiple parameters. In other words, a parameter error might
affect other parameters by backpropagating through the camera projection loss.

To get around this issue, we separate the camera projection loss and assess it
separately for each parameter similar to [4]:

$\displaystyle\begin{split}L_{f_{x}}&=L((f_{x},f_{y}^{GT},u_{0}^{GT},v_{0}^{GT},b^{GT},d^{GT},\theta_{p}^{GT},\\\
&t_{x}^{GT},t_{y}^{GT},t_{z}^{GT},X^{GT},Y^{GT},Z^{GT}),\omega)\\\
L_{f_{y}}&=L((f_{x}^{GT},f_{y},u_{0}^{GT},v_{0}^{GT},b^{GT},d^{GT},\theta_{p}^{GT},\\\
&t_{x}^{GT},t_{y}^{GT},t_{z}^{GT},X^{GT},Y^{GT},Z^{GT}),\omega)\\\ \ldots\\\
L_{Z}&=L((f_{x}^{GT},f_{y}^{GT},u_{0}^{GT},v_{0}^{GT},b^{GT},d^{GT},\theta_{p}^{GT},\\\
&t_{x}^{GT},t_{y}^{GT},t_{z}^{GT},X^{GT},Y^{GT},Z),\omega)\end{split}$
$\displaystyle\begin{split}L^{*}&=\frac{L_{f_{x}}+L_{f_{y}}+L_{u_{0}}+...+L_{Z}}{13}\end{split}$
(17)

By adding weights, the loss function is further normalised to prevent the
unneeded bias caused by one or more error terms $\alpha_{i}$ with each of the
parameters. Due to the varied ranges of several parameters, this bias is
introduced. These weights $\alpha_{i}$ are learned adaptively during the
training process. The updated loss function is defined as:

$\displaystyle\begin{split}L^{*}&=\alpha_{f_{x}}L_{f_{x}}+\alpha_{f_{y}}L_{f_{y}}+\alpha_{u_{0}}L_{u_{0}}+...+\alpha_{Z}L_{Z}\end{split}$
(18)

|
---|---
(a) | (b)
|
(c) | (d)
|
(e) | (f)
Fig. 10: Some representative images from the synthetic and real datasets. (a)
Town 1 - CARLA (b) Town 2 - CARLA (c-d) Tsinghua-Daimler (e-f) Cityscapes.
Fig. 11: Camera Projection Loss in the form of Lambda layers. Lambda layers
have been used to implement the loss using (Eq. 14a \- Eq. 15c).

##  4 Results and Evaluation

Table 3: Table showing Normalized MAE in predicted parameters on CVGL test set
comprising of 23,796 images.

Method $f_{x}$ $f_{y}$ $u_{0}$ $v_{0}$ $b$ $d$ $t_{x}$ $t_{y}$ $t_{z}$
$\theta_{p}$ Average [19] 0.840 0.786 0.432 0.542 6.552 3.607 6.552 9.372
5.361 0.744 Deep-Homo [33] 0.062 0.062 0.008 0.008 0.156 0.065 0.156 0.161
0.155 0.045 Deep-Calib [12] 0.015 0.016 0.008 0.011 0.041 0.059 0.048 0.048
0.057 0.012 MN-CPL-U (Ours) 0.935 0.685 0.892 0.737 0.938 0.423 0.400 0.329
0.432 1.060 MN-Baseline (Ours) 0.030 0.029 0.017 0.007 0.057 0.013 0.064 0.076
0.071 0.024 MN-CPL-A (Ours) 0.022 0.022 0.004 0.006 0.093 0.007 0.097 0.116
0.098 0.017 SN-CPL-U (Ours) 0.020 0.017 0.003 0.012 0.644 0.112 0.106 0.068
0.193 0.207 SN-Baseline (Ours) 0.012 0.011 0.009 0.013 0.031 0.015 0.031 0.035
0.038 0.020 SN-CPL-A (Ours) 0.037 0.034 0.020 0.016 0.402 0.079 0.118 0.113
0.320 0.089

Table 4: Table showing Normalized MAE in predicted parameters on Tsinghua-
Daimler test set comprising of 2,914 images. For this experiment, we just did
a forward pass without any transfer learning or training.

Method $f_{x}$ $f_{y}$ $u_{0}$ $v_{0}$ $b$ $d$ $t_{x}$ $t_{y}$ $t_{z}$
$\theta_{p}$ Average [19] 0.994 0.991 0.969 0.951 112.438 0.492 10.843 271.935
13.798 982.413 Deep-Homo [33] 0.958 0.958 0.946 0.895 9.985 1.233 0.166 27.141
0.862 2746.994 Deep-Calib [12] 0.968 0.968 0.946 0.895 13.156 1.258 0.562
30.371 0.951 4769.592 MN-CPL-U (Ours) 0.872 0.888 0.782 0.795 0.081 1.271
0.147 23.836 0.635 7700.968 MN-Baseline (Ours) 0.957 0.958 0.944 0.893 18.323
1.258 1.035 32.946 0.999 2418.250 MN-CPL-A (Ours) 0.938 0.938 0.946 0.895
14.182 1.259 0.727 30.640 1.418 1995.353 SN-CPL-U (Ours) 0.872 0.888 0.782
0.795 0.081 1.271 0.147 23.836 0.635 7700.968 SN-Baseline (Ours) 0.968 0.968
0.946 0.896 16.010 1.257 0.688 31.591 1.585 4191.964 SN-CPL-A (Ours) 0.939
0.939 0.945 0.892 4.425 1.096 0.694 35.572 1.066 1935.330

Table 5: Table showing Normalized MAE in predicted parameters on Cityscapes
test set comprising of 1,525 images. For this experiment, we just did a
forward pass without any transfer learning or training.

Method $f_{x}$ $f_{y}$ $u_{0}$ $v_{0}$ $b$ $d$ $t_{x}$ $t_{y}$ $t_{z}$
$\theta_{p}$ Average [19] 0.994 0.991 0.970 0.950 110.05 0.487 12.933 886.848
13.818 517.895 Deep-Homo [33] 0.957 0.957 0.948 0.891 9.656 1.380 0.345 88.574
0.895 1379.635 Deep-Calib [12] 0.975 0.974 0.948 0.892 12.358 1.409 0.731
98.387 0.859 2324.758 MN-CPL-U (Ours) 0.857 0.876 0.767 0.774 0.101 1.444
0.349 78.088 0.635 4337.016 MN-Baseline (Ours) 0.956 0.957 0.946 0.890 20.008
1.416 1.592 114.156 0.933 871.623 MN-CPL-A (Ours) 0.944 0.943 0.948 0.892
12.687 1.418 0.958 100.959 1.388 960.736 SN-CPL-U (Ours) 0.946 0.947 0.948
0.892 4.382 1.382 0.877 103.374 0.962 2067.501 SN-Baseline (Ours) 0.964 0.964
0.948 0.893 15.981 1.416 1.001 107.610 1.526 2062.671 SN-CPL-A (Ours) 0.938
0.937 0.946 0.889 2.695 1.165 0.889 120.788 1.098 1001.019

Table 6: Table showing Accuracy on different thresholds using Horizontal Field
of View in predicted parameters on CVGL test set comprising of 23,796 images.

Method 0 1 2 3 4 5 Average [19] 0.0 0.467 0.766 1.0 1.0 1.0 Deep-Homo [33] 0.0
0.999 1.0 1.0 1.0 1.0 Deep-Calib [12] 0.0 0.999 1.0 1.0 1.0 1.0 MN-CPL-U
(Ours) 0.0 0.929 0.992 1.0 1.0 1.0 MN-Baseline (Ours) 0.0 0.998 1.0 1.0 1.0
1.0 MN-CPL-A (Ours) 0.0 0.996 0.999 1.0 1.0 1.0 SN-CPL-U (Ours) 0.0 0.997
0.999 1.0 1.0 1.0 SN-Baseline (Ours) 0.0 0.999 0.999 1.0 1.0 1.0 SN-CPL-A
(Ours) 0.0 0.997 0.999 1.0 1.0 1.0

Table 7: Table showing Accuracy on different thresholds using Horizontal Field
of View in predicted parameters on Tsinghua-Daimler test set comprising of
2,914 images.

Method 0 1 2 3 4 5 Average [19] 0.0 0.0 0.0 1.0 1.0 1.0 Deep-Homo [33] 0.0
0.484 0.949 1.0 1.0 1.0 Deep-Calib [12] 0.0 0.266 0.486 1.0 1.0 1.0 MN-CPL-U
(Ours) 0.0 0.808 0.899 1.0 1.0 1.0 MN-Baseline (Ours) 0.0 0.408 0.917 1.0 1.0
1.0 MN-CPL-A (Ours) 0.0 0.628 0.954 1.0 1.0 1.0 SN-CPL-U (Ours) 0.0 0.559
0.910 1.0 1.0 1.0 SN-Baseline (Ours) 0.0 0.229 0.573 1.0 1.0 1.0 SN-CPL-A
(Ours) 0.0 0.668 0.765 1.0 1.0 1.0

Table 8: Table showing Accuracy on different thresholds using Horizontal Field
of View in predicted parameters on Cityscapes test set comprising of 1,525
images.

Method 0 1 2 3 4 5 Average [19] 0.0 0.0 0.0 1.0 1.0 1.0 Deep-Homo [33] 0.0
0.469 0.982 1.0 1.0 1.0 Deep-Calib [12] 0.0 0.191 0.370 1.0 1.0 1.0 MN-CPL-U
(Ours) 0.0 0.895 0.935 1.0 1.0 1.0 MN-Baseline (Ours) 0.0 0.358 0.991 1.0 1.0
1.0 MN-CPL-A (Ours) 0.0 0.512 0.910 1.0 1.0 1.0 SN-CPL-U (Ours) 0.0 0.536
0.904 1.0 1.0 1.0 SN-Baseline (Ours) 0.0 0.280 0.634 1.0 1.0 1.0 SN-CPL-A
(Ours) 0.0 0.697 0.868 1.0 1.0 1.0

###  4.1 Datasets

SUN360 is the name of the dataset that is frequently used in literature as
benchmark data for the estimate of camera parameters [54]. Many of the state-
of-the-art methods [34, 12, 4] have been evaluated on this dataset however to
the best of our knowledge the SUN360 database [54] is no more publicly
available. For this reason, we have generated a new dataset using CARLA [2]
having 2 towns, sample images along with town maps are shown in Fig. 10 and
Fig. 2 respectively.

CVGL Camera Calibration Dataset: By creating a fresh CVGL Camera Calibration
dataset, we trained and assessed [1] using Town 1 and Town 2 of CARLA [2]
Simulator. The dataset consists of $49$ camera configurations with town 1
having $25$ configurations while town 2 having $24$ configurations . The
parameters modified for generating the configurations include $fov$, $x$, $y$,
$z$, pitch, yaw, and roll. Here, $fov$ is the field of view, (x, y, z) is the
translation while (pitch, yaw, and roll) is the rotation between the cameras.
The total number of image pairs is $79,320$, out of which $18,083$ belong to
Town 1 while $61,237$ belong to Town 2, the difference in the number of images
is due to the length of the tracks.

Cyclist Detection Dataset: Cyclists are divided into three classes as shown in
Fig. 3. We have used a recent Cyclist Detection dataset [37] for evaluating
our approach on real world data. We have used the test set provided by the
authors containing $2,914$ images by first deriving the right image using left
and disparity images and then use the pair as input to compare different
methods.

Cityscapes Dataset: We have used the Cityscapes dataset [55] for evaluating
our approach on real world data. We have used the test set provided by the
authors containing $1,525$ images by first deriving the right image using left
and disparity images and then use the pair as input to compare different
methods.

Implementation Details: Keras is used to implement and train our loss [56], an
open-source deep learning framework. All networks are trained on GeForce GTX
$1050$ Ti GPU for $200$ epochs with early stopping using ADAM optimizer [57]
with Mean Absolute Error (MAE) loss function and a base learning rate $\eta$
of ${10}^{-3}$ with a batch size of $16$.

When creating Sequential and Functional API models, Keras’ Lambda layer
enables the use of any expression as a Layer. Lambda layers have been used in
the suggested design for fundamental operations like addition, subtraction,
multiplication, division, negation, cosine, and sine. It makes sense to
include mathematical equations in the framework for learning. Lambda layer
representation of $x_{cam}$, $y_{cam}$ and $z_{cam}$ is shown in Fig. 6, Fig.
7 and Fig. 8 respectively.

There is no universal agreement on how to evaluate calibration networks; some
earlier works use a classification approach and provide accuracy values
directly [35]. Others set a cutoff for the regression mistakes and report
accuracy values [18, 19, 4]. Yin  _et al._ [50] report peak signal-to-noise
ratio structural similarity errors. Rong  _et al._ [31] use a metric based on
straight line segment lengths that is only meaningful for radial distortion
correction.

We have used Normalized Mean Absolute Error for evaluation as follows:

$NMAE(y,\hat{y})=\frac{MAE(y,\hat{y})}{\frac{1}{n}\sum_{i=1}^{n}|y_{i}|}=\frac{MAE(y,\hat{y})}{mean(|y|)}$
(19)

where $y$ and $\hat{y}$ are the target and estimated values respectively.

We have also used Horizontal feild of view as follows:

$H_{\theta}=2tan^{-1}(\frac{w}{2f})$ (20)

for a given image width, $w$.

Fig. 12: Three network architectures suggested by Deep-Calib are shown in an
illustration: SingleNet (a), DualNet (b), and SeqNet (c). The calibration
image serves as the input, and the distortion parameter and focal length are
the outputs. In SeqNet, one of the second network’s dense layers is combined
with the focal length estimate from the first network. [12]

###  4.2 Comparative Analysis

Experimental Setup: We contrasted our suggested strategy with three cutting-
edge techniques, including Average field of view [19], Deep-Homo [33] and
Deep-Calib [12]. Average field of view [19] is a standard method that predicts
the average field of view of the training set given a query image [19]. Deep-
Homo [33] estimates the homography of two photos at 8 degrees of freedom. An
overview of the approach can be seen in Fig. 9. We have modified Deep-Homo[33]
must use 4-point parameterization and then convert the result into the
homography matrix in order to predict the necessary 13 parameters for
comparison purposes as by default, it only predicted 8 values corresponding to
the four corners. We have modified Deep-Calib [12], while by default, it only
anticipated 2 values for focal length and distortion, it was necessary to
forecast the necessary 13 characteristics in order to do comparisons. Overview
of the architectures proposed by Deep-Calib [12] is shown in Fig. 12. We also
developed Baseline, CPL-U, and CPL-A versions of our multi-task learning
technique specifically for the ablative study. Baseline is an end-to-end
learning architecture based on mean absolute error and does not include any
further layers to incorporate camera model equations (MAE). It has 13
regressors that all use the same feature extractor to forecast the necessary
values. To investigate the impact of the suggested camera projection loss,
baseline is used. In order to balance the disparate ranges of calibration
parameters, we additionally used two variations of camera projection loss, one
with uniform weighting (CPL-U) in the loss function and the other with
adaptive weighting (CPL-A). MN is for Multi-net, where two networks—one for
each right and left image—are used, and the features are then combined,
whereas SN stands for Single-net, where the combined images are sent to only
one network.

Error Analysis on CVGL Camera Calibration Dataset: We contrast the normalised
mean absolute error (NMAE) of each parameter using all available techniques
with the suggested strategy. It can be seen from Table 3 that for ($v_{0}$)
and disparity ($d$), MN-CPL-A approach resulted in minimum values for NMAE and
for ($u_{0}$) SN-CPL-U performs the best and Deep-Calib [12] performs best on
pitch ($\theta_{p}$) due to bias in loss introduced as a result of the
heterogeneous range of values among parameters. For all other parameters, SN-
Baseline has lowest NMAE values. This indicates that incorporating camera
model geometry in the learning framework not only resulted in a more
interpretable learning framework but it also outperforms the state-of-the-art
methods. It can be seen from Table 6 that Deep-Homo performs best as it has
the highest accuracy for thresholds of 1 and 2 as far as horizontal field of
view is concerned. It can also be seen that SN-Baseline shows very competitive
results on both thresholds which means our method is at par with other
methods.

Error Analysis on Cyclist Detection Dataset: For this experiment, we didn’t
trained on the Tsinghua-Daimler dataset but just performed a forward pass to
test the generalizability and the results further strengthen our argument. It
can be seen from Table 4 that our proposed multi-task learning approach, MN-
CPL-U, outperforms other methods on $8$ out of $10$ parameters. For disparity
($d$) and pitch ($\theta_{p}$) , MN-CPL-U resulted in higher NMAE values due
to bias in loss introduced as a result of the heterogeneous range of values
among parameters. For all the remaining parameters our proposed multi-task
learning approach resulted in minimum values for NMAE which further solidifies
our argument of incorporating camera model geometry in the learning framework.
It can be seen from Table 7 that MN-CPL-U performs for threshold of 1 while
MN-CPL-A performs best for threshold of 2 which shows the superiority of our
proposed approach.

Error Analysis on Cityscapes Dataset: For this experiment, we didn’t trained
on the Cityscapes dataset but just performed a forward pass to test the
generalizability and the results further strengthen our argument. It can be
seen from Table 5 that our proposed multi-task learning approach, MN-CPL-U,
outperforms other methods on $7$ out of $10$ parameters. For disparity ($d$),
pitch ($\theta_{p}$) and ($t_{x}$), MN-CPL-U resulted in higher NMAE values
due to bias in loss introduced as a result of the heterogeneous range of
values among parameters. For all the remaining parameters our proposed multi-
task learning approach resulted in minimum values for NMAE which further
solidifies our argument of incorporating camera model geometry in the learning
framework. It can be seen from Table 8 that MN-CPL-U performs best for
threshold of 1 while MN-Baseline works best for threshold of 2 which suggests
that our approach is better as compared to other methods.

##  5 Conclusion

On both synthetic and actual data, our suggested strategy surpasses numerous
benchmarks, including CNN-based techniques. It combines Deep Learning and
closed-form analytical approaches. As a result, this work presents a
fundamentally novel concept in the fields of machine learning and computer
vision. Despite the fact that we used the suggested technique to estimate
camera calibration parameters, it provides a basic framework for many similar
issues. For instance, Kalman Filter tracking, homography estimation, and many
other applications are among those we want to use in the future.

## References

  * [1] Talha Hanif Butt and Murtaza Taj,  “Camera calibration through camera projection loss,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 2649–2653.
  * [2] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun,  “CARLA: An open urban driving simulator,”  in 1st Annual Conference on Robot Learning, 2017, pp. 1–16.
  * [3] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna,  “Rethinking the inception architecture for computer vision,”  in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818–2826.
  * [4] Manuel López-Antequera, Roger Marí, Pau Gargallo, Yubin Kuang, Javier Gonzalez-Jimenez, and Gloria Haro,  “Deep single image camera calibration with radial distortion,”  in Computer Vision and Pattern Recognition (CVPR), 2019, vol. 8.
  * [5] Christoph Schöller, Maximilian Schnettler, Annkathrin Krämmer, Gereon Hinz, Maida Bakovic, Müge Güzet, and Alois Knoll,  “Targetless rotational auto-calibration of radar and camera for intelligent transportation systems,”  in 2019 IEEE Intelligent Transportation Systems Conference (ITSC). IEEE, 2019, pp. 3934–3941.
  * [6] Ganesh Iyer, R Karnik Ram, J Krishna Murthy, and K Madhava Krishna,  “Calibnet: Geometrically supervised extrinsic calibration using 3d spatial transformer networks,”  in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018, pp. 1110–1117.
  * [7] Omid Poursaeed, Guandao Yang, Aditya Prakash, Qiuren Fang, Hanqing Jiang, Bharath Hariharan, and Serge Belongie,  “Deep fundamental matrix estimation without correspondences,”  in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0–0.
  * [8] Sovann En, Alexis Lechervy, and Frédéric Jurie,  “Rpnet: An end-to-end network for relative camera pose estimation,”  in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0–0.
  * [9] Iaroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu,  “Relative camera pose estimation using convolutional neural networks,”  in International Conference on Advanced Concepts for Intelligent Vision Systems. Springer, 2017, pp. 675–687.
  * [10] Jorge L Charco, Boris X Vintimilla, and Angel D Sappa,  “Deep learning based camera pose estimation in multi-view environment,”  in 2018 14th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS). IEEE, 2018, pp. 224–228.
  * [11] Peter Hansen, Hatem Alismail, Peter Rander, and Brett Browning,  “Online continuous stereo extrinsic parameter estimation,”  in 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012, pp. 1059–1066.
  * [12] Oleksandr Bogdan, Viktor Eckstein, Francois Rameau, and Jean-Charles Bazin,  “Deepcalib: A deep learning approach for automatic intrinsic calibration of wide field-of-view cameras,”  in Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production, 2018, pp. 1–10.
  * [13] Romil Bhardwaj, Gopi Krishna Tummala, Ganesan Ramalingam, Ramachandran Ramjee, and Prasun Sinha,  “Autocalib: Automatic traffic camera calibration at scale,”  ACM Transactions on Sensor Networks (TOSN), vol. 14, no. 3-4, pp. 1–27, 2018.
  * [14] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox,  “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,”  arXiv preprint arXiv:1711.00199, 2017.
  * [15] Yoshikatsu Nakajima and Hideo Saito,  “Robust camera pose estimation by viewpoint classification using deep learning,”  Computational Visual Media, vol. 3, no. 2, pp. 189–198, 2017.
  * [16] Jacek Komorowski and Przemysław Rokita,  “Extrinsic camera calibration method and its performance evaluation,”  in International Conference on Computer Vision and Graphics. Springer, 2012, pp. 129–138.
  * [17] Alex Kendall, Matthew Grimes, and Roberto Cipolla,  “Posenet: A convolutional network for real-time 6-dof camera relocalization,”  in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2938–2946.
  * [18] Yannick Hold-Geoffroy, Kalyan Sunkavalli, Jonathan Eisenmann, Matthew Fisher, Emiliano Gambaretto, Sunil Hadap, and Jean-François Lalonde,  “A perceptual measure for deep single image camera calibration,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2354–2363.
  * [19] Scott Workman, Connor Greenwell, Menghua Zhai, Ryan Baltenberger, and Nathan Jacobs,  “Deepfocal: A method for direct focal length estimation,”  in 2015 IEEE International Conference on Image Processing (ICIP). IEEE, 2015, pp. 1369–1373.
  * [20] EV Shalnov and AS Konushin,  “Convolutional neural network for camera pose estimation from object detections,”  The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 42, pp. 1, 2017.
  * [21] Joo-Haeng Lee,  “Camera calibration from a single image based on coupled line cameras and rectangle constraint,”  in Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). IEEE, 2012, pp. 758–762.
  * [22] René Ranftl and Vladlen Koltun,  “Deep fundamental matrix estimation,”  in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 284–299.
  * [23] Richard Hartley and Andrew Zisserman,  Multiple view geometry in computer vision,  Cambridge university press, 2003.
  * [24] Bruno Caprile and Vincent Torre,  “Using vanishing points for camera calibration,”  International journal of computer vision, vol. 4, no. 2, pp. 127–139, 1990.
  * [25] Jonathan Deutscher, Michael Isard, and John MacCormick,  “Automatic camera calibration from a single manhattan image,”  in European Conference on Computer Vision. Springer, 2002, pp. 175–188.
  * [26] Ranjith Unnikrishnan and Martial Hebert,  “Fast extrinsic calibration of a laser rangefinder to a camera,”  Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RI-TR-05-09, 2005\.
  * [27] Andreas Geiger, Frank Moosmann, Ömer Car, and Bernhard Schuster,  “Automatic camera and range sensor calibration using a single shot,”  in 2012 IEEE international conference on robotics and automation. IEEE, 2012, pp. 3936–3943.
  * [28] Jesse Levinson and Sebastian Thrun,  “Automatic online calibration of cameras and lasers.,”  in Robotics: Science and Systems, 2013, vol. 2, p. 7.
  * [29] Gaurav Pandey, James R McBride, Silvio Savarese, and Ryan M Eustice,  “Automatic targetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information,”  in Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012\.
  * [30] Zachary Taylor and Juan Nieto,  “Motion-based calibration of multimodal sensor arrays,”  in 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2015, pp. 4843–4850.
  * [31] Jiangpeng Rong, Shiyao Huang, Zeyu Shang, and Xianghua Ying,  “Radial lens distortion correction using convolutional neural networks trained with synthesized images,”  in Asian Conference on Computer Vision. Springer, 2016, pp. 35–49.
  * [32] Menghua Zhai, Scott Workman, and Nathan Jacobs,  “Detecting vanishing points using global image context in a non-manhattan world,”  in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 5657–5665.
  * [33] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich,  “Deep image homography estimation,”  arXiv preprint arXiv:1606.03798, 2016.
  * [34] Chaoning Zhang, Francois Rameau, Junsik Kim, Dawit Mureja Argaw, Jean-Charles Bazin, and In So Kweon,  “Deepptz: Deep self-calibration for ptz cameras,”  in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 1041–1049.
  * [35] Scott Workman, Menghua Zhai, and Nathan Jacobs,  “Horizon lines in the wild,”  arXiv preprint arXiv:1604.02129, 2016.
  * [36] João P Barreto,  “A unifying geometric representation for central projection systems,”  Computer Vision and Image Understanding, vol. 103, no. 3, pp. 208–217, 2006.
  * [37] Xiaofei Li, Fabian Flohr, Yue Yang, Hui Xiong, Markus Braun, Shuyue Pan, Keqiang Li, and Dariu M Gavrila,  “A new benchmark for vision-based cyclist detection,”  in IEEE Intelligent Vehicles Symposium, 2016, pp. 1028–1033.
  * [38] Andrew W Fitzgibbon,  “Simultaneous linear estimation of multiple view geometry and lens distortion,”  in 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. IEEE, 2001, vol. 1, pp. I–I.
  * [39] Zhengdong Zhang, Yasuyuki Matsushita, and Yi Ma,  “Camera calibration with lens distortion from low-rank textures,”  in CVPR 2011. IEEE, 2011, pp. 2321–2328.
  * [40] Rich Caruana,  “Multitask learning,”  Machine learning, vol. 28, no. 1, pp. 41–75, 1997.
  * [41] Alex Kendall, Yarin Gal, and Roberto Cipolla,  “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,”  in IEEE conference on computer vision and pattern recognition, 2018, pp. 7482–7491.
  * [42] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich,  “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,”  arXiv preprint arXiv:1711.02257, 2017.
  * [43] Christoph Strecha, Wolfgang Von Hansen, Luc Van Gool, Pascal Fua, and Ulrich Thoennessen,  “On benchmarking camera calibration and multi-view stereo for high resolution imagery,”  in 2008 IEEE conference on computer vision and pattern recognition. Ieee, 2008, pp. 1–8.
  * [44] Yasutaka Furukawa and Jean Ponce,  “Accurate camera calibration from multi-view stereo and bundle adjustment,”  International Journal of Computer Vision, vol. 84, no. 3, pp. 257–268, 2009.
  * [45] Yuji Yokochi, Sei Ikeda, Tomokazu Sato, and Naokazu Yokoya,  “Extrinsic camera parameter estimation based-on feature tracking and gps data,”  in Asian Conference on Computer Vision. Springer, 2006, pp. 369–378.
  * [46] Gang Xu, Yuki Inoue, and Hirohisa Teramoto,  “Determining camera intrinsic and extrinsic parameters from multiple images of multiple balls,”  in Workshop on science of computer vision. Citeseer, 2002.
  * [47] Moumen T Ahmed, ElSayed E Hemayed, and Aly A Farag,  “Neurocalibration: a neural network that can tell camera calibration parameters,”  in Proceedings of the Seventh IEEE International Conference on Computer Vision. IEEE, 1999, vol. 1, pp. 463–468.
  * [48] Lang Nie, Chunyu Lin, Kang Liao, Shuaicheng Liu, and Yao Zhao,  “Depth-aware multi-grid deep homography estimation with contextual correlation,”  arXiv preprint arXiv:2107.02524, 2021.
  * [49] Nianjin Ye, Chuan Wang, Haoqiang Fan, and Shuaicheng Liu,  “Motion basis learning for unsupervised deep homography estimation with subspace projection,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 13117–13125.
  * [50] Xiaoqing Yin, Xinchao Wang, Jun Yu, Maojun Zhang, Pascal Fua, and Dacheng Tao,  “Fisheyerecnet: A multi-context collaborative deep network for fisheye image rectification,”  in European Conference on Computer Vision, 2018, pp. 469–484.
  * [51] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.,  “Imagenet large scale visual recognition challenge,”  International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.
  * [52] Richard Hartley and Andrew Zisserman,  Multiple View Geometry in Computer Vision,  Cambridge University Press, New York, NY, USA, 2 edition, 2003.
  * [53] Olivier Faugeras and Olivier Autor Faugeras,  Three-Dimensional Computer Vision: A Geometric Viewpoint,  MIT press, 1993.
  * [54] Jianxiong Xiao, Krista A Ehinger, Aude Oliva, and Antonio Torralba,  “Recognizing scene viewpoint using panoramic place representation,”  in 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012, pp. 2695–2702.
  * [55] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele,  “The cityscapes dataset for semantic urban scene understanding,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3213–3223.
  * [56] Nikhil Ketkar,  “Introduction to Keras,”  in Deep learning with Python, pp. 97–111. Springer, 2017.
  * [57] Diederik P. Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  in International Conference on Learning Representations, San Diego, CA, USA, 2015.