#  SmartDepthSync: Open Source Synchronized Video Recording System of
Smartphone RGB and Depth Camera Range Image Frames with Sub-millisecond
Precision

Marsel Faizullin  Anastasiia Kornilova  Azat Akhmetyanov
Konstantin Pakulev  Andrey Sadkov and Gonzalo Ferrer
Skolkovo Institute of Science and Technology  Moscow  Russia This research is
based on the work supported by Samsung Research, Samsung Electronics.The
author is with Skolkovo Institute of Science and Technology.
{marsel.faizullin, anastasiia.kornilova, a.akhmetyanov, konstantin.pakulev,
a.sadkov<EMAIL_ADDRESS>
###### Abstract

Nowadays, smartphones can produce a synchronized (synced) stream of high-
quality data, including RGB images, inertial measurements, and other data.
Therefore, smartphones are becoming appealing sensor systems in the robotics
community. Unfortunately, there is still the need for external supporting
sensing hardware, such as a depth camera precisely synced with the smartphone
sensors. In this paper, we propose a hardware-software recording system that
presents a heterogeneous structure and contains a smartphone and an external
depth camera for recording visual, depth, and inertial data that are mutually
synchronized. The system is synced at the time and the frame levels: every RGB
image frame from the smartphone camera is exposed at the same moment of time
with a depth camera frame with sub-millisecond precision. We provide a method
and a tool for sync performance evaluation that can be applied to any pair of
depth and RGB cameras. Our system could be replicated, modified, or extended
by employing our open-sourced materials.

time synchronization, clock synchronization, smartphone sensors, depth camera,
IMU, Android app

##  1 Introduction

A sensor system that consists of various combinations of multiple types of
sensors, such as visual or depth cameras, LiDARs, radars, or inertial
measurement units (IMUs), generally produces a richer stream of data than a
system with a single type of sensor. These data can produce more diverse
information that is highly valuable for solving problems of state estimation
[1, 2, 3, 4] or 3D reconstruction [5, 6, 7], among many. In some of these
problems, e.g., Lidar-Inertial Odometry (LIO) [8, 9], the data can be properly
fused to obtain more accurate localization and mapping results, while for
other problems, part of data can be utilized as reference data for training of
machine learning or deep learning algorithms, e.g., in single-image depth
estimation [10, 11], or to be used for benchmarking [12, 13, 14].

The quality of sensor synchronization (sync) can severely determine the
quality of the obtained results. Poor synchronization directly impacts the
ability of the methods to perform sensor fusion: in a moving sensor system,
the sync error results in a projection error of the exteroceptive sensors
(multiple cameras) due to the parasitic displacement occurred during unwanted
time offset between frame shots from different cameras [15]. Out of sync can
drastically degrade the performance of IMU-based image stabilisation [16].
Visual-Inertial Odometry (VIO) algorithms cannot correctly track image
keypoints when IMU measurements and images are not correctly synced [17].

Figure 1: These images show a person’s upper body capturing while carrying the
system around the person with low dynamic (about 0.2 s-1 angular velocity and
1 m/s tangential linear velocity). Every image depicts a depth frame overlaid
on a visual frame. The left image frames are synchronized on the frame level,
while the frames on the right image are exposed with 33 ms offset. Because
some background objects have low reflectance (empty depth frame pixels), it is
possible to estimate frame mismatch. Thus labels of the plastic cases on a
shelf in the top right corner of the right image (red box (a)) do not coincide
with their depth representation; the empty depth frame pixel area related to
the right rack of the shelf (b) or a heap of objects on a table (c) has a
shift of 5-10 cm from its visual twin. At the same time, there is no such
frame mismatch in the left image.

In recent years, smartphones have become an available option for dedicated
sensor systems [18, 19]; therefore, they began to draw considerable attention
in the robotics community thanks to the continuous improvement of their sensor
integration. A modern smartphone provides tightly synced visual cameras with
an IMU sensor making this gadget an ideal independent data gathering and
processing unit. However, a smartphone might need some support from external
high-quality sensors that expand data modality, such as a depth camera to
collect 3D-data.

To this end, a smartphone RGB camera must be synced with the depth camera: a
delay of only a few milliseconds might result in depth-to-visual frame
mismatch of several image pixels that corresponds to reprojection error of
several centimeters for objects at the distance of a couple of meters, when
capturing a dynamic scene or in case of a moving recording system (Fig. 1).
Unfortunately, smartphones are closed systems that do not allow wired
triggering as in most industrial settings. Therefore, combining a smartphone
and external sensors requires hybrid hardware-software approaches to sync the
overall recording system correctly.

Current solutions of smartphone sync are based on network protocols such as
the Precision Time Protocol (PTP) [20] or the Network Time Protocol (NTP) [21]
However, for precise sync, the protocols require specific conditions of a
network connection, including constraints on latency and symmetry; complete
implementation of the sync protocols is not always available in smartphones;
background processes launched on smartphones negatively impact on protocols
sync mechanism [22]. Alternatively, data-driven sync methods analyze data
properties to align the data from different sensors. However, their direct
application provides low (several millisecond) sync precision [23] or needs
auxiliary tools [24] and non-typical events that may not be obtained [25].

This paper aims to propose a synced recording system that includes both
external sensors and sensors embedded in a smartphone. We call such a system a
heterogeneous system due to different ways of mutual sync of the sensors and
different ways of their integration into the overall system. The main content
of the SmartDepthSync is aimed at describing the integration and sync of the
system. Below, we list the contributions.

In our first contribution, we propose a multi-modal heterogeneous hardware-
software system for data collection that consists of a combination of
smartphone RGB-camera and IMU and an individual high-grade depth camera. We
made the design of our system publicly available and shared all the materials
and code on our project
page111https://github.com/MobileRoboticsSkoltech/bandeja-platform or
https://github.com/Marselka/bandeja-platform. To gather smartphone data, we
developed an open-source OpenCamera Sensors mobile application that enforces
low-level application programming interface (API) to achieve highly accurate
and precise data timing. This application has been made publicly available for
the community222https://github.com/MobileRoboticsSkoltech/OpenCamera-Sensors.

In our second contribution, we provide a two-level method for sync of all the
sensors: (i) time sync, where the entire obtained data are presented in a
common time domain, (ii) frame sync – smartphone and depth camera frames are
exposed at the same moments of time with sub-millisecond precision and a low
frame sync drift that maintains the precision order for more than 60 minutes.
The time sync level is obtained by our previously proposed gyroscope based
time sync algorithm called Twist-n-Sync [22]. Now, we made its implementation
publicly available as a Python
package333https://github.com/MobileRoboticsSkoltech/twistnsync-python or
https://github.com/Marselka/twistnsync-python.

Our third contribution consists of a novel approach of an RGB to depth camera
sync precision evaluation, making use of the depth camera infrared projector
features.

The present paper is structured as follows: in Sec. 2 we review the related
work, Sec. 3 overviews the main components of the recording system and Sec. 4
describes the software components. Sec. 5.2 describes the time and frame sync
process between all the components. Sec. 6 evaluates the results and, finally,
Sec. 7 concludes the paper.

##  2 Related Work

Since the SmartDepthSync lies in the intersection of multiple fields: data
acquisition platform, sensors synchronization, smartphone as a sensor, we
briefly introduce current research related to our work.

Hardware platforms. Hardware (HW) sync is the most precise and flexible
synchronization method that is frequently utilized for sensor platforms. In
this paragraph, we count state-of-the-art HW platforms for data acquisition.
Tschopp et al.[17] propose an open-source sensor suite with HW triggering
based on a microcontroller unit (MCU). The system supports a number of
sensors, including visual, depth, and inertial sensors. The authors achieve 1
ms sync accuracy and demonstrate sync importance on Visual-Inertial (VI)
Simultaneous Localization and Mapping (SLAM), multi-modal mapping, navigation,
and reconstruction tasks. Liu et al.[26] share principles to follow for
sensors sync and consider intra- and inter-machine sync levels. To realize the
principles, the authors design a field-programmable gate array (FPGA) based
platform for sync of visual cameras, lidar, radar, and IMU. For intra-machine
sync they state 100 $\mathrm{\SIUnitSymbolMicro s}$ accuracy (lidar to the
rest sensors). Nikolic et al. [27] develop an HW-synced VI sensor unit for
real-time SLAM based on FPGA. Due to tight integration, inter-sensor time
offset achieves down to several $\mathrm{\SIUnitSymbolMicro s}$. Schuber et
al.[28] present a VI dataset and provide a description of a VI sensor system
(two visual cameras and an IMU) used for data gathering. The system is HW-
synced; however, the time offset between the IMU and cameras is estimated by
software sync using grid-search. Faizullin et al. [29] provide an open-source
HW system design for IMU-to-lidar sync based on MCU. The system achieves 1
$\mathrm{\SIUnitSymbolMicro s}$ sync precision by emulating GPS clock by MCU
in contrast to less precise PTP based sync in [26]. Schneider et al. [30]
propose a method for lidar to camera sync that makes a visual camera expose
the images when the lidar laser beam coincides with the camera’s optical axis.

Network-based time sync. An interface for HW sensors sync is not always
available. The main well-developed alternative approach to sync devices is to
use network-based protocols. Those protocols are based on exchanging messages
with time among devices in a sensor network and gathering statistics on
arrival time between messages. The main representative of this class is the
NTP protocol, supported on many operating systems, including embedded devices.
Among other protocols and algorithms Simple Network Time Protocol (SNTP) [31],
PTP, Reference Broadcast Synchronization (RBS) [32], and Lightweight Time
Synchronization (LTS) [33] provide sub-millisecond clock sync accuracy between
devices. The main disadvantage of network-based protocols is the requirements
on latency and symmetry of the network connection in order to perform fast and
precise sync. Also, an additional barrier of using network protocols is the
unavailability of protocol implementation on different platforms and the level
of their control from the user side — i.e., in the case of Android
smartphones, there is no flexible control on sync process from the Android API
even though the NTP daemon is provided on the platform.

Data-driven time sync. If the techniques above are unavailable or unsuitable,
data-driven methods of sync are used as an alternative to them. The data-
driven methods utilize properties of data streams from multiple sensors to
identify mutual temporal dependence between the streams. For instance, Lukac
et al. [34] applied this concept to the synchronization of seismic sensors
during the MesoAmerica Subduction Experiment. Below, we focus on methods of
image frame sync from independent cameras that do not have a common HW trigger
or clock. A number of approaches to synchronize images are based on the
analysis of common events captured by the cameras. Abrupt brightness change
made by external light sources (natural or artificial) is a perfect companion
for sync. Although the need for the light source narrows down the usage of
these techniques. Shrestha et al. [23] sync video recordings by detecting
light intensity change on images produced by still camera flashes. This
approach provides about 40 ms accuracy that is dictated by the framing period.
A simple though powerful method for the sync of independent video streams with
a precision better than one millisecond is proposed in [25]. There, the
authors employ the rolling-shutter effect of cameras to increase the time
resolution of external flashing events. Shrstha et al. [35] analyze audio
tracks to sync independent video records of the same event and achieve sub-
frame accuracy. Bradley et al. [24] set up an external stroboscope to create a
common exposure for multiple rolling-shutter cameras. Thus, cameras receive an
avalanche of photons reflected from the scene at the same instant of time.
This approach is useful in very controlled settings. In contrast to light-
based methods, Caspi et al. [36] propose a temporal-spatial matching of images
from cameras by processing trajectories of moving objects and obtain sub-frame
sync accuracy.

Smartphones as synchronized sensors. Since the quality of modern smartphones
is being improved, these devices are becoming more popular as systems of
sensors with a wide range of sensing modalities. They can provide more usage
when coupled within a network. These types of networks need time sync as well.
Currently, sync of smartphones is provided by network-based or data-driven
techniques. Callebaut et al.[19] consider smartphones as sensing platforms and
sync them by GPS time data and NTP. Their sync performance analysis shows
millisecond order accuracy and precision. Sandha et al.[37] carry out an
evaluation of smartphone-to-smartphone sync performance based on Bluetooth,
Wi-Fi, and audio peripheral and state (sub-)millisecond sync accuracy. Akiyama
et al.[38] apply acoustic-light beacon for smartphone localization utilizing
smartphone’s camera (rolling shutter effect) and microphone. Latimer et
al.[39] – sub-frame video sync in a smartphone camera network based on NTP.
Ansari et al.[40] propose an Android app for simultaneous photo shooting from
multiple smartphones with sub-millisecond sync accuracy. They use custom NTP
in a local net with one smartphone as a server. In [22], the authors propose
gyroscope-based smartphones sync technique with microseconds accuracy and
implement it for synced photo shooting. The smartphones must be rigidly
attached for such high performance.

To the best of our knowledge, none of the existing platforms provide
synchronization of heterogeneous sensors, including smartphone and other
sensors (RGB, depth camera, lidar, IMU, etc.). In contrast, our platform
provides synchronous data gathering, including smartphone video and IMU
measurements and standalone depth camera images. Moreover, our technique
allows sensors to be synced at sub-frame level – it makes RGB and depth frames
be exposed at the same instance of time with tens of microseconds precision.

##  3 System Overview

Figure 2: The system common view. All the sensors are attached to the aluminum
square tube to have rigid connection. Figure 3: Example of the platform
application in a human recording process. The system operator (on the left)
moves the system while the laptop operator (not shown) launches and monitors
the recording over the SSH connection.

The system is designed to record smartphone videos synchronized at the time
and frame levels with depth images from an independent external depth camera.
Synchronized IMU measurements are also available. A common view of the system
from the front and back sides is presented in Fig. 2. The main components of
the recording system and their principal roles are as follows:

  * •

Smartphone provides data from the RGB camera and built-in IMU (gyroscope,
accelerometer).

  * •

Depth camera provides high-quality depth data.

  * •

Standalone IMU along with the smartphone built-in IMU provide gyroscope data
necessary for the gyroscope-based synchronization algorithm Twist-n-Sync [22]
between smartphone and depth camera data. Accelerometer data are also
available.

  * •

MCU acts as a hardware level mediator: it controls triggering of the depth
camera frames, obtains standalone IMU measurements, and collects metadata
essential for sync.

  * •

Mini PC (mini personal computer) controls the recording process, including
interaction with the smartphone, depth camera, and MCU.

All the system’s components are mounted on the hand-held plywood platform
designed to carry them comfortably during recordings in the way presented in
Fig. 3. A metal rig strengthens the platform body to have constant relative
transformations between every sensor. The system is battery-powered and can
work several hours after a full charge.

An additional laptop allows the operator to easily control the recording
process via an SSH connection. During the recording, the smartphone data are
saved on the smartphone, whereas other data (depth, inertial data) is saved on
the mini PC. Once the recording finishes, all the data are transferred to a
common machine, where post-processing re-assigns timestamps into a common time
domain.

Figure 4: The system topology. Arrows show data/command directions. The depth
camera and MCU board are connected to the PC via a USB connection, the
smartphone and the laptop are connected to the Wi-Fi network launched on the
mini PC.

###  3.1 System Topology

The block scheme of the system is presented in Fig. 4 and shows connections
and data flow between the components. The depth camera and standalone IMU
(through the MCU board) are connected to the mini PC via a USB interface. The
depth camera is also connected to the MCU board for synced frame triggering.
The smartphone is connected to the mini PC via a Wi-Fi network, using an
access point launched on the PC. The communication between the smartphone and
mini PC is provided by our Android OpenCamera Sensors application (Sec. 4.1)
that runs on the smartphone. Due to remote control API, the app allows a user
to run and stop smartphone video and IMU recording over the network commands
from mini PC.

###  3.2 Hardware Specifics

In our system configuration, we use the Samsung S10e smartphone (RGB-camera
and IMU), the Azure Kinect DK depth camera (other sensors besides depth are
not used), a standalone MPU-9150 IMU-sensor, the Intel(R) NUC mini PC, powered
by Ubuntu 18.04 OS, and the STM32F4DISCOVERY MCU board based on STM32F407 MCU.
The smartphone is powered by Android 10 OS and utilizes a standard camera for
video recording with 1920x1080 pixel resolution and 30 fps frame rate. The
depth camera is set up to shoot depth images with narrow field of view with
640x576 pixel resolution and a 30 or 5 fps frame rate. The smartphone and
standalone IMU sensors (gyroscope, accelerometer) collect inertial
measurements at a 500 Hz data rate.

##  4 System Software

For our system, we have developed the following software components: (i)
smartphone application, (ii) MCU firmware, (iii) mini PC recorder, and (iv)
data extractor. The first three components are designed for the recording
process, the last one is employed in the post-processing step.

###  4.1 Smartphone Application

For recording smartphone data and providing communication between the
smartphone and the rest of the system, we developed an open-source OpenCamera
Sensors app publicly available on GitHub [41]. It is based on the OpenCamera
app [42] that provides flexible control of camera parameters. OpenCamera
Sensors extends it with tools for a synced recording of the camera and IMU
data, as well as provides remote control over the Wi-Fi network.

The following features of the OpenCamera Sensors are used in the recording
pipeline: (1) start/stop recording video frames and IMU data — for data
collection, (2) start/stop IMU recording and return recorded data — for gyro-
based time sync, (3) provide the timestamp of the first video frame at the
beginning of the recording — for the frame sync.

To improve time and frame sync precision, we adhere to the following setups
during every recording:

  1. 1.

We disable sound recording to avoid the addition of undesirable extra frames
with unknown timestamps by Android Camera2 API codec.

  2. 2.

We lock exposure to achieve a constant frame rate because auto or manual
exposure produces a non-stable frame rate.

In addition, we disable optical video stabilization to keep a constant
extrinsic relation between the cameras.

###  4.2 MCU Firmware

MCU firmware is developed to perform several tasks aimed at sync and
recording: it (i) gathers timestamped inertial data from standalone IMU — for
MCU to smartphone time sync by Twist-n-Sync, (ii) triggers depth camera frames
— to sync them at the frame level with smartphone RGB-camera frames, (iii)
collects and sends timestamps of triggering pulses — for MCU to depth camera
time sync during data extraction.

MCU board provides triggering pulses to a dedicated built-in HW interface of
the depth camera with a specific phase for frame sync with the smartphone
camera. This process is discussed in more details in Sec. 5.4. IMU data
gathering process is described in [29].

###  4.3 Mini PC Recorder

The mini PC recorder software is developed for sync and recording processes.
During the sync process, the mini PC interacts with the smartphone and MCU to
perform time- and frame-level sync between the smartphone camera and the depth
camera, as described in detail in Sec. 5.4. During the recording process, the
software spawns multiple ROS nodes that collect depth camera and standalone
IMU data and meta-data for post-processing. The collected data are saved to a
common rosbag file.

The depth camera data gathering is based on Robot Operating System (ROS)
driver [43] that is modified to obtain depth-images with depth camera internal
timestamps instead of ROS timestamps. This modification allows sync to avoid
intermediate non-precise ROS timestamping [44, 29]. The MCU-IMU ROS driver
performs communications with MCU, and it is entirely developed from scratch.
All the code is available on our project page.

###  4.4 Data Extraction Software

When data collection is finished, corresponding files are manually uploaded
from the smartphone and mini PC to the external machine. Using software for
data extraction, they are unpacked, and all the data timestamps are
transformed to a single time domain. The tool puts all the extracted files
into sub-directories named after corresponding topics. The extractor uses
rosbag API to extract messages from the required ROS topics and ffmpeg for
smartphone images extraction.

##  5 Synchronization

The collected data from smartphone and depth cameras has little value if the
sensors do not have a common time reference. In order to get the reference,
time sync must be performed among the sensors. However, only time sync is not
enough for some tasks where sensors must obtain data at the same moment of
time (e.g. RGB-D SLAM [45]). In particular, for our setup and due to low frame
rates of the smartphone and depth camera (30 fps at most), the neighbouring
frames from both sensors may be exposed with the offset of up to 16.7 ms (half
of the framing period). This mismatch may create adverse artifacts when
simultaneously processing visual and depth images that capture dynamic scenes
or when the platform is moving (rotating) during the capture. To overcome
this, frame sync also should be performed between the sensors. In Fig. 1 an
example of the benefit of the frame sync is shown, see the description for
details. This section describes both sync sub-tasks and how they are reflected
in the design of our system.

###  5.1 Clock Reference

To perform time sync, a common clock reference should be chosen. Our system
has three different clock sources — mini PC, smartphone, and MCU. The mini PC
clock is not suitable for reference due to its loose relation to the other two
clocks: neither the MCU nor the smartphone has a precise sync technique to the
mini PC clock. Moreover, the mini PC does not have a precise timestamping
procedure for the sensors connected to it (depth camera, standalone IMU). Both
the MCU clock and the smartphone clock could be chosen as the reference clock.
In our work, we use the MCU-based clock for certainty.

###  5.2 Time Sync

A direct wired time sync of all the system’s components is not available
because smartphones do not allow for such an interface. Common network-based
protocols for time sync, such as NTP or PTP, are also not suitable candidates
for our system because (1) they rely on network symmetry and constant latency,
which neither the PC nor the smartphone guarantee because of uncontrollable
background network operations, (2) smartphones cannot provide flexible
operation with NTP/PTP daemons. Moreover, even in case of solving these two
problems and syncing the smartphone and mini PC clocks, network-based
protocols cannot solve the depth camera to mini PC relation (the depth camera
does not have such a feature), and it makes us return to the ROS timestamping
problem discussed in Sec. 4.3. In order to avoid these hindrances, we apply
the IMU data-driven solution that syncs smartphone and depth cameras directly,
avoiding the mini PC. This solution is described in Sec. 5.4.

###  5.3 Frame Sync

The canonical way to sync data on the frame level is to use hardware
triggering of target devices. In our case, the smartphone doesn’t provide such
an interface, whereas the depth camera supports it. Therefore, to sync devices
on the frame level, we use the following approach:

  1. 1.

The smartphone starts video recording and shares a timestamp of the first
video frame with the mini PC over the Wi-Fi interface.

  2. 2.

Taking into account the timestamp, the mini PC adjusts the phase of the depth
sensor using hardware triggering via the MCU.

The phase is defined as the relative offset between the closest smartphone RGB
frame exposure middle point and the depth camera frame exposure middle point.
This approach requires both the smartphone and the depth camera to have a
constant frame rate. On the smartphone, this property is guaranteed by blocked
exposure, on depth camera – by hardware triggering with a constant period.
Moreover, the frame rates must be equal or multiple of each other to keep
frames aligned over time.

###  5.4 Recording Pipeline with Synchronization

This sub-section describes the overall recording pipeline and implementation
details that provide synced data from the smartphone and the depth camera at
the time and frame levels.

After launching the smartphone app, mini PC software, and MCU and depth camera
drivers, the system is ready for sync process. The whole pipeline from sync up
to data extraction is divided into five steps: (1) depth camera to MCU time
sync, (2) smartphone to MCU time sync, (3) depth to smartphone camera frame
alignment, (4) recording, (5) final timestamps mapping to a common MCU time
domain during the extraction.

Step 1. Depth camera to MCU time sync. After the system is launched, the MCU
starts to trigger depth camera frames and provides their timestamps in the MCU
time domain. We denote the first captured depth frame timestamp in the MCU
time domain as $t_{D_{0}}^{M}$. A unique association of the MCU timestamps
with the depth frames that finishes the sync process is done later, during the
fifth step.

Step 2. Smartphone to MCU time sync. In order to obtain the time offset
between the MCU and smartphone clocks, a data-driven gyroscope-based software
sync method is applied. The system is hand-twisted, and both the smartphone
and standalone IMUs catch this common event. Then the data from both IMUs with
their timestamps are analyzed by the Twist-n-Sync algorithm to obtain the
offset denoted as $\Delta t_{SM}$. This means that time mapping between the
MCU and smartphone is expressed by:

$\displaystyle t^{S}=t^{M}+\Delta t_{SM},$ (1)

where $t^{S}$ and $t^{M}$ is time in the smartphone and MCU time domains
respectively.

Step 3. Depth to smartphone camera frame alignment. In this step, smartphone
starts video recording and generates the initial frame timestamp in the
smartphone time domain $t_{S_{0}}^{S}$ that according to (1) is expressed in
the MCU time domain by:

$\displaystyle t_{S_{0}}^{M}=t_{S_{0}}^{S}-\Delta t_{SM},$ (2)

where $t_{S_{0}}^{S}$ and $t_{S_{0}}^{M}$ is the time moment of the initial
smartphone frame shot $S_{0}$ in the smartphone and MCU time domains
respectively.

By this moment, the initial smartphone and depth frame timestamps
$t_{S_{0}}^{M}$ and $t_{D_{0}}^{M}$ have been estimated and are now expressed
in the same reference frame $M$, the MCU clock domain. We remove the frame
superscript for clarity. Under the assumption that the smartphone and depth
camera framing periods are constant and equal, the exposure moments can be
expressed by:

$\displaystyle t_{S_{n}}$ $\displaystyle=t_{S_{0}}+T\,n,$ (3) $\displaystyle
t_{D_{m}}$ $\displaystyle=t_{D_{0}}+T\,m+\Delta t,$

where $T$ is the framing period, $n$ and $m$ are frame counts, $\Delta t$ is
the time offset that has to be found for frame alignment, initially it equals
zero. The smartphone and the depth camera are considered aligned if
$t_{S_{n}}=t_{D_{m}}$ for some pair $(n,m)$; then $\Delta t$ for frame
alignment can be found from (3) as:

$\displaystyle\Delta t$ $\displaystyle=t_{S_{0}}-t_{D_{0}}+T\,(n-m).$ (4)

The $\Delta t$ then to be transformed to the target phase shift $\Delta
t_{phase}$ by:

$\displaystyle\Delta t_{phase}$ $\displaystyle=\Delta t\pmod{T}$ (5)
$\displaystyle=t_{S_{0}}-t_{D_{0}}\pmod{T}.$

The obtained $\Delta t_{phase}$ is sent to the MCU that finally corrects the
phase on the fly. At this stage, smartphone-to-depth camera frame sync is
completed.

MCU HW can set up the phase with a discrete step size equals about 390 ns that
is negligible to the overall precision of the sync that is discussed in the
next section.

Steps 4 and 5. Recording and extraction. After obtaining offsets and
performing frame sync, the system then records the data. After recording,
during the extraction, data timestamps are finally mapped into the common MCU
time domain.

##  6 Evaluation

In this section, we evaluate the precision of the frame alignment, as well as
its accuracy drift over time.

The previous works [40, 25, 22] (Sec. 2) utilize controlled light sources for
evaluation of the sync performance of rolling shutter cameras by their
methods. The light sources emit short light flashes (strobes) in specific time
moments. The strobes are then caught by the cameras, and the analysis of the
synced images is performed. Due to the rolling shutter effect [25], the
strobes are visible only on some pixel rows of images that are exposed at this
moment while the rest areas remain darker. By row position difference between
the synced images, one can estimate the time offset between the synced image
frames. For this estimation, the row readout time must be known for every
image sensor to transform the row position difference to the time offset. More
detailed explanation can be found in [22]. We propose a variation of the above
principle in order to measure depth-RGB frame alignment that can be applied to
any pair of visual rolling shutter camera and time-of-flight based depth
camera. In our case, the controlled light source is the depth camera’s
projector that emits infrared (IR) strobes of light. The strobes are normally
used by Azure Kinect DK as a light source in the depth estimation process. The
projector generates the strobe train of 9 strobes at every depth image frame
exposure, the fifth strobe coincides with the middle point of the exposure
time [46].

Figure 5: Perfect sync case – the 5th projector strobe coincides with the
exposure of the middle (540th) row of the smartphone camera frame. The 1st and
9th strobes are invisible on the frame because the strobe train duration is
longer than the entire row-by-row smartphone frame exposure.

We consider the coincidence of the projector’s fifth infrared strobe flash
moment with the middle exposure point of the middle row of a smartphone frame
as perfect sync. The 540th row is chosen as the middle row, additional
description can be found in Fig. 5. Fluctuations of the row position around
the middle one determine precision of the smartphone-to-depth cameras frame
sync, whereas a steady shift of the mean bright row position over time defines
the accuracy drift.

The problem of infrared (IR) light is that it is invisible for the smartphone
camera image sensor because of the IR filter in the camera’s optical scheme.
To bypass this issue, we designed a light re-transmitter that receives the IR
strobes and repeats them in the visible spectrum. The re-transmitter
transforms IR-light to the electrical signal by a photodiode, then amplifies
the signal power to flash a LED-strip. The visible light is then captured by
the smartphone camera with no constraints. A common view of the re-transmitter
and experimental setup used for the evaluation are shown in Fig. 6.

Figure 6: Common view of the re-transmitter (left) and experimental setup
(right). The amplifier is used to strengthen the photodiode signal to flash
the LED-strip. The smartphone camera is set up to the lowest exposure time to
resolve the train of short strobes. It could be noticed that no more than
seven strobes can be observable on the single smartphone image because the
whole image readout time is less than the strobes train duration.

Below, we firstly evaluate inter-launch sync precision. This means we monitor
the row position corresponding to the certain strobe at every recording start
for a number of system launches, then we analyze the obtained distribution of
inter-launch row positions. To collect the inter-launch row positions, we
recorded 52 short smartphone video sequences and extracted 16 consequent
frames from every video, as depicted in Fig. 7. For each of these frames from
a single video, we (i) extract per-row intensity, then (ii) average the
intensities over images, and (iii) apply the smoothing to the averaged per-row
intensity by the Gaussian filter with a kernel size of 7 lines (or 71.4
$\mathrm{\SIUnitSymbolMicro s}$) to get more robust peak positions. After
smoothing, we choose the region that includes a certain peak for every frame
in every recording (the fourth peak is chosen as the highest peak among the
others in the experiment) and obtain the peak position for every trial. The
obtained peak distribution provides sub-millisecond precision, stated in Table
1.

Figure 7: Statistics gathering for frame sync precision evaluation. Row
position determination is shown for a single record from 52 in total. The
number of frames for the determination is 16.

The row-to-time transformation is obtained by measuring the average number of
rows between every available pair of neighbouring bright peaks (determined by
pairs of consecutive strobes in a strobe train), knowing the time delay
between the strobes from [46]. During the evaluation, we manually injected
several milliseconds offset to the depth camera phase to always observe the
first strobe on the smartphone images. This is to be sure that the fourth peak
is picked up on every image. Otherwise, it is not possible to enumerate
strobes because the whole train does not fit into an image (this is described
in Fig. 5 and can be seen on the smartphone screen in Fig. 6).

Figure 8: Frame sync drift estimation over 5 minutes. For clarity, the right
axis indicates time scale of the drift.

For frame sync accuracy drift evaluation, we recorded a 5-minute video and
gathered the pixel row position for some strobe for every sixth smartphone
frame. Then linear regression is applied to 2100 pairs (frame timestamp – row
position) to compute frame sync drift over time. Fig. 8 depicts the row
position drift. The numerical result is shown in Table 1. We do not provide
the initial sync accuracy value estimation because the offset between
smartphone and depth camera exposure starts is set up manually to see the
fifth strobe in the middle of the smartphone images.

Table 1: Sync precision and drift. The interquartile range (IQR) and standard deviation (SD) measure inter-launch sync precision. The accuracy drift over time is for a single recording. Such performance allows recording with sub-millisecond precision during an hour. IQR(SD) | Accuracy drift
---|---
66.3 (61.6) $\mathrm{\SIUnitSymbolMicro s}$ | 16.34 $\mathrm{\SIUnitSymbolMicro s}$/m

##  7 Conclusion

In this paper, we have proposed an open-source recording system of
heterogeneous sensors including, but not limited to smartphone visual camera,
standalone depth camera, and IMUs. The main contribution of our work is on the
hybrid approach to sync different time authorities: MCU and smartphone in our
case, but easily extendable to other configurations. To achieve this, we have
combined traditional wired-based trigger sync with software sync for the
smartphone. This allows us to sync the sensors not only at the time but also
at the frame level guaranteeing synchronous frame triggering of the smartphone
and depth camera. The sync performance thanks to the unified sync approach
does not degrade when increasing the number of sensors: any number of sensors
that support hardware sync can be directly connected to the trigger; any
number of sensors that include IMU can be synchronized by our software sync
with microseconds accuracy as have been shown in our previous work.

We have shown that our system achieves a sync precision of 66
$\mathrm{\SIUnitSymbolMicro s}$ and can maintain stable sync conditions for
recording during an hour due to low sync drift. In addition, we have proposed
a new measurement process to establish precise sync between a smartphone RGB
and a standalone depth camera employing properties of a depth camera infrared
projector. We have released a synchronization package and a smartphone
recording app; all code, design, and scripts necessary to replicate our
recording platform or modify it with new configurations have been made
publicly available.

## References

  * [1] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” _IEEE Transactions on Robotics_ , 2021\.
  * [2] T. Schops, T. Sattler, and M. Pollefeys, “Bad slam: Bundle adjusted direct rgb-d slam,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 134–144.
  * [3] W. Shao, S. Vijayarangan, C. Li, and G. Kantor, “Stereo visual inertial lidar simultaneous localization and mapping,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 370–377.
  * [4] J. Zhang and S. Singh, “Visual-lidar odometry and mapping: Low-drift, robust, and fast,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 2174–2181.
  * [5] A. Dai, M. Nießner, M. Zollhöfer, S. Izadi, and C. Theobalt, “Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration,” _ACM Transactions on Graphics (ToG)_ , vol. 36, no. 4, p. 1, 2017.
  * [6] M. Labbé and F. Michaud, “Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation,” _Journal of Field Robotics_ , vol. 36, no. 2, pp. 416–446, 2019.
  * [7] M. Zollhöfer, P. Stotko, A. Görlitz, C. Theobalt, M. Nießner, R. Klein, and A. Kolb, “State of the art on 3d reconstruction with rgb-d cameras,” in _Computer graphics forum_ , vol. 37, no. 2. Wiley Online Library, 2018, pp. 625–652.
  * [8] H. Ye, Y. Chen, and M. Liu, “Tightly coupled 3d lidar inertial odometry and mapping,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 3144–3150.
  * [9] T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and D. Rus, “Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 5135–5142.
  * [10] A. Ignatov, G. Malivenko, D. Plowman, S. Shukla, and R. Timofte, “Fast and accurate single-image depth estimation on mobile devices, mobile ai 2021 challenge: Report,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2545–2557.
  * [11] J. Hu, M. Ozay, Y. Zhang, and T. Okatani, “Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 1043–1051.
  * [12] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5828–5839.
  * [13] A. Knapitsch, J. Park, Q.-Y. Zhou, and V. Koltun, “Tanks and temples: Benchmarking large-scale scene reconstruction,” _ACM Transactions on Graphics (ToG)_ , vol. 36, no. 4, pp. 1–13, 2017.
  * [14] S. Cortés, A. Solin, E. Rahtu, and J. Kannala, “Advio: An authentic dataset for visual-inertial odometry,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 419–434.
  * [15] E. Olson, “A passive solution to the sensor synchronization problem,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 1059–1064.
  * [16] A. English, P. Ross, D. Ball, B. Upcroft, and P. Corke, “Triggersync: A time synchronisation tool,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 6220–6226.
  * [17] F. Tschopp, M. Riner, M. Fehr, L. Bernreiter, F. Furrer, T. Novkovic, A. Pfrunder, C. Cadena, R. Siegwart, and J. Nieto, “Versavis—an open versatile multi-camera visual-inertial sensor suite,” _Sensors_ , vol. 20, no. 5, p. 1439, 2020.
  * [18] M. Grossi, “A sensor-centric survey on the development of smartphone measurement and sensing systems,” _Measurement_ , vol. 135, pp. 572–592, 2019.
  * [19] G. Callebaut, G. Ottoy, and L. De Strycker, “Bring your own sensor: Use your android smartphone as a sensing platform,” in _2019 IEEE Sensors Applications Symposium (SAS)_. IEEE, 2019, pp. 1–5.
  * [20] “Ieee standard for a precision clock synchronization protocol for networked measurement and control systems,” _IEEE Std 1588-2008 (Revision of IEEE Std 1588-2002)_ , pp. 1–300, 2008.
  * [21] D. Mills, “Network time protocol version 4: Protocol and algorithms specification,” Internet Requests for Comments, RFC Editor, RFC 5905, 6 2010\. [Online]. Available: https://www.rfc-editor.org/info/rfc5905
  * [22] M. Faizullin, A. Kornilova, A. Akhmetyanov, and G. Ferrer, “Twist-n-sync: Software clock synchronization with microseconds accuracy using mems-gyroscopes,” _Sensors_ , vol. 21, no. 1, p. 68, 2021.
  * [23] P. Shrestha, H. Weda, M. Barbieri, and D. Sekulovski, “Synchronization of multiple video recordings based on still camera flashes,” in _Proceedings of the 14th ACM international conference on Multimedia_ , 2006, pp. 137–140.
  * [24] D. Bradley, B. Atcheson, I. Ihrke, and W. Heidrich, “Synchronization and rolling shutter compensation for consumer video camera arrays,” in _2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_. IEEE, 2009, pp. 1–8.
  * [25] M. Šmıd and J. Matas, “Rolling shutter camera synchronization with sub-millisecond accuracy,” in _International Conference on Computer Vision Theory and Applications_ , 2017, p. 8.
  * [26] S. Liu, B. Yu, Y. Liu, K. Zhang, Y. Qiao, T. Y. Li, J. Tang, and Y. Zhu, “The matter of time–a general and efficient system for precise sensor synchronization in robotic computing,” _arXiv preprint arXiv:2103.16045_ , 2021.
  * [27] J. Nikolic, J. Rehder, M. Burri, P. Gohl, S. Leutenegger, P. T. Furgale, and R. Siegwart, “A synchronized visual-inertial sensor system with fpga pre-processing for accurate real-time slam,” in _2014 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2014, pp. 431–437.
  * [28] D. Schubert, T. Goll, N. Demmel, V. Usenko, J. Stueckler, and D. Cremers, “The tum vi benchmark for evaluating visual-inertial odometry,” in _International Conference on Intelligent Robots and Systems (IROS)_ , October 2018.
  * [29] M. Faizullin, A. Kornilova, and G. Ferrer, “Open-source lidar time synchronization system by mimicking gps-clock,” 2021.
  * [30] S. Schneider, M. Himmelsbach, T. Luettel, and H.-J. Wuensche, “Fusing vision and lidar-synchronization, correction and occlusion reasoning,” in _2010 IEEE Intelligent Vehicles Symposium_. IEEE, 2010, pp. 388–393.
  * [31] D. Mills, “Simple network time protocol (sntp) version 4 for ipv4, ipv6 and osi,” Internet Requests for Comments, RFC Editor, RFC 4330, 1 2006. [Online]. Available: https://www.rfc-editor.org/info/rfc4330
  * [32] Hyojung Lee, Wonpil Yu, and Youngmi Kwon, “Efficient rbs in sensor networks,” in _Third International Conference on Information Technology: New Generations (ITNG’06)_ , 2006, pp. 279–284.
  * [33] J. van Greunen and J. Rabaey, “Lightweight time synchronization for sensor networks,” in _Proceedings of the 2nd ACM International Conference on Wireless Sensor Networks and Applications_ , ser. WSNA ’03. New York, NY, USA: Association for Computing Machinery, 2003, p. 11–19. [Online]. Available: https://doi.org/10.1145/941350.941353
  * [34] M. Lukac, P. Davis, R. Clayton, and D. Estrin, “Recovering temporal integrity with data driven time synchronization,” in _2009 International Conference on Information Processing in Sensor Networks_. IEEE, 2009, pp. 61–72.
  * [35] P. Shrstha, M. Barbieri, and H. Weda, “Synchronization of multi-camera video recordings based on audio,” in _Proceedings of the 15th ACM international conference on Multimedia_ , 2007, pp. 545–548.
  * [36] Y. Caspi, D. Simakov, and M. Irani, “Feature-based sequence-to-sequence matching,” _International Journal of Computer Vision_ , vol. 68, no. 1, pp. 53–64, 2006.
  * [37] S. S. Sandha, J. Noor, F. M. Anwar, and M. Srivastava, “Exploiting smartphone peripherals for precise time synchronization,” in _2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP)_. IEEE, 2019, pp. 1–6.
  * [38] T. Akiyama, M. Sugimoto, and H. Hashizume, “Syncsync: Time-of-arrival based localization method using light-synchronized acoustic waves for smartphones,” in _2015 International Conference on Indoor Positioning and Indoor Navigation (IPIN)_. IEEE, 2015, pp. 1–9.
  * [39] R. Latimer, J. Holloway, A. Veeraraghavan, and A. Sabharwal, “Socialsync: Sub-frame synchronization in a smartphone camera network,” in _European Conference on Computer Vision_. Springer, 2014, pp. 561–575.
  * [40] S. Ansari, N. Wadhwa, R. Garg, and J. Chen, “Wireless software synchronization of multiple distributed cameras,” in _2019 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2019, pp. 1–9.
  * [41] Opencamera sensors – an android application for synchronized recording of video and imu data. Mobile Robotics Lab, Skoltech. Accessed Jul. 13, 2021. [Online]. Available: https://github.com/MobileRoboticsSkoltech/OpenCamera-Sensors
  * [42] OpenCamera, 2019. [Online]. Available: https://opencamera.org.uk/
  * [43] “Azure Kinect ROS Driver,” https://github.com/microsoft/Azure_Kinect_ROS_Driver, accessed: 2021-01-30.
  * [44] J. Park, R. Delgado, and B. W. Choi, “Real-time characteristics of ros 2.0 in multiagent robot systems: An empirical study,” _IEEE Access_ , vol. 8, pp. 154 637–154 651, 2020.
  * [45] R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras,” _IEEE Transactions on Robotics_ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * [46] Microsoft, Azure Kinect DK, 2015. [Online]. Available: https://azure.microsoft.com/en-us/services/kinect-dk/