#  Event-Based Structured Light for Depth Reconstruction using Frequency
Tagged Light Patterns

T. Leroux, S.-H. Ieng and R. Benosman University of Pittburgh, Carnegie Mellon
Univeristy, Sorbonne Universitas
<EMAIL_ADDRESS>
###### Abstract

This paper presents a new method for 3D depth estimation using the output of
an asynchronous time driven image sensor. In association with a high speed
Digital Light Processing projection system, our method achieves real-time
reconstruction of 3D points cloud, up to several hundreds of hertz. Unlike
state of the art methodology, we introduce a method that relies on the use of
frequency tagged light pattern that make use of the high temporal resolution
of event based sensors. This approch eases matching as each pattern unique
frequency allow for any easy matching between displayed patterns and the event
based sensor. Results are show on real scenes.

###### Index Terms:

Neuromorphic sensing, Event-based vision, Structured light, 3D imaging

##  I Introduction

###  I-A Motivation

Structured light is considered the most reliable technique to estimate depth
and is behind most commonly used methods in the field of non-contact 3D
measurement. Because of its accuracy and simplicity, its the method of choice
in many applications including autonomous vehicle navigation[1, 2], factory
inspection processes [3, 4], object recognition[5, 6] or reverse engineering
[7]. The term _structured light_ itself refers to the projection of simple or
encoded light patterns onto the illuminated scene [8]. Its main purpose is to
simplify the extraction of features in the image of the scene captured by a
camera, and facilitate the pairing process. Like in classic stereoscopic
systems, the measure of 3D information using structured light is done by
triangulation when corresponding points are identified.

This paper presents a method based on a completely new neuromorphic vision
sensor, whose dynamic and temporal properties are unmatched. Using a totally
different approach of vision, it allows to push back the limitations of former
systems. Uses of structured light technology can be tracked back to the early
80’s [9], but its only since the developement of the Kinect sensor by
Microsoft in 2008 that structured light became affordable. A large number of
devices have emerged since. However, most of them share two major limitations
resulting from the use of classic frame-grabbing cameras. Because they have
short dynamic ranges, captured intensity of the projected signal can be
extremely low in strong ambient illumination, thus resulting in poor depth
estimations. This is especially true in outdoor environnement where sunlight
is often 2-5 orders of magnitude brighter than the projected light. Another
contrainst is the frequency at which classic cameras operate (between 30 and
$120\mathrm{Hz}$ for commonly used cameras), which is not suited to follow and
reconstruct fast moving objects. Some of the latest methods using high speed
frame-grabbing cameras achieve 3D reconstruction up to $500\mathrm{Hz}$ but at
high energy and data cost.

Recent developement in neuromorphic vision opens up new alternatives to
classic frame-grabbing cameras and a new way to process visual information.
The presented system uses an biomimetic artificial retina with a 143dB dynamic
range able to acquire asynchronous data at a microsecond precision.

###  I-B State of the Art

Structured light techniques differs from one another essentially because of
the encoding method used to create the projected pattern, or sequence of
patterns. In the litterature, several comparative studies can be found that
analyse pattern encoding strategies over the last decades [10, 11, 12].
Methods based on multiple patterns (i.e time multiplexing) are generally
better when dealing with static objects and offers highly accurate
measurements. Posdamer et al. [13] used a succession of patterns generating a
pure binary code, followed by works to decrease errors at brightness
boundaries [14], increase the resolution [15] or reduce the number of frame
needed [16]. For these methods, a high resolution is obtained as depth
information is calculated at every pixel. Noise is mostly generated by the
movement of objects between successive patterns of the projection. Other
methods based on single pattern projection, using spatial multiplexing, are
more robust for a moving object. Those includes works on patterns composed of
multiple slits of different sizes [17, 18] or colors [19, 20], patterns using
specific shapes or colors organized in pseudorandom arrays [21, 22] or direct
coding patterns in which each pixel is associated with a unique codeword by
either spatial grading [23] or phase shifting [24, 25]. Recently, work has
been done aiming to reach real-time reconstruction. Most of these attempts
were made using a high speed frame based camera coupled with a fast DLP
projector to allow projection of binary patterns at speeds over
$1\mathrm{k}\mathrm{Hz}$ [26, 27]. These methods use both spatial and time
multiplexing to get the best from both worlds and achieve up to
$500\mathrm{Hz}$ reconstructions, while the current average reconstruction
rate is somewhere between 30-100 Hz .

##  II Hardware

###  II-A Time encoded imaging

Biomimetic event-based cameras are a novel type of vision devices that - like
their biological counterparts - are driven by ”events” happening within the
scene. They are not like conventional vision sensors which, by contrast, are
driven by artificially created timing and control signals (e.g. frame clock)
which have no relation whatsoever to the source of the visual information
[28]. Over the past few years, a variety of these event-based devices have
been implemented, including temporal contrast vision sensors that are
sensitive to relative luminance change [29, 30], gradient-based sensors
sensitive to static edges [31], edge-orientation sensitive devices and
optical-flow sensors [32]. Most of these vision sensors output visual
information about the scene in the form of asynchronous address events (AER)
[33] and encode the visual information in the time dimension and not as
voltage, charge or current. The presented depth estimation method is designed
to work on the data delivered by such a time-encoding sensor.

The ATIS used in this work is a time-domain encoding vision sensors with
$304\times 240$ pixels resolution. [28]. The sensor contains an array of fully
autonomous pixels that combine an illuminance relative change detector circuit
and a conditional exposure measurement block.

Figure 1: Functional diagram of an ATIS pixel [29]. Two types of asynchronous
events, encoding change and brightness information, are generated and
transmitted individually by each pixel in the imaging array.

As shown in the functional diagram of the ATIS pixel in Fig. 1, the relative
change detector individually and asynchronously initiates the measurement of
an exposure/gray scale value only if \- and immediately after - a brightness
change of a certain magnitude has been detected in the field-of-view of the
respective pixel. The exposure measurement circuit in each pixel individually
encodes the absolute instantaneous pixel illuminance into the timing of
asynchronous event pulses, represented as inter-event intervals.

Since the ATIS is not clocked like a conventional camera, the timing of events
can be conveyed with a very accurate temporal resolution in the order of
microseconds. The time-domain encoding of the intensity information
automatically optimizes the exposure time separately for each pixel instead of
imposing a fixed integration time for the entire array, resulting in an
exceptionally high dynamic range and improved signal to noise ratio. The
pixel-wise change detector driven operation yields almost ideal temporal
redundancy suppression, resulting in a maximally sparse encoding of the image
data.

###  II-B Digital Light Processing

Figure 2: The DLP lightcrafter module ®

Digital Light Processing (DLP) is an image projection technology originally
developed by Texas Instrument, relying heavily on a chip containing orientable
micromirrors called Digital Micromirror Device (DMD). A light bulb sends light
on the chip’s mirror array that reflect light back in a lens to generate an
image on the screen. DMD was invented by Hornbeck and Nelson in 1987. In our
setup we use a DLP lightcrafter model (Fig. 2) which is a complete projection
system embedding a WVGA micromirror array. It is able to project binary frames
of $304\times 240$ pixels at a $0.7\mathrm{m}\mathrm{s}$ time resolution.
Faster and bigger models exist, but this model benefits from the
implementation of an asynchronous control driver that enables to manipulate up
to $5000$ specific mirrors per time step instead of having to modify the
complete pixel array synchronously.

###  II-C Setup

For the experiments in this paper we developed an active stereoscopic vision
system using the association of an ATIS camera and a DLP projector (Fig. 3).
With its very high frame rate, the DMD used to produce binary projection can
make profit of the high temporal resolution of the ATIS. An event is conveyed
to the output buffer in a few hundred nanoseconds. This means that the time
needed to change the position of DMD mirrors is fast enough to obtain
spatially and temporally dense information. However limitation arises when
exposing the whole pixel array of the ATIS to high frequency changes. An
estimation of the maximal event flow the ATIS can currently manage is around 8
mega events per second. Even if this limitation will probably be overcome in
the future, a $1\mathrm{k}\mathrm{Hz}$ stimulation of the sensor is limited to
800 events per step which represents only $1\%$ of the camera’s pixel array.
Due to this limitation a tradeoff has to be made between temporal and spatial
resolution.

Figure 3: Experimental setup

##  III Pattern codification strategies

To get the better out of the ATIS sensor, it is mandatory to choose a suitable
pattern codification. In the event domain, the only information available is
the polarity and timestamp of pixels, thus the correspondance problem can’t be
solve by color or grey level coding. Moreover, the need for an asynchronous
system impose that our pattern can be segmented in small independent patches
that we could use to extract local information only when it is needed. This
constraint has a direct impact on the spatial organisation of the pattern and
direct methods like the one used in LABEL:NorthwesternUni2015 must be
discarded. In the next section we expand over three different methods that
could be used to gather local data independently as well as being decodable
without too much computation from the sensor’s output to ensure a decent
reconstruction speed. The first method we will develop is based on the well
known time-multiplexing binary coding (refs?) but encode the binary frames as
a signal with set frequency and dutycycle, each element is then part of a
spatial codification that can be decoded locally. Our second method aim to
make use of spatio-temporal continuity of events generated from fast moving
objects to create codewords based on spatial orientation of multiple moving
dots. This approach differs only from the first in the way information is
extracted from the sensor, but the decoding process is done the same way.
Lastly, we developed an approach based on phase-shifting algorithms, but
instead of evaluating the phase from several shifted patterns (ref), we
continuously move a series of stripes and record precise time difference
between stripe positions from an arbitrary origin, which gives us a robust
measurement. This last method is able to gather information at the pixel level
which is more accurate than previous ones but in exchange the pattern can only
be segmented roughly for asynchronous projection.

Figure 4: Structured light principle. A coded light pattern is projected onto
the scene that is observed by a camera. Decoding the pattern allows the
matching of paired points in the two views ($p_{1},p_{2}$) and by
triangulation of the rays coming from optical centers $C_{1}$ and $C_{2}$, the
position of the real point $P$ is found. Colors in the picture are for clarity
only and in our case refers to different binary signal’s dutycycle for the
first method or different motion orientation for the second.

##  IV Period and Dutycycle

As the ATIS sensor is able to perceive really fast changes in illumination, it
is possible to exploit the temporal redudancy of a blinking pattern. This
approach can be viewed as an extension of standard binary time-multiplexing
methods with the specificity that the different code-words are given by the
frequency (and dutycycle) of the signal projected onto a pixel instead of
being computed through a sequence of intensity values. Fig.
LABEL:fig:dutycycle shows the pattern spatial organisation, each color refers
to a specific dutycycle. It uses a Debruijn sequence that is repeated for each
line. Ideally we should have projected full lines instead of small spots but
practical limitations limited our choices. A periodic projection generates, at
the pixel level, an event stream composed of successive ON and OFF event
bursts following the signal’s periodic behavior. The length and the number of
events constituting each burst is defined by the contrast and illumination
condition as well as the set of biases used to operate the sensor.

In a perfect case, each edge of the signal would trigger only one event of the
corresponding polarity for each pixel. In this situation, measuring the inter-
event time between successive ON and OFF events would be enough to get a
correct estimation of the signal’s frequency and dutycycle. However, because
of small imprecisions in the attribution of timestamps by the sensor’s arbiter
and possible variations in the global illumination, the number of events
triggered at each edge of the signal is not consistent. To ensure at least one
event per period, we have two choices: increase the sensor’s sensibility to
generate bursts at each edge of the signal so that faulty events are less
prevalent, or consider a spatial neighborhood instead of a single pixel. The
first solution has a major drawback as increasing the number of event at high
speed may generate congestion in the event stream and results in less accurate
timestamps attribution. The second solution, even if lowering the spatial
resolution, is more appropriate.

As a result, instead of considering event streams from each pixel, we create a
set of two streams (one for each polarity) composed of events from a local
pixel neighborhood of size $\mathcal{N}$. This operation increase the
probability of having at least one event per edge but also increase the size
of bursts. Eventually, using this spatial neighborhood also increase the
precision of the timestamp of the first event in each burst as it is always
the pixel with the smallest latency that shoot first, reducing the impact of
electronic noise in the sensor’s chip.

####  IV-1 Burst filter

Figure 5:  Example of the burst filter on a sinusoidal input. The input signal
is transformed into an event stream by the sensor and at each new event the
inter-spike delay is added to the filter’s action potential. Between events,
the AP follows an exponential decay. At any given polarity state, when the
opposite polarity threshold is reached, the output toggles, effectively
reflecting the changes of the input signal.

As we are working on binary pattern, every information related to variations
of higher frequency than the signal’s one has to be removed. The first step in
this process is to reduce the sensor’s output, from a series of event clusters
following the light variations, to a single event at each edge of the input
signal. To recover a robust estimation of the signal period (and dutycycle),
it is necesssary to clean the bursts and extract, a single, precise event with
the correct polarity. A filter is assignated to each pixel’s output that
integrates inter-event delays as shown on Fig. 5. The filter behaves like two
integrate and fire neuron models inhibiting each other but modelized as a
single filter, instead of having each neuron triggering when their threshold
is reached, it uses a hysteresis function to generate outputs from a single
action potential. The filter’s action potential is sent through a hysteresis
comparator with two different thresholds that enables the control of the
sensor dynamics for each polarity. Beginning in an uncertain state at time
$t_{n}$, when a first threshold $thresh(up,down)$ is reached the output state
is changed accordingly and it remains untill the opposite polarity threshold
is crossed. The variations of the filter’s action potential $AP$ then follows
the rule: for an event $e_{i}$ arriving at time $t_{n+1}$ with polarity
$p(e_{i})$, first the exponential decay is computed as

$\widehat{AP}(t_{n})=AP(t_{n})\exp{\frac{(t_{n+1}-t_{n})}{\tau}}$ (1)

with $\tau$ a time constant set according to the stimulus frequency, then the
update is done by

$AP(t_{n+1})=\widehat{AP}(t_{n})\cdot(1+(t_{n+1}-t_{n}).p(e_{i})),$ (2)

where $p(e_{i})=\\{-1,1\\}$. Filter’s output state $S_{out}(t)$ is modified
when the following condition is met:

$S_{out}(t_{n+1})=|AP_{n+1}|>thresh(\mathrm{bar}{S}_{out}(t_{n})),$ (3)

where $thresh(\mathrm{bar}{S}_{out}(t_{n}))$ is the absolute value of the
opposite polarity threshold for a given output state. Each time this condition
is reached, a new event $e(x,y,t,p)$ is triggered and sent to the rest of the
filtering chain with $(x,y)$ as the position of the pixel, timestamp $t_{n+1}$
at which the output state changes and the polarity $p$ reflecting filter’s new
state. The output stream of this filter is a spike train $S_{e(x,y,t,p)}$ with
accurate timestamps from which we can easily estimate the signal’s frequency
$\widehat{f}(x,y)$ after each spike of same polarity by

$\widehat{f}(x,y)=\frac{1}{\widehat{T}}=\frac{1}{t_{k}^{p}-t_{k-1}^{p}}.$ (4)

The signal’s dutycycle is estimated from the computation of its two half
periods by comparing the successive alternations of positive and negative
filtered events as

$\widehat{T}_{k}^{ON}=(1-\lambda)\widehat{T}_{k-1}^{ON}+\lambda(t_{k}^{OFF}-t_{k-1}^{ON}),$
(5)
$\widehat{T}_{k}^{OFF}=(1-\lambda)\widehat{T}_{k-1}^{OFF}+\lambda(t_{k}^{ON}-t_{k-1}^{OFF}),$
(6)

with $\lambda$ a smoothing parameter.

The signal’s dutycycle is given by

$\widehat{\alpha}=\frac{\widehat{T}^{ON}}{\widehat{T}^{ON}+\widehat{T}^{OFF}}.$
(7)

Results of the method are shown in Fig. 6 for a wide range of frequencies
going from $40\mathrm{Hz}$ and up to $1\mathrm{k}\mathrm{Hz}$. Bias settings
of the sensor were fixed during the experiment and chosen to cover the
complete range of frequencies without changing the behavior of event bursts to
reduce the influence of background noise, however the threshold for the OFF
channel was a bit too low at $1\mathrm{k}\mathrm{Hz}$ and no output could be
recorded. This doesn’t mean that the algorithm can’t extract frequencies over
$1\mathrm{k}\mathrm{Hz}$, but such high speed requires a particular bias
setting. The scope of this paper is to present a pattern design for structured
light reconstruction and as such, bias settings should be set in order to
optimize the behavior in a narrower range of frequencies around the desired
projection rate.

Error bars shows the standard deviation over the $100$ periods measured at
each frequency. Looking at the ON events results for $500\mathrm{Hz}$ and
$666\mathrm{Hz}$, we can see that the consistency of the frequency estimation
is heavily dependent on the good quality of the data recorded: at those two
frequencies, the number of ON events recorded fell to only one third of those
of negative polarity, which translates into missed edges or inaccurate
timestamps on the filter’s ouput. The reason of this drop in events number at
particular frequencies is yet unknown but could lay in ambient light condition
or background noise. The OFF channel offers more consistency and we can
observe a slight increase in variance for high frequencies that express the
progressive influence of small imprecision in timestamp attribution as the
period decrease closer to the sensor’s temporal resolution.

In Fig. 7 we show the error in percentage of the projected frequency, obtained
for a signal blinking at $1\mathrm{k}\mathrm{Hz}$, over $500$ periods. We
obtain a mean extracted frequency $\widehat{f}=999.92\mathrm{Hz}$ for a
$3\times 3$ neighborhood and the error stays under $3\%$, whitch corresponds
to a standard deviation of $\sigma=9.19\mathrm{Hz}$.

Figure 6: Results for frequencies ranging from $40\mathrm{Hz}$ to
$1\mathrm{k}\mathrm{Hz}$ over 100 periods. Error bars shows the standard
deviation observed in each set. Dutycycle was fixed at $50\%$. The third graph
shows the precentage of ON events in each set and can explain variations of
the variance observed at $500\mathrm{Hz}$ and $666\mathrm{Hz}$. Figure 7:
Error percentage in frequency estimation, for $500$ periods of a signal at
$1\mathrm{k}\mathrm{Hz}$ in a $3\times 3$ pixel neighborhood. The algorithm
gives an estimation maintained under less than $3\%$ error.

###  IV-A Random shifting

Given a set of $n$ signals following a square wave of fixed frequency, with no
other treatment done, every signals switch at the same time and will generate
at least $2n$ events (ON and OFF alternance) for each period $T$ if used in
projection on the ATIS. What we desire is an asynchronous stimulus. Here, not
only it is totally synchronous, but it is also eventually a large number of
changes for a single time step. We need to lower this number as much as
possible to stay under practical limitations and generate smooth activity on
the sensor. To ensure that, we introduce a random phase shift for every pixels
in the set with uniform distribution so that the probability $p$ of having a
change event $X$ in a time window of size $\Delta t$ is

$\begin{split}p(X)&=p(t\leq X\leq t+\Delta t)\\\ &=\int_{t}^{t+\Delta
t}\frac{2}{T}d\tau\\\ &=\frac{2\Delta t}{T}\end{split}$ (8)

The number of changes generated by the whole set of phase shifted pixels
follows a binomial distribution $\mathcal{B}(n,p)$ with average value $\mu=np$
and variance $\sigma^{2}=np(1-p)$ as shown in Fig.LABEL:phasehist.

In a coded pattern constructed from an alphabet of size $k$, we have $k$ sets
of pixel with same frequency so the distribution of the total number of
changes would be the sum of each binomial functions
$\mathcal{B}_{k}(n_{k},p_{k})$. An approximation of this sum can be found by a
normal distribution $\mathcal{N}(\mu,\sigma)$ as long as $\sigma^{2}$ is not
too small with

$\mu=\sum{n_{k}p_{k}}\qquad\sigma^{2}=\sum{n_{k}p_{k}(1-p_{k})}$ (9)

##  V Orientation tracking

This second method aims to take advantage of the powerful tracking
capabilities of the ATIS sensor as it has been demonstrated in
LABEL:Lagorce2014. Instead of using only time to code the symbols of the
pattern, a spatio-temporal code is designed. Spots are moved along different
oriented axis in a pattern organized in the same way than the previous method.
Tracking is done using gaussian kernels designed in the previously cited paper
and, for each determined position, orientation of the moving axis is computed
from the covariance matrix by determining the highest correlation axis.
Compared to the first method’s codification, tracking enables to gather
information much faster, as each successive projection step improve the
current tracker covariance instead of having to wait a complete period. The
acquisition can be made each time the trackers are updated and this delay can
be set lower than a complete movement revolution. However, the spatial
resolution is lowered by the size of the moving object and symbol density
cannot be too high or the tracking would fail to separate symbols from each
other.

###  V-A Perfect Maps

Perfect Maps, are random arrays of dimensions $r\times v$ in which a sub-
matrix of dimensions $n\times m$ appears only once in the whole pattern.
Theoretically, PM are constructed having dimensions $rv=2^{mn}$ but usually
the zero-matrix is not considered giving $rv=2^{mn}-1$ unique sub matrices of
$m\times n$ dimensions. In , Morano defines a brute force algorithm (non De
Bruijn based) to generate these maps by adding successive pseudo-random
elements to the pattern, row by row starting from the top left corner. Each
new element corresponds to a new sub matrix with an associated code with
alphabet of size $k$, and its Hamming distance between all codes already
included in the pattern is calculated. If this distance is superior or equal
to the minimum Hamming distance specified, the test is passed and the code is
added, otherwise the other $k-1$ symbols are tested. When the algorithm gets
stuck, the whole pattern is canceled and the algorithm starts over. Since the
constructed pattern can contains only a subset of all possible combinations,
such PM is called Perfect SubMaps (PSM). Other works used this brute force
approach like Claes using a colour codification or Albitar with geometrical
features. More recently, Maurice and al. proposed a hybrid algorithm to build
large patterns of more than $200\times 200$ features in a very short time with
high Hamming distance, using a splitting strategy to perform the Hamming test
in the codeword space instead of the pattern array. We choose to use in this
paper an algorithm inspired by this last method.

To generate a matrix M, with a number of symbols $k$ and a codeword length n,
we first create an array with $k^{n}$ boolean flags for all possible
codewords, initialised to zero. Starting on the top left corner of M, we
randomly generate a first $3\times 3$ patch and perform a Hamming test on
every codewords with a distance $H<H_{min}$. Each flag array entry is accessed
by a single codeword such that the $n$ symbols are the coefficients in $k$
basis. If all the corresponding flags are set to false, the patch is added to
M and a next patch is generated until the map is complete
(Fig.LABEL:fig:PMgen). If the algorithm gets stuck in a situation were the
added symbol generates only forbidden codes, we shuffle recently added codes
until all neighboring codes affected by this modification pass the Hamming
test.

##  VI Phase shifting

Phase shifting is a well known method since 1966 (Carré, 1966) that has been
widely used in interferometry for its better accuracy and speed as opposed to
static images analysis methods. PS algorithms records a series of images that
encode the wavefront phase in the variations of the intensity pattern. Various
algorithms has been developped over the years using different number of images
and phase unwrapping methods. Similarly, fringe projection for 3D shape
reconstruction has been exhaustively studied because of its simple setup and
processing associated with relatively high speed and resolution and among
those, phase-shifting methods have proven very effective. In standard imaging,
intensity values of successive shifted patterns are compared point-to-point to
recover a unique phase value for each parrallel line along the coding axis.
The existence of a depth discontinuity in the scene will cause a deformation
of the projected pattern that will be recorded as a phase deviation as shown
on Fig. LABEL:PSexample. The unique phase value recovered is matched with the
projected pattern and the shape of the scene objects can be computed.

Figure 8:  When a periodic signal is continusously shifted in front of the
sensor, the event stream of two different pixels gives precise temporal
information about the phase of the signal.

###  VI-A Event-base phase-shifting

Commonly, in order to recover phase deviation, several phase-shifted patterns
are projected and recorded. Using a set number of shifted images, a system of
equations is defined and solving it allows to retrieve the phase value. With
AES sensors however, no intensity value can be measured directly and no
complete frames can be acquired which makes common methods unusable. Instead,
a continuous shifting of the pattern is used to produce on each line along the
coding axis, a series of events with precise timestamps following the
variations of the pattern fringes. Every period of the signal projected
generates on each pixels a succession of alternatively positive and negative
spike bursts $\mathcal{S}(p)$ as shown in Fig. 8. Taking a given pixel $p_{i}$
as reference on a line, for every pixels $p_{j,\forall j\neq i}$, we can
measure the time-shift between bursts $\mathcal{S}(p_{j})$ and
$\mathcal{S}(p_{i})$. To recover the wrapped phase $\phi$ , a modulo operator
is applied on this value to constrain it between 0 and the signal’s period,
then it is projected on the interval $[0,2\pi)$ by :

$\phi=\frac{2\pi}{P_{signal}}mod(\Delta
t(\mathcal{S}(p_{j}),\mathcal{S}(p_{i})),P_{signal})$ (10)

An unwrapping algorithm is used to compute the absolute phase value $\Phi$ for
every pixels. On each line along the coding axis, the following reccursive
formula is applied :

$\Phi_{x}=\Phi_{x-1}+\lambda*mod(\phi_{x-1}-\Phi_{x-1},2\pi)$ (11)

Once the unwrapped phase is obtained, a mapping between the projected pattern
and the phase image can be found and the 3D measurement is done.

##  VII Results

###  VII-A Experiment

The method is applied on a scene containing two objects placed around one
meter away from the system. We used a SPSM of $20\times 30$ symbols and the
pattern codification is done using the dutycycle of the signal. The
neighborhood in the burst filter is set to 1 ($3\times 3$ pixels) and signal’s
frequency is set to $20\mathrm{Hz}$ for performance issues. The algorithm runs
on MatLab and even if it could be heavily parallelized, it wasn’t the case on
this experiment and real time wasn’t achieve. An image of accumulated outputs
from filters is given in Fig. 9, colors in the image reflect the estimation of
the dutycycle by each pixel associated filter. Point correspondance is
performed by extracting the $3\times 3$ dutycycle based codewords and as
pattern’s dots are bigger than a single pixel, we perform a spatial averaging
to obtain a sub-pixel position of the dot in the camera frame. After
triangulation, a 3D point cloud is obtained (Fig. 10)

Figure 9: Image of the projected pattern as extracted by the algorithm. Colors
correspond to the estimated dutycycle. Figure 10: 3D point cloud. Figure 11:
Reconstructed depth

##  VIII Conclusion and Discussion

This paper introduced a methodology that makes use of the hguh temporal
resolution of the event based sensor to rethink the problem of structured
light depth reconstruction. We used a unique spatial distribution of light
patterns composed of elements each flickering at a unique frequency. We also
used the idea of random shift allowing each neighboorhood to be unique. Each
methodology relying on the combined use of an event based camera and a light
projector coding each spatial position in the frequency domain can make use of
the developed approach. There are several ways to decode frequencies from the
output of the event based camera. The method used here could be bettered,
using more precise event based camera allowing fequency to be extracted from
the timing of the oscillation of each pattern from inter spike information. If
that condition is fulfilled even simpler patterns could be used without the
need to rely on unique neighborhoods.

## References

  * [1] A. D. L. Escalera, L. Moreno, M. A. Salichs, and J. M. Armingol, “Continuous mobile robot localization by using structured light and a geometric map,” _International Journal of Systems Science_ , vol. 27, no. 8, pp. 771–782, 1996. [Online]. Available: http://www.tandfonline.com/doi/abs/10.1080/00207729608929276
  * [2] M. Asada, “Map building for a mobile robot from sensory data,” _Systems, Man and Cybernetics, IEEE Transactions on_ , vol. 20, no. 6, pp. 1326–1336, 1990\.
  * [3] Y. Kakinoki, T. Koezuka, S. Hashinami, and M. Nakashima, “Wide-area, high dynamic range 3-d imager,” in _Proc. SPIE 1194, Optics, Illumination, and Image Sensing for Machine Vision IV_ , 1990.
  * [4] R. T. Chin, “Automated visual inspection: 1981 to 1987,” _Computer Vision, Graphics, and Image Processing_ , vol. 41, no. 3, pp. 346–381, 1988. [Online]. Available: http://www.sciencedirect.com/science/article/pii/0734189X88901089
  * [5] E. M. Petriu, Z. Sakr, H. J. W. Spoelder, and A. Moica, “Object recognition using pseudo-random color encoded structured light,” in _Instrumentation and Measurement Technology Conference, 2000. IMTC 2000\. Proceedings of the 17th IEEE_ , vol. 3. IEEE, 2000, pp. 1237–1241. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=848675
  * [6] J. Park, G. N. DeSouza, and A. C. Kak, “Dual-beam structured-light scanning for 3-d object modeling,” in _3-D Digital Imaging and Modeling, 2001. Proceedings. Third International Conference on_. IEEE, 2001, pp. 65–72. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=924399
  * [7] D. Page, A. Koschan, Y. Sun, and M. Abidi, “Laser-based imaging for reverse engineering,” _Sensor Review_ , vol. 23, no. 3, pp. 223–229, 2003. [Online]. Available: http://www.emeraldinsight.com/journals.htm?issn=0260-2288&volume=23&issue=3&articleid=876385&articletitle=Laser-based+imaging+for+reverse+engineering&
  * [8] R. Klette, K. Schlüns, and A. Koschan, _Computer vision: three-dimensional data from images_. Springer Singapore, 1998, vol. 20. [Online]. Available: http://www.researchgate.net/publication/220689403_Computer_vision_-_three-dimensional_data_from_images/file/32bfe50d203e8594ea.pdf
  * [9] J. Batlle, E. Mouaddib, and J. Salvi, “Recent progress in coded structured light as a technique to solve the correspondence problem: a survey,” _Pattern recognition_ , vol. 31, no. 7, pp. 963–982, 1998. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0031320397000745
  * [10] X. Su and W. Chen, “Fourier transform profilometry:: a review,” _Optics and Lasers in Engineering_ , vol. 35, no. 5, pp. 263–284, may 2001. [Online]. Available: https://doi.org/10.1016/S0143-8166(01)00023-9
  * [11] J. Salvi, J. Pags, and J. Batlle, “Pattern codification strategies in structured light systems,” _Pattern Recognition_ , vol. 37, no. 4, pp. 827–849, april 2004. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0031320303003303
  * [12] J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” _Pattern Recognition_ , vol. 43, no. 8, pp. 2666–2680, aug 2010. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S003132031000124X
  * [13] J. L. Posdamer and M. D. Altschuler, “Surface measurement by space-encoded projected beam systems,” _Computer graphics and image processing_ , vol. 18, no. 1, pp. 1–17, 1982. [Online]. Available: http://www.sciencedirect.com/science/article/pii/0146664X8290096X
  * [14] S. Inokuchi, K. Sato, and F. Matsuda, “Range imaging system for 3-d object recognition,” in _Proceedings of the International Conference on Pattern Recognition_ , vol. 48, 1984, pp. 806–808.
  * [15] D. Bergmann, “New approach for automatic surface reconstruction with coded light,” in _SPIE’s 1995 International Symposium on Optical Science, Engineering, and Instrumentation_ , 1995, pp. 2–9. [Online]. Available: http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1007846
  * [16] a. D.Caspi, N.Kiryati, “Range imaging with adaptive color structured light,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 20, no. 5, pp. 470–480, 1998.
  * [17] M. Maruyama and S. Abe, “Range sensing by projecting multiple slits with random cuts,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 15, no. 6, pp. 647–651, 1993.
  * [18] J. T. N. Durdle and V.Raso, “Animprovedstructuredlighttechnique for surface reconstruction of the human trunk,” in _IEEE Canadian Conference on Electrical and Computer Engineering_ , IEEE, Ed., vol. 2, 1998, pp. 874–877.
  * [19] L. Zhang, B. Curless, and S. Seitz, “Rapid shape acquisition using color structured light and multi-pass dynamic programming,” in _First International Symposium on 3D Data Processing Visualization and Transmission_ , 2002, pp. 24–36.
  * [20] J. Salvi, J. Batlle, and E. Mouaddib, “A robust-coded pattern projection for dynamic 3d scene measurement,” _Pattern Recognition Letters_ , vol. 19, no. 11, pp. 1055–1065, 1998. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0167865598000853
  * [21] P. G. C. Albitar and C.Doignon, “Designofamonochromaticpattern for a robust structured light coding,” in _IEEE International Conference on Image Processing_ , IEEE, Ed., vol. 6, 2007, pp. 529–532.
  * [22] X. Maurice, P. Graebling, and C. Doignon, “A pattern framework driven by the hamming distance for structured light-based reconstruction with a single image,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , IEEE, Ed., 2011, pp. 2497–2504.
  * [23] J. Tajima and M. Iwakawa, “3-d data acquisition by rainbow range finder,” in _International Conference on Pattern Recognition_ , IEEE, Ed., vol. 1, 1990, pp. 309–313.
  * [24] C. Guan, L. Hassebrook, and D. Lau, “Composite structured light pattern for three-dimensional video,” _Optics Express_ , vol. 11, no. 5, pp. 406–417, 2003. [Online]. Available: http://www.opticsexpress.org/abstract.cfm?URI=oe-11-5-406
  * [25] T. K. N. Ono, T. Shimizu and S.Ando, “Real-time3dimagerbasedon spatio-temporal phase unwrapping,” in _SICE 2004 Annual Conference_ , vol. 3, 2004, pp. 2544–2547.
  * [26] K. D. I. Ishii, K. Yamamoto and T. Tsuji, “High-speed3dimageacquisition using coded structured light projection,” in _International Confer- ence on Intelligent Robots and Systems_ , IEEE, Ed., 2007, pp. 925–930.
  * [27] S. Zhang, D. V. D. Weide, and J. Oliver, “Superfast phase- shifting method for 3-d shape measurement,” _Optics Express_ , vol. 18, no. 9, pp. 9684–9689, 2010. [Online]. Available: http://www.opticsexpress.org/abstract.cfm?URI=oe-18-9-9684
  * [28] P. Lichtsteiner, C. Posch, and T. Delbruck, “A 128 $\times$ 128 120 db 15 $\mu$s latency asynchronous temporal contrast vision sensor,” _IEEE Journal of Solid-State Circuits_ , vol. 43, no. 2, pp. 566–576, 2008.
  * [29] D. M. C. Posch and R. Wohlgenannt, “Aqvga143dbdynamicrange frame-free pwm image sensor with lossless pixel-level video compression and time-domain cds,” _IEEE Journal of Solid-State Circuits_ , vol. 46, no. 1, pp. 259–275, 2011.
  * [30] J. Leero-Bardallo, T. Serrano-Gotarredona, and B. Linares-Barranco, “A 3.6 s latency asynchronous frame-free event-driven dynamic-vision-sensor,” _IEEE Journal of Solid-State Circuits_ , vol. 46, no. 6, pp. 1443–1455, 2011\.
  * [31] A. G. Andreou and K. A. Boahen, “A contrast sensitive silicon retina with reciprocal synapses,” in _Advances in Neural Information Processing Systems_ , vol. 4, 1991, pp. 764–772. [Online]. Available: http://www.stanford.edu/group/brainsinsilicon/pdf/92confN IP SC ontrast.pdf
  * [32] A. Stocker and R. Douglas, “Analog integrated 2-d optical flow sensor with programmable pixels,” in _Proceedings of the 2004 International Symposium on Circuits and Systems_ , IEEE, Ed., vol. 3, 2004, pp. 9–12.
  * [33] K. Boahen, “Point-to-point connectivity between neuromorphic chips using address events,” _IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing_ , vol. 47, no. 5, pp. 416–434, May 2000.