# Temporal-Spatial Processing of Event Camera Data via Delay-Loop Reservoir
Neural Network

Richard Lau (clau@peratonlabs.com), Anthony Tylan-Tyler (anthony.tylan-
tyler@peratonlabs.com),
Lihan Yao (lihan.yao@peratonlabs.com), Rey de Castro Roberto
(rreydecastro@peratonlabs.com),
Robert Taylor (robert.m.taylor@peratonlabs.com), Isaiah Jones
<EMAIL_ADDRESS>
(Peraton Labs
331 Newman Springs Road, Red Bank NJ, 07701, USA)

###### Abstract

This paper describes a temporal-spatial model for video processing with
special applications to processing event camera videos. We propose to study a
conjecture motivated by our previous study of video processing with delay loop
reservoir (DLR) neural network [1, 2], which we call _Temporal-Spatial
Conjecture_ (TSC). The TSC postulates that there is significant information
content carried in the temporal representation of a video signal and that
machine learning algorithms would benefit from separate optimization of the
spatial and temporal components for intelligent processing. To verify or
refute the TSC, we propose a _Visual Markov Model_ (VMM) which decompose the
video into spatial and temporal components and estimate the mutual information
(MI) of these components. Since computation of video mutual information is
complex and time consuming, we use a Mutual Information Neural Network[10] to
estimate the bounds of the mutual information. Our result shows that the
temporal component carries significant MI compared to that of the spatial
component. This finding has often been overlooked in neural network
literature. In this paper, we will exploit this new finding to guide our
design of a _delay-loop reservoir_ neural network for event camera
classification, which results in a _18% improvement_ on classification
accuracy.

_Keywords_ Temporal-Spatial Conjecture, Video Markov Model, Mutual
Information, Mutual Information Neural Estimate, Delay Loop Reservoir, Event
Camera, Edge Application.

##  1 Introduction

While there are many Artificial Intelligence/Machine Learning (AI/ML)
techniques in literature that are used in practice for classification and
prediction of video signals, most employ techniques that are handcrafted by
experts in the field, often without theoretical justifications [5, 6]. In this
paper, which is based upon work supported by the Defense Advanced Research
Projects Agency (DARPA) on “Reservoir-based Event Camera for Intelligent
Processing on the Edge (RECIPE)”, we decompose a video signal into temporal
and spatial component and postulate a Temporal-Spatial Conjecture (TSC) that
suggests benefits by separating temporal and spatial processing to reduce
training time and complexity. To validate or refute the TSC, we propose a
Video Markov Model (VMM) that captures inter-dependence of key components of
the visual signal. The VMM quantifies the dependence by measuring by their
mutual information (MI). Results from the VMM provide insight to help explain
why certain approaches work and possibly guidance for the construction of
improved AI/ML algorithms. The TSC study was motivated by a prior result from
our previous work in HyDDENN [1, 2], where we had found that seperate
temporal/spatial processing in a drone/bird classifier leads to an improvement
of Signal-to-Noise ratio compared to a classifier that processes the combined
temporal-spatial information. In this paper, we aim to quantify such
advantage.

To demonstrate the TSC, we apply the lessons learned from the VMM to process
event camera videos. Recent years have seen growing research efforts in “Event
Cameras”, which provides a new way of capturing visual sensing in the emerging
field of neuromorphic engineering. Event cameras are fundamentally different
from traditional visual cameras, and has created a paradigm shift in how
visual information is acquired. A traditional camera samples the scene
periodically (e.g. 30 frames/sec) and captures light intensity on all the
pixels. In contrast, an event camera records only differential pixel intensity
information upon scene changes. This represents a fundamentally different
method of capturing visual information and leads to the following benefits: 1)
the sensed data is sparse and significantly smaller and thus reduces storage
and transport requirements by a factor of 1000×; 2) ultra-high temporal
resolution and low latency (in microseconds); and 3) very high dynamic range
(140dB), and 4) low power consumption. These characteristics have great impact
on the next generation of sensors deployed in both military and commercial
systems, including self-driving automobiles, high-speed Unmanned Aerial
Vehicles (UAV), robotics, ultra-high-speed data transfer, storage reduction,
and smart surveillance.

To qualify the potential improvement, we will apply the TSC-inspired
architecture to process event video via a Delay-Loop Reservoir Neural Network
(DLR) for classifying and predicting spike signals from event-based cameras.
Advantages of DLR for edge networks have been studied and reported in [2]. In
this paper, we will demonstrate how TSC is used in improving DLR for
processing event videos.

This paper is organized as follows. In Section 2, we describe the TSC and its
motivation, followed by defining the VMM structure. Section 2 also includes a
description of the DVS event dataset that we use to perform the study. Section
3 goes into detailed description of the Mutual Information Neural Estimate
approach for estimating the Mutual Information for the VMM. It then describes
the results when the VMM is applied to the DVS data. Motivated by the TSC
result and insight, Section 4 applies the Delay-Loop Reservoir (DLR) for
classification of the DVS data. It compares performance results between a
direct DLR vs. a modified DLR as inspired by the TSC and explains how lessons
learned from the TSC helps to design a better architecture for the DLR.
Section 5 discusses future research in extension of the study and implications
of the TSC. Section 6 gives the final conclusion.

##  2 Temporal Spatial Conjecture

The study of the TSC is motivated by understanding the fundamental theory
behind processing video signals. Video processing algorithms often process the
video signal as a 3-dimensional tensor of space and time, which requires large
amount of memory and computational resource. Since a video signal is made up
of consecutive video frames, each of which are static images, the spatial
content of nearby frames is often highly correlated or even repetitive.
However, most current AI/ML techniques used for classification and prediction
of video signals do not exploit this correlation. While well-designed AI/ML
algorithms, often handcrafted by experts in the field, can extract this
correlation to achieve the goal, the algorithms are excessively
computationally demanding, which becomes a significant problem for efficient
edge applications.

The current study asks the following question: Are there efficient ways to
break up the video signal before processing which may lead to more efficient
resource usage? We start with postulating a conjecture, called Temporal-
Spatial Conjecture (TSC), which states that separating the temporal and
spatial processing is beneficial in visual processing to reduce training time
and complexity. The TSC was motivated by a prior result from our previous work
in HyDDENN [1, 2], where we had found that separating temporal/spatial
processing in a drone/bird classifier leads to an improvement of Signal-to-
Noise ratio of 4dB compared to a classifier that processes the combined
temporal-spatial information. In this paper, we will build an analytic tool to
quantify such advantage.

###  2.1 Video Model

To analyze the TSC, we build a Video Markov Model (VMM) for analysis of
information content of the components of the video signal. We first represent
an event video ($V$) as a set of contiguous event frames, $F_{t},$for
$t=1,...,D$ are the time samples. The frames are sampled in time, which can be
non-uniform. For notational simplicity we assume uniform sampling in the
following. Thus, we write,

$V={F_{t}}\hskip 150.0ptt=1,...,D$

The VMM is created via two procedures. First it decomposes the event-based
video signal ($V$) into a spatial component ($S$) and a temporal component
($T$). The decomposition should satisfy two criteria:

  1. 1.

Each component $S$, $T$, is a subset of the events of the original signal $V$.

  2. 2.

The union of $S$ and $T$, carries information that approaches that of the
entire signal. Such information is measured with respect to an AI/ML
application such as predicting or classifying the label of the video.

Figure 1: Video Markov Model

As shown in Figure 1, the spatial component $S$ is defined to be a set of
sampled event frames such that the frames have negligible correlation among
them. Thus, we write

$S=\\{F_{t_{u}}\\}\hskip 100.0ptt_{u}=\text{time index of uncorrelated
frames}$

There are different ways to obtain the spatial component. One way is to
randomly shuffle the event frames so that they are uncorrelated. Another
method is to sparsely sample the event frames such that there is little
correlation between the samples. Note that both methods satisfy criteria 1
given above. However, the later method is usually preferred since it reduces
processing and storage complexity.

While the spatial component modifies and reduces temporal content, the
temporal component retains temporal information but allows reduction of the
spatial information. There are numerous ways that $T$ can be defined to
satisfy the VMM criteria. We propose to define $T$ as clusters of events in
contiguous frames. Thus, we write

$T=\\{C_{t}\\}\hskip 150.0ptt=1,...,D$

where $C_{t}s$ are clusters of the events at frame of time $t$. This
definition does not place stringent requirements on the structure of the
clusters but requires that the temporal information be unchanged from that of
the original video signal. The second part of the VMM is to compute or learn
the mutual information for 3 configurations as shown in Figure 2. We describe
in more detailed in the following section on the computation and estimation of
these MIs _I_. The amount of MI carried in these models provide answer to the
TSC and hints on how to design efficient AI/ML architectures.

Figure 2: VMM sub structure diagrams for study of TSC

###  2.2 Event Data

For testing of the VMM and testing of subsequent even data classification, we
will use Dynamic Vision Sensor (DVS) dataset [7]. DVS is a low-power, real-
time, event-based hand gesture recognition system. It is the first implemented
entirely on event-based hardware. DVS captures changes in pixel brightness,
transmitting data only when a change is detected, which significantly reduces
power consumption compared to traditional frame-based cameras. The DVS dataset
comprises of 11 hand gesture categories from 29 subjects under 3 illumination
conditions (see Figure 3).

Figure 3: Examples from 8 of the 11 classes in the DVS dataset

Figure 4: VMM RGB captures of gestures, the corresponding event stream, and
the aggregation of events into frames.

To process the data, the event stream is first partitioned in time as shown in
Figure 4, where events in a partition, i.e. a time window, is aggregated into
one frame. In the DVS gesture dataset, each frame collects events over a time
window of 30 milliseconds. By sequencing 200 consecutive frames, an
observation represents $\sim$6 seconds in time.

##  3 Mutual Information Extraction

Solving the TSC requires computation of Mutual Information, $I$, which is a
difficult task due to the large size of the data. This section describes how
we use the Mutual Information Neural Estimate (MINE) [10] method to estimate
$I$.

###  3.1 Mutual Information

The mutual information between two variables is a measure of how dependent
they are on each other. For a pair of variables where one is known, the mutual
information quantifies how much is known about the unknown variable from the
known variable. If the mutual information is high, then the unknown variable
is well constrained by the known variable. If the mutual information is low,
then the unknown variable is relatively unconstrained by the known variable.
Applying this to the TSC, we can use the mutual information to see how well
the separate spatial and temporal information can be associated with the video
label.

The mutual information is defined as the difference between the entropy of a
single variable, $H)X=\mathbb{E}_{p}[logp(x)]$, where $\mathbb{E}_{P}[x]$ is
the expectation value of $x$ for the distribution $P$, and the joint entropy
of both, $H(X|Y):$

$I(X,Y)=H(X)-H(X|Y)$

The mutual information between the two variables is constrained from above by
$I(X,Y)\leq min(H(X),H(Y))$ This constraint arises from $I(X,Y)=I(Y,X)$ and
that $min(H(X|Y))=0$ when $X$ is fully determined by $Y$. Thus the maximum
mutual information occurs when one variable is fuly determined by the other
and is then constrained by which variable has the lowest entropy. This
constraint will allow us to determine when the mutual information is ’high’ as
well as compare $I(V_{S},V_{C})$ and $I(V_{T},V_{C})$ where $V_{S}$ is the
spacial information of the video, $V_{T}$ is the temporal information of the
video, and $V_{C}$ is the video class label.

###  3.2 Mutual Information Neural Estimate

Determining the mutual information $I(X,Y)$ can be done when the joint
distribution of the variables is known. In our case, these distributions are
not known and, further complicating matters, the video data is very high
dimensional. For the individual frames, for example, the probability
distribution is over a 32768 dimensional space. Due to its high
dimensionality, the sample frames are likely to lie very far from each other,
making the estimation of the whole distribution unfeasible. In order to
overcome this limitation, we use the Mutual Information Neural Estimation
(MINE) [10] to estimate the mutual information between pairs of variables.

Figure 5: A diagrammatic representation of the MINE algorithm

MINE is a neural network which estimates the mutual information using an
approximate calculation of the Kullback-Leiber (KL) divergence. To begin, the
mutual information can be rewritten in terms of the KL diverence between the
joint distribution $P(X|Y)$ and the product of the marginal distribution
$P_{X}\oplus P_{Y}$

$I(X,Y)=D_{KL}(P(X|Y)\hskip 2.0pt||\hskip 2.0ptP(X)\oplus P(Y))$

The KL divergence between two distributions is defined as

$I(X,Y)=D_{KL}(P||Q)=\mathbb{E}_{P}[log(\frac{P}{Q})]$

$D_{KL}$ can be estimated using the Donsker-Varadhan representation as

$D_{KL}(P||Q)\geq\sup_{T^{\prime}\in\mathcal{F}}\mathbb{E}_{P}[T^{\prime}]-log(\mathbb{E}_{Q}[e^{T^{\prime}-1}])$

where $T^{\prime}$ is a function and $\mathcal{F}$ is a subset of functions.
Using this inequality, the neural network shown in Figure 2 attempts to learn
the function $T^{\prime}$ which maximizes the Donsker-Varadhan representation
of the KL divergence.

Figure 6: An example of the trajectories making up $V_{T}$ from a video of the
‘clapping hands’ class of the DVS128 Gesture dataset

Applying MINE to video data requires us to rigorously define the video spatial
($V_{S}$) and temporal ($V_{T}$) information. We choose to represent the
$V_{S}$ as a segment of 5 full resolution images covering 0.5s of video (a
flattened 5x2x128x128 tensor). $V_{T}$, is represented as a flattened 8x200
matrix of the trajectories of 4 centroids defined by $k$-means clustering on
the event stream which has been sliced along time into segments of 0.027s (see
Figure 6).

Using these definitions, the MINE can be used to calculate the entropy of each
variable as $I(X,X)=H(X)$. This gives us $H(V_{S})>24$, and $H(V_{C})\geq
2.49$. As the MINE estimate is established to be tight, we use $H(V_{T})$ as
the maximum value for both $I(V_{S},V_{C})$ and $I(V_{T},V_{C})$. Thus we can
directly compare the values of $I(V_{S},V_{C})$ and $I(V_{T},V_{C})$ to
understand the relative information content of each. Next, we can use MINE to
estimate $I(V_{S},V_{C})\geq 2.13$ (85% of maximum) and $I(V_{T},V_{C})\geq
2.31$ (92% of maximum)

###  3.3 Implication of Mutual Information on TSC

These results suggest that for event camera data, the bulk of the information
for labelling the video data is in the temporal information. As we highlighted
before, it is difficult to separate the spatial and temporal information. For
example, $V_{T}$ is composed of the spatial location of cluster centroids over
time which still includes spatial information. Similarly, our definition of
$V_{S}$ covers 0.5s of events over 5 frames (0.1s of events compiled into each
frame), which includes changes over time. If, instead, a single frame compiled
from 0.1s of events is used for $V_{S}$, the mutual information drops to 1.26
(50% of maximum), suggesting that the spatial information alone contains
significantly less information necessary for labelling the videos.

An additional complication of the spatial information is its high
dimensionality. At a resolution of 2x128x128, the resulting vector has 32768
components. In order to simplify the spatial information while retaining the
temporal information which appears to contain the bulk of the video
information, we propose to down sample the spatial resolution to 2x8x8 over
200 frames, giving vectors of 25600 a 21% reduction in size to cover the whole
video compared to a single frame while still retaining the bulk of the spatial
information. Labelling this the full video information, $V$, we find that
$I(V,V_{C})\geq 2.44$ (98% of maximum). Thus, by simplifying the spatial
information through down sampling while retaining all of the temporal
information, we are able to improve our result by 6%. This result is
significant as it motivates us to design a corresponding DLR to take advange
of the insight learned. This will be verified in an DLR application in the
following section.

##  4 DLR Classification of Event Camera Data

This section will illustrate how the insight gained from the TSC study
motivate an improvement on the Delay Loop Reservoir (DLR) algorithm for event
camera processing.

###  4.1 DLR Algorithm

Figure 7: Delay-Loop Reservoir structure

As shown in Figure 7, the Delay Loop Reservoir (DLR) [2] is a nonlinear, high
dimensional model within the reservoir computing framework. The DLR operates
on serialized stream data by sequentially upsampling each sample into high
dimensions, via random weights. The resulting high dimensional representation
is then processed by the core delay loop, which serves two functions: 1)
inject nonlinearity using hyperbolic tangent; and 2) allows processing
temporal processing via a leaky factor which combine past temporal data with
current data. The leaky factor thus controls the degree of “memory” of the
DLR, and can be used to select frequency components for processing. After loop
processing, the data is classified by a linear classifier such as a ridge
regressor. Note that the entire DLR has only training at the ridge regressor
stage and the delay loop itself does not required training. This is a main
difference between reservoir based NN and other NN’s. The simplicity of the
DLR architecture leads to practical implementation in hardware, which has been
demonstrated in FPGA implementation as reported in [2].

###  4.2 Initial Result DLR Algorithm

We first apply the DLR on the 11-class classification task of the DVS dataset
with tuning parameters including the leaky factor, loop dimensionality, and
ridge regression regularization. Our initial result showed an accuracy of 76%.
We noted that this sub-par performance can be attributed to an overfitting
problem. Overfitting occurs when a model learns the training data too well,
including its noise and outliers, and therefore performs poorly on unseen
data. This is especially prevalent for event camera data, where noise among
events in an event stream may become a ‘crutch’ for the model. As an
illustration, Figure 8 shows the spurious noisy events that cause overfitting.
The red blob in the event stream has been memorized by the model during
training. Whenever the spurious event appears, the model predicts class $i$.
However, in the test set, an event stream of class $i$ does not contain this
specific event.

Figure 8: Example of spurious events in the event stream

###  4.3 TSC-inspired modification of the DLR Algorithm

As illustrated above, the problem of overfitting in event classification is
that the DLR tries to classify using the noisy data. It does a good job for
the training data and indeed this is what had found. During training, the DLR
performs close to 100% accuracy while its accuracy falls to 76

In traditional NN, methods to mitigate overfitting in neural networks include
reducing the number of weights in the NN or generation of new training data by
data augmentation. These would not work in DLR as the loop does not have
training parameter and regularization in the ridge regressor was already used.
Data augmentation is also not desirable as it adds complexity to the model.

Figure 9: TSC-inspired preprocessing for DLR improvement

Instead, we will apply a data reduction module to the DLR as inspired by the
TSC result. This modification is shown in Figure 9. The Sparse Spatial module
suggests lowering the sampling rate of the events while the Temporal module
captures frames from the event. To tackle the overfitting problem, we will
substantially reduce the spatial sampling rate, but retaining sufficient
temporal resolution, which is consistent with the TSC result that the temporal
component carries more information for label classification. Using this
modification, the new DLR architecture is shown in Figure 10.

Figure 10: Modified DLR Architecture

The main addition of Figure 10 is the Spatial Filter and Down-sample module.
We apply a low-pass filter to remove high-frequency noise that could lead to
aliasing artifacts before subsampling. The subsampling process reduces the
image size by keeping every $n$-th pixel in each row and column, and
discarding the rest. The procedure $g$ is applied to pixel coordinate $(i,j)$,
given by

$g(i,j)=\sum_{k,l}^{n}f(k,l)h(i-\frac{k}{r},j-\frac{l}{r})$

where $f(k,l)$ is the smoothing filter, $h(i-\frac{k}{r},j-\frac{l}{r})$
subsamples the blurred image to dimensions r by r. Figure 11 shows effect of
the sampling process for various subsampling rates.

Figure 11: Image of subsampling effect

Figure 12: DLR Accuracy with respect to different frame sub-sampling

Figure 12 shows the performance improvement of the modified DLR architecture.
It shows that the largest improvement is achieved by downsampling the event
frame to an 8 × 8 event pixel frames. This result cannot be adequately
explained by traditional methods to tackle overfitting in machine learning, as
traditional NN methods focus on spatial learning. However, this result can be
fully explained from the viewpoint of the TSC as temporal component carries
more information. Since noise is most prominent in the spatial component, we
can substantially reduce the frame size and thus reducing overfitting to a
minimum without affecting the temporal component.

Using the modified DLR architecture, with fine tuning of other DLR parameters
including leaky factor, loop dimensionality, we were able to achieve an
accuracy of 89.6% for the 11-class DVS data classification. This is in
comparison with an accuracy of 96% [11] when attention neural networks are
used. Thus, the DLR has achieved on-par performance with State of the Art
Attention Neural Network but with substantial reduction in training time and
computation complexity as measured by Multiply-and-Accumulate units [2].

##  5 Future Work

This paper is motivated by exploring processing of the temporal component of a
video signal as this area of research seems to have not been the focus in
machine learning algorithm development. The proposed VMM validates an
important result for applying DLR to event data by optimizing the spatial
component. We have not investigated optimization with respect to the temporal
domain. For example, what would be an optimum frame structure? And what would
be a joint spatial-temporal optimum structure? Also, we would also like to
study the impact of TSC on other applications such as object segmentation and
object recognition.

##  6 Conclusion

This paper describes a temporal-spatial model for video processing with
special applications to processing event camera videos. We first describe the
Temporal-Spatial Conjecture which postulates that there is significant
information content carried in the temporal representation of a video signal.
TSC also advocates that machine learning algorithms would benefit from
separate optimization of the spatial and temporal components for intelligent
processing. We described the Visual Markov Model which decomposed videos into
spatial and temporal components and estimate the mutual information of these
components with respect to machine learning functions. The VMM study found
that temporal component carries more information than its spatial counterpart.
The finding is important, which was exploited to help design a modification to
the delay-loop reservoir algorithm for classification of DVS event data. The
modified DLR algorithm tackles the overfitting problem successfully and
achieved 18% improvement on classification accuracy, thereby validating the
TSC.

##  7 Acknowledgement

This paper is based upon work supported by the DARPA MTO contract
N6600122C4025 on “Reservoir-based Event Camera for Intelligent Processing on
the Edge (RECIPE)". We thank Dr. Whitney Mason and Dr. Timothy Klausutis,
Program Managers, for their vision on the TSC and DLR and guidance of the
program, Greg Jones for his guidance and technical discussions on the program,
and Justin Mauger (CIV USN NIWC PACIFIC CA (USA)) for many technical
discussions. The views, opinions and/or findings expressed are those of the
authors and should not be interpreted as representing the official views or
policies of DARPA or the U.S. Government.

## References

  * [1] DARPA AIE.  on Hyper Dimensional Data Enabled Neural Networks (HyDDENN) A.I. Exploratory PA-19-03-03, December 2019.
  * [2] Richard Lau, Lihan Yao, Todd Huster, William Johnson, Stephen Arleth, Justin Wong, Devin Ridge, Michael Fletcher, William C. Headley.  Scaled-Time-Attention Robust Edge Network.  CoRR abs/2107.04688 (2021)
  * [3] DARPA AIE.  on Photonic Edge AI Compact Hardware (PEACH) DARPA-PA-18-02-05
  * [4] Guy Van der Sande, Daniel Brunner and Miguel C. Soriano  Advances in photonic reservoir computing  Nanophotonics, 6(3), 2017.
  * [5] He Wang, Edmond S. L. Ho, Hubert P. H. Shum, Zhanxing Zhu  Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling  IEEE Trans. Vis. Comput. Graph. 27(1): 216-227 (2021).
  * [6] Li Yao, Atousa Torabi, Kyunghyun Cho  “Describing Videos by Exploiting Temporal Structure,”  arXiv:1502.08029v5 [stat.ML] 1 Oct 2015.
  * [7] A. Amir, B. Taba, D. Berg, T. Melano, J. McKinstry, C. Di Nolfo, T. Nayak, A. Andreopoulos, G. Garreau, M. Mendoza, J. Kusnitz, M. Debole, S. Esser, T. Delbruck, M. Flickner, and D. Modha.  "A Low Power, Fully Event-Based Gesture Recognition System,"  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017.
  * [8] Pierot et al  Prophesee 1 Megapixel Automotive Detection Dataset.  "Learning to Detect Objects with a 1 Megapixel Event Camera," NeuroIPS 2020.
  * [9] Lukoševičius, M., & Jaeger, H  Reservoir computing approaches to recurrent neural network training. Computer Science Review,  3(3), 127-149. 2009.
  * [10] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, Devon Hjelm.  "Mutual Information Neural Estimation,"  Proceedings of the 35th International Conference on Machine Learning, PMLR 80:531-540, 2018.
  * [11] Sabater, L. Montesano, A. Murillo  "Event Transformer. A sparse-aware solution for efficient event data processing"  arXiv preprint arXiv:1804.09028, 2018.
  * [12] T. J. O’Shea, T. Roy and T. C. Clancy  "Over-the-Air Deep Learning Based Radio Signal Classification,"  IEEE Journal of Selected Topics in Signal Processing, vol. 12, no. 1, Feb. 2018.
  * [13] Deepsig18 data: https://www.deepsig.ai/datasets. Vectorized deepsig18 data https://drive.google.com/open?id=1vrzz1Dbf98E-Q79-3CFjGNS7sw1HJVM6.
  * [14] Richard Lau, T. Woodward  “See-through Obscurants via Compressive Sensing in Degraded Visual Environment,”  SPIE April 2015. doi:10.1117/12.2178039


# Managing Clouds in Cloud Platforms

Hassan Gobjuka  Verizon
919 Hidden Ridge
Irving, TX 75083
Email<EMAIL_ADDRESS>Kamal A. Ahmat Hassan Gobjuka Department of
Information Technology Verizon City University of New York 919 Hidden Ridge
New York, NY 11101 Irving, TX 75038<EMAIL_ADDRESS><EMAIL_ADDRESS>
##  I Motivation

Managing cloud services is a fundamental challenge in today s virtualized
environments. These challenges equally face both providers and consumers of
cloud services. The issue becomes even more challenging in virtualized
environments that support mobile clouds. Cloud computing platforms such as
Amazon EC2 provide customers with flexible, on demand resources at low cost.
However, they fail to provide seamless infrastructure management and
monitoring capabilities that many customers may need. For instance, Amazon EC2
doesn’t fully support cloud services automated discovery and it requires a
private set of authentication credentials. Salesforce.com, on the other hand,
do not provide monitoring access to their underlying systems. Moreover, these
systems fail to provide infrastructure monitoring of heterogenous and legacy
systems that don’t support agents. In this work, we explore how to build a
cloud management system that combines heterogeneous management of virtual
resources with comprehensive management of physical devices. We propose an
initial prototype for automated cloud management and monitoring framework. Our
ultimate goal is to develop a framework that have the capability of
automatically tracking configuration and relationships while providing full
event management, measuring performance and testing thresholds, and measuring
availability consistently. Armed with such a framework, operators can make
better decisions quickly and more efficiently.

##  II Challenges

These tasks are achieved through an agentless monitoring of the cloud’s
infrastructure. While traditional network management methods suffer from
inherited difficulties [1, 2], implementing seamless network management and
monitoring framework entails several new challenges:

  * •

Discovering the relationship of virtualized resources to underlying physical
infrastructure.

  * •

Minimizing the overhead of monitoring and problem determination across a
physical and virtualized infrastructure.

  * •

Handling security-related constraints that may affect data collection is
probably one of the most serious issues.

  * •

Response action should be taken regarding a particular virtual or physical
device within the hard response deadline time frame. In agentless-based
monitoring systems, this can be insured only by implementing high number of
threads, which in turn increases complexity.

  * •

Dealing with infrastructure management issues such as root-cause analysis
becomes more complex.

##  III Design and Implementation

We propose an event-based model where events are placed on an in-memory
publish/subscribe bus on the Management Server, enabling a high throughput of
events.

The event bus architecture, depicted in Figure 1 enables any authorized
mediator to create events on the bus, and any authorized consumer to access
events from the bus. Events on the bus show current status of infrastructure
components.

Figure 1: An initial prototype of our cloud management and monitoring system.

The framework will provide a set of APIs to simplify creation of consumer and
mediator applications. A set of language extensions and Web services will be
used to enable Perl, Ruby, or Java scripts to create events on the bus. To
support high level of reliability and scalability, the Distributed Collector
subsystem will be multi-threaded. Furthermore, events will are normalized from
any source into a common format, which will enable consistent processing.

## References

  * [1] T. Benson, A. Akella, and D. A. Maltz,  Unraveling the complexity of network management,  In NSDI, 2009.
  * [2] H. Gobjuka, and Y. Breitbart,  Ethernet Topology Discovery for Networks with Incomplete Information,  IEEE/ACM Transactions on Networking, 2010, In Press.