# A thermodynamic origin for the Cohen-Kaplan-Nelson bound

Satish Ramakrishna<EMAIL_ADDRESS>Department of Physics &
Astronomy, Rutgers, The State University of New Jersey, 136 Frelinghuysen Road
Piscataway, NJ 08854-8019

(31-Oct-2021)

###### Abstract

The Cohen-Kaplan-Nelson bound is imposed on the grounds of logical consistency
(with classical General Relativity) upon local quantum field theories. This
paper puts the bound into the context of a thermodynamic principle applicable
to a field with a particular equation of state in an expanding universe. This
is achieved without overtly appealing to either a decreasing density of states
or a minimum coupling requirement, though they might still be consistent with
the results described. We do so by defining an appropriate Helmholtz free
energy which when extremized relative to a key parameter (the Hubble radius
$L$) provides a scaling formula for the entropy with the Hubble radius (an
exponent $r$ used in the text). We deduce that the CKN bound is one possible
solution to this extremization problem (with $r=\frac{3}{2}$), but there are
others consistent with $r=2$.

The paper establishes that the holographic principle applied to cosmology is
consistent with minimizing the free energy of the universe in the canonical
ensemble, upon the assumption that the ultraviolet cutoff is a function of the
causal horizon scale.

##  I INTRODUCTION

The Cohen-Kaplan-Nelson CKN  bound hints at a connection between the
ultraviolet cutoff applicable to local quantum field theory and the infrared
cutoff applicable to the same theory. It is obtained by placing a limit on
when gravitational effects should dominate the problem and collapse the
relevant field configuration into a black hole. The limit is actually ensured
by stating, by observation, that gravitational effects do not actually appear
to play a significant role. The bound therfore constrains the behavior of
local quantum field theories in the universe, by connecting the ultraviolet
cutoff to the size of the causal horizon - the infrared limit of the theory.
Essentially, the connection between the ultraviolet cutoff and the causal
horizon size connects the number of possible internal states (which scales
with some power of the length) to the smallest possible wavelength in the bulk
of the region.

Some recent work by Banks and Draper Banks  shows how to deduce the
$\frac{3}{2}$-power of the entropy/length-scale dependence (that the CKN bound
leads to) from an appropriately chosen ansatz for the density of states in the
bulk.

A parallel piece of (prior) work, especially that of Fischler and Susskind
FischSuss  deduces various cosmological consequences of equations of state. It
finds that the quadratic entropy/length-scale dependence required by
holography is realized only with the equation of state $P=\rho$, which is
achieved in the special case of a massless scalar field with no potential
function, i.e., with only a kinetic term. They deduce this from a very general
starting point using classical gravity, i.e., the Friedmann equation combined
with the equation of state.

The question arises - how are the above pieces of work connected? Is it an
accident that the CKN bound gives a $\frac{3}{2}$ power scaling with length
for entropy while the Fischler-Susskind bound gives quadratic scaling for
$w=1$ and $\frac{3}{2}$-power length scaling for $w=\frac{1}{3}$? Can one
derive both from one framework? Are they actually just two ways of looking at
the same problem?

###  I.1 Summary of the work presented

This work finds a common ground between the above sequence of thoughts as well
as posits the CKN bound as the consequence of a physical principle. To this
end, we frame the problem as one of equilibrium thermodynamics of the entire
universe bathed in a thermal gas that obeys, in the simplest case, the
equation of state $P=w\rho$ for various values of $w$. We define the Helmholtz
free energy for this thermal gas and also (subsequently) apply the same
condition to a radiation-dominated universe. From just this, we deduce that
the dependence of entropy on the length scale is indeed the $\frac{3}{2}$
power. The resulting equation turns out to be exactly the CKN bound.

We conclude that the CKN bound is exactly the equivalent of extremizing the
free-energy in an expanding universe (exactly what it is extremized versus is
explained later). After this, we apply the methodology to the case $w=1$ and
re-derive the Fischler/Susskind conditions (and the area-entropy connection),
in the canonical ensemble, by minimizing the Helmholtz free energy. We also
confirm the bound that they found in this case.

In what follows, we will focus on a parameter $r$, which is the power of the
Hubble radius that the entropy scales in accordance with. Standard holographic
ideas would require $r=2$. The CKN bound yields $r=\frac{3}{2}$. As we will
note in the discussions that follow, there are other solutions to the
extremization problem that yield $r=2$ under differing assumptions about
densities of states and non-linear equations of state.

##  II The Fischler-Susskind bound

Let us study the thermodynamics of an entropy-saturating field in an expanding
universe as in FischSuss . The authors impose the holographic condition that,
operating with Planck units

$\displaystyle\sigma R_{H}^{D}<[a(t)R_{H}]^{D-1}$ (1)

where $t$ is the cosmic time, $R_{H}=\int_{0}^{t}\frac{dt}{a(t)}$ is the
horizon, $\sigma$ is the (assumed) constant comoving entropy density and $D$
is the number of spatial dimensions. Saturating the limit is achieved by
setting $a(t)\sim t^{p}\rightarrow p=\frac{1}{D}$, which is consistent with an
entropy-saturating equation of state $(P=w\rho$ with $w=1)$. In this limit,
the R.H.S of Equation (1) is proportional to the square of $L=a(t)R_{H}$; in
$3$ spatial dimensions, this is the area.

We can derive these from applying the thermodynamic principle of extremizing
the free energy as follows. We consider the following cases, for the radiation
gas ($w=\frac{1}{3}$) and the scalar field with only kinetic term ($w=1$).

We will use the following general formulas, viz. if $P=w\rho$, then, in $D$
spatial dimensions, with $t$ the cosmic time and $a(t)$ the scale factor

$\displaystyle\rho\sim\frac{1}{a^{D(1+w)}}\>\>\>,\>\>\>a(t)\sim
t^{\frac{2}{D(1+w)}}\>\>\>,\>\>\>T\sim\frac{1}{a(t)^{Dw}}$ (2)

The causal horizon is defined as

$\displaystyle R_{H}=\int_{0}^{t}\frac{dt}{a(t)}\sim t^{1-\frac{2}{D(1+w)}}$
$\displaystyle L=a(t)\times R_{H}=t\>\>\>\>$ (3)

In these terms, for a general equation of state (the temperature dependence is
derived under the constant entropy density assumption Stornaiolo  ),

$\displaystyle\rho\sim\frac{1}{L^{2}}\>\>\>,\>\>\>a(t)\sim
L^{\frac{2}{D(1+w)}}\>\>\>,\>\>\>T\sim\frac{1}{L^{\frac{2w}{1+w}}}\>\>\>,\>\>\>\rho\sim
T^{\frac{1+w}{w}}$ (4)

These formulas apply under the condition that the equation of state is linear,
i.e., that $w$ is a number and not itself a function of $\rho$ or $a$. In
addition, we are assuming homogeneity and isotropic expansion with constant
entropy density; hence, the quantities $\rho,a(t),T$ cannot deviate from the
functional form above. We are exploring in a future publication the effect of
non-linear equations of state upon the arguments in this paper (some remarks
are made in Sections IV and VI).

In what follows, we will keep dimensions in check with appropriate powers of
$M_{P}$, the Planck energy and work in natural units ($c=1,\hbar=1)$.

###  II.1 The thermodynamic framework

We apply the usual laws of thermodynamics to an expanding universe bathed in a
thermal gas with equation of state $P=w\rho$ and a temperature $T$. We work in
the canonical ensemble - this consistent choice implies we are assuming that
we can define an equilibriated temperature for this thermal gas, though we
allow processes (eg. expansion) where the temperature changes. We define the
Helmholtz free energy

$\displaystyle F=\rho V-TS$ (5)

where $\rho$ is the energy density, $V=L^{3}$ is the volume of the causal
horizon, $T$ is the temperature of the field and $S$ its entropy.

All these quantities (including the entropy $S$) are functions of $L$, the
size of the causal horizon. We are asserting that the universe remains in
equilibrium while it is expanding. In equilibrium, the free energy is at an
extremum and the universe’s expansion needs to be consistent with the free
energy being extremized. All the quantities in the free-energy expression are
functions of $L$. It is natural that we should extremize the free energy with
respect to the causal horizon $L$.

These relationships are detailed in Equation (4) above for a linear equation
of state. For instance, the temperature in a radiation-dominated universe, is
$T\sim\frac{M_{P}}{a(t)}\sim\sqrt{\frac{M_{P}}{L}}$. In addition, $S$, the
entropy, is assumed to depend on it as $S\sim L^{r}$. Since entropy is
dimensionless, we will write it as $S=(M_{P}L)^{r}$.

The energy density has two components, generically. There will be a
temperature dependent piece and there will be a vacuum energy piece. For
instance, in a radiation-dominated universe with an ultraviolet cutoff
$\Lambda$, the total energy is

$\displaystyle\rho V=L^{3}\bigg{(}\sigma T^{4}+\Lambda^{4}\bigg{)}$ (6)

where $\sigma$ is the Stefan-Boltzmann constant. It is tempting to think that
this term is dominated in the late(r) universe by the ultraviolet cutoff term,
but we are going to allow $\Lambda$ to scale with some power of $L$, say
$\Lambda(L)\sim M_{P}(\frac{1}{M_{P}L})^{\mu}$. Here we have inserted $M_{P}$
to produce proper dimensions for $\Lambda$.

As explained earlier, we will extremize the free energy with respect to $L$ \-
physically, this implies that the universe is in equilibrium as it expands.
Since $\rho V$ and $TS$ are proportional to powers of $L$, the extremization
produces, upto factors of order $\sim 1$, that $\rho V\sim TS$.

Under this analysis, the vacuum energy is a function of $L$, so we are forced
to assume if will participate and equilibriate with the fluid that pervades
the universe. It is indeed an energy density and in this picture, it does
participate in the equilibrium thermodynamics of the universe.

Hence, we are going to impose the uniform condition $\rho V\sim TS$.

##  III The case $w=\frac{1}{3}$

We are going to make some simplifications to match dimensions correctly. We
start with (as explained previously),

$\displaystyle\rho
V=TS\>\>\>\>\>\rightarrow\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>L^{3}\bigg{(}\sigma
T^{4}+\Lambda^{4}(L)\bigg{)}\sim\sqrt{\frac{M_{P}}{L}}\>(M_{P}L)^{r}$ (7)

This approximate equality can be written as

$\displaystyle\sigma\bigg{(}\frac{T}{M_{P}}\bigg{)}^{4}+\bigg{(}\frac{\Lambda(L)}{M_{P}}\bigg{)}^{4}\sim\frac{1}{(M_{P}L)^{\frac{7}{2}-r}}$
(8)

We notice that

$\displaystyle\bigg{(}\frac{T}{M_{P}}\bigg{)}^{4}\sim\frac{1}{(M_{P}L)^{2}}\>\>\>\>\>,\>\>\>\>\>\bigg{(}\frac{\Lambda(L)}{M_{P}}\bigg{)}^{4}\sim\bigg{(}\frac{1}{M_{P}L}\bigg{)}^{4\mu}$
(9)

By inspection, an easy solution to this equation by comparing powers of $L$ is
given by $\mu=\frac{1}{2}$ and $r=\frac{3}{2}$.

This case (with $r=\frac{3}{2}$) is also the one obtained by the CKN
condition, i.e., since the two terms in Equation (8) have the same dependence
on $L$, we can write (upto factors of order $\sim 1$),

$\displaystyle\bigg{(}\frac{\Lambda(L)}{M_{P}}\bigg{)}^{4}\sim\frac{1}{(M_{P}L)^{2}}$
(10)

which is exactly the CKN condition.

Of course, this makes sense - in this particular case, with $w=\frac{1}{3}$,
the causal horizon is $L$. In addition, the energy inside the universe is
$\sim\rho L^{3}\sim\frac{1}{L^{2}}L^{3}\sim L$. The Schwarzschild radius is of
the same order as the causal horizon, which is indeed the CKN condition, so it
is indeed not surprising that this case leads to the above result.

As it turns out (and this is originally due to the model’s authorsCKN ), this
solution can also be obtained by counting states in a self-consistent manner.
If we count the states in the quantum field theory, we’d obtain (from a
standard local field theory calculation), and approximate the solution for
$\Lambda$ by ignoring the non-vacuum energy part,

$\displaystyle(M_{P}L)^{r}=(dimensionless\>constants)\>\times\>L^{3}\int_{\frac{1}{L}}^{\Lambda(L)}dk\>k^{2}\sim
L^{3}\Lambda^{3}(L)=(M_{P}L)^{3}(\frac{\Lambda(L)}{M_{P}})^{3}$
$\displaystyle\rightarrow(M_{P}L)^{r}\sim(M_{P}L)^{3}\frac{1}{(M_{P}L)^{\frac{3}{4}(\frac{7}{2}-r)}}\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>$
$\displaystyle\rightarrow
r=\frac{3}{2}\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>$
(11)

which again indicates that the entropy scales as the $3/2$ power of the Hubble
length scale, as is known. Note that we have derived this from an application
of the minimization of free energy principle rather than the CKN condition,
though, as may be noted.

##  IV Other solutions to Equation (8)

Apart from the condition $\mu=\frac{1}{2}$, we can consider other solutions,
classified as

  1. 1.

$\mu>\frac{1}{2}$: Here, in Equation (8), the temperature term dominates for
large $L$, we can easily see that $r=\frac{3}{2}$ is the only consistent
solution.

  2. 2.

$\mu<\frac{1}{2}$: In this case, the ultraviolet cutoff term dominates for
large $L$. We might then proceed to find a relation between $\mu$ and $r$,
however, note that by Equation (4), the energy density is forced to scale as
$\rho\sim\frac{1}{L^{2}}$. This constrains the problem massively and prevents
there being any other solution (other than $r=\frac{3}{2}$) in the linear
equation of state case.

  3. 3.

If we assumed that the density of states (as used in Equation (11)) is $\sim
k^{m}$ ($m=2$ is the usual case) and we further write $\Lambda(L)\sim
M_{P}\frac{1}{(M_{P}L)^{q}}$, we obtain the following consequences, i.e.,

$\displaystyle
q(m+2)=2\>\>\>\>,\>\>\>\>3-q(m+1)=r\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>$
$\displaystyle\rightarrow\>\>\>\>q=\frac{2}{m+2}\>\>\>\>\>\>\>\>\>r=\frac{m+4}{m+2}\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>$
(12)

which are obtained by requiring that the energy density scales as
$\sim\frac{1}{L^{2}}$ while the number of states (the entropy) scales as
$L^{r}$. We then obtain different possible combinations of parameters (see
Table 1). For consistency with the minimization of free energy principle, we
still expect $r=\frac{3}{2}$. However, with a non-linear equation of state, we
could get a different temperature dependence with $L$ (the Hubble horizon) and
this can lead to other values of $r$ corresponding to different values of $m$
and $q$. This rich array of solutions is being exploredSatishFut , but a quick
summary follows.

###  IV.1 Nonlinear equations of state

Generally speaking, with an equation of state, i.e.,

$\displaystyle P=(w_{0}+w_{1}\rho)\rho$ $\displaystyle
T\sim\frac{1}{a^{D(w_{0}+w_{1}\times a^{-D(1+w_{0})})}}$ (13)

hence, if we define $T\sim\frac{1}{L^{\gamma}}$ and if $w_{1}>0$,
$\gamma>\frac{2w_{0}}{1+w_{0}}$ (conversely, $\gamma$ is smaller, if $w_{1}$
were negative). Then, we find

$\displaystyle
q=\frac{2}{m+2}\>\>\>\>\>\>\>\>\>r=\frac{m+4}{m+2}\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>$
$\displaystyle
1+\gamma=r\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>$
$\displaystyle\rightarrow
q=\gamma\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>\>$
(14)

Interestingly, the inverse power of temperature is the same as the inverse
Hubble-length scaling exponent of the ultraviolet cutoff. In general, we’d
expect $r$ to be higher with an equation of state $P=(w_{0}+w_{1}\rho)\rho$,
if $w_{1}>0$.

Table 1: Combinations of Density of States and Length Scale Dependence of $\Lambda$ (Density of States exponent) | Scaling of $\Lambda$ | Entropy Scaling
---|---|---
$m$ | $q$ | $r$
2 | $\frac{1}{2}$ | $\frac{3}{2}$
1 | $\frac{2}{3}$ | $\frac{5}{3}$
0 | 1 | 2

To demonstrate the generality of the free-energy principle, we next apply it
to the case of the “perfect” gas, i.e., with $w=1$, exemplified by the scalar
field with only a kinetic term.

##  V The case $w=1$

In this case, the total energy again is written as the sum of the field energy
plus the vacuum energy upto the ultraviolet cutoff, i.e.,

$\displaystyle\rho V=L^{3}(\alpha T^{2}+\Lambda^{4}(L))$ (15)

Again, the ultraviolet cutoff is assumed to possess some dependence upon $L$,
of a form similar to the previous section, i.e., $\Lambda(L)\sim
M_{P}(\frac{1}{M_{P}L})^{\mu}$.

The $T^{2}$ dependence in Equation (15) is a straightforward consequence of
the thermodynamics of expansion, see Equation (4). The equation has an
appropriate (dimensionful) constant $\alpha$. In this case, however, the
temperature term in the universe dominated by this type of radiation is
$T\sim\frac{M_{P}}{a(t)^{3}}\sim\frac{1}{L}$. Again, assuming that the entropy
$S\sim(M_{P}L)^{r}$, we write the equation in a dimensionally simple way
(${\hat{\alpha}}$ is a dimensionless constant constructed from $\alpha$ and
$M_{P}$), i.e.,

$\displaystyle\big{(}{\hat{\alpha}}\big{(}\frac{T}{M_{P}}\big{)}^{2}+\big{(}\frac{\Lambda}{M_{P}}\big{)}^{4}\big{)}\sim\frac{1}{(M_{P}L)^{4-r}}$
(16)

A similar observation to the above is in order, i.e.,

$\displaystyle\bigg{(}\frac{T}{M_{P}}\bigg{)}^{2}\sim\frac{1}{(M_{P}L)^{2}}\>\>\>\>\>,\>\>\>\>\>\bigg{(}\frac{\Lambda(L)}{M_{P}}\bigg{)}^{4}\sim\bigg{(}\frac{1}{M_{P}L}\bigg{)}^{4\mu}$
(17)

Again, by inspection of the terms, one possible self-consistent solution is

$\displaystyle\mu=\frac{1}{2}\>\>\>\>\>,\>\>\>\>\>r=2$ (18)

is a self-consistent solution. We thus find, in agreement with Fischler and
Susskind FischSuss , that the entropy/area relation is reached for the case
$P=\rho$.

##  VI Other solutions to Equation (16)

Apart from the condition $\mu=\frac{1}{2},r=2$, we can consider other
solutions, classified as

  1. 1.

$\mu>\frac{1}{2}$: Here, in Equation (16), the temperature term dominates for
large $L$, we can easily see that $r=2$ is the only consistent solution.

  2. 2.

$\mu<\frac{1}{2}$: In this case, the ultraviolet cutoff term dominates for
large $L$. We again find no other possible solutions (other than $r=2$) due to
the constraints on the evolution of the energy density
($\rho\sim\frac{1}{L^{2}}$). Once again, the general analysis in the previous
section applies and we can get other values of the entropy exponent $r$, with
suitable choices of $q$ and $m$ and a density of states calculation. However,
we still need to stay consistent with the minimization of free energy
principle that we have mandated and we again find that with a non-linear
equation of state, a modified dependence of the temperature $T$ upon $L$ and
other values of the entropy exponent $r$.

All this can be generalized to $D$ spatial dimensions, i.e., with the simplest
solution in all cases,

$\displaystyle r=D-\frac{2}{1+w}$ (19)

In the terminology of Banks & Draper Banks , the power dependence of the
density of states is derived from $L(\epsilon)\sim\frac{1}{\epsilon^{n}}$ and
we will find $n=1+\frac{2}{3}(D-\frac{2}{1+w})$.

Figure 1: Entropy Exponent vs. Pressure/Energy Density Ratio

##  VII Numerical Estimates

The usual objections to the CKN bound are due to the implied size of the
ultraviolet cutoff. If $\Lambda\sim\frac{1}{\sqrt{L}}$, then for the current
Hubble radius, we would expect $\Lambda\sim 0.7eV$. This is a regime that
appears to be well tested to not be a sensible ultraviolet limit. However, if
$\gamma=q=\frac{1}{3}$, for instance, which is a result consistent with a
nonlinear equation of state for $w_{1}<0$, then we could well find
$\Lambda\sim 1\>GeV$, which is a more reasonable limit.

##  VIII Conclusions

We have established a simple free-energy minimization principle in the
canonical ensemble to derive the CKN bound. To some extent, these calculations
use a blend of classical gravitational physics with standard quantum field
theory. They show that (in this case, at least) applying the holographic bound
to the problem is exactly the same as minimizing the free-energy of the
universe in the canonical ensemble. In this manner, they are similar in spirit
to the entropic views of gravity and immediately shed light on entropy-length
scale relations. We extend the method to compute the bound applicable to other
equations of state and demonstrate that one can deduce (in the radiation-
dominated case) the entropy/length-scale relations without amending the
density of states of the underlying quantum field theory. Adding the vacuum
energy into the mix of energies in the thermodynamic problem allows one to
enrich the space of possible entropy/length-scale relationships in a way that
has not been previously appreciated.

##  IX Data Disclosure

Data sharing is not applicable to this article as no new data were created or
analyzed in this study. No conflicts exist.

##  X Acknowledgments

Detailed discussions with Tom Banks and Scott Thomas were especially useful.
An especially lucid colloquium by Patrick Draper motivated this study and
discussions with him after are also gratefully acknowledged. The hospitality
of the Rutgers Physics Department and the NHETC is also acknowledged. A rather
detailed and deep review by an anonymous reviewer is acknowledged for fixing
an error in a preliminary version of this paper.

## References

  * (1) Cohen, A. G., Kaplan D. B., Nelson, A. E. Effective Field Theory, Black Holes and the Cosmological Constant Phys. Rev. Lett. 82, 4971 (1999)
  * (2) Fischler, W., Susskind, L. Holography and Cosmology arXiv:hep-th 9806039
  * (3) Thomas, Scott. Holography Stabilizes the Vacuum Energy Phys. Rev. Lett. 89, 081301 (2002)
  * (4) Banks, T. and Draper, P. Remarks on the Cohen-Kaplan-Nelson bound Phys. Rev. D 101, 126010 (2020)
  * (5) Blinov, N. and Draper, P. Densities of States and the CKN Bound arXiv:hep-th 2107.03530v1
  * (6) Stornaiola, C. Cosmological fluids with general equations of state Physics Letters A, 189 351-360 (1994)
  * (7) Ramakrishna, S. in preparation.
  * (8) Banks, T. On the limits of effective quantum field theory: Eternal inflation, landscapes and other mythical beasts arXiv: 1910.12817
  * (9) Davoudiasl, H. Cohen-Kaplan-Nelson Bound and Minimum Coupling in Effective Field Theories arXiv: 2105.01116


# Dynamic Vision Sensors for Human Activity Recognition

Stefanie Anna Baby1, Bimal Vinod2, Chaitanya Chinni3, Kaushik Mitra4
Computational Imaging Lab
IIT Madras, Chennai, India
<EMAIL_ADDRESS>
###### Abstract

Unlike conventional cameras which capture video at a fixed frame rate, Dynamic
Vision Sensors (DVS) record only changes in pixel intensity values. The output
of DVS is simply a stream of discrete ON/OFF events based on the polarity of
change in its pixel values. DVS has many attractive features such as low power
consumption, high temporal resolution, high dynamic range and less storage
requirements. All these make DVS a very promising camera for potential
applications in wearable platforms where power consumption is a major concern.
In this paper we explore the feasibility of using DVS for Human Activity
Recognition (HAR). We propose to use the various slices (such as $x-y$, $x-t$
and $y-t$) of the DVS video as a feature map for HAR and denote them as Motion
Maps. We show that fusing motion maps with Motion Boundary Histogram (MBH)
gives good performance on the benchmark DVS dataset as well as on a real DVS
gesture dataset collected by us. Interestingly, the performance of DVS is
comparable to that of conventional videos although DVS captures only sparse
motion information.

##  1 Introduction

Conventional video camera uses frame based visual acquisition where each pixel
is sampled at a fixed frame rate irrespective of whether or not their value
changed. This leads to data redundancy and hence increased bandwidth and
memory requirements. Dynamic Vision Sensor (DVS) [9] is a recent innovation in
machine vision that mimics some of the functionalities of the human retinal
vision. Instead of capturing the whole frame, it records only those pixels
that see a change in intensity values. If the magnitude of change in log
intensity value at a pixel is beyond a threshold an ON or OFF event is
generated.

A major advantage of DVS is its ultra low power consumption. This is because
it only generates ON/OFF events and avoids the use of ADCs which consumes the
most power in conventional cameras. Hence DVS could be used to boost the
battery life in wearable or portable devices like untethered Augmented Reality
(AR) devices, which currently use conventional cameras for various purposes
such as gesture/activity recognition and building $3-$D maps. With this idea
in mind, we explore performing activity/gesture recognition using DVS.

DVS is intrinsically suitable for gesture/activity recognition since it does
not record any static information about the scene. Thus, we can avoid the
overhead of preprocessing algorithms such as background subtraction and
contour extraction used in conventional image processing. For the task of
human activity recognition, we propose a simple method of using various slices
($x-y$, $x-t$ and $y-t$) of the DVS video as feature maps. We denote these
maps as motion maps and employ Bag of Visual Words framework to extract
critical features. Recognition rates obtained were similar to that of existing
descriptors under this setting. We also combined the motion maps’ features
with state-of-the-art motion descriptor Motion Boundary Histogram (MBH) to
obtain the best recognition rates, much higher than the HAR performance of
individual descriptors. The results on DVS data are even comparable with the
recognition rates seen in conventional videos. This is quite surprising given
that DVS data is a very compressed version of the original video data with a
remarkably sparse encoding. We experimented on two datasets: the DVS
recordings of UCF11 [7] and a hand gesture DVS dataset collected by us. In
both the datasets our results have been promising for DVS.

###  1.1 Related Work

There are several works in literature for human activity recognition, of which
we mention here a few relevant ones. For a typical activity recognition task,
two types of features are classically extracted - descriptors based on motion
and those based on shape. Motion History Images (MHI) from videos accumulate
foreground regions of a person and accounts for its shape and stance [3].
Several more contour-based approaches such as Cartesian Coordinate Features,
Fourier Descriptors Features [8, 6], Centroid-Distance Features and Chord-
Length Features provide shape description [19]. For motion based descriptors,
Histogram of Optical Flow (HOF) computes optical flow of pixels between
consecutive frames using brightness constancy assumption [11, 4]. Motion
boundary histograms take one step further by performing derivative operation
on the optical flow field. This makes the feature invariant to local
translation motion of the camera and captures only relative motion in the
video [5, 17].

Several other descriptors work by extracting the scene (background), color/hue
and texture based features in a video. But texture and hue information is
unavailable in DVS data because of its binary encoding scheme. The scene
context based descriptors can also not be used with DVS videos since scenes
usually are static in a video, unless there is significant camera motion.
Nevertheless, volume based features like motion and shape often provide
sufficient information required to perform decent recognition and are more
popular than surface features like color and texture.

Human activity recognition has been popularly solved by extracting local
features from videos on which Bag of Visual Words model (BoVW) is learnt and a
classifier, typically SVM is trained [18, 14]. As against this, recent works
on HAR has focussed on deep learning techniques for improving recognition
rates. Deep Convolutional and LSTM Recurrent Neural network units can be
trained to automate feature extraction and directly perform natural sensor
fusion [13] on human videos. Two-stream Convolutional Neural Networks learn
the spatial and temporal information extracted from RGB and optical flow
images of videos and are also becoming common for activity recognition [12,
15]. However our method is simple and easy to implement, providing an
intuitive framework for activity recognition.

Since DVS recording provide us with both motion and shape cues, we exploit
these critical information by proposing a fusion of simple shape and motion
based feature descriptors.

##  2 DVS Based Activity Recognition

Figure 1:  Our proposed method: The event stream from DVS is converted into
video at $30fps$. Motion maps are generated through various projections of
this event video and SURF features are extracted. MBH features using dense
trajectory are also extracted. Bag of features encoding from both these
descriptors are combined and given to linear SVM classifier (one-vs-all).

Unlike conventional camera, DVS captures only motion and thus avoids the need
to perform background subtraction. Also DVS output is usually sparse because
change in pixel intensity occurs only at texture edges. To exploit this
sparsity and motion cues captured by DVS, we propose to extract various
projections of the DVS data (the motion maps) and use them as feature
descriptor for activity recognition. Finally we fuse the motion maps with a
state-of-the-art motion descriptor MBH [5] to further improve the recognition
accuracy. The overall architecture is shown in Figure 1.

We first convert DVS event streams into a video by accumulating events over a
time window. For our experiments we made videos at $30fps$ framerate. From
this, we obtain different $2$-D projections: $x-y$, $x-t$ and $y-t$ by
averaging over each left-out dimension. Thus, $x-y$ projection is obtained by
averaging over the $time$ axis, $x-t$ averages $y-axis$ and $y-t$ by averages
$x-axis$. We call these $2$-D projections as motion maps since DVS captures
the direction of motion of the foreground object.

The $x-y$ motion map gives us the mean pose and stance of an object in the
scene whereas the $x-t$ and the $y-t$ motion maps record the manner in which
the object had moved over the video’s duration. Our proposed $x-y$ motion map
is similar to the idea of motion history images [1] but we have two additional
maps that account for the movement of the object along the horizontal and
vertical directions.

From the motion maps, we extract Bag of Features (BoF), where we use Speeded
Up Robust Features (SURF) [2] extracted through grid search on the maps. This
is followed by $k-means$ clustering of the train data’s features to create a
visual vocabulary of $k$ words. Then features from each video are binned to
these $k$ clusters and are $L_{2}$ normalized. Finally, a linear SVM
classifier under one-vs-all encoding scheme is trained on the encoded features
to predict the performed activity. Since the motion maps inherently complement
each another with the $x-y$ map encoding the shape and pose of the object
while the $x-t$ and the $y-t$ motion maps describing its motion, we combine
all the three motion maps’ descriptors to obtain better classification
accuracy. We tried fusion of features before as well as after performing BoF
and observed that fusion after BoF performs better. This result is in line
with that of [14] where the authors give a comprehensive study of fusing
feature descriptors at several stages of BoF.

For the final recognition task, the motion maps descriptors are further
combined with the MBH descriptor since MBH also encodes the appearance of
objects and local motion in video but in a method that is distinctly
different. MBH takes derivative of optical flow which in turn is computed
using derivative of the video frames with respect to its spatial and temporal
coordinates. Hence MBH employs second order statistics for its feature
extraction while the motion maps use simple zero order statistics. Thus these
two descriptors supplement one another and their combined feature set
outperforms the individual recognition rates. In the next section, we evaluate
these descriptors and the loss in performance of HAR on using DVS data
compared to conventional videos on a benchmark dataset. We also report the
performance on the DVS gesture dataset we have collected. From the results, we
assess the usability of DVS for activity recognition and conclude with its
shortcomings.

##  3 Datasets

We performed our experiments on two datasets - the UCF YouTube Action Data Set
or UCF11 [10] and a DVS gesture dataset collected by us using DVS128.

The UCF11 data was chosen because it is one of the few human action datasets
whose benchmark DVS counterpart is publicly available [7]. The DVS data was
created by the authors [7] by re-recording the existing benchmark UCF11 videos
played on a monitor using a DAViS240C vision sensor. Since the data was not
directly recorded from the wild, this would mean that time resolution greater
than that provided by the UCF11 video is not available in DVS under this
simulated setting. Nonetheless, the dataset is sufficient for our experiments
since it captures the sensor noise in DVS and is used on action videos that by
themselves are not very fast paced.

The UCF11 dataset contains eleven action classes as shown in Figure 1, viz.
basketball shooting, biking, diving, golf swinging, horse riding, soccer
juggling, swinging, tennis swinging, trampoline jumping, volleyball spiking
and walking dog. Each class is further subdivided in to $25$ groups that allow
us to perform Leave One Out (LOO) cross validation twenty five times on the
actions as suggested by the creators of the data.

Figure 2: YouTube Action Data Set 111Image source:
http://crcv.ucf.edu/data/UCF_YouTube_Action.php
Figure 3: Gestures from the DVS dataset collected by us. Ground truth from an
RGB camera is also shown.

For a second round of experiments, we used the DVS hand gesture dataset
collected by us using DVS128 and we refer them as the DVS Gesture data (see
Figure 3). The dataset contains $10$ different hand gestures, each performed
$10$ times by $12$ subjects constituting a total of $1200$ gestures. The hand
gestures used are left swipe, right swipe, beckon, counter-clock wise
rotation, clock wise rotation, swipe down, swipe up, swipe V, wave X and wave
Z. We performed $12$-fold cross-validation for all experiments on this dataset
leaving out one subject each time.

Figure 4 shows the motion maps created from randomly picked videos of the
eleven classes of UCF11 data. Note that in the $x-y$ map, much of the shape
and pose of the object is captured. Similarly, the $x-t$ and $y-t$ slices show
rhythmic patterns based on the movement involved typical for a given action
category. Notable ones among these are winding river-like $y-t$ motion map for
action class swinging and the rhythmic up and down spikes in the $x-t$ motion
map for trampoline class.

(a) $x-y$ Motion Map

(b) $x-t$ Motion Map

(c) $y-t$ Motion Map

Figure 4:  The three Motion maps for $11$ randomly picked UCF11 videos.

##  4 Feature Extraction and classification

This section describes the steps that we used for feature extraction and
classification. To evaluate existing motion descriptors like HoG, HOF and MBH,
we extracted local spatio-temporal features using dense trajectories from the
videos [16]. Dense trajectories are created by dense sampling of images frame
wise. The sampled points are tracked along frames using dense optical flow
field and the trajectory length is limited to $15$ frames to avoid drifting of
tracked points. Along the path tracked, descriptors like HoG, HOF or MBH are
computed using the neighborhood volume of $32\times 32\times 15$ pixels. This
volume is further divided into cells of size $16\times 16$ pixels $\times 5$
frames. So each tracked tube gives a $2\times 2\times 3$ cells. Within each
cell, the histograms of descriptors are found. For HoG and MBH, we used $8$
orientation bins per cell and the magnitude of the feature values were used
for weighting. For HOF, an additional bin was added to account for pixels
whose flow value was smaller than a threshold. All the descriptors were also
$L_{2}$ normalized before performing bag of features. In total, HoG gave
feature descriptors of size $96$ per tracked volume ($2\times 2\times 3$ cells
per tracked path times $8$ bins) while HOF produced $108$ features ($2\times
2\times 3$ cells times $9$ bins). MBH also gave $96$ features similar to HoG,
but in both horizontal and vertical directions. Thus, overall it had twice the
number of features for a chosen trajectory. Bag of features was individually
performed on each of these descriptors. Since each video produced about
$\approx 500,000$ dense tracks, most of them in close proximity to one
another, BoF was done on a subset of training features on $100,000$
trajectories randomly selected. To ensure that every video in the train set
contributes to the codebook, we selected features randomly from each video
instead of pooling all extracted features first and performing random
selection. The codebook dimension in the clustering step was maintained at
$500$. After learning the cluster centers, all features of the video were used
to generate the histograms of the same $500$ bins. Finally the segregated
features were $L_{2}$ normalized and SVM classifier was trained.

On each motion map also we individually performed bag of features with a
codebook of dimension $500$. We have used Matlab’s built-in function
bagOfFeatures for this step and trained one-vs-all linear SVM for the multi-
class recognition. The results under Leave One Out cross-validation method for
all these descriptors are given in the next section.

##  5 Experimental Results

We have conducted our experiments on two datasets as explained in the
following section222For our code and DVS gesture dataset refer -
https://github.com/Computational-Imaging-Lab-IITM/HAR-DVS.

###  5.1 HAR on UCF11 and its DVS counterpart

In this experiment, HAR was performed on the original UCF11 dataset (RGB) and
its corresponding DVS recordings. Table 1(a) provides the recognition rates
obtained with 25 fold Leave One Out cross-validation method.

Dataset | HoG | HOF | MBH |  x-y Motion Map |  x-t Motion Map |  y-t Motion Map |  Combined Motion Maps |  Motion Maps \+ HOF |  Motion Maps \+ MBH
---|---|---|---|---|---|---|---|---|---
Original UCF11 | 0.6319 | 0.5754 | 0.7707 | 0.4397 | 0.4567 | 0.4077 | 0.5867 | 0.6922 | 0.7933
DVS recordings of UCF11 | 0.5358 | 0.6043 | 0.7016 | 0.4943 | 0.451 | 0.4629 | 0.6727 | 0.7299 | 0.7513

(a) Results on UCF11 and its DVS counterpart

Dataset | HoG | HOF | MBH |  x-y Motion Map |  x-t Motion Map |  y-t Motion Map |  Combined Motion Maps |  Motion Maps \+ HOF |  Motion Maps \+ MBH
---|---|---|---|---|---|---|---|---|---
DVS gesture dataset | 0.8768 | 0.9689 | 0.9468 | 0.7748 | 0.8349 | 0.7899 | 0.9529 | 0.9809 | 0.9880

(b) Results on the DVS gesture dataset collected by us

Table 1: Recognition rates for various motion and shape descriptors on the
UCF11 dataset, its corresponding DVS data and the DVS gesture dataset
collected by us. Note that MBH features give $70\%$ accuracy while addition of
motion maps give $75\%$ accuracy on the DVS recordings of UCF11 data Figure 5:
Confusion matrix for UCF11-DVS dataset on combining motion maps and MBH

The results show that fusion of motion maps from the DVS data gave a HAR rate
of $67.27\%$, comparable to the rates for HOF and HoG on the original UCF11
data. Interestingly, with the individual motion map descriptors the DVS
recordings of UCF11 gave higher recognition rates while descriptors like HoG
and HOF performed better on the original videos. This is because there is no
background clutter and scene information in DVS recording for distracting its
bag of features encoding. KNN classifier was also used for the final
predictions, but it gave consistently about $5\%$ lower HAR rates. Similarly,
it was observed that simply using a larger dimension codebook of size $4000$
improves recognition rates by $2-3\%$. Because our aim is to study the
performance of DVS data for HAR compared to original data, we limited our
codebook size to 500 words since using higher sized codebook simply improved
both the results.

To further boost HAR rates, we separately included the MBH and HOF descriptors
along with the motion maps and trained SVM classifier in light of the
complementarity they offer. The HAR values in Table 1(a) show that the
features from motion maps better complement the second order statistics of MBH
than the first order HOF features on both the UCF11 datasets. The results also
show that the fusion of MBH and motion maps gave the highest recognition rate
among all descriptors and nearly bridged the performance gap between DVS and
conventional videos on the benchmark UCF11 data. Given the sparsity of DVS
data, it is remarkable that the final descriptor has provided a near-
equivalent performance on DVS when compared to the results on conventional
videos.

###  5.2 Recognition on our DVS gesture dataset

(a) Accuracies on original UCF11 and its DVS counterpart

(b) Accuracies on our DVS gesture dataset

Figure 6:  Accuracy plots on UCF11 and our DVS gesture dataset.

With our DVS gesture dataset also, we obtained decent recognition rates by
combining the motion maps alone, nearly same as that given by existing feature
descriptors. Combining motion maps with the MBH descriptor again augmented HAR
performance to give the highest recognition rates as seen in Table 1(b).

##  6 Conclusion and Future Work

In this paper, we have analyzed the performance of DVS data in human activity
recognition and compared it with its conventional frame-based counterpart
using traditional feature extraction techniques. We also proposed a new
encoding technique (motion maps) that is suited especially for DVS data in
light of its sparse and concise recording scheme. Combining the existing MBH
descriptor with motion maps gave the best recognition results.

Based on the feature descriptors available for its encoding, HAR results from
DVS recordings have been nearly equal to that of RGB videos on the benchmark
UCF11 data. Additional features based on the scene, texture and hue have
enabled better recognition rates with actual videos. But these are more
complex and unavailable for use with the DVS data from the very beginning.
Hence respecting the limitations that come with DVS, we conclude that within
the framework of its possible descriptors it is just as useful for HAR as
conventional videos and could efficiently be used in place of the latter,
especially in low power and high speed applications.

As future work, we can look at improving performance of simple bag-of-features
where location based relations are not preserved due to its pooling step.
Rather than destroying spatial information between the extracted features in
the image, methods like Spatial Correlogram and matching can be employed on
the DVS data. Also, we noted that similar to recognition rates in conventional
videos, dense trajectories with MBH gave the best results on using traditional
features in DVS as well. Much of the success of dense tracking comes from the
fact that it generates too many interest points given any video sample.
Visualization of the interest points found by dense sampling showed that some
of these are randomly fired noisy events in DVS unrelated to the object in
foreground. A simple median filtering pre-processing before finding dense
interest points however did not improve recognition rate. In order to truly
address the problem, a new method specifically for finding and tracking DVS
events should itself be invented. This would act as the true initial step for
improving the performance of HAR on using optical flow, MBH as well as dense
trajectories with dynamic vision sensors.

## References

  * [1] M. A. Ahad.  Motion history image.  In Motion history images for action recognition and understanding, pages 31–76. Springer, 2013.
  * [2] H. Bay, T. Tuytelaars, and L. Van Gool.  Surf: Speeded up robust features.  Computer vision–ECCV 2006, pages 404–417, 2006.
  * [3] A. F. Bobick and J. W. Davis.  The recognition of human movement using temporal templates.  IEEE Transactions on pattern analysis and machine intelligence, 23(3):257–267, 2001.
  * [4] A. Bruhn, J. Weickert, and C. Schnörr.  Lucas/kanade meets horn/schunck: Combining local and global optic flow methods.  International journal of computer vision, 61(3):211–231, 2005.
  * [5] N. Dalal, B. Triggs, and C. Schmid.  Human detection using oriented histograms of flow and appearance.  In European conference on computer vision, pages 428–441. Springer, 2006.
  * [6] R. D. De Leon and L. E. Sucar.  Human silhouette recognition with fourier descriptors.  In Pattern Recognition, 2000. Proceedings. 15th International Conference on, volume 3, pages 709–712. IEEE, 2000.
  * [7] Y. Hu, H. Liu, M. Pfeiffer, and T. Delbruck.  Dvs benchmark datasets for object tracking, action recognition, and object recognition.  Frontiers in Neuroscience, 10, 2016.
  * [8] H. Kauppinen, T. Seppanen, and M. Pietikainen.  An experimental comparison of autoregressive and fourier-based descriptors in 2d shape classification.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(2):201–207, 1995.
  * [9] P. Lichtsteiner, C. Posch, and T. Delbruck.  A 128x128 120 db 15 $\mu$ s latency asynchronous temporal contrast vision sensor.  IEEE journal of solid-state circuits, 43(2):566–576, 2008.
  * [10] J. Liu, J. Luo, and M. Shah.  Recognizing realistic actions from videos “in the wild”.  In Computer vision and pattern recognition, 2009. CVPR 2009. IEEE conference on, pages 1996–2003. IEEE, 2009.
  * [11] B. D. Lucas, T. Kanade, et al.  An iterative image registration technique with an application to stereo vision.  1981\.
  * [12] C.-Y. Ma, M.-H. Chen, Z. Kira, and G. AlRegib.  Ts-lstm and temporal-inception: Exploiting spatiotemporal dynamics for activity recognition.  arXiv preprint arXiv:1703.10667, 2017.
  * [13] F. J. Ordóñez and D. Roggen.  Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition.  Sensors, 16(1):115, 2016.
  * [14] X. Peng, L. Wang, X. Wang, and Y. Qiao.  Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice.  Computer Vision and Image Understanding, 150:109–125, 2016.
  * [15] K. Simonyan and A. Zisserman.  Two-stream convolutional networks for action recognition in videos.  In Advances in neural information processing systems, pages 568–576, 2014.
  * [16] H. Wang, A. Kläser, C. Schmid, and C.-L. Liu.  Action Recognition by Dense Trajectories.  In IEEE Conference on Computer Vision & Pattern Recognition, pages 3169–3176, Colorado Springs, United States, June 2011.
  * [17] H. Wang, A. Kläser, C. Schmid, and C.-L. Liu.  Dense trajectories and motion boundary descriptors for action recognition.  International journal of computer vision, 103(1):60–79, 2013.
  * [18] J. Yang, Y.-G. Jiang, A. G. Hauptmann, and C.-W. Ngo.  Evaluating bag-of-visual-words representations in scene classification.  In Proceedings of the international workshop on Workshop on multimedia information retrieval, pages 197–206. ACM, 2007.
  * [19] D. Zhang and G. Lu.  Review of shape representation and description techniques.  Pattern recognition, 37(1):1–19, 2004.