USTC-ICTS/PCFT-21-23

Boson-Fermion Correspondence

and Holomorphic Anomaly Equation

in 2d Yang-Mills Theory on Torus

Min-xin<EMAIL_ADDRESS>

Interdisciplinary Center for Theoretical Study,

University of Science and Technology of China, Hefei, Anhui 230026, China

Peng Huanwu Center for Fundamental Theory,

Hefei, Anhui 230026, China

Recently, Okuyama and Sakai proposed a novel holomorphic anomaly equation for
the partition function of 2d Yang-Mills theory on a torus, based on an
anholomorphic deformation of the propagator in the bosonic formulation. Using
the boson-fermion correspondence, we derive the formula for the deformed
partition function in fermionic description and give a proof of the
holomorphic anomaly equation.

###### Contents

  1. 1 Introduction
  2. 2 Bosonic Formulation
  3. 3 Fermionic Formulation
    1. 3.1 Boson-Fermion Correspondence and Feynman Rules
    2. 3.2 Deformed Partition Function
    3. 3.3 Holomorphic Anomaly Equation
  4. 4 Conclusion

##  1 Introduction

The 2d Yang-Mills theories are useful toy models for understanding the
physical properties of the more realistic 4d Yang-Mills theories. In
particular, they were studied as the early examples of the gauge/string
duality before the more famous AdS/CFT correspondence appeared, see e.g. [1,
2]. We are interested in the $U(N)$ Yang-Mills theory on a torus, whose large
$N$ expansion turns out to be equivalent to topological strings on elliptic
curves. In this case, the partition function has remarkable modular
properties, studied in the early literature, e.g. [3, 4, 5, 6, 7].

The partition function can be computed in two formulations in terms of free
fermions and bosons. This is probably one of the simplest examples of mirror
symmetry where the fermionic and bosonic formulations roughly correspond to
the topological A-model and B-model on the elliptic curve. The higher genus
mirror symmetry of the partition function including the gravitational
descendants has been mathematically proven by Li in [8]. The topological
string partition functions on Calabi-Yau three-folds satisfies the powerful
Bershadsky-Cecotti-Ooguri-Vafa (BCOV) holomorphic anomaly equations [9].
However, in the case of elliptic curve with modulus $\tau$, although the
partition function can be written in terms of quasi-modular forms, i.e.
polynomials of Eisenstein series $E_{2},E_{4},E_{6}$, the naive anholomorphic
deformation by replacing the holomorphic $E_{2}$ with the anholomorphic but
modular $\hat{E}_{2}(\tau)\equiv E_{2}(\tau)-\frac{3}{\pi{\rm Im}(\tau)}$ does
not give a nice holomorphic anomaly equation similar to the BCOV equation. See
some studies in [10]. This seems very different from most examples in Seiberg-
Witten theories and topological strings on elliptic Calabi-Yau three-folds,
where one can show that the holomorphic anomaly and modular anomaly equations
are essentially equivalent [11, 12, 13].

Okuyama and Sakai proposed an anholomorphic deformation of the propagators in
the Feynman diagrams of the bosonic formulation, based on some physical
arguments [14]. Although the partition function is not modular with this
deformation, a nice BCOV-like holomorphic anomaly equation can be obtained. In
this paper, we derive the corresponding formula in fermionic formulation,
which was also already conjectured in [14]. It is then easy to prove the
conjectured holomorphic anomaly equation. We will use the well-known boson-
fermion correspondence, where some nice expositions are available in e.g. [15,
8]. A classic introduction to the theory of (quasi)-modular forms is [16].

The paper is organized as the followings. In Sec. 2 we review the formulation
of holomorphic anomaly equation in [14]. In Sec. 3, after briefly reviewing
the boson-fermion correspondence, we derive the formula for the deformed
partition function in the fermionic formulation. It is then straightforward to
prove the holomorphic anomaly equation. in Sec. 4 we conclude with some
potential future directions.

##  2 Bosonic Formulation

The partition function can be expressed by the path integral of a compactified
boson field $\varphi$ on the torus with a cubic interaction

$\mathcal{Z}=\int\mathcal{D}\varphi\exp[\int_{T^{2}}(\bar{\partial}\varphi\partial\varphi+\frac{g_{s}}{6}(\partial\varphi)^{3})].$
(2.1)

We can expand the cubic interaction for small $g_{s}$ and compute by Feynman
diagrams. The propagator is given by the two-point function on torus, which is
basally determined by the double periodicity and singularities, in terms of
the Weierstrass’s elliptic function and Eisenstein series

$\langle\partial\varphi(z_{1})\partial\varphi(z_{2})\rangle=-\wp(z_{1}-z_{2})-\frac{E_{2}}{12}+S,$
(2.2)

where $S$ is an anholomorphic parameter relating to the imaginary part of the
torus modulus

$S:=\frac{1}{4\pi{\rm Im}(\tau)}.$ (2.3)

The appearance of $S$ depends on whether one takes into account of the winding
mode contributions [3, 17]. The partition function would be entirely
holomorphic if one omits the $S$-dependence in the propagator. Here we follow
the prescription in [14] of keeping the $S$-dependence, so that a BCOV-like
holomorphic anomaly equation can be obtained.

The partition function can be expanded in terms of free energy

$\log(\mathcal{Z})=\mathcal{F}=\sum_{g=1}^{\infty}g_{s}^{2g-2}\mathcal{F}_{g}.$
(2.4)

This corresponds to the large $N$ expansion of 2d $U(N)$ gauge theory where
the genus $g$ appears in the powers of $N$. The genus one term, which is
independent of the cubic coupling constant $g_{s}$, comes from the
normalization of the path integral with the quadratic action. It is convenient
to also include the anholomorphic dependence so that it is modular

$\mathcal{F}_{1}=\frac{1}{2}\log(S)-\log\eta(\tau),$ (2.5)

where $\eta(\tau)=Q^{\frac{1}{24}}\prod_{n=1}^{\infty}(1-Q^{n})$ with
$Q:=e^{2\pi i\tau}$ is the well-known Dedekind eta function. The shift of the
genus one free energy by an additive constant will not affect our calculations
here.

The higher genus free energy $\mathcal{F}_{g}$ is then computed by summing
over all connected Feynman diagrams with $2g-2$ cubic vertices and $3g-3$
propagators. The propagator (2.2) needs to be integrated over the torus, and
for exploring the relation with fermionic formulation, it is convenient to
express it as a power series of $Q$ [18]. The contribution of a Feynman
diagram $\Gamma$ is

$I_{\Gamma}=\oint\prod_{i=1}^{2g-2}\frac{dx_{i}}{2\pi
ix_{i}}\prod_{k=1}^{3g-3}G(x^{+}_{k}/x^{-}_{k}),$ (2.6)

where the propagator is given by

$G(x)=\sum_{n=1}^{\infty}\frac{n(x^{n}+x^{-n}Q^{n})}{1-Q^{n}}+S.$ (2.7)

Each vertex is associated with a $x_{i}$ integral. The $x_{\pm}$ in the
integral correspond to the two vertices connected by the propagator. Notice
the propagator is not invariant under the transformation
$x\rightarrow\frac{1}{x}$. A prescription is to assign an order to the $2g-2$
vertices, then for the propagator connecting different vertices, one can
always choose $x_{+}$ to be the vertex with the higher order. For the
propagator connecting two edges in the same vertex, a zeta-function
regularization $\zeta(-1)=-\frac{1}{12}$ is needed to compute the expression

$G(1)=\sum_{n=1}^{\infty}n+\sum_{n=1}^{\infty}\frac{2nQ^{n}}{1-Q^{n}}+S=-\frac{E_{2}}{12}+S.$
(2.8)

It is familiar in Feynman diagram calculations that the same Feynman rules
also apply to the computations of partition functions by simply including the
disconnected Feynman diagrams.

As an example, the genus two free energy has contributions from two Feynman
diagrams. The result is computed in [14] as

$\mathcal{F}_{2}=\frac{5}{24}S^{3}-\frac{E_{2}}{48}S^{2}-\frac{E_{2}^{2}-2E_{4}}{1152}S+\frac{5E_{2}^{3}-3E_{2}E_{4}-2E_{6}}{51840}.$
(2.9)

One can check that the $E_{2}$ and $S$ can not be completely combined into the
modular $\hat{E}_{2}$. However, there is a nice holomorphic anomaly equation.
One introduces the derivative $D:=Q\partial_{Q}=\frac{1}{2\pi
i}\partial_{\tau}$, so we have

$DS=S^{2},~{}~{}~{}DQ^{n}=nQ^{n}.$ (2.10)

The derivative of genus one free energy is

$D\mathcal{F}_{1}=\frac{1}{2}S-\frac{E_{2}}{24}.$ (2.11)

The anholomorphic dependences in the free energy and partition function come
entirely from the $S$ parameter. It is then easy to check the following genus
two holomorphic anomaly equation

$2\partial_{S}\mathcal{F}_{2}=(D+S)D\mathcal{F}_{1}+(D\mathcal{F}_{1})^{2}.$
(2.12)

Some higher order free energies were calculated in [14] and similar
holomorphic anomaly equations were checked. It is most convenient to organize
the holomorphic anomaly equation in terms of the partition function as

$(2\partial_{S}-S^{-1})\mathcal{Z}=g_{s}^{2}(D+S)D\mathcal{Z}.$ (2.13)

This looks tantalizingly similar to the celebrated BCOV equation.

##  3 Fermionic Formulation

###  3.1 Boson-Fermion Correspondence and Feynman Rules

We briefly review the boson-fermion correspondence, following mostly the
notation in [8]. The system of free fermions is described by two sets of
Grassmann operators with half integer indices, satisfying the following anti-
commutation relations

$\\{b_{n},c_{m}\\}=\delta_{m+n,0},~{}~{}\\{b_{n},b_{m}\\}=\\{c_{n},c_{m}\\}=0,~{}~{}~{}m,n\in\mathbb{Z}+\frac{1}{2}.$
(3.1)

The operators with positive and negative indices are regarded as annihilation
and creation operators. The vacuum state $|0\rangle$ is annihilated by all
annihilation operators

$b_{n}|0\rangle=c_{n}|0\rangle=0,~{}~{}~{}n>0.$ (3.2)

One can then construct the linear basis of the Fock space by acting the
creation operators on the vacuum. The dual states can be defined with the dual
vacuum $\langle 0|$ annihilated by negative mode operators from the left. The
dual state of $b_{-n}|0\rangle$ is $\langle 0|c_{n}$ for $n>0$ so that the
inner product is normalized to be one. The charge of a state is defined to be
the difference of the numbers of $b,c$ creation operators acting on the
vacuum. The Hilbert space is decomposed into the charged sectors as
$H=\oplus_{p\in\mathbb{Z}}H_{p}$. We will be interested in the zero-charge
sector $H_{0}$, which are constructed by acting the same number of $b$ and $c$
creation operators on the vacuum.

It is convenient to define the fermionic fields which are generating functions
of the fermionic operators

$b(z)=\sum_{n\in\mathbb{Z}+\frac{1}{2}}b_{n}z^{-n-\frac{1}{2}},~{}~{}~{}c(z)=\sum_{n\in\mathbb{Z}+\frac{1}{2}}c_{n}z^{-n-\frac{1}{2}}.$
(3.3)

The duality of states can be conveniently denoted as
$b(z)^{\dagger}=\bar{z}^{-1}c(1/\bar{z})$. We have the following relation,
appeared in the well-known operator product expansion (OPE) in conformal field
theory

$b(z)c(w)=\frac{1}{z-w}+:b(z)c(w):,~{}~{}~{}~{}{\rm for}~{}|z|>|w|,$ (3.4)

where $::$ is the normal ordering which moves the annihilation operator to
right, with sign for the fermionic case.

The system of free bosons can be constructed from the above fermionic system
by

$\alpha(z)=\sum_{n\in\mathbb{Z}}\alpha_{n}z^{-n-1}=:b(z)c(z):.$ (3.5)

The bosonic mode operators can be expressed in terms of fermions

$\alpha_{n}=\sum_{k\in\mathbb{Z}+\frac{1}{2}}:b_{k}c_{n-k}:.$ (3.6)

They form a set of infinite harmonic oscillators with the commutation
relations

$[\alpha_{m},\alpha_{n}]=m\delta_{m+n,0},~{}~{}~{}m,n\in\mathbb{Z}.$ (3.7)

Again we identify the bosonic operators with positive and negative indices as
annihilation and creation operators, while for the zero mode operator
$\alpha_{0}$, it is easy to see that a fermionic state with charge $p$ is an
eigenstate of with eigenvalue $p$. For each sector $H_{p}$, there is a unique
fermionic state which is created by the lowest fermionic creation operators
and is annihilated by all bosonic annihilation operators. We can then
construct the bosonic Fock space by acting the creation operators on the
vacuum state in each sector $H_{p}$. In this way there is an isomorphism of
the bosonic and fermionic Hilbert spaces.

One defines another bosonic field $\phi(z)$ by

$\phi(z)=P+\alpha_{0}\log z+\sum_{n\neq 0}\frac{\alpha_{-n}}{n}z^{n},$ (3.8)

which is related to the bosonic field in (3.5) by
$\alpha(z)=\partial_{z}\phi(z)$. The momentum operator $P$ is the conjugate of
the zero mode operator $[\alpha_{0},P]=1$. Its exponential $e^{P}$ increases
the charge of a state by one. Since we will be interested only in the zero
charge sector, the operators $P,\alpha_{0}$ will not be important in our
context. It is straightforward to check the relation, which is familiar from
OPE

$\phi(z)\phi(w)=\log(z-w)+:\phi(z)\phi(w):,~{}~{}~{}~{}{\rm for}~{}|z|>|w|,$
(3.9)

where $::$ is the bosonic normal ordering.

The theorem of boson-fermion correspondence [15] states that the fermionic
fields can be expressed

$b(z)=:e^{\phi(z)}:,~{}~{}~{}c(z)=:e^{-\phi(z)}:,$ (3.10)

It is also familiar in CFT computations that

$:e^{\phi(z)}::e^{-\phi(w)}:=\frac{1}{z-w}:e^{\phi(z)-\phi(w)}:.$ (3.11)

Together with the fermionic formula (3.4), we arrive at the following useful
formula

$\displaystyle\mathcal{E}(z,\lambda)$
$\displaystyle\equiv\frac{1}{(e^{\lambda/2}-e^{-\lambda/2})z}:e^{\phi(e^{\lambda/2}z)-\phi(e^{-\lambda/2}z)}:$
(3.12)
$\displaystyle=:b(e^{\lambda/2}z)c(e^{-\lambda/2}z):+\frac{1}{(e^{\lambda/2}-e^{-\lambda/2})z}.$

The exponent in bosonic expression can be written in terms of mode expansion

$\phi(e^{\lambda/2}z)-\phi(e^{-\lambda/2}z)=\lambda\alpha_{0}+\sum_{n\neq
0}\frac{\alpha_{-n}}{n}z^{n}(e^{n\lambda/2}-e^{-n\lambda/2}).$ (3.13)

One can then expand the above formula (3.12) as Laurent series of $\lambda,z$
and compare the coefficients. In this way, many non-trivial identities between
the bosonic and fermionic mode operators can be obtained. We will focus only
on the $z^{-1}$ coefficient, which does not change the energy of a state. The
formula for fermionic modes is

$\oint\frac{dz}{2\pi
i}\mathcal{E}(z,\lambda)=\sum_{k\in\mathbb{Z}+\frac{1}{2}}e^{\lambda
k}:b_{-k}c_{k}:+\frac{1}{e^{\lambda/2}-e^{-\lambda/2}}.$ (3.14)

Comparing the coefficients of $\lambda z^{-1}$ in the formula (3.12) gives the
the zero mode Virasoro operator, or the energy

$L_{0}=\frac{1}{2}\alpha_{0}^{2}+\sum_{n=1}^{\infty}\alpha_{-n}\alpha_{n}=\sum_{k\in\mathbb{Z}^{\geq
0}+1/2}k(b_{-k}c_{k}+c_{-k}b_{k}).$ (3.15)

This familiar relation can be also directly derived using (3.6) after a
careful calculation with the anti-commutation relations. The generating
function with trace over the zero charge sector is related to the Dedekind eta
function

$\textrm{Tr}_{H_{0}}Q^{L_{0}-\frac{1}{24}}=\eta(\tau)^{-1},$ (3.16)

which can be computed by both bosonic and fermionic formulas in (3.15), whose
equivalence is implied by the classic Jacobi triple identity. The other
Virasoro operators can be also defined in terms of both bosons and fermions
but will not be needed in our context.

For our purpose, we will also compare the coefficients of $\lambda^{2}z^{-1}$.
In this case the bosonic exponential function in (3.12) needs to be expanded
to the cubic order. Since we will only consider the zero charge sector, we can
omit the $\alpha_{0}$ in the formulas. We denote the operator

$\displaystyle\mathcal{O}$ $\displaystyle\equiv\frac{1}{6}\oint\frac{dz}{2\pi
iz}:(\sum_{n\neq 0}\alpha_{-n}z^{n})^{3}:$ (3.17)
$\displaystyle=\sum_{k\in\mathbb{Z}+\frac{1}{2}}\frac{k^{2}}{2}:b_{-k}c_{k}:.$

The trace of $Q^{L_{0}-\frac{1}{24}}\mathcal{O}^{2g-2}$ over the zero charge
sector $H_{0}$ gives the $g_{s}^{2g-2}$ order contributions to the partition
function (2.1) without the $S$ deformation. This can be computed according to
the Wick theorem by contracting the bosonic mode operators, including the
disconnected diagrams. Usually the Wick contraction for a bosonic harmonic
oscillator is between a creation and annihilation operator with the creation
operator on the right. We are using an “effective Wick contraction” with
trace, where the formula for two modes is computed as

$\displaystyle\textrm{Tr}_{H_{0}}(Q^{L_{0}-\frac{1}{24}}\alpha_{m}\alpha_{n})$
$\displaystyle=\delta_{m+n,0}\eta(\tau)^{-1}(1-Q^{n})\sum_{k=0}^{\infty}nkQ^{nk}$
(3.18) $\displaystyle=\delta_{m+n,0}\eta(\tau)^{-1}\frac{nQ^{n}}{1-Q^{n}},$
$\displaystyle\text{for }n\geq 0.$

While the case of $n<0$ can be obtained by the canonical commutation relation
with an extra term

$\displaystyle\textrm{Tr}_{H_{0}}(Q^{L_{0}-\frac{1}{24}}\alpha_{m}\alpha_{n})$
$\displaystyle=\delta_{m+n,0}\eta(\tau)^{-1}(1-Q^{n})\sum_{k=0}^{\infty}n(k+1)Q^{nk}$
(3.19) $\displaystyle=\delta_{m+n,0}\eta(\tau)^{-1}\frac{m}{1-Q^{m}},$
$\displaystyle\text{for }n<0.$

It is then straightforward to compute

$\displaystyle~{}~{}\textrm{Tr}_{H_{0}}(Q^{L_{0}-\frac{1}{24}}\sum_{m,n}\alpha_{m}\alpha_{n}x_{-}^{-m}x_{+}^{-n})=\eta(\tau)^{-1}\sum_{n=1}^{\infty}\frac{n[(\frac{x_{+}}{x_{-}})^{n}+(\frac{x_{-}}{x_{+}})^{n}Q^{n}]}{1-Q^{n}},$
(3.20)

which matches with the propagator (2.7) without the $S$-deformation, while the
factor $\eta(\tau)^{-1}$ accounts for the contribution of genus one free
energy (2.5). The effective Wick contraction between two edges from the same
vertex does not contribute in this case since we restrict to the zero charge
sector.

The calculations (LABEL:compute3.17) can be independently done for all
possible effective Wick contractions of the edges of the cubic vertices in
(3.17), and there is only one overall factor of $\eta(\tau)^{-1}$ for a
Feynman diagram. To illustrate, let us for simplicity consider the case of
$\alpha_{-1},\alpha_{1}$ operators. Focusing on the sector of Hilbert space
created by $\alpha_{-1}$, we denote
$|k\rangle=\frac{(\alpha_{-1})^{k}}{\sqrt{k!}}|0\rangle$ the properly
normalized $k$-particle state. Then we compute schematically

$\displaystyle\sum_{k=0}^{\infty}\langle\wick{k|Q^{\alpha_{-1}\alpha_{1}}\cdots\c{1}\alpha_{-1}\cdots\c{1}\alpha_{1}\cdots|k}\rangle=\sum_{k=0}^{\infty}k\langle
k|Q^{\alpha_{-1}\alpha_{1}}\cdots|k\rangle=Q\partial_{Q}\sum_{k=0}^{\infty}\langle
k|Q^{\alpha_{-1}\alpha_{1}}\cdots|k\rangle.$ (3.21)

The calculation can be understood as first performing an implicit “usual Wick
contraction” of $\alpha_{1}$ with the creation operators in $|k\rangle$, then
we use the $\alpha_{-1}$ to fill in for the creation operator. So it looks
effectively we are contracting $\alpha_{-1},\alpha_{1}$ as displayed in the
above expression. After all contractions are done, the trace is simply
$\sum_{k=0}^{\infty}\langle k|Q^{\alpha_{-1}\alpha_{1}}|k\rangle=(1-Q)^{-1}$.
So the effect of the $\alpha_{-1},\alpha_{1}$ contraction can be derived as
$(1-Q)Q\partial_{Q}(1-Q)^{-1}=\frac{Q}{1-Q}$, consistent with (3.18) for
$n=1$, independently of the other possible operators denoted as $\cdots$ in
the above expression. Similarly, the effective Wick contraction of
$\alpha_{1},\alpha_{-1}$ with $\alpha_{-1}$ on the right gives a factor of
$\frac{1}{1-Q}$.

As a check of the somewhat tricky reasonings, we can compute directly a simple
example

$\sum_{k=0}^{\infty}\langle
k|Q^{\alpha_{-1}\alpha_{1}}(\alpha_{-1}\alpha_{1})^{2}|k\rangle=\sum_{k=0}^{\infty}k^{2}Q^{k}=\frac{Q(1+Q)}{(1-Q)^{3}}.$
(3.22)

On the other hand, we can compute by the effective Wick contractions as

$\displaystyle\sum_{k=0}^{\infty}\langle
k|Q^{\alpha_{-1}\alpha_{1}}(\wick{\c{1}\alpha_{-1}\c{1}\alpha_{1}}\wick{\c{1}\alpha_{-1}\c{1}\alpha_{1}}+\wick{\c{1}\alpha_{-1}\c{2}\alpha_{1}\c{2}\alpha_{-1}\c{1}\alpha_{1}}|k\rangle$
(3.23) $\displaystyle=(1-Q)^{-1}[\frac{Q^{2}}{(1-Q)^{2}}+\frac{Q}{(1-Q)^{2}}]$
$\displaystyle=\frac{Q(1+Q)}{(1-Q)^{3}},$

which gives the same result.

So the effective Wick contractions give the same Feynman rules described in
the previous Sec. 2 without the $S$-deformation. The paper [8] considers the
much more general cases with insertions of descendant operators in the
correlation functions in the mathematical definition of Gromov-Witten
invariants of elliptic curves. Here we focus on the special case where the
first Chern class of the line bundles of marked points in the Riemann surface
have degree one. In this case the genus $g$ contribution must have $2g-2$
insertions, enforced by the selection rule due to the dimension of the virtual
fundamental class of the moduli space. This corresponds to the $2g-2$ cubic
vertices in the Feynman diagrams.

The computations of the trace of $Q^{L_{0}-\frac{1}{24}}\mathcal{O}^{2g-2}$
with the fermionic operators in the formula (3.17) give the free fermion
formulation of the partition function (without the $S$-deformation), well
studied in e.g. [3, 5]. We will give the detailed analysis for the case with
$S$-deformation in the next subsection 3.2.

###  3.2 Deformed Partition Function

To produce the $S$-dependence in the propagator (2.7), we introduce an
auxiliary parameter $a$. It is easy to compute the Gaussian integrals

$\int_{-\infty}^{\infty}\frac{e^{-\frac{a^{2}}{2}}da}{\sqrt{2\pi}}a^{2n}=(2n-1)!!,~{}~{}~{}n\in\mathbb{N}.$
(3.24)

The formula can be understood in terms of Feynman rule as the number of Wick
contractions between $a$’s.

We multiply the formula (3.12) by a factor $e^{\lambda a\sqrt{S}}$, and again
compare the coefficients of $\lambda^{2}z^{-1}$ in the Laurent expansion. We
denote the operator as

$\displaystyle\mathcal{O}(S)$
$\displaystyle\equiv-\frac{1}{24}aS^{\frac{1}{2}}+\frac{1}{6}\oint\frac{dz}{2\pi
iz}:(aS^{\frac{1}{2}}+\sum_{n\neq 0}\alpha_{-n}z^{n})^{3}:$ (3.25)
$\displaystyle=\sum_{k\in\mathbb{Z}+\frac{1}{2}}\frac{(k+a\sqrt{S})^{2}}{2}:b_{-k}c_{k}:+\frac{1}{6}a^{3}S^{\frac{3}{2}}-\frac{1}{24}aS^{\frac{1}{2}}.$

Performing the integral
$\int_{-\infty}^{\infty}\frac{e^{-\frac{a^{2}}{2}}da}{\sqrt{2\pi}}$ on the
auxiliary parameter $a$ will enforce a Wick contraction between
$S^{\frac{1}{2}}$’s in the cubic vertex, giving rise the deformed propagator
(2.7) with the $S$ term. Our prescription has an extra term
$-\frac{1}{24}aS^{\frac{1}{2}}$ besides the cubic vertex. It accounts for the
normal ordering constant when two edges from the same vertex are Wick
contracted. In this case, we can calculate

$\displaystyle~{}~{}\textrm{Tr}_{H_{0}}(Q^{L_{0}-\frac{1}{24}}\sum_{m,n}:\alpha_{m}\alpha_{n}:z^{-m-n})=\eta(\tau)^{-1}\sum_{n=1}^{\infty}\frac{2nQ^{n}}{1-Q^{n}}.$
(3.26)

Comparing with (LABEL:compute3.17) for $x_{+}=x_{-}$ or (2.8), there is a
difference with normal ordering constant $\sum_{n=1}^{\infty}n=-\frac{1}{12}$.
This is exactly compensated by the extra term $-\frac{1}{24}aS^{\frac{1}{2}}$,
since $3\times\frac{1}{6}(-\frac{1}{12})=-\frac{1}{24}$.

Now using the fermionic formula in (3.25), we can finally write the deformed
partition function (2.1) in terms of summation over 2d Young tableaux

$\displaystyle\mathcal{Z}$
$\displaystyle=\sqrt{S}\int_{-\infty}^{\infty}\frac{e^{-\frac{a^{2}}{2}}da}{\sqrt{2\pi}}\sum_{g=1}^{\infty}\frac{g_{s}^{2g-2}}{(2g-2)!}\textrm{Tr}_{H_{0}}[Q^{L_{0}-\frac{1}{24}}\mathcal{O}(S)^{2g-2}]$
(3.27)
$\displaystyle=\sqrt{S}\int_{-\infty}^{\infty}\frac{e^{-\frac{a^{2}}{2}}da}{\sqrt{2\pi}}\oint\frac{dx}{2\pi
ix}Q^{-\frac{1}{24}}e^{g_{s}(\frac{1}{6}a^{3}S^{\frac{3}{2}}-\frac{1}{24}aS^{\frac{1}{2}})}$
$\displaystyle\times\prod_{p\in\mathbb{Z}^{\geq
0}+\frac{1}{2}}[1+xQ^{p}e^{g_{s}(\frac{p^{2}}{2}+pa\sqrt{S})}][1+x^{-1}Q^{p}e^{g_{s}(-\frac{p^{2}}{2}+pa\sqrt{S})}]$
$\displaystyle=\sqrt{S}\int_{-\infty}^{\infty}\frac{e^{-\frac{a^{2}}{2}}da}{\sqrt{2\pi}}\sum_{R}Q^{|R|-\frac{1}{24}}e^{g_{s}\sum_{i}R_{i}(R_{i}-2i+1)}e^{g_{s}[\frac{1}{6}a^{3}S^{\frac{3}{2}}+(|R|-\frac{1}{24})aS^{\frac{1}{2}}]}.$

Some explanations are in orders. We have include a factor of $\sqrt{S}$ to
match the convention of genus one free energy in (2.5). The last expression
sums over all 2d Young tableaux $R$, labelled by a sequence of non-increasing
positive integers $R_{1}\geq R_{2}\geq\cdots$. The number of boxes is denoted
$|R|=\sum_{i}R_{i}$. It is well known that a 2d Young tableau has a Frobenius
representation in terms of two sets of strictly decreasing positive half
integers as
$(p_{1},p_{2},\cdots,p_{n}|\tilde{p}_{1},\tilde{p}_{2},\cdots,\tilde{p}_{n})$.
The map can be obtained by dissecting the Young tableau through its diagonal
elements. It is obvious that $|R|=\sum_{i=1}^{n}(p_{i}+\tilde{p}_{i})$. There
is an isomorphism with the zero charge Hilbert space $H_{0}$, where the rows
of the Young tableau corresponds to the bosonic creation operators, while the
Frobenius representation gives the fermionic Fock space. For example, the
Frobenius representation
$(p_{1},p_{2},\cdots,p_{n}|\tilde{p}_{1},\tilde{p}_{2},\cdots,\tilde{p}_{n})$
corresponds to the state

$|\psi\rangle=\prod_{i=1}^{n}b_{-p_{i}}c_{-\tilde{p}_{i}}|0\rangle.$ (3.28)

In the second equality in (3.27) we have used the fact that the fermionic
state corresponding to the Frobenius representation is an eigenstate of the
fermionic operator in (3.25). In fact, for the state (3.28) it is easy to see

$\displaystyle\sum_{k\in\mathbb{Z}+\frac{1}{2}}f_{k}:b_{-k}c_{k}:|\psi\rangle=\sum_{j=1}^{n}(f_{p_{j}}-f_{-\tilde{p}_{j}})|\psi\rangle,$
(3.29)

for any coefficients $f_{k}$’s.

The computation in (3.27) also uses the following formula of the Frobenius
representation. For a 2d Young tableau $R$, we put the numbers
$0,1,\cdots,R_{1}-1$ in the first row, then $-1,0,1,\cdots,R_{2}-2$ in the
second row, etc. The diagonal elements in the Frobenius partition are always
0. Suppose the Frobenius representation is
$(p_{1},p_{2},\cdots,p_{n}|\tilde{p}_{1},\tilde{p}_{2},\cdots,\tilde{p}_{n})$,
we can sum over the numbers in the boxes in two ways to easily prove

$\sum_{i}p_{i}^{2}-\sum_{i}\tilde{p}_{i}^{2}=\sum_{i}R_{i}(R_{i}-2i+1).$
(3.30)

Since the Frobenius representation of the transposition of $R$ is simply
obtained by switching the two sets of half integers, the half integer powers
of $S$ always cancel between a Young tableau and its transposition in the
formula (3.27). So it is clear that the free energies are polynomials of $S$.
It is also obvious that the odd $g_{s}$ power terms vanish by a trivial change
of the integration variable $a\rightarrow-a$.

We can perform some simple checks of the formula (3.27). For example, we can
expand to $g_{s}^{2}$ and compare with the $S^{2},S^{3}$ terms in the genus
two formula (2.9). We find consistent results

$\displaystyle\mathcal{Z}$
$\displaystyle=\sqrt{S}\int_{-\infty}^{\infty}\frac{e^{-\frac{a^{2}}{2}}da}{\sqrt{2\pi}}\sum_{R}Q^{|R|-\frac{1}{24}}\\{1+g_{s}^{2}[\frac{a^{6}S^{3}}{72}+(|R|-\frac{1}{24})\frac{a^{4}S^{2}}{6}+\cdots]\\}+\mathcal{O}(g_{s}^{4})$
(3.31)
$\displaystyle=\sqrt{S}\eta(\tau)^{-1}[1+g_{s}^{2}(\frac{5}{24}S^{3}-\frac{E_{2}}{48}S^{2}+\cdots)+\mathcal{O}(g_{s}^{4})].$

We note that the fermionic formula (3.27) was already essentially also
conjectured by Okuyama and Sakai [14] in a somewhat different notation. Here
we have given a proof using the boson-fermion correspondence formula (3.25).

###  3.3 Holomorphic Anomaly Equation

As in [14], it is now straightforward to analyze the holomorphic anomaly
equation (2.13) with the fermionic formula (3.27). Here we perform a more
direct computation using the differential rules (2.10

$\displaystyle~{}~{}\left[-2\partial_{S}+S^{-1}+g_{s}^{2}(D+S)D\right]\mathcal{Z}$
(3.32)
$\displaystyle=\sum_{R}(|R|-\frac{1}{24})^{-\frac{1}{2}}Q^{|R|-\frac{1}{24}}e^{g_{s}\sum_{i}R_{i}(R_{i}-2i+1)}$
$\displaystyle\times\int_{-\infty}^{\infty}\frac{e^{-\frac{a^{2}}{2}}da}{\sqrt{2\pi}}e^{\tilde{g}_{s}(\frac{1}{6}a^{3}\tilde{S}^{\frac{3}{2}}+a\tilde{S}^{\frac{1}{2}})}\\{-\tilde{g}_{s}a(1+\frac{1}{2}\tilde{S}a^{2})+\tilde{g}_{s}^{2}\tilde{S}^{\frac{1}{2}}(1+2\tilde{S}+\frac{5}{4}\tilde{S}^{2})$
$\displaystyle+\tilde{g}_{s}^{3}\tilde{S}^{2}[a+\tilde{S}(\frac{7}{4}a+\frac{1}{2}a^{3})+\frac{9}{8}\tilde{S}^{2}a^{3}]+\frac{1}{16}\tilde{g}_{s}^{4}\tilde{S}^{\frac{7}{2}}a^{2}(2+\tilde{S}a^{2})^{2}\\},$

where for convenience we have changed variables

$S=\tilde{S}(|R|-\frac{1}{24}),~{}~{}~{}g_{s}=\tilde{g}_{s}(|R|-\frac{1}{24})^{-\frac{3}{2}}.$
(3.33)

We expect that the integrand in the last two lines in the formula
(LABEL:formula3.24) can be written as a total derivative, so that the integral
always vanishes. It is not difficult to make an ansatz and find the solution

$\partial_{a}\\{\frac{\tilde{g}_{s}}{4}e^{-\frac{a^{2}}{2}+\tilde{g}_{s}(\frac{1}{6}a^{3}\tilde{S}^{\frac{3}{2}}+a\tilde{S}^{\frac{1}{2}})}[4(1+\tilde{S})+\tilde{g}_{s}\tilde{S}^{\frac{3}{2}}(4+5\tilde{S})a+\tilde{S}(2+\tilde{g}_{s}^{2}\tilde{S}^{2})a^{2}+\tilde{g}_{s}\tilde{S}^{\frac{5}{2}}a^{3}+\frac{1}{2}\tilde{g}_{s}^{2}\tilde{S}^{4}a^{4}]\\}.$
(3.34)

Of course there seem to be some potential divergences in the integral in
(LABEL:formula3.24) as $a\rightarrow\pm\infty$ due to the cubic term $a^{3}$
in the exponential. However it is at least completely well defined as a formal
power series of $g_{s}$, as mentioned in [14], and the fact that the integrand
is a total derivative is sufficient to ensure the vanishing of the formal
power series.

##  4 Conclusion

We have derived the fermionic formula (3.27) for the deformed partition
function of 2d Yang-Mills theory on a torus, and prove the conjectured
holomorphic anomaly equation (2.13). It may be interesting to consider further
possible generalizations of the holomorphic anomaly equation, e.g. including
the gravitational descendants [8], with $q$-deformation [19]. It would be also
interesting to explore whether the techniques used here have some interesting
applications in more sophisticated situations, e.g. for topological strings on
Calabi-Yau three-folds.

As we mentioned, although it is justified by some physical arguments, Okuyama
and Sakai’s anholomorphic deformation [14] does not form a modular combination
with the quasi-modular forms. On the other hand, it appears that in the
proposed mathematical definitions of BCOV-like theories associated with
elliptic curves, e.g. [8, 20, 21], the partition functions are always modular.
It would be interesting to explore whether there are more useful anholomorphic
deformations in more general settings.

Acknowledgments

We thank Bao-ning Du, Jun-Hao Li, Gao-fu Ren, Pei-xuan Zeng for helpful
discussions, and Sheldon Katz, Albrech Klemm, Yuji Sugimoto, Xin Wang for
collaborations on related topics. This work was supported in parts by the
national Natural Science Foundation of China (Grants No.11947301 and
No.12047502).

## References

  * [1] E. Witten, “Two-dimensional gauge theories revisited,” J. Geom. Phys. 9 (1992) 303–368, arXiv:hep-th/9204083.
  * [2] D. J. Gross and W. Taylor, “Two-dimensional QCD is a string theory,” Nucl. Phys. B 400 (1993) 181–208, arXiv:hep-th/9301068.
  * [3] M. R. Douglas, “Conformal field theory techniques in large N Yang-Mills theory,” in NATO Advanced Research Workshop on New Developments in String Theory, Conformal Models and Topological Field Theory.  5, 1993.  arXiv:hep-th/9311130.
  * [4] R. Dijkgraaf, “Mirror symmetry and elliptic curves,” in The Moduli Space of Curves, R. H. Dijkgraaf, C. F. Faber, and G. B. M. van der Geer, eds., pp. 149–163.  Birkhäuser Boston, Boston, MA, 1995.
  * [5] M. Kaneko and D. Zagier, “A generalized jacobi theta function and quasimodular forms,” in The Moduli Space of Curves, R. H. Dijkgraaf, C. F. Faber, and G. B. M. van der Geer, eds., pp. 165–172.  Birkhäuser Boston, Boston, MA, 1995.
  * [6] R. E. Rudd, “The String partition function for QCD on the torus,” arXiv:hep-th/9407176.
  * [7] S. Cordes, G. W. Moore, and S. Ramgoolam, “Lectures on 2-d Yang-Mills theory, equivariant cohomology and topological field theories,” Nucl. Phys. B Proc. Suppl. 41 (1995) 184–244, arXiv:hep-th/9411210.
  * [8] S. Li, “BCOV theory on the elliptic curve and higher genus mirror symmetry,” arXiv:1112.4063 [math.QA].
  * [9] M. Bershadsky, S. Cecotti, H. Ooguri, and C. Vafa, “Kodaira-Spencer theory of gravity and exact results for quantum string amplitudes,” Commun. Math. Phys. 165 (1994) 311–428, arXiv:hep-th/9309140.
  * [10] R. Dijkgraaf, “Chiral deformations of conformal field theories,” Nucl. Phys. B 493 (1997) 588–612, arXiv:hep-th/9609022.
  * [11] M.-x. Huang and A. Klemm, “Holomorphic Anomaly in Gauge Theories and Matrix Models,” JHEP 09 (2007) 054, arXiv:hep-th/0605195.
  * [12] M.-x. Huang, “Modular anomaly from holomorphic anomaly in mass deformed $\mathcal{N}=2$ superconformal field theories,” Phys. Rev. D 87 no. 10, (2013) 105010, arXiv:1302.6095 [hep-th].
  * [13] M.-X. Huang, S. Katz, and A. Klemm, “Towards refining the topological strings on compact Calabi-Yau 3-folds,” JHEP 03 (2021) 266, arXiv:2010.02910 [hep-th].
  * [14] K. Okuyama and K. Sakai, “Holomorphic anomaly of 2d Yang-Mills theory on a torus revisited,” JHEP 08 (2019) 025, arXiv:1903.10510 [hep-th].
  * [15] T. Miwa, E. Date, and M. Jimbo, Solitons: Differential Equations, Symmetries and Infinite Dimensional Algebras.  Cambridge Tracts in Mathematics.
  * [16] D. Zagier, “Elliptic modular forms and their applications,” in The 1-2-3 of modular forms, Universitext, pp. 1–103.  Springer, Berlin, 2008.  https://doi.org/10.1007/978-3-540-74119-0_1.
  * [17] S. Datta, J. R. David, and S. P. Kumar, “Conformal perturbation theory and higher spin entanglement entropy on the torus,” JHEP 04 (2015) 041, arXiv:1412.3946 [hep-th].
  * [18] J. Böhm, K. Bringmann, A. Buchholz, and H. Markwig, “Tropical mirror symmetry for elliptic curves,” Journal für die reine und angewandte Mathematik 2017 no. 732, (2017) 211–246, arXiv:1309.5893 [math.AG].
  * [19] M. Aganagic, H. Ooguri, N. Saulina, and C. Vafa, “Black holes, q-deformed 2d Yang-Mills, and non-perturbative topological strings,” Nucl. Phys. B 715 (2005) 304–348, arXiv:hep-th/0411280.
  * [20] K. J. Costello and S. Li, “Quantum BCOV theory on Calabi-Yau manifolds and the higher genus B-model,” arXiv:1201.4501 [math.QA].
  * [21] S. Li and J. Zhou, “Regularized Integrals on Riemann Surfaces and Modular Forms,” arXiv:2008.07503 [math.DG].


# Privacy-Preserving Action Recognition using Coded Aperture Videos

Zihao W. Wang1 Vibhav Vineet2 Francesco Pittaluga3
Sudipta N. Sinha2 Oliver Cossairt1 Sing Bing Kang4
1Northwestern University 2 Microsoft Research 3 University of Florida 4Zillow
Group

###### Abstract

The risk of unauthorized remote access of streaming video from networked
cameras underlines the need for stronger privacy safeguards. We propose a
lens-free coded aperture camera system for human action recognition that is
privacy-preserving. While coded aperture systems exist, we believe ours is the
first system designed for action recognition without the need for image
restoration as an intermediate step. Action recognition is done using a deep
network that takes in as input, non-invertible motion features between pairs
of frames computed using phase correlation and log-polar transformation. Phase
correlation encodes translation while the log polar transformation encodes in-
plane rotation and scaling. We show that the translation features are
independent of the coded aperture design, as long as its spectral response
within the bandwidth has no zeros. Stacking motion features computed on frames
at multiple different strides in the video can improve accuracy. Preliminary
results on simulated data based on a subset of the UCF and NTU datasets are
promising. We also describe our prototype lens-free coded aperture camera
system, and results for real captured videos are mixed.

##  1 Introduction

Cameras as monitoring systems inside and outside the home or business is an
important area of growth. However, as cameras that are connected online are
prone to hacking, with images and videos illegally acquired potentially
resulting in loss of privacy and breach of security.

In this paper111Most of the work was done when Z.W. Wang, F. Pittaluga, and
S.B. Kang were at Microsoft Research., we describe initial work on a novel
privacy-preserving action recognition system. Our system enhances the
preservation of privacy from capture to executing visual tasks, as shown in
Figure 1. By using a lensless coded aperture (CA) camera, which places only a
coded aperture in front of an image sensor, the resulting CA image would be
visually unrecognizable and are difficult to restore with high fidelity.
Instead of decoding the image as a preprocessing step, which is ill-posed and
requires expensive computation if the mask is non-separable, we extract motion
features (translation, rotation, and scaling) using the Fourier-Mellin
transform and use them as inputs to a deep neural network.

We show that the translation features are invariant to the coded aperture (2D
mask pattern) design, as long as its Fourier transform is broadband (_i.e_.,
no zeros in the spectral magnitude). Specifically, the term “invariance”
refers to the fact that the translational features are only dependent on the
type of motion in the scene, not on the choice of the coded aperture design.
To promote the invariance property for all features, we design a training
mechanism which arbitrarily changes masks for each sample batch and observe
performance improvements when testing with a new random mask. The ”mask-
invariant” feature is important for two reasons: (1) training can be done
without reliance on a specific coded aperture design, and (2) from a
commercial perspective, no two random cameras are likely to have the same
coded aperture design, which makes image restoration virtually impossible
through reverse engineering.

Figure 1: Comparison of action recognition systems. The conventional system
(top) may be vulnerable to a privacy attack by an adversary. Our lensless
coded aperture camera system (bottom) preserves privacy by making the video
incomprehensible while allowing action recognition.

From a privacy perspective, the CA camera acts as the first layer of privacy
protection, as CA images are visually incomprehensible. Our motion features
provides a second layer of privacy. These features are based on phase
correlation between pairs of video frames, which whitens signal in Fourier
space and only leaves motion signal intact. Please note that from here on, we
use the terms “coded aperture” and “mask” interchangeably.

##  2 Related work

Our work is multi-disciplinary. The relevant areas are compressive sensing,
optics and sensors, coded apertures and action recognition. Here, we briefly
survey each area.

###  2.1 Reconstruction-free visual inference

Executing visual tasks without reconstructing the original visual data is an
interesting direction for data collected not in the form of visual
images/videos as reconstruction problems are usually ill-posed and
computationally expensive. One reconstruction heavy scenario is Compressive
Sensing (CS), where the measurements are far fewer than required by Shannon-
Nyquist requirement [11]. Tasks that can be solved by directly processing CS
data include optical flow [41], dynamic textures [35], face recognition [26,
25], and action recognition [20], _etc_. Our work considers a similar problem
to [20], _i.e_., performing action recognition without reconstructing images.
In the smashed filters approach, every frame of the scene is compressively
sensed by optically correlating random patterns with the frame to obtain CS
measurements. Therefore, the approach requires multiple sequential frame
capture and a DMD array (which is costly and has fragile moving parts). Our
approach uses a single coded aperture camera. Reconstruction-free methods do
not reveal the appearance of the scene and can therefore safeguard privacy in
sensitive environments.

###  2.2 Privacy-preserving optics and cameras

Optics and imaging sensors. There are imaging sensors and modalities whose
direct output is not visually recognizable. This achieves the purpose of
privacy preservation at the optics/sensor level. A popular approach for
preserving privacy is by defocusing [29]. Alternative optical solution is to
put optical elements in front of sensors, _e.g_., cylindrical lens [27],
diffraction gratings [39], or diffusers [2] in front of the sensor. Recovery
of these images requires careful calibration of the imaging system and
adequate computation.

Firmware. Sensor firmware can be modified to protect privacy before or during
the sensing process. For example, in PrivacyCam [6], regions of interest are
first identified based on background subtraction before being encrypted using
AES. Other implementations involve embedding watermarks into the captured data
[8, 19].

Coded apertures. Coded aperture imaging originates from the field of
astronomical X-ray and gamma-ray imaging in the 1960s [5, 10, 15]. By
extending pinholes to cameras with masks consisting of designed patterns,
coded apertures has been used for eliminating issues imposed by lenses and has
found novel applications in extending depth-of-field [9, 12], extracting scene
depth and light fields [22, 24, 43], and miniaturizing camera architectures
[1, 4]. Unlike conventional RGB images, lensless coded aperture images
obfuscates visual features familiar to human. Our work is inspired by this
distinctive effect. We explore the feasibility of using coded aperture data to
execute visual tasks such as action recognition, for the purpose of preserving
privacy.

###  2.3 Privacy-preserving action recognition

Action recognition is a long-standing computer vision task with wide
applications in video surveillance, autonomous vehicles and real-time patient
monitoring. Early approaches use handcrafted motion features, _e.g_., HOG/HOF
[21] and dense trajectories [44]. Recent works utilize two input streams for
appearance and motion [37] and 3D CNN architectures [42] to learn spatio-
temporal features [13]. State-of-the-art approaches for video-based action
recognition require both appearance and optical flow based motion features.
These systems are training on large video datasets, _e.g_., ImageNet and
Kinetics.

Privacy-preserving action recognition is becoming important due to the risk of
privacy breaches in surveillance systems in sensitive areas such as
healthcare. Approaches that use multiple extremely low resolution cameras have
been explored [7, 34]. Recently, Ren _et al_. used adversarial training to
anonymize human faces in videos, without affecting action recognition
performance [32]. Furthermore, adversarial learning has been explored to
jointly optimize privacy attributes and utility objectives [30, 28, 45].

##  3 Image formation for coded aperture camera

We consider a lens-free coded aperture imaging architecture, where a planar
coded aperture (mask) is placed in front of an imaging sensor. The encoding
mask can be considered as an array of pinholes located at various lateral
locations. The acquired image $\bm{d}$ can be numerically modeled as a
convolution between the object image $\bm{o}$ and the point spread function
(PSF) $\bm{a}$, i.e.,

$\bm{d}=\bm{o}\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{$\ast$}}}\bm{a}+\bm{e},$
(1)

with $\bm{e}$ being noise. The convolution is applicable if the mask is far
enough from the sensor, such that each sensor pixel is able to see the entire
mask pattern. If the mask-sensor distance is small (as in the case of FlatCam
[4]), the mask design should consist of a smaller pattern replicated in a 2D
array. The size of the smaller pattern should be such that each sensor pixel
sees a version of it locally. Then the output can be considered a result of
convolution.

We first implement the convolution based on FFT, which we refer as the
_without_ boundary effect (BE) version. However, we observe that real CA
images have boundary effect. We then incorporate boundary effect by zero-
padding both image and mask. The FFT-based convolution remains the same. We
then crop to the original size after convolution. This would generate
simulated CA frames that are more consistent with ones captured with a real
camera. However, this procedure is significantly more computationally
expensive. In experiments, we use the _without_ BE version for analysis of the
motion features and optimizing feature representation as DNN input, and both
versions are used for final testing.

##  4 Extraction of motion features

In this section, we describe how we compute features for action recognition
without having to first restore the images from a lenless coded aperture
camera. We refer to them as TRS (translation, rotation, scale) features. They
are computed from pairs of frames captured at different moments in time.

###  4.1 Translational (T) features

Phase correlation was used first for global image registration [31] and then
for motion/flow estimation [3, 17]. Compared to other motion estimation
methods [40], phase correlation has the advantages of being computational
efficient and invariant to illumination changes and moving shadows. We show
how phase correlation can be used to characterize motion in coded aperture
observations without knowing the mask design.

Assume there exists a translation between two video frames:

$\bm{o}_{1}(\mathbf{p})=\bm{o}_{2}(\mathbf{p}+\Delta\mathbf{p}),$ (2)

where $\mathbf{p}=[x,y]^{T}$ and $\Delta\mathbf{p}=[\Delta x,\Delta y]^{T}$
are the spatial coordinates and displacement, respectively.

In frequency domain, translation gives rise to a phase shift:

$\mathcal{O}_{1}(\mathbf{\nu})=\phi(\Delta\mathbf{p})\mathcal{O}_{2}(\mathbf{\nu}),$
(3)

where $\nu=[\xi,\eta]^{T}$ and $\phi(\Delta\mathbf{p})=\exp^{i2\pi(\xi\Delta
x+\eta\Delta y)}$. $\xi$ and $\eta$ are the frequency coordinates in Fourier
space. $\mathcal{O}_{1}$ and $\mathcal{O}_{2}$ represent Fourier spectra of
$\bm{o}_{1}$ and $\bm{o}_{2}$. By computing the cross-power spectrum and
taking an inverse Fourier transform, the translation yields a delta signal:

$\displaystyle\mathcal{C}_{o}(\xi,\eta)=\frac{\mathcal{O}_{1}^{*}\cdot\mathcal{O}_{2}}{|\mathcal{O}_{1}^{*}\cdot\mathcal{O}_{2}|}=\phi^{*}\frac{\mathcal{O}_{2}^{*}\cdot\mathcal{O}_{2}}{|\mathcal{O}_{2}^{*}\cdot\mathcal{O}_{2}|}=\phi(-\Delta\mathbf{p}),$
(4) $\displaystyle\bm{c}(\mathbf{p})=\delta(\mathbf{p}+\Delta\mathbf{p}).$ (5)

The translation can be located by finding the peak signal; this feature is the
basis of the original work [31], assuming a single global translation.
Multiple translations result in an ensemble of delta functions.

###  4.2 T features independent of coded apertures

The convolutional transformation that generates a CA image encodes local
motion in the original video to global motion in the resulting CA video. This
makes the localization of the motion very challenging without restoration.
However, we demonstrate that the global translation can still be retrieved
using phase correlation, and is _independent_ of the mask design, as long as
they have broadband spectrum. Following Eqs. (1) and (3), a translation
relationship ($\Delta\mathbf{p}$) also exists:

$\mathcal{D}_{1}(\nu)=\mathcal{O}_{1}\cdot\mathcal{A}=\phi\mathcal{O}_{2}(\nu)\cdot\mathcal{A}=\phi\mathcal{D}_{2}(\nu),$
(6)

where $\mathcal{A}$ denotes the Fourier spectrum of mask $\bm{a}$. The cross-
power spectrum is then

$\mathcal{C}_{d}(\nu)=\frac{\mathcal{D}_{1}^{*}\cdot\mathcal{D}_{2}}{|\mathcal{D}_{1}^{*}\cdot\mathcal{D}_{2}|}=\phi^{*}\frac{\mathcal{O}_{2}^{*}\cdot\mathcal{A}^{*}\cdot\mathcal{A}\cdot\mathcal{O}_{2}}{|\mathcal{O}_{2}^{*}\cdot\mathcal{A}^{*}\cdot\mathcal{A}\cdot\mathcal{O}_{2}|}\simeq\mathcal{C}_{o}.$
(7)

Note that phase correlation has a magnitude normalization procedure while
computing the cross-power spectrum. This step can effectively whiten the
spectrum so as to eliminate global changes in appearance. This property
provides an additional layer of privacy protection. In our implementation, we
add a small number $\epsilon$ in the denominator of Eq. (7) to prevent
division by zero. Regardless, the object spectrum will be unstable if
$\mathcal{A}$ has near-zero elements.

###  4.3 Coded aperture design

We focus on 2D intensity binary mask patterns as they enable practical
implementations. As shown in Figure 2, the randomness in the mask pattern,
which result in broadband spectra, preserves the T features compared to the T
map computed from RGB frames. Figure 2 show representative masks that are
considered. The pseudorandom mask (mask 1) provides a relatively uniform
magnitude distribution. The separable mask (mask 2) based on maximum length
sequence (MLS) have much stronger frequency response along the horizontal and
vertical axes. Mask 3 is a round aperture and has undesirable dropoffs at
higher frequencies. We use pseudorandom masks in our evaluation.

Figure 2: T features from different CA observations. 3 different mask patterns
(all 50% clear) are investigated (Row 2). Row 1 shows the cross-section of
Fourier spectra. Rows 3 and 4 show example RGB images and their corresponding
synthetic CA frames (_withoout_ BE). Row 5: T feature maps based on Eq. (7).
Row 6: error maps, with the “ground truth” being the T map for RGB frames.
$\epsilon=10^{-3}$.

Note that since these masks are spatially as large as the image and non-
separable in x and y (except row 1), high fidelity image restoration would be
difficult and computationally-expensive [9]. We did not implement a
restoration algorithm for these reasons.

We will show later that using only T features is less effective for action
recognition (Figure 3). We investigate two extensions of the T features,
namely rotation and scale features, and multiple strides.

###  4.4 Rotation and scale features in log-polar space

Given global translation, rotation, and scaling, we have
$\bm{o}_{1}(\mathbf{p})=\bm{o}_{2}(s\mathbf{Rp}+\Delta\mathbf{p})$, where $s$
is a scaling factor and $\mathbf{R}$ is a rotation matrix with angle
$\Delta\theta$. Translation $\Delta\mathbf{p}$ can be eliminated by taking the
magnitude of the Fourier spectrum,

$|\mathcal{O}_{1}(\nu)|=|\mathcal{O}_{2}(s\mathbf{R\nu})|.$ (8)

If we treat the Fourier spectra as images and transform them into log-polar
representations, i.e.,
$\mathbf{p}=[x,y]^{T}\Rightarrow\mathbf{q}=[\log(\rho),\theta]^{T}$, rotation
and scaling become additive shifts on the two axes

$|\mathcal{O}_{1}(\mathbf{q})|=|\mathcal{O}_{2}(\mathbf{q}+\Delta\mathbf{q})|.$
(9)

This enables us to use phase correlation once again to locate rotation and
scale. Note that the mask invariant property is not preserved in RS space.
This is because the mask spectrum contributes to a strong static signal to the
observed images. However, we later show that the mask-invariant property for
RS features can be realized by training with varying random masks.

###  4.5 Multi-stride TRS (MS-TRS)

We make a further extension to compute TRS features based on multiple strides
in each video clip. This is to account for varying speeds of motion. For a
video clip with length $l$, the TRS features in stride $s$ are computed by:

$T_{i}^{(s)},RS_{i}^{(s)}=\mathcal{TRS}\\{\bm{d}_{i\times s},\bm{d}_{i\times
s+s}\\},$ (10)

where $i\in\\{0,1,...,\lfloor\frac{l-s}{s}\rfloor+1\\}$ denotes all the
possible consecutive indices within length $l$. For example, if a video clip
of length 13 is given, the resulting $s2$ TRS features have 12 channels, 6 for
T, and 6 for RS. In our case, we compare evaluation results for strides of 2,
3, 4, 6, with clip lengths of 13 and 19.

##  5 Experimental results on simulated data

We now report the results for the following experiments:

  * •

We compare the performance of our method based on CA videos with a baseline
that uses regular videos.

  * •

We evaluate the performance of our method when the proposed T, TRS, and MS-TRS
features are used.

  * •

We compare the effect of using the same versus different or varying masks on
training and validation data.

  * •

We also compare the effect of using different MS-TRS configurations. This
experiment is used to select an appropriate configuration for the final
evaluation.

  * •

We report results for the best MS-TRS configuration.

We first describe the datasets and protocols used.

Datasets. We have evaluated our approach on the UCF-101 [38] and NTU [36]
datasets. UCF-101 [38] contains 101 action classes with 13k videos. In our
initial evaluation, we focus on indoor settings (more important from a privacy
standpoint). Therefore, we created four different subsets from the 101 classes
by selecting actions relevant to indoors.

  * •

UCF-05: Writing on board, Wall pushups, blowing candles, pushups, mopping
floor;

  * •

UCF-body (09): Hula hoop, mopping floor, baby crawling, body weight squat,
jumping jack, wall push up, punch, push ups and lunges;

  * •

UCF-subtle (13): Apply eye makeup, apply lipsticks, blow dry hair, blowing
candles, brushing teeth, cutting in kitchen, mixing batter, typing, writing on
board, hair cut, head assage, shaving beard, knitting;

  * •

UCF-indoor (22): combination of UCF-body and UCF-subtle.

We also use the NTU [36] dataset which contains videos of indoor actions. We
choose this dataset as it collects data using stationary cameras (we handle
only static background for now). From our initial evaluation, we found that
our proposed approach is better suited for more significant body motions.
Because of this, we choose ten classes (with a mix of whole and partial body
motions) for our final testing. Eight classes come from the NTU dataset and
two classes are from the UCF dataset.

###  5.1 Protocol

Definitions. We use letters $s$ and $l$ to denote the stride and length of a
video. For example, $s1,l4$ denotes four consecutive video frames. The number
of input channels depends on the training mode.

Training and Validation. We use the first official train/test split from the
UCF dataset and randomly select 20% of the training set for validation. Both
the training and validation data is expanded using data augmentation to
prevent over-fitting. The data augmentation process is as follows.

  * •

gray clips: Each video frame is loaded in as grayscale image at a resolution
between 224 and 256. The aspect ratio is fixed at ($240\times 320$). The clip
is then vertically flipped with 50% chance. A ($224\times 224\times l$) clip
is then cropped and used as input.

  * •

CA clips: Each CA clip first experiences the same augmentation step as gray
clips. The CA simulation is computed at the resolution of $256\times 256$ and
rescaled back to $224\times 224$. We simulate CA observations by computing
element-wise multiplication in Fourier space between the Fourier transforms of
the image and the mask kernel. We did not implement boundary effect for
computation consideration. The diffraction effect is not accounted for as we
observe minimal impact on the TRS features. Another reason is that simulating
PSF for non-separable masks by matrix multiplication [9] is expensive.

  * •

T features: The T features are generated from CA clips at the resolution of
$256\times 256$. The central $224\times 224$ area is cropped as input. An
$l$-frame CA clip results in ($l-1$) T channels.

  * •

TRS/MS-TRS features: In the TRS setting, the T features follow the same
cropping. For RS, the R-axis uses center cropping while the S-axis is
downsized to 224. An $l$-frame CA clip results in $2l$ channels, with $l$ T
channels and $l$ RS channels stacked together. For MS-TRS, the resulting
channels depend on the selected strides.

We use a batch size of 16 or 32. Each epoch, for both training and validation,
prepares samples randomly from approximately 20% of all the possible frame
combinations. 50 Epochs are used in our evaluation experiments. The percentage
of accurate samples is reported. When reporting, we compute the running
average accuracy of 5 epochs for better visualization.

Testing. During testing, we resampled each video at 3 spatial scales
($\mu\times\mu$ pixels, with $\mu=224,256,300$) and 5 temporal starting frames
evenly distributed across the video length. For example, using MS-
TRS-$s346$-$l19$ configuration, a video with 100 frames will be used to
generate five clips, starting at frames 1, 21, 41, 61, and 81, with each clip
being 19 frames long. Each clip will be used to compute MS-TRS at three
spatial scales. The final score for each video is computed by averaging the
scores of the 15 clips.

Others. We use the VGG-16 CNN architecture, which contains approximately 134
million parameters. Adam optimizer is used with learning rate 0.0001,
$\beta_{1}=0.9,\beta_{2}=0.999$. Since the CA observation is computed on-the-
fly, we can change the underlying masks used in each batch. In this paper, we
use “m1/m1” to refer to the setting where training and validation using the
same fixed mask and “m1/m2” to refer to when training and validation uses two
different masks. Finally, “dm1/dm2” denotes the setting where training and
validation is done using variable masks. A pseudo-random binary mask is
randomly generated for each batch. Note that the mask is fixed for all frames
of a single video.

###  5.2 Initial evaluation (without BE)

The goal of our initial evaluation is to validate our proposed training
framework, as well as to find the optimal feature representation. Such
experiments are implemented using CA simulations _without_ BE, as accounting
for boundary effect is computationally more expensive.

Baselines. We first train one network on the original videos and three
networks on the simulated CA videos as our four baselines. See the results in
Table 1. The top-1 classification accuracy of 95% (row 1) for the original
videos is our upper bound of what we can expect. The performance of the
baselines trained directly on CA videos (rows 2 to 4), will serve as our lower
bounds. We expect our proposed features, which involve computation based on
CA, to perform better than CA. The CA baselines show instability even when
training and validation phases have the same mask. The network corresponding
to the second row suffers from overfitting. Changing training masks for each
batch does not improve the performance.

| training | validation
---|---|---
gray video | 99.56 (99.86) | 94.39 (95.91)
CA (m1/m1) | 79.06 (92.65) | 63.21 (86.96)
CA (m1/m2) | 94.66 (95.17) | 27.95 (40.55)
CA (dm1/dm2) | 34.93 (36.61) | 27.23 (36.96)
Table 1: Baseline comparison for UCF-05. Here, for the CA cases, training and
validation are done directly on CA videos. The numbers are: average accuracy %
of the last 5 epochs (maximum accuracy %). All clips have length 3.

Variable masks during training. Our goal is to maximize the robustness of the
designed features to the mask patterns. In order to achieve this, we change
the training and validation masks by randomly generating a pseudo-random mask
during each batch. We compare this dynamic training mechanism with two other
modalities, _i.e_., (1) training and validation using the same mask (m1/m1)
and (2) training and validation using two different masks, no mask variation
during training (m1/m2). The results are presented in Figure 3.

Figure 3: Comparison of validation accuracy for UCF-05, with training and
validation: using the same mask (m1/m1), using two different masks (m1/m2),
and based on a random mask per batch and a different random mask for
validation (dm1/dm2). Note: s3 = stride of 3, s2346 = strides of 2, 3, 4, and
6.

For T features, the validation accuracy plateaus at about 60%. Dynamic
training with variable masks does not improve the accuracy. This supports the
fact that T features are invariant to the choice of masks.

For TRS and MS-TRS features, using the same stride and length of the clips,
the performance improves to around 70% for m1/m1. However, since the RS
features are not mask-invariant, validation using a different mask does not
have the same accuracy. Varying the masks during training does not improve the
performance compared to training using the same mask. This is an interesting
effect as, theoretically, the RS features do not have the same mask-invariant
property. This drawback appears to be mitigated by changing the masks during
training. This, in turn, enables us to test using an arbitrary mask. MS-TRS
trained with varying mask achieves the highest validation accuracy 77.8%.

Strides and clip length. In the case of TRS, we found that increasing the
strides and clip lengths can improve the performance. We evaluated different
combinations of MS-TRS features. The training and validation for MS-TRS is
under dm1/dm2 mode. The results are summarized in Table 2. For the same video
length, using larger strides improves validation accuracy. For the same stride
setting, _e.g_., $s346$, processing more video frames improves performance.
However, using longer stride and longer video, such as _i.e_. $s46,l19$,
suffers from overfitting. The combination $s2346,l19$ is not evaluated as
generating the 44-channel input on-the-fly becomes computationally expensive.

| input shape | training | validation
---|---|---|---
$s2346,l13$ | (224, 224, 30) | 96.67 | 83.59
$s346,l13$ | (224, 224, 18) | 93.69 | 83.66
$s46,l13$ | (224, 224, 10) | 92.94 | 86.59
$s346,l19$ | (224, 224, 26) | 96.00 | 86.26
$s46,l19$ | (224, 224, 14) | 89.91 | 79.23
Table 2: Comparison of training and validation performances for MS-TRS,
dm1/dm2 for UCF-05. Numbers are max accuracy percentage within the first 50
epochs.

More action classes. We selected three MS-TRS settings from Table 2 and then
trained networks for three larger datasets. These datasets are also subsets of
UCF-101 actions focused on indoor settings and include body motions and subtle
motions which primarily involve hand & face. The evaluation results are shown
in Table 3.

| UCF-body | UCF-subtle | UCF-indoor
---|---|---|---
$s346,l13$ | 88.4 / 81.2 | 84.9 / 73.2 | 84.8 / 70.8
$s346,l19$ | 90.5 / 83.4 | 86.1 / 76.4 | 88.6 / 72.8
$s46,l13$ | 89.9 / 79.1 | 80.9 / 66.5 | 83.8 / 66.3
Table 3: Training and validation accuracies on different UCF subsets for
networks trained on different MS-TRS configurations. UCF-body, UCF-subtle and
UCF-indoor has 9, 13 and 22 classes respectively.

###  5.3 Testing results

Based on the experiments on the UCF subset datasets, we selected _i.e_., MS-
TRS-$s346$-$l19$ as the best feature representation. Next, we computed MS-
TRS-$s346$-$l19$ features on the 10-class combined dataset of NTU and UCF to
examine the feasibility of our representation for daily activities. We used
about one-sixth of the NTU videos for the eight classes for training to ensure
we have a similar number of training examples as for the two UCF classes. In
training phase, each class consists of 100 videos with more than 10K frames.
We use a different data augmentation scheme for the NTU dataset. Each NTU
video is loaded at random height resolution between 460 and 520. The aspect
ratio is fixed at $1080:1920=9:16$.

The central $240\times 320$ region (same as the UCF classes) is cropped and
used to compute CA and MS-TRS. For testing, each NTU video is loaded at
$522\times 928$ resolution. The central $256\times 256$ video is cropped and
used to compute CA and MS-TRS at different scales as described in the testing
protocol.

For synthetic CA testing, the overall top-1 accuracy is 60.1% _without_ BE and
35.5% _with_ BE. The top-1, 2, 3 accuracies for each class is reported in
Table 4. The results indicate a large variation across classes. Our trained
model is able to correctly recognize body motions such as hopping and
staggering but is less accurate at differentiating between subtle hand motions
such as clapping and hand waving.

class top-1 top-2 top-3 1 hopping 97.1 / 97.1 100 / 97.1 100 / 100 2
staggering 94.3 / 65.7 97.1 / 91.4 100 / 100 3 jumping up 91.4 / 0.00 97.1 /
71.4 97.1 / 88.6 4 JJ $\dagger$ 81.1 / 16.2 91.9 / 83.8 100 / 91.9 5 BWS
$\dagger$ 76.7 / 33.3 86.7 / 73.3 93.3 / 90.0 6 standing up 57.1 / 20.0 88.6 /
40.0 94.3 / 54.3 7 sitting down 51.4 / 11.4 82.9 / 22.9 100 / 31.4 8 throw
31.4 / 20.0 57.1 / 48.6 68.6 / 80.0 9 clapping 11.4 / 20.0 14.3 / 68.6 31.4 /
77.1 10 hand waving 5.70 / 71.4 14.3 / 88.6 20.0 / 88.6 average 60.1 / 35.5
73.4 / 68.6 80.8 / 80.2

Table 4: Testing results for combined NTU and UCF 10 classes dataset. Data
format: accuracy % _without_ BE / _with_ BE. BWS: body weight squats; JJ:
jumping jack. $\dagger$ indicates the class comes from UCF dataset, others are
from NTU dataset. Ranking according to top-1 accuracy _without_ BE.

##  6 Experimental results on real data

###  6.1 Prototype

To validate our ideas, we built an imaging system as shown in Figure 4. Our
system consists of a monochrome board-level imaging sensor (XIMEA MQ042,
$2048\times 2048$) and a spatial light modulator or SLM (LC2012, $1024\times
768$) sandwiched between two polarizing filters. The distance between the
sensor and SLM is approximately 6mm. The pixel size for the XIMEA camera is
5.5um while that for the SLM is 36um. A long-pass filter is required to remove
light frequencies that have low extinction factors with the SLM-filter combo.
In addition, we use a cover with a square opening ($12$mm $\times$ $12$mm) to
cut out stray oblique rays and reduce inter-reflection on the side walls
between the SLM and sensor.

Figure 4: Prototype consisting of monochrome camera XIMEA MQ042 and spatial
light modulator LC2012.

###  6.2 Testing results

We collect several CA videos using our prototype system and test using both
models _with_ and _without_ BE. These models are trained on a subset of NTU
and UCF data as discussed in Section 5.3. Each testing video consists of 100
consecutively captured frames.

Quantitative results are shown in Table 5. We observe that “body weight
squats” is a dominating class, and has been correctly classified. Other
classes such as “jumping jack” and “standing up” are only correctly classified
in the top-2 and top-3 choices. “Sitting down” and “hand waving” have not been
correctly classified. Examples of our successful and failed videos are shown
in Fig. 5.

We hypothesize that a possible reason for such failure could be due to the
fact that failure videos are much darker than the successful videos. Further,
we note that the models used for testing the prototype data has been trained
only on NTU and UCF data. The model has not seen a single sample from the real
prototype system. We believe this could have caused domain gap between the
prototype and simulated models, that led to loss in accuracy. Such performance
drop has been observed in other recognition problems as well. For example,
loss in accuracy has been observed when a deep model trained on computer
graphics data is tested on real world data [33].

In order to resolve the domain gap issue, we will investigate two future
research directions. First, we will capture a large set of CA training data
from our prototype system. Currently there is no publicly available CA dataset
for action recognition. Collecting such a large scale coded aperture dataset
is an interesting direction, and will be really valuable for wider research
community working on privacy-preserving action recognition problem. Second, we
will fine-tune the model that has been pre-trained on large scale simulated CA
data, e.g., on NTU-UCF data. Such fine-tuning should help to achieve better
robustness and generalization, as shown in RGB based action recognition tasks
[14]. These are interesting future research directions.

class | top-1 | top-2 | top-3
---|---|---|---
BWS (3) | 100 / 100 | 100 / 100 | 100 / 100
jumping jack (5) | 0.0 / 0.0 | 100 / 0.0 | 100 / 40.0
standing up (1) | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 100
sitting down (2) | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0
hand waving (8) | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0
Table 5: Results on captured CA videos. Accuracies (in percentage) using
_with_ BE / _without_ BE model are reported. Figure 5: Examples of captured
videos used for testing. The four rows from top to bottom show one example of
the ”jumping jack”, ”body weight squats”, ”hand waving” and ”sitting down”
classes respectively.

##  7 Discussion

Restoration of coded aperture images. Restoration from CA images is a non-
trivial task. Deconvolution can be done if the mask design is known (including
PSF or mask code, pixel pitch, distance between the SLM and the sensor) [4,
9], although their masks are separable in x and y whereas ours are not. Even
when the mask and camera parameters are known, restoring our CA images can be
expected to be substantially more computational expensive.

If the mask pattern is unknown, reconstruction approaches can be designed by
incorporating several properties of the encoding mask. Correlation-based
approaches can be used for recovery as the pseudorandom masks have
approximately a delta function as their autocorrelation. The autocorrelation
of a CA image is equivalent to the autocorrelation of the scene image:
$\bm{d}\star\bm{d}\simeq(\bm{o}\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{$\ast$}}}\bm{a})\star(\bm{o}\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{$\ast$}}}\bm{a})=(\bm{o}\star\bm{o})\mathop{\scalebox{1.5}{\raisebox{-0.86108pt}{$\ast$}}}(\bm{a}\star\bm{a})\propto\bm{o}\star\bm{o}$.
The object signal can thus be recovered using a phase retrieval algorithm [16,
18]. However, such methods can only restore a coarse image (specifically, near
binary quality at high contrast areas). Other constraints such as coprime blur
pairs (CBP) [23] can be applied for on/post capture video blurring and
recovery. Although the polynomial CBP kernels can be estimated, it imposes
higher numerical precision for the captured images.

Attacking our system through deep learning is plausible. A deep neural network
may be designed to estimate the underlying optical parameters and mask
pattern, or to reconstruct the original image; this assumes enough training
data can be collected. Since a lensless coded aperture result in a global
image transformation, a fully-connected layer may well be required.

Limitations. In our work, we assume that our camera is perfectly stationary,
which is typically the case for indoor surveillance. Our FFT-based features
are sensitive to extraneous global motion that is not related to body action;
a source of such motion is camera shake. As noted earlier, our system is also
unable to discern local multiple complex motions such as hand-waving and head
scratching.

##  8 Conclusions

There are several interesting takeaways from our experiments. First, training
directly on the CA videos results in poor performance. Second, varying the
mask at random during training reduces overfitting and improves performance.
Third, using multiple strides with TRS (MS-TRS) as input works the best. This
is likely attributed to its ability to adapt to different speeds of motion. We
also described our prototype, and results for real CA sequences are mixed.
However, we believe this is a good first step towards proving the viability of
using CA cameras for privacy-preserving action recognition.

## References

  * [1] Jesse K Adams, Vivek Boominathan, Benjamin W Avants, Daniel G Vercosa, Fan Ye, Richard G Baraniuk, Jacob T Robinson, and Ashok Veeraraghavan.  Single-frame 3D fluorescence microscopy with ultraminiature lensless FlatScope.  Science Advances, 3(12):e1701548, 2017.
  * [2] Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller.  Diffusercam: lensless single-exposure 3d imaging.  Optica, 5(1):1–9, 2018.
  * [3] Vasileios Argyriou and Theodore Vlachos.  A study of sub-pixel motion estimation using phase correlation.  In BMVC, pages 387–396, 2006.
  * [4] M Salman Asif, Ali Ayremlou, Aswin Sankaranarayanan, Ashok Veeraraghavan, and Richard G Baraniuk.  Flatcam: Thin, lensless cameras using coded aperture and computation.  IEEE Transactions on Computational Imaging, 3(3):384–397, 2017\.
  * [5] TM Cannon and EE Fenimore.  Coded aperture imaging: Many holes make light work.  Optical Engineering, 19(3):193283, 1980.
  * [6] Ankur Chattopadhyay and Terrance E Boult.  Privacycam: A privacy preserving camera using uCLinux on the Blackfin DSP.  In CVPR, pages 1–8, 2007.
  * [7] Ji Dai, Jonathan Wu, Behrouz Saghafi, Janusz Konrad, and Prakash Ishwar.  Towards privacy-preserving activity recognition using extremely low temporal and spatial resolution cameras.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 68–76, 2015.
  * [8] Lieven De Strycker, Pascale Termont, Jan Vandewege, J Haitsma, AACM Kalker, Marc Maes, and Geert Depovere.  Implementation of a real-time digital watermarking process for broadcast monitoring on a TriMedia VLIW processor.  IEE Proceedings-Vision, Image and Signal Processing, 147(4):371–376, 2000.
  * [9] Michael J DeWeert and Brian P Farm.  Lensless coded-aperture imaging with separable Doubly-Toeplitz masks.  Optical Engineering, 54(2):023102, 2015.
  * [10] RH Dicke.  Scatter-hole cameras for x-rays and gamma rays.  The Astrophysical Journal, 153:L101, 1968.
  * [11] David L Donoho et al.  Compressed sensing.  IEEE Transactions on information theory, 52(4):1289–1306, 2006\.
  * [12] Edward R Dowski and W Thomas Cathey.  Extended depth of field through wave-front coding.  Applied Optics, 34(11):1859–1866, 1995.
  * [13] Christoph Feichtenhofer, Axel Pinz, Richard P Wildes, and Andrew Zisserman.  What have we learned from deep representations for action recognition?  Connections, 19:29, 2018.
  * [14] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Convolutional two-stream network fusion for video action recognition.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [15] Edward E Fenimore and Thomas M Cannon.  Coded aperture imaging with uniformly redundant arrays.  Applied Optics, 17(3):337–347, 1978.
  * [16] James R Fienup.  Phase retrieval algorithms: A comparison.  Applied Optics, 21(15):2758–2769, 1982.
  * [17] Huy Tho Ho and Roland Goecke.  Optical flow estimation using Fourier Mellin transform.  In CVPR, pages 1–8. IEEE, 2008.
  * [18] Ori Katz, Pierre Heidmann, Mathias Fink, and Sylvain Gigan.  Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations.  Nature Photonics, 8(10):784, 2014.
  * [19] Elias Kougianos, Saraju P Mohanty, and Rabi N Mahapatra.  Hardware assisted watermarking for multimedia.  Computers & Electrical Engineering, 35(2):339–358, 2009.
  * [20] Kuldeep Kulkarni and Pavan Turaga.  Reconstruction-free action inference from compressive imagers.  IEEE transactions on pattern analysis and machine intelligence, 38(4):772–784, 2016.
  * [21] Ivan Laptev, Marcin Marszalek, Cordelia Schmid, and Benjamin Rozenfeld.  Learning realistic human actions from movies.  In CVPR, pages 1–8, 2008.
  * [22] Anat Levin, Rob Fergus, Frédo Durand, and William T Freeman.  Image and depth from a conventional camera with a coded aperture.  ACM Transactions on Graphics (TOG), 26(3):70, 2007.
  * [23] Feng Li, Zijia Li, David Saunders, and Jingyi Yu.  A theory of coprime blurred pairs.  In 2011 International Conference on Computer Vision, pages 217–224. IEEE, 2011.
  * [24] Chia-Kai Liang, Tai-Hsu Lin, Bing-Yi Wong, Chi Liu, and Homer H Chen.  Programmable aperture photography: Multiplexed light field acquisition.  In ACM Transactions on Graphics (TOG), volume 27, page 55, 2008\.
  * [25] Suhas Lohit, Kuldeep Kulkarni, Pavan Turaga, Jian Wang, and Aswin C Sankaranarayanan.  Reconstruction-free inference on compressive measurements.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16–24, 2015.
  * [26] Pradeep Nagesh and Baoxin Li.  A compressive sensing approach for expression-invariant face recognition.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1518–1525. IEEE, 2009.
  * [27] Shota Nakashima, Yuhki Kitazono, Lifeng Zhang, and Seiichi Serikawa.  Development of privacy-preserving sensor for person detection.  Procedia-Social and Behavioral Sciences, 2(1):213–217, 2010.
  * [28] Francesco Pittaluga, Sanjeev Koppal, and Ayan Chakrabarti.  Learning privacy preserving encodings through adversarial training.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 791–799. IEEE, 2019.
  * [29] Francesco Pittaluga and Sanjeev Jagannatha Koppal.  Pre-capture privacy for small vision sensors.  IEEE TPAMI, 39(11):2215–2226, 2017.
  * [30] Nisarg Raval, Ashwin Machanavajjhala, and Landon P Cox.  Protecting visual secrets using adversarial nets.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1329–1332, 2017.
  * [31] B Srinivasa Reddy and Biswanath N Chatterji.  An FFT-based technique for translation, rotation, and scale-invariant image registration.  IEEE Transactions on Image Processing, 5(8):1266–1271, 1996.
  * [32] Zhongzheng Ren, Yong Jae Lee, and Michael S Ryoo.  Learning to anonymize faces for privacy preserving action detection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 620–636, 2018.
  * [33] Stephan Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for data: Ground truth from computer games.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [34] Michael S Ryoo, Kiyoon Kim, and Hyun Jong Yang.  Extreme low resolution activity recognition with multi-siamese embedding learning.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [35] Aswin C Sankaranarayanan, Pavan K Turaga, Richard G Baraniuk, and Rama Chellappa.  Compressive acquisition of dynamic scenes.  In European Conference on Computer Vision, pages 129–142. Springer, 2010.
  * [36] Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang.  NTU RGB+D: A large scale dataset for 3D human activity analysis.  In CVPR, June 2016.
  * [37] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In Advances in Neural Information Processing Systems, pages 568–576, 2014.
  * [38] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  UCF101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [39] Leonidas Spinoulas, Oliver S Cossairt, Aggelos K Katsaggelos, Patrick Gill, and David G Stork.  Performance comparison of ultra-miniature diffraction gratings with lenses and zone plates.  In Computational Optical Sensing and Imaging, pages CM3E–1. Optical Society of America, 2015.
  * [40] A Murat Tekalp.  Digital video processing.  Prentice Hall Press, 2015.
  * [41] Vijayaraghavan Thirumalai and Pascal Frossard.  Correlation estimation from compressed images.  Journal of Visual Communication and Image Representation, 24(6):649–660, 2013.
  * [42] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, pages 4489–4497, 2015.
  * [43] Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal, Ankit Mohan, and Jack Tumblin.  Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing.  ACM Transactions on Graphics (TOG), 26(3):69, 2007.
  * [44] Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng-Lin Liu.  Action recognition by dense trajectories.  In CVPR, pages 3169–3176, 2011.
  * [45] Zhenyu Wu, Zhangyang Wang, Zhaowen Wang, and Hailin Jin.  Towards privacy-preserving visual recognition via adversarial training: A pilot study.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 606–624, 2018.