# Sum-Rate Optimization for RIS-Aided Multiuser Communications with Movable
Antenna

Yunan Sun, Hao Xu, Chongjun Ouyang, and Hongwen Yang The authors are with the
School of Information and Communication Engineering, Beijing University of
Posts and Telecommunications, Beijing, 100876, China (e-mail: {sunyunan,
Xu_Hao, DragonAim, yanghong}@bupt.edu.cn).(Corresponding author: Hongwen Yang)

###### Abstract

Reconfigurable intelligent surface (RIS) is known as a promising technology to
improve the performance of wireless communication networks, which has been
extensively studied. Movable antenna (MA) is a novel technology that fully
exploits the antenna position for enhancing the channel capacity. In this
paper, we propose a new RIS-aided multiuser communication system with MAs. The
sum-rate is maximized by jointly optimizing the beamforming, the reflection
coefficient (RC) values of RIS and the positions of MAs. A fractional
programming-based iterative algorithm is proposed to solve the formulated non-
convex problem, considering three assumptions for the RIS. Numerical results
are presented to verify the effectiveness of the proposed algorithm and the
superiority of the proposed MA-based system in terms of sum-rate.

###### Index Terms:

Movable antenna (MA), multiuser communications, reconfigurable intelligent
surfaces (RIS), sum-rate optimization.

##  I INTRODUCTION

With the development of metasurfaces, reconfigurable intelligent surface (RIS)
has attracted a great deal of attentions as a promising technology to realize
the smart radio environments (SRE)[1]. An RIS is a two-dimensional surface
consisting of many low-cost passive elements and a controller. By adjusting
the propagation of the incident signals smartly, RIS is able to improve the
performance of wireless communication networks with increasing number of
degrees of freedom (DoFs) into the channel, especially when the line-of-sight
(LoS) paths between the base station (BS) and user terminals (UTs) are
blocked[2]. Thus, RIS proves to be the most promising technology to enhance
the efficiency of data transmission. A lot of research has been done to
introduce RIS as an aid to conventional multiuser communication system[3]-[5].

On the other hand, the concept of movable antennas (MAs) was recently
conceived to harness additional spatial DoFs, which can overcome some
constraints of conventional fixed-position antennas (FPAs)[6]. The positions
of MAs can be adjusted by controllers in real time, such as stepper motors or
servos, since the MAs are connected to radio frequency (RF) chains via
flexible cables. Compared with conventional FPA-based wireless communication
systems, the MA-based systems can reasonably change the positions of transmit
antennas depending on the channel state information (CSI) such that the
channel between MAs and UTs is reshaped for achieving higher capacity[7].
Moreover, the MAs can be flexibly moved in a three-dimensional (3D) region to
fully exploit the channel variation therein[8].Thus the MA-based system can
exploit the full spatial diversity in a given region with much fewer antennas
than the conventional antenna selection (AS) technique.Due to the superiority
of MA, the concept of MA has attracted increasing attention. The recieve
signal-to-noise ratio (SNR) of a MA-based system was analyzed in [6]. In [9],
the authors initially characterized the capacity of a point-to-point MIMO
channel with MAs, where both the BS and the UTs are equipped with MAs. An
fractional programming (FP)-based algorithm was first introduced to an MA-
based downlink transmission framework to tackle the joint optimization problem
of transmit beamforming and MA positions for maximizing the sum-rate in [10].

As no existing works focus on the MA-based multiuser wireless communication
system with the aid of RIS, we investigate an RIS-aided MA enabled downlink
multiuser multiple-input single output (MU-MISO) system in this paper, where
only the BS is equipped with MAs and the LoS paths between the BS and UTs are
blocked. Under this scenario, we focus on the joint optimization problem of
beamforming, reflection coefficient (RC) values of RIS and the positions of
MAs aiming to maximize the sum-rate of the presented system.

The main contributions of this work is summarized as follows:

  * •

This paper is one of the early attempts to study the sum-rate optimization for
the RIS-aided MA enabled downlink MU-MISO system. We propose this framework
that harness both the RIS and MAs to improve sum-rate.

  * •

We propose an iterative FP-based algorithm to solve the joint optimization
problem of beamforming, RC values of RIS and the positions of MAs.

  * •

Numerical results have verified the effectiveness of the proposed algorithm
and the superiority of the proposed MA-based system in terms of sum-rate.

##  II SYSTEM MODEL

###  II-A System Description

We investigate an RIS-aided multiuser MISO downlink communication system with
movable-antenna BS as shown in Figure 1, where the BS equipped with $N$
transmit antennas sends signals to $K$ single antenna UTs with an RIS which
has $M$ reflection elements. The UTs are denoted by
$\mathcal{K}\triangleq\\{1,...,K\\}$. Each UT has a receive FPA while the BS
has N transmit MAs. All the MAs can move to change their positions in real
time since they are connected to RF chains via flexible cables. The position
of $n$-th MA is represented by Cartesian coordinates
$\textbf{t}_{n}=[x_{n},y_{n}]^{T}\in\mathcal{C}$ for
$n\in\mathcal{N}\triangleq\\{1,...,N\\}$, where $\mathcal{C}$ is a square
region with size $A\times A$.

Figure 1: The RIS-aided communication system with MA-BS.

For simplicity, we assume that all the channels experience quasi-static flat-
fading. In addition, we assume that the CSI of all channels involved is
perfectly known by the BS and the IRS. The RIS-aided link can be modeled as a
concatenation of three components, i.e., the BS-RIS link, RIS phase-shift
matrix and RIS-UT link. Thus the channel vector
$\textbf{H}_{k}\in\mathbbmss{C}^{N\times 1}$ from the BS to $k$-th UT follows
the structure as:

$\displaystyle\textbf{H}_{k}=\textbf{G}\bm{\Phi}^{H}\textbf{h}_{k},$ (1)

where $\textbf{G}=\textbf{G}_{t}^{H}\bm{\Lambda}^{H}\textbf{G}_{r}$. The terms
mentioned are defined as follows:

  * •

$\textbf{G}_{t}=[\textbf{g}_{1},...,\textbf{g}_{N}]\in\mathbbmss{C}^{L\times
N}$ is the transmit field response vector(FRV) at the BS, where $L$ is the
number of channel paths between the BS and RIS and
$\textbf{g}_{n}\in\mathbbmss{C}^{L\times 1}$ is the transmit FRV between the
$n$-th MA and the RIS for $n=1,...,N$.

  * •

$\textbf{g}_{n}=[{\rm{e}}^{j\frac{2\pi}{\lambda}\textbf{t}_{n}^{T}\bm{\rho}_{n,1}},...,{\rm
e}^{j\frac{2\pi}{\lambda}\textbf{t}_{n}^{T}\bm{\rho}_{n,L}}]^{T}$, where
$\bm{\rho}_{n,l}=[{\rm sin}\theta_{n,l}{\rm cos}\phi_{n,l},{\rm
cos}\theta_{n,l}]^{T},\theta_{k,l}\in[0,2\pi]$ and $\phi_{k,l}\in[0,2\pi]$ are
the elevation and azimuth angles of the $l$-th path, respectively, for
$l=1,...,L$. And $\lambda$ is the wavelength.

  * •

$\bm{\Lambda}={\rm diag}\\{[\nu_{1},...,\nu_{L}]\\}\in\mathbbmss{C}^{L\times
L}$, where $\nu_{l}$ is the complex response of the $l$-th path.

  * •

$\textbf{G}_{r}\in\mathbbmss{C}^{L\times M}$ is the receive FRV of the RIS.

  * •

$\bm{\Phi}={\rm diag}(\psi_{1},...,\psi_{M})\in\mathbbmss{C}^{M\times M}$ is
the phase-shift matrix of the RIS, where $\psi_{m}$ is the reflection
coeffecient (RC) of $m$-th RIS element for $m=1,...,M$.

  * •

$\textbf{h}_{k}=[h_{k_{1}},...,h_{k_{M}}]^{T}\in\mathbbmss{C}^{M\times 1}$ is
the channel response between the RIS and the $k$-th UT for $k=1,...,K$.

Besides, we consider following three assumptions for the feasible set of RC in
this paper:

  * •

Ideal RC(IRC): It only restricts that the RC is peak-power constrained:

$\displaystyle\mathcal{F}_{1}=\\{\psi_{m}|\mid\psi_{m}\mid^{2}\leq 1\\}.$ (2)

  * •

Continuous Phase Shifter(CPS): It is assumed that the strength of the
reflection signal from each reflection element is maximized, thus
$|\psi_{m}|^{2}=1$, which can be represented as:

$\displaystyle\mathcal{F}_{2}=\\{\psi_{m}|\psi_{m}={\rm
e}^{j\xi_{m}},\xi_{m}\in[0,2\pi)\\}.$ (3)

  * •

Discrete Phase Shifter(DPS): In practice, the phase of RC has finite levels.
We assume that the phase of $\psi_{m}$ only takes $\kappa$ arithmetic values,
which can be represented as:

$\displaystyle{\mathcal{F}_{3}=\\{\psi_{m}|\psi_{n}={\rm
e}^{j\xi_{m}},\xi_{m}\in\\{0,\frac{2\pi}{\kappa},...,\frac{2\pi(\kappa-1)}{\kappa}\\}\\}}.$
(4)

Denoted by $\textbf{x}\in\mathbbmss{C}^{N\times 1}$ the input of the downlink
transmission, then the received signal at $k$-th UT is expressed as:

$\displaystyle y_{k}=\textbf{H}_{k}^{H}\textbf{x}+n_{k},$ (5)

where $n_{k}$ is the circularly symmetric complex Gaussian noise with zero
mean and covariance $\sigma_{k}^{2}$. And
$\textbf{x}=\sum_{k=1}^{K}\textbf{w}_{k}x_{k}$, where $x_{k}\in\mathbbmss{C}$
denotes the transmit data symbol to $k$-th UT with
$\textbf{w}_{k}\in\mathbbmss{C}^{N\times 1}$ being corresponding transmit
beamforming vector. It is assumed that all the $x_{k}$ are independent random
variables with zero mean and unit variance. The received signal-to-
interference-plus-noise ratio(SINR) at $k$-th UT is expressed as:

$\displaystyle\gamma_{k}=\frac{|\textbf{h}_{k}^{H}\bm{\Phi}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{k}|^{2}}{\sum_{i\neq
k}|\textbf{h}_{k}^{H}\bm{\Phi}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{i}|^{2}+\sigma_{k}^{2}},$
(6)

Note that $\textbf{G}_{t}$ depends on the position of MAs. Then the sum-rate
can be expressed as:

$\displaystyle R=\sum_{k=1}^{K}{\rm log}(1+\gamma_{k}).$ (7)

###  II-B Problem Formulation

This paper aims to improve the sum-rate of the system by jointly optimizing
the transmit beamforming, the RC of RIS and the positions of MAs, subject to
appropriate constraints. The transmit power constraint of BS is
$\sum_{k=1}^{K}|\textbf{w}_{k}|^{2}\leq P_{max}.$

A minimum distance $D$ is required between each pair of MAs in order to avoid
the coupling effect between the antennas in the transmit region. Thus we can
formulate the sum-rate optimization problem as:

$\displaystyle{\mathcal{P}}_{\text{1}}:$
$\displaystyle~{}\max_{{\textbf{W}},{\textbf{T}},{\bf\Phi}}~{}{R}$ (8a) s.t.
$\displaystyle~{}\sum\nolimits_{k=1}^{K}\textbf{w}_{k}^{H}\textbf{w}_{k}\leq
P_{\max},$ (8b) $\displaystyle~{}\psi_{m}\in\mathcal{F},\forall m=1,...,M,$
(8c) $\displaystyle~{}\textbf{t}_{n}\in\mathcal{C},\forall
n=1,...,N,|\textbf{t}_{n}-\textbf{t}_{n^{\prime}}|\leq D,n\neq n^{\prime}$
(8d)

where $\textbf{W}=[\textbf{w}_{1},...,\textbf{w}_{K}]\in\mathbbmss{C}^{N\times
K}$,$\mathcal{F}\in\\{\mathcal{F}_{1},\mathcal{F}_{2},\mathcal{F}_{3}\\}$ and
$\textbf{T}=[\textbf{t}_{1},...,\textbf{t}_{N}]\in\mathbbmss{R}^{2\times N}$.
The above problem is difficult to tackle directly due to the fact that the
objective function (8a) is not convex and the constraints (8d) is not a convex
set. Moreover, the coupling among W, $\bm{\Phi}$ and T makes this problem more
challenging to solve.

##  III PROPOSED ALGORITHM

In this section, we present a joint optimization algorithm to solve problem
(8). Firstly, (8a) is simplified into a more trackable form by invoking the FP
method, i.e., the Lagrangian dual transform and quadratic transform proposed
in [11]. Then, the variables to be optimized are updated in an alternating
manner, with other variables fixed. We present three solutions for different
constraint conditions in (8c).

###  III-A Optimizing beamforming matrix W

We firstly introduce two auxiliary variable
$\bm{\alpha}=[\alpha_{1},...,\alpha_{K}]$ and
$\bm{\beta}=[\beta_{1},...,\beta_{K}]$ to turn (8a) into a convex equivalent
form. Then the problem can be written as:

$\displaystyle{\mathcal{P}}_{\text{2}}:$
$\displaystyle~{}\max_{{\textbf{W}},{\textbf{T}},{\bm{\Phi}},{\bm{\alpha}},{\bm{\beta}}}~{}{R^{\prime}=\sum_{k=1}^{K}{\rm
log}(1+\alpha_{k})}-\sum_{k=1}^{K}\alpha_{k}$
$\displaystyle~{}+\sum_{k=1}^{K}(1+\alpha_{k})[2\Re\\{\beta_{k}^{*}A_{k}\\}-|\beta_{k}|^{2}B_{k}]$
(9a) s.t.
$\displaystyle~{}\eqref{P_X_Cons1},~{}\eqref{P_X_Cons2},~{}\eqref{P_X_Cons3},$
(9b) $\displaystyle~{}\alpha_{k}>0,\beta_{k}\in\mathbbmss{C},k\in\mathcal{K}.$
(9c)

where
$A_{k}=\textbf{h}_{k}^{H}\bm{\Phi}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{k}$
and
$B_{k}=\sigma_{k}^{2}+\sum_{i=1}^{K}|\textbf{h}_{k}^{H}\bm{\Phi}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{i}|^{2}$.
In (9), when W, $\bm{\Phi}$ and T hold fixed, the optimal $\alpha_{k}$ and
$\beta_{k}$ are given by

$\displaystyle\alpha_{k}^{\circ}=\gamma_{k}$ (10)
$\displaystyle\beta_{k}^{\circ}=\frac{\sqrt{1+\alpha_{k}}A_{k}}{B_{k}}$ (11)

After updating $\alpha_{k}$ and $\beta_{k}$, (9) becomes a standard convex
quadratic optimization problem whose solution is [11] when $\bm{\Phi}$ and T
are fixed. The optimal $\textbf{w}_{k}$ is given by

$\displaystyle\textbf{w}_{k}^{\circ}=\sqrt{1+\alpha_{k}}\beta_{k}(\lambda_{0}\textbf{I}+\sum_{i=1}^{K}|\beta_{i}|^{2}\textbf{C}_{i})^{-1}\textbf{G}_{t}^{H}\bm{\Lambda}\textbf{G}_{r}\bm{\Phi}^{H}\textbf{h}_{k}$
(12)

where
$\textbf{C}_{i}=\textbf{G}_{t}^{H}\bm{\Lambda}\textbf{G}_{r}\bm{\Phi}^{H}\textbf{h}_{i}\textbf{h}_{i}^{H}\bm{\Phi}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}$
and $\lambda_{0}$ is the dual variable introduced for the power constraint,
which is optimally determined by

$\displaystyle\lambda_{0}^{\circ}=min\\{\lambda_{0}\geq
0:\sum_{k=1}^{K}|\textbf{w}_{k}|^{2}\leq P_{max}\\}.$ (13)

The dual variable $\lambda_{0}$ is chosen, such that the complementarity
slackness, i.e.,
$\lambda_{0}\left(\sum_{k=1}^{K}|\textbf{w}_{k}|^{2}-P_{max}\right)$, is
satisfied. It can be seen that $\sum_{k=1}^{K}|\textbf{w}_{k}|^{2}$ is a
monotonic decreasing function with $\lambda_{0}\geq 0$. Thus, we can find
$\lambda_{0}$ via bisection-based search.

###  III-B Optimizing Reflection Response Matrix $\bm{\Phi}$

With $\textbf{W},\alpha$ and $\beta$ fixed, the problem (9) can be expressed
as:

$\displaystyle{\mathcal{P}}_{\text{3}}:$
$\displaystyle~{}\max_{{\textbf{T}},{\bf\Phi}}~{}{R^{\prime\prime}=\sum_{k=1}^{K}\left(1+\alpha_{k}\right)\frac{\gamma_{k}}{1+\gamma_{k}}}$
(14a) s.t. $\displaystyle~{}\eqref{P_X_Cons2},~{}\eqref{P_X_Cons3}.$ (14b)

Simiarly, we introduce an auxiliary variable
$\bm{\epsilon}=[\epsilon_{1},...,\epsilon_{K}]$ and optimize $\bm{\epsilon}$
and $\bm{\Phi}$ alternatively. Then (14) can be translated to the following
problem based on the quadratic transform proposed in [12]:

$\displaystyle{\mathcal{P}}_{\text{4}}:$
$\displaystyle~{}\max_{{\textbf{T}},{\bf\Phi},{\bm{\epsilon}}}~{}{f\left(\textbf{T},\bm{\Phi},\bm{\epsilon}\right)}$
(15a) s.t. $\displaystyle~{}\eqref{P_X_Cons2},~{}\eqref{P_X_Cons3}.$ (15b)

where the translated objective function is

$\displaystyle~{}f\left(\textbf{T},\bm{\Phi},\bm{\epsilon}\right)=\sum_{k=1}^{K}[2\sqrt{1+\alpha_{k}}\Re\\{\epsilon_{k}^{H}\bm{\varphi}^{H}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{k}\\}$
$\displaystyle~{}-|\epsilon_{k}|^{2}\sigma_{k}^{2}+|\epsilon_{k}|^{2}\sum_{i=1}^{K}|\bm{\varphi}^{H}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{i}|^{2}]$
(16)

where $\bm{\varphi}=diag(\bm{\Phi}^{H})\in\mathbbmss{C}^{M\times
1},\textbf{S}_{k}={\rm diag}(\textbf{h}_{k})\in\mathbbmss{C}^{M\times M}$.
When $\bm{\Phi}$ hold fixed, the optimal $\bm{\epsilon}$ can be calculated by
setting $\partial f/\partial\epsilon_{k}$ to zero, which can be expressed as:

$\displaystyle\epsilon_{k}^{\circ}=\frac{\sqrt{1+\alpha_{k}}\bm{\varphi}^{H}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{k}}{\sum_{i=1}^{K}|\bm{\varphi}^{H}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{i}|^{2}+\sigma_{k}^{2}}$
(17)

With $\bm{\epsilon}$ fixed, the objective function can be written as:

$\displaystyle
f_{2}(\bm{\varphi})=-\bm{\varphi}^{H}\textbf{U}\bm{\varphi}+2\Re\\{\textbf{V}^{H}\bm{\varphi}\\}$
(18)

where

$\displaystyle\textbf{U}=\sum_{k=1}^{K}|\epsilon_{k}|^{2}(\sum_{i=1}^{K}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{i}\textbf{w}_{i}^{H}\textbf{G}_{t}^{H}\bm{\Lambda}^{H}\textbf{G}_{r}\textbf{S}_{k}),$
$\displaystyle\textbf{V}=\sum_{k=1}^{K}\epsilon_{k}^{H}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{k}$

The function (18) is a quadratic concave function of $\bm{\varphi}$ since U is
a positive-definite matrix. If $\mathcal{F}=\mathcal{F}_{1}$, then the
optimization of $\bm{\varphi}$ is a convex problem due to the convexity of
$\mathcal{F}_{1}$. Then the problem can be transformed to its dual problem via
Lagrange dual decomposition:

$\displaystyle{\mathcal{P}}_{\text{5}}:$
$\displaystyle~{}\max_{{\bm{\varphi}}}~{}{f_{3}(\bm{\varphi},\bm{\eta})=f_{2}\left(\bm{\varphi}\right)-\sum_{m=1}^{M}\eta_{m}(\bm{\varphi}^{H}\textbf{e}_{m}\textbf{e}_{m}^{H}\bm{\varphi}-1)}$
(19a) s.t. $\displaystyle~{}\psi_{m}\in\mathcal{F}_{1},\forall m=1,...,M,$
(19b)

where $\textbf{e}_{m}\in\mathbbmss{R}^{M\times 1}$ is an elementary vector
with a one at the $m$-th position, $\bm{\eta}=[\eta_{1},...,\eta_{M}]$ is the
dual variable for the constraint of the $m$-th RIS element, i.e.,
$\bm{\varphi}^{H}\textbf{e}_{m}\textbf{e}_{m}^{H}\bm{\varphi}\leq 1$. And the
optimal $\bm{\varphi}$ can be obtained by setting $\partial f_{3}/\varphi$ to
zero, which can be expressed as:

$\displaystyle\bm{\varphi}=(\sum_{m=1}^{M}\eta_{m}\textbf{e}_{m}\textbf{e}_{m}^{H}+\textbf{U})^{-1}\textbf{V}$
(20)

Therefore, the optimal dual variable vector $\bm{\eta}$ can be determined
according to the constraints in (19b) via ellipsoid method [2].

If $\mathcal{F}=\mathcal{F}_{2}$ or $\mathcal{F}=\mathcal{F}_{3}$, then the
optimization of $\bm{\varphi}$ is not a convex problem due to the non-
convexity of $\mathcal{F}_{2}$. It can be written as:

$\displaystyle{\mathcal{P}}_{\text{6}}:$
$\displaystyle~{}\max_{{\bm{\varphi}}}~{}{f_{2}\left(\bm{\varphi}\right)}$
(21a) s.t. $\displaystyle~{}\psi_{m}\in\mathcal{F}_{2},\forall m=1,...,M,$
(21b)

Thus, we optimize $\bm{\varphi}$ via minimum-maximum (MM) method, which has
the following iterative form:

$\displaystyle\textbf{Q}_{\tau}(\bm{\varphi})=-\bm{\varphi}^{H}\bm{\Omega}\bm{\varphi}+2\Re\\{\bm{\varphi}^{H}(\bm{\Omega}-\textbf{U})\bm{\varphi}_{\tau}\\}$
$\displaystyle-\bm{\varphi}_{\tau}^{H}(\bm{\Omega}-\textbf{U})\bm{\varphi}_{\tau}+2\Re\\{\textbf{V}^{H}\bm{\varphi}\\}$
(22)

where $\textbf{Q}_{\tau}$ is a base value of (21a) for the in the $\tau$-th
iteration, $\bm{\Omega}=\lambda_{max}I$ and $\lambda_{max}$ is the maximum
eigenvalue of U. Since the contraint (21b) guarantees that
$\bm{\varphi}^{H}\bm{\varphi}=M$, $\bm{\varphi}^{H}\bm{\Omega}\bm{\varphi}$
and $\bm{\varphi}_{\tau}^{H}(\bm{\Omega}-\textbf{U})\bm{\varphi}_{\tau}$ are
not related to the value of $\bm{\varphi}$. If $\mathcal{F}=\mathcal{F}_{2}$,
it is obvious that the optimal $\bm{\varphi}$ in the $\tau$-th iteration can
be expressed as:

$\displaystyle{\bm{\varphi}_{\tau+1}={\rm
e}^{j\angle\\{(\bm{\Omega}-\textbf{U})\bm{\varphi}_{\tau}-\textbf{V}\\}}}.$
(23)

If $\mathcal{F}=\mathcal{F}_{3}$, the optimal phase of $m$-th element of
$\bm{\varphi}$ in the $\tau$-th iteration can be expressed as:

$\displaystyle{{\xi_{m_{\tau+1}}}={\min_{\xi_{m}\in\\{0,...,\frac{2\pi(\kappa-1)}{\kappa}\\}}~{}{|\xi_{m}-\angle\\{(\bm{\Omega}-\textbf{U})\bm{\varphi}_{\tau}-\textbf{V}\\}}|}}.$
(24)

###  III-C Optimizing MA Position Matrix T

With other variables fixed, we translate the problem (14) to the following
problem by introducing an auxiliary $\bm{\delta}=[\delta_{1},...,\delta_{K}]$.

$\displaystyle{\mathcal{P}}_{\text{7}}:$
$\displaystyle~{}\max_{{\textbf{T}},{\bm{\delta}}}~{}{f_{4}(\textbf{T})=\sum_{k=1}^{K}2\sqrt{1+\alpha_{k}}\Re\\{\delta_{k}^{H}\bm{\Phi}^{H}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{k}\\}}$
$\displaystyle~{}{-|\delta_{k}|^{2}(\sum_{i=1}^{K}|\bm{\Phi}^{H}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{i}|^{2}+\sigma_{k}^{2})}$
(25a) s.t. $\displaystyle~{}\eqref{P_X_Cons3}.$ (25b)

Similar to (17), the optimal $\bm{\delta}$ with fixed T is:

$\displaystyle\delta_{k}^{\circ}=\frac{\sqrt{1+\alpha_{k}}\bm{\varphi}^{H}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{k}}{\sum_{i=1}^{K}|\bm{\varphi}^{H}\textbf{S}_{k}^{H}\textbf{G}_{r}^{H}\bm{\Lambda}\textbf{G}_{t}\textbf{w}_{i}|^{2}+\sigma_{k}^{2}}$
(26)

Then the remaining problem is to optimize T. This optimization problem can be
expressed as:

$\displaystyle{\mathcal{P}}_{\text{8}}:$
$\displaystyle~{}\max_{{\textbf{T}}}~{}f_{5}(\textbf{T})=\sum_{k=1}^{K}2\sqrt{1+\alpha_{k}}\Re\\{\textbf{P}^{H}_{k}\textbf{G}_{t}\textbf{w}_{k}\\}$
$\displaystyle~{}-\textbf{P}_{k}^{H}\textbf{G}_{t}\bm{\Pi}\textbf{G}_{t}^{H}\textbf{P}_{k}$
(27a) s.t. $\displaystyle~{}\eqref{P_X_Cons3}.$ (27b)

where
$\textbf{P}_{k}=\Lambda^{H}\textbf{G}_{r}{\textbf{S}_{k}}\bm{\varphi}\delta_{k}$
and $\bm{\Pi}=\sum_{i=1}^{K}\textbf{w}_{i}\textbf{w}_{i}^{H}$. The objective
function (27a) can also be written as

$f_{4}(\textbf{T})=\sum_{k=1}^{K}2\sqrt{1+\alpha_{k}}\Re\left\\{\sum_{l}^{L}{P^{*}_{k_{l}}}{\rm
e}^{j\frac{2\pi}{\lambda}\textbf{t}_{n}^{T}\bm{\rho}_{n,l}}{w_{k_{n}}}\right\\}-\sum_{n=1}^{N}\sum_{l=1}^{L}\sum_{n^{\prime}=1}^{N}\sum_{l^{\prime}=1}^{L}P^{*}_{k_{l}}P_{k_{l^{\prime}}}{\rm
e}^{j\frac{2\pi}{\lambda}(\textbf{t}_{n}^{T}\bm{\rho}_{n,l}-\textbf{t}_{n^{\prime}}^{T}\bm{\rho}_{n^{\prime},l^{\prime}})}\Pi_{n,n^{\prime}}$
where $P_{k_{l}}$ is the $l$-th element of $\textbf{P}_{k}$, $w_{k_{n}}$ is
the $n$-th element of $\textbf{w}_{k}$ and $\Pi_{n,m}$ is the $(n,m)$-th
element of $\bm{\Pi}$. Due to the intractability of $f_{4}(\textbf{T})$, we
adopt a gradient descent (GD) algorithm with backtracking line search [13] to
optimize T. The gradient values of $f_{4}(\textbf{T})$ w.r.t. $\textbf{t}_{n}$
are calculated as follows [14]:

$\displaystyle~{}\nabla_{\textbf{t}_{n}}f_{4}=\sum_{k=1}^{K}\sqrt{1+\alpha_{k}}C_{k}+E_{k}+F_{k}$
(28)

where
$C_{k}=\sum_{l=1}^{L}\frac{-4\pi\rho_{n,l}}{\lambda}|P^{*}_{k_{l}}w_{k_{n}}|{\rm
sin}(\frac{2\pi}{\lambda}\textbf{t}_{n}^{T}\rho_{n,l}+\angle{P^{*}_{k_{l}}w_{k_{n}}})$,
$E_{k}=\sum_{n^{\prime}\neq
n}^{N}\sum_{l=1}^{L}\sum_{l^{\prime}=1}^{L}\frac{4\pi\rho_{n,l}}{\lambda}|P^{*}_{k_{l}}P_{k_{l^{\prime}}}\Pi_{n^{\prime},n}\hfill{{\rm
e}^{-j\frac{2\pi}{\lambda}\textbf{t}_{n^{\prime}}^{T}\bm{\rho}_{n^{\prime},l^{\prime}}}|}$
${\rm
sin}(\frac{2\pi}{\lambda}\textbf{t}_{n}^{T}\bm{\rho}_{n,l}+\angle{P^{*}_{k_{l}}P_{k_{l^{\prime}}}\Pi_{n^{\prime},n}{\rm
e}^{-j\frac{2\pi}{\lambda}\textbf{t}_{n^{\prime}}^{T}\bm{\rho}_{n^{\prime},l^{\prime}}}})$
and ${F_{k}=}$
${\sum_{l=1}^{L}\sum_{l^{\prime}=1}^{L}\frac{2\pi(\rho_{n,l}-\rho_{n,l^{\prime}})}{\lambda}|P^{*}_{k_{l}}P_{k_{l^{\prime}}}\Pi_{n,n}|{\rm
sin}(\frac{2\pi}{\lambda}\textbf{t}_{n}^{T}(\bm{\rho}_{n,l}-\bm{\rho}_{n,l^{\prime}})}$
$+\angle{P^{*}_{k_{l}}P_{k_{l^{\prime}}}\Pi_{n,n}})$. The algorithm of
optimizing T is given in Algorithm 1, which is guaranteed to converge since
the sum-rate is upper bounded. The proposed joint optimization algorithm is
summarized in Algorithm 2, which is also guaranteed to converge to a
stationary solution of Problem $\mathcal{P}_{1}$.

1 Initialize $\textbf{T}^{0}=[\textbf{t}_{1}^{0},...,\textbf{t}_{N}^{0}]$,
step size $\mu_{0}$, the minimum step size $\mu_{min}$, the iteration index
$q=0$ and the maximum iteration number $q_{max}$. repeat

2 for  _n=1:N_ do

3 Calculate $\nabla_{\textbf{t}_{n}}f_{4}$ by (28) and set $\mu=\mu_{0}$;

4 repeat

5 Calculate
$\hat{\textbf{t}}_{n}=\textbf{t}_{n}^{q}+\mu\nabla_{\textbf{t}_{n}}{f_{4}}$;

6 Set $\mu=\mu/2$;

7 until  _( 8d)& $f_{4}(\hat{t}_{n})>f_{4}(t_{n}^{q})or\mu<\mu_{min}$_;

8 Set $q=q+1$.

9

10

11until  _converge or reach the predefined maximum number of iterations
$q_{max}$_;

Algorithm 1 GD-Based Algorithm for Optimizing T

1 Initialize $\\{\textbf{W}^{0},\bm{\Phi}^{0},\textbf{T}^{0}\\}$, the
iteration index $r=0$ and the maximum iteration number $r_{max}$.

2repeat

3 Update $\bm{\alpha}^{r}$ and $\bm{\beta}^{r}$ by (10) and (11)

4 Calculate $\textbf{W}^{r+1}$ by (12)

5 Update $\bm{\epsilon}^{r}$ by (17)

6 if  _$\mathcal{F}=\mathcal{F}_{1}$_ then

7 Update $\bm{\varphi}^{r}$ by (20)

8 else if  _$\mathcal{F}=\mathcal{F}_{2}$ or $\mathcal{F}=\mathcal{F}_{3}$_
then

9 Set $\tau=0$ and $\tau_{max}$ for MM method

10 repeat

11 Update $\bm{\varphi}_{\tau}$ by (23) or (24)

12 Set $\tau=\tau+1$

13 until  _converge or reach $\tau_{max}$_;

14 Update $\bm{\varphi}^{r}=\bm{\varphi}_{\tau}$ and obtain $\bm{\Phi}^{r+1}$

15 Update $\bm{\delta}$ by (26)

16 Solve $\mathcal{P}_{8}$ by Algorithm 1, and obtain $\textbf{T}^{r+1}$

17 Set $r=r+1$.

18

19until  _converge or reach the predefined maximum number of iterations
$r_{max}$_;

Algorithm 2 FP-Based Algorithm for Solving $\mathcal{P}_{1}$

##  IV Numerical Results

In this section, numerical results are provided to confirm the effectiveness
of the proposed algorithm. In the simulation, we set $N=4$, $K=4$, $M=16$,
$D=\frac{\lambda}{2}$ and $\sigma_{k}^{2}=-100$dBm($\forall k$). The distance
between the BS and RIS is 50 m. The UTs are uniformly and randomly distributed
around a center, which is 100 m away from RIS, with radius 10 m. Moreover, we
have incorporated the free-space path loss model for this system. It is widely
known that the signal reflected by the RIS suffered from the “double-fading”
effect. Thanks to the recent advances in meta-materials, the reflection gain
of the RIS elements is usually high. We assume $\varsigma=10$dB denotes the
reflection gain of RIS. Then the equivalent free-space path loss model of
$k$-th UT is given by
$-10{\rm{log}}_{10}v=92.5+20{\rm{log}}_{10}[f_{0}({\rm{GHZ}})]+20{\rm{log}}_{10}[d_{0}({\rm{km}})]-\varsigma$
and
$-10{\rm{log}}_{10}u_{k}=92.5+20{\rm{log}}_{10}[f_{0}({\rm{GHZ}})]+20{\rm{log}}_{10}[d_{k}({\rm{km}})]-\varsigma$,
where $f_{0}=2$GHz is the carrier frequency ,$d_{0}$ is the distance from the
BS to the RIS and $d_{k}$ is the distance from the RIS to the $k$-th UT. As
for the channel model, we assume that $L=4$,
$\nu_{l}\sim\mathcal{CN}(0,\frac{v}{L})(\forall l)$ and
$h_{k_{m}}\sim\mathcal{CN}(0,u_{k})(\forall k,m)$. The elevation and azimuth
angles are randomly set within $[0\pi]$. We compare the performance of MA and
FPA-based scheme, where the BS is equipped with a uniform linear array
composed of $N$ FPAs spaced by $\frac{\lambda}{2}$. We also compare the
performance of RIS optimization and fixed RIS case, where the RC of the RIS is
initialized with random phase and unit amplitude.

Firstly, we depict the convergence of the proposed methods for MAs in Figure
2. From this picture, it is observed that the sum-rate tends to converge to a
stable value and all the three algorithms converge in about 12 iterations.

Then, we present the sum-rate versus the transmit power budget $P_{max}$ in
Figure 3. By comparing the performance of MA and FPA, it can be seen that the
proposed algorithm achieves a larger sum-rate than the schemes with FPAs.
Furthermore, we compare the performance of different RIS set. The IRC
constraint have the best performance, but the sum-rate loss is negligible when
it reduces to CPS constraint. It means that when the RIS is set with CPS
constraint, the amplitude of the optimal RC is close to 1. The DPS constraint
scheme achieves lower sum-rate than IRC and CPS since the inaccurate phase of
RC.

Lastly, we show the sum-rate versus the normalized region size $A/\lambda$
when $P_{max}=10$dBm in Figure 4. The achievable sum rate increases when the
normalized region size gets larger. For CPS constraint, the performance tend
to converge when $A/\lambda$ is over 3 while for IRC and DPS, the gain of
region size is not obvious. This reveals that the optimal sum-rate performance
for MA enbaled systems can be achieved with a finit transmit region and it is
necessary to set a large enough region size for MA especially when the RIS is
taken into account.

Figure 2: Sum-rate versus the number of iterations. Figure 3: Sum-rate versus
the power budget. $A=2\lambda$ Figure 4: Sum-rate versus the normalized region
size.

##  V Conclusion

In this paper, we investigated the RIS-aided movable antenna enabled multiuser
communication system with the aim of optimizing the sum-rate through
beamforming, RC of RIS and the positions of MAs. We presented a FP-based
algorithm to solve the joint optimization problem. Numerical results suggested
that the RIS-aided MA enabled architecture outperforms conventional schemes in
terms of sum-rate.

## References

  * [1] Y. Liu, X. Liu, X. Mu, T. Hou, J. Xu, M. Di Renzo, and N. Al-Dhahir, “Reconfigurable intelligent surfaces: Principles and opportunities,” _IEEE Communications Surveys & Tutorials_, vol. 23, no. 3, pp. 1546–1577, 2021.
  * [2] H. Guo, Y. -C. Liang, J. Chen and E. G. Larsson, “Weighted sum-rate maximization for intelligent reflecting surface enhanced wireless networks,” in _Proc. IEEE GLOBECOM_ , 2019, pp. 1–6.
  * [3] Y. Xiu, Y. Zhao, Y. Liu, J. Zhao, O. Yagan, and N. Wei, “IRS assisted millimeter wave communications: Joint power allocation and beamforming design,” in _Proc. IEEE Wireless Commun. Netw. Conf. Workshops (WCNCW)_ , Mar. 2021, pp. 1–6.
  * [4] Y. Liu, X. Mu, X. Liu, M. Di Renzo, Z. Ding and R. Schober, “Reconfigurable intelligent surface-aided multi-user networks: Interplay between NOMA and RIS,” _IEEE Wireless Commun._ , vol. 29, no. 2, pp. 169–176, Apr. 2022.
  * [5] C. Pan, H. Ren, K. Wang, W. Xu, M. Elkashlan, A. Nallanathan, and L. Hanzo, “Multicell MIMO communications relying on intelligent reflecting surfaces,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 8, pp.5218–5233, Aug. 2020.
  * [6] L. Zhu, W. Ma, and R. Zhang, “Modeling and performance analysis for movable antenna enabled wireless communications,” _arXiv:_ 2210.05325, 2022.
  * [7] X. Pi, L. Zhu, Z. Xiao, and R. Zhang, “Multiuser communications with movable-antenna base station via antenna position optimization,” _arXiv:_ 2308.05546,2023.
  * [8] L. Zhu ,W. Ma, and R. Zhang, “Movable antennas for wireless communication: Opportunities and challenges,” _arXiv:_ 2306.02331, 2023.
  * [9] W. Ma, L. Zhu, and R. Zhang, “MIMO capacity characterization for movable antenna systems,” _IEEE Tran. Wireless Commun._ , Early Access, 2023.
  * [10] Z. Cheng, N. Li, J. Zhu, and C. Ouyang, “Sum-rate maximization for movable antenna enabled multiuser communications” _arXiv:_ 2309.11135, 2023.
  * [11] K. Shen, and W. Yu, “Fractional programming for communication systems–Part I: Power control and beamforming,” _IEEE Trans. Signal Process._ , vol. 66, no. 10, pp. 2616–2630, May. 2018.
  * [12] K. Shen and W. Yu, “Fractional programming for communication systems-Part II: Uplink scheduling via matching,” _IEEE Trans. Signal Process._ , vol. 66, no. 10, pp. 2631–2644, May 2018.
  * [13] X. D. Zhang, Matrix Analysis and Applications. Cambridge, U.K.: CambridgeUniv.Press,2017.
  * [14] A. Hjorungnes and D. Gesbert, “Complex-valued matrix differentiation: Techniques and key results,” _IEEE Trans. Signal Process._ , vol. 55, no. 6, pp. 2740–2746, 2007.


# CodedEvents: Optimal Point-Spread-Function Engineering for
3D-Tracking with Event Cameras

Sachin Shah  Matthew A. Chan  Haoming Cai  Jingxi Chen  Sakshum Kulshrestha
Chahat Deep Singh  Yiannis Aloimonos  Christopher A. Metzler
University of Maryland, College Park
<EMAIL_ADDRESS>
###### Abstract

Point-spread-function (PSF) engineering is a well-established computational
imaging technique that uses phase masks and other optical elements to embed
extra information (e.g., depth) into the images captured by conventional CMOS
image sensors. To date, however, PSF-engineering has not been applied to
neuromorphic event cameras; a powerful new image sensing technology that
responds to changes in the log-intensity of light.

This paper establishes theoretical limits (Cramér Rao bounds) on 3D point
localization and tracking with PSF-engineered event cameras. Using these
bounds, we first demonstrate that existing Fisher phase masks are already
near-optimal for localizing static flashing point sources (e.g., blinking
fluorescent molecules). We then demonstrate that existing designs are sub-
optimal for tracking moving point sources and proceed to use our theory to
design optimal phase masks and binary amplitude masks for this task. To
overcome the non-convexity of the design problem, we leverage novel implicit
neural representation based parameterizations of the phase and amplitude
masks. We demonstrate the efficacy of our designs through extensive
simulations. We also validate our method with a simple prototype.

Figure 1:  CodedEvent Tracking. Left: example recovered trajectory using
designed optics for an event camera. Right: top row, optimal phase mask design
and PSFs for a CMOS sensor, bottom row, our optimal phase mask design and PSFs
for an event sensor.

##  1 Introduction

Single-molecule localization microscopy (SMLM) is a vital tool for resolving
nano-scale structures with applications in analysis of protein clusters [39],
cell dynamics [62], and electromagnetic effects [30]. Traditional SMLM
experiments are limited by the slow capturing process of frame-based CMOS
sensors, preventing use in capturing high-speed, dynamic interactions.
Recently, [9] showed event cameras are key to enabling high-speed 2D SMLM.

In contrast to traditional CMOS cameras, event cameras are an emerging class
of bio-inspired neuromorphic sensors that operate with a high temporal
resolution on the order of $\mu$s. These sensors are comprised of an
asynchronous pixel array, where each pixel records an event when the log
intensity change exceeds a set threshold. In addition to having kilohertz time
resolution, these sensors are low-power, resistant to constant background
noise, and can operate over a high dynamic range [14]. Already, these sensors
have proven useful in a range of applications including object tracking [4,
60], gesture recognition [32, 3], and robotics [24, 48].

Just as PSF-engineering allows one to extract additional information using
conventional CMOS sensors [52], we believe that event-camera-specific PSF
engineering will be the key to enabling high-speed 3D SMLM with event cameras.
Unfortunately, existing PSF design theory is not equipped for the event space.
In this work, we bridge this gap by developing Cramér Rao Bounds on 3D
position estimation for event camera measurements. Leveraging these bounds, we
subsequently develop a novel implicit neural representation for optical
elements to design components with improved 3D particle localization
capabilities.

Specifically, our principal contributions are as follows:

  * •

We derive the Fisher Information and Cramér Rao Bounds for event camera
measurements parameterized by 3D spatial positions.

  * •

We develop novel implicit neural representations for learning both amplitude
and phase masks.

  * •

We identify new phase and amplitude designs for optimally encoding 3D
information with event cameras.

  * •

We demonstrate in simulation that our designs outperform existing methods at
3D particle tracking.

##  2 Related Work

###  2.1 Coded Optics

Specialized lenses have been shown to encode additional depth information in
CMOS image frames. A ‘coded aperture’ can produce depth-dependent blurs that
enable one to extract depth by looking at the per-pixel defocus pattern [33].
Future works extend the ‘depth from defocus’ idea by leveraging information
theory to design an optimal lens [52, 27]. More recently, researchers have
proposed optimizing optical parameters in conjunction with a neural network
reconstruction algorithm in an ‘end-to-end’ fashion. This joint-optimization
problem is difficult to optimize due to local minima. Many works have
discussed mask parameterizations to stabilize optimization: Zernike basis [64,
10] and rotationally symmetric [25]. However, direct pixel-wise methods should
be preferred due to their expressiveness [36]. Dynamic pixel-wise masks have
been proposed as a training stabilization mechanism [50]. Specialized optics
have been explored for other applications such as super resolution [55], high-
dynamic-range imaging [40], hyper-spectral sensing [34], and privacy-
preservation [22]. To our knowledge, PSF engineering specifically for event-
based sensors has been relatively unexplored.

###  2.2 Microscopy Tracking

Originally, single-particle localization was limited to 2D dimensions, where
only the $x,y$ coordinates of an emitter are recovered [57]. Similar to works
on depth from defocus, the depth of an emitter can be recovered from 2D
measurements by considering a microscope’s PSF. A standard microscope
typically has a PSF resembling the circular Airy pattern; however, because it
spreads out quickly its depth resolving range is limited. A few engineered
PSFs—such as the double-helix PSF [46]—have since been proposed to improve the
imaging range. In particular, Shechtman _et al_. finds the optimally
informative PSF (dubbed the Fisher PSF) for a CMOS sensor to localize the 3D
position of a single emitter [52]. A few other techniques for resolving the 3D
location of particles have been proposed such as light-field-microscopy [37]
and lensless imaging [35].

Unfortunately, these techniques are limited by the sub-kilohertz readout of
conventional CMOS sensors. This hinders their use in imaging fast, dynamic
processes such as blood flow [8] and voltage signals [1]. A few ultrafast
imaging methods have also been proposed [15, 63, 38, 65] but require high-
power illumination which can be phototoxic to certain organic samples.
Recently, event cameras have been proposed as an alternative to CMOS sensors
for 2D SMLM [9]. Another work proposes extending light-field-microscopy to
event cameras to resolve 3D position but requires complex optical setups and
sacrifices spatial resolution [20]. By designing optics to encode depth
information into event streams, we can enable high-speed 3D SMLM.

###  2.3 Depth Estimation

Extracting 2D information from images tends to be a significantly easier task
than extracting depth, hence, monocular depth estimation is often the
bottleneck in 3D tracking performance. Structured light projectors [16] or
time-of-flight sensors [13] use active illumination to extract depth
information. Given these methods’ reliance on an internal light source,
performance can degrade in adverse lighting conditions. If we allow multiple
views, stereo [21] or structure from motion [61] can triangulate 3D position.
These methods are sensitive to occlusion and texture-less scenes and require
multiple calibrated cameras. Many neural network approaches with all-in-focus
CMOS images as input have been proposed [59, 68, 47, 67]. Recently, event-
based depth estimation has made significant progress with neural networks [72,
26, 42, 44, 53]. Spiking neural networks have been proposed for spiking
cameras, which similar to event cameras, offer asynchronous readout of pixels
[69].

##  3 Theory

###  3.1 Event Camera Simulation

Let $(x(t),y(t),z(t))$ be the location of a point light source at time $t$. We
focus on tracking points around some focal plane $z$, with $z(t)=z+\Delta
z(t)$ and $z\gg|z(t)|$. In this context, a pin-hole camera would capture,

$\displaystyle I_{t}(u,v)$ $\displaystyle=\delta\left(u-f\frac{x(t)}{z+\Delta
z(t)},v-f\frac{y(t)}{z+\Delta z(t)}\right)$ (1)
$\displaystyle\approx\delta\left(u-\frac{f}{z}x(t),v-\frac{f}{z}y(t)\right)$
(2)

where $\delta$ is the Dirac Delta function. Because $f$ and $z$ are constant,
we will consider $x(t)$ and $y(t)$ pre-scaled for notation sake. In practice,
a camera captures a blurry image depending on the point-spread-function (PSF)
it induces. A PSF $h$ can be modeled with Fourier optics theory as a function
of 3D-position $x,y,z$, amplitude modulation $A$ caused by blocking light, and
phase modulation $\phi^{M}$ caused by phase mask height variation [18].

$\displaystyle
h=\left|{\mathcal{F}}\left[A\exp\left(i\phi^{DF}(x,y,z)+i\phi^{M}\right)\right]\right|^{2}$
(3)

where $\phi^{DF}(x,y,z)$ is the defocus aberration due to the distance from
the camera. Then, a point light source at location $(x(t),y(t),z(t))$ captured
by a regular camera is

$\displaystyle I^{b}_{t}(u,v)$ $\displaystyle=[h_{z(t)}*I_{t}](u,v)$ (4)
$\displaystyle=h(x(t),y(t);z(t)).$ (5)

Note that because this PSF depends on depth, it can be used to encode depth
information into $I^{b}$. Event cameras trigger events with respect to the log
of photocurrent $L=\log(I^{b})$ [14] where a pixel’s photocurrent is linearly
related to the wave intensity at that pixel. Specifically, an event is
triggered when the absolute difference between the current intensity at
$t+\tau$ and the reference intensity from $t$, $\Delta
L(u,v)=L_{t+\tau}(u,v)-L_{t}(u,v)$, is greater than some threshold $T$.

$O_{t}(u,v)=\begin{cases}+1&\Delta L(u,v)>T\\\ -1&\Delta L(u,v)<-T\\\
\textnormal{none}&\textnormal{otherwise}\end{cases}$ (6)

In isolation, each event contains little information; however, a sequence of
events can be highly informative [31, 54, 2]. Notably the inceptive event
time-surfaces representation suggests the trailing events that occur after the
first event correspond to the log-intensity change [6]. Therefore, by binning
events over time, one can approximately recover the change in log intensity
$\Delta L$. Visuallly, we show the accumulated event frame approaches $\Delta
L$ as the number of intermediate frames accumulated increases in Figure 2. We
prove this approximation is at most off by $1$ for an idealized event camera
in Section S4 of the supplement. Therefore, our event measurement (6) can be
simplified as,

$\displaystyle O_{t}$
$\displaystyle=\log\left(I_{t}^{b}\right)-\log\left(I^{b}_{t-\tau}\right).$
(7)

Figure 2:  Binning events approximates the log difference as the number of
accumulated frames increases.  Consider a point source moving from the blue
location to the red location at depth plane $1\mu$m over a fixed time interval
in the first image. The second image illustrates the direct access to the
difference in (7), while the subsequent images demonstrate the effect of
accumulating $N$ event frames across the time interval. Observe how large $N$
nearly recovers $\Delta L$, demonstrating the validity of the approximation.

###  3.2 Information

In the field of statistical information theory, the Fisher Information (FI)
reports the amount of information gained about the parameters of a
distribution, given a measurement. As such, we can use FI to express the
effectiveness of PSFs at encoding depth information. The multi-parameter FI is
represented as an $N\times N$ matrix where the $i,j$ entry is defined as the
variance of the score:

$\displaystyle\mathcal{I}(\theta)_{i,j}=\operatorname*{\mathbb{E}}{\left[{\left(\frac{\partial}{\partial\theta_{i}}\log{f(X;\theta)}\right)\left(\frac{\partial}{\partial\theta_{j}}\log{f(X;\theta)}\right)\mid\theta}\right]}$
(8)

where $\theta$ is the set of parameters, $\theta_{i}$ is the $i$th parameter,
and $f(X;\theta)$ is a probability density function for the distribution
observation $X$ is drawn from.

For traditional CMOS sensors, FI has been used to compare coded apertures and
phase masks for a wide range of tasks such as depth estimation [45], hyper-
spectral imaging [5], and detecting linear structures [17]. Those works have
shown that the intrinsic photon shot noise in $I^{b}$ can be modeled as a
Poisson random variable with mean $\lambda=h(x,y,z)$. We derive the FI matrix
for an event sensor.

Flashing light. As a warm-up, consider the SMLM technique for event cameras
presented in [9], which assumes a blinking labeling model similar to STORM
(stochastic optical reconstruction microscopy) [49], PALM (photoactivated
localization microscopy) [7] and DNA-PAINT (DNA point accumulation for imaging
in nano-scale topography) [51]. With this idealized model of an event camera,
$\log I^{b}_{t-\tau}=0$, so (7) reduces to

$\displaystyle O_{t}=\log I^{b}_{t}.$ (9)

By applying $e^{x}$ to the measurement, we can indirectly measure $I^{b}_{t}$.
Moreover, by applying standard results for FI of a Poisson distribution [58,
28], we can write the FI matrix for an event camera capturing a blinking
particle as:

$\mathcal{I}(\theta)_{i,j}=\sum_{n}^{N}\frac{1}{h(n)+\beta}\left(\frac{\partial
h(n)}{\partial\theta_{i}}\right)\left(\frac{\partial
h_{z}(n)}{\partial\theta_{j}}\right)$ (10)

where $N$ is the number of pixels, $h(n)$ is the PSF intensity at pixel $n$,
$\beta$ is background noise, and $\theta=\\{x,y,z\\}$ corresponds to the 3D
location of a point source. Notice that this is the same result as in [45],
suggesting that — in the context of blinking particles — the Fisher mask found
in [52] for a traditional CMOS camera is also optimal for an event-based
sensor.

Generalization. We now derive the positional information content for any event
measurement. Rewriting (7) with logarithmic rules, we obtain,

$\displaystyle O_{t}=\log{\frac{I^{b}_{t}}{I^{b}_{t-\tau}}}.$ (11)

The inner expression is drawn from the ratio of Poisson random variables with
means $\lambda_{t}$ and $\lambda_{t-\tau}$. This can be approximated as a
single Normal distribution [19]:

$\displaystyle\frac{I^{b}_{t}}{I^{b}_{t-\tau}}\sim{\mathcal{N}}\left(\frac{\lambda_{t}}{\lambda_{t-\tau}},\frac{\lambda_{t}}{\lambda_{t-\tau}^{2}}+\frac{\lambda_{t}^{2}}{\lambda_{t-\tau}^{3}}\right).$
(12)

Similar to the flashing light example, we can exponentiate the measurement to
recover this ratio. Using the symbolic mathematics solver SymPy [41], we
evaluate the expectation in (10) with
$\theta=\\{x_{t},y_{t},z_{t},x_{t-\tau},y_{t-\tau},z_{t-\tau}\\}$ and
$f(X;\theta)$ as the PDF of the normal distribution, yielding

$\displaystyle\mathcal{I}(\theta)=\sum_{n}^{N}\frac{\mathcal{D}^{T}\mathcal{D}}{2\left(\mu+\nu\right)^{2}}\odot\begin{bmatrix}a&a&a&b&b&b\\\
a&a&a&b&b&b\\\ a&a&a&b&b&b\\\ b&b&b&c&c&c\\\ b&b&b&c&c&c\\\
b&b&b&c&c&c\end{bmatrix}$ (13)

where

$\displaystyle\mu$
$\displaystyle=\lambda_{t-\tau}=h(x_{t-\tau},y_{t-\tau},z_{t-\tau})+\beta$
(14) $\displaystyle\nu$ $\displaystyle=\lambda_{t}=h(x_{t},y_{t},z_{t})+\beta$
(15) $\displaystyle\mu_{i}$
$\displaystyle=\frac{\partial}{\partial\theta_{i}}\mu$ (16)
$\displaystyle\nu_{i}$ $\displaystyle=\frac{\partial}{\partial\theta_{i}}\nu$
(17) $\displaystyle\mathcal{D}$
$\displaystyle=\begin{bmatrix}\mu_{x}/\mu&\mu_{y}/\mu&\mu_{z}/\mu&\nu_{x}/\nu&\nu_{y}/\nu&\nu_{z}/\nu\end{bmatrix}$
(18) $\displaystyle a$
$\displaystyle=2\mu^{2}\nu+4\mu^{2}+2\mu\nu^{2}+12\mu\nu+9\nu^{2}$ (19)
$\displaystyle b$
$\displaystyle=-\left(2\mu^{2}\nu+2\mu^{2}+2\mu\nu^{2}+7\mu\nu+6\nu^{2}\right)$
(20) $\displaystyle c$
$\displaystyle=2\mu^{2}\nu+\mu^{2}+2\mu\nu^{2}+4\mu\nu+4\nu^{2}$ (21)

##  4 Method

(a)

(b)

Figure 3: System overview. (a) An MLP produces a phase or amplitude mask based
on a grid of $x,y$ coordinates. The weights are updated through back-
propagation of the CRB computed with Brownian Motion. (b) In simulation, coded
events are generated by first rendering high-frame-rate coded CMOS frames and
converting them to event frames. These measurements are passed to a
3D-tracking algorithm.

###  4.1 Objective Function

Similar to existing work on 3D tracking for CMOS sensors, we can leverage the
FI matrix to optimize optical parameters that efficiently encode depth
information [52, 64]. Specifically, we compute the Cramér Rao Bound (CRB),
which provides a fundamental bound on how accurately parameters can be
estimated given a measurement. If $T(X)$ is the unbiased estimator for
parameters $\theta$, then the CRB is

$\displaystyle
CRB_{i}\equiv{\left[{\mathcal{I}(\theta)^{-1}}\right]}_{i}\leq\text{cov}_{\theta}\left(T(X)\right)_{i}.$
(22)

Then, the objective function we wish to minimize is

$\displaystyle\mathcal{L}_{CRB}=\sum_{z\in
Z}\sum_{i\in\theta}\sqrt{{\left[{\mathcal{I}(\theta)^{-1}}\right]}_{i,i}}$
(23)

where $Z$ is a set of depth planes.

###  4.2 Optical Parameter Representation

PSF manipulation is typically achieved through designed optical elements such
as phase and amplitude masks. In general, phase masks are preferred over
binary amplitude masks for their photon efficiency and continuous parametric
representation, allowing for optimization via standard gradient descent
methods. Inspired by [12], we demonstrate that implicit neural representations
can model phase masks in such a way that results in more stable optimization
and better-optimized mask designs. We use an architecture similar to the
sinusoidal representation network (SIREN) presented in [56] to predict the
phase delay caused by the mask at each location $(u,v)$. Input data in
${\mathbb{R}}^{2}$ is processed by a four-layer multi-layer perceptron (MLP)
with hidden feature size $128$, and $\sin$ activation. We refer to this method
as Neural Phase Mask (NPM).

Phase masks offer many degrees of freedom and excellent light throughput, but
can be relatively expensive to manufacture and are only effective for some
frequencies. Meanwhile binary amplitude masks are cheap to manufacture (such
as with consumer-grade 3D printers) and can operate across all frequencies
(including x-ray), but offer fewer degrees of freedom.

Historically, methods for designing optimal binary apertures have been
fundamentally limited due to the lack of optimization techniques for discrete
binary parameters. As a result, prior works [33, 71, 70] walk over a
restricted search space, leaving ample room for improvement. To solve this
issue, we propose a novel implicit neural representation for binary amplitude
masks. We use an MLP to predict the percent of photons blocked at each mask
location $(u,v)$. The input in ${\mathbb{R}}^{2}$ is processed by a four-layer
MLP with hidden feature size $128$ and SoftPlus [43] activation. The output to
the network is passed through a sigmoid. We refer to this method as Neural
Amplitude Mask (NAM).

##  5 Experimental Details

PSFs are simulated for a microscope imaging system with NA$=1.4$, index of
refraction $n=1.518$, wavelength $\lambda=550$nm, magnification $M=111.11$,
$4$f lens focal length $f=150$mm, pixel pitch of $49.58\mu$m, and resolution
of $256\times 256$. Each phase and amplitude mask is optimized using
$\mathcal{L}_{CRB}$ for 10,000 epochs. Because particle motion influences FI,
we leverage Monte Carlo sampling while training to maximize information
content for all motion directions. For each epoch, we compute the total CRB
for $3$ random orthogonal motions across $11$ depth planes. We use the Adam
[29] optimizer with parameters $\beta_{1}=0.99$, $\beta_{2}=0.999$, and a
learning rate of $10^{-3}$. Training and testing were conducted on NVIDIA RTX
A5000 GPUs.

To validate our design’s ability to track point sources, we train a
Convolutional Neural Network (CNN) to map binned event frames to 3D locations.
Events are accumulated over $16$ refresh cycles to produce an accumulated
event frame. These $256\times 256$ single-channel images are processed by a
CNN with 5 convolutional blocks and a linear output head. Each block is
followed by batch normalization, ELU activation [11], and max pooling. The
output is a normalized length $3$ vector representing the position of the
particle at a given time step. The CNN is trained on $3$ Brownian motion
trajectories. Each trajectory is sampled at 16,000 time steps. A ‘coded’ CMOS
video frame is simulated by blurring a $300$nm emitter with the optical
component’s PSF for the location and adding Gaussian noise (to simulate other
noise sources such as thermal). Next, we generate a ‘coded-event-stream’ from
the high-speed video using standard event camera simulator methods by tracking
the per-pixel reference signal [23]. Finally, we bin every $16$ frames to
produce a $1000$-frame ‘coded-event-video’. The particle location at the end
of the $16$-frame bin is considered the ground truth position. We supplement
this training with $2000$ random starting positions and corresponding motion
vectors. Each motion is scaled to have magnitude drawn from
${\mathcal{N}}\left(100\text{nm},20\text{nm}\right)$. For each position-motion
pair, we generate a $16$ frame ‘coded’ CMOS video to accumulate into a ‘coded-
event-frame’. The CNN is trained for 100 epochs with the Adam optimizer. We
also manufacture a lab prototype to the demonstrate practical benefits of
coded apertures for event cameras (see Section S1 in the supplementary
materials for details).

##  6 Results

Because designed optics for event cameras is an emerging field, we compare our
optimized phase and amplitude mask designs to components designed for
traditional CMOS sensors: open aperture/Fresnel lens, Fisher phase mask [52]
and Levin _et al_.’s amplitude mask [33] (Figure 4).

Figure 4: Visualization of non-event camera-specific optical components. Each
component is placed in the same plane as a $150$mm focal length lens.

###  6.1 Cramér Rao Bound

We simulate Brownian motion by sampling $1000$ unit direction vectors and
independently scaling them by a magnitude drawn from
${\mathcal{N}}\left(100\text{nm},20\text{nm}\right)$. The speed is relative to
the event camera refresh rate, with a $1000$ accumulated-event-frame per
second system, this motion simulates a range of biological processes such as
molecular diffusion [66]. We then evaluate the average CRB over the $1000$
motions at $30$ depth planes spaced evenly on a $3\mu$m range around the focal
plane. For all $6$ position parameters, we plot the CRB trend with respect to
depth (Figure 5). Observe that each optical system performs worse as a point
source moves away from the focal plane as the defocus change decreases.
Although an open-aperture lens is slightly better around the focal plane, its
bound increases at a higher rate than the other designs. We also report the
average CRB over all parameters and depth slices to demonstrate our neural-
based phase mask is best overall (Table 1).

| Component | CRB (nm) $\downarrow$
---|---|---
| Open Aperture | 80.8
Amplitude | Levin _et al_. | 263.3
NAM (Ours) | 50.5
Phase | Fisher | 36.3
NPM (Ours) | 33.1
Table 1:  Average CRB for each optical component across a $3\mu$m depth range
for all $6$ position parameters. Phase masks outperform amplitude masks due to
higher light efficiency, and our neural-designed phase mask is best.  Figure
5: 3D localization CRB with respect to depth. First row: particle’s $x,y,z$
position at time $t-\tau$. Second row: particle’s $x,y,z$ position at time
$t$. Observe the bound increases as the source drifts from the focal plane.

###  6.2 3D Tracking

We validate our theoretical results in simulation by tracking a 3D moving
emitter across a $8\mu\text{m}\times 8\mu\text{m}\times 4\mu\text{m}$ volume.
After training a CNN to decode 3D position from coded event frames, we
evaluate our network tracking performance on $5$ sequences of Brownian motion,
each consisting of $1000$ binned frames. Table 2 shows our event camera-
specific optical designs minimize 3D tracking error more than conventional
designs. Additionally, our method is substantially better at depth plane
recovery. Qualitative results in Figure 6 demonstrate that 3D positions
recovered using our designs more tightly fit ground-truth trajectories.

|  | RMSE (nm) $\downarrow$ |  $L_{1}$ (nm) $\downarrow$
---|---|---|---
| Component | 3D | $z$
| Open Aperture | 617 | 936
Amplitude | Levin _et al_. | 764 | 1036
NAM (Ours) | 66.0 | 49.2
Phase | Fisher | 52.6 | 44.2
NPM (Ours) | 51.2 | 39.2
Table 2:  Tracking accuracy comparison. We present quantitative results on 3D
trajectory recovery for known optical designs. Our event CRB loss function
found the best-performing design. Although only slightly improved in overall
3D tracking, our design noticeably improves depth recovery.  Figure 6:
Recovered 3D position over Brownian motion sequence with coded event frames.
Left: phase mask methods, right: amplitude mask methods. Observe trajectories
reconstructed from phase mask-coded events more closely align with ground-
truth positions. Units in microns.

##  7 Ablation Studies

###  7.1 Optical Representations

Additionally, we compare 3D tracking results using two different amplitude
mask representations: pixel-wise and neural amplitude mask (Figure 7) and
three different phase mask representations: pixel-wise, Zernike basis, and
neural phase mask (Figure 8). As shown in Table 3, our implicit neural
representation-based methods achieve a lower average error bound than
alternative representations, despite being two times smaller than pixel-wise
representations with respect to the number of parameters. As expected, phase
mask results generally outperform the amplitude mask results (Figure 9).
However, our novel neural binary aperture makes optimizing amplitude masks
more tractable. We observe that pixel-wise representations not only yield
difficult-to-manufacture apertures but also suboptimal performance. In terms
of 3D tracking, the implicit neural representations produce a smaller error on
average (Table 4) and more accurately match sampled 3D trajectories (Figure
10).

Figure 7: Designed amplitude masks and corresponding PSFs. Top: pixel-wise representation. Bottom: implicit neural representation. Figure 8: Designed phase masks and corresponding PSFs. Top: pixel-wise representation. Middle: first 55 Zernike coefficients representation. Bottom: implicit neural representation. | Representation | CRB (nm) $\downarrow$
---|---|---
Amplitude | Pixel-Wise | 65.5
NAM | 50.5
Phase | Pixel-Wise | 34.2
Zernike | 34.8
NPM | 33.1
Table 3:  Average CRB of different optimized representations across a $3\mu$m depth range. Notice the neural representations outperform their pixel-wise counterparts.  Figure 9: Effect of optical parameterization on 3D localization CRB. First row: particle’s $x,y,z$ position at time $t-\tau$. Second row: particle’s $x,y,z$ position at time $t$. Our implicit neural representations are particularly advantageous for amplitude masks. |  | RMSE (nm) $\downarrow$ |  $L_{1}$ (nm) $\downarrow$
---|---|---|---
| Component | 3D | $z$
Amplitude | Pixel-Wise | 120 | 103
NAM | 66.0 | 49.2
Phase | Pixel-Wise | 56.5 | 45.9
Zernike | 51.3 | 50.2
| Our NPM | 51.2 | 39.2
Table 4:  Effect of optimized mask parameterization on tracking accuracy.
Average distance between ground-truth Brownian motion and the recovered 3D
position is minimized with our neural-based designs. Figure 10: Effect of
optical representation on 3D trajectory recovery. Left: phase mask methods,
right: amplitude mask methods. Observe that neural representations produce
tighter reconstructions. Units in microns.

###  7.2 Tracking Limits

In this section, we explore the limits of 3D tracking with variable external
factors. For each experiment, we compute the average CRB over $30$ depth
slices and $6$ parameters for $3$ orthogonal unit directions ($x$, $y$, and
$z$). First, as the number of available photons increases, the lower bound on
3D position estimation monotonically decreases (Figure 11). More available
photons equate to a higher signal-to-noise ratio. Additionally, this result
helps explain why phase masks outperform amplitude masks. Second, we show
extremely slow-moving particles (less than nanometers per refresh rate)
experience a significantly higher CRB (Figure 12). Minimal movement indicates
smaller intensity changes and thus an event camera would trigger fewer events.
On the other side, as a particle moves faster, the number of events will
decrease as there is a non-zero delay between when an event camera can trigger
sequential events. Our learned phase mask is more robust to speed changes than
an open aperture and our learned amplitude mask. Third, when the percentage of
photons due to background noise increases, the bound on error also increases
(Figure 13). We design our masks with $1\%$ of captured photons attributable
to the background, but the learned designs are more resistant to degraded
conditions than an open aperture.

We also explore the effect of modifying the accumulation period in Section S2
and how the optimal design changes with respect to speed in Section S3 of the
supplement.

Figure 11:  Flux effect on CRB. With more available photons, the signal-to-
noise ratio increases, so the 3D information content is more reliable, and the
bound on 3D tracking error decreases.  Figure 12:  Speed effect on CRB. Too-
slow moving particles trigger fewer events yielding a worse CRB. Similarly, as
a particle moves faster the delay between triggers leads to fewer events.
Figure 13:  Background photon effect on CRB. As the percentage of photons
hitting the sensor due to background noise increases, CRB also increases. The
impact is minimal in our method.

##  8 Limitations

While we were successful in designing optics to improve performance on 3D
tracking with event cameras, our method carries some limitations. First,
although our binned event frames can be obtained at kHz refresh rates, they do
not take full advantage of the asynchronous nature of event cameras. Second,
our bounds are for an idealized event camera model with no read-noise. It
would be impossible to outperform these bounds, but there might exist a
tighter bound that accounts for these hardware imperfections. Lastly, we only
consider single-emitter images. With multiple point sources, the resolving
accuracy between single points may be more limited.

##  9 Conclusion

This work introduces PSF-engineering to neuromorphic event-based sensors. We
first derive information theoretical limits on 3D point localization and
tracking. We demonstrate that existing amplitude and phase mask designs are
suboptimal for tracking moving emitters and design new optical elements for
this task. Additionally, to overcome the non-convexity of this optimization
problem, we introduce a novel implicit neural representation for optical
components. Finally, we validate the effectiveness of our designs in
simulation and compare against state-of-the-art mask designs. Our work unlocks
not only highly performant optics for event cameras but also the ability to
design highly expressive elements for other sensors.

## Acknowledgements

This work was supported in part by the Joint Directed Energy Transition
Office, AFOSR Young Investigator Program award no. FA9550-22-1-0208, ONR award
no. N00014-23-1-2752 and N00014-17-1-2622, Dolby Labs, SAAB, Inc, and National
Science Foundation grants BCS 1824198 and CNS 1544787. The support of the
Maryland Robotics Center under a postdoctoral fellowship to C.S., is also
gratefully acknowledged.

## References

  * Abdelfattah et al. [2023] Ahmed S. Abdelfattah, Jihong Zheng, Amrita Singh, Yi-Chieh Huang, Daniel Reep, Getahun Tsegaye, Arthur Tsang, Benjamin J. Arthur, Monika Rehorova, Carl V. L. Olson, Yichun Shuai, Lixia Zhang, Tian-Ming Fu, Daniel E. Milkie, Maria V. Moya, Timothy D. Weber, Andrew L. Lemire, Christopher A. Baker, Natalie Falco, Qinsi Zheng, Jonathan B. Grimm, Mighten C. Yip, Deepika Walpita, Martin Chase, Luke Campagnola, Gabe J. Murphy, Allan M. Wong, Craig R. Forest, Jerome Mertz, Michael N. Economo, Glenn C. Turner, Minoru Koyama, Bei-Jung Lin, Eric Betzig, Ondrej Novak, Luke D. Lavis, Karel Svoboda, Wyatt Korff, Tsai-Wen Chen, Eric R. Schreiter, Jeremy P. Hasseman, and Ilya Kolb.  Sensitivity optimization of a rhodopsin-based fluorescent voltage indicator.  _Neuron_ , 111(10):1547–1563.e9, 2023.
  * Alzugaray and Chli [2018] Ignacio Alzugaray and Margarita Chli.  Asynchronous corner detection and tracking for event cameras in real time.  _IEEE Robotics and Automation Letters_ , 3(4):3177–3184, 2018.
  * Amir et al. [2017] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha.  A low power, fully event-based gesture recognition system.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Angelopoulos et al. [2021] Anastasios N. Angelopoulos, Julien N.P. Martel, Amit P.S. Kohli, Jorg Conradt, and Gordon Wetzstein.  Event based, near-eye gaze tracking beyond 10,000 hz.  _IEEE Transactions on Visualization and Computer Graphics (Proc. VR)_ , 2021.
  * Baek et al. [2021] Seung-Hwan Baek, Hayato Ikoma, Daniel S. Jeon, Yuqi Li, Wolfgang Heidrich, Gordon Wetzstein, and Min H. Kim.  Single-shot hyperspectral-depth imaging with learned diffractive optics.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2651–2660, 2021.
  * Baldwin et al. [2019] R. Wes Baldwin, Mohammed Almatrafi, Jason R. Kaufman, Vijayan Asari, and Keigo Hirakawa.  Inceptive event time-surfaces for object classification using neuromorphic cameras.  In _Image Analysis and Recognition - 16th International Conference, ICIAR 2019, Proceedings_ , pages 395–403, Germany, 2019. Springer Verlag.
  * Betzig et al. [2006] Eric Betzig, George H. Patterson, Rachid Sougrat, O. Wolf Lindwasser, Scott Olenych, Juan S. Bonifacino, Michael W. Davidson, Jennifer Lippincott-Schwartz, and Harald F. Hess.  Imaging intracellular fluorescent proteins at nanometer resolution.  _Science_ , 313(5793):1642–1645, 2006.
  * Bouchard et al. [2009] Matthew B Bouchard, Brenda R Chen, Sean A Burgess, and Elizabeth M C Hillman.  Ultra-fast multispectral optical imaging of cortical oxygenation, blood flow, and intracellular calcium dynamics.  _Opt Express_ , 17(18):15670–15678, 2009.
  * Cabriel et al. [2023] Clément Cabriel, Tual Monfort, Christian G. Specht, and Ignacio Izeddin.  Event-based vision sensor for fast and dense single-molecule localization microscopy.  _Nature Photonics_ , 2023.
  * Chang and Wetzstein [2019] Julie Chang and Gordon Wetzstein.  Deep optics for monocular depth estimation and 3d object detection.  In _Proc. IEEE ICCV_ , 2019.
  * Clevert et al. [2016] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus), 2016.
  * Feng et al. [2023] Brandon Y. Feng, Haiyun Guo, Mingyang Xie, Vivek Boominathan, Manoj K. Sharma, Ashok Veeraraghavan, and Christopher A. Metzler.  Neuws: Neural wavefront shaping for guidestar-free imaging through static and dynamic scattering media.  _Science Advances_ , 9(26):eadg4671, 2023.
  * Foix et al. [2011] Sergi Foix, Guillem Alenya, and Carme Torras.  Lock-in time-of-flight (tof) cameras: A survey.  _IEEE Sensors Journal_ , 11(9):1917–1926, 2011\.
  * Gallego et al. [2020] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al.  Event-based vision: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(1):154–180, 2020.
  * Gao et al. [2014] Liang Gao, Jinyang Liang, Chiye Li, and Lihong V. Wang.  Single-shot compressed ultrafast photography at one hundred billion frames per second.  _Nature_ , 516(7529):74–77, 2014.
  * Geng [2011] Jason Geng.  Structured-light 3d surface imaging: a tutorial.  _Adv. Opt. Photon._ , 3(2):128–160, 2011.
  * Ghanekar et al. [2022] Bhargav Ghanekar, Vishwanath Saragadam, Dushyant Mehra, Anna-Karin Gustavsson, Aswin C. Sankaranarayanan, and Ashok Veeraraghavan.  Ps 2 f: Polarized spiral point spread function for single-shot 3d sensing.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–12, 2022.
  * Goodman [2017] Joseph W. Goodman.  _Introduction to fourier optics_.  Freeman, 2017.
  * Griffin [1992] Tralissa F Griffin.  Distribution of the ratio of two poisson random variables.  Master’s thesis, Texas Tech University, 1992.
  * Guo et al. [2023] Ruipeng Guo, Qianwan Yang, Andrew S. Chang, Guorong Hu, Joseph Greene, Christopher V. Gabel, Sixian You, and Lei Tian.  Eventlfm: Event camera integrated fourier light field microscopy for ultrafast 3d imaging, 2023.
  * Hartley and Zisserman [2004] R. I. Hartley and A. Zisserman.  _Multiple View Geometry in Computer Vision_.  Cambridge University Press, ISBN: 0521540518, second edition, 2004.
  * Hinojosa et al. [2021] Carlos Hinojosa, Juan Carlos Niebles, and Henry Arguello.  Learning privacy-preserving optics for human pose estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2573–2582, 2021.
  * Hu et al. [2021] Y Hu, S C Liu, and T Delbruck.  v2e: From video frames to realistic DVS events.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 2021.
  * Iaboni et al. [2021] Craig Iaboni, Himanshu Patel, Deepan Lobo, Ji-Won Choi, and Pramod Abichandani.  Event camera based real-time detection and tracking of indoor ground robots.  _IEEE Access_ , 9:166588–166602, 2021.
  * Ikoma et al. [2021] Hayato Ikoma, Cindy M. Nguyen, Christopher A. Metzler, Yifan Peng, and Gordon Wetzstein.  Depth from defocus with learned optics for imaging and occlusion-aware depth estimation.  _IEEE International Conference on Computational Photography (ICCP)_ , 2021.
  * Javier Hidalgo-Carrio and Scaramuzza [2020] Daniel Gehrig Javier Hidalgo-Carrio and Davide Scaramuzza.  Learning monocular dense depth from events.  _IEEE International Conference on 3D Vision.(3DV)_ , 2020.
  * Jusuf and Lew [2022] James M. Jusuf and Matthew D. Lew.  Towards optimal point spread function design for resolving closely spaced emitters in three dimensions.  _Opt. Express_ , 30(20):37154–37174, 2022.
  * Kay [1993] Steven M. Kay.  _Fundamentals of Statistical Signal Processing_.  Prentice-Hall, 1 edition, 1993.
  * Kingma and Ba [2015] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , San Diega, CA, USA, 2015.
  * Kinkhabwala et al. [2009] Anika Kinkhabwala, Zongfu Yu, Shanhui Fan, Yuri Avlasevich, Klaus Müllen, and W. E. Moerner.  Large single-molecule fluorescence enhancements produced by a bowtie nanoantenna.  _Nature Photonics_ , 3(11):654–657, 2009.
  * Lagorce et al. [2017] Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E Shi, and Ryad B Benosman.  Hots: A hierarchy of event-based time-surfaces for pattern recognition.  _IEEE Trans Pattern Anal Mach Intell_ , 39(7):1346–1359, 2017.
  * Lee et al. [2014] Jun Haeng Lee, Tobi Delbruck, Michael Pfeiffer, Paul K J Park, Chang-Woo Shin, Hyunsurk Eric Ryu, and Byung Chang Kang.  Real-time gesture interface based on event-driven processing from stereo silicon retinas.  _IEEE Trans Neural Netw Learn Syst_ , 25(12):2250–2263, 2014.
  * Levin et al. [2007] Anat Levin, Rob Fergus, Frédo Durand, and William T Freeman.  Image and depth from a conventional camera with a coded aperture.  _ACM transactions on graphics (TOG)_ , 26(3):70–es, 2007.
  * Li et al. [2022] Lingen Li, Lizhi Wang, Weitao Song, Lei Zhang, Zhiwei Xiong, and Hua Huang.  Quantization-aware deep optics for diffractive snapshot hyperspectral imaging.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 19748–19757, 2022.
  * Liu et al. [2020] Fanglin Linda Liu, Grace Kuo, Nick Antipa, Kyrollos Yanny, and Laura Waller.  Fourier diffuserscope: single-shot 3d fourier light field microscopy with a diffuser.  _Opt. Express_ , 28(20):28969–28986, 2020.
  * Liu et al. [2022] Xin Liu, Linpei Li, Xu Liu, Xiang Hao, and Yifan Peng.  Investigating deep optics model representation in affecting resolved all-in-focus image quality and depth estimation fidelity.  _Opt. Express_ , 30(20):36973–36984, 2022.
  * Llavador et al. [2016] A. Llavador, J. Sola-Pikabea, G. Saavedra, B. Javidi, and M. Martínez-Corral.  Resolution improvements in integral microscopy with fourier plane recording.  _Opt. Express_ , 24(18):20792–20798, 2016.
  * Ma et al. [2021] Yayao Ma, Youngjae Lee, Catherine Best-Popescu, and Liang Gao.  High-speed compressed-sensing fluorescence lifetime imaging microscopy of live cells.  _Proceedings of the National Academy of Sciences_ , 118(3):e2004176118, 2021.
  * Maynard et al. [2021] Stephanie A Maynard, Philippe Rostaing, Natascha Schaefer, Olivier Gemin, Adrien Candat, Andréa Dumoulin, Carmen Villmann, Antoine Triller, and Christian G Specht.  Identification of a stereotypic molecular arrangement of endogenous glycine receptors at spinal cord synapses.  _eLife_ , 10:e74441, 2021.
  * Metzler et al. [2020] Christopher A. Metzler, Hayato Ikoma, Yifan Peng, and Gordon Wetzstein.  Deep optics for single-shot high-dynamic-range imaging.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Meurer et al. [2017] Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Štěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz.  Sympy: symbolic computing in python.  _PeerJ Computer Science_ , 3:e103, 2017.
  * Mostafavi et al. [2021] Mohammad Mostafavi, Kuk-Jin Yoon, and Jonghyun Choi.  Event-intensity stereo: Estimating depth by the best of both worlds.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 4258–4267, 2021.
  * Nair and Hinton [2010] Vinod Nair and Geoffrey E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Proceedings of the 27th International Conference on International Conference on Machine Learning_ , page 807–814, Madison, WI, USA, 2010. Omnipress.
  * Nam et al. [2022] Yeongwoo Nam, Mohammad Mostafavi, Kuk-Jin Yoon, and Jonghyun Choi.  Stereo depth from events cameras: Concentrate and focus on the future.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6114–6123, 2022.
  * Ober et al. [2004] Raimund J Ober, Sripad Ram, and E Sally Ward.  Localization accuracy in single-molecule microscopy.  _Biophysical journal_ , 86(2):1185–1200, 2004\.
  * Pavani et al. [2009] Sri Rama Prasanna Pavani, Michael A. Thompson, Julie S. Biteen, Samuel J. Lord, Na Liu, Robert J. Twieg, Rafael Piestun, and W. E. Moerner.  Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function.  _Proceedings of the National Academy of Sciences_ , 106(9):2995–2999, 2009.
  * Ranftl et al. [2022] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(3), 2022.
  * Rodríguez-Gómez et al. [2022] Juan Pablo Rodríguez-Gómez, Raul Tapia, Maria del Mar Guzmán Garcia, Jose Ramiro Martínez-de Dios, and Anibal Ollero.  Free as a bird: Event-based dynamic sense-and-avoid for ornithopter robot flight.  _IEEE Robotics and Automation Letters_ , 7(2):5413–5420, 2022.
  * Rust et al. [2006] Michael J Rust, Mark Bates, and Xiaowei Zhuang.  Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm).  _Nature Methods_ , 3(10):793–796, 2006.
  * Shah et al. [2023] Sachin Shah, Sakshum Kulshrestha, and Christopher A. Metzler.  Tidy-psfs: Computational imaging with time-averaged dynamic point-spread-functions.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 10657–10667, 2023.
  * Sharonov and Hochstrasser [2006] Alexey Sharonov and Robin M. Hochstrasser.  Wide-field subdiffraction imaging by accumulated binding of diffusing probes.  _Proceedings of the National Academy of Sciences_ , 103(50):18911–18916, 2006.
  * Shechtman et al. [2014] Yoav Shechtman, Steffen J. Sahl, Adam S. Backer, and W. E. Moerner.  Optimal point spread function design for 3d imaging.  _Phys. Rev. Lett._ , 113:133902, 2014.
  * Shi et al. [2023] Peilun Shi, Jiachuan Peng, Jianing Qiu, Xinwei Ju, Frank Po Wen Lo, and Benny Lo.  Even: An event-based framework for monocular depth estimation at adverse night conditions, 2023.
  * Sironi et al. [2018] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman.  Hats: Histograms of averaged time surfaces for robust event-based object classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Sitzmann et al. [2018] Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein.  End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging.  _ACM Transactions on Graphics (TOG)_ , 37(4):114, 2018.
  * Sitzmann et al. [2020] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In _Proc. NeurIPS_ , 2020.
  * Small and Stahlheber [2014] Alex Small and Shane Stahlheber.  Fluorophore localization algorithms for super-resolution microscopy.  _Nature Methods_ , 11(3):267–279, 2014.
  * Snyder and Miller [1991] Donald L. Snyder and Michael I. Miller.  _Random Point Processes in time and space_.  Springer, 2 edition, 1991.
  * Spencer et al. [2020] Jaime Spencer, Richard Bowden, and Simon Hadfield.  Defeat-net: General monocular depth via simultaneous unsupervised representation learning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Tinch et al. [2022] Luc Tinch, Nitesh Menon, Keigo Hirakawa, and Scott McCloskey.  Event-based detection, tracking, and recognition of unresolved moving objects.  _Advanced Maui Optical and Space Surveillance Technologies (AMOS) Conference_ , 2022.
  * Tomasi and Kanade [1992] Carlo Tomasi and Takeo Kanade.  Shape and motion from image streams under orthography: a factorization method.  _International Journal of Computer Vision_ , 9(2):137–154, 1992.
  * Verdier et al. [2022] Hippolyte Verdier, François Laurent, Alhassan Cassé, Christian L. Vestergaard, Christian G. Specht, and Jean-Baptiste Masson.  A maximum mean discrepancy approach reveals subtle changes in $\alpha$-synuclein dynamics.  _bioRxiv_ , 2022.
  * Wu et al. [2020] Jianglai Wu, Yajie Liang, Shuo Chen, Ching-Lung Hsu, Mariya Chavarha, Stephen W Evans, Dongqing Shi, Michael Z Lin, Kevin K Tsia, and Na Ji.  Kilohertz two-photon fluorescence microscopy imaging of neural activity in vivo.  _Nat Methods_ , 17(3):287–290, 2020.
  * Wu et al. [2019] Yicheng Wu, Vivek Boominathan, Huaijin Chen, Aswin Sankaranarayanan, and Ashok Veeraraghavan.  Phasecam3d — learning phase masks for passive single view depth estimation.  In _2019 IEEE International Conference on Computational Photography (ICCP)_ , pages 1–12, 2019.
  * Xiao et al. [2023] Sheng Xiao, John T. Giblin, David A. Boas, and Jerome Mertz.  High-throughput deep tissue two-photon microscopy at kilohertz frame rates.  _Optica_ , 10(6):763–769, 2023.
  * Yang and Hinner [2015] Nicole J Yang and Marlon J Hinner.  Getting across the cell membrane: an overview for small molecules, peptides, and proteins.  _Methods Mol Biol_ , 1266:29–53, 2015.
  * Yin et al. [2023] Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen.  Metric3d: Towards zero-shot metric 3d prediction from a single image.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9043–9053, 2023.
  * You et al. [2021] Zunzhi You, Yi-Hsuan Tsai, Wei-Chen Chiu, and Guanbin Li.  Towards interpretable deep networks for monocular depth estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 12879–12888, 2021.
  * Zhang et al. [2022] Jiyuan Zhang, Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang.  Spike transformer: Monocular depth estimation for spiking camera.  In _Computer Vision – ECCV 2022_ , pages 34–52, Cham, 2022. Springer Nature Switzerland.
  * Zhou and Nayar [2009] Changyin Zhou and Shree Nayar.  What are good apertures for defocus deblurring?  In _2009 IEEE International Conference on Computational Photography (ICCP)_ , pages 1–8, 2009.
  * Zhou et al. [2009] Changyin Zhou, Stephen Lin, and Shree Nayar.  Coded aperture pairs for depth from defocus.  In _2009 IEEE 12th International Conference on Computer Vision_ , pages 325–332, 2009.
  * Zhu et al. [2019] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis.  Unsupervised event-based learning of optical flow, depth, and egomotion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.

Supplementary Material

##  S1 Hardware Prototype

We performed a real-world experiment for tracking a point light source at
meter scale using a binary amplitude mask and a Prophessee EVK3 event camera.
Specifically, we fabricated the NAM mask at 20mm diameter scale on a Creality
Ender 3 S1 Pro using 1.75mm PLA filament (see Figure 1). Then, we captured an
event dataset by moving a point source at discrete depth planes ranging
between $75$cm and $125$cm with and without our coded aperture. For all
measurements, the camera was focused at $100$cm. We binned events in $1$ms
intervals to achieve an effective frame rate of $1000$ FPS and trained a CNN
to estimate the event frame’s depth. Results in Figure 2 demonstrate improved
tracking performance compared to an open aperture, particularly at depths
where the point source is defocused.

Figure 1:  Prototype. Top: The fabricated mask is placed at the aperture plane
of an event camera with a $50$mm focal length lens. Bottom: Sample captured
event frames for a point source.  Figure 2:  Real-world 3D tracking.
Comparison between NAM and Open apertures for depth estimation at 1000FPS.
Error bars show the 90% interquartile range.

##  S2 Accumulation Time

Cutting-edge event cameras offer $10$kHz fresh rates; even with $16$-frame
accumulation, the camera effectively operates at $625$FPS — much faster than
conventional CMOS sensors. We also retrained our CNN-based tracking algorithm
on ‘pure’ event frames with no accumulation. Overall performance degraded: NPM
by $+45$% RMSE and NAM by $+54$% RMSE. Alternative architectures such as
Spiking Neural Networks designed for sparse binary measurements may be better
suited for processing ‘pure’ events.

##  S3 The Effects of Particle Speed

We have shown CRB depends on particle speed; a natural question is does the
optimal design change with respect to speed. We optimize our neural phase mask
using the CRB objective function with fixed particle
speeds—$\\{50,100,500,1000\\}$nm per time step. Our learned designs are shown
in Figure 3. When a particle moves quickly relative to the binned interval,
the optimal design resembles the Fisher phase pattern found for traditional
CMOS sensors.

One can explain this collapse to the original Fisher mask design as follows.
As a particle moves faster, the captured binned event frame looks more similar
to the composition of a negative PSF at the start location and a positive PSF
at the end location (Figure 4). This suggests that single-point event tracking
mirrors two-point CMOS tracking.

Figure 3:  Designed Phase Masks and corresponding PSFs for specific speeds.
Each row visualizes the neural phase mask designed for tracking particles
moving at $N$ nanometers per time interval. Observe that the optimal design
for ‘fast’ moving particles is the Fisher design.  Figure 4:  Event camera
measurements of a moving particle with the Fisher mask.  Motion is simulated
over a fixed time interval with $100$ event samples. Observe a ‘fast’ moving
particle produces an event frame with two copies of a regular PSF: a negative
copy at the start location, and a positive copy at the end location. Row 1:
negative event count over the time interval. Row 2: positive event count over
the time interval. Row 3: the red channel visualizes negative events and the
blue channel visualizes positive events. The pink regions represent where the
events cancel in a binned measurement. Row 4: binned event frame $pos-neg$.
Row 5: log-intensity difference $\Delta L$.

##  S4 Log-Intensity Difference Approximation

In this section, we prove the log-intensity difference approximation we
consider when deriving the Cramér Rao Bound is proportional to binned event
frames.

Assume an idealized event camera model, where an event is triggered as soon as
the log-intensity change between the reference and the current intensity
equals some threshold, $\mathcal{T}$. Consider producing a binned event frame
for a time interval $[t_{\text{start}},t_{\text{end}}]$. For a single pixel,
let the sequence of events over this interval occur at times
$t_{1},t_{2},\ldots t_{n}$ and have polarities
$p_{1},p_{2},\ldots,p_{n}\in\\{-1,1\\}$. Let $f(t)$ be the log-intensity at
time $t$ for the same pixel and be continuous over the interval.

######  Lemma S4.1.

The log-intensity difference, $f(t_{\text{end}})-f(t_{\text{start}})$, is
proportional to the binned event pixel value, $\sum_{i=1}^{n}p_{i}$, with
error $|\epsilon|<1$.

$\displaystyle
f(t_{\text{end}})-f(t_{\text{start}})\propto\epsilon+\sum_{i=1}^{n}p_{i}$ (24)

###### Proof.

By assumption, the magnitude of the change corresponding to each event is
$\mathcal{T}$. Notice that $\mathcal{T}p_{i}$ is the log-intensity difference
between the previous event time (the reference) and the current event time.

$\sum_{i=1}^{n}p_{i}=\frac{1}{\mathcal{T}}\sum_{i=1}^{n}f(t_{i})-f(t_{i-1})$
(25)

The right-hand side is a telescoping sum,

$\sum_{i=1}^{n}f(t_{i})-f(t_{i-1})=f(t_{n})-f(t_{0}).$ (26)

$t_{0}=t_{\text{start}}$ because the first event must occur $t_{1}-t_{0}$
after the start of the interval. Then, the binned event frame is

$\sum_{i=1}^{n}p_{i}=\frac{1}{\mathcal{T}}\left(f(t_{n})-f(t_{\text{start}})\right).$
(27)

Finally, $|f(t_{n})-f(t_{\text{end}})|=|\delta|<\mathcal{T}$ because if the
quantity exceeded the threshold, an additional event would be triggered.
Substitute $t_{\text{end}}$ for $t_{n}$.

$\displaystyle\sum_{i=1}^{n}p_{i}$
$\displaystyle=\frac{1}{\mathcal{T}}\left(f(t_{\text{end}})-f(t_{\text{start}})+\delta\right)$
(28)
$\displaystyle=\frac{1}{\mathcal{T}}\left(f(t_{\text{end}})-f(t_{\text{start}})\right)+\epsilon$
(29)

Thus, a binned event frame can be approximated as log-intensity difference
divided by $\mathcal{T}$ with error $|\epsilon|<1$. ∎

As an event camera becomes more sensitive to change ($\mathcal{T}$ decreases),
the approximation’s percent error decreases because the magnitude of the
binned event frame increases but the total absolute error is fixed at most
$1$.