# Falcon: Accelerating Homomorphically Encrypted Convolutions for Efficient
Private Mobile
Network Inference

Tianshi Xu1,2, Meng Li1,2∗, Runsheng Wang1,3,4, Ru Huang1,3,4 This work was
supported in part by the NSFC (62125401) and the 111 Project (B18001). 1
School of Integrated Circuits, Peking University, Beijing, China 2 Institute
for Artificial Intelligence, Peking University, Beijing, China 3 Institute of
Electronic Design Automation, Peking University, wuxi, China 4 Beijing
Advanced Innovation Center for Integrated Circuits, Beijing, China
<EMAIL_ADDRESS>
###### Abstract

Efficient networks, e.g., MobileNetV2, EfficientNet, etc, achieves state-of-
the-art (SOTA) accuracy with lightweight computation. However, existing
homomorphic encryption (HE)-based two-party computation (2PC) frameworks are
not optimized for these networks and suffer from a high inference overhead. We
observe the inefficiency mainly comes from the packing algorithm, which
ignores the computation characteristics and the communication bottleneck of
homomorphically encrypted depthwise convolutions. Therefore, in this paper, we
propose Falcon, an effective dense packing algorithm for HE-based 2PC
frameworks. Falcon features a zero-aware greedy packing algorithm and a
communication-aware operator tiling strategy to improve the packing density
for depthwise convolutions. Compared to SOTA HE-based 2PC frameworks, e.g.,
CrypTFlow2, Iron and Cheetah, Falcon achieves more than $15.6\times$,
$5.1\times$ and $1.8\times$ latency reduction, respectively, at operator
level. Meanwhile, at network level, Falcon allows for $1.4$% and $4.2$%
accuracy improvement over Cheetah on CIFAR-100 and Tiny Imagenet datasets with
iso-communication, respecitvely.

###### Index Terms:

Secure Two-Party Computation, Homomorphic Encryption, Depthwise Convolution,
Convolution Packing

##  I Introduction

Recent years have witnessed the algorithmic breakthroughs as well as the
compute and parameter explosion of convolutional neural networks (CNNs).
State-of-the-art (SOTA) efficient networks, e.g., MobileNetV2 [1],
EfficientNet [2], ConvNext [3], etc, achieve superhuman accuracy with
lightweight computation. Therefore, they are getting increasingly adopted in
real-world applications, including sensitive and private tasks such as face
authentication [4], medical diagnosis [5], etc. Privacy has thus emerged as a
major concern in the network deployment and there is a growing demand for
privacy-preserving deep neural network (DNN) inference [6, 7, 8, 9].

Homomorphic encryption (HE)-based secure two-party computation (2PC) helps
solve the following dilemma: the server, who owns a private model, and the
client, who owns private data, want to jointly apply the model to the data
without giving out the model or data. HE-based 2PC not only enables the
inference but also permits a cryptographically-strong privacy protection,
attracting a lot of attention in recent years [10, 11, 12].

As CNNs process high-dimensional tensors while HE computes over polynomials,
the first step of HE-based inference is the mapping from tensors to
polynomials, whose coefficients are usually regarded as a one-dimensional
vector, denoted as packing [13]. The packing algorithm directly impacts the
inference correctness and complexity and has been widely studied to improve
the HE-based 2PC efficiency for CNNs [13, 14, 15, 10, 16, 17].

However, existing HE-based 2PC frameworks are not optimized for efficient
networks, which extensively use depthwise and group convolutions to reduce the
compute and latency [18, 1, 2, 19]. As shown in Figure 1, in two prior-art 2PC
frameworks, the latency of a depthwise convolution is almost the same as a
standard convolution, indicating a high inference inefficiency.

Figure 1: Compare a depthwise convolution with a standard convolution on
compute (measured by multiplication accumulations, i.e., MACs), latency on
GPU, as well as latency of two prior-art and our 2PC frameworks.

We observe the inefficiency comes from two limitations of the packing
algorithms in existing HE-based 2PC frameworks. On one hand, prior-art methods
cannot leverage the computation characteristics of depthwise convolutions.
They have to pad the depthwise filter with zeros and convert it back to a
dense filter during packing [15, 10], leading to low packing density and high
computation and communication overhead. On the other hand, they often ignore
the communication to transfer input and output ciphertexts are imbalanced and
only focus on optimizing the input communication [14, 15, 10].

In this paper, we propose Falcon, an effective dense packing algorithm for HE-
based 2PC frameworks for privacy-preserving efficient network inference.
Falcon features two main techniques to reduce inference latency for depthwise
and group convolutions. First, while zero padding is necessary for computation
correctness, Falcon reuses the zero channels between padded filters to improve
the overall packing density. Secondly, we find the output ciphertext transfer
often dominates the overall communication and propose to tile the input tensor
and the depthwise convolution. Although the tiling increases the input
ciphertext transfer communication slightly, it drastically improves the output
transfer efficiency. Our main contributions can be summarized as follows:

  * •

We observe the bottleneck of existing HE-based 2PC frameworks when processing
depthwise convolutions and propose Falcon to improve the inference efficiency
of SOTA lightweight networks.

  * •

We propose a zero-aware greedy packing algorithm, which formulates the
depthwise filter packing as a shortest common superstring problem and proposes
a greedy algorithm to improve the packing density.

  * •

We propose a communication-aware operator tiling algorithm and achieve further
communication reduction by balancing the input and output ciphertext transfer
communication.

  * •

Falcon outperforms SOTA HE-based 2PC frameworks, including Cheetah and Iron,
with $1.4\sim 5.4\times$ and $3.6\sim 15.7\times$ communication reduction,
respectively, at operator level, especially for high polynomial orders. With
iso-latency, Falcon allows for $1.4$% and $4.2$% accuracy improvement on the
CIFAR-100 and Tiny Imagenet datasets over Cheetah, respectively.

##  II Preliminaries

###  II-A Threat Model

We focus on efficient privacy-preserving DNN inference involving two parites,
i.e., server and client, in which the server holds the proprietary DNN model
and the client owns private data [10, 16]. The model architecture, including
the number of layers as well as the types, dimensions, and bit widths for each
layer, is public to both the server and the client [9, 14, 15]. At the end of
the protocol execution, the client learns the inference results without
leaking any information to the server. Consistent with previous works [20, 21,
22, 23], we adopt an honest-but-curious security model in which both parties
follow the specification of the protocol but also try to learn more from the
protocol than allowed.

###  II-B Notations

TABLE I: Notations used in the paper.

Notations | Meanings
---|---
$[n]$ |  $\\{0,..,n-1\\}$ for a non-negative integer $n$
$\lceil\cdot\rceil$,$\lfloor\cdot\rfloor$,$\lfloor\cdot\rceil$ | Ceiling, flooring, and rounding operations
$N,q$ | Polynomials degree, ciphertext bitwidth
$\hat{a}$ |  A polynomial and $\hat{a}[i]$ denotes the $i$-th coefficient of the polynomial
$\lambda$ | Security parameter that measures the attack hardness
$\boxplus,\boxminus,\boxtimes$ |  Homomorphic addition, subtraction and multiplication
$\mathrm{Enc}(\cdot),\mathrm{Dec}(\cdot)$ | Homomorphic encryption and decryption
$H,W,C$ | Height, width, and channels of convolution input tensor
$H^{\prime},W^{\prime},K$ | Height, width, and channels of convolution output tensor
$R,G,s$ | Convolution kernel size, group size, and stride

Table I summarizes the notations used in this paper. The input, weight, and
output tensors of a convolution operation are denoted as
$\bm{X}\in\mathbb{Z}^{C\times H\times W}$, $\bm{W}\in\mathbb{Z}^{K\times
C\times R\times R}$, and $\bm{Y}\in\mathbb{Z}^{K\times H^{\prime}\times
W^{\prime}}$, where $\mathbb{Z}$ denotes the integer domain. A convolution
operation can be converted to a matrix-matrix multiplication
$\bm{X}^{\prime}\bm{W}^{\prime}$ through the im2col operation [24], where
$\bm{X}^{\prime}\in\mathbb{Z}^{H^{\prime}W^{\prime}\times CR^{2}}$,
$\bm{W}^{\prime}\in\mathbb{Z}^{CR^{2}\times K}$. For a group convolution,
$\bm{W}\in\mathbb{Z}^{K\times G\times R\times R}$. When $G=1$ and $K=C$, it
becomes a depthwise convolution. We also use $\bm{X}_{i}$ to represent the
$i$-th channel of $\bm{X}$, i.e., $\bm{X}[i,:,:]$, $\bm{W}_{i,j}$ to represent
the $j$-th channel of $i$-th filter, i.e., $\bm{W}[i,j,:,:]$. For simplicity,
we also use $\bm{W}_{i}$ denote $\bm{W}_{i,i}$.

###  II-C HE-based 2PC Inference

We review the basic flow of HE-based 2PC inference [25, 14, 26]. The 2PC
framework mainly involves two types of cryptographic primitives, including
arithmetic secret sharing (ASS) and HE. For ASS, an $l$-bit feature $x$ is
shared by the server and the client additively as the sum of two values, say
$\langle x\rangle^{S}_{2^{l}}$ and $\langle x\rangle^{C}_{2^{l}}$, and
$x\equiv\langle x\rangle^{S}_{2^{l}}+\langle x\rangle^{C}_{2^{l}}$[22]. For
simplicity, we omit $l$ in the rest of the paper. To compute the convolutions
between the filter and the shared features, HE is leveraged. HE enables to
compute homomorphically on the ciphertext without the need of decryption and
thus, protects the data privacy. As in Table I, HE supports homomorphic
addition, substruction, and multiplication.

In Figure 2, the parameter of the $i$-th layer is represented as $w_{i}$, and
the layer input and convolution output are denoted as $x_{i}$ and $y_{i}$,
respectively. During inference, the client and server hold additive shares
denoted as $\langle\cdot\rangle^{C},\langle\cdot\rangle^{S}$. The client
encrypts its share as $\mathrm{Enc}(\langle x_{i}\rangle^{C})$ and sends it to
the server. The server perform homomorphic operations to obtain the output
$\mathrm{Enc}(y_{i})=(\mathrm{Enc}(\langle x_{i}\rangle^{C})\boxplus\langle
x_{i}\rangle^{S})\boxtimes w_{i}$. Blinding of $y_{i}$ is done by subtracting
a randomly sampled $s_{i}$ from $\mathrm{Enc}(y_{i})$, and then, the result is
sent to the client. The client decrypts it to obtain $\langle
y_{i}\rangle^{C}=y_{i}-s_{i}$. The server outputs $s_{i}$ as $\langle
y_{i}\rangle^{S}$. We omit the details of the ReLU protocols and refer
interested readers to [15, 23, 10]. Communication costs for a MobileNetV2
network performing secure inference on the CIFAR-100 dataset using Cheetah
framework are highlighted as shown in Figure 2: linear layers contribute
$498\mathrm{MB}$ of communication in total, with $30\mathrm{MB}$ for input
ciphertext transfer and $468\mathrm{MB}$ for output ciphertext transfer.

Figure 2: Secure neural network inference based on HE (linear).

###  II-D Depthwise Convolution and Efficient Network

As illustrated in Figure 3, group convolution is a type of sparsely connected
convolution, which is first introduced in AlexNet [27]. Standard convolution
(Figure 3(a)) generates $K$ output features by applying each filter over all
$C$ input channels, resulting in a computational cost of $O(KC)$. In contrast,
group convolution (Figure 3(b)) reduces this computational cost by dividing
the input channels into $C/G$ mutually exclusive groups. Each group has $G$
channels and produces its own set of outputs, resulting in $O(KG)$
computational cost. When $G=1$, a group convolution becomes a depthwise
convolution (Figure 3(c)), where a single filter is applied to each input
channel. Depthwise convolution is widely employed in efficient networks,
including MobileNetV1 [18], MobileNetV2 [1], EfficientNet-Lite [2], etc. These
networks achieve SOTA accuracy with a high parameter and compute efficiency.
For instance, EfficientNet-Lite achieve similar accuracy compared to ResNet50
with $4\times$ latency reduction [2].

Figure 3: (a) Standard convolution, (b) group convolution and (c) depthwise
convolution.

###  II-E HE-based Convolution Protocols

Since our approach primarily focuses on the HE-based convolution layer, we
first introduce how baseline methods, including CrypTFlow2 [15], Cheetah [10],
and Iron [16] perform a convolution with a single input channel and single
output channel (SISO).

Figure 4: Illustration of SIMD packing: rotation is required to enable
summation and leads to a high computation complexity.

CrypTFlow2 leverages the SIMD packing to perform HE-based convolution, which
is also used by Gazelle [13], Delphi [14], etc. Figure 4 illustrates an
example of using SIMD packing to accomplish SISO convolution. It requires
im2col to transform a convolution into a matrix multiplication first, where
$\bm{X}^{\prime}\in\mathbb{Z}^{H^{\prime}W^{\prime}\times R^{2}}$ and
$\bm{W}^{\prime}\in\mathbb{Z}^{R^{2}\times 1}$. Then each row of
$\bm{X}^{\prime}$ is encoded into a separate ciphertext vector, denoted as
$\\{\hat{x}_{0},\ldots,\hat{x}_{H^{\prime}\times W^{\prime}-1}\\}$.
$\bm{W}^{\prime}$ is encoded as a plaintext vector $\hat{w}$. With SIMD
packing, a single homomorphic multiplication can realize the element-wise
product between the $\hat{x}_{i}$ and $\hat{w}$, resulting in
$\hat{u}_{i}=\hat{x}_{i}\odot\hat{w}$. To further sum all the elements in
$\hat{u}_{i}$, because $\hat{u}_{i}$ is encrypted, it can only be re-arranged
by a rotation. Hence, a “rotation-and-sum” step is needed before the final
results are calculated as in Figure 4. Compared to a homomorphic
multiplication, the computation complexity of a rotation is usually much
higher, leading to a high computation latency [13, 10].

Figure 5: Polynomial coefficient packing proposed by Cheetah.

To get rid of the expensive rotation, Cheetah [10] discovers that a polynomial
multiplication already encompasses convolution operations and proposes a
polynomial coefficient encoding method [28]. Specifically, Cheetah carefully
designed two mappings: $\pi^{i}$ and $\pi^{w}$ to convert the input and weight
tensor to the polynomial coefficients, i.e., one-dimensional vectors. The two
polynomials are then multiplied, and the correct convolution results can be
directly extracted from the polynomial coefficients as shown in Figure 5. Iron
[16] shares the same coefficient packing algorithm as Cheetah and further
optimizes the algorithm for matrix-matrix multiplications. To support
convolutions, Iron also requires running im2col first as shown in Figure 6.

Figure 6: Polynomial coefficient packing proposed by Iron.

##  III Inefficiency of Existing Solutions for Efficient Networks

From Figure 1, we observe for prior-art 2PC frameworks, Cheetah and Iron, the
computation latency of a depthwise convolution is almost the same as that of a
standard convolution, indicating a high inference overhead. We now analyze the
origin of the overhead to shed light on the directions of further
optimization. Note that we do not discuss the SIMD packing algorithm used in
CrypTFlow2 as its rotation operations may result in a latency gap of over
10$\times$ compared to Cheetah and Iron following [10, 16] and as shown in the
experimental results. Thus, it is not the focus of our analysis.

###  III-A Cost Analysis of HE-based Depthwise Convolution

The cost of a HE-based depthwise convolution primarily comes from computation
and communication between the client and the server. The computational
complexity is mainly determined by the number of homomorphic multiplications,
while the communication consists of two parts. As in Figure 2, the first part
is the transfer of input ciphertext polynomials from the client to the server,
which equals to the product of the number of polynomials and the communication
of each polynomial. The communication of a single polynomial scales with the
polynomial degree and the bit width of each coefficient, i.e., $\\#Poly\times
Nq$. The second part is the transfer of output ciphertext polynomails from the
server to the client. Following the optimization of Cheetah, the output
transfer communication is $\\#Poly\times(N+n)q$, where $n$ is the number of
output elements in each polynomial.

###  III-B Inefficiency of Existing Packing Algorithms

Figure 7: Packing algorithm of (a) Iron and (b) Cheetah to support depthwise
convolutions with multiple channels. Each square box represents a channel.
Blank squares represent zero channels. Channels packed in the same polynomial
are enclosed in the same dashed box.

In Section II-E, we have already introduced how Iron and Cheetah perform a
single channel convolution and now, we focus on analyzing their complexity for
multi-channel cases. For ease of explanation, we consider each channel as a
basic unit and assume $HW\leq N$, i.e., each channel convolution can be packed
within a polynomial. When $HW>N$, we can follow [10] to tile the input
tensors.

Iron When extending to depthwise convolution with multiple channels, Iron
needs to perform im2col operation for each channel, transforming a single
depthwise convolution into $C$ matrix multiplications. Hence, Iron generates
$C$ input polynomials, $C$ output polynomials, and in total $C$ homomorphic
multiplications. Following Section III-A, the input transfer communication
becomes $C\times Nq$, while the output transfer communication is
$C\times(N+H^{\prime}W^{\prime})q$. Iron only compute one channel at one time,
and each input polynomial and weight polynomial only packs $HW$ and $R^{2}$
elements, respectively, which demonstrates a high packing redundancy.

Cheetah Cheetah implements the protocol for standard convolution, and there
are two approaches to extend Cheetah to support depthwise convolution. The
first approach is to execute the convolution separately for each channel,
which is essentially the same as Iron. A more efficient approach is to restore
the depthwise convolution to a standard convolution by padding zero channels
as shown in Figure 7. Then we can invoke Cheetah’s standard convolution
protocol, where the non-zero channels are convolved with the input, while the
zero channels can be skipped directly. The advantage of this approach is that
it allows multiple channels to be packed into the same polynomial. Hence, for
a depthwise convolution, Cheetah produces $\lceil\frac{CHW}{N}\rceil$ input
polynomials and generates $C/\lceil\frac{N}{CHW}\rceil$ output polynomials,
and requires $C/\lceil\frac{N}{CHW}\rceil$ homomorphic multiplications. The
communication for input polynomials and output polynomials is
$\lceil\frac{CHW}{N}\rceil Nq$ and
$((C/\lceil\frac{N}{HWC}\rceil)N+CH^{\prime}W^{\prime})q$, respectively.
Because $HW\leq N$ and $\lceil\frac{N}{CHW}\rceil\geq 1$, Cheetah’s approach
is strictly superior to Iron’s approach in terms of both computation and
communication. However, in Cheetah’s packing method, due to the significant
amount of zero channels introduced, $\frac{C-1}{C}$ of the channels in weight
polynomials are zero channels, leading to waste of compute and communication
for output polynomials.

###  III-C Inefficiency of Tiling

From the communication analysis in Section III-B, we observe that both Iron
and Cheetah have a higher theoretical communication cost for output
polynomials compared to input polynomials. Our experiments also confirm this
observation as in Figure 2. However, both Iron and Cheetah’s tiling strategies
focus on optimizing the communication cost of input polynomials: they try to
pack as many input channels as possible, resulting in more compact input
ciphertext. However, with the increase of the number of channels in the input
polynomials, Cheetah needs to add more zero channels, which reduces the number
of channels that can be packed into a weight polynomial. This further
increases the number of output polynomials, and raises the output transfer
communication.

##  IV Depthwise Convolution Friendly Protocol

###  IV-A Overview

We first provide an overview of our proposed method, Falcon. We build Falcon
on top of Cheetah and address the significant overhead induced by the padded
zero channels by two steps, including zero-aware greedy packing and
communication-aware operator tiling.

###  IV-B Zero-aware Greedy Packing

Figure 8: Illustration of (a) zero-aware greedy packing and (b) combining
zero-aware greedy packing and communication-aware operator tiling. Dashed
boxes represent the zero channels reused by neighboring depthwise filters.

As mentioned in Section III-B, the inefficiency of Cheetah arises from a
significant number of packed zero channels. As shown in Figure 8, for the
depthwise convolution with $4$ channels, after padding it to a standard
convolution, the naive packing algorithm in Cheetah maps the $4$ filters into
$\bm{W}_{3}\bm{000}\bm{0}\bm{W}_{2}\bm{00}\bm{00}\bm{W}_{1}\bm{0}\bm{000}\bm{W}_{0}$.
While the padded zero channels is necessary to eliminate the interference
among channels to ensure the correctness of the computation, we make the
following two observations:

  * •

The computation order of different channels does not impact the computation
correctness. For example, the 4 padded filters can be mapped as
$\bm{W}_{3}\bm{000}\bm{000}\bm{W}_{0}\bm{0}\bm{W}_{2}\bm{00}\bm{00}\bm{W}_{1}\bm{0}$
without impacting the computation of each filter.

  * •

The zero channels of neighboring packed filters can be reused, e.g.,
$\bm{W}_{3}\bm{000}$ can share the 3 zero channels with $\bm{000}\bm{W}_{0}$,
resulting in
$\bm{W}_{3}\bm{000}\bm{W}_{0}\bm{0}\bm{W}_{2}\bm{00}\bm{W}_{1}\bm{0}$.

The above two observations enable much denser packing to drastically reduce
the number of padded zero channels.

More formally, let $\bm{0}^{(k)}$ denote a sequence of $k$ consecutive zero
channels, then, the one-dimensional vector of $i$-th padded filter, denoted as
$\tilde{\bm{W}}_{i}$, can be represented as
$\bm{0}^{(C-1-i)}\bm{W}_{i}\bm{0}^{(i)}$, $\forall i\in[C-1]$. Let
$\tilde{\bm{W}}$ denote the final mapping of the whole kernel, then, we
observe as long as $\tilde{\bm{W}}_{i}$ is a subsequence of $\tilde{\bm{W}}$,
the computation correctness can be guaranteed. We omit the proof due to space
constraints. Let $\mathrm{Len}(\tilde{\bm{W}})$ denote the vector length of
the final mapping, then, we can formulate our zero-aware greedy packing as
follows.

###### Problem IV.1

Given a set of vectors of padded filter
$\\{\tilde{\bm{W}}_{0},\ldots,\tilde{\bm{W}}_{C-1}\\}$, find the optimal
vector $\tilde{\bm{W}}$ that has the smallest length while preserving the
computation correctness, i.e.,

$\displaystyle\mathrm{minimize}\quad$
$\displaystyle\mathrm{Len}(\tilde{\bm{W}})$ $\displaystyle\mathrm{s.t.}\quad$
$\displaystyle\tilde{\bm{W}}_{i}\text{ is a subsequence of
}\tilde{\bm{W}},\forall i\in[C-1]$

Figure 9: Illustration of greedy algorithm for SCS problem.

Problem IV.1 is equivalent to the shortest common superstring (SCS) problem,
which is proved to be NP-hard. Lots of approximation algorithms have been
developed to solve the problem. We use Ukkonen’s 1990 algorithm[29]. As shown
in Figure 9. First, we build a directed graph where vertices are
$\\{\tilde{\bm{W}}_{0},\tilde{\bm{W}}_{1},...,\tilde{\bm{W}}_{C-1}\\}$, and
the weight of the edge from $\tilde{\bm{W}}_{i}$ to $\tilde{\bm{W}}_{j}$ is
the length of the longest suffix zero channels of $\tilde{\bm{W}}_{i}$, which
is also a prefix zero channels of $\tilde{\bm{W}}_{j}$. Next, we randomly
select one of the longest edges and merge the nodes at both ends of the edge
to form a new node and update the graph. Repeat this selecting and merging
process until there are no more edges in the graph. Finally, merge the
remaining nodes together to obtain the results $\tilde{\bm{W}}$.

Figure 10: Illustration of how zero-aware greedy packing combines multiple
channels.

After applying the greedy algorithm, we discover that the solution always
merges the $\tilde{\bm{W}}_{i}$ and $\tilde{\bm{W}}_{C-1-i}$ since they are
symmetric in the location of zero channels. Furthermore, if the sequence
starts with zeros, those zeros can be removed without affecting the
corresponding coefficients in the output. Based on the observation, we now
formally derive the mapping function for our greedy packing algorithm
described above. Suppose an input polynomial packs $C_{x}$ channels, and a
weight polynomial packs $C_{w}$ channels. The sequence of weight polynomials
is illustrated in Figure 10. Under the same $N$, we have $C_{x}\geq C_{w}$
because additional zero channels need to be added for the weight polynomials.
Therefore, we let $C_{x}=kC_{w}$, where $k\in\mathbb{Z^{+}}$, to prevent the
channels corresponding to a weight polynomial from being distributed across
different input polynomials, which would result in a doubling of the
homomorphic multiplications. Now, we present the coefficient mappings
$\pi_{\mathrm{dwconv}}^{i}$ and $\pi_{\mathrm{dwconv}}^{w}$ to pack the three-
dimensional input and weight tensors into one-dimensional vector. For
simplicity, we require $\max(C_{x}\times H\times W,C_{w}\times R\times R)\leq
N$, $C_{x}=C_{w}$ and $C_{w}$ is even.

$\displaystyle\hat{x}=\pi_{\mathrm{dwconv}}^{i}(\mathbf{X})\text{ st.
}\hat{x}[cHW+iW+j]=\mathbf{X}[c,i,j]$
$\displaystyle\hat{w}=\pi_{\mathrm{dwconv}}^{w}(\bm{W})\text{ st.
}\hat{w}[\mathrm{offset}(c^{\prime})\times HW+O-lW-l^{\prime}]$
$\displaystyle=\bm{W}[c^{\prime},l,l^{\prime}]$

where $O=W(h-1)+h-1$ and $\mathrm{offset}$ is:

$\mathrm{offset}(c^{\prime})=\left\\{\begin{array}[]{l}\left(C_{x}+2\right)\times\left(C_{w}-1-c^{\prime}\right),c^{\prime}\geqslant\frac{C_{w}}{2}\\\
C_{x}+c^{\prime}\times\left(C_{x}+1\right),c^{\prime}<\frac{C_{w}}{2}\end{array}\right.$

The multiplication $\hat{y}=\hat{x}\cdot\hat{w}$ directly gives the
2-dimension depthwise convolution in some of the coefficients of the resulting
polynomial:

$\mathbf{Y}\left[k^{\prime},i^{\prime},j^{\prime}\right]=\hat{y}\left[(\mathrm{offset}(k^{\prime})+k^{\prime})\times
HW+O+i^{\prime}sW+j^{\prime}s\right]$

where $\mathbf{Y}\in\mathbb{Z}^{C_{x}\times H^{\prime}\times W^{\prime}}$.
$\mathrm{offset}(k^{\prime})$ calculates the offset of weights at the channel
level, while $k^{\prime}$ calculates the offset of inputs at the channel
level. This is used to determine the starting position of each output channel,
while the encoding within a single channel remains consistent with Cheetah.

When $k\neq 1$, it means that the same input polynomial $\hat{x}$ needs to be
multiplied with multiple weight polynomials
$\\{\hat{w}_{0},\hat{w}_{1},...,\hat{w}_{k-1}\\}$ to obtain the outputs
$\\{\hat{y}_{0},\hat{y}_{1},...,\hat{y}_{k-1}\\}$. In order to perform packing
in this case, we first concatenate these weight polynomials end-to-end to form
a larger polynomial $\hat{w}=\hat{w}_{0}\hat{w}_{1}...\hat{w}_{k-1}$. Then, we
apply packing to $\hat{w}$, and $\hat{w}$ still satisfies
$\hat{w}=\pi_{\mathrm{dwconv}}^{w}(\bm{W})$. Afterwards, $\hat{w}$ is divided
into $k$ smaller polynomials, and the leading zeros of each polynomial are
removed.

Complexity Analysis. Now we analyze the computational and communication
complexity of zero-aware greedy packing theoretically. We first analyze the
utilization of weight polynomial coefficients. Evidently, Cheetah’s
utilization rate is $\frac{1}{C_{x}}$. Based on the analysis shown in Figure
10, our theoretical utilization is
$\frac{C_{w}}{C_{x}+1+(C_{w}/2-1)(C_{x}+2)}>\frac{2}{C_{x}+2}$. Considering
the range of values for $C_{x}$ in neural networks, zero-aware greedy packing
can achieve approximately twice the channel utilization rate compared to
Cheetah.

In terms of computational complexity, zero-aware greedy packing reduces the
number of polynomial multiplications by a factor of $\frac{2}{C_{x}+2}$
compared to Cheetah. As for communication, zero-aware greedy packing reduces
the communication of the output polynomials from
$(\lceil\frac{C_{x}HW}{N}\rceil N+C_{x}H^{\prime}W^{\prime})q$ to
$(\lceil\frac{(C_{x}+2)HW}{2N}\rceil N+C_{x}H^{\prime}W^{\prime})q$ compared
with Cheetah. In Section III-B, it has been proven that the computational and
communication complexity of Cheetah is strictly superior to Iron. Therefore,
our method outperforms both Cheetah and Iron in computation and communication.

###  IV-C Communication-aware Operator Tiling

As shown in Figure 2, the communication of the output polynomials is much
higher than that of the input polynomials. This observation aligns with the
communication cost formula derived in Section III-B. Previous algorithms aim
to maximize the number of channels packed in one input polynomial
($C_{x}$)[10, 16, 13, 14, 15], but this leads to reduced utilization of weight
polynomial coefficients and smaller $C_{w}$ because more zero channels need to
be padded. Moreover, the reduction in $C_{w}$ significantly increases the
communication cost of output polynomials, resulting in excessively high
communication costs.

To address this issue, we propose a novel communication-aware operator tiling
method. Unlike previous methods, we conduct a theoretical analysis of
communication cost in depthwise convolution and formulate it as a nonlinear
programming problem. Our objective is to minimize communication while adhering
to polynomial coefficient encoding rules. Formally, the communication cost is
given by the equation:

$\lceil\frac{C}{C_{x}}\rceil
Nq+\lceil\frac{C}{C_{w}}\rceil(N+H^{\prime}W^{\prime}C_{w})q$

The first part of the equation represents the communication of the input
polynomials, where $\lceil\frac{C}{C_{x}}\rceil$ is the number of ciphertexts.
The second part corresponds to the output polynomials communication as
discussed in Section III-B. By approximating the ceiling function as $\lceil
x\rceil\approx x$, we obtain:

$NCq(\frac{1}{C_{x}}+\frac{1}{C_{w}})+H^{\prime}W^{\prime}Cq$

Among them, only $\frac{1}{C_{x}}+\frac{1}{C_{w}}$ needs to be minimized, with
all other parameters known before executing the convolution protocol. We also
have two constraints: 1) polynomial coefficient encoding should not exceed the
degree $N$, and 2) $C_{x}$ must be a positive integer multiple of $C_{w}$.
Hence, we can formulate the communication optimization problem as follows:

$\displaystyle\mathrm{minimize}\ \ \ \ \ $ $\displaystyle\ \
||\frac{1}{C_{x}}+\frac{1}{C_{w}}||$ $\displaystyle\mathrm{s.t.}\ \ \ \ $
$\displaystyle(C_{x}+2)C_{w}\leq\frac{2N}{HW}+2$ $\displaystyle
C_{x}=kC_{w}\text{ where }k\in\mathbb{Z^{+}}$

This is a solvable nonlinear programming problem with a relatively small
solution space. With $C_{x}\leq\frac{N}{HW}$ and $C_{x}=kC_{w}$, the solution
space is at most $\frac{N}{HW}\times\frac{N}{HW}$, allowing us to directly
solve it using a search algorithm with a complexity of
$O((\frac{N}{HW})^{2})$. An example is illustrated in Figure 8 (b), after
tiling, we reduce the size of weight polynomials and output polynomials.

###  IV-D Extending to group convolution

Furthermore, we discover that our method can be naturally extended to group
convolution. In group convolution, $C$ convolutional filters are divided into
$\frac{C}{G}$ groups, with each group containing $G$ convolutional filters,
and each filter consisting of $G$ channels. In this case, instead of treating
individual channels or filters as basic units, we consider a whole group of
convolutional filters (including $G$ kernels and $G^{2}$ channels) as a basic
unit. The packing between groups follows the rules of zero-aware greedy
packing, while within each group, it essentially becomes a standard
convolution and can be directly packed using the Cheetah approach for standard
convolutions.

##  V Experimental Results

###  V-A Experimental Setup

Falcon is built on top of the SEAL library [30], the EMP toolkit [31] and
OpenCheetah[10] in C++. We also use the Ezpc library[32] to evaluate
CrypTFlow2[15]. Consistent with [33, 22], we simulate a WAN network setting
via Linux Traffic Control, where the bandwidth is 9 MBps. All the following
experiments are performed on machines with 2.2 GHz Intel Xeon CPU and 256GB
RAM. Following [16, 10], we set $N=4096$ for most of the cases.

###  V-B Microbenchmark Evaluation

#### Single Depthwise Convolution Evalution

We compare the performance of Falcon with CrypTFlow2, Iron, and Cheetah for a
single depthwise convolution. As shown in Figure 11, Falcon achieves a
communication reduction of $1.4\sim 4.0\times$ and a latency reduction of
$1.8\sim 3.2\times$ compared to Cheetah. Compared to Iron, Falcon achieves a
communication reduction of $3.6\sim 11.8\times$. It is important to note that
CrypTFlow2 (with SIMD) has relatively low communication. However, as mentioned
in Section III, the SIMD method has high computational complexity, resulting
in significant latency. In comparison, our method achieves a latency reduction
of $15.6\sim 19.9\times$.

Figure 11: (a) Communication and (b) latency comparison of depthwise
convolutions of different dimensions (the tuple represents the input feature
height, channels, and kernel size).

#### Depthwise Convolution under Different Polynomial Degree

Figure 12: (a) Communication and (b) latency comparison of a depthwise
convolution under different polynomial degrees.

We benchmark Falcon against CrypTFlow2, Cheetah, and Iron, for different
polynomial degrees. We select a depthwise convolution with input dimension of
$(14,576,3)$ for evaluation. As shown in Figure 12, overall, with the increase
of $N$, Falcon achieves a higher speedup compared to existing methods due to a
larger packing optimization space. Compared to Cheetah, we achieve a
communication reduction of $4.0\times$ at $N=4096$ and an even greater
reduction of $11.1\times$ at $N=32768$. Compared to Iron, our method achieves
a remarkable communication reduction of $32.9\times$ when $N=32768$.

#### Depthwise Convolution under Different Bandwidth

We also benchmark the latency of Falcon against CrypTFlow2, Cheetah, and Iron
under different bandwidth settings: $9\mathrm{MBps}$, $44\mathrm{MBps}$, and
$384\mathrm{MBps}$, representing WAN settings in [33, 22], WAN and LAN
settings in Cheetah[10], respectively. In Table II, our methods achieve a
significant latency reduction of approximately $5.6\sim 7.35\times$ compared
to Cheetah, and this advantage increases as the bandwidth decreases. We also
observe that CrypTFlow2 and Iron are not sensitive to bandwidth. For
CrypTFlow2, this is because it suffers from a high computational complexity
while for Iron, we hypothesize the latency bottleneck comes from the frequent
encryption and decryption.

TABLE II: Latency (s) comparison under different bandwidths. Bandwidth | Methods
---|---
CrypTFlow2 | Iron | Cheetah | Falcon (ours)
384MB/s [10] | 29.66 | 10.23 | 2.44 | 0.43
44MB/s [10] | 30.24 | 10.49 | 3.04 | 0.45
9MB/s[33, 22] | 30.86 | 10.68 | 5.56 | 0.76

#### Group Convolution Evaluation

We also evaluate the performance of Falcon for group convolutions. We select a
depthwise convolution with dimension of $(14,576,3)$ and benchmark its
performance under different group sizes. As shown in Table III, our method
consistently achieves better communication for different group sizes.
Additionally, we notice that, except for Iron, the other methods are not
sensitive to the group size. The main reason is that both CrypTFlow2 and
Cheetah expand group convolutions into standard convolutions. In contrast,
Iron’s approach leads to an increase in matrix multiplication dimension as the
group size grows. When the dimension exceed the polynomial degree $N$, it
results in a significant increase of communication.

TABLE III: Communication (MB) comparison under different group sizes. Group size | Methods
---|---
CrypTFlow2 | Iron | Cheetah | Falcon (ours)
1 | 9.00 | 51.70 | 18.94 | 8.98
2 | 9.00 | 51.70 | 18.94 | 8.98
4 | 9.00 | 68.44 | 18.94 | 8.98
8 | 9.00 | 102.73 | 18.94 | 8.98
TABLE IV: End-to-end comparison with prior-art works on different architectures and datasets (the inference complexity on Cifar10 and Cifar100 is essentially the same). Network | Dataset | CrypTFlow2 | Cheetah | Falcon (Ours)
---|---|---|---|---
Comm. (GB) | Latency (s) | Comm. (GB) | Latency (s) | Comm. (GB) | Latency (s)
MobileNetV2 | Cifar | 2.32 | 622.43 | 0.60 | 132.03 | 0.46 | 109.27
MobileNetV2 | Tiny Imagenet | 2.42 | 1067.34 | 0.70 | 168.35 | 0.54 | 130.41
EfficientNet-lite | Cifar | 0.83 | 563.90 | 0.50 | 103.48 | 0.34 | 76.61
EfficientNet-lite | Tiny Imagenet | 1.28 | 915.36 | 0.64 | 146.37 | 0.45 | 107.44

Figure 13: Accuracy comparison with Cheetah and CrypTFlow2 on (a) Tiny
Imagenet, N=4096; (b) Tiny Imagenet, N=32768; (c) Cifar-100, N=4096.

z

Figure 14: Block-wise comparison of latency on MobileNetV2/Tinyimagenet,
N=4096.

###  V-C End-to-end Inference Evaluation

#### Communication and Latency Comparison

We now perform an end-to-end inference evaluation compared to prior-art
methods, including CrypTFlow2 and Cheetah. Due to the unavailability of open-
source code for Iron, we find it difficult to reproduce all of its linear and
nonlinear protocols. As a result, we are unable to evaluate its performance in
the end-to-end inference. We apply Falcon to two widely used efficient
networks, MobileNetV2 and EfficientNet-lite, and evaluate their performance on
the Cifar10, Cifar100 [34], and Tiny Imagenet [35] datasets. We set the
bandwidth to $9\mathrm{MBps}$ and $N=4096$.

Results and analysis. As shown in Table IV, compared to Cheetah, Falcon
reduces the communication by $1.31\times$ and the latency by more than
$1.2\times$ on the Cifar10 and Cifar100 datasets. On the Tiny Imagenet
dataset, the advantage of Falcon is greater, with a communication reduction of
$1.44\sim 1.48\times$ and a latency reduction of $1.35\sim 1.36\times$. These
experimental results resonate with the theoretical expectations. For Tiny
Imagenet, the networks have more layers which have smaller dimensions $H$ and
$W$ but larger channel size $C$, resulting in a larger packing optimization
space.

#### Networks Accuracy Comparison

We also benchmark the accuracy of MobileNetV2 networks with different channel-
widths on Cifar100 and Tiny Imagenet for CrypTFlow2, Cheetah, and Falcon
(Ours). Additionally, we selecte $N=4096$ and $N=32768$ to demonstrate the
superiority of our method with larger $N$.

Results and analysis. As shown in Figure 13, Falcon consistently outperforms
the other methods across different network widths. In Tiny Imagenet,when
$N=4096$, with approximately $0.5\mathrm{GB}$ of communication costs, Falcon
achieves $60.75\%$ top-1 accuracy, surpassing Cheetah and CrypTFlow2 by
$3.76\%$ and $10.65\%$, respectively. When $N$ is further increased to
$32768$, Falcon maintains a substantial lead in top-1 accuracy over Cheetah
and CrypTFlow2. With approximately $2.8\mathrm{GB}$ of communication, Falcon
outperforms them by $6.43\%$ and $10.81\%$, respectively. For the Cifar100
dataset, as the network exhibits strong capability on it, increasing the width
does not bring significant accuracy improvement. However, Falcon still
achieves an average accuracy improvement of $1.4$%.

#### Block-wise Latency Comparison

We show the block-wise latency comparison in Figure 14. As we can observe,
Falcon achieves more latency reduction in deeper layers of the network, as
$H,W$ becomes smaller and $C$ becomes larger, resulting in a larger packing
optimization space. Additionally, Falcon reduces the latency for all the
blocks, highlighting the effectiveness of Falcon.

##  VI Conclusion

In this paper, we propose Falcon, a dense packing algorithm for
homomorphically encrypted depthwise and group convolution for HE-based 2PC
network inference. Falcon leverages the computation characteristics of the
depthwise convolution and directly optimize the communication bottleneck.
Falcon features a zero-aware greedy packing algorithm and a communication-
aware operator tiling strategy to significantly reduce the communication for
efficient networks. Compared with prior-art works, e.g., Cheetah and Iron,
Falcon achieves a latency reduction of $1.8\sim 3.2\times$ at the operator
level and $1.21\sim 1.36\times$ at the network level, which translates to 1.4%
and 4.2% accuracy improvement with iso-communication. These improvements
enhance the practicality of HE-based 2PC inference for efficient networks.

## References

  * [1] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _IEEE/CVF conference on computer vision and pattern recognition (CVPR)_ , 2018, pp. 4510–4520.
  * [2] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International conference on machine learning (ICML)_. PMLR, 2019, pp. 6105–6114.
  * [3] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 11 976–11 986.
  * [4] N. Azouji, A. Sami, and M. Taheri, “Efficientmask-net for face authentication in the era of covid-19 pandemic,” _Signal, Image and Video Processing_ , vol. 16, no. 7, pp. 1991–1999, 2022.
  * [5] G. Kaissis, A. Ziller, J. Passerat-Palmbach, T. Ryffel, D. Usynin, A. Trask, I. Lima Jr, J. Mancuso, F. Jungmann, M.-M. Steinborn _et al._ , “End-to-end privacy preserving deep learning on multi-institutional medical imaging,” _Nature Machine Intelligence_ , vol. 3, no. 6, pp. 473–484, 2021\.
  * [6] W.-S. Choi, B. Reagen, G.-Y. Wei, and D. Brooks, “Impala: Low-latency, communication-efficient private deep learning inference,” May 2022.
  * [7] D. Demmler, T. Schneider, and M. Zohner, “Aby -a framework for efficient mixed-protocol secure two-party computation,” in _Proceedings 2015 Network and Distributed System Security Symposium_ , Feb 2015. [Online]. Available: http://dx.doi.org/10.14722/ndss.2015.23113
  * [8] K. Gupta, D. Kumaraswamy, N. Chandran, and D. Gupta, “Llama: A low latency math library for secure inference,” _Proceedings on Privacy Enhancing Technologies_ , vol. 2022, no. 4, p. 274–294, Sep 2022. [Online]. Available: http://dx.doi.org/10.56553/popets-2022-0109
  * [9] N. Kumar, M. Rathee, N. Chandran, D. Gupta, A. Rastogi, and R. Sharma, “Cryptflow: Secure tensorflow inference.” in _2020 IEEE Symposium on Security and Privacy (SP)_ , Jul 2020. [Online]. Available: http://dx.doi.org/10.1109/sp40000.2020.00092
  * [10] Z. Huang, W.-j. Lu, C. Hong, and J. Ding, “Cheetah: Lean and fast secure $\\{$Two-Party$\\}$ deep neural network inference,” in _31st USENIX Security Symposium (USENIX Security 22)_ , 2022, pp. 809–826.
  * [11] R. Dathathri, O. Saarikivi, H. Chen, K. Laine, K. Lauter, S. Maleki, M. Musuvathi, and T. Mytkowicz, “Chet: an optimizing compiler for fully-homomorphic neural-network inferencing,” in _Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation_ , Jun 2019. [Online]. Available: http://dx.doi.org/10.1145/3314221.3314628
  * [12] D. Kim, J. Park, J. Kim, S. Kim, and J. Ahn, “Hyphen: A hybrid packing method and optimizations for homomorphic encryption-based neural networks,” Feb 2023.
  * [13] C. Juvekar, V. Vaikuntanathan, and A. Chandrakasan, “GAZELLE: A low latency framework for secure neural network inference,” Jan 2018.
  * [14] P. Mishra, R. Lehmkuhl, A. Srinivasan, W. Zheng, and R. Popa, “Delphi: A cryptographic inference service for neural networks,” Jan 2020.
  * [15] D. Rathee, M. Rathee, N. Kumar, N. Chandran, D. Gupta, A. Rastogi, and R. Sharma, “Cryptflow2: Practical 2-party secure inference,” in _Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security_ , 2020, pp. 325–342.
  * [16] M. Hao, H. Li, H. Chen, P. Xing, G. Xu, and T. Zhang, “Iron: Private inference on transformers,” in _Advances in Neural Information Processing Systems_ , 2022.
  * [17] Q. Lou and L. Jiang, “Hemet: A homomorphic-encryption-friendly privacy-preserving mobile neural network architecture,” May 2021.
  * [18] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [19] M. Tan and Q. Le, “Efficientnetv2: Smaller models and faster training,” in _International conference on machine learning_. PMLR, 2021, pp. 10 096–10 106.
  * [20] R. Gilad-Bachrach, N. Dowlin, K. Laine, K. Lauter, M. Naehrig, and J. Wernsing, “Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy,” in _International conference on machine learning_. PMLR, 2016, pp. 201–210.
  * [21] S. U. Hussain, M. Javaheripi, M. Samragh, and F. Koushanfar, “Coinn: Crypto/ml codesign for oblivious inference via neural networks,” in _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ , 2021, pp. 3266–3281.
  * [22] P. Mohassel and Y. Zhang, “Secureml: A system for scalable privacy-preserving machine learning,” in _2017 IEEE Symposium on Security and Privacy (SP)_ , Jun 2017.
  * [23] D. Rathee, M. Rathee, R. K. K. Goli, D. Gupta, R. Sharma, N. Chandran, and A. Rastogi, “Sirnn: A math library for secure rnn inference,” in _2021 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2021, pp. 1003–1020.
  * [24] K. Chellapilla, S. Puri, and P. Simard, “High performance convolutional neural networks for document processing,” in _Tenth international workshop on frontiers in handwriting recognition_. Suvisoft, 2006.
  * [25] J. Liu, M. Juuti, Y. Lu, and N. Asokan, “Oblivious neural network predictions via minionn transformations,” in _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , Oct 2017.
  * [26] K. Garimella, Z. Ghodsi, N. Jha, S. Garg, and B. Reagen, “Characterizing and optimizing end-to-end systems for private inference,” Jul 2022.
  * [27] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2017.
  * [28] S. Bian, D. E. S. Kundi, K. Hirozawa, W. Liu, and T. Sato, “Apas: application-specific accelerators for rlwe-based homomorphic linear transformations,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 4663–4678, 2021.
  * [29] E. Ukkonen, “A linear-time algorithm for finding approximate shortest common superstrings,” _Algorithmica_ , vol. 5, no. 1-4, pp. 313–323, 1990.
  * [30] “Microsoft SEAL (release 3.6),” https://github.com/Microsoft/SEAL, Nov. 2020, microsoft Research, Redmond, WA.
  * [31] X. Wang, A. J. Malozemoff, and J. Katz, “EMP-toolkit: Efficient MultiParty computation toolkit,” https://github.com/emp-toolkit, 2016.
  * [32] N. Chandran, D. Gupta, A. Rastogi, R. Sharma, and S. Tripathi, “Ezpc: Programmable, efficient, and scalable secure two-party computation for machine learning,” _Cryptology ePrint Archive_ , 2017.
  * [33] L. Shen, Y. Dong, B. Fang, J. Shi, X. Wang, S. Pan, and R. Shi, “Abnn 2,” in _Proceedings of the 59th ACM/IEEE Design Automation Conference_ , Aug 2022.
  * [34] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [35] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.


# A Framework for Pupil Tracking with Event Cameras

Khadija Iddrisu  Dublin City University, Ireland  Waseem Shariff  University
of Galway, Ireland  Suzanne Little  Dublin City University, Ireland

###### Abstract

Saccades are extremely rapid movements of both eyes that occur simultaneously,
typically observed when an individual shifts their focus from one object to
another. These movements are among the swiftest produced by humans and possess
the potential to achieve velocities greater than that of blinks. The peak
angular speed of the eye during a saccade can reach as high as 700°/s in
humans, especially during larger saccades that cover a visual angle of 25°.
Previous research has demonstrated encouraging outcomes in comprehending
neurological conditions through the study of saccades. A necessary step in
saccade detection involves accurately identifying the precise location of the
pupil within the eye, from which additional information such as gaze angles
can be inferred. Conventional frame-based cameras often struggle with the high
temporal precision necessary for tracking very fast movements, resulting in
motion blur and latency issues. Event cameras, on the other hand, offer a
promising alternative by recording changes in the visual scene asynchronously
and providing high temporal resolution and low latency. By bridging the gap
between traditional computer vision and event-based vision, we present events
as frames that can be readily utilized by standard deep learning algorithms.
This approach harnesses YOLOv8, a state-of-the-art object detection
technology, to process these frames for pupil tracking using the publicly
accessible Ev-Eye dataset. Experimental results demonstrate the framework’s
effectiveness, highlighting its potential applications in neuroscience,
ophthalmology, and human-computer interaction.

Keywords: Event Camera, Pupil Tracking, Saccade, Object Detection.

##  1 Introduction

Saccades have been extensively studied in the context of various neurological
and psychiatric disorders, such as schizophrenia, concussions, traumatic brain
injuries (TBI), and Parkinson’s disease [Bittencourt et al., 2013].
Additionally, the relationship between saccades and cognition has shown strong
correlation in identifying neurological disorders [Yang et al., 2024].
Assessing saccade changes during non-visual cognitive tasks may help to
identify subtle brain changes due to aging and neuro-degenerative diseases. To
make saccade analysis broadly useful for both diagnostic and research
purposes, it is crucial to identify specific eye movement parameters. Key
metrics, including velocity and duration, must be extracted from data that
reflects a wide range of patient characteristics, such as differing eye
shapes, and iris, hair, and skin pigmentation. However, to analyze saccades,
the first step is to detect the pupil.

Automatic and non-invasive pupil tracking is a significant task in computer
vision with applicability in areas such as Human-Computer Interactions (HCI),
Virtual Reality and Extended Reality. This technology can contribute to
greater understanding of psychological conditions by analysing the behaviour
and tracking the movement of the pupil position over time. Recently, a few
studies to improve pupil tracking have emerged [Kang et al., 2019, Khan et
al., 2020, Lee et al., 2020]. The majority of this research falls into two
categories: remote pupil tracking and near-eye pupil tracking. Remote-eye
pupil tracking involves pupil tracking from a distance from the eye which is
applicable in areas such as driver monitoring while the latter involves pupil
tracking with a setup in direct contact with the eyes (e.g., AR/VR). For
effective pupil detection in non-clinical, real-world, environments, optimized
sensors, such as event cameras, are essential.

Event cameras (ECs) have gained significant attention from researchers for
pupil tracking due to their high sampling rate [Gallego et al., 2020, Kang et
al., 2023, Shariff et al., 2024, Wang et al., 2024]. ECs record changes in
light intensity per pixel, offering unique features such as high dynamic range
(HDR), high temporal resolution, and low latency. These characteristics open
new opportunities for tasks like pupil tracking. The HDR capability of ECs
allows them to capture and report changes occurring in very rapid motion,
which often results in motion blur in traditional cameras. Additionally, since
ECs only capture changes in brightness levels, they protect user privacy by
not collecting detailed iris data, as long as no reconstruction is done on the
event stream [Gallego et al., 2020, Shariff et al., 2024]. Nonetheless,
sensitive information might still be recoverable from event data through high
resolution reconstruction [Rebecq et al., 2019]. Techniques such as "event
encryption" [Zhang et al., 2024a], "Event-based person identification", "event
scrambling" [Ahmad et al., 2024] among others have been proposed to enhance
privacy by encrypting the event data to prevent reconstruction.

Early studies on pupil tracking utilized computer vision and machine learning
techniques, including pupil localization, eye center estimation, deep learning
approaches, and support vector machines [Klaib et al., 2021]. Most of these
studies relied on traditional frame-based camera systems and sometimes
integrated near-infrared (NIR) pupil-tracking tasks. Inspired by recent
advancements [Rebecq et al., 2019] which aims to bridge the gap between event
cameras and modern computer vision, we propose using event-to-video conversion
to generate event windows. These windows are then processed using YOLOv8, a
state-of-the-art object detector, to detect and track the pupil accurately.

The contributions of this work are as follows:

  1. 1.

Integration of advanced computer vision techniques with the unique
capabilities of event cameras by applying the YOLOv8 object detection model to
event camera data. This combination enhances the accuracy and efficiency of
pupil detection and tracking, leveraging the high temporal resolution and low
latency of event cameras.

  2. 2.

Training of four different neural networks to process and analyze event camera
data. This allows for a comprehensive evaluation of various models,
identifying the most effective methods for pupil tracking and ensuring robust
performance across different scenarios.

  3. 3.

A significant initial step towards the more complex goal of analyzing
saccades. Focusing on the foundational task of accurate pupil detection and
tracking using event cameras, the research sets the stage for future studies
to investigate saccadic movements that are crucial for diagnosing and
understanding various neurological and psychiatric disorders.

##  2 Related Work

Despite the demonstrated potential of event-based pupil tracking, the body of
research addressing this area remains limited. To date, only a small number of
recent studies have undertaken this task.

Kang et al [Kang and Kang, 2023] developed a large-scale event face training
database through RGB-to-Event domain translation. They achieved this by using
the StyleFlow algorithm to generate event-like images from RGB face images,
leveraging existing annotations. Subsequently, the study evaluated a cross-
modal learning-based pupil localization algorithm on real event camera images,
captured with a DAVIS 346 camera, using a mixed dataset of RGB and event-like
images. The algorithm, based on the RetinaFace algorithm and trained under
this strategy, achieved a high accuracy in pupil localization, outperforming
methods trained solely on RGB or event-like databases. In another study, the
authors accumulated events over 33 ms intervals into frames [Kang et al.,
2023]. Following this representation, they use cascaded Adaboost classifiers
with multi-block local binary patterns (LBPs) to detect eye-nose region on
which a Shape-Deformation Model (SDM) is applied for precise alignment,
utilizing SIFT features and a Support Vector Machine (SVM) to maintain
tracking across frames.

Kagemoto et al [Kagemoto and Takemura, 2023] presents the first purely event-
based approach for pupil tracking with the bright and dark pupil effect
elicited by alternately blinking two near-infrared illumination sources. The
pupil center is detected by manually defining the region of interest and
calculating the center of gravity of the more eventful polarity. This method
processes events in real-time at 2000 Hz, significantly faster than
traditional ellipse fitting methods, which operate at 30 Hz. Comparative
experiments demonstrated that this approach accurately tracks eye movements
with a high correlation to target movements and reduces gaze detection delay
to 0.04 seconds compared to 1.18 seconds with ellipse fitting.

Finally, Swift-Eye [Zhang et al., 2024b] proposed a method to handle occlusion
by leveraging Timelens, an event-based frame interpolation network, to
transform low frame-rate video and asynchronous event streams into high-speed
video up to 5000fps. Multi-scale spatial feature extraction is performed using
a swin-transformer and feature pyramid network (FPN) to create hierarchical
features for pupil estimation. Additionally, a rotated bounding box detects
the pupil region by accommodating the ellipse’s rotation in near-eye images.
An occlusion-ratio-based approach adapts the detection method according to the
level of occlusion, while invalid estimates are linearly interpolated.

Our proposed approach diverges from Swift-Eye by not relying on information
from RGB frames, which many event-based pupil tracking methods still utilize.
Instead, we rely exclusively on events converted into frames in a manner that
preserves the asynchronicity of events while mitigating issues such as
undersampling. This approach aligns with existing event frame-based methods
but aims to enhance the reliability and accuracy of pupil tracking without the
need for additional RGB data.

##  3 Methodology

###  3.1 Event Representation

Deep learning algorithms such as Convolutional Neural Networks (CNNs) accept
data inputs of fixed size such as images. Hence to perform pupil tracking with
CNNs, there is a need to convert events into representations such as frames or
voxel grids which can be readily used by these networks. In this study, we
convert events into 2D frames by accumulating polarity pixel-wise.

Event cameras (ECs) possess an extremely high temporal resolution,
theoretically capable of achieving up to 1 million frames per second (FPS) due
to their microsecond ($1\mu$) temporal precision [Wang et al., 2019]. This
contrasts with conventional frame-based cameras, which are typically limited
to a maximum of about 100-145 FPS. We make use of the high temporal resolution
by accumulating events over a 10 ms period to generate equivalent frames at
100 FPS. This approach allows us to significantly reduce the sampling time
while preserving the temporal information inherent in the events. By doing so,
we mitigate the risk of under-sampling and ensure that relevant information is
maintained within the frames, all while preserving the asynchronous nature of
the events. Additionally, this approach allows us to generate more frames for
training (over 600 frames per event file) as compared to a few frames when
accumulating based on a fixed size window. Sampling at 10 ms also aided in
generation of frames captured during very rapid movements such as a
half/almost full blink periods which may otherwise be lost in traditional
imaging.

There are two methods commonly employed for 2D event accumulation; by
accumulating in a fixed time window or fixed size window. The fixed time
window approach conserves the temporal information of events, however for
tasks such as pupil/eye tracking, it may lead to overlapping of eye features
in time windows where there are many events generated due to motion. Therefore
a fixed size approach is often employed for eye tracking applications, however
due to the sparse nature of events, a larger window size is needed to make the
pupil visible which results to low frame rate and subsequently can lead to
under-sampling.

In this study, we employ a fusion of both approaches such that, we sample a
fixed duration window, i.e The total duration of events, from the minimum
timestamp $\big{(}T_{\text{min}}\big{)}$ to the maximum timestamp
$\big{(}T_{\text{max}}\big{)}$, is divided into frames of duration 10
milliseconds (ms) each. The number of frames $\big{(}N_{\text{frames}}\big{)}$
is determined as
$(N_{\text{frames}}=\left\lceil\frac{T_{\text{max}}-T_{\text{min}}}{\text{duration\\_ms}}\right\rceil)$.
For each frame, events occurring within the time window
$([T_{\text{start}},T_{\text{end}}))$ are selected, where
$(T_{\text{start}}=T_{\text{min}}+i\times\text{duration\\_ms})$ and
$(T_{\text{end}}=T_{\text{start}}+\text{duration\\_ms})$.

This process ensures that each frame represents a specific interval of time.
To avoid creating frames with too few events, a threshold is applied. Frames
are only generated if the number of events within the current time window
exceeds the specified event threshold (set to 2000). This helps to filter out
frames that might be empty or contain too little information. The events
within the selected time window are then processed to update the pixel values
in the frame. For each event $(t,y,x,p)$, we check the event’s coordinates and
polarity. If the polarity $p$ is +1, indicating an ON event, the corresponding
pixel at $(y,x)$ is set to 255 (white). Conversely, if the polarity $p$ is -1,
indicating an OFF event, the pixel is set to 0 (black).

###  3.2 Pupil Tracking with Yolo

Data Preparation: Following the method for event representations, we generated
frames from the EV-Eye Dataset for training with YOLOv8. EV-Eye [Zhao et al.,
2024] consist of event data collected from 48 participants. Each participant
participates in four sessions; fixation, saccades, and the last two session
capturing smooth pursuit movement. We utilise the raw unprocessed data and
generate 20 random frames from each participant from both the left and right
eye which allows us to generate a diverse range of frames from different eye
movements for the resulting dataset. We select 38 subjects for training while
reserving the remaining 10 for validation and testing. Subsequently, we
generate 2400 frames from the training set and 400 each for the validation and
test sets forming a total of 3200 frames. We then manually label these images
with LabelImg [Tzutalin, 2015] annotation tool with resulting bounding boxes
in YOLOv8 format for training. Fig 1 illustrates the process of manual
annotation.

Figure 1: Labelling process of event frames with LabelImg tool.

Training: YOLOv8 [Jocher et al., 2023] offers enhanced accuracy and speed,
making it suitable for real-time applications. YOLOv8 utilizes a novel
architecture that builds upon the success of previous YOLO versions while
introducing several key improvements. The backbone of YOLOv8 is a modified
version of the CSPDarknet53 feature extractor, which employs cross-stage
partial connections to enhance information flow between layers. Instead of the
traditional Feature Pyramid Network (FPN), YOLOv8 utilizes a C2f module to
combine high-level semantic features. C2f represents two $3\times 3$
convolutions with a residual connection which accepts outputs from the
bottleneck and concatenates it. The bottleneck is the same as in YOLOv5 with
changes made to the size of the first convolutional layer from $1\times 1$ to
$3\times 3$ [Sohan et al., 2024]. The head of the network consists of multiple
detection heads, each responsible for predicting bounding boxes, class
probabilities, and objectness scores at different scales. YOLOv8 also
incorporates a self-attention mechanism in the head to focus on relevant
features and adjust their importance based on the task. In our study, we
utilised the event frames generated to train YOLOv8. The dataset, comprising
3200 accumulated frames, was divided into a training, validation and test set
with a ratio of 70:15:15 respectively. The training process was implemented in
PyTorch and trained on an NVIDIA GeForce RTX 2080 Ti GPU. An AdamW optimizer
was utilised in both models to achieve the highest performance with a learning
rate of $1\times 10^{-3}$ and a weight decay of $1\times 10^{-3}$.

##  4 Experiments and Results

Table 1 presents the performance metrics of four different variants of the
YOLOv8 model (YOLOv8n, YOLOv8s, YOLOv8m, and YOLOv8l) on the task of pupil
tracking using the EV-EYE data. The models are evaluated based on mean average
precision (mAP), precision, recall, F1 score, and the number of parameters (in
millions).

In terms of mean average precision (mAP), which measures the accuracy and
completeness of the model’s predictions by calculating the average precision
across all classes, YOLOv8n achieved the highest score of 0.981, indicating
its superior performance in accurately identifying the pupil. The other
models, YOLOv8s, YOLOv8m, and YOLOv8l, achieved slightly lower mAP scores, all
around 0.975 to 0.976. Precision, which measures the accuracy of the positive
predictions, was also highest for YOLOv8n at 0.965, signifying the lowest
false positive rate among the models. The precision for the other models was
slightly lower, with YOLOv8s at 0.950, YOLOv8m at 0.949, and YOLOv8l at 0.944.

Recall, which measures the ability to detect true positives, was highest for
YOLOv8l at 0.938. True positive parameters are defined by two key thresholds:
the Intersection over Union (IoU) threshold and the confidence threshold. The
IoU threshold determines the minimum overlap required between a predicted
bounding box and a ground truth bounding box for the prediction to be
considered a true positive. An IoU threshold of 0.5 is used, meaning the
predicted box must have at least 50% overlap with the ground truth box to be
counted as a true positive. The confidence threshold, on the other hand,
defines the minimum confidence score a prediction must have to be considered a
true positive. YOLOv8n had a recall of 0.919, which was slightly lower than
the other models, ranging from 0.92 to 0.927.

The F1 score, which is the harmonic mean of precision and recall, was highest
for YOLOv8n and YOLOv8l at 0.94. YOLOv8s and YOLOv8m had a slightly lower F1
score of 0.93. Regarding the number of parameters, YOLOv8n had the fewest at
3.0 million, making it the most lightweight model. The number of parameters
increased with model size: YOLOv8s had 11.1 million, YOLOv8m had 25.8 million,
and YOLOv8l had 43.6 million.

In summary, the YOLOv8n model, despite having the fewest parameters, performed
exceptionally well across all metrics, particularly excelling in mAP and
precision. This makes it a highly efficient model for pupil tracking,
balancing accuracy with computational efficiency. The larger models, YOLOv8s,
YOLOv8m, and YOLOv8l, showed strong performance as well, with slight
variations in precision, recall, and F1 score, suggesting that while they may
offer slight improvements in certain areas, they do so at the cost of
increased computational complexity.

YOLOv8 Model | Mean Average Precision | Precision | Recall | F1 Score | Parameters (M)
---|---|---|---|---|---
YOLOv8-n (nano) | 0.981 | 0.965 | 0.919 | 0.94 | 3.0
YOLOv8-s (small) | 0.976 | 0.950 | 0.920 | 0.93 | 11.1
YOLOv8-m (medium) | 0.976 | 0.949 | 0.927 | 0.93 | 25.8
YOLOv8-l (large) | 0.975 | 0.944 | 0.938 | 0.94 | 43.6
Table 1: Results of 4 different YOLOv8 variants trained using the EV-EYE data

Figure 2 shows the training and validation loss curves for YOLOv8-n on the
pupil tracking task. These curves illustrate how the models’ performance
improved over the training period, indicating the effectiveness of the
training process and the stability of the models. The upper rows present the
training outcomes, whereas the lower rows show the results on the validation
set. It is observed that the loss overall decreases with more epochs
indicating high accuracy. The first three columns depict the box loss, object
loss, and classification loss of the improved YOLOv8 model. Additionally, it
is observed that the validation losses exhibit greater noise and are
marginally higher than the training losses, indicating some degree of
overfitting. The precision, recall, and mAP metrics display more variability
in the validation sets compared to the training sets. Therefore, although the
model demonstrates good learning behavior, the fluctuations and slight
inconsistencies in the validation metrics suggest a susceptibility to
overfitting. Potential solutions to address this issue in future work include
incorporating attention models and assembling YOLO variants.

Figure 2: Training and Validation Losses for YOLOv8 on pupil tracking. The
horizontal axes represent the number of epochs, while the vertical axes
represent the value of each metric during training.

Figure 3 displays the qualitative results of pupil detection. As seen in the
labels on the right hand side of the image, some images were not annotated to
avoid inaccuracies in frames where the pupil were not fully visible. This
happens during certain eye movements such as blinks when the events generated
from movement from other eye parts is lesser than the pupil. However, the
model demonstrates robust performance by localizing the position of the pupil
even when it is half visible. It is evident that the model is highly robust,
maintaining pupil detection across various eye movements, even when the eye is
nearly (frame_1092) closed. Whether moving left or right(frame_1052), the
model consistently demonstrates robust performance.

Figure 3: Qualitative results of the best performing proposed model
(YOLOv8-n).The image on the left indicates ground truth labels while the image
on the right indicates predicted labels along with confidence scores.

##  5 Limitations

Our current model faces challenges when tested on remote datasets, which is
not surprising given that we trained the models using a near-eye dataset.
Although our focus is on applications in both near and remote eye sensing, we
did not have a sufficient remote-eye dataset for training. As a result, the
model’s performance in identifying pupils in the presence of occlusions is
less robust, and this may lead to significant errors. To address this issue,
we plan to focus on acquiring and incorporating comprehensive remote eye
datasets for further experiments, and we also intend to incorporate Region-of-
Interest (ROI) techniques for directly extracting eye features to ensure
robustness in all conditions and even in the presence of occlusions. This
addition is expected to enhance the accuracy and robustness of our pupil
tracking pipeline, paving the way for a saccade detection algorithm that can
provide more reliable and precise outcomes. However, it is important to note
that the model still performs fairly well in terms of pupil localization. We
show this in the results presented in supplementary materials.

Additionally, while the YOLOv8n model performed exceptionally well in terms of
mean average precision (mAP) and precision, its recall was slightly lower
compared to the other models. Specifically, the YOLOv8n had a recall of 0.919,
which is lower than the recall of 0.938 for YOLOv8l, and slightly below the
recall values for YOLOv8s and YOLOv8m. This suggests that the YOLOv8n model
may miss some true positives, which can be critical in applications where
detecting every instance is essential.

##  6 Conclusion

In this study, we demonstrated pupil tracking in event cameras using a frame-
based representation of events. By converting data from Event Cameras into a
format understood by standard deep learning algorithms, we overcame issues
such as under-sampling by generating events with duration of 10ms (100 fps)
and highlighted the potential of the high frame rate offered by ECs. The
presented results affirm the reliability and applicability of event camera
imaging in pupil tracking. Our findings and evaluation of various iterations
of YOLOv8 highlights how deep learning algorithms can be adopted and
transferred to dense representations such as event frames. ECs have
demonstrated promising potential for object detection, especially in scenarios
involving rapid motion and have the potential for the detection of high-speed
changes. The use of ECs for pupil tracking has offered opportunities for high
speed tracking in applications in XR, VR, Human Computer Interactions, e.t.c.

Moving forward, the integration of ECs in pupil tracking presents exciting
prospects for other areas involving analyzing eye movement. For instance,
saccade detection which is essential for understanding human cognition and
improving visual diagnostics of neurological conditions. The potential of such
applications can lead to new avenues and breakthroughs in non-invasive
diagnosis of neurological conditions such as Dementia and Parkinson’s disease
with further research and development required.

## Acknowledgments

This research was conducted with the financial support of Science Foundation
Ireland under grant no. [12/RC/2289_P2] at the Insight SFI Research Centre for
Data Analytics, Dublin City University in collaboration with FotoNation
(Tobii).

##  Appendix A Appendix 1

We evaluate the robustness of the proposed approach by testing on data from
different variations of ECs and include the results in supplementary
materials.

## References

  * [Ahmad et al., 2024] Ahmad, S., Morerio, P., and Del Bue, A. (2024).  Event anonymization: Privacy-preserving person re-identification and pose estimation in event-based vision.  IEEE Access.
  * [Bittencourt et al., 2013] Bittencourt, J., Velasques, B., Teixeira, S., Basile, L. F., Salles, J. I., Nardi, A. E., Budde, H., Cagy, M., Piedade, R., and Ribeiro, P. (2013).  Saccadic eye movement applications for psychiatric disorders.  Neuropsychiatric Disease and Treatment, pages 1393–1409.
  * [Gallego et al., 2020] Gallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., et al. (2020).  Event-based vision: A survey.  IEEE transactions on pattern analysis and machine intelligence, 44(1):154–180.
  * [Jocher et al., 2023] Jocher, G., Chaurasia, A., and Jiu, J. (2023).  Yolo by ultralytics (version 8.0.0).  https://github.com/ultralytics/ultralytics.  Accessed: 2024-07-09.
  * [Kagemoto and Takemura, 2023] Kagemoto, T. and Takemura, K. (2023).  Event-based pupil tracking using bright and dark pupil effect.  In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–3.
  * [Kang et al., 2019] Kang, D., Heo, J., Kang, B., and Nam, D. (2019).  Pupil detection and tracking for ar 3d under various circumstances.  Electronic Imaging, 31:1–5.
  * [Kang and Kang, 2023] Kang, D. and Kang, D. (2023).  Event camera-based pupil localization: Facilitating training with event-style translation of rgb faces.  IEEE Access.
  * [Kang et al., 2023] Kang, D., Lee, Y. K., and Jeong, J. (2023).  Exploring the potential of event camera imaging for advancing remote pupil-tracking techniques.  Applied Sciences, 13(18):10357.
  * [Khan et al., 2020] Khan, W., Hussain, A., Kuru, K., and Al-Askar, H. (2020).  Pupil localisation and eye centre estimation using machine learning and computer vision.  Sensors, 20(13):3785.
  * [Klaib et al., 2021] Klaib, A. F., Alsrehin, N. O., Melhem, W. Y., Bashtawi, H. O., and Magableh, A. A. (2021).  Eye tracking algorithms, techniques, tools, and applications with an emphasis on machine learning and internet of things technologies.  Expert Systems with Applications, 166:114037.
  * [Lee et al., 2020] Lee, K. I., Jeon, J. H., and Song, B. C. (2020).  Deep learning-based pupil center detection for fast and accurate eye tracking system.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16, pages 36–52. Springer.
  * [Rebecq et al., 2019] Rebecq, H., Ranftl, R., Koltun, V., and Scaramuzza, D. (2019).  Events-to-video: Bringing modern computer vision to event cameras.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3857–3866.
  * [Shariff et al., 2024] Shariff, W., Dilmaghani, M. S., Kielty, P., Moustafa, M., Lemley, J., and Corcoran, P. (2024).  Event cameras in automotive sensing: A review.  IEEE Access.
  * [Sohan et al., 2024] Sohan, M., Sai Ram, T., Reddy, R., and Venkata, C. (2024).  A review on yolov8 and its advancements.  In International Conference on Data Intelligence and Cognitive Informatics, pages 529–545. Springer.
  * [Tzutalin, 2015] Tzutalin, D. (2015).  Labelimg. git code.
  * [Wang et al., 2019] Wang, L., Ho, Y.-S., Yoon, K.-J., et al. (2019).  Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10081–10090.
  * [Wang et al., 2024] Wang, Z., Gao, C., Wu, Z., Conde, M. V., Timofte, R., Liu, S.-C., Chen, Q., Zha, Z.-j., Zhai, W., Han, H., et al. (2024).  Event-based eye tracking. ais 2024 challenge survey.  arXiv preprint arXiv:2404.11770.
  * [Yang et al., 2024] Yang, H. W., Choe, J. Y., Noh, S. R., Kim, J. L., Han, J. W., and Kim, K. W. (2024).  Exploring age-related changes in saccades during cognitive tasks in healthy adults.  Frontiers in Behavioral Neuroscience, 17:1301318.
  * [Zhang et al., 2024a] Zhang, P., Zhu, S., and Lam, E. Y. (2024a).  Event encryption: rethinking privacy exposure for neuromorphic imaging.  Neuromorphic Computing and Engineering, 4(1):014002.
  * [Zhang et al., 2024b] Zhang, T., Shen, Y., Zhao, G., Wang, L., Chen, X., Bai, L., and Zhou, Y. (2024b).  Swift-eye: Towards anti-blink pupil tracking for precise and robust high-frequency near-eye movement analysis with event cameras.  IEEE Transactions on Visualization and Computer Graphics.
  * [Zhao et al., 2024] Zhao, G., Yang, Y., Liu, J., Chen, N., Shen, Y., Wen, H., and Lan, G. (2024).  Ev-eye: Rethinking high-frequency eye tracking through the lenses of event cameras.  Advances in Neural Information Processing Systems, 36.