# Sequential minimum optimization algorithm with small sample size estimators

Wojciech Roga1<EMAIL_ADDRESS>Takafumi Ono2,3
<EMAIL_ADDRESS>Masahiro Takeoka1,4<EMAIL_ADDRESS>
1Department of Electronics and Electrical Engineering, Keio University, 3-14-1
Hiyoshi, Kohoku, Yokohama, Kanagawa 223-8522, Japan  2Program in Advanced
Materials Science Faculty of Engineering and Design, Kagawa University,
2217-20 Hayashi-cho, Takamatsu, Kagawa 761-0396, Japan  3JST, PRESTO, 4-1-8
Honcho, Kawaguchi, Saitama 332-0012, Japan  4Advanced ICT Research Institute,
National Institute of Information and Communications Technology (NICT), 4-2-1
Nukuikita, Koganei, Tokyo 184-8795, Japan

###### Abstract

Sequential minimum optimization is a machine-learning global search training
algorithm. It is applicable when the functional dependence of the cost
function on a tunable parameter given the other parameters can be cheaply
determined. This assumption is satisfied by quantum circuits built of known
gates. We apply it to photonics circuits where the additional challenge
appears: low frequency of coincidence events lowers the speed of the
algorithm. We propose to modify the algorithm such that small sample size
estimators are enough to successfully run the machine learning task. We
demonstrate the effectiveness of the modified algorithm applying it to a
quantum optics classifier with data reuploading.

##  I Motivation

There are a variety of proposals how to use quantum devices in machine
learning [1, 2, 3, 4, 5, 6]. The research is motivated by the expectation that
quantum features can bring benefits for the performance. Indeed, many
algorithms use linear algebra modules, which where shown to be solved faster
under some conditions when applied on a quantum device [7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 20]. Also Grover search algorithm if applicable
can speed up the algorithm [21]. Or, sampling from a programmable quantum
device can be faster than calculating the expectation values of certain
observables [22, 23, 24, 25, 26, 27, 28, 29]. Moreover, the access to
entangled states allow one to produce different statistical dependencies than
only classical states on a fixed processors as the entangled states occupy a
significant volume of all quantum states of a given system [30, 31].

Among the supervised quantum machine learning algorithms [32, 33, 34, 35, 36,
37, 38] we find hybrid quantum-classical ones with classically programmable
circuits. Their parameters are trained by classical algorithms such as
gradient descent, and the cost function is constructed based on the
expectation values of observables that are measured. The speed at which the
training algorithms find the global minimum of the cost function is part of
the efficiency of the total algorithm.

There are local search algorithms such as gradient descent, in which the
search starts from a chosen starting point and explores the cost function
landscape further from that point gradually. Also global search algorithms
recognise and explore some global features of the landscape.

One of the global search algorithms proposed for the quantum circuits recently
is the so-called sequential minimum optimization [39] (SMO). It is based on
the idea, that from a small number of measurements, one can infer the
functional dependence of the cost function along one parameter that typically
is a known quantum gate or an optical element like a phase shifter [40].
Knowing this, we can in one step update this parameter with the value that
corresponds to the minimum of the cost function along this variable. This
procedure can be applied sequentially to all parameters of the circuit
converging in a few rounds to the minimum of the cost function landscape,
which is possibly far from the starting point. This algorithm, unlike local
search algorithms, is not affected by barren plateaus and does not rely on
additional parameters such as the size of the step. In [39] where the authors
introduced this procedure for the qubit quantum circuits, the convergence was
also discussed.

In [41], sequential minimum optimization was used to an integrated photonics
based quantum classifier in an experiment that showed computational abilities
of two-mode quantum circuit. This paper followed [31], in which the authors
first showed that a sufficiently long single qubit quantum circuit satisfies
the universal approximation theorem [42], known from the theory of a single-
layer neural networks, and is able to classify data points divided by an
arbitrary border function. The authors performed simulations of several
problems using gradient descent training.

However in the photonic circuit experiment with two photons performed in [41]
the sequential minimal optimization was preferred as the time of measuring
expectation values of coincident photons is longer. Coincidence counts in this
experiment allowed for reducing decoherence errors. However, this type of
experiment is strongly affected by losses which induce a long time of
gathering enough data in postselection.

The experimental run-time needed to train the circuit with the SMO algorithms
is proportional to:

$\displaystyle t$ $\displaystyle=$
$\displaystyle\\#rounds*\\#circuit\\_parameters*\\#training\\_points$
$\displaystyle*\\#function\\_coefficients*\\#estimator\\_measurements,$

where the last term is the number of measurements, we need to estimate the
expectation value of the observable measured in the experiment and used in the
cost function. The number of function coefficients is the number of values for
a given training point needed to specify the functional dependence of the cost
function on one parameter of the circuit in the SMO algorithm. The number of
rounds is how many times the algorithm updates all parameters. The number of
circuit parameters and the number of training points are clear.

The number of rounds can be kept small due to the efficiency of the sequential
minimal optimization algorithm [39]. In the experiment reported in [41], it
was equal to 10. The number of function parameters, as we will explain later
discussing details of the algorithm, is $O(n)$, where $n$ is the number of
photons in the circuit. The number of detector clicks to estimate the
expectation value of an observable, in principle, can be arbitrary large and
in experiment with coincidence measurements can dominate the total time of the
training process.

The goal of this paper is to modify the sequential minimal optimization
algorithm such that the expectation value of the observable is replaced there
by the estimator of this value based on a small sample size. If the number of
coincidence events used to estimate a given expectation value of the measured
observation was around 200, our modification can reduce the experiment run-
time even by 100 times. With this reduction the weak sources generating for
instance, entangled states can still be efficiently used in machine learning
experiments with many training data points.

This approach is inspired by [43], where the authors discuss in detail the
problem of replacing expectation values by unbiased estimators for the
gradient descent type training. Then, estimation of the first and sometimes
second derivatives of the cost functions at given points are needed. The
authors show that the training based on the estimators works well and the
computational time related to the measurements can be significantly reduced.
The driving idea of the method is the observation that since the cost function
is an unbiased estimator of the sum of expectation values of independent
variables, it can be replaced by the sum of unbiased estimators of the
variables. The error in using estimators from a finite number of measurements
depends on the number of measurements and the number of training points.
Therefore, if the number of training points is sufficiently large, the number
of measurements used for the estimators of the expectation value can be kept
small.

In this manuscript, we discuss in detail the idea of [43] adapted to the
sequential minimal optimization. As [43] covers gradient descent-type
training, we add something novel to the development of the SMO method.

The main result is formulated as Theorem 1, it is proven and supported by the
simulation of a 2-mode photonic optical classifier with a NOON state as the
input. Although the main message of our paper does not require the NOON state,
and could be shown with other fixed numbers of photon states, we decided to
use the NOON state as a tribute to Jonathan Dowling [44], our friend and
collaborator. There are other advantages from using entangled states in this
kind of experiments, but it is not our goal to discuss them here.

The paper is organized as follows. First, we give details of the sequential
minimal optimization for a photonic circuit. This method follows the analogous
algorithm derived earlier in [39] for qubit circuits however our notation and
conventions are slightly different. Then, we present the main result of our
paper in the form of a Theorem; we give a detailed proof and discuss the
convergence of the estimator of the cost function to the real cost function
when the number of training points increases. In the following section we
present numerical simulator of a photonic classifier. We summarize our work
with conclusions and an outlook.

##  II Sequential minimal optimization in a photonic circuit

Figure 1: General passive photonic circuit with the input state
$|\Psi_{in}\rangle$ of a fixed number of photons. We distinguish a phase
shifter depending on a parameter $\varphi$. We want to know the dependence of
the probability of $|\Psi_{out}\rangle$ at the end of the circuit on
$\varphi$. Here, unitary transformations $U_{1}$ and $U_{2}$ can be arbitrary
photon number preserving transformations.

We consider a general passive optical circuit. Without losing generality, we
can compose the circuit with phase-shifters and 50-50 beam-splitters applying
for example, the so-called Reck configuration [45, 46]. Therefore, we can
assume that the parameters of the circuit are phases of the phase-shifters.
However, the method we discuss below can be easily adapted to circuits with
any passive optical elements single- or multi-mode.

We assume that an $n$-photon state $|\Psi_{in}\rangle$ is put as the input and
an $n$-photon state $|\Phi_{out}\rangle$ is measured. They can be arbitrary.
We assume that a given data point, possibly repeatedly (data-reuploading), is
encoded in some phases.

We consider the cost function to be a function of the probability of measuring
an $n$-photon state $|\Psi_{out}\rangle$, trainable phases, and data. For a
training point indexed by $i$, let us write the probability of measuring
$|\Psi_{out}\rangle$ distinguishing one of the trainable variables $\varphi$
to as in figure 1,

$p^{i}(\varphi)=|\langle\Psi_{out}|U_{2}^{\dagger}U_{\varphi}U_{1}|\Psi_{in}\rangle|^{2}.$
(1)

Here, $U_{1}$ and $U_{2}$ are unitaries before and after a phase-shifter
$U_{\varphi}$ parameterized by $\varphi$. The idea is to update each parameter
$\varphi$ to the value where the cost function expressed as the function of
this parameter achieves its minimum. To do that we need to derive an explicit
formula for $p^{i}(\varphi)$ for each $\varphi$ in each step of the training.
Therefore, assume we are interested in the functional dependence of $p^{i}$ on
phase $\varphi$ of a phase-shifter in mode $m$. Let us expand
$U_{1}|\Psi_{in}\rangle$ and $U_{2}|\Psi_{out}\rangle$ in the Fock basis
writing mode $m$ as the first one

$\displaystyle U_{1}|\Psi_{in}\rangle$ $\displaystyle=$
$\displaystyle\alpha_{0}|0\rangle_{m}|\xi_{0}\rangle+\alpha_{1}|1\rangle_{m}|\xi_{1}\rangle+...+\alpha_{n}|n\rangle|\xi_{n}\rangle,$
$\displaystyle U_{2}|\Psi_{out}\rangle$ $\displaystyle=$
$\displaystyle\beta_{0}|0\rangle_{m}|\xi^{\prime}_{0}\rangle+\beta_{1}|1\rangle_{m}|\xi^{\prime}_{1}\rangle+...+\beta_{n}|n\rangle|\xi^{\prime}_{n}\rangle.$

Acting by the phase-shifter on $U_{1}|\Psi_{in}\rangle$ we get

$\displaystyle U_{\phi}U_{1}|\Psi_{in}\rangle$ $\displaystyle=$
$\displaystyle\alpha_{0}|0\rangle_{m}|\xi_{0}\rangle+\alpha_{1}e^{i\varphi}|1\rangle_{m}|\xi_{1}\rangle$
$\displaystyle+$
$\displaystyle\alpha_{2}e^{2i\varphi}|2\rangle_{m}|\xi_{2}\rangle+...+\alpha_{n}e^{ni\varphi}|n\rangle|\xi_{n}\rangle.$

Hence, it easily follows from (1) that

$\displaystyle p^{i}(\varphi)$ $\displaystyle=$ $\displaystyle
A_{0}+A_{1}\cos(\varphi)+A_{2}\sin(\varphi)$ (2) $\displaystyle+$
$\displaystyle A_{3}\cos(2\varphi)+A_{4}\sin(2\varphi)+...,$

If we knew $A_{0},...,A_{2n+1}$, we would have $p^{i}(\varphi)$. Values
$A_{j}$ can be obtained by solving the set of $2n+1$ equations given by (2)
measured for $2n+1$ different $\varphi$. For simplicity, let us assume $n=2$.
Then, we can figure out the values of $A_{j}$ from measuring probabilities
denoted here by $r_{j}=p^{i}(\varphi_{j})$ for $5$ different values of the
phase (generalization of this procedure is straightforward)

$\begin{pmatrix}1&\cos\varphi_{0}&\sin\varphi_{0}&\cos 2\varphi_{0}&\sin
2\varphi_{0}\\\ 1&\cos\varphi_{1}&\sin\varphi_{1}&\cos 2\varphi_{1}&\sin
2\varphi_{1}\\\ 1&\cos\varphi_{2}&\sin\varphi_{2}&\cos 2\varphi_{2}&\sin
2\varphi_{2}\\\ 1&\cos\varphi_{3}&\sin\varphi_{3}&\cos 2\varphi_{3}&\sin
2\varphi_{3}\\\ 1&\cos\varphi_{4}&\sin\varphi_{4}&\cos 2\varphi_{4}&\sin
2\varphi_{4}\end{pmatrix}\begin{pmatrix}A_{0}\\\ A_{1}\\\ A_{2}\\\ A_{3}\\\
A_{4}\end{pmatrix}=\begin{pmatrix}r_{0}\\\ r_{1}\\\ r_{2}\\\ r_{3}\\\
r_{4}\end{pmatrix}.$ (3)

Values $r_{k}$ are the probabilities measured by the same circuit with 5
chosen phases $\varphi_{k}$. In particular, if one choses

$\displaystyle\varphi_{0}=0,\quad\varphi_{1}=\frac{2\pi}{5},\quad\varphi_{2}=-\frac{2\pi}{5},\quad\varphi_{3}=\frac{4\pi}{5},\quad\varphi_{4}=-\frac{4\pi}{5},$

equation (3) takes the form of the easily invertible Fourier transform

$F\begin{pmatrix}(A_{3}-iA_{4})/2\\\ (A_{1}-iA_{2})/2\\\ A_{0}\\\
(A_{1}+iA_{2})/2\\\ (A_{3}+iA_{4})/2\end{pmatrix}=\begin{pmatrix}r_{3}\\\
r_{1}\\\ r_{0}\\\ r_{2}\\\ r_{4}\end{pmatrix},$ (4)

where the Fourier transform matrix is

$F=\begin{pmatrix}\omega^{4}&\omega^{2}&\omega^{0}&\omega^{-2}&\omega^{-4}\\\
\omega^{2}&\omega^{1}&\omega^{0}&\omega^{-1}&\omega^{-2}\\\
\omega^{0}&\omega^{0}&\omega^{0}&\omega^{0}&\omega^{0}\\\
\omega^{-2}&\omega^{-1}&\omega^{0}&\omega^{1}&\omega^{2}\\\
\omega^{-4}&\omega^{-2}&\omega^{0}&\omega^{2}&\omega^{4}\\\ \end{pmatrix},$
(5)

and $\omega=e^{i\frac{2\pi}{5}}$. Hence

$\begin{pmatrix}A_{3}\\\ A_{1}\\\ A_{0}\\\ A_{2}\\\
A_{4}\end{pmatrix}=\frac{2}{5}\begin{pmatrix}\cos\frac{8\pi}{5}&\cos\frac{4\pi}{5}&1&\cos\frac{4\pi}{5}&\cos\frac{8\pi}{5}\\\
\cos\frac{4\pi}{5}&\cos\frac{2\pi}{5}&1&\cos\frac{2\pi}{5}&\cos\frac{4\pi}{5}\\\
0.5&0.5&0.5&0.5&0.5\\\
\sin\frac{4\pi}{5}&\sin\frac{2\pi}{5}&0&-\sin\frac{2\pi}{5}&-\sin\frac{4\pi}{5}\\\
\sin\frac{8\pi}{5}&\sin\frac{4\pi}{5}&0&-\sin\frac{4\pi}{5}&-\sin\frac{8\pi}{5}\end{pmatrix}\begin{pmatrix}r_{3}\\\
r_{1}\\\ r_{0}\\\ r_{2}\\\ r_{4}\end{pmatrix}.$ (6)

This shows that to obtain the functional dependence of the probability
$p(\varphi)$ in (2) on each adjustable circuit parameter $\varphi$, we need to
measure $2n+1$ probabilities $r_{j}$, where $n$ is the number of photons, with
the same circuit but with different values of the parameter under
investigation.

Notice that with the effort of finding $A_{j}$ for each training point, we can
find an explicit formula for the cost function and its minimum for each
adjustable parameter. Hence, we can update the parameter with the value in
which the cost function achieves the minimum, which is the key idea of SMO.
This procedure is applied in the analysis of the following section.

##  III Sequential Minimum Optimization with small sample size estimators

For each training point enumerated by index $i$, for each phase shifter
characterized by its $\varphi$, we measure $2n+1$ probabilities $r_{j}$ for
$2n+1$ phases, as in the previous section. From the probabilities we calculate
parameters $A_{k}^{i}$ that allow us to define the function $p^{i}(\varphi)$
as in (2) that enters to the cost function

$C(\varphi)=\frac{1}{N}\sum_{i=1}^{N}(p^{i}(\varphi)-y^{i})^{2},$ (7)

which is still an easy function of $\varphi$ that can be efficiently minimized
numerically. The phase of the phase shifter is updated as follows

$\varphi\rightarrow{\rm argmin}\ C(\varphi),$ (8)

and we move to the next phase shifter. We repeat everything until a stopping
criterion is met.

However, in an experiment with photonic circuits to estimate a real single
probability $r_{j}$, we use its estimator $\tilde{r}_{j}=N_{success}/N_{all}$,
which is assumed to be a random variable with a Binomial distribution. The
related mean square error is scaled with the sample size $N_{all}$ as
$O(1/N_{all})$. To keep the error small, we typically collect data from a few
hundreds of coincidence events. The sample size depends on the intensity of
the source and time we want to spend for the total experiment. For example, in
[41], estimating a single probability $\tilde{r}_{j}$ took around a second
with a weak coherent state source that produced around 3000 two-photon
coincidences in this interval. In experiments with multiple photons or with
sources of entangled photons, the time of gathering similar statistics would
be significantly increased and running the experiment would be inefficient.

However, in what follows, we show that we can successfully run our experiment
with strongly smaller sample size estimators, i.e., when $N_{all}$ is small.
Indeed, some tasks can be satisfactory solve with as little sample size as
$N_{all}=1$ in which case $\tilde{r}_{j}$ is binary.

Therefore, for each training point indexed by $i$, using the estimators
$\tilde{r}_{j}^{i}$, we can calculate the estimators of $A^{i}_{k}$ denoted by
$\tilde{A}^{i}_{k}$ as in Eq. (6) and define the estimator of $p^{i}(\varphi)$
analogous to (2) as follows

$\tilde{p}^{i}(\varphi)=\tilde{A}^{i}_{0}+\tilde{A}_{1}^{i}\cos(\phi)+\tilde{A}_{2}^{i}\sin(\varphi)+...$
(9)

We prove the following theory that shows that for a large number of the
training points $N$ a function built from these estimators converges to the
cost function built from their expectation values.

###### Theorem 1

Let $N$ be the number of training points and

$\tilde{C}(\varphi)=\frac{1}{N}\sum_{i}^{N}\Big{(}\tilde{p}^{i}(\varphi)(\tilde{p}^{i})^{\prime}(\varphi)-2\tilde{p}^{i}(\varphi)y^{i}+(y^{i})^{2}\Big{)},$
(10)

where $\tilde{p}^{i}(\varphi)$ and $(\tilde{p}^{i})^{\prime}(\varphi)$ are
functions (9) generated from separate independent measurements for the same
training point. For an arbitrary unbiased estimator of the probabilities of
experimental outcomes and for sufficiently large $N$

${\rm argmin}\ \tilde{C}(\varphi)\rightarrow{\rm argmin}\ C(\varphi)$ (11)

Proof. Let us expand $\tilde{C}(\varphi)$

$\displaystyle\tilde{C}(\varphi)$ $\displaystyle=$
$\displaystyle\frac{1}{N}\sum_{i}^{N}\Big{[}\Big{(}\tilde{A}_{0}^{i}+\tilde{A}_{1}^{i}\cos(\varphi)+\tilde{A}_{2}^{i}\sin(\varphi)+...\Big{)}$
$\displaystyle\ \ \ \ \ \ \ \
*\Big{(}(\tilde{A}_{0}^{i})^{\prime}+(\tilde{A}_{1}^{i})^{\prime}\cos(\varphi)+(\tilde{A}_{2}^{i})^{\prime}\sin(\varphi)+...\Big{)}$
$\displaystyle-$ $\displaystyle
2\Big{(}\tilde{A}_{0}^{i}+\tilde{A}_{1}^{i}\cos(\varphi)+\tilde{A}_{2}^{i}\sin(\varphi)+...\Big{)}y^{i}+(y^{i})^{2}\Big{]}$
$\displaystyle=$
$\displaystyle\frac{1}{N}\sum_{i}^{N}\tilde{A}^{i}_{0}(\tilde{A}^{i}_{0})^{\prime}+\frac{1}{N}\sum_{i}^{N}\tilde{A}^{i}_{0}(\tilde{A}^{i}_{1})^{\prime}\cos(\varphi)+...$
$\displaystyle-$ $\displaystyle
2\frac{1}{N}\sum_{i}^{N}\tilde{A}_{0}^{i}y_{i}-2\frac{1}{N}\sum_{i}^{N}\tilde{A}_{1}^{i}y^{i}\cos(\varphi)+...$
$\displaystyle+$ $\displaystyle\frac{1}{N}\sum_{i}^{N}(y^{i})^{2}.$

Let us look first at the term linear in $\tilde{A}^{i}_{0}$. We express it by
the estimators of the probabilities $\tilde{r}^{i}_{k}$

$X=\frac{1}{N}\sum_{i}^{N}\tilde{A}_{0}^{i}y_{i}=\frac{1}{N}\sum_{i}^{N}(w^{0}_{0}\tilde{r}^{i}_{0}+w^{0}_{1}\tilde{r}^{i}_{1}+...)y^{i},$
(12)

where $w$ are appropriate constants from equation (6). Notice that as
$\tilde{r}^{i}_{j}$ are random variables with expectation values $r^{i}_{j}$,
also $X$ is a random variable with the expectation value

$\frac{1}{N}\sum_{i}^{N}(w^{0}_{0}r^{i}_{0}+w^{0}_{1}r^{i}_{1}+...)y^{i}$ (13)

here $r^{i}_{j}$ are the values of the real probabilities from the experiment.
The standard deviation of $X$ scales with $N$ such as
$\frac{\sigma^{i}}{\sqrt{N}}$, where $(\sigma^{i})^{2}$ is the variance of
$\tilde{r}^{i}_{j}$. The expectation value of $X$ is the same as the term
computed from the perfect probabilities $r^{i}_{j}$, with the standard
deviation vanishing with $N$. The same reasoning holds for other linear terms
of $\tilde{C}(\varphi)$. This implies that the linear terms of
$\tilde{C}(\varphi)$ tend to the analogous terms of $C(\varphi)$.

Let us consider now the quadratic term

$\displaystyle Y$ $\displaystyle=$
$\displaystyle\frac{1}{N}\sum_{i}^{N}\tilde{A}^{i}_{0}(\tilde{A}^{i}_{0})^{\prime}$
(14) $\displaystyle=$
$\displaystyle\frac{1}{N}\sum_{i}^{N}(w^{0}_{0}\tilde{r}^{i}_{0}+w^{0}_{1}\tilde{r}^{i}_{1}+...)(w^{0}_{0}(\tilde{r}^{i}_{0})^{\prime}+w^{0}_{1}(\tilde{r}^{i}_{1})^{\prime}+...)$
$\displaystyle=$
$\displaystyle\frac{1}{N}\sum_{i}^{N}w^{0}_{0}w^{0}_{0}\tilde{r}^{i}_{0}(\tilde{r}^{i}_{0})^{\prime}+...$

For $\tilde{r}^{i}_{0}$ and $(\tilde{r}^{i}_{0})^{\prime}$ being independent
random variables,
$\frac{1}{N}\sum_{i}^{N}w^{0}_{0}w^{0}_{0}\tilde{r}^{i}_{0}(\tilde{r}^{i}_{0})^{\prime}$
is a random variable with the expectation value equal to
$(w^{0}_{0})^{2}(r^{i}_{0})^{2}$. Similarly, other terms of $Y$. Therefore,
the expectation value of $Y$ is
$\frac{1}{N}\sum_{i}^{N}(w^{0}_{0})^{2}(r^{i}_{0})^{2}+...$ that is the same
as the analogous term in $C(\varphi)$. By the same argument as previously, the
variance of $Y$ vanishes with $N$. The same reasoning can be applied to the
other quadratic terms.

Concluding, the coefficients in front of trigonometric functions in
$\tilde{C}(\varphi)$ tend to the appropriate coefficients of $C(\varphi)$ when
$N$ is sufficiently large. QED.

##  IV Simulation of photonic classifier with data reuploading

As a demonstration of this method, let us simulate a fragment of a simple
optical universal classifier. The idea of a universal quantum classifier with
a single qubit was developed in [31]. In this scenario, two-dimensional data
points $x$ to be classified are encoded as parameters of a single qubit
unitary transformation $U(\varphi,x)$ of the qubit initialized in the ground
state $|0\rangle$. This transformation also contains the free parameters
$\varphi$, which are tuned during the learning process. Similarly, more
unitary gates can be introduced sequentially with different free parameters,
but the same data $x$ as in figure 2. This provides the nonlinearity of the
expectation values of the output states in the variable encoding the data.
After the circuit, the qubit is measured. The classification is based on the
results. The free parameters $\varphi$ are adjusted during the training of the
system.

Figure 2: Layers of the single qubit classifier with data re-uploading [31].

The authors of [31] provide arguments that in this way, one can arbitrarily
approximate any function of the input points achieving a universal classifier.
The arguments are based on the analogy to the universality of neural networks.

Figure 3: A two mode unitary transformation built of phase-shifters and the
Mach-Zender interferometer.

In [41] the idea of the universal quantum classifier has been implemented on
an optical circuit of integrated photonics using the advantages of this
platform. The original scheme from [31] was adopted for bosonic systems. The
classification based on the data reuploading scheme has been performed
experimentally on an integrated photonic quantum optical circuit. Instead of
the qubit, two mode state of two photons was used. In this way losses has been
eliminated from the system as the coincidences of two photons where measured.

We simulate an optical classifier shown in figure 3 to classify two
dimensional points in two regions as shown in figure 4 $a)$. The input state
is taken as a NOON state

$|\Psi_{in}\rangle=\frac{1}{\sqrt{2}}(|20\rangle+|02\rangle)$ (15)

(although we would obtain similar results with other input states). The
circuit consists of phase-shifters and fixed beam-splitters. The phases are
functions of the free parameters adjustable during the training and data.
Specifically, one phase $\varphi_{j}=\phi_{j_{1}}+\phi_{j_{2}}x_{k}$, where
$x_{k}$ is either $x_{1}$ – the horizontal coordinate of a 2D data point or
$x_{2}$ – the vertical coordinate. So, the circuit from figure 3 contains 6
trainable parameters $\phi$. Notice that the same data are re-uploaded
repetitively. During the training at the end of the circuit, we measure the
estimator of the probability of state $|11\rangle$. We train the circuit
minimizing the cost function defined in (10).

After the training we generate new testing points. To test the classifier we
calculate the proper probability of $|11\rangle$ and classify a given point as
0 if the probability is smaller than a threshold, otherwise the class is
denoted as 1. We check the number of results which are: true positive (TP,
classifier indicates 1 for class 1), true negative (TN, classifier indicates 0
for class 0), false positive (FP, classifier indicates 1 for class 0), and
false negative (FN, classifier indicates 0 for class 1). The quality of
performance is characterised by the average success probability, which is an
average of the true positive and true negative ratios:

$P=\frac{1}{2}TPR+\frac{1}{2}TNR,$ (16)

where

$\displaystyle TPR$ $\displaystyle=$ $\displaystyle\frac{TP}{TP+FN},$
$\displaystyle TNR$ $\displaystyle=$ $\displaystyle\frac{TN}{TN+FP}.$ (17)

We repeat the simulation for different estimators of the probability given by
the sample size $N_{all}=200,50,10,5,2,1$. The number of training points is
taken as $N=300$. The training points and the starting point are the same in
experiments different $N_{all}$. The results are shown in figures 4 and 5.

Figure 4: a) Training points in two classes. b)-h) Classifier trained with
respectively: probabilites, estimators from $N_{all}=200$, estimators from
$N_{all}=50$, estimators from $N_{all}=10$, estimators from $N_{all}=5$,
estimators from $N_{all}=2$, estimators from $N_{all}=1$. In the upper right
corner of each figure the average success probability is written.

The numerical tests were performed with 300 training data, as shown in figure
4 $a)$ with 10 rounds of updating all 6 parameters of the circuit. We observe
that using an estimator of the probability based on 200 measurements give the
same efficiency as the algorithm with the exact probabilities, figure 4 $b)$.
However, we observe that the efficiency only slightly changes when $N_{all}$
is reduced till $N_{all}=1$ when the estimator of a single probability is a
binary variable. For all estimators features of the sets are clearly visible.
Figure 5 shows the plot of the probability of success averaged over different
series of training data versus the number of measurements $N_{all}$ of the
estimators. The averages were calculated by repeating all experiments for 10
different training data sets.

Figure 5: Average of the probability of success as a function of the number of
measurements used by the estimator. The average is over different sets of
training data. The task and the structure of the circuit are fixed. The mark
”inf” denotes the real value of the probability. The error bars show the
stadard deviation.

Based on our numerical example, we conclude that an estimator based on 1
measurement is still sufficiently good to recognize points from different
classes with a high probability of success. Taking these estimators allows us
to reduce the time of the experiment based on 200 measurements per
probability, 100 times.

##  V Discussion

In the context of machine learning with quantum optical circuits, we study a
training algorithm, the so-called sequential minimal optimization [39, 41]
that gives us an update rule for parameters in the optimization of the cost
function. This algorithm is a non local search alternative to gradient
descent. In the original algorithm, the cost function is constructed from
measured estimators of the expectation values of variables. In some types of
experiments in which the interesting events are rare, building estimators that
well approximate the expectation values takes time that may be prohibitive.
Therefore, following the research on gradient descent algorithms with small
sample size estimators [43], we develop the theory of the sequential minimum
optimization algorithm with small sample size estimators. Our theory allows
for using week sources of light to run still efficient supervised machine
learning tasks.

Our theory was demonstrated in the photonic circuit experiment but is not
restricted to it. It can be used as well in other platforms where the events
under interest are rare.

_Acknowledgements_ The authors are grateful to Jonathan P. Dowling for his
insightful suggestions and encouragements (and plenty of humor!) to this work.
This work was initially started when the authors were working at National
Institute of Information and Communications Technology (NICT). Jon was a
frequent visitor to NICT and initiated many projects and collaborations, where
this work is a part of them. He will be immenselly missed. This work was
supported by JST PRESTO Grant No. JPMJPR1864, The Murata Science Foundation,
JST CREST Grant No. JPMJCR1772, and JST COI-NEXT Grant No. JPMJPF2221.

## References

  * Mitarai  _et al._ [2018] K. Mitarai, M. Negoro, M. Kitagawa, and K. Fujii, Physical Review A 98, 032309 (2018).
  * Lloyd  _et al._ [2020] S. Lloyd, M. Schuld, A. Ijaz, J. Izaac, and N. Killoran, arXiv:2001.03622 (2020).
  * Schuld  _et al._ [2020] M. Schuld, A. Bocharov, K. M. Svore, and N. Wiebe, Physical Review A 101, 032308 (2020).
  * Schuld  _et al._ [2021a] M. Schuld, R. Sweke, and J. J. Meyer, Physical Review A 103, 032430 (2021a).
  * Chen  _et al._ [2021] S. Y. C. Chen, C. M. Huang, C. W. Hsing, and Y. J. Kao, Machine Learning: Science and Technology 2, 045021 (2021).
  * Cerezo  _et al._ [2021] M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, and P. J. Coles, Nature Reviews Physics 3, 625 (2021).
  * Lewenstein and Olko [1991] M. Lewenstein and M. Olko, Network: Computation in Neural Systems 2, 207 (1991).
  * Harrow  _et al._ [2009] A. W. Harrow, A. Hassidim, and S. Lloyd, Phys. Rev. Lett. 103, 150502 (2009).
  * Lloyd  _et al._ [2013] S. Lloyd, M. Mohseni, and P. Rebentrost, arXiv:1307.0411 (2013).
  * Rebentrost  _et al._ [2014] P. Rebentrost, M. Mohseni, and S. Lloyd, Phys. Rev. Lett. 113, 130503 (2014).
  * Paparo  _et al._ [2014] G. D. Paparo, V. Dunjko, A. Makmal, M. A. Martin-Delgado, and H. J. Briegel, Physical Review X 4, 031002 (2014).
  * Georgescu  _et al._ [2014] I. M. Georgescu, S. Ashhab, and F. Nori, Reviews of Modern Physics 86, 153 (2014).
  * Biamonte  _et al._ [2017] J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, Nature 549, 195 (2017).
  * Harris  _et al._ [2017] N. C. Harris, G. R. Steinbrecher, M. Prabhu, Y. Lahini, J. Mower, D. Bunandar, C. Chen, F. N. Wong, T. Baehr-Jones, M. Hochberg, S. Lloyd, and D. Englund, Nature Photonics 11, 447 (2017).
  * Shen  _et al._ [2017] Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, Nature Photonics 11, 441 (2017).
  * Steinbrecher  _et al._ [2019] G. R. Steinbrecher, J. P. Olson, D. Englund, and J. Carolan, npj Quantum Information 5, 60 (2019).
  * Yu  _et al._ [2019] S. Yu, F. Albarrán-Arriagada, J. C. Retamal, Y. T. Wang, W. Liu, Z. J. Ke, Y. Meng, Z. P. Li, J. S. Tang, E. Solano, L. Lamata, C. F. Li, and G. C. Guo, Advanced Quantum Technologies 2, 1800074 (2019).
  * Cong  _et al._ [2019] I. Cong, S. Choi, and M. D. Lukin, Nature Physics 15, 1273 (2019).
  * Shastri  _et al._ [2021] B. J. Shastri, A. N. Tait, T. Ferreira de Lima, W. H. Pernice, H. Bhaskaran, C. D. Wright, and P. R. Prucnal, Nature Photonics 15, 102 (2021).
  * Zhao  _et al._ [2021] L. Zhao, Z. Zhao, P. Rebentrost, and J. Fitzsimons, Quantum Mach. Intell. 3, 21 (2021).
  * Grover [1997] L. K. Grover, Phys. Rev. Lett. 79, 325 (1997).
  * Aaronson and Arkhipov [2011] S. Aaronson and A. Arkhipov,  _The computational complexity of linear optics, In Proceedings of the Forty-Third Annual ACM Symposium on Theory of Computing_ (ACM, New York, 2011).
  * Tillmann  _et al._ [2013] M. Tillmann, B. Dakić, R. Heilmann, S. Nolte, A. Szameit, and P. Walther, Nature Photonics 7, 540 (2013).
  * Lund  _et al._ [2014] A. P. Lund, A. Laing, S. Rahimi-Keshari, T. Rudolph, J. L. O’Brien, and T. C. Ralph, Phys. Rev. Lett. , 100502 (2014).
  * Gard  _et al._ [2015] B. T. Gard, K. R. Motes, J. P. Olson, P. P. Rohde, and J. P. Dowling,  _An Introduction to boson-bampling, In: edited by Malinovskaya, S. A., Novikova, I., From atomic to mesoscale: The role of quantum coherence in systems of various complexities_ (WSPC, New Jersey, 2015).
  * Huh  _et al._ [2015] J. Huh, G. G. Guerreschi, B. Peropadre, J. R. McClean, and A. Aspuru-Guzik, Nature Photon 9, 615 (2015).
  * Wang  _et al._ [2017] J. Wang, S. Paesani, R. Santagati, S. Knauer, A. A. Gentile, N. Wiebe, M. Petruzzella, J. L. O’brien, J. G. Rarity, A. Laing, and M. G. Thompson, Nature Physics 13, 551 (2017).
  * Sparrow  _et al._ [2018] C. Sparrow, E. Martín-López, N. Maraviglia, A. Neville, C. Harrold, J. Carolan, Y. N. Joglekar, T. Hashimoto, N. Matsuda, J. L. O’Brien, D. P. Tew, and A. Laing, Nature 557, 660 (2018).
  * Quesada [2019] N. Quesada, J. Chem. Phys. , 164113 (2019).
  * Życzkowski  _et al._ [1998] K. Życzkowski, P. Horodecki, A. Sanpera, and M. Lewenstein, Physical Review A 58, 883 (1998).
  * Pérez-Salinas  _et al._ [2020] A. Pérez-Salinas, A. Cervera-Lierta, E. Gil-Fuster, and J. I. Latorre, Quantum 4, 226 (2020).
  * Havlíček  _et al._ [2019] V. Havlíček, A. D. Córcoles, K. Temme, A. W. Harrow, A. Kandala, J. M. Chow, and J. M. Gambetta, Nature 567, 209 (2019).
  * Benedetti  _et al._ [2019] M. Benedetti, E. Lloyd, S. Sack, and M. Fiorentini, Quantum Science and Technology 4, 043001 (2019).
  * LaRose and Coyle [2020] R. LaRose and B. Coyle, Phys. Rev. A 102, 032420 (2020).
  * Schuld  _et al._ [2021b] M. Schuld, R. Sweke, and J. J. Meyer, Phys. Rev. A 103, 032430 (2021b).
  * Ortiz Marrero  _et al._ [2021] C. Ortiz Marrero, M. Kieferová, and N. Wiebe, PRX Quantum 2, 040316 (2021).
  * Huang  _et al._ [2021] H.-Y. Huang, R. Kueng, and J. Preskill, Phys. Rev. Lett. 126, 190505 (2021).
  * Uvarov and Biamonte [2021] A. V. Uvarov and J. D. Biamonte, Journal of Physics A: Mathematical and Theoretical 54, 245301 (2021).
  * Nakanishi  _et al._ [2020] K. M. Nakanishi, K. Fujii, and S. Todo, Phys. Rev. Res. 2, 043158 (2020).
  * Matthews  _et al._ [2009] J. C. F. Matthews, A. Politi, A. Stefanov, and J. L. O’brien, Nature Photonics 3, 346 (2009).
  * Ono  _et al._ [2022] T. Ono, W. Roga, K. Wakui, M. Fujiwara, S. Miki, H. Terai, and M. Takeoka, arXiv:2207.06614 (2022).
  * Cybenko [1989] G. Cybenko, Math. Control. Signals, Syst. 2, 303 (1989).
  * Sweke  _et al._ [2020] R. Sweke, F. Wilde, J. Meyer, M. Schuld, P. K. Faehrmann, B. Meynard-Piganeau, and J. Eisert, Quantum 4, 314 (2020).
  * Dowling and Milburn [2003] J. P. Dowling and G. J. Milburn, Phil. Trans. R. Soc. A 361, 1655–1674 (2003).
  * Reck  _et al._ [1994] M. Reck, A. Zeilinger, H. J. Bernstein, and P. Bertani, Phys. Rev. Lett. 73, 58 (1994).
  * Clements  _et al._ [2016] W. R. Clements, P. C. Humphreys, B. J. Metcalf, W. S. Kolthammer, and I. A. Walmsley, Optica 3, 1460 (2016).


# PeQES: A Platform for Privacy-enhanced
Quantitative Empirical Studies

Dominik Meißner  0000-0002-2937-6306<EMAIL_ADDRESS>Institute of
Distributed SystemsUlm University, GermanyAlbert-Einstein-Allee 1189081 ,
Felix Engelmann  0000-0001-9356-0231<EMAIL_ADDRESS>Institute of
Distributed SystemsUlm University, GermanyAlbert-Einstein-Allee 1189081 ,
Frank Kargl  0000-0003-3800-8369<EMAIL_ADDRESS>Institute of
Distributed SystemsUlm University, GermanyAlbert-Einstein-Allee 1189081 and
Benjamin Erb  0000-0002-5432-4989<EMAIL_ADDRESS>Institute of
Distributed SystemsUlm University, GermanyAlbert-Einstein-Allee 1189081

(2021)

###### Abstract.

Empirical sciences and in particular psychology suffer a methodological crisis
due to the non-reproducibility of results, and in rare cases, questionable
research practices. Pre-registered studies and the publication of raw data
sets have emerged as effective countermeasures. However, this approach
represents only a conceptual procedure and may in some cases exacerbate
privacy issues associated with data publications. We establish a novel,
privacy-enhanced workflow for pre-registered studies. We also introduce PeQES,
a corresponding platform that technically enforces the appropriate execution
while at the same time protecting the participants’ data from unauthorized use
or data repurposing. Our PeQES prototype proves the overall feasibility of our
privacy-enhanced workflow while introducing only a negligible performance
overhead for data acquisition and data analysis of an actual study. Using
trusted computing mechanisms, PeQES is the first platform to enable privacy-
enhanced studies, to ensure the integrity of study protocols, and to safeguard
the confidentiality of participants’ data at the same time.

empirical studies, privacy architecture

††journalyear: 2021††copyright: acmlicensed††conference: The 36th ACM/SIGAPP
Symposium on Applied Computing; March 22–26, 2021; Virtual Event, Republic of
Korea††booktitle: The 36th ACM/SIGAPP Symposium on Applied Computing (SAC
’21), March 22–26, 2021, Virtual Event, Republic of Korea††price: 15.00††doi:
10.1145/3412841.3441997††isbn: 978-1-4503-8104-8/21/03††ccs: Security and
privacy Domain-specific security and privacy architectures††ccs: Security and
privacy Privacy protections††ccs: Security and privacy Privacy-preserving
protocols††ccs: Security and privacy Trusted computing

##  1\. Introduction

Empirical research disciplines such as psychology rely on the use of surveys
to assess phenomena that are not directly measurable. Appropriate
operationalizations enable statistical analyses of these phenomena, e.g.,
through survey items. For instance, a clinical psychological study might
evaluate the outcome and efficiency of a therapeutical intervention by
comparing self-reports of subjects before and after the treatment. Similarly,
a socio-psychological study might explore the association of personality
characteristics with certain behavioral tendencies using the correlation of
operationalized values.

However, empirical research disciplines — and in particular psychology — are
increasingly facing two challenges in the realization and evaluation of
studies and experiments, namely issues of the research process and privacy
issues:  (1) Non-reproducible study outcomes (Shrout and Rodgers, 2018) (i.e.,
replication crisis) as well as questionable work methods such as HARKing
(Kerr, 1998) (i.e., post-hoc hypothesizing) or p-hacking (Head et al., 2015)
(i.e., data dredging for significant outcomes) are raising the question of the
extent to which research results are conclusive;  (2) Privacy requirements for
empirical researchers are becoming more demanding, particularly due to the
increasing use of new technology in their research (Kargl et al., 2019).
E.g., studies that utilize mobile sensing differ almost completely from
traditional pen-and-paper surveys in terms of data collection, data handling,
and data processing.

Pre-registration has now been suggested to tackle the former challenge: the
registration, publication, and review of a study protocol and hypotheses prior
to the actual data collection and analysis (Shrout and Rodgers, 2018). This
practice is sometimes complemented with the publication of primary data sets
to verify analyses and to foster re-use (Open Science Collaboration and
others, 2015). This practice yields new privacy issues (Boronow et al., 2020),
though (e.g., de-anonymization and re-identification attacks against study
participants).

To overcome both challenges at once for online studies, we suggest PeQES — a
unified platform that embraces the conceptual idea of pre-registered studies,
provides a technical runtime for empirical study procedures, ensures the
integrity of the study protocols, and enforces the protection of the
participants’ privacy. To introduce PeQES, we first discuss a general process
of empirical studies and illustrate potential threats for data privacy and
study integrity in Section 2. In Section 3, we introduce our privacy-enhanced
workflow which takes these threats into account. Next in Section 4, we
demonstrate our platform prototype that implements the privacy-enhanced
workflow. Section 5 compares our solution to existing approaches. Finally, we
discuss the implications of our PeQES platform in Section 6 and summarize our
contributions in Section 7.

##  2\. Quantitative Empirical Studies

Empirical research in psychology can be separated into quantitative approaches
(e.g., surveys with standardized self-report inventories) and qualitative
approaches (e.g., unstructured interviews). Although both approaches are based
on direct or indirect observations, only the quantitative method is primarily
founded in mathematical modeling and necessitates statistical analysis. For
the remainder of this work, we only focus on quantitative psychological
research that is conducted with surveys. This covers both correlation studies
and experimental study designs. While our ideas can also be applied to fields
other than psychology, we believe that strong privacy protection is most
needed where highly sensitive personal data are processed.

###  2.1. Research Methodology of Empirical Studies

A streamlined process for quantitative psychological studies covers the
following steps.  (S1:) Based on an overarching research question, related
work, and available information and resources, the _researchers_ form a
hypothesis;  (S2:) The _researchers_ design a study that addresses the
hypothesis. This includes the choice of appropriate operationalizations,
inventories, and variables for the phenomena, as well as the selection of the
target sample. Experimental studies also require the choice of experimental
manipulations or interventions;  (S3:) The _ethics board_ of the researchers’
institution reviews the study design and approves or rejects it based on
ethical aspects;  (S4:) The _researchers_ perform the actual study. This
includes the recruitment of _participants_ and the collection of data;  (S5:)
The analysis of the data allows the researchers to draw conclusions. In
particular, statistical hypothesis testing determines whether the formulated
hypothesis needs to be rejected or not, based on a certain significance level;
(S6:) By publishing the results, the _researchers_ can communicate their
findings to their _research community_.

Pre-registration extends the previous process by adding additional steps (Open
Science Collaboration and others, 2015):  (P1:) After S3, but before S4, the
_researchers_ register the study by publishing their study design with peer
reviewing;  (P2:) After S4, the _researchers_ document the actual course of
the study and any deviations from the original design;  (P3:) Due to the prior
acceptance of the study design, _researchers_ can now assume to publish their
results in S6 — even in case of negative outcomes — as long as they adhered to
the original study protocol. The publication step often requires the release
of the raw data sets and corresponding analysis code.

###  2.2. Involved Parties

Next, we take a look at the different parties that are involved in this
research process, consider their motivations, and hint to potential threats,
if we assume malicious actors.

Researchers:

advance the state of knowledge in their discipline by designing and conducting
studies and by publishing the outcomes of their studies. Therefore,
researchers follow principles and best practices of their discipline, apply
rigorous methods, and work honestly.

Ethics Boards:

supervise the ethical integrity of any study conducted by researchers
affiliated with their institutions. An ethics board is a trustworthy body of
an institution as it guarantees participants their safety, integrity, and
privacy in the setting of the study. The board wants to protect the reputation
of the institution and prevent scientific misconduct.

Participants:

take part in studies, either by their own initiative, or because of
participation rewards (e.g., crediting of subject hours, monetary rewards).

Dishonest Participants:

(so-called lurkers) might want to receive the reward without actual effort.
This can lead to careless, haphazard, or random survey responses which reduce
the data quality. Repeated participation by the same person represents another
potential threat in which the attacker exploits the reward mechanism (so-
called sybil attacks).

Research Communities:

believe in the growth in knowledge in their disciplines. Therefore, a
community critically reviews and evaluates studies, analyses, and outcomes
published by their researchers. A community promotes good scientific practices
and condemns scientific misconduct of its members.

Malicious Researchers:

are primarily interested in the successful publication of their work.
Therefore, they deviate from honest researchers when it comes to scientific
conduct. The playbook of malicious researchers includes the following
techniques:

(1):  Application of ethically questionable methods, either by skipping the
ethics boards111As most journals in psychology, medicine, and social sciences
require approvals by local institutional review boards, this technique also
requires a forged approval. (skipping S3), or by deliberate deviation from the
original study design in S4;  (2):  HARKing: exploring the data set after data
collection in order to generate promising hypotheses, then using the very same
data sets for statistical hypothesis testing. In this technique, the malicious
researchers either ignore the general research methodology, or they modify
their hypotheses post-hoc during S4;  (3):  p-hacking: e.g., exploring only
subsets of the collected data sets to boost significance of outcomes. Here,
the malicious researchers deliberately remove “outliers” in S4 to construct
their desired outcome;  (4):  The manipulation of data sets or the generation
of synthetic data rows during S4 represents an even more severe variant.

Another attack vector for malicious researchers emerges through the potential
linkage of data rows and participants. This enables malicious researchers to
de-anonymize participants and explore individual responses, e.g., for personal
purposes.

Data Sniffers:

access the publicly available data sets of studies with malicious intents.
They try to link individual data rows to individual subjects using de-
anonymization and re-identification techniques. Data sniffers take advantage
of contextual information about the execution of the study (e.g., recruitment
procedure, time, and location). Furthermore, they use additional, external
data sources (e.g., social media profiles) for matching and linking.

PlatformResearcherEthics BoardResearch CommunityParticipants Study Design (i)
Attested Study (ii) Pre-registration (iii) Study (iv) , Study (v) Data (vi)
Results (vii) Publication (viii) Figure 1. The privacy-enhanced workflow for
quantitative empirical studies.

##  3\. Privacy-enhanced Study Workflow

We propose an enhanced workflow for pre-registered studies in which the
correctness of the procedure, the integrity of the study design, and the
confidentiality of participant responses is protected in a technical manner.

First, our workflow requires a structured specification of the study design
and the associated data analysis. Second, the workflow introduces an
institutional platform that handles the implementation of a study design and
the actual execution of the study, namely the data collection and the
statistical data analysis. Technically, the platform represents a trusted
execution environment. Third, the workflow empowers certain parties to sign or
verify study specifications or to encrypt data sets.

As shown in Figure 1, the resulting workflow consists of eight consecutive
steps:

  1. (i)

Based on a hypothesis (S1), the researchers specify a suitable study design
(S2). This includes the specification of the study procedures and survey
design as well as a pre-defined script for the statistical data analysis. The
researchers then send the study design to their ethics board for approval.

  2. (ii)

The ethics board reviews the submitted study by considering the structured
specification (S3). Furthermore, the ethics board audits the intended analysis
and assesses its privacy impact for participants (e.g., risk of re-
identifiable records in the analysis output). Upon approval, the ethics board
cryptographically signs the study design.

  3. (iii)

After approval, the researchers pre-register the study and present it to the
research community (P1). This also includes the publication of the study
specification so that other researchers can check the analysis in use.
Additionally, the research community can use its specific domain knowledge to
peer review the analysis code and complement the ethics board’s assessment.

  4. (iv)

Next, the researchers submit the study to the platform. The platform only
accepts studies that have undergone prior acceptance by the ethics board. For
valid studies, it creates corresponding survey forms based on the
specification.

  5. (v)

Participants are intivited to take part in active studies (S4). Therefore, the
platform provides them with information about the study and demonstrates them
the approval of the ethics board.

  6. (vi)

The survey responses of a participant are encrypted and securely transmitted
to the platform for data collection.

  7. (vii)

By a pre-defined trigger (e.g., time period, total number of participations)
or upon instruction of the researchers, the platform eventually transitions
the study from the data collection phase to the data analysis phase.
Therefore, the platform closes the data collection and then executes the pre-
defined and signed analysis script exactly once. That is, the researchers only
receive the final outputs of their pre-defined analyses (S5). However, they
cannot explore the raw data sets, run interactive analyses, or make changes to
their scripts.

  8. (viii)

Drawing on the provided results, the researchers compile a publication of the
study outcomes (S6/P3).

##  4\. The PeQES Platform

In this section, we introduce our platform for privacy-enhanced quantitative
empirical studies, or PeQES for short. In our prototype, the platform is
responsible for study management, participant data aggregation, and execution
of the analysis script. The platform has to be trustworthy for participants
and researchers, as it receives the raw data of all participants and is tasked
with computing the results of the statistical data. There are multiple
technical approaches to implement this kind of trusted service for privacy
preserving data processing, such as secure multiparty computation, fully
homomorphic encryption, and trusted execution environments. Each of these
approaches have different properties, advantages, and disadvantages, which we
briefly discuss in Section 5. For PeQES, we opted for a trusted execution
environment.

_Trusted execution environments_ (TEE) are a promising technology to ensure
data integrity and confidentiality, even if the underlying operating system is
compromised. A TEE may be implemented by _hardware enclaves_ that are based on
a secure section of the processor with access to a protected memory area that
is inaccessible to the OS. This allows code running inside the enclave to keep
secrets from the OS and other parts of the system. I/O operations and
networking is provided by the untrusted OS and has to be secured by other
mechanisms. Such enclaves usually allow remote parties to remotely attest the
authenticity of the hardware as well as the code running inside the enclave,
thus enabling secure and trusted communication with software inside the
enclave. To persistently store data, enclaves rely on the untrusted OS. To
ensure confidentiality and integrity of stored data, enclaves have the
capability to derive sealing keys that are based on the currently executed
software and the unique hardware. Before passing data for storage to the OS,
its authenticity and confidentiality can be assured through an authenticated
encryption with a derived sealing key. The stored data can only be decrypted
by the same version of the platform running on the same processor. We
implemented a prototype of our platform using the widely available Intel
Software Guard Extensions (Intel SGX)222https://software.intel.com/en-us/sgx,
an extension to the instruction set of Intel CPUs that enables the usage of
secure enclaves. As our PeQES protocols are weakly coupled to the enclave, it
can easily be ported to a different TEE in the case of serious vulnerabilities
in SGX.

###  4.1. Architecture

PeQES Platform Data Store TLS Termination Trusted SGX Enclave HTTP Request
Handler Studies Statistics Engine Intel IASResearcherParticipantEthics Board
Figure 2. PeQES prototype architecture.

Our prototypical realization of the PeQES platform is depicted in Figure 2.
The platform provides an HTTP-based API that allows different parties
(researcher, ethics board, participants) to interface with it. To keep the
participation hurdle as low as possible, we assume that participants do not
wish to install dedicated software to participate and instead use a modern web
browser. The HTTP communication applies common TLS transport encryption, which
is terminated outside the hardware enclave.

We argue that moving the TLS termination inside the enclave would not be
sufficient to establish trust in the enclave, without performing a remote
attestation during the TLS handshake or pinning a public key for the enclave.
The platform can simply eclipse the enclave and obtain another valid
certificate for the domain. As we assume the clients to be standard web
browsers, a remote attestation during the TLS handshake is not feasible.
Neither the deprecated HTTP Public Key Pinning (Evans et al., 2015), nor its
replacement Certificate Transparency (Laurie et al., 2013) enable a client to
differentiate a malicious certificate issued by the platform from a legitimate
certificate used by the enclave, as both are valid.

Instead, PeQES is built on transitive trust — participants trust the ethics
board to work diligently, attesting the enclave, and therefore trust approved
studies.

###  4.2. Threat Model

As generally discussed in Section 2.2, we map the entities to the following
security model: While most researchers $(\mathcal{R})$ have noble intentions,
nevertheless we model them as fully malicious, trying to deviate from the
protocol, e.g., to gain survey information or alter their analysis. We assume
that they all perform pre-registered studies and the study design itself is
considered public. The ethics board $(\mathcal{B})$ acts honestly in any
aspect. As we require honest answers to the study, the participants
$(\mathcal{P})$ have to act honestly. Sybil attacks may be dealt with by an
authentication provider, which is out of scope. Dubious responses by lurkers
(e.g., always selecting the first option) are expected to be excluded as part
of the actual data analysis phase (i.e., data cleansing). The platform
provider is modeled as covert, i.e. tries to extract information and only
deviates from the protocol, if it is unlikely to be caught. It is trusted with
the distribution of the correct web application to the participants but is
interested in extracting raw survey results. The platform does not engage in a
denial-of-service attack against the enclave. The secure enclave
$(\mathcal{E})$ is assumed to be secure and unbreachable, and the code inside
attested. Attacks on the confidentiality of the SGX enclave and side-channel
attacks are out of scope. Data sniffers are modeled as any external malicious
attacker, who aims to get information about the study answers.

###  4.3. Protocol Flow

#### Setup

To prepare our PeQES platform, the ethics board $\mathcal{B}$ generates a key
pair $(\mathcal{B}_{\textsf{sk}},\mathcal{B}_{\textsf{pk}})$. The public key
$\mathcal{B}_{\textsf{pk}}$ is published for verification of study approvals.
Multiple ethics boards may form a PKI-like structure. The researcher
$\mathcal{R}$ generates a key-pair
$(\mathcal{R}_{\textsf{sk}},\mathcal{R}_{\textsf{pk}})$. In addition, a secure
signature scheme $(\textsf{Sig},\textsf{Ver})$ and a symmetric encryption
scheme $(\textsf{Enc},\textsf{Dec})$ is set up.

#### Register Study

A researcher registers a new study $\mathcal{S}$ by uploading its
specification (i.e., name, description, survey, analysis script) and the
researcher’s public key $\mathcal{R}_{\textsf{pk}}$ to an enclave
$\mathcal{E}$ of the PeQES platform. A signature
$\textsf{Sig}_{\mathcal{R}_{\textsf{sk}}}(\mathcal{S})$ attests the intent of
$\mathcal{R}$. Once uploaded, $\mathcal{E}$ verifies the signature and
generates a study key-pair
$(\mathcal{S}_{\textsf{sk}},\mathcal{S}_{\textsf{pk}})$ for the study, which
is kept secure in the protected memory of the enclave.

#### Approve Study

To approve a study $\mathcal{S}$ (Figure 3), the ethics board $\mathcal{B}$
performs a remote attestation to establish a secure channel with the platform
enclave $\mathcal{E}$ through a shared master key $(\textsf{SMK})$. During the
attestation, $\mathcal{B}$ verifies that $\mathcal{E}$ is executed in a
genuine trusted execution environment and that the platform software has not
been tampered with. $\mathcal{B}$ uses the secure channel to verify that
$\mathcal{S}_{\textsf{sk}}$ is only known to $\mathcal{E}$ through a
combination of the assurances of the remote attestation and a proof of
knowledge (i.e., signing a nonce $n$, provided by $\mathcal{B}$, with
$\mathcal{S}_{\textsf{sk}}$). If successful and if $\mathcal{S}$ is compliant,
$\mathcal{B}$ signs
$(\mathcal{S}_{\sigma}=\textsf{Sig}_{\mathcal{B}_{\textsf{sk}}}(\mathcal{S},\mathcal{S}_{\textsf{pk}}))$
the study as approved.

$\mathcal{E}$: $\mathcal{S}_{\textsf{sk}}$$\mathcal{B}$:
$\mathcal{B}_{\textsf{sk}}$$\textsf{SMK}\leftarrow$Remote
attestation$\textsf{Enc}_{\textsf{SMK}}(n)$$\textsf{Enc}_{\textsf{SMK}}(\textsf{Sig}_{\mathcal{S}_{\textsf{sk}}}(n))$$\mathcal{S}_{\sigma}\leftarrow\textsf{Sig}_{\mathcal{B}_{sk}}(\mathcal{S},\mathcal{S}_{\textsf{pk}})$
Figure 3. Protocol flow for approving a study.

#### Study Participation

To participate in the study (Figure 4), the participant $\mathcal{P}$ fetches
$\mathcal{S}$, $\mathcal{S}_{\textsf{pk}}$, and $\mathcal{S}_{\sigma}$ from
the platform, together with a trusted JavaScript web application. They
validate the approval of $\mathcal{B}$ by
$\textsf{Ver}_{\mathcal{B}_{\textsf{pk}}}(\mathcal{S}_{\sigma},\mathcal{S})\stackrel{{\scriptstyle?}}{{=}}1$
and abort on failure.

After completion of the study, $\mathcal{P}$ prepares to upload their response
$\mathcal{S}_{\mathcal{P}}$ for $\mathcal{S}$. First, $\mathcal{P}$ generates
a temporary key pair $(G_{\mathcal{P},\textsf{sk}},$
$G_{\mathcal{P},\textsf{pk}})$ and establishes an ephemeral key EK with
$\mathcal{E}$ and its temporary key $(G_{\mathcal{E},\textsf{sk}},$
$G_{\mathcal{E},\textsf{pk}})$ over an authenticated Diffie-Hellman exchange
$\textsf{DH}(G_{\mathcal{E}},G_{\mathcal{P}})$. I.e., the temporary public key
$G_{\mathcal{E},\textsf{pk}}$ of $\mathcal{E}$ is signed
$G_{\sigma}=\textsf{Sig}_{\mathcal{S}_{\textsf{sk}}}(G_{\mathcal{E},\textsf{pk}})$.
Then $\mathcal{P}$ verifies
$\textsf{Ver}_{\mathcal{S}_{\textsf{pk}}}(G_{\sigma},G_{\mathcal{E},\textsf{pk}})$.
By verifying $\mathcal{S}_{\sigma}$ and $G_{\sigma}$, $\mathcal{P}$ can skip
the remote attestation of $\mathcal{E}$ during this step and still validate
the enclave’s authenticity by transitive trust and the assumption, that
$\mathcal{B}$ performed a remote attestation. After successful verification,
the participant’s response $\mathcal{S}_{\mathcal{P}}$ is encrypted
$\textsf{Enc}_{\textsf{EK}}(\mathcal{S}_{\mathcal{P}})$ and uploaded to
$\mathcal{E}$ and then processed there.

$\mathcal{E}$: $\mathcal{S}_{\textsf{sk}}$$\mathcal{P}$:
$\mathcal{B}_{\textsf{pk}}$$\mathcal{S},\mathcal{S}_{\textsf{pk}},\mathcal{S}_{\sigma}$$\textsf{Ver}_{\mathcal{B}_{\textsf{pk}}}(\mathcal{S}_{\sigma},\mathcal{S})$gen
$G_{\mathcal{E}}$$G_{\mathcal{E},\textsf{pk}},G_{\sigma}$
$\textsf{Ver}_{\mathcal{S}_{\textsf{pk}}}(G_{\sigma},G_{\mathcal{E},\textsf{pk}})$,
gen $G_{\mathcal{P}}$, $\textsf{DH}(G_{\mathcal{P}},G_{\mathcal{E}})$
$G_{\mathcal{P},\textsf{pk}},\textsf{Enc}_{\textsf{EK}}(\mathcal{S}_{\mathcal{P}})$
Figure 4. Protocol flow for participating in a study.

Next, $\mathcal{E}$ uses EK to decrypt
$\textsf{Dec}_{\textsf{EK}}(\mathcal{S}_{\mathcal{P}})$. Due to storage
limitations, the results are persisted outside of $\mathcal{E}$. They are
encrypted with a symmetric key derived from $\mathcal{S}_{\textsf{sk}}$.
Integrity and protection against roll-back attacks by the platform provider is
achieved by storing the encrypted objects in a Merkle tree. Thereby the
enclave merely stores the root hash of the tree and performs efficient updates
on the data structure.

#### Study Completion

To complete a study $\mathcal{S}$, the researcher authenticates to
$\mathcal{E}$ with the help of $\mathcal{R}_{sk}$. If valid, $\mathcal{E}$
stops accepting new participants and executes the predefined analysis script
of $\mathcal{S}$. The result $\mathcal{S}_{\text{result}}$ of the script is
transferred to $\mathcal{R}$ for publication. Optionally, $\mathcal{E}$ signs
$\textsf{Sig}_{\mathcal{S}_{\textsf{sk}}}(\mathcal{S}_{\text{result}})$ for
public acceptance of the results. Optionally, if $\mathcal{B}$ mandates, the
raw results of the study are deleted by $\mathcal{E}$ so that it is impossible
to run the evaluation script twice.

###  4.4. Implementation

We have implemented PeQES using the Rust-based Fortanix Enclave Development
Platform333https://edp.fortanix.com/. The prototype implementation of the
PeQES platform is available under an open source
license444https://github.com/vs-uulm/peqes/tree/sac21. In total we implemented
our platform prototype in 1459 lines of code.

Next, we implemented the client as a Vue.js web application in 871 lines of
code (c.f., Figure 5). It relies on native cryptography primitives provided by
the WebCryptography API (Watson, 2017) to encrypt participants’ responses.
Survey specifications are represented as JSON and rendered using the SurveyJS
library555https://surveyjs.io/Overview/Library/. The analysis scripts can be
composed in a JavaScript-based DSL with access to the statistics library
jstat.

Figure 5. Study preview for the researcher and ethics board that with
additional meta information to identify the researcher and enclave.

The statistics component of the platform is based on the embedded JavaScript
engine QuickJS666https://bellard.org/quickjs/ with custom Rust bindings. The
JavaScript interpreter is limited to standard JavaScript functions with the
exception of statistics primitives from the jstat
library777https://jstat.github.io/ that is embedded in the enclave. Especially
I/O access, such as accessing the file system or accessing network resources,
is not possible to prevent obvious side channel attacks. The memory of the
engine is located in the encrypted section of the enclave. The JavaScript
environment is sandboxed and has only access to its internal state but not to
other memory inside or outside the enclave. Prior to execution, the input data
is copied into the memory of engine in form of a JSON array. The results
computed by the analysis script are collected as a JSON object that is copied
back to the enclave following the execution.

###  4.5. Security Evaluation

The PeQES platform is secure against our specified threat model, as the
following measures are implemented. The enclave $(\mathcal{E})$, participants
$(\mathcal{P})$, and ethics board $(\mathcal{B})$ act honestly and therefore
need no further consideration. As the raw data of a study are not published,
which they classically are for reproduction purposes, data sniffers have no
access at all.

The platform provider is assumed to act covertly and tries to learn as much as
possible from the interactions. The delivery of the website including the
cryptography code contains no sensitive information but allows to ship code to
exfiltrate participant’s data. However this inflicts a prohibitive risk of
getting caught and losing reputation. During the _Register Study_ workflow,
the platform provider is able to impersonate a legit enclave by emulation and
get access to an already public study information. In addition, such an
emulation is detected by the ethics board during the remote attestation of the
_Approve Study_ workflow. Any further communication through the platform to
the enclave is encrypted and authenticated by the signature of the ethics
board. This restricts the platform to merely collect meta-data of all
connections without knowing the association to a specific study. With storage
inside enclaves limited, the platform provider acts as a storage device for
the enclave. As the data is authenticated and encrypted by the enclave, the
only attack vector is to correlate encrypted communication with the size of
the encrypted data. This leaks at most, if a submitted answer is accepted and
persisted by the enclave, which all of the submissions should be.

The malicious researchers are confined to legitimate actions by our protocol.
During _Register Study_ , a malicious research may upload a flawed study under
the identity of someone else to harm reputation or cause a penalty. This is
prevented by the mandatory signature of the researcher requiring the secret
key of the impersonated one. The malicious act more difficult to detect is an
untruthful study design. A common goal is the extraction of participants’
answers and the possibility to change the analysis retrospectively to align
the study results with the hypothesis or improve the confidence. Our platform
includes the following countermeasures. The participants’ answers are stored
encrypted by the enclave without access by anyone outside the enclave. The
analysis script provided can only be executed once, which is also enforced by
the enclave. This prevents data reconstruction from multiple samplings of
random subsets or similar. It remains to verify that the study itself complies
with ethical regulations and common sense. This includes that the results
properly anonymize answers. In PeQES, the ethics board is tasked to ensure
these properties for all studies they approve. The questionnaire part of the
study is a finite set of questions and possible answers, similar to classical
studies, and therefore easy to audit.

The analysis script remains as the most difficult part to verify if it adheres
to a benevolent cause. Not only is it a new component, many ethics boards are
not prepared for, but even from a computer science perspective it is difficult
to assess if the analysis will do only what is evident from looking at it.
Especially dynamic, interpreted environments allow for subtle side effects,
difficult to find even after thorough inspection. Even though the interpreter
is sandboxed as described in the previous section, we acknowledge that a
Turing-complete analysis/query language has inherent caveats. If analyzing the
script is not feasible for ethics boards, PeQES is modular by design and the
statistics component can be adapted to provide an interpreter for a
specifically restricted domain specific language (DSL) instead of the current
JavaScript version. Therefore, enabling a trade-off between functionality and
computational expressiveness on the one hand, and security and reviewability
on the other hand. An example of such an alternative statistical component is
a very limited flow-based DSL that only allows statistical primitives as pre-
audited building blocks. This enables easier composition of analysis scripts
without coding experience and a more streamlined auditing process at the
expense of limiting the feature set that is available for analysis scripts.

###  4.6. Performance Evaluation

We have conducted a set of experiments to compare the performance of the
secure enclave of PeQES to a variant running without an enclave. We have
released all evaluation artifacts888https://github.com/vs-
uulm/peqes/tree/sac21 under an open source license, following the Popper
convention (Jimenez et al., 2017) for reproducible evaluations. The
experiments were carried out on two machines equipped with an Intel SGX
capable Intel Core i7-7700 (quad-core with SMT; 3.60 GHz) CPU and 32 GB RAM,
running Ubuntu 18.04.3 LTS, and connected using gigabit ethernet.

The sample study consisted of a short demographic questionnaire and BFI-10, a
10-item personality scale (Rammstedt and John, 2007). The analysis script
explored differences in personality traits of two demographic sub-groups
(split by age) using an independent two-sample t-test (see Listing 1). Using a
synthetic workload generator on the second host, we sequentially submitted up
to 10,000 participations to the platform for each evaluation run before
transitioning to the the statistical analysis execution. For each submission
of a survey response, we measured the total response time at the client side.
For each study run, we also measured the total execution time of the analysis
after collecting all responses.

The unencrypted baseline implementation without enclave took an average of
$M$=15.8 ms to handle a response submission ($P_{50}$=15.8 ms, $P_{99}$=19.8
ms). In turn, PeQES required $M$=29.7 ms ($P_{50}$=29.6 ms, $P_{99}$=35.2 ms).
Figure 6 illustrates the impact of different sample sizes ($N$=10 …1,000) and
the platform implementations (usage of SGX). For a sample size of $N$=10,000
the mean computation time using SGX is 15.7 times as high as without SGX
($t_{\overline{\textrm{sgx}}}$=4.6 s, $t_{\textrm{sgx}}$=72.4 s).

Figure 6. 95% confidence interval of the mean execution time of the
statistical analysis as a function of platform implementation type and sample
size based on 35 executions per variant.

⬇

1const s = [

2 // demographic group with age <= 50

3 { E: [], N: [], O: [], G: [], V: [] },

4 // demographic group with age > 50

5 { E: [], N: [], O: [], G: [], V: [] }

6];

8// iterate over participant responses

9data.forEach((row) => {

10 // recode items

11 row['BFI'].item01 = 6 - row['BFI'].item01;

12 row['BFI'].item03 = 6 - row['BFI'].item03;

13 row['BFI'].item04 = 6 - row['BFI'].item04;

14 row['BFI'].item05 = 6 - row['BFI'].item05;

15 row['BFI'].item07 = 6 - row['BFI'].item07;

17 // add response to respective sub demographic

18 const g = row['age'] <= 50 ? 0 : 1;

19 s[g].E.push(row['BFI'].item01, row['BFI'].item06);

20 s[g].N.push(row['BFI'].item04, row['BFI'].item09);

21 s[g].O.push(row['BFI'].item05, row['BFI'].item10);

22 s[g].G.push(row['BFI'].item03, row['BFI'].item08);

23 s[g].V.push(row['BFI'].item02, row['BFI'].item07);

24});

26// expose result object to platform

27pushResult({

28 E: ttest(s[0].E, s[1].E),

29 N: ttest(s[0].N, s[1].N),

30 O: ttest(s[0].O, s[1].O),

31 G: ttest(s[0].G, s[1].G),

32 V: ttest(s[0].V, s[1].V)

33});

Listing 1:  Exemplified analysis script to compute an independent two-sample
t-test on the personality traits of two demographic sub-groups.

All evaluation results indicate a noticeable performance degradation when
using SGX. However, we argue that the absolute values are completely within
reasonable limits. The increase of submission latency is hardly noticeable for
participants and is far below the recommended limit of acceptable server
response times of 200
ms999https://developers.google.com/speed/docs/insights/Server by Google’s
project PageSpeed Insights. The longer computation times become only apparent
in larger samples sizes which are primarily used in specific sociological
studies. But even then the superlinear overhead is negligible as the
computation is executed only once and lacks strict timing requirements.

###  4.7. Limitations

While our PeQES platform enables sound and privacy friendly quantitative
studies, we acknowledge the following technical limitations. All communication
is passed through the platform provider, which is responsible for TLS
termination, hosting of the web applications and storage of encrypted survey
results. Thereby a lot of meta-data is accumulated, such as which IP address
participated in which study. Due to easy usability for participants through a
browser, we are susceptible to malicious JavaScript files exfiltrating answers
to researchers directly or interested third parties. WAIT (Meißner et al.,
2021) showcases an extension for web browsers that enables the verification of
provided web applications against a public transparency log, similar to
certificate transparency. Approaches like this, further limit the attack
possibilities of the platform provider and may be applied to multiple similar
scenarios to alleviate this limitation. Another possible attack vector is a
compromise of the enclave by breaking the sandbox of the JavaScript engine via
the statistical analysis script.

##  5\. Related Work

Quite recently, researchers from psychology (e.g., Peikert et al. (Peikert and
Brandmaier, 2019), Arslan et al. (Arslan et al., 2020)) have started to
suggest standardized and partially automated workflows to support their
research process based on open source technologies. They primarily take into
account reproducibility, replicability, and more transparency for the research
process, but their workflows are also designed to streamline research efforts.
However, they have neither advanced privacy protection mechanisms for
collected data, nor do they comprehensively protect the integrity of the study
design and the data analysis.

Lapets et al. (Lapets et al., 2016) suggest a system for secure analytics that
can be deployed as a web application. Data contributors add a random mask to
their data before uploading it to a service provider, which aggregates all
data entries. The mask is encrypted using the public key of an analyzer and
also uploaded. The analyzer can retrieve the aggregated masked data and the
encrypted masks, which can be decrypted, aggregated, and subtracted from the
aggregated masked data to retrieve the true aggregate data.

Rmind (Bogdanov et al., 2018) is a system for privacy-preserving statistical
analysis based on secure multi-party computation. Input data is secret shared
across three computation servers that are assumed to be non-colluding.
Statistical queries can be evaluated using an R-like statistics language.

STAR (Servan-Schreiber et al., 2019) enables statistical tests with auditable
results based on secret sharing, secure multiparty computation, and tamper-
proof ledgers. The main focus of STAR is to address the issue of false
discoveries in scientific studies, by generating a publicly verifiable
certificate for the performed statistical tests.

Lu et al. (Lu et al., 2017) designed a system to enable privacy-preserving
statistical analysis with fully homomorphic encryption. While they proposed
protocols for multiple statistics, performance is inferior and the privacy
policies by the ethics boards cannot be enforced.

The above systems all assume non-colluding computation parties. However, for
quantitative empirical studies this assumption does not hold, because the
computing parties have no incentive not to collude, as they are disjunct from
the data subjects. In case the computing parties are research institutions,
they may even have a strong incentive to collude to gain the raw data for
exploration. In contrast, PeQES embraces and extends the existing research
workflow and integrates ethics committees to review studies for privacy risks.

##  6\. Discussion and Future Work

The practicality of our platform depends essentially on the acceptance by the
researchers and the integratability of technical steps into existing
organizational procedures.

We assume that researchers might perceive the platform rather skeptically at
first. After successful submission of a study to the platform, the subsequent
study progress — i.e., data collection and data analysis — becomes neither
observable nor fully controllable by them. The platform appears more like a
black box that ingests participations and eventually yields the output of a
pre-defined analysis. The platform also takes away the possibility of
interactive data exploration from the researchers. Furthermore, it prevents
any post-hoc modifications of statistical analyses. However, we argue that
these restrictions are in line with the concept of pre-registration and
encourage good scientific practices. The workflow deliberately decouples
hypothesis generation and hypothesis testing, data collection and data
analysis, as well as study design and study execution. Adhering to this
approach ensures a rigorous conduct and thus increases the trust in its
outcomes. There are indeed methods to circumvent negative implications of this
approach when conducting studies in practice. Before implementing
comprehensive confirmatory studies, researchers typically apply purely
explorative analyses to generate hypothesis ideas and rely on pilot studies to
test operationalizations, inventories, and surveys. So in such a two-step
approach, PeQES will only be used in the second step for the larger,
confirmatory study, but not for the initial exploration. Regarding the
statistical analysis scripts, we argue that such scripts should be tested in
advance against synthetically generated survey responses. This is not only
helpful for development and debugging of the analysis code. A prior execution
with simulated and controllable responses for test purposes also helps to
verify the correctness of the intended analysis steps in the first place.

Note that the analysis scripts can include data wrangling and data cleansing
steps as part of the overall analysis. These steps are therefore also subject
to approval by the ethics board. This also allows researchers to apply tests
for certain assumptions to their samples (e.g., Kolmogorov-Smirnov test for
normality). To include such pre-tests is particularly relevant for non-
interactive data analysis procedures as potential biases in the samples cannot
be observed exploratively anymore. By performing appropriate tests, biased
samples can either be excluded, or the data analysis process would stop its
execution as soon as the distribution assumptions for the sample do not hold.

Regarding the integration into existing procedures, we assume that the review
step of ethics board represents the main barrier. Whereas until now ethics
boards only review a written description of a study, the new workflow requires
them to audit and sign a structured study specification and to assess the
privacy impact of a designated data analysis and its estimated output. Again,
we believe that this change eventually strengthens the position of reviewing
boards, as it increases compliance and strengthens their influence on the
actual study execution. A possible step towards the simplification of this
procedure is the specification and enforcement of data policies. Here, the
system would derive required data usage policies from the study specification.
In turn, the ethics board would review these policies and compare them with
general guidelines. Ultimately, the platform would prevent any usage of
participants’ data that is not in line with the pre-defined policies or that
interferes with specific policies configured by a participant. An orthogonal
approach would be to not only require the review of the ethics board, but
additionally necessitate peer reviews of the pre-registered study design.
Peers in the research community can be expected to have more domain expertise
for the specific research and study design and can complement the judgment of
the ethics board.

PeQES and its workflow is also advantageous for the reproducibility and
replicability of studies. Reproducibility assures that the same original data
with the same statistical analysis code reproduce the same findings. This is
inherently provided by PeQES, as the original data cannot be changed, and the
execution of the statistical analysis is also taking place within the
protected boundaries of PeQES. Replicability describes a repetition of a study
with the same methods, but with newly collected original data and independent
from the original research efforts. In this case, the published study design
artifacts can be used to exactly replicate an existing study at another
institution that runs a PeQES instance. This will require an attestation of
the ethics board which is responsible for the replicating instance. However,
PeQES guarantees that the exact same methods are used during the study
replication. In this way, PeQES excludes confounding factors during the
replication and makes it possible to attribute the replicated results solely
to the new samples.

The success of PeQES ultimately depends on the degree to which it can be
extended into a comprehensive solution for privacy-enhanced empirical
research. Our next step is the design of a privacy-enhanced participant
management component to tackle longitudinal studies with recurrent
participations, attribute-based participant recruitment (e.g., only targeting
left-handed subjects in a certain group), and reward payoffs — in each case
with appropriate privacy protection. Inclusion of such a system can also
further strengthen the authenticity of participants’ data by limiting the
possibility of sybil attacks by participants and creation of fake participants
by malicious researchers. Data access policies for analysis scripts and data
usage policies of study responses would also facilitate secondary data usage.
E.g., researchers could submit supplementary studies that operate on the same
data sets, provided that the ethics board has approved the supplementary
analysis and provided that the participant has agreed to secondary use of
their data in the first place.

For the future, we also expect our platform to handle data collections beyond
online surveys. This includes scenarios that are inherently more sensitive to
data privacy issues, such as mobile sensing. E.g., the use of apps to collect
individual behavioral patterns of participants throughout the day makes strong
technical data protection imperative.

##  7\. Conclusion

We have suggested a privacy-enhanced workflow for quantitative empirical
studies that embraces the idea of pre-registration and adds additional steps
to enforce the correct workflow execution. Furthermore, we have introduced
PeQES, a trusted execution platform that implements this workflow based on
appropriate protocols and Intel SGX. Our solution addresses the orderly
execution of a pre-defined study design, guarantees the confidentiality of
participants’ data, ensures the integrity of the study and its analysis
procedures, and thus increases trust in corresponding outcomes and results.
The evaluation of the PeQES prototype showed a noticeable performance overhead
of our solution, which in our opinion is still negligible in practice. PeQES
represents a core building block towards a more comprehensive technical
solution that improves privacy protection for quantitative empirical studies
by adding complementary components such as participation management.

## References

  * (1)
  * Arslan et al. (2020) Ruben C Arslan, Matthias P Walther, and Cyril S Tata. 2020\.  formr: A study framework allowing for automated feedback generation and complex longitudinal experience-sampling studies using R.  _Behavior Research Methods_ 52, 1 (2020), 376–387.  https://doi.org/10.3758/s13428-019-01236-y
  * Bogdanov et al. (2018) Dan Bogdanov, Liina Kamm, Sven Laur, and Ville Sokk. 2018\.  Rmind: A Tool for Cryptographically Secure Statistical Analysis.  _IEEE Transactions on Dependable and Secure Computing_ 15, 3 (May 2018), 481–495.  https://doi.org/10.1109/TDSC.2016.2587623
  * Boronow et al. (2020) Katherine E. Boronow, Laura J. Perovich, Latanya Sweeney, Ji Su Yoo, Ruthann A. Rudel, Phil Brown, and Julia Green Brody. 2020\.  Privacy Risks of Sharing Data from Environmental Health Studies.  _Environmental Health Perspectives_ 128, 1 (2020), 17008\.  https://doi.org/10.1289/EHP4817
  * Evans et al. (2015) C. Evans, C. Palmer, and R. Sleevi. 2015.  _Public Key Pinning Extension for HTTP_.  RFC 7469.  https://doi.org/10.17487/RFC7469
  * Head et al. (2015) Megan L. Head, Luke Holman, Rob Lanfear, Andrew T. Kahn, and Michael D. Jennions. 2015\.  The Extent and Consequences of P-Hacking in Science.  _PLOS Biology_ 13, 3 (March 2015), 1–15.  https://doi.org/10.1371/journal.pbio.1002106
  * Jimenez et al. (2017) Ivo Jimenez, Michael Sevilla, Noah Watkins, Carlos Maltzahn, Jay Lofstead, Kathryn Mohror, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. 2017.  The Popper Convention: Making Reproducible Systems Evaluation Practical. In _2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)_. IEEE, 1561–1570.  https://doi.org/10.1109/IPDPSW.2017.157
  * Kargl et al. (2019) Frank Kargl, Rens W van der Heijden, Benjamin Erb, and Christoph Bösch. 2019.  Privacy in Mobile Sensing.  In _Digital Phenotyping and Mobile Sensing_. Springer, 3–12.  https://doi.org/10.1007/978-3-030-31620-4_1
  * Kerr (1998) Norbert L Kerr. 1998\.  HARKing: Hypothesizing After the Results are Known.  _Personality and Social Psychology Review_ 2, 3 (1998), 196–217.  https://doi.org/10.1207/s15327957pspr0203_4
  * Lapets et al. (2016) Andrei Lapets, Nikolaj Volgushev, Azer Bestavros, Frederick Jansen, and Mayank Varia. 2016\.  Secure MPC for Analytics as a Web Application. In _2016 IEEE Cybersecurity Development (SecDev)_. 73–74.  https://doi.org/10.1109/SecDev.2016.027
  * Laurie et al. (2013) B. Laurie, A. Langley, and E. Kasper. 2013.  _Certificate Transparency_.  RFC 6962.  https://doi.org/10.17487/RFC6962
  * Lu et al. (2017) Wenjie Lu, Shohei Kawasaki, and Jun Sakuma. 2017\.  Using Fully Homomorphic Encryption for Statistical Analysis of Categorical, Ordinal and Numerical Data. In _24th Annual Network and Distributed System Security Symposium, NDSS 2017, San Diego, California, USA, February 26 - March 1, 2017_. The Internet Society.  https://doi.org/10.14722/ndss.2017.23119
  * Meißner et al. (2021) Dominik Meißner, Frank Kargl, and Benjamin Erb. 2021\.  WAIT: Protecting the Integrity of Web Applications with Binary-Equivalent Transparency. In _The 36th ACM/SIGAPP Symposium on Applied Computing_ (Virtual Event, Republic of Korea) _(SAC ’21)_. ACM.  https://doi.org/10.1145/3412841.3442143
  * Open Science Collaboration and others (2015) Open Science Collaboration and others. 2015\.  Estimating the reproducibility of psychological science.  _Science_ 349, 6251 (2015).  https://doi.org/10.1126/science.aac4716
  * Peikert and Brandmaier (2019) Aaron Peikert and Andreas M Brandmaier. 2019.  A Reproducible Data Analysis Workflow with R Markdown, Git, Make, and Docker.  https://doi.org/10.31234/osf.io/8xzqy
  * Rammstedt and John (2007) Beatrice Rammstedt and Oliver P. John. 2007.  Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German.  _Journal of Research in Personality_ 41, 1 (2007), 203–212.  https://doi.org/10.1016/j.jrp.2006.02.001
  * Servan-Schreiber et al. (2019) Sacha Servan-Schreiber, Olga Ohrimenko, Tim Kraska, and Emanuel Zgraggen. 2019.  Custodes: Auditable Hypothesis Testing.  _CoRR_ abs/1901.10875 (2019).  arXiv:1901.10875 http://arxiv.org/abs/1901.10875
  * Shrout and Rodgers (2018) Patrick E. Shrout and Joseph L. Rodgers. 2018.  Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis.  _Annual Review of Psychology_ 69, 1 (2018), 487–510.  https://doi.org/10.1146/annurev-psych-122216-011845
  * Watson (2017) Mark Watson. 2017\.  _Web Cryptography API_.  W3C Recommendation. W3C.  https://www.w3.org/TR/2017/REC-WebCryptoAPI-20170126/.