# Emulating quantum computing with optical matrix multiplication

Mwezi Koni  School of Physics, University of the Witwatersrand, Private Bag 3,
Wits 2050, South Africa  Hadrian Bezuidenhout  School of Physics, University
of the Witwatersrand, Private Bag 3, Wits 2050, South Africa  Isaac Nape
School of Physics, University of the Witwatersrand, Private Bag 3, Wits 2050,
South Africa<EMAIL_ADDRESS>
###### Abstract

Optical computing harnesses the speed of light to perform vector-matrix
operations efficiently. It leverages interference, a cornerstone of quantum
computing algorithms, to enable parallel computations. In this work, we
interweave quantum computing with classical structured light by formulating
the process of photonic matrix multiplication using quantum mechanical
principles such as state superposition and subsequently demonstrate a well
known algorithm, namely the Deutsch-Jozsa’s algorithm. This is accomplished by
elucidating the inherent tensor product structure within the Cartesian
transverse degrees of freedom of light, which is the main resource for optical
vector-matrix multiplication. To this end, we establish a discrete basis using
localized Gaussian modes arranged in a lattice formation and demonstrate the
operation of a Hadamard Gate. Leveraging the reprogrammable and digital
capabilities of spatial light modulators, coupled with Fourier transforms by
lenses, our approach proves adaptable to various algorithms. Therefore our
work advances the use of structured light for quantum information processing.

††preprint: APS/123-QED

##  I Introduction

Controlling various degrees of freedom of light, i.e. time, frequency, spatial
and momentum, has become an emerging and promising tool for numerous
information processing tasks in classical and quantum domains, ranging from
novel imaging methods [1, 2, 3, 4, 5, 6, 7], communications [8, 9, 10, 11, 12,
13] and computation [14, 15, 16, 17], all harnessing the high dimensional
nature of structured light fields [18, 19, 20, 21, 22]. This is because
spatial modes enable for information encoding in qudit spaces for dimensions
$d>2$ per particle as opposed to the $d=2$ (qubits) encoding levels offered by
traditional qubit encoding such as with polarisation states. Because qubits
are easy to control, they are a reliable resource for numerous protocols.

In traditional quantum computing, encoding basis states are typically formed
from multiple qubits (d=2, states) and the operations needed to construct
arbitrary unitary gates in $d>2$ dimensions often involve a mix of two-
dimensional gate operations (such as the Hadamard gate and T-gate) along with
a multi-qubit operations (like the Control-not gate) [23]. This allows for
precise and efficient execution of arbitrary computations that are required to
demonstrate quantum advantage [24]. Given that each qubit is restricted to two
dimensions, the only way to enhance encoding capacity is by increasing the
number of qubits and ensuring optimal connectivity among them. An alternative
is to utilize qudits, which are particles that occupy $d>2$ dimensions,
because they offer larger encoding state spaces [25] and promise robustness
against noise [26]. As a result, structured photon states emerge as a
promising candidate for achieving this, as they occupy higher-dimensional
Hilbert spaces.

There have been numerous impressive proposals and implementations of quantum
computing algorithms that utilize the higher dimensional nature of the
internal degrees of freedom of photons [27, 28, 29]. This field has developed
tremendously, aided by the advent of multiplane light conversion technology
[30], where digital holography and free-space diffraction are employed to
realize arbitrary unitary gate operations [31]. Inspired by these advances,
diffraction-based deep neural network architectures have also been proposed
[32]. Additionally, harnessing mode mixing in complex media such as multimodal
optical fibers, as opposed to diffractive MPLCs, has emerged as a potential
avenue (see review on this topic [33]) to achieve similar operations but with
discrete pixel-like states [34]. To realise the full potential of qudits for
quantum computing, efficient single photon sources and the tools for
entangling and controlloing them are neccesary.

On the other hand, harnessing classical optical fields for quantum computing
has also been an active area of research[35, 36, 37, 38, 39, 14]. The drive in
this field has been inspired by the observation that some of the essential
resources of quantum computing can be found in classical coherent fields. Like
quantum states, classical waves can be prepared in superpositions and can be
interfered allowing for parallel information processing. Notably Jozsa also
argued that classical waves can provide an efficient simulation platform since
not all quantum algorithms require entanglement as a resource [40]. This opens
up new possibilities for studying or implementing quantum computational tasks
without relying on complex quantum hardware, by merely exploiting coherent
fields and performing the required matrix-vector operations on them.

An intriguing and unexplored approach to utilizing coherent fields in quantum
computing is Tamura’s optical matrix-vector multiplication from the 1970s [41]
, which was long forgotten but has recently resurfaced. This method harnesses
point-wise multiplication (Hadamard product) of light and the Fourier
transform capabilities of lenses. With this approach, operations have been
applied in dimensions reaching up to $d=56$ dimensions [42]. Hidden in this
method, is the capability to encode information in Hilbert spaces formed from
transverse modes of coherent laser fields.

In this work, we use optical matrix multiplication as a foundation for
immulating quantum algorithms therefore interacing quantum computing with
structure light. we unveil the inherent tensor product structure of transverse
spatial modes that is exploited in coherent optical matrix-vector
multiplication and show that it can be harnessed as a tool to emulate quantum
computing algorithms. Our logical basis is constructed from lattices of
displaced Gaussian modes, eigenmodes of free space, that are locally modulated
with digital holograms encoded on spatial light modulators. We treat the
Cartesian location coordinates $x$ and $y$ of each mode as our degrees of
freedom, forming the equivalent of two qudit registers. The $x$ component is
used to embed our state vectors, while the $y$-components acts as an ancilla
that facilities the unitary gate operator implementation. After a cylindrical
lens integrates over the $x$-component, the final resulting state is measured
in the $y$-coordinate. We test this using our basis and the Hadamard transform
as example and demonstrate the Deutsch-Jozsa algorithm with our encoding
scheme, showing it can be used to query balanced and constant function, with
average fidelity above $90\%$. In our scheme, we exploit the inherent
parallelism offered by optical matrix-vector multiplication, allowing to
prepare multiple identical spatial modes as a superposition, and encode the
oracle matrix as local phase transformations on each state.

Figure 1: (a) Analogue of quantum computing using optical vector-matrix
multiplication. $\text{SLM}_{1}$ encodes the lattice state via an operation,
$U$, by copying the vector components encoded in the x-direction into the
y-direction following Ref. [42]. The second SLM encodes the matrix operation
$M$. After this a cylindrical lens focuses the field in x-direction to produce
the final matrix-vector product at the output farfield plane. (b) The
encodings performed by the spatial light modulators (SLMs) operate on the
tensor product states $\ket{x_{k}}\otimes\ket{y_{j}}$ corresponding to the
discrete position states that can be obtained by partitioning the Cartesian
plane ($\mathbb{R}_{2}$) into non-overlapping segments. (c) A quantum circuit
for performing matrix multiplication using our basis states
$\ket{x_{k}}\otimes\ket{y_{j}}$ ,corresponding to the coordinates of the
partitions. The operations are represented as operators that act on the
x-coordinate and y-coordinate degrees of freedom. (d) An example of the
Hadamard basis states encoded onto the lattice states for N=4. Here, the
lattice is formed from propagation invariant Gaussian modes, each encoded into
the independent partitions. To demonstrate that they form a tensor product
space, $\mathcal{H}_{N}\otimes\mathcal{H}_{N}$, we show the individual states
for the x and y coordinates, where the final lattice is obtained from their
tensor products. These states are orthorgonal. (e) Example of one of the
states being acted on by the Hadamard transform and then focused to perform
the column wise summation. The embedded state vector is modulated, pointwise,
with an SLM encoding the Hadamard matrix. This is then focused in the farfield
with a cylindrical lens where the zero order component of the pattern contains
the result in the vertical direction. The outcomes are shown for other states
in the Hadamard basis.

##  II Theory

###  II.1 Matrix-vector multiplication via pointwise multiplication

Consider a matrix, $M$, with elements $M_{jk}$, and a vector $u$ with elements
$u_{k}$. We can compute the matrix-vector product to produce the vector
$v\equiv Mu$, where the $j^{\text{th}}$ component can be computed from
$v_{j}=\sum_{k}M_{jk}u_{k}$. The goal is to perform such an operation using
light fields. While this technique has already been introduced in classical
optics [42], here it will be reformulated for quantum computing, where the
vector $v$ now encodes a normalized quantum state and $M$ encodes a unitary
gate operator. Firstly, notice that the matrix-vector product can be rewritten
as:

$v_{j}=\sum_{k}M_{jk}u_{k}=\sum_{k}(M\odot U)_{jk},$ (1)

representing the summation over the columns of the element-wise product (or
Hadamard product [43]), denoted by the symbol $\odot$, between our matrix $M$
and another matrix $U$, which encodes or copies $u$ into its rows. The matrix
$U$ can be obtained from the outer product,
$U=u^{\intercal}\otimes\textbf{1}$, with
$\textbf{1}=(1,1,1,1,\ldots)^{\intercal}$.

We see that the matrix-vector multiplication can be decomposed into the
elementwise product $M\odot U$ and a subsequent summation of the columns.
These operations can be achieved optically using spatial light modulators
(SLM) and a cylindrical lens (CL) as shown in Fig. 1(a) as demonstrated in
[42]. On one SLM, a uniform field can be modulated with a hologram that
encodes (point-wise) the matrix $U$ and is imaged onto a second SLM, which is
encoded with the matrix $M$. The imaging system between the two SLMs enables
the elementwise multiplication of the encoded matrices. The final operation is
performed using a cylindrical lens that focuses the field in the direction
that coincides with the columns of $M$, performing a column-wise
summation—therefore completing the product to produce the elements $v_{j}$

Remarkably, hidden in this approach is the fact that the transverse Cartesian
coordinates are used as independent degrees of freedom that can be used to
enact operations, analogous to non-interacting multi-particle systems in
quantum computing. Next, we show how this idea can be used to emulate quantum
computing with transverse fields.

###  II.2 Formulation using quantum states

In our approach, we emulate quantum computing by formulating optical vector-
matrix multiplication in the language of quantum mechanics by representing the
optical fields that encode the vectors as states on Hilbert space and
operations acting on the fields to represent the matrices that enact the
equivilant of gate operations as shown in Fig. 1 (b) and (c). Firstly, the x-y
coordinate system can be decomposed into a tensor-product space using the
continuous degrees of freedom mapping the states
$\ket{x}\in\mathcal{H}_{\infty}$ and $\ket{y}\in\mathcal{H}_{\infty}$ so that
any field $\psi(x,y)$ is described by the state
$\ket{\psi}=\int\psi(x,y)\ket{x}\ket{y}dxdy$ where the basis states satisfy
$\innerproduct{x}{x^{\prime}}=\delta(x-x^{\prime})$ and
$\innerproduct{y}{y^{\prime}}=\delta(y-y^{\prime})$. This means that each
photon in a laser field is defined on a Hilbert space
$\mathcal{H}_{\infty}\otimes\mathcal{H}_{\infty}$ spanned by the cartesian
internal degrees of freedom (i.e. the coordinate states). By partitioning the
transverse plane into $N$ bounded and non-overlap intervals ,
$\ket{x}\rightarrow\ket{x_{j}}$ and $\ket{y}\rightarrow\ket{y_{k}}$, shown in
Fig. 1 (b), we can now define qudit states that form a high dimensional qudit
space on the combined Hilbert space, i.e.
$\mathcal{H}_{N}\otimes\mathcal{H}_{N}$ that has dimensions $N^{2}$. This
means that the coordinates spaces are analogues of independent qudits. From
these states, a uniform superposition can be prepared as

$\ket{\Psi}=\frac{1}{\sqrt{N}}\sum_{jk}\ket{x_{k}}\otimes\ket{y_{j}},$ (2)

where each state in the partition has a non-zero coefficient. From this
description, we see that the operations that can be applied on them can be
enacted on each individual degree of freedom or on both simultaneously. Using
our matrix vector product from Eq. (1), we can show that using such states and
treating the x and y degrees of freedom as analogues of qudit registers, that
we can execute matrix vector (equivalently, operator and state) products. For
instance, we can map the matrix $U$ that was defined earlier onto an operator
that encodes information into the lattice as,

$\hat{U}=\sum_{m}u_{m}\ket{x_{m}}\bra{x_{m}}\otimes\mathbb{I},$ (3)

this is the analogue for the matrix $\hat{U}$ in Eq. (1) - assuming that the
initial state is the superposition $\ket{\Psi}$. Here, the coefficients mark
the $x$-components but copy all the elements, $u_{m}$, across the
$y$-components - this still leaves the $y$-components independent of the
$x$-components. Similarly, for matrix $M$ we have the operator

$\hat{M}=\sum_{mn}M_{nm}\ket{x_{m}}\ket{y_{n}}\bra{y_{n}}\bra{x_{m}},$ (4)

, which instead interacts with both components (registers), therefore encoding
the matrix component’s $M_{nm}$ into each state $\ket{x_{m}}\ket{y_{n}}$ of
the lattice. The combined operation $\hat{M}\hat{U}$ produces the state,

$\ket{\Psi}=\frac{1}{\sqrt{N}}\sum_{jk}M_{jk}u_{k}\ket{x_{k}}\otimes\ket{y_{j}}.$
(5)

which has the same components as Eq. (1) $v_{k}=\sum_{k}M_{jk}u_{k}$ from
before. While the result of the multiplication is encoded into the lattice,
which is $N\times N$ in dimensions, the result can be read out as a column in
the y-direction once the summation $k$ (in the x-component) is contracted.
This can be done using the Fourier transforming (FT) lens in the x-direction -
the analogue of a quantum Fourier transform, therefore completing the curcuit
in Fig. 1 (c). The result of this is an interference pattern, having a zero
order component that corresponds to the amplitude,
$\sum_{jk}e^{if_{x_{l}}x_{k}}M_{jk}v_{k}$ for the frequency component
$f_{x_{l}}=0$ . Measuring the zeroth order interference pattern in x-direction
integrates the x-component, therefore leaving the y-components in the state

$\ket{v}=\sum_{jk}M_{jk}u_{k}\ket{y_{j}}.$ (6)

Figure 2: (a) Conceptual illustration of the balanced and constant functions
involved in the Deutsch-Jozsa Algorithm for binary strings of length two, i.e.
00, 01, 10, 11. The two top figures showcase a balanced function which maps
half of its inputs to $0$ and the other half to $1$ and the two bottom figure,
a constant function that maps all its inputs to either $0$ or to $1$. (b)
Illustration of our optical implementation of the Deutsch-Jozsa Algorithm. A
lattice of Gaussian beams, representing the superposition of states, is
simultaneously prepared by Alice. The state is then queried by Bob in parallel
by the action of the compound oracle and hadarmard operations, which impart
function evaluations spatially on the state through phase transformations and
map us back into the computational basis. A constant function returns the
state mapping onto the binary string $00$, indicated by the vector lighting up
on the first elements. A balanced function returns a state orthogonal to $00$,
shown by light lobes at different positions.

as expected. Next, measuring each y-components produces the probability
amplitudes $v_{j}=M_{jk}u_{k}$, the same outcome as v = Mu from Eq. (1).
Because this scheme allows us to express operations of the x-y coordinates
analogous to gate based computing platforms, we will exploit it for quantum
computing, where the matrix mechanics that describes quantum computing can be
formulated using the transverse spatial modes in Cartesian coordinates. Next
we introduce our basis of choice for doing this and illustrate the Hadamard
gate.

##  III Hadamard gate

Before we introduce the Hadamard gate we first present our encoding basis.
Instead of using arrays of square pixel like states for each bounded interval
on the x-y plane, we encode arrays of Gaussian modes, as shown in Fig. 1(c).
The illustrations show that our Gaussian lattices/arrays are encoded as tensor
products of arrays of one dimensional Gaussian modes, where
$(x_{j},y_{k})\rightarrow\ket{x_{j}}\ket{y_{k}}$ denote the centres of each
Gaussian mode given by the states,

$\displaystyle\ket{x_{j}}\ket{y_{k}}$
$\displaystyle\propto\int_{A_{j}}e^{-\frac{(x-x_{j})^{2}}{w}}\ket{x}dx\otimes\int_{A_{k}}e^{-\frac{(y-y_{k})^{2})}{w}}\ket{y}dy$
$\displaystyle=\int_{A_{j}\times
A_{k}}e^{-\frac{(x-x_{j})^{2}+(y-y_{k})^{2})}{w}}\ket{x}\ket{y}dxdy$ (7)

where each mode is centred at coordinates $(x_{j},y_{k})$ within a closed
boundaries $A_{j}\times A_{k}\subset\mathbb{R}_{2}$ that are non overlapping.
In this way, uniform superpositions such as in Eq. (2) can also be prepared,
i.e. resulting in the field in the first panel of Fig. 1 (d). The rest of the
fields in the figure represent instance of the Hadamard basis states, which
include the uniform superposition. Next, we show how the Hadamard gate can be
enacted on these fields.

The N-dimensional Hadamard gate, $\hat{H}_{N}$, is commonly used for quantum
computing to map between the logical and the superposition basis (or vice-
versa). For example, for two-dimensional states ($N=2$), we can describe the
logical basis as $\ket{v_{0}}\equiv\ket{0}=(1,0)^{\intercal}$ and
$\ket{v_{1}}\equiv\ket{1}=(0,1)^{\intercal}$,. In this basis the Hadamard gate
is given by, $\hat{H}_{2}=\frac{1}{\sqrt{2}}\begin{pmatrix}1&1\\\
1&-1\end{pmatrix}$. Applying the Hadamard gate maps the logical basis states
onto the Hadamard basis states $\ket{u_{1,2}}=1/\sqrt{2}(\ket{0}\pm\ket{1})$,
respectively. The states $\ket{u_{1,2}}$ are orthogonal and form a basis. The
Hadamard gate performs a change of basis, however when applied twice it leaves
the state unchanged, i.e $\hat{H}\hat{H}=\mathbb{I}$, true for all dimensions
N. For qudits, $N>2$, the Hadamard matrix can be computed for dimensions
$N=2^{n}$ ($n$ power of 2) as $\hat{H}_{N}=\hat{H}^{\otimes n}$, using tensor
products of $n$ Hadamard gates. Thus, the $k^{th}$ Hadamard basis state can be
obtained by applying the Hadamard gate to the $k^{th}$ state in the logical
basis following,

$\displaystyle\hat{H}_{N}\ket{v_{k}}$ $\displaystyle=\ket{u_{k}},$
$\displaystyle=\sum^{N-1}_{m=0}(-1)^{m\cdot k}\ket{m},$ (8)

where the states $\ket{m}$ are the standard basis states with coefficients
$(-1)^{m\cdot k}$ with $m\cdot k$ corresponding to the dot product of the
binary representation of the integers $m$ and $k$. Accordingly, the Hadamard
basis states can be mapped onto our lattice states as

$\ket{u_{k}}=\frac{1}{\sqrt{N}}\sum^{N-1}_{mn=0}(-1)^{m\cdot
k}\ket{x_{m}}\ket{y_{n}}.$ (9)

Figure 1 (d) shows examples of the lattice states that have been encoded with
the Hadamard basis states in $N=4$ dimensions. To encode these coefficients
into the lattice we use the operator $\hat{U}$ in Eq. 3, but with the
coefficients corresponding to our Hadamard basis states. Applying the Hadamard
gate to each of the states in the lattice should map them back to the logical
basis states, following $\hat{H}_{N}\ket{u_{k}}=\ket{v_{k}}$. Each basis state
$\ket{u_{k}}$ that is embedded in the lattice (as in Fig. 1 (d)) can be mapped
onto a unique element vector with one non-zero entry, as shown in Fig. 1 (e).
To encode the Hadamard gate, we can map it’s matrix components onto the
lattice, using Eq. ( 4) where $M_{jk}$ will represent the matrix elements of
the Hadamard gate. Thereafter, we apply the Fourier transform and the zero
order component filtering to complete the operator-state (matrix-vector)
multiplication.

Next we show how this scheme can be used to perform a well known quantum
algorithm.

###  III.1 Application: Deutsch-Jozsa algorithm

The Deutsch-Jozsa algorithm [44] provides an efficient quantum solution for
determining whether a boolean function, $f(\cdot)$, that takes binary values
as inputs, is balanced (half of the inputs map to 1 and the other half to 0)
or constant (all inputs either map to 0 or 1). To illustrate this, consider a
collection of 2-bit input strings chosen from the set
$\mathcal{B}=\\{00,01,10,11\\}$; each element $k\in\mathcal{B}$ can be used an
input to the function, i.e. F(k). A balanced function would return 0 for half
the entries, e.g., $k\in\\{00,11\\}$, and 1 for the other half,
$k\in\\{01,10\\}$, as shown in the first left panel of Fig. 2 (a). The second
example (top right panel of Fig. 2 (a)) shows an alternative combination that
can also represent a balanced function. For the constant case, all binary
values map to a single output, i.e., $\\{00,01,10,11\\}\rightarrow 0$ or
$\\{00,01,10,11\\}\rightarrow 1$ as shown in the bottom panel of Fig. 2 (b)
for two instances of a such a function.

To query the nature of such a function, the quantum computer prepares a
superposition of all possible binary values in $\mathcal{B}$, queries the
function values simultaneously, and through interference it returns a result
indicating whether the function is balanced or not. We adapt the algorithm to
our encoding scheme using the lattice basis and our operator-state (matrix-
vector) multiplication scheme.

Figure 3: Experimental setup showcasing the optical vector matrix
multiplication process. A laser beam is expanded using a telescope to overfill
the active area of the first Spatial Light Modulator ($\text{SLM}_{1}$). At
this point the beam is transformed to encode the input vector as repeated rows
in a matrix of Gaussian beams. This output is then relayed to $\text{SLM}_{2}$
using the 4f system of lenses (L1 and L2) where the lattice undergoes an
additional phase encoding. Subsequently, the beam is propagated through a
cylindrical lens (CL) and undergoes a one-dimensional Fourier transform in the
x-coordinate. The final outcome is captured by a CCD camera. The result of the
multiplication is found on the central fringe, which we filter out (shown in
the inset). The filtered mode image on the CCD camera shows the result of the
matrix-vector multiplication, where the regions of no intensity correspond to
zero elements and bright regions correspond to non-zero elements.

We illustrate the concept using four-dimensional (N=4) states as an example.
We can map the binary values of the inputs to the function $F(\cdot)$ in the
set $\mathcal{B}$ to the logical basis states

$\displaystyle\ket{0}\rightarrow\ket{00}$ $\displaystyle=\begin{pmatrix}1\\\
0\\\ 0\\\ 0\end{pmatrix},\ket{1}\rightarrow\ket{01}=\begin{pmatrix}0\\\ 1\\\
0\\\ 0\end{pmatrix},$ $\displaystyle\ket{2}\rightarrow\ket{10}$
$\displaystyle=\begin{pmatrix}0\\\ 0\\\ 1\\\
0\end{pmatrix},\ket{3}\rightarrow\ket{11}=\begin{pmatrix}0\\\ 0\\\ 0\\\
1\end{pmatrix}.$ (10)

Using this basis, we can prepare all the possible inputs as the unnormalised
superposition state $\ket{0}+\ket{1}+\ket{2}+\ket{3}$ where the integer
representation of the binary values are mapped onto their integer
representation. These correspond to the logical basis states $\ket{v_{k}}$
from the previous section. In the algorithm, we will see that the mappings of
the gates convert between the Hadamard basis states ($\ket{u_{k}}$) and the
logical basis states ($\ket{v_{k}}$). To prepare the uniform superposition
($\ket{0}+\ket{1}+\ket{2}+\ket{3}$) using our lattice, we can use the first
element of the Hadamard basis following Eq. (9)

$\ket{u_{0}}=\frac{1}{\sqrt{N}}\sum^{N-1}_{jk=0}\ket{x_{k}}\otimes\ket{y_{j}}.$
(11)

This indeed encodes the uniform superposition into our lattice as shown in
Fig. 2 (b). Next, the features of the function $F(k)$, are encoded onto the
unitary operator as $U_{F}$ so that the states in the superposition become
$\sum_{k}(-1)^{F(k)}\ket{k}$, showing that $U_{F}$ is diagonal, and has the
form

$U_{F}=\begin{pmatrix}(-1)^{F(0)}&0&\dots&0\\\ 0&(-1)^{F(1)}&\dots&0\\\
\vdots&\vdots&\ddots&\vdots\\\ 0&0&\dots&(-1)^{F(N-1)}\end{pmatrix}$ (12)

where each state in the superposition is marked with a coefficient
$(-1)^{F(k)}$. As such, we see that $U_{F}$ encodes a $\pi$ phases depending
on whether $F(k)$ is 0 or 1. Motivated by this, we encode the unitary as
$\hat{U}_{F}=\sum_{k}(U_{F})_{kk}\ket{x_{k}}\bra{x_{k}}\otimes\mathbb{I}$ onto
our lattice, where $(U_{F})_{kk}$ are the diagonal elements of $U_{F}$ ,
marking the x-components of the lattice with the desired phases. In this way
we do not need ancillary particles thanks to the higher dimensional nature of
our degrees of freedom.

Accordingly, given the uniform superposition of the lattice, we obtain

$\displaystyle\hat{U}_{F}\ket{u_{0}}$
$\displaystyle=\frac{1}{2}((-1)^{F(0)}\ket{x_{0}}\ket{y_{0}}+(-1)^{F(1)}\ket{x_{1}}\ket{y_{0}}+..)$
(13)
$\displaystyle=\frac{1}{\sqrt{N}}\sum^{N-1}_{jk=0}(-1)^{F(k)}\ket{x_{k}}\ket{y_{j}}.$
(14)

As a result, we can express the state as a piece wise function

$\hat{U}_{F}\ket{u_{0}}=\frac{1}{\sqrt{N}}\begin{cases}\pm\ket{u_{0}},&\text{Constant}\\\
\sum_{k}(-1)^{F(k)}\ket{x_{k}}\ket{y_{j}}&\text{Balanced},\end{cases}$ (15)

showing that the state remains unchanged if the function is constant, while it
can be in an arbitrary superposition if it is balanced. Finally encoding the
Hadamard matrix and applying the cylindrical lens and performing the filtering
leaves the y-coordinate in the state

$\ket{v}=\begin{cases}\pm\ket{y_{0}},&\text{Constant}\\\ \sum_{{j\neq
0}}u_{j}\ket{y_{j}}&\text{Balanced},\end{cases}$ (16)

illustrated graphically in Fig. 2 (b), showing the expected output intensities
mapped onto the vertical rows of the output field. For the constant case we
have that the algorithm returns the first element of the basis ,
$\ket{v_{0}}\equiv\ket{y_{0}}$. In the N=4 binary example, this is similar to
obtaining the $\ket{v_{0}}\equiv\ket{00}$ state. The balanced case returns an
arbitrary superposition of the logical basis states with coefficients given by
$u_{j}=\frac{1}{\sqrt{N}}\sum_{m}(-1)^{m\cdot j+F(j)}$ where $m\cdot j$ is the
binary inner product using the binary representation of $m$ and $j$. For
example, for a balanced function that maps as $\\{00,11\\}\rightarrow 1$ and
$\\{01,10\\}\rightarrow 0$, corresponds to a unitary

$U_{F}=\begin{pmatrix}-1&0&0&0\\\ 0&1&0&0\\\ 0&0&1&0\\\
0&0&0&-1\par\end{pmatrix},$ (17)

represented in the logical basis. The output for this example is
$-\ket{v_{3}}\equiv-\ket{11}$. In general, the output state is one of the
logical basis states that exclude, $\ket{v_{0}}\equiv\ket{00}$ as shown in
Fig. 2 (b). In our setup, we execute the protocol by encoding $\ket{u_{0}}$ on
the first SLM and then subsequently encode $\hat{M}=\hat{H}_{N}\hat{U}$ onto
the second SLM.

Figure 4: Setup Characterisation:Intensity images collected by the CCD camera
when encoding a Hadamard matrix on the second SLM and the vector built from
Hadamard columns on the first SLM. In (a), the extraction of the central lobe
containing our solution vector from the multiplication of a 2x2 Hadamard
matrix with the first column is highlighted. (b) We then stack these to form a
diagonal matrix consisting of the light fringes as our elements. This diagonal
matrix is then used to form a crosstalk matrix, from which the fidelity of the
computation is computed. (c-e) Extension of this process to higher dimensional
matrices, up to 16D.

##  IV Experiment

Figure 5: Deutsch-Jozsa algorithm results: Algorithm results for the Deutsch-
Jozsa algorithm implementation. Panels (a), (b), and (c) show the intensity
distributions for single-qubit, two-qubit, and three-qubit systems,
respectively, with the form of the oracle diagonal matrix $U_{F}$. In each
panel: (1) Unfiltered images representing the initial light distribution, and
(2) Filtered intensity images corresponding to the central fringe at $k=0$.
Subpanels (i) correspond to the constant function, and subpanels (ii)
correspond to the balanced function. The bar graphs depict the normalized
intensity against the row position, allowing the determination of the function
type. The presence or absence of light in the first vector element’s spatial
position indicates whether the function is balanced or constant.

In FIG 3 we illustrate the setup used for our demonstration. To ensure a
uniform field intensity across the transverse plane, we overfilled the first
spatial light modulator (SLM1) by magnifying a laser beam from a Helium Neon
(HeNe) source at a wavelength of 633 nm, using the telescope. On
$\text{SLM}_{1}$, the input vector for the lattice of Gaussian modes was
encoded as an array of displaced Gaussian beams, producing the superposition
state, $\ket{\Psi}\equiv\ket{u_{0}}$ from Eq. (2). To alter the coefficients
of the lattice, each row of the matrix (U) mapping the vector was encoded
using a digital hologram that encodes amplitudes and phases [45] onto of each
Gaussian mode. Because the SLM encodes the vector state as a matrix, $U$,
given any state $\ket{u}$, the elements were copied into its rows following
the procedure outlined in the theory.

The matrix generated on $\text{SLM}_{1}$ was then transferred to
$\text{SLM}_{2}$ using an imaging system consisting of lenses $L_{1}$ and
$L_{2}$. An aperture was placed between the two lenses to filter the modulated
first order diffraction pattern from $\text{SLM}_{1}$. On $\text{SLM}_{2}$, we
applied an additional phase encoding to the lattice/matrix elements, thereby
completing the optical representation of the matrix-vector multiplication
process. Following the encoding on $\text{SLM}_{2}$, the beam was passed
through a cylindrical lens to undergo a one-dimensional Fourier transform.
This operation focused the outcome of the matrix-vector multiplication within
the central fringe (shown as an inset in Fig. 3), specifically at the zeroth
order Fourier component in coordinate. To extract this outcome accurately, we
employed digital filtering, enabling precise capture of the resultant vector
as shown in the inset of Fig. 3.

##  V Results

###  V.1 Matrix-Multiplication

To characterise our system, we exploit the orthogonality of the Hadamard
basis. In order to do this, we prepared the Hadamard basis states
${\ket{u_{j}}}$ on $\text{SLM}_{1}$ and encoded the Hadamard transform on
$\text{SLM}_{2}$. Thereafter, we propagated the field through the cylindrical
lens and collected the resulting interference pattern on a CCD camera. The
multiplication between the Hadamard basis states and the Hadamard gate enable
us to clearly distinguish between different columns of the output vector
state. Accordingly, we obtain orthogonal states ($\ket{v_{j}}$) with distinct
positions marked by bright lobes. We show a measured example of this for $N=2$
dimensions in Fig. 4 (a) where the first basis state, $\ket{u_{0}}$, is
encoded on the lattice and measured using the Hadamard gate. The outcome is a
measured interference pattern, where the resulting filtered zero order
produces a bright lobe in the first entry - for this specific example, the
result corresponds to the state $\ket{v_{0}}$. Summing and normalising the
intensity for each region of the output vector, corresponded to extracting the
output vector elements.

We repeated the process for all the basis states for a given dimension. In
Fig. 4 (b) we show the measured lobes for all the states in $N=2$, confirming
that the Hadamard basis states ($\ket{u_{0}}$ and $\ket{u_{1}}$) are mapped on
to the basis states $\ket{v_{0}}$ and $\ket{v_{1}}$. From the measured lobes,
we extracted the intensity from each component and produce a crosstalk matrix
(C.M). For $N=2$. The cross talk matrix resembles an identity matrix showing
that the measure basis modes are distinguishable and therefore confirms
orthogonality between the measured vectors. To quantify this, we measured a
fidelity of F = $0.99\pm 0.01$, showing that the Hadamard matrix can map the
basis elements $\ket{u_{j}}$ onto the basis states $\ket{v_{j}}$. This was
calculated by normalising each row, and computing the average over the
diagonal components.

Figures. 4 (c)-(e) show the extension of this process to higher-dimensional
matrices. For dimensions $N=4$, $N=8$ and $N=16$ we obtain average fidelities
of $0.96\pm 0.01$, $0.94\pm 0.05$ and $0.81\pm 0.02$ multiplication,
respectively. However, as the dimensions increase, we note that the fidelity
decreases. We attribute this to the diffraction of Gaussian beams. Increasing
the dimension or number of Gaussian modes for the same physical size of the
matrix means decreasing the beam waist $w_{0}$ of each mode. As the beam
waists get smaller, they expand at a faster rate during propagation, according
to $w(z)=w_{0}\sqrt{1+\left(\frac{\lambda z}{\pi w_{0}^{2}}\right)^{2}}$.
Given that the cylindrical lens focuses light in one direction while
neglecting the other, we expect the unfocused direction to continue expanding,
more rapidly as $w_{0}$ gets smaller, resulting in overlapping vector elements
which are hard to distinguish. This diffraction effect impacts the fidelity of
higher-dimensional computations.

###  V.2 Deutsch-Jozsa Algorithm demonstration

Using our scheme, we demonstrate the Deutsch-Jozsa algorithm for a function
that encodes information using N=2, N=4 and N=8 basis elements, shown in Fig.
5 (a)-(c), respectively. We initialized our system in the superposition state
$\ket{u_{0}}$ by digitally displaying a hologram encoding the uniform lattice
of Gaussian modes on $\text{SLM}_{1}$. The action of the oracle phase
transformation, and the subsequent Hadamard gate are encoded as a compound
unitary transformation $H\hat{U}_{F}$ on the second ($\text{SLM}_{2}$), the
light beam is then sent through a cylindrical lens which acts as a summing
operator. The resulting interference pattern that was measured with the CCD
camera and filtered zero order vector components are shown as insets in each
figure.

From these images, we then plot the cross-section of the normalized intensity
against the vector element number. In this case, we can distinguish whether
the function is balanced or constant by interrogating the presence or absence
of light in the spatial position of the vector element. Using our basis
vectors from the characterisation, this means a constant function produces the
vector $\ket{v_{0}}$ and if the function is balanced $\ket{v_{j\neq 0}}$. The
instance of the oracle operator ($\hat{U}_{F}$) used for each type of function
is shown in each figure; this is shown as an inset in the intensity plots
representing the measured components.

For each case, we accurately determined whether the function encoded in the
unitary operation is constant or balanced. The average fidelity of these
measurements exceeded $90\%$ across all cases. Specifically, for N=2, 4, and
8, we achieved fidelities of $0.99\pm 0.01$, $0.97\pm 0.01$, and $0.93\pm
0.05$, respectively.

##  VI Discussion and Conclusion

In this study, we utilized optical matrix-vector multiplication to emulate
quantum computing by treating the $x$-$y$ components of a coherent field as a
tensor product space, $\mathcal{H}_{N}\otimes\mathcal{H}_{N}$, derived from
individual coordinates. The encoding basis was defined within this Hilbert
space using a lattice of Gaussian modes, where the positions of the centers in
the $x$-$y$ coordinates established localized states. Vector states were
encoded along the $x$-coordinate, while both coordinates were employed to
encode the matrix operator. The output vector was measured along the
$y$-coordinate after filtering out the zero-order component of the output
interference pattern. We demonstrated the use of Hadamard basis elements and
the Hadamard gate, achieving encoding and decoding fidelities of high as
$95\%$ for $N=8$ dimensions and above $80\%$ for $N=16$.

Further, our implementation of the Deutsch-Jozsa algorithm using optical
vector matrix multipliers leverages the representation of quantum states as
vectors and quantum operators as matrices that can be imprinted on our lattice
basis. Optically, we prepare multiple identical Gaussian modes in parallel,
with each mode representing an element of the states that are inputs to the
function $F(k)$. The high-dimensional nature of our encoding scheme enables
the oracle function in our demonstration to be encoded as a diagonal operator.

We acknowledge the proposal by Perez-Garcia et al. [39], which employs
classical light following Dragoman’s method. However, this approach
necessitates multiple optical elements to prepare the superposition state. In
contrast, our method efficiently prepares the state in a single step by
projecting light onto a spatial light modulator (SLM) in a reprogrammable
manner - the digital encoding allows for dynamic control and can therefore be
adapted for any operation. Therefore, we anticipate that any transformation
(operator/gate) can be encoded, including X-gate, Z-gate, ect., because the
method allows any matrix operation to be encoded into the lattice.

##  VII Acknowledgements

The authors acknowledge the South African Quantum initiative, National
research Foundation (South Africa) and Optica for funding.

## References

  * Cameron et al. [2024] Patrick Cameron, Baptiste Courme, Chloé Vernière, Raj Pandya, Daniele Faccio, and Hugo Defienne.  Adaptive optical imaging with entangled photons.  _Science_ , 383(6687):1142–1148, 2024.
  * Moodley and Forbes [2024] Chané Moodley and Andrew Forbes.  Advances in quantum imaging with machine intelligence.  _Laser & Photonics Reviews_, page 2300939, 2024.
  * Geng [2011] Jason Geng.  Structured-light 3d surface imaging: a tutorial.  _Advances in optics and photonics_ , 3(2):128–160, 2011.
  * Erkmen and Shapiro [2010] Baris I Erkmen and Jeffrey H Shapiro.  Ghost imaging: from quantum to classical to computational.  _Advances in Optics and Photonics_ , 2(4):405–450, 2010.
  * Ferri et al. [2010] Fabio Ferri, D Magatti, LA Lugiato, and A Gatti.  Differential ghost imaging.  _Physical review letters_ , 104(25):253603, 2010.
  * Zia et al. [2023] Danilo Zia, Nazanin Dehghan, Alessio D’Errico, Fabio Sciarrino, and Ebrahim Karimi.  Interferometric imaging of amplitude and phase of spatial biphoton states.  _Nature Photonics_ , 17(11):1009–1016, 2023.
  * Kim et al. [2022] Gyeongtae Kim, Yeseul Kim, Jooyeong Yun, Seong-Won Moon, Seokwoo Kim, Jaekyung Kim, Junkyeong Park, Trevon Badloe, Inki Kim, and Junsuk Rho.  Metasurface-driven full-space structured light for three-dimensional imaging.  _Nature Communications_ , 13(1):5920, 2022.
  * Mirhosseini et al. [2015] Mohammad Mirhosseini, Omar S Magaña-Loaiza, Malcolm N O’Sullivan, Brandon Rodenburg, Mehul Malik, Martin PJ Lavery, Miles J Padgett, Daniel J Gauthier, and Robert W Boyd.  High-dimensional quantum cryptography with twisted light.  _New Journal of Physics_ , 17(3):033033, 2015.
  * Wang et al. [2012] Jian Wang, Jeng-Yuan Yang, Irfan M Fazal, Nisar Ahmed, Yan Yan, Hao Huang, Yongxiong Ren, Yang Yue, Samuel Dolinar, Moshe Tur, et al.  Terabit free-space data transmission employing orbital angular momentum multiplexing.  _Nature photonics_ , 6(7):488–496, 2012.
  * Krenn et al. [2014] Mario Krenn, Robert Fickler, Matthias Fink, Johannes Handsteiner, Mehul Malik, Thomas Scheidl, Rupert Ursin, and Anton Zeilinger.  Communication with spatially modulated light through turbulent air across vienna.  _New Journal of Physics_ , 16(11):113028, 2014.
  * Wang et al. [2022] Jian Wang, Jun Liu, Shuhui Li, Yifan Zhao, Jing Du, and Long Zhu.  Orbital angular momentum and beyond in free-space optical communications.  _Nanophotonics_ , 11(4):645–680, 2022.
  * Forbes et al. [2024] Andrew Forbes, Mostafa Youssef, Sachleen Singh, Isaac Nape, and Bora Ung.  Quantum cryptography with structured photons.  _Applied Physics Letters_ , 124(11), 2024.
  * Lavery et al. [2017] Martin PJ Lavery, Christian Peuntinger, Kevin Günthner, Peter Banzer, Dominique Elser, Robert W Boyd, Miles J Padgett, Christoph Marquardt, and Gerd Leuchs.  Free-space propagation of high-dimensional structured optical fields in an urban environment.  _Science Advances_ , 3(10):e1700552, 2017.
  * Perez-Garcia et al. [2018] Benjamin Perez-Garcia, Raul I Hernandez-Aranda, Andrew Forbes, and Thomas Konrad.  The first iteration of grover’s algorithm using classical light with orbital angular momentum.  _Journal of Modern Optics_ , 65(16):1942–1948, 2018.
  * Peruzzo et al. [2014] Alberto Peruzzo, Jarrod McClean, Peter Shadbolt, Man-Hong Yung, Xiao-Qi Zhou, Peter J Love, Alán Aspuru-Guzik, and Jeremy L O’brien.  A variational eigenvalue solver on a photonic quantum processor.  _Nature communications_ , 5(1):4213, 2014.
  * Konno et al. [2024] Shunya Konno, Warit Asavanant, Fumiya Hanamura, Hironari Nagayoshi, Kosuke Fukui, Atsushi Sakaguchi, Ryuhoh Ide, Fumihiro China, Masahiro Yabuno, Shigehito Miki, et al.  Logical states for fault-tolerant quantum computation with propagating light.  _Science_ , 383(6680):289–293, 2024.
  * McMahon [2023] Peter L McMahon.  The physics of optical computing.  _Nature Reviews Physics_ , 5(12):717–734, 2023.
  * Nape et al. [2023] Isaac Nape, Bereneice Sephton, Pedro Ornelas, Chane Moodley, and Andrew Forbes.  Quantum structured light in high dimensions.  _APL Photonics_ , 8(5), 2023.
  * Forbes et al. [2021] Andrew Forbes, Michael de Oliveira, and Mark R Dennis.  Structured light.  _Nature Photonics_ , 15(4):253–262, 2021.
  * Rubinsztein-Dunlop et al. [2016] Halina Rubinsztein-Dunlop, Andrew Forbes, Michael V Berry, Mark R Dennis, David L Andrews, Masud Mansuripur, Cornelia Denz, Christina Alpmann, Peter Banzer, Thomas Bauer, et al.  Roadmap on structured light.  _Journal of Optics_ , 19(1):013001, 2016.
  * Piccardo et al. [2021] Marco Piccardo, Vincent Ginis, Andrew Forbes, Simon Mahler, Asher A Friesem, Nir Davidson, Haoran Ren, Ahmed H Dorrah, Federico Capasso, Firehun T Dullo, et al.  Roadmap on multimode light shaping.  _Journal of Optics_ , 24(1):013001, 2021.
  * Bliokh et al. [2023] Konstantin Y Bliokh, Ebrahim Karimi, Miles J Padgett, Miguel A Alonso, Mark R Dennis, Angela Dudley, Andrew Forbes, Sina Zahedpour, Scott W Hancock, Howard M Milchberg, et al.  Roadmap on structured waves.  _Journal of Optics_ , 25(10):103001, 2023.
  * Boykin et al. [2000] P Oscar Boykin, Tal Mor, Matthew Pulver, Vwani Roychowdhury, and Farrokh Vatan.  A new universal and fault-tolerant quantum basis.  _Information Processing Letters_ , 75(3):101–107, 2000.
  * Kitaev et al. [2002] Alexei Yu Kitaev, Alexander Shen, and Mikhail N Vyalyi.  _Classical and quantum computation_.  Number 47. American Mathematical Soc., 2002.
  * Wang et al. [2020] Yuchen Wang, Zixuan Hu, Barry C Sanders, and Sabre Kais.  Qudits and high-dimensional quantum computing.  _Frontiers in Physics_ , 8:589504, 2020.
  * Ecker et al. [2019] Sebastian Ecker, Frédéric Bouchard, Lukas Bulla, Florian Brandt, Oskar Kohout, Fabian Steinlechner, Robert Fickler, Mehul Malik, Yelena Guryanova, Rupert Ursin, et al.  Overcoming noise in entanglement distribution.  _Physical Review X_ , 9(4):041042, 2019.
  * Garcia-Escartin and Chamorro-Posada [2012] Juan Carlos Garcia-Escartin and Pedro Chamorro-Posada.  Quantum computer networks with the orbital angular momentum of light.  _Physical Review A_ , 86(3):032334, 2012.
  * O’brien [2007] Jeremy L O’brien.  Optical quantum computing.  _Science_ , 318(5856):1567–1570, 2007.
  * Gao et al. [2019] Xiaoqin Gao, Mario Krenn, Jaroslav Kysela, and Anton Zeilinger.  Arbitrary d-dimensional pauli x gates of a flying qudit.  _Physical review A_ , 99(2):023825, 2019.
  * Fontaine et al. [2022] Nicolas K Fontaine, Joel Carpenter, Simon Gross, Sergio Leon-Saval, Yongmin Jung, David J Richardson, and Rodrigo Amezcua-Correa.  Photonic lanterns, 3-d waveguides, multiplane light conversion, and other components that enable space-division multiplexing.  _Proceedings of the IEEE_ , 110(11):1821–1834, 2022.
  * Brandt et al. [2020] Florian Brandt, Markus Hiekkamäki, Frédéric Bouchard, Marcus Huber, and Robert Fickler.  High-dimensional quantum gates using full-field spatial modes of photons.  _Optica_ , 7(2):98–107, 2020.
  * Wang et al. [2024] Qianke Wang, Jun Liu, Dawei Lyu, and Jian Wang.  Ultrahigh-fidelity spatial mode quantum gates in high-dimensional space by diffractive deep neural networks.  _Light: Science & Applications_, 13(1):10, 2024.
  * Lib and Bromberg [2022] Ohad Lib and Yaron Bromberg.  Quantum light in complex media and its applications.  _Nature Physics_ , 18(9):986–993, 2022.
  * Goel et al. [2024] Suraj Goel, Saroch Leedumrongwatthanakun, Natalia Herrera Valencia, Will McCutcheon, Armin Tavakoli, Claudio Conti, Pepijn WH Pinkse, and Mehul Malik.  Inverse design of high-dimensional quantum optical circuits in a complex medium.  _Nature Physics_ , pages 1–8, 2024.
  * Cerf et al. [1998] Nicolas J Cerf, Christoph Adami, and Paul G Kwiat.  Optical simulation of quantum logic.  _Physical Review A_ , 57(3):R1477, 1998.
  * Spreeuw [2001] Robert JC Spreeuw.  Classical wave-optics analogy of quantum-information processing.  _Physical Review A_ , 63(6):062302, 2001.
  * Londero et al. [2004] P Londero, C Dorrer, M Anderson, S Wallentowitz, K Banaszek, and IA Walmsley.  Efficient optical implementation of the bernstein-vazirani algorithm.  _Physical Review A_ , 69(1):010302, 2004.
  * Kaur et al. [2007] Gurpreet Kaur, Geetu Narang, et al.  Optical implementations, oracle equivalence, and the bernstein-vazirani algorithm.  _JOSA B_ , 24(2):221–225, 2007.
  * Perez-Garcia et al. [2016] Benjamin Perez-Garcia, Melanie McLaren, Sandeep K Goyal, Raul I Hernandez-Aranda, Andrew Forbes, and Thomas Konrad.  Quantum computation with classical light: implementation of the deutsch–jozsa algorithm.  _Physics Letters A_ , 380(22-23):1925–1931, 2016.
  * Jozsa [1997] Richard Jozsa.  Entanglement and quantum computation.  _arXiv preprint quant-ph/9707034_ , 1997.
  * Tamura and Wyant [1979] Poohsan N Tamura and James C Wyant.  Two-dimensional matrix multiplication using coherent optical techniques.  _Optical Engineering_ , 18(2):198–204, 1979.
  * Spall et al. [2020] James Spall, Xianxin Guo, Thomas D Barrett, and AI Lvovsky.  Fully reconfigurable coherent optical vector–matrix multiplication.  _Optics Letters_ , 45(20):5752–5755, 2020.
  * Horn [1990] Roger A Horn.  The hadamard product.  In _Proc. Symp. Appl. Math_ , volume 40, pages 87–169, 1990.
  * Deutsch and Jozsa [1992] David Deutsch and Richard Jozsa.  Rapid solution of problems by quantum computation.  _Proceedings of the Royal Society of London. Series A: Mathematical and Physical Sciences_ , 439(1907):553–558, 1992.
  * Arrizón et al. [2007] Victor Arrizón, Ulises Ruiz, Rosibel Carrada, and Luis A González.  Pixelated phase computer holograms for the accurate encoding of scalar complex fields.  _JOSA A_ , 24(11):3500–3507, 2007.


# FedMVT: Semi-supervised Vertical Federated Learning with MultiView Training
††thanks: International Workshop on Federated Learning for User Privacy and
Data Confidentiality in Conjunction with IJCAI 2020 (FL-IJCAI’20)

Yan Kang,1 Yang Liu,1 Tianjian Chen 1 1WeBank, Shenzhen, China
{yangkang, yangliu, tobychen<EMAIL_ADDRESS>
###### Abstract

Federated learning allows many parties to collaboratively build a model
without exposing data. Particularly, vertical federated learning (VFL) enables
parties to build a robust shared machine learning model based upon distributed
features about the same samples. However, VFL requires all parties to share a
sufficient amount of overlapping samples. In reality, the set of overlapping
samples may be small, leaving the majority of the non-overlapping data
unutilized. In this paper, we propose Federated Multi-View Training (FedMVT),
a semi-supervised learning approach that improves the performance of VFL with
limited overlapping samples. FedMVT estimates representations for missing
features and predicts pseudo-labels for unlabeled samples to expand training
set, and trains three classifiers jointly based upon different views of the
input to improve model’s representation learning. FedMVT does not require
parties to share their original data and model parameters, thus preserving
data privacy. We conduct experiments on the NUS-WIDE and the CIFAR10. The
experimental results demonstrate that FedMVT significantly outperforms vanilla
VFL that only utilizes overlapping samples, and improves the performance of
the local model in the party that owns labels.

##  1 Introduction

With the increasingly stricter privacy-protection laws implemented worldwide
GDPR (2018); DLA Piper (2018), federated learning has received significant
attention and became a popular research topic recently Kairouz et al. (2019).
With the research goes deeper and wider, the practice of federated learning
has been expanded from building powerful mobile applications based on data
resided in millions of mobile devices McMahan et al. (2016) to solving the
problem of data fragmentation and isolation among or within organizations Yang
et al. (2019a). For example, many business decisions of a bank may rely on the
purchasing preferences of its customers. This bank may share some customers
with a local retail company, who owns purchasing preference data of the local
people. Thus, the bank can invite the retail company to collaboratively build
a shared model leveraging the data features of both sides to improve its
businesses. The retail company can also benefit from the shared model.

This and many other similar practical demands Yang et al. (2019a) motivate the
development of vertical federated learning (VFL)Yang et al. (2019b) that
enables participating parties to collaboratively train a machine learning
model by fully utilizing scattered features of their overlapping samples
without exposing data. However, a critical prerequisite of VFL is that it
requires all parties to share a sufficient amount of overlapping samples in
order to achieve decent performance. Liu et al. (2018) propose a federated
transfer learning framework to address weak supervision (few labels) problems
in the VFL setting. Nonetheless, It does not utilize all labeled data for
supervised training, nor does it take full advantage of unlabeled samples for
improving learning quality. Many other similar approaches such as domain
adaption Peterson et al. (2019) and knowledge distillation Li and Wang (2019),
have been applied in the federated learning setting. However, they mainly
focus on scenarios where all parties share the same feature space (also known
as horizontal federated learning Yang et al. (2019b)). Therefore, research on
applying semi-supervised techniques in the VFL setting is insufficient.

In this paper, we propose a novel semi-supervised algorithm in VFL setting,
termed FedMVT, to address the limitations of existing vertical federated
learning approaches. Our contributions are as follows:

  1. 1.

FedMVT significantly boosts up the performance of the federated model when the
overlapping samples between parties are limited;

  2. 2.

FedMVT helps improve the performance of the local classifier of the party that
owns labels, and thus it enables the party to make local predictions with more
confidence;

  3. 3.

FedMVT works with data of various types. This is important in practice since
real-world VFL applications often need to deal with heterogeneous features;

  4. 4.

FedMVT does not require participating parties to share their original data and
model parameters, but only intermediate representations and gradients, which
can be further protected by the VFL-DNN (Deep Neural Network) framework
implemented by FATE111https://github.com/FederatedAI/FATE.

##  2 Related Work

In this section, we focus on reviewing closely related fields and approaches.

#### Vertical Federated Learning (VFL).

VFL is also known as feature-partitioned machine learning in some literature.
The secure linear machine learning with data partitioned in the feature space
has been well studied in Gascón et al. (2016); Hardy et al. (2017); Mohassel
and Zhang (2017). They apply either hybrid MPC (Secure Multi-party
Computation) protocol or additively homomorphic encryption Rivest et al.
(1978) for secure linear model training. Cheng et al. (2019) proposed a secure
federated tree-boosting (SecureBoost) approach in the VFL setting that enables
participating parties with different features to collaboratively build a set
of boosting trees, and they prove that the SecureBoost provides the same level
of accuracy as its non-privacy-preserving centralized counterparts. FATE
designed and implemented a VFL-DNN (Deep Neural Network) framework that
supports DNN in the VFL setting. VFL-DNN leverages a hybrid encryption scheme
in the forward stage and backward stage of the VFL training process to protect
data privacy and model security. Vepakomma et al. (2018) proposed several
configurations of splitting a deep neural network to support various forms of
collaborations among health entities that each holds a partial deep network
and owns a different modality of patient data.

#### Semi-supervised Learning (SSL).

SSL aims to alleviate the need for a large amount of labeled data by allowing
a model to leverage unlabeled data. Many recent approaches for semi-supervised
learning use transfer learning Pan et al. (2010), consistency regularization
Tarvainen and Valpola (2017); Laine and Aila (2017) or pseudo-labeling hyun
Lee (2013); Clark et al. (2018) to learn from unlabeled data. Berthelot et al.
(2019) proposed MixMatch that unifies three dominant semi-supervised methods:
entropy minimization Grandvalet and Bengio (2004), consistency regularization
and generic regularization. They demonstrate that MixMatch can approach the
error rate achieved by fully supervised training with significantly fewer
data. Clark et al. (2018) proposed a semi-supervised Cross-View Training (CVT)
approach that simultaneously trains a full model based upon all labeled data
and multiple auxiliary models that only see restricted views of the unlabeled
data. During training, CVT teaches auxiliary models to match predictions made
by the full model to improve representation learning and in turn improve the
full model. While successful, these works are not designed for VFL scenarios
where features are scattered among parties. Liu et al. (2018) proposed a
secure Federated Transfer Learning (FTL) framework, which is the first
framework that enables VFL to benefit from transfer learning. FTL helps the
party in the target domain build a prediction model by utilizing rich label
resources from the source domain.

##  3 Problem Definition

Consider party A having dataset
$\mathcal{D}^{A}:=\\{(x^{A,i},y^{A,i})\\}_{i=1}^{N^{A}}$ where $x^{A,i}\in
R^{a}$ is the feature of the $i\text{th}$ sample and $y^{A,i}\in\\{0,1\\}^{C}$
is the corresponding one-hot encoding ground-truth label for $C$ classes
corresponding to $x^{A,i}$, party B having
$\mathcal{D}^{B}:=\\{x^{B,i}\\}_{i=1}^{N^{B}}$ where $x^{B,i}\in R^{a}$.
$\mathcal{D}^{A}$ and $\mathcal{D}^{B}$ are held privately by the two parties
and cannot be exposed to each other. $N^{A}$ and $N^{B}$ are numbers of
samples for $D^{A}$ and $D^{B}$ respectively.

We assume that there exists a limited set of overlapping samples
$\mathcal{D}_{ol}:=\\{x_{ol}^{B,i},x_{ol}^{A,i},y_{ol}^{A,i}\\}_{i=1}^{N_{ol}}$
between the two parties, where party B owns the partition
$\mathcal{D}^{B}_{ol}:=\\{x_{ol}^{B,i}\\}_{i=1}^{N_{ol}}$ and party A owns the
rest $\mathcal{D}^{A}_{ol}:=\\{x_{ol}^{A,i},y_{ol}^{A,i}\\}_{i=1}^{N_{ol}}$.
$N_{ol}$ is the number of overlapping samples. One can find the set of
overlapping samples through encrypted entity alignment techniques in a
privacy-preserving setting Nock et al. (2018). Here, we assume that party A
and party B have already found or known the IDs of their overlapping samples.
We denote $D^{A}_{nl}:=\\{x_{nl}^{A,j},y_{nl}^{A,j}\\}_{j=1}^{N^{A}_{nl}}$ as
the non-overlapping samples for party A, while
$D^{B}_{nl}:=\\{x_{nl}^{B,z}\\}_{z=1}^{N^{B}_{nl}}$ for party B.

If we concatenate $\mathcal{D}^{A}$ and $\mathcal{D}^{B}$ together as one big
virtual dataset in a tabular view, this virtual dataset is vertically
partitioned, and each party owns one vertical partition of this dataset. This
is where the term ”vertical federated learning” comes from. Figure 1 shows the
tabular view of the vertically partitioned virtual dataset owned by two
parties, A and B.

Figure 1: View of the virtual dataset in Vertical Federated Learning. Each
party owns a vertical partition of this dataset.

The vanilla VFL is trying to build a federated machine learning model
utilizing only overlapping samples $D_{ol}$, leaving non-overlapping samples
$D^{A}_{nl}$ and $D^{B}_{nl}$ unused. We, therefore, propose a semi-supervised
VFL with Multi-View Training (FedMVT) approach that not only fully utilizes
all available data but also estimates representation of missing features
(shown in figure 1) for not only significantly improving the performance of
the federated model but also helping improve the performance of the local
model of party A.

##  4 The Proposed Approach

Figure 2: Diagram of learning, estimating and selecting representations from
original features in $D$, and then forming an enlarged training set $\chi$.
(a) learning representations from overlapping samples between party A and
party B. (b) learning representations from non-overlapping samples of party B
and estimating representations for corresponding missing features (the stripe
with dashed red line) in party A. (c) learning representations from non-
overlapping samples of party A and estimating representations for
corresponding missing features (the stripe with dashed blue line) in party B.

Deep neural networks have been widely used to learn feature representation
Oquab et al. (2014). For each party, we utilize two neural networks to learn
feature representations from raw input data. One is to learn feature
representations weakly shared between the two parties and the other is to
learn feature representations unique to each party.

More specifically, we denote $r_{u}^{p}=\it{h}_{u}^{p}(x^{p})$ as the _unique_
feature representations and $r_{c}^{p}=\it{h}_{c}^{p}(x^{p})$ as the _shared_
feature representations that are learned from features $x^{p}$ through neural
networks $\it{h}_{u}^{p}$ and $\it{h}_{c}^{p}$ respectively in party
$p\in\\{A,B\\}$, where $r_{u}^{p}\in\mathbb{R}^{N^{p}\times d^{p}}$ and
$r_{c}^{p}\in\mathbb{R}^{N^{p}\times d^{p}}$, and $\it{d^{p}}$ is the
dimension of the top hidden representation layer of neural networks in party
$p$. Then, the complete feature representations for $x^{p}$ is denoted as
$r^{p}=[r_{u}^{p};r_{c}^{p}]$, where $[;]$ is the concatenation operator that
concatenates two matrices along the feature axis.

For convenience, we denote $r^{p}_{u,ol}$ and $r^{p}_{c,ol}$ as the feature
representations learned from overlapping samples $x_{ol}^{p}$ in party
$p\in\\{A,B\\}$ (Figure 2(a)), while $r^{p}_{u,nl}$ and $r^{p}_{c,nl}$ as the
feature representations learned from non-overlapping samples $x_{nl}^{p}$
(Figure 2(b) and 2(c)). Correspondingly, the complete feature representations
for $x_{ol}^{p}$ and $x_{ul}^{p}$ are denoted as
$r_{ol}^{p}=[r_{u,ol}^{p};r_{c,ol}^{p}]$ and
$r_{nl}^{p}=[r_{u,nl}^{p};r_{c,nl}^{p}]$, respectively.

Intuitively, $\it{h}_{c}^{p}$ aims to capture common feature representations
between the two parties while $\it{h}_{u}^{p}$ helps learn domain-specific
representations. We propose the following three loss terms to enforce neural
networks to learn the desired representations.

$L^{AB}_{dist}(r^{A}_{c,ol},r^{B}_{c,ol})=\frac{1}{N_{ol}}\sum_{i}\left\|r^{A,i}_{c,ol}-r^{B,i}_{c,ol}\right\|^{2}_{F}$
(1)
$L^{A}_{orth}(r^{A}_{u},r^{A}_{c})=\frac{1}{N^{A}}\sum_{i}\left\|r^{A,i}_{u}\otimes
r^{A,i}_{c}\right\|^{2}_{F}$ (2)
$L^{B}_{orth}(r^{B}_{u},r^{B}_{c})=\frac{1}{N^{B}}\sum_{i}\left\|r^{B,i}_{u}\otimes
r^{B,i}_{c}\right\|^{2}_{F}$ (3)

where $\otimes$ denotes matrix multiplication operator and
$\left\|\cdot\right\|^{2}_{F}$ is the squared Frobenius norm. By minimizing
(1), neural networks $h_{c}^{A}$ and $h_{c}^{B}$ are pushed to learn common
feature representations from the raw data of the two parties. While (2) and
(3) are orthogonality constraints Bousmalis et al. (2016) that encourage
neural networks $h_{c}^{p}$ and $h_{u}^{p}$ belonging to party $p\in\\{A,B\\}$
to learn distinct feature representations.

Inspired by multi-view learning Clark et al. (2018) to share representations
across models and improve models’ representation learning, we train three
softmax classifiers $f^{A}$, $f^{B}$ and $f^{AB}$ jointly and each takes as
input a feature representation learned from a different view of the virtual
dateset (Figure 1) and outputs estimated class distributions. The loss
functions for the three classifiers are defined as follows:

$L^{A}(r^{A},y^{A})=\frac{1}{N^{A}}\sum_{i}L_{ce}(f^{A}(r^{A,i}),y^{A,i})$ (4)
$L^{B}(r^{B}_{ol},y^{A}_{ol})=\frac{1}{N_{ol}}\sum_{i}L_{ce}(f^{B}(r^{B,i}_{ol}),y^{A,i}_{ol})$
(5)
$L^{fed}([r^{B}_{ol};r^{A}_{ol}],y^{A}_{ol})=\frac{1}{N_{ol}}\sum_{i}L_{ce}(f^{AB}([r^{B,i}_{ol};r^{A,i}_{ol}]),y^{A,i}_{ol})$
(6)

Formulas (4), (5) and (6) compute the cross-entropy between estimated class
distributions and ground-truth labels. Then, we have the objective loss:

$L_{obj}=L^{fed}+L^{A}+L^{B}+\lambda_{1}L^{AB}_{dist}+\lambda_{2}L^{A}_{orth}+\lambda_{3}L^{B}_{orth}$
(7)

where $\lambda s$ are loss weights. Formula (7) forms the backbone of the
FedMVT approach. For now, it only uses existing data without considering
missing features and labels. In sections 4.1 and 4.2, we will elaborate on how
we estimate representations of missing features and determine pseudo-labels
for unlabeled samples. The workflow is overviewed in figure 2.

###  4.1 Feature Representation Estimation

We estimate representations for the missing features in the virtual dataset
(figure 1) by adopting the scaled dot-product attention function Vaswani et
al. (2017). We explain how the estimation is calculated by walking through the
procedure of estimating representations for missing features in party A
(figure 2(b)).

Formula (8) estimates the _shared_ representation, denoted as
$\tilde{r}^{A,i}_{c}$, for party A’s missing feature corresponding to the
$i\text{th}$ sample $x^{B,i}$ of party B by leveraging _shared_ feature
representation $r^{B,i}_{c}$ learned from $x^{B,i}$ and $r^{A}_{c}$ learned
from $x^{A}$.

$\tilde{r}^{A,i}_{c}=g^{A}_{c}(r^{B,i}_{c})=softmax(\frac{r^{B,i}_{c}\otimes
r^{A}_{c}}{\sqrt{d}})\otimes r^{A}_{c}$ (8)

Intuitively, $g^{A}_{c}$ can be interpreted as a search engine such that given
$r^{B,i}_{c}$ by party B, it gives a result of $\tilde{r}^{A,i}_{c}$ in party
A.

Figure 3 illustrates the steps of applying formula (8) to estimate
$\tilde{r}^{A,3}_{c,nl}$ for party A’s missing feature corresponding to the
$3\text{rd}$ non-overlapping feature $x^{B,3}_{nl}$ of party B: first,
original features $x^{B,3}_{nl}$, $x^{A,3}$ and $x^{A,4}$ from both parties
are transformed into corresponding shared representations $r^{B,3}_{c,nl}$,
$r^{A,3}_{c}$ and $r^{A,4}_{c}$ in a common feature representation space
through neural networks $h^{B}_{c}$ and $h^{A}_{c}$. Then, the similarities
between representation $r^{B,3}_{c,nl}$ and the other two (i.e., $r^{A,3}_{c}$
and $r^{A,3}_{c}$) are calculated, denoted as $s_{3}$ and $s_{4}$
respectively. Further, step 3 is applied to calculate
$\tilde{r}^{A,3}_{c,nl}$.

Figure 3: Estimate shared feature representation $\tilde{r}^{A,3}_{c,nl}$ for
party A’s missing feature corresponding to feature $x^{B,3}_{nl}$ of party B.
$x^{A,3}$ and $x^{A,4}$ in $x^{A}$ represent all features in $x^{A}$.

We apply formula (9) to estimate _unique_ representation, denoted as
$\tilde{r}^{A,i}_{u}$, for party A’s missing feature corresponding to the
$i\text{th}$ feature $x^{B,i}$ of party B by leveraging _unique_ feature
representations $r^{B,i}_{u}$ and $r^{B}_{u,ol}$ learned from $x^{B,i}$ and
$x^{B}_{ol}$ respectively in party B, and $r^{A}_{u,ol}$ learned from
$x^{A}_{ol}$ in party A.

$\tilde{r}^{A,i}_{u}=g^{A}_{u}(r^{B,i}_{u})=softmax(\frac{r^{B,i}_{u}\otimes
r^{B}_{u,ol}}{\sqrt{d}})\otimes r^{A}_{u,ol}$ (9)

Figure 4: Estimate unique feature representation $\tilde{r}^{A,3}_{u,nl}$ for
party A’s missing feature corresponding to feature $x^{B,3}_{nl}$ of party B.
$x^{A,1}_{ol}$ and $x^{A,2}_{ol}$ in $x^{A}_{ol}$ represent all features in
$x^{A}_{ol}$.

Figure 4 illustrates the steps of applying formula (9) to estimate
representation $\tilde{r}^{A,3}_{u,nl}$ for party A’s missing feature
corresponding to $x^{B,3}_{nl}$ of party B: first, original features from each
party are projected into a unique feature representation space. Then, the
similarities between representation $r^{B,3}_{u,nl}$ and the other two (i.e.,
$r^{A,3}_{u,ol}$ and $r^{A,3}_{u,ol}$) in space B are calculated, denoted as
$s_{1}$ and $s_{2}$ respectively. Further, $s_{1}$ and $s_{2}$ are transferred
to space A. Specifically, $s_{1}$ is taken as similarity between
$r^{B,3}_{u,nl}$ and $r^{A,1}_{u,ol}$, while $s_{2}$ is taken as similarity
between $r^{B,3}_{u,nl}$ and $r^{A,2}_{u,ol}$. Last, step 4 is applied to
calculate $\tilde{r}^{A,3}_{u,nl}$.

Following the same logic, we can estimate _shared_ representation
$\tilde{r}^{B,j}_{c}=g^{B}_{c}(r^{A,j}_{c})$ and _unique_ representation
$\tilde{r}^{B,j}_{u}=g^{B}_{u}(r^{A,j}_{u})$ for party B’s missing feature
corresponding to the feature $x^{A,j}$ of party A.

Thereby, we obtain representations
$\tilde{r}^{A}_{nl}=[\tilde{r}^{A}_{u,nl}:\tilde{r}^{A}_{c,nl}]$ and
$\tilde{r}^{B}_{nl}=[\tilde{r}^{B}_{u,nl};\tilde{r}^{B}_{c,nl}]$ estimated for
missing features of party A and party B, respectively, and construct the
enlarged training set $\chi$ consisting of three parts:
$\chi^{AB}_{ol}=\\{r^{B,i}_{ol},r^{A,i}_{ol},y^{A,i}_{ol}\\}^{N_{ol}}_{i=1}$;
$\chi^{A\tilde{B}}_{nl}=\\{\tilde{r}^{B,j}_{nl},r^{A,j}_{nl},y^{A,j}_{nl}\\}^{N^{A}_{nl}}_{j=1}$
;
$\chi^{\tilde{A}B}_{nl}=\\{r^{B,z}_{nl};\tilde{r}^{A,z}_{nl}\\}^{N^{B}_{nl}}_{z=1}$,
where $\chi^{AB}_{ol}$ is learned from overlapping samples of A and B;
$\chi^{A\tilde{B}}_{nl}$ is learned from non-overlapping samples of party A
with estimated representations for corresponding missing features in Party
B;$\chi^{\tilde{A}B}_{nl}$ is the other way around. $\chi^{AB}_{ol}$ and
$\chi^{A\tilde{B}}_{nl}$ have ground-truth labels located in party A, while
$\chi^{\tilde{A}B}_{nl}$ has no labels. We will estimate labels for
$\chi^{\tilde{A}B}_{nl}$ in section 4.2.

We add two loss terms to force the estimated feature representations to
approximate the representations learned from original features:
$L^{A}_{dist}(\tilde{r}^{A}_{ol},r^{A}_{ol})=\frac{1}{N_{ol}}\sum_{i}\left\|\tilde{r}^{A,i}_{ol}-r^{A,i}_{ol}\right\|^{2}_{F}$,
$L^{B}_{dist}(\tilde{r}^{B}_{ol},r^{B}_{ol})=\frac{1}{N_{ol}}\sum_{i}\left\|\tilde{r}^{B,i}_{ol}-r^{B,i}_{ol}\right\|^{2}_{F}$,
where $\tilde{r}^{p}_{ol}$ $p\in\\{A,B\\}$ is the representations estimated
for overlapping samples in party $p$ and it should be as close to $r^{p}_{ol}$
as possible.

###  4.2 Pseudo-label Prediction

As discussed in section 4.1,
$\chi^{\tilde{A}B}_{nl}=\\{r^{B,z}_{nl};\tilde{r}^{A,z}_{nl}\\}^{N^{B}_{nl}}_{z=1}$
has no labels. We apply the three trained softmax classifiers $f^{A}$, $f^{B}$
and $f^{AB}$ that take as inputs $\tilde{r}^{A,z}_{nl}$, $r^{B,z}_{nl}$ and
$[r^{B,z}_{nl};\tilde{r}^{A,z}_{nl}]$ respectively to produce three candidate
pseudo-labels for each unlabeled sample in $\chi^{\tilde{A}B}_{nl}$ (algorithm
1, line 6). Only when at least two of the three candidate pseudo-labels are
equal and their probabilities are higher than a predefined threshold $t$, do
we add this pseudo-labeled sample to the training set (algorithm 1, line 10).
With all samples in $\chi$ have labels, $f^{A}$, $f^{B}$ and $f^{AB}$ are
trained on training sets $\chi^{A}_{ol+nl}$, $\chi^{B}_{ol+nl}$ and $\chi$
respectively, where $\chi^{A}_{ol+nl}$ is the combination of $\chi^{A}_{ol}$
and $\chi^{A}_{nl}$ and $\chi^{B}_{ol+nl}$ is the combination of
$\chi^{B}_{ol}$ and $\chi^{B}_{nl}$.

Algorithm 1 gives an overview of the full FedMVT algorithm, in which the
concat function concatenates input matrices along the feature axis and the
combine function concatenates the input matrices along sample axis.

Algorithm 1 FedMVT algorithm

1: Input: Datasets $D^{A}$ and $D^{B}$; Batch index sets $T_{ol}$,
$T^{A}_{nl}$ and $T^{B}_{nl}$ of $D_{ol}$, $D^{A}_{nl}$ and $D^{B}_{nl}$
respectively.Neural networks $h^{B}_{u},h^{B}_{c},h^{A}_{u}$ and $h^{A}_{c}$;
Representation estimators $g^{B}_{u},g^{B}_{c},g^{A}_{u}$ and $g^{A}_{c}$;
Softmax classifiers $f^{A}$, $f^{B}$ and $f^{AB}$ Class probability threshold
$t$ and the epoch number $K$

2: for $e=1,2,...,K$ do

3: Select a mini-batch:
$\\{x^{B,i}_{ol},x^{A,i}_{ol},y^{A,i}_{ol}\\},i\in{T_{ol}}$ from
$D_{ol}$;$\\{x^{A,j}_{nl},y^{A,j}_{nl}\\},j\in{T^{A}_{nl}}$ from
$D^{A}_{nl}$;$\\{x^{B,z}_{nl}\\},z\in{T^{B}_{nl}}$ from $D^{B}_{nl}$

4: Learn $r^{B,i}_{ol},r^{B,z}_{nl},r^{A,i}_{ol},r^{A,j}_{nl}$ through
$h^{B}_{u},h^{B}_{c},h^{A}_{u},h^{A}_{c}$;

5: Estimate $\tilde{r}^{B,j}_{nl},\tilde{r}^{A,z}_{nl}$ through
$g^{B}_{u},g^{B}_{c},g^{A}_{u},g^{A}_{c}$;

6: Estimate pseudo-labels:$\tilde{y}^{z}_{1}=f^{A}(\tilde{r}^{A,z}_{nl})$;
$\tilde{y}^{z}_{2}=f^{B}(r^{B,z}_{nl})$;$\tilde{y}^{z}_{3}=f^{AB}([r^{B,z}_{nl};\tilde{r}^{A,z}_{nl}]);$

7: $\chi^{AB}=concat(r^{B,i}_{ol},r^{A,i}_{ol},y^{A,i}_{ol})$;

8: $\chi^{A\tilde{B}}=concat(\tilde{r}^{B,j}_{nl},r^{A,j}_{nl},y^{A,j}_{nl})$;

9: $\chi^{\tilde{A}B}=concat(r^{B,z}_{nl};\tilde{r}^{A,z}_{nl})$

10:
$\chi^{\tilde{A}B}=select(\chi^{\tilde{A}B},(\tilde{y}^{z}_{1},\tilde{y}^{z}_{2},\tilde{y}^{z}_{3}),t)$;

11: $\chi=combine(\chi^{AB},\chi^{A\tilde{B}},\chi^{\tilde{A}B})$;

12: $f^{A}$, $f^{B}$, $f^{AB}$ based on $\chi^{A}_{ol+nl}$,
$\chi^{B}_{ol+nl}$, $\chi$ for training;

13: Compute loss:
$L_{obj}=L^{fed}+L^{A}+L^{B}+\lambda_{1}L^{A}_{dist}+\lambda_{2}L^{B}_{dist}+\lambda_{3}L^{AB}_{dist}+\lambda_{4}L^{A}_{orth}+\lambda_{5}L^{B}_{orth}$;

14: Compute gradients $\nabla L_{obj}$ and update models.

15: end for

###  4.3 Security Analysis

FedMVT does not require parties to share their original data and model
parameters, but only intermediate representations and gradients. The
intermediate representations are results transformed from original features
through deep neural networks with multiple layers of transformation functions.
Therefore, there is little chance for the other party to reverse-engineer the
original features Gupta and Raskar (2018). Recently, there has been much
discussion on the potential risks associated with privacy leakage through
gradients Bonawitz et al. (2017); Phong et al. (2018). To prevent intermediate
representations and gradients from being exposed, FATE designed and
implemented a VFL-DNN (Deep Neural Network) framework that can efficiently
perform encryption in both the forward stage and backward stage of the VFL
training process. For experimental convenience, we did not implement FedMVT
with FATE. However, FedMVT is compatible with the VFL-DNN framework and can be
migrated to FATE with no much effort.

##  5 Experimental Evaluation

We report experiments conducted on public datasets including: 1) NUS-WIDE
dataset Chua et al. (2009) 2) CIFAR-10 Krizhevsky (2009) to validate our
proposed approach.

NUS-WIDE. The NUS-WIDE dataset consists of 634 low-level image features
extracted from Flickr images and their associated 1000 textual tags as well as
81 ground truth labels. Here, we consider solving a 10-label classification
problem with a data federation formed between party A and party B. Each party
utilizes two local neural networks that each has one hidden layer with 32
units to learn feature representations from their raw inputs. Then, each party
feeds the learned feature representations into its local softmax classifier
$f^{p},p\in\\{A,B\\}$ and the federated softmax layer $f^{AB}$ with 128
(32x2x2) hidden nodes, respectively, for jointly training.

On NUS-WIDE, we run experiments with two scenarios. In the first scenario,
denoted as scenario-1, we put 634 image features on party A and 1000 textual
features on party B. The second scenario, denoted as scenario-2, is the other
way around. In both scenarios, party A owns the labels.

CIFAR-10. We partition each CIFAR-10 image with shape $32\times 32\times 3$
vertically into two parts (each part has shape $32\times 16\times 3$). Each
party uses two local VGG-like CNN models to learn representations from raw
image features, and then it feeds the learned representations into its local
softmax classifier $f^{p},p\in\\{A,B\\}$ and the federated softmax classifier
$f^{AB}$, respectively, for jointly training. The VGG-like CNN model consists
of 2x2 max-pooling layers, 3x3 convolutional layers with stride 1 and padding
1, and fully connected layers. The architecture is conv32-conv32-pool-
conv64-conv64-pool-conv128-conv128-pool-fc64. Thus, the federated softmax
classifier has 256 (64x2x2) nodes.

Figure 5: Test accuracy (%) comparison of FedMVT models to vanilla models on
NUS-WIDE for a varying number of overlapping samples. Exact numbers are
provided in table 1. The number of local data is slightly different for
different size of overlapping samples, and is around $30000\pm 2000$. (a)
shows results when party A holds image data, while (b) shows results when
party A hold text data. scenario-1: Party A with image
---
Model | 250 | 500 | 1000 | 2000 | 4000 | 8000
Vanilla-Local |  58.3 |  58.3 |  58.5 |  58.8 |  58.9 |  58.9
Vanilla-VFL |  29.7 |  49.0 |  63.9 |  68.3 |  70.9 |  71.5
FedMVT-local |  57.2 |  58.7 |  59.0 |  59.6 |  60.1 |  62.1
FedMVT-VFL |  63.8 |  67.3 |  68.4 |  71.7 |  74.4 |  75.0
scenario-2: Party A with text
Model | 250 | 500 | 1000 | 2000 | 4000 | 8000
Vanilla-Local |  69.8 |  69.9 |  70.1 |  70.2 |  70.3 |  70.3
Vanilla-VFL |  29.7 |  49.0 |  63.9 |  68.3 |  70.9 |  71.5
FedMVT-local |  73.2 |  73.3 |  73.6 |  74.0 |  74.4 |  74.9
FedMVT-VFL |  68.7 |  71.2 |  72.7 |  73.9 |  75.9 |  77.4
Table 1: Test accuracy (%) comparison on NUS-WIDE corresponding to figure 5.

FedMVT and Baselines. FedMVT-VFL and FedMVT-local are $f^{AB}$ and $f^{A}$,
respectively, trained with algorithm 1. Vanilla-VFL is $f^{AB}$ trained on
overlapping samples, while Vanilla-local is $f^{A}$ trained on local data of
party A. For each dataset, they use the same local neural networks.

For NUS-WIDE, figure 5 shows the accuracy comparison of FedMVT to vanilla
models with a varying number of overlapping samples from 250 to 8000.

For both scenarios, FedMVT-VFL outperforms vanilla-VFL by a significantly
large margin. For example, with 250 overlapping samples, FedMVT-VFL makes
improvements of 34 and 39 points in accuracy over vanilla-VFL in scenario-1
and scenario-2, respectively. With 8000 overlapping samples, FedMVT-VFL
improves the accuracy by 3.5 and 5.9 points compared with vanilla-VFL,
respectively. FedMVT also enhances the performance of the vanilla-local model
of party A. with 8000 overlapping samples, FedMVT-local outperforms vanilla-
local by 3.2 and 4.6 points in scenario-1 and scenario-2 respectively, which
demonstrates that the local model performs increasingly well with better
feature representation.

As shown in table 1, scenario-2 outperforms scenario-1 across all models,
which demonstrates that textual features have more discriminative power than
image features on the NUS-WIDE. This partially explains why VFL models do not
outperform their corresponding local models (blue v.s. red in figure 5) in
scenario-2 when the size of overlapping samples is not sufficiently large
enough (e.g., $\leq 2000$). While in scenario-1, party A benefits
significantly from applying VFL (blue v.s. red in figure 5). This suggests
that, in practice, when the size of the overlapping samples is limited, the
party owning labels may be better off using its local model if it has features
with strong discriminative power. Otherwise, it may benefit from leveraging
features of the other party.

Figure 6: Test accuracy (%) comparison of FedMVT models to vanilla models on CIFAR10 for a varying number of overlapping samples. Exact numbers are provided in table 2. The number of local data is slightly different for different size of overlapping samples, and is around $25000\pm 2500$. Model | 500 | 1000 | 2000 | 4000 | 6000 | 8000
---|---|---|---|---|---|---
Vanilla-local |  63.4 |  64.4 |  65.3 |  65.8 |  66.2 |  67.2
Vanilla-VFL |  40.0 |  44.2 |  50.4 |  56.5 |  60.7 |  62.9
FedMVT-local |  61.3 |  64.0 |  64.9 |  66.9 |  68.0 |  69.1
FedMVT-VFL |  62.7 |  65.0 |  66.0 |  67.8 |  70.5 |  71.4
Table 2: Test accuracy (%) comparison on CIFAR10 corresponding to figure 6.

The experimental setup on CIFAR10 is the same as the one on NUS-WIDE, except
we do not swap image partitions between the two parties since they are cut
from the same images, and thus we assume they have similar discriminative
power.

For CIFAR10, FedMVT-powered models also outperform vanilla models. For
example, with 8000 overlapping samples (table 2), the FedMVT-VFL and FedMVT-
local improve the accuracy by 8.5 and 1.9 points compared with vanilla-VFL and
vanilla-local, respectively.

Figure 6 shows that the vanilla-VFL cannot beat vanilla-local with 10000
overlapping samples (over 1/3 of total samples). One explanation for this is
that VFL only leverages the compressed information of the top dense layer of
local CNN models of the two parties, thereby losing many useful feature
patterns, for trading minimal information leakage. FedMVT equips VFL with an
effective way of improving models’ representation learning.

##  6 Conclusion

We propose Federated Multi-View Training (FedMVT), a semi-supervised learning
approach that improves the performance of VFL with limited overlapping
samples. FedMVT leverages feature representation estimation and pseudo-labels
prediction to expand the training set and trains three classifiers jointly to
improve models’ representation learning. FedMVT not only significantly
improves the performance of the federated model in VFL, but also enhances the
performance of the local model of the party that owns labels.

## References

  * Berthelot et al. [2019] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  CoRR, abs/1905.02249, 2019.
  * Bonawitz et al. [2017] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth.  Practical secure aggregation for privacy-preserving machine learning.  In CCS, pages 1175–1191, 2017.
  * Bousmalis et al. [2016] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan.  Domain separation networks.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 343–351. Curran Associates, Inc., 2016.
  * Cheng et al. [2019] Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, and Qiang Yang.  Secureboost: A lossless federated learning framework.  CoRR, abs/1901.08755, 2019.
  * Chua et al. [2009] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng.  NUS-WIDE: A real-world web image database from National University of Singapore.  In CIVR, 2009.
  * Clark et al. [2018] Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le.  Semi-supervised sequence modeling with cross-view training.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914–1925, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
  * DLA Piper [2018] DLA Piper.  Data protection laws of the world: Full handbook, 2018.
  * Gascón et al. [2016] Adrià Gascón, Phillipp Schoppmann, Borja Balle, Mariana Raykova, Jack Doerner, Samee Zahur, and David Evans.  Secure linear regression on vertically partitioned datasets.  IACR Cryptology ePrint Archive, page 892, 2016.
  * GDPR [2018] GDPR.  General data protection regulation, 2018.
  * Grandvalet and Bengio [2004] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  In Proceedings of the 17th International Conference on Neural Information Processing Systems, NIPS’04, page 529–536, Cambridge, MA, USA, 2004. MIT Press.
  * Gupta and Raskar [2018] Otkrist Gupta and Ramesh Raskar.  Distributed learning of deep neural network over multiple agents.  CoRR, abs/1810.06060, 2018.
  * Hardy et al. [2017] Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thorne.  Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption.  CoRR, abs/1711.10677, 2017.
  * hyun Lee [2013] Dong hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In ICML Workshop on Challenges in Representation Learning. 2013\.
  * Kairouz et al. [2019] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, and Arjun Nitin Bhagoji.  Advances and open problems in federated learning.  2019\.
  * Krizhevsky [2009] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * Laine and Aila [2017] Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  In ICLR (Poster). OpenReview.net, 2017.
  * Li and Wang [2019] Daliang Li and Junpu Wang.  Fedmd: Heterogenous federated learning via model distillation.  In NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality. 2019.
  * Liu et al. [2018] Yang Liu, Tianjian Chen, and Qiang Yang.  Secure federated transfer learning.  CoRR, abs/1812.03337, 2018.
  * McMahan et al. [2016] H. Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas.  Federated learning of deep networks using model averaging.  CoRR, abs/1602.05629, 2016.
  * Mohassel and Zhang [2017] Payman Mohassel and Yupeng Zhang.  SecureML: A system for scalable privacy-preserving machine learning.  IACR Cryptology ePrint Archive, page 396, 2017.
  * Nock et al. [2018] Richard Nock, Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Giorgio Patrini, Guillaume Smith, and Brian Thorne.  Entity resolution and federated learning get a federated resolution.  CoRR, abs/1803.04035, 2018.
  * Oquab et al. [2014] M. Oquab, L. Bottou, I. Laptev, and J. Sivic.  Learning and transferring mid-level image representations using convolutional neural networks.  In CVPR, 2014.
  * Pan et al. [2010] Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen.  Cross-domain sentiment classification via spectral feature alignment.  In WWW, pages 751–760, 2010.
  * Peterson et al. [2019] Daniel Peterson, Pallika Kanani, and Virendra J. Marathe.  Private federated learning with domain adaptation.  In NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality. 2019.
  * Phong et al. [2018] Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai.  Privacy-preserving deep learning via additively homomorphic encryption.  IEEE Transactions on Information Forensics and Security, 13(5):1333–1345, 2018.
  * Rivest et al. [1978] R L Rivest, L Adleman, and M L Dertouzos.  On data banks and privacy homomorphisms.  Foundations of Secure Computation, pages 169–179, 1978.
  * Tarvainen and Valpola [2017] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 1195–1204. Curran Associates, Inc., 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, and Aidan N Gomez.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc., 2017.
  * Vepakomma et al. [2018] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar.  Split learning for health: Distributed deep learning without sharing raw patient data.  CoRR, abs/1812.00564, 2018.
  * Yang et al. [2019a] Q. Yang, Y. Liu, Y. Cheng, Y. Kang, T. Chen, and H. Yu.  Federated Learning.  Morgan & Claypool, 2019.
  * Yang et al. [2019b] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong.  Federated machine learning: Concept and applications.  ACM Transactions on Intelligent Systems and Technology, 10(2):12:1–12:19, 2019.