# Parity-sensitive inhomogeneous dephasing of macroscopic spin ensembles

Wai-Keong Mok  Institute for Quantum Information and Matter, California
Institute of Technology, Pasadena, CA 91125, USA  Centre for Quantum
Technologies, National University of Singapore, 3 Science Drive 2, Singapore
117543  Steven Touzard  Centre for Quantum Technologies, National University
of Singapore, 3 Science Drive 2, Singapore 117543  Department of Materials
Science and Engineering, National University of Singapore, Singapore
Department of Physics, National University of Singapore, Singapore  MajuLab,
CNRS-UNS-NUS-NTU International Joint Research Unit, Singapore UMI 3654,
Singapore  Leong-Chuan Kwek  Centre for Quantum Technologies, National
University of Singapore, 3 Science Drive 2, Singapore 117543  MajuLab, CNRS-
UNS-NUS-NTU International Joint Research Unit, Singapore UMI 3654, Singapore
National Institute of Education, Nanyang Technological University, Singapore
637616, Singapore  Quantum Science and Engineering Centre (QSec), Nanyang
Technological University, Singapore

###### Abstract

Spin ensembles play a pivotal role in various quantum applications such as
metrology and simulating many-body physics. Recent research has proposed
utilizing spin cat states to encode logical quantum information, with
potentially logical lifetimes on the order of seconds via enhanced collective
interactions that scale with system size. We investigate the dynamics of spin
cat states under inhomogeneous broadening, revealing a phenomenon termed
‘parity-sensitive inhomogeneous dephasing’: odd cat states are significantly
more susceptible to inhomogeneous dephasing compared to even cat states due to
parity symmetry. Additionally, from a mean-field analysis of the driven-
dissipative dynamics, we identify a synchronization phase transition wherein
the ensemble becomes completely dephased beyond a critical inhomogeneous
linewidth. Our findings shed light on the stability of collective spin states,
important for advancing quantum technologies.

##  I Introduction

Coherent manipulation of spin ensembles is crucial for scalable
implementations of many quantum technologies, such as quantum metrology [1, 2,
3], computation [4, 5, 6] and simulation [7, 8]. In particular, spin ensembles
can be engineered to behave collectively as a single high-dimensional quantum
system. This allows for enhanced precision in quantum sensors [9], and is also
very useful in quantum repeater protocols [10, 11]. Collective spin dynamics
has also led to the discovery of interesting physical phenomena such as Dicke
superradiance [12, 13], which remains an active field of study in both
theoretical [14, 15, 16, 17, 18, 19, 20] and experimental [21, 22, 23]
frontiers to harness it for various physical applications.

Recently, it was proposed that collective spin states are potentially useful
for encoding logical quantum information. Motivated by the experimental
success of bosonic cat qubits [24, 25] for quantum error correction in
superconducting circuits, the idea of spin cat qubits is based on encoding the
logical qubit in macroscopic superpositions of spin coherent states [26],
i.e., cat states. The spin cat states are dissipatively stabilized by
engineering collective two-body losses in the spin ensemble [27], analogous to
the protocol developed for bosonic systems. For realistic experimental
parameters, it was estimated in Ref. [27] that the spin cat qubit has a
lifetime on the order of seconds, several orders of magnitude larger than the
state-of-the-art lifetimes for bosonic cat qubits. This substantial
improvement fundamentally stems from the enhanced collective interactions in
the spin ensemble which scales as $\sqrt{N}$, where $N$ is the system size.

In this work, we study the robustness of spin cat states in the presence of
inhomogeneous broadening. This can arise for example from Doppler shifts in
atomic gas clouds or spatial inhomogeneity in the electric or magnetic fields
in solid state systems. Such imperfections break the permutation symmetry of
the spin system, which inhibits the collective behavior. We consider the
quantum driven-dissipative dynamics proposed in Ref. [27] which stabilizes the
spin cat states at long times, described in Sec. II. By analytically solving
for the free evolution of the spin ensemble under the sole effect of the
inhomogeneous broadening in Sec. III, we uncover an effect which we term
parity-sensitive inhomogeneous dephasing. We find that the robustness of the
spin cat states to inhomogeneous broadening depends critically on the parity
symmetry of the state, such that the even cat state is significantly more
robust to inhomogeneous dephasing. This effect persists even when we consider
the full quantum dynamics, suggesting that the odd cat state is indeed fragile
against inhomogeneous dephasing, limiting its usefulness in encoding logical
information.

In Sec. IV, we study the semiclassical mean-field dynamics derived from the
full quantum model. The resulting dynamics can then be physically interpreted
as a competition between spin synchronization and dephasing. We show that the
system undergoes a synchronization phase transition, where the synchronization
in the spin ensemble is completely broken in the long-time limit beyond a
critical inhomogeneous broadening linewidth. This sets a limitation to the
stability of the collective spin states against inhomogeneous dephasing. Our
results provide a physical understanding of the robustness of spin cat states
in realistic environments, which would be important in developing quantum
technologies based on spin ensembles. We provide an outlook in Sec. V.

##  II Driven-dissipative spin model

We consider a system of $N$ two-level systems (TLS), i.e., pseudospin-$1/2$
particles, which we henceforth refer to as spins. The system is described by
the Lindblad master equation

$\dot{\rho}=-i[H,\rho]+\frac{\Gamma_{2}}{N^{2}}\mathcal{D}\left[\sum_{i,j=1}^{N}\sigma_{i}^{-}\sigma_{j}^{-}\right]\rho$
(1)

with the Hamiltonian in the rotating frame (setting $\hbar=1$)

$H=\frac{1}{2}\sum_{i}\delta_{i}\sigma_{i}^{z}+\frac{\eta}{N}\left(\sum_{i,j}e^{i\varphi}\sigma_{i}^{-}\sigma_{j}^{-}+e^{-i\varphi}\sigma_{i}^{+}\sigma_{j}^{+}\right).$
(2)

The parameters $\eta,\varphi$ and $\Gamma_{2}$ describe the squeezing
strength, squeezing phase and nonlinear two-excitation loss rate respectively.
$\sigma_{i}^{-}=\ket{g_{i}}\bra{e_{i}}$ is the lowering operator for the
$i$-th particle from the excited state $\ket{e_{i}}$ to the ground state
$\ket{g_{i}}$, and $\sigma_{i}^{+}=\ket{e_{i}}\bra{g_{i}}$ is the
corresponding raising operator.
$\sigma_{i}^{z}=\ket{e_{i}}\bra{e_{i}}-\ket{g_{i}}\bra{g_{i}}$ Each spin has a
detuning of $\delta_{i}$ compared to the ensemble mean frequency, which models
the inhomogeneous broadening in the system. The dissipator
$\mathcal{D}[A]\rho\equiv A\rho A^{\dagger}-\\{A^{\dagger}A,\rho\\}/2$ governs
the dissipative interactions between the particles, with the jump operator
$A$. Eq. (1) can be physically realized by collective two-photon coupling
between the spins and a bosonic mode, such as an optical cavity, which is
strongly dissipative. The bosonic mode can then be adiabatically eliminated,
resulting in effective two-body dissipative interactions between the spins
[27].

In the absence of inhomogeneous broadening (i.e., $\delta_{i}=0$) and the
regime $\eta\ll\Gamma_{2}$, the system is weakly excited and behaves as the
bosonic model studied in [28]. This can be seen by performing the Holstein-
Primakoff transformation $S_{-}=(N-a^{\dagger}a)^{1/2}a$, where
$S_{\pm}=\sum_{i}\sigma_{i}^{\pm}$ are the collective spin lowering and
raising operators, while $a,a^{\dagger}$ are the bosonic annihilation and
creation operators respectively. The spins behave collectively when
$\delta_{i}=0$ due to permutation symmetry. In the weak excitation regime
$\braket{a^{\dagger}a}\ll N$, we have approximately $S_{-}\approx\sqrt{N}a$
and $S_{+}\approx\sqrt{N}a^{\dagger}$. The model can be approximately
described (in the rotating frame) by

$\dot{\rho}\approx-i[H_{\text{bosonic}},\rho]+\Gamma_{2}\mathcal{D}[a^{2}]\rho$
(3)

where

$H_{\text{bosonic}}\approx\eta(e^{i\varphi}a^{2}+e^{-i\varphi}a^{{\dagger}2}).$
(4)

The steady state of this bosonic model is spanned by the coherent states
$\ket{\pm\alpha}$ with complex amplitude
$\alpha=\sqrt{2\eta/\Gamma_{2}}\exp{(-i(\varphi/2+\pi/4))}$ [28, 24]. By
forming even and odd superpositions of $\ket{\pm\alpha}$, one obtains
stabilized cat states which can be used to encode a logical qubit, where the
errors can be corrected autonomously. These cat states are robust against
single-photon loss, which is the dominant noise source in superconducting
cavities. By mapping single-photon loss to either a logical bit or phase flip,
this scheme generates a biased noise qubit, with the noise bias increasing
with the amplitude $|\alpha|$ of the cat state [25].

In Ref. [27], it was proposed to use collective spin systems in the weak
excitation limit for a similar logical encoding, where the steady states are
now spin coherent states [26] which can be superposed to obtain spin cat
states analogous to the bosonic case. The main motivation behind this is to
leverage the collective enhancement of the coherent spin interactions, such
that the amplitude of the spin cat states scale as $\sim\sqrt{N}$ which
translates to stronger protection against noise. It was argued that the spin
cat states are robust to inhomogeneous broadening even though the permutation
symmetry is broken. We now show that this depends heavily on the parity
symmetry of the cat state.

(a)  Figure 1: Fidelity $F(t)$ for spin coherent, even cat and odd cat states
under the free evolution of the dephasing Hamiltonian $H_{0}$. The light solid
lines depict the random realizations of the detunings (10 realizations shown),
while the dashed lines correspond to the analytical result for the mean
fidelity $\overline{F(t)}$. The parameters are $N=200$,
$\theta=1/\sqrt{N}\approx 0.0707,\phi=0$.

##  III Parity-sensitive inhomogeneous dephasing

The spin coherent states are defined as

$\ket{\theta,\phi}=\bigotimes_{n=1}^{N}\left(\cos\frac{\theta}{2}\ket{g_{n}}+e^{i\phi}\sin\frac{\theta}{2}\ket{e_{n}}\right).$
(5)

Let us first consider the dynamics of the spin ensemble under the free
evolution governed by the dephasing (inhomogeneous broadening) Hamiltonian
$H_{0}=\sum_{n}\delta_{n}\sigma_{n}^{z}$. For concreteness, the detuning
frequencies $\delta_{n}\sim\mathcal{N}(0,\delta^{2})$ are independently and
identically distributed random variables drawn from a Gaussian distribution
with zero mean and variance $\delta^{2}$. The overlap between the evolved
state and the initial spin coherent state is given by

$\begin{split}c(t)&\equiv\braket{\theta,\phi}{e^{-iH_{0}t}}{\theta,\phi}\\\
&=\prod_{n=1}^{N}e^{i\delta_{n}t/2}\left(\cos\frac{\theta}{2}+e^{-i\delta_{n}t}\sin\frac{\theta}{2}\right).\end{split}$
(6)

The fidelity $F(t)=|c(t)|^{2}$ indicates the survival probability of the
initial state at time $t$. Using the Gaussian average
$\mathbb{E}[\cos\delta_{n}t]=e^{-\delta^{2}t^{2}/2}$, we get the average
fidelity

$\overline{F(t)}=\left[1-\frac{1}{2}(1-e^{-\delta^{2}t^{2}/2})\sin^{2}\theta\right]^{N},$
(7)

where the average is taken over random realizations of the frequencies
$\delta_{n}$. In the weak excitation regime, $\theta$ is related to the
amplitude of the bosonic coherent state via
$|\alpha|\approx\sqrt{N}\tan(\theta/2)$ [27]. Hence, we make the small
amplitude expansion $N\theta^{2}\ll 1$ to get

$\overline{F(t)}=1-\frac{N\theta^{2}}{2}(1-e^{-\delta^{2}t^{2}/2})+O(N^{2}\theta^{4})$
(8)

which implies $\overline{F(\infty)}\approx 1-(N\theta^{2})/2$ is of order
unity since $N\theta^{2}\ll 1$. Using
$\mathbb{E}[\cos^{2}\delta_{n}t]=(1+e^{-2\delta^{2}t^{2}})/2$, we can also
evaluate the variance of the fidelity,

$\text{Var}[F(t)]=\frac{N\theta^{4}}{8}(1-e^{-\delta^{2}t^{2}})+O(\theta^{6}).$
(9)

Thus, the fluctuations of $F(t)$ are $\sim\sqrt{N}\theta^{2}$, which is much
smaller than $\overline{F(t)}$. This implies that for sufficiently large
systems, the random variable $F(t)$ concentrates around the mean
$\overline{F(t)}$.

The spin cat states are defined as

$\ket{\text{Cat}_{\pm}}=\frac{1}{\mathcal{N}_{\pm}}(\ket{\theta,\phi}\pm\ket{\theta,\phi+\pi}),$
(10)

with the normalization factors

$\mathcal{N}_{\pm}=\sqrt{2(1\pm\cos^{N}\theta)}.$ (11)

As with the spin coherent states, we compute the fidelity of the cat states
under the dephasing dynamics

$F_{\pm}(t)\equiv|\braket{\text{Cat}_{\pm}}{e^{-iH_{0}t}}{\text{Cat}_{\pm}}|^{2}.$
(12)

Using the Gaussian average $\mathbb{E}[\sin\delta_{n}t]=0$ from symmetry, and
after some algebra, we obtain

$\begin{split}\overline{F_{\pm}(t)}&=\frac{4}{\mathcal{N}_{\pm}^{4}}\bigg{[}\bigg{(}1-\frac{1}{2}(1-e^{-\delta^{2}t^{2}/2})\sin^{2}\theta\bigg{)}^{N}\\\
&+\bigg{(}1-\frac{1}{2}(1+e^{-\delta^{2}t^{2}/2})\sin^{2}\theta\bigg{)}^{N}\pm
2\cos^{N}\theta\bigg{]}.\end{split}$ (13)

In the small amplitude regime $N\theta^{2}\ll 1$ we obtain qualitatively
different results for the even cat state

$\overline{F_{+}(t)}=1-\frac{N^{2}\theta^{4}}{16}(1-e^{-\delta^{2}t^{2}})+O(N^{3}\theta^{6}),$
(14)

and the odd cat state

$\overline{F_{-}(t)}=e^{-\delta^{2}t^{2}}+O(N^{2}\theta^{4}).$ (15)

The above expressions are simplified using $N\gg 1$. Note that the leading
order term in Eq. (15) should be interpreted as the limiting case of
$N\theta^{2}\to 0^{+}$, since $\theta=0$ is not well defined for the odd cat
state. Similar to Eq. (9), one can also show that the relative fluctuations
for the fidelity vanish as $N\to\infty$.

This reveals an important fact that the robustness of the cat state against
inhomogeneous broadening is very sensitive to its parity symmetry. For the
even cat state, the fidelity saturates at $F_{+}\approx 1-(N\theta^{2}/4)^{2}$
which is of order unity, while for the odd cat state the fidelity vanishes
rapidly on the timescale of $\delta^{-1}$. Moreover, the even cat state is
more robust than the spin coherent state, since the infidelity
$1-\overline{F_{+}(t)}$ is quadratic in $N\theta^{2}$ (as compared to linear
for the spin coherent state). Physically, this means that the dephasing
effects are suppressed by even symmetry and amplified by odd symmetry.

###  III.1 Adding the stabilization dynamics

(a)  Figure 2: Mean fidelity $\overline{F(t)}$ under the full dissipative
dynamics (1). The $N=8$ spins are initialized in the even (blue circles) and
odd (red squares) cat states $\ket{\text{Cat}_{\pm}}$ with
$\eta/\Gamma_{2}=0.2$, subject to inhomogeneous broadening with strengths
$\delta/\Gamma_{2}=\\{10^{-3},10^{-2}\\}$. The fidelities are computed with
respect to the initial state and averaged over $10$ realizations of the
detunings. In the absence of inhomogeneous broadening, the fidelities of the
steady states are $0.998$ (even) and $0.995$ (odd).

So far, we have only considered the free evolution governed by the dephasing
Hamiltonian $H_{0}$. We now study numerically the full dissipative evolution
governed by the master equation (1), using the QuTiP package [29]. Fig. 2
shows the mean fidelities $\overline{F(t)}$ for the cases of even and odd cat
states, for $N=8$. In each case, we initialize the system in the even/odd cat
state and measure the fidelity with respect to the initial state. In the
absence of inhomogeneous broadening, the steady states achieve fidelities of
$0.998$ and $0.995$ for the even and odd cat states respectively. We see that
the odd cat state is significantly more susceptible to the dephasing effects.
For example, at time $\Gamma_{2}t\approx 200$ with
$\delta/\Gamma_{2}=10^{-2}$, the even cat state has a fidelity around $0.74$
while the fidelity of the odd cat state is only around $0.1$. This shows that
the parity-sensitive dephasing derived for the simple case of only free
Hamiltonian evolution applies even with the driven-dissipative stabilization
terms. In Ref. [27], it was shown numerically that the even cat state
$\ket{\text{Cat}_{+}}$ is robust against inhomogeneous broadening (and other
imperfections). This is consistent with our findings. We argue here that the
same robustness does not apply to the odd cat state, even at larger system
sizes beyond the reach of numerical simulations. For larger $N$, one expects
that the effects of inhomogeneous broadening are suppressed due to stronger
collective interactions, but the separation of timescales in the fidelity
decay between the even and odd cat state should remain valid.

(a)  Figure 3: (Top) Steady state amplitude $|\braket{a}|$ against
$\eta/\Gamma_{2}$ for the quantum model with $N=100$. The initial state is the
bosonic coherent state with complex amplitude
$\alpha=\sqrt{2\eta/\Gamma_{2}}e^{-i\pi/4}$. The dashed line corresponds to
the mean field solution for $\sqrt{N}A$ in Eq. (19). (Bottom) Steady state
Wigner functions for $\eta/\Gamma_{2}\in\\{5,12\\}$.

##  IV Mean-field analysis: Synchronization phase transition

Since a direct analytical treatment of Eq. (1) is not feasible, we consider
instead a mean-field approximation where we assume a product state ansatz for
$\rho$. The spin coherent state in Eq. (5) is a product state, so we expect
this to be a reasonable approximation for large $N$. The mean-field equations
can be physically interpreted as describing the synchronization dynamics of
$N$ globally coupled classical spins, containing both dissipative and coherent
(also called reactive) couplings. As we will show, this system exhibits a
synchronization transition with both $\eta/\Gamma_{2}$ and
$\delta/\Gamma_{2}$, where the spins become desynchronized beyond a critical
parameter value. This provides important insights on the robustness of the
spin coherent states to inhomogeneous broadening with driven-dissipative
stabilization, for large system sizes.

The mean-field equations governing the dynamics of the $m$-th spin read (see
Appendix A for a detailed derivation):

$\begin{split}\frac{d}{dt}\braket{\sigma_{m}^{+}}&=i\delta_{m}\braket{\sigma_{m}^{+}}-2i\eta
e^{i\theta}\braket{\sigma_{m}^{z}}c_{1,m}^{*}\\\
&-2\Gamma_{2}\braket{\sigma_{m}^{+}}\left(\frac{c_{2,m}}{N}+|c_{1,m}|^{2}\right)\\\
&+\Gamma_{2}\braket{\sigma_{m}^{z}}c_{1,m}(2c_{2,m}+N|c_{1,m}|^{2})\end{split}$
(16)

and

$\begin{split}\frac{d}{dt}\braket{\sigma_{m}^{z}}&=8\eta\text{Im}\left(e^{-i\theta}\braket{\sigma_{m}^{+}}c_{1,m}\right)\\\
&-4\Gamma_{2}(1+\braket{\sigma_{m}^{z}})\left(\frac{c_{2,m}}{N}+|c_{1,m}|^{2}\right)\\\
&-4\Gamma_{2}\text{Re}[\braket{\sigma_{m}^{+}}c_{1,m}^{*}(2c_{2}+N|c_{1,m}|^{2})]\end{split}$
(17)

where $c_{1,m}\equiv\frac{1}{N}\sum_{j\neq m}\braket{\sigma_{j}^{+}}$ is the
average coherence and $c_{2,m}\equiv\frac{1}{2N}\sum_{j\neq
m}(1+\braket{\sigma_{j}^{z}})$ is the spin excitation density, excluding the
$m$-th spin. Re and Im denote the real and imaginary parts respectively. The
squeezing phase $\varphi$ is set to zero without loss of generality.

(a)  Figure 4: Steady state value of the phase spread $\zeta$ against the
dimensionless detuning $\tilde{\delta}=N^{2}\delta/\Gamma_{2}$ and
dimensionless squeezing $\tilde{\eta}=\eta/\Gamma_{2}$, for $N=10^{4}$ spins.
The white region denote data points where the spins fail to converge to a
synchronized state at long times. The red line marks the elliptical boundary
of the synchronization region given by Eq. (22), with fitting parameters
$a\approx 0.03125$ and $b\approx 2.2077$. For small values of $\tilde{\eta}$,
some points within the synchronization region fail to converge to the steady
state within the simulation time due to the slow dynamics.

###  IV.1 Identical frequencies

First, we consider the case where all the spins have the same frequency
($\delta=0$). This recovers the permutation symmetry in the system, which
allows us to compare the mean-field solution to the exact quantum dynamics.
From symmetry arguments, we denote
$\braket{\sigma_{j}^{+}}=\braket{\sigma^{+}}=A\exp(i\phi)$ where $A$ and
$\phi$ are to be determined, and
$\braket{\sigma_{j}^{z}}=\braket{\sigma^{z}}=z$. Then, $c_{1,m}=(N-1)/N\times
A\exp(i\phi)$ and $c_{2,m}=(N-1)/2N\times(1+z)$. This reduces the problem
drastically from $3N$ real variables to just $3$ real variables $A$, $\phi$
and $z$. For large $N$, the reduced equations of motion become

$\begin{split}\frac{\dot{A}}{A}&=-2\eta z\sin
2\phi-2\Gamma_{2}A^{2}+N\Gamma_{2}zA^{2}\\\ \dot{\phi}&=-2\eta zA\cos 2\phi\\\
\frac{\dot{z}}{A^{2}}&=8\eta\sin
2\phi-4\Gamma_{2}(1+z)-4\Gamma_{2}NA^{2}.\end{split}$ (18)

In the low excitation limit, the steady state of the master equation (1) is
approximately the bosonic coherent state with
$|\braket{a}|=\sqrt{N}A=\sqrt{2\eta/\Gamma_{2}},z\approx-1,\phi=\pi/4$. This
agrees with the steady state solution of Eq. (18):

$\begin{split}A^{2}&\approx\frac{4\eta-1+\sqrt{1-\frac{16\eta}{N\Gamma_{2}}}}{2N}\\\
\phi&=\pi/4\\\
z&\approx-\frac{1+\sqrt{1-\frac{16\eta}{N\Gamma_{2}}}}{2}\end{split}$ (19)

which is valid in the range $0\leq\eta/(N\Gamma_{2})\leq 1/16$. To leading
order in $\eta/{N\Gamma_{2}}$, we recover $A^{2}=2\eta/N\Gamma_{2}$ and $z=-1$
which matches the quantum results in the low excitation limit. The solution in
Eq. (19) describes the synchronized state where the spins are all phase locked
to one another. The range of validity of this solution can be interpreted as
the synchronization region, where the synchronization is broken at
$\eta/(N\Gamma_{2})=1/16$. This sharp behavior in $\eta/\Gamma$ can also be
understood for the quantum system. Making the exact Holstein-Primakoff
transformation as introduced in Sec. II, we plot the steady state bosonic
amplitude $|\braket{a}|$ in Fig. 3 for $N=100$ spins, initializing in the
coherent state with complex amplitude
$\alpha=\sqrt{2\eta/\Gamma_{2}}e^{-i\pi/4}$. For small $\eta/\Gamma_{2}$, the
mean-field approximation agrees well with the quantum model. Interestingly,
the bosonic amplitude increases and drops sharply at $\eta/\Gamma_{2}\approx
10$, although the drop occurs at a larger value of $\eta/\Gamma_{2}$ compared
to the mean field model ($\eta/\Gamma=6.25$). In the quantum case, the drop in
amplitude has a different physical interpretation: the bosonic Hilbert space
has a dimension of $N+1$. Due to the boundedness of the quantum phase space,
the bosonic amplitude cannot increase indefinitely with $\eta/\Gamma_{2}$ as
$\sqrt{2\eta/\Gamma_{2}}$. Thus, for sufficiently large amplitudes, a
secondary blob emerges in the Wigner function at a phase difference of $\pi$
and causing $|\braket{a}|$ to collapse. As a simple estimate, this collapse
occurs when $\braket{a^{\dagger}a}\propto N$ such that the boundaries of the
phase space become relevant. Since $\braket{a^{\dagger}a}\sim\eta/\Gamma_{2}$,
the amplitude collapse occurs when $\eta/\Gamma_{2}\propto N$, the same
scaling as the mean-field prediction (even though the exact values differ).

###  IV.2 Oppositely detuned sub-ensembles

Now, we break permutation symmetry by adding detunings to the spins. For
analytical tractability, we consider two sub-ensembles, each with $N/2$ spins
(assuming $N$ is even), with detunings $\pm\delta$. The results derived for
this model should hold qualitatively for the more realistic model of random
detunings (e.g., Gaussian distributed) which describes inhomogeneous
broadening. Assuming that $\braket{\sigma^{+}}=A\exp[i(\pi/4\pm\zeta)]$ for
the two sub-ensembles respectively, where $\zeta$ is the deviation of the
phase from $\pi/4$, and $\braket{\sigma_{j}^{z}}=z$ for all spins, we can
reduce the problem once again to just 3 nonlinear coupled equations

$\begin{split}\frac{\dot{A}}{A}&=-2\eta z\left(\cos^{2}\zeta-\frac{1}{N}\cos
2\zeta\right)\\\
&+\Gamma_{2}z\left(\cos^{2}\zeta-\frac{1}{N}\right)\left[1+z+NA^{2}\left(1-\frac{2}{N}\right)\cos^{2}\zeta\right]\\\
\dot{\zeta}&=\delta+\eta z\left(1-\frac{2}{N}\right)\sin 2\zeta\\\
&-\frac{\Gamma_{2}}{2}z\left[1+z+NA^{2}\left(1-\frac{2}{N}\right)\cos^{2}\zeta\right]\sin
2\zeta\\\ \frac{\dot{z}}{A^{2}}&=8\eta\left(\cos^{2}\zeta-\frac{1}{N}\cos
2\zeta\right)\\\
&-4\Gamma_{2}\left(\cos^{2}\zeta-\frac{1}{N}\right)\left[1+z+NA^{2}\left(1-\frac{2}{N}\right)\cos^{2}\zeta\right]\\\
&-4\Gamma_{2}(1+z)\cos^{2}\zeta\end{split}$ (20)

To verify the accuracy of Eq. (20), we compare numerical simulations of Eq.
(20) with the full mean-field equations (16) and (17). It is also simple to
see that this is consistent with Eq. (19) by setting
$\delta=0,\phi\approx\pi/4,z\approx-1$ and taking the large $N$ limit.

Compared to the previous case of identical frequencies, this set of equations
is much harder to solve analytically. For small $\zeta$, we can get an
approximate solution by expanding Eq. (20) to first order in $\zeta$ and also
first order in $1/N$, the approximate steady state solution corresponding to
the synchronized state is (see Appendix B)

$\begin{split}A^{2}&\approx\frac{4\eta-1+\sqrt{1-\frac{16\eta}{N\Gamma_{2}}}}{2N}\\\
\zeta&\approx\frac{1+\sqrt{1-\frac{16\eta}{N\Gamma_{2}}}}{32\eta^{2}}N^{2}\Gamma_{2}\delta\\\
z&\approx-\frac{1+\sqrt{1-\frac{16\eta}{N\Gamma_{2}}}}{2}\end{split}$ (21)

This solution is valid for small $\delta$ (where the dephasing of the ensemble
is slow) or large $\eta$ (where the synchronization effect is strong). For
$\eta/N\Gamma_{2}\ll 1$, the formula for phase spread simplifies to
$\zeta\approx N^{2}\Gamma_{2}\delta/16\eta^{2}$. Interestingly, while the
phase spread is linear in $\delta$, the steady state values for $A^{2}$ and
$z$ are independent of $\delta$ up to linear order.

For larger values of $\delta$, we perform numerical simulations of the mean-
field equations for $N=10^{4}$ spins, shown in Fig. 4. The steady state value
of the phase spread $\zeta$ is small for small $\delta$ and large $\eta$,
which agrees with our physical intuition. For the case of identical
frequencies $\delta=0$, there is a maximum $\eta/\Gamma_{2}$ of $N/16=625$
beyond which the synchronization breaks, agreeing with our analytical
calculations. As $\delta$ increases, the region of synchronization narrows
until a threshold value such that no synchronization is possible for any
$\eta$. The boundary of the synchronization region appears to be elliptical.
We postulate that the boundary curve takes the form

$\tilde{\delta}=b\sqrt{a^{2}-(\tilde{\eta}-a)^{2}}$ (22)

where $\tilde{\delta}\equiv N^{2}\delta/\Gamma_{2}$ and
$\tilde{\eta}\equiv\eta/\Gamma_{2}$ are the dimensionless frequency detuning
and squeezing respectively, and $a,b$ are the fitting parameters. The
threshold values are thus $\tilde{\eta}_{c}=2a$ and $\tilde{\delta}_{c}=ab$.
From numerical fitting, we find that $a\approx 0.03125$ and $b\approx 2.2077$
which gives $\tilde{\eta}_{c}\approx 0.0625N$ and $\tilde{\delta}_{c}\approx
0.06888N^{2}$. The value of $\tilde{\eta}_{c}$ agrees excellently with the
theoretical prediction of $N/16$.

Since $\tilde{\delta}_{c}\propto N^{2}$, the critical detuning
$\delta_{c}/\Gamma_{2}\approx 0.06888$ is independent of $N$. This suggests
that the robustness of the spin coherent state to inhomogeneous broadening is
independent of $N$, for a fixed $\eta/\Gamma_{2}$. Since the odd cat state is
more fragile than the spin coherent state by the parity-sensitive dephasing
effect, this supports the lack of robustness for the odd cat state, consistent
with the numerical results in Fig. 2 for the quantum model.

##  V Discussion and Outlook

In this work, we analyze the effects of symmetry-breaking inhomogeneous
dephasing on collective spin states such as spin coherent states and cat
states $|\text{Cat}_{\pm}\rangle$ formed from the superposition of two spin
coherent states offset by a $\pi$-phase. We show that the spin dephasing is
highly sensitive to the parity symmetry of the collective spin state: For
$|\text{Cat}_{+}\rangle$, even symmetry provides enhanced protection against
inhomogeneous dephasing which grows with the system size $N$. On the contrary,
the odd symmetry in $|\text{Cat}_{-}\rangle$ renders it significantly more
fragile to inhomogeneous dephasing, causing the fidelity to decay rapidly.

This has important implications in using macroscopic spin ensembles as ‘cat
qubits’ to encode logical quantum information, motivated by recent successful
experiments in superconducting quantum circuits. Due to the collectively
enhanced interactions in spin ensembles scaling as $\sqrt{N}$, they stand to
benefit from greater error-correcting capabilities compared to their bosonic
counterpart. However, using spin ensembles also come with a different set of
experimental challenges like particle loss which has to be carefully
addressed. In certain implementations such as rare-earth ions, inhomogeneous
broadening arising from spatially-varying magnetic fields break permutation
symmetry which is required for the spins to behave collectively, and causes
spin dephasing. In Ref. [27], it was demonstrated explicitly that
$|\text{Cat}_{+}\rangle$ is robust against such inhomogeneous dephasing, which
is consistent with our results. However, our results also suggest that the odd
cat state $|\text{Cat}_{-}\rangle$ is significantly more fragile. This means
that the lifetime of $|\text{Cat}_{-}\rangle$ is limited by the dephasing
timescale from the inhomogeneous broadening, which is on the order of $1$
$\mu\text{s}$ (taking $\delta\sim 1$ MHz). Consequently, encoding the logical
qubit using $|\text{Cat}_{\pm}\rangle$ is unlikely to result in logical
lifetimes significantly longer than the break-even point.

An alternative logical encoding is to define the basis states
$\ket{0}_{L}\propto\ket{\theta,0}+\ket{\theta,\pi}$ and
$\ket{1}_{L}\propto\ket{\theta,\pi/2}+\ket{\theta,3\pi/2}$, where
$\ket{\theta,\phi}$ are the spin coherent states (5). Here, the logical qubit
is encoded in the even parity subspace which enjoys the enhanced protection
against inhomogeneous broadening. This encoding is analogous to the ‘four-
legged cat code’ proposed in bosonic quantum error correction [30]. To
stabilize this as the steady state, one approach would be to replace the
collective two-body terms in Eq. (1) with collective four-body terms. However,
it is very challenging in practice to implement quartic dissipators of the
form $\mathcal{D}[S_{-}^{4}]\rho$, where $S_{-}$ is the collective spin
lowering operator, while suppressing all unwanted dissipative terms.

Apart from quantum error correction, our results are also relevant for other
applications of collective spin systems such as metrology and quantum
simulation of dissipative phase transitions. As a future work, it would be
interesting to study the parity-sensitive dephasing effect in more general
dynamics beyond the specific master equation studied here, and also for
general spin states with well-defined parity symmetry. One can also go beyond
the product state ansatz and include quantum correlations in the numerical
simulations using higher-order mean-field methods such as cumulant expansion
[31, 32], which has been widely employed to study many-body spin dynamics [33,
34, 35, 36, 37, 38].

## Acknowledgments

L.C.K. acknowledges support from the Ministry of Education, Singapore and the
National Research Foundation, Singapore. S.T acknowledges support from the
National Research Foundation, Singapore. The Institute for Quantum Information
and Matter is an NSF Physics Frontiers Center.

## References

  * Pezzè  _et al._ [2018] L. Pezzè, A. Smerzi, M. K. Oberthaler, R. Schmied, and P. Treutlein, Quantum metrology with nonclassical states of atomic ensembles, Rev. Mod. Phys. 90, 035005 (2018).
  * Zhou  _et al._ [2020] H. Zhou, J. Choi, S. Choi, R. Landig, A. M. Douglas, J. Isoya, F. Jelezko, S. Onoda, H. Sumiya, P. Cappellaro, H. S. Knowles, H. Park, and M. D. Lukin, Quantum metrology with strongly interacting spin systems, Phys. Rev. X 10, 031003 (2020).
  * Arunkumar  _et al._ [2023] N. Arunkumar, K. S. Olsson, J. T. Oon, C. A. Hart, D. B. Bucher, D. R. Glenn, M. D. Lukin, H. Park, D. Ham, and R. L. Walsworth, Quantum logic enhanced sensing in solid-state spin ensembles, Phys. Rev. Lett. 131, 100801 (2023).
  * Wesenberg  _et al._ [2009] J. H. Wesenberg, A. Ardavan, G. A. D. Briggs, J. J. L. Morton, R. J. Schoelkopf, D. I. Schuster, and K. Mølmer, Quantum computing with an electron spin ensemble, Phys. Rev. Lett. 103, 070502 (2009).
  * Brion  _et al._ [2007] E. Brion, K. Mølmer, and M. Saffman, Quantum computing with collective ensembles of multilevel systems, Phys. Rev. Lett. 99, 260501 (2007).
  * Barrett  _et al._ [2010] S. D. Barrett, P. P. Rohde, and T. M. Stace, Scalable quantum computing with atomic ensembles, New J. Phys. 12, 093032 (2010).
  * Georgescu  _et al._ [2014] I. M. Georgescu, S. Ashhab, and F. Nori, Quantum simulation, Rev. Mod. Phys. 86, 153 (2014).
  * Islam  _et al._ [2011] R. Islam, E. E. Edwards, K. Kim, S. Korenblit, C. Noh, H. Carmichael, G.-D. Lin, L.-M. Duan, C.-C. Joseph Wang, J. K. Freericks, and C. Monroe, Onset of a quantum phase transition with a trapped ion quantum simulator, Nat. Commun. 2, 377 (2011).
  * Koppenhöfer  _et al._ [2022] M. Koppenhöfer, P. Groszkowski, H.-K. Lau, and A. Clerk, Dissipative superradiant spin amplifier for enhanced quantum sensing, PRX Quantum 3, 030330 (2022).
  * Duan  _et al._ [2001] L.-M. Duan, M. D. Lukin, J. I. Cirac, and P. Zoller, Long-distance quantum communication with atomic ensembles and linear optics, Nature 414, 413 (2001).
  * Sangouard  _et al._ [2011] N. Sangouard, C. Simon, H. de Riedmatten, and N. Gisin, Quantum repeaters based on atomic ensembles and linear optics, Rev. Mod. Phys. 83, 33 (2011).
  * Dicke [1954] R. H. Dicke, Coherence in spontaneous radiation processes, Phys. Rev. 93, 99 (1954).
  * Gross and Haroche [1982] M. Gross and S. Haroche, Superradiance: An essay on the theory of collective spontaneous emission, Phys. Rep. 93, 301 (1982).
  * Masson  _et al._ [2020] S. J. Masson, I. Ferrier-Barbut, L. A. Orozco, A. Browaeys, and A. Asenjo-Garcia, Many-body signatures of collective decay in atomic chains, Phys. Rev. Lett. 125, 263601 (2020).
  * Masson and Asenjo-Garcia [2022] S. J. Masson and A. Asenjo-Garcia, Universality of dicke superradiance in arrays of quantum emitters, Nat. Commun. 13, 2285 (2022).
  * Sierra  _et al._ [2022] E. Sierra, S. J. Masson, and A. Asenjo-Garcia, Dicke superradiance in ordered lattices: Dimensionality matters, Phys. Rev. Research 4, 023207 (2022).
  * Robicheaux [2021] F. Robicheaux, Theoretical study of early-time superradiance for atom clouds and arrays, Phys. Rev. A 104, 063706 (2021).
  * Malz  _et al._ [2022] D. Malz, R. Trivedi, and J. I. Cirac, Large-$n$ limit of dicke superradiance, Phys. Rev. A 106, 013716 (2022).
  * Cardenas-Lopez  _et al._ [2023] S. Cardenas-Lopez, S. J. Masson, Z. Zager, and A. Asenjo-Garcia, Many-body superradiance and dynamical mirror symmetry breaking in waveguide qed, Phys. Rev. Lett. 131, 033605 (2023).
  * Mok  _et al._ [2023] W.-K. Mok, A. Asenjo-Garcia, T. C. Sum, and L.-C. Kwek, Dicke superradiance requires interactions beyond nearest neighbors, Phys. Rev. Lett. 130, 213605 (2023).
  * Rainò  _et al._ [2020] G. Rainò, H. Utzat, M. Bawendi, and M. Kovalenko, Superradiant emission from self-assembled light emitters: From molecules to quantum dots, MRS Bulletin 45, 841–848 (2020).
  * Rainò  _et al._ [2018] G. Rainò, M. A. Becker, M. I. Bodnarchuk, R. F. Mahrt, M. V. Kovalenko, and T. Stöferle, Superfluorescence from lead halide perovskite quantum dot superlattices, Nature 563, 671 (2018).
  * Lei  _et al._ [2023] M. Lei, R. Fukumori, J. Rochman, B. Zhu, M. Endres, J. Choi, and A. Faraon, Many-body cavity quantum electrodynamics with driven inhomogeneous emitters, Nature 617, 271 (2023).
  * Leghtas  _et al._ [2015] Z. Leghtas, S. Touzard, I. M. Pop, A. Kou, B. Vlastakis, A. Petrenko, K. M. Sliwa, A. Narla, S. Shankar, M. J. Hatridge, M. Reagor, L. Frunzio, R. J. Schoelkopf, M. Mirrahimi, and M. H. Devoret, Confining the state of light to a quantum manifold by engineered two-photon loss, Science 347, 853 (2015).
  * Lescanne  _et al._ [2020] R. Lescanne, M. Villiers, T. Peronnin, A. Sarlette, M. Delbecq, B. Huard, T. Kontos, M. Mirrahimi, and Z. Leghtas, Exponential suppression of bit-flips in a qubit encoded in an oscillator, Nat. Phys. 16, 509 (2020).
  * Radcliffe [1971] J. M. Radcliffe, Some properties of coherent spin states, J. Phys. A: Gen. Phys. 4, 313 (1971).
  * Qin  _et al._ [2021] W. Qin, A. Miranowicz, H. Jing, and F. Nori, Generating long-lived macroscopically distinct superposition states in atomic ensembles, Phys. Rev. Lett. 127, 093602 (2021).
  * Mirrahimi  _et al._ [2014] M. Mirrahimi, Z. Leghtas, V. V. Albert, S. Touzard, R. J. Schoelkopf, L. Jiang, and M. H. Devoret, Dynamically protected cat-qubits: a new paradigm for universal quantum computation, New J. Phys. 16, 045014 (2014).
  * Johansson  _et al._ [2013] J. Johansson, P. Nation, and F. Nori, QuTiP 2: A python framework for the dynamics of open quantum systems, Comput. Phys. Commun. 184, 1234 (2013).
  * Leghtas  _et al._ [2013] Z. Leghtas, G. Kirchmair, B. Vlastakis, R. J. Schoelkopf, M. H. Devoret, and M. Mirrahimi, Hardware-efficient autonomous quantum memory protection, Phys. Rev. Lett. 111, 120501 (2013).
  * Kubo [1962] R. Kubo, Generalized cumulant expansion method, J. Phys. Soc. Jpn. 17, 1100 (1962).
  * Plankensteiner  _et al._ [2022] D. Plankensteiner, C. Hotter, and H. Ritsch, QuantumCumulants.jl: A Julia framework for generalized mean-field equations in open quantum systems, Quantum 6, 617 (2022).
  * Hotter  _et al._ [2023] C. Hotter, L. Ostermann, and H. Ritsch, Cavity sub- and superradiance for transversely driven atomic ensembles, Phys. Rev. Res. 5, 013056 (2023).
  * Debnath  _et al._ [2018] K. Debnath, Y. Zhang, and K. Mølmer, Lasing in the superradiant crossover regime, Phys. Rev. A 98, 063837 (2018).
  * Debnath  _et al._ [2019] K. Debnath, Y. Zhang, and K. Mølmer, Collective dynamics of inhomogeneously broadened emitters coupled to an optical cavity with narrow linewidth, Phys. Rev. A 100, 053821 (2019).
  * Debnath  _et al._ [2020] K. Debnath, G. Dold, J. J. L. Morton, and K. Mølmer, Self-stimulated pulse echo trains from inhomogeneously broadened spin ensembles, Phys. Rev. Lett. 125, 137702 (2020).
  * Rubies-Bigorda  _et al._ [2023] O. Rubies-Bigorda, S. Ostermann, and S. F. Yelin, Characterizing superradiant dynamics in atomic arrays via a cumulant expansion approach, Phys. Rev. Res. 5, 013091 (2023).
  * Masson  _et al._ [2024] S. J. Masson, J. P. Covey, S. Will, and A. Asenjo-Garcia, Dicke superradiance in ordered arrays of multilevel atoms, PRX Quantum 5, 010344 (2024).

##  Appendix A Derivation of the semiclassical mean-field equations

Starting from the quantum master equation (1), we make the mean-field ansatz
$\rho=\bigotimes_{i=1}^{N}\rho_{i}$ which assumes that the quantum
correlations are negligible. This allows us to derive the single-spin dynamics
for $\braket{\sigma_{m}^{+}}\equiv\text{Tr}(\sigma_{m}^{+}\rho_{m})$ and
$\braket{\sigma_{m}^{z}}\equiv\text{Tr}(\sigma_{m}^{z}\rho_{m})$.
Equivalently, we factorize spin correlations
$\braket{\sigma_{i}^{+}\sigma_{j}^{-}}\approx\braket{\sigma_{i}^{+}}\braket{\sigma_{j}^{-}}$
and
$\braket{\sigma_{i}^{+}\sigma_{i}^{-}\sigma_{j}^{-}}\approx\braket{\sigma_{i}^{+}\sigma_{i}^{-}}\braket{\sigma_{j}^{-}}=(1+\braket{\sigma_{i}^{z}})\braket{\sigma_{j}^{-}}/2$
for $i\neq j$. Note that we treat the population and coherence separately,
i.e., we do not assume
$\braket{\sigma_{i}^{+}\sigma_{i}^{-}}\approx\braket{\sigma_{i}^{+}}\braket{\sigma_{i}^{-}}$,
which allows us to capture some effects of quantum coherence while the spins
are weakly excited. The contributions from the squeezing Hamiltonian are

$\frac{d}{dt}\braket{\sigma_{m}^{+}}=i\frac{\eta}{N}\sum_{i,j}e^{i\theta}\braket{[\sigma_{i}^{-}\sigma_{j}^{-},\sigma_{m}^{+}]}+e^{-i\theta}\cancel{\braket{[\sigma_{i}^{+}\sigma_{j}^{+},\sigma_{m}^{+}]}}=\frac{2i\eta}{N}e^{i\theta}\sum_{j\neq
m}\braket{[\sigma_{m}^{-},\sigma_{m}^{+}]\sigma_{j}^{-}}\approx\frac{-2i\eta}{N}e^{i\theta}\braket{\sigma_{m}^{z}}\sum_{j\neq
m}\braket{\sigma_{j}^{-}}$ (23)

where the factor of 2 comes from the symmetry of $i\leftrightarrow j$. For
$\braket{\sigma_{m}^{z}}$, we have

$\begin{split}\frac{d}{dt}\braket{\sigma_{m}^{z}}&=\frac{i\eta}{N}\left\\{\sum_{i,j}e^{i\theta}\braket{[\sigma_{i}^{-}\sigma_{j}^{-},\sigma_{m}^{z}]}+e^{-i\theta}\braket{[\sigma_{i}^{+}\sigma_{j}^{+},\sigma_{m}^{z}]}\right\\}=\frac{2i\eta}{N}\left\\{\sum_{j\neq
m}e^{i\theta}\braket{[\sigma_{m}^{-},\sigma_{m}^{z}]\sigma_{j}^{-}}+e^{-i\theta}\braket{[\sigma_{m}^{+},\sigma_{m}^{z}]\sigma_{j}^{+}}\right\\}\\\
&\approx\frac{4i\eta}{N}\sum_{j\neq
m}e^{i\theta}\braket{\sigma_{m}^{-}}\braket{\sigma_{j}^{-}}-e^{-i\theta}\braket{\sigma_{m}^{+}}\braket{\sigma_{j}^{+}}=\frac{8\eta}{N}\text{Im}\left(e^{-i\theta}\braket{\sigma_{m}^{+}}\sum_{j\neq
m}\braket{\sigma_{j}^{+}}\right).\end{split}$ (24)

Next, we compute the contribution from the collective two-body dissipator

$\begin{split}\frac{N^{2}}{\Gamma_{2}}\frac{d}{dt}\braket{\sigma_{m}^{+}}&=\sum_{i,j,k,l}\braket{\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{m}^{+}\sigma_{k}^{-}\sigma_{l}^{-}-\frac{1}{2}\sigma_{m}^{+}\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}\sigma_{l}^{-}-\frac{1}{2}\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}\sigma_{l}^{-}\sigma_{m}^{+}}\\\
&\approx\braket{\sigma_{m}^{+}\sigma_{m}^{-}}\sum_{\begin{subarray}{c}i,j,k\\\
ijk\neq
m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}}-\braket{\sigma_{m}^{-}\sigma_{m}^{+}}\sum_{\begin{subarray}{c}i,j,k\\\
ijk\neq
m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}}-2\braket{\sigma_{m}^{+}\sigma_{m}^{-}\sigma_{m}^{+}}\sum_{\begin{subarray}{c}i,j\\\
ij\neq m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{-}}\\\
&=\braket{\sigma_{m}^{z}}\sum_{\begin{subarray}{c}i,j,k\\\ ijk\neq
m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}}-2\braket{\sigma_{m}^{+}}\sum_{\begin{subarray}{c}i,j\\\
ij\neq m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{-}}\end{split}$ (25)

Eq. (25) contains terms such as
$\braket{\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}}$ and
$\braket{\sigma_{i}^{+}\sigma_{j}^{-}}$. To obtain a closed set of mean-field
equations, we have to reduce these in terms of $\braket{\sigma_{i}^{+}}$ and
$\braket{\sigma_{i}^{z}}$. Defining $c_{1,m}\equiv\frac{1}{N}\sum_{j\neq
m}\braket{\sigma_{j}^{+}}$ and $c_{2,m}\equiv\frac{1}{2N}\sum_{j\neq
m}(1+\braket{\sigma_{j}^{z}})$, we then obtain

$\begin{split}\frac{N^{2}}{\Gamma_{2}}\frac{d}{dt}\braket{\sigma_{m}^{+}}&\approx\braket{\sigma_{m}^{z}}\left(2\sum_{\begin{subarray}{c}i,j\\\
ij\neq m\\\ i\neq
j\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{i}^{-}}\braket{\sigma_{j}^{+}}+\sum_{\begin{subarray}{c}i,j,k\\\
ijk\neq m\\\ i\neq j\neq
m\end{subarray}}\braket{\sigma_{i}^{+}}\braket{\sigma_{j}^{+}}\braket{\sigma_{k}^{-}}\right)-2\braket{\sigma_{m}^{+}}\left(\sum_{i\neq
m}\braket{\sigma_{i}^{+}\sigma_{i}^{-}}+\sum_{\begin{subarray}{c}i,j\\\ ij\neq
m\\\ i\neq
j\end{subarray}}\braket{\sigma_{i}^{+}}\braket{\sigma_{j}^{-}}\right)\\\
&=-2\braket{\sigma_{m}^{+}}(Nc_{2,m}+N^{2}|c_{1,m}|^{2})+\braket{\sigma_{m}^{z}}(2N^{2}c_{1,m}c_{2,m}+N^{3}|c_{1,m}|^{2}c_{1,m})\end{split}$
(26)

Similarly, we can obtain the equation of motion for $\braket{\sigma_{m}^{z}}$:

$\begin{split}\frac{1}{\Gamma_{2}}\frac{d}{dt}\braket{\sigma_{m}^{z}}&=\sum_{i,j,k,l}\braket{\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{m}^{z}\sigma_{k}^{-}\sigma_{l}^{-}-\frac{1}{2}\sigma_{m}^{z}\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}\sigma_{l}^{-}-\frac{1}{2}\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}\sigma_{l}^{-}\sigma_{m}^{z}}\\\
&\approx(\braket{\sigma_{m}^{+}\sigma_{m}^{z}}-\braket{\sigma_{m}^{z}\sigma_{m}^{+}})\sum_{\begin{subarray}{c}i,j,k\\\
ijk\neq
m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{-}\sigma_{k}^{-}}+(\braket{\sigma_{m}^{z}\sigma_{m}^{-}}-\braket{\sigma_{m}^{-}\sigma_{m}^{z}}\sum_{\begin{subarray}{c}i,j,k\\\
ijk\neq m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}}\\\
&+(4\braket{\sigma_{m}^{+}\sigma_{m}^{z}\sigma_{m}^{-}}-2\braket{\sigma_{m}^{z}\sigma_{m}^{+}\sigma_{m}^{-}}-2\braket{\sigma_{m}^{+}\sigma_{m}^{-}\sigma_{m}^{z}})\sum_{\begin{subarray}{c}i,j\\\
ij\neq m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{-}}\\\
&=-2\braket{\sigma_{m}^{+}}\sum_{\begin{subarray}{c}i,j,k\\\ ijk\neq
m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{-}\sigma_{k}^{-}}-2\braket{\sigma_{m}^{-}}\sum_{\begin{subarray}{c}i,j,k\\\
ijk\neq
m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{+}\sigma_{k}^{-}}-4(1+\braket{\sigma_{m}^{z}})\sum_{\begin{subarray}{c}i,j\\\
ij\neq m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{-}}\\\
&=-4\text{Re}\left(\braket{\sigma_{m}^{+}}\sum_{\begin{subarray}{c}i,j,k\\\
ijk\neq
m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{-}\sigma_{k}^{-}}\right)-4(1+\braket{\sigma_{m}^{z}})\sum_{\begin{subarray}{c}i,j\\\
ij\neq m\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{j}^{-}}\\\
&\approx-4\text{Re}\left[\braket{\sigma_{m}^{+}}\left(2\sum_{\begin{subarray}{c}i,j\\\
ij\neq m\\\ i\neq
j\end{subarray}}\braket{\sigma_{i}^{+}\sigma_{i}^{-}}\braket{\sigma_{j}^{-}}+\sum_{\begin{subarray}{c}i,j,k\\\
ijk\neq m\\\ i\neq j\neq
k\end{subarray}}\braket{\sigma_{i}^{+}}\braket{\sigma_{j}^{-}}\braket{\sigma_{k}^{-}}\right)\right]-4(1+\braket{\sigma_{m}^{z}})\left(\sum_{i\neq
m}\braket{\sigma_{i}^{+}\sigma_{i}^{-}}+\sum_{\begin{subarray}{c}i,j\\\ ij\neq
m\end{subarray}}\braket{\sigma_{i}^{+}}\braket{\sigma_{j}^{-}}\right)\\\
&=-4(1+\braket{\sigma_{m}^{z}})(Nc_{2,m}+N^{2}|c_{1,m}|^{2})+\text{Re}[\braket{\sigma_{m}^{z}}(2N^{2}c_{1,m}^{*}c_{2,m}+N^{3}|c_{1,m}|^{2}c_{1,m}^{*}]\end{split}$
(27)

Combining Eqs. (23), (24), (26) and (27), we have the mean-field equations:

$\begin{split}\frac{d}{dt}\braket{\sigma_{m}^{+}}&=i\omega_{m}\braket{\sigma_{m}^{+}}-2i\eta
e^{i\theta}\braket{\sigma_{m}^{z}}c_{1,m}^{*}-2\Gamma_{2}\braket{\sigma_{m}^{+}}\left(\frac{c_{2,m}}{N}+|c_{1,m}|^{2}\right)+\Gamma_{2}\braket{\sigma_{m}^{z}}c_{1,m}(2c_{2,m}+N|c_{1,m}|^{2})\end{split}$
(28)

$\begin{split}\frac{d}{dt}\braket{\sigma_{m}^{z}}&=8\eta\text{Im}\left(e^{-i\theta}\braket{\sigma_{m}^{+}}c_{1,m}\right)-4\Gamma_{2}(1+\braket{\sigma_{m}^{z}})\left(\frac{c_{2,m}}{N}+|c_{1,m}|^{2}\right)-4\Gamma_{2}\text{Re}[\braket{\sigma_{m}^{+}}c_{1,m}^{*}(2c_{2}+N|c_{1,m}|^{2})]\end{split}$
(29)

as described in Eqs. (16) and (17) in the main text.

##  Appendix B Two detuned ensembles, small $\delta$ limit

When the detuning $\delta$ is small, we expect the steady state value for the
phase $\phi$ to be close to $\pi/4$, as was shown in Eq. (19). Writing
$\phi=\pi/4\pm\zeta$, we obtain the mean-field equations (20). Expanding the
equations to linear order in $\zeta$, and then to order $1/N$, we obtain the
simplifed equations (setting $\Gamma_{2}=1$ for notational simplicity)

$\begin{split}\frac{\dot{A}}{A}&\approx-2A^{2}+\frac{N-1}{N}z(1-2\eta+(N-2)A^{2})+\frac{N-1}{N}z^{2}\\\
\dot{\zeta}&\approx\delta-\frac{z}{N}(N+4\eta-2N\eta+(N-2)A^{2}+Nz)\zeta\\\
\frac{\dot{z}}{A^{2}}&\approx\frac{4A^{2}}{N}(1+2N(\eta-1)-2\eta-(N-2)(N-1)A^{2}+z-2Nz).\end{split}$
(30)

Since $A^{2}$ is of order $1/N$, one has to be careful here when making the
large $N$ approximation. We verify the accuracy of the above equation by
comparing numerical simulations against those from the full mean-field
equations (16) and (17). The steady state solution is complicated, but we can
expand in powers of $1/N$ to get

$A^{2}\approx\frac{2\eta}{N}-\frac{4\eta}{N^{2}}+\frac{4\eta-32\eta^{2}}{N^{2}}+\ldots\approx\frac{2\eta}{N},$
(31)
$\zeta\approx\delta\left(\frac{1+\sqrt{1-16(\eta/N)}}{32(\eta/N)^{2}}+\frac{1+32(\eta/N)}{128N(\eta/N)^{3}}+\ldots\right)\approx\frac{1+\sqrt{1-16\eta/N}}{32(\eta/N)^{2}}\delta$
(32)

and

$z\approx-\frac{1+\sqrt{1-16\eta/N}}{2}$ (33)


# Efficient Fine-Tuning with Domain Adaptation for Privacy-Preserving Vision
Transformer

1st Teru Nagamori  Tokyo Metropolitan University
Tokyo,Japan
<EMAIL_ADDRESS>2nd Sayaka Shiota  Tokyo Metropolitan University
Tokyo,Japan
<EMAIL_ADDRESS>3rd Hitoshi Kiya  Tokyo Metropolitan University
Tokyo,Japan
<EMAIL_ADDRESS>
###### Abstract

We propose a novel method for privacy-preserving deep neural networks (DNNs)
with the Vision Transformer (ViT). The method allows us not only to train
models and test with visually protected images but to also avoid the
performance degradation caused from the use of encrypted images, whereas
conventional methods cannot avoid the influence of image encryption. A domain
adaptation method is used to efficiently fine-tune ViT with encrypted images.
In experiments, the method is demonstrated to outperform conventional methods
in an image classification task on the CIFAR-10 and ImageNet datasets in terms
of classification accuracy.

###### Index Terms:

privacy-preserving, domain adaptation, image classification, Vision
Transformer

##  I Introduction

Deep neural networks (DNNs) have been deployed in many applications including
security-critical ones such as biometric authentication and medical image
analysis. In addition, training a deep learning model requires a huge amount
of data and fast computing resources, so cloud environments are increasingly
used in various applications of DNN models. However, since cloud providers are
not always reliable in general, privacy-preserving deep learning has become an
urgent problem [1, 2, 3].

One of the privacy-preserving solutions for DNNs is to use encrypted images to
protect visual information in images for training and testing models [4, 5].
In this approach, images are transformed by using a learnable encryption
method, and encrypted images are used as training and testing data. The use of
models trained with encrypted images enables us to use state-of-the-art
learning algorithms without any modification. However, all conventional
learnable encryption methods [6, 7, 8, 9, 10, 11] have a problem, that is, the
performance of encrypted models degrades compared with models without
encryption.

Traditional cryptographic methods such as homomorphic encryption [12, 13] are
one of the other privacy-preserving approaches, but the computation and memory
costs are expensive, and it is not easy to apply these methods to state-of-
the-art DNNs directly. Federated learning [3, 14] allows users to train a
global model without centralizing the training data on one machine, but it
cannot protect privacy during inference for test data when a model is deployed
in an untrusted cloud server.

Accordingly, we focus on the problem of degraded model performance that occurs
when models are trained with encrypted images. In this paper, we consider
reducing the performance degradation of encrypted models when using encrypted
images in the Vision Transformer (ViT) [15], which has high performance in
image classification, and we propose a domain adaptation method to reduce the
influence of encryption. In experiments, the proposed method is demonstrated
to maintain almost the same performance as models trained with plain images in
terms of image classification accuracy even when using encrypted images.

The rest of this paper is structured as follows. Section II presents related
work on image encryption for deep learning and the Vision Transformer.
Regarding the proposed method, Section III includes an overview, image
encryption, and fine-tuning with domain adaptation. Experiments for verifying
the effectiveness of the method, including classification performance,
training efficiency, and comparison with conventional methods, are presented
in Section IV, and Section V concludes this paper.

Figure 1: Architecture of Vision Transformer ($N$=9)

##  II Related work

###  II-A Image Encryption for Deep Learning

Various image transformation methods with a secret key, often referred to as
perceptual image encryption or image cryptography, have been studied so far
for many applications [1]. In this paper, we focus on learnable images
transformed with a secret key, which have been studied for deep learning.
Learnable encryption enables us to directly apply encrypted data to a model as
training and testing data. Encrypted images have no visual information on
plain images in general, so privacy-preserving learning can be carried out by
using visually protected images. In addition, the use of a secret key allows
us to embed unique features controlled with the key into images. Adversarial
defenses [16] and access control [17] are carried out with encrypted data
using unique features.

Tanaka first introduced a block-wise learnable image encryption method (LE)
with an adaptation layer [6], which is used prior to a classifier to reduce
the influence of image encryption. Another encryption method is a pixel-wise
encryption (PE) method in which negative-positive transformation (NP) and
color component shuffling are applied without using any adaptation layer [7].
However, both encryption methods are not robust enough against ciphertext-only
attacks, as reported in [18, 4]. To enhance the security of encryption, LE was
extended to an extended learnable image encryption method (ELE) by adding a
block scrambling (permutation) step and a pixel encryption operation with
multiple keys [8]. However, ELE still has an inferior accuracy compared with
using plain images, even when an additional adaptation network (denoted as
ELE-AdaptNet hereinafter) is applied to reduce the influence of the encryption
[8]. Moreover, large-size images cannot be applied to ELE because of the high
computation cost of ELE-AdaptNet.

Recently, block-wise encryption was also pointed out to have a high similarity
with isotropic networks such as ViT and ConvMixer [19], and the similarity
enables us to reduce performance degradation [9, 10, 19]. In addition, neural
network-based image encryption methods without any block-wise encryption,
called NeuraCrypt [20] and Color-NeuraCrypt [11], have been proposed, but
these methods also have the same performance degradation problem as
conventional block-wise encryption methods.

Accordingly, we propose a domain adaptation method so that the performance
degradation problem is solved by combining block-wise encryption with domain
adaptation under the use of ViT.

###  II-B Vision Transformer

The Vision Transformer (ViT) [15] is commonly used in image classification
tasks and is known to provide a high classification performance. As shown in
Fig. 1, in ViT, an input image $x\in\mathbb{R}^{h\times w\times c}$ is divided
into $N$ patches with a size of $p\times p$, where $h$, $w$, and $c$ are the
height, width, and number of channels of the image. Also, $N$ is given as
$hw/p^{2}$. Afterward, each patch is flattened into
$x_{p}^{i}=[x_{p}^{i}(1),x_{p}^{i}(2),\dots,x_{p}^{i}(L)]$. Finally, a
sequence of embedded patches is given as

$\displaystyle z_{0}=$
$\displaystyle[x_{class};x_{p}^{1}\mathbf{E};x_{p}^{2}\mathbf{E};\dots
x_{p}^{i}\mathbf{E};\dots x_{p}^{N}\mathbf{E}]+\mathbf{E_{pos}},$ (1)

where

$\displaystyle\mathbf{E_{pos}}=$
$\displaystyle((e_{pos}^{0})(e_{pos}^{1})\dots(e_{pos}^{i})\dots(e_{pos}^{N})),$
$\displaystyle L=$ $\displaystyle p^{2}c,x_{class}\in\mathbb{R}^{D},\
x_{p}^{i}\in\mathbb{R}^{L},\ e_{pos}^{i}\in\mathbb{R}^{D},$
$\displaystyle\mathbf{E}\in$ $\displaystyle\mathbb{R}^{L\times D},\
\mathbf{E_{pos}}\in\mathbb{R}^{(N+1)\times D}.$

$x_{class}$ is a class token, $\mathbf{E}$ is an embedding (patch embedding)
that linearly maps each patch to dimensions $D$, $\mathbf{E_{pos}}$ is an
embedding (position embedding) that gives position information to patches in
the image, $e_{pos}^{0}$ is the position information of a class token,
$e_{pos}^{i}$ is the position information of each patch, and $z_{0}$ is a
sequence of embedded patches. Afterward, $z_{0}$ is input into the transformer
encoder. The encoder outputs only the class token, which is a vector of
condensed information on the entire image, and it is used for classification.

Previous studies have indicated that when DNN models are trained with
encrypted images, the performance of the models is degraded compared with
models trained with plain images. In contrast, it has been pointed out that
block-wise encryption has similarity with models that have embedding
structures such as ViT [9, 19]. Accordingly, we focus on the relationship
between block-wise image encryption and the embedding structure of ViT to
avoid the influence of image encryption even when encrypted images are used
for training models.

Figure 2: Overview of proposed fine-tuning

##  III Proposed Method

###  III-A Overview

Figure 2 shows the procedure for fine-tuning a model by using the proposed
domain adaptation. Since a model pre-trained with plain images as in [8] is
prepared in advance, the fine-tuning with encrypted images that we will
discuss is highlighted in the figure.

A model creator applies the domain adaptation to the pre-trained model
$f_{\theta}$ to generate a model $\hat{f}_{\theta}$ with a key. Model
$\hat{f}_{\theta}$ is then uploaded to a cloud service such as Amazon Web
Services (AWS), which may be untrusted. Model $\hat{f}_{\theta}$ is fine-tuned
by using images encrypted with the key. In this paper, we use a block-wise
encryption method, which consists of block scrambling and pixel shuffling, to
generate encrypted images as an example of block-wise encryption [1]. In block
scrambling, an image is divided into non-overlapped blocks (patches), and the
divided blocks are randomly permutated with a key. In contrast, pixel
shuffling is used to randomly permutate the position of pixels in each block
as described later.

Finally, test images are encrypted with the same keys as those used for fine-
tuning by a user, and the encrypted test images are input to the fine-tuned
model.

Figure 3: Procedure of image encryption

###  III-B Image Encryption

Model $f_{\theta}$ is fine-tuned by using encrypted images as shown in Fig. 2.
Figure 3 shows the encryption procedure used in this paper. Details on the
procedure are given below.

  * (a)

Divide an image $x\in\mathbb{R}^{h\times w\times c}$ into non-overlapped
blocks with a size of $p\times p$ such that
$B=(\left\\{B_{1},\dots,B_{i},\dots,B_{N}\right\\})^{\top}$,
$B_{i}\in\mathbb{R}^{p^{2}\times c}$.

  * (b)

Permutate blocks with secret key $k_{1}$ such as

$\displaystyle\hat{B}=$ $\displaystyle\mathbf{E_{bs}}B$ (2) $\displaystyle=$
$\displaystyle(\left\\{\hat{B}_{1},\dots,\hat{B}_{i},\dots,\hat{B}_{N}\right\\})^{\top},\hat{B}_{i}\in\mathbb{R}^{p^{2}\times
c}$

where $\hat{B}_{i}\in\left\\{B_{1},\dots,B_{N}\right\\}$, and a permutation
matrix $\mathbf{E_{bs}}$ is generated with key $k_{1}$ as follows.

    * 1)

Using key $k_{1}$, generate a random integer vector with a length of $N$ as

$l_{e}=[l_{e}(1),l_{e}(2),\dots,l_{e}(i)\ ,\dots,l_{e}(N)]\ ,$ (3)

where

$\displaystyle l_{e}(i)\in$ $\displaystyle\left\\{1,2,...,N\right\\},$
$\displaystyle l_{e}(i)\neq$ $\displaystyle l_{e}(j)\ \text{if}\ i\neq j,$
$\displaystyle i,j\in$ $\displaystyle\left\\{1,\dots,N\right\\}.$

    * 2)

$k_{(i,j)}$ is given as

$k_{(i,j)}=\left\\{\begin{matrix}0&(j\neq l_{e}(i))\\\
1&(j=l_{e}(i))\end{matrix}\right..$ (4)

    * 3)

Define $\mathbf{E_{bs}}\in\mathbb{R}^{N\times N}$ as

$\displaystyle\mathbf{E_{bs}}=\begin{bmatrix}k_{(1,1)}&k_{(1,2)}&\dots&k_{(1,N)}\\\
k_{(2,1)}&k_{(2,2)}&\dots&k_{(2,N)}\\\ \vdots&\vdots&\ddots&\vdots\\\
k_{(N,1)}&k_{(N,2)}&\dots&k_{(N,N)}\\\ \end{bmatrix},$ (5)

For example, $\mathbf{E_{bs}}$ is given by, for $N=3$,

$\displaystyle\mathbf{E_{bs}}=\begin{bmatrix}1&0&0\\\ 0&0&1\\\ 0&1&0\\\
\end{bmatrix}.$ (6)

  * (c)

Flatten each block $\hat{B_{i}}$ into a vector $\hat{b}_{i}$ with a length of
$L=p\times p\times c$ such that

$\hat{b}_{i}=[\hat{b}_{i}(1),\dots,\hat{b}_{i}(L)]^{\top},$ (7)

where $\hat{b}_{i}$ is the same as $(x_{p}^{i})^{\top}$ in Eq. 1.

  * (d)

Shuffle pixels in $\hat{b}_{i}$ with secret key $k_{2}$ such as

$\displaystyle b^{\prime}_{i}=$
$\displaystyle(\mathbf{E}_{ps}\hat{b}_{i})^{\top}$ (8) $\displaystyle=$
$\displaystyle[b^{\prime}_{i}(1),\dots,b^{\prime}_{i}(L)],$

where $\mathbf{E_{ps}}$ is a matrix generated by the same procedure as
$\mathbf{E_{bs}}$ ($N$ is replaced with $L$).

Also, Eq. 8 is expressed as

$b^{\prime}_{i}=(\mathbf{E}_{ps}\hat{b}_{i})^{\top}=(\mathbf{E}_{ps}(x_{p}^{i})^{\top})^{\top}=x^{{}^{\prime}i}_{p}.$
(9)

  * (e)

Concatenate the encrypted vectors
$b^{\prime}_{i}(i\in\left\\{1,\dots,N\right\\})$ into an encrypted image
$x^{{}^{\prime}}$.

Figure 4 shows an example of images encrypted with this procedure, where block
size $p$ was 16.

|  |  |
---|---|---|---
| (a) Original
---
(224 $\times$ 224 $\times$ 3)
| (b) Block scrambling
---
(block size = 16)
| (c) Pixel shuffling
---
(block size = 16)
| (d) Block scrambling + Pixel shuffling
---
(block size = 16)

Figure 4: Example of encrypted images

###  III-C Fine-tuning with Domain Adaptation

As shown in Fig. 2, model $f_{\theta}$ pre-trained in the plain domain is
transformed into model $\hat{f}_{\theta}$ to reduce the influence of
encryption prior to the fine-tuning of $f_{\theta}$ by using the proposed
domain adaptation. The procedure of the adaptation is summarized here.

The domain adaptation is carried out in accordance with the embedding
structure of ViT. $\mathbf{E_{bs}}$ in Eq. 5 is used to permutate blocks in an
image, so it has a close relationship with position embedding
$\mathbf{E_{pos}}$ in Eq. 1, where $\mathbf{E_{pos}}$ includes the position
information of a class token $x_{class}$. In contrast, $\mathbf{E_{bs}}$ does
not consider the information of $x_{class}$. To fill the gap between
$\mathbf{E_{pos}}$ and $\mathbf{E_{bs}}$, $\mathbf{E_{bs}}$ is extended as

$\mathbf{E_{bs}^{\prime}}=\begin{bmatrix}1&0&0&\dots&0\\\
0&k_{(1,1)}&k_{(1,2)}&\dots&k_{(1,N)}\\\
0&k_{(2,1)}&k_{(2,2)}&\dots&k_{(2,N)}\\\ \vdots&\vdots&\vdots&\ddots&\vdots\\\
0&k_{(N,1)}&k_{(N,2)}&\dots&k_{(N,N)}\\\ \end{bmatrix},$ (10)
$\displaystyle\mathbf{E_{bs}^{\prime}}$
$\displaystyle\in\mathbb{R}^{(N+1)\times(N+1)}.$

In the domain adaptation, $\mathbf{E_{pos}}$ is transformed as in

$\mathbf{\hat{E}_{pos}}=\mathbf{E_{bs}^{\prime}}\mathbf{E_{pos}}.$ (11)

By using Eq. 11, $\mathbf{E_{pos}}$ can be adapted to images permutated by
block scrambling. Similarly, $\mathbf{E}$ is adapted to pixels that are
randomly replaced by pixel shuffling, as in

$\mathbf{\hat{E}}=\mathbf{E_{ps}}\mathbf{E}.$ (12)

From Eqs. 2, 9, 11, and 12, under the use of image encryption and domain
adaptation, an adapted sequence of embedded patches in Eq. 1 is replaced with

$\displaystyle\hat{z}_{0}=$
$\displaystyle\mathbf{E_{bs}^{\prime}}[x_{class};(\mathbf{E_{ps}}(x_{p}^{1})^{\top})^{\top}\mathbf{E_{ps}}\mathbf{E};\dots(\mathbf{E_{ps}}(x_{p}^{N})^{\top})^{\top}\mathbf{E_{ps}}\mathbf{E}]+\mathbf{E_{bs}^{\prime}}\mathbf{E_{pos}}$
(13) $\displaystyle=$
$\displaystyle\mathbf{E_{bs}^{\prime}}[x_{class};x_{p}^{{}^{\prime}1}\mathbf{\hat{E}};\dots
x_{p}^{{}^{\prime}N}\mathbf{\hat{E}}]+\mathbf{E_{bs}^{\prime}}\mathbf{E_{pos}}.$

In accordance with Eqs. 11 and 12, $f_{\theta}$ with $\mathbf{E_{pos}}$ and
$\mathbf{E}$ is transformed to $\hat{f}_{\theta}$ with
$\mathbf{\hat{E}_{pos}}$ and $\mathbf{\hat{E}}$ as shown in Fig. 2, where the
difference between $f_{\theta}$ and $\hat{f}_{\theta}$ is only a sequence of
embedding patches $\hat{z}_{0}$ in Eq. 13.

Finally, test images are applied to $\hat{f}_{\theta}$ in the adaptation
domain. In our framework, as shown Fig. 5, test images are input to a model
after fine-tuning $\hat{f}_{\theta}$ with encrypted images. In addition, test
images are required to be encrypted by using the same keys as those used for
fine-tuning the model.

Figure 5: Test procedure

##  IV Experiments

The effectiveness of the proposed method is verified in terms of the accuracy
of image classification and the efficiency of model training.

###  IV-A Setup

We conducted image classification experiments on the CIFAR-10 [21] and
CIFAR-100 datasets [21]. CIFAR-10 consists of 60,000 images with 10 classes
(6000 images for each class), and CIFAR-100 consists of 60,000 images with 100
classes (600 images for each class), where 50,000 images were used for
training, and 10,000 images were used for testing. In addition, we used the
Imagenette dataset [22], which is a subset of the ImageNet dataset with 10
classes. We also fine-tuned a ViT-B 16 model pre-trained with the ImageNet21k
dataset, so we resized the image size from $32\times 32\times 3$ to $224\times
224\times 3$. Training was carried out under the use of a batch size of 32, a
learning rate of 0.001, a momentum of 0.9, and a weight decay of 0.0005 using
the stochastic gradient descent (SGD) algorithm for 15 epochs. A cross-entropy
loss function was used as the loss function. The block size of the encryption
was set to 16 to match the ViT patch size.

TABLE I: Classification accuracy (%) with and without proposed method (CIFAR-10) | Domain Adaptation
---|---
Encryption | Yes | No
Pixel shuffling | 99.05 | 97.78
Block scrambling | 98.93 | 94.25
Pixel shuffling + Block scrambling | 98.98 | 72.86
Plain (Baseline) | 99.00
|  |
---|---|---
(a)Training
|  |
(b)Testing
Figure 6: Learning curves during training and testing

###  IV-B Classification Performance

Table I shows the classification accuracy of models trained with encrypted
images. When training models without the proposed domain adaptation, the
accuracy decreased compared with that of using plain images. In particular,
when applying the combined use of pixel shuffling and block scrambling, the
classification accuracy significantly decreased due to the influence of
encryption. In contrast, when training models with the domain adaptation, the
accuracy was almost the same as that of using plain images (Baseline).
Accordingly, we confirmed that the proposed domain adaptation was effective in
fine-tuning pre-trained models with encrypted images.

###  IV-C Training Efficiency

When training a model, it is important to consider not only the classification
performance but also the training efficiency. Training efficiency means the
time required for model training and loss convergence. The use of encrypted
images should be avoided as it significantly increases the time required for
training and loss convergence compared with models trained using plain images.

Figure 6 shows the classification accuracy and loss value of three models per
epoch during training and testing where the number of epochs was increased to
30. For training, from the figure, a model fine-tuned with only encrypted
images (without DA) needed a larger number of epochs to achieve the same
values as those of the plain model (Plain). In contrast, when using the domain
adaptation, the model fine-tuned with encrypted images (Proposed) achieved
almost the same performances in each epoch as the model trained with plain
images. This means that even if encrypted images are used, the domain
adaptation allows us not only to train privacy-preserving models with almost
the same accuracy as plain models but to also avoid increasing the model
training time.

For testing, the model without the domain adaptation was confirmed to have
degraded accuracy and loss performances compared with the plain model. In
contrast, the model with domain adaptation had almost the same performances as
the plain model.

Accordingly, the proposed method allows us not only to train privacy-
preserving models without performance degradation but to also avoid an
increase in training time even when using encrypted images.

TABLE II: Comparison with existing methods (CIFAR-10 and CIFAR-100) Encryption | Model | Accuracy (%) $\uparrow$
---|---|---
CIFAR-10 | CIFAR-100
Block scrambling $+$ Pixel shuffling $+$ Domain adaptation (Proposed) | ViT-B 16 | 98.98 | 92.58
LE [6] | Shakedrop $+$ AdaptNet | 94.49 | 75.48
ELE [8] | Shakedrop $+$ AdaptNet | 83.06 | 62.97
PE [7] | ResNet-18 | 92.03 | -
Block scrambling $+$ Pixel shuffling $+$ Negative/Positive transformation [23] | ConvMixer-512/16 $+$ Adaptive matrix | 92.65 | -
Color-NeuraCrypt [11] | ViT-B 16 | 96.20 | -
Block scrambling $+$ Pixel position shuffling [10] | ViT-B 16 | 96.64 | 84.42
EtC [9] | ViT-B 16 | 87.89 | -
| ConvMixer-256/8 | 92.72 | -
Plain | ShakeDrop | 96.70 | 83.59
ResNet-18 | 95.53 | -
ViT-B 16 | 99.00 | 92.60
ConvMixer-256/8 | 96.07 | -
TABLE III: Comparison with existing methods (Imagenette) Encryption | Model | Accuracy(%) $\uparrow$
---|---|---
Block scrambling $+$ Pixel shuffling $+$ Domain adaptation (Proposed) | ViT-B 16 | 99.2
EtC [9] | ViT-B 16 | 90.62
| ConvMixer-384/8 | 90.11
Plain | ViT-B 16 | 99.2
ConvMixer-384/8 | 96.10

###  IV-D Comparison with Conventional Methods

In Tables II and III, the proposed method is compared with the state-of-the-
art methods for privacy preserving image classification with encrypted images
such as LE [6], PE [7], and ELE [8]. From the tables, the proposed method not
only has the lowest performance degradation on all datasets but also
outperforms all existing methods in terms of the image classification
accuracy.

##  V Conclusion

In this paper, we proposed a domain adaptation method to reduce the influence
of image encryption for privacy-preserving ViT. The method is applied to a
model pre-trained in the plain domain prior to the fine-tuning of the pre-
trained model. The domain adaptation is carried out with two random matrices
generated with secret keys, and test images are also encrypted by using the
keys. In experiments, the method was demonstrated not only to reduce the
performance degradation of models but to also achieve the highest
classification accuracy among conventional methods on the CIFAR-10, CIFAR-100,
and Imagenette datasets.

## Acknowledgment

This study was partially supported by JSPS KAKENHI (Grant Number JP21H01327)
and JST CREST (Grant Number JPMJCR20D3).

## References

  * [1] H. Kiya, M. AprilPyone, Y. Kinoshita, S. Imaizumi, and S. Shiota, “An overview of compressible and learnable image transformation with secret key and its applications,” _APSIPA Transactions on Signal and Information Processing_ , vol. 11, no. 1, e11, 2022.
  * [2] T. Chuman, W. Sirichotedumrong, and H. Kiya, “Encryption-then-compression systems using grayscale-based image encryption for jpeg images,” _IEEE Transactions on Information Forensics and Security_ , vol. 14, no. 6, pp. 1515–1525, 2019.
  * [3] K. Wei, J. Li, C. Ma, M. Ding, W. Chen, J. Wu, M. Tao, and H. V. Poor, “Personalized federated learning with differential privacy and convergence guarantee,” _IEEE Transactions on Information Forensics and Security_ , vol. 18, pp. 4488–4503, 2023.
  * [4] H. Ito, Y. Kinoshita, M. Aprilpyone, and H. Kiya, “Image to perturbation: An image transformation network for generating visually protected images for privacy-preserving deep neural networks,” _IEEE Access_ , vol. 9, pp. 64 629–64 638, 2021.
  * [5] M. Maung, A. Pyone, and H. Kiya, “Encryption inspired adversarial defense for visual classification,” in _2020 IEEE International Conference on Image Processing (ICIP)_ , 2020, pp. 1681–1685.
  * [6] M. Tanaka, “Learnable image encryption,” in _2018 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW)_ , 2018, pp. 1–2.
  * [7] W. Sirichotedumrong, Y. Kinoshita, and H. Kiya, “Pixel-based image encryption without key management for privacy-preserving deep neural networks,” _IEEE Access_ , vol. 7, pp. 177 844–177 855, 2019.
  * [8] K. Madono, M. Tanaka, O. Masaki, and O. Tetsuji, “Block-wise scrambled image recognition using adaptation network,” in _Workshop on Artificial Intelligence of Things (AAAI WS)_ , 2020.
  * [9] M. AprilPyone and H. Kiya, “Privacy-preserving image classification using an isotropic network,” _IEEE MultiMedia_ , vol. 29, no. 2, pp. 23–33, 2022.
  * [10] Z. Qi, M. AprilPyone, Y. Kinoshita, and H. Kiya, “Privacy-preserving image classification using vision transformer,” _EURASIP European Signal Processing Conference, Belgrade, Serbia, August 31_ , pp. 543–547, 2022.
  * [11] Z. Qi, M. AprilPyone, and H. Kiya, “Color-neuracrypt: Privacy-preserving color-image classification using extended random neural networks,” 2023.
  * [12] L. T. Phong, Y. Aono, T. Hayashi, L. Wang, and S. Moriai, “Privacy-preserving deep learning via additively homomorphic encryption,” _IEEE Transactions on Information Forensics and Security_ , vol. 13, no. 5, pp. 1333–1345, 2018.
  * [13] Y. Wang, J. Lin, and Z. Wang, “An efficient convolution core architecture for privacy-preserving deep learning,” in _2018 IEEE International Symposium on Circuits and Systems (ISCAS)_ , 2018, pp. 1–5.
  * [14] J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” _arXiv_ , 2016. [Online]. Available: https://arxiv.org/abs/1610.05492
  * [15] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7_. OpenReview.net, 2021.
  * [16] M. AprilPyone and H. Kiya, “Block-wise image transformation with secret key for adversarially robust defense,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 2709–2723, 2021.
  * [17] H. Kiya, R. Iijima, M. AprilPyone, and Y. Kinoshita, “Image and model transformation with secret key for vision transformer,” _IEICE Transactions on Information and Systems_ , vol. E106.D, no. 1, pp. 2–11, 2023.
  * [18] A. H. Chang and B. M. Case, “Attacks on image encryption schemes for privacy-preserving deep neural networks,” 2020.
  * [19] H. Kiya, T. Nagamori, S. Imaizumi, and S. Shiota, “Privacy-preserving semantic segmentation using vision transformer,” _Journal of Imaging_ , vol. 8, no. 9, 2022. [Online]. Available: https://www.mdpi.com/2313-433X/8/9/233
  * [20] A. Yala, H. Esfahanizadeh, R. G. L. D. Oliveira, K. R. Duffy, M. Ghobadi, T. S. Jaakkola, V. Vaikuntanathan, R. Barzilay, and M. Medard, “Neuracrypt: Hiding private health data via random neural networks for public training,” 2021.
  * [21] A. Krizhevsky, “Learning multiple layers of features from tiny images.” University of Toronto, Tech. Rep., 2009.
  * [22] J. Howard, “imagenette.” [Online]. Available: https://github.com/fastai/imagenette/
  * [23] Z. Qi, A. MaungMaung, and H. Kiya, “Privacy-preserving image classification using convmixer with adaptative permutation matrix and block-wise scrambled image encryption,” _Journal of Imaging_ , vol. 9, no. 4, 2023. [Online]. Available: https://www.mdpi.com/2313-433X/9/4/85