Corresponding author<EMAIL_ADDRESS>
#  Enhanced Spectral Density of a Single Germanium Vacancy Center in a
Nanodiamond by Cavity-Integration

Florian Feuchtmayr  F.F. and R.B. contributed equally to this work.  Institute
for Quantum Optics, Ulm University, Albert-Einstein-Allee 11, 89081 Ulm,
Germany  Robert Berghaus  F.F. and R.B. contributed equally to this work.
Institute for Quantum Optics, Ulm University, Albert-Einstein-Allee 11, 89081
Ulm, Germany  Selene Sachero  Institute for Quantum Optics, Ulm University,
Albert-Einstein-Allee 11, 89081 Ulm, Germany  Gregor Bayer  Institute for
Quantum Optics, Ulm University, Albert-Einstein-Allee 11, 89081 Ulm, Germany
Niklas Lettner  Institute for Quantum Optics, Ulm University, Albert-Einstein-
Allee 11, 89081 Ulm, Germany  Center for Integrated Quantum Science and
Technology (IQst), Ulm University, Albert-Einstein-Allee 11, 89081 Ulm,
Germany  Richard Waltrich  Institute for Quantum Optics, Ulm University,
Albert-Einstein-Allee 11, 89081 Ulm, Germany  Patrick Maier  Institute for
Quantum Optics, Ulm University, Albert-Einstein-Allee 11, 89081 Ulm, Germany
Viatcheslav Agafonov  GREMAN, UMR 7347 CNRS, INSA-CVL, Tours University, 37200
Tours, France  Alexander Kubanek  [  Institute for Quantum Optics, Ulm
University, Albert-Einstein-Allee 11, 89081 Ulm, Germany  Center for
Integrated Quantum Science and Technology (IQst), Ulm University, Albert-
Einstein-Allee 11, 89081 Ulm, Germany

###### Abstract

Color centers in diamond, among them the negatively-charged germanium vacancy
(GeV-), are promising candidates for many applications of quantum optics such
as a quantum network. For efficient implementation, the optical transitions
need to be coupled to a single optical mode. Here, we demonstrate the transfer
of a nanodiamond containing a single ingrown GeV- center with excellent
optical properties to an open Fabry-Pérot microcavity by nanomanipulation
utilizing an atomic force microscope. Coupling of the GeV- defect to the
cavity mode is achieved, while the optical resonator maintains a high finesse
of $\mathcal{F}=7{,}700$ and a 48-fold spectral density enhancement is
observed. This article demonstrates the integration of a GeV- defect with a
Fabry-Pérot microcavity under ambient conditions with the potential to extend
the experiments to cryogenic temperatures towards an efficient spin-photon
platform.

††preprint: AIP/123-QED

The following article has been accepted by Applied Physics Letters. It is
found at https://doi.org/10.1063/5.0156787

With increasing advances in quantum technologies and the importance of secure
communication, there is great interest in developing a quantum network. Kimble
(2008) In such a network, the communication is based on quantum entanglement,
and by using quantum key distribution it provides guaranteed security. Scarani
_et al._ (2009); Pirandola  _et al._ (2020) However, to increase the range of
the network on a global scale, Wehner, Elkouss, and Hanson (2018) nodes are
required due to lossy photon channels that limit the range to about 100 km.
Jiang  _et al._ (2009)
Essential for the realization of such nodes in a scalable quantum network are
a quantum memory capable of storing and processing quantum information, Lim
_et al._ (2006) and high rates of coherent photons to establish entanglement
in communication schemes. For the interaction between photons and the quantum
memory, an efficient interface is necessary. Consequent challenges of
overcoming photon losses, maintaining coherence, attaining robustness and
enabling scalability can be tackled by using a quantum repeater as a node in
the quantum network. Briegel  _et al._ (1998) One approach to realizing such a
repeater is to address a single emitter to gain access to a quantum memory via
its spin. By placing the emitter inside a high quality optical resonator,
Vahala (2003); Janitz, Bhaskar, and Childress (2020) quantum state writing and
reading can be performed efficiently at high rates due to an enhanced
spontaneous emission rate. Purcell (1946)
Promising single emitter candidates for realizing a quantum network are color
centers, optically active defect centers in diamond. Ruf  _et al._ (2021a);
Chen, Zheludev, and Gao (2020) In order to couple them to the mode of a
microcavity diamond membranes (DMs) Häußler  _et al._ (2019a); Ruf  _et al._
(2021b) and nanodiamonds (NDs) Bayer  _et al._ (2022) are mainly used.
Currently, negatively-charged nitrogen vacancies (NV-) Doherty  _et al._
(2013) and silicon vacancies (SiV-) Rogers  _et al._ (2014) are the most
studied color centers. While NV- centers show good spin properties and great
strides have been made to entangle them, Pompili  _et al._ (2021) they show
poor optical access, e.g. a low emission in the zero phonon line (ZPL).
Jelezko and Wrachtrup (2006) In contrast, group IV defects, with their
interstitial doping atom in the diamond lattice, are promising alternative
emitters for this concept. Bradac  _et al._ (2019) Among them is the SiV-,
which compared to NV-, has superior optical properties with a high Debye-
Waller (DW) factor of 0.7. Bradac  _et al._ (2019) A drawback is the technical
overhead of mK temperatures, which is required for receiving good spin
properties. Lagomarsino  _et al._ (2021)
The negatively-charged germanium vacancy (GeV-) Iwasaki  _et al._ (2015);
Nahra  _et al._ (2021) is another group IV defect with a heavier impurity atom
than the SiV- center. The GeV- center exhibits a longer excited state lifetime
and a comparable DW factor of 0.3-0.6. Bradac  _et al._ (2019); Palyanov  _et
al._ (2015); Nahra  _et al._ (2021); Häußler  _et al._ (2017) At seven Kelvin,
the phonon-induced spin dephasing is exponentially suppressed resulting in
good spin properties already below one Kelvin; higher than for the SiV-
center. Janitz, Bhaskar, and Childress (2020); Bhaskar  _et al._ (2017) Beside
their implementation in diamond waveguides Siampour  _et al._ (2020); Bhaskar
_et al._ (2017) the GeV- defect was coupled to photonic resonators Bray  _et
al._ (2018); Kumar  _et al._ (2021) and to one tunable Fabry-Pérot (FP)
cavity. Høy Jensen  _et al._ (2020) With a cavity formed by the end-facet of
an optical fiber and a flat mirror with a bonded DM a spectral density
enhancement (SDE) of 31 was reported. Høy Jensen  _et al._ (2020)
In this work, we demonstrate the transfer of a 200 nm sized ND containing a
single GeV- defect with non-blinking and narrow emission to an optical open FP
microcavity with a high quality factor by utilizing the pick and place
technique with an atomic force microscope (AFM). We realized coupling of the
strong ZPL of the GeV- center to the cavity mode, while observing a 48-fold
SDE and maintaining a high finesse. In our approach, the ND with the GeV-
center is intrinsically aligned with the cavity field, making lateral scanning
of the cavity mirror redundant. Therefore, the system is passively very stable
with reduced technical overhead.

Figure 1: Characterizing GeV- center in NDs. (a) SEM image of the synthesized
NDs. (b) Distribution of the ZPL and FWHM of the acquired GeV- defects. (c) PL
spectrum with a coarse grating of 150 g/mm revealing the ZPL and PSB of the
selected ND GeV-I with inserted atomic structure of a GeV-. (d) PL with a fine
grating of 1200 g/mm showing the ZPL of ND GeV-I at (599.11 $\pm$ 0.03) nm and
a FWHM of (1.24 $\pm$ 0.03) nm under green excitation with (930 $\pm$ 20)
$\upmu$W. (e) Saturation curve of ND GeV-I with the saturation intensity
$I_{\textup{sat}}$=(0.15 $\pm$ 0.07) $\frac{\text{Mc}}{\text{s}}$ and
saturation power $P_{\textup{sat}}$=(3.0 $\pm$ 2.2) mW. (f) Off-resonant
autocorrelation measurement without background correction of ND GeV-I showing
strong anti-bunching of $g^{(2)}(0)$=0.11 $\pm$ 0.04 and an excited state
lifetime of $\tau_{\textup{LT}}$=(2.53 $\pm$ 0.20) ns at (60 $\pm$ 2)
$\upmu$W.

First, NDs with single GeV- defect centers were grown by high pressure high
temperature (HPHT) Palyanov  _et al._ (2016) synthesis under 1,450∘C and 8GPa.
In this process, detonation NDs, fluoroadamantane
(C${}_{\text{10}}$H${}_{\text{15}}$F) and hepta-fluoronaphtalene
(C${}_{\text{10}}$F${}_{\text{8}}$) were used as carbon sources. Germanium
triphenyl-chloride (GeC${}_{\text{18}}$H${}_{\text{15}}$Cl) was added as
doping component for germanium. Fig. 1(a) shows a scanning electron microscope
(SEM) image of the NDs of varying sizes after the synthesis. Subsequently, the
NDs were cleaned by acid boiling and diluted in ultrapure water. After an
ultrasonic bath for deagglomeration, the ND solution was drop-casted on a
sapphire bulk substrate.
The sample was characterized in a confocal microscope (NA = 0.9) at room
temperature with an excitation wavelength of 532 nm. NDs with emission from
GeV- centers showed a strong ZPL at a median wavelength of (602 $\pm$ 2) nm
and are presented in fig. 1(b). The observed spectral distribution of the ZPL
suggests strain in the NDs.Meesala  _et al._ (2018) With a mean FWHM of (1.3
$\pm$ 0.3) nm and a distribution between 0.8 - 2.2 nm the defects showed
narrow linewidths compared to reported values. Nahra  _et al._ (2021) The ND
with the identifier GeV-I had good optical properties and was selected for
further investigation. Its photoluminescence (PL) is displayed in figures
1(c,d). This color center demonstrated a high and stable emission rate in the
ZPL with a high quality factor of
$Q_{\textup{GeV}}=\lambda/\delta\lambda=483\,\pm\,12$. The spectrum shows a
small phonon sideband (PSB) with a DW factor above 0.6, which results from
different converging evaluations, see supplementary material. At 658 nm and
693 nm emission originated from an unrelated laser and the sapphire bulk,
respectively, which was not taken into account for the DW factor. In addition,
its saturation curve is illustrated in fig. 1(e) and was fitted with the
function

$I(P)=\frac{I_{\textup{sat}}\times P}{P+P_{\textup{sat}}}.$ (1)

The determined values for saturation intensity and saturation power subside
the presented data in Ref. Siampour  _et al._ , 2020. To evaluate the
properties of the emission signal from the emitter, we performed an
autocorrelation measurement under off-resonant excitation. The time resolved
correlation without any background correction and with the fitting model

$g^{(2)}(\tau)=1-A\times\exp\biggl{(}-\biggl{|}\frac{\tau}{\tau_{\textup{LT}}}\biggr{|}\biggr{)}$
(2)

can be seen in fig. 1(f), for other emitters see supplementary material. A
strong anti-bunching of $g^{(2)}(0)=0.11\,\pm\,0.04$ was observed. This shows
a high rate of emission arising from only a single GeV- center within the ND
under green excitation, making this a good candidate for quantum optics
experiments and the use as a single photon source. An excited state lifetime
of $\tau_{\textup{LT}}=(2.53\,\pm\,0.20)\,$ns at an excitation power of 60
$\upmu$W was determined, in agreement with literature. Iwasaki  _et al._
(2015)

Figure 2: Assembly of an FP-resonator with a GeV- center containing ND. (a)
Microscopic image of the mirror containing the curved structure (circled
white) intended for the ND. (b) Confocal scan and AFM images of NDs showing
GeV-I. (c) Schematic illustration of the pick and place technique for placing
NDs inside a mirror structure using the AFM cantilever. (d) Tilt subtracted
AFM image of the spherical mirror containing the transferred ND GeV-I in its
center with respective cross sections in x- and y-directions. (e) Confocal
scan image of the spherical mirror containing the same ND in its center after
transfer. (f) PL of ND GeV-I in the mirror structure with a ZPL at (599.25
$\pm$ 0.03) nm and a FWHM of (1.10 $\pm$ 0.04) nm using an excitation power of
(20 $\pm$ 2) $\upmu$W.

Next, the GeV- center containing ND was integrated in the field of the
hemispherical FP cavity by placing it in the center of the curved mirror. The
concave mirror was fabricated by a CO2 laser ablation process of a SiO2
substrate. By creating multiple mirror structures on one substrate, a
versatile mirror array could be fabricated. Varying the pulse lengths results
in different structure sizes, as shown in fig. 2(a). Interferometric analysis
of the structure intended for the ND revealed a depth of 2.7 $\upmu$m and a
diameter of 26.4 $\upmu$m. By coating the structured substrate a distributed
Bragg reflector (DBR) was formed, yielding a low transmission between
595-690nm with a minimum of $T\,<\,$310 ppm at 601 nm. At this wavelength the
mirror coating forms a field antinode around 150 nm in air and 62 nm in
diamond away from the mirror surface.
Before the mirror integration, the emitter could be identified by comparing
the confocal scan with the AFM image in fig. 2(b). Using the AFM based pick
and place technique Fehler  _et al._ (2021); Häußler  _et al._ (2019b)
nanomanipulation of ND GeV-I was achieved by transferring the emitter to the
center of the desired structure, see fig. 2(c).
The mirror containing the ND was scanned with the AFM in fig. 2(d), whereby
the cross sections reveal the size of the ND of (190 x 180 x 130) nm. It was
also determined that the structure used here is elliptical, with radii of
curvature RoC${}_{\textup{X}}=(21.48\,\pm\,0.01)\,\upmu$m and
RoC${}_{\textup{Y}}=(28.94\,\pm\,0.01)\,\upmu$m. With the help of a confocal
microscope, it could be confirmed that the optical properties were preserved.
Fig. 2(e) shows the spherical shape of the mirror due to minor fluorescence of
the coating. Pollution and other impurities are visible at the edge of the
structure, outside the relevant cavity mode. In the center, a peak of
fluorescence originating from ND GeV-I is observable, verified by the PL
spectrum in fig. 2(f). The surrounding area of the ND shows a bleached
background fluorescence due to previous laser scans with 532 nm.
The spectral density (SD) of the spectrum

$\textup{SD}=\frac{\textup{Photoncounts}}{\textup{FWHM}\times P}$ (3)

could be extracted by using the integrated photoncounts of the Lorentzian fit
and the excitation power that lay within the linear range of the saturation
curve. The spectrum in figure 2(f) yields a SD of $(12.7\,\pm\,1.0)$ c/(s GHz
mW), which could be compared to the free space emitter that demonstrates a SD
of $(2.03\,\pm\,0.08)$ c/(s GHz mW) in fig. 1(d). The measurements were
performed on different confocal microscopes, so differences in the setup
efficiencies had to be taken into account. With the reflected laser signal
from the curved mirror and a different excitation beam width at the emitter, a
relative excitation efficiency of
$\tilde{\eta}_{\textup{Con}}^{\textup{exc}}=\eta_{\textup{Con2}}^{\textup{exc}}/\eta_{\textup{Con1}}^{\textup{exc}}=2.3\,\pm\,0.2$
followed. Different optical components led to a relative detection efficiency
of $\tilde{\eta}_{\textup{Con}}^{\textup{det}}=1.3\,\pm\,0.2$. Considering
these efficiencies and comparing the resulting SD from free space emission and
the ND placed in the curved mirror suggested a spectral density enhancement of
$\textup{SDE}=\textup{SD}_{\textup{Con2}}/(\textup{SD}_{\textup{Con1}}\times\tilde{\eta}_{\textup{Con}}^{\textup{det}}\times\tilde{\eta}_{\textup{Con}}^{\textup{exc}})$$=2.1\,\pm\,0.4$,
see supplementary material for complete calculations. This enhancement is
attributed to the reflected fluorescence signal of the curved mirror,
indicating a beneficial position of the emitter. The 2-fold SDE suggesting
efficient emitter coupling indicates that the emitter is located near the
electric field maximum, which is estimated to be at a distance of 62 nm from
the mirror surface.

Figure 3: Characterizing the assembled cavity with a GeV- ND. (a) Schematic
illustration of the operated optical microcavity setup. (b) Exemplary finesse
measurement showing the reflected laser signal over the z piezo voltage of the
flat mirror corresponding to the cavity length. (c) Possible finesse values of
the microcavity determined by the transmission of the coating compared to the
measurement with the ND inside the cavity. (d) PL of resonance n = 51 of the
optical microcavity with an excitation wavelength of 587.78 nm and excitation
power of (80 $\pm$ 12) $\upmu$W. The spectrum (spectrometer resolution of
$0.03$ nm) shows a cavity modulated signal of the GeV- with a peak of emission
at (599.20 $\pm$ 0.03) nm.

For characterizing the GeV- center in the cavity, the curved mirror containing
the transferred ND and a second plane mirror with the same substrate and
coating were assembled and formed the resonator, as illustrated in fig. 3(a).
After passing a laser clean up filter (600 SP) and the reflection of a
dichroic mirror, the excitation laser light (Sirah Matisse 2 DS Dye laser:
linewidth below $100$ kHz at 600 nm) of a singlemode fiber (IVG Fiber: SM
CU600PSC) was coupled into one of the cavity structures with the help of a
lens (Asphericon: AHL10-08-P-U-780, f = 8mm, NA = 0.55) mounted to
nanopositioners in x, y and z (Attocube ANP series). An additional
nanopositioner with z piezo enabled tunability of the cavity length by
scanning the plane mirror. The cavity signal and the reflected light passed
the lens, the dichroic mirror, a filter (605/15 BP) and were collected by a
multimode fiber (Thorlabs: MM FG050LGA) in order to guide it to an APD or
spectrometer.
The cavity finesse was extracted from the reflection signal by scanning the
piezo of the plane mirror, as seen in fig. 3(b). The obtained values were
compared to the theoretical possible finesse given by the coating limit in
fig. 3(c). The cavity maintained a high finesse close to the coating limit.
One could receive a maximum finesse of $\mathcal{F}>10{,}300$ at 601 nm, while
in the measurements a value of $\mathcal{F}=7{,}700\pm 1{,}800$ was achieved.
Comparing the finesse at 648.6 nm before $\mathcal{F}=4{,}100\pm 600$ and
after $\mathcal{F}=4{,}000\pm 800$ placing the ND in the cavity showed no
significant losses due to reduced scattering of the small sized nanoparticle.
With an excitation wavelength of 587.8 nm the cavity modulated ZPL signal of
the single GeV- center was detected, as shown in fig. 3(d). With the signal of
the emitter at 599.2 nm and the excitation wavelength at neighboring
resonances of the resonator, a cavity length of $L=(15.72\,\pm\,0.06)\,\upmu$m
was determined. Considering the RoC of $(25\,\pm\,5)\,\upmu$m, this resulted
in a beam waist on the flat side of $\omega_{0}=(1.5\,\pm\,0.2)\,\upmu$m and
on the curved side of $\omega(L)=(2.5\,\pm\,0.6)\,\upmu$m. From this a mode
volume of
$V=\frac{\pi}{4}L\omega_{0}^{2}=(140\,\pm\,40)\,\lambda_{\textup{Las}}^{3}$
followed and with the finesse of $\mathcal{F}=5{,}000\,\pm\,1{,}200$ at the
emitter wavelength the quality factor of the cavity was evaluated to
$Q_{\textup{Cav}}=\,260{,}000\,\pm\,60{,}000$. Finally, these parameters of
the cavity enabled calculating the Purcell-factor with

$\mathcal{P}=\frac{3}{4\pi^{2}}\bigg{(}\frac{\lambda}{n}\bigg{)}^{3}\frac{Q_{\textup{Cav}}}{V},$
(4)

and the refraction index $n_{\textup{air}}=1$, giving the values
$\mathcal{P}_{\textup{flat}}=152\,\pm\,56$ on the flat side and
$\mathcal{P}_{\textup{curv}}=57\,\pm\,29$ on the curved side.
However, this enhancement is only achieved for
$Q_{\textup{Cav}}<Q_{\textup{GeV}}$, which was not the case for our room
temperature experiments due to thermal broadening of the emitter.Albrecht  _et
al._ (2013) Therefore, the corrected formula for the Purcell effect is given
by

$\mathcal{P}^{*}=\mathcal{P}\times\frac{Q_{\textup{GeV}}}{Q_{\textup{Cav}}},$
(5)

yielding the corrected values $\mathcal{P}_{\textup{flat}}^{*}=$
0.28$\,\pm\,$0.03 on the flat side and $\mathcal{P}_{\textup{curv}}^{*}=$
0.12$\,\pm\,$0.02 on the curved side.
In order to determine the SDE experimentally, we extracted the SD
$(470\,\pm\,120)\,$c/(s GHz mW) of the emitter in the microcavity from fig.
3(d) and compared it to the free space emitter from fig. 1(d). We evaluated
the altered excitation efficiency
$\tilde{\eta}_{\textup{Cav}}^{\textup{exc}}=11\,\pm\,4$ in relation to the
confocal microscope setup, which was caused by the finesse of the cavity, a
more efficient excitation wavelength Häußler  _et al._ (2017) and a wider
excitation beam at the emitter. The detection efficiency was corrected by a
factor of $\tilde{\eta}_{\textup{Cav}}^{\textup{det}}=0.45\,\pm\,0.07$ due to
different optical components. This resulted in a (48 $\pm$ 20)-fold SDE, see
supplementary material for complete calculations. Considering that the Purcell
effect is limited at room temperature, the enhancement can be attributed to
cavity funneling.
In conclusion, we demonstrated the assembly of a tunable and versatile cavity
system containing a single GeV- inside one ND. The ingrown GeV- defect center
showed strong, stable and narrow emission in the ZPL with a small PSB yielding
a DW of above 0.6. With a $g^{(2)}(0)$ value of 0.11 $\pm$ 0.04 it proved to
be a high purity single photon source. The small size of the ND enabled
integration into the optical resonator mode without significant scattering
losses, maintaining a high quality factor $Q_{\textup{Cav}}$ above
$250{,}000$. The nanomanipulation capabilities allowed highly accurate
alignment of the pre-characterized ND with GeV- center to the cavity field.
With this integration into the open, fully tunable FP microcavity
photoluminescence measurements could be performed under off-resonant
excitation at room temperature, taking advantage of the single emitter
emission. We calculated a 48-fold SDE of the emitter and retained a high
finesse of $\mathcal{F}=7{,}700$ including the nanoparticle.

The passively stable platform, with a mode volume V of
$(140\,\pm\,40)\,\lambda_{\textup{Las}}^{3}$, does not require additional
lateral scanning of the cavity mirror, profits from a low thermal expansion
due to the main setup material choice of titanium, and allows for remote
realignment with the nanopositioners. Altogether this offers robustness even
at cryogenic temperatures. Bayer  _et al._ (2022) Cooling this platform to low
temperatures, as shown in earlier work with SiV-centers, Bayer  _et al._
(2022) offers a decreased thermal broadening of the emitter towards the FT
limit, which suggests an increased Purcell effect of $\mathcal{P}>50$. In
combination with improved coherence of the emission into the ZPL, this results
in a higher spectral enhancement of the signal out of the cavity. In order to
receive access to a quantum memory, spin access would be a desirable next goal
to establish an efficient spin photon interface. With those advances this
hybrid microcavity system is a robust and promising approach for many quantum
optic applications such as a quantum repeater node.

## Supplementary Material

For additional information on the DW factor, additional autocorrelation
measurements for other NDs, nanomanipulation of the ND, and comprehensive
calculations of the spectral density enhancement, please refer to our
supplementary material.

###### Acknowledgements.

The authors thank V.A. Davydov for synthesis and processing of the ND
material. The authors thank Jan Schimmel for the fabrication of the CO2
structures and Lukas Antoniuk for experimental support. Funding by the German
Federal Ministry of Education and Research (BMBF) within the project QR.X
(16KISQ006) is gratefully acknowledged. S.S. acknowledges support of the Marie
Curie ITN project LasIonDef (GA n.956387). N.L. acknowledges support of IQst.
Most measurements were performed on the basis of the Qudi software suite.
Binder  _et al._ (2017)

## DATA AVAILABILITY

The data that support the findings of this study are available from the
corresponding authors upon reasonable request.

## AUTHOR DECLARATIONS

### Conflict of Interest

The authors have no conflicts to disclose.

### Author Contributions

Florian Feuchtmayr and Robert Berghaus contributed equally to this work.
Florian Feuchtmayr: Investigation (equal); Data Curation (lead); Visualization
(lead); Writing – original draft (lead); Writing – review and editing (equal).
Robert Berghaus: Conceptualization (supporting); Methodology (equal);
Investigation (equal); Data Curation (supporting); Visualization (supporting);
Writing – original draft (supporting); Writing – review and editing (equal).
Selene Sachero: Methodology (equal); Investigation (equal). Gregor Bayer:
Conceptualization (supporting); Methodology (equal). Niklas Lettner:
Investigation (equal). Richard Waltrich: Methodology (supporting). Patrick
Maier: Methodology (supporting). Viatcheslav Agafonov: Methodology (equal).
Alexander Kubanek: Conceptualization (lead); Supervision (lead); Funding
acquisition (lead); Writing – review and editing (supporting).

## References

## References

  * Kimble (2008) H. J. Kimble, “The quantum internet,” Nature 453, 1023–1030 (2008).
  * Scarani  _et al._ (2009) V. Scarani, H. Bechmann-Pasquinucci, N. J. Cerf, M. Dušek, N. Lütkenhaus, and M. Peev, “The security of practical quantum key distribution,” Reviews of modern physics 81, 1301 (2009).
  * Pirandola  _et al._ (2020) S. Pirandola, U. L. Andersen, L. Banchi, M. Berta, D. Bunandar, R. Colbeck, D. Englund, T. Gehring, C. Lupo, C. Ottaviani, J. L. Pereira, M. Razavi, J. S. Shaari, M. Tomamichel, V. C. Usenko, G. Vallone, P. Villoresi, and P. Wallden, “Advances in quantum cryptography,” Adv. Opt. Photon. 12, 1012–1236 (2020).
  * Wehner, Elkouss, and Hanson (2018) S. Wehner, D. Elkouss, and R. Hanson, “Quantum internet: A vision for the road ahead,” Science 362, eaam9288 (2018).
  * Jiang  _et al._ (2009) L. Jiang, J. M. Taylor, K. Nemoto, W. J. Munro, R. Van Meter, and M. D. Lukin, “Quantum repeater with encoding,” Physical Review A 79, 032325 (2009).
  * Lim  _et al._ (2006) Y. L. Lim, S. D. Barrett, A. Beige, P. Kok, and L. C. Kwek, “Repeat-until-success quantum computing using stationary and flying qubits,” Physical Review A 73, 012304 (2006).
  * Briegel  _et al._ (1998) H.-J. Briegel, W. Dür, J. I. Cirac, and P. Zoller, “Quantum repeaters: The role of imperfect local operations in quantum communication,” Phys. Rev. Lett. 81, 5932–5935 (1998).
  * Vahala (2003) K. J. Vahala, “Optical microcavities,” nature 424, 839–846 (2003).
  * Janitz, Bhaskar, and Childress (2020) E. Janitz, M. K. Bhaskar, and L. Childress, “Cavity quantum electrodynamics with color centers in diamond,” Optica 7, 1232–1252 (2020).
  * Purcell (1946) E. M. Purcell, “Proceedings of the american physical society, b10. spontaneous emission probabilities at radio frequencies,” Phys. Rev 69, 674 (1946).
  * Ruf  _et al._ (2021a) M. Ruf, N. H. Wan, H. Choi, D. Englund, and R. Hanson, “Quantum networks based on color centers in diamond,” Journal of Applied Physics 130, 070901 (2021a).
  * Chen, Zheludev, and Gao (2020) D. Chen, N. Zheludev, and W.-b. Gao, “Building blocks for quantum network based on group-iv split-vacancy centers in diamond,” Advanced Quantum Technologies 3, 1900069 (2020).
  * Häußler  _et al._ (2019a) S. Häußler, J. Benedikter, K. Bray, B. Regan, A. Dietrich, J. Twamley, I. Aharonovich, D. Hunger, and A. Kubanek, “Diamond photonics platform based on silicon vacancy centers in a single-crystal diamond membrane and a fiber cavity,” Physical Review B 99, 165310 (2019a).
  * Ruf  _et al._ (2021b) M. Ruf, M. Weaver, S. van Dam, and R. Hanson, “Resonant excitation and purcell enhancement of coherent nitrogen-vacancy centers coupled to a fabry-perot microcavity,” Phys. Rev. Appl. 15, 024049 (2021b).
  * Bayer  _et al._ (2022) G. Bayer, R. Berghaus, S. Sachero, A. B. Filipovski, L. Antoniuk, N. Lettner, R. Waltrich, M. Klotz, P. Maier, V. Agafonov, and A. Kubanek, “A quantum repeater platform based on single siv- centers in diamond with cavity-assisted, all-optical spin access and fast coherent driving,” arXiv (2022).
  * Doherty  _et al._ (2013) M. W. Doherty, N. B. Manson, P. Delaney, F. Jelezko, J. Wrachtrup, and L. C. Hollenberg, “The nitrogen-vacancy colour centre in diamond,” Physics Reports 528, 1–45 (2013).
  * Rogers  _et al._ (2014) L. J. Rogers, K. D. Jahnke, M. W. Doherty, A. Dietrich, L. P. McGuinness, C. Müller, T. Teraji, H. Sumiya, J. Isoya, N. B. Manson, and F. Jelezko, “Electronic structure of the negatively charged silicon-vacancy center in diamond,” Phys. Rev. B 89, 235101 (2014).
  * Pompili  _et al._ (2021) M. Pompili, S. L. N. Hermans, S. Baier, H. K. C. Beukers, P. C. Humphreys, R. N. Schouten, R. F. L. Vermeulen, M. J. Tiggelman, L. dos Santos Martins, B. Dirkse, S. Wehner, and R. Hanson, “Realization of a multinode quantum network of remote solid-state qubits,” Science 372, 259–264 (2021).
  * Jelezko and Wrachtrup (2006) F. Jelezko and J. Wrachtrup, “Single defect centres in diamond: A review,” physica status solidi (a) 203, 3207–3225 (2006).
  * Bradac  _et al._ (2019) C. Bradac, W. Gao, J. Forneris, M. E. Trusheim, and I. Aharonovich, “Quantum nanophotonics with group iv defects in diamond,” Nature communications 10, 1–13 (2019).
  * Lagomarsino  _et al._ (2021) S. Lagomarsino, A. Flatae, H. Kambalathmana, F. Sledz, L. Hunold, N. Soltani, P. Reuschel, S. Sciortino, N. Gelli, M. Massi, _et al._ , “Creation of silicon-vacancy color centers in diamond by ion implantation,” Frontiers in Physics 8, 601362 (2021).
  * Iwasaki  _et al._ (2015) T. Iwasaki, F. Ishibashi, Y. Miyamoto, S. Kobayashi, T. Miyazaki, K. Tahara, K. D. Jahnke, L. J. Rogers, B. Naydenov, F. Jelezko, _et al._ , “Germanium-vacancy single color centers in diamond,” Scientific reports 5, 1–7 (2015).
  * Nahra  _et al._ (2021) M. Nahra, D. Alshamaa, R. Deturche, V. Davydov, L. Kulikova, V. Agafonov, and C. Couteau, “Single germanium vacancy centers in nanodiamonds with bulk-like spectral stability,” AVS Quantum Science 3 (2021).
  * Palyanov  _et al._ (2015) Y. N. Palyanov, I. N. Kupriyanov, Y. M. Borzdov, and N. V. Surovtsev, “Germanium: a new catalyst for diamond synthesis and a new optically active impurity in diamond.” Sci Rep 5, 14789 (2015).
  * Häußler  _et al._ (2017) S. Häußler, G. Thiering, A. Dietrich, N. Waasem, T. Teraji, J. Isoya, T. Iwasaki, M. Hatano, F. Jelezko, A. Gali, _et al._ , “Photoluminescence excitation spectroscopy of siv- and gev- color center in diamond,” New Journal of Physics 19, 063036 (2017).
  * Bhaskar  _et al._ (2017) M. K. Bhaskar, D. D. Sukachev, A. Sipahigil, R. E. Evans, M. J. Burek, C. T. Nguyen, L. J. Rogers, P. Siyushev, M. H. Metsch, H. Park, F. Jelezko, M. Lončar, and M. D. Lukin, “Quantum nonlinear optics with a germanium-vacancy color center in a nanoscale diamond waveguide,” Phys. Rev. Lett. 118, 223603 (2017).
  * Siampour  _et al._ (2020) H. Siampour, O. Wang, V. A. Zenin, S. Boroviks, P. Siyushev, Y. Yang, V. A. Davydov, L. F. Kulikova, V. N. Agafonov, A. Kubanek, N. A. Mortensen, F. Jelezko, and S. I. Bozhevolnyi, “Ultrabright single-photon emission from germanium-vacancy zero-phonon lines: deterministic emitter-waveguide interfacing at plasmonic hot spots,” Nanophotonics 9, 953–962 (2020).
  * Bray  _et al._ (2018) K. Bray, B. Regan, A. Trycz, R. Previdi, G. Seniutinas, K. Ganesan, M. Kianinia, S. Kim, and I. Aharonovich, “Single crystal diamond membranes and photonic resonators containing germanium vacancy color centers,” ACS Photonics 5, 4817–4822 (2018).
  * Kumar  _et al._ (2021) S. Kumar, C. Wu, D. Komisar, Y. Kan, L. F. Kulikova, V. A. Davydov, V. N. Agafonov, and S. I. Bozhevolnyi, “Fluorescence enhancement of a single germanium vacancy center in a nanodiamond by a plasmonic bragg cavity,” The Journal of Chemical Physics 154, 044303 (2021).
  * Høy Jensen  _et al._ (2020) R. Høy Jensen, E. Janitz, Y. Fontana, Y. He, O. Gobron, I. P. Radko, M. Bhaskar, R. Evans, C. D. Rodríguez Rosenblueth, L. Childress, A. Huck, and U. Lund Andersen, “Cavity-enhanced photon emission from a single germanium-vacancy center in a diamond membrane,” Phys. Rev. Appl. 13, 064016 (2020).
  * Palyanov  _et al._ (2016) Y. N. Palyanov, I. N. Kupriyanov, Y. M. Borzdov, A. F. Khokhryakov, and N. V. Surovtsev, “High-pressure synthesis and characterization of ge-doped single crystal diamond,” Crystal Growth & Design 16, 3510–3518 (2016).
  * Meesala  _et al._ (2018) S. Meesala, Y.-I. Sohn, B. Pingault, L. Shao, H. A. Atikian, J. Holzgrafe, M. Gündoğan, C. Stavrakas, A. Sipahigil, C. Chia, R. Evans, M. J. Burek, M. Zhang, L. Wu, J. L. Pacheco, J. Abraham, E. Bielejec, M. D. Lukin, M. Atatüre, and M. Lončar, “Strain engineering of the silicon-vacancy center in diamond,” Phys. Rev. B 97, 205444 (2018).
  * Fehler  _et al._ (2021) K. G. Fehler, L. Antoniuk, N. Lettner, A. P. Ovvyan, R. Waltrich, N. Gruhler, V. A. Davydov, V. N. Agafonov, W. H. P. Pernice, and A. Kubanek, “Hybrid quantum photonics based on artificial atoms placed inside one hole of a photonic crystal cavity,” ACS Photonics 8, 2635–2641 (2021).
  * Häußler  _et al._ (2019b) S. Häußler, L. Hartung, K. G. Fehler, L. Antoniuk, L. F. Kulikova, V. A. Davydov, V. N. Agafonov, F. Jelezko, and A. Kubanek, “Preparing single siv- center in nanodiamonds for external, optical coupling with access to all degrees of freedom,” New Journal of Physics 21, 103047 (2019b).
  * Albrecht  _et al._ (2013) R. Albrecht, A. Bommer, C. Deutsch, J. Reichel, and C. Becher, “Coupling of a single nitrogen-vacancy center in diamond to a fiber-based microcavity,” Physical review letters 110, 243602 (2013).
  * Binder  _et al._ (2017) J. M. Binder, A. Stark, N. Tomek, J. Scheuer, F. Frank, K. D. Jahnke, C. Müller, S. Schmitt, M. H. Metsch, T. Unden, T. Gehring, A. Huck, U. L. Andersen, L. J. Rogers, and F. Jelezko, “Qudi: A modular python suite for experiment control and data processing,” SoftwareX 6, 85–90 (2017).


# THE-X: Privacy-Preserving Transformer Inference with
Homomorphic Encryption

Tianyu Chen♣ ♠ $\star$, Hangbo Bao♥, Shaohan Huang♥, Li Dong♥, Binxing Jiao♠,
Daxin Jiang♠, Haoyi Zhou♣ ♠, Jianxin Li♣ ♠ ✉, Furu Wei♥
BDBC, Beihang University, China♣
SKLSDE Lab, Beihang University, China♠
Microsoft Research♥ NLP Group, Microsoft STCA♠
{tianyuc<EMAIL_ADDRESS>
{t-habao,shaohanh, lidong1, binxjia, djiang<EMAIL_ADDRESS>
###### Abstract

As more and more pre-trained language models adopt on-cloud deployment, the
privacy issues grow quickly, mainly for the exposure of plain-text user data
(e.g., search history, medical record, bank account). Privacy-preserving
inference of transformer models is on the demand of cloud service users. To
protect privacy, it is an attractive choice to compute only with ciphertext in
homomorphic encryption (HE). However, enabling pre-trained models inference on
ciphertext data is difficult due to the complex computations in transformer
blocks, which are not supported by current HE tools yet. In this work, we
introduce THE-X, an approximation approach for transformers, which enables
privacy-preserving inference of pre-trained models developed by popular
frameworks. THE-X proposes a workflow to deal with complex computation in
transformer networks, including all the non-polynomial functions like GELU,
softmax, and LayerNorm. Experiments reveal our proposed THE-X can enable
transformer inference on encrypted data for different downstream tasks, all
with negligible performance drop but enjoying the theory-guaranteed privacy-
preserving advantage.††${\star}$ Contribution during internship at MSRA. ✉The
corresponding author is Jianxin Li<EMAIL_ADDRESS>
##  1 Introduction

Accompanying the revolution of pre-trained models in many NLP applications,
such as sentiment analysis Xu et al. (2019a), question answering Yang et al.
(2019b), information retrieval Yang et al. (2019c), and text generation Raffel
et al. (2020), many related technologies have been deployed on the cloud to
process user data from personal customers, small businesses, and large
enterprises by industrial service providers. However, the convenience of the
on-cloud pre-training technology also comes with a series of privacy
challenges due to the sensitive nature of user data. For example, the input
text or even text vector representations in user requests can leak private
information, which may cause the specific user to be identified Schwartz and
Solove (2011); Zhu and Han (2020). This lack of privacy guarantees may impede
privacy-conscious users from releasing their data to service providers. Thus,
service providers may suffer from the deficiency of evolving models with user
data. Besides, unintended data disclosure and other privacy breaches may
result in litigation, fines, and reputation damages for service providers.
These concerns spark our proposal of THE-X, to enable privacy-preserving
inference of transformer.

Figure 1: An overview of our THE-X. The transformer-based model could
inference on encrypted data with our THE-X, enabling theory-guaranteed privacy
protection for users.

Specifically, we identify two challenges for the privacy-preserving inference
of pre-trained models. The first challenge is how to protect users’ plain text
data from access by third-party service providers. (e.g., the clinic record or
shopping history). Prior work has applied Differential Privacy (DP) Dwork et
al. (2006) and its variants to address similar privatization issues -
originally for statistical databases and more recently for DL Abadi et al.
(2016) and NLP Qu et al. (2021); Basu et al. (2021b); Fernandes et al. (2019);
Lyu et al. (2020); Basu et al. (2021a). However, this solution may suffer from
eavesdropping attackers. A handful of research Zhu and Han (2020); Zhao et al.
(2020) demonstrated it possible to recover raw data from gradient leakage.
Also, privacy protection could never be theory-guaranteed. The second
challenge is the performance concern, recent works like TextHide Huang et al.
(2020) and FedNLP Lin et al. (2021) leverages the federated learning Yang et
al. (2019a) to train model on encrypted data, at cost of considerable
performance dropping. Focusing on the privacy of training data, they have not
fully explored privacy-preserving inference.

To solve the concerns above, we depict one practice of privacy-preserving
inference in Figure 1, where a fine-tuned language model could be converted
into the cloud service mode with THE-X, and process users’ data with its eyes
blind. During inference, the content of the user query is anonymous to the
transformer model. The results of computation are also ciphertext, which only
can be decrypted by the user’s private key.

In addition, we need a theory-guaranteed encryption solution like the
homomorphic encryption (HE) Gentry (2009) to convince both service providers
and users of the privacy security in production scenarios. The semantic
security of HE is guaranteed by lattice-based cryptography, and the HE
computation results on ciphertext could be decrypted to the same results in
plaintext, preventing performance reduction cost. The basic idea of
homomorphic encryption is to perform computations on encrypted data without
first decrypting it, which could fully ensure privacy in cloud-serving
scenarios. It allows user data to be encrypted and out-sourced to commercial
cloud environments for processing.

However, due to the complex operations (e.g., GELU activation) in transformer-
based models, the popular partially homomorphic encryption solution, which
only supports addition or multiplication, can not easily be adapted into
scenarios of pre-trained models. Based on HE transformer backend Boemer et al.
(2019b, a, 2020), we designed a series of approximation components to fulfill
the whole inference pipeline of the mainstream transformer backbone. We
evaluate THE-X for BERT-tiny on the GLUE benchmark Wang et al. (2019) and the
CONLL2003 task Tjong Kim Sang and De Meulder (2003). Our results show that
THE-X can achieve the privacy-preserving inference with the averaged
performance reduction of only 1.49%.

Our contributions include:

  * •

We are the first work to explore the privacy-preserving transformer inference
with HE.

  * •

We design a practical and effective approximation workflow for converting
transformer-based models into a function that consists of fully HE operations.

  * •

A thorough set of experiments confirms the negligible performance reduction
with our proposed THE-X approximation.

##  2 Background

###  2.1 Security and Privacy Concern of Pre-trained Models

Pre-trained models like BERT Devlin et al. (2019) and GPT-3 Brown et al.
(2020) rely heavily on the use of plain text data to get human-like
performance. Despite the remarkable achievements of pre-trained models, these
state-of-the-art models can not directly answer some sensitive use cases,
including the medical record Christoph et al. (2015), search history Shen et
al. (2007) and other personally identifiable information (PII).

To avoid the direct computation on plain-text data, recent works like TextHide
Huang et al. (2020) and DP-finetuning Kerrigan et al. (2020) introduce the
classical federated learning and differential privacy (DP) to protect the
sensitive data. However, TextHide Huang et al. (2020) can only be applied to
sentence-level tasks. Due to the mix-up operation, TextHide fails to model
token-level tasks like named entity recognition or semantic role labelling.
DP-finetuning would greatly sacrifice the performance of fine-tuned model by
20% perplexity for a generation model like GPT-2.

###  2.2 Practical Homomorphic Encryption

The classic definition of homomorphic encryption is a form of encryption that
permits users to perform computations on its encrypted data without first
decrypting it. These computations results are retained in an encrypted form,
which could be decrypted into identical output produced by the same
computations on the unencrypted data. Let $F$ be a function or the entire pre-
trained model, $E$ as an encryption function, $D$ as a decryption function.
Then for any allowed plain text input $x$, we have:

$F(x)=D(g(E(x)),$ (1)

where $g$ is a constructed function to play the same role of function $F$,
except on encrypted data. Figure 1 shows how a user performs inference using a
cloud-deployed pre-trained model which is not trusted. First, the pre-trained
model receives a ciphertext encrypted by the user private key and performs
inference function $g$ on the ciphertext. Then, the server will send an
encrypted result to the user, which can only be decrypted by the user key. At
no point does the cloud service provider gain access to the plain text.

Figure 2: The Approximation Workflow of THE-X. To replace the non-polynomial
operations, we split the fine-tuning stage into several subphases. Given a
pre-trained checkpoint, we drop the pooler of the pre-trained model and
replace softmax and GeLU. Afterward, we follow the standard fine-tuning for
classification or regression tasks. We add LayerNorm approximation into the
fine-tuned model and distill knowledge from original LN layers. After dropping
the original LN, we convert the model into fully HE-supported ops with the HE
transformer.

The Intel HE transformer for nGraph Boemer et al. (2019b, a) is a Homomorphic
Encryption backend to the deep learning models. Currently, it supports the
CKKS Cheon et al. (2017) encryption scheme, implemented by the Simple
Encrypted Arithmetic Library (SEAL) SEAL  from Microsoft Research. It is a
research tool to demonstrate the feasibility of HE on deep learning.

###  2.3 Challenges of Transformer Inference with HE

Some HE schemes only support a single algebraic operation, such as addition or
multiplication. These are known as "partially homomorphic" schemes (PHE).
Other schemes, called "fully homomorphic"(FHE), support two such as addition
and multiplication. Note that composing addition and multiplication suffices
to construct polynomial functions, and hence polynomial approximations to non-
polynomial functions such as GELU Hendrycks and Gimpel (2016) or LayerNorm Xu
et al. (2019b). Notably, this limitation prevents the exact computation of any
comparison-based operations such as Max, Min, as well as common functions such
as exponential or sigmoid. Finally, "leveled homomorphic" schemes (LHE)
support addition and multiplication, only up to a fixed computational depth.

##  3 THE-X: Formal Description

There are two core ideas in THE-X. The first one is to incorporate the user
device into the HE inference, and the second is using "simplified computation"
to approximate the non-polynomial functions.

In the following, we will describe how to enable homomorphic encryption of
transformer-based models with THE-X.

###  3.1 Approximation Workflow

Data: labeled task data $\mathcal{D}$.

Input: pre-trained Transformer model $\mathcal{M}$, softmax estimation model
$\mathcal{S}$.

1$\widehat{\mathcal{M}}\leftarrow\mathcal{M}\odot(\mathcal{S},ReLU)$.

// replace GELU and Softmax

2while  _not done_ do

3 sample batches $(x_{i},y_{i})$ from $\mathcal{D}$,

4 let $(x_{i},y_{i})$ optimize $\widehat{\mathcal{M}}$ with $\mathcal{S}$
frozen.

end while

5$\tilde{\mathcal{M}}\leftarrow\widehat{\mathcal{M}}\oplus\tilde{\mathcal{N}}$.

// add the layernorm approximation

6while  _not done_ do

7 sample batches $(x_{i},y_{i})$ from $\mathcal{D}$,

8 freeze the parameters of $\tilde{\mathcal{M}}$ except $\tilde{\mathcal{N}}$.

9 compute $k$-th layernorm output $O_{k}$, $\tilde{O_{k}}$.

10 compute loss $\ell_{k}$ = MSELoss($O_{k}$, $\tilde{O_{k}}$).

11 update $\tilde{\mathcal{N}}$ with loss $\mathcal{L}=\sum_{k}{\ell_{k}}$.

end while

12$\bar{\mathcal{M}}\leftarrow\tilde{\mathcal{M}}\ominus\mathcal{N}$.

// discard the origin layernorm

return $\bar{\mathcal{M}}$.

Algorithm 1 Approximation Workflow

First, we present the approximation workflow of THE-X, which consists of two
stages: Standard Finetuning and LN Distill as depicted in Figure 2. Given a
pre-trained model $\mathcal{M}$ and corresponding downstream data, we aim to
produce a fully HE supported $\bar{\mathcal{M}}$ which is fine-tuned and ready
for deployment.

The two-stage optimization of algorithm 1 aims to find the best approximation
checkpoint. For computation efficiency, pre-trained models can also be fine-
tuned together with the layernorm approximation, and it needs only a single
optimization loop. We will discuss the schedule of the different approximation
workflow in Sec 4.6. There are three major non-polynomial functions in the
transformer block, where we will study in detail.

####  3.1.1 Gaussion Error Linear Units (GLEU)

With a computation of Gaussian error, Gaussian Error Linear Units (GLEUs)
Hendrycks and Gimpel (2016) is not suitable to serve as an active function in
HE state. The Gaussian kernel includes unsupported functions like exponential.
While in the implementation of the transformer, GELU is defined as a fast
approximated version, where the $tanh$ function is still non-polynomial,
unsupported by HE.

$G(x)=0.5x(1+tanh[\sqrt{2/\pi}(x+0.044715x^{3})]).$ (2)

We illustrate the numerical comparison between GELU and RELU in Figure 3 ,
where the outputs of GELU are very close to RELU. Hence, we propose to replace
the GELU layer in the model with a ReLU activation function. Despite the $Max$
function in ReLU, other computations are well supported by HE. To enable the
computation of $Max$, we implement the first key idea, incorporating the user
device into the inference. The server will convey ciphertext input to the user
for local $Max$ computation. Once received the connection, a user device
decrypts the ciphertext input and calls the local $Max$ function to get the
results and return re-encrypted results to the server. Despite the
communication cost, no plaintext is leaked during the TLS connection and
semantic security is guaranteed.

Figure 3: The activation results of GELU compared with ReLU. With an input
around zero, the activation results are very close. With a larger or smaller
input value, the activation results tend to converge.

####  3.1.2 Softmax

The second non-polynomial function is softmax, which includes the exponential
and division computation.

$Softmax(x_{i})=\frac{exp(x_{i})}{\sum_{j}exp(x_{j})}.$ (3)

The first thought to approximate softmax is to find alternatives of softmax
operation in transformer, which include Taylor series approximation Vincent et
al. (2015), softmax-free linear attention Lu et al. (2021). However, both of
them have some limitations. The Taylor series approximation can only
approximate the exponential operation. Softmax-free linear attention utilizes
newton-inverse to approximate division, but the approximation error is
unbounded in full-scale attention settings.

For these considerations, we have no choice but to design an estimation
network with addition and multiplication.

$S(x_{i})=x_{i}\ast T(\sum_{j}ReLU(((x_{j})/2+1)^{3})).$ (4)

Equation 4 is the formal description of our softmax estimation network. Same
as the approximation of GELU, ReLU operation here is realized by communication
with the client. Instead of a division operation, we approximate reciprocal
operation with a three-layer linear neural network denoted as $T$.

To get a better estimation of softmax, we randomly generate input tensors
whose values are between $[-3,3]$ and use their softmax scores as MSE targets.
Then we optimize the $T$ for 100k steps with a learning rate of 1e-3 until the
MSE loss drop down to 1e-6.

An under-explored problem here is the Infinite value of Masked Attention,
where the input of softmax is always the masked attention scores. To prevent
the attention of masked tokens, the origin transformer model fills the masked
attention scores with negative infinity before softmax. When fed with an
infinite value, the softmax estimation model may face numerical disaster. We
will discuss this phenomenon and the corresponding solution in Sec 4.5.

####  3.1.3 LayerNorm

Recall that the layer normalization Ba et al. (2016) in transformer is
implemented over a mini-batch of inputs, which could be formulated as:

$y=\frac{x-E[x]}{\sqrt{Var[x]+\epsilon}}\ast\gamma+\beta.$ (5)

The mean and standard deviation are calculated over division operations where
the approximation is needed. $\gamma$ and $\beta$ are learnable affine
transform parameters. To avoid the introduction of new parameters, we keep the
learnable parameters while leaving the mean and standard deviation achieved by
regression.

$\hat{y}=x\circ\boldsymbol{\gamma}+\beta.$ (6)

The new parameter $\boldsymbol{\gamma}$ predicts the value of standard
deviation by regression from origin $\hat{\gamma}$. We find the simple linear
replacement is enough for values with a small scale of bias. Here
$\boldsymbol{\gamma},\beta\in\mathbb{R}$ and $\circ$ denotes the Hadamard
product.

The layer normalization will be applied in each multi-head attention block and
after the output dense layer. So the approximation error tends to accumulate
when the transformer stacks with too many layers.

We treat the layernorm approximation as an individual stage in Figure 1 as LN-
Distill to learn from origin LN layers. A challenge here is the Attention
Overflow, where the input attention score before normalization may have an
unbounded scale, leading to numerical problems. We will discuss the detail of
Attention Overflow in Sec 4.5.

####  3.1.4 Other Practical Replacement

After the approximation workflow, a fine-tuned model consists of only addition
and multiplication operations, which is fully compatible with homomorphic
encryption. We power the model by HE transformer backend. Since the HE
transformer backend could only work for TensorFlow checkpoint, any pre-trained
transformers inherited from PyTorch building version need to be converted into
TensorFlow format first. There are some other details worth mentioning here.

  * •

For the $softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$ operation in attention score
computation, we absorb the value of $\frac{1}{\sqrt{d_{k}}}$ into the weights
of query projection layer.

  * •

We use a fully kernel convolution layer instead of linear projection due to
the lack of supported dense operation.

  * •

All matrix multiplication will be converted into the element-wise style.

  * •

We drop the pooler layer for the unsupported operation of tanh.

###  3.2 Privacy-preserving Inference

In this section, we describe the behavior of HE models during privacy-
preserving inference. Note that inference is completed by the joint effort of
the server and the user device.

Input: user plain text query $\mathcal{P}_{q}$, private key $\mathcal{K}$
generated under server protocol, encrypted server model $\mathcal{M}$.

1 client computes embeddings: $\mathcal{E}_{q}\leftarrow\mathcal{P}_{q}$.

2client encrypts query embeddings: $\mathcal{C}_{q}\leftarrow
Encrypt(\mathcal{E}_{q},\mathcal{K})$.

3server forwards the model: $\mathcal{C}_{i}$ =
$\mathcal{M}(\mathcal{C}_{q})$.

4client handles activation: $\mathcal{C}_{a}$ = $ReLU(\mathcal{C}_{i})$.

5server continues forwarding: $\mathcal{C}_{o}=\mathcal{M}(\mathcal{C}_{a})$.

6client decrypts results:
$\mathcal{P}_{o}=Decrypt(\mathcal{C}_{o},\mathcal{K})$.

Algorithm 2 Inference with HE

In Algorithm 2, notably absent is the support of ReLU operations, where the
server exchanges the activation results with the client. However, all the
communication between client and server is in ciphertext, ensuring the privacy
of user queries and may prevent eavesdropping attackers from recovering
private text data.

##  4 Experiments

In this section, we design both sequence-level and token-level tasks to
evaluate the approximation performance of our THE-X solution. We also discuss
several identified factors which greatly affect approximation workflow.

Table 1: Performance on the GLUE11footnotemark: 1 tasks for both baseline
(standard finetuning) and THE-X with BERT-tiny, measured on the development
sets. We report the best results by hyper-parameter search. $|\mathcal{D}|$
denotes the number of training examples. THE-X only suffers average utility
performance loss: < 1.5% in most tasks. ‘P/S corr.’ is Pearson/Spearman
correlation and ‘m/mm’ denotes the accuracy scores on matched/mismatched set.

Tasks | $|\mathcal{D}|$ | Type | Metrics | Baseline | ReLU | ReLU-S | ReLU-S-L | HE | Perf $\downarrow$
---|---|---|---|---|---|---|---|---|---
SST-2 | 67k | Sentiment | Acc. | 82.45 | 82.40 | 82.34 | 82.11 | 82.11 | 0.34
MRPC | 3.7k | Paraphrase | F1/Acc. | 81.57/70.10 | 81.69/70.34 | 80.81/69.85 | 79.93/68.87 | 79.94/68.87 | 1.63/1.23
STS-B | 7k | Similarity | P/S corr. | 72.83/73.66 | 72.89/73.03 | 74.19/74.27 | 68.38/70.96 | 68.39/70.97 | 4.44/2.69
QQP | 364k | Paraphrase | F1 | 80.28/84.03 | 79.55/82.89 | 79.38/83.36 | 78.28/83.75 | 78.33/83.63 | 1.95/0.40
MNLI | 393k | NLI | m/mm | 69.75/70.75 | 69.51/70.60 | 68.61/69.13 | 68.59/69.41 | 68.47/69.08 | 1.28/1.67
QNLI | 108k | NLI | ACC. | 78.38 | 78.35 | 78.33 | 78.33 | 78.20 | 0.18
RTE | 2.5k | NLI | ACC. | 58.56 | 58.32 | 58.27 | 58.12 | 58.12 | 0.44
Average Perf $\downarrow$ | 0.00 | 0.25 | 0.34 | 1.42 | 1.48 | 1.48

Table 2: Performance on the CONLL2003 task for both baseline and THE-X with
BERT-tiny, measured on the development sets. We find that the replacement with
ReLU has a slight effect on performance and even gets a better F1 score by
0.12 than original GELU activation.

Metrics | Precision | Recall | F1 | Perf $\downarrow$
---|---|---|---|---
Raw | 82.34 | 84.85 | 83.57 | 0
ReLU | 82.29 | 85.13 | 83.69 | -0.12
ReLU-S | 82.08 | 84.73 | 83.38 | 0.19
ReLU-S-L | 79.65 | 83.79 | 81.67 | 1.90
HE | 79.65 | 83.79 | 81.67 | 1.90

###  4.1 Evaluation Tasks

GLUE Wang et al. (2019), the General Language Understanding Evaluation
benchmark, is a collection of tools for evaluating the performance of models
across a diverse set of existing NLU tasks. We choose a subset of GLUE111CoLA
task is not reported because of the limited capacity of BERT-tiny. tasks,
which include: MRPC Dolan and Brockett (2005), SST-2 Socher et al. (2013),
QQP222https://www.quora.com/profile/Ricky-Riche-2/First-Quora-Dataset-Release-
Question-Pairs, STS-B Cer et al. (2017), MNLI Williams et al. (2018), QNLI
Rajpurkar et al. (2016), and RTE Dagan et al. (2005); Haim et al. (2006);
Giampiccolo et al. (2007); Bentivogli et al. (2009).

Following previous work Devlin et al. (2019); Turc et al. (2019), we exclude
the WNLI task from the GLUE benchmark. We also use the famous CoNLL-2003 Tjong
Kim Sang and De Meulder (2003) named entity recognition task as our additional
token-level evaluation. In conclusion, we include the most varieties of NLU
tasks, covering both sequence-level and sentence-level tasks, in both
regression and classification format.

###  4.2 Experiment Settings

For computation efficiency and energy-saving consideration, we use the
released BERT-tiny Turc et al. (2019) as our demo model, which is a standard
transformer-based language model with only 2 layers and a hidden size of 128.
We provide four settings to evaluate different parts of our approximation
components.

  * •

Baseline. In this setting, we make no replacement or approximation. We use the
raw pre-trained checkpoint to fine-tune on downstream tasks.

  * •

ReLU. We fine-tune the pre-trained model with all GELU activation replaced
with ReLU.

  * •

ReLU-S. In addition to ReLU, we fine-tune the model with the softmax operation
replaced by the softmax estimation model.

  * •

ReLU-S-L. We implement full approximation including a layer normalization
replacement.

  * •

HE. We convert the fine-tuned checkpoint with HE-transformer and power the
inference with SEAL backend.

Implementation. To reduce the variance of results under different settings, we
choose hyper-parameters from a fixed set during approximation fine-tuning and
HE inference runtime.

  * •

For fine-tuning the approximation components, we choose a batch size from {4,
8, 16, 32, 128} and a learning rate from 1e-4, 3e-4, 3e-5, 5e-5 as mentioned
in the initial bert code Turc et al. (2019). We use an Adam optimizer with
weight decay chosen from {0.05, 0.1, 0.2, 0.4, 0.5}

  * •

For HE evaluation, we use the HE-transformer backend, where two parameters are
recommended searching by Intel, the poly modules and coeff-modules. We choose
the poly modules degree from {1024, 2048, 4096, 8192, 16384} and choose the
coeff-modules from {20, 30, 60}.

###  4.3 Approximation Results

Table 1 shows the results of the baseline and THE-X on the GLUE benchmark. The
averaged performance reduction of THE-X is 1.48% when compared to the baseline
model. We observe the most performance reduction comes from the approximation
of layernorm, which incurs a reduction of 1.08%. The softmax estimation model
contributes the least performance drop among the approximation components, for
only 0.09% on average, indicating the softmax function could be well imitated
by neural networks. We also find the average performance reduction of HE is
quite negligible, where the slight drop may be due to the sequence truncation.

The results of THE-X on token-level NER task are reported in Table 2. The
replacement of GELU with ReLU even improves the performance of the F1 score.
We assume the slight improvement may come from unexpected bias. However, the
layernorm approximation incurs the most performance reduction. We assume
token-level tasks need a more detailed pattern in attention score. After all,
THE-X still works well in the token-level task with a merely F1 reduction of
1.9%.

Across different types of tasks, we find our THE-X yields the best performance
on the classification tasks, including paraphrase, sentiment and NLI. Among
the classification tasks, the performance of QNLI drops the least, for only
0.18%. We also find the performance drops most on the regression tasks, such
as the similarity task STS-B, for 4.44% pearson correlation and 2.69% spearman
correlation. We assume the regression task needs a higher numerical precision
than the classification task.

Figure 4: Performance on CONLL2003 task with different mask values. We find
the “Negative Infinity” value of the 0 mask greatly reduces approximation
performance. In THE-X using a mask value in [-3, -5] might be a default
choice. Figure 5: Performance on all tasks with different weight decay values,
measured on the development sets. Metrics are marked on the y-axis and weight
decay values are marked on the x-axis.

###  4.4 Negative Infinity

Recall in Equation 4, we replace softmax with a neural estimation model. To
prevent the attention of masked tokens, the origin transformer model fills the
masked attention scores with negative infinity before softmax, where the
numerical disaster occurs in our approximation method. In Figure 4, to solve
this problem, we give an empirical study of how "negative" the masked
attention scores should be. Despite the indistinguishable F1 score change of
raw model fine-tune with different attention mask values, the approximation
method is extremely sensitive to the numerical changes. We assume the softmax
estimation model fails to deal with large input values and leads to a credible
performance drop. However, when the value of the attention mask becomes too
small, it serves as a bias to attention scores, which also leads to a certain
performance drop. We recommend using a moderate mask value between -2 and -5.

###  4.5 Attention Overflow

Another challenge of THE-X is the attention score input of layer
normalization. In most cases, the scale of multi-head attention output is very
dense around $[-1,1]$. However, before normalization, we also observe the
attention scores are scarily sparse, with some extreme values reaching $1e4$,
which is difficult for our LN-distill stage. To prevent the overflow attention
scores, we use the weight decay of Adam optimizer as regularization.

In Figure 5, we present the attention overflow phenomenon across different
tasks. Without any regularization, our approximation method yields
uncontrolled attention scores, leading to poor performance. As the weight
decay increases, the attention scores tend to converge and benefit better
approximation results. We also observe that the larger weight decay may harm
the performance on NLI tasks, where the regularization could be seen as trade-
off between better approximation results and higher performance upper bound.
For the NER task, larger weight decay may even benefit the performance and
also boost our approximation method.

###  4.6 Schedule of Approximation workflow

There are still doubts about how to organize the several optimization steps
for the best approximation performance. We investigate four schedule plans:

  * •

Two Stages. Where we freeze the softmax estimation model during standard fine-
tuning. We select the best checkpoint to implement the second stage - distill
the layer normalization network.

  * •

Joint FT S. We optimize the softmax estimation model during standard fine-
tuning and apply the LN-distillation after.

  * •

Joint FT LN. We apply one-pass optimization with the softmax estimation model
frozen but update the other parameters including layer normalization network.
No further LN-distill will be implemented.

  * •

Joint FT S + LN. A total one-pass optimization with all approximation
parameters updated with the model together.

Figure 6: Performance on all tasks with different organizations of
approximation workflow. Jointly fine-tuning the softmax estimation model or
approximated layernorm leads to a performance drop across all tasks.

As illustrated in Figure 6, we observe that fine-tuning the different
approximation components individually (aka. "Two stages") may be a good
default to keep the best performance of approximation. For the regression task
STS-B, jointly fine-tuning the softmax estimation model and approximated
layernorm even fails to fulfill the approximation pipeline, pulling the
performance down to 0.4%. We assume fine-tuning different components may fall
into a bi-level optimization problem and it is hard to achieve satisfying
results. In conclusion, the softmax estimation model and the approximated
layernorm are both critical components to the performance of THE-X, deserving
individual optimization.

##  5 Conclusions

We present THE-X, a practical approach to enable pre-trained transformer
models to infer under homomorphic encryption. It requires several
approximation components to replace the original operations in the transformer
model. It imposes a slight burden in terms of performance cost but enjoys the
full advantage of homomorphic encryption - the theory-guaranteed user privacy.

We see this as a first step in combing homomorphic encryption to address
emerging privacy issues in pre-trained models. We hope our work motivates
further research, including better approximation solutions on different NLP
applications.

##  6 Acknowledgments

This paper is supported in part by the NSFC through grant No.U20B2053. We also
thanks the support from Beijing Advanced Innovation Center for Future
Blockchain and Privacy Computing.

## References

  * Abadi et al. (2016) Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016_ , pages 308–318. ACM.
  * Ba et al. (2016) Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016.  Layer normalization.  _CoRR_ , abs/1607.06450.
  * Basu et al. (2021a) Priyam Basu, Tiasa Singha Roy, Rakshit Naidu, and Zumrut Muftuoglu. 2021a.  Privacy enabled financial text classification using differential privacy and federated learning.  In _Proceedings of the Third Workshop on Economics and Natural Language Processing_ , pages 50–55, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Basu et al. (2021b) Priyam Basu, Tiasa Singha Roy, Rakshit Naidu, Zümrüt Müftüoglu, Sahib Singh, and Fatemehsadat Mireshghallah. 2021b.  Benchmarking differential privacy and federated learning for BERT models.  _CoRR_ , abs/2106.13973.
  * Bentivogli et al. (2009) Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2009.  The fifth PASCAL recognizing textual entailment challenge.  In _Proceedings of the Second Text Analysis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009_. NIST.
  * Boemer et al. (2020) Fabian Boemer, Rosario Cammarota, Daniel Demmler, Thomas Schneider, and Hossein Yalame. 2020.  MP2ML: A mixed-protocol machine learning framework for private inference.  In _PPMLP’20: Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, Virtual Event, USA, November, 2020_ , pages 43–45. ACM.
  * Boemer et al. (2019a) Fabian Boemer, Anamaria Costache, Rosario Cammarota, and Casimir Wierzynski. 2019a.  ngraph-he2: A high-throughput framework for neural network inference on encrypted data.  In _Proceedings of the 7th ACM Workshop on Encrypted Computing & Applied Homomorphic Cryptography, WAHC@CCS 2019, London, UK, November 11-15, 2019_, pages 45–56. ACM.
  * Boemer et al. (2019b) Fabian Boemer, Yixing Lao, Rosario Cammarota, and Casimir Wierzynski. 2019b.  ngraph-he: a graph compiler for deep learning on homomorphically encrypted data.  In _Proceedings of the 16th ACM International Conference on Computing Frontiers, CF 2019, Alghero, Italy, April 30 - May 2, 2019_ , pages 3–13. ACM.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Cheon et al. (2017) Jung Hee Cheon, Andrey Kim, Miran Kim, and Yong Soo Song. 2017.  Homomorphic encryption for arithmetic of approximate numbers.  In _Advances in Cryptology - ASIACRYPT 2017 - 23rd International Conference on the Theory and Applications of Cryptology and Information Security, Hong Kong, China, December 3-7, 2017, Proceedings, Part I_ , volume 10624 of _Lecture Notes in Computer Science_ , pages 409–437. Springer.
  * Christoph et al. (2015) J Christoph, L Griebel, I Leb, I Engel, F Köpcke, D Toddenroth, H-U Prokosch, J Laufer, K Marquardt, and M Sedlmayr. 2015.  Secure secondary use of clinical data with cloud-based nlp services.  _Methods of information in medicine_ , 54(03):276–282.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The PASCAL recognising textual entailment challenge.  In _Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers_ , volume 3944 of _Lecture Notes in Computer Science_ , pages 177–190. Springer.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005_. Asian Federation of Natural Language Processing.
  * Dwork et al. (2006) Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. 2006.  Calibrating noise to sensitivity in private data analysis.  In _Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings_ , volume 3876 of _Lecture Notes in Computer Science_ , pages 265–284. Springer.
  * Fernandes et al. (2019) Natasha Fernandes, Mark Dras, and Annabelle McIver. 2019.  Generalised differential privacy for text document processing.  In _Principles of Security and Trust - 8th International Conference, POST 2019, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2019, Prague, Czech Republic, April 6-11, 2019, Proceedings_ , volume 11426 of _Lecture Notes in Computer Science_ , pages 123–148. Springer.
  * Gentry (2009) Craig Gentry. 2009.  Fully homomorphic encryption using ideal lattices.  In _Proceedings of the 41st Annual ACM Symposium on Theory of Computing, STOC 2009, Bethesda, MD, USA, May 31 - June 2, 2009_ , pages 169–178. ACM.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.  The third PASCAL recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL@ACL 2007 Workshop on Textual Entailment and Paraphrasing, Prague, Czech Republic, June 28-29, 2007_ , pages 1–9. Association for Computational Linguistics.
  * Haim et al. (2006) R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006.  The second pascal recognising textual entailment challenge.  In _Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment_.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  Bridging nonlinearities and stochastic regularizers with gaussian error linear units.  _CoRR_ , abs/1606.08415.
  * Huang et al. (2020) Yangsibo Huang, Zhao Song, Danqi Chen, Kai Li, and Sanjeev Arora. 2020.  TextHide: Tackling data privacy in language understanding tasks.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1368–1382, Online. Association for Computational Linguistics.
  * Kerrigan et al. (2020) Gavin Kerrigan, Dylan Slack, and Jens Tuyls. 2020.  Differentially private language models benefit from public pre-training.  In _Proceedings of the Second Workshop on Privacy in NLP_ , pages 39–45, Online. Association for Computational Linguistics.
  * Lin et al. (2021) Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. 2021.  Fednlp: A research platform for federated learning in natural language processing.  _CoRR_ , abs/2104.08815.
  * Lu et al. (2021) Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. 2021.  SOFT: softmax-free transformer with linear complexity.  _CoRR_ , abs/2110.11945.
  * Lyu et al. (2020) Lingjuan Lyu, Xuanli He, and Yitong Li. 2020.  Differentially private representation for NLP: formal guarantee and an empirical study on privacy and fairness.  In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , volume EMNLP 2020 of _Findings of ACL_ , pages 2355–2365. Association for Computational Linguistics.
  * Qu et al. (2021) Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Marc Najork. 2021.  Natural language understanding with privacy-preserving bert.  _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100, 000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 2383–2392. The Association for Computational Linguistics.
  * Schwartz and Solove (2011) Paul M Schwartz and Daniel J Solove. 2011.  The pii problem: Privacy and a new concept of personally identifiable information.  _NYUL rev._ , 86:1814.
  * (31) SEAL. 2020.  Microsoft SEAL (release 3.6).  https://github.com/Microsoft/SEAL.  Microsoft Research, Redmond, WA.
  * Shen et al. (2007) Xuehua Shen, Bin Tan, and ChengXiang Zhai. 2007.  Privacy protection in personalized search.  _SIGIR Forum_ , 41(1):4–17.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1631–1642. ACL.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Well-read students learn better: On the importance of pre-training compact models.  _arXiv preprint arXiv:1908.08962_.
  * Vincent et al. (2015) Pascal Vincent, Alexandre de Brébisson, and Xavier Bouthillier. 2015.  Efficient exact gradient update for training deep networks with very large sparse targets.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1108–1116.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Xu et al. (2019a) Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019a.  BERT post-training for review reading comprehension and aspect-based sentiment analysis.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2324–2335. Association for Computational Linguistics.
  * Xu et al. (2019b) Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. 2019b.  Understanding and improving layer normalization.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 4383–4393.
  * Yang et al. (2019a) Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. 2019a.  _Federated Learning_.  Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers.
  * Yang et al. (2019b) Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019b.  End-to-end open-domain question answering with bertserini.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Demonstrations_ , pages 72–77. Association for Computational Linguistics.
  * Yang et al. (2019c) Wei Yang, Haotian Zhang, and Jimmy Lin. 2019c.  Simple applications of BERT for ad hoc document retrieval.  _CoRR_ , abs/1903.10972.
  * Zhao et al. (2020) Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. 2020.  idlg: Improved deep leakage from gradients.  _CoRR_ , abs/2001.02610.
  * Zhu and Han (2020) Ligeng Zhu and Song Han. 2020.  Deep leakage from gradients.  In Qiang Yang, Lixin Fan, and Han Yu, editors, _Federated Learning - Privacy and Incentive_ , volume 12500 of _Lecture Notes in Computer Science_ , pages 17–31. Springer.


# Trieste: Efficiently Exploring The Depths of Black-box Functions with
TensorFlow

Victor Picheny<EMAIL_ADDRESS>
Joel Berkley
Henry B. Moss
Hrvoje Stojic
Uri Granta
Artem Artemev
Sebastian W. Ober
Khurram Ghani
Alexander Goodall
Andrei Paleyes
Sattar Vakili
Sergio Pascual-Diaz
Stratis Markou
Secondmind, Cambridge, UK Jixiang Qing
Nasrulloh R.B.S Loka
Ivo Couckuyt
Ghent University, Ghent, Belgium

###### Abstract

We present Trieste, an open-source Python package for Bayesian optimization
and active learning written in TensorFlow. Our library enables the use of
popular TensorFlow-based models within sequential decision making loops, e.g.
Gaussian processes from GPflow or GPflux, or neural networks from Keras. This
plug-and-play mindset is central to the package and extends to our acquisition
functions and the internal dynamics of the decision making loop, both of which
can be tailored by researchers or engineers when tackling custom use-cases.
Trieste is a research-friendly and production-ready toolkit backed by a
comprehensive test suite, extensive documentation, and available at
https://github.com/secondmind-labs/trieste.

##  1 Introduction

A popular solution to high-cost black-box optimisation tasks has arisen in
Bayesian Optimization (BO) (see shahriari2016taking for an introduction). By
using probabilistic surrogate models to predict the value of objective
functions at not-yet-observed input locations, BO is able to focus subsequent
evaluations into areas of the search space where uncertainty is high or that
contain promising evaluations. The exact balance of exploration and
exploitation followed by the search is controlled by heuristic search
strategies known as acquisition functions. BO’s ability to find high-scoring
regions of black-box functions within heavily constrained optimization budgets
has led to an explosion in BO research and has had recent successes across a
diverse range of tasks including teaching robots to cook (junge2020improving),
fine-tuning Amazon Alexa (moss2020boffin), performing molecular search
(griffiths2022gauche), designing chemical additives (rankovic2022bayesian),
and many more besides.

In this paper, we present Trieste111Trieste was the first crewed vessel to
reach the bottom of the Mariana trench — the literal global minima., a highly
modular library designed to enable users working within TensorFlow ecosystems
to 1) deploy their own existing models to drive BO loops and 2) build BO
pipelines that can harness the ease and computational efficiency provided by
TensorFlow’s automatic differentiation and support for modern compute
resources like GPUs. Our library contains a wide-range of advanced Bayesian
optimization functionalities matched only by Torch-based BoTorch of
balandat2020botorch (see Section 4) which, although widely regarded as the
state-of-the-art BO Python library, does not support the large number
researchers and engineers with TensorFlow models and pipelines. As such,
Trieste has been taken up by researchers to develop new BO methodology
(moss2021gibbon; vakili2021scalable; picheny2022bayesian; qing2022robust;
qing2022spectral; qing2022text; moss2022information; chang2022fantasizing;
heidari2022finding) whilst also being used across a range of applications
including designing effective yet light-weight automotive heat-exchangers
(paleyes2022penalisation) and improving adhesive bonding
(jordens2023optimization).

##  2 Related Work

Many open-source libraries have been built to support the recent increase in
use and development of BO methodology. For example, Python users with models
written in Torch or NumPy can easily find compatible BO libraries such as
BoTorch (balandat2020botorch), GPyOpt (gpyopt2016), RoBo (klein2017robo) or
Emukit (paleyes2021emulation). Similarly, those running R or C++ can use
DiceOptim (roustant2012dicekriging) or BayesOpt (martinez2014bayesopt).
Unfortunately, missing from the above list is a BO library that supports
TensorFlow 2, one of the most popular frameworks for performing machine
learning in Python. It is likely that a lack of BO support has acted as
barrier for the wider take-up of BO by the large number of researchers and
engineers working within modern TensorFlow-based ecosystems, an environment
which is fundamentally different to the now defunct TensorFlow 1 previously
supported only by the now also deprecated GPflowOpt (knudde2017gpflowopt).
Note that due to the interactive nature between model and BO mechanics (e.g.
the optimization of acquisition functions that depend on predictions of
models), full realization of the TensorFlow’s key strengths, like auto-
differentiation and GPU support, can be achieved only when both model and BO
mechanics share the same back-end.

##  3 Overview of Trieste

Trieste has a modular structure with a full BO loop primarily consists of four
building blocks, a choice of interface (either AskTellOptimizer or
BayesianOptimizer), a choice of ProbabilisticModel and a pairing of
AcqusitionRule and AcquisitionFunction.

###  3.1 Interfaces

A key design choice of Trieste is that its BO loop does need not have direct
access to the objective function. In many libraries, the objective must be a
query-able function which is to be passed into the loop and called for each BO
step. Unfortunately, this is rarely suitable when performing BO in the real-
world. For example, when evaluating the objective function requires laboratory
or test-bench experiments or when experiments require distributed computing
resources separate to the host node. To this end, Trieste provides an
AskTellOptimizer interface, which separates the BO loop from the objective
function. For each BO step, an AskTellOptimizer outputs some recommended query
points to the user (the ask) and then expects to receive some new evaluations
(the tell). Our AskTellOptimizer provides a highly flexible BO library,
enabling Trieste to be used across a range of non-standard settings, e.g. when
evaluations arrive asynchronously for batch BO under varying evaluation costs
(kandasamy2018parallelised) or when when users only have partial control over
objective function evaluations (qing2022robust). However, for settings where
it is appropriate for the objective function to be passed into the BO, e.g.
when researchers are running toy experiments, we also provide a more standard
BayesianOptimizer interface that will run multiple BO steps, outputting only
the final model. See Figure 1 for a high-level overview of the distinction
between our AskTellOptimizer and BayesianOptimizer.

(a)

(b)

Figure 1: Trieste’s two different interfaces for an $N$-step optimization. For
the AskTellOptimizer users must manually evaluate the black-box objective
function for each iteration whereas the BayesianOptimizer queries the black-
box directly, performing all $N$ BO iterations without user interaction.

###  3.2 Multi-model Support

Trieste’s BO loops allow the modeling of multiple quantities (using either
multiple separate models or a single multi-output model) and so naturally
supports common BO extensions like multi-objective knowles2006parego, multi-
fidelity (swersky2013multi) and optimization under unknown constraints
(gardner2014bayesian). The surrogate models themselves can be any model
written in TensorFlow through our general ProbabilisticModel wrapper. However,
for convenience, we provide wrappers for standard models like Gaussian
processes from GPflow (matthews2017gpflow), for deep Gaussian processes from
GPflux (dutordoir2021gpflux), as well as neural networks from Keras.

###  3.3 Rules and Acquisition Functions

Regardless of the interface and model choice, the recommendation of query
points is controlled by an AcquisitionRule. While the vanilla BO rule is to
query the point that optimizes a particular AcquisitionFunction, the rule is a
useful abstraction to handle complex cases, for instance a varying
optimization space for the AcquisitionFunction

which usually builds and optimizes a particular AcquisitionFunction.

As detailed in our documentation, it is straight-forward for a user to specify
a new acquisition function, however, special care is taken to allow
AcquisitionFunctions to be updated without expensive retracing of computation
graphs for each BO step.

A wide range of acquisition functions are already provided in Trieste. Many
are based on the gold-standard Expected Improvement (EI; jones1998efficient)
including, exact and Monte-Carlo multi-point EI (chevalier2013fast;
balandat2020botorch) and local penalization (gonzalez2016batch) for batch BO,
augmented EI (huang2006global) for noisy optimization, Monte-Carlo Expected
Hyper-volume Improvement (daulton2020differentiable) for (batch) multi-
objective BO, and constrained EI (gardner2014bayesian) for constrained
optimization. To demonstrate the ease of designing new acquisition functions
in Trieste, we also include implementations of recent information-theoretic
approaches like MES (wang2017max) and MUMBO (moss2021mumbo), as well as a
recent extensions of Thompson sampling (wilson2020efficiently). For those
interested in active learning rather than BO, Trieste also includes popular
methods such as BALD (houlsby2011bayesian), Integrated variance reduction
(picheny2010adaptive) and Expected feasibility (bichon2008efficient).

##  4 Feature Benchmark

Table 1 compares Trieste with two other popular BO frameworks. Trieste has
comparable features to the Torch-based BoTorch and provides substantially more
functionality than the NumPy-based Emukit. The other Python BO libraries
introduced in Section 2 like GPyOpt, RoBO and GPflowOpt are not included in
this comparison as they have even less functionality and are now deprecated.

| Auto-diff | GPU | Ask-tell | Batch |  | Multi-
---
fidelity
| Multi-
---
objective
Back-end
Trieste | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | TensorFlow
BoTorch | ✓ | ✓ |  ×222The Ax library (https://ax.dev/) is a high-level interface for BoTorch somewhat similar to an ask-tell. | ✓ | ✓ | ✓ | Torch
Emukit | × | × | × | ✓ | ✓ | × | NumPy
Table 1: A comparison of feature support (according to their documentation) of
the Python BO libraries still under active maintenance.

##  5 Conclusions and Future Plans

Trieste is an open-source project that allows TensorFlow practitioners to use
their existing models within BO pipelines. Backed by continuous integration
and comprehensive unit tests, Trieste is a reliable and robust framework used
for both real-world deployment and research. As Trieste is built on features
added and improved by the community and so we gladly welcome feature requests
and code contributions.

## Acknowledgments

We want to thank Secondmind for supporting the open-sourcing effort.