Mazharul Islam

University of Wisconsin–Madison

Part of the work done while  author was an intern at Visa Research

Sunpreet S. Arora

Visa Research

Rahul Chatterjee

University of Wisconsin–Madison

Peter Rindal

Visa Research

Maliheh Shirvanian

Work done while  author was at Visa Research

@ACM@instpresentclassnameNo institution present for an affiliation
@ACM@citypresentclassnameNo city present for an affiliation
@ACM@countrypresentclassnameNo country present for an affiliation

Secure multi-party computation (MPC) techniques can be used
to provide data privacy when users query deep neural network (DNN) models hosted on a public cloud. State-of-the-art MPC techniques can be directly leveraged for DNN models that use simple activation functions (AFs) such as $\relu$. However, DNN model architectures designed for cutting-edge applications often use complex and highly non-linear AFs. Designing efficient MPC techniques for such complex AFs is an open problem.

Towards this, we propose , which produces piece-wise polynomial approximations of complex AFs to enable their efficient use with state-of-the-art MPC techniques. neither requires nor imposes any restriction on model training and results in near-identical model accuracy.
We extensively evaluate on four different machine-learning tasks
with DNN architectures that use popular complex AFs $\silu$, $\gelu$, and $\mish$. Our experimental results show that $\sysname$ incurs
negligible accuracy loss compared to DNN-specific approaches for handling complex non-linear AFs.

We also incorporate in two state-of-the-art MPC libraries for privacy-preserving inference and demonstrate that provides 2$\times$-5$\times$ speedup in computation compared to the state-of-the-art approximation approach for non-linear functions — while providing similar or better accuracy for DNN models with large number of hidden layers.

§ INTRODUCTION

Deep neural networks (DNNs) based inference services
are being increasingly adopted in various emerging applications, such as early disease discovery from personal health records [69], personalized product recommendations [10], media translations [9], image recognition [3], and even biometric authentication [15].
Trained DNN models are typically hosted on a cloud server for applications or users
to query for inference tasks. Although very useful, these services can pose serious privacy concerns.

For instance, users are required to share their facial images with an online service
hosting a face recognition DNN model.

Indeed, recently, IRS has to remove their identity verification service based on facial recognition due to the concern of the privacy of users' facial images [15].

DNN models used for inference cannot be transferred to the client devices because they can be proprietary and trained on private training data [20, 38], for example, user's medical records. Clients also would like to avoid sharing their private data for inference purposes.

This problem is generally referred to as secure inference, where a client is able to obtain the inference results on their private input without sharing the private input with the server, nor learning anything about the DNN model parameters.  Secure inference can resolve privacy risks associated with DNN-based inference services.

Recent progress in secure
multi-party computation (MPC)
has resulted in a number of investigations
to solve the secure inference problem [74].
A central challenge in this regard is handling the non-linear activation functions (AFs) efficiently, as  MPC techniques are not optimized for handling such nonlinear AFs. Indeed, now it is well known that AFs are the bottleneck of performing secure inference [21, 32, 63, 51, 22].

To address this challenge, prior attempts

have mostly focused on DNN models using $\relu$
AF [63, 8, 42, 23, 64, 11] — a relatively simple non-linear function.

However, these protocols still remain unsuitable in real-world applications for performing secure inference over DNN models, which are already trained over non-linear complex AFs

such as  $\silu, \gelu, \mish$. One possible approach
is to approximate these complex AFs using a number of piece-wise polynomials.  We can call the generated approximation via such approach MPC-friendly as these piece-wise polynomials can be easily computed securely
using existing MPC libraries that support only three math operations $\addmathoperation~(+)$, $\mulmathoperation~(\times)$, and $\comparemathoperation~(>)$. However, state-of-the-art generic MPC-friendly approximation approach $\nfgen$ [18] — a work contemporary to ours — is not tailored for complex AFs used in DNN models. Therefore, perhaps unsurprisingly, as our work demonstrates, it fails to provide the expected performance efficacy as the DNN models get deeper.

This raises an open question “can an efficient protocol be devised to enable secure inference over DNN models that extensively use complex and widely used non-linear AFs?”. We also want to achieve this without necessitating any modifications to the way DNN models are trained.
In this work, we make progress toward answering this question affirmatively by designing a scheme called $\sysname$. Our scheme follows an approach similar to $\nfgen$ — approximating non-linear functions using a number of piece-wise polynomials. However, $\sysname$ is specifically tailored for widely used complex AFs and thus provides better performance efficacy as the DNN models get deeper while maintaining a similar or better inference accuracy.
However, designing an MPC-friendly approximation approach tailored for complex non-linear AFs requires addressing some unique challenges.

First, unlike $\relu$ like simple non-linear AFs,  approximating complex more non-linear AFs (e.g., $\silu$, $\gelu$, $\mish$) accurately is hard near the region close to zero as illustrated in fig:actfuncs.
We argue ensuring precise approximation in this harder to approximate region is crucial to minimize inference accuracy loss.
This is because the majority of the input to the complex AF also falls to this region, primarily due to a normalization step known as batch normalization (BN) applied to the inputs prior to forwarding them to these complex AFs (refer to sec:related-work-dnn for more details about this phenomenon).

Second, it is unclear how approximation error, introduced by such MPC-friendly approximations of complex AFs, affects the inference accuracy of the DNN models. There is a chance that slight errors introduced by such approximation in each layer can accumulate
and consequently affect the inference accuracy of DNN models significantly. This effect may be even more pronounced
for deeper and wider models with more hidden layers and neurons per layer.

Lastly, the approximation approach needs to handle the trade-offs between performance overhead and inference accuracy loss carefully.
Presumably, a generated MPC-friendly approximation with fewer low-degree polynomials would reduce performance overhead but might lead to significant inference accuracy loss.
Conversely, an approximation with many high-degree polynomials would reduce inference accuracy loss but suffer from elevated performance overhead.
Additionally, during approximation, it's essential to account for the ring size used by the MPC library to represent fixed points (FXP) as
existing MPC libraries typically perform computation over FXP representation instead of commonly used floating point (FLP) numbers.
While working with a smaller range and resolution in FXP enhances efficiency, this can trigger overflow or underflow during conversion from  FLP to FXP, significantly compromising inference accuracy with high probability.
On the other hand, using a larger FXP range avoids such overflow/underflow issues but introduces performance overhead.

We handle these challenges empirically and via careful synergy of machine learning techniques and MPC.
More specifically, we tailor our approximation scheme to harmonize the intricate architecture of the state-of-the-art DNN models while maintaining compatibility with existing general-purpose MPC libraries.
As a first step, we recognize that instead of relying on cubic spline-based piece-wise interpolation as prior work on handing complex AFs [47],
Chebyshev sequence-based interpolation [48] is more suited for approximating complex AFs. This is because Chebyshev interpolation generally outperforms other techniques when approximated functions are smooth and monotonic [50], as is the case with complex AFs.
We note $\nfgen$ [18], being a contemporary work to ours, also adopts a similar insight for generating MPC-friendly approximation of non-linear functions.
Nevertheless, as we describe next, our distinctive optimizations tailored for DNN models make sustain inference accuracy loss even as DNN models increase in depth without sacrificing performance.

Next, we observe that state-of-the-art DNN models apply normalization in between performing linear transformations
and applying non-linear AFs (fig:normalization).
Such normalization leads to
majority of the inputs to the non-linear AFs fall into specific places near the region close to zero with high probability — while a small portion of inputs will fall into places with low probability.
Thus, we hypothesize that
an MPC-friendly approximation scheme for complex AFs; taking this observation into account will help mitigate the cumulative impact of errors introduced by MPC-friendly approximations from one layer to subsequent layers of a deeper/wider DNN model.

Lastly, to balance performance overhead and inference accuracy loss, we devise an empirical strategy employing simulated annealing (SA) search technique [35].
This approach revolves around strategically adjusting the following three parameters: the numbers of polynomials ($\npieces$), degrees associated with each polynomial ($\maxdegree$), and ring sizes of FXP ($\ring$) used for approximation.
Lowering any of these parameters inherently reduces performance overhead.
Leveraging this progressive relationship, we initiate the searching process with an initial solution featuring high $\npieces$, $\maxdegree$, and $\ring$ — resulting in pronounced performance overhead and an imbalance between performance and accuracy. To rectify this, we explore adjacent solutions through random local adjustments to mitigate performance overhead while preserving minimal accuracy loss, all within the framework of the SA search technique.
Importantly, exhaustive iterations over reasonable $\npieces$, $\maxdegree$, and $\ring$ values would be computationally infeasible due to the time-intensive nature of evaluating inference loss for each generated MPC-friendly approximation.
Additionally, we propose a systematic way to find the appropriate approximation threshold — an important component of the approximation-based approaches — through binary search, moving away from setting a fixed threshold as $\nfgen$ [18] does (refer to sec:selecting-threshold). Furthermore, we introduce a DNN-specific modification (refer to  sec:designing-fcrude) to enhance the performance efficacy of $\sysname$.

We implement $\sysname$ and incorporate its generated MPC-friendly approximation of complex AFs
to two state-of-the-art secure inference MPC libraries  [53] and CryptFlow2 [63].
Our experiments reveal that
$\sysname$ generated MPC-friendly approximation of complex AFs
shows negligible inference accuracy loss while being computationally efficient for four image classification tasks of various complexities. We are in the process of open-sourcing $\sysname$ with the next version of the paper.

Summary. Our contributions, in summary, are as follows:

* We present , that can generate MPC-friendly piece-wise polynomial approximations for
popular complex non-linear AFs. The generated approximation is generic and can be easily incorporated into state-of-the-art multi-party computation scenarios (sec:overview).

* The approximation technique used in our method is input density aware and accurately approximates regions with high input probability density while coarsely estimating regions with low input probability density (sec:accurate-approximations).

* We empirically handle the innate challenge to find an approximation that does not degrade inference accuracy and introduce much performance overhead using an application-specific design of simulated annealing (SA) searching technique (sec:time).

* Extensive experiments using four different state-of-the-art DNN models on diverse classification tasks demonstrate that $\sysname$ and $\nfgen$ incur negligible accuracy loss compared to existing other DNN-specific approaches [47, 43, 62] (sec:accuracy). We further compare the performance overhead of $\sysname$ and $\nfgen$ by incorporating their generated approximation to two state-of-the-art MPC libraries. We
find that our DNN model-specific optimizations make $\sysname$ 2$\times$-5$\times$ computationally efficient than $\nfgen$ [18] — for DNN models having a high number of hidden layers, all while maintaining negligible accuracy degradation (sec:performance).

§ BACKGROUND AND RELATED WORK

This section summarizes relevant background and prior work from deep neural networks (sec:related-work-dnn) and
cryptographic techniques developed for solving the secure inference problem (sec:related-work-mpc).

§.§ Deep Neural Network Preliminaries

Activation Functions (AFs).

AFs are used for adding non-linearity to the learning process and play a major role
in enhancing the training capabilities and accuracy of the DNN models.
Many contemporary models use $\relu$ AF, which makes a hard gating decision based on the input sign (fig:actfuncs). Despite being theoretically simple, $\relu$ provides remarkably faster convergence and performance in practice [40, 56].
However, $\relu$ outputs a value of zero whenever the input is negative, and as such, the neural network loses a certain
amount of valid information as soon as inputs become negative.
This drawback prompted ML communities to develop complex AFs, overcoming the limitations of $\relu$.

Complex AFs.

In recent years, a range of complex AFs, such as $\silu$ [16],
$\gelu$ [27], and $\mish$ [52], has emerged surpassing the performance of $\relu$ in state-of-the-art DNN models applied across
computer vision, natural language processing, and reinforcement learning applications.
These AFs as shown in fig:actfuncs,
are smooth and continuous, can handle small weight changes, and aid in effectively regularizing DNN models. For example, Hendrycks et al. empirically illustrated the robustness of $\gelu$-trained DNN models
against noisy inputs, often surpassing the accuracy of $\relu$-trained models [27].
Ramachandran et al. used automatic search techniques to uncover $\silu$ (also called Swish). This complex AF improved image classification accuracy of Inception-ResNet-v2 model by 0.6% and by 0.9% of Mobile NASNET-A model [61] by simply substituting it with $\relu$. Misra et al. proposed the self-regularized AF $\mish$ that exhibits superior performance compared to AFs for YOLOv4, ResNet models [52].

Hence, complex AFs offer a compelling advantage in building better-performing models in terms of
convergence and classification accuracy when compared to $\relu$. Unfortunately, unlike $\relu$, which is relatively easy to compute for secure evaluation, these complex AFs exhibit a higher degree of non-linearity as shown in fig:actfuncs.
This makes their use with existing MPC techniques challenging.
In this work, we address this limitation by designing an MPC-friendly version of these three complex AFs. We refer more interested readers to app:related-work for additional details on other complex AFs used in neural networks that lie outside the scope of this work.

Batch normalization.
Batch normalization (BN) is used to address internal covariance shift problem in neural networks — which happens when a layer's input distribution changes abruptly due to its dependency on previous layers [34]. BN lends stability to the training process by reducing dependence on initial parameter selection and requiring a lower learning rate and number of epochs. BN is performed on the outputs of the linear transformations, and normalized outputs are forwarded to non-linear AFs.
Thus, non-linear AFs receive normalized inputs.

fig:normalization illustrates BN process
for  $\ell^{\textsf{th}}$ layer where input to the linear operations is $h^\ell$ and output is
$\mathbf{a}^\ell = {w^\ell}^Th^{l}$.
Assume $\mathbf{a}^\ell = \Big(a^\ell_1, a^\ell_2, \cdots, a^\ell_d\Big)$  is $d$-dimensional.
If the population mean  and variance are $\mathsf{E}[\mathbf{a}^{\ell}], \mathsf{Var[\mathbf{a}^{\ell}]}$ respectively,
then $\mathbf{a}^\ell$ is normalized to $\overline{\mathbf{a}}^l$  using eqn:bn such that the probability distribution of $\overline{a}^l$  follows a normal distribution with mean and variance zero and 1, respectively.

\begin{equation}
\overline{\mathbf{a}}^{\ell}_{k} =  (\mathbf{a}^{\ell}_{k} - \mathsf{E}[\mathbf{a}^{\ell}_{k}])/\mathsf{Var[\mathbf{a}^{\ell}_{k}]}
\label{eqn:bn}
\end{equation}

BN is widely used in state-of-the-art DNN models to calibrate the input to the
non-linear AFs during both training and inference phase.
This makes it a good estimator of the input density to complex AFs in DNN models during inference.
Our scheme leverages this estimation to improve the generated
MPC-friendly approximation.

The output of the linear operations ($\mathbf{a}^\ell$) are normalized to $\overline{\mathbf{a}}^\ell$ using  eqn:bn before they are forwarded for applying non-linear operations involving complex activation functions (AFs).

§.§ Secure Inference for DNN models

State-of-the-art MPC techniques enable computation on encrypted data and have been used to address the secure inference problem. Generally, a client encrypts their input and sends the encrypted input to a cloud service. The cloud service performs inference using trained DNN models over the encrypted input. Typically, MPC techniques are optimized for linear transformations (e.g., addition, vector multiplications, etc.). Therefore, computing non-linear operations involved in secure inference (e.g., non-linear AFs) is one of the main challenges.

$\relu$ specific secure inference.
Given $\relu$ is one of the most popular AFs used in practical deployments of DNNs, recent research has mostly focused on the use of
in MPC paradigms [71, 70, 63, 8, 31].
For example, Rathee et al. propose a novel 2PC protocol
for secure comparison and division for efficient evaluation of $\relu$
in semi-honest settings [63].
Follow-up works
extend this protocol to the malicious client threat model [8, 31].
However, $\relu$ specific optimizations proposed in the aforementioned methods do not generalize to other complex AFs.
Another set of methods uses Garbled Circuits (GC) for secure evaluation
of $\relu$
[51, 42, 36, 65]. However, communication overhead limits its applicability to
shallow DNN models (less than seven layers). It is challenging to generalize these methods to wide DNN models that use complex AFs other than $\relu$ for
secure inference.

A different approach for computing non-linear AFs
efficiently in the encrypted domain is by  restricting
the way DNN models are trained.
For example, Riazi et al. [64] leverage GC based protocol
for secure
inference on binary neural networks (BNN).
However,  retraining proprietary models with these restrictions is costly
and oftentimes practically infeasible.
Imposing such limitations on the training process can also impact the performance of DNNs in practice.
Pereteanu et al. [60] introduce the notion of partially private DNN models
such that the middle part of the model is sent in plaintext to clients to reduce communication overhead.
However, in practice,  cloud service providers would want to keep their full part of the DNN model
secret lest revealing any part of the property model leaks sensitive information, resulting in business consequences.

In summary, a plethora of research work has focused on secure inference for $\relu$-based DNNs.
Our work focuses on novel AFs that have been shown to outperform $\relu$
and are getting traction in the ML community. We refer the readers to  [58] for a recent comprehensive survey.

Secure inference for other non-linear AFs.
A common approach for secure inference involving non-linear AFs is by
approximating them with low-degree polynomials. These
polynomials are easy to compute for MPC frameworks and
thus are MPC-friendly. The challenge is not
to degrade the inference accuracy, as the approximation error can cause incorrect results.
Delphi [51], for example, runs a planner that balances which AF
can be replaced with low-degree polynomials without introducing too many inaccuracies and achieving a significant communication benefit. CryptoNet [23]
CryptoDL [28],  MiniONN [47] also, use similar ideas for approximating non-linear AFs.
However, they are application-specific, and switching to another
application degrades accuracy significantly [21]
due to small errors getting propagated resulting in numeric instability.
In addition, MiniONN [47] is heavily focused on $\sigmoid$ AF — which is essential for logistic regression models.
However, as we will show in sec:accuracy, using their recipe of piece-wise polynomial
approximation drastically decreases the inference accuracy
when we use their approach for approximating the complex AFs that we focus in this work.

Recently, Fan et al. proposed $\nfgen$ [18] a technique capable of converting popular non-linear functions into MPC-friendly ones.
One may also choose to use this approximation-based approach to do the same for complex non-linear AFs.
In fact, $\nfgen$ is the closest related work to ours as we also follow a similar approach — generating MPC-friendly approximations of the complex AFs using a number of piece-wise polynomials. However,
$\nfgen$ is not specifically customized for widely used complex AFs inside DNN models.
Absence of such customized techniques makes $\nfgen$ computationally less efficient
when we compare it with our scheme through extensive experiments (sec:performance).

§ PROBLEM OVERVIEW

Secure inference in cloud-based deployment setting.
(a) Proprietary DNN model trained over private data that is not MPC-friendly due to complex non-linear activation functions (AFs) (e.g., $\silu, \gelu, \mish$). An MPC-friendly model is generated by replacing the complex AFs with their approximations using $\sysname$. (b) Next, we generate secret shares of the MPC-friendly model and distribute them with $N=3$ cloud servers. (c) To get the inference result, the client generates shares of its private input and sends them to the cloud servers. Cloud servers perform inference and return the shares of the inference result to the client, and the client uses them to reconstruct the original inference result.

§.§ Problem Formulation and Scenario Setup

Problem formulation.
We refer to the server holding the DNN model by $\modelowner$.
The DNN model consists of $L$ layers, each comprising first linear transformations and then applying non-linear complex activation function (AF) $\fx$.
In between linear and non-linear complex AF, batch normalization is also present.
We assume a machine learning as a service (MLasS) [1] inspired scenario where weights $\weights$ of all layers have already been trained, and the trained model is being used to provide cloud-based inference $z$ over client ($\client$)  uploaded input $\clientinput$  using eqn:problem-formulation.

\begin{equation}
z := \fx(\weights_L \cdot \cdots \fx(\weights_2 \cdot \fx(\weights_1 \cdot \clientinput)))
\label{eqn:problem-formulation}
\end{equation}

The problem secure inference tackles is how to compute eqn:problem-formulation obliviously to satisfy the privacy needs of both $\client$ and $\modelowner$.
This requires designing a system such that $\client$ knows nothing about model weights $\mathcal{W}= [\weights_1, \weights_2, \cdots, \weights_L]$ of the DNN model and $\modelowner$ learns nothing about $\clientinput$; yet $\client$ can get the inference result $z$ from these received SS.
Moreover, we need to achieve this privacy need without significantly degrading inference accuracy and performance overhead.

Scenario Setup.
MPC techniques provide a generic solution to solve this problem using the $N$ server scenario. In this scenario, the DNN model owner $\modelowner$ and the client $\client$ uses a set of $N$ non-colluding servers to achieve their privacy requirements. $\modelowner$ generates $N$ secret shares (SS) of the model weights (i.e., $\mathcal{W} = \llbracket \mathcal{W}_1, \mathcal{W}_2, \cdots \mathcal{W}_N \rrbracket$), and $\client$ does the same for its private input (i.e., $\clientinput = \llbracket \clientinput_1, \clientinput_2, \cdots, \clientinput_N \rrbracket$. Then, these $N$ SS are distributed by $\modelowner$ and $\client$ with the $N$ servers. The servers then perform the secure inference computing eqn:problem-formulation securely on their shares of the information and return the results $\llbracket z_1, z_2, \cdots, z_N \rrbracket$ to $\client$, who can combine the shares for constructing $z$.
We summarize the adversary's power $\adv$  for the $N$ server scenario typically assumed by MPC protocols in sec:threat-model. Following other works, we focus on running experiments for two most common scenarios: 2PC $(N=2)$ [63, 51, 64] and 3PC $(N=3)$ [53, 70, 71] — although our scheme does not necessarily depend on the specifics of $N$.
We pictorially present a 3PC scenario in fig:overview.

One of the motivating realizations of this scenario can be in the medical domain.
In particular, where a  DNN model has been trained by a trusted organization (e.g., NIH[National Institutes of Health at: <https://www.nih.gov/>]) leveraging substantial computational resources and exclusive access to users' private health records. To preserve the privacy of the proprietary DNN model, NIH can generate SS of the model and distribute them across
$N$ different non-colluding servers, possibly hosted by different hospitals. When patients submit their private health data, they can generate $N$ SS and share them with the $N$ different hospitals. In this way, the patient learns the final result without learning anything about the model or revealing their private information to any hospital.

Difficulty in computing non-linear AFs.

A major bottleneck is securely computing $\fx(x)$ shown in eqn:problem-formulation. This is because $\fx(x)$ is non-linear, which consumes most of the
communication and latency costs of the overall total protocol execution, as illustrated by many prior work
(e.g., Rathee et al. c.f., [63] Table 6).
Linear operations (i.e., vector multiplication) are relatively
less expensive.

§.§ Design Goals

>angle=#1, left, minipage=#3, lap=-(#2)

20.8cmMethod     21.2cm1 Supp. cmplx. AF     21.2cm2 Supp. many HL     21.2cm3 Comp. w/ MPC libs     21.8cm4 Supp. any training proc.

CryptFlow2 [63]         $^\dag$

SIMC [8]         $^\dag$

Cheeta [31]         $^\dag$

Delphi [51]         $^\dag$

XONN [64]         $^\dag$

SIRNN [62]

MUSE [42]         $^\dag$

SecureNN [70]         $^\dag$

FALCON [71]         $^\dag$

$\nfgen$ [18]

electriclime This ()

Yes;  Unclear;   No

19cm$^\ddag$Refer to  sec:requirements for more details about the design goals indicated by columns 1-4

$^\dag$Number of hidden layers (HLs) support is experimentally validated for DNN models with $\relu$.

Comparison of related work with .

While designing , we want to ensure DNN model designers are not restricted to the set of AFs and model architectures that MPC platforms support. We distill four criteria for this and show how prior work on secure inference, mainly from 2019 onwards, fail to satisfy one or more of these design goals in tab:comparison.

1 Support complex AF. We want our scheme to be compatible with the majority of the DNN models used by inference services. Therefore, in this work, we do not use $\relu$-specific optimizations.

A large amount of prior work are devoted to optimize $\relu$ and fail to satisfy this design goal [63, 31, 51, 70, 71]. Few works rely on garbled circuits (GC) to evaluate AFs, but experimental evaluations are limited to $\relu$ AF [8, 42, 64, 62][Although Delphi uses GC to evaluate non-linear layers, the MPC friendly
square function used to replace $\relu$ is specific towards $\relu$. Therefore, we do not
consider it compatible with complex AFs.].
Therefore, it is unclear if these GC-based protocols can generalize to other complex AFs such as $\silu$, $\gelu$, and $\mish$. We marked them as unclear in the first column of tab:comparison.

2 Supports large number of hidden layers.
The error introduced due to replacing $\fx$ with its MPC-friendly approximation $\fmpc$ in eqn:problem-formulation can accumulate and possibly lead to a significant loss in accuracy as the number of hidden layers increases.
Unfortunately, few prior works <cit.> that support complex AFs show significant accuracy loss for DNN models with a high number of
hidden layers. $\nfgen$, however, does not exhibit this accuracy loss as the number of hidden layers increases, but this negligible accuracy loss comes at the cost of paying high-performance overhead.
We want our scheme to endure such accuracy loss as the number of hidden layers increases without increasing the performance overhead significantly.

3 Compatible with MPC libraries.
The secure inference procedure we develop should not only support a wide variety of AFs but also should be easy to implement. Implementations that require new cryptographic primitives for secure inference will be hard and slow to deploy.
Therefore, in this work, we aim to design a scheme that is generic to MPC library currently in use. To realize our solution, the MPC library only needs to support addition, multiplication, and comparison operations.
This would also allow seamless transitioning from inference service using $\relu$ based DNN models to complex AF-based DNNs. Except $\nfgen$, other prior works do not satisfy this design goal.

4 No restriction on training. To handle accuracy loss with an increasing number of hidden layers, a few prior works change how DNN models are traditionally trained. For example, XONN requires restricting the weights of the DNN model to binary values (i.e., $\pm 1$); similarly, Delphi replaces certain AFs (i.e., $\relu$) with a square function during training. We believe this type of restriction poses additional constraints as already trained DNN models are most likely trained traditionally without these restrictions, and further attempts to adjust the weight of the already trained DNN models (e.g., fine-tuning the trained DNN model by applying these restrictions) to comply with these protocols would be expensive. Therefore, we aim to design without any restriction on the training process of the DNN models.

In summary, recent proposals in secure inference literature are making great strides toward realizing secure inference; but they do not focus on satisfying the above-mentioned generability, deployability, and scalability aspects important for realizing secure inference in the real world.

We aim to bridge this gap via our designed scheme .

§.§ Threat Model

We assume a general setup of an MPC scheme and henceforth inherit their security requirements.
More specifically, in MPC schemes, the adversary $\adv$ is parameterized by four dimensions [45].
They are i) corruption strategy (static, adaptive, proactive), ii) type of $\adv$ in terms of how they are following the protocol (semi-honest, malicious, covert), iii) corruption ability $\adv$ has (honest, dishonest majority), and iv) power of the $\adv$ (informational, computational-secure).
Our techniques do not apply any restriction on how the adversary $\adv$ is modeled by the MPC scheme. In our experiments, we use MPC schemes that are secure against honest-but-curious adversaries.

§ : OVERVIEW AND DESIGN

In this section, we first give an overview of our scheme in sec:overview. Then, we detail
our scheme gradually in sec:accurate-approximations and sec:time. We sketch our scheme in algo:candidateListGen with a summary of notations used in tbl:notations.

§.§ Overview of $\sysname$

Piece-wise polynomial approximation approach. Our scheme $\sysname$ follows the idea of approximating a complex activation function (AF) using a number of piece-wise polynomials.
First, we observe that complex AF can be approximated easily using linear functions outside a certain range. (Fan et al. made similar observations for $\sigmoid$ [18].)
Therefore, we only need to focus on a small range of $x$ values, say $[s, e]$, which require approximation.
We will approximate $\fx$ using a piece-wise polynomial function, with $\npieces$ pieces $[\poly_1, \poly_2, \cdots, \poly_m]$ defined as follows:

\begin{equation}
\fmpc(x) = \sum_{i=0}^{m+1} \indicator_i\cdot \poly_i(x)
\label{eqn:approx}
\end{equation}

where $\indicator_i(x) = 1$ if $x\in (x_{i-1}, x_i]$, and $0$ otherwise, for all $i \in \{0, 1, \ldots, m\}$, $x_{-1} = -\infty$, $x_0 = s$, $x_m = e$, and $I_{m+1}(x) = 1$, if $x>e$, and 0 otherwise.  The functions $\indicator_i$ define the pieces, and functions $\poly_i$ define the polynomials.

We impose an additional constrain that all polynomials
must be of degree
$\maxdegree$ or less,   $\forall i \ \deg(\poly_i) \le \maxdegree$ as shown in eqn:polynomial

\begin{equation}
\poly_i(x) = a_0 + a_1x + a_2x^2 + \cdots +   a_{k}x^k
\label{eqn:polynomial}
\end{equation}

The above-mentioned approach is not specific towards $\relu$  and can approximate complex AFs (e.g., $\silu$, $\gelu$, $\mish$) — satisfying design goal 1 as described in sec:requirements. Furthermore eqn:approx,<ref> comprise of three math operations $\addmathoperation$, $\mulmathoperation$, and $\comparemathoperation$ and the majority of the MPC libraries support these three math operations — making $\sysname$ generic to the MPC library being used and thus satisfying design goal 3.
However, generally, approximation-based approaches tend to be inaccurate [37].

Thus, maintaining negligible accuracy loss with increasing hidden layers (design goal 2) and imposing no restriction on training (design goal 4) become challenging.
We address this challenge by developing the techniques described next.

§.§ Generating Accurate Approximations

To approximate $\fx$  for a given region $[s,e]$ using  $\npieces$  piece-wise polynomials with degree at most $\maxdegree$ and has negligible accuracy loss, we use an opportunistic approach $\candidatelistgenerationalgo$ as shown in algo:candidateListGen.
Note that our approach, as mentioned earlier, shares similarities with $\nfgen$ [18] on the interpolation aspect only (described in sec:interpolation for completeness).
Other techniques, described next, are tailored to complex AF used in DNN models and unique to $\sysname$.
This makes computationally more efficient than $\nfgen$ as the number of hidden layers increases (experimentally illustrated in sec:performance).
We now describe these techniques employed by in detail as follows.

§.§.§ Computing $\density(x)$

We aim to design approximate polynomials that are close to accurate on likely values of $x$, meaning higher probability according to $\density(x)$, while may have higher error on values of $x$, which are less likely. A challenge, however, is how to estimate the distribution $\density$.
Interestingly, in DNN models, the inputs to an AF are first batch normalized (BN) using eqn:bn, to help the network converge faster during training (discussed earlier in sec:normalization).
Therefore, the set of values the AF is computed on is distributed (approximately) as a Gaussian distribution with zero mean and unit standard deviation.
The approximating piecewise polynomial, therefore, should ensure low error on highly likely inputs, whereas on low probable inputs, it may make a higher error.

Our key insight is that $\density(x)$ can guide us to focus
on approximating those regions more accurately
where $\density(x)$  is high. In contrast, we can get away with
a less accurate approximation where $\density(x)$
is close to zero without degrading the accuracy significantly.
Moreover, due to BN applied on DNN layers prior to applying the AFs, we estimate $\density(x)$ using a standard normal distribution $\normal(0, 1)$.
However, incorporating $\density(x)$ to the approximation procedure is not straightforward, and we use a customized function to compute the approximation error that takes into account $\density(x)$ as we describe next.
We remark one caveat of this design choice — becomes reliant on BN (discussed further in sec:discussion).

§.§.§ Designing $\Error$

To incorporate $\density(x)$ to the approximation procedure, we customize an
approximation error  which we refer to as weighted mean approximation error denoted by $\MeanApproxError$.

\begin{multline}
\MeanApproxError(\density, \fmpc, \alpha, \beta) = \\
\frac{1}{(\beta-\alpha)} \int \limits_{\alpha}^{\beta}\density(x) \cdot |\fx(x) - \fmpc(x)| dx
\label{eqn:mean_approximation_error}
\end{multline}

As shown in eqn:mean_approximation_error,  in addition to considering how accurately $\fmpc$ estimates $\fx(x)$ for a given region between $s$ and $c$, $\MeanApproxError$   also takes $\density(x)$ into account.
Prior work $\nfgen$ [18] uses max approximation error, which we denote by $\MaxApproxError$ — as a way to design $\Error$.
(, α, β) = max_x ∈[α, β] |(x) - (x)|

We choose to use $\MeanApproxError$ over $\MaxApproxError$ as it is easy
to guide the approximation process via $\density(x)$ using $\MeanApproxError$.

§.§.§ Selecting a threshold $\ethreshold$ for approximation error $\MeanApproxError$

A straightforward ad hoc way to ensure the accurateness of the approximation is to set a fixed approximation error threshold  ($\ethreshold$) and consider an approximation accurate if approximation error calculated via $\Error$ is $\le \ethreshold$.
$\nfgen$ also follows this ad hoc approach and sets $\ethreshold = 10^{-3}$.
Via empirical experimentation, they observed that if  $\MaxApproxError \le 10^{-3}$, then the generated approximation, when used in logistic regression and $\chi^2$ testing, does not degrade accuracy without adding much performance overhead.

We refrain from setting a fixed $\ethreshold$ for as the appropriate $\ethreshold$ may vary well from one DNN model/dataset to another.
Also, should systematically find the appropriate $\ethreshold$, relieving the practitioners of the additional burden of finding an appropriate $\ethreshold$ on their own.
Thus, discovers an appropriate $\ethreshold$ by performing a binary search (BS) over $\ethreshold$ and finding the highest $\ethreshold$ such that the approximation corresponding to $\ethreshold$ incurs a negligible inference accuracy loss. This is sound due to the monotonic relationship between approximation error and inference accuracy.
Lastly, one challenge of this approach is checking if the inference accuracy loss is negligible or not —  at each step of BS. We describe a solution to this challenge next.

§.§.§ Measuring accuracy loss

One of the crucial components of $\candidatelistgenerationalgo$is for finding the appropriate $\ethreshold$ is to tell apart if the generated $\fmpc$ renders a negligible accuracy loss.
Unfortunately, it is difficult to tackle this challenge analytically.

We attempt to handle this challenge empirically by relying on that well-known closed-world assumption used in machine learning — that is, if something performs well on the training/validation dataset, it will perform equally well on the test data.
More specifically, we replace the original $\fx$ with the generated MPC-friendly approximation $\fmpc$ and calculate the inference accuracy over the training dataset.
We call this  inference accuracy
$\eta_2$ and compare it with the plaintext inference accuracy $\eta_1$ which uses the original $\fx$ over the same training dataset.
If $(\eta_1 - \eta_2)/\eta_1 \le \negligibleaccuracy$, we consider $\fmpc$ to be accurate enough, where $\negligibleaccuracy$ is a small value representing the accuracy loss the practitioner can tolerate.

§.§.§ Designing $\fcrude$

We also added another DNN model-specific optional optimization.
Instead of approximating the original $\fx(x)$, we manually
introduce a crude MPC-friendly approximation[For simplicity this is not shown in algo:candidateListGen $\candidatelistgenerationalgo$.] of $\fx(x)$, which we call $\fcrude$.
Then, set to approximate $\fx(x) - \fcrude(x)$ using $\candidatelistgenerationalgo$ instead. The final approximation of an AF would be $\fcrude(x) + \fmpc(x)$.  Note that $\fcrude$ is designed to be simple and linear, making it easy to use with standard MPC libraries. We found this approach significantly improves the approximation procedure

For $\silu$ AF, since $\silu(x) = x \cdot \sigmoid(x)$, we can simply
borrow the
structure of the MPC-friendly approximation for
$\sigmoid(x) \approx  \max(0, \min(x + 0.5, 1))$ from [54]. We tweak it slightly to be more precise and multiply it by $x$ to get $\newfcrude{silu}$ as shown in eqn:crude-silu.

\begin{equation}
\label{eqn:crude-silu}
\newfcrude{silu}(x)  = x \cdot \max\Big(0, \min(6x + 0.5, 1)\Big) \\
\end{equation}

For $\gelu$ AF, since $\gelu(x) \approx x \cdot \sigmoid(1.702x)$, in a similar way we can write crude MPC-friendly approximation of $\gelu$ AF by leveraging the same structure of MPC approximation for $\sigmoid$ as shown in eqn:crude-gelu.

\begin{equation}
\label{eqn:crude-gelu}
\newfcrude{\gelu}(x)  = x \cdot  \max\Big( 0, \min(10x, 0.5)\Big)
\end{equation}

Since $\mish$ cannot be expressed easily in terms of $\sigmoid$, we denote crude MPC
friendly approximation of it by $\relu$ as shown in eqn:crude-mish.

\begin{equation}
\newfcrude{\mish}(x) = \max(0,x)
\label{eqn:crude-mish}
\end{equation}

§.§.§ Performing interpolation

We  interpolate $f(x)$  between range $[\alpha, \beta]$  by a $\maxdegree$ degree polynomial $\poly$ (eqn:polynomial)
using the  $\interpolate$ procedure.
To find the best performing $\poly(x)$, similar to $\nfgen$, we adopt
Chebyshev interpolation [48]
over other alternatives, such as cubic spline or uniform polynomial.
This is due to an established fact in the area of function approximation theory that Chebyshev  polynomial interpolation generally has superior performance to
cubic spline or uniform polynomials interpolation when $f(x)$
is smooth and monotonic (c.f., [50] Table 5.1) as it is in our case for complex AF used in DNN models (fig:actfuncs).

$\candidatelistgenerationalgo$ procedure.
Now we can piece together the above-mentioned techniques and describe the procedure to approximate
$\fx$ within region $[\alpha, \beta]$ using a number of piece-wise polynomials
in detail (as shown in  $\candidatelistgenerationalgo$ algo:candidateListGen).
First, we set a step size $\stepsize$, and at each step, we increase the pointer $\beta$ by $\stepsize$.
Before moving $\beta$, we check if
the adjusted approximation error $\ethreshold'$ in the region $[\alpha, \beta]$ is more than the  expected approximation error $\ethreshold/\npieces$.
If this is the case, we approximate the region $[\alpha, \beta]$ using a polynomial determined using the Chebyshev interpolation algorithm, add that polynomial piece to $\fmpc$,  and update $\alpha$ to $\beta$.
Next, we update $\beta$ by $\stepsize$ and perform the above-mentioned check again until we have approximated
the whole region $[s, e]$.

§.§ Finding Computationally Efficient Approximation

Now that we can generate MPC-friendly approximations $\fmpc$
using $\candidatelistgenerationalgo$ procedure that have negligible accuracy loss, we can search over all possible values of  $\langle \npieces, \maxdegree, \ring \rangle$ and select the  $\fmpc$ that is computationally efficient. We abuse the notation slightly and use $\theta$ to represent $\langle \npieces, \maxdegree, \ring \rangle$.

Unfortunately, because of the systemic approach we take to find the appropriate $\ethreshold$, $\candidatelistgenerationalgo$ becomes time consuming.
This is because determining if the accuracy loss is negligible at each step of binary search with reasonable confidence requires performing inference over the large training dataset (as explained in sec:accuracy-loss-measure), and it makes exhaustively iterating over all possible $\theta$ infeasible.
Instead, we devise an application-specific searching technique based on simulated annealing (SA) [35].
One advantage of sketching SA-based searching for optimal $\theta$ is that it is gradient-free — suiting our needs, overcoming the difficulty to underpin an analytical formula of $\nabla_{\theta = \langle \npieces, \maxdegree, \ring \rangle} \candidatelistgenerationalgo(.)$.
That being said, other gradient-free searching techniques may also work as well [17], and we detail additional discussions in app:minmax.

One important characteristic of SA — we need to model for this case — is how to avoid being trapped in a local suboptimal solution.
To this extent, we follow suggestions from prior work [12, 33], and probabilistically move towards a new solution $\theta_{\step}$ even if $\theta_{\step}$ is computationally less efficient approximation than the current best solution ($\tcur$).
More precisely, if  at $\step$-th iteration, we denote the MPC-friendly approximation from $\theta_{\step}$ as $\fmpc^{\step}$, then we always update our current best solution $\tcur$ to
$\theta_{\step}$ if $\fmpc^{\step}$ is computationally more efficient than $\fcur$ (i.e., $ \Time(\fcur) > \Time(\fmpc^{\step})$). Otherwise, we update $\tcur$ to $\theta_{\step}$ with a certain acceptance probability.
This probability depends on two factors. First, the temperature
at $\step$-th iteration called $T_\step$ — which is initially high, meaning we have a high tendency to accept a less computationally efficient solution, but after a few more iterations $T_\step$ decreases and so does our tendency to accept a computationally less efficient solution.
Second, the amount of computation less efficient $\fmpc^{\step}$ is compared to $\fcur$.
In other words, we accept $\theta_{\step}$ using the following equation.

θ_      () > (^)  or

exp((()-(^))/T_) >

r U_[0, 1]

$\Time(\cdot)$  is the procedure that returns the average time it takes to complete secure inference using the corresponding approximation, which can be logged simultaneously when we run the secure inference using $\fmpc$ inside $\candidatelistgenerationalgo$.

We have to design two more parameters carefully. One is the neighborhood generation heuristic for $\theta$, and the other is setting a cooling schedule for the temperature $T_\step$.
Without careful handling of these two parameters
SA may lead to undesired approximations [5].

Neighbour generation heuristic.
At iteration $\step$,
we generate a new neighbor  $\theta_\step = \langle \npieces', \maxdegree', \ring' \rangle$ from  $\theta = \langle \npieces, \maxdegree, \ring \rangle$
in the following way: for $\npieces', \maxdegree'$ we randomly sample two integers numbers $z_1, z_2\in\Z$
such that
$P(X = z) = 1/3\cdot2^{|z|}$ and set $\npieces' \gets \npieces + z_1$ and  $\maxdegree' \gets \maxdegree + z_2$.
This means the chances of moving further away from the current value  $\npieces$ and  $\maxdegree$ decreases exponentially.

Handling $\ring_{\langle n,d\rangle}$ requires a bit more consideration.
Note to specify a $\ring$, we need two numbers: i) $n$, the size of the ring used in MPC library, and ii) $d$, the number of last bits to represent the fractional parts.
Typically MPC libraries uses $\ring$ sizes of $\{128, 84, 64, 32\}$.
We i.i.d sample a ring size from these for $n$, and regarding values of $d$ we set it to $d \gets \floor{n/\gamma_2}$ where $\gamma_2$ is i.i.d sample from  $\gamma_2 \in_{R}\{3/2, 2, 5/2,3, 7/2, 4\}$.

Cooling schedule.
As for the cooling schedule, we use the classical logarithmic series  $T_\step \gets \chi_0/\log(\step + 1)$ at iteration $\step$ from Hajek et al. [24]. This choice ensures that initially, $T_\step$ would be high, thereby increasing the chances of accepting a computationally less efficient approximation during the early iterations. But as the number of iterations increases, $T_\step$ progressively decreases, lowering this chance.

We simply set $\chi_0 = 0.2$ for all of our experiments, yielding $T_{1} \approx 0.67$ and  $T_{10} \approx 0.2$.

We show the pseudocode for finding computationally efficient approximation  $\ApproximationAlgo$ and the procedure for generating neighbors at each iteration $\generateNeighbour$ in algo:candidateListGen.

§ EXPERIMENTAL EVALUATION

We conduct experiments to address the following questions:

* Model Accuracy (sec:accuracy):
What is the impact on model inference accuracy of using MPC-friendly activation functions $\fmpc(x)$ generated using our scheme $\sysname$ and
other existing approaches  [18, 47, 43, 62]?

* Inference Time
What is the inference time overhead of $\sysname$ compared to $\nfgen$ [18]  as the number of hidden layers increases without losing any significant loss in inference accuracy?

§.§ Implementation Details

Our Scheme.
We implement our scheme
using Python 3.8 in $\approx 1,200$ LoC. We approximate the region between $x \in [-10, 10]$ for all activation functions (AFs) as beyond that region, they can be easily approximated using polynomials.
We also use   [49]
library for the majority of mathematical operations, including calculating the approximation error between a given region of a polynomial using eqn:mean_approximation_error and performing  Chebyshev interpolation as mentioned in sec:interpolation.

Our scheme requires testing if the generated approximation has negligible accuracy loss by checking $(\eta_1 - \eta_2)/\eta_1 \le \negligibleaccuracy$ (as described in sec:accuracy-loss-measure).

We also configure $\ApproximationAlgo$ with ten iterations ($\step_{max} = 10$) to find a computationally efficient approximation and set $\chi_0=0.2$.
For the initial solution $\theta_0 = \langle \npieces_0, \maxdegree_0, \ring_0 \rangle$, we set  $\npieces_0 = 10^4$ and  $\maxdegree_0 = 10$; default parameters taken from $\nfgen$ [18].
For $\ring_{\langle n_0, d_0 \rangle}$, we used
$\langle n_0, d_0 \rangle = \langle 128, 64 \rangle$  —  a popular choice of ring size by many MPC libraries.
With this configuration, $\ApproximationAlgo$ took less than 25 minutes on commodity hardware to finish the four tasks and three complex AFs we detail in sec:image-classification.

tbl:configurations shows the appropriate $\npieces, \maxdegree, \ring$ we find via $\ApproximationAlgo$ for all tasks and complex AFs.

We note can also be implemented by
adding our DNN-specific optimizations (as discussed in sec:our-piece-wise-poly) on top of $\nfgen$ code base, and we expect to see similar efficacy over  $\nfgen$ — to what Fan et al. propose in [18].
However, when $\nfgen$ was publicly released[<https://github.com/Fannxy/NFGen>], we were already towards the end of the development cycle of our code base, and hence, we did not reuse their code base for implementing techniques employed by .

Other Approaches.
We consider four state-of-the-art approaches for comparison:
$\nfgen$ [18], MiniONN [47], MPCFormer [43] and SIRNN [62].
Additionally, we consider a rudimentary base approach: replacing the complex AF with a popular MPC-friendly AF $\relu$.
We consider this approach as $\relu$ is relatively MPC-friendly because it can be computed using only two piece-wise polynomials.

For $\nfgen$, we add a wrapper class to the author's open-source implementation to measure the inference accuracy and computational overhead for the four tasks. Besides that, we keep their implementation unchanged — using  $\MaxApproxError$ (eqn:max_approximation_error) to measure the approximation error, setting $\ethreshold = 10^{-3}$, $\maxdegree = 10$, and $\npieces = 10^4$.

In  [47], Liu et al. describe an approach called MiniONN for generating MPC-friendly approximations of $\sigmoid$
AF. Since there is no publicly available implementation of MiniONN, we implement it ourselves to the best of our abilities and extend the approach to generate MPC-friendly versions of complex AF $\fx \in \{\silu, \mish, \gelu\}$ (see  app:MiniONN for further details of their approach).

MPCFormer [43] approximates $\gelu$
using a polynomial $\gelu(x)= 0.125x^2 + 0.25x + 0.5$. This approximation was motivated by the need to perform secure inference for transformer-based DNN models where $\gelu$ activation is used extensively.
Since Li et al.  [43] did not provide any recipes that can be generalized directly to other AF, we only compare the accuracy and computational overhead for $\gelu$.

Lastly, Rathee et al. present a library called SIRNN [62] that computes complex mathematical operations (e.g., $e^{x}, \ln(x), \frac{1}{x}$) securely using a combination of
lookup tables and numerical methods (e.g., Goldschmidt's iterations).
Thus, complex AFs can be computed sequentially by performing
the aforementioned operations and combining the intermediate results using $\addmathoperation, \mulmathoperation, \comparemathoperation$ operators to evaluate $\fx$. Recently, Hao et al. [25] extended their approach to computing $\gelu$ activation function in an efficient manner by reducing one additional network call. Nevertheless, this work uses the open source  implementation of SIRNN [55].

§.§ Experimental Setup

Task details.
To demonstrate that inference accuracy and performance overhead
is negligible
for secure inference using
our scheme,
we  consider four state-of-the-art image classification
tasks as shown in tab:classification-accuracy and three complex activation functions (AFs) $\fx \in \{\silu, \gelu, \mish\}$.
We train the four models corresponding to each complex AF for each task.
While training these models, we preserve
the widely use parameters as proposed in the
literature for all models (e.g., the overall architecture of the model,
# of epochs, learning rate, optimizer, etc.) —  including a batch normalization layer before inputs are being fed to
complex AFs of each hidden layer as illustrated in fig:normalization.
Below, we provide brief details about these four classification tasks, and further details are in app:dnn.

Four classification tasks.
For the first task,
we consider a simple classification task of MNIST dataset [41] using
a three-layer deep fully connected network (FCN) with one input, output, and hidden layer. MNIST dataset contains  70 K 28x28 handwritten digits grey images, and the three-layer deep FCN achieves
close to $0.99$ training accuracy for the three complex AFs. We refer to this task as
in the paper.

Next,  we move towards a more complex classification task of
CIFAR-10 dataset [39] — which we refer to as .
CIFAR-10 consists of
60 K 32x32 color images with 6 K images per 10 classes.
For performing classification on this dataset, we use a
a convolutional
neural network (ConvNet) [59] with five hidden layers
and train it over the 50 K training images of CIFAR-10 dataset using three different complex AF.

For the third task, we consider performing classification on ImageNet-1K dataset which has been one of the challenging benchmark dataset in image classification  [13]. The dataset contains around 1 million annotated images with 50 K validation images and 100 K test images. We use a deep
residual neural network (ResNet9) [26] model having eight hidden layers over the training images for 50 epochs for three complex AFs and achieved a validation accuracy of around 0.74.  We refer to this task as  in this paper.

Lastly, we perform experiments to detect spoofed images in CelebA-Spoof [76] dataset.
We refer to this task as . This is a large-scale face anti-spoofing dataset use to train anti-spoofing DNN models. CelebA-Spoof contains 625 K facial images from $>$10 K subjects with each image has 43 attributes; 40 of them correspond to indicating facial components of real images and three of them correspond to attributes of spoofed facial images.
For training, we perform an 80-20 split of the CelebA-Spoof dataset and
adopted the EfficientNetB0 [67] model, which is the state-of-the-art top-performing anti-spoofing detection model and winner of the CVPR challenge of detecting spoofed face images[46].
EfficientNetB0 model consists of 17 hidden layers, and after training the model for 25 epochs, it achieved a training accuracy of 0.98.

Machine specification.
We train the models on a Linux machine with an Intel Core i9 processor having 128 GB RAM and Nvidia GTX 1080 GPU. We use the training split of each dataset for training the models, and after the training is completed, we save these models.  We assume $\modelowner$ holds these saved models and does not want to reveal them to $\client$ while performing secure inference.
We simulate the $\client$'s input $\clientinput$
using the testing split of the corresponding datasets for each task.

§.§ Model Accuracy

We first measure the inference accuracy of the trained models over the testing split of the dataset by using the (non-MPC-friendly) complex AF as it is and refer to it as plaintext accuracy $(\eta_1)$.
Then, we replace the complex AF with its MPC-friendly approximation generated by different approaches and measure its inference accuracy $(\eta_2)$. Thus, $(\eta_1-\eta_2)/\eta_1$ gives the inference accuracy loss introduced by MPC-friendly approximations.

shows the inference accuracy loss (in percentage) for generated MPC-friendly approximation
and other state-of-the-art approaches generated MPC-friendly approximation [18, 25, 47] — for each task across the three complex AFs $\silu$, $\gelu$, $\mish$.
Now we discuss the inference accuracy loss for different approaches, and throughout the discussion, we conservatively consider accuracy loss negligible if $(\eta_1-\eta_2)/\eta_1 < 10^{-2}$.

$\relu$ based rudimentary approach.

We observe that although for the first  task, the inference accuracy loss is within 1.54%-2.68% for the last three tasks

accuracy loss is higher — at least 45.66% — making this approach unsatisfactory.

SIRNN [62].
For SIRNN, we observe that for  task, we observe less significant accuracy loss ($0.95\%-2.37\%)$.
Furthermore, for  the accuracy does not degrade too much — by 0.48%-1.78%.
However, for  and  task the accuracy degradation is evidently not negligible — suffering from an accuracy loss
of 2.58%-16.31%.

We  hypothesize such accuracy degradation is primarily
due to two reasons:
1) intermediate steps overflow in the fixed point representation,
and 2) error is introduced while one computing one complex math operation propagates and
accumulates while using that result as input for another complex math operation.
This further motivates the need to  take a piece-wise polynomial approximation-based
approach for designing MPC-friendly approximation of complex AF when state-of-the-art DNN models are used, confirming findings from prior work [18].

MPCFormer [43]

For MPCFormer, we observe a negligible accuracy loss for  and  tasks of 0.18% and 0.09% respectively. However, similar to SIRNN, it exhibits a non-negligible
accuracy loss of 9.4% and 7.07% for  and  task respectively.
We suspect this is because $\gelu$ activation approximation by MPCFormer
relies on knowledge distillation (KD) [29]– which is essentially fine-tuning the sequence-to-sequence-based pre-trained model for efficiency. In absence of KD, a simple plug-and-play replacement of polynomial approximation of $\gelu$ activation proposed by MPCFormer does not work well.

MiniONN [47].
For MiniONN, we observe that the inference accuracy loss becomes significant when we use their recipe to generate a friendly approximation of complex AFs $\silu$, $\gelu$, $\mish$.
The accuracy loss becomes catastrophically high, especially for , (27.12%-39.89%).
This shows that although the recipe proposed by MiniONN does not show accuracy degradation
for $\sigmoid$ AF for simplistic logistic regression models, there is a generalization gap when such recipes are used
for DNN models trained on diverse datasets involving complex AFs.

$\sysname$ and $\nfgen$ [18]
We observe that for all tasks, in general, and generated MPC-friendly approximations have
negligible accuracy loss of $< 1\%$. For one instance, though,   task involving $\silu$ AF, $\nfgen$ has an accuracy loss of 1.36% — marginally higher than the aforementioned threshold. When
When comparing the two approaches, generally, $\sysname$ generated approximation has lower accuracy loss, except for two instances showing a slight deviation.
The first one is  task involving $\gelu$ (0.37% vs 0.23%) and  task involving $\mish$ AF (0.66% vs 0.53%).

Results summary. We conclude from these experiments that $\nfgen$ and $\sysname$ are resistant towards significant accuracy loss — when we use their generated MPC-friendly approximation instead of the original complex AF — compared to other approaches we consider. Keeping that in mind, we can now investigate the next important aspect of secure inference, measuring performance overhead. We narrow down our experiments to $\nfgen$ and $\sysname$ — excluding the other approaches — as their accuracy loss is significantly high.

§.§ Inference Time

We benchmark the inference time of  $\nfgen$ and $\sysname$ to measure the performance overhead.
While benchmarking, we instantiate each party in the protocol by machines running on commodity-type hardware  —  having an Intel Core i7 processor with 64 GB RAM and connected over a 252 Mbits/sec network link.
We use the average inference time for a single image calculated over the testing split of the datasets and include both computational and communication costs while reporting the results.

We consider two state-of-the-art MPC libraries [53, 63] designed for secure inference  —  one for a 2PC scenario and the other for a 3PC scenario (both scenarios are described earlier in sec:problem-formulation-and-scenario-setup).

3PC results.
First, for the 3PC scenario, we consider  [53] that uses replicated secret sharing (SS) based secure inference protocol.
Table <ref> compares the performance overhead
of and $\nfgen$ for the 3PC setting using  library.
We observe that $\sysname$ outperforms  $\nfgen$ 2$\times$-5$\times$ for the last three classification tasks involving a high number of layers. However, $\sysname$'s performance efficacy for the first task  is comparable to $\nfgen$ — exhibiting similar inference time.
We hypothesize this is because the number of hidden layers is only one for the DNN model used in this
first task. In contrast, the number of hidden layers for the other classification tasks is 5, 8, and 17 respectively. Because of this, there is a higher chance of the approximation errors introduced in one hidden layer propagating to the next hidden layers.
Our DNN-specific techniques discussed in sec:our-piece-wise-poly can effectively curl out this approximation error from propagating to the hidden next layers without sacrificing much performance overhead compared to $\nfgen$. Thus,  $\sysname$'s  superior performance   becomes more pronounced as
number of hidden layers becomes high.

2PC results.
For the 2PC scenario, we consider  CryptFlow2 [63]  —  another state-of-the-art library for secure inference based on a novel protocol for millionaries' problem [75] and division over fixed-point arithmetic
We experiment with the oblivious transfer (OT) based construction of CryptoFlow2 but believe performance results would also be similar for homomorphic encryption (HE) based construction.
We observe a performance efficiency trend of with similar to the 3PC scenario. We present detail results in app:crytpflow2.

§ DISCUSSION AND FUTURE WORK

enables fast secure inference for DNN models that use complex activation functions (AFs) while protecting the secrecy of the client's inputs as well as the proprietary DNN model.
Deploying is straightforward as it is compatible with standard MPC libraries. Here we discuss other deployment considerations for practitioners to use .

Accelerating secure inference time using GPU.
In the plaintext setting, impressive ML inference time has been achieved by harnessing

GPUs, which support highly parallelizable workloads.
This boost in inference speed also extends to secure scenarios. Indeed, recent works have shown how to run MPC operations inside GPUs and gain a significant speedup in machine learning training and inference [73, 68]. In our experiments, we did not use GPUs for inference, and thus
the inference times reported in tab:performance can be further improved by porting these techniques to and CryptFlow2. We note that is compatible with any MPC protocol that supports the three basic operations, and both and CryptFlow2 support them. We acknowledge that this endeavor of making and CryptFlow2 GPU-friendly requires considerable development time and thus we leave that for future explorations. Nevertheless, we believe the relative gain in performance improvement will remain the same, as the operations $(\ADD, \MULT, \COMP)$ used in our technique are very similar to those used in prior work.

Easy Adoption of $\sysname$ in practice.
Our approach focuses on complex AFs, and we believe complex AF gets precedence over
$\relu$ for two key reasons.
Firstly, complex AFs, in many cases, result in performing robust, noise-resistant, better-performing DNN models (e.g., $\mish$ have higher accuracy in computer vision and object detection tasks than $\relu$ [52], DNN models trained on $\silu$ are more noise-resistant than the ones trained on $\relu$ [27], etc.). Secondly, the DNN models have been trained over a complex AF in many cases. Retraining/fine-tuning the DNN model further using  $\relu$ AF to make it compatible with secure inference protocol specific to $\relu$  for enabling privacy-preserving inference is difficult for practitioners (e.g., state-of-the-art face anti-spoofing detection model EfficientNetB0 by default is pre-trained on $\silu$ AF [72]).

$\sysname$ is more useful than the state-of-the-art generic MPC-friendly approximation approach for non-linear functions  $\nfgen$ when the number of hidden layers is more than one. This phenomenon is illustrated by $\sysname$'s similar performance to $\nfgen$ for  where the number of hidden layers is one. However, as the number of hidden layers increases for the other three tasks, $\sysname$ outperforms $\nfgen$ by 2$\times$-5$\times$.  It is also worth noting that our work is experimentally evaluated on the three most popular complex AFs used in the ML community, and one can easily use our approach to approximate other less widely used complex AFs (e.g., $\tanh$, Smish, etc.).

Dependence on batch normalization.
Our piece-wise approximation is dependent on the presence of batch normalization (BN).
Generally, BN is employed before AFs by the majority of state-of-the-art DNN models in computer vision. $\sysname$ leverages the phenomenon that BN shifts the input distribution to have zero mean and unit variance (similar to standard normal distribution $\normal(0,1)$).
We believe another type of normalization, typically used in natural language processing, called layer normalization, can also be leveraged to design an approach similar to ours in the future.

For now, it is clear that if one has a DNN model with a large number of hidden layers, which is already trained using complex AF and batch normalization,
would be useful to do secure inference on that model without sacrificing the accuracy or requiring any retraining. This will significantly simplify deployability. Moreover, compatibility with existing MPC libraries makes it easy for practitioners to easily switch to if they are already using a secure inference.

§ CONCLUSION

To protect the privacy of clients' private input sent to a proprietary  DNN model hosted on a cloud-based inference service, we present a scheme to generate MPC-friendly piece-wise polynomial approximations of complex activation functions (AFs).

The generated MPC-friendly approximations can directly replace existing
complex AFs used in DNNs in a plug-and-play manner and obviate the need to retrain DNN models. Our extensive experiments show that the proposed approach achieves better performance overhead computer vision models trained on challenging datasets while maintaining similar model accuracy.

[1]
Machine Learning as a Service: What It Is, When to Use It and What Are the
Best Tools Out There.
July 2022.

[2]
Toluwani Aremu and Karthik Nandakumar.
Polykervnets: Activation-free neural networks for efficient private
In 2023 IEEE Conference on Secure and Trustworthy Machine
Learning (SaTML), pages 593–604, 2023.

[3]
Amazon AWS.
Image Classification - MXNet.
June 2023.

[4]
Shayan Aziznejad and Michael Unser.
Deep spline networks with control of lipschitz regularity.
In ICASSP 2019-2019 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pages 3242–3246. IEEE, 2019.

[5]
Walid Ben-Ameur.
Computing the initial temperature of simulated annealing.
Computational optimization and applications, 29:369–385, 2004.

[6]
Pakshal Bohra, Joaquim Campos, Harshit Gupta, Shayan Aziznejad, and Michael
Learning activation functions in deep (spline) neural networks.
IEEE Open Journal of Signal Processing, 1:295–309, 2020.

[7]
Pakshal Bohra, Dimitris Perdios, Alexis Goujon, Sébastien Emery, and
Michael Unser.
Learning lipschitz-controlled activation functions in neural networks
for plug-and-play image reconstruction methods.
In NeurIPS 2021 Workshop on Deep Learning and Inverse Problems,

[8]
Nishanth Chandran, Divya Gupta, Sai Lakshmi Bhavana Obbattu, and Akash Shah.
SIMC: ML inference secure against malicious clients at
Semi-Honest cost.
In 31st USENIX Security Symposium (USENIX Security 22), pages
1361–1378, Boston, MA, August 2022. USENIX Association.

[9]
Google Cloud.
Media Translation: Add real-time audio translation directly to your
content and applications.
<https://cloud.google.com/media-translation>, June 2023.

[10]
Google Cloud.
Recommendations AI: Deliver highly personalized product
recommendations at scale.
<https://cloud.google.com/recommendations>, June 2023.

[11]
Anders Dalskov, Daniel Escudero, and Marcel Keller.
Secure evaluation of quantized neural networks.
Proceedings on Privacy Enhancing Technologies,
2020(4):355–375, 2020.

[12]
Kalyanmoy Deb.
Optimization for engineering design: Algorithms and examples.
PHI Learning Pvt. Ltd., 2012.

[13]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In 2009 IEEE conference on computer vision and pattern
recognition, pages 248–255. Ieee, 2009.

[14]
Abdulrahman Diaa, Lucas Fenaux, Thomas Humphries, Marian Dietz, Faezeh
Ebrahimianghazani, Bailey Kacsmar, Xinda Li, Nils Lukas, Rasoul Akhavan
Mahdavi, Simon Oya, et al.
Fast and private inference of deep neural networks by co-designing
activation functions.
arXiv preprint arXiv:2306.08538, 2023.

[15]
Ashlea Ebeling.
IRS To Drop Facial Scan ID.me Verification For Online Accounts.

[16]
Stefan Elfwing, Eiji Uchibe, and Kenji Doya.
Sigmoid-weighted linear units for neural network function
approximation in reinforcement learning.
Neural Networks, 107:3–11, 2018.

[17]
Yavuz Eren, İbrahim B Küçükdemiral, and İlker
Introduction to optimization.
In Optimization in renewable energy systems, pages 27–74.
Elsevier, 2017.

[18]
Xiaoyu Fan, Kun Chen, Guosai Wang, Mingchun Zhuang, Yi Li, and Wei Xu.
NFGen: Automatic Non-Linear Function Evaluation Code Generator for
General-Purpose MPC Platforms.
In Proceedings of the 2022 ACM SIGSAC Conference on Computer and
Communications Security, CCS '22, page 995–1008, New York, NY, USA, 2022.
Association for Computing Machinery.

[19]
Frederic B Fitch.
Warren s. mcculloch and walter pitts. a logical calculus of the ideas
immanent in nervous activity. bulletin of mathematical biophysics, vol. 5
(1943), pp. 115–133.
The Journal of Symbolic Logic, 9(2):49–50, 1944.

[20]
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart.
Model inversion attacks that exploit confidence information and basic
In Proceedings of the 22nd ACM SIGSAC conference on computer and
communications security, pages 1322–1333, 2015.

[21]
Karthik Garimella, Nandan Kumar Jha, and Brandon Reagen.
Sisyphus: A cautionary tale of using low-degree polynomial
activations in privacy-preserving deep learning.
arXiv preprint arXiv:2107.12342, 2021.

[22]
Zahra Ghodsi, Akshaj Kumar Veldanda, Brandon Reagen, and Siddharth Garg.
Cryptonas: Private inference on a relu budget.
Advances in Neural Information Processing Systems,
33:16961–16971, 2020.

[23]
Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig,
and John Wernsing.
Cryptonets: Applying neural networks to encrypted data with high
throughput and accuracy.
In International conference on machine learning, pages
201–210. PMLR, 2016.

[24]
Bruce Hajek.
Cooling schedules for optimal annealing.
Mathematics of operations research, 13(2):311–329, 1988.

[25]
Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang.
Iron: Private inference on transformers.
In Advances in Neural Information Processing Systems, 2022.

[26]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 770–778, 2016.

[27]
Dan Hendrycks and Kevin Gimpel.
Gaussian Error Linear Units (Gelus).
arXiv preprint arXiv:1606.08415, 2016.

[28]
Ehsan Hesamifard, Hassan Takabi, and Mehdi Ghasemi.
CryptoDL: Deep neural networks over encrypted data.
arXiv preprint arXiv:1711.05189, 2017.

[29]
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh.
A fast learning algorithm for deep belief nets.
Neural computation, 18(7):1527–1554, 2006.

[30]
John J Hopfield.
Neural networks and physical systems with emergent collective
computational abilities.
Proceedings of the national academy of sciences,
79(8):2554–2558, 1982.

[31]
Zhicong Huang, Wen jie Lu, Cheng Hong, and Jiansheng Ding.
Cheetah: Lean and Fast Secure Two-Party Deep Neural Network
In 31st USENIX Security Symposium (USENIX Security 22), pages
809–826, Boston, MA, August 2022. USENIX Association.

[32]
Siam Umar Hussain, Mojan Javaheripi, Mohammad Samragh, and Farinaz Koushanfar.
Coinn: Crypto/ml codesign for oblivious inference via neural
In Proceedings of the 2021 ACM SIGSAC Conference on Computer and
Communications Security, pages 3266–3281, 2021.

[33]
Chii-Ruey Hwang.
Simulated annealing: Theory and applications: Pjm van laarhoven and
ehl aarts: D. reidel, dordrecht, 1987, 198 pp., isbn 90-277-2513-6, dfl. 120,

[34]
Sergey Ioffe and Christian Szegedy.
Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
In International conference on machine learning, pages
448–456. PMLR, 2015.

[35]
David S Johnson, Cecilia R Aragon, Lyle A McGeoch, and Catherine Schevon.
Optimization by simulated annealing: An experimental evaluation; part
i, graph partitioning.
Operations research, 37(6):865–892, 1989.

[36]
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan.
$\{$GAZELLE$\}$: A low latency framework for secure neural network
In 27th USENIX Security Symposium (USENIX Security 18), pages
1651–1669, 2018.

[37]
Mahimna Kelkar, Phi Hung Le, Mariana Raykova, and Karn Seth.
Secure poisson regression.
In 31st USENIX Security Symposium (USENIX Security 22), pages
791–808, 2022.

[38]
Manish Kesarwani, Bhaskar Mukhoty, Vijay Arya, and Sameep Mehta.
Model Extraction Warning in MLaaS Paradigm.
In Proceedings of the 34th Annual Computer Security Applications
Conference, ACSAC '18, page 371–380, New York, NY, USA, 2018. Association
for Computing Machinery.

[39]
Alex Krizhevsky and Geoffrey Hinton.
Learning multiple layers of features from tiny images.
Technical Report 0, University of Toronto, Toronto, Ontario, 2009.

[40]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks.
Communications of the ACM, 60(6):84–90, 2017.

[41]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.
Gradient-based learning applied to document recognition.
Proceedings of the IEEE, 86(11):2278–2324, 1998.

[42]
Ryan Lehmkuhl, Pratyush Mishra, Akshayaram Srinivasan, and Raluca Ada Popa.
Muse: Secure inference resilient to malicious clients.
In 30th USENIX Security Symposium (USENIX Security 21), pages
2201–2218, 2021.

[43]
Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P Xing, and Hao Zhang.
Mpcformer: fast, performant and private transformer inference with
arXiv preprint arXiv:2211.01452, 2022.

[44]
Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and
Jörn-Henrik Jacobsen.
Preventing gradient attenuation in lipschitz constrained
convolutional networks.
Advances in neural information processing systems, 32, 2019.

[45]
Yehuda Lindell.
Secure multiparty computation.
Communications of the ACM, 64(1):86–96, 2020.

[46]
Ajian Liu, Chenxu Zhao, Zitong Yu, Anyang Su, Xing Liu, Zijian Kong, Jun Wan,
Sergio Escalera, Hugo Jair Escalante, Zhen Lei, et al.
3d high-fidelity mask face presentation attack detection challenge.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 814–823, 2021.

[47]
Jian Liu, Mika Juuti, Yao Lu, and Nadarajah Asokan.
Oblivious neural network predictions via minionn transformations.
In Proceedings of the 2017 ACM SIGSAC conference on computer and
communications security, pages 619–631, 2017.

[48]
Nicholas Marshall.
Chebyshev interpolation.
<https://web.math.princeton.edu/ nfm2/chebyshev-interpolation.pdf>.

[49]
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej
Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu
Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E.
Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats,
Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel,
Štěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal,
Robert Cimrman, and Anthony Scopatz.
SymPy: symbolic computing in Python.
PeerJ Computer Science, 3:e103, January 2017.

[50]
Mario J Miranda and Paul R Fackler.
Lecture notes in computational economic dynamics; chapter 5 function
<http://fmwww.bc.edu/ec-p/software/Miranda/chapt5.pdf>, 1996.

[51]
Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and
Raluca Ada Popa.
Delphi: A cryptographic inference service for neural networks.
In 29th USENIX Security Symposium (USENIX Security 20), pages
2505–2522. USENIX Association, August 2020.

[52]
Diganta Misra.
Mish: A self regularized non-monotonic neural activation function.
arXiv preprint arXiv:1908.08681, 2019.

[53]
Payman Mohassel and Peter Rindal.
Aby3: A mixed protocol framework for machine learning.
In Proceedings of the 2018 ACM SIGSAC Conference on Computer and
Communications Security, CCS '18, page 35–52, New York, NY, USA, 2018.
Association for Computing Machinery.

[54]
Payman Mohassel and Yupeng Zhang.
Secureml: A system for scalable privacy-preserving machine learning.
In 2017 IEEE symposium on security and privacy (SP), pages
19–38. IEEE, 2017.

[55]
mpc msri.
EzPC - a language for secure machine learning.
<https://github.com/mpc-msri/EzPC/tree/master/EzPC>, June 2023.

[56]
Vinod Nair and Geoffrey E Hinton.
Rectified linear units improve restricted boltzmann machines.
In Icml, 2010.

[57]
Sebastian Neumayer, Alexis Goujon, Pakshal Bohra, and Michael Unser.
Approximation of lipschitz functions using deep spline neural
arXiv preprint arXiv:2204.06233, 2022.

[58]
Lucien KL Ng and Sherman SM Chow.
Sok: Cryptographic neural-network computation.
In 2023 IEEE Symposium on Security and Privacy (SP), pages
497–514. IEEE Computer Society, 2023.

[59]
Keiron O'Shea and Ryan Nash.
An introduction to convolutional neural networks.
arXiv preprint arXiv:1511.08458, 2015.

[60]
George-Liviu Pereteanu, Amir Alansary, and Jonathan Passerat-Palmbach.
Split he: Fast secure inference combining split learning and
homomorphic encryption.
arXiv preprint arXiv:2202.13351, 2022.

[61]
Prajit Ramachandran, Barret Zoph, and Quoc V Le.
Searching for activation functions.
arXiv preprint arXiv:1710.05941, 2017.

[62]
Deevashwer Rathee, Mayank Rathee, Rahul Kranti Kiran Goli, Divya Gupta, Rahul
Sharma, Nishanth Chandran, and Aseem Rastogi.
SiRnn: A Math Library for Secure RNN Inference.
In 2021 IEEE Symposium on Security and Privacy (SP), pages
1003–1020, 2021.

[63]
Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya
Gupta, Aseem Rastogi, and Rahul Sharma.
CrypTFlow2: Practical 2-party secure inference.
In Proceedings of the 2020 ACM SIGSAC Conference on Computer and
Communications Security, pages 325–342, 2020.

[64]
M Sadegh Riazi, Mohammad Samragh, Hao Chen, Kim Laine, Kristin Lauter, and
Farinaz Koushanfar.
$\{$XONN$\}$:$\{$XNOR-based$\}$ Oblivious Deep Neural Network
In 28th USENIX Security Symposium (USENIX Security 19), pages
1501–1518, 2019.

[65]
M Sadegh Riazi, Christian Weinert, Oleksandr Tkachenko, Ebrahim M Songhori,
Thomas Schneider, and Farinaz Koushanfar.
Chameleon: A hybrid secure computation framework for machine learning
In Proceedings of the 2018 on Asia conference on computer and
communications security, pages 707–721, 2018.

[66]
Bart Selman and Carla P Gomes.
Hill-climbing search.
Encyclopedia of cognitive science, 81:82, 2006.

[67]
Mingxing Tan and Quoc Le.
Efficientnet: Rethinking model scaling for convolutional neural
In International conference on machine learning, pages
6105–6114. PMLR, 2019.

[68]
Sijun Tan, Brian Knott, Yuan Tian, and David J Wu.
Cryptgpu: Fast privacy-preserving machine learning on the gpu.
In 2021 IEEE Symposium on Security and Privacy (SP), pages
1021–1038. IEEE, 2021.

[69]
Nenad Tomašev, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham,
Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk,
et al.
A clinically applicable approach to continuous prediction of future
acute kidney injury.
Nature, 572(7767):116–119, 2019.

[70]
Sameer Wagh, Divya Gupta, and Nishanth Chandran.
SecureNN: 3-Party Secure Computation for Neural Network Training.
Proc. Priv. Enhancing Technol., 2019(3):26–49, 2019.

[71]
Sameer Wagh, Shruti Tople, Fabrice Benhamouda, Eyal Kushilevitz, Prateek
Mittal, and Tal Rabin.
Falcon: Honest-majority maliciously secure framework for private deep
arXiv preprint arXiv:2004.02229, 2020.

[72]
Allen Wang.
June 2023.

[73]
Jean-Luc Watson, Sameer Wagh, and Raluca Ada Popa.
Piranha: A $\{$GPU$\}$ platform for secure computation.
In 31st USENIX Security Symposium (USENIX Security 22), pages
827–844, 2022.

[74]
Runhua Xu, Nathalie Baracaldo, and James Joshi.
Privacy-preserving machine learning: Methods, challenges and
arXiv preprint arXiv:2108.04417, 2021.

[75]
Andrew Chi-Chih Yao.
How to generate and exchange secrets.
In 27th annual symposium on foundations of computer science
(Sfcs 1986), pages 162–167. IEEE, 1986.

[76]
Yuanhan Zhang, ZhenFei Yin, Yidong Li, Guojun Yin, Junjie Yan, Jing Shao, and
Ziwei Liu.
Celeba-spoof: Large-scale face anti-spoofing dataset with rich
In European Conference on Computer Vision, pages 70–85.
Springer, 2020.

§ DNN MODELS CONSIDERED FOR DIFFERENT TASKS

For experimental evaluation, in this work, we consider four
DNN models for four image classification tasks.
For  task, we consider a  fully connected neural network (FCN) with one hidden layer as shown in fig:fcn.
For the second task , we consider a convolutional neural network (CNN) with five hidden layers (fig:convnet).
Since the datasets we consider for the last two tasks are relatively more complex, we select more complex DNN models. In particular, for , we consider a residual neural network (ResNet9) having eight hidden layers (fig:resnet9). For  task  we choose EfficientNetB0 model having 17 hidden layers (fig:efficientNetB0Model).
We refer the interested reader to [26, 67] for more details. Reader should observe that inputs to the activation function are batch normalized for each of the four models – a standard practice in DNN models.

§ DETAILS OF MINIONN

We briefly show the recipe from MiniONN [47] to generate MPC-friendly piece-wise polynomials approximations for
$\sigmoid$ activation function in fig:MiniONN. We  implement the recipe ourselves to generate MPC-friendly approximations for complex activation functions (e.g., $\silu$, $\gelu$, $\mish$).

§ 2PC RESULTS

We use the oblivious transfer-based construction for CryptFlow2. We observe a similar performance gain as the 3PC scenario — 2$\times$-5$\times$ speedup of compared to — as shown in tab:performance-appendix.

§ RELATED WORK ON COMPLEX ACTIVATION FUNCTIONS

In this section, we review some work on activation functions (AFs) that fall outside the scope of our work—but nevertheless be interesting to readers.

Complex AFs used in early NN.
Early neural network (NN) models
used binary threshold units [30, 19] and subsequently $\sigmoid$ and $\tanhh$
as AFs. The AFs are complex as well.
However, research in the last decades has exhibited
$\relu$ AF outperforms such complex used by primary NN models convincingly. Hence, in our work, we focus on more recent complex AFs that have superior performance than $\relu$.

Linear splines based AFs.
Methods presented in [57, 4, 6, 7]
uses a set of linear learnable splines as AFs in 1-Lipschitz constraint neural network.
These linear splines are MCP-friendly.
But 1-Lipschitz constraint neural network, due to being prone to vanishing gradients problem and having less expressiveness [44], is not generally used for cloud-based inference services. Thus, they fall outside the scope of our paper.

Polynominal AFs.
Recently, another trendy line of work attempts to redesign AF with a polynomial function [14, 2]. During training, these new AF are used instead of the traditional AF. However, in many real-world scenarios, we assume the DNN model has already been trained over traditionally used complex AF, and retraining further is considered expensive and time-consuming. Hence, these works are outside the scope of our work as well.

Task Name     $\fx$     $\npieces$     $\maxdegree$     $\ring$

3*     $\silu$     78     3     $\langle 64,32 \rangle$

$\gelu$     82     5     $\langle 64,32 \rangle$

$\mish$     85     3     $\langle 64,32 \rangle$

3*     $\silu$     77     3     $\langle 84, 42 \rangle$

$\gelu$     34     3     $\langle 84, 63 \rangle$

$\mish$     93     4     $\langle 84, 42 \rangle$

3*     $\silu$     91     5     $\langle 84, 42 \rangle$

$\gelu$     101     5     $\langle 84, 42\rangle$

$\mish$     96     5     $\langle 84, 42 \rangle$

3*     $\silu$     81     5     $\langle 64, 32 \rangle$

$\gelu$     90     5     $\langle 64, 32 \rangle$

$\mish$     93     5     $\langle 64, 32\rangle$

$\npieces, \maxdegree, \ring$  for four tasks using $\ApproximationAlgo$.
We consider accuracy loss $\negligibleaccuracy < 10^{-2}$ as negligible. $\ring$ is presented as $\langle n,d \rangle$.

§ DISCUSSION ON GRADIENT FREE LOCAL OPTIMIZATIONS

We want $\sysname$ to generate accurate and computationally efficient MPC-friendly approximation. If we denote the set of MPC-friendly approximations that does not degrade accuracy significantly by $\ffancy$, then we can pose this as the following minimization problem.
min_∈ ()
Unfortunately, it is not computationally feasible to fully populate the search space $\ffancy$.
Therefore, we look toward local optimizations that do not require access to
full search space. We also need the local optimization to be gradient-free (or equivalently treat the objective function $\Time$ as a blackbox).

Our initial attempt to use a hill climbing (HC) [66] type approach failed as HC tends to get stuck while searching for solutions. We, therefore, choose a simulated annealing-based approach since it is scholastic and generally resistant to getting stuck during solution searches. One may explore other existing gradient-free local optimizations in the literature (e.g., ant colony optimization, tabu search, genetic algorithm, etc.) [17], and we believe they will work equally well given they are successfully this application's custom techniques can be designed.