# A novel method of computing the Volume of
Hyperspheres in finite and infinite dimensions.

Cyril Belardinelli  Kantonsschule Solothurn
Solothurn
Switzerland<EMAIL_ADDRESS>
###### Abstract.

In the present article, the volume of the hypersphere in n-dimensional
euclidean space is recalculated in a rather original way by using the theory
of generalized functions (tempered distributions).
The calculation is performed by applying the integral representation of the
Heaviside unit step function and furthermore by using the distributional
Fourier transform of general power-law functions. The same method (added by a
regularization procedure of the occurring infinite-dimensional integral via
the Riemann-zeta function) is applied in the case of infinite dimensions
giving rise to some sort of anti-measure of spheres in the Hilbert space of
square-summable complex sequences.

###### Key words and phrases:

Generalized functions; tempered distributions; Fourier transforms

###### 2020 Mathematics Subject Classification:

Primary 46F10; Secondary 01A90

##  1\. Historical Remarks

In this article, we recompute the volume of the n-ball in a direct way using
generalized functions. Obviously, the interest lies in the methodology and not
in the formula itself, as it has been known for a long time. The formula for
the volume is used frequently in many branches of physics and mathematics,
ranging from Quantum Field Theory (e. g. when applied to dimensional
regularization, see e. g. [12]) to Probability Theory. The volume formula has
been derived for the first time by the relatively unknown Swiss mathematician
L. Schläfli in the mid 19th century (1850-52). The formula is exposed in
Schläfli’s opus magnum _Theorie der vielfachen Kontinuität_ [13]. This work
has been published posthumously only in 1901 after it has been rejected for
publication several times. However, in the years between 1850-1900 most of the
basic results of n-dimensional geometry have already been introduced and
generally accepted. It is mainly due to this publication delay that Schläfli’s
results went largely unnoticed. In fact, there are no comments at all on his
results in the classical historical literature such as e. g. [5, 4, 9].
Nevertheless, Schläfli must be considered (together with H. Grassmann, A.
Cayley, B. Riemann and to a certain extent also W. R. Hamilton) as one of the
co-founders of n-dimensional geometry. The main results have been developped
by these authors [6, 7] between the years 1844-1854.

##  2\. Preliminaries

###  2.1. Fourier transform of power-law functions

The present work makes use of the theory of generalized functions (also known
as tempered distributions). More precisely, this paper is basically concerned
with an application of a recently derived formula for the distributional
Fourier transform of a general power-law function $f(x)=x^{\alpha}$, (with a
real exponent $\alpha$). This formula has been derived in [3] for the purpose
of defining derivatives of fractional order. The formula for real (i. e. not
integer) exponents seems not to be known in the literature. In any case it
does not appear in classical works about generalized functions such as e. g.
[14, 15, 8, 17, 11].
The classical Fourier transform of the function $f(x)=x^{\alpha}$ exists only
for those exponents $\alpha$ which lie in the interval between minus one and
zero. By contrast, the Fourier transform of the corresponding tempered
distribution $\mathrm{Pf}\,x^{\alpha}$ is well-defined for every real exponent
$\alpha$. Note that the abbreviation _Pf_ denotes Hadamard’s Partie fini. The
precise definition of $\mathrm{Pf}\,x^{\alpha}$ is given by:

(2.1) $\mathrm{Pf}\,x^{\alpha}:=\mathrm{finite\,part\,of}\lim_{\epsilon\to
0^{+}}\int\limits_{|x|>\epsilon}\varphi(x)\,x^{\alpha}\mathrm{d}x$

The formula for $\mathscr{F}\,\mathrm{Pf}\,x^{\alpha}$ needed below is given
by the following Theorem:

######  Theorem 2.1.

Let $f(x)=x^{\alpha}$, ($\alpha\in\mathbb{R}$) be the monomial which generates
the regular distribution $\mathrm{Pf}\,x^{\alpha}$. We have then:

(2.2)
$\mathscr{F}\,\mathrm{Pf}\,x^{\alpha}=\begin{cases}2\pi\,i^{\alpha}\,\delta^{(\alpha)}(x)&\alpha\in\\{0,1,2,3...\\}\\\
\frac{\pi\,i^{\alpha}}{(-\alpha-1)!}\,\,\mathrm{Pf}\,x^{-\alpha-1}\,\mathrm{sgn}{(x)}&\alpha\in\\{-1,-2,-3...\\}\\\
T[\varphi]&\alpha\in\mathbb{R}\smallsetminus\mathbb{Z}\end{cases}$

where $\delta^{(\alpha)}(x)$ denotes the $\alpha^{\mathrm{th}}$ derivative of
the Dirac-distribution $\delta(x)$. $T$ denotes the regular distribution given
by:

$T=\Gamma(\alpha+1)\,\left(\mathrm{e}^{-i\pi\alpha}\,\mathrm{Pf}\,\frac{1}{(-ix)^{\alpha+1}}+\mathrm{Pf}\,\frac{1}{(ix)^{\alpha+1}}\right)$

###### Proof.

See [3] ∎

In the specific case of integer and half integer exponents Theorem 2.1 reduces
to:

(2.3)
$\mathscr{F}\,\mathrm{Pf}\,x^{-m}=\mathrm{Pf}\,\begin{cases}\frac{\pi}{i^{m}(m-1)!}\,x^{m-1}\,\mathrm{sgn}{(x)}&m\in\\{1,2,3...\\}\\\
&\\\
\sqrt{\pi}\,\,2^{m+\frac{1}{2}}\,(-2m)!!\,\Theta(-x)\,(-i|x|)^{m-1}&m\in\\{\pm\frac{1}{2},\pm\frac{3}{2},\pm\frac{5}{2},...\\}\end{cases}$

Where $\Theta(\cdot)$ denotes Heaviside’s unit step function defined by:

$\Theta(x)=\begin{cases}1&\text{if}\quad x>0\\\ \frac{1}{2}&\text{if}\quad
x=0\\\ 0&\text{if}\quad x<0\end{cases}$

The double factorial in (2.3) is defined in the usual manner:

$n!!=n\cdot(n-2)\cdot(n-4)\cdots 2\,\,(1\mathrm{,respectively})$

Note that the double factorial is also defined for negative odd arguments. The
values can be evaluated by virtue of the formula (restricted to odd numbers
$n$):

$(-n)!!\,n!!=(-1)^{\frac{n-1}{2}}\,n$

A few examples are given by:

######  Example 2.1.

$(-1)!!=1\quad(-3)!!=-1\quad(-5)!!=\frac{1}{3}$

###  2.2. Branch of the complex Logarithm

We stress the fact that formula of Theorem 2.1 is based on the following
branch of the complex logarithm:

$\displaystyle\log\colon\mathbb{C}^{\times}$ $\displaystyle\to\mathbb{C}$
$\displaystyle r\mathrm{e}^{i\varphi}$ $\displaystyle\mapsto\log{r}+i\varphi$

where
$\mathbb{C}^{\times}:=\\{r\mathrm{e}^{i\varphi}|\,r>0,\,-\pi\leqslant\varphi<\pi\\}$

######  Remark 2.1.

As a consequence of the chosen branch we have the following identities which
we use in the paper.

(2.4)
$\log{i}=i\frac{\pi}{2}\mathrm{,}\quad\log{(-i)}=-i\frac{\pi}{2}\mathrm{,}\quad\log{(-1)}=-i\pi$

##  3\. Volume of the n-Ball

In this paper, we are concerned with the volume of n-dimensional balls in
$\mathbb{R}^{n}$ defined by:

(3.1)
$\mathbb{B}^{n}(r)=\\{(x_{1},x_{2},\dots,x_{n})\in\mathbb{R}^{n}\,|\,x_{1}^{2}+x_{2}^{2}+\dots+x_{n}^{2}\leqslant
r^{2}\\}$

The Volume of $\mathbb{B}^{n}(r)$ can be expressed in the form:

(3.2)
$\mathrm{Vol}\,(\mathbb{B}^{n}(r))=\int\limits_{\mathbb{R}^{n}}\mathrm{d}x_{1}\mathrm{d}x_{2}\cdots\mathrm{d}x_{n}\,\Theta(r^{2}-x_{1}^{2}-x_{2}^{2}-\cdots-
x_{n}^{2})$

In the next step we use the integral representation of $\Theta(x)$:

(3.3) $\Theta(x)=\frac{1}{2\pi i}\lim_{\epsilon\to
0}\int\limits_{\mathbb{R}}\mathrm{d}k\,\frac{\mathrm{e}^{ikx}}{k-i\epsilon}$

By inserting this expression into (3.2) we get:

(3.4) $\mathrm{Vol}\,(\mathbb{B}^{n}(1))=\frac{1}{2\pi i}\lim_{\epsilon\to
0}\int\limits_{\mathbb{R}^{n}}\mathrm{d}x_{1}\mathrm{d}x_{2}\cdots\mathrm{d}x_{n}\int\limits_{\mathbb{R}}\mathrm{d}k\,\frac{\mathrm{e}^{-k(\epsilon-i)}}{k-i\epsilon}\mathrm{e}^{-(\epsilon+ik)(x_{1}^{2}+x_{2}^{2}+\cdots+x_{n}^{2})}$

Note that we inserted factors $\mathrm{e}^{-\epsilon(\cdot)}$ into the
integral in (3.4) in order to ensure convergence.
According to Fubini’s Theorem we obtain (leaving out the factor
$\mathrm{e}^{-\epsilon k}$):

(3.5)
$\mathrm{Vol}\,(\mathbb{B}^{n}(1))=\frac{\pi^{\frac{n}{2}-1}}{2}\lim_{\epsilon\to
0}\int\limits_{\mathbb{R}}\mathrm{d}k\,\frac{\mathrm{e}^{ik}}{(\epsilon+ik)^{\frac{n}{2}+1}}$

Next, we reinterpret the integral in 3.5 as the Fourier backtransform
(evaluated at $x=1$) of the following _tempered distribution_ :

(3.6) $T_{\epsilon}=\mathrm{Pf}\frac{1}{(\epsilon+ik)^{\frac{n}{2}+1}}$

The volume can then be rewritten in the form:

(3.7)
$\mathrm{Vol}(\mathbb{B}^{n}(1))=\pi^{\frac{n}{2}}\,\mathscr{S}^{\prime}-\lim_{\epsilon\to
0}\left[\mathscr{F}^{-1}T_{\epsilon}\,(x=1)\right]$

Where the notation $\mathscr{S}^{\prime}-\lim_{\epsilon\to 0}$ denotes the
limit within the space of distributions $\mathscr{S}^{\prime}$. Due to the
continuity of the operator $\mathscr{F}^{-1}$ we are allowed to write:

(3.8) $\mathscr{S}^{\prime}-\lim_{\epsilon\to
0}\,[\mathscr{F}^{-1}T_{\epsilon}]=\mathscr{F}^{-1}[\mathscr{S}^{\prime}-\lim_{\epsilon\to
0}T_{\epsilon}]$

The limit on the right hand side of (3.8) can be determined by using the
following Lemma [3]:

######  Lemma 3.1.

$\mathscr{S}^{\prime}-\lim_{\epsilon\to
0^{+}}\frac{1}{(\epsilon+ix)^{m+1}}=\begin{cases}\pi\,\frac{i^{m}}{m!}\,\delta^{(m)}(x)+\mathrm{Pf}\,(ix)^{-m-1}&\,m\in\\{1,2,3...\\}\\\
&\\\
\mathrm{Pf}\,(ix)^{-m-1}&\,m\in\\{\pm\frac{1}{2},\pm\frac{3}{2},\pm\frac{5}{2},...\\}\\\
\end{cases}$

###### Proof.

See [3] ∎

Applying formula (2.3) we obtain:

$\mathscr{F}^{-1}\left[\mathscr{S}^{\prime}-\lim_{\epsilon\to
0}T_{\epsilon}\right]=\begin{cases}\frac{1}{m!}&\,m\in\\{1,2,3,...\\}\\\ &\\\
\frac{2^{m+\frac{1}{2}}i^{-m-1}(-i)^{m}}{\sqrt{\pi}}(-2m-2)!!&\,m\in\\{\frac{1}{2},\frac{3}{2},\frac{5}{2},...\\}\\\
\end{cases}$

######  Remark 3.1.

The expression for half integer values of $m$ can be simplified by using the
following identities (valid for
$m\in\left\\{\frac{1}{2},\frac{3}{2},\frac{5}{2},...\right\\}$):

$(-2m-2)!!=\frac{(-1)^{m+\frac{1}{2}}}{(2m)!!}$

and

$(-1)^{m+\frac{1}{2}}(-i)^{m}i^{-m-1}=1$

The simplified formula reads then:

$\mathscr{F}^{-1}\left[\mathscr{S}^{\prime}-\lim_{\epsilon\to
0}T_{\epsilon}\right]=\begin{cases}\frac{1}{m!}&\,m\in\\{1,2,3,...\\}\\\ &\\\
\frac{2^{m+\frac{1}{2}}}{\sqrt{\pi}(2m)!!}&\,m\in\\{\frac{1}{2},\frac{3}{2},\frac{5}{2},...\\}\\\
\end{cases}$

The final result for the volume formulated in terms of space dimension $n$ is
then:

$\mathrm{Vol}\,(\mathbb{B}^{n}(1))=\begin{cases}\frac{\pi^{\frac{n}{2}}}{(\frac{n}{2})!}&\,n\in\\{2,4,...\\}\\\
&\\\ 2\frac{(2\pi)^{\frac{n-1}{2}}}{(n)!!}&\,n\in\\{1,3,5,...\\}\\\
\end{cases}$

which can be subsumed under the well-known formula:

$\mathrm{Vol}\,(\mathbb{B}^{n}(1))=\frac{\pi^{\frac{n}{2}}}{\Gamma(\frac{n}{2}+1)}\mathrm{,}\quad
n\in\\{1,2,3...\\}$

##  4\. Volume of Spheres in infinite dimensions

In the present section we try to calculate the volume of balls in the Hilbert
space of square-summable complex sequences
$\ell^{2}(\mathbb{C}):=\\{(c_{n}):c_{n}\in\mathbb{C}\,|\sum_{n=1}^{\infty}|c_{n}|^{2}\leqslant\infty\\}$.
However, it is known that a $\sigma$-additive Lebesgue-type measure which is
translationally and rotationally invariant does not exist in a Hilbert space
of infinite dimensions (See e. g. [10]). The volume we are calculating in this
section does therefore not represent a measure in the usual sense $\grave{a}$
la Lebesgue.
The volume to be calculated is formally given by an infinite-dimensional
integral which we regularize by Riemann’s $\zeta$-function. We remark that an
analogous calculation is given in [2] where one calculates a certain path
integral $\grave{a}$ la Feynman by using the integral representation of delta-
function $\delta(x)$ and applying a $\zeta$-regularization.

The ball of radius $r$ within $\ell^{2}(\mathbb{C})$ is defined by:

(4.1)
$\mathbb{B}_{\mathbb{C}}^{\infty}(r):=\\{(c_{1},c_{2},\dots)\in\ell^{2}(\mathbb{C})\,|\sum_{n=1}^{\infty}|c_{n}|^{2}\leqslant
r^{2}\\}$

Analogous to (3.2) we have:

(4.2)
$\mathrm{Vol}\,(\mathbb{B}_{\mathbb{C}}^{\infty}(r))=\int\limits_{\mathbb{C}\times\mathbb{C}\times\cdots}\prod_{n=1}^{\infty}\mathrm{d}c_{n}\,\Theta(r^{2}-|c_{1}|^{2}-|c_{2}|^{2}-\cdots)$

Where we define the integration over the complex variable $c_{n}=x_{n}+iy_{n}$
in the following canonical way:

(4.3)
$\int_{\mathbb{C}}\mathrm{d}c_{n}:=\int_{\mathbb{R}^{2}}\mathrm{d}x_{n}\mathrm{d}y_{n}$

Next, we proceed along the same track as in Sec. 3

$\mathrm{Vol}\,(\mathbb{B}_{\mathbb{C}}^{\infty}(r))=\frac{1}{2\pi
i}\lim_{\epsilon\to
0}\int\limits_{\mathbb{C}\times\mathbb{C}\times\cdots}(\mathrm{d}c_{n})\int\limits_{\mathbb{R}}\mathrm{d}k\,\frac{\mathrm{e}^{ikr^{2}}}{k-i\epsilon}\,\mathrm{e}^{-(\epsilon+ik)(|c_{1}|^{2}+|c_{2}|^{2}+\cdots}$

Where we use the short notation
$(\mathrm{d}c_{n}):=\prod_{n=1}^{\infty}\mathrm{d}c_{n}$

By the same procedure as in Sec. 3 we interchange the order of integration to
obtain:

(4.4) $\mathrm{Vol}\,(\mathbb{B}_{\mathbb{C}}^{\infty}(r))=\frac{1}{2\pi
i}\lim_{\epsilon\to
0}\int\limits_{\mathbb{R}}\mathrm{d}k\,\frac{\mathrm{e}^{ikr^{2}}}{k-i\epsilon}\int\limits_{\mathbb{C}\times\mathbb{C}\times\cdots}(\mathrm{d}c_{n})\,\mathrm{e}^{-(\epsilon+ik)(|c_{1}|^{2}+|c_{2}|^{2}+\cdots}$

The latter integral can be evaluated formally:

(4.5)
$\int\limits_{\mathbb{C}\times\mathbb{C}\times\cdots}(\mathrm{d}c_{n})\,\mathrm{e}^{-(\epsilon+ik)(|c_{1}|^{2}+|c_{2}|^{2}+\cdots}=\prod_{n=1}^{\infty}\frac{\pi}{\epsilon+ik}$

We regularize the latter infinite product by means of Riemann’s
$\zeta$-function [16, 1]. We stress the fact that the same regularization
procedure has already been applied successfully in evaluating Feynman path
integrals with quadratic Lagrangians. See e. g. [2].
The Logarithm of the product in Eq. (4.5) reads formally :

$\log{\prod_{n=1}^{\infty}\frac{\pi}{\epsilon+ik}}=\left(\log{\frac{\pi}{\epsilon+ik}}\right)\,\sum_{n=1}^{\infty}1$

By applying the $\zeta$-regularized sum (See e. g. [2]):

$\sum_{n=1}^{\infty}1=\zeta(0)=-\frac{1}{2}$

we obtain:

$\prod_{n=1}^{\infty}\frac{\pi}{\epsilon+ik}=\sqrt{\frac{\epsilon+ik}{\pi}}$

Finally, we insert the latter expression into (4.4) which gives:

$\mathrm{Vol}\,(\mathbb{B}_{\mathbb{C}}^{\infty}(r))=\frac{1}{2\pi^{\frac{3}{2}}}\lim_{\epsilon\to
0}\int\limits_{\mathbb{R}}\mathrm{d}k\,\frac{\mathrm{e}^{ikr^{2}}}{\sqrt{\epsilon+ik}}$

The latter expression can be rewritten formally as:

$\mathrm{Vol}\,(\mathbb{B}_{\mathbb{C}}^{\infty}(r))=\frac{1}{\sqrt{\pi}}\,\mathscr{S}^{\prime}-\lim_{\epsilon\to
0}\left[\mathscr{F}^{-1}(\epsilon+ik)^{-\frac{1}{2}}\right]_{x=r^{2}}$

Once again, the limit $\epsilon\to 0$ can be interchanged with the operator
$\mathscr{F}^{-1}$ to obtain:

$\mathrm{Vol}\,(\mathbb{B}_{\mathbb{C}}^{\infty}(r))=\frac{1}{\sqrt{i\pi}}\,\left[\mathscr{F}^{-1}\frac{1}{\sqrt{k}}\right]_{x=r^{2}}$

According to (2.3):

$\mathscr{F}^{-1}\frac{1}{\sqrt{k}}=\frac{1}{\sqrt{\pi}\,0!!}\,\Theta(x)\,(-i|x|)^{-1/2}$

Which leads to to the result:

(4.6) $\mathrm{Vol}\,(\mathbb{B}_{\mathbb{C}}^{\infty}(r))=\frac{1}{\pi r}$

######  Remark 4.1.

Formally, the same result is obtained when one inserts the value $n=-1$ into
formula:

$\mathrm{Vol}\,(\mathbb{B}^{n}(r))=\frac{\pi^{\frac{n}{2}}}{\Gamma(\frac{n}{2}+1)}\,r^{n}$

Formula (4.6) states that spheres with a greater radius have less volume.
Evidently, this feature is incompatible with the usual properties of a
Lebesgue-measure. It must therefore represent something different than a usual
measure.

## Appendix A

We prove the following elementary identity:

$(-1)^{m+\frac{1}{2}}(-i)^{m}i^{-m-1}=1\mathrm{,}\quad
m\in\left\\{\frac{1}{2},\frac{3}{2},\frac{5}{2},...\right\\}$

###### Proof.

Rewriting the left hand side of the latter expression:

$\mathrm{e}^{(m+\frac{1}{2})\log{(-1)}}\mathrm{e}^{m\log{(-i)}}\mathrm{e}^{-(m+1)\log{i}}$

According to 2.4 we get:

$\mathrm{e}^{(m+\frac{1}{2})(-i\pi)}\mathrm{e}^{m(-i\frac{\pi}{2})}\mathrm{e}^{-(m+1)i\frac{\pi}{2}}=\mathrm{e}^{i2m\pi}\,\mathrm{e}^{-i\pi}=1$

∎

*

## References

  * [1] L. V. Ahlfors.  Complex analysis, An introduction to the theory of analytic functions of one complex variable, third ed. McGraw-Hill Book Co., New York, 1978.
  * [2] C. Belardinelli.  arXiv:1908.05226, 2019.
  * [3] C. Belardinelli.  arXiv:2105.00276, 2021.
  * [4] C. B. Boyer.  A History of Mathematics. John Wiley & Sons, Inc., 1968.
  * [5] F. Cajori.  A history of Mathematics, 2nd ed. The Macmillan Company, London, 1919.
  * [6] A. Cayley.  On linear transformations.  Cambridge and Dublin mathematical journal, 1:104–122, 1846.
  * [7] H. Grassmann.  Lineale Ausdehnungslehre, volume 1.  Verlag von Otto Wigand, 1844.
  * [8] G. E. Shilov I. M. Gel’fand.  Generalized Functions, Volume 1: Properties and Operations, volume 1.  AMS Chelsea Publishing: An Imprint of the American Mathematical Society, 1964.
  * [9] M. Kline.  Mathematical Thought from Ancient to Modern Times. Oxford University Press, 1973.
  * [10] S. Mazzucchi.  Mathematical Feynman Path Integrals And Their Applications.  World Scientific, 2009.
  * [11] R. Sikorski P. Antosik, J. Mikusinski.  Theory of Distributions, The Sequential Approach. Elsevier Scientific Publishing Company, Amsterdam, 1973.
  * [12] M. E. Peskin and D. V. Schröder.  Am Introduction to Quantum Field Theory. Perseus Books Publishing L. L. C., 1995.
  * [13] L. Schläfli.  Theorie der vielfachen Kontinuität. Herausgegeben im Auftrag der Schweiz. Naturforschenden Gesellschaft von J. H. Graf, Bern, 1901.
  * [14] L. Schwartz.  Théorie Des Distributions. Tome I.  Herrmann Paris, 1950.
  * [15] L. Schwartz.  Théorie Des Distributions. Tome II.  Herrmann Paris, 1951.
  * [16] E. C. Titchmarsh.  The Theory of the Riemann Zeta-function.  Oxford University Press, 1951.
  * [17] V.S. Vladimirov.  Methods of the theory of generalized functions.  Taylor & Francis, London 2002, pages 19–21, 2002.


#  Waffle: Watermarking in Federated Learning
††thanks: This work was supported in part by Intel (in the context of the
Private-AI Institute) and SAPPAN, a project funded by the European Union’s
Horizon 2020 Research and Innovation Programme under Grant Agreement no.
833418. We also thank Sebastian Szyller for interesting discussions, and Aalto
Science-IT project for computational resources.

Buse G. A. Tekgul1, Yuxi Xia1, Samuel Marchal2, and N. Asokan 31  1Department
of Computer Science, Aalto University, 02150, Espoo, Finland
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>2F-Secure
Corporation, 00180, Helsinki, Finland
Email<EMAIL_ADDRESS>3 University of Waterloo, Waterloo, ON N2L
3G1, Canada
Email<EMAIL_ADDRESS>
###### Abstract

_Federated learning_ is a distributed learning technique where machine
learning models are trained on client devices in which the local training data
resides. The training is coordinated via a central server which is, typically,
controlled by the intended owner of the resulting model. By avoiding the need
to transport the training data to the central server, federated learning
improves privacy and efficiency. But it raises the risk of _model theft_ by
clients because the resulting model is available on every client device. Even
if the application software used for local training may attempt to prevent
direct access to the model, a malicious client may bypass any such
restrictions by reverse engineering the application software. _Watermarking_
is a well-known deterrence method against model theft by providing the means
for model owners to demonstrate ownership of their models. Several recent deep
neural network (DNN) watermarking techniques use _backdooring_ : training the
models with additional mislabeled data. Backdooring requires full access to
the training data and control of the training process. This is feasible when a
single party trains the model in a centralized manner, but not in a federated
learning setting where the training process and training data are distributed
among several client devices. In this paper, we present Waffle, the first
approach to watermark DNN models trained using federated learning. It
introduces a re-training step at the server after each aggregation of local
models into the global model. We show that Waffle efficiently embeds a
resilient watermark into models incurring only negligible degradation in test
accuracy ($-0.17\%$), and does not require access to training data. We also
introduce a novel technique to generate the backdoor used as a watermark. It
outperforms prior techniques, imposing no communication, and low computational
($+3.2\%$) overhead111The code for reproducing our work can be found at
https://github.com/ssg-research/WAFFLE.

###### Index Terms:

Federated learning, ownership demonstration, watermarking, deep learning

##  I Introduction

Distributed machine learning has gained considerable attention in many big
data processing applications such as recommendation systems, smart assistants,
and health-care, due to its efficient parallelization and scalability
properties. Federated learning [1] is an instance of privacy-preserving
distributed machine learning that allows decentralized training of deep neural
networks (DNNs) by many parties holding local data samples. Major companies
have already utilized federated learning in various large-scale applications
such as improving the Google Keyboard (Gboard) suggestions [2, 3] or Apple’s
voice recognition models [4]. In these applications, clients only need to
download a set of tools or a specific software (e.g., Tensorflow Lite222ML for
Mobile and Edge Devices, https://www.tensorflow.org/lite/) into their mobile
devices, and execute the training with the on-device, cached data. Hence,
instead of using publicly available or synthetic datasets that often do not
represent the real-world distribution, model owners can deploy high-quality
models with no access to the possibly sensitive training data.

The most common federated learning setting in commercial, global-scale
applications is client-server involving three parties: model owner, aggregator
(hosted on a server), and a large number of clients training a common global
model [5]. Federated learning consists of iterative aggregation rounds where
(1) the aggregator sends the global model parameters to clients, (2) each
client replaces parameters of its local model using the global model, re-
trains it using the on-device data, and sends updated parameters back to the
aggregator, which (3) combines them into a new global model. The final global
model is delivered to the model owner when the training is completed. Since
each client’s training data stays in place, federated learning preserves
privacy of clients’ data and avoids incurring substantial costs for
transferring training data from clients to a centralized trainer.

Despite its advantages, federated learning brings forth the issue of
preserving ownership. In large-scale, client-server federated learning
applications, there are multiple clients who are data-owners and _can use the
model_ in their local devices, but there is _only one model owner._ However, a
side-effect of the training process is that each client can gain complete
access to the global model in every round, including the final one by reverse
engineering the application software [6] or on-device dynamic analysis [7].
Although the model owner can deploy further mechanisms for hiding model
parameters during the local training (e.g. homomorphic encryption), this might
lead to high latency, bandwidth costs, and additional requirements for benign
clients [8]. It is therefore useful to have a means of demonstrating the
intended ownership in case a malicious client uses the global model residing
on their devices in unauthorized ways such as monetizing the model or making
it available to their own customers.

Recently, different watermarking techniques [9, 10, 11, 12, 13] have been
proposed to demonstrate ownership of DNN models. In a typical backdoor-based
watermarking procedure, the model owner first designs a secret _watermark_
which consists of mislabeled input output pairs. Then, the model owner trains
the model with both the training dataset and the watermark in order to embed
the watermark into the model. This watermark can be subsequently used to
demonstrate ownership. However, existing DNN watermarking solutions cannot be
directly applied in federated learning for two reasons. First, model owners
cannot use techniques for generating watermarks using training data because
they lack access to the training data (e.g., Gboard [3]). Second, training is
performed in parallel by several clients, some of them being potentially
malicious. Although some “trusted clients” can join the watermark embedding
phase, the training setup and aggregation rules have to be changed to handle
them differently. Thus, watermark embedding cannot be implemented easily by
clients.

Our goal is to design a procedure for effectively embedding watermarks into
DNN models trained via client-server federated learning, without decreasing
the accuracy of the resulting global model while minimizing the computational
and communication overhead imposed on the distributed training process. We
claim the following contributions:

  1. 1.

Problem definition: We identify the problem of demonstrating ownership of
models trained via client-server federated learning and define requirements
for a solution addressing this problem (Section III-B).

  2. 2.

Watermarking procedure for federated learning: We introduce Waffle: the first
solution for addressing ownership problem in client-server federated learning
(Section IV). Waffle leverages capabilities of the aggregator to embed a
backdoor-based watermark [9] by re-training the global model with the
watermark during each aggregation round. We show on MNIST and CIFAR10 using
two DNN architectures that Waffle embeds watermarks without degrading the
model performance (Section VI-C).

  3. 3.

Data-independent watermark generation method: We introduce WafflePattern, a
novel data-independent method to generate watermarks for DNN image
classification models (Section IV-C). It generates images having random but
class-consistent patterns on a random background, which is suitable for
federated learning. Compared to prior watermark generation methods [9, 13,
14], WafflePattern retains model performance better (Section VI-C) and imposes
lower communication and computational overhead (Section VI-D). We also show
that Waffle with WafflePattern is resilient to watermark removal techniques
including fine-tuning, pruning and reverse-engineering (Section VI-B) if no
more than 10% of the clients collude to defeat the watermarking procedure.

Our procedure focuses on embedding effective watermarks into models trained
with federated learning. We do not discuss the demonstration of ownership
using our watermarks because secure schemes [9, 11] for registering and using
backdoor-based watermarks to reliably demonstrate ownership of DNN models
already exist and these schemes can be used with our watermarks. We also leave
discussions on the legal validity of DNN model watermarks as out of scope.

##  II Background

###  II-A Federated Learning

The client-server federated learning is composed of three main parties: 1) A
large number of clients $C=\\{c_{j}\\}_{j=1}^{K}$ who are data owners and keep
their datasets $D_{c_{j}}$ private, 2) a model owner $O$ providing a randomly
initialized global model $w_{G}$ at the beginning of federated learning and
getting it trained at the end, and 3) a secure aggregator $\mathit{Agg}$ [15]
located between $O$ and $C$. We focus on training a DNN model, which is a
function $F(x,w):\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ that we simply refer
to as its parameters $w$, e.g., $w_{G}$. $O$ cannot obtain any information
about $D_{c_{j}}$ due to the secure aggregation protocol [15] implemented by
$\mathit{Agg}$. In this work, we focus on federated learning using the
FederatedAveraging (FedAvg) algorithm [1], a widely used aggregation rule. In
FedAvg, clients train their models using stochastic gradient descent (SGD).
Before federated learning starts, $w_{G}$ is initialized by $O$. In
aggregation round $t$,

  1. 1.

$\mathit{Agg}$ sends $w_{G(t)}$ to a subset of clients
$C_{sub}=\\{c_{i}\\}_{i=1}^{L}$, where $(L\ll K)$.

  2. 2.

Each $c_{i}$ updates $w_{c_{i}(t-1)}$ with $w_{G(t)}$, re-trains the updated
$w_{c_{i}(t)}$ by a pre-determined number of local passes over its local
dataset $D_{c_{i}}$, and sends the re-trained local model $w_{c_{i}(t)}$ to
$\mathit{Agg}$.

  3. 3.

$\mathit{Agg}$ averages all local models into a new $w_{G(t+1)}$.

###  II-B Watermarking DNN Models by Backdooring

Several works [9, 14, 10, 16, 12, 13] have proven the feasibility of embedding
watermarks into DNNs for demonstration of ownership. In general, a watermark
set $\mathrm{WM}_{w}$ consists of samples $\\{x,B(x)\\}$ designed by the owner
of the DNN model $w$. The model owner embeds $\mathrm{WM}_{w}$ into $w$ by
optimizing $w$ on both the training set and $\mathrm{WM}_{w}$ such that
$B(x)=w^{+}(x)$ and $B(x)\neq w(x)$ for almost all $x\in\mathrm{WM}_{w}$,
where $w^{+}(x)$ is the watermarked model. If the model owner suspects that
another model $w_{adv}$ is possibly derived from $w^{+}$, $\mathrm{WM}_{w}$
with a pre-defined verification algorithm Verify is used to demonstrate
ownership if $\textsc{Verify}(w_{adv},\mathrm{WM}_{w})$ returns $True$.

In this paper, we focus on watermarking via backdooring [9, 10, 13]. A
backdoor [17] consists of a trigger set of samples with incorrect labels. In
computer vision, a trigger set usually contains specific patterns that can be
added to an image [10] or a set of images unrelated to the actual task [9,
10], both of which serve a similar purpose. The backdoor is injected into a
DNN by training it with both the correct training data and the backdoor. At
inference time, the backdoored DNN performs normally on clean inputs but
returns the expected incorrect label when a sample from the trigger set is
given as an input. To reliably demonstrate that a model $w_{adv}$ is derived
from $w^{+}$, i.e., to obtain
$\textsc{Verify}(w_{adv},\mathrm{WM}_{w})\rightarrow True$, $\mathrm{WM}_{w}$
must be secret and registered in a timestamped public bulletin [9, 11]. In
addition, the accuracy of $w_{adv}$ on $\mathrm{WM}_{w}$ must be over a
threshold $T_{acc}$: $Acc(w_{adv},\mathrm{WM}_{w})\geq T_{acc}$. The value for
$T_{acc}$ is computed using the size of the watermark set $|\mathrm{WM}_{w}|$
and the number of classes $m$ for $w$ [9, 11].

##  III Ownership Demonstration in Federated Learning

###  III-A Adversary Model

The ability to demonstrate ownership of machine learning models has emerged as
an important concern because models can be monetized and represent a business
advantage [18, 11, 19]. Figure 1 shows our adversary model. We consider
malicious clients to be our primary adversaries. Following prior work [15], we
assume that $\mathit{Agg}$ is the only trusted party in federated learning.
$\mathit{Agg}$ resides in a central server, and is defined as an incorruptible
third party [15, 20] between $C$ and $O$. In some applications, the cloud
server can belong to $O$ (e.g., Gboard [3], Apple’s Siri [4]) but
$\mathit{Agg}$ is still considered a secure party: it implements a secure
aggregation protocol [15] and uses cryptographic primitives while routing
messages between the server and clients. In unauthenticated or unencrypted
network models like federated learning, $\mathit{Agg}$ provides the strongest
possible security for protecting data confidentiality [5, 15, 20].

Figure 1: Architecture and adversary model for watermarking in federated
learning. Parties under the control of an adversary are highlighted in red.

####  III-A1 Adversaries’ goal

Adversaries aim to obtain a $w_{adv}$ having the same performance as $w_{G}$
on a test dataset $D_{test}$, so that $Acc(w_{adv},D_{test})\approx
Acc(w_{G},D_{test})$ while evading watermark detection, i.e.,
$\textsc{Verify}(w_{adv},\mathrm{WM}_{w_{G}})\rightarrow False$, where
$\mathrm{WM}_{w_{G}}$ is the watermark generated by $O$. We assume that an
adversary $adv=c_{i}$ has the following capabilities:

  * •

has access to its own relevant training data $D_{adv}$, but does not have
access to other clients’ training data. Each $D_{c_{i}}$ is independent,
identically distributed (IID), balanced and limited in size such that every
$c_{i}$ is incentivized to participate in federated learning. $adv$ having a
large $D_{adv}$ would rather train its own model instead of stealing $w_{G}$.

  * •

gains white-box access to $w_{adv(t)}$, and can save $w_{adv(t)}$ at any
aggregation round $t$.

  * •

can apply any post processing technique (e.g., fine-tuning, pruning, reverse
engineering) to $w_{adv(t)}$ in order to remove watermarks.

####  III-A2 Assumptions

We assume that every $c_{i}$ is incentivized to maximize the accuracy of
$w_{G}$. $c_{i}$ does not backdoor [21], poison or embed its own watermark
sets in $w_{G}$. These attacks might decrease the performance of $w_{G}$ and
$w_{adv}$, which is contrary to the goal of every $c_{i}$, including $adv$,
who aims to obtain a model $w_{G}/w_{adv}$ having the best performance
possible. Furthermore, clients does not change the training scheme and use the
same hyper-parameters provided by $O$ (e.g., number of local passes $E_{c}$,
loss function $\ell_{c}$, learning rate $\eta_{c}$, batch size $|b_{c}|$),
since they are optimized to maximize the accuracy of $w_{G}$.

###  III-B Requirements

We define the following requirements for designing an effective watermarking
scheme (W1:3) as well as performance conditions (P1:3) which must be met in
federated learning. Performance conditions should be also taken into account
for building other type of solutions for ownership demonstration of federated
learning models.

  1. W1

Demonstration of ownership:At any aggregation round $t$,
$\textsc{Verify}(w_{adv(t)},\mathrm{WM}_{w_{G}})$ should return $True$ , which
requires $Acc(w_{adv(t)},\mathrm{WM}_{w_{G}})\geq T_{acc}$.

  2. W2

Robustness: $\mathrm{WM}_{w_{G}}$ embedded in $w_{adv}$ should be resilient
against attacks that try to remove it without destroying the performance of
$w_{adv}$. This requirement is satisfied when W1 still holds or
$Acc(w_{adv},D_{test})\ll Acc(w^{+}_{adv},D_{test})$ after applying a
watermark removal attack to $w_{adv}$.

  3. W3

Data independence: Watermarking procedure should not require training data
knowledge, and $O$ cannot use clients’ training data in order to generate
their own watermark set $\mathrm{WM}_{w_{G}}$.

  1. P1

Model utility: Watermarking should not degrade the performance of the local
model, i.e., $Acc(w^{+}_{adv},D_{test})\approx Acc(w_{adv},D_{test})$.

  2. P2

Low communication overhead: Embedding $\mathrm{WM}_{w_{G}}$ should not
increase the total number of aggregation rounds or the amount of the data
exchanged to reach the convergence of $w_{G}$.

  3. P3

Low computation overhead: Implementing watermark embedding with
$\mathrm{WM}_{w_{G}}$ inside any party should incur minimal additional
computation.

##  IV Watermarking in Federated Learning

###  IV-A Current Challenges

Watermarking can help model owners to prove ownership of their DNN models.
However, state-of-the-art techniques [9, 10, 12, 13] cannot be directly
implemented in client-server federated learning, since they assume full
control over the training process as in centralized machine learning.
Moreover, clients might be malicious; therefore, they should be excluded from
any involvement in watermarking and should not have access to the watermark
set. $O$ has two options for embedding the watermark using existing
techniques:

  * •

“Pre-embedding” into $w_{G(0)}$ before training starts.

  * •

“Post-embedding” into $w_{G(t)}$ before deploying the model.

Pre-embedding achieves a watermark accuracy of $100\%$ during the first
aggregation round, while Post-embedding achieves that at the last aggregation
round $t$. However, both techniques have limitations. Pre-embedded watermarks
are easily removed from the global model after several aggregation rounds.
Post-embedding demonstrates a reliable proof of ownership when the model is
deployed, but $adv$ can use updates from the global model, $w_{G(t-1)}$, one
aggregation round before the training is completed. $w_{adv(t-1)}=w_{G(t-1)}$
contains no watermark and $Acc(w_{adv(t-1)},D_{test})\approx
Acc(w_{adv(t)},D_{test})$. Finally, post-embedded watermarks are not resilient
to removal attacks such as fine-tuning and pruning as shown in [9]. Both
techniques fail to satisfy W1 and W2 and are not feasible for watermarking
federated learning models.

###  IV-B Waffle Procedure

Considering the capabilities of $\mathit{Agg}$ presented in Section III-A, we
add a new responsibility for it: _watermarking_. In Waffle, $O$ shares its
watermark set $\mathrm{WM}_{w_{G}}$ only with $\mathit{Agg}$. $\mathit{Agg}$
re-trains $w_{G}$ to guarantee that the watermark is embedded and Verify
returns $True$ for watermarked models: $w^{+}_{G(t)}$ and $w^{+}_{c_{i}(t)}$.
Waffle makes no modification to client operations or the secure aggregation.
Inside $\mathit{Agg}$, we introduce two new functions: Pretrain and Retrain.
Pretrain is a one time operation performed before federated learning starts
and Retrain is a recurrent operation performed at each aggregation round $t$.
The pseudo-code for Pretrain and Retrain is given in Algorithm 1 (Appendix
A-A).

Pretrain gets a randomly initialized $w_{G(0)}$ and returns a watermarked
version $w^{+}_{G(0)}$, $Acc(w^{+}_{G}(0),\mathrm{WM}_{w_{G}})=100\%$. Retrain
first implements the secure aggregation: averaging re-trained
$w^{-}_{c_{i}(t)}$333$w^{-}$ refers to a re-trained version of a previously
watermarked model $w^{+}$. received from $C_{sub}$ and updating
$w^{-}_{G(t+1)}$ with this average. Then, it re-trains $w^{-}_{G(t+1)}$ using
$\mathrm{WM}_{w_{G}}$ until $Acc(w^{-}_{G(t+1)},\mathrm{WM}_{w_{G}})$ reaches
threshold value $th$. We also define the maximum number of re-training rounds
$E_{r}=100$ in Retrain to not slow down model upload speed. Therefore, Retrain
terminates when the number of re-training rounds reaches $E_{r}$, even if
$Acc(w^{+}_{G(t+1)},\mathrm{WM}_{w_{G}})<th$.

Waffle starts with $\mathit{Agg}$ executing Pretrain. Then $\mathit{Agg}$
executes Retrain and sends $w^{+}_{G(t+1)}$ to $C_{sub}$ in each $t$. By
starting federated learning with a global model already converged to the
watermark, we can decrease the number of re-training rounds required to reach
high $Acc(w^{+}_{G(t)},\mathrm{WM}_{w_{G}})$ satisfying P3. Waffle ensures
high enough $Acc(w^{+}_{G(t)},\mathrm{WM}_{w_{G}})$ to enable demonstration of
ownership at anytime, satisfying W1. Waffle also ensures that $w^{+}_{G(t)}$
and $w^{+}_{adv(t)}$ converge for both the watermark and the actual task,
satisfying P1. In addition, Waffle does not require any client’s training data
to operate, since Waffle re-trains $w^{+}_{G(t)}$ using only the watermark but
no other data samples. Therefore, it satisfies the requirement W3.

Waffle is executed after the aggregation step of federated learning, and is
independent of the aggregation method. Therefore, any other robust aggregation
method such as Krum [22], trimmed mean or median [23] can be easily combined
with Waffle. While Waffle can use any existing watermark set, we introduce
WafflePattern, a novel way to generate (training) data-independent watermarks.

###  IV-C WafflePattern

Inspired by the prior work [9, 13] and assuming that $O$ has no access to the
training data in large-scale federated learning applications, we propose
WafflePattern: adding specific patterns to images containing only noise. While
creating WafflePattern, we first generate different images using Gaussian
noise. Then, each image is embedded with a certain pattern and labeled with a
class that is related to the original task. Each class has a different pattern
and every pattern is unique in terms of color, shape, orientation and
position. We assume that $O$ knows the dimensions of the model’s inputs and
outputs, since $O$ selects the model architecture according to these
specifications. Therefore, this information is enough to construct
WafflePattern. Figure 2b illustrates samples of WafflePattern generated for
colored images.

WafflePattern is independent of training data and suitable for federated
learning. This property satisfies the requirement W3. Additionally,
WafflePattern is easy to learn. Using the same pattern for each class helps
$w_{G}$ to converge and overfit to WafflePattern, since samples of each class
include features that are easy to learn and memorize. Therefore, this property
helps satisfying P2 and P3.

##  V Experimental Setup

(a) Clean

(b) WafflePattern

(c) Embedded C.

(d) unRelate

(e) unStruct

Figure 2: Original images (a) in CIFAR10 dataset for classes airplane,
automobile and bird as well as sample of watermark sets WafflePattern (b),
Embedded Content [13] (c), unRelate [9, 13] (d) and unStruct [14] (e)
constructed for these three classes.

###  V-A Datasets and Models

Considering that reaching a high accuracy in federated learning is very
challenging, we replicated image classification tasks in [1] to provide
baseline models with a sufficiently high test accuracy. We choose MNIST [24]
and CIFAR10 [25] as datasets since prior work [21, 26, 1] has obtained good
models with these datasets using FedAvg. For MNIST, we use a 5-layer
convolutional neural network [18]. For CIFAR10, we use VGG16 [27]444VGG16 has
no batch normalization layer that uses training data statistics. Therefore, it
is a suitable model for federated learning. which is an off-the-shelf complex
DNN model trained over the ImageNet dataset [28]. For the federated learning
setup, we use 100 clients participating the training and select 10 clients
randomly as $C_{sub}$ in each $t$. We repeat our experiments for different
number of local passes $E_{c}=\\{1,5,10,20\\}$ used by each client. We
distribute the training data over clients in IID fashion. Each client receives
600 and 500 training images for MNIST and CIFAR10, respectively. We measure
the test accuracy using the test subset of datasets, assuming a similar setup
as in [2]: the model is evaluated at different client devices not involved in
the federated learning process. Details about the training configuration
settings can be found in Appendix A-B.

We compare Waffle to other methods presented in IV-A and to baseline models
that do not use a watermarking scheme. Baseline models are constructed using
FedAvg and we use the tuple of $\\{E_{c},E_{a}\\}$ to define different
baseline models. We achieve $99\%$ test accuracy on MNIST in all 4 baseline
models. For CIFAR10, we achieve $85\%$ test accuracy similar to [1].

###  V-B Watermark Sets

For generating WafflePattern we choose 100 for the size of the watermark set
since it is sufficient to provide a high confidence $>1-2^{-64}$ during
demonstration of ownership [9, 11] while not degrading the overall performance
of the model. In both MNIST and CIFAR10, each class contains 10 watermark
samples. In order to construct WafflePattern, we first generate 100 images
with Gaussian noise, then randomly create 10 different patterns and label each
pattern with a different class. Finally, each noisy image is combined with a
random pattern. Figure 2b shows samples of WafflePattern generated for the
CIFAR10 task.

We also compare the performance of WafflePattern to other state-of-the-art
watermark generation methods: Embedded Content [13], unRelate [9, 13] and
unStruct [14]. Figure 2 shows samples of these watermarks generated for the
CIFAR10 task. Details about these watermark sets are given in Appendix A-C.

##  VI Evaluation

We evaluate the performance of Waffle and WafflePattern using the requirements
defined in Section III-B. We compare the performance of models watermarked
using Waffle to Pre-embedded models and baseline models without watermark (see
Section V). We also analyze how well WafflePattern performs by comparing it to
Embedded Content [13], unRelate [9, 13] and unStruct [14] (see Figure 2).
Further evaluation with additional experimental results can be found in
Appendix A-D.

###  VI-A W1 Demonstration of Ownership

W1 requires Verify to return $True$ at any $t$. We explained in Section IV
that $\mathrm{WM}_{w_{G}}$ is secret, only known by the model owner and
$\mathit{Agg}$. It can be registered in a public bulletin as proposed in [9].
Thus, W1 is satisfied if $Acc(w^{+}_{G(t)},\mathrm{WM}_{w_{G}})\geq T_{acc}$.
Using the formula defined in [11], we compute that a reliable demonstration of
ownership with confidence $>1-2^{-64}$, given a watermark set of size 100 for
a 10-classes classifier (MNIST and CIFAR10 models), is provided by
$T_{acc}=47\%$.

Table I shows that Pre-embedding achieves on average
$Acc(w^{+}_{G(E_{a})},\mathrm{WM}_{w_{G}})=27\%$ for MNIST and $15\%$ for
CIFAR10. Watermarks embedded via Pre-embedding are not resilient to re-
training. In contrast, Waffle achieves high watermark accuracy ($\sim 99.0\%$)
at the last aggregation round in both tasks and Verify returns $True$ before
$w_{G}$ starts to improve (after 10 aggregation rounds on average). We can
successfully embed all four types of watermark sets long before the global
model converges. Therefore, in the remaining experiments, we only evaluate
Waffle with different watermark sets.

TABLE I: Average watermark accuracy (over 4 different watermarks) at the final
aggregation round $E_{a}$. Waffle satisfies W1 while Pre-embedding does not:
$Acc(w^{+}_{G(E_{a})},\mathrm{WM}_{w_{G}})<47\%$.

|  $Acc(w^{+}_{G(E_{a})},\mathrm{WM}_{w_{G}})$ for MNIST |  $Acc(w^{+}_{G(E_{a})},\mathrm{WM}_{w_{G}})$ for CIFAR10
---|---|---
$\\{E_{c},E_{a}\\}$ | Pre-embedding | Waffle | Pre-embedding | Waffle
$\\{1,250\\}$ | 24.00 | 99.00 | 15.00 | 99.00
$\\{5,200\\}$ | 30.00 | 99.00 | 14.00 | 99.50
$\\{10,150\\}$ | 22.75 | 98.50 | 15.00 | 99.00
$\\{20,100\\}$ | 31.00 | 98.75 | 16.00 | 99.75

###  VI-B W2 Robustness

W2 states that embedded watermarks should be robust against post-processing
watermark removal techniques. For evaluating the robustness, we use three
state-of-the-art defenses against backdooring. Fine-tuning [29] and pruning
[30, 31] are generic watermark removal techniques that do not require
knowledge of the trigger pattern or the watermarking method. Both techniques
utilize some clean dataset (e.g., subset of a training set) and re-train the
model in order to remove the watermark. Neural Cleanse [32] is a technique to
detect, reverse-engineer and subsequently remove potential backdoors from DNN
models, and it can be used for watermark removal. To evaluate these
techniques, we set the acceptable utility drop of 5 percentage point (pp): the
watermark must be removed while keeping the test accuracy degradation less
than 5pp.

####  VI-B1 Fine-tuning attack

(a)  Figure 3: Comparison of watermark accuracy for MNIST and CIFAR10 at
different $\\{E_{c},E_{a}\\}$ tuples when fine-tuning is implemented by an
increasing number of malicious clients. TABLE II: Test and watermark (WM)
accuracy (%) for different numbers of fine-tuning epochs on watermarked MNIST
and CIFAR10 models using WafflePattern at different $\\{E_{c},E_{a}\\}$
tuples. 1 adversary out of 100 clients.

| MNIST | CIFAR10
---|---|---
$\\{E_{c},E_{a}\\}$ | $\\{5,200\\}$ | $\\{10,150\\}$ | $\\{5,200\\}$ | $\\{10,150\\}$
epoch | Test | WM | Test | WM | Test | WM | Test | WM
0 | 99.0 | 99.0 | 98.9 | 100.0 | 85.6 | 100.0 | 85.8 | 99.0
20 | 98.9 | 98.8 | 98.7 | 98.8 | 85.5 | 99.8 | 85.5 | 96.8
40 | 98.9 | 98.5 | 98.7 | 98.8 | 85.5 | 99.8 | 85.5 | 96.2
60 | 98.9 | 98.2 | 98.7 | 98.2 | 85.5 | 99.8 | 85.6 | 96.0
80 | 98.9 | 98.2 | 98.7 | 98.0 | 85.5 | 99.8 | 85.6 | 96.0
100 | 98.9 | 98.0 | 98.7 | 98.0 | 85.5 | 99.8 | 85.6 | 96.2

We tested fine-tuning attack using an increasing number of malicious clients
combining their local datasets. Figure 3 and A1 (Appendix A-D) show that
WafflePattern is the most resilient watermark to fine-tuning for CIFAR10 in
all experiments while it is the second most resilient watermark for MNIST when
$\\{E_{c},E_{a}\\}=\\{5,200\\}$ and $\\{10,150\\}$. Table II and A1 (Appendix
A-D) provide a detailed evaluation of resilience according to the number of
fine-tuning epochs run by $adv$. Tables show that WafflePattern is resilient
to fine-tuning even for a large number of fine-tuning epochs.

####  VI-B2 Pruning attack

(a)  Figure 4: Comparison of the test and watermark (wm) accuracy for MNIST
and CIFAR10 at different $\\{E_{c},E_{a}\\}$ tuples when the pruning attack is
implemented by one adversary.

We implemented parameter pruning proposed in [30] followed by fine-tuning by
one $adv$ using different pruning rates. We chose the pruning technique in
[30] that first removes a number of connections with magnitudes close to zero
from the dense model and obtains a more sparse model. Then, it re-trains this
sparse model to recover the test accuracy. Figure 4 and A2 Appendix A-D) show
that WafflePattern is the most resilient watermark for CIFAR10, since the
watermark accuracy decreases below $T_{acc}$ when a higher percentage of
neurons removed from CIFAR10 models compared to other watermark types.
Although it is not the most resilient in MNIST, Figure 4 shows that if the
adversary removes more than 70% of the neurons from MNIST models (respectively
50% in CIFAR10 models), test accuracy starts to decline. Therefore, even
though $adv$ can evade demonstration of ownership, i.e.,
$\textsc{Verify}(w_{adv},\mathrm{WM}_{G})\rightarrow False$, the pruned model
shows a drop in test accuracy larger than 5pp compared to the original
$w_{adv}$. We also measured that WafflePattern satisfies W2 if the proportion
of malicious clients is smaller than 40%.

####  VI-B3 Neural Cleanse attack

Neural Cleanse first finds potential small trigger patterns needed to
misclassify all inputs as a specific label and repeats this step for all
classes. Then, it outputs an anomaly index based on this analysis. If the
anomaly index is above 2.0, the model is considered backdoored. In the case of
a suspected backdoor, Neural Cleanse also marks any class that is likely to be
infected. We found that adversaries cannot detect the presence of watermarks
in MNIST (Table A4, Appendix A-D). On the other hand, $adv$ can recognize that
the model is watermarked in CIFAR10. However, it can only mark up to 3
possibly infected classes, while all ten classes are infected by our
watermarks. Moreover, the reversed triggers of marked classes look completely
different from the original patterns, as shown in Figure A4 (Appendix A-D).

We also implemented patching-via-unlearning [32], a mitigation technique using
reversed triggers obtained via Neural Cleanse. This method requires relaxing
some assumptions of our adversary model and leaking information about the
triggers to $adv$. We assumed that $adv$ is aware of the presence of
watermarks in $w_{adv}$ and it knows that the watermark set includes
different, distinctive patterns for each class. Performing this attack, a
single $adv$ cannot remove the watermark from MNIST model and at least 40% of
clients should be malicious for a successful removal. (Figure A3, Appendix
A-D). For CIFAR10, one $adv$ might evade the verification but the performance
drop for watermarked models using WafflePattern is over 5pp after patching as
shown in Table III. If less than 10% of clients are malicious, they cannot
recover the performance of $w_{G}$ and WafflePattern satisfies W2.

TABLE III: Robustness against NeuralCleanse patching via unlearning. Test and
watermark (WM) accuracy (%) for CIFAR10 models watermarked using
WafflePattern.

$\\{E_{c},E_{a}\\}$ | $\\{1,250\\}$ | $\\{5,200\\}$ | $\\{10,150\\}$ | $\\{20,100\\}$
---|---|---|---|---
# of adv. | Test | WM | Test | WM | Test | WM | Test | WM
0 (Baseline) | 85.7 | 100.0 | 85.6 | 100.0 | 85.8 | 99.0 | 85.6 | 100.0
1 | 74.0 | 27.2 | 75.9 | 25.2 | 75.6 | 28.0 | 76.2 | 35.0
2 | 78.8 | 29.5 | 73.4 | 31.8 | 72.6 | 26.0 | 74.4 | 26.5
5 | 78.1 | 29.8 | 80.5 | 26.0 | 78.5 | 31.2 | 78.8 | 24.8
10 | 80.2 | 27.5 | 80.8 | 34.5 | 80.9 | 20.0 | 78.4 | 38.8
20 | 79.3 | 36.5 | 81.6 | 30.2 | 81.5 | 25.2 | 80.8 | 36.5
30 | 80.6 | 17.8 | 82.4 | 26.5 | 81.0 | 34.8 | 82.0 | 37.5
40 | 83.3 | 30.8 | 81.8 | 27.8 | 81.1 | 40.2 | 81.5 | 30.0

###  VI-C P1 Model Utility

According to P1, watermarking should not degrade the test accuracy of the
converged model, i.e., $w^{+}_{G(t=E_{a})}$ and
$Acc(w^{+}_{G(t=E_{a})},D_{test})\approx Acc(w_{adv(t=E_{a})},D_{test})$. For
evaluating utility, we measured the test accuracy of watermarked federated
learning models using four different watermark sets, and compared it to non-
watermarked baseline models. Table IV presents the test accuracy of baseline
and watermarked models using different $\\{E_{c},E_{a}\\}$ combinations.
Results show that WafflePattern reaches a sufficient test accuracy similar to
baseline models ($\leq 0.2pp$ for MNIST and $\leq 0.7pp$ for CIFAR10.) and
does not degrade the test accuracy as much as unStruct in MNIST or Embedded
Content in CIFAR10. WafflePattern satisfies P1.

###  VI-D P2-P3 Communication and computational overhead

P2 and P3 state that both the watermarking procedure and the watermark set
should not increase the communication overhead and incur minimal additional
computation. Waffle increases computation by re-training the global model in
each $t$. Different watermark sets may be easier/harder to learn and incur
additional communication (aggregation rounds) for the model to converge for
both the watermark set and its main task: satisfying both W1 and P1.

We evaluated the communication overhead by measuring the increase in test
accuracy of watermarked models according to $E_{a}$. Reaching a high test
accuracy in a smaller $E_{a}$ means that the model can be trained with minimal
communications. Figure 5 and A5 (Appendix A-D) compares the test accuracy
progression of non-watermarked baseline models and watermarked models using
different watermark sets. In MNIST, the test accuracy quickly converges in
baseline models and all watermarked models except for unStruct. In CIFAR10,
Embedded Content converges slower than other watermark sets and requires more
aggregation rounds for obtaining a performance similar to baseline models. In
both cases, WafflePattern satisfies P2.

We also calculated the computational overhead by dividing the total number of
retraining rounds to the total number of local passes performed by clients.
Table V gives the computational overhead in Waffle using different watermarks.
While unStruct has the lowest computational overhead, it requires twice as
much aggregation rounds compared to other models to reach $99\%$ test accuracy
in MNIST. WafflePattern usually needs fewer retraining rounds than other
watermark sets, since it contains similar features that can be learned easily
for each class. Therefore, WafflePattern satisfies P3.

TABLE IV: The test accuracy (%) (at $t=E_{a}$) of watermarked models using
different watermark sets.

$\\{E_{c},E_{a}\\}$ | Watermark pattern
---|---
MNIST | Baseline | WafflePattern | Embedded C. | unRelate | unStruct
$\\{1,250\\}$ | 98.97 | 98.88 | 99.05 | 98.92 | 97.59
$\\{5,200\\}$ | 98.91 | 98.94 | 98.98 | 98.79 | 98.13
$\\{10,150\\}$ | 99.11 | 99.06 | 98.97 | 99.06 | 97.97
$\\{20,100\\}$ | 99.02 | 98.85 | 98.97 | 98.79 | 97.77
CIFAR10 | Baseline | WafflePattern | Embedded C. | unRelate | unStruct
$\\{1,250\\}$ | 86.27 | 85.70 | 85.19 | 85.81 | 86.53
$\\{5,200\\}$ | 86.24 | 85.61 | 86.21 | 86.25 | 85.99
$\\{10,150\\}$ | 85.90 | 85.89 | 85.69 | 85.76 | 85.91
$\\{20,100\\}$ | 85.85 | 85.67 | 85.47 | 85.74 | 85.72

(a)  Figure 5: Progression of the test accuracy for MNIST and CIFAR10.
Baseline and watermarked models using different watermark sets are shown in
the figure. TABLE V: Average computational overhead (%) incurred by the
retraining rounds in watermarked models using different watermark sets.

Dataset | WafflePattern | Embedded C. | unRelate | unStruct
---|---|---|---|---
MNIST |  3.06 |  2.02 |  10.39 |  0.91
CIFAR10 |  2.97 |  5.72 |  6.10 |  1.47

###  VI-E Evasion of Verification

In addition to methods to recover and remove watermarks, $adv$ can also attack
the verification mechanisms [9, 11] used to demonstrate model ownership. Even
though verification is not in the scope of our paper, we discuss possible
attacks.

$adv$ may try to evade verification by detecting queries for watermark samples
as out-of-distribution (OOD) samples. We implemented and tested this attack
using the threshold-based detector model introduced in [16] for CIFAR10. This
method is shown [16] to be strong enough to evade verification against
backdoor-based watermarking methods [9, 10] with a negligible false positive
rate (FPR). As a possible watermark set, we use a subset of the
TinyImageNet555https://tiny-imagenet.herokuapp.com, which is similar to
unRelate. $adv$ trains the detector with both its training data which
represents in-distribution data, and TinyImageNet subset representing OOD
data. We investigate two different scenarios: 1) Each client including
adversaries, has a balanced, IID dataset as defined in the adversary model,
and 2) a more realistic scenario where clients as well as adversaries have
non-IID, unbalanced datasets [1]. In both scenarios, CIFAR10 models are
watermarked with WafflePattern where $\\{E_{c_{1}},E_{a_{1}}\\}=\\{1,250\\}$,
$\\{E_{c_{2}},E_{a_{2}}\\}=\\{1,450\\}$ and $w_{adv}^{2}$ has 82.5% test
accuracy so the two models have similar performance. Table VI reports both the
lowest FPR calculated over multiple thresholds and the true positive rate
(TPR, the ratio of watermark samples correctly identified as OOD to the
watermark set) at that FPR. As can be seen from the table, Waffle watermark
verification could be evaded if $adv$ has high quality IID data. However,
Waffle is resilient to evasion in a non-IID setting. In the non-IID scenario,
verification can be evaded with 5% FPR only if more than 50% clients are
adversaries and share their datasets for training the detector. Moreover,
since $adv$ having only a limited training dataset might choose poor OOD data,
this affects the detection performance and increases FPR to a degree such that
the resulting prediction model is unusable [33]. Therefore, evasion of
verification is not a concern for real-world client-server federated learning
with non-IID data.

TABLE VI: Evasion of verification in both IID and non-IID settings. The lowest false positive rate (FPR) and true positive rate (TPR) calculated at that FPR is reported for watermarked CIFAR10 models using WafflePattern. | IID setting | non-IID setting
---|---|---
# of adv. | TPR | (lowest) FPR | TPR | (lowest) FPR
1 | 64.0 | 0.8 | 89.5 | 53.0
2 | 78.7 | 1.3 | 80.8 | 39.3
5 | 88.0 | 1.6 | 92.2 | 22.9
10 | 94.7 | 2.5 | 90.8 | 19.7
20 | 90.0 | 1.1 | 91.8 | 7.0
30 | 96.5 | 1.6 | 88.0 | 15.3
40 | 81.0 | 1.0 | 91.8 | 6.8
50 | 80.0 | 0.6 | 84.0 | 4.8

##  VII Discussion and Takeaways

In contrast to existing Pre- and Post-embedding techniques, Waffle meets W1
enabling to reliably demonstrate ownership of $w_{G}$ at any aggregation
round. Figure 6 summarizes the overall effectiveness of different watermark
types, and it is evident that WafflePattern gives the best trade-off (big area
and never the worst one) considering all six requirements.

Figure 6: Trade-off between meeting the requirements in Section III-B for
different watermark sets. All watermarks are ranked between 1-4 (the higher,
the better) based on the results obtained from all experiments.

Waffle is also resilient to almost all watermark removal attacks if less than
40% clients are malicious, as summarized in Table VII. While Neural Cleanse
requires only 10% malicious clients to be successful against the CIFAR10
model, adversaries would have to share their datasets and collaborate with
each other to recover the triggers. The collaboration between adversaries in a
distributed attack is different from a sybil attack in our case. Creating
multiple identities does not improve removal and evasion attacks since the
performance of these attacks is directly related to the quantity and the
diversity of data held by the adversary. Collaboration is impractical since
adversaries need to reveal their highly-sensitive dataset to other
untrustworthy parties. We also empirically evaluated that watermark removal
attacks fail to decrease watermark accuracy or recover test accuracy, if
performed in a federated learning setup without sharing their datasets. On the
other hand, if an adversary holds more than 10% of the whole training data, it
could successfully remove watermarks from $w_{adv}$ without decreasing the
model performance. This scenario is not typical in large-scale distributed
learning settings, where there are a very large number of clients, each
holding a tiny fraction of the overall training data. We conclude that under
reasonable assumptions, Waffle is resilient to distributed attacks of several
malicious federated learning clients.

TABLE VII: Minimum ratio of malicious clients required to evade Waffle.

Attack type | MNIST | CIFAR10
---|---|---
Fine-tuning [29] | 50% | 50%
Pruning [30] | 40% | 40%
Neural Cleanse, patching-via-unlearning [32] | 40% | 10%
Evasion of verification with OOD detection [16] | - | 50%

Adversaries can also try to cast model ownership into doubt by embedding its
own watermark into the model: ownership piracy. This issue can be addressed by
registering any watermark into a time-stamped bulletin (e.g., blockchain) for
it to be valid [9, 11]. We can enforce our watermark to be registered together
with some private artifacts of the model before federated learning starts as a
proof of authenticity. Such a private artifact can be the architecture of the
model (number of layers, number of neurons, type of activation functions,
etc.) since all these parameters can provide a large entropy and they are
unknown to any federated learning client before they receive the initial
model.

Finally, a model stealing attack can be performed against $w_{adv}$ by $adv$
itself to create a surrogate model without watermark or embed its own
watermark [34, 11]. Model stealing typically requires a large amount of data
(more than specified by our adversary model) and it causes accuracy drops
larger than 5pp [35]. Thus, we consider these attacks impractical in our
adversary model.

We assume that each client’s training data $D_{c_{i}}$ is IID (Section III-A),
and based this assumption on state-of-the-art papers [1, 22, 36] Although IID
assumption leads to a better setting to study the impact of model utility
requirement P1, in real world federated learning applications, $D_{c_{i}}$’s
are typically unbalanced and non-IID. Therefore, we evaluated the performance
of Waffle and WafflePattern in non-IID scenarios, and concluded that we met
requirements W1 and P1 with a slight increase in communication and
computational overhead. Detailed experimental results can be found in Appendix
A-E.

While we focused on a single owner $O$, Waffle can also be used if several
clients would all be owners of the trained model. In this case, $\mathit{Agg}$
can generate and distribute different subsets of the watermark set to each
client for individual demonstration of ownership. There are challenges to be
addressed in the case of collective ownership since the size of the watermark
set increases linearly with the number of owners and there might be a decrease
in utility. We consider extending Waffle to this case as future work. We will
also explore how malicious clients can try to recover and degrade watermarks
during the training phase of the federated learning and provide
detection/mitigation techniques against these adversaries.

##  VIII Related Work

Watermarking DNNs for ownership verification was first proposed in [12] by
using backdooring techniques. Deepmarks [37] presents a collusion-secure
watermarking method that encodes watermarks into the probability density
function of weights using a specific regularization loss during the training
phase. However, these techniques require direct access to model weights
(white-box access) for ownership verification. DeepSigns [14] is the first
watermarking method applicable with both white-box and black-box access, and
it embeds watermarks to the activation maps of selected layers. DeepIPR [38]
proposes a passport-based DNN ownership verification scheme that tries to
embed watermarks into a special passport layer of DNNs. Although DeepIPR is
robust against watermark removal attacks, it is expensive and imposes a
significant computational cost. A zero-bit watermarking algorithm [39] embeds
watermarks into models, that are stolen via black-box methods, by leveraging
adversarial perturbations. Although this approach is feasible, it heavily
depends on adversarial examples and their transferability property across
different models. Authors in [16] propose a blind-watermark based approach
that inserts a specifically designed logo into the original sample via encoder
such that the resulting sample is almost indistinguishable from the original
one. All these proposals require full control of the training process and
cannot be applied in federated learning.

Federated learning is vulnerable to adversarial attacks that alters the
training and inference phase of the system. Poisoning attacks are first
introduced in [21], where a malicious client trains its local model on the
backdoor and attempts to replace the global model with the poisoned model.
Another powerful attack is model update poisoning attacks, where the adversary
aims to prevent the global model from converging to a desirable state by
sending poisoned model updates to the server [36]. [40] states that clients
might suffer Byzantine failures, which leads to arbitrary behavior across
communication rounds affecting the convergence of the global model.

Client-server federated learning also suffers from privacy leakage [41, 42,
43]. For example, in [41], a malicious client can learn about class
representatives of other clients’ training data by using generative
adversarial networks. Attackers in client-server federated learning implement
passive [43] and active [42] membership inference attacks in order to detect
whether a sample belongs to the overall training set or a specific client. In
passive attacks, the attacker could be the aggregator that can only observe
individual model updates. In active attacks, malicious clients try to
influence the global model in order to extract more information about other
clients’ training dataset. There have been a few attempts to prevent the
privacy leakage applying differential privacy [44] or using trusted execution
environments in clients’ devices [45]. However, these methods trade-off either
computational overhead or performance for privacy.

## References

  * [1] H. B. McMahan, E. Moore, D. Ramage, S. Hampson _et al._ , “Communication-efficient learning of deep networks from decentralized data,” _arXiv preprint arXiv:1602.05629_ , 2016.
  * [2] A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage, “Federated learning for mobile keyboard prediction,” _arXiv preprint arXiv:1811.03604_ , 2018.
  * [3] T. Yang, G. Andrew, H. Eichner, H. Sun, W. Li, N. Kong, D. Ramage, and F. Beaufays, “Applied federated learning: Improving google keyboard query suggestions,” _arXiv preprint arXiv:1812.02903_ , 2018.
  * [4] F. Granqvist, M. Seigel, R. van Dalen, Áine Cahill, S. Shum, and M. Paulik, “Improving on-device speaker verification using federated learning with privacy,” 2020. [Online]. Available: https://arxiv.org/pdf/2008.02651.pdf
  * [5] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konecny, S. Mazzocchi, H. B. McMahan _et al._ , “Towards federated learning at scale: System design,” _arXiv preprint arXiv:1902.01046_ , 2019.
  * [6] K. Li, “Reverse engineering ai models,” 2018, hITBSecConf. [Online]. Available: https://conference.hitb.org/hitbsecconf2018dxb/sessions/ai-model-security-reverse-engineering-machine-learning-models/
  * [7] Z. Sun, R. Sun, and L. Lu, “Mind your weight (s): A large-scale study on insufficient machine learning model protection in mobile apps,” _arXiv preprint arXiv:2002.07687_ , 2020.
  * [8] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _arXiv preprint arXiv:1912.04977_ , 2019.
  * [9] Y. Adi, C. Baum, M. Cisse, B. Pinkas, and J. Keshet, “Turning your weakness into a strength: Watermarking deep neural networks by backdooring,” in _27th Security Symposium (Security 18)_ , 2018, pp. 1615–1631.
  * [10] J. Guo and M. Potkonjak, “Watermarking deep neural networks for embedded systems,” in _2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_. IEEE, 2018, pp. 1–8.
  * [11] S. Szyller, B. G. Atli, S. Marchal, and N. Asokan, “Dawn: Dynamic adversarial watermarking of neural networks,” _arXiv preprint arXiv:1906.00830_ , 2019\.
  * [12] Y. Uchida, Y. Nagai, S. Sakazawa, and S. Satoh, “Embedding watermarks into deep neural networks,” in _Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval_ , 2017, pp. 269–277.
  * [13] J. Zhang, Z. Gu, J. Jang, H. Wu, M. P. Stoecklin, H. Huang, and I. Molloy, “Protecting intellectual property of deep neural networks with watermarking,” in _Proceedings of the 2018 on Asia Conference on Computer and Communications Security_ , 2018, pp. 159–172.
  * [14] B. Darvish Rouhani, H. Chen, and F. Koushanfar, “Deepsigns: An end-to-end watermarking framework for ownership protection of deep neural networks,” in _Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2019, pp. 485–497.
  * [15] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 1175–1191.
  * [16] Z. Li, C. Hu, Y. Zhang, and S. Guo, “How to prove your model belongs to you: a blind-watermark based framework to protect intellectual property of dnn,” in _Proceedings of the 35th Annual Computer Security Applications Conference_ , 2019, pp. 126–137.
  * [17] Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks.” in _ISOC Network and Distributed System Security Symposium_ , 2018.
  * [18] M. Juuti, S. Szyller, S. Marchal, and N. Asokan, “Prada: protecting against dnn model stealing attacks,” in _2019 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 2019, pp. 512–527.
  * [19] F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction apis,” in _25th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 16)_, 2016, pp. 601–618.
  * [20] W. A. Group, “Federated learning white paper v1.0,” https://www.fedai.org/static/flwp-en.pdf, 2018, online; accessed 29 January 2020\.
  * [21] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” _arXiv preprint arXiv:1807.00459_ , 2018.
  * [22] P. Blanchard, R. Guerraoui, J. Stainer _et al._ , “Machine learning with adversaries: Byzantine tolerant gradient descent,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 119–129.
  * [23] D. Yin, Y. Chen, K. Ramchandran, and P. Bartlett, “Byzantine-robust distributed learning: Towards optimal statistical rates,” _arXiv preprint arXiv:1803.01498_ , 2018.
  * [24] Y. LeCun, C. Cortes, and C. Burges, “Mnist handwritten digit database,” _ATT Labs [Online]. Available:http://yann.lecun.com/exdb/mnist_, vol. 2, 2010.
  * [25] A. Krizhevsky, V. Nair, and G. Hinton, “Cifar-10 (canadian institute for advanced research).” [Online]. Available: http://www.cs.toronto.edu/~kriz/cifar.html
  * [26] D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” _arXiv preprint arXiv:1910.03581_ , 2019.
  * [27] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [28] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [29] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” in _Advances in neural information processing systems_ , 2014, pp. 3320–3328.
  * [30] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in _Advances in neural information processing systems_ , 2015, pp. 1135–1143.
  * [31] Y. He, X. Zhang, and J. Sun, “Channel pruning for accelerating very deep neural networks,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * [32] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao, “Neural cleanse: Identifying and mitigating backdoor attacks in neural networks,” in _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2019, pp. 707–723.
  * [33] L. Song, V. Sehwag, A. N. Bhagoji, and P. Mittal, “A critical evaluation of open-world machine learning,” _arXiv preprint arXiv:2007.04391_ , 2020.
  * [34] M. Shafieinejad, J. Wang, N. Lukas, and F. Kerschbaum, “On the robustness of the backdoor-based watermarking in deep neural networks,” _arXiv preprint arXiv:1906.07745_ , 2019.
  * [35] B. G. Atli, S. Szyller, M. Juuti, S. Marchal, and N. Asokan, “Extraction of complex dnn models: Real threat or boogeyman?” _arXiv preprint arXiv:1910.05429_ , 2019.
  * [36] E. M. E. Mhamdi, R. Guerraoui, and S. Rouault, “The hidden vulnerability of distributed learning in byzantium,” in _Proceedings of the 35th International Conference on Machine Learning_ , 2018, pp. 3518–3527.
  * [37] H. Chen, B. D. Rouhani, C. Fu, J. Zhao, and F. Koushanfar, “Deepmarks: A secure fingerprinting framework for digital rights management of deep learning models,” in _Proceedings of the 2019 on International Conference on Multimedia Retrieval_. ACM, 2019, pp. 105–113.
  * [38] L. Fan, K. W. Ng, and C. S. Chan, “Rethinking deep neural network ownership verification: Embedding passports to defeat ambiguity attacks,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 4716–4725.
  * [39] E. L. Merrer, P. Perez, and G. Trédan, “Adversarial frontier stitching for remote neural network watermarking,” _arXiv preprint arXiv:1711.01894_ , 2017\.
  * [40] Y. Chen, L. Su, and J. Xu, “Distributed statistical machine learning in adversarial settings: Byzantine gradient descent,” _Proceedings of the ACM on Measurement and Analysis of Computing Systems_ , vol. 1, no. 2, pp. 1–25, 2017.
  * [41] B. Hitaj, G. Ateniese, and F. Perez-Cruz, “Deep models under the gan: information leakage from collaborative deep learning,” in _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 603–618.
  * [42] L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2019, pp. 691–706.
  * [43] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Stand-alone and federated learning under passive and active white-box inference attacks,” _arXiv preprint arXiv:1812.00910_ , 2018.
  * [44] R. C. Geyer, T. Klein, and M. Nabi, “Differentially private federated learning: A client level perspective,” _arXiv preprint arXiv:1712.07557_ , 2017.
  * [45] F. Mo, A. S. Shamsabadi, K. Katevas, S. Demetriou, I. Leontiadis, A. Cavallaro, and H. Haddadi, “Darknetz: towards model privacy at the edge using trusted execution environments,” _arXiv preprint arXiv:2004.05703_ , 2020.
  * [46] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * [47] T. Ryffel, A. Trask, M. Dahl, B. Wagner, J. Mancuso, D. Rueckert, and J. Passerat-Palmbach, “A generic framework for privacy preserving deep learning,” _arXiv preprint arXiv:1811.04017_ , 2018.
  * [48] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [49] R. Namba and J. Sakuma, “Robust watermarking of neural network with exponential weighting,” in _Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security_ , 2019, pp. 228–240.

##  Appendix A Appendix

###  A-A Algorithm for Waffle

Algorithm 1 Functions and notations used in Waffle

$t$ | aggregation round
---|---
$w_{G(0)}$ | randomly initialized global model
$w^{-}_{G(t+1)}$ | updated global model at $t$
$w^{+}_{G(t+1)}$ | watermarked global model at $t$
$\mathrm{WM}_{w_{G}}$ | watermark set designed by $O$
$th$ | min threshold value for $Acc(w^{-}_{G(t+1)},\mathrm{WM}_{w_{G}})$
$E_{r}$ | max number of re-training rounds
$E_{i}$ | number of pre-training rounds
$b_{G}$ | mini batch of $\mathit{Agg}$
$\eta_{G}$ | learning rate in Retrain
$\eta_{i}$ | learning rate in Pretrain
$\ell_{G}$ | loss function
$\triangledown\ell_{G}$ | gradient of the loss function $\ell_{G}$

1: Pretrain$(w_{G(0)},\mathrm{WM}_{w_{G}}):$

2: for round $i=1\cdots E_{i}$ do

3: for $b_{G}\in\mathrm{WM}_{w_{G}}$ do

4: $w^{+}_{G(0)}\leftarrow
w_{G(0)}-\eta_{i}\triangledown\ell_{G}(w_{G(0)};b_{G})$

5: end for

6: end for

7: return $w^{+}_{G(0)}$

1: Retrain$(w_{c_{i}(t)},\forall c_{i}\in C_{sub}):$

2: $w^{-}_{G(t+1)}\leftarrow FedAvg(w_{c_{i}(t)})$

3: $tr\leftarrow 0$

4: while $Acc(w^{-}_{G(t+1)},\mathrm{WM}_{w_{G}})<th$ and $t_{r}\leq E_{r}$ do

5: for $b_{G}\in\mathrm{WM}_{w_{G}}$ do

6: $w^{+}_{G(t+1)}\leftarrow
w^{-}_{G(t+1)}-\eta_{G}\triangledown\ell_{G}(w^{-}_{G(t+1)};b_{G})$

7: end for

8: $t_{r}\leftarrow t_{r}+1$

9: end while

10: return $w^{+}_{G(t+1)}$

###  A-B Detailed Experimental Setup and Hyperparameter Selection

To facilitate the comparative performance evaluation of Waffle (and its
variants) and state-of-the-art watermark generation techniques, we used the
following experimental setup: PyTorch (version 1.4.0) [46] and Pysyft
(0.1.21a1) [47] library. Pysyft is a secure and private machine learning
library that is used for applications including federated learning. All
experiments are done in a computer with 2x12 core Intel(R) Xeon(R) CPUs (32GB
RAM) and NVIDIA Quadro P5000 with 16GB memory.

For all models and experiments, total number clients involved in the training
is 100. In each aggregation round $t$, aggregator averages model updates
uploaded from 10 randomly selected $c_{i}$ to recompute a new global model
$w_{G(t+1)}$. For all experiments, we used the cross-entropy loss, Stochastic
Gradient Descent (SGD) [48] with a batch size of $50$ and learning rate of
$0.1$ and $0.01$ for training MNIST and CIFAR10 models, respectively. We used
a learning rate of $0.001$ at only one experiment to protect the model from
diverging: watermarking MNIST models using unStruct.

During Pretrain and Retrain, we trained models using the cross entropy loss
and SGD with a batch size of $50$. During Pretrain, we used a learning rate of
$0.1$ and $5\times 10^{-4}$ for MNIST and CIFAR10, respectively. During
Retrain, we used a small learning rate of $0.005$ and $5\times 10^{-4}$ for
MNIST and CIFAR10 to ensure that the shift in the local minimum is as small as
possible so that optimizing the model on the watermark set does not corrupt
the actual task. We also used a momentum of $0.5$ and a weight decay of
$5\times 10^{-5}$ in Pretrain. During Pretrain, we trained MNIST models for 25
epochs with WafflePattern, 90 epochs with Embedded Content, 80 epochs with
unRelate and 150 epochs with unConstruct. Similarly, we pre-trained CIFAR10
models for 30 epochs with WafflePattern, 60 epochs with Embedded Content, 55
epochs with unRelate and 200 epochs with unConstruct. For all experiments, the
stop condition for Retrain is $Acc(w^{-}_{G(t+1)},\mathrm{WM}_{w_{G}})<98\%$
or the maximum number of training rounds $E_{r}=100$ is reached.

###  A-C Generating State-of-the-art Watermark Sets

Embedded Content [13] takes a subset of the training data and modifies samples
of this subset by adding a meaningful content (e.g., logo, text, a
specifically designed pattern) into them. While original samples are labeled
correctly, modified samples have incorrect labels that are pre-defined by the
model owner. We construct Embedded Content by randomly selecting 10 images
from the training data for each class, 100 images in total. We modify these
images by adding a specifically designed pattern to them and assign incorrect
labels to this modified set. We emphasize that Embedded Content requires
training data knowledge, so it is not a suitable method for watermarking
large-scale federated learning models. Nevertheless, we compare WafflePattern
to Embedded Content, since it is proved to be a robust watermark [13] against
post-processing techniques. unRelate [13, 9] is either designed as
unstructured abstract images [9] or as structured images from another data
distribution [13], both of which are unrelated to the original task. These
images are labeled with classes from the original task. For example, if the
task is face recognition, the model owner might use different handwriting
images to construct the watermark. We construct unRelate by randomly sampling
100 images from the ImageNet dataset which are unrelated to CIFAR10 and MNIST
[13]. unStruct [14] generates a random watermark set and adds it into
different layers of DNN models. In order to imitate unStruct, we produce
watermark samples with purely Gaussian noise as in [49]. In both unRelate and
unStruct, 10 classes are assigned to randomly selected 10 images, 100 in
total.

###  A-D Additional Experimental Results

####  A-D1 Fine-tuning attack

(a)  Figure A1: Comparison of watermark accuracy for MNIST and CIFAR10 at
different $\\{E_{c},E_{a}\\}$ tuples when fine-tuning is implemented by an
increasing number of malicious clients. TABLE A1: Test and watermark (WM)
accuracy (%) for different numbers of fine-tuning epochs on watermarked MNIST
and CIFAR10 models using WafflePattern at different $\\{E_{c},E_{a}\\}$
tuples. 1 adversary out of 100 clients.

| MNIST | CIFAR10
---|---|---
$\\{E_{c},E_{a}\\}$ | $\\{1,250\\}$ | $\\{20,100\\}$ | $\\{1,250\\}$ | $\\{20,100\\}$
epoch | Test | WM | Test | WM | Test | WM | Test | WM
0 | 98.8 | 99.0 | 98.8 | 99.0 | 85.7 | 100.0 | 85.6 | 100.0
20 | 98.6 | 97.8 | 98.5 | 88.2 | 85.4 | 94.2 | 84.4 | 95.2
40 | 98.6 | 97.2 | 98.5 | 88.0 | 85.4 | 94.0 | 84.5 | 94.8
60 | 98.6 | 95.5 | 98.5 | 88.0 | 85.4 | 93.0 | 84.6 | 95.0
80 | 98.6 | 95.2 | 98.5 | 88.0 | 85.4 | 93.0 | 84.6 | 95.0
100 | 98.6 | 95.0 | 98.5 | 88.0 | 85.4 | 93.0 | 84.6 | 94.8

Fine-tuning is the most likely attack that an adversary might attempt, since
it only involves re-training the model on its original training dataset. We
tested fine-tuning by making adversaries run the same local training procedure
as used during federated learning once more on the final watermarked model
$w_{adv}={w^{+}_{G(t)}}$. Figure A1 shows that all watermark sets except
Embedded Content are robust to fine-tuning even when up to 50% of clients are
malicious. Table A1 further shows the evolution of the test and watermark
accuracy when running an increasing number of fine-tuning epochs against
WafflePattern. We can see that the watermark accuracy initially decreases by
at most 10 percentage points and then reaches a plateau after a maximum of 100
epochs. The final watermark accuracy is always high enough (>85%) to enable
reliable proof of ownership showing that WafflePattern is resilient to a large
number of fine-tuning epochs.

####  A-D2 Fine-pruning attack

(a)  Figure A2: Comparison of the test and watermark (wm) accuracy for MNIST
and CIFAR10 at different $\\{E_{c},E_{a}\\}$ tuples when the pruning attack is
implemented by one adversary.

(a) MNIST (b) CIFAR10

Figure A3: Comparison of the test and watermark (wm) accuracies for the MNIST
(a) and CIFAR10 (b) models when patching via unlearning is implemented against
watermarked models.

The pruning technique in [30] first removes a number of connections with
magnitudes close to zero from the dense model and obtains a more sparse model.
Then, it re-trains this sparse model to recover the test accuracy. In our
setup, adversaries prune their model $w_{adv}={w^{+}_{G(t)}}$ and then
implement the fine-tuning attack. Figure A2 illustrates the test and watermark
accuracy of $w_{adv}$ when pruning is implemented by one adversary using
different pruning rates. We see that the pruning is effective at decreasing
the watermark accuracy for all watermark sets when a high percentage of
neurons are removed from the model (70% for MNIST and 50% for CIFAR10).
However, as explained in Section VI-B2, when the watermark accuracy is below
$T_{acc}$, the test accuracy also starts to decrease and the pruned model
cannot achieve a high test accuracy as the original model $w_{adv}$.

####  A-D3 Neural Cleanse attack

Table A2 shows that in MNIST, Neural Cleanse returns an anomaly index below
2.0 for all watermarked models using all four watermark sets. In CIFAR10, we
measure an anomaly index around 2.5, but Figure A4 shows that the reversed
triggers of marked classes and original watermarks are dissimilar. Patching-
via-unlearning results are plotted in Figure A3. As can be seen from figure,
in MNIST, WafflePattern is robust when less than 40% of clients are
adversaries. In CIFAR10, one adversary might evade the verification but the
performance drop is more than 5%.

(a) Clean

(b) WM

(c) Reversed

(d) Clean

(e) WM

(f) Reversed

Figure A4: Visualization of reversed triggers where Neural Cleanse is
implemented by one adversary against watermarked models using WafflePattern.
We show the original images (a) MNIST class 1 and (d) CIFAR10 class 9; example
watermark samples for them (b) and (e) respectively; and reversed triggers via
Neural Cleanse (c) and (f) respectively.

####  A-D4 Communication overhead

Figure A5 shows the convergence in test accuracy for MNIST and CIFAR10 during
the training at different $\\{E_{c},E_{a}\\}$ tuples. In MNIST, unStruct has
the slowest convergence rate, while Embedded Content converges slightly slower
than other watermarks in CIFAR10 except $\\{E_{c},E_{a}\\}=\\{5,200\\}$.
Embedded Content has also the second slowest convergence rate in MNIST.
WafflePattern satisfies P2 in both MNIST and CIFAR10.

TABLE A2:  Average anomaly index for watermarked models watermarked using
different watermark sets. Results are averaged over watermarked models with
different $\\{E_{c},E_{a}\\}$ tuples and various number of adversaries.

Dataset | WafflePattern | Embedded C. | unRelate | unStruct
---|---|---|---|---
MNIST |  1.27 |  1.32 |  1.47 |  1.54
CIFAR10 |  2.35 |  2.16 |  2.32 |  2.15

(a)  (b)  (c)

Figure A5: Progression of the test accuracy for MNIST and CIFAR10. Baseline
and watermarked models using different watermark sets are shown in the figure.

###  A-E Waffle in Non-IID scenarios

As explained in Section VII, we evaluated the performance of Waffle and
WafflePattern in a more realistic federated learning setup, where each client
holds a non-IID training data. For both MNIST and CIFAR10, we implemented the
same partitioning in [1] in order to distribute the training set to clients in
a non-IID fashion. In this partitioning, clients and adversaries will have
samples for only two classes, so each resulting $D_{c_{i}}$ is still balanced
(i.e. includes the same number of samples for each class) but highly non-IID.

We trained both baseline and watermarked models using WafflePattern, where the
learning rate is decreased by 2 and $E_{a}$ is increased in order to obtain a
model with a reasonable performance and prevent the loss from diverging. As
shown in Table A3, the test accuracy of baseline and watermarked models are
quite similar while the watermark accuracy is still high. Therefore, Waffle
with WafflePattern satisfies demonstration of ownership W1 and model utility
P1 requirements in extreme non-IIDness. However, the computational overhead is
higher in CIFAR10, since the convergence to both training and watermark sets
slows down due the skewness in client’s data distribution and the watermark
set. We also should note that the extreme non-IIDness also negatively affects
the watermark removal techniques presented in Section VI-B. For example, in
Neural Cleanse, one $adv$ can only find reverse triggers for only two classes,
which reduces the effect of patching-via-unlearning. In pruning, one $adv$
retrains the pruned model with only samples from two classes, and thus cannot
recover the test accuracy as much as in the IID scenario.

TABLE A3: Test accuracy (%), watermark accuracy (%) and computational overhead (%) of watermarked models using WafflePattern. | Baseline models | Watermarked models
---|---|---
MNIST | Test | Test | WM | Comp.
$\\{E_{c},E_{a}\\}$ | Acc. | Acc. | Acc. | overhead
$\\{1,500\\}$ | 98.44 | 97.61 | 100.0 | 0.80
$\\{1,450\\}$ | 98.49 | 98.60 | 100.0 | 0.23
$\\{1,300\\}$ | 98.14 | 98.38 | 100.0 | 0.16
$\\{1,350\\}$ | 98.51 | 98.74 | 100.0 | 0.15
CIFAR10 | Test | Test | WM | Avg comp.
$\\{E_{c},E_{a}\\}$ | Acc. | Acc. | Acc. | overhead
$\\{1,450\\}$ | 81.88 | 81.97 | 99.00 | 3.24
$\\{1,400\\}$ | 83.30 | 83.24 | 100.0 | 4.17
$\\{1,350\\}$ | 83.49 | 82.76 | 99.00 | 4.02
$\\{1,300\\}$ | 81.28 | 80.59 | 100.0 | 2.04