# Riesz inequality for harmonic quasiregular mappings

David Kalaj  University of Montenegro, Faculty of Natural Sciences and
Mathematics, Cetinjski put b.b. 81000 Podgorica, Montenegro<EMAIL_ADDRESS>
###### Abstract.

Let $K\geqslant 1$ and $p\in(1,2]$. We obtain asymptotically sharp constant
$c(K,p)$, when $K\to 1$ in the inequality

$\|\Im f\|_{p}\leq c(K,p)\|\Re(f)\|_{p}$

where $f\in\mathbf{h}^{p}$ is $K-$quasiregular harmonic mapping in the unit
disk belonging to the Hardy space $\mathbf{h}^{p}$, under the conditions
$\arg(f(0))\in(-\pi/(2p),\pi/(2p))$ and
$f(\mathbb{D})\cap(-\infty,0)=\emptyset$. The paper improves a recent result
by Liu and Zhu in [13]. We also extend this result for the quasiregular
harmonic mappings in the unit ball in $\mathbb{R}^{n}$.

###### Key words and phrases:

Harmonic mappings, minimal surfaces

###### Contents

  1. 1 Introduction
  2. 2 Main results
  3. 3 Proof of Theorem 2.1
  4. 4 Power of a quasiregular function and proof of Theorem 2.2

##  1\. Introduction

Let $\mathbb{D}$ denote the unit disk and $\mathbf{T}$ the unit circle in the
complex plane. For $p>1$, we define the Hardy class $\mathbf{h}^{p}$ as the
class of harmonic mappings $f=g+\bar{h}$, where $g$ and $h$ are holomorphic
mappings defined on the unit disk $\mathbb{D},$ so that

$\|f\|_{p}=\|f\|_{\mathbf{h}^{p}}=\sup_{0<r<1}M_{p}(f,r)<\infty,$

where

$M_{p}(f,r)=\left(\int_{\mathbf{T}}|f(r\zeta)|^{p}d\sigma(\zeta)\right)^{1/p}.$

Here $d\sigma(\zeta)=\frac{dt}{2\pi},$ if $\zeta=e^{it}\in\mathbf{T}$. The
subclass of holomorphic mappings that belongs to the class $\mathbf{h}^{p}$ is
denoted by $H^{p}$.

If $f\in\mathbf{h}^{p}$, then it is well-known that there exists

$f(e^{it})=\lim_{r\to 1}f(re^{it}),a.e.$

and $f\in L^{p}(\mathbf{T}).$ Then there hold

(1.1) $\|f\|^{p}_{{\mathbf{h}^{p}}}=\lim_{r\to
1}\int_{0}^{2\pi}|f(re^{it})|^{p}\frac{dt}{2\pi}=\int_{0}^{2\pi}|f(e^{it})|^{p}\frac{dt}{2\pi}.$

Similarly, we define the Hardy space $\mathbf{h}^{p}$ of harmonic functions in
the unit ball $\mathbb{B}\subset\mathbb{R}^{n}$. Let
$\mathbb{S}=\partial\mathbb{B}$. Then we say a harmonic function $u$ in
$\mathbb{B}$ belongs to the Hardy space $\mathbf{h}^{p}$ if

$\|f\|_{p}=\|f\|_{\mathbf{h}^{p}}=\sup_{0<r<1}M_{p}(f,r)<\infty,$

where

$M_{p}(f,r)=\left(\int_{\mathbb{S}}|f(\zeta)|^{p}d\sigma(\zeta)\right)^{1/p}.$

Here $d\sigma$ is the surface $n-1$ dimensional measure of the Euclidean
sphere which satisfies the condition: $\sigma(\mathbb{S})=1$. The hardy space
of harmonic function in the space has similar properties as in the plane (see
e.g. [1]).

Let $1<p<\infty$ and let $\overline{p}=\max\\{p,p/(p-1)\\}$. Verbitsky in [17]
proved the following results. If $f=u+iv\in H^{p}$ and $v(0)=0$, then

(1.2) $\sec(\pi/(2\overline{p}))\|v\|_{p}\leq\|f\|_{p},$

and

(1.3) $\|f\|_{p}\leq\csc(\pi/(2\overline{p}))\|u\|_{p},$

and both estimates are sharp. Those results improve the sharp inequality

(1.4) $\|v\|_{p}\leq\cot(\pi/(2\overline{p}))\|u\|_{p}$

found by S. Pichorides ([16]). For some related results see [4, 6, 10, 8].

Then those results have been extended by the author in [11]. As a byproduct,
the author proved a Hollenbeck-Verbitsky conjecture for the case $s=2$.

Further, those results have been extended by Marković and Melentijević in [14]
and Melentijević in [12]. Melentijević proved a Hollenbeck-Verbitsky
conjecture for the case $s<\sec^{2}(\pi/(2p))$, $p\leq 4/3$ or $p\geqslant 2$.

Recently Liu and Zhu in [13] generalized Riesz conjugate functions theorem for
planar harmonic $K$-quasiregular mappings (when $1<p\leq 2$) provided that the
real part does not vanish at the unit disk. Their result is asymptotically
sharp when $K\to 1$. Moreover, they extended this result for invariant
harmonic quasiconformal mappings in the unit ball also assuming that the first
coordinate is non-vanishing.

In this paper, we will remove the assumption that the real part of the mapping
does not vanish. Moreover, our approach works for harmonic quasiregular
mappings in the space.

###  1.1. Quasiregular and harmonic mappings

A continuous and nonconstant mapping $f:G\to\mathbb{R}^{n}$, $n\geqslant 2$,
in the local Sobolev space $W^{1,n}_{loc}(G,\mathbb{R}^{n})$ is
$K$-quasiregular, $K\geqslant 1$, if

$|Df(x)|\leq K\ell(f^{\prime}(x))$

for almost every $x\in G$, where $G$ is an open subset of $\mathbb{R}^{n}$.
Here $Df(x)$ is the formal differential matrix and

$|Df(x)|=\sup_{|h|=1}|f^{\prime}(x)h|,\ \
\ell(Df(x))=\inf_{|h|=1}|f^{\prime}(x)h|.$

Let $(Df(x))^{t}$ be the transpose of the matrix $Df(x)$. Then we define the
Hilbert norm of $Df(x)$ by the formula

$\|Df(x)\|=\sqrt{\mathrm{trace}Df^{t}Df}=\sqrt{\sum_{k=1}^{n}\lambda_{k}},$

where $\lambda_{1}\leq\dots\leq\lambda_{n}$ are eigenvalues of the matrix
$Df^{t}Df$.

Observe that

$|Df^{t}(x)|=|Df(x)|,\ \ \ell(Df(x))=\ell(Df^{t}(x)).$

Further if $f$ is $K$-quasiregular, then

$\nabla|f(x)|=\nabla\sqrt{\left<f(x),f(x)\right>}=Df^{t}(x)\frac{f(x)}{|f(x)|}.$

Thus

(1.5) $|\nabla|f(x)||\geqslant\ell(Df(x))\geqslant\frac{|Df(x)|}{K}.$

If $n=2$, then we use the notation $k-$quasiconformal mapping for
$k=(K-1)/(K+1)$.

A smooth mapping $w:G\to\mathbb{R}^{n}$ is called harmonic if it satisfies the
Laplace equation $\Delta u=0$. The solution of the equation $\Delta w=g$ (in
the sense of distributions see [9]) in the unit ball, satisfying the boundary
condition $w|_{\mathbb{S}}=f\in L^{1}(\mathbb{S})$, is given by

(1.6)
$w(x)=\int_{\mathbb{S}}P(x,\eta)f(\eta)d\sigma(\eta)-\int_{\mathbb{B}}G(x,y)g(y)dV(y),\,|x|<1.$

Here

(1.7) $P(x,\eta)=\frac{1-|x|^{2}}{|x-\eta|^{n}}$

is the Poisson kernel and $d\sigma$ is the surface $n-1$ dimensional measure
of the Euclidean sphere which satisfies the condition:
$\int_{\mathbb{S}}P(x,\eta)d\sigma(\eta)\equiv 1$. The first integral in (1.6)
is called the Poisson integral and is usually denoted by $P[f](x)$. It is a
harmonic mapping. The function

(1.8)
$G(x,y)=\left\\{\begin{array}[]{ll}\frac{1}{2\pi}\log\frac{|x-y|}{|1-x\overline{y}|},&\hbox{for
$n=2$;}\\\
c_{n}\left(\frac{1}{|x-y|^{n-2}}-\frac{1}{(1+|x|^{2}|y|^{2}-2\left<x,y\right>)^{(n-2)/2}}\right),&\hbox{for
$n\geqslant 3$.}\end{array}\right.,$

where

(1.9) $c_{n}=\frac{1}{(n-2)\omega_{n-1}}$

and $\omega_{n-1}$ is the measure of $\mathbb{S}$, is the Green function of
the unit ball. The Poisson kernel and the Green function are harmonic in $x$.
If $f\in L^{1}$ and $g\in L^{\infty}$ then $u-P[f]\in C^{1,\alpha}$; see [5,
Theorem 8.33].

##  2\. Main results

The main result of this paper are following theorems

######  Theorem 2.1.

Let

$c_{n}^{p}(K,p)=\frac{(1+(n-1)K^{2})(1+\frac{(p-2)}{nK^{2}})}{(p-1)}.$

a) Assume that $f$ is a $K-$quasiregular harmonic mapping in the unit disk so
that $\Im f(0)=0$. Assume that $\Re f\in\mathbf{h}^{p}$ for some $p\in(1,2]$.
Then $f\in\mathbf{h}^{p}$ and we have the inequality

$\|f\|_{p}\leq c_{2}(K,p)\|\Re f\|_{p}.$

b) Assume that $f=(f_{1},\dots,f_{n})$ is a $K-$quasiregular harmonic mapping
of the unit ball into $\mathbb{R}^{n}$. Assume that $f_{1}\in\mathbf{h}^{p}$
for some $p\in(1,2]$. Then $f\in\mathbf{h}^{p}$ and we have the inequality

$\|f\|^{p}_{p}\leq|f(0)|^{p}+c_{n}(K,p)(\|f_{1}\|^{p}_{p}-|f_{1}(0)|^{p}).$

The constant $C_{k}(K,p)$ is asymptotically sharp when $K\to 1$ and $p\to 2$.

The following theorem is an improvement of the main result of Liu and Zhu
[13].

######  Theorem 2.2.

a) Let

$c(p,K)=\left(\frac{\tan^{p-1}\frac{\pi}{2p}}{\cot(\frac{\pi}{2p})}+(K^{2}-1)\frac{\sin^{p-1}\frac{\pi}{2p}}{\cos\frac{\pi}{2p}}\right)^{1/p}.$

Assume that $f$ is a $K-$quasiregular harmonic mapping in the unit disk so
that $\arg(f(0))\in(-\pi/(2p),\pi/(2p))$. Assume that $\Re f\in\mathbf{h}^{p}$
for some $p\in(1,2]$ and that $f(\mathbb{D})\cap(-\infty,0)=\emptyset$. Then
$f\in\mathbf{h}^{p}$ and we have the inequality

$\|\Im f\|_{p}\leq c(K,p)\|\Re f\|_{p}.$

The constant $c(K,p)$ is asymptotically sharp.

b) Under condition of a) for

$d(p,K)=\left(\cos\left[\frac{\pi}{2p}\right]^{-p}+(K^{2}-1)\tan\left[\frac{\pi}{2p}\right]\right)^{1/p},$

we have the asymptotically sharp inequality

$\|f\|_{p}\leq d(K,p)\|\Re f\|_{p}.$

######  Remark 2.3.

Astala and Koskela proved in [2, Theorem 6.1] that if $f$ is quasiconformal in
$\mathbb{B}$ with one of its coordinate functions belongs to $\mathbf{h}^{p}$,
then $f\in h^{q}$ for all $q<p$. Furthermore, they showed that $q$ can not
reach $p$ by giving a concrete example. Here we have a better outcome, but
under essentially stronger conditions. The proofs of those theorems are
similar to the proof of [3, Theorem 4.1], and this method has been also used
by Liu and Zhu in [13]. Theorem 2.2 improves the main result in [13], because
we do not assume that $u$ is positive and get the same outcome.F

##  3\. Proof of Theorem 2.1

###### Proof.

We will assume that the mapping $f$ has a smooth extension to the boundary. If
not, then we take the dilatation $F(x)=f(rx)$, with $r<1$ and let $r\to 1$.
The case $n=2$. Since

$\|f\|_{p}^{p}=\frac{1}{2\pi}\int_{-\pi}^{\pi}|f(e^{it})|^{p}dt,$

by using (1.6) to the function $w(z)=|f(z)|^{p}$ we get

$|f(0)|^{p}=\int_{0}^{2\pi}|f(e^{it})|^{p}\frac{dt}{2\pi}-\frac{1}{2\pi}\int_{\mathbb{D}}\log\frac{1}{|w|}\Delta|f(w)|^{p}dudv.$

Let $u=\Re f$ and assume that $u>0$. Let $dA(z)=dxdy$. Then

(3.1)
$\|u\|_{p}^{p}=|u(0)|^{p}+\frac{p(p-1)}{2\pi}\int_{\mathbb{D}}|u|^{p-2}|\nabla
u|^{2}\log\frac{1}{|z|}dA(z).$

Let $v(0)=0$,

$\|f\|_{p}^{p}=|f(0)|^{p}+\frac{1}{2\pi}\int_{\mathbb{D}}\Delta|f(z)|^{p}\log\frac{1}{|z|}dA(z)$

Since

$\Delta|f|^{p}=\frac{p(p-2)}{4}|f|^{p-4}\|\nabla|f|^{2}\|^{2}+\frac{p}{2}|f|^{p-2}\Delta|f|^{2}$

and

$\frac{p}{2}|f|^{p-2}\Delta|f|^{2}=4\frac{p}{2}|f|^{p-2}(|g^{\prime}(z)|^{2}+|h^{\prime}(z)|^{2}),$

and because $f$ is $K-$quasiregular, by (1.5) we have

$|\nabla|f|^{2}|^{2}=4|f|^{2}|\nabla|f||^{2}\geqslant
4|f|^{2}\|Df\|^{2}K^{-2}.$

So

$\begin{split}\Delta|f|^{p}&\leq\left(\frac{p(p-2)}{K^{2}}+2p\right)|f|^{p-2}(|g^{\prime}(z)|^{2}+|h^{\prime}(z)|^{2})\\\
&\leq\left(\frac{p(p-2)}{K^{2}}+2p\right)|u|^{p-2}(|g^{\prime}(z)|^{2}+|h^{\prime}(z)|^{2})\\\
&\leq\frac{1+K^{2}}{2}\left(\frac{p(p-2)}{K^{2}}+2p\right)|u|^{p-2}|g^{\prime}(z)+h^{\prime}(z)|^{2}.\end{split}$

For $p\in(1,2)$ and $v(0)=0$ since $|\nabla
u|^{2}=|g^{\prime}+h^{\prime}|^{2}$, by the previous estimate we obtain that

$\begin{split}\|f\|_{p}^{p}&\leq|f(0)|^{p}+\left(\frac{p(p-2)}{K^{2}}+2p\right)\int_{\mathbb{D}}|f|^{p-2}(|g^{\prime}(z)|^{2}+|h^{\prime}(z)|^{2})\log\frac{1}{|z|}\frac{dA(z)}{2\pi}\\\
&\leq|f(0)|^{p}+\frac{1+K^{2}}{2}\left(\frac{p(p-2)}{K^{2}}+2p\right)\int_{\mathbb{D}}|u|^{p-2}|\nabla
u|^{2}\log\frac{1}{|z|}\frac{dA(z)}{2\pi}\\\
&=|f(0)|^{p}+\frac{1+K^{2}}{2}\left(\frac{p(p-2)}{K^{2}}+2p\right)\frac{1}{p(p-1)}(\|u\|_{p}^{p}-|u(0)|^{p})\\\
&\leq\left(\frac{(p-2)}{K^{2}}+2\right)\frac{1+K^{2}}{2(p-1)}\|u\|_{p}^{p}.\end{split}$

If exists $a$ so that $u(a)=0$, then take
$w=w_{\epsilon}=\sqrt{\epsilon^{2}+u^{2}(x,y)}$, $\epsilon>0$, and get

$\Delta
w^{p}=p\left(\epsilon^{2}+u^{2}\right)^{-2+\frac{p}{2}}\left(\epsilon^{2}+(-1+p)u^{2}\right)|\nabla
u|^{2}.$

Observe that

$\partial_{\epsilon}w_{\epsilon}=\left(\epsilon^{2}+u^{2}\right)^{1-\frac{p}{2}}\left(\epsilon^{2}+u^{2}+\left(2-\frac{p}{2}\right)\left(\epsilon^{2}+(-1+p)u^{2}\right)\right)|\nabla
u|^{2}.$

It follows that $\epsilon\to w_{\epsilon}$ is increasing. Then by Green
formula we have

$\|w\|_{p}^{p}=|w(0)|^{p}+\frac{1}{2\pi}\int_{\mathbb{D}}\Delta
w^{p}\log\frac{1}{|z|}dA(z),$

and then by Fatou Lemma, letting $\epsilon\to 0$ we have

$\|u\|_{p}^{p}=|u(0)|^{p}+\frac{1}{2\pi}\int_{\mathbb{D}}\Delta|u|^{p}\log\frac{1}{|z|}dA(z),$

because $h_{n}=\Delta w^{p}\log\frac{1}{|z|}$, $\epsilon=1/n$, is positive and
decreasing sequence of functions converging to
$\Delta|u|^{p}\log\frac{1}{|z|}$. Then we repeat the previous proof, but with
$|u|$ instead of $u$. Notice that (3.1) can be also found in Pavlović book
[15, Lemma 6.2.5]. Here $u=\Re(f)=\Re(g+h)$.

The case $n>2$. This proof is similar to the proof of the case $n=2$. The only
difference appears due to the different Green functions. Assume that $u>0$ and
assume that $\omega_{n}$ is the $n-1$ area of $\mathbb{S}$ and let
$c_{n}=1/(\omega_{n}(n-2))$. Then by (1.6) we have

$\|f\|_{p}^{p}=|f(0)|^{p}+c_{n}\int_{\mathbb{B}}\Delta|f(x)|^{p}\left(|x|^{2-n}-1\right)dV(x).$

Further, we have

$\begin{split}\Delta|f|^{p}&=\frac{p(p-2)}{4}|f|^{p-4}\|\nabla|f|^{2}\|^{2}+\frac{p}{2}|f|^{p-2}\Delta|f|^{2}\\\
&=\frac{p(p-2)}{4}|f|^{p-4}\|\nabla|f|^{2}\|^{2}+p|f|^{p-2}\|Df\|^{2}.\end{split}$

Since

$|Df|^{2}=\max\\{{\lambda_{k}},k\in\\{1,\dots,n\\}\\}\geqslant\frac{1}{n}\sum_{k=1}^{m}\lambda_{k}=\frac{1}{n}\|Df\|^{2},$

where $\lambda_{k}$ are the eigenvalues of $(Df)^{\ast}Df$, in view of (1.5),
we get that

$\|\nabla|f|^{2}\|=4|f|^{2}\|\nabla|f|\|^{2}\geqslant
4|f|^{2}\frac{|Df|^{2}}{K^{2}}\geqslant 4|f|^{2}\frac{\|Df\|^{2}}{nK^{2}}.$

Thus

$\Delta|f|^{p}\leq(p+\frac{p(p-2)}{nK^{2}})|f|^{p-2}\|Df\|^{2}.$

On the other let $u=f_{1}$ and assume first that $u(x)>0$ for every $x$. Then
we have

$\Delta u^{p}=p(p-1)|\nabla u|^{2}u^{p-2}$

Thus

$\Delta|f|^{p}\leq\frac{(1+(n-1)K^{2})(1+\frac{(p-2)}{nK^{2}})}{(p-1)}\Delta
u^{p}.$

Thus for

$C(K,p)=\frac{(1+(n-1)K^{2})(1+\frac{(p-2)}{nK^{2}})}{(p-1)}$
$\begin{split}\|f\|_{p}^{p}&\leq|f(0)|^{p}+c_{n}\int_{\mathbb{B}}\Delta|f|^{p}\left(|x|^{2-n}-1\right)dV(x)\\\
&\leq|f(0)|^{p}+C(K,p)+c_{n}\int_{\mathbb{D}}\Delta|u|^{p}\left(|x|^{2-n}-1\right)dV(x)\\\
&\leq|f(0)|^{p}+C(K,p)\left(\|u\|_{p}^{p}-|u(0)|^{p}\right).\end{split}$

If $u$ vanishes in some points, then take
$w=w_{\epsilon}=\sqrt{\epsilon^{2}+u^{2}(x)}$ and get as before

$\Delta
w^{p}=p\left(\epsilon^{2}+u^{2}\right)^{-2+\frac{p}{2}}\left(\epsilon^{2}+(-1+p)u^{2}\right)|\nabla
u|^{2}.$

Observe also that $\epsilon\to w_{\epsilon}$ is increasing. Then by Green
formula we have

$\|w\|_{p}^{p}=|w(0)|^{p}+c_{n}\int_{\mathbb{B}}\Delta
w^{p}\left(\frac{1}{|x|^{n-2}}-1\right)dV(x),$

and then again by Fatou Lemma, letting $\epsilon\to 0$ we have

$\|u\|_{p}^{p}=|u(0)|^{p}+c_{n}\int_{\mathbb{B}}\Delta|u|^{p}\left(\frac{1}{|x|^{n-2}}-1\right)dV(x),$

because $h_{n}=\Delta w^{p}\log\frac{1}{|z|}$, $\epsilon=1/n$, is positive and
decreasing sequence of functions converging to

$w(x)=\Delta|u|^{p}\left(\frac{1}{|x|^{n-2}}-1\right)=p(p-1)|\nabla
u|^{2}|u|^{p-2}\left(\frac{1}{|x|^{n-2}}-1\right).$

The proof shows that $w(x)$ is integrable in the unit ball $\mathbb{B}$
provided that $u\in\mathbf{h}^{p}$ for $p\in(1,2]$. By repeating the previous
proof and noticing that

$\Delta|u|^{p}\left(\frac{1}{|x|^{n-2}}-1\right)=p(p-1)|\nabla
u|^{2}|u|^{p-2}\left(\frac{1}{|x|^{n-2}}-1\right)$

we get the desired conclusion.‘

The last statement of the theorem can be proved because for $I(x)=x$, the
equality holds in $\|I\|_{2}=C_{2}(1,2)\|I_{1}\|_{2}$, where $I_{1}(x)=x_{1}$.
Here $C_{2}(1,2)=\sqrt{n}$. In order to see this observe that

$\frac{1}{\omega_{n}}\int_{\mathbb{S}}x_{1}^{2}d\sigma(x)=\frac{1}{\omega_{n}}\int_{\mathbb{S}}x_{2}^{2}d\sigma(x)=\dots=\frac{1}{\omega_{n}}\int_{\mathbb{S}}x_{n}^{2}d\sigma(x).$

So summing altogether we get

$\frac{1}{\omega_{n}}\int_{\mathbb{S}}x_{1}^{2}d\sigma(x)=\frac{1}{n}.$

∎

##  4\. Power of a quasiregular function and proof of Theorem 2.2

Observe that as in Remark 2.3, $\Re(f^{p}(z))=|f(z)|^{p}\cos(p\arg(f(z)))$ is
smooth in $\mathbb{D}$, because
$\arg:\mathbb{C}\setminus(-\infty,0]\to(-\pi,\pi)$ is smooth. Then after
straightforward calculation, we get

$\Delta(f^{p})=p(p-1)f^{p-2}(f_{x}^{2}+f_{y}^{2}).$

The previous formula is correct, even though the function $f^{p}$ is
multivalent, so the previous equality is true only locally. This implies the
following formula

$\Delta\Re(f^{p})=p(p-1)\Re(f^{p-2}(f_{x}^{2}+f_{y}^{2})).$

Now we have

$f_{x}=g^{\prime}+\overline{h^{\prime}}$

and

$f_{y}=i(g^{\prime}-\overline{h^{\prime}}).$

So

$f_{x}^{2}+f_{y}^{2}=4g^{\prime}\overline{h^{\prime}}.$

Thus for $r=|f|$,

$|\Delta\Re(f^{p})|\leq
p(p-1)r^{p-2}(|f_{x}^{2}+f_{y}^{2}|=4p(p-1)r^{p-2}|g^{\prime}h^{\prime}|.$

Therefore

(4.1) $|\Delta\Re(f^{p})|\leq 4p(p-1)r^{p-2}|g^{\prime}\cdot h^{\prime}|\leq
4kp(p-1)r^{p-2}|g^{\prime}|^{2}.$

###### Proof of Theorem 2.2.

As in Theorem 2.1, we will assume that the mapping $f$ has a smooth extension
to the boundary. If not, then we take the dilatation $F(z)=f(rz)$, with $r<1$
and let $r\to 1$.

a) For $p\in[1,2]$ by [16, Lemma 2.1], we have for $p\in[1,2]$ and
$|x|\leq\pi$,

(4.2) $|\sin x|^{p}\leq A(p)|\cos x|^{p}-B(p)\cos(px),$

where

$A(p)=\frac{\tan^{p-1}\frac{\pi}{2p}}{\cot(\frac{\pi}{2p})},\ \
B(p)=\frac{\sin^{p-1}\frac{\pi}{2p}}{\cos\frac{\pi}{2p}}.$

Notice that the corresponding inequality in [16] is formulated only for
$|x|\leq\pi/2$, but the inequality (4.2) continues to hold for $|x|\leq\pi$.
Indeed, just observe that if $x\in[\pi/2,\pi]$, then $y=\pi-x\in[0,\pi/2]$ and
(4.2) hold for $y$ instead of $x$. But we also have
$\cos(p(\pi-x))\geqslant\cos px$ for $x\in[\pi/2,\pi]$. This proves the
claimed inequality.

Thus

$|v(re^{it})|^{p}\leq A(p)|u(re^{it})|^{p}-B(p)\Re(f^{p}((re^{it}))).$

In particular

(4.3) $\|v\|_{p}^{p}\leq
A(p)\|u\|_{p}^{p}-B(p)\int_{-\pi}^{\pi}\Re(f^{p}((e^{it})))\frac{dt}{2\pi}.$

Further, because $\Re(f^{p}((e^{it})))$ is smooth,

$\int_{-\pi}^{\pi}\Re(f^{p}((e^{it})))\frac{dt}{2\pi}=\Re(f^{p}(0))+\frac{1}{2\pi}\int_{\mathbb{D}}\Delta\Re
f^{p}(w)\log\frac{1}{|w|}dudv,$

and

$\int_{-\pi}^{\pi}|u(e^{it})|^{p}\frac{dt}{2\pi}=|u(0)|^{p}+\frac{1}{2\pi}\int_{\mathbb{D}}\Delta|u^{p}(w)|\log\frac{1}{|w|}dudv.$

Further, because $u=\Re(g+h)$, we have

(4.4) $\begin{split}\Delta|u|^{p}&=p(p-1)|\nabla u|^{2}|u|^{p-2}\\\
&=p(p-1)|g^{\prime}+h^{\prime}|^{2}|u|^{p-2}\\\ &\geqslant
p(p-1)(1-k)^{2}|g^{\prime}|^{2}|u|^{p-2}.\end{split}$

Then (4.1) and (4.4) imply that

$|\Delta\Re(f^{p})|\leq\frac{4k}{(1-k)^{2}}\Delta|u|^{p}.$

Thus

$\begin{split}\int_{-\pi}^{\pi}\Re(f^{p}((e^{it})))\frac{dt}{2\pi}&=\Re(f^{p}(0))+\frac{1}{2\pi}\int_{\mathbb{D}}\Delta\Re
f^{p}(w)\log\frac{1}{|w|}dudv,\\\
&\geqslant\Re(f^{p}(0))-\frac{1}{2\pi}\int_{\mathbb{D}}|\Delta\Re
f^{p}(w)|\log\frac{1}{|w|}dudv\\\ \
&\geqslant\Re(f^{p}(0))-\frac{4k}{(1-k)^{2}}\frac{1}{2\pi}\int_{\mathbb{D}}\Delta|u^{p}(w)|\log\frac{1}{|w|}dudv\\\
&=\Re(f^{p}(0))-\frac{4k}{(1-k)^{2}}\left(-|u(0)|^{p}+\int_{-\pi}^{\pi}|u(e^{it})|^{p}\frac{dt}{2\pi}\right)\\\
&=\Re(f^{p}(0))+\frac{4k}{(1-k)^{2}}|u(0)|^{p}-\frac{4k}{(1-k)^{2}}\int_{-\pi}^{\pi}|u(e^{it})|^{p}\frac{dt}{2\pi}.\end{split}$

Now if $\theta=\arg(f(0))\in(-\pi/(2p),\pi/(2p)$ then

$\Re(f^{p}(0))+\frac{4k}{(1-k)^{2}}|u(0)|^{p}\geqslant|f(0)|^{p}\cos(\theta
p)\geqslant 0.$

Thus

$\int_{-\pi}^{\pi}\Re(f^{p}((e^{it})))\frac{dt}{2\pi}\geqslant-\frac{4k}{(1-k)^{2}}\int_{-\pi}^{\pi}|u(e^{it})|^{p}\frac{dt}{2\pi}.$

Hence

$\begin{split}\|v\|_{p}^{p}&\leq
A(p)\|u\|_{p}^{p}-B(p)\int_{-\pi}^{\pi}\Re(f^{p}((e^{it})))\frac{dt}{2\pi}\\\
&\leq A(p)\|u\|_{p}^{p}+B(p)\frac{4k}{(1-k)^{2}}\|u\|_{p}^{p}\\\
&=\left(A(p)+(K^{2}-1)B(p)\right)\|u\|_{p}^{p}.\end{split}$

b) In this case we use the following inequality ([17]): for every
$t\in[-\pi,\pi]$ we have

$-1+\cos\left[\frac{\pi}{2p}\right]^{-p}|\cos
t|^{p}-\cos(pt)\tan\left[\frac{\pi}{2p}\right]\geqslant 0.$

Then the proof is very similar to the previous proof, but instead of (4.3) we
use the inequality

(4.5) $\|f\|_{p}^{p}\leq
C(p)\|u\|_{p}^{p}-D(p)\int_{-\pi}^{\pi}\Re(f^{p}((e^{it})))\frac{dt}{2\pi},$

where

$C(p)=\cos\left[\frac{\pi}{2p}\right]^{-p}\text{ and
}D(p)=\tan\left[\frac{\pi}{2p}\right].$

∎

######  Remark 4.1.

The proof of Theorem 2.2 also works under more general condition

$\cos(p\theta)+\frac{4k}{(1-k)^{2}}|\cos\theta|^{p}\geqslant 0,$

where $\theta=\arg(f(0))\in(-\pi,\pi]$.

### Acknowledgments

I would like to thank Professor Shaolin Chen for the useful discussion on this
topic.

## References

  * [1] S. Axler; P. Bourdon; W. Ramey: Harmonic function theory, Springer Verlag New York 1992.
  * [2] K. Astala, P. Koskela, _$H^{p}$ -theory for quasiconformal mappings._ Pure Appl. Math. Q. 7, No. 1, 19-50 (2011).
  * [3] P. Duren, _Theory of $H^{p}$ spaces._ Pure and Applied Mathematics (Academic Press) 38. New York and London: Academic Press. XII, 258 p. (1970).
  * [4] M. Essén, _A superharmonic proof of the M. Riesz conjugate function theorem._ Ark. Mat. 22 (1984), no. 2, 241–249.
  * [5] D. Gilbarg; N. Trudinger: _Elliptic Partial Differential Equations of Second Order_ , Vol. 224, 2nd Edition, Springer 1983.
  * [6] B. Hollenbeck, N. J. Kalton, I. E. Verbitsky,  _Best constants for some operators associated with the Fourier and Hilbert transforms._ Studia Math. 157 (2003), no. 3, 237–278.
  * [7] B. Hollenbeck, I. E. Verbitsky, _Best constants for the Riesz projection._ J. Funct. Anal. 175 (2000), no. 2, 370–392.
  * [8] B. Hollenbeck, I. E. Verbitsky, _Best constant inequalities involving the analytic and co-analytic projection._ Operator Theory: Advances and Applications 202, 285-295 (2010).
  * [9] L. Hörmander: _Notions of convexity_ , Progress in Mathematics, Birkhäuser Boston Inc., Boston, MA, 1994.
  * [10] L. Grafakos, _Best bounds for the Hilbert transform on $L^{p}(\mathbf{R})$._ Math. Res. Lett. 4 (1997), no. 4, 469–471.
  * [11] D. Kalaj, _On Riesz type inequalities for harmonic mappings on the unit disk,_ Trans. Am. Math. Soc. 372, No. 6, 4031–4051 (2019).
  * [12] P. Melentijević, _Hollenbeck-Verbitsky conjecture on best constant inequalities for analytic and co-analytic projections._ Math. Ann. (2023). https://doi.org/10.1007/s00208-023-02639-1
  * [13] J. Liu, J. F. Zhu, _Riesz conjugate functions theorem for harmonic, quasiconformal mappings_ , Advances in Mathematics 434 (2023) 109321.
  * [14] P. Melentijević, M. Marković, _Best Constants in Inequalities Involving Analytic and Co-Analytic Projections and Riesz’s Theorem in Various Function Spaces._ Potential Anal (2022). https://doi.org/10.1007/s11118-022-10021-0
  * [15] M. Pavlović.  Introduction to function spaces on the disk.  Belgrade: Matematički Institut SANU, 2004.
  * [16] S. K. Pichorides, S. _On the best values of the constants in the theorems of M. Riesz, Zygmund and Kolmogorov._ Collection of articles honoring the completion by Antoni Zygmund of 50 years of scientific activity, II. Studia Math. 44 (1972), 165–179.
  * [17] I. E. Verbitsky, _Estimate of the norm of a function in a Hardy space in terms of the norms of its real and imaginary parts._ Linear operators. Mat. Issled. No. 54 (1980), 16–20, 16–165, ”Fifteen Papers on Functional Analysis,” Amer. Math. Soci. Transl. Ser. 124 (1984), 11–15 (English transl.)


# FedSDD: Scalable and Diversity-enhanced Distillation for Model Aggregation
in Federated Learning

Ho Man Kwan, Shenghui Song
<EMAIL_ADDRESS><EMAIL_ADDRESS>
The Hong Kong University of Science and Technology

###### Abstract

Recently, innovative model aggregation methods based on knowledge distillation
(KD) have been proposed for federated learning (FL). These methods not only
improved the robustness of model aggregation over heterogeneous learning
environment, but also allowed training heterogeneous models on client devices.
However, the scalability of existing methods is not satisfactory, because the
training cost on the server increases with the number of clients, which limits
their application in large scale systems. Furthermore, the ensemble of
existing methods is built from a set of client models initialized from the
same checkpoint, causing low diversity. In this paper, we propose a scalable
and diversity-enhanced federated distillation scheme, FedSDD, which decouples
the training complexity from the number of clients to enhance the scalability,
and builds the ensemble from a set of aggregated models with enhanced
diversity. In particular, the teacher model in FedSDD is an ensemble built by
a small group of aggregated (global) models, instead of all client models,
such that the computation cost will not scale with the number of clients.
Furthermore, to enhance diversity, FedSDD only performs KD to enhance one of
the global models, i.e., the main global model, which improves the performance
of both the ensemble and the main global model. While partitioning client
model into more groups allow building an ensemble with more aggregated models,
the convergence of individual aggregated models will be slow down. We
introduce the temporal ensembling which leverage the issues, and provide
significant improvement with the heterogeneous settings. Experiment results
show that FedSDD outperforms other FL methods, including FedAvg and FedDF, on
the benchmark datasets.

##  1 Introduction

Federated learning (FL) [20] allows users to jointly train a deep learning
model without sharing their data. Recent works [19, 25, 12, 5] adopted
knowledge distillation (KD) [9] for model aggregation to tackle data and
device heterogeneity issues. Such distillation-based aggregation methods have
been shown to outperform weight averaging methods like FedAvg [20]. However,
existing works utilized all client models to build the ensemble for KD,
leading to poor scalability in real world applications, where a large number,
e.g., thousands, of clients are available. Furthermore, existing methods built
the ensemble by client models that were initialized from the same global
model, which may have low diversity and thus limit the performance of the
ensemble.

To tackle the above issues, we propose a new algorithm named Federated
Scalable and Diversity-enhanced Distillation (FedSDD). The key innovations of
FedSDD include: (i). Improving the scalability of distillation-based
aggregation by building the ensemble (i.e. the teacher model) from a set of
aggregated models, each of which is obtained by performing weight averaging
over a sub-group of client models. As a result, the computation complexity for
distillation at the server is independent of the number of clients but
dictated by the number of aggregated models. (ii). Maximizing the capacity of
the ensemble and the global model by enhancing the diversity among the
compositing models of the ensemble. This is achieved by training multiple
global models simultaneously, but only transferring the knowledge from the
ensemble to one of them. (iii). Ideally, a larger ensemble, i.e., more
involved models, has better capacity. However, in the scheme of leveraging
aggregated models as an ensemble ((i).), this results in fewer client models
per group, implicating a trade-off between individual global model convergence
and ensemble capacity. We propose alleviating this issue by introducing the
temporal ensembling. Furthermore, FedSDD has favorable properties in terms of
efficiency and privacy. In particular, FedSDD can achieve higher efficiency by
parallel training across the server and client sides. Different from existing
distillation-based methods, FedSDD does not require accessing client models,
which provides further protection to the user privacy. Experiment results show
that FedSDD can achieve the superior results on CIFAR10/100 datasets [16],
especially when the data of different clients are not independent and
identically distributed (Non-IID).

The contributions of this paper include:

1.) We propose FedSDD, a highly efficient and scalable distillation-based FL
algorithm, to tackle the limitation of existing FL algorithms from the
perspective of model aggregation. Building the ensemble by aggregated models
over a group of clients improves the scalability while distillation to only
one of the group models enhances the diversity among the compositing models of
the ensemble.

2.) Experiment results demonstrate that FedSDD can achieve superior
performance with lower complexity and latency compared to existing
distillation-based FL algorithms. Ablation studies highlight the impact of the
key elements in FedSDD, which can potentially benefit further works along the
same line of research.

##  2 Related works

Federated learning. Training deep learning models require a large amount of
data. In many practical scenarios, e.g. healthcare, data are distributed over
the user devices, and these user data may not be able to be shared due to
privacy regulations and/or communication constraints. To tackle the above
issues, federated learning [20] was proposed to train a shared model, normally
referred to as the global model, by utilizing the data distributed on the
client devices without data sharing. The simplest approach is FedAvg [20],
which performs multiple epochs of local training on the client side and then
aggregates the learned models on the server side by weight averaging. FedProx
[18] also utilizes weight averaging to perform model aggregation, but with
added regularization to tackle the data heterogeneity issue.

Knowledge distillation in federated learning. KD [9] has been proposed in deep
learning to compress deep neural networks. Student models, which typically
have a smaller size, are forced to mimic the output of the teacher model. In
some recent works, KD has been applied in FL with two lines of approaches. The
first utilizes distillation to perform model aggregation. For example, FedDF
[19] and FedBE [4] utilized client models as the teacher model to update the
global model on the server, and the purpose is to improve the model
aggregation performance with heterogeneous data. Some other works, including
FedFTG [25] and Fed-ET [5], utilized distillation for the same purpose, but in
a different setting, i.e. in a data-free or model heterogeneous environment.
The second approach shares model predictions between clients and the server
for training purposes. For example, in FD [13], model predictions were shared
between clients to regularize the local training. In FedAD [7], model outputs
of the clients were used to train a global model on the server through
distillation.

In this paper, we focus on distillation-based model aggregation for FL. The
proposed FedSDD utilizes distillation to perform model aggregation, and can be
classified into the first approach. In particular, FedSDD first performs local
training and weight averaging within multiple groups clients to obtain
multiple global models. Then, it builds an ensemble as the teacher model and
enhance one of the global model via distillation. The key innovations lie in
the enhanced scalability due to the way of building of ensemble and the
improved diversity across the compositing models of the ensemble by the
designated KD to only one of the global models. Furthermore, the process in
the first step is fully modularized, such that averaging based methods like
FedAvg [20] and FedProx [18] can be directly applied. As a result, the
proposed framework can easily adopt existing FL algorithms for further
enhancement.

##  3 Scalable and diversity-enhanced distillation for model aggregation

###  3.1 Overview

Figure 1: The workflow of FedSSD (w/o temporal ensembling). In FedSSD, there
are $K$ global models, where one of which is the main global model and we
optimize the main global model thought both local training and knowledge
distillation. In each round, all global models will be distributed to
different groups of clients for performing local training, and the updated
weights from the clients will be aggregated into the new weights of the
corresponding global models. Then, these updated global models will form an
ensemble, which will be utilized as the teacher model in knowledge
distillation, for enhancing the student model, i.e. the main global model.
Note that the temporal ensembling is skipped here for ease of presentation.

For ease of illustration, we assume the same training setting as FedDF [19],
where the labeled data are distributed over a set of clients, in a Non-IID
manner, and the server only has access to a small unlabeled dataset. The
training target is to learn a global model through multiple rounds of
optimization. Note that the key techniques of the proposed method can be
extended to different settings, e.g. with heterogeneous devices [19, 5].

In FedSDD, we perform $T$ rounds of training and, in each round, $K$ global
models will be trained by $K$ groups of clients. The detailed grouping and
training procedure will be introduced later. For simplicity, we assume that
these global models share the same architecture, but a distinct set of weights
where the initial weights of the $k$-th global model is denoted by
$w^{*}_{0,k}$ for $0\leq k<K$. In the $t$-th round, all $K$ global models are
updated and the new weights are denoted by $w^{*}_{t,k}$ for the $k$-th model.
The first global model, which has weights $w^{*}_{t,0}$, is also referred to
as the main global model in FedSDD, and our target is to maximize the
performance of this global model. There are two major operations in each round
of the proposed algorithm: (i). local training and weight averaging, and (ii).
knowledge distillation. The workflow of FedSSD is shown in Fig. 1. In the
following, we introduce the details of the two operations and highlight their
difference from existing methods.

####  3.1.1 Local training and weight averaging

Assume there are in total $N_{t}$ participating clients in the $t$-th round
with $1\leq t<T$. These clients are randomly but evenly distributed into $K$
groups, where $N_{t,k}$ denotes the number of clients in the $k$-th group.
Then, the server broadcasts the weights of the $k$-th global model from last
round, $w^{*}_{t-1,k}$, to the $k$-th group of clients to initialize their
local models. Thus, the initial model weights for the $n$-th client in the
$k$-th group is given by:

$w^{l}_{t,k,n}=w^{*}_{t-1,k},$ (1)

where $0\leq n<N_{t,k}$. After that, all clients perform local training with
their private datasets $X_{t,k,n}$. The updated weights, denoted by
$w^{l*}_{t,k,n}$, will then be collected by the server for weight averaging
[20] within each group to obtain $K$ averaged models. In particular, the
averaged model of the $k$-th group is given by:

$w_{t,k}=\sum^{N_{t,k}-1}_{i=0}\frac{|X_{t,k,i}|}{\sum^{N_{t,k}-1}_{j=0}|X_{t,k,j}|}w^{l*}_{t,k,i}.$
(2)

It is worth noticing that, the above workflow performs weights broadcasting,
local training, and aggregation in different groups, independently. The
workflow within one group is essentially the same as FedAvg [20], and shares
similar procedures as many common FL algorithms. Thus, the actual
implementation can be easily replaced by other federated learning algorithms,
to bring additional benefits, e.g. replacing FedAvg by FedProx [18] to tackle
the Non-IID issue by introducing regularization. Although FedDF [19] has
demonstrated a similar compatibility, FedSDD further improves it by the full
modularization of the weight averaging and local training step. As a result,
the direct access to the client models are not required by the server. For
example, by utilizing secure aggregation [2], the server can obtain the
results of aggregation but without accessing the individual client models,
which provide better privacy protection to the clients. In this paper, our
focus is not on the local training and weight averaging step, thus we use
FedAvg [20] as the default option. In the experiments, we will demonstrate
that FedSDD works well with different alternative FL algorithms.

Algorithm 1 FedSDD

$T$: Number of training rounds

$N_{t,k}$: Number of participating clients in the $k$-th group at the $t$-th
round.

$C_{t,k,n}$: The $n$-th client in the $k$-th group at the $t$-th round.

$K$: Number of global models, $R$: Number of checkpoints for building
ensemble.

// initialize $K$ global models $w^{*}_{0,0},...,w^{*}_{0,K-1}$

for $t\in\\{1,...,T\\}$ do

for $k\in\\{0,...,K-1\\}$ do in parallel

for $n\in\\{0,...,N_{t,k}-1\\}$ do in parallel

$w^{l}_{t,k,n}\leftarrow w^{*}_{t-1,k}$

$w^{l*}_{t,k,n}\leftarrow\textit{LocalTraining}(C_{t,k,n},w^{l}_{t,k,n})$

end for

$w_{t,k}\leftarrow\textit{Aggregation}(w^{l*}_{t,k,0},...,w^{l*}_{t,k,N_{t,k}-1})$

end for

for $k\in\\{0,...,K-1\\}$ do in parallel

if $k==0$ then

$w^{*}_{t,k}\leftarrow\textit{Distillation}(w_{t,0},\\{w_{t,0},...,w_{t,K-1},...,w_{t-R+1,0},...,w_{t-R+1,K-1}\\})$

else

$w^{*}_{t,k}\leftarrow w_{t,k}$

end if

end for

end for

####  3.1.2 Diversity-enhanced knowledge distillation

After weight averaging, the server builds an ensemble with the $K$ global
models, and utilizes it to perform distillation to enhance the performance of
the main global model (i.e. the model with weight $w_{t,0}$). Note that only
the weight of the main global model will be updated in this stage. The purpose
is to prevent different global models from learning similar knowledge and thus
enhance the diversity. This is expected to generate an ensemble that can
outperform individual global models. Let $F(x|w)$ denote the logit output of
the model with weight $w$ and input $x$, and denote $\sigma(.)$ as the softmax
function. Then, the output of the ensemble $F_{ensemble}$ can be determined by
the outputs of $K$ group models as:

$y=\sigma\left(F_{ensemble}(x|w_{0,t},...,w_{K-1,t})\right)=\sigma\left(\sum^{K-1}_{k=0}\frac{1}{K}F(x|w_{k,t})\right).$
(3)

For classification tasks, the final model weights at round $t$ can be
determined as:

$\begin{cases}w^{*}_{t,k}=\operatorname*{argmin}_{w}E_{x}\left[KL(\sigma(F(x|w)),\sigma(F_{ensemble}(x|w_{0,t},...,w_{K-1,t})))\right],&\text{if
}k=0\\\ w^{*}_{t,k}=w_{t,k},&\text{otherwise}\end{cases}$ (4)

where $KL$ denotes the KL-divergence loss [9, 18], and the expectation is
empirically minimized.

Remark 1: In FedSDD, KD is only performed for the main global model to avoid
forcing all clients to learn the same knowledge, i.e., to enhance the
diversity. It is also worth mentioning that the members of all groups will be
resampled and reshuffled at the beginning of each round. As a result, every
global model has a chance to learn from all client datasets.

Remark 2: The size of the ensemble is determined by the number of groups, $K$,
instead of the number of participating clients, $N_{t}$. As a result, the
computational cost of KD in FedSDD will not increase with the number of
participating clients, offering a better scalability than exiting methods.

####  3.1.3 Temporal ensembling

Although a large number of global models (i.e. a large $K$) may enhance the
ensemble capability, it may also slow down the convergence of individual
global models. To enhance the ensemble capability without sacrificing the
convergence of individual global models, we propose to build the ensemble by
leveraging multiple checkpoints of different training rounds. Intuitively,
global models from consecutive rounds were trained based on the data from
different subsets of the clients. Thus, ensembling global models from multiple
checkpoints of different rounds can emulate the case where more clients are
participating. This allows us to build a larger ensemble without slowing down
individual models’ convergence.

With temporal ensembling, the output of the ensemble can be determined as:

$y=\sigma(F_{ensemble}(x|w_{0,t},...,w_{K-1,t-R+1}))=\sigma\left(\sum^{K-1}_{k=0}\sum^{R-1}_{r=0}\frac{1}{KR}F(x|w_{k,t-r})\right)$
(5)

where $R$ denotes the number of rounds utilized for building the ensemble. In
our experiments, we found that building the ensemble from multiple rounds is
highly effective, especially when the degree of Non-IID is high.

By combining the above techniques, the FedSDD algorithm is shown Algo. 1 whose
work flow is given in Fig. 1.

###  3.2 Comparison with existing works

In this section, we compare FedSDD with existing methods.

Table 1: Comparison between different distillation-based methods.  Method | Ensemble size | Parallelism between server/clients | No access to client models
---|---|---|---
FedDF | $C$ | ✗ | ✗
FedBE | $C+S+1$ | ✗ | ✗
FedSDD (w/ FedAvg) | $KR$ | ✓ | ✓
$K$: Number of global models $R$: Number of checkpoint rounds $C$: Number of
active clients $S$: Number of sampled models

Distillation-based aggregation. Although existing works [19, 25, 12, 5] have
already utilised distillation to enhance the global model, they typically did
not concern about the scalability of the methods. For example, with FedDF
[19], the cost of performing inference with the ensemble will be high
especially for systems with a large number of clients, because the ensemble
output is the average of that from all client models.

As mentioned in Remark 2, FedSDD trains multiple global models and builds the
ensemble from them directly. By training $K$ global models and storing $R$
checkpoints for each of them, the ensemble is built by $K\times R$ models. As
a result, the complexity of the ensemble can be controlled by adjusting $K$
and $R$, instead of scaling with the number of clients.

Multiple global models. Multiple models have been utilized for both inference
and learning purposes. For example, Fed-ensemble [21] learns multiple models
and build the ensemble for inference. However, if the ensemble is large, it
can not be directly utilized by clients due to their limited computation
resources. The heterogeneous version of FedDF [19] trains multiple global
models with different architectures and scales, and performs distillation to
enhance all of them, which may ruin the diversity among models.

Different from existing works, FedSSD (i). targets on training a single,
compact, and high performance global model; and (ii). uses KD to update only
the main global model, which preserves the diversity and achieves enhanced
performance.

Diversity preserving knowledge distillation. Prior works in online KD (i.e.,
not for FL) have proposed different ways in preserving the models’ diversity
during training. Codistillation [1] applied the distillation term in its loss
function, after the training has been started for a while. OKDDip [3] applied
a similar scheme with one main student for distillation, but still utilized KD
to enhance the remaining student models. Comparing with OKDDip, the KD scheme
in this paper is not only simpler with reduced computation complexity (no
forward/backward pass for the remaining $K-1$ global models), but also allows
parallel training between the server-side and the client-side which can
benefit FL. See supplementary materials for more details.

Temporal ensembling. Prior works utilized multiple checkpoints to build
ensembles for prediction [11] or semi-supervised learning [17]. These works
either trained models for a number of epochs with learning rate scheduling to
obtain a diverse set of checkpoints at different local minimas, or utilized
data augmentation and model regularization to produce predictions with the
ensemble. FedGKD [23] utilized historical checkpoints for regularizing the
training of client models. In this paper, temporal ensembling is utilized in
FL systems for enhancing the capability of the teacher model. Moreover,
temporal ensembling in FedSDD can be simply achieved by combining model
checkpoints of consecutive rounds without any additional efforts. This is
possible due to the Non-IID learning environment in FL, and can prevent the
slowing down of convergence that may occur with the above-mentioned
techniques.

Additional advantage of FedSDD. FedSDD also provides additional benefits
beyond scalability and performance. Unlike previous works, FedSDD fully
modularizes the local training and the weight averaging step, and does not
require direct access to the client models. As a result, FedSDD can obtain the
same privacy protection level as weight averaging methods such as FedAvg [20].
For example, secure aggregation [2] can be used to prevent the server from
obtaining sensitive information from individual client models, which can’t be
achieved by existing knowledge distillation based methods [22]. Furthermore,
in FedSDD, only one global model, i.e. the main global model, is updated by
KD, thus the training of the other $K-1$ global models can start immediately,
after the previous round of training is completed. This allows for parallelism
between the server-side distillation and the client-side local training. The
parallelism can significantly improve the efficiency of federated learning.

We summarise the advantages of FedSDD compared with FedDF [19] and FedBE [4]
in Table 1.

##  4 Experiments

###  4.1 Image Classification with CIFAR10/100.

Table 2: CIFAR10/100 results. |  | CIFAR10 | CIFAR100
---|---|---|---
Model | Method | $\alpha=1.0$ | $\alpha=0.1$ | $\alpha=1.0$ | $\alpha=0.1$
ResNet20 | FedAvg | $88.53\pm 0.31$ | $78.72\pm 2.31$ | $58.84\pm 0.42$ | $52.98\pm 1.41$
| FedProx | $88.36\pm 0.18$ | $79.44\pm 2.17$ | $58.76\pm 0.93$ | $53.57\pm 0.37$
| SCAFFOLD | $89.85\pm 0.50$ | $80.08\pm 1.53$ | $61.09\pm 0.46$ | $54.74\pm 1.26$
| FedDF | $87.98\pm 0.16$ | $80.04\pm 1.87$ | $59.74\pm 0.38$ | $51.83\pm 0.83$
|  FedSDD ($R=1$) | $89.06\pm 0.19$ | $80.18\pm 2.38$ | $61.90\pm 0.83$ | $54.72\pm 0.87$
|  FedSDD ($R=2$) | $89.01\pm 0.18$ | $81.27\pm 2.16$ | $62.42\pm 0.57$ | $56.39\pm 1.10$
|  FedSDD ($R=4$) | $89.01\pm 0.07$ | $82.21\pm 1.10$ | $62.84\pm 0.45$ | $56.90\pm 0.77$
ResNet56 | FedAvg | $89.25\pm 0.11$ | $80.05\pm 2.65$ | $59.47\pm 1.23$ | $54.37\pm 1.24$
| FedProx | $89.65\pm 0.23$ | $80.84\pm 1.32$ | $61.01\pm 1.02$ | $57.18\pm 0.92$
| SCAFFOLD | $90.73\pm 0.27$ | $82.22\pm 0.92$ | $62.94\pm 0.91$ | $56.36\pm 0.31$
| FedDF | $88.87\pm 0.29$ | $81.10\pm 2.31$ | $60.71\pm 0.68$ | $52.52\pm 1.37$
|  FedSDD ($R=1$) | $89.86\pm 0.14$ | $81.30\pm 2.17$ | $62.95\pm 2.43$ | $57.48\pm 2.03$
|  FedSDD ($R=2$) | $90.23\pm 0.23$ | $82.77\pm 1.67$ | $65.28\pm 0.64$ | $59.37\pm 0.64$
|  FedSDD ($R=4$) | $90.00\pm 0.26$ | $83.20\pm 0.99$ | $65.63\pm 0.89$ | $59.56\pm 0.43$
WRN16-2 | FedAvg | $90.26\pm 0.33$ | $81.52\pm 2.66$ | $63.57\pm 1.06$ | $58.77\pm 0.55$
| FedProx | $90.38\pm 0.29$ | $81.78\pm 2.58$ | $63.75\pm 0.84$ | $59.24\pm 1.00$
| SCAFFOLD | $91.32\pm 0.11$ | $82.90\pm 2.19$ | $65.33\pm 0.48$ | $60.66\pm 0.85$
| FedDF | $89.52\pm 0.10$ | $81.32\pm 2.91$ | $63.54\pm 0.53$ | $56.89\pm 1.28$
|  FedSDD ($R=1$) | $90.69\pm 0.41$ | $82.62\pm 2.46$ | $66.46\pm 0.30$ | $61.16\pm 1.20$
|  FedSDD ($R=2$) | $90.77\pm 0.16$ | $83.39\pm 2.17$ | $67.11\pm 0.41$ | $62.37\pm 0.84$
|  FedSDD ($R=4$) | $90.70\pm 0.31$ | $84.02\pm 1.75$ | $67.48\pm 0.22$ | $63.05\pm 0.80$
Table 3: Round time comparison between FedAvg[20], FedDF[19] and FedSSD. Round times of FedDF and FedSDD are reported by the relative time to FedAvg, i.e. the time spent on KD. All results are in seconds. | Number of clients
---|---
Method | 8 | 14 | 20
FedAvg | 123.6 | 214.9 | 301.6
FedDF | +223.6 | +337.0 | +447.1
FedSDD | +174.1 | +177.1 | +175.5

In the experiments, we will perform classification tasks following the same
setting as [19]. In particular, the clients have the CIFAR10/100 [16] datasets
for local training. Similar to other FL algorithms with distillation (i.e.
FedDF [19] and FedSDD), the server holds unlabelled data from the
CIFAR100/ImageNet (resized to $32\times 32$) [6] to perform distillation for
classifying images from the CIFAR10/100 datasets, respectively. There are in
total 20 clients, where 40% of them participate in the training in each round,
with 100 rounds of training in total. We sampled data for each client by the
Dirichlet distribution [10], with $\alpha=\\{1.0,0.1\\}$ for different levels
of data heterogeneity. For each setting, we reported the average top-1
accuracy of three runs with different seeds, where the same seeds were shared
across different methods to generate the same data partition and sequence of
clients’ participation for a fair comparison. We trained ResNet20/56 [8] and
WRN16-2 [24], which have been proven capable to achieve high accuracy on
CIFAR10/100 datasets. For the optimizer, we used SGD with learning rate of 0.8
and 0.1 and batch size of 64 and 256, for the client-side and the server-side
training, respectively. We did not apply weight decay or learning rate
scheduling in our experiments.

We compared FedSDD with FedAvg [20], FedProx[18], SCAFFOLD[14] and FedDF[19].
We set the number of local training epochs to be 40, and the regularizer
parameter $\mu$ of FedProx to be 0.001. For FedSDD, we set the number of
global models $K$ to be 4 and the number of checkpoints $R$ to be 1/2/4 to
build the ensemble. For both FedDF and FedSDD, we performed distillation with
5000 steps in each training round, with the temperature $\tau$ to be 4.0.

The results are shown in Table 2. It can be observed that FedSDD outperformed
all other methods on the harder dataset, CIFAR100, for all settings with high
degree of data heterogeneity. It suggests that FedSDD is a good candidate for
tackling the Non-IID issue. We also found that the value of $R$ has little
impact to the performance of FedSDD in the CIFAR10 experiments with low degree
of data heterogeneity, but a higher value of $R$ improved the performance of
FedSDD significantly in the CIFAR10 runs with more Non-IID data, and on
CIFAR100 datasets with both high and low degrees of data heterogeneity. Note
that in the above experiments, FedSDD with $R\leq 2$ has an ensemble size no
larger than that of FedDF, thereby maintaining an overall lower computational
cost. In such cases, FedSDD can still outperform FedDF.

Next, we illustrate the advantage of FedSSD in terms of scalability. Due to
the lack of access to large scale FL system, we can only provide the
comparison with small scale experiments. Table 3 shows the comparison of the
round time (execution time per communication round) between FedAvg[20],
FedDF[19] and FedSSD. The experiment is conduced on an A100 GPU, with the
setting of the CIFAR10/100 experiment, where the model is ResNet-20. It can be
observed that, with 8 clients, the time spent by FedSDD on KD is around 78% of
that for FedDF. However, since the ensemble size depends on only the number of
global models ($K=4$ for this experiment), when increasing the number of
clients to 14/20, the time spent on KD reduces to 52%/39% of FedDf. This
demonstrates the scalability advantage of FedSDD, and the scalability is
essential for applying KD in large scale FL systems.

##  5 Ablation Studies

###  5.1 Compatibility of FedSDD with other FL algorithms

Table 4: CIFAR10 results of FedSDD with different local training settings.  |  | CIFAR10 (8 clients per round) | CIFAR10 (20 clients per round)
---|---|---|---
Model | Method | $\alpha=1.0$ | $\alpha=0.1$ | $\alpha=1.0$ | $\alpha=0.1$
ResNet20 | FedAvg | $88.53\pm 0.31$ | $78.72\pm 2.31$ | $88.59\pm 0.28$ | $79.84\pm 1.50$
| FedProx | $88.36\pm 0.18$ | $79.44\pm 2.17$ | $88.52\pm 0.07$ | $79.63\pm 2.48$
| SCAFFOLD | $89.85\pm 0.50$ | $80.08\pm 1.53$ | $90.10\pm 0.07$ | $82.83\pm 2.40$
| FedSDD w/ FedAvg | $89.06\pm 0.19$ | $80.18\pm 2.38$ | $89.04\pm 0.11$ | $82.89\pm 1.30$
| FedSDD w/ FedProx | $88.98\pm 0.22$ | $81.21\pm 1.30$ | $89.28\pm 0.17$ | $82.12\pm 2.26$
| FedSDD w/ SCAFFOLD | $88.80\pm 0.16$ | $76.51\pm 3.74$ | $90.66\pm 0.04$ | $84.30\pm 1.88$

As mentioned in section 3.1.1, the local training and model aggregation step
of FedSDD can be replaced by many different existing methods. We evaluate the
compatibility of FedSDD with different choices of local training scheme by
combining FedSDD with FedAvg [20] (default option in this paper), FedProx
[18], and SCAFFOLD [14]. The results in Table 4 (left, with 8 clients) show
that FedSDD is compatible with FedProx and the combination achieves better
performance with high degree of data heterogeneity.

However, the combination of FedSDD and SCAFFOLD did not improve the
performance under the concerned setting. We conjectured this may be due to the
fact that the number of clients per global model is only 2 in FedSDD which is
too small for SCAFFOLD. We thus performed additional experiments with 20
clients per round, i.e. the number of clients per global models became 5, as
shown in Table 4 (right). With these settings, the combination of FedSDD and
SCAFFOLD achieved much higher accuracy than the stand-alone FedSDD and
SCAFFOLD. The above results suggest that FedSDD is flexible with the
implementation of the local training scheme.

###  5.2 Comparison between different ways of building the ensemble

Table 5: CIFAR10 results of different ensemble settings. |  | CIFAR10
---|---|---
Model | Method | $\alpha=1.0$ | $\alpha=0.1$
ResNet20 |  Global model ($K=1$) | $88.53\pm 0.31$ | $78.72\pm 2.31$
|  Ensemble ($K=1$) Clients) | $88.82\pm 0.21$ | $80.22\pm 0.88$
|  Ensemble ($K=1$, Bayesian, Gaussian) | $88.72\pm 0.14$ | $79.60\pm 1.26$
|  Ensemble ($K=1$, Bayesian, Dirichlet) | $88.70\pm 0.11$ | $80.25\pm 1.43$
|  Global model ($K=4$) | $86.69\pm 0.54$ | $70.74\pm 5.11$
|  Ensemble ($K=4$, Clients) | $90.49\pm 0.17$ | $82.12\pm 1.65$
|  Ensemble ($K=4,R=1$, Aggregated) | $90.58\pm 0.31$ | $81.76\pm 2.40$
|  Ensemble ($K=4,R=2$, Aggregated) | $90.75\pm 0.24$ | $83.50\pm 1.56$
|  Ensemble ($K=4,R=4$, Aggregated) | $90.69\pm 0.19$ | $83.99\pm 0.88$

We performed experiments to compare different ways of building the ensemble
from global models, including the methods that use client models [19, 4] and
the proposed method which utilizes multiple global models. Following the
notations of FedSDD, we use $K$ to denote the number of global models and $R$
to represent the number of checkpoints in building the ensemble. In this
experiment, we set $K\in\\{1,4\\}$ and $R\in\\{1,2,4\\}$. Note that, to avoid
the possible negative impact to the ensembles, we did not use distillation in
this experiment.

We considered two existing methods that used only one global model ($K=1$):
(i). Following FedDF [19], we utilized all client models to build the ensemble
and only trained one global model; (ii). Following FedBE [4], we trained a
global model and built the ensemble by combining the client models, the
averaged model, and $10$ models sampled from the Gaussian distribution or the
Dirichlet distribution.

Then, we considered two strategies which utilize multiple global models
($K>1$): (iii). Similar to (i), we utilized client models (initialized from
different global models) to build the ensemble. (iv). Similar to FedSDD, we
built the ensemble from multiple checkpoints of global models, which is also
the case for Fed-ensemble when $R=1$.

The results are shown in Table 5, where we also provide the performance of the
standalone global model for reference purposes. With low degree of
heterogeneity ($\alpha=1.0$), despite slowing down the convergence of
individual global models, a larger number of global models offers better
results. All ensembles with multiple global models performed closely, where
the ensembles built by global models from multiple checkpoints achieved
slightly higher accuracy. With $K=1$, all methods did not show clear
performance advantage over the global model, which agrees with our
understanding about the diversity among the compositing models.

With highly Non-IID data, all ensemble strategies showed significant
improvement over the baseline global model with $K=1$. However, strategies
with multiple global models still outperformed the methods with single global
model, and the best results were obtained by strategy (iv) with $R=4$. It
indicates that direct access to client models is not necessary to build a high
performance ensemble.

In the above experiments, ensembles built by the multiple checkpoints of the
global models achieved the best performance, which is the strategy being used
in FedSDD.

###  5.3 Comparison between different ways of distillation

Table 6: CIFAR10 results of different distillation settings. |  | CIFAR10
---|---|---
Model | Method | $\alpha=1.0$ | $\alpha=0.1$
ResNet20 | w/o distillation | $86.69\pm 0.54$ | $70.74\pm 5.11$
| Basic distillation | $88.63\pm 0.34$ | $79.98\pm 2.42$
| Basic distillation w/ warm-up (20 rounds) | $88.66\pm 0.22$ | $80.20\pm 1.81$
| Basic distillation w/ warm-up (40 rounds) | $88.47\pm 0.04$ | $79.27\pm 2.14$
| Diversity preserved distillation | $89.06\pm 0.19$ | $80.18\pm 2.38$
ResNet20 (Ensemble) | w/o distillation | $90.58\pm 0.31$ | $81.76\pm 2.40$
| Basic distillation | $89.23\pm 0.36$ | $80.57\pm 2.24$
| Basic distillation w/ warm-up (20 rounds) | $89.38\pm 0.42$ | $80.80\pm 1.95$
| Basic distillation w/ warm-up (40 rounds) | $89.19\pm 0.24$ | $80.32\pm 2.55$
| Diversity preserving distillation | $90.31\pm 0.20$ | $81.39\pm 2.54$

In FedSDD, distillation is performed to enhance only the main global model,
which enhances the diversity. To evaluate the effectiveness of the proposed
method, we performed comparison between different ways of distillation: (i). A
basic KD scheme, which utilizes the ensemble as the teacher model and all
global models are trained by mimicking the output of the ensemble. Note that
we fixed the weights of the ensemble during the distillation process. This is
similar to the heterogeneous version of FedDF [19], where all global models
are improved by KD; (ii). A scheme following Codistillation [1], which skips
the distillation in the early rounds of training, such that the models will
not learn from each other at the beginning to maintain potentially higher
diversity. We set the number of warm-up epochs to 20/40; and (iii). The
proposed diversity-enhanced distillation, where only the main global model is
enhanced by KD.

Results in Table 6 show that, the global model trained by the proposed scheme
outperformed its counterparts with low degree of data heterogeneity, and
performed similarly as the others with high degree of data heterogeneity. In
fact, the ensemble trained by the proposed scheme obtained the best result for
all the cases, which is close to the ensemble without distillation. These
observations support our expectation that diversity enhancing schemes can
improve the performance of both the ensemble and the distilled global model.

##  6 Conclusion

In this paper, we proposed FedSSD, a distillation-based federated learning
algorithm to tackle the limitation of existing KD-based FL algorithms for
model aggregation. The proposed scheme is highly scalable as it utilizes
multiple global models, instead of all client models, for building the
ensemble. The ensemble in FedSSD has a better performance due to the
diversity-enhancing KD scheme, which in turn improves the performance of the
global model through KD. Furthermore, the main modules of FedSSD are
compatible with different FL algorithms, whose adaptation can further improve
the performance of FL systems.

## References

  * [1] R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton.  Large scale distributed neural network training through online distillation.  arXiv preprint arXiv:1804.03235, 2018.
  * [2] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth.  Practical secure aggregation for privacy-preserving machine learning.  In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 1175–1191, 2017.
  * [3] D. Chen, J.-P. Mei, C. Wang, Y. Feng, and C. Chen.  Online knowledge distillation with diverse peers.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3430–3437, 2020.
  * [4] H.-Y. Chen and W.-L. Chao.  Fedbe: Making bayesian model ensemble applicable to federated learning.  In ICLR, 2021.
  * [5] Y. J. Cho, A. Manoel, G. Joshi, R. Sim, and D. Dimitriadis.  Heterogeneous ensemble knowledge transfer for training large models in federated learning.  arXiv preprint arXiv:2204.12703, 2022.
  * [6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [7] X. Gong, A. Sharma, S. Karanam, Z. Wu, T. Chen, D. Doermann, and A. Innanje.  Ensemble attention distillation for privacy-preserving federated learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15076–15086, 2021.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [9] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  In NIPS Deep Learning and Representation Learning Workshop, 2015.
  * [10] T.-M. H. Hsu, H. Qi, and M. Brown.  Measuring the effects of non-identical data distribution for federated visual classification.  arXiv preprint arXiv:1909.06335, 2019.
  * [11] G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger.  Snapshot ensembles: Train 1, get m for free.  In International Conference on Learning Representations.
  * [12] W. Huang, M. Ye, and B. Du.  Learn from others and be yourself in heterogeneous federated learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10143–10153, 2022.
  * [13] E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim.  Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data.  arXiv preprint arXiv:1811.11479, 2018.
  * [14] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh.  Scaffold: Stochastic controlled averaging for federated learning.  In International Conference on Machine Learning, pages 5132–5143. PMLR, 2020.
  * [15] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [16] A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [17] S. Laine and T. Aila.  Temporal ensembling for semi-supervised learning.  In International Conference on Learning Representations.
  * [18] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith.  Federated optimization in heterogeneous networks.  Proceedings of Machine Learning and Systems, 2:429–450, 2020.
  * [19] T. Lin, L. Kong, S. U. Stich, and M. Jaggi.  Ensemble distillation for robust model fusion in federated learning.  Advances in Neural Information Processing Systems, 33:2351–2363, 2020.
  * [20] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  * [21] N. Shi, F. Lai, R. A. Kontar, and M. Chowdhury.  Fed-ensemble: Improving generalization through model ensembling in federated learning.  arXiv preprint arXiv:2107.10663, 2021.
  * [22] J. Wang, Z. Charles, Z. Xu, G. Joshi, H. B. McMahan, M. Al-Shedivat, G. Andrew, S. Avestimehr, K. Daly, D. Data, et al.  A field guide to federated optimization.  arXiv preprint arXiv:2107.06917, 2021.
  * [23] D. Yao, W. Pan, Y. Dai, Y. Wan, X. Ding, H. Jin, Z. Xu, and L. Sun.  Local-global knowledge distillation in heterogeneous federated learning with non-iid data.  arXiv preprint arXiv:2107.00051, 2021.
  * [24] S. Zagoruyko and N. Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.
  * [25] L. Zhang, L. Shen, L. Ding, D. Tao, and L.-Y. Duan.  Fine-tuning global model via data-free knowledge distillation for non-iid federated learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10174–10183, 2022.

##  Appendix A Appendix

###  A.1 Experiment with different communication intervals

Table 7: CIFAR10 results with different communication intervals |  |  | CIFAR10
---|---|---|---
# rounds / # local epochs | Model | Method | $\alpha=1.0$ | $\alpha=0.1$
25 / 160 | ResNet20 | FedAvg | $85.81\pm 0.12$ | $73.17\pm 3.66$
|  | FedDF | $86.78\pm 0.12$ | $77.79\pm 1.73$
|  | FedSDD | $86.27\pm 0.68$ | $73.99\pm 2.15$
100 / 40 | ResNet20 | FedAvg | $88.53\pm 0.31$ | $78.72\pm 2.37$
|  | FedDF | $87.98\pm 0.16$ | $80.04\pm 1.87$
|  | FedSDD | $89.06\pm 0.19$ | $80.18\pm 2.38$
400 / 10 | ResNet20 | FedAvg | $89.69\pm 0.27$ | $79.19\pm 0.42$
|  | FedDF | $88.90\pm 0.19$ | $78.92\pm 0.59$
|  | FedSDD | $90.37\pm 0.08$ | $81.34\pm 0.81$

We compared the performance of FedSDD with FedAvg [20] and FedDF [19], with
different communication intervals. To fix the total computation workload, we
set the number of rounds and the number of local epochs per round as
25/100/400 and 160/40/10, respectively. We also scaled the number of
distillation steps to 20000/5000/1250. From Table 7, it can be observed that
FedSDD performed better than FedAvg in all the cases, and achieved better
results than FedDF in most of the cases. But, with 25 communication rounds,
FedDF outperformed FedSDD. We expect this is due to the extremely small number
of clients per global models in FedSDD, i.e. two clients per global models,
which slowed down the convergence when the communication frequency is too low.

###  A.2 Experiment with different number of global models

Table 8: CIFAR10 results of FedSDD with different number of global models. |  |  | CIFAR10
---|---|---|---
Model | Method | $K$ | $\alpha=1.0$ | $\alpha=0.1$
ResNet20 | FedSDD | $2$ | $88.34\pm 0.32$ | $79.38\pm 2.54$
|  | $3$ | $88.89\pm 0.22$ | $79.63\pm 6.15$
|  | $4$ | $89.06\pm 0.19$ | $80.18\pm 2.38$

In the main paper, we used $K=4$ as the default setting of FedSDD. Here, we
provide results with $K=2/3/4$. Given there are only 8 clients per round in
the default configuration, there are at least two clients per global model,
and all global models are aggregated models. For $K=3$, we allocated one more
client to the main global model in FedSDD. The results in Table 8 show that
$K=4$ is the best configuration among the tested cases. However, the optimal
value of $K$ may vary with different tasks.

###  A.3 Experiment with different number of clients

Table 9: CIFAR10 results of FedSDD with different scaling schemes. |  |  |  | CIFAR10
---|---|---|---|---
# Clients | Model | Method | $K$ | $\alpha=1.0$ | $\alpha=0.1$
8 | ResNet20 | FedAvg | $-$ | $88.53\pm 0.31$ | $78.72\pm 2.37$
|  | FedDF | $-$ | $87.98\pm 0.16$ | $80.04\pm 1.87$
|  | FedSDD | $4$ | $89.06\pm 0.19$ | $80.18\pm 2.38$
14 | ResNet20 | FedAvg | $-$ | $88.67\pm 0.22$ | $80.25\pm 1.64$
|  | FedDF | $-$ | $88.38\pm 0.16$ | $81.61\pm 1.94$
|  | FedSDD | $4$ | $89.14\pm 0.09$ | $81.26\pm 1.99$
|  |  | $7$ | $89.31\pm 0.06$ | $81.90\pm 1.80$
20 | ResNet20 | FedAvg | $-$ | $88.59\pm 0.28$ | $79.84\pm 1.50$
|  | FedDF | $-$ | $88.12\pm 0.25$ | $81.55\pm 2.02$
|  | FedSDD | $4$ | $89.04\pm 0.11$ | $82.89\pm 1.30$
|  |  | $10$ | $89.14\pm 0.11$ | $81.97\pm 1.54$
Table 10: The difference between FedDF and FedSDD. The method in the last row is identical to the FedSDD with $R=1$. |  | CIFAR10
---|---|---
Model | Method | $\alpha=1.0$ | $\alpha=0.1$
ResNet20 | FedDF | $82.53\pm 0.24$ | $68.72\pm 4.23$
| \+ Improved training configuration | $87.93\pm 0.47$ | $79.01\pm 2.15$
| \+ Removal of drop-worst & early-stopping | $87.42\pm 0.19$ | $78.55\pm 3.00$
|  \+ Aggregated ensemble + $K=4$ \+ diversity-enhanced distillation | $88.35\pm 0.18$ | $79.59\pm 2.93$

In Table 9, we show how the performance of FedSDD scales with the number of
clients, and compared the result with that of FedAvg [20] and FedDF [19]. As
the number of clients increases, we considered two options: 1.) Scaling the
number of clients per global model, i.e. fixing the value of $K$; and 2.)
Scaling the number of global models, i.e. fixing the number of clients per
group ($K=7$ and $10$ in Table 9). We found that FedSDD outperformed FedAvg
and FedDF with both scaling schemes, and obtained a better result when the
number of global models increases, except the case with high degree of data
heterogeneity and 20 clients. In practice, the scaling scheme is subject to
the server’s capacity, as increasing the number of global models will also
increase the cost of distillation.

###  A.4 Improvement from FedDF

Both FedSDD and FedDF [19] utilize KD for model aggretation with an unlabelled
dataset. In this section, we progressively modify FedDF to approach FedSDD,
and show the effect of each modification. Note that, following [19], we keep
10% of the training data from CIFAR10 as the validation set of the server in
this experiment.

The original FedDF utilized Adam optimizer [15] for server training, and
applied the ‘drop-worst’ mechanism and the early-stopping. We performed the
following modifications: 1) Adopt the training configuration from the main
paper (including the choice of learning rate, the use of SGD optimizer for
distillation, etc.); 2) Remove the ‘drop-worst’ and early-stopping operations;
and 3) Utilize four aggregated global models to build the ensemble with the
proposed distillation scheme, which enhances the model diversity. By
introducing the three modifications, the modified FedDF is effectively the
same as FedSDD (without temporal ensembling).

Table 10 shows the results of different modifications, from which we have
three observations. First, the configuration we used in this paper improved
FedDF. Second, the removal of the ‘drop-worst’ and early-stopping operations
led to a worse result, but it saved the need for the validation set at the
server. Third, the main mechanisms we used in FedSDD, i.e. the ensemble from
aggregated models and the proposed distillation scheme, provided significant
performance improvement.

###  A.5 Possible extensions of FedSDD

To perform distillation, a server dataset or generator is mandatory. In this
paper, we consider the use of FedSDD with unlabelled server dataset, following
[19, 4]. Since the dataset is not labelled and can be independent of the
client datasets, it is cheap to collect. Note that extending FedSDD to include
labeled dataset or generator is straightforward. Existing works [19, 25] have
utilized generator for knowledge distillation. We can also extend FedSDD to
the setting with heterogeneous models [19, 5], where we can train multiple
global models for each model type, and select one main global model for each
model type. As long as only global models are utilized to form the ensemble,
and KD is only performed for enhancing a subset of the global models, the
scalability of the algorithm and the diversity among the global models can be
preserved.

###  A.6 Parallel server-side and client-side training of FedSDD

In practical federated learning systems, clients are not always available for
training. When there are limited number of available clients, the parallelism
introduced in FedSSD can significantly reduce the round time by performing the
server-side KD and the client-side local training simultaneously. In
particular, the training of the global model $k$, with $k>0$, i.e. except the
main global model, in the $t$-th round can start, if its $t-1$-th round
training is completed. This independence can also be observed from Figure 1 in
the main paper, which shows that the operation in each group only depends on
the previous process within the same group.

In Figure 2 of this supplementary material, we demonstrate one scheduling
example for FedSDD and compare it with other distillation based methods, e.g.
FedDF [19], to illustrate the benefits of parallelism. In this example, we
assume there are four clients, and only one is available at any given time.
For the sake of simplicity, we further assume there are four global models for
FedSSD and the main global model is always assigned to the first client. Under
such circumstances, existing methods need to perform KD and local training
sequentially, while FedSSD can perform them simultaneously to reduce the round
time.

(a) Existing Methods (e.g. FedDF)

(b) FedSDD

Figure 2: An example for the training scheduling comparison between FedSDD and
other methods. When clients are not always available, the parallelism
introduced in FedSSD can reduce the round time.