# Topological chiral kagome lattice

Jing-Yang You  Department of Physics, National University of Singapore, 2
Science Drive 3, Singapore 117551  Xiaoting Zhou<EMAIL_ADDRESS>Department of Physics, Northeastern University, Boston, MA, 02115, USA  Tao
Hou  Division of Physics and Applied Physics, School of Physical and
Mathematical Sciences, Nanyang Technological University, Singapore 637371,
Singapore  Mohammad Yahyavi  Division of Physics and Applied Physics, School
of Physical and Mathematical Sciences, Nanyang Technological University,
Singapore 637371, Singapore  Yuanjun Jin  Division of Physics and Applied
Physics, School of Physical and Mathematical Sciences, Nanyang Technological
University, Singapore 637371, Singapore  Yi-Chun Hung  Department of Physics,
Northeastern University, Boston, MA, 02115, USA  Institute of Physics,
Academia Sinica, Taipei 115229, Taiwan  Bahadur Singh  Department of Condensed
Matter Physics and Materials Science, Tata Institute of Fundamental Research,
Mumbai, India  Chun Zhang  Department of Physics, National University of
Singapore, 2 Science Drive 3, Singapore 117551  Jia-Xin Yin  Department of
Physics, Southern University of Science and Technology, Shenzhen, Guangdong
518055, China  Arun Bansil  Department of Physics, Northeastern University,
Boston, MA, 02115, USA  Guoqing Chang<EMAIL_ADDRESS>Division of
Physics and Applied Physics, School of Physical and Mathematical Sciences,
Nanyang Technological University, Singapore 637371, Singapore

###### Abstract

Chirality, a fundamental structural property of crystals, can induce many
unique topological quantum phenomena. In kagome lattice, unconventional
transports have been reported under tantalizing chiral charge order. Here, we
show how by deforming the kagome lattice to obtain a three-dimensional (3D)
chiral kagome lattice in which the key band features of the non-chiral 2D
kagome lattice – flat energy bands, van Hove singularities (VHSs), and
degeneracies – remain robust in both the $k_{z}=0$ and $\pi$ planes in
momentum space. Given the handedness of our kagome lattice, degenerate
momentum points possess quantized Chern numbers, ushering in the realization
of Weyl fermions. Our 3D chiral kagome lattice surprisingly exhibits 1D
behavior on its surface, where topological surface Fermi arc states connecting
Weyl fermions are dispersive in one momentum direction and flat in the other
direction. These 1D Fermi arcs open up unique possibilities for generating
unconventional non-local transport phenomena at the interfaces of domains with
different handedness, and the associated enhanced conductance as the
separation of the leads on the surface is increased. Employing first-
principles calculations, we investigate in-depth the electronic and phononic
structures of representative materials within the ten space groups that can
support topological chiral kagome lattices. Our study opens a new research
direction that integrates the advantages of structural chirality with those of
a kagome lattice and thus provides a new materials platform for exploring
unique aspects of correlated topological physics in chiral lattices.

In the past decade, there has been a surge of research interest in topological
materials characterized by non-trivial geometric phases Hasan and Kane (2010);
Yu  _et al._ (2010); Fu  _et al._ (2007); Chang  _et al._ (2013); Qi and Zhang
(2011); Burkov (2016); Bansil  _et al._ (2016); Zhong  _et al._ (2016);
Bradlyn  _et al._ (2017); Schindler  _et al._ (2018); Wen (2017); Armitage
_et al._ (2018); Lv  _et al._ (2021); Hasan  _et al._ (2021). Another parallel
area of focus has revolved around correlated systems with strong interactions
Lee  _et al._ (2006); Zhou  _et al._ (2021); Pesin and Balents (2010); Imada
_et al._ (1998); Tokura  _et al._ (1993); Sachdev (1999). The convergence of
these two fields presents a cutting-edge frontier for unraveling the study of
interacting topological phases of quantum matter Yankowitz  _et al._ (2019);
Kerelsky  _et al._ (2019); Lisi  _et al._ (2020); Cao  _et al._ (2020); Serlin
_et al._ (2020); Choi  _et al._ (2021); Chen  _et al._ (2021); Tan  _et al._
(2021); Jiang  _et al._ (2021); Yang  _et al._ (2020); Yu  _et al._ (2021);
Teng  _et al._ (2022); Yin  _et al._ (2022a); Teng  _et al._ (2023); Shao  _et
al._ (2023); Zhou  _et al._ (2023). However, a challenge in advancing this
field is that electron-electron correlations are generally negligible in many
well-known topological materials. One effective strategy for enhancing
electron-electron correlations involves introducing a high density of states
in the vicinity of the Fermi level. Here the kagome lattice stands out as
particularly promising candidate due to its unique electronic structures that
features flat bands and van Hove singularities (VHSs). Although the initial
focus was on two-dimensional (2D) lattices, recent work considers three-
dimensional (3D) kagome systems in which the kagome layers coexist with other
atomic layers in the unit cell Ye  _et al._ (2018); Liu  _et al._ (2018); Yin
_et al._ (2019); Ortiz  _et al._ (2019); Yin  _et al._ (2022b); Yang  _et al._
(2023). The focus has been a scenario in which the coupling along the stacking
direction remains sufficiently weak so that the flat bands and VHSs of the
kagome layers are preserved. Unfortunately, the complex interlayer couplings
in these systems often lead to the disappearance of the distinctive kagome
band characteristics.

The high symmetry of the kagome lattice also limits its potential
applications. Recall that the breaking of symmetries has dramatically
contributed to the emergence of numerous topological phases of quantum matter.
For instance, chiral crystals that break inversion and mirror symmetries have
been found to exhibit unique topological properties, including multi-fold
chiral fermions and related quantized nonlinear optical responses de Juan  _et
al._ (2017); Chang  _et al._ (2017); Tang  _et al._ (2017); Chang  _et al._
(2018); Sanchez  _et al._ (2019); Rao  _et al._ (2019); Takane  _et al._
(2019); Yang  _et al._ (2019); Schröter  _et al._ (2020). Kagome KV3Sb5, with
chiral charge order at low temperatures, has shown promise in realizing large
orbital magnetization and unconventional superconductivity Jiang  _et al._
(2021); Luo  _et al._ (2022); Li  _et al._ (2022a). However, prior to inducing
strong correlation effects, KV3Sb5 exhibits non-chiral behavior at high
temperatures. Novel correlated topological phenomena can thus be expected to
arise from symmetry breaking in chiral crystals Herbut and Janssen (2014);
Jian and Yao (2017); Li  _et al._ (2022b); Chiu  _et al._ (2023); Sanchez  _et
al._ (2023); Rao  _et al._ (2023).

Figure 1: (a) A 2D kagome lattice with nearest-neighboring hopping $t$.
Symbols ’+’ and ’-’ signify wave functions with equal amplitude but opposite
phases, giving rise to compact localized states. Dashed arrows indicate
canceled hoppings. (b) A 3D chiral kagome lattice, where its projection on the
(001) plane is a 2D kagome. (c) Electronic band structure along the high-
symmetry paths in the Brillouin zone, and the corresponding density of states
(DOS). (d) $\mathbf{q}$-dependent static charge-density susceptibility
$\chi_{0}(\mathbf{q})$ at low-temperature $T=0.00001t$ with an enhanced
amplitude at the CDW wave vectors $\mathbf{Q}$ on the (001) plane with
($k_{1}$, $k_{2}$, $k_{3}$) = (0, $\pi$, 0); ($\pi$, 0, 0); (-$\pi$, $\pi$, 0)
(upper panel) and on the (010) plane with (0, 0, $\pi$) (bottom panel).

The 2D kagome lattice comprises three atoms in the primitive cell represented
by red, green, and blue spheres in Fig. 1(a). The band structure of a single-
orbital kagome model with only the nearest-neighboring (NN) hoppings exhibits
a flat band, VHSs, and Dirac degeneracies, where one branch of the dispersive
Dirac bands intersects the flat band at the $\Gamma$ point Yin  _et al._
(2019). This is attributed to the wave function localization that forms
compact localized states within the hexagon due to destructive interference.
To extend this model to 3D, we can maintain the NN hopping but translate two
atoms in 2D kagome lattice along the $z$ axis (out-of-plane direction) by
$c$/3 and 2$c$/3, respectively, as depicted in Fig. 1(b). In this
configuration, it can be inferred that the system should preserve the
characteristic kagome band features.

For the 2D kagome lattice, the green and blue atoms can be linked through
mirror symmetry ($\mathcal{M}$). However, this symmetry is naturally broken
when translating green and blue atoms along the $z$ axis at varying distances.
Furthermore, unlike the 2D kagome lattice with inversion symmetry, the
inversion centers for the three atoms in the 3D kagome lattice are not at the
same height, thus resulting in inversion-symmetry-breaking. The 3D kagome
lattice lacks both mirror and center inversion symmetries, classifying it as a
chiral lattice possessing the screw symmetry $S_{6/3}$ with the combination of
a six/three-fold rotation $C_{6/3}$ and a one-third fractional translation
along the $z$ axis.

We now undertake an analytical verification to ascertain whether the kagome
band characteristics remain preserved within the new lattice by constructing
the Hamiltonian including only the NN hopping:

$H_{k}\\!\\!=\\!-2t\\!\\!\left(\begin{array}[]{ccc}\\!d/t&\\!{\rm
cos}(\frac{k_{2}}{2})e^{-\frac{ik_{3}}{3}}&\\!{\rm
cos}(\frac{k_{1}+k_{2}}{2})e^{\frac{ik_{3}}{3}}\\\ \\!{\rm
cos}(\frac{k_{2}}{2})e^{\frac{ik_{3}}{3}}&\\!d/t&\\!{\rm
cos}(\frac{k_{1}}{2})e^{-\frac{ik_{3}}{3}}\\\ \\!{\rm
cos}(\frac{k_{1}+k_{2}}{2})e^{-\frac{ik_{3}}{3}}&\\!{\rm
cos}(\frac{k_{1}}{2})e^{\frac{ik_{3}}{3}}&\\!d/t\end{array}\right),$ (1)

where $\mathbf{a_{1}}=a(1,0,0)$, $\mathbf{a_{2}}=a(-1/2,\sqrt{3}/2,0)$,
$\mathbf{a_{3}}=c(0,0,1)$, and $k_{i}=\mathbf{k}\cdot\mathbf{a_{i}}$. Figure
1(c) presents the electronic band structure of the new lattice described by
Eq. (1) with the parameters $d=0$ and $t=0.1$. Indeed, as expected, the kagome
bands are faithfully reproduced, including the flat band, VHSs, and the band
degeneracies at $k_{z}(k_{3})=0$ and $\pi$. We also note that the energy bands
in $k_{z}$= 0 and $\pi$ planes are reversed by a $\pi$ phase difference
accumulated by the $z$-direction translation. Along $k_{z}$=0 to $\pi$,
ladder-shaped bands connect the two kagome sets. We further calculate the
density of states (DOS) for the system, similar to the 2D kagome system,
revealing very sharp divergent peaks at the flat band and VHSs. This indicates
potential strong electron-electron correlations at the energy of VHSs and flat
bands.

At the energy of VHSs, we compute the Lindhard function on the (001) and (010)
planes, where exceptionally high-density electronic susceptibilities nested by
these VHSs are observed [Fig. 1(d)]. Within the (001) plane, the VHS at $M(L)$
favors particle-hole scatterings linked by the nesting vectors $\mathbf{Q}$,
characterized by (0, $\pi$, 0), ($\pi$, 0, 0) and (-$\pi$, $\pi$, 0). Akin to
2D kagome materials, the enhanced susceptibility in the chiral kagome lattice
can also prompt the emergence of an in-plane correlated charge density wave
(CDW). In addition, because of the existence of a third dimension, on the
(010) plane, the nearest VHSs at $M$ and $L$ in the $k_{z}$ direction can be
further associated with the vector $\mathbf{Q}$(0, 0, $\pi$), suggesting a
potential CDW along the $z$ direction.

We note that our formulation is applicable not only to electronic structures
but it can also handle phononic lattices, see Supplemental Materials (SM) and
Fig. S1 for details. Our results suggest the presence of strong electron-
electron and electron-phonon interactions in the chiral kagome lattice.

Figure 2: (a) Right-handed Crystal structure, (b) 3D energy dispersion of the
degeneracy at the $\Gamma$ point, along with Berry curvature $\Omega_{x}$
around the $\Gamma$ points on the (001) plane with flow direction indicated,
and (c) distribution of Weyl points in the Brillouin zone (BZ), where colored
spheres with different sizes label Weyl points with different topological
charges, and associated projected surface BZs for the 3D chiral kagome lattice
with right-handedness. In (a), the upward arrow illustrates the evolution
process of atoms moving from the plane along the $c$ direction, while the
spiral arrow projected onto the bottom surface indicates chirality. (d) Right-
handed Crystal structure, and its (e) Berry curvature $\Omega_{x}$ around the
$\Gamma$ and $A$ points, and (f) distribution of Weyl points for the 3D chiral
kagome lattice with left-handedness.

Based on the preceding considerations, we have constructed a 3D chiral kagome
lattice. To discuss topological effects arising from symmetry breaking, we
first consider a right-handed crystal [Fig. 2(a)], where the atoms follow an
anti-clockwise rotation with respect to the $z$ axis following the right-hand
rule. By analyzing the Berry curvature of the bands, we observe that the two-
fold degeneracies at the $\Gamma$, $A$, $K$, and $H$ points possess quantized
Chern numbers. Specifically, in the right-handed kagome lattice, a Chern
number $C=2$ is ascribed to the $\Gamma$ point, serving as a source of Berry
curvature [Fig. 2(b)]. Each of the two equivalent $H$ points carries a Chern
number $C=-1$ to compensate for the positive chiral charge at the $\Gamma$
point [Fig. 2(c)]. Similarly, the right-handed 3D kagome lattice exhibits a
$C=-2$ at the $A$ point and $C=1$ at the $K$ point [Fig. 2(c)]. Furthermore,
we observe that one of the branches of the double Weyl cone at both $\Gamma$
and $A$ points comprises flat bands [Fig. 2(b), top panel].

Applying a vertical mirror operation to the right-handed 3D kagome lattice
[Fig. 2(a)] results in the lattice with left-handedness [Fig. 2(d)], wherein
the atoms rotate clockwise along the $z$ axis. Interestingly, the electronic
dispersions of the left-handed crystal exactly mirror those of the right-
handed counterpart, with both structures representing ground states. However,
a significant distinction arises in the flow of Berry curvature in momentum
space, which undergoes a complete reversal due to the change in structural
chirality in real space [Figs. 2(b),(e)]. This reversal, in turn, affects the
sign of the Chern numbers associated with the massless Weyl fermions [Figs.
2(c),(f)]. Our analysis demonstrates that the topological Chern number in the
kagome lattice is inherently dictated by its structural chirality.
Consequently, by tuning the handedness of the lattice, one gains the ability
to manipulate the Chern numbers and thus the related topological quantum
responses, including the direction of photocurrents de Juan  _et al._ (2017).

We now turn to discuss the unique topological surface responses of our 3D
chiral kagome lattice. Typically, the surfaces of a 3D material exhibit 2D
electronic structures. However, intriguingly, despite the inherent 3D nature
of our chiral kagome lattice, its surfaces manifest distinct one-dimensional
(1D) features. This is evident on the (001) surface of the lattice, wherein
conducting electrons from the top layer are bound to the inner layer through
NN hopping along a 1D chain [Fig. 3(a)]. Consequently, the emergence of 1D
Fermi arc surface states is anticipated. Indeed, on the constant energy
contours of the (001) surface, we observe the presence of two 1D Fermi arc
surface states [Fig. 3(b), top panel]. This distinctive 1D nature of Fermi
arcs also extends to other terminations, such as the (010) surface, signifying
the general occurrence of 1D surface states in the 3D chiral kagome lattice
[Fig. 3(b), bottom panel]. A detailed analysis of the energy dispersion of the
Fermi arc surface states reveals that they exhibit dispersion in only one
direction [Fig. 3(c), top panel], while remaining completely flat in the
perpendicular direction [Fig. 3(c), bottom panel]. The surface states on the
(001) plane also display dispersion in one direction while remaining flat in
the orthogonal direction [Fig. S2]. This further substantiates the 1D nature
of these topological Fermi arcs in our 3D chiral kagome lattice.

The presence of 1D surface states characterized by parallel Fermi arcs
suggests a pronounced Fermi nesting effect, hinting at potential strong
correlation effects on the surface of the chiral kagome lattice. It is
essential to recognize that while both the bulk and surface states can exhibit
strong correlations, the underlying mechanisms driving these effects are
fundamentally distinct. The correlation of bulk states originates from VHSs
and flat bands, whereas for surface states, it arises from the specific 1D
configurations on the surface. As a consequence, it is plausible that two
independent correlation effects are induced, operating separately in the bulk
and on the surface of the chiral kagome lattice. For instance, considering the
CDW phenomenon as an illustration, the VHSs in bulk may lead to a
2$\times$2$\times$1 or 2$\times$2$\times$2 CDW order [Fig. 1(d)] Jiang  _et
al._ (2021); Teng  _et al._ (2022). In contrast, on the surface, the
$\mathbf{Q}$-vector characterizing the charge order could acquire a relatively
arbitrary value, which is determined by the separation of Fermi arcs Li  _et
al._ (2022b); Sanchez  _et al._ (2023); Rao  _et al._ (2023). The distinct
behaviors of bulk and surface correlations open new possibilities for
correlated topological effects on the chiral kagome lattice.

Figure 3: (a) Schematic diagram of 1D channels on the (001) surface, where the
green arrows represent the 1D channels. (b) The (001) and (010) surface state
spectral functions at 0.12 eV, where a path perpendicular to Fermi arcs
($k_{\rm path1}$) and a path parallel to Fermi arcs ($k_{\rm path2}$) are
highlighted in (b). (c) The surface spectral functions along the specific $k$
paths for the (010) surface. (d) The domain wall formed by different-handed
structures that are mirror symmetrical about the domain wall. (e) The current
transport at the interface (domain wall) on the (010) plane. Solid curves are
fermi arcs at 0.12 eV, while dashed curves are fermi arcs at slightly higher
energy. (f) Schematic diagram of non-local transport for finite size materials
and three-terminal device, where the red arrows indicate the propagation
process of surface current. Lead1 and Lead2 are connected to the (010) plane,
and Lead3 is connected to the (0$\bar{1}$0) plane. The red and blue regions
represent chiral opposite lattices. (g) and (h) The conductance of devices
with uniform chirality and opposite chirality (corresponding to (f)) in the
energy range of (010) surface states (around 0.125 eV), respectively.
$G_{2(3)1}$ is the conductance between Lead1 and Lead2(3).

In addition to the potential strong correlations induced on the surface, the
presence of 1D Fermi arcs in the 3D chiral kagome lattice offers opportunities
for realizing exotic nonlocal transport phenomena. Unlike its 2D counterpart
with inversion symmetry, the 3D chiral kagome lattice can readily form left-
and right-handed chiral domains that are mirror symmetric on either side of
the domain wall [Fig. 3(d)]. The connectivity of the Fermi arcs is also
reversed on the two sides of the domain wall, as shown by the (010) surface
states in Fig. 3(e). Consider the following example, when a current originates
from the Fermi arc of the left domain and moves toward the right domain [Fig.
3(e)], it encounters a unique situation. Due to the inherent absence of
reflection surface modes, this current cannot be reflected back. Additionally,
it cannot pass through the domain wall to reach the (010) surface of the right
domain, as the group velocity of states on the right domain opposes the
electron’s direction of motion. Consequently, the current can only be
transmitted along the interface of the domain wall, entering the right domain
from the surface on the other side [Fig. 3(f)]. The current transport on the
(001) surface can exhibit negative reflection and refraction as discussed in
Fig. S3.

In conventional diffusion electronics, transport adheres to Ohm’s law, where
the conductance decreases with increasing separation between two leads.
However, in our proposed scenario, we expect the conductance to offer novel
perspectives for designing materials with tailored transport properties and
potential applications in advanced electronic devices. Surprisingly, we
anticipate the conductance between Lead2 and Lead1 to be smaller than the
conductance between Lead3 and Lead1, even though Leads1 and 3 are more widely
separated from each other [Fig. 3(f)]. To verify this picture, we directly
calculate the conductance of two devices of the same size for comparison
[Figs. 3(g),(h)], one device has a uniform chirality, while the other consists
of two regions with opposite chirality [Fig. 3(d)]. In the 3D kagome device
with uniform chirality, the conductance $G_{21}$ is larger than $G_{31}$ [Fig.
3(g)], indicating that the incoming current from Lead1 tends to partition more
towards Lead2. This preference can be attributed to Lead2 being closer to
Lead1 compared to Lead3. In contrast, in the device consisting of two
chiralities, the conductance $G_{31}$ is larger than $G_{21}$ [Fig. 3(h)]. Our
calculations indeed reveal the emergence of unconventional nonlocal transport
effects in the 3D chiral kagome lattice because of the lattice’s chirality and
unique 1D Fermi arcs. These findings open up new avenues for engineering
materials with intriguing transport properties, suggesting the potential for
exciting applications in future electronic devices.

The most basic symmetry operation in the 3D chiral kagome lattice is the screw
symmetry $S_{6}$ (or $S_{3}$). Space groups (SGs) that satisfy this symmetry
include SGs 144, 145, 151-154, 171, 172, 180 and 181. By consulting the
Inorganic Crystal Structure Database (ICSD) Lev , we discover a large number
of experimentally synthesized materials that hold the 3D chiral kagome lattice
with the 3D kagome band characteristics [Fig. 1(c)] in their electronic or
phonon band structures, such as CePO4, TiC4H8NO10, BeF2, BaCoO2, BaZnO2, GeO2
and CuH20N6OF2, InH8C4NO10, SiO2 and KScH4(C2O5)2, and Li(BH)5 as shown in
Figs. S4 and S5. Among these known materials, we find various types, including
metals, insulators, nonmagnets, and ferromagnets. This diversity highlights
the potential of 3D chiral kagome materials for a wide range of applications
and investigations across different material types and properties.

Figure 4: (a) Crystal structure of CePO4 in SG 180, where Ce atoms form a 3D
chiral kagome lattice. (b) Electronic band structure and (c) phonon spectrum
of CePO4. (d) The (001) and (100) surface state spectral functions at -0.04
eV.

Finally, we focus on a representative material candidate for the 3D chiral
kagome lattice CePO4, where the 3D chiral kagome lattice is composed of Ce
atoms [Fig. 4(a)]. This material is a ferromagnetic semiconductor with a
magnetic moment of about 1 $\mu_{B}$ per Ce atom. Interestingly, only one spin
species (spin up) is distributed near the Fermi level, while the other one
remains far away from the Fermi level [Fig. 4(b)]. The three valence bands
near the Fermi level in CePO4 form the ideal 3D kagome bands, characterized by
flat bands with a bandwidth of less than 3 meV on $k_{z}=0$ and $\pi$ planes.
By introducing about 0.8 holes through doping, the Fermi level can be
effectively tuned to align with the flat band. This can be achieved, for
instance, by substituting part of the P atoms in CePO4 with Si or C atoms. The
phonon spectrum of CePO4 further adds to its unique properties, revealing
several groups of 3D chiral kagome bands in the frequency range of 10 to 18
THz [Fig. 4(c)]. It is worth noting that although our model only considers the
NN hopping, the robust nature of the flat bands for electrons and phonons in
real materials suggests that our model is highly compatible with the behavior
exhibited by the real materials. Additionally, the surface states of CePO4
exhibit 1D Fermi arcs [Fig. 4(d)], which is consistent with our model. This
distinctive combination of electronic and phonon properties in CePO4 positions
it as a promising candidate for exploring novel quantum phenomena and
potential applications in the realm of 3D chiral kagome materials.

## Acknowledgement

Work at Nanyang Technological University was supported by the National
Research Foundation, Singapore under its Fellowship Award (NRF-
NRFF13-2021-0010) and the Nanyang Assistant Professorship grant (NTUSUG). Work
at the National University of Singapore was supported by the Ministry of
Education, Singapore, under its MOE AcRF Tier 3 Award MOE2018-T3-1-002. The
work at Northeastern University was supported by the Air Force Office of
Scientific Research under award number FA9550-20-1-0322 and benefited from the
computational resources of Northeastern University’s Advanced Scientific
Computation Center (ASCC) and the Discovery Cluster. The work at TIFR Mumbai
was supported by the Department of Atomic Energy of the Government of India
under Project No. 12-R$\&$D-TFR-5.10-0100 and benefited from the computational
resources of TIFR Mumbai.

J-Y. Y., X. Z., and T. H. contributed equally to this work.

## References

  * Hasan and Kane (2010) M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010).
  * Yu  _et al._ (2010) R. Yu, W. Zhang, H.-J. Zhang, S.-C. Zhang, X. Dai, and Z. Fang, Science 329, 61 (2010).
  * Fu  _et al._ (2007) L. Fu, C. L. Kane, and E. J. Mele, Phys. Rev. Lett. 98, 106803 (2007).
  * Chang  _et al._ (2013) C.-Z. Chang, J. Zhang, X. Feng, J. Shen, Z. Zhang, M. Guo, K. Li, Y. Ou, P. Wei, L.-L. Wang, Z.-Q. Ji, Y. Feng, S. Ji, X. Chen, J. Jia, X. Dai, Z. Fang, S.-C. Zhang, K. He, Y. Wang, L. Lu, X.-C. Ma, and Q.-K. Xue, Science 340, 167 (2013).
  * Qi and Zhang (2011) X.-L. Qi and S.-C. Zhang, Rev. Mod. Phys. 83, 1057 (2011).
  * Burkov (2016) A. A. Burkov, Nat. Mater. 15, 1145 (2016).
  * Bansil  _et al._ (2016) A. Bansil, H. Lin, and T. Das, Rev. Mod. Phys. 88, 021004 (2016).
  * Zhong  _et al._ (2016) S. Zhong, J. E. Moore, and I. Souza, Phys. Rev. Lett. 116, 077201 (2016).
  * Bradlyn  _et al._ (2017) B. Bradlyn, L. Elcoro, J. Cano, M. G. Vergniory, Z. Wang, C. Felser, M. I. Aroyo, and B. A. Bernevig, Nature 547, 298 (2017).
  * Schindler  _et al._ (2018) F. Schindler, A. M. Cook, M. G. Vergniory, Z. Wang, S. S. P. Parkin, B. A. Bernevig, and T. Neupert, Sci. Adv. 4, eaat034 (2018).
  * Wen (2017) X.-G. Wen, Rev. Mod. Phys. 89, 041004 (2017).
  * Armitage  _et al._ (2018) N. P. Armitage, E. J. Mele, and A. Vishwanath, Rev. Mod. Phys. 90, 015001 (2018).
  * Lv  _et al._ (2021) B. Q. Lv, T. Qian, and H. Ding, Rev. Mod. Phys. 93, 025002 (2021).
  * Hasan  _et al._ (2021) M. Z. Hasan, G. Chang, I. Belopolski, G. Bian, S.-Y. Xu, and J.-X. Yin, Nat. Rev. Mater. 6, 784 (2021).
  * Lee  _et al._ (2006) P. A. Lee, N. Nagaosa, and X.-G. Wen, Rev. Mod. Phys. 78, 17 (2006).
  * Zhou  _et al._ (2021) X. Zhou, W.-S. Lee, M. Imada, N. Trivedi, P. Phillips, H.-Y. Kee, P. Törmä, and M. Eremets, Nature Reviews Physics 3, 462 (2021).
  * Pesin and Balents (2010) D. Pesin and L. Balents, Nature Physics 6, 376 (2010).
  * Imada  _et al._ (1998) M. Imada, A. Fujimori, and Y. Tokura, Rev. Mod. Phys. 70, 1039 (1998).
  * Tokura  _et al._ (1993) Y. Tokura, Y. Taguchi, Y. Okada, Y. Fujishima, T. Arima, K. Kumagai, and Y. Iye, Phys. Rev. Lett. 70, 2126 (1993).
  * Sachdev (1999) S. Sachdev, Phys. World 12, 33 (1999).
  * Yankowitz  _et al._ (2019) M. Yankowitz, S. Chen, H. Polshyn, Y. Zhang, K. Watanabe, T. Taniguchi, D. Graf, A. F. Young, and C. R. Dean, Science 363, 1059 (2019).
  * Kerelsky  _et al._ (2019) A. Kerelsky, L. J. McGilly, D. M. Kennes, L. Xian, M. Yankowitz, S. Chen, K. Watanabe, T. Taniguchi, J. Hone, C. Dean, A. Rubio, and A. N. Pasupathy, Nature 572, 95 (2019).
  * Lisi  _et al._ (2020) S. Lisi, X. Lu, T. Benschop, T. A. de Jong, P. Stepanov, J. R. Duran, F. Margot, I. Cucchi, E. Cappelli, A. Hunter, A. Tamai, V. Kandyba, A. Giampietri, A. Barinov, J. Jobst, V. Stalman, M. Leeuwenhoek, K. Watanabe, T. Taniguchi, L. Rademaker, S. J. van der Molen, M. P. Allan, D. K. Efetov, and F. Baumberger, Nat. Phys. 17, 189 (2020).
  * Cao  _et al._ (2020) Y. Cao, D. Rodan-Legrain, O. Rubies-Bigorda, J. M. Park, K. Watanabe, T. Taniguchi, and P. Jarillo-Herrero, Nature 583, 215 (2020).
  * Serlin  _et al._ (2020) M. Serlin, C. L. Tschirhart, H. Polshyn, Y. Zhang, J. Zhu, K. Watanabe, T. Taniguchi, L. Balents, and A. F. Young, Science 367, 900 (2020).
  * Choi  _et al._ (2021) Y. Choi, H. Kim, Y. Peng, A. Thomson, C. Lewandowski, R. Polski, Y. Zhang, H. S. Arora, K. Watanabe, T. Taniguchi, J. Alicea, and S. Nadj-Perge, Nature 589, 536 (2021).
  * Chen  _et al._ (2021) H. Chen, H. Yang, B. Hu, Z. Zhao, J. Yuan, Y. Xing, G. Qian, Z. Huang, G. Li, Y. Ye, S. Ma, S. Ni, H. Zhang, Q. Yin, C. Gong, Z. Tu, H. Lei, H. Tan, S. Zhou, C. Shen, X. Dong, B. Yan, Z. Wang, and H.-J. Gao, Nature 599, 222 (2021).
  * Tan  _et al._ (2021) H. Tan, Y. Liu, Z. Wang, and B. Yan, Phys. Rev. Lett. 127, 046401 (2021).
  * Jiang  _et al._ (2021) Y.-X. Jiang, J.-X. Yin, M. M. Denner, N. Shumiya, B. R. Ortiz, G. Xu, Z. Guguchia, J. He, M. S. Hossain, X. Liu, J. Ruff, L. Kautzsch, S. S. Zhang, G. Chang, I. Belopolski, Q. Zhang, T. A. Cochran, D. Multer, M. Litskevich, Z.-J. Cheng, X. P. Yang, Z. Wang, R. Thomale, T. Neupert, S. D. Wilson, and M. Z. Hasan, Nat. Mater. 20, 1353 (2021).
  * Yang  _et al._ (2020) S.-Y. Yang, Y. Wang, B. R. Ortiz, D. Liu, J. Gayles, E. Derunova, R. Gonzalez-Hernandez, L. Šmejkal, Y. Chen, S. S. P. Parkin, S. D. Wilson, E. S. Toberer, T. McQueen, and M. N. Ali, Sci. Adv. 6, eabb6003 (2020).
  * Yu  _et al._ (2021) F. H. Yu, T. Wu, Z. Y. Wang, B. Lei, W. Z. Zhuo, J. J. Ying, and X. H. Chen, Phys. Rev. B 104, L041103 (2021).
  * Teng  _et al._ (2022) X. Teng, L. Chen, F. Ye, E. Rosenberg, Z. Liu, J.-X. Yin, Y.-X. Jiang, J. S. Oh, M. Z. Hasan, K. J. Neubauer, B. Gao, Y. Xie, M. Hashimoto, D. Lu, C. Jozwiak, A. Bostwick, E. Rotenberg, R. J. Birgeneau, J.-H. Chu, M. Yi, and P. Dai, Nature 609, 490 (2022).
  * Yin  _et al._ (2022a) J.-X. Yin, Y.-X. Jiang, X. Teng, M. S. Hossain, S. Mardanya, T.-R. Chang, Z. Ye, G. Xu, M. M. Denner, T. Neupert, B. Lienhard, H.-B. Deng, C. Setty, Q. Si, G. Chang, Z. Guguchia, B. Gao, N. Shumiya, Q. Zhang, T. A. Cochran, D. Multer, M. Yi, P. Dai, and M. Z. Hasan, Phys. Rev. Lett. 129, 166401 (2022a).
  * Teng  _et al._ (2023) X. Teng, J. S. Oh, H. Tan, L. Chen, J. Huang, B. Gao, J.-X. Yin, J.-H. Chu, M. Hashimoto, D. Lu, C. Jozwiak, A. Bostwick, E. Rotenberg, G. E. Granroth, B. Yan, R. J. Birgeneau, P. Dai, and M. Yi, Nat. Phys. 19, 814 (2023).
  * Shao  _et al._ (2023) S. Shao, J.-X. Yin, I. Belopolski, J.-Y. You, T. Hou, H. Chen, Y. Jiang, M. S. Hossain, M. Yahyavi, C.-H. Hsu, Y. P. Feng, A. Bansil, M. Z. Hasan, and G. Chang, ACS Nano 17, 10164 (2023).
  * Zhou  _et al._ (2023) H. Zhou, S. Yan, D. Fan, D. Wang, and X. Wan, Phys. Rev. B 108, 035138 (2023).
  * Ye  _et al._ (2018) L. Ye, M. Kang, J. Liu, F. von Cube, C. R. Wicker, T. Suzuki, C. Jozwiak, A. Bostwick, E. Rotenberg, D. C. Bell, L. Fu, R. Comin, and J. G. Checkelsky, Nature 555, 638 (2018).
  * Liu  _et al._ (2018) E. Liu, Y. Sun, N. Kumar, L. Muechler, A. Sun, L. Jiao, S.-Y. Yang, D. Liu, A. Liang, Q. Xu, J. Kroder, V. Süß, H. Borrmann, C. Shekhar, Z. Wang, C. Xi, W. Wang, W. Schnelle, S. Wirth, Y. Chen, S. T. B. Goennenwein, and C. Felser, Nat. Phys. 14, 1125 (2018).
  * Yin  _et al._ (2019) J.-X. Yin, S. S. Zhang, G. Chang, Q. Wang, S. S. Tsirkin, Z. Guguchia, B. Lian, H. Zhou, K. Jiang, I. Belopolski, N. Shumiya, D. Multer, M. Litskevich, T. A. Cochran, H. Lin, Z. Wang, T. Neupert, S. Jia, H. Lei, and M. Z. Hasan, Nat. Phys. 15, 443 (2019).
  * Ortiz  _et al._ (2019) B. R. Ortiz, L. C. Gomes, J. R. Morey, M. Winiarski, M. Bordelon, J. S. Mangum, I. W. H. Oswald, J. A. Rodriguez-Rivera, J. R. Neilson, S. D. Wilson, E. Ertekin, T. M. McQueen, and E. S. Toberer, Phys. Rev. Materials 3, 094407 (2019).
  * Yin  _et al._ (2022b) J.-X. Yin, B. Lian, and M. Z. Hasan, Nature 612, 647 (2022b).
  * Yang  _et al._ (2023) J. Yang, X. Yi, Z. Zhao, Y. Xie, T. Miao, H. Luo, H. Chen, B. Liang, W. Zhu, Y. Ye, J.-Y. You, B. Gu, S. Zhang, F. Zhang, F. Yang, Z. Wang, Q. Peng, H. Mao, G. Liu, Z. Xu, H. Chen, H. Yang, G. Su, H. Gao, L. Zhao, and X. J. Zhou, Nat. Commun. 14, 4089 (2023).
  * de Juan  _et al._ (2017) F. de Juan, A. G. Grushin, T. Morimoto, and J. E. Moore, Nat. Commun. 8, 15995 (2017).
  * Chang  _et al._ (2017) G. Chang, S.-Y. Xu, B. J. Wieder, D. S. Sanchez, S.-M. Huang, I. Belopolski, T.-R. Chang, S. Zhang, A. Bansil, H. Lin, and M. Z. Hasan, Phys. Rev. Lett. 119, 206401 (2017).
  * Tang  _et al._ (2017) P. Tang, Q. Zhou, and S.-C. Zhang, Phys. Rev. Lett. 119, 206402 (2017).
  * Chang  _et al._ (2018) G. Chang, B. J. Wieder, F. Schindler, D. S. Sanchez, I. Belopolski, S.-M. Huang, B. Singh, D. Wu, T.-R. Chang, T. Neupert, S.-Y. Xu, H. Lin, and M. Z. Hasan, Nat. Mater. 17, 978 (2018).
  * Sanchez  _et al._ (2019) D. S. Sanchez, I. Belopolski, T. A. Cochran, X. Xu, J.-X. Yin, G. Chang, W. Xie, K. Manna, V. Süß, C.-Y. Huang, N. Alidoust, D. Multer, S. S. Zhang, N. Shumiya, X. Wang, G.-Q. Wang, T.-R. Chang, C. Felser, S.-Y. Xu, S. Jia, H. Lin, and M. Z. Hasan, Nature 567, 500 (2019).
  * Rao  _et al._ (2019) Z. Rao, H. Li, T. Zhang, S. Tian, C. Li, B. Fu, C. Tang, L. Wang, Z. Li, W. Fan, J. Li, Y. Huang, Z. Liu, Y. Long, C. Fang, H. Weng, Y. Shi, H. Lei, Y. Sun, T. Qian, and H. Ding, Nature 567, 496 (2019).
  * Takane  _et al._ (2019) D. Takane, Z. Wang, S. Souma, K. Nakayama, T. Nakamura, H. Oinuma, Y. Nakata, H. Iwasawa, C. Cacho, T. Kim, K. Horiba, H. Kumigashira, T. Takahashi, Y. Ando, and T. Sato, Phys. Rev. Lett. 122, 076402 (2019).
  * Yang  _et al._ (2019) Y. Yang, H. xiang Sun, J. ping Xia, H. Xue, Z. Gao, Y. Ge, D. Jia, S. qi Yuan, Y. Chong, and B. Zhang, Nat. Phys. 15, 645 (2019).
  * Schröter  _et al._ (2020) N. B. M. Schröter, S. Stolz, K. Manna, F. de Juan, M. G. Vergniory, J. A. Krieger, D. Pei, T. Schmitt, P. Dudin, T. K. Kim, C. Cacho, B. Bradlyn, H. Borrmann, M. Schmidt, R. Widmer, V. N. Strocov, and C. Felser, Science 369, 179 (2020).
  * Luo  _et al._ (2022) H. Luo, Q. Gao, H. Liu, Y. Gu, D. Wu, C. Yi, J. Jia, S. Wu, X. Luo, Y. Xu, L. Zhao, Q. Wang, H. Mao, G. Liu, Z. Zhu, Y. Shi, K. Jiang, J. Hu, Z. Xu, and X. J. Zhou, Nat. Commun. 13 (2022), 10.1038/s41467-021-27946-6.
  * Li  _et al._ (2022a) H. Li, H. Zhao, B. R. Ortiz, T. Park, M. Ye, L. Balents, Z. Wang, S. D. Wilson, and I. Zeljkovic, Nat. Phys. 18, 265 (2022a).
  * Herbut and Janssen (2014) I. F. Herbut and L. Janssen, Phys. Rev. Lett. 113, 106401 (2014).
  * Jian and Yao (2017) S.-K. Jian and H. Yao, Phys. Rev. B 96, 155112 (2017).
  * Li  _et al._ (2022b) G. Li, H. Yang, P. Jiang, C. Wang, Q. Cheng, S. Tian, G. Han, C. Shen, X. Lin, H. Lei, W. Ji, Z. Wang, and H.-J. Gao, Nat. Commun. 13, 2914 (2022b).
  * Chiu  _et al._ (2023) W.-C. Chiu, G. Chang, G. Macam, I. Belopolski, S.-M. Huang, R. Markiewicz, J.-X. Yin, Z.-J. Cheng, C.-C. Lee, T.-R. Chang, F.-C. Chuang, S.-Y. Xu, H. Lin, M. Z. Hasan, and A. Bansil, Nat. Commun. 14, 2228 (2023).
  * Sanchez  _et al._ (2023) D. S. Sanchez, T. A. Cochran, I. Belopolski, Z.-J. Cheng, X. P. Yang, Y. Liu, T. Hou, X. Xu, K. Manna, C. Shekhar, J.-X. Yin, H. Borrmann, A. Chikina, J. D. Denlinger, V. N. Strocov, W. Xie, C. Felser, S. Jia, G. Chang, and M. Z. Hasan, Nat. Phys. 19, 682 (2023).
  * Rao  _et al._ (2023) Z. Rao, Q. Hu, S. Tian, Q. Qu, C. Chen, S. Gao, Z. Yuan, C. Tang, W. Fan, J. Huang, Y. Huang, L. Wang, L. Zhang, F. Li, K. Wang, H. Yang, H. Weng, T. Qian, J. Xu, K. Jiang, H. Lei, Y.-J. Sun, and H. Ding, Science Bulletin 68, 165 (2023).
  * (60) Inorganic Crystal Structure Database (FIZ Karlsruhe, 2014); http://icsd.fz-karlsruhe.de/icsd .


# Closing the Gap between Client and Global Model Performance in Heterogeneous
Federated Learning

Hongrui Shi<EMAIL_ADDRESS>University of Sheffield ,  Valentin Radu
<EMAIL_ADDRESS>University of Sheffield and  Po Yang
<EMAIL_ADDRESS>University of Sheffield

(2022)

###### Abstract.

The heterogeneity of hardware and data is a well-known and studied problem in
the community of Federated Learning (FL) as running under heterogeneous
settings. Recently, custom-size client models trained with Knowledge
Distillation (KD) has emerged as a viable strategy for tackling the
heterogeneity challenge. However, previous efforts in this direction are aimed
at client model tuning rather than their impact onto the knowledge aggregation
of the global model. Despite performance of global models being the primary
objective of FL systems, under heterogeneous settings client models have
received more attention. Here, we provide more insights into how the chosen
approach for training custom client models has an impact on the global model,
which is essential for any FL application. We show the global model can fully
leverage the strength of KD with heterogeneous data. Driven by empirical
observations, we further propose a new approach that combines KD and Learning
without Forgetting (LwoF) to produce improved personalised models. We bring
heterogeneous FL on pair with the mighty FedAvg of homogeneous FL, in
realistic deployment scenarios with dropping clients.

federated learning, distributed training, knowledge transfer, heterogeneous
systems

††copyright: none††journalyear: 2022††doi: XXXXXXX.XXXXXXX††ccs: Computing
methodologies Distributed algorithms††ccs: Computer systems organization
Embedded systems††ccs: Information systems Information integration

##  1\. Introduction

Federated Learning (FL) has become a popular machine learning paradigm for
training models on many edge computing devices using locally generated data
without compromising privacy. However, expanding FL to include millions of
clients is a double-edge sword. While beneficial to model generalisation by
experiencing diverse data distributions, working with many heterogeneous
devices exposes the challenges of straggler clients and optimisation
inconsistency (Wang et al., 2020; Li et al., 2018). Empirical studies show
that extreme heterogeneous environments can degrade the performance of FL by
up to 4$\times$ (Abdelmoniem et al., 2022).

A major issue causing FL to underperform in heterogeneous settings is the
distribution and aggregation of a single-architecture (structure) model across
clients. To address this, emerging studies are focusing on distributing
custom-size (personalised) models to each client. In (Smith et al., 2017;
Liang et al., 2020; Jiang et al., 2019), FL aggregates client updates of a fix
model structure into a global model. Another solution is to distribute a
custom-size models to each client, and train with knowledge distillation (KD),
thus having heterogeneous models across the federated learning pool. This
latter approach offers the opportunity to customise models on the computing
characteristic of local hardware and data. KD-based FL has been explored in
recent works, either for improving performance of FL with heterogeneity (Li
and Wang, 2019; Lin et al., 2020; Itahara et al., 2020; Chen and Chao, 2020;
He et al., 2020) or for achieving communication efficiency (Jeong et al.,
2018; Guha et al., 2019) and resilience against adversarial attacks (Chang et
al., 2019; Sun and Lyu, 2020), as well surveyed in (McMahan et al., 2017;
Zhang et al., 2021).

Our work focuses on training a server-side global model using an ensemble of
knowledge distillations from personalised client models. The aforementioned
research effort in the community either ignores the effect of client side
choices onto the global model of the server (Li and Wang, 2019; Chang et al.,
2019) or relies on homogeneous model architectures for improving client
performance (Jeong et al., 2018; Lin et al., 2020; Chen and Chao, 2020). Our
expectation is that both server and client models should improve performance
via KD-based ensemble training in heterogeneous settings. In our case, KD is
performed on both server and client side to train the global model and client
models respectively. The knowledge used by KD are the logits created by client
models on a public dataset, aggregating on the server and distributing to
clients every FL round. Following the KD training, custom client models are
updated on the client side using the local data.

In our evaluation, Cifar-100 (Krizhevsky and Hinton, 2009) is assigned as the
public dataset to facilitate knowledge distillation, and fractions of Cifar-10
(Krizhevsky and Hinton, 2009) as local data on the client side. The two chosen
datasets are disjoint to strengthen the data privacy characteristic of our
solution. We simulate non-IID local data settings for the vanilla FedAvg
(McMahan et al., 2017) but considering a realistic scenario with computational
impaired clients not participating in global updates. With our method, both
global and client models have similar performance in global updates, with the
global model even outperforming client models by 3% to 4%. In local updates
with extreme non-IID settings, the performance of custom client models
degrades substantially. To reduce the performance gap, we boost the training
by integrating Learning without Forgetting (LwoF) (Li and Hoiem, 2017) in our
solution. LwoF has the effect of reducing sudden shifts in model
representations, which results in improved client model performance by 5%, and
narrows the performance gap with the global model.

Our work making the following contributions:

  * •

We relax the condition of FL with a single architecture model distributed
across clients by customising the architecture of each client model to match
the computation resources of the target device. Our solution integrates KD to
aggregate and transfer the knowledge from heterogeneous client models.

  * •

Our evaluation is designed to reflect realistic conditions of heterogeneous
client hardware by assuming a part of clients dropping from FL training loop
due to their insufficient computation resources.

  * •

We bring the focus to global model performance during training, not just the
client performance. We control the gap between global model performance and
client model performance via LwoF, an effective solution to prevent models
from forgetting previously learned knowledge.

##  2\. Related Work

One of the major challenges FL faces is the heterogeneity of clients. In real
world, both client hardware and data may vary significantly in terms of
computation resources, network capacity and data distribution. This fact makes
general FL working on hundreds or even millions of clients struggle to fit the
local data characteristics and to meet fast synchronization time frames.
Recently, a growing number of works has sought to approach heterogeneity via
model personalisation, by tuning the global model in the local environment.

###  2.1. Personalised Clients

In contrast to using a universal model, personalised FL (pFL) adopts
heterogeneous models that calibrate both model weights or architectures. The
information exchanged in the network depends on the chosen underlying
techniques.

Tasks in Multi-Task Learning (MTL) and Meta-Learning are comparable to the
clients in FL. MOCHA (Smith et al., 2017) treats local updates of a client
model as an individual learning task in the pipeline of MTL. It characterizes
global updates by client parameter correlation, and local updates for
personalising client models. The Model-Agnostic Meta-Learning (MAML) (Finn et
al., 2017) is connected to pFL by comparing the samples of learning tasks from
the meta-training phase to local updates in FL. Authors of (Jiang et al.,
2019) show that FL forms a good initial model, which can be further
personalised via Meta-Learning.

Split learning is also adopted to personalise models. In (Liang et al., 2020)
the lower part of the global model is personalised locally, while the upper
parts are updated using standard model fusion (McMahan et al., 2017). The
personalised lower part is essentially a feature extractor adapted by each
client.

The aforementioned pFLs either fully or partially rely on a homogeneous model
of a single size architecture. To broaden the application of FL to devices
with smaller computing capability, research in custom size model for client
side is just emerging in the community.

###  2.2. Knowledge Distillation in FL

One approach for allowing heterogeneous model architectures in FL is building
on knowledge distillation (KD), also known as knowledge transfer (KT). As
architectural differences prohibit the server from using direct model fusion
to transfer the learned knowledge, KD exploits the inner knowledge of client
models exposed by their outputs, without needing to align the architectures,
just their outputs.

Early applications of KD to FL (Jeong et al., 2018) are motivated by improving
the communication efficiency. Compared to transmitting the whole model in
general FL, KD-based FL features the transmission of logits, model outputs of
far smaller size. Later efforts (Chang et al., 2019; Sun and Lyu, 2020)
demonstrate the robustness of KD in adversary attacks and noise pollution,
further solidifying its advantageous position. Even more, KD is helping to
address the heterogeneity challenge due to its construction on personalised
models. FedDF (Lin et al., 2020) highlights architectural independence of
client models by integrating KD. However, it still includes the model fusion
before performing ensemble distillation, styling the model heterogeneity with
architecture clusters. FedMD (Li and Wang, 2019) and Cronus (Chang et al.,
2019) produce good client models by relying on the knowledge transfer across
heterogeneous models, but they do not showcase a global model for the server
side, which is common to FL. At a different end, DS-FL (Itahara et al., 2020),
FedBE (Chen and Chao, 2020) and KT-pFL (Zhang et al., 2021) contribute
advanced methods to aggregating the logits in KD-based FL, mitigating the
problem of data heterogeneity.

##  3\. Methods

###  3.1. System Setup

Our objective is to train a global model on the server using knowledge
distilled from personalised client models of different architectures.
Algorithm 1 presents the proposed solution. All client models
$M_{k},\;k=1\dots N$ and the global model $M_{g}$ have the same output size
(to apply KD), but with varying lower architectures. The learning task is
defined by the local data $\mathcal{D}_{k},\;k=1\dots N$ accommodated by
clients. Knowledge distillation is used to transfer the global knowledge to
individual models. To facilitate the knowledge transfer, we follow prior
approaches (Li and Wang, 2019; Chen and Chao, 2020; Lin et al., 2020),
employing a public dataset $\mathcal{D}_{p}$ and indiscriminately distributing
it to the server and clients. We organise our solution into two stages, the
global update and the local update, introduced in Section 3.2 and Section 3.3
respectively.

Input: Public data $\mathcal{D}_{p}$, Local data $\mathcal{D}_{k}$,
personalised client model $M_{k},\;k=1\ldots N$;

Initialization: Train $M_{k},\;k=1\ldots N$ on $\mathcal{D}_{p}$ and then on
$\mathcal{D}_{k}$ until converge (a pre-FL stage detailed in Section 4);

for  _FL round $t=1,2,...$_ do

Global Updates:

for  _Client $M_{k},\;k=1\ldots N$_ do

$\bm{z}^{p}_{k}\leftarrow$ Logits $\left(M_{k},\>\mathcal{D}_{p}\right)$;

end for

Server collects $\bm{z}^{p}_{k},\;k=1\dots N$, create the global knowledge
$\hat{\bm{z}}^{p}=\frac{1}{N}\sum_{k}\bm{z}^{p}_{k}$;

$M^{\prime}_{g}\leftarrow$ Train
$\left(M_{g},\>\hat{\bm{z}}^{p},\>\bm{z}^{p}_{g}\right)$ with Eq. 1 as loss;

for  _Client $M_{k},\;k=1\ldots N$_ do

Downloads $\hat{\bm{z}}^{p}$ from server;

$M^{\prime}_{k}\leftarrow$ Train
$\left(M_{k},\>\hat{\bm{z}}^{p},\>\bm{z}^{p}_{k}\right)$ with Eq. 1 as loss;

end for

Local Updates:

for  _$M_{k},\;k=1\ldots N$_ do

$M_{k}\leftarrow$ Train $\left(M^{\prime}_{k},\>\mathcal{D}_{k}\right)$ with
Eq. 2 or Eq. 4 if LwoF is applied;

end for

end for

Return $M_{k},k=1\ldots m$

Algorithm 1 FedMD+Global

###  3.2. Global Updates

Global updates perform ensemble knowledge distillation for global model and
client models, gathering and transferring the global knowledge. The global
knowledge is in the form of logits distilled by the client models on the
public data, $\mathcal{D}_{p}$. Every client model $k\in\\{1,2,...,N\\}$
computes a forward pass on the instances of $\mathcal{D}_{p}$ to generate its
logits, $\bm{z}^{p}_{k}$, model outputs ahead of the top softmax layer. The
server collects $\bm{z}^{p}_{k},\;k=1\dots N$ from clients and averages them,
$\hat{\bm{z}}^{p}=\frac{1}{N}\sum_{k}\bm{z}^{p}_{k}$. $\hat{\bm{z}}^{p}$ is
used as the global knowledge to update $M_{g}$ and $M_{k},\;k=1\dots N$ in
parallel using a loss function as follows.

(1)
$\mathcal{L}_{g}\left(\hat{\bm{z}}^{p},\bm{z}^{p}_{s}\right)=\|\hat{\bm{z}}^{p}-\bm{z}^{p}_{s}\|_{1}$

where $\mathcal{L}_{g}$ 111Our work employs $\mathcal{L}_{1}(\cdot)$ loss
function. However, other loss functions such as $\mathcal{L}_{CE}(\cdot)$ and
$\mathcal{L}_{2}(\cdot)$ can also be applied. is the loss function for model
training,
$\bm{z}^{p}_{s}\in\\{\bm{z}^{p}_{1},\bm{z}^{p}_{2},...,\bm{z}^{p}_{N},\bm{z}^{p}_{g}\\}$
222The notation is abused for simplicity. Client logits in training are
different to previous ones used for aggregation because the model is being
updated. are the logits from client models and the global model. Training with
the loss function, the model is optimised to minimise the differences of its
own logits $\bm{z}^{p}_{s}$ and the aggregated logits $\hat{\bm{z}}^{p}$ on
$\mathcal{D}_{p}$.

The global updates do not break the privacy-persevering principle of FL
because they only rely on $\mathcal{D}_{p}$ and communicate the logits. By
approaching $\hat{\bm{z}}^{p}$, $M_{g}$ and $M_{k},\;k=1\dots N$ learn the
distilled global knowledge without visiting local data. The performance at
this stage will be evaluated in Section 4.

###  3.3. Local Updates and LwoF

The global updates are followed by local updates that optimise client model
$M_{k}$ on its associated local data $\mathcal{D}_{k}$, personalising the
model performance as follows.

(2)
$\underset{\bm{W}_{k}}{\operatorname{arg}\,\operatorname{min}}\;F(\bm{W}_{k},\>\mathcal{D}_{k})=\mathcal{L}_{t}(f_{k}(\bm{W}_{k},\>\bm{x}_{k}),\>\bm{y}_{k})$

where $\mathcal{L}_{t}$ is the loss function for the target task. $\bm{W}_{k}$
and $f_{k}$ are the weights and function represented by client model $M_{k}$.
$(\bm{x}_{k},\bm{y}_{k})$ is the instance of input-target pairs of local data
$\mathcal{D}_{k}$.

The learned global knowledge could be lost in local updates. To protect it, we
add a regularisation term, shown in Equation 3, which is commonly known as
Learning without Forgetting (LwoF) (Li and Hoiem, 2017), to Equation 2. On
instances of $\mathcal{D}_{k}$, the regularisation term calculates the cross
entropy of the soft labels created by a copy of $M_{k}$ in the previous global
update stage and the $M_{k}$ currently being updated.

(3)
$\mathcal{L}_{CE}(\bm{x}_{k};\rho)=-p(\bm{x}_{k};\rho)\log(q(\bm{x}_{k};\rho))$

where
$p(\bm{x}_{k};\rho)=\frac{\exp\left(\frac{\bm{z}^{\prime}_{k}}{\rho}\right)}{\sum_{j}\exp\left(\frac{\bm{z}^{\prime}_{k}}{\rho}\right)}$
and
$q(\bm{x}_{k};\rho)=\frac{\exp\left(\frac{\bm{z}_{k}}{\rho}\right)}{\sum_{j}\exp\left(\frac{\bm{z}_{k}}{\rho}\right)}$,
$\bm{z}_{k}$ stands for the logits of $M_{k}$, $\bm{z}^{\prime}_{k}$
represents the logits calculated by the $M_{k}$ copied from the global update
stage. $\rho$ is the temperature, a hyperparameter used to further soften the
outputs of the softmax. With the additional term, the objective function of
local updates, Equation 2, becomes

(4)
$\underset{\bm{W}_{k}}{\operatorname{arg}\,\operatorname{min}}\;F(\bm{W}_{k},\>\mathcal{D}_{k})=\mathcal{L}_{t}(f_{k}(\bm{W}_{k},\>\bm{x}_{k}),\bm{y}_{k})+\beta\mathcal{L}_{CE}(\bm{x}_{k};\rho)$

The regularisation term added to the local update prevents $M_{k}$ from
deviating significantly from the $M_{k}$ updated by the global knowledge
$\hat{\bm{z}}^{p}$, protecting the global knowledge. $\beta$ is the
coefficient for the regularisation term. In our evaluation, it is set to 1 as
originally introduced in (Li and Hoiem, 2017).

##  4\. Experiments

Task and Data. We evaluate our method on an image classification task with
Cifar-10 and Cifar-100 (Krizhevsky and Hinton, 2009) as the local (client)
data and public (server) data respectively. Cifar-10 has a total number of
60,000 images of size $32\times 32$ pixels, split across 10 classes with 6,000
images each. Its training set consists of 50,000 images and the test set
contains 10,000 images. Cifar-100 is a harder dataset than Cifar-10 in the
number of classes, containing 100 classes, each with just 600 samples. For
simplicity, we use the coarse labels of Cifar-100 instead of its fine labels,
by grouping the 100 fine classes into 20 coarse classes, each group containing
5 fine classes under a single label. Throughout this section, the performance
of models is evaluated on the test set of Cifar-10.

Baselines. We compare our method with FedAvg (Li et al., 2019), one of the
most popular FL baselines in previous studies of heterogeneous FL. We design
the evaluation to consider the impact of clients running on small devices with
constrained computing resources. For our method, we customise the client model
size such that its participation in the training loop is guaranteed by fitting
the available computing resources of the client. In contrast, FedAvg
distributes a single-size model to all its clients. To simulate smaller
clients struggling to perform the required computations on the single size
model of FedAvg, we introduce client dropping from the training round. This is
equivalent to some clients not meeting the round deadline before the global
aggregation. We experiment with three FedAvg client participation settings:
100%, 60%, 40%. Considering that vanilla FedAvg (McMahan et al., 2017) was
designed to operate with as low as 10% client participation, our baseline is
not impairing FedAvg.

Heterogeneous Models. All our models are constructed on the Wide Residual
Networks (WRN) (Zagoruyko and Komodakis, 2016) with varying depth and width.
For the global (server) model, we choose a WRN-40-1 with depth of 40 (layers)
and width 1 (full). For the 20 custom-size client models we vary the depth of
WRN between 10 and 40, as presented in Table 1. Clients with less computation
resources receive a shallower WRN, while the largest client model is a
WRN-34-1 for the more capable computing clients. In the case of FedAvg
evaluation, all the client models share the same structure as the global
model, a WRN-40-1. By using heterogeneous models, our method enjoys a smaller
overall memory footprint compared with the baseline FedAvg of 60% and 100%
client participation.

Heterogeneous Data. Our evaluation adopts the Dirichlet distribution in
splitting the non-IID local data, following the path set by previous studies
(Lin et al., 2020; He et al., 2020; Hsu et al., 2019). The hyperparameter
$\alpha$ of the distribution embodies the level of non-IIDness. The larger
$\alpha$ is, the more uniformly distributed classes are in client local data.
Figure 1 illustrates the two data distribution scenarios, $\alpha=0.5$ and
$\alpha=0.1$, with splits of data across the 20 clients. A larger bubble
indicates a larger number of data samples of the same class present in the
local data of a client ID. We chose these two scenarios to show the impact of
data heterogeneity.

(a) non-IID distribution with $\alpha=0.5$.

(b) non-IID distribution with $\alpha=0.1$.

Figure 1. The visualization of non-IID local data across 20 clients. Client ID are presented on the Ox axis, and on Oy the Cifar-10 classes. The size of the bubble indicates the number of samples. For the higher $\alpha=0.5$, data is approaching an IID case. In contrast, $\alpha=0.1$ introduces a high degree of non-IIDness, shown by the varying sizes of bubbles. Table 1. Model configuration of our method and for the Fedavg baselines with 20 clients. Our method features WRNs with varying depths, custom to the client computing resources, significantly reducing the total number of trainable parameters. Method | Parties | Depth | Width | #Para
---|---|---|---|---
Ours |  | Client 0-5
---
Client 6-9
Client 10-13
Client 14-17
Client 18, 19
Global
Total
| 10
---
16
22
28
34
40
-
| 1
---
1
1
1
1
1
-
| 79k
---
177k
274k
372k
469k
567k
4.7m
FedAvg |  | Global&Client
---
40%/60%/100% Total
| 40
---
-
| 1
---
-
| 567k
---
3.8/5.6/9.4m

Optimisation Setup. Our experiments apply a uniform optimisation setup in
global updates, with varying conditions for the local update hyperparameters.
A round of global FL update include 5 epochs for knowledge distillation. To
achieve a faster global updates, only 5000 data points are randomly sampled
from the public data (10% of the Cifar-100 training set). For the local
updates, we experiment with 5 and 10 as the number of local epochs, for both
levels of heterogeneity $\alpha=0.5/0.1$. SGD with a learning rate of 0.1 is
employed as the optimiser for both global and local updates. The batch sizes
for global and local updates are tuned to 64 and 128 respectively.

Model Initialisation. To accelerate the system convergence, we follow the
practice introduced by (Li and Wang, 2019), initializing client models before
the commencement of FL. Every client model is trained on the public data and
then on its local data. We define as _Initial (performance)_ the performance
of the client model after transfer from server from global model. For
initialization, the size of the WRN output is set to 30, equal to the sum
between the number of classes in Cifar-100 (20) and the number of classes in
Cifar-10 (10). This multi-task approach joins the use of two disjoint
datasets. The initial performance is not influenced by the number of local
update epochs. For the FedAvg baseline, the initialisation involves only the
local data as FedAvg does not use a public dataset.

Experiment Results. We run all experiments for 50 global rounds. Table 2 and
Figure 2 present the experiment results for our method introduced in the
previous sections and that of the FedAvg baseline. The performance of the
global model is presented as _global performance_ , while the performance of
client models is presented as the average performance at the stage of local
updates and global updates with distillation, as _personalised performance_
and _distilled performance_ respectively. Given $\alpha=0.1$, the global model
in our method outperforms the baseline with 40% client participation by a
large margin (7.28%) and similar performance to the baseline with 60% client
participation, just 0.7% lower. The huge client performance increase from
initialisation to distillation, more than 31% ($\alpha=0.1$) and 23%
($\alpha=0.5$), denote the effectiveness of KD in transferring the global
knowledge. We also see the global model trained by KD outperforms client
models at global updates stage, by 3% to 4%. These results evidence the
feasibility of training a global model without relying on single-size model
aggregation as commonly done in previous works. Finally, the number of local
epochs does not affect the global model performance greatly, except for the
client models, which enjoys a minor performance increase (around 2%) for
$le=10$.

The Impact of Data Heterogeneity. The performance of the distilled model is
constantly better than the performance of the personalised model, while a high
degree of data heterogeneity leads to a larger gap between them. The gap jumps
from 11.26% to 22.64%. This indicates that the client model is better pruned
to forget the learned global knowledge during its training on local data with
heterogeneity. Additionally, when given a mild data heterogeneity
($\alpha=0.5$), we find our solution is outperformed by the baseline with 40%
client participation. From this, we observe that in low data heterogeneity the
performance of FedAvg is not much affected by clients dropping, but if data
heterogeneity is high, our method works better.

Figure 2. Visualizing the performance of the global model and the average across the 20 client models on the Cifar-10 test data, as detailed in Table 2. Table 2. The performance of the global model and that of client models on the Cifar-10 test data. _Global performance_ is the test accuracy achieved by the global model at the distillation stage. _Distilled performance_ and _personalisation performance_ are the average test accuracy of the 20 client models at the global update stage and local update stage, respectively. _Initial performance_ is the average test accuracy of client models from initialisation. Baseline performance is the test accuracy achieved by the global model of FedAvg with 100%/60%40% client participation. Bold values for our method indicate the best performance achieved by the global model with our method. FedAvg (60%) has some values close to our method, while FedAvg (100%) is an unrealistic scenario to have no client drops. Methods |  | $le=5$
---
| $\alpha=0.5$ | $\alpha=0.1$
---|---
| $le=10$
---
| $\alpha=0.5$ | $\alpha=0.1$
---|---
FedAvg (100%) |  | 79.29% | 65.59%
---|---
| 79.15% | 66.84%
---|---
FedAvg (60%) |  | 76.07% |  60.4%
---|---
| 76.00% |  60.29%
---|---
FedAvg (40%) |  | 72.20% | 52.48%
---|---
| 71.97% | 49.16%
---|---
Global (ours) |  |  68.34% |  59.76%
---|---
|  68.74% |  59.53%
---|---
Initial (ours) |  | 44.87% | 27.95%
---|---
| 44.87% | 27.95%
---|---
Distilled (ours) |  | 64.98% | 54.77%
---|---
| 66.01% | 56.27%
---|---
Personalised (ours) |  | 53.72% | 32.13%
---|---
| 54.68% | 32.69%
---|---
Gap (Dis. & Per.) |  | 11.26% | 22.64%
---|---
| 11.33% | 23.58%
---|---

Application of LwoF. We further adopt LwoF in local updates to address the
forgetting problem of global knowledge. Table 3 and Figure 3 summarise the
experiment results. Here our method uses equation 4. First, the performance
gap between distillation and personalisation is largely reduced, dropping to
less than 3.15% from over 23% in all scenarios, proving the effectiveness of
LwoF. Second, the performance of personalised client model is substantially
improved, particularly given a high degree of local data heterogeneity, 5%
improvement for $\alpha=0.1$. However, narrowing the performance gap
sacrifices the global and distilled performance, decreasing by up to 17%. As
such, LwoF is applicable in our proposed method if the client values the
performance of the personalised model. Otherwise, the client model learned at
the distillation stage is more favorable.

Figure 3. Visualising the performance of the global model and the average across the 20 clients when applying the LwoF to local updates, as detailed in Table 3. Table 3. The performance of the global model and of the 20 client models on Cifar-10 test data when applying LwoF. Bold figures present the best performance achieved by the global model with our method. The arrows indicate the change of performance compared with Table 2.  Performance |  | $le=5$
---
| $\alpha=0.5$ | $\alpha=0.1$
---|---
| $le=10$
---
| $\alpha=0.5$ | $\alpha=0.1$
---|---
Initial |  | 44.87% | 27.95%
---|---
| 44.87% | 27.95%
---|---
Global |  |  61.18% $\downarrow$ |  43.84% $\downarrow$
---|---
|  62.3% $\downarrow$ |  41.60% $\downarrow$
---|---
Distilled |  | 56.04% $\downarrow$ | 39.41% $\downarrow$
---|---
| 58.54% $\downarrow$ | 40.62% $\downarrow$
---|---
Personalised |  | 55.46% $\uparrow$ | 37.71% $\uparrow$
---|---
| 56.75% $\uparrow$ | 37.47% $\uparrow$
---|---
Gap (Dis. & Per.) |  | 0.58% $\downarrow$ | 1.70% $\downarrow$
---|---
| 1.79% $\downarrow$ | 3.15% $\downarrow$
---|---

##  5\. Conclusion

In this paper, we focus on the training of a global model in heterogeneous
client settings, driven by knowledge distillation. The performance is
evaluated at the stages both distillation on the server and in local updates
on the client. Using non-IID local data from Cifar-10, and Cifar-100 for
knowledge distillation on the server, the global model achieves comparable
performance to that of FedAvg, which benefits from a high client participation
of 60%. Our experiments also highlight the susceptibility of client models
being affected by the forgetting in extreme non-IID settings. We show that the
performance of client models can be immediately improved with the adoption of
LwoF, though in detriment to global model performance.

## References

  * (1)
  * Abdelmoniem et al. (2022) Ahmed M. Abdelmoniem, Chen-Yu Ho, Pantelis Papageorgiou, and Marco Canini. 2022.  Empirical Analysis of Federated Learning in Heterogeneous Environments. In _Proceedings of the 2nd European Workshop on Machine Learning and Systems_ (Rennes, France) _(EuroMLSys ’22)_. Association for Computing Machinery, New York, NY, USA, 1–9.  https://doi.org/10.1145/3517207.3526969
  * Chang et al. (2019) Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. 2019.  Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer.  _arXiv preprint arXiv:1912.11279_ (2019).
  * Chen and Chao (2020) Hong-You Chen and Wei-Lun Chao. 2020.  Fedbe: Making bayesian model ensemble applicable to federated learning.  _arXiv preprint arXiv:2009.01974_ (2020).
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017\.  Model-agnostic meta-learning for fast adaptation of deep networks. In _International Conference on Machine Learning_. PMLR, 1126–1135.
  * Guha et al. (2019) Neel Guha, Ameet Talwalkar, and Virginia Smith. 2019\.  One-shot federated learning.  _arXiv preprint arXiv:1902.11175_ (2019).
  * He et al. (2020) Chaoyang He, Murali Annavaram, and Salman Avestimehr. 2020\.  Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Hsu et al. (2019) Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019.  Measuring the effects of non-identical data distribution for federated visual classification.  _arXiv preprint arXiv:1909.06335_ (2019).
  * Itahara et al. (2020) Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. 2020.  Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data.  _arXiv preprint arXiv:2008.06180_ (2020).
  * Jeong et al. (2018) Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. 2018\.  Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data.  _arXiv preprint arXiv:1811.11479_ (2018).
  * Jiang et al. (2019) Yihan Jiang, Jakub Konečnỳ, Keith Rush, and Sreeram Kannan. 2019.  Improving federated learning personalization via model agnostic meta learning.  _arXiv preprint arXiv:1909.12488_ (2019).
  * Krizhevsky and Hinton (2009) Alex Krizhevsky and Geoffrey Hinton. 2009.  Learning multiple layers of features from tiny images.  (2009).
  * Li and Wang (2019) Daliang Li and Junpu Wang. 2019.  Fedmd: Heterogenous federated learning via model distillation.  _arXiv preprint arXiv:1910.03581_ (2019).
  * Li et al. (2018) Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2018\.  Federated optimization in heterogeneous networks.  _arXiv preprint arXiv:1812.06127_ (2018).
  * Li et al. (2019) Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2019.  On the convergence of fedavg on non-iid data.  _arXiv preprint arXiv:1907.02189_ (2019).
  * Li and Hoiem (2017) Zhizhong Li and Derek Hoiem. 2017.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 12 (2017), 2935–2947.
  * Liang et al. (2020) Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020.  Think locally, act globally: Federated learning with local and global representations.  _arXiv preprint arXiv:2001.01523_ (2020).
  * Lin et al. (2020) Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. 2020\.  Ensemble distillation for robust model fusion in federated learning.  _Advances in Neural Information Processing Systems_ 33 (2020), 2351–2363.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017\.  Communication-efficient learning of deep networks from decentralized data. In _AIStat_. PMLR.
  * Smith et al. (2017) Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. 2017.  Federated multi-task learning.  _arXiv preprint arXiv:1705.10467_ (2017).
  * Sun and Lyu (2020) Lichao Sun and Lingjuan Lyu. 2020.  Federated model distillation with noise-free differential privacy.  _arXiv preprint arXiv:2009.05537_ (2020).
  * Wang et al. (2020) Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. 2020.  Tackling the objective inconsistency problem in heterogeneous federated optimization.  _arXiv preprint arXiv:2007.07481_ (2020).
  * Zagoruyko and Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis. 2016.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ (2016).
  * Zhang et al. (2021) Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu. 2021\.  Parameterized knowledge transfer for personalized federated learning.  _Advances in Neural Information Processing Systems_ 34 (2021), 10092–10104.