# Federated Few-Shot Learning with Adversarial Learning

Chenyou Fan<EMAIL_ADDRESS>Shenzhen Institute of Artificial
Intelligence and Robotics for SocietyChina and  Jianwei Huang
<EMAIL_ADDRESS>The Chinese University of Hong Kong, ShenzhenChina

(2021)

###### Abstract.

We are interested in developing a unified machine learning model over many
mobile devices for practical learning tasks, where each device only has very
few training data. This is a commonly encountered situation in mobile
computing scenarios, where data is scarce and distributed while the tasks are
distinct. In this paper, we propose a federated few-shot learning (FedFSL)
framework to learn a few-shot classification model that can classify unseen
data classes with only a few labeled samples. With the federated learning
strategy, FedFSL can utilize many data sources while keeping data privacy and
communication efficiency. There are two technical challenges: 1) directly
using the existing federated learning approach may lead to misaligned decision
boundaries produced by client models, and 2) constraining the decision
boundaries to be similar over clients would overfit to training tasks but not
adapt well to unseen tasks. To address these issues, we propose to regularize
local updates by minimizing the divergence of client models. We also formulate
the training in an adversarial fashion and optimize the client models to
produce a discriminative feature space that can better represent unseen data
samples. We demonstrate the intuitions and conduct experiments to show our
approaches outperform baselines by more than 10% in learning vision tasks and
5% in language tasks.

federated learning, few-shot learning, adversarial optimization

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††conference: ; submission††booktitle: ;††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Artificial
intelligence††ccs: Computing methodologies Distributed artificial
intelligence††ccs: Computing methodologies Computer vision

##  1\. Introduction

Figure 1. Overview of a FedFSL system. Distributed client devices train client
models with sampled FSL tasks from local data. Then client models are sent to
the central server and aggregated to a central model, which is sent back to
clients for the next round of local updates.

In recent years, mobile devices such as smartphones and tablet computers have
become perhaps the most frequent and convenient way of connecting users to the
Internet. A capable mobile device can now get access to first-hand user data
such as photos, voice, typing records, and so on, thus it becomes an ideal
platform to deploy machine learning models to better understand user intents
and assist in completing daily tasks.

Conventional distributed machine learning approaches (Li et al., 2014; Zhang
et al., 2015) require the data to be transferred from clients to a central
server, which raises serious concerns of data privacy. A recent proposed
approach to address this issue is federated learning (FedL) (McMahan et al.,
2017; Zhao et al., 2018; Li et al., 2020c). In the FedL paradigm, each
participating client computes a local machine learning model with its own
data, while a central server periodically coordinates client models by model
aggregation (without collecting the actual data). FedL provides a promising
way of learning models on distributed devices while preserving data privacy
and locality.

However, existing FedL approaches assume each participating client has
sufficient training data for the tasks of interest. The realistic situation is
that data collected with mobile devices can be insufficient and almost always
unlabeled. This severely limits the practicality and scalability of FedL in
realistic applications. For example, image classification with federated deep
neural networks (DNNs) is a commonly compared task in recent FedL studies
(McMahan et al., 2017; Zhao et al., 2018; Li et al., 2020c). Training such a
DNN in supervised fashion requires hundreds of labeled training samples or
more for every class. In reality, each mobile user may own just one or a few
samples of each interested image category, and the mobile user often does not
have time or interest to label these images. The huge gap between lab
scenarios with abundant labeled data and real situations with scarce and
mostly unlabeled data motivates us to consider the following question: How to
make FedL effective in data-scarce scenarios?

Method | Federated learning | Non-convex model | SGD solver | Few-shot learning | Divergence | Feature learning
---|---|---|---|---|---|---
FedAvg(McMahan et al., 2017) / FedProx(Li et al., 2018) | ✓ | ✓ | ✓ |  |  |
FEDL (Dinh et al., 2019) | ✓ |  | ✓ |  |  |
Mutual learning (Zhang et al., 2018b) |  | ✓ | ✓ |  | ✓ |
Local adaptation (Yu et al., 2020) | ✓ | ✓ | ✓ |  | ✓ |
Gradient-based FSL(Finn et al., 2017) |  | ✓ | ✓ | ✓ |  |
Metric-based FSL (Gidaris and Komodakis, 2018) |  | ✓ | ✓ | ✓ |  | ✓
FedFSL-naive ((Chen et al., 2018), 3.3) | ✓ | ✓ | ✓ | ✓ |  |
FedFSL-MI (ours, 4.1) | ✓ | ✓ | ✓ | ✓ | ✓ |
FedFSL-MI-Adv (ours, 4.2) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 1. Comparisons of relevant approaches with supported features.

A recently developed approach to address the issue of insufficient training
data is few-shot learning (FSL) (Vinyals et al., 2016; Finn et al., 2017; Li
et al., 2017). FSL aims to develop machine learning models to solve unseen
tasks with very few labeled training data, but often in the context of a
single data source. In this paper, we propose a federated few-shot learning
(FedFSL) framework for efficiently training few-shot classification models
from many data sources. The few-shot model can tackle novel tasks with just a
few labeled data for unseen classes. As shown in Fig. 1, the paradigm of
FedFSL is to first perform local updates with few-shot tasks sampled from
local data, then send local models to a central server for model aggregation
and coordination. Through inheriting the merits of FedL, FedFSL also preserves
the data privacy and communication efficiency.

FedFSL has many potential applications for utilizing machine learning models
on mobile devices. For example, a few-shot language model can be used to
suggest words by learning from just a few typing records from each of many
users; a few-shot sentiment analysis model can be used to depict user profiles
given only a few tweets posted by each of the many users; a few-shot face
recognition model can identify users and their friends by learning from just
few annotated photos by each of many mobile users.

There are two technical challenges we encounter during the development of the
FedFSL framework: 1) directly using the existing FedL approaches to the data-
scarce scenarios may lead to misaligned decision boundaries produced by client
models, and 2) constraining the decision boundaries to be similar over clients
would develop a classifier overfit to training tasks but not transferable to
unseen tasks. To address these issues, we propose to regularize local updates
by minimizing the divergence between client models and the central model. We
then formulate the training of the feature generator part and classifier part
of the model in adversarial fashion. In this way, the client models are
explicitly optimized to produce a discriminative feature space that can better
represent unseen data samples. We demonstrate the intuitions and conduct
experiments to show the effectiveness of our approaches.

Our contributions can be summarized as follows:

  * •

We propose a novel federated few-shot learning framework (FedFSL) that can
perform effective federated learning on few-shot tasks. This represents the
first step in addressing the commonly encountered but overlooked scenarios in
mobile computing where training data is scarce and testing tasks are distinct.

  * •

We present the key innovations in formulating FSL with federated clients as
well as explicitly optimizing the federated model by minimizing model
discrepancy in challenging non-IID scenarios.

  * •

We define a novel concept of mutual divergence of federated client models,
which can be minimized to better coordinate the client training on scarce
local data.

  * •

We design a dedicated adversarial learning approach to construct a
discriminative feature space, which better generalizes to unseen tasks
compared with existing training procedures of FSL models.

  * •

We evaluate our framework by modelling different types of structured data
(such as images and sentences) with both CNN and RNN models, showing its
effectiveness and practical usability in modelling various learning tasks in
machine vision and NLP.

  * •

Our approaches significantly outperform baselines that are either non-
distributed or not aligning the feature space across the clients by more than
10% on vision tasks and 5% on language tasks.

##  2\. Related work

We will briefly review recent related work in two categories: (i) studies
either federated learning (FedL) (e.g., (McMahan et al., 2017; Zhao et al.,
2018; Li et al., 2018; Li et al., 2020c; Dinh et al., 2019; Zhang et al.,
2020)) or few-shot learning (FSL) (e.g., (Vinyals et al., 2016; Snell et al.,
2017; Munkhdalai and Yu, 2017; Li et al., 2017; Sun et al., 2019; Gidaris and
Komodakis, 2018; Sung et al., 2018; Li et al., 2020a)), or both of them (Chen
et al., 2018) (ii) studies proposing similar ideas of minimizing model
divergence to better learn individual models or an ensemble model.

To our best knowledge, training FSL models on distributed devices is still an
under-explored open problem. The first work of this topic was from Chen et al.
(Chen et al., 2018) who explored federated meta-learning by applying FedAvg on
meta-learning approaches such as MAML (Finn et al., 2017) in a straightforward
way. However, their goal is to improve supervised learning by better sharing
models among federated clients, instead of learning few-shot tasks. They
neither evaluated their models on FSL tasks, nor explicitly considered dealing
with the underlying data heterogeneity in different devices (e.g., non-IID
case) which can severely harm FSL. Our model captures the idea of federating
transferable knowledge among distributed clients. We further explore the
practical data-scarce scenarios and evaluate our models on challenging
benchmark FSL datasets. In addition, we explicitly resolve the data
heterogeneity issue by proposing a family of more effective meta-learning
approaches designed for federated settings.

The other work that loosely connects FSL with FedL is Li et al. (Li et al.,
2020b), which proposed a differentially private algorithm for securing
parameter transfer across devices or learning stages. The authors considered
FedL and FSL as two separated applications of their technique, and their goal
is to secure data privacy during model sharing instead of performing FSL with
federated devices.

There are two recent studies relating to the techniques that we utilize to
coordinate client model training and to learn the consistent feature space,
though neither of them considered few-shot learning. Yu et al. (Yu et al.,
2020) used knowledge distillation (Hinton et al., 2015) (a variant of KL-
divergence) in federated learning to learn different client models to better
fit local data. Our approach is different from theirs such that 1) they did
not perform model aggregation as they want to train client models that fit on
local data, but our method performs aggregation for learning a unified global
model, 2) their method followed the normal supervised learning paradigm which
cannot solve few-shot tasks, and 3) they did not consider feature space
learning but our method does. Zhang et al. (Zhang et al., 2018b) proposed to
minimize the KL-divergence of every client model pair to enhance ensemble
learning. However, this imposes heavy computation and communication costs in
distributed scenarios. On the contrary, we propose to approximate the client
pairwise KL-divergence with the divergence between the client model and the
federated global model, which integrates into FedL seamlessly.

It’s also worth to clarify that several recent studies (Guha et al., 2019;
Salehkaleybar et al., 2019; Shin et al., 2020) focused on reducing the
communications of distributed learning with one or a few communication rounds
under federated learning settings. Though this topic is also referred as “few-
shot federated learning”, it’s totally different from our work which studies
federated learning under data-scarce scenarios.

##  3\. Federated Few-shot Learning

In this section, we formulate the Federated Few-Shot Learning (FedFSL)
framework in details. Specifically, we will study the few-shot classification
task which learns to classify on novel classes with few training samples. We
will first review the general federated learning (FedL) objective in 3.1 and
the general centralized few-shot learning (FSL) procedures in 3.2
respectively, based on which we propose the FedFSL formally in 3.3.

###  3.1. General Federated Learning Objective

First, we briefly review federated learning (FedL) paradigm and its common
implementations in this subsection. We consider a distributed system of $K$
clients, each owning a local data source. In FedL, each client trains a local
machine learning model based on its local data, while a central server
coordinates the clients periodically by collecting their parameters,
aggregating them into a central model and sending its parameter back to all
clients. This process will repeat for multiple communication rounds until
convergence or timeout.

Formally, let $n_{k}$ be number of data samples of client $k$,
$n=\sum_{k}n_{k}$ be total samples across the devices, $w$ be the learning
model. We consider a local objective for client $k$ as the average loss over
all data samples

(1)
$\small\mathcal{L}_{k}(w)=\frac{1}{n_{k}}\sum_{i=1}^{n_{k}}f(x_{i},y_{i};w),$

in which $f$ is a loss function that evaluates the prediction of model $w$ on
a data sample $(x_{i},y_{i})$. The type of loss function $f$ depends on the
task and is known by all clients. For example, in a classification task with
deep neural networks (DNNs), $f$ is often chosen as the cross-entropy loss
applied on the models’ probabilistic outputs.

The global target is a weighted average of local objectives

(2) $\small\underset{w}{\text{min}}\
\mathcal{L}(w)=\sum_{k=1}^{K}p_{k}\mathcal{L}_{k}(w),$

in which $p_{k}=n_{k}/n$. However, as direct data exchange is prohibited in
FedL, directly optimizing the global objective (2) needs to perform a full
batch gradient descent on all data that each client holds, and perform model
aggregation after each client update. This requires high memory usage (if the
client model is large) and excessive communications for exchanging models
between clients and the server. A common practice to resolve this issue is to
approximate the global objective, such as in the FedAvg (McMahan et al., 2017)
algorithm which optimizes each local objective $\mathcal{L}_{k}$ individually
and in the FedProx (Li et al., 2018) which solves the local objectives with
proximal terms to regularize training.

Existing FedL approaches often assume that the clients always hold sufficient
training data for a same task, e.g., all the clients should own enough data
samples of the same categories in a classification task. However, the
realistic situation is each client may own a few labeled data samples for
certain categories for training, and may encounter unlabeled data samples for
testing with unseen true categories. This leads us to study the few-shot
classification task which learns to classify on novel classes with few
training samples in the following sections.

###  3.2. General Centralized Few-Shot Learning

Next, we briefly review centralized few-shot learning (FSL) procedures. FSL
aims to learn a generic model which can adapt to unseen tasks with only a few
labeled training samples. In this paper, we study the few-shot classification
task which aims to classify novel classes. We define an $N$-way $P$-shot
$Q$-query FSL task as a task of training a model with $P$ labeled images for
each of $N$ classes and then evaluating the model with $Q$ unlabeled query
images for each class. $P$ is typically very small such as 1 or 5 as “few-
shot” implies.

Let us consider a toy example of classifying animal pictures. The training
data are images of cats and dogs which we call base classes. The testing data
are images of tigers and wolves which we call novel classes; each novel class
owns one labeled image and many unlabeled images. We wish to develop a model
trained with base classes (cat and dog) that can predict on tiger and wolf
(2-way) samples by observing each category just one labeled image (1-shot),
i.e., a 2-way 1-shot FSL task.

Figure 2. Three-step meta-learning of FSL.

To train a capable FSL model, recent state-of-the-art work (Gidaris and
Komodakis, 2018; Sung et al., 2018; Finn et al., 2017; Li et al., 2017)
adopted a training strategy called meta-learning which samples various few-
shot tasks from training data and optimizes the model to fast adapt to these
new tasks. The key idea of meta-learning is to learn some transferable
knowledge from few data samples in training data that can apply on unseen
data. State-of-the-art gradient-based approaches (Finn et al., 2017; Sun et
al., 2019) consider a good model initialization as transferable knowledge if
it could adapt to various few-shot tasks with just a few gradient steps. They
propose to explicitly search for such an initial model as follows.

The training objective is to minimize the training loss over a batch of tasks
$\mathcal{T}\in\mathcal{B}$ as follows

(3) $\begin{split}w^{*}&=\underset{w}{\text{min}}\
\mathcal{L}(w)=\frac{1}{|\mathcal{B}|}\sum_{\mathcal{T}\in\mathcal{B}}\ell_{\mathcal{T}}(w),\\\
\end{split}$

in which $w^{*}$ is the optimized model that trains to fast adapt to new tasks
and $\ell$ is a task loss. This can be tackled with an iterative approach in
which each iteration can be decomposed into three steps, as shown in Fig. 2.

  * •

Sampling step: The first step is to sample a few-shot task $\mathcal{T}$, also
called an episode, from base classes. For an $N$-way $P$-shot $Q$-query few-
shot task, an episode consists of $P$ data instances sampled from each of $N$
distinct base classes as a support set $\mathcal{T}^{(s)}$, and $Q$ data
instances sampled from the same $N$ classes as a query set
$\mathcal{T}^{(q)}$, which gives a total of $(P+Q)\cdot N$ instances.

  * •

Adaptation step: The second step is to adapt the current model to the sampled
task with gradient descents. This step uses the few labeled data in the
support set $\mathcal{T}^{(s)}$ and performs one or several gradient steps
towards optimizing the model weights to the sampled task such that

(4)
$\begin{split}w^{{}^{\prime}}=w-\alpha\nabla_{w}f_{\mathcal{T}^{(s)}}(w),\end{split}$

in which $w^{\prime}$ is the adapted model, $\alpha$ is the step size.

  * •

Optimization step: The final step is to evaluate $w^{\prime}$ with more
samples in the query set $\mathcal{T}^{(q)}$ with the empirical loss function

(5)
$\begin{split}\ell_{\mathcal{T}}(w)&=f_{\mathcal{T}^{(q)}}(w^{\prime})=f_{\mathcal{T}^{(q)}}(w-\alpha\nabla
f_{\mathcal{T}^{(s)}}(w)),\\\ \end{split}$

which can be solved by another gradient descent

(6) $\begin{split}w\leftarrow w-\beta\
\nabla_{w}\ell_{\mathcal{T}}(w),\end{split}$

in which $\beta$ is the learning rate.

The above procedures are summarized in Fig. 2. For a centralized FSL to
converge, the sampling-adaptation-optimization procedures are repeated for
many iterations. This will produce the optimal parameter that best adapts to
few-shot tasks.

During inference, a learned FSL model firstly adapts to unseen tasks with a
few labeled samples with (4), then predicts the labels on query samples. In
our toy example mentioned above, the model is trained with cat and dog samples
to discern two species, then is used to classify tiger and wolf samples with
its learned capacity of distinguishing patterns.

Figure 3. Demo of a two-client case of FedFSL based on meta-learning
procedure.

###  3.3. Federated Few-shot Learning (FedFSL)

As our purpose is to facilitate distributed devices to learn models for few-
shot tasks, we need study how to design such a framework in which meta-
learning procedures can be integrated in the federated learning. We propose
Federated Few-shot Learning (FedFSL) in this section. The goal of FedFSL is to
search for a global optimal model $w^{*}$ learned on distributed data sources
that can best perform few-shot tasks.

Suppose we have $K$ participating clients and each of them can sample batches
of few-shot tasks $\mathcal{T}_{k}\in\mathcal{B}_{k}$ from their local data
sources as discussed in previous section. We first define $k$-th client’s
local FSL objective by extending from (3)

(7)
$\begin{split}\mathcal{L}_{k}(w)&=\frac{1}{|\mathcal{B}_{k}|}\sum_{\mathcal{T}_{k}\in\mathcal{B}_{k}}\ell_{\mathcal{T}_{k}}(w),\\\
\end{split}$

in which the subscript $k$ of $\mathcal{T}_{k}$ emphasizes that it’s sampled
from local data source of client $k$.

Our goal is to find a global optimal model $w^{*}$ which minimizes the
weighted average of local FSL objectives. We formulate this global target as
an FedL problem similar to (2), i.e.,

(8) $\begin{split}w^{*}&=\underset{w}{\text{min}}\
\mathcal{L}(w)=\sum_{k=1}^{K}p_{k}\mathcal{L}_{k}(w)=\sum_{k=1}^{K}\frac{|\mathcal{B}_{k}|}{|\mathcal{B}|}\mathcal{L}_{k}(w)\\\
\end{split}$

Directly optimizing (8) would be difficult, as learning an optimal $w^{*}$ for
all distributed clients would require excessive communications across clients,
as discussed in 3.1. We thus provide an efficient algorithm for tackling this
issue.

Motivated by FedAvg (McMahan et al., 2017), we propose a straightforward way
of solving a surrogate objective of (8) to approximate the global solution,
which we call FedFSL-naive. As shown in Fig. 3, FedFSL-naive iteratively
updates the central model $w$ by (i) first optimizing each local objective of
(7) in parallel, and (ii) aggregating local models to the central model, which
update the global model and send it back to clients for the next round of
optimization. Formally,

  * •

At the $t$-th optimization round, each client $k$ optimizes the following
local objective

(9) $\begin{split}w^{*,t}_{k}&=\underset{w}{\text{argmin}}\
\mathcal{L}_{k}(w)=\underset{w}{\text{argmin}}\
\frac{1}{|\mathcal{B}_{k}|}\sum_{\mathcal{T}_{k}\in\mathcal{B}_{k}}\ell_{\mathcal{T}_{k}}(w),\\\
\end{split}$

in which the FSL loss $\ell(w)$ is given by (5). Fig. 3 shows a two-client
example, in which each client updates on three sampled tasks with (9) and
obtains local optimal models $w^{*,t}_{1}$ and $w^{*,t}_{2}$. The clients then
send these local parameters to the central server.

  * •

Then the central server approximates the optimal global solution by averaging
the client models such that

(10)
$w^{t+1}=\sum_{k=1}^{C}\frac{|\mathcal{B}_{k}|}{|\mathcal{B}|}w^{*,t}_{k},$

which will be synchronized to all clients for next round of optimization.

The above steps (9) and (10) are repeated for multiple rounds until
convergence. We summarize the procedures in Algorithm 1.

Input: A set of $K$ federated clients. A local FSL objective $\mathcal{L}_{k}$
for each client $k$.

Output: A global model $w$ optimized for FSL task.

Server executes:

Initialize global model $w^{0}$

$t\leftarrow 1$

while  _t $\leq$ maximum rounds $T$_ do

for  _each client $k$ in parallel_ do

$w_{k}^{t}\leftarrow$ ClientUpdate$(w^{t})$

end for

Clients send model parameters $w_{1...K}^{t}$ to server

$w^{t+1}\leftarrow\sum_{k=1}^{K}\frac{|\mathcal{B}_{k}|}{|\mathcal{B}|}w^{t}_{k}$

// model avg

The server sends $w^{t+1}$ back to clients

$t\leftarrow t+1$

end while

Return $w^{t}$

ClientUpdate$(w)$:

Input: global model from previous round $w^{t}$

Output: updated local model $w^{t}_{k}$

$w\leftarrow w^{t}$

Sample a batch of episodes
$\mathcal{B}_{k}=\\{\mathcal{T}_{1},...,\mathcal{T}_{n}\\}$

$w^{k}_{t}\leftarrow$ Solve Eq.(9) with SGD

Return $w^{t}_{k}$

Algorithm 1 FedFSL-naive framework.

###### Proposition 1.

If loss function $f_{\mathcal{T}}(w)$ in (4) satisfies the strongly-convex
conditions as in Corollary 1, Finn et al. (Finn et al., 2019) 111 $f$ is
G-Lipschitz, $\beta$-smooth, $\rho$-Lipschitz Hessians and $\mu$-strongly
convex , Algorithm 1 converges at a rate of $\mathcal{O}(\frac{1}{T})$ in
which $T$ is the total number of every device’s gradient updates during
training.

###### Proof.

As $f_{\mathcal{T}}(w)$ is strongly-convex, Corollary 1, (Finn et al., 2019)
implies that the local FSL objective $\mathcal{L}_{k}$ in (7) is also
strongly-convex. By taking Algorithm 1 as a FedAvg algorithm with a strongly-
convex objective, Theorem 3, (Li et al., 2019) implies that it converges at a
rate of $\mathcal{O}(\frac{1}{T})$ in which $T$ is total number of local
gradient updates of all devices during training. ∎

Proposition 1 shows that for a convex FSL objective, e.g., the cross-entropy
loss of a linear or logistic model with $L_{2}$-regularization, Algorithm 1
converges. For a broader family of non-convex models, such as deep neural
networks, convergence of FedL is still an open research topic though some
attempts have been made (Smith et al., 2017; Li et al., 2018).

We note that solving (9) requires computing a gradient through $w^{\prime}$,
which is another function of gradient of $w$ as given by (4), and thus
requires computing the Hessian. Fortunately, fast Hessian-vector products
(Ghorbani et al., 2019) are widely adopted to approximate the second-order
information, which is equivalent of performing backward passes twice with SGD.
If we denote the number of parameters of the model as $W$, this results in
$\mathcal{O}(W)$ computational time for one iteration of model update. Thus
the total computational time of FedFSL-naive is $\mathcal{O}(W\cdot T)$. Note
that Chen et al. (Chen et al., 2018) proposed similar federated meta-learning
procedures for supervised learning tasks.

##  4\. Improving FedFSL with better coordination

So far, we have provided FedFSL-naive as a straightforward way of performing
distributed few-shot learning. However, one unresolved technical challenge is
that meta-learning depends on sampled episodes that contain only very few
labeled data points. In data-scarce scenarios, even the data distribution over
the clients could be the same, the high variance of the data may lead to quite
distinct gradient descent directions, and thus the trained few-shot models
could become quite distinct over the clients. This results in model divergence
in aggregation. Similar observations were also found in FedL tasks with non-
IID data (Zhao et al., 2018; Li et al., 2018) but this problem has been
amplified in the data-scarce scenarios we consider.

Figure 4. Illustration of decision boundaries learned by (a) FedFSL-naive and
(b) FedFSL-MI in two-client case.

In Fig. 4(a), we illustrate a two-client case which follows FedFSL-naive
scheme to learn models individually (left) first and average the models to
obtain the federated decision boundary (right). However, the discrepancy
between two client models makes them provide misaligned individual decision
boundaries (left). Thus the aggregated central model provides less optimal
federated decision boundary (right) with lots of misclassified data samples.

In this section, we will discuss how to better coordinate client models with
mutual information in 4.1, and we propose an adversarial learning procedures
to further learn a discriminative feature space in 4.2.

###  4.1. FedFSL with Mutual Information (MI)

To better coordinate client models learned on distinct data sources, we
propose to regularize the local updates by minimizing the divergence between
client models and the central model. As the central model is shared with the
clients at each round, it serves as an intermediate way of training the
clients collaboratively without overfitting local data.

####  4.1.1. Mutual information (MI)

Kullback-Leibler (KL) divergence is commonly used as a measure of the
difference of two probability distributions. In a collection of multiple
distributions, a summed pair-wise KL-divergence is used in recent studies
(Belghazi et al., 2018; Zhang et al., 2018b) in ensemble learning to measure
the total discrepancy of all those distributions, which we term it mutual
information (MI). In FedL, MI can be utilized to measure the discrepancy of
all the participating client models. However, calculating the pairwise KL-
divergence requires $\mathcal{O}(K^{2})$ calculations which could impose heavy
burdens to the central server. We will propose a simplified MI proximal term
and integrate it in FedFSL, and we target to minimize it for reducing internal
discrepancy over client models.

####  4.1.2. FedFSL-MI

Formally, at $t$-th optimization round, we ask the central server to produce a
k-exclusive global model $w^{t}_{\text{-}k}$ such that

(11) $w^{t}_{\text{-}k}=\sum_{c=1,c\neq
k}^{K}\frac{|\mathcal{B}_{c}|}{|\mathcal{B}_{\textit{-}k}|}w^{t}_{c}\ ,$

and send back to $k$-th client. The k-exclusive global model
$w^{t}_{\text{-}k}$ is taken as an ensemble of all other client models except
the $k$-th client.

We now define the MI loss as the Kullback-Leibler (KL) divergence of
probability outputs produced by the k-exclusive global model
$w^{t}_{\text{-}k}$ and the client model $w_{k}$ over sampled tasks such that

(12)
$\begin{split}&\mathcal{L}_{k}^{MI}(w^{t}_{-k},w_{k})=\frac{1}{|\mathcal{B}_{k}|}\sum_{\mathcal{T}_{k}}D_{KL}\left(p(w^{t}_{-k})~{}||~{}p(w_{k})\right)\\\
=&\frac{1}{|\mathcal{B}_{k}|}\sum_{\mathcal{T}_{k}}(p(w^{t}_{-k})\cdot\log
p(w^{t}_{-k})-p(w^{t}_{-k})\cdot\log p(w_{k})),\\\ \end{split}$

in which $p(\cdot)$ is the probability outputs of an FSL model. Given an
$N$-way FSL task $\mathcal{T}_{k}$, $p(w)$ is the normalized $N$-way
predictions over $N$ classes that sums to one. We aim to minimize MI in order
to reduce the discrepancy.

By integrating MI into the original local FedFSL objective function (7), our
new target is to jointly minimize the MI loss together with the local FSL task
loss such that

(13) $\begin{split}&w^{*,t}_{k}=\underset{w_{k}}{\text{min}}\
\mathcal{L}_{k}(w_{k})+\gamma\mathcal{L}_{k}^{MI}(w^{t}_{\text{-}k},w_{k}),\end{split}$

in which the weight $\gamma>0$ can be searched by cross-validation. We call
this new method FedFSL-MI (FedFSL with Mutual Information regularization).

As the k-exclusive global model is different from client to client, the
central server needs to compute the global model $K$ times. In practice, when
$K$ is large (e.g., $\geq$10), we can conveniently reuse the aggregated
central model $w^{t}$ as in (10) to approximate $w^{t}_{\text{-}k}$, leading
to no additional computation cost. We will default to use $w^{t}$ to
approximate $w^{t}_{\text{-}k}$ in our experiments to reduce computations in
simulated mobile devices, and we will compare them in Sec.5.4.4.

We illustrate the intuition of using MI in Figure 4(b). As we minimize the
discrepancy among client models, we encourage the decision boundaries to be
consistent across the clients. Thus the federated model could produce a better
aligned decision boundary. In the empirical study, we will show that it leads
to a significant improvement over FedFSL-naive. However, the decision
boundaries could become complex due to the consistency constraint (Figure 4(b)
right), which we will discuss in next section.

###  4.2. Improving feature space with adversarial learning

One technical disadvantage of FedFSL-MI is that constraining the decision
boundaries to be similar over clients would develop a complex classifier that
overfits to training tasks. However, the classes of testing data for FSL are
different from the base classes of training data, which makes the complex
decision boundary not useful to unseen tasks. This also presents a key
difference between FSL and conventional supervised learning.

####  4.2.1. Feature space

We aim to improve the FedFSL-MI by learning a central model that can produce a
better-aligned decision boundary for unseen tasks. Recent studies of metric
learning (Schroff et al., 2015; Gidaris and Komodakis, 2018) have shown that
learning a good feature space is beneficial to various tasks as it provides
good representations (also known as features or feature embeddings) for data
samples. In an ideal feature space, samples of the same class or similar
classes are close to each other, while samples of different classes are far
away. For example, images of cats and tigers are close in feature space, while
tigers and wolves could be far away due to their distinct visual features.

Researchers have also found that a representative feature space is a kind of
transferable knowledge that can be used to learn unseen data samples. For
example, pre-trained vision recognition models (i.e., ResNet) (Sharif Razavian
et al., 2014; Ren et al., 2015) and language models (i.e. BERT) (Pennington et
al., 2014; Vaswani et al., 2017) can produce off-the-shelf image/language
representations for various tasks. Few-shot classification, as we consider in
this paper, will especially benefit from a discriminative and transferable
feature space if such a feature space can be derived properly and efficiently
in distributed scenarios. We will show it is feasible in next sections.

####  4.2.2. Learn a consistent feature space

To our best knowledge, how to learn a consistent and discriminative feature
space with FedL has never been studied before. The difficulty is how to
construct a consistent feature space over many clients without sharing data.
Motivated by recent progress in Generative Adversarial Networks (GANs)
(Goodfellow et al., 2014; Saito et al., 2018), we will decompose an FSL model
as a feature generator and a classifier (i.e., discriminator) which can be
optimized in an alternative and iterative fashion. This new adversarial
learning approach is named as FedFSL-MI-Adv (FedFSL with Mutual Information
regularization and Adversarial learning).

We first introduce some new notations to facilitate discussion. Without loss
of generality, a few-shot classification model can be represented as a feature
generator $\Theta$ and a classifier $\theta$. For a given data sample $x$, we
denote its generated feature as $f_{\Theta}(x)$. The output logits of the
classification model is derived by applying the classifier on the feature such
that $f_{\theta}\circ f_{\Theta}(x)$. Thus the predicted probabilistic
distribution over $N$ classes is denoted as
$p(\Theta,\theta)=\sigma(f_{\theta}\circ f_{\Theta}(x))$ in which $\sigma$ is
the softmax function.

In centralized training, the feature generator and the classifier could be
learned with supervised learning without many tricks. However, in distributed
scenarios, we have to additionally consider aligning feature space learned
with many clients. We propose a novel procedure that alternatively trains the
classifier and the generator as two opponents. We train the client model
classifier to maximize the difference between its predictions and central
model predictions, while train the client feature generator to minimize the
difference. We will explain the details and intuitions.

Figure 5. An example of federated decision boundaries learned by FedFSL-MI-adv
with two-stage adversarial learning on two clients of different data
distributions.

####  4.2.3. Adversarial learning procedure

We design a two-stage adversarial learning procedure for the local update for
explicitly learning a consistent feature space. In overall, the
$(t\text{-}1)$-th communication round ends up by aggregating the client models
to a central model in (10) and sending it back to clients as
$w_{k}=[\Theta_{k},\theta_{k}]$. At the beginning of next round of local
update, each client initializes a new classifier $\theta_{k}^{\prime}$. The
feature generator $\Theta_{k}$ and two classifiers $\theta_{k}$ and
$\theta_{k}^{\prime}$ are all trainable and will involve in a two-stage
adversarial training procedure as follows.

  * •

Training stage-1 is to train two classifiers to produce _distinct_ decision
boundaries, in the motivation of detecting ambiguous data samples in current
feature space. Ambiguous samples are those lying near the decision boundaries
that tend to be misclassified by two different classifiers, as shown in Fig.
5(stage-1). Intuitively, detecting those samples is the prerequisite of
optimizing a feature space that resolves the ambiguity. During this stage, the
feature generator $\Theta_{k}$ remains fixed, while $\theta_{k}$ and
$\theta_{k}^{\prime}$ are updated.

We first define the adversarial loss to measure the difference of two
classifiers $\theta_{k}$ and $\theta_{k}^{\prime}$ by the KL divergence of
their probabilistic outputs $p(\Theta,\theta)$ such that

(14)
$\begin{split}&\mathcal{L}_{k}^{adv}(\theta_{k},\theta_{k}^{\prime},\Theta_{k})=\frac{1}{|\mathcal{B}_{k}|}\sum_{\mathcal{T}_{k}}D_{KL}\left(p(\Theta_{k},\theta_{k})~{}||~{}p(\Theta_{k},\theta_{k}^{\prime})\right).\end{split}$

We simultaneously minimize the FSL local objective (7) while _maximize_ the
adversarial loss to encourage the disagreement of the two classifiers.
Formally, we define the objective of stage-1 as a combination of local task
objective $\mathcal{L}_{k}$ (7) and adversarial loss $\mathcal{L}_{k}^{adv}$
(14) with weight $\eta>0$ such that

(15) $\underset{\theta_{k},\theta_{k}^{\prime}}{\text{min}}\
\mathcal{L}_{k}^{st\text{-}1}(\theta_{k},\theta_{k}^{\prime};\Theta_{k})\\\
=\mathcal{L}_{k}(\theta_{k};\Theta_{k})+\mathcal{L}_{k}(\theta_{k}^{\prime};\Theta_{k})-\eta\mathcal{L}_{k}^{adv}(\theta_{k},\theta_{k}^{\prime};\Theta_{k}).$

  * •

Training stage-2 is to minimize adversarial loss for learning the
discriminative feature generator. In this stage, we fix the classifiers
$\theta_{k}$ and $\theta_{k}^{\prime}$ and train the generator $\Theta_{k}$ to
minimize the discrepancy of the two classifiers measured by the adversarial
loss. The intuition is shown in Fig. 5(stage-2): by minimizing (14), the
feature generator $\Theta$ is learning to push ambiguous data samples away
from the decision boundaries, so that both classifiers could make the right
predictions and their discrepancy gets reduced. As a result, the feature space
(dashed circles) generated by $\Theta$ is trained to be discriminative which
produces larger inter-class margins.

Formally, we define the objective of stage-2 as a combination of local task
objective $\mathcal{L}_{k}$ and adversarial loss $\mathcal{L}_{k}^{adv}$ with
weight $\lambda>0$ such that

(16) $\underset{\Theta_{k}}{\text{min}}\
\mathcal{L}_{k}^{st\text{-}2}(\Theta_{k};\theta_{k},\theta_{k}^{\prime})\\\
=\mathcal{L}_{k}(\Theta_{k};\theta_{k})+\mathcal{L}_{k}(\Theta_{k};\theta_{k}^{\prime})+\lambda\mathcal{L}_{k}^{adv}(\Theta_{k};\theta_{k},\theta_{k}^{\prime}).$

By training the classifiers and the feature generator in an adversarial
manner, we iteratively optimize the model to learn a discriminative feature
generator which helps boost few-shot learning on unseen tasks.

In our toy example, the feature generator learned by cat and dog images is
likely to distinguish these two categories by eyes and ears and other unique
features of these two species. Thus, cat-like and dog-like images are
projected with large margins in the feature space learned explicitly by our
approach. This space is transferable to tasks such as classifying tiger and
wolf images. In reality, the richer and more representative are the base
classes, the more discriminative the feature space will be. Recently,
centralized machine learning models (Sharif Razavian et al., 2014; Vaswani et
al., 2017) have been shown to be capable of learning generic and versatile
feature spaces on complex structured data such as images and texts. We have
shown that such a feature space can also be efficiently learned in distributed
scenarios.

In conclusion, we have proposed a novel way of learning a discriminative
feature space in FedFSL with an adversarial learning strategy. We summarize
FedFSL-MI-Adv in Algorithm 2.

Input: A set of $K$ federated clients. A local FSL objective $\mathcal{L}_{k}$
for each client $k$.

Output: A global model $w=[\Theta,\theta]$ optimized for FSL task.

Server executes:

Initialize global model $w^{0}=[\Theta^{0},\theta^{0}]$

$t\leftarrow 1$

while  _t $\leq$ maximum rounds $T$_ do

for  _each client $k$ in parallel_ do

$[\Theta^{t}_{k},\theta^{t}_{k}]\leftarrow$
ClientUpdate$([\Theta^{t},\theta^{t}])$

end for

Clients send models $[\Theta^{t}_{1...K},\theta^{t}_{1...K}]$ to server

$[\Theta^{t+1},\theta^{t+1}]\leftarrow\sum_{k=1}^{K}\frac{|\mathcal{B}_{k}|}{|\mathcal{B}|}[\Theta^{t}_{k},\theta^{t}_{k}]$

The server sends $[\Theta^{t+1},\theta^{t+1}]$ back to clients

$t\leftarrow t+1$

end while

Return $[\Theta^{t},\theta^{t}]$

ClientUpdate$([\Theta,\theta])$:

Input: global model from previous round $[\Theta^{t},\theta^{t}]$

Output: updated local model $[\Theta^{t}_{k},\theta^{t}_{k}]$

Sample a batch of episodes
$\mathcal{B}_{k}=\\{\mathcal{T}_{1},...,\mathcal{T}_{n}\\}$

$[\Theta_{k},\theta_{k}]\leftarrow[\Theta^{t},\theta^{t}]$

Initialize a new classifier $\theta_{k}^{\prime}$

$\theta_{k},\theta_{k}^{\prime}\leftarrow$ Solve Eq.(15) with SGD

$\Theta_{k}\leftarrow$ Solve Eq.(16) with SGD

$[\Theta^{t}_{k},\theta^{t}_{k}]\leftarrow[\Theta_{k},\theta_{k}]$

Return $[\Theta^{t}_{k},\theta^{t}_{k}]$

Algorithm 2 FedFSL-MI-Adv algorithm.

##  5\. Experiments and Discussions

We first provide details of the model architecture, parameter settings, and
datasets that we use in the experiments. Then we visualize the decision
boundaries of our approaches with a toy example. We then demonstrate the
performance of our proposed algorithms with two typical few-shot
classification tasks – 5-way 1-shot and 5-way 5-shot – on three benchmark
datasets which cover machine vision and NLP tasks. We will make in-depth
discussions.

###  5.1. Model architecture

We utilize a 12-layer deep neural network (DNN) as our base model (adopted
from ResNet-12 (He et al., 2016) which is commonly used for image
classification tasks (Sun et al., 2019; Oreshkin et al., 2018; Zhang et al.,
2018a)). It consists of the feature extractor with 12 convolutional layers and
about 5 million parameters in total, as well as two fully-connected layers
with a ReLU nonlinearity as the classifier. We update model parameters by Adam
solver (Kingma and Ba, 2015) with a fixed learning rate $10^{-3}$. We set the
adaptation step size $\alpha=0.01$ in (4), the mutual information weight
$\gamma=0.2$, and the stage-1/-2 discrepancy loss weight $\eta=\lambda=0.1$ in
(15) and (16).

###  5.2. Datasets

We briefly describe three benchmark datasets that are commonly used in
studying FSL and FedL (Finn et al., 2017; Munkhdalai and Yu, 2017; Li et al.,
2017; Sun et al., 2019; Chen et al., 2018).

  * •

miniImageNet (Vinyals et al., 2016) is based on a small portion of the full
ImageNet images (Deng et al., 2009). It has 100 classes of images split to
64/16/20 as train/val/test sets. Each class has 600 images with a resolution
of 84 $\times$ 84.

  * •

FC100 (Oreshkin et al., 2018) is based on CIFAR-100 images that has 100
classes split to 60/20/20 as train/val/test splits. Each class has 600 images
with a low resolution 32$\times$32\. It is more challenging because of the low
image quality.

  * •

Sent140 (Caldas et al., 2018) is a benchmark federated learning dataset for
2-way sentiment classification (positive and negative). We sampled from this
dataset 10,000 annotated tweets provided by 310 twitter users and split them
to train/val/test sets with provided tools. Each tweet has 1-20 English words.
We tokenize the sentences and keep only common words which have GloVe
representations (Pennington et al., 2014).

###  5.3. MNIST Example

We provide a simple example on MNIST digit dataset in Fig. (6), to visualize
and compare the decision boundaries of FedFSL-MI and FedFSL-MI-Adv. We
consider the 5-way 1-shot FSL task here: train a digit classification model on
data of digits 0-4, and test its few-shot classification capability on digits
5-9 by observing just one labeled sample per class. To better visualize the
results, we manipulate the feature generator to produce a 2-dim feature for
each input digit sample.

In Fig. 6, we plot the testing data samples from digit class 5-9 by projecting
their features produced by the feature generators. We also depict the decision
boundaries of the classifiers. Data samples of different classes are with
different colors. We observe that FedFSL-MI-Adv (left) produces more
distinguishable decision boundaries than FedFSL-MI (right) as expected. The
two algorithms achieve a few-shot classification accuracy of 87.5% and 83.6%,
respectively. The least accurate class recognized by FedFSL-MI-Adv is digit
’9’ (purple) of 73% correctness rate, with 18% misclassified as ’6’ (orange);
while for FedFSL-MI is digit ’6’ (orange) of 64% correctness rate, with 23%
misclassified as ’8’ (red). This indicates that our adversarial learning
approach proposed in Section 4.2 boosts the FedFSL task by constructing a more
discriminative and transferable feature space for FSL.

Figure 6. Visualization of decision boundaries of FedFSL-KD and FedFSL-KD-Adv
at different epochs.

###  5.4. Results on benchmark datasets

We experiment with our proposed three methods (FedFSL-naive, -MI, -MI-Adv) and
two additional baselines (FSL-local and FedFSL-prox) for comparison.

  * •

FSL-local is a non-distributed baseline of training an individual FSL model
for each client on local data and averaging their results on the shared
testing tasks.

  * •

FedFSL-prox is a variant of FedFSL-naive by adding a weight regularization
term as FedProx (Li et al., 2018) in objective.

We partition the data samples in IID and non-IID ways. For IID partition, data
samples of each class are uniformly distributed to each client. To perform
non-IID partition, we follow (Hsu et al., 2019; Yu et al., 2020) by dividing
data samples to all clients class-by-class with Dirichlet distribution of
concentration parameter $\alpha=1.0$. In Fig.5.4, we show an example of such a
partition of 64 training classes of miniImageNet on a randomly chosen client,
when total device number is 2 to 30.

Figure 7. Non-IID data. Method | Non-IID
---|---
1-shot | 5-shot
FedFSL-local | 59.70% | 66.68%
FedFSL-naive | 68.85% | 70.62%
FedFSL-prox | 70.77% | 72.25%
FedFSL-MI | 70.37% | 73.25%
FedFSL-MI-Adv | 71.35% | 76.00%
Table 2. Sent140 results. Table 3. Results on benchmark datasets.

Method | IID | Non-IID
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
FSL-local | 50.83% | 67.47% | 48.08% | 63.25%
FedFSL-naive (Chen et al., 2018) | 53.00% | 67.63% | 49.95% | 66.11%
FedFSL-prox (Li et al., 2018) | 53.03% | 69.05% | 50.08% | 68.53%
FedFSL-MI (ours) | 54.98% | 69.07% | 51.07% | 68.57%
FedFSL-MI-Adv (ours) | 56.42% | 70.92% | 53.69% | 69.61%
(a) MiniImageNet results.

Method | IID | Non-IID
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
FSL-local | 36.45% | 47.68% | 35.90% | 52.93%
FedFSL-naive (Chen et al., 2018) | 38.42% | 49.97% | 36.11% | 51.58%
FedFSL-prox (Li et al., 2018) | 37.62% | 48.99% | 36.95% | 53.02%
FedFSL-MI (ours) | 39.78% | 50.65% | 38.08% | 52.98%
FedFSL-MI-Adv (ours) | 40.22% | 51.18% | 38.51% | 54.43%
(b) FC100 results.

####  5.4.1. Results on Image Classification

In Table 3(b), we compare our methods on miniImageNet (a) and FC100 (b) with
1-shot / 5-shot tasks learned by a federation of 10 clients. We observe that

  * •

_FedFSL-MI-Adv outperforms others_. For both IID and non-IID case, FedFSL-MI-
Adv consistently outperforms others on both 1-shot and 5-shot tasks for both
datasets.

  * •

For 1-shot and 5-shot IID task on miniImageNet, FedFSL-MI-Adv achieves the
best accuracy of 56.42% and 70.92%, which outperforms the second best FedFSL-
MI by more than 2.6% and 2.7% respectively and relatively. A similar trend has
also been observed for FC100.

  * •

For non-IID task, FedFSL-MI-Adv outperforms the second best FedFSL-MI by more
than 5% and 3.1% in 1-shot case on miniImageNet and FC100 datasets
respectively, and outperforms FedFSL-naive by more than 7.2% and 5.5%
respectively. This indicates that our designed modules indeed help achieve a
better federated model especially for non-IID case.

  * •

For FC100 5-shot task, FedFSL-MI-Adv performs better on non-IID partitions
than IID partitions, with an accuracy 54.43% (non-IID) v.s. 51.18% (IID) shown
by last line in Table 3(b)(b). One explanation is that non-IID partitions
force each client model to learn on distinct local tasks where certain data
classes get sampled more times and thus get represented well. As FedFSL-MI-Adv
further aligns the feature spaces of all clients, we could derive a more
representative joint feature space with the global model.

  * •

Using FedProx (Li et al., 2018) performs no better than our FedFSL-MI. This is
because FedProx directly constrains client model weights to be closer to
global model, while our FedFSL-MI softly optimizes the model outputs of them
to be closer, which makes the training end-to-end and easier to optimize.

Figure 8. FSL accuracy of 1-shot task on MiniImageNet w.r.t. number of devices
in federation.

####  5.4.2. Results on Text Classification

In Table 5.4, we compare our methods on Sent140 dataset with 1-shot / 5-shot
tasks learned by a federation of 5 clients. Following the provided tool of
partitioning the dataset, we distribute different users’ data to each client
without replacement. Since the data distributions vary for users, this
sampling process provides non-IID data partitions. Our goal is to train an
effective global sentiment classification model on one portion of users, which
can be used to detect the sentiment on disjoint new users. This is
particularly challenging because different users can use very distinct words
and exclamations to express feelings, and many of them are rare.

In this task, our backbone model is a GRU (RNN) network with hidden size 128.
We convert tweet sentences to sequences of 300-D GloVe (Pennington et al.,
2014) word vectors as input to the GRU model. We add a binary classifier upon
GRU hidden output to classify negative and positive sentiment. We examine the
performance of baselines and our models and observe that

  * •

FedFSL-MI-Adv outperforms the other approaches in this natural language
understanding task, similar as in image classification task. It also shows
that our FedFSL framework can be applicable to both CNN and RNN models.

  * •

We found that the performance increases from 1-shot to 5-shot tasks are
generally less than image classification tasks, e.g., less than 5% on Sent140
while more than 10% on miniImageNet. This is because the few-shot labelled
sentences can only provide a few more words to help adapt to a user’s emotion,
while images can provide much richer details and patterns of a given object.

####  5.4.3. Different device number

We study the trend of accuracy of FedFSL with different number of
participating devices $K=2,5,10,20,30$, as well as $K=1$ to simulate complete
centralized training. We illustrate the results for non-IID 5-way 1-shot task
with miniImageNet in Fig. 8 with detailed numbers. Note that the more
participating devices, the fewer training samples each device holds. We
observe that

  * •

The overall trend is that more participating devices yielded decreased
accuracy for all 3 approaches. The task becomes more difficult when $K$
increases as the device coordination grows harder and the client model becomes
less capable with less training data.

  * •

_FedFSL-MI-Adv still achieves the best results_ on all cases, leading the
second best FedFSL-MI by more 2-5% relatively.

  * •

The performance of _FedFSL-MI-Adv decreases more slowly than other approaches_
with the increase of $K$, which indicates the beneficial of learning a
consistent feature space over the clients.

  * •

_FedFSL-MI-Adv in 2-device federation works even better than 1-device
centralized training_ , with accuracy 54.96% v.s. 54.41%. Note that the total
training samples of each device get halved on each device when $K=2$. The
surprising result that distributed training outperforms centralized training
can be explained that FSL is aiming to learn with very few training samples,
instead of fitting a task with many samples as in supervised learning.
Therefore, FSL is relative less sensitive to the number of examples in _base_
classes on each client. Moreover, by utilizing our approach to align decision
boundaries well, the two client models form an effective ensemble to enhance
the overall performance, compared with a single-model case.

####  5.4.4. Mutual information with $k$-exclusive global model

In Section 4.1, we discussed that we can estimate the mutual information
produced by $w_{-k}^{t}$ with the global model $w^{t}$ when the number of
devices are large. We conducted experiments to compare using the original
$w_{-k}^{t}$ with $w^{t}$ in (13). Surprisingly, we found that using $w^{t}$
either outperforms or ties with using $w_{-k}^{t}$ as original definition in
both 1-/5-shot tasks in all datasets, with the only exception of miniImageNet
5-shot task (69.61% v.s. 70.70%). We found that this phenomenon could be
explained by“self-knowledge distillation” such that a trained model can be
used to improve itself (Yun et al., 2020; Zhang and Sabuncu, 2020) as it
provides a kind of label smoothing which can make training processes robust to
noises.

##  6\. Conclusion

In this paper, we proposed a framework that makes federated learning effective
in data-scarce scenarios. We designed an adversarial learning strategy to
construct a consistent feature space over the clients, to better learn from
scarce data. Experimental results show that our adversarial learning based
method outperforms baseline methods by 5%$\sim$15% on benchmark datasets.
Future work can investigate the theoretical convergence analysis of FedFSL
with non-convex models and how to further extend FedFSL to regression and
reinforcement learning tasks.

## References

  * (1)
  * Belghazi et al. (2018) Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. 2018\.  Mutual information neural estimation. In _ICML_.
  * Caldas et al. (2018) Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečnỳ, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. 2018.  Leaf: A benchmark for federated settings.  _arXiv preprint arXiv:1812.01097_ (2018).
  * Chen et al. (2018) Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. 2018.  Federated meta-learning with fast convergence and efficient communication.  _arXiv preprint arXiv:1802.07876_ (2018).
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _CVPR_.
  * Dinh et al. (2019) Canh Dinh, Nguyen H Tran, Minh NH Nguyen, Choong Seon Hong, Wei Bao, Albert Zomaya, and Vincent Gramoli. 2019.  Federated learning over wireless networks: Convergence analysis and resource allocation. In _Proc. IEEE INFOCOM’19_.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017\.  Model-agnostic meta-learning for fast adaptation of deep networks.  _ICML_ (2017).
  * Finn et al. (2019) Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. 2019.  Online meta-learning.  _NIPS_ (2019).
  * Ghorbani et al. (2019) Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. 2019\.  An Investigation into Neural Net Optimization via Hessian Eigenvalue Density. In _ICML_.
  * Gidaris and Komodakis (2018) Spyros Gidaris and Nikos Komodakis. 2018.  Dynamic Few-Shot Visual Learning Without Forgetting. In _CVPR_.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets. In _NIPS_.
  * Guha et al. (2019) Neel Guha, Ameet Talwalkar, and Virginia Smith. 2019\.  One-shot federated learning.  _arXiv preprint arXiv:1902.11175_ (2019).
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _CVPR_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the knowledge in a neural network.  _NIPS Deep Learning and Representation Learning Workshop_ (2015).
  * Hsu et al. (2019) Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019.  Measuring the effects of non-identical data distribution for federated visual classification.  _arXiv preprint arXiv:1909.06335_ (2019).
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization. In _ICLR_.
  * Li et al. (2020a) Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. 2020a.  Boosting Few-Shot Learning With Adaptive Margin Loss. In _CVPR_.
  * Li et al. (2020b) Jeffrey Li, Mikhail Khodak, Sebastian Caldas, and Ameet Talwalkar. 2020b.  Differentially private meta-learning. In _ICLR_.
  * Li et al. (2014) Mu Li, Bor-Yiing Su, et al. 2014\.  Scaling distributed machine learning with the parameter server. In _OSDI_.
  * Li et al. (2020c) Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020c.  Federated learning: Challenges, methods, and future directions.  _IEEE Signal Processing Magazine_ (2020).
  * Li et al. (2018) Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2018\.  Federated optimization in heterogeneous networks.  _arXiv preprint arXiv:1812.06127_ (2018).
  * Li et al. (2019) Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2019.  On the convergence of fedavg on non-iid data. In _ICLR_.
  * Li et al. (2017) Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. 2017\.  Meta-sgd: Learning to learn quickly for few-shot learning.  _arXiv preprint arXiv:1707.09835_ (2017).
  * McMahan et al. (2017) H Brendan McMahan, Seth Hampson, et al. 2017\.  Communication-efficient learning of deep networks from decentralized data. In _AISTATS_.
  * Munkhdalai and Yu (2017) Tsendsuren Munkhdalai and Hong Yu. 2017.  Meta networks. In _ICML_.
  * Oreshkin et al. (2018) Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. 2018.  Tadam: Task dependent adaptive metric for improved few-shot learning. In _NIPS_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _EMNLP_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015\.  Faster r-cnn: Towards real-time object detection with region proposal networks. In _NIPS_.
  * Saito et al. (2018) Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. 2018.  Maximum classifier discrepancy for unsupervised domain adaptation. In _CVPR_.
  * Salehkaleybar et al. (2019) Saber Salehkaleybar, Arsalan Sharifnassab, and S Jamaloddin Golestani. 2019.  One-shot federated learning: theoretical limits and algorithms to achieve them.  _arXiv preprint arXiv:1905.04634_ (2019).
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015\.  Facenet: A unified embedding for face recognition and clustering. In _CVPR_.
  * Sharif Razavian et al. (2014) Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014.  CNN features off-the-shelf: an astounding baseline for recognition. In _CVPR_.
  * Shin et al. (2020) MyungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. 2020\.  XOR Mixup: Privacy-Preserving Data Augmentation for One-Shot Federated Learning.  _arXiv preprint arXiv:2006.05148_ (2020).
  * Smith et al. (2017) Virginia Smith, Simone Forte, Chenxin Ma, Martin Takáč, Michael I Jordan, and Martin Jaggi. 2017.  CoCoA: A general framework for communication-efficient distributed optimization.  _JMLR_ (2017).
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017\.  Prototypical networks for few-shot learning. In _NIPS_.
  * Sun et al. (2019) Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. 2019\.  Meta-transfer learning for few-shot learning.  _CVPR_ (2019).
  * Sung et al. (2018) Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018.  Learning to compare: Relation network for few-shot learning.  _CVPR_ (2018).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NIPS_.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016\.  Matching networks for one shot learning. In _NIPS_.
  * Yu et al. (2020) Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. 2020\.  Salvaging federated learning by local adaptation.  _arXiv preprint arXiv:2002.04758_ (2020).
  * Yun et al. (2020) Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. 2020\.  Regularizing class-wise predictions via self-knowledge distillation. In _CVPR_. 13876–13885.
  * Zhang et al. (2018a) Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. 2018a.  Metagan: An adversarial approach to few-shot learning. In _NIPS_.
  * Zhang et al. (2015) Sixin Zhang, Anna E Choromanska, and Yann LeCun. 2015\.  Deep learning with elastic averaging SGD. In _NeurIPS_.
  * Zhang et al. (2020) Xiaoli Zhang, Fengting Li, Zeyu Zhang, Qi Li, Cong Wang, and Jianping Wu. 2020\.  Enabling Execution Assurance of Federated Learning at Untrusted Participants. In _Proc. IEEE INFOCOM’20_.
  * Zhang et al. (2018b) Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018b.  Deep mutual learning. In _CVPR_.
  * Zhang and Sabuncu (2020) Zhilu Zhang and Mert R Sabuncu. 2020.  Self-Distillation as Instance-Specific Label Smoothing.  _arXiv preprint arXiv:2006.05065_ (2020).
  * Zhao et al. (2018) Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. 2018\.  Federated learning with non-iid data.  _arXiv preprint arXiv:1806.00582_ (2018).