# CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated
Learning

Minh N. H. Nguyen, Huy Q. Le, Shashi Raj Pandey, and Choong Seon Hong
Department of Computer Science and Engineering,
Kyung Hee University, Yongin-si 17104, South Korea {minhnhn, quanghuy69,
shashiraj<EMAIL_ADDRESS>M. N. H. Nguyen, H. Q. Le, S. R. Pandey, and C.
S. Hong are with the Department of Computer Science and Engineering, Kyung Hee
University, Yongin-si 17104, South Korea.

###### Abstract

In a practical setting towards better generalization abilities of client
models for realizing robust personalized Federated Learning (FL) systems,
efficient model aggregation methods have been considered as a critical
research objective. It is a challenging issue due to the consequences of
non-i.i.d. properties of client’s data, often referred to as statistical
heterogeneity and small local data samples from the various data
distributions. Therefore, to develop robust generalized global and
personalized models, conventional FL methods need redesigning the knowledge
aggregation from biased local models while considering huge divergence of
learning parameters due to skewed client data. In this work, we demonstrate
that the knowledge transfer mechanism is a de facto technique to achieve these
objectives and develop a novel knowledge distillation-based approach to study
the extent of knowledge transfer between the global model and local models.
Henceforth, our method considers the suitability of transferring the outcome
distribution and (or) the embedding vector of representation from trained
models during cross-device knowledge transfer using a small proxy dataset in
heterogeneous FL. In doing so, we alternatively perform cross-device knowledge
transfer following general formulations as 1) global knowledge transfer and 2)
on-device knowledge transfer. Through simulations on four federated datasets,
we show the proposed method achieves significant speedups and high
personalized performance of local models. Furthermore, the proposed approach
offers a more stable algorithm than FedAvg during the training, with minimal
communication data load when exchanging the trained model’s outcomes and
representation.

##  1 Introduction

Nowadays, myriad mobile applications incorporate on-device AI modules to
utilize the user’s data and facilitate new experiences on their smartphones,
such as AI-powered cameras, Extended Reality (XR) applications, and
intelligent assistants Samsung (2021). However, data privacy concerns have
been an Achille’s heels in the centralized machine learning (ML) paradigm.
Thus, in recent years, there’s been a sudden surge in developing federated
learning (FL) frameworks that’s able to limit data privacy leakage Kairouz et
al. (2019); McMahan et al. (2017); Lim et al. (2020). In FL, a set of devices
are connected with a central node (edge/cloud server) to collaboratively train
a learning model without sharing their local data Kairouz et al. (2019). In
doing so, the FL schemes require to share only the learning model parameters
to the server; hence, the user data privacy can be kept intact. However, such
a federated training scheme raises concerns about the heterogeneity of
computing devices and distributed data samples’ properties leading to poor
learning performance and fluctuation of the trained models across devices. To
fill this research gap, recent FL algorithm designs are departing from
heterogeneous FL settings towards personalized FL, which considers the
performance of the global model and the personalized client models. In
heterogeneous FL, the underlying data distribution is not identical across
devices, i.e., non-i.i.d. Hence, the solution to the global empirical risk
minimization problem, i.e., the trained global model, will not personalize
well for each client. Meanwhile, the local training process in FL algorithms
with their small private datasets often results in biased client models that
produce high accuracy on the local data but with low generalization
capabilities. Also, the client model parameters and data features are diverse
across devices. Such as in FedAvg McMahan et al. (2017), the aggregation of
model parameters based on the conventional coordinates-based averaging scheme
from highly skewed data could slow down the learning process due to extensive
divergence of model parameters to be averaged Yu et al. (2020).

In this paper, we redesign knowledge transfer mechanisms in FL and examine the
underlying fundamental correlation between learning performances of FL
algorithms. To that end, we show the proposed method achieves significant
speedups and obtains better personalized performance of local models in the
fixed users scenario and the subset of users selection scenario. Furthermore,
we show our approach offers better stability than the FedAvg algorithm and
tackles privacy leakage issues with minimal communication data load when
exchanging the trained model’s outcomes and (or) representation solely. In the
following subsection, we first outline related approaches to handle the
heterogeneity issues in FL literature.

Structural alignment in heterogeneous FL: One of the promising research
directions to resolve unmatched parameters of learning models in FL is using
the structural alignment methods Mostafa (2019); Wang et al. (2020); Yu et al.
(2020). The representation matching in Mostafa (2019) and “permutation
invariance" in FedMA Wang et al. (2020) are proposed to evaluate parameter
similarity across local models and reorganize the permutation accordingly via
matching schemes; thus, obtaining parameters that could be averaged together
in the model aggregation. In Yu et al. (2020), the authors proposed structure-
information alignment based on the feature-oriented interpretation and
regulation method to ensure explicit feature information allocation in
different neural network structures. Based on the feature interpretation, the
proposed regulation method adjusts the local model architecture according to
their data and task distribution at the very early training stage and
continuously maintains structure and information alignment. In this manner,
structural alignment methods showed their efficiency in improving the FL
performance and applicability in various heterogeneous settings.

Common Representation Learning: This is a popular method to handle the uneven
distributions and representations of different modalities or tasks Bengio et
al. (2013); Huang et al. (2017); Peng and Qi (2019); Xu et al. (2020). In this
approach, the training of shared features across tasks/modalities constructs a
common representation; thereby, cross-modal common representational learning
allows knowledge transfer across different modalities Bengio et al. (2013). In
doing so, the heterogeneous data from different modalities resulting in
diverse representations are projected into a common space. Then, the cross-
modal knowledge transfer can be performed by reducing the discrepancy between
representations of pairwise cross-modal data from high-level layers Huang et
al. (2017). Furthermore, it can preserve the relevant semantics and allow
heterogeneous data from different modalities to be correlated more easily Peng
and Qi (2019); Xu et al. (2020). Recently, a similar design, the FedRep
algorithm Collins et al. (2021), was applied in FL that alternatively trains
the shared low-dimensional representation, which is aggregated across clients
by averaging and private local heads to improve the learning performance of
client models.

Figure 1: Cross-Device knowledge transfer scheme in FL.

The convergence of the alignment methods and knowledge distillation: Even
though the alignment approaches are widely studied in the representation
learning literature, the knowledge distillation (KD) approach has emerged as
one of the promising alignment approaches. KD approach demonstrated superior
performance via knowledge transfer using outcome distribution (i.e., soft
targets) from the teacher to student models Hinton et al. (2015). The
alignment of outcome distribution using Kullback–Leibler (KL) divergence helps
the student model imitate and learn from a teacher model or ensemble of
models. KD was first introduced in Buciluǎ et al. (2006); Ba and Caruana
(2014) and then developed by Hinton et al. (2015) to compress the smaller
learning model from a large model. In KD, the soft targets $z_{t}$ and $z_{s}$
are the softmax outputs of the student and teacher model, respectively. The
loss function of the student model between the prediction $z_{s}$ and ground-
truth label $y_{s}$ is the cross-entropy loss denoted as $\ell_{CE}$. In
addition, the distillation loss using the KL divergence is defined as

$\ell_{KD}=\tau^{2}KL(z_{t},z_{s}),$ (1)

where $\tau$ is the temperature parameter used for softening the output
values. The higher the value of the temperature, the softer the outputs of
learning models. Therefore, the loss function is used to train the student
model is defined as

$\ell_{\textrm{student}}=\ell_{CE}+\alpha\ell_{KD},$ (2)

where $\alpha$ is the hyper-parameter that control the penalty KD loss.

In Zhang et al. (2018), the authors introduced a deep mutual learning strategy
with an ensemble of student models to collaboratively learn and teach each
other throughout the training process. Furthermore, KD was recently adopted to
transfer the knowledge between the global and local models and exhibited
promising results in FL. The alignment of outcome distribution with KL
divergence in KD helps the global model and local models to transfer their
learning knowledge and further improve the performance of the learning models.
To overcome communication cost and heterogeneity issues, the authors in Jeong
et al. (2018) applied KD with the average logits from clients using their
private datasets, and balancing the private datasets with data augmentation
can further improve the learning performance. The works Li and Wang (2019);
Bistritz et al. (2020) leveraged a large public dataset in the local knowledge
distillation using the aggregation/consensus of soft targets from clients as
the teacher knowledge. In Lin et al. (2020), the authors proposed ensemble
distillation for global model fusion enabling different sizes, numerical
precision, or structure of learning models. It is possible to apply KD in the
global and local model updates, as shown in He et al. (2020), but the server
requires sequentially distilling the knowledge from each client and vice
versa. However, the authors do not consider the proxy dataset as well as the
knowledge aggregation.

Inspired by the discussion on alignment methods and knowledge distillation
approaches, it is fundamental to find the answer to the following research
question: “what transferable knowledge and regularizers should be used in
knowledge transfer between the global model and local models in FL?" Answering
this question opens opportunities to provide more efficient knowledge transfer
mechanisms in FL.

Contributions: In the typical FL algorithms such as FedAvg McMahan et al.
(2017), FedProx Li et al. (2020), the global model is updated via model
aggregation (i.e., parameters averaging), which is then broadcast back to
clients and used as the initial model for local learning. However, such
approaches ignore the fundamentals of interleaved properties of data across
clients; thus, resulting in poor generalization. To address this research gap,
in this work, we propose cross-device knowledge transfer (CDKT) with two
mechanisms: 1) global knowledge transfer to transfer the knowledge from client
models to the global model, 2) on-device knowledge transfer to transfer the
generalized knowledge of the global model to the client models. In doing so,
we study the knowledge transfer mechanism using generic distance regularizers
to align the knowledge between the models. Unlike knowledge distillation
approaches, the transferable knowledge is the outcomes (i.e., activation
results $z$ of the final layer) and (or) embedding representation features
(i.e., activation results $e$ of the intermediate layer) given the proxy
samples as shown in Fig. 1. We assume this proxy data is of small size and
available by clients to follow a practical scenario where the system can pre-
collect a small amount of labeled data at the beginning.

Our experiments reveal that (i) the outcome distribution and embedding
features’ representation of the proxy data shared by all devices and servers
are transferable and (ii) quickly improves the global model performance and
personalized performance of the client models. Different from many existing
works, we evaluate the personalized performance of local models with
specialization (i.e., performance on local testing data), generalization
capabilities (i.e., cross-device testing performance). These evaluation
metrics provide more insights into the behaviors of the heterogeneous FL
systems with different transferring methods and learning settings. The
proposed CDKT-FL algorithm enables the flexibility of heterogeneous learning
model design and knowledge transfer mechanism for future personalized FL
applications. CDKT also improves stability and faster convergence while
requiring fewer communication data with outcome and representation knowledge
than full complex model parameters compared to the FedAvg algorithm. These
observations provide solid motivation to study knowledge transfer approaches
and related metrics in FL.

##  2 Cross-Device Knowledge Transfer in Federated Learning

In a general scheme of FL algorithms Reddi et al. (2020); Li et al. (2021),
the global model is updated via a gradient-based optimizer or model parameter
aggregation in the server, while the local/personalized model in user device
$n$ is updated using their private dataset $D_{n}$. We first define the neural
network with the prediction outcomes (i.e., $z=h\circ w(x)$), where $h$ is the
projection head, which is a small neural network (i.e., the last fully-
connected layers; a small classifier). We denote the embedding features of the
representation sub-net parameters $w$ for the image $x$ as $e=w(x)$. Hence,
the learning model parameters $w,h$ can be trained and used to extract the
embedding features (i.e., $e_{n}$) and (or) classes’ prediction distribution
(i.e., $z_{n}$) of input images at device $n$. In this work, instead of using
model parameters from clients in every round to aggregate into the global
model, cross-device knowledge transfer is being used to transfer the
collective knowledge from all clients to the server and construct a
generalized model. Note that the global model can be updated using an
additional regularizer to align its knowledge with the collective knowledge of
clients. In doing so, it is necessary to have a fixed small proxy data $D_{r}$
such that all models can access and provide their learning knowledge as the
outcomes (i.e., $z$) and (or) embedding features (i.e., $e$), as shown in Fig.
1. In addition, the knowledge transfer also helps the global model transfer
its generalization capability to all client models. This way, firstly, we can
reduce communication data load and also further enhance users’ data privacy,
which is often compromised during full parameters exchange against reverse
engineering tricks. Secondly, since the knowledge transfer is sufficient to
transfer learning capabilities, the learning model architectures can differ
between the server model (with more layers) and client models (with fewer
layers). Towards that end, in Fig. 1, we illustrate the proposed scheme of
CDKT-FL.

The proposed CDKT-FL algorithm (Alg. 1) enables the transfer of knowledge
between clients and global models in the FL setting. At each global round $t$,
the server broadcasts the knowledge from the updated global model to all
clients. Then, each client performs $K$ local epochs with its private and
proxy dataset based on the gradient of the loss function in the on-device
learning problem (8), with the local learning rate $\eta$. Note that we use
mini-batch of private data in the cross-entropy loss, whereas proxy data in
the on-device CDKT regularizer. Then, the clients send their latest knowledge
(i.e., $z_{n}$ and (or) $e_{n}$) to the server. At the global level, the
server builds the generalized model by solving the global learning problem
(5). For this, we use a gradient-based update for $R$ global epochs with the
global learning rate $\gamma$. In particular, to perform CDKT-FL, we design
the knowledge transfer in FL with two general problems: generalized model
construction and on-device learning in the following subsections.

###  2.1 Generalized Model Construction

In the generalized model construction problem, the main target is utilizing
the proxy data (i.e., $D_{r}$) and collective knowledge from all devices to
build the global model. If the transferable knowledge is only the outcome of
client models, then the problem is in the form of knowledge distillation (KD)
Hinton et al. (2015). Similar to the design of Li and Wang (2019), we consider
the averaged results of the outcomes as the collective knowledge that results
in the equivalent performance of the learning problem. This approach, however,
requires less computation than the sum of individual KL loss between the
student model (global model) and each teacher model (client model) in ensemble
KD. In the following, we review the KD method and discuss the proposed cross-
device knowledge transfer method.

  * •

KD Method:

${\ell}_{s}(D_{r})={\ell}_{CE}(z_{s}|D_{r})+\beta\,{\ell}_{KD}\bigg{(}z_{s},\sum_{n\in{N}}{\frac{1}{N}}z_{n}|D_{r}\bigg{)},$
(3)

where $\beta$ is the parameter to control the trade-off between proxy data
loss and the KD regularizer; $z$ are the soft targets of the learning models.

  * •

Cross-device knowledge transfer Method (CDKT): We formulate the general
generalized model construction problem with the global CDKT regularizer using
a generic function $d$. Thereby, the proposed formulation measures the
discrepancy between the knowledge of $n$ client models ($g_{1},\dots,g_{n}$)
and the knowledge of server model $g_{s}$ as

${\ell}_{s}(D_{r})={\ell}_{CE}(z_{s}|D_{r})+\beta\,d\big{(}g_{s},g_{1},\dots,g_{n}|D_{r}\big{)}.$
(4)

The collective knowledge from clients are constructed as the averaged results
of the outcomes $z$ and (or) embedding features $e$. To improve the limitation
in transferred knowledge from clients, we adjust the collective outcomes $z$
of clients by averaging it with the true labels $y_{r}$ when using their final
outcomes.

$\displaystyle{\ell}_{s}(D_{r})$
$\displaystyle={\ell}_{CE}(z_{s}|D_{r})+\beta\,d\bigg{(}e_{s},\sum_{n\in{N}}{\frac{1}{N}}e_{n}|D_{r}\bigg{)}$
$\displaystyle+\beta\,d\bigg{(}z_{s},\lambda
y_{r}+(1-\lambda)\sum_{n\in{N}}{\frac{1}{N}}z_{n}|D_{r}\bigg{)},$ (5)

where $\lambda\in[0,1]$, and the parameter $\beta$ allows the server to
control the trade-off between proxy data loss and the global CDKT regularizer.
Here, we note that various distance functions $d$ can be used for knowledge
transfer, such as Norm, KL divergence, and Jensen-Shannon (JS) divergence
Kullback (1997). We provide the concrete mathematical formulations of these
distances in our supplementary materials.

Algorithm 1 CDKT-FL Algorithm

1:Input: $T,K,R$

2:for $t=0,\dots,T-1$ do

3: On-Device Learning: Each device $n$ receives the generalized knowledge
$\boldsymbol{g}_{s}$ from the server and updates local model with $D_{n}$,
$D_{r}$;

4: for $k=0,\dots,K-1$ do

5: We loop for each batch of private and proxy data (i.e., $S_{n}$ and
$S_{r}$): $\displaystyle{w}^{t}_{n}=$
$\displaystyle{w}^{t}_{n}-\eta\nabla\ell_{c}^{n}(S_{n},S_{r});$ (6)

6: end for

7: Device $n$ extracts and sends the local knowledge $\boldsymbol{g}_{n}$
given the proxy dataset to the server;

8: Generalized Model Construction: Server updates the global model with the
received local knowledge from the selected devices:

9: for $r=0,\dots,R-1$ do

10: We loop through batches of proxy data (i.e., $S_{r}$):
$\displaystyle{w}^{t}_{g}=$
$\displaystyle{w}^{t}_{g}-\gamma\nabla\ell_{s}({S}_{r})$ (7)

11: end for

12:end for

###  2.2 On-device Learning

Using a similar design, the on-device learning problem helps the client models
to improve their generalization capabilities by imitating the generalized
knowledge from the global model. In this approach, the client $n$ utilizes the
private dataset $D_{n}$ and also the generalized knowledge from the global
model obtained using the proxy dataset $D_{r}$. In particular, we formulate
the on-device loss function for each device $n$ as follows:

$\displaystyle{\ell}^{n}_{c}(D_{n},D_{r})$
$\displaystyle={\ell}^{n}_{CE}(z_{n}|D_{n})+\alpha\,d(g_{n},g_{s}|D_{r})$
$\displaystyle={\ell}^{n}_{CE}(z_{n}|D_{n})+\alpha\,d(e_{n},e_{s}|D_{r})$
$\displaystyle+\alpha\,d(z_{n},\lambda y_{s}+(1-\lambda)z_{s}|D_{r})$ (8)

where $\alpha$ is the parameter to control the trade-off between the local
learning objective’s biased knowledge and generalization capabilities.
Further, the on-device CDKT regularizer helps to reduce the bias and over-
fitting issues of local models when training on the small private dataset.

Table 1: The median of testing accuracy on four datasets from $90$ to $100$ rounds | Fashion-MNIST | MNIST | CIFAR-10 | CIFAR-100
---|---|---|---|---
Global | C-Per | Global | C-Per | Global | C-Per | Global | C-Per
Fixed Users | No Transfer | 83.72 | 59.09 | 81.14 | 59.70 | 55.92 | 50.43 | 17.27 | 18.76
FedAvg | 82.56 | 76.41 | 90.12 | 87.20 | 59.85 | 52.95 | 15.53 | 17.13
CDKT (Rep) | 84.88 | 86.06 | 86.22 | 90.09 | 58.44 | 59.25 | 17.75 | 21.81
CDKT (Full) | 85.81 | 81.66 | 88.02 | 85.70 | 58.91 | 55.83 | 17.84 | 21.93
CDKT (RepFull) | 86.51 | 84.08 | 88.32 | 87.25 | 59.09 | 57.74 | 18.49 | 22.22
Subset of Users | No Transfer | 78.19 | 58.98 | 80.50 | 58.06 | 52.17 | 46.75 | 16.47 | 8.13
FedAvg | 69.44 | 70.62 | 88.03 | 89.18 | 48.83 | 47.20 | 12.25 | 12.57
CDKT (Rep) | 78.50 | 80.90 | 84.75 | 84.61 | 55.67 | 53.92 | 16.72 | 15.74
CDKT (Full) | 79.22 | 79.02 | 85.91 | 79.32 | 56.60 | 54.35 | 17.33 | 15.76
CDKT (RepFull) | 79.94 | 81.39 | 84.62 | 81.82 | 56.57 | 54.03 | 17.60 | 17.74

(a) Fixed Users scenario

(b) Subset of Users scenario

Figure 2: Comparison of CDKT-FL with different settings versus FedAvg and No
Transfer with Fashion-MNIST dataset

##  3 Experiments

###  3.1 Setting

In this section, we validate the efficacy of the CDKT-FL algorithm with the
MNIST LeCun et al. (1998), Fashion-MNIST Xiao et al. (2017), CIFAR-10, and
CIFAR-100 Krizhevsky et al. (2009) datasets for handwritten digits, fashion
images, and objects recognition, respectively. We conduct the experiments with
$10$ selected clients in each round, where each client has median numbers of
data samples at $63$, $70.5$, $966.5$, and $1163.5$ with MNIST, Fashion-MNIST,
CIFAR-10, and CIFAR-100 respectively. To simulate the properties of non-i.i.d
data distribution, we set each client’s data from random $20$ classes in $100$
classes for CIFAR-100 dataset and only two classes in total $10$ classes with
the other datasets, and use $20\%$ of private data samples for evaluating the
testing performance. Thereby, the private datasets are unbalanced and have a
small number of training samples. Further, the small proxy datasets have all
classes data with, respectively, $355$, $330$, $4200$, and $4200$ samples for
MNIST, Fashion-MNIST, and CIFAR-10 datasets. The global CNN model in the
server consists of two convolution layers in MNIST and Fashion-MNIST dataset,
whereas three convolution layers are used in the CIFAR-10 and CIFAR-100
dataset, followed by the pooling and fully connected layers. The client CNN
models have a similar design or one CNN layer fewer than the global model one
in the heterogeneous model setting. The learning parameters such as learning
rates, trade-off parameters are tuned to achieve good results for different
settings of algorithms; the number of local epochs $K=2$, global epochs $R=2$,
and the batch size is $20$.

Evaluation metrics: To the best of our knowledge, FL approaches focus more on
the learning performance of the global model while personalized FL algorithms
evaluate the learning performance of the local models. In this work, to better
capture the learning behavior of FL algorithms, we conduct evaluations with
the specialization (C-Spec) and generalization (C-Gen) performance of client
models on average, defined by the accuracy in their local testing data only,
and the collective testing data from all clients, respectively. To summarize
the performance of local models, we calculate the average of C-Gen and C-Spec
performance as the C-Per metric. Meanwhile, the Global metric is the accuracy
of the global model for the collective testing data from all clients.

Baselines and CDKT settings: For comparison, we evaluate different settings of
the CDKT-FL algorithm and compared them with the FedAvg algorithm, no transfer
knowledge scheme. Accordingly, the setting Rep, Full, and RepFull stand for
using only the embedding features of representation, outcomes of full models,
and both in the global knowledge transfer, respectively. Furthermore,
different distance functions were also validated, such as KL, JS, and Norm-2
(N). The setting KL-N indicates the use of KL divergence in the global CDKT
regularizer, while the on-device CDKT regularizer uses Norm-2.

###  3.2 Experimental Results

In summary, we obtained experimental results on four datasets for different
settings in the CDKT-FL algorithm and baselines with the median testing
accuracy from round $90$ to $100$, as in Table 1. There are two principal
scenarios: (1) fixed $10$ users, and (2) selected $10$ from the total $50$
users. Compared to the FedAvg algorithm, CDKT-FL outperforms $10\%$ with
Fashion-MNIST, $7\%$ with CIFAR-10, $5\%$ for both scenarios according to
C-Per performance. Meanwhile in MNIST, CDKT-FL obtains $3\%$ better in
scenario (1) but $4\%$ lower C-Per performance than FedAvg in scenario (2). In
addition to the main target to improve personalized performance, CDKT-FL also
significantly improves the performance of global model in most of the settings
except MNIST dataset. According our detailed figures, in scenario (1), CDKT-FL
obtains higher C-Gen performance, a comparable C-Spec performance of client
models, and a comparable or higher Global performance. In scenario (2), our
proposed algorithm achieves higher C-Spec and Global performance with the
Fashion-MNIST and CIFAR datasets, but FedAvg exceptionally exhibits a good
performance on a simple learning task such as hand digit recognition on MNIST
dataset. In principle, CDKT helps the client models improve C-Gen and slightly
improve the Global performance than that of no transfer setting. The detailed
results for MNIST and CIFAR-10 datasets are shown in our supplementary
material. In addition, as the final performance metric, CDKT-FL algorithm
exhibits better stability and faster convergence in general while improving
the C-Per, and preserves high Global performance in most of the settings.
These attributes also enable the client or global models to be usable early
after $50$ communication rounds. As expected, in the subset of users selection
scenario, the FedAvg algorithm shows an extensive fluctuation due to
irregularity in learning parameters across the client models stemmed from
statistical heterogeneity.

In Fig. 2, we evaluate different settings knowledge transfer schemes of CDKT-
FL to compare with No transfer and FedAvg in both of the scenarios on Fashion-
MNIST dataset. In scenario (1), as shown in Fig. 2(a), experimental results
demonstrate that our proposed schemes outperform the baselines in terms of
speedups C-Gen and Global performance. The Rep+KL-N setting achieves the
highest C-Gen performance with $75.37\%$ but slightly lower C-Spec than that
of FedAvg. In scenario (2), Fig. 2(b) shows that the CDKT-FL algorithms with
three settings significantly improve C-Spec, Global and more stable than
FedAvg algorithm.

In the supplemental material, we evaluate the effects of the trade-off
parameters used to control the compensation between learning loss and the CDKT
regularizer. As a result, when increasing $\alpha$, we can boost C-Gen
performance; however, that slightly degrades the C-Spec performance. Such
learning behavior shows the interesting compensation effect between C-Gen and
C-Spec performances, such as till some threshold the specialized capability
may decrease as increasing the generalized capability of client models. On the
other hand, $\beta$ does not show significant effects on the performance,
however, when $\beta$ is too small, it makes the global performance is
fluctuated due to less contribution from CDKT regularizers and transferred
knowledge. Finally, we evaluate the effectiveness of the CDKT-FL algorithm
using heterogeneous local models, i.e., using one CNN layer less than global
model. The CDKT-FL algorithm with smaller client models achieves comparable
performance in Fashion-MNIST dataset but slight degradation in the other
datasets.

##  4 Conclusion

This work bridges the gap between existing alignment methods and knowledge
distillation to develop a general form of cross-device knowledge transfer in
FL. Numerous experiments, settings, regularizers, and scenarios are validated
to study the performance of client and global models for a better
understanding of their behaviors in heterogeneous FL. The detailed results
following extensive simulations on four datasets have shown the advantages of
the proposed algorithms. Compared to the FedAvg algorithm, we demonstrate that
the CDKT-FL algorithm is more stable and can speed up the global model
performance, higher personalized performance of client models. In addition to
the learning performance, the proposed algorithm enables (i) the flexibility
of heterogeneous model architectures, (ii) reduces communication data with the
outcome and (or) representation knowledge instead of full model parameters
compared to the conventional FL algorithms, and (iii) tackles privacy leakage
issues (via reverse engineering techniques) of current FL algorithms. Finally,
we advocate the knowledge transfer method works as a hybrid design with recent
advanced techniques to improve learning performance.

## References

  * Ba and Caruana [2014] Jimmy Ba and Rich Caruana.  Do deep nets really need to be deep?  In Advances in Neural Information Processing Systems, volume 27, 2014.
  * Bengio et al. [2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013.
  * Bistritz et al. [2020] Ilai Bistritz, Ariana Mann, and Nicholas Bambos.  Distributed distillation for on-device learning.  In Advances in Neural Information Processing Systems, volume 33, pages 22593–22604, 2020.
  * Buciluǎ et al. [2006] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil.  Model compression.  In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541, 2006.
  * Collins et al. [2021] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai.  Exploiting shared representations for personalized federated learning.  In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 2089–2099, 18–24 Jul 2021.
  * He et al. [2020] Chaoyang He, Murali Annavaram, and Salman Avestimehr.  Group knowledge transfer: Federated learning of large cnns at the edge.  In Advances in Neural Information Processing Systems, volume 33, 2020.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  In NIPS Deep Learning and Representation Learning Workshop, 2015\.
  * Huang et al. [2017] Xin Huang, Yuxin Peng, and Mingkuan Yuan.  Cross-modal common representation learning by hybrid transfer network.  In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 1893–1900, 2017.
  * Jeong et al. [2018] Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim.  Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data.  arXiv:1811.11479, 2018.
  * Kairouz et al. [2019] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin BhagCanhoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al.  Advances and open problems in federated learning.  arXiv:1912.04977, 2019.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kullback [1997] Solomon Kullback.  Information theory and statistics.  Courier Corporation, 1997.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * Li and Wang [2019] Daliang Li and Junpu Wang.  Fedmd: Heterogenous federated learning via model distillation.  In The 2nd International Workshop on Federated Learning for User Privacy and Data Confidentiality, 2019.
  * Li et al. [2020] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks.  In Proceedings of Machine Learning and Systems 2020, pages 429–450. 2020.
  * Li et al. [2021] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith.  Ditto: Fair and robust federated learning through personalization.  In International Conference on Machine Learning, pages 6357–6368. PMLR, 2021.
  * Lim et al. [2020] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao.  Federated learning in mobile edge networks: A comprehensive survey.  IEEE Communications Surveys & Tutorials, 22(3):2031–2063, 2020\.
  * Lin et al. [2020] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi.  Ensemble distillation for robust model fusion in federated learning.  In Advances in Neural Information Processing Systems, volume 33, pages 2351–2363, 2020.
  * McMahan et al. [2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial Intelligence and Statistics, pages 1273–1282, 2017\.
  * Mostafa [2019] Hesham Mostafa.  Robust federated learning through representation matching and adaptive hyper-parameters.  arXiv:1912.13075, 2019.
  * Peng and Qi [2019] Yuxin Peng and Jinwei Qi.  Cm-gans: Cross-modal generative adversarial networks for common representation learning.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 15(1):1–24, 2019.
  * Reddi et al. [2020] Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and Hugh Brendan McMahan.  Adaptive federated optimization.  In International Conference on Learning Representations, 2020.
  * Samsung [2021] Samsung.  Smarter future powered by ai.  samsung.com/semiconductor/minisite/exynos/technology/ai/, 2021.
  * Wang et al. [2020] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni.  Federated learning with matched averaging.  In International Conference on Learning Representations, 2020.
  * Xiao et al. [2017] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv:1708.07747, 2017.
  * Xu et al. [2020] Xing Xu, Kaiyi Lin, Lianli Gao, Huimin Lu, Heng Tao Shen, and Xuelong Li.  Learning cross-modal common representations by private-shared subspaces separation.  IEEE Transactions on Cybernetics, 2020.
  * Yu et al. [2020] Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen.  Heterogeneous federated learning.  arXiv:2008.06767, 2020.
  * Zhang et al. [2018] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu.  Deep mutual learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4320–4328, 2018.

##  5 Supplemental Results

In this material, we added Table 1, 2 for the detailed C-Gen and C-Spec of
algorithms with four datasets. For each dataset, we also use the box plots to
illustrate the fluctuation of algorithm from $90$ to $100$ rounds. As can be
seen, our algorithm obtains a stable convergence with better performance while
FedAvg depicts the extensive fluctuation, especially in the subset of users
scenario.

###  5.1 Cross Entropy Loss:

${\ell}_{CE}(z|D_{r})=-\sum_{x\in D_{r}}\sum_{c}P(y_{c}|x)\ln P(z_{c}|x),$

where $P(z_{c}|x)$ is the prediction outcome probability and $P(y_{c}|x)$ is
the target label of class $c$ given data sample $x$.

###  5.2 Distance for the CDKT regularizers:

A generic distance function $d$ in CDKT regularizer can be adopted with three
following distance metrics

  * •

Norm2 distance:

$d_{norm}(g_{n},g_{s}|D_{r})=\sum_{x\in D_{r}}\|g_{n}(x)-g_{s}(x)\|_{2},$

where $g_{n}(x)$ is the prediction outcome (or embedding features) of client
$n$ and $g_{s}(x)$ is the knowledge from global model given data sample $x$.

  * •

Kullback–Leibler (KL) divergence:

$d_{KL}(g_{n},g_{s}|D_{r})=\sum_{x\in
D_{r}}\sum_{c}P(g_{s,c}|x)\frac{P(g_{s,c}|x)}{P(g_{n,c}|x)},$

where $P(g_{n,c}|x)$ is the prediction outcome (or embedding features) of
client $n$ and $P(g_{s,c}|x)$ is the knowledge from global model of class $c$
given data sample $x$.

  * •

Jashon-Shannon (JS) divergence:

$d_{JS}(g_{n},g_{s}|D_{r})=\frac{1}{2}d_{KL}(g_{n},g_{m}|D_{r})+\frac{1}{2}d_{KL}(g_{s},g_{m}|D_{r})$

where $g_{m}=\frac{1}{2}(g_{n}+g_{s})$.

###  5.3 Additional Fashion-MNIST results

For Fashion-MNIST, we evaluate the effect of trade-off parameters $\alpha$ and
$\beta$ as shown in Fig. 4, 5. We also evaluate the effect of different
distance metrics in CDKT-regularizers in both scenarios in Fig. 6. We observe
that the setting KL-N obtains stable and higher performance among other
metrics. In the KL-N setting, KL divergence is used in the global CDKT
regularizer, while the on-device CDKT regularizer uses Norm.

###  5.4 MNIST results

Similar to Fashion-MNIST, we also provide the detailed figures in this
subsection. In Fig. 8, we evaluate the different transferred knowledge with
CDKT-FL algorithm and compare with FedAvg and No transfer in both scenarios.
In scenario (1), our algorithm depicts the fast convergence on C-Gen
performance after just $30$ rounds while FedAvg requires nearly $80$ rounds to
reach stability and comparable performance. Regarding the subset of users
scenario, our algorithm obtains the better C-Spec, global performance and
lower C-Gen performance than FedAvg. Nevertheless, FedAvg experiences an
extensive fluctuation despite the high C-Gen performance. Similar to the
Fashion-MNIST, the results indicate that the setting KL-N achieves the best
performance among others metrics in both scenarios as shown in Fig. 9.
Finally, we observe that our method obtains the comparable performance when
using heterogeneous models in Fig. 10 with just a slight degradation in fixed
users scenario.

###  5.5 CIFAR-10 and CIFAR-100 results

Similar to the learning behaviors with the Fashion-MNIST dataset in the main
manuscript, CIFAR-10 and CIFAR-100 exhibit faster convergence of the Global
and C-Gen performance and its stability with different transferred knowledge
as shown in Fig. 12 and Fig. 15, respectively, in scenario (1). Regarding
scenario (2), FedAvg depicts the slightly higher C-Gen performance, meanwhile
CDKT-FL outperforms FedAvg in terms of Global and C-Spec performance. CDKT-FL
shows a slight degradation in the last $20$ global rounds with the highest
C-Gen performance. Also, FedAvg extensively fluctuates since the learned
models are highly different between clients. Moreover, as shown in Fig. 13 and
Fig. 16, using smaller learning models at clients with two CNN layers while
three CNN layers in the server model shows a slight degradation due to lower
learning capacities in both datasets.

Table 2: The median of testing accuracy on four datasets from $90$ to $100$ rounds in fixed users scenario | Fashion-MNIST | MNIST | CIFAR-10 | CIFAR-100
---|---|---|---|---
Global | C-Gen | C-Spe | Global | C-Gen | C-Spe | Global | C-Gen | C-Spe | Global | C-Gen | C-Spe
No Transfer | 83.72 | 19.58 | 98.60 | 81.14 | 21.80 | 97.60 | 55.92 | 22.74 | 78.11 | 17.27 | 8.29 | 29.23
FedAvg | 82.56 | 53.05 | 98.14 | 90.12 | 76.30 | 98.20 | 59.85 | 26.24 | 79.66 | 15.53 | 7.58 | 26.67
| CDKT
---
(Rep)
84.88 | 75.37 | 96.74 | 86.22 | 81.98 | 98.20 | 58.44 | 40.59 | 77.91 | 17.75 | 12.27 | 31.34
| CDKT
---
(Full)
85.81 | 65.19 | 98.14 | 88.02 | 73.20 | 98.20 | 58.91 | 37.93 | 73.74 | 17.84 | 11.06 | 32.80
| CDKT
---
(RepFull)
86.51 | 69.70 | 98.37 | 88.32 | 76.29 | 98.20 | 59.09 | 39.88 | 75.59 | 18.49 | 11.69 | 32.76
Table 3: The median of testing accuracy on four datasets from $90$ to $100$ rounds in subset of users scenario | Fashion-MNIST | MNIST | CIFAR-10 | CIFAR-100
---|---|---|---|---
Global | C-Gen | C-Spe | Global | C-Gen | C-Spe | Global | C-Gen | C-Spe | Global | C-Gen | C-Spe
No Transfer | 78.19 | 19.09 | 98.86 | 80.50 | 18.68 | 97.45 | 52.17 | 17.14 | 76.35 | 16.47 | 3.04 | 13.22
FedAvg | 69.44 | 65.33 | 75.90 | 88.03 | 86.06 | 92.30 | 48.83 | 41.69 | 52.70 | 12.25 | 10.81 | 14.34
| CDKT
---
(Rep)
78.50 | 69.70 | 92.10 | 84.75 | 72.38 | 96.85 | 55.67 | 38.33 | 69.52 | 16.72 | 7.41 | 24.07
| CDKT
---
(Full)
79.22 | 62.86 | 95.17 | 85.91 | 62.81 | 95.83 | 56.60 | 38.02 | 70.69 | 17.33 | 7.76 | 23.76
| CDKT
---
(RepFull)
79.94 | 67.82 | 94.96 | 84.62 | 66.84 | 96.81 | 56.57 | 38.85 | 69.22 | 17.60 | 10.32 | 25.15

(a) Fixed Users Scenario

(b) Subset of Users Scenario

Figure 3: Convergence of CDKT-FL with different settings versus FedAvg with
Fashion-MNIST dataset

Figure 4: Effects of trade-off parameters in CDKT-FL with Fashion-MNIST
dataset

Figure 5: Effects of trade-off parameters in CDKT-FL with Fashion-MNIST
dataset

(a) Fixed Users Scenario

(b) Subset of Users Scenario

Figure 6: Distance metric comparison of CDKT-FL with Fashion-MNIST dataset
Figure 7: Comparison between Homogeneous and Heterogeneous model with Fashion-
MNIST dataset

(a) Fixed Users Scenario

(b) Subset of Users Scenario

Figure 8: Comparison of CDKT-FL with different settings versus FedAvg and No
Transfer with MNIST dataset

(a) Fixed Users Scenario

(b) Subset of Users Scenario

Figure 9: Distance metric comparison of CDKT-FL with MNIST dataset  Figure 10:
Comparison between Homogeneous and Heterogeneous Model with MNIST dataset

(a) Fixed Users Scenario

(b) Subset of Users Scenario

Figure 11: Convergence of CDKT-FL with different settings versus FedAvg with
MNIST dataset

(a) Fixed Users Scenario

(b) Subset of Users Scenario

Figure 12: Comparison of CDKT-FL with different settings versus FedAvg and No
Transfer with CIFAR-10 dataset  Figure 13: Comparison between Homogeneous and
Heterogeneous Model with CIFAR-10 dataset

(a) Fixed Users Scenario

(b) Subset of Users Scenario

Figure 14: Convergence of CDKT-FL with different settings versus FedAvg with
CIFAR-10 dataset

(a) Fixed Users Scenario

(b) Subset of Users Scenario

Figure 15: Comparison of CDKT-FL with different settings versus FedAvg and No
Transfer with CIFAR-100 dataset  Figure 16: Comparison between Homogeneous and
Heterogeneous Model with CIFAR-100 dataset

(a) Fixed Users Scenario

(b) Subset of Users Scenario

Figure 17: Convergence of CDKT-FL with different settings versus FedAvg with
CIFAR-100 dataset


# Etymo: A New Discovery Engine for AI Research

Weijian Zhang  The University of ManchesterOxford RdManchesterM13 9PL
<EMAIL_ADDRESS>,  Jonathan Deakin  The University of
ManchesterOxford RdManchesterM13 9PL<EMAIL_ADDRESS>,  Nicholas J. Higham  The University of ManchesterOxford RdManchester
<EMAIL_ADDRESS>and  Shuaiqiang Wang  JD.comBeijing
<EMAIL_ADDRESS>
(2018)

###### Abstract.

We present Etymo (https://etymo.io), a discovery engine to facilitate
artificial intelligence (AI) research and development. It aims to help readers
navigate a large number of AI-related papers published every week by using a
novel form of search that finds relevant papers and displays related papers in
a graphical interface. Etymo constructs and maintains an adaptive similarity-
based network of research papers as an all-purpose knowledge graph for
ranking, recommendation, and visualisation. The network is constantly evolving
and can learn from user feedback to adjust itself.

web search, content analysis, similarity-based network, graph centrality, data
visualisation

††conference: The Web Conference; April 2018; Lyon, France††journalyear:
2018††copyright: none

##  1\. Introduction

The rapid growth of global scientific output creates new challenges for
information retrieval. The problem is particularly acute in AI (artificial
intelligence) research. ArXiv (https://arxiv.og), for example, gains around
500 new AI-related papers every week and the number is growing. As a result,
it is difficult for researchers to keep up-to-date with the latest
developments in AI research. We have built a new discovery engine for
scholarly research called Etymo that addresses this challenge.

Citations of scientific papers are generally considered an important indicator
of a paper’s impact (Hirsch, 2005). Google Scholar’s ranking algorithm is not
publicly known, but research (Beel and Gipp, 2009) has shown that citation
counts have the highest weighting in its ranking algorithm. However, recent
publications have few or no citations so it is difficult to use citations to
judge the importance of very recent papers; thus, recent insightful
publications are difficult to rank. Our idea is to build a similarity-based
network and use this information for information retrieval tasks.

How can we obtain links in a non-hypertext setting? Our approach is to infer
links from the distributed vector representation of the full-text papers,
i.e., if the cosine similarity between the vector representations is large, we
link these two papers. It turns out that inferred links are similar to the
citation network because papers talking about the same subject tend to cite
one another. However, the analogy between hyperlinks and generated links is
not perfect. In particular, auto-generated links are a noisier source of
information and much more prone to spam.

He et al. (He et al., 2017) proposed a meta-approach called HICODE (HIdden
COmmunity DEtection) for discovering the hidden community structure in a
network. By removing certain edges from the network (weakening the strength of
certain structures), one can uncover other structures in the network.
Similarly, we can strengthen and weaken the connectivity of the network
structure so as to improve our ranking algorithm and filter out unwanted
papers. We do this by exploiting papers’ social media activities (such as the
number of retweets on Twitter) and certain information from user feedback. The
resulting graph is used for ranking, recommendation, and visualisation. Note
also that inferred links can be generated faster than citations (papers may
take 1 year to be cited, but inferred links can be generated almost
instantly).

We use a combination of PageRank and Reverse PageRank for ranking papers,
which we find gives better search results than pure PageRank or HITS
(Kleinberg, 1999). In Reverse PageRank, we compute PageRank on the graph with
reversed direction, i.e., reverse the direction of each edge $(i,j)$ to
$(j,i)$. Fogaras (Fogaras, 2003) shows that Reversed Page Rank scores express
hub quality. We have also designed and implemented a new search interface
where we display search results as a combination of a list and relationship
visualisation. This new interface allows readers to quickly locate relevant
and related papers.

Our Etymo discovery engine provides a way to evaluate new research papers by
exploiting the full-text of research papers. It challenges the traditional
list-based search interface by combining an item list with item relationship
visualisation. Etymo updates its database on a daily basis and is free to use
for all (demo available at https://etymo.io). Registered users can also write
notes and receive recommendations.

##  2\. Related Works

Recently, researchers have realised that the full text of scientific papers is
an important resource for search and other applications. Indeed, Salatino et
al. (Salatino and Motta, 2016) use the semantic enhanced topic network (where
nodes are topics and edges are their co-occurrences in a sample of
publications) to identify the appearance of new topics. Sateli et al. (Sateli
et al., 2017) analyze full-text research articles to generate semantic user
profiles. Semantic Scholar (https://www.semanticscholar.org)
111https://techcrunch.com/2016/11/11/scientists-gain-a-versatile-modern-
search-engine-with-the-ai-powered-semantic-scholar/ is a search engine for
scholarly research that analyzes and links key information from the full text
of research papers for improving search results. Similarly, Etymo makes use of
the full text of papers to generate a similarity-based network, which is then
used for information retrieval tasks.

##  3\. Architecture Overview

The dependency graph of Etymo is shown in Figure 1. Etymo has several crawlers
for downloading research papers from different journal websites. For each
paper in our database, we store both the PDF version of the paper and the
metadata, including author name, journal name, paper abstract, and the date of
publication.

In the Analysis stage, we convert all the PDFs to text using pdftotext
222https://en.wikipedia.org/wiki/Pdftotext. We then apply Doc2Vec (Le and
Mikolov, 2014) and TF-IDF (Manning et al., 2008, Chap 6) to represent a
document $d$ as numeric vectors $v_{Doc2Vec}(d)$ and $v_{tf-idf}(d)$
respectively. Both algorithms represent a paper by a numeric vector such that
papers with similar content are close to each other in the vector space. This
content similarity information is then used for building a similarity-based
network of all the papers in the database. We generate two networks using the
two alorithms and find in practice a combination of the two networks can
product better results than just using one of the two. We use t-SNE (Maaten
and Hinton, 2008) to find the paper locations and network centrality
algorithms for the paper ranking. We also generate a lexicon from the TF-IDF’s
global term weights, which is later used in search. The main components of
Etymo consist of a search engine and a feed engine. Results from both engines
are displayed as a list and graph visualisation.

Research Papers Social Media Crawler Analysis Lexicon Similarity-based Network
Paper Ranking Paper Location Feed Search Figure 1. The dependency graph of
Etymo. The direction of an arrow indicates the dependence of the two
components connected by it. The red rectangles represent our data source; the
green rectangles are the two main components of Etymo.

##  4\. System Features

Etymo has two important features that help it produce useful search results.
First, it uses a document vector representation of the full-text papers to
build a similarity-based network, where papers are nodes and similar papers
are linked. This network is adaptive because of a user feedback mechanism:
users’ stars, clicks and Twitter mentions are used to reinforce the ‘correct’
connections and weaken the ‘unimportant’ ones. The resulting network is then
used for ranking and recommendation. Second, we have designed and implemented
a novel search interface, with results presented both in a traditional item
list and with a visualisation showing paper relationships in order to help
users quickly find related papers and have a general idea of a research area.

###  4.1. Similarity-based Network

To construct the similarity-based network, one first needs to represent
documents using numeric vectors. We use a distributed representation of the
documents called Doc2Vec (Le and Mikolov, 2014) and a bag of words model
called TF-IDF. We then construct the similarity-based network using the cosine
similarity measure. One potential problem with this similarity-based network
approach is that it does not distinguish a high quality paper from a bad one.
We argue that we can use user feedback to adjust the network structure in
order to give important papers higher weights.

####  4.1.1. Adaptive Network and Ranking

Each paper is a node in the network and similar papers are linked together. We
compute the cosine similarity of every pair of paper vectors in the database.
If the cosine similarity score of two papers’ vector representations is larger
than a given threshold, we link these two papers. In other words, if
$u\in\mathbb{R}^{n}$ and $v\in\mathbb{R}^{n}$ (where $n=1000$ in practice) are
the vector representation of two papers $p_{u}$ and $p_{v}$, we define

$\operatorname{cosine\\_similarity}(u,v)=\frac{\sum_{i=1}^{n}u_{i}v_{i}}{\|u\|\|v\|},$

where $\|u\|=\left(\sum_{i}u_{i}^{2}\right)^{1/2}$. If
$\operatorname{cosine\\_similarity}(u,v)=w>\alpha$, where $\alpha$ is a
threshold, we link paper $p_{u}$ and $p_{v}$ by an edge with weight $w$.

Calculating the similarity scores of two papers has a time complexity $O(n)$.
Adding a new paper when there are already $m$ papers in the network therefore
costs $O(mn)$, which is clearly prohibitive for large $m$. One potential
solution is to calculate a new paper’s similarity on a representative subset
of the existing papers, i.e., find top $k$ high quality papers, where $k\ll
n$. Since our graph centrality ranking provides a measure of paper quality, we
use the top $k$ high ranking papers as a representative subset. Then for a new
paper, we only calculate its similarity with these $k$ papers.

Similarity-based networks are vulnerable to spam. For example, if a paper
contains a large number of important key words in AI research, it may have a
high connectivity on the network hence a high score in the ranking. We use
authors’ votes to adjust the structure of the network in three main ways:

  1. (1)

We use user stars to increase the edge weights to a node, this increases the
number of edges to that node. In other words, a paper with many user stars
connects to more papers than a paper with few.

  2. (2)

We use user libraries to infer connectivities: increasing edge weights between
the papers in a user’s library.

  3. (3)

We weaken the connectivities of a highly ranking paper if it has poor click
rates.

Finally, we turn the undirected similarity-based network into a directed
network using the temporal information from the paper published date. As a
result, a new paper on this network ‘recommends’ a old paper if they have
similar content or user data suggests that they are related. Intuitively we
attempt to predict the citation network structure of new research papers when
their citations are not available.

###  4.2. Go Beyond List: Relationship Visualisation

A few recent scholar engines incorporate some form of visualisation in
displaying their search results. AMiner (https://aminer.org/)(Tang et al.,
2008) shows similar authors and the ego network, which consists of a centre
node (“ego”) and the nodes to whom the ego node is directly connected to.
AceMap (Tan et al., 2016) displays the citation relationships between academic
papers on a map, in a similar way to Google Maps. What we do differently is
that we provide a combination of the traditional list of results with a
content similarity-based relationship visualisation. Figure 2 shows the web
interface of Etymo.

Figure 2. The web interface of Etymo. The left part is a list of search
results ordered by importance, while the right part is the corresponding
visualisation of each paper in the list. Each paper is represented by a node
and related papers are close to each other. The size of the node represents
the importance and colour represents journal.

Why do we need a new interface? The most important reason is that it saves our
time in finding interesting research papers. The information we usually need
is the top ten papers from the search results and the papers related to them,
but there is no easy way to access all of that information at once using the
traditional list interface. For example, to see related papers of the current
search result in Google Scholar, a user needs to click on multiple ’related
article’ links. In Etymo, a user can check the top ten rated papers on the
list and locate all the related papers on the graph at the same time.

##  5\. Experiments

Etymo has over $36000$ papers in the database and we typically add $500$ new
papers every week. The Analysis uses an instance of Amazon Elastic Compute
Cloud (Amazon EC2) m4.xlarge, which has $16$ vCPUs and $64$GB memory.

We update our database on a daily basis. During each update, we need to first
find two sets of vector representations for all the newly added papers using
Doc2Vec and TF-IDF. Training of both models are done on a weekly basis. We
then use t-SNE to find the x,y location of all the papers, i.e., reduce these
$1000$ dimensional vectors to $2$ dimensional vectors. The computation of
t-SNE can be done in $O(N\log N)$ and requires $O(N)$ memory (Van Der Maaten,
2014), which makes it possible to learn embedding of data sets with millions
of objects. The number of nodes in our similarity-based network is equal to
the number of papers in our database. We apply PageRank and Reverse PageRank
on this network. The predominant method for computing the PageRank is the
power method. At each iteration, we do a sparse matrix vector multiplication,
which has complexity $O(pN)$, where $p$ is the average number of non-zero
elements on every row of the matrix and $N$ is the dimension of the matrix.
Usually $10$ iterations can produce a good approximate ranking (Langville and
Meyer, 2011, Chap 8.2).

In general, we found that network-based ratings can improve search results by
highlighting historically important papers. For the query “t-sne” (a popular
machine learning algorithm for dimensionality reduction) we show the top $5$
search results in the tables below. Table 1 shows the results which include
the PageRank and Reverse PageRank ratings on the similarity-based network,
while Table 2 does not. We noticed that the one with network-based ratings
gives more weight to important papers. Note that Maaten and Hinton’s
”Visualizing Data using t-SNE” is the original t-SNE paper. Comparing with
Google Scholar’s search results in Table 3, Etymo’s top search results include
more recent publications.

Table 1. Top $5$ search results of the search query ”t-sne”. The results include a combination of PageRank and Reverse PageRank ratings. Authors | Year |  Title
---|---|---
Laurens van der Maaten, Geoffrey Hinton | 2008 |  Visualizing Data using t-SNE
Laurens van der Maaten | 2014 |  Accelerating t-SNE using Tree-Based Algorithms
Yanshuai Cao, Luyu Wang | 2017 |  Automatic Selection of t-SNE Perplexity
George C. Linderman, Manas Rachh, Jeremy G. Hoskins, Stefan Steinerberger, Yuval Kluger | 2017 |  Efficient Algorithms for t-distributed Stochastic Neighborhood Embedding
Yukun Chen, Jianbo Ye, Jia Li | 2017 |  Aggregated Wasserstein Metric and State Registration for Hidden Markov Models
Table 2. Top $5$ search results of the search query ”t-sne”. The results do not include any network-based ratings. Authors | Year |  Title
---|---|---
Yanshuai Cao, Luyu Wang | 2017 |  Automatic Selection of t-SNE Perplexity
Laurens van der Maaten | 2014 |  Accelerating t-SNE using Tree-Based Algorithms
Maaten, Laurens van der, Hinton, Geoffrey | 2008 |  Visualizing Data using t-SNE
Richard R. Yang, Mike Borowczak | 2017 |  Assessing Retail Employee Risk Through Unsupervised Learning Techniques
Martin Renqiang Min, Hongyu Guo, Dinghan Shen | 2017 |  Parametric t-Distributed Stochastic Exemplar-centered Embedding
Table 3. Top $5$ search results of the search query ”t-sne” in Google Scholar. Authors | Year |  Title
---|---|---
Maaten, Laurens van der, Hinton, Geoffrey | 2008 |  Visualizing Data using t-SNE
Laurens van der Maaten | 2014 |  Accelerating t-SNE using Tree-Based Algorithms
AR Jamieson ML Giger, K Drukker, H Li | 2010 |  Exploring nonlinear feature space dimension reduction and data representation in breast CADx with Laplacian eigenmaps and t-SNE
K Bunte, S Hasse, M Biehl, T Villmann | 2012 |  Stochastic neighbor embedding (SNE) for dimension reduction and visualization using arbitrary divergences
Maaten, Laurens van der, Hinton, Geoffrey | 2008 |  [PDF] Visualizing Data using t-SNE

##  6\. Conclusion

It is hard to quantify new things. In research, the value of a newly published
work is usually unknown until citations become available. The value of this
work is to provide a new approach to improve search results on new papers by
exploiting the paper full text content and social media data. Our user
interface combines the item list with item relationship visualisation, which
saves researchers time in finding interesting research papers.

## References

  * (1)
  * Beel and Gipp (2009) Jöran Beel and Bela Gipp. 2009.  Google Scholar’s Ranking Algorithm: An Introductory Overview. In Proceedings of the 12th International Conference on Scientometrics and Informetrics (ISSI’09), Vol. 1. ISSI, Rio de Janeiro (Brazil), 230–241.
  * Fogaras (2003) Dániel Fogaras. 2003\.  Where to start browsing the web?. In International Workshop on Innovative Internet Community Systems. Springer, 65–79.
  * He et al. (2017) Kun He, Yingru Li, Sucheta Soundarajan, and John E. Hopcroft. 2017.  Hidden Community Detection in Social Networks.  (Feb. 2017).  https://arxiv.org/abs/1702.07462 ArXiv preprint arXiv:1702.07462.
  * Hirsch (2005) Jorge E. Hirsch. 2005\.  An Index to Quantify An Individual’s Scientific Research Output.  Proceedings of the National Academy of Sciences of the United States of America 102, 46 (2005), 16569.  https://doi.org/10.1073/pnas.0507655102
  * Kleinberg (1999) Jon M. Kleinberg. 1999\.  Authoritative Sources in a Hyperlinked Environment.  J. Assoc. Comput. Mach. 46, 5 (Sept. 1999), 604–632.  https://doi.org/10.1145/324133.324140
  * Langville and Meyer (2011) Amy N Langville and Carl D Meyer. 2011.  Google’s PageRank and beyond: The science of search engine rankings.  Princeton University Press.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed Representations of Sentences and Documents. In Proceedings of the 31st International Conference on Machine Learning (ICML-14). 1188–1196.  http://www.jmlr.org/proceedings/papers/v32/le14.pdf
  * Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  Journal of Machine Learning Research 9, Nov (2008), 2579–2605.
  * Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schutze. 2008.  Introduction to Information Retrieval.  Cambridge University Press, Cambridge.
  * Salatino and Motta (2016) Angelo Antonio Salatino and Enrico Motta. 2016.  Detection of Embryonic Research Topics by Analysing Semantic Topic Networks. In International Workshop on Semantic, Analytics, Visualization. Springer, 131–146.
  * Sateli et al. (2017) Bahar Sateli, Felicitas Löffler, Birgitta König-Ries, and René Witte. 2017.  ScholarLens: Extracting Competences from Research Publications for the Automatic Generation of Semantic User Profiles.  PeerJ Computer Science 3 (July 2017), e121.  https://doi.org/10.7717/peerj-cs.121
  * Tan et al. (2016) Zhaowei Tan, Changfeng Liu, Yuning Mao, Yunqi Guo, Jiaming Shen, and Xinbing Wang. 2016\.  AceMap: A Novel Approach Towards Displaying Relationship Among Academic Literatures. In Proceedings of the 25th International Conference Companion on World Wide Web (WWW ’16 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 437–442.  https://doi.org/10.1145/2872518.2890514
  * Tang et al. (2008) Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008.  Arnetminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 990–998.
  * Van Der Maaten (2014) Laurens Van Der Maaten. 2014\.  Accelerating t-SNE using tree-based algorithms.  Journal of machine learning research 15, 1 (2014), 3221–3245.