#  Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated
Learning

Gwen Legate
Concordia University, Mila
<EMAIL_ADDRESS>
&Lucas Caccia
McGill University, Mila
<EMAIL_ADDRESS>
&Eugene Belilovsky
Concordia University, Mila
<EMAIL_ADDRESS>

###### Abstract

In Federated Learning, a global model is learned by aggregating model updates
computed at a set of independent client nodes, to reduce communication costs
multiple gradient steps are performed at each node prior to aggregation. A key
challenge in this setting is data heterogeneity across clients resulting in
differing local objectives which can lead clients to overly minimize their own
local objective, diverging from the global solution. We demonstrate that
individual client models experience a catastrophic forgetting with respect to
data from other clients and propose an efficient approach that modifies the
cross-entropy objective on a per-client basis by re-weighting the softmax
logits prior to computing the loss. This approach shields classes outside a
client’s label set from abrupt representation change and we empirically
demonstrate it can alleviate client forgetting and provide consistent
improvements to standard federated learning algorithms. Our method is
particularly beneficial under the most challenging federated learning settings
where data heterogeneity is high and client participation in each round is
low.

##  1 Introduction

Federated Learning (FL) is a distributed machine learning paradigm in which a
shared global model is learned from a decentralized set of data located at a
number of independent client nodes (McMahan et al., 2017; Konečnỳ et al.,
2016). Driven by communication constraints, FL algorithms typically perform a
number of local gradient update steps before synchronizing with the global
model. This reduced communication strategy is very effective under independent
and identically distributed (i.i.d.) settings, but data heterogeneity across
clients has direct implications on the convergence and performance of FL
algorithms (Zhao et al., 2018). FL was conceptualized as a learning technique
to train a shared model without sharing user sensitive data, while allowing
users to benefit from data stored at other nodes, such as phones and tablets
of decentralized users. Under realistic settings, client data will often have
non-i.i.d. distributions. For the case of supervised multi-class
classification, users may frequently have no data whatsoever from one or
several classes. Data in-homogeneity across clients frequently induces client
drift, a phenomenon in which clients progress too far towards optimizing their
own local objective, leading to a solution that has severely "drifted" from an
optimal global solution (Karimireddy et al., 2020).

In continual learning, a model is trained on a number of tasks sequentially
and the learner needs to learn each new task without forgetting knowledge
obtained from the preceding tasks. The tendency of a continual learning model
to forget previously learned information when learning a new task is termed
catastrophic forgetting (McCloskey & Cohen, 1989) and it is typically a focus
of study in continual learning literature. Similar to FL, data heterogeneity
in continual learning presents a challenge since different tasks typically
contain data drawn from different underlying distributions. We can draw a
connection between the catastrophic forgetting problem in continual learning
and the client drift problem in federated learning. We consider one round of
federated learning in which $K$ random clients are selected and initialized
with a copy of the current global model. Each client performs a pre-determined
number of local update steps to optimize the objective on their local data. A
round ends with an update to the global model achieved by aggregating the
updates from each client. At the beginning of a round, the clients receive a
model previously derived from training on other clients data. As local
training proceeds, the model becomes increasingly biased towards a given
client. As discussed in Lesort (2022); Gupta et al. (2022), this can cause
client models to rely on spurious correlations to improve their in-
distribution performance, creating a situation where local models experience
catastrophic forgetting with respect to data of the other clients (drawn from
distinctly different distributions). Naturally, aggregating models that have
deviated from a joint solution will lead to degraded results with respect to
the global objective. We denote this problem as local client forgetting.
Reducing local client forgetting would moderate the decrease in performance
with respect to other clients data at individual client models. We hypothesize
this would increase the ability of local models to generalize to the
distribution of data from other clients and improve the loss of individual
models over the combined data and we therefore propose to reduce client drift
by tackling local client forgetting. Figure 1 illustrates local client
forgetting within a round of federated learning.

Figure 1:  Illustration of catastrophic forgetting within client rounds. A
global model with knowledge of all classes is sent to all clients
participating in a given FL round. Local training increases the client model
performance on the client’s local distribution but tends to simultaneously
decrease performance with respect other clients distributions which leads to
poor aggregation and overall model performance.

There are numerous approaches to tackle catastrophic forgetting in the
continual learning literature (Kirkpatrick et al., 2017; Li & Hoiem, 2017;
Chaudhry et al., 2019; Schwarz et al., 2018; Davari et al., 2022). Many of
these however are largely impractical in the FL setting. Experience Replay
methods (Chaudhry et al., 2019) require access to other clients’ data,
violating the fundamental data communication constraints of FL. Similar
concerns exist for many regularization methods such as elastic weight
consolidation (EWC) Kirkpatrick et al. (2017) which require communicating
additional information. Regularization methods can additionally require many
steps to converge due to the additional conflicting objectives (Aljundi et
al., 2019). This computational constraint can hurt convergence of the FL
algorithm, a key desideratum. For the supervised continual learning setting,
Caccia et al. (2022); Ahn et al. (2021) proposed a modification of the
standard cross entropy objective function that truncates the softmax
denominator, removing terms corresponding to classes from old tasks. A variant
of this method inspired by the long-tailed recognition methods (Ren et al.,
2020) was also recently introduced in Jodelet et al. (2022). This simple
approach mitigates catastrophic forgetting by reducing the bias on the model
to avoid predicting old classes.

Inspired by the parallels between client drift in FL and catastrophic
forgetting in the continual learning case, we propose an adaptation of the CL
method from Caccia et al. (2022); Ahn et al. (2021); Jodelet et al. (2022) to
modify the loss function of each client based on its class distribution using
a re-weighted softmax. We will empirically demonstrate that this approach can
drastically reduce client level forgetting in the heterogeneous setting,
leading to substantially improved overall global model convergence and final
performance. We apply the re-weighted softmax to baseline methods FedAvg
(McMahan et al., 2017), SCAFFOLD (Karimireddy et al., 2020), FedNova (Wang et
al., 2020) and FedProx (Li et al., 2020) demonstrating performance
improvements in all cases.

##  2 Related Work

#### Federated Learning

The most commonly used baseline in federated learning is the FedAvg algorithm
proposed by McMahan et al. (2017), which improves on local SGD (Stich, 2018)
by allowing clients to train multiple local iterations successively, thus
reducing communication costs. Since the communication cost between two nodes
is orders of magnitude larger than the cost between processor and memory on
the same node, communication efficiency in federated learning of upmost
importance (Konečnỳ et al., 2016). Convergence of FedAvg has been widely
studied for both i.i.d. (Stich, 2018; Wang & Joshi, 2018) and non i.i.d.
settings (Karimireddy et al., 2020; Li et al., 2020; Fallah et al., 2020; Yu
et al., 2019) and settings under which client data is heterogeneous offer
additional challenges since convergence has been shown to deteriorate as a
function of increasing heterogeneity (Li et al., 2020; Hsu et al., 2019).

#### Non-Heterogeneous Data Partitions and Client Drift

One significant challenge encountered when training on decentralized data is
heterogeneity of samples across clients (Konečnỳ et al., 2016; Li et al.,
2021). Partitions contain data generated under different conditions, which can
reasonably be expected to create different local distributions at each client.
For the case of supervised multi-class classification, which is the focus of
this work, users may frequently be missing data from an entire class or
multiple classes of the global underlying distribution. During FL training,
data at each client are sampled from these local distributions, creating
different local objectives. When clients progress too far towards minimizing
their own objective, local models drift from one another, degrading the
performance of the shared global model and slowing down convergence (Yao et
al., 2021; Li et al., 2019; Karimireddy et al., 2020; Diao et al., 2020).

Several attempts have been made to alleviate client drift through various
methods. One approach centers around knowledge distillation to regulate local
training, (Zhu et al., 2021) and (Lin et al., 2020) ensemble information about
the global data distribution and disseminate it to clients via additional
models trained at the server. These methods possess the added risk of privacy
attacks and while Zhu et al. (2021) take steps to mitigate this risk, their
method requires the existence of an unlabeled dataset, which may not be
available in all settings. Other approaches attempt to constrain gradient
updates from the clients to reduce the impact of client drift. Karimireddy et
al. (2020) propose SCAFFOLD, an algorithm to control client drift using
control norms to modify client gradients. The control norms estimate the drift
at each client and use that estimate to correct the local updates. FedProx (Li
et al., 2020) adds a proximal term to the local objective to limit the impact
of variation in local updates. This term is weighted with an additional
hyperparameter, $\mu$ which must be tuned appropriately. Both SCAFFOLD and
FedProx require careful hyperparameter tuning to be effective (Li et al.,
2022). Like SCAFFOLD and FedProx Acar et al. (2021) also estimate client drift
and use it as a means to constrain updates. However, their operations take
place at the server level and corrections are applied to the server updates
thus avoiding SCAFFOLD’s use control norms and saving on the inherent
communication burden. The general strategy for each of these methods is
similar. They attempt to estimate client drift using gradient updates and then
constrain the updates to reduce the drift. SCAFFOLD has additionally been
shown to have unstable accuracy and has twice the communication burden of
FedAvg due to the control norms required for each client Li et al. (2022). Our
proposed method modifies the objective functions locally, creating no
additional communication burdens and introducing no additional
hyperparameters. It also directly tackles the problem of the underlying
distribution shift, unlike the other methods mentioned which attempt to
address client heterogeneity through constrained gradient optimization and/or
knowledge distillation. Tenison et al. (2022) propose a gradient masking
technique that modifies the aggregation of updates on the server side. This
method does directly tackle the problem of distribution shift and while it has
been shown to be effective at achieving better generalization on non i.i.d.
data, our re-weighted softmax is able to take advantage of the structure of
the commonly used cross-entropy loss, making it simpler to implement.
Additionally, since we modify the objective functions locally, the re-weighted
softmax method is compatible with any federated optimization method in the
literature.

#### Continual Learning

Continual learning is a process by which tasks are learned sequentially over a
period of time and knowledge of previous tasks is retained and leveraged to
learn new tasks (Chen & Liu, 2018). Continual learning is made difficult by
the fact that neural networks suffer from catastrophic forgetting, in which
learning a new task overrides weights learned from past training, thus
degrading model performance on previously learned tasks (McCloskey & Cohen,
1989). Several families of methods have been developed to mitigate
catastrophic forgetting. The first class of methods are architecture based
approaches Schwarz et al. (2018) that attempt to grow or modify an
architecture over time to expand its knowledge. In the second class of methods
approaches which store some subset of old data for rehearsal are applied
(Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Rebuffi et al., 2017).
Finally, a third class of methods regularizes the learned parameters to limit
drastic weight changes when learning a new task (Kirkpatrick et al., 2017;
Zenke et al., 2017). None of these solutions are widely applicable in a FL
setting since they typically require some sort of information sharing across
client nodes which is prohibited in the FL framework. A federated continual
learning setting has been considered in the literature (Yoon et al., 2021).
Here each client in the federated network continuously collects data. Our work
on the other hand considers the standard FL setting where each client
maintains a fixed set of data and draws connections to a notion of forgetting
across clients to motivate a modification of the loss function. Shoham et al.
(2019) have used ideas from continual learning to propose FedCurv, based on
the EWC algorithm (Kirkpatrick et al., 2017) from continual learning. FedCurv
requires sending additional information and is not compatible with all FL
methods. Along this line, Xu et al. (2022) also proposed an approach inspired
from rehearsal methods, generating pseudo data and adding an additional
regularization term. This requires an expensive pseudo data generating
procedure and increases the local training time.

##  3  Methods

#### Background

In federated optimization, training data is distributed and optimization
occurs over $K$ clients with each client $k\in{1,...,K}$ having data
$\mathbf{X}_{k}$ drawn from distribution $D_{k}$. We define
$n_{k}=|\mathbf{X}_{k}|$ and $n=\sum_{k=1}^{K}n_{k}$ for $n$ samples. The data
$\mathbf{X}_{k}$ at each node may be drawn from different distributions and/or
may be unbalanced with some clients possessing more training samples than
others. The typical objective function for federated optimization is given by

$\min_{\mathbf{w}\in\mathbb{R}^{d}}\sum_{k=1}^{K}\frac{n_{k}}{n}\mathcal{L}(\mathbf{w},\mathbf{X}_{k}),$
(1)

with $\mathcal{L}(\mathbf{w},\mathbf{X}_{k})$ measuring client $k$’s local
objective, and $\mathbf{w}$ representing the global parameters.

In this work we will restrict ourselves to the common case where $\mathcal{L}$
is the cross entropy loss. There are many possible variations of FL
algorithms. In general, they follow the similar structure to FedAvg (McMahan
et al., 2017), which proceeds as follows :

  * •

Client selection: for a set of $K$ clients, $K*p$ are selected at each round
$\\{t_{i}\\}_{i=1}^{T}$ , where $0<p\leq 1$ is a pre-determined proportion of
clients.

  * •

Client updates: At the beginning of round, client models are initialized with
the current weights of the server model. Each client selected for the round
performs $E$ local iterations of SGD.

  * •

Cerver update: The weights of the individual client models are aggregated to
form an update to the shared global model.

#### Re-weighted Softmax Cross Entropy

Consider a neural network $f:\mathcal{R}^{D}\to\mathcal{R}^{C}$ where $C$ is
the total number of classes. The standard cross entropy is given by equation 2
where $y(\mathbf{x})$ is the label of $\mathbf{x}$ and $\mathcal{C}$ is the
set of all classes available to the clients.

$\displaystyle\mathcal{L}_{CE}(\mathbf{X}_{k},\mathbf{w})$
$\displaystyle=-\sum_{\mathbf{x}\in\mathbf{X}}\log\frac{\exp(f_{\mathbf{w}}(\mathbf{x})_{y(\mathbf{x})})}{\sum_{c\in\mathcal{C}}\exp(f_{\mathbf{w}}(\mathbf{x})_{c})}$
(2)
$\displaystyle=-\sum_{\mathbf{x}\in\mathbf{X}}\Bigl{[}f_{\mathbf{w}}(\mathbf{x})_{y(\mathbf{x})}-\log\Bigl{(}\sum_{c\in\mathcal{C}}\exp(f_{\mathbf{w}}(\mathbf{x})_{c}\Bigr{)}\Bigr{]}$
(3)

One interpretation of this classical loss function considers the two terms as
a tightness term (the first term) which brings samples close to their
representative classes and a contrast term (the second term) which pushes them
apart from other classes (Boudiaf et al., 2020). We note similar loss
functions can be interpreted from an energy modeling view (Liu, 2020).

We now modify the standard cross entropy using the re-weighted softmax (WSM)
to give our per-client objective function in Equation 4 where
$\bm{\mathbf{\beta}_{k}}$ is a vector containing the proportions of each class
present in the client dataset and $\mathbf{\beta}_{c}\in\bm{\beta_{k}}$ is the
proportion of label $c$ present in the dataset.

$\mathcal{L}_{WSM}(\mathbf{X}_{k},\mathbf{w})=-\sum_{\mathbf{x}\in\mathbf{X}}\Bigl{[}f_{\mathbf{w}}(\mathbf{x})_{y(\mathbf{x})}-\log\Bigl{(}\sum_{c\in\mathcal{C}}\beta_{c}\exp(f_{\mathbf{w}}(\mathbf{x})_{c})\Bigr{)}\Bigr{]}$
(4)

We note that in Equation 4 the weighting $\beta$ introduces in the second term
is a function only of the labels present in the client data $\mathbf{Y}_{k}$.
In a highly imbalanced class scenario, as often studied for FL, many
$\beta_{c}$ will be zero or contain very small values, thus removing or
substantially degrading the contribution of that class to the contrast term.
Intuitively, aggressively optimizing $\mathcal{L}_{CE}$ through multiple
gradient steps during a client round can lead to a drastic increase in
$\mathbb{E}_{x,y\sim D_{j\neq k}}[l_{CE}(\mathbf{x},\mathbf{y})]$ where
$D_{j}$ are the distribution of clients other than client $k$. This is because
the contrast term encourages classes not present on the client to never be
predicted. The re-weighted softmax approach modifies the original local
objective function to avoid excessive pressure that drives up the loss of
other client data. Indeed, classes not present at the current client are
ignored by the local optimization which forces the client to learn by adapting
the model’s internal representation of the classes present in its training
data, rather than abruptly shifting representations of classes outside its
training set (Caccia et al., 2022). We will demonstrate empirically this leads
to a reduction in local client forgetting defined below.

#### Local client forgetting

We formalize the notion of local client forgetting discussed in Sec 1 for a
multi-class classification problem. Denoting the accuracy on a client $k$’s
local test data $Acc_{k}(\mathbf{w})$, where $\mathbf{w}$ are the model
parameters. We can define local client forgetting according to equation 5
where $\mathbf{w}_{t}^{i}$ refers to the model of client $i$ at round $t$
after it has completed local training (prior to aggregation) and
$\mathbf{w}_{t-1}$ is the global model (after aggregation) at the end of round
$t-1$.

$F_{ki}=Acc_{k}(\mathbf{w}_{t-1})-Acc_{k}(\mathbf{w}_{t}^{i})$ (5)

.

We define an average forgetting for a client $k$’s model according to equation
6

$F_{k}=\frac{1}{K-1}\sum_{i\neq k}F_{ki}$ (6)

In what follows we will study these quantities for a standard FL setting.

##  4 Experiments

In this section, we present the empirical results for the WSM framework. We
start by analyzing the notion of forgetting in the context of standard FedAvg,
and then show how the application of the re-weighted softmax objective can
resolve this. Subsequently, we study how WSM can enhance standard FL
algorithms, improving their overall performance.

#### Datasets and Data Partitions

We utilize CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009) and FEMNIST (Caldas
et al., 2018) datasets in our experiments, each of these datasets come pre-
separated into training and testing sets. Our primary evaluations consider 100
clients where each client requires their own training and validation sets
according to their own unique distribution. To facilitate this, the entire
training set is separated into equally sized non-i.i.d. partitions using the
Dirichlet distribution parameterized by $\alpha=0.1$, similar to the method of
Hsu et al. (2019). To provide intuition into what client partitions
parameterized by $\alpha=0.1$ look like in practice, Fig. 2 shows the data
distributions of ten randomly selected clients. These client partitions are
then further separated into training ($90\%$) and validation ($10\%$) sets for
each client. For example, 100 clients being trained using CIFAR-10 which
contains 50 000 training samples would each have 500 of these training
samples. Of those 500 samples, 450 would be used for local model updates and
50 would be used exclusively for validation.

Figure 2: CIFAR-10 - Distributions of ten randomly selected clients with data
partitioned according to a Dirichlet distribution parameterized by
$\alpha=0.1$.

#### Settings

We follow the experimental settings of Reddi et al. (2020). Clients are
sampled without replacement for each round but can be selected again in
subsequent rounds. The fraction of clients sampled is $10\%$ for CIFAR-10 and
FEMNIST datasets and $2\%$ for CIFAR-100. Our primary evaluations train a
ResNet-18 over 4000 communication rounds for 3 local epochs, using a mini-
batch of size 64 and a learning rate of 0.05 for CIFAR-10 and CIFAR-100.
FEMNIST, which converges faster due to its large size, is trained for 3000
rounds with all other settings the same as for the CIFAR datasets. We use SGD
as our optimizer, with weight decay of $1\times 10^{-4}$ following Yao et al.
(2021); Hsu et al. (2019). In these experiments, we observe that FedAvg often
(though not always) performs better with group normalization as indicated by
Hsieh et al. (2020) while FedAvg+WSM is able to perform well with both group
and batch normalization, very frequently achieving the best results with
batchnormalization. We therefore treat the normalization method as a
hyperparameter and provide the best results for both batch and group
normalization for both methods at each learning rate.

#### Validation

Throughout the training process the global model is periodically evaluated on
the aggregation of the client validation sets to gauge overall training
progress, the test set on the other hand is only used at the end of the
training process. For lower values of $\alpha$, as client distributions become
more skewed, there can be significant changes in accuracy between training
runs (Hsu et al., 2019). Since we focus our analysis on the highly
heterogeneous case in which the Dirichlet distributions at each client are
parameterized by $\alpha=0.1$, we observe higher variance in our results. This
is especially true for smaller datasets such as CIFAR-10 and CIFAR-100. To
mitigate these effects on the validation statistics, we follow the lead of
Reddi et al. (2020) and report our final accuracy as the average of the test
accuracies taken over the last 100 rounds of training.

Figure 3: Illustrating local client forgetting for a given round with and
without WSM. For each heatmap the y-axis indicates the model of client i and
the x-axis indicates the local data of client k. In the left column we show
the $F_{ik}$ as defined in Eq. 5. Positive values of forgetting (green)
indicate high forgetting. The right column show the accuracy of client i’s
model after training when evaluated on client k’s data. We note that in some
cases the accuracy can completely collapse on other client’s data
(particularly when they don’t overlap in any classes). We see that FedAvg+WSM
(top row) significantly reduces forgetting across clients.

###  4.1 Forgetting During a Federated Round

We first study and build intuition on the effects of forgetting in a federated
round for CIFAR-10 dataset. In Figure 3 we show the effects of forgetting
during a round of federated learning. The heatmaps on the bottom row of Figure
3 are for round 1200 of training. In the forgetting heatmap we observe a lot
of green in the off-diagonal terms indicating that forgetting is very high
when using standard cross entropy.

The post local update heatmap for standard cross entropy shows a strong trend
of better accuracies along the diagonal indicating model $i$ does much better
on dataset $k$ when $i=k$, its own dataset.

Figure 4: Average forgetting $F_{k}$ for selected rounds of training. We
observe high forgetting for FedAvg which is substnatially reduced (especially
towards the end of training when applying WSM.

When using WSM, we observe no preference for better accuracy along the
diagonal as is the case without using WSM. In fact, we notice lower accuracies
to be concentrated along the column, indicating a particularly difficult
dataset for all local models or along the row, indicating a model that does
badly on all datasets including its own. These observations are supported by
the forgetting heatmap for WSM which is predominantly yellow indicating very
low forgetting values. These results show WSM has the ability to greatly limit
the effects of local client forgetting by ignoring classes outside of its data
distribution and focusing learning on the classes present. This narrower focus
leads to better overall performance after aggregation. The heatmaps are shown
for round 1200 of training, approximately a third of the way through training,
however, the observation is further confirmed for other rounds as shown in
Figure 4 where we plot forgetting throughout training. We observe on average
it is high in FedAVG and is substantially reduced especially at the end of
training for WSM. Having shown that WSM can indeed reduce the local client
forgetting, we now study its effect on the aggregated models.

###  4.2 Evaluation of WSM

In this section we demonstrate how WSM used in combination with FedAvg can
improve model performance and convergence. Table 1 shows model performance
across a range of learning rates, using CIFAR-10, CIFAR-100, and FEMNIST
datasets. The reported values are the average $\pm$ the standard deviation of
three runs for each setting. The best preforming model for each learning rate
is shown in bold and the best overall result for each dataset is indicated by
a green (red) box for WSM (FedAvg). This table demonstrates that WSM has a few
benefits to offer during training. WSM substantially improves performance for
both CIFAR datasets with a $2.2\%$ and $1.3\%$ improvement for CIFAR-10 and
CIFAR-100, respectively. For FEMNIST dataset the results show the best
performing models are within statistical error of one another. We do however
observe strong results using WSM with higher learning rates under which regime
we are able to obtain faster convergence. WSM also makes the hyperparameters
easier to tune since it performs well over a large range of learning rates,
for example for learning rates between $0.03$ and $0.1$ FedAvg+WSM has
remarkably steady performance between $85.5\%$ to $85.7\%$ while we observe no
such consistent performance for vanilla FedAvg.

Table 1: Accuracy results of FedAvg with and without WSM for different hyperparameters. We observe that FedAvg+WSM consistently improves performance over FedAvg over a wider range, as well as having the highest overall accuracy by a substantial margin on the cifar datasets. WSM also makes the learning rate easier to tune since we observe a large hyperparameter range  Method |  | _Dataset_
---|---|---
lr | CIFAR-10 | CIFAR-100 | FEMNIST
FedAvg | 0.5 | $0.326\pm 0.098$ | $0.292\pm 0.012$ | $0.542\pm 0.087$
FedAvg+WSM (ours) |  | $\mathbf{0.792\pm 0.006}$ | $\mathbf{0.426\pm 0.003}$ | $\mathbf{0.837\pm 0.002}$
1-5  FedAvg | 0.3 | $0.791\pm 0.013$ | $0.384\pm 0.013$ | $0.769\pm 0.006$
FedAvg+WSM (ours) |  | $\mathbf{0.834\pm 0.008}$ | $\mathbf{0.467\pm 0.015}$ | $\mathbf{0.844\pm 0.010}$
1-5  FedAvg | 0.1 | $0.724\pm 0.027$ | $0.500\pm 0.016$ | $0.835\pm 0.002$
FedAvg+WSM (ours) |  | $\mathbf{0.855\pm 0.004}$ | $\mathbf{0.514\pm 0.009}$ | $\mathbf{0.848\pm 0.006}$
1-5  FedAvg | 0.07 | $0.826\pm 0.007$ | $0.437\pm 0.007$ | $\mathbf{0.827\pm 0.006}$
FedAvg+WSM (ours) |  | $\mathbf{0.856\pm 0.005}$ | $\mathbf{0.553\pm 0.018}$ | $0.826\pm 0.019$
1-5  FedAvg | 0.05 | $0.827\pm 0.004$ | $0.464\pm 0.001$ | $\mathbf{0.853\pm 0.004}$
FedAvg+WSM (ours) |  | $\mathbf{0.858\pm 0.003}$ | $\mathbf{0.564\pm 0.007}$ | $0.842\pm 0.005$
1-5  FedAvg | 0.03 | $0.836\pm 0.005$ | $0.431\pm 0.020$ | $\mathbf{0.835\pm 0.006}$
FedAvg+WSM (ours) |  | $\mathbf{0.857\pm 0.005}$ | $\mathbf{0.581\pm 0.005}$ | $0.834\pm 0.003$
1-5  FedAvg | 0.01 | $0.815\pm 0.003$ | $0.431\pm 0.005$ | $\mathbf{0.830\pm 0.002}$
FedAvg+WSM (ours) |  | $\mathbf{0.845\pm 0.006}$ | $\mathbf{0.574\pm 0.006}$ | $0.800\pm 0.019$
1-5  FedAvg | 0.007 | $0.817\pm 0.007$ | $0.426\pm 0.005$ | $\mathbf{0.821\pm 0.011}$
FedAvg+WSM (ours) |  | $\mathbf{0.841\pm 0.004}$ | $\mathbf{0.568\pm 0.003}$ | $0.800\pm 0.007$
1-5  FedAvg | 0.005 | $0.802\pm 0.010$ | $0.426\pm 0.002$ | $\mathbf{0.819\pm 0.022}$
FedAvg+WSM (ours) |  | $\mathbf{0.826\pm 0.009}$ | $\mathbf{0.554\pm 0.004}$ | $0.773\pm 0.013$

Figure 5: Convergence plots of different algorithms with and without WSM. We observer that WSM variants lead to substantially better convergence for all the compared methods. Table 2: Accuracy results of FedAvg (McMahan et al., 2017), SCAFFOLD (Karimireddy et al., 2020) FedNova (Wang et al., 2020) and FedProx (Li et al., 2020) with and without WSM. We observe that even combined with FL optimization based methods designed to address heterogeneity, WSM provides consistent performance gains. Furthermore, the best performance overall is highlighted in red. We find that although SCAFFOLD and FedPROX can handle heterogeneity better than FedAvg, when combined with WSM FedAvg can obtain the best performance amongst all methods. Method | CIFAR-10 | CIFAR-100
---|---|---
FedAvg | $0.800\pm 0.033$ | $0.494\pm 0.008$
FedAvg+WSM (ours) | $\mathbf{0.864\pm 0.008}$ | $\mathbf{0.640\pm 0.002}$
1-3  Scaffold | $0.794\pm 0.033$ | $0.532\pm 0.003$
Scaffold+WSM (ours) | $\mathbf{0.812\pm 0.012}$ | $\mathbf{0.572\pm 0.028}$
1-3  FedNova | $0.780\pm 0.010$ | $0.500\pm 0.005$
FedNova+WSM (ours) | $\mathbf{0.789\pm 0.13}$ | $\mathbf{0.575\pm 0.002}$
1-3  FedProx | $0.776\pm 0.028$ | $0.487\pm 0.10$
FedProx+WSM (ours) | $\mathbf{0.818\pm 0.023}$ | $\mathbf{0.562\pm 0.005}$

#### WSM for Heterogenous FL methods

So far we have demonstrated the ability of WSM to improve model preformance
for the FedAvg algorithm, now we demonstrate its effectiveness over a range of
FL algorithms. In Table 2 we apply WSM to SCAFFOLD, Fed Nova and FedProx.
SCAFFOLD and FedProx are constrained optimization based methods, specifically
designed to address the problem of data heterogeneity and Fed Nova as well is
designed to improve performance on heterogeneous data. In these experiments we
use 30 clients and train the models over only 2000 rounds, since some of these
algorithms have a high overhead and many hyperparameters to search. We keep
the same local batch size of 64 and 3 local iterations from the base case.
Combining each of the methods with WSM we confirm our hypothesis that using
WSM to mitigate local client forgetting provides improvement over the base
cases for each method, in most cases the improvement is substantial. From
Figure 5 we remark that combining WSM with a baseline method generally
provides a stronger performance from the very beginning of training since for
both CIFAR-10 and CIFAR-100 the curves showing the training progression for
the methods combined with WSM all start off with higher reported accuracy than
the baselines and this improvement in performance persists for the duration of
training. Work on critical learning periods, where critical learning periods
are defined as the early epochs of a training regime, have shown they can
determine the final quality of a deep neural network for traditional ML
methods (Jastrzębski et al., 2018; Achille et al., 2017; Yan et al., 2021)
investigate critical learning periods in the FL setting and discover they do
indeed exist consistently in FL. We can thus hypothesize that the early
training advantage we see when applying WSM may be having a positive impact on
its consistent ability to outperform other FL algorithms it is applied to.

The ease of application of WSM is a significant advantage since this feature
makes it an easy way to achieve a substantial improvement over a baseline
algorithm. Finally, we observe that combining WSM with FedAvg has the largest
overall effect since the best overall method in table 2 is FedAvg+WSM. While
WSM does offer improvements over all baseline algorithms, this result is
somewhat surprising since our intuition was that WSM would work independently
alleviate local client forgetting and would augment the value of the
modifications offered by the different federated learning algorithms. We
theorize that the superior performance of FedAvg+WSM over SCAFFOLD+WSM or
FedProx+WSM or Fed Nova+WSM is because while SCAFFOLD, Fed Nova and FedProx
with WSM all individually improve the baseline, they may need to be further
optimized to work effectively together.

Figure 6: Plots of ablations of dataset heterogeneity, number of local
iterations and the fraction of clients selected at each round studied with
WSM. We observe WSM provides performance increases under most of the
conditions studied but it’s most significant advantages are in scenarios with
very heterogeneous client data and smaller fractions of clients are selected
for each training round.

###  4.3 Ablations

We now further study the behavior of WSM in combination with FedAvg under
different data distributions, client participation settings, different numbers
of local iterations and different model architectures. In Figure 6 ablations
for the CIFAR-10 setting are shown where we ablate one setting at a time.
Except for where we specify the value of the parameter being ablated, the
hyperparameters for the ablation studies is the same as for our base case
described in section 4. We observe that under conditions of low $\alpha$ (high
heterogeneity) and low fractions of clients selected at each round the use of
WSM is particularly advantageous.

#### Parameter $\alpha$ of the Dirichlet Distribution

The Dirichlet distribution is parameterized by $\alpha$ as $\alpha\rightarrow
0$ the client distributions will become increasingly heterogeneous and as
$\alpha\rightarrow\infty$, each client data edges closer towards the same
i.i.d. distribution. In our base case we use $\alpha=0.1$ and we now
investigate how model performance is affected by changing the heterogeneity of
the data partitions. Similar to the work of Hsu et al. (2019), we investigate
$\alpha=\\{0.01,0.1,0.2,0.5,1,10,100\\}$, for a more explicit indication of
how client distributions change as a function of $\alpha$ we refer the reader
to section A.4. From Figure 6 we observe WSM has a more significant effect as
$\alpha$ decreases with the biggest margin of improvement over vanilla FedAvg
occurring when $\alpha=0.01$. As the data becomes increasingly homogeneous,
the gap between cross entropy with and without WSM shrinks until the data is
i.i.d. and the two methods are equivalent. From Figure 8 we see that by the
time $\alpha=0.5$ most of the clients possess all of the classes, albeit in
very skewed proportions. This is also the point at which the performance of
FedAvg and FedAvg+WSM become very close. While WSM continues to offer a
performance increase over the entire range of $\alpha$ except for
$\alpha=100$, we conclude WSM is most advantageous when clients labels
distributions are imbalanced.

#### Fraction of Participating Clients

One of the characteristics of FL is that data is massively distributed and
nodes may have limited communication with the central server as nodes may be
offline or have slow connections limiting their communication(McMahan et al.,
2017). The direct result of the limited capacity for connection is that only a
small number of client nodes may participate in updates at any given time.
These ablations focus on the fraction of clients participating in an update
round, for each experiment the fraction of participating clients is constant
for all rounds of training and we explore fractions from $1\%$, where only one
client participates in the update, up to $60\%$. For the fraction of
participating clients $p$, we observe the largest performance gap between
FedAvg+WSM and FedAvg when the number of participating clients is low. This
feature is significant since as explained above, low client participation is a
known feature of real world federated learning settings. We hypothesize the
performance gap as a function of client fraction between the FedAvg and
FedAvg+WSM is due to the larger impact of local client forgetting when we
limit communication capability. Unless we actively take steps to control
forgetting, non-participating clients will have their data distributions
forgotten because unlike participating clients, they will be unable to
contribute their updates to the global model. As the fraction of clients
selected at each round increases, we observe the performance gap between the
two methods narrow since more clients will have the opportunity to be selected
at each round and "remind" the model of their data distributions.

#### Local Iterations

A local iteration is defined as one gradient step on the client during a
federated round. In our reference setup $~{}7$ local iterations is equivalent
to one local epoch. After 21 local iterations we observe a sharp decrease in
accuracy for FedAvg as local iterations increase. While FedAvg+WSM does also
experience a drop in accuracy as local iterations increase, its drop in
accuracy is much less pronounced. This result is in line with what we would
expect: since WSM helps to alleviate the effects of local client forgetting
increasing the number of local training iterations will cause the effects of
local client forgetting become more serious. The effect of WSM in allowing the
local models to train for more local iterations without forgetting has
important implications for the communication costs in FL. Since communication
is a serious bottleneck and is limited by client drift, our method offers a
valuable option to speed up training in a federated setting.

#### Architecture

We additionally investigate the performance of WSM using the LeNet
architecture (LeCun et al., 1998) on CIFAR-10 and CIFAR-100. While this model
is not the considered state-of-the-art on these CIFAR datasets, it allows us
to eliminate the dependence on the normalization as a hyperparameter to
investigate relative performance for the purposes of this investigation. As
before we train for 4000 rounds with each client training for three local
epochs. The data is divided between clients using a Dirichlet distribution
parameterized by $\alpha=0.1$. The reported values are the result of the
average of three different seeds with the standard deviation indicating the
variation between runs. Experiments were performed across a range of learning
rates and the results of the best performing models with and without WSM are
shown in Table 3. The complete set of results across all learning rates is
provided in the appendix, section A.3. The difference between the best
performing models of FedAvg and and FedAvg+WSM is $1.02\%$ for CIFAR-10 and an
impressive $7.6\%$ on CIFAR-100.

Table 3: Accuracy results of FedAvg with and without WSM for different settings of client learning rates using the LeNet architecture. | CIFAR-10 | CIFAR-100
---|---|---
FedAvg | $0.608\pm 0.010$ | $0.198\pm 0.004$
FedAvg+WSM (ours) | $\mathbf{0.624\pm 0.009}$ | $\mathbf{0.274\pm 0.007}$

##  5 Conclusion

In this paper we took a deeper look at the local client forgetting problem, we
show that when a client performs local updates during FL, it risks overly
optimizing its local objective leading to forgetting on other subsets of data.
Local client forgetting degrades the performance of the global model and we
show this phenomenon is especially severe in cases where there is a
significant distribution mismatch across clients. First making the connection
with the catastrophic forgetting problem in the continual learning, we propose
a client level modification of the objective function which we call the
weighted softmax. We show empirically that WSM allows us to mitigate client
level forgetting by demonstrating improved performance for the FedAvg
algorithm when combined with WSM across a range of learning rates. We also
demonstrate these improvements are not limited to FedAvg since we also observe
significant performance increases when WSM is applied to SCAFFOLD and FedProx.
An ablation study demonstrated WSM is particularly effective in the regime of
highly heterogeneous client datasets and/or when a small percentage of clients
are selected at each round. Our results indicate that addressing local client
forgetting in general is an important consideration for federated learning
optimization, one that bears closer scrutiny.

## References

  * Acar et al. (2021) Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama.  Federated learning based on dynamic regularization.  _arXiv preprint arXiv:2111.04263_ , 2021.
  * Achille et al. (2017) Alessandro Achille, Matteo Rovere, and Stefano Soatto.  Critical learning periods in deep neural networks.  _arXiv preprint arXiv:1711.08856_ , 2017.
  * Ahn et al. (2021) Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon.  Ss-il: Separated softmax for incremental learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 844–853, 2021.
  * Aljundi et al. (2019) Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia.  Online continual learning with maximal interfered retrieval.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/15825aee15eb335cc13f9b559f166ee8-Paper.pdf.
  * Boudiaf et al. (2020) Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed.  A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses.  In _European conference on computer vision_ , pp. 548–564. Springer, 2020.
  * Caccia et al. (2022) Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky.  New insights on reducing abrupt representation change in online continual learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=N8MaByOzUfb.
  * Caldas et al. (2018) Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečnỳ, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar.  Leaf: A benchmark for federated settings.  _arXiv preprint arXiv:1812.01097_ , 2018.
  * Chaudhry et al. (2019) Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning.  _arXiv preprint arXiv:1902.10486_ , 2019.
  * Chen & Liu (2018) Zhiyuan Chen and Bing Liu.  Lifelong supervised learning.  In _Lifelong Machine Learning_ , pp. 33–74. Springer, 2018.
  * Davari et al. (2022) MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky.  Probing representation forgetting in supervised and unsupervised continual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16712–16721, 2022.
  * Diao et al. (2020) Enmao Diao, Jie Ding, and Vahid Tarokh.  Heterofl: Computation and communication efficient federated learning for heterogeneous clients.  _arXiv preprint arXiv:2010.01264_ , 2020.
  * Fallah et al. (2020) Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar.  Personalized federated learning: A meta-learning approach.  _arXiv preprint arXiv:2002.07948_ , 2020.
  * Gupta et al. (2022) Sharut Gupta, Kartik Ahuja, Mohammad Havaei, Niladri Chatterjee, and Yoshua Bengio.  FL games: A federated learning framework for distribution shifts.  In _Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022)_ , 2022.  URL https://openreview.net/forum?id=UyfdXCeelfy.
  * Hsieh et al. (2020) Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons.  The non-iid data quagmire of decentralized machine learning.  In _International Conference on Machine Learning_ , pp. 4387–4398. PMLR, 2020.
  * Hsu et al. (2019) Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown.  Measuring the effects of non-identical data distribution for federated visual classification.  _arXiv preprint arXiv:1909.06335_ , 2019.
  * Jastrzębski et al. (2018) Stanisław Jastrzębski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey.  On the relation between the sharpest directions of dnn loss and the sgd step length.  _arXiv preprint arXiv:1807.05031_ , 2018.
  * Jodelet et al. (2022) Quentin Jodelet, Xin Liu, and Tsuyoshi Murata.  Balanced softmax cross-entropy for incremental learning with and without memory.  _Computer Vision and Image Understanding_ , 225:103582, 2022\.
  * Karimireddy et al. (2020) Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh.  Scaffold: Stochastic controlled averaging for federated learning.  In _International Conference on Machine Learning_ , pp. 5132–5143. PMLR, 2020.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Konečnỳ et al. (2016) Jakub Konečnỳ, H Brendan McMahan, Daniel Ramage, and Peter Richtárik.  Federated optimization: Distributed machine learning for on-device intelligence.  _arXiv preprint arXiv:1610.02527_ , 2016.
  * Krizhevsky & Hinton (2009) Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Lesort (2022) Timothée Lesort.  Continual feature selection: Spurious features in continual learning.  _arXiv preprint arXiv:2203.01012_ , 2022.
  * Li et al. (2021) Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He.  A survey on federated learning systems: vision, hype and reality for data privacy and protection.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * Li et al. (2022) Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He.  Federated learning on non-iid data silos: An experimental study.  In _2022 IEEE 38th International Conference on Data Engineering (ICDE)_ , pp. 965–978. IEEE, 2022.
  * Li et al. (2020) Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks.  _Proceedings of Machine Learning and Systems_ , 2:429–450, 2020.
  * Li et al. (2019) Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang.  On the convergence of fedavg on non-iid data.  _arXiv preprint arXiv:1907.02189_ , 2019.
  * Li & Hoiem (2017) Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(12):2935–2947, 2017.
  * Lin et al. (2020) Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi.  Ensemble distillation for robust model fusion in federated learning.  _Advances in Neural Information Processing Systems_ , 33:2351–2363, 2020.
  * Liu (2020) Weitang et al. Liu.  Energy-based out-of-distribution detection.  _Neural information processing systems_ , 2020.
  * Lopez-Paz & Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ , 30, 2017.
  * McCloskey & Cohen (1989) Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pp. 109–165. Elsevier, 1989.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pp. 1273–1282. PMLR, 2017.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * Reddi et al. (2020) Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and H Brendan McMahan.  Adaptive federated optimization.  _arXiv preprint arXiv:2003.00295_ , 2020.
  * Ren et al. (2020) Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al.  Balanced meta-softmax for long-tailed visual recognition.  _Advances in neural information processing systems_ , 33:4175–4186, 2020.
  * Schwarz et al. (2018) Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning.  In _International Conference on Machine Learning_ , pp. 4528–4537. PMLR, 2018.
  * Shoham et al. (2019) Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor-Yosef, and Itai Zeitak.  Overcoming forgetting in federated learning on non-iid data.  _arXiv preprint arXiv:1910.07796_ , 2019.
  * Stich (2018) Sebastian U Stich.  Local sgd converges fast and communicates little.  _arXiv preprint arXiv:1805.09767_ , 2018.
  * Tenison et al. (2022) Irene Tenison, Sai Aravind Sreeramadas, Vaikkunth Mugunthan, Edouard Oyallon, Eugene Belilovsky, and Irina Rish.  Gradient masked averaging for federated learning.  _arXiv preprint arXiv:2201.11986_ , 2022.
  * Wang & Joshi (2018) Jianyu Wang and Gauri Joshi.  Cooperative sgd: A unified framework for the design and analysis of communication-efficient sgd algorithms.  _arXiv preprint arXiv:1808.07576_ , 2018.
  * Wang et al. (2020) Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor.  Tackling the objective inconsistency problem in heterogeneous federated optimization.  _Advances in neural information processing systems_ , 33:7611–7623, 2020.
  * Xu et al. (2022) Chencheng Xu, Zhiwei Hong, Minlie Huang, and Tao Jiang.  Acceleration of federated learning with alleviated forgetting in local training.  _arXiv preprint arXiv:2203.02645_ , 2022.
  * Yan et al. (2021) Gang Yan, Hao Wang, and Jian Li.  Critical learning periods in federated learning.  _arXiv preprint arXiv:2109.05613_ , 2021.
  * Yao et al. (2021) Dezhong Yao, Wanning Pan, Yutong Dai, Yao Wan, Xiaofeng Ding, Hai Jin, Zheng Xu, and Lichao Sun.  Local-global knowledge distillation in heterogeneous federated learning with non-iid data.  _arXiv preprint arXiv:2107.00051_ , 2021.
  * Yoon et al. (2021) Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang.  Federated continual learning with weighted inter-client transfer.  In _International Conference on Machine Learning_ , pp. 12073–12086. PMLR, 2021.
  * Yu et al. (2019) Hao Yu, Sen Yang, and Shenghuo Zhu.  Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 5693–5700, 2019.
  * Zenke et al. (2017) Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _International Conference on Machine Learning_ , pp. 3987–3995. PMLR, 2017.
  * Zhao et al. (2018) Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.  Federated learning with non-iid data.  _arXiv preprint arXiv:1806.00582_ , 2018.
  * Zhu et al. (2021) Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou.  Data-free knowledge distillation for heterogeneous federated learning.  In _International Conference on Machine Learning_ , pp. 12878–12889. PMLR, 2021.

##  Appendix A Appendix

###  A.1 Additional Forgetting studies

In this section we show an additional heatmaps, similar to those from figure
3. The heatmaps shown in figure 3 is for round 1200, $\frac{3}{10}$ of the way
through training. Figure 7 shows heatmaps for round 3600, $\frac{9}{10}$ of
the way through training which illustrates local client forgetting and the
ability of WSM to reduce its effects are present throughout the training
cycle.

Figure 7: We show local client forgetting for round 3600 with and without WSM.
The heatmap is structured the same way as for figure 3 described previously
where the y-axis indicates the model of client i and the x-axis indicates the
local data of client k. Again at round 3600, much later in training than round
1200 shown in figure 3 we observe FedAvg+WSM (top row) significantly reduces
forgetting across clients.

###  A.2 Complete Result Set Across Learning Rates and Normalization Methods

Table 4 shows the complete set of results across learning rates and
normalization methods. We observe FedAvg+WSM has a strong performance with
batch normalization contrary to the findings of Hsieh et al. (2020) while
FedAvg preforms better using group norm.

Table 4: Accuracy results of FedAvg with and without WSM for different hyperparameters. We observe that FedAvg+WSM with batch normalization consistently improves performance over FedAvg, as well as having the highest overall accuracy by a large margin. WSM also makes the learning rate easier to tune since we observe a large hyperparameter range  Method | _Hyper-params_ | _Dataset_
---|---|---
lr | norm | CIFAR-10 | CIFAR-100 | FEMNIST
FedAvg | 0.5 | group | $0.326\pm 0.098$ | $0.292\pm 0.012$ | $0.542\pm 0.087$
FedAvg+WSM (ours) | $0.452\pm 0.173$ | $0.234\pm 0.191$ | $0.418\pm 0.177$
FedAvg | batch | $0.742\pm 0.004$ | $0.386\pm 0.003$ | $0.812\pm 0.020$
FedAvg+WSM (ours) | $\mathbf{0.792\pm 0.006}$ | $\mathbf{0.426\pm 0.003}$ | $\mathbf{0.837\pm 0.002}$
1-6  FedAvg | 0.3 | group | $0.791\pm 0.013$ | $0.384\pm 0.013$ | $0.769\pm 0.006$
FedAvg+WSM (ours) | $0.744\pm 0.007$ | $0.388\pm 0.027$ | $0.761\pm 0.118$
FedAvg | batch | $0.742\pm 0.004$ | $0.412\pm 0.014$ | $0.815\pm 0.008$
FedAvg+WSM (ours) | $\mathbf{0.834\pm 0.008}$ | $\mathbf{0.467\pm 0.015}$ | $\mathbf{0.844\pm 0.010}$
1-6  FedAvg | 0.1 | group | $0.724\pm 0.027$ | $0.500\pm 0.016$ | $0.835\pm 0.002$
FedAvg+WSM (ours) | $0.794\pm 0.022$ | $0.446\pm 0.004$ | $0.827\pm 0.012$
FedAvg | batch | $0.820\pm 0.006$ | $0.442\pm 0.016$ | $0.806\pm 0.031$
FedAvg+WSM (ours) | $\mathbf{0.855\pm 0.004}$ | $\mathbf{0.514\pm 0.009}$ | $\mathbf{0.848\pm 0.006}$
1-6  FedAvg | 0.07 | group | $0.826\pm 0.007$ | $0.437\pm 0.007$ | $\mathbf{0.827\pm 0.006}$
FedAvg+WSM (ours) | $0.805\pm 0.007$ | $0.484\pm 0.041$ | $0.823\pm 0.006$
FedAvg | batch | $0.787\pm 0.006$ | $0.513\pm 0.006$ | $0.789\pm 0.003$
FedAvg+WSM (ours) | $\mathbf{0.856\pm 0.005}$ | $\mathbf{0.553\pm 0.018}$ | $0.826\pm 0.019$
1-6  FedAvg | 0.05 | group | $0.827\pm 0.004$ | $0.464\pm 0.001$ | $\mathbf{0.853\pm 0.004}$
FedAvg+WSM (ours) | $0.791\pm 0.019$ | $0.454\pm 0.015$ | $0.841\pm 0.002$
FedAvg | batch | $0.790\pm 0.012$ | $0.531\pm 0.007$ | $0.833\pm 0.024$
FedAvg+WSM (ours) | $\mathbf{0.858\pm 0.003}$ | $\mathbf{0.564\pm 0.007}$ | $0.842\pm 0.005$
1-6  FedAvg | 0.03 | group | $0.836\pm 0.005$ | $0.431\pm 0.020$ | $\mathbf{0.835\pm 0.006}$
FedAvg+WSM (ours) | $0.774\pm 0.035$ | $0.472\pm 0.007$ | $0.830\pm 0.006$
FedAvg | batch | $0.779\pm 0.028$ | $0.561\pm 0.010$ | $0.756\pm 0.015$
FedAvg+WSM (ours) | $\mathbf{0.857\pm 0.005}$ | $\mathbf{0.581\pm 0.005}$ | $0.834\pm 0.003$
1-6  FedAvg | 0.01 | group | $0.815\pm 0.003$ | $0.431\pm 0.005$ | $\mathbf{0.830\pm 0.002}$
FedAvg+WSM (ours) | $0.785\pm 0.011$ | $0.471\pm 0.010$ | $0.800\pm 0.018$
FedAvg | batch | $0.787\pm 0.003$ | $0.566\pm 0.009$ | $0.744\pm 0.011$
FedAvg+WSM (ours) | $\mathbf{0.845\pm 0.006}$ | $\mathbf{0.574\pm 0.006}$ | $0.800\pm 0.019$
1-6  FedAvg | 0.007 | group | $0.817\pm 0.007$ | $0.426\pm 0.005$ | $\mathbf{0.821\pm 0.011}$
FedAvg+WSM (ours) | $0.773\pm 0.014$ | $0.476\pm 0.010$ | $0.794\pm 0.006$
FedAvg | batch | $0.797\pm 0.008$ | $\mathbf{0.568\pm 0.003}$ | $0.734\pm 0.018$
FedAvg+WSM (ours) | $\mathbf{0.841\pm 0.004}$ | $\mathbf{0.568\pm 0.003}$ | $0.800\pm 0.007$
1-6  FedAvg | 0.005 | group | $0.802\pm 0.010$ | $0.426\pm 0.002$ | $\mathbf{0.819\pm 0.022}$
FedAvg+WSM (ours) | $0.768\pm 0.019$ | $0.474\pm 0.006$ | $0.781\pm 0.017$
FedAvg | batch | $0.783\pm 0.009$ | $0.553\pm 0.005$ | $0.743\pm 0.053$
FedAvg+WSM (ours) | $\mathbf{0.826\pm 0.009}$ | $\mathbf{0.554\pm 0.004}$ | $0.773\pm 0.013$

###  A.3 LeNet Performance Across Multiple Learning Rates

Table A.3 shows model performance using the LeNet architecture across learning
rates. WSM outperforms vanilla FedAvg for each learning rate for both
datasets. As with the ResNet-18 case, we continue to observe that WSM provides
good performance over a larger range of learning rates than FedAvg which makes
it easier to tune. We also observe WSM reduces the variance in model accuracy
as evidenced by the typically lower standard deviations reported for the runs
in each set.

Table 5: Accuracy results of FedAvg with and without WSM for different settings of client learning rates using the LeNet architecture. | lr | CIFAR-10 | CIFAR-100
---|---|---|---
FedAvg |  | $0.589\pm 0.012$ | $0.119\pm 0.002$
FedAvg+WSM (ours) | 0.07 | $\mathbf{0.624\pm 0.009}$ | $\mathbf{0.180\pm 0.013}$
1-5  FedAvg |  | $0.580\pm 0.051$ | $0.154\pm 0.011$
FedAvg+WSM (ours) | 0.05 | $\mathbf{0.615\pm 0.004}$ | $\mathbf{0.217\pm 0.009}$
1-5  FedAvg |  | $0.605\pm 0.014$ | $0.168\pm 0.001$
FedAvg+WSM (ours) | 0.03 | $\mathbf{0.621\pm 0.009}$ | $\mathbf{0.251\pm 0.007}$
1-5  FedAvg |  | $0.608\pm 0.010$ | $0.168\pm 0.044$
FedAvg+WSM (ours) | 0.01 | $\mathbf{0.624\pm 0.009}$ | $\mathbf{0.274\pm 0.007}$
1-5  FedAvg |  | $0.591\pm 0.014$ | $0.195\pm 0.004$
FedAvg+WSM (ours) | 0.007 | $\mathbf{0.620\pm 0.022}$ | $\mathbf{0.274\pm 0.001}$
1-5  FedAvg |  | $0.585\pm 0.019$ | $0.198\pm 0.004$
FedAvg+WSM (ours) | 0.005 | $\mathbf{0.615\pm 0.003}$ | $\mathbf{0.270\pm 0.001}$
1-5  FedAvg |  | $0.545\pm 0.018$ | $0.181\pm 0.009$
FedAvg+WSM (ours) | 0.003 | $\mathbf{0.566\pm 0.001}$ | $\mathbf{0.247\pm 0.004}$
1-5  FedAvg |  | $0.441\pm 0.019$ | $0.115\pm 0.007$
FedAvg+WSM (ours) | 0.001 | $\mathbf{0.478\pm 0.009}$ | $\mathbf{0.186\pm 0.003}$
1-5 |  |  |

###  A.4 Illustrating different dirichlet parameters

Figure 8 offers a practical illustration of how client partitions change as a
function of $\alpha$. Clients with $\alpha=0.01$ have only a small percentage
of the classes while at $\alpha=100$ clients have all 10 classes in
proportions that are much more equal than we observe with the other two
parameterizations.

Figure 8: Percentages of each class label for ten randomly selected clients
with $\alpha=0.01,0.5,100$ from left to right


# Information Retrieval from the Digitized Books

Riya Gupta
IIIT Hyderabad
India
<EMAIL_ADDRESS>C.V. Jawahar
IIIT Hyderabad
India
<EMAIL_ADDRESS>
###### Abstract

Extracting the relevant information out of a large number of documents is a
challenging and tedious task. The quality of results generated by the
traditionally available full-text search engine and text-based image retrieval
systems is not optimal. Information retrieval (IR) tasks become more
challenging with the nontraditional language scripts, as in the case of Indic
scripts. The authors have developed OCR (Optical Character Recognition) Search
Engine to make an Information Retrieval & Extraction (IRE) system that
replicates the current state-of-the-art methods using the IRE and Natural
Language Processing (NLP) techniques. Here we have presented the study of the
methods used for performing search and retrieval tasks. The details of this
system, along with the statistics of the dataset (source: National Digital
Library of India or NDLI), is also presented. Additionally, the ideas to
further explore and add value to research in IRE are also discussed.

## Keywords

Optical Character Recognition, Information Retrieval & Extraction, Metadata,
Search Engine, Search Engine Optimization

##  1 Introduction

Often the information retrieval systems designed to provide and assist the
general populace with texts in different languages that are either not
optimized or lack advanced language features, like getting similar results to
a query. Even for the big digital database like Google Books, the contextual
and semantic information is not available for the desired query. Short queries
might not explain the full intent behind the search. However, suppose the
Information Retrieval and Extraction systems (IRE) can be made capable in
terms of obtaining semantically and contextually rich information from the
ocean of data. In that case, it will improve the user experience. It will also
help in the direction of extracting the required query.

This work focuses on multimedia retrieval (image and text in Indic languages)
in this IRE system, where the user enters a text query for which the system
returns books that are indexed and ranked. Multilingual OCR[11, 8] is used to
achieve this task. We know that optical Character Recognition (OCR) is the
process of conversion of text or text-containing documents such as handwritten
text, printed or scanned text images into a digital format that can be used
for further processes. In this system, text recognition is performed on
ancient Indic language documents. The final information retrieval system’s
goal is to demonstrate the high accuracy of the Indic OCR and the retrieval of
the ancient texts in Indic languages.

The challenges faced by IRE systems in Indic language scenarios are discussed
in Section 2. Section 3 consists of the details and functionalities of the OCR
Search Engine designed by the authors. Section 4 represents the ideas that can
be implemented on top of the existing system.

##  2 Practical Challenges associated with Indian IRE systems

Modern state-of-the-art IRE systems suffer from various challenges for Indian
data. These challenges can vary among different areas within information
retrieval. These areas majorly include cross-lingual information retrieval
(CLIR)[10], Web search, user modeling, filtering, classification,
summarization, question answering, metasearch, semantic, and context-based
search and extraction[1]. Based on the current trends and research work, these
are some of the challenges that the community faces at the moment:

###  2.1 Limited Research Aid

According to the latest Indian Census, India has 24 officially recorded
languages. These include Hindi and English. Although most of these languages
have a similar grammatical structure and layout, the semantics and script
differ drastically between them. Furthermore, a particular language varies
widely in fonts, rendering schemes, dialects and other aspects depending on
the region.

Moreover, the number of researchers contributing in the field of information
retrieval and text-based challenges for Indian languages (Hindi, Bangla,
Sanskrit, Tamil, Telugu and Malayalam) are comparatively smaller due to the
diversity in them, as well as due to the involvement of various modalities for
instance images, text or audio.

###  2.2 Limited amount of available data

One of the reasons for the lack of research in building quality IRE systems
for Indian languages is the lack of data for information retrieval. Even when
the data is publicly available, it is not readily usable and requires
extensive cleaning and preprocessing to make it suitable for the retrieval
models. Current state-of-the-art methods like WordNet[13], ConceptNet[18] are
trained on large datasets for Roman and Latin scripts. Similar datasets are
tedious to build due to the reasons stated in 2.1. Thus, the unavailability of
quality data for this task hinders the progress in the field.

Recently, several groups from NDLI, IIT Kharagpur, have converted and
digitalized a large number of historical documents like ”Bhagawadgitha”. These
documents can act as a large dataset if annotated. However, even if it is
present, the dataset is majorly noisy. Modern text recognition algorithms in
this space are not designed to handle such noisy data and hence perform
poorly.

|
---|---
Figure 1: In this system, old and degraded document images are passed through
an OCR, which hampers the quality of the generated text. The figure shows an
example of a sample degraded document in Bangla language.

Figure 1 shows a few examples of the data from the Bangla documents. The noise
might include the warped lines within the documents, watermarked pages and
traces of ink from the backside of the page.

###  2.3 Semantics and Context based Retrievals

Web Search Engines and Information Retrieval are two aspects where web search
is just a spectrum of Information retrieval. The information which is given as
an input to the search bar does not always represent the practical and
required information that a user demands. Understanding the rich semantic and
context behind a text query is extremely important for the search engines to
return relevant results. For instance, in the search engines Yahoo and Bing,
if a person enters a search query ’Nokia’, the results generated will be all
the links containing Nokia, irrespective of their usability for the user.
Whereas in Google search, the links are ranked in order of details about Nokia
and currently available cellphones of Nokia showing a higher semantic
understanding of the Google search engine than that of Yahoo and Bing. Similar
features are needed to be implemented for the Indian language based IRE
system.

However, due to the complexity of Indian languages, newer techniques
specifically designed for them are required for generating rich semantic
embedding. Hence, building algorithms that can extract rich information from
Indic texts and later combine them with user search history and search
patterns can be beneficial.

##  3 What is OCR Search Engine?

In recent years, there has been a lot of research in the field of text
recognition, and efforts have been put into achieve an OCR that can perform
well on Indian scripts. Indic scripts pose unique challenges for state-of-the-
art OCR algorithms trained in English. For example, Sandhi rules and the
difference in glyphs make it harder for the state-of-the-art OCR algorithms to
work for such languages. makes it harder for the State-of-the-art OCR
algorithms to work for such languages. Unavailability of IRE algorithms and
dictionaries in Indic languages that can fetch similar words due to
differences in the script and colossal amount of data to a given query hampers
the search engine’s performance. The developed OCR Search Engine that
demonstrates the accuracy of the multilingual Indic OCR along with the
retrieval accuracy.

This OCR Search Engine is an IRE system that uses Optical Character
Recognition designed for Indian scripts to generate the text from the document
images in the Indic languages. Currently, it supports Hindi, Tamil, and Telugu
languages.

Figure 2: Basic pipeline followed by the OCR Search Engine

The system is built using Python and PSQL, forming the back-end and HTML/CSS
being used as the front end. It uses the text generated by the Indic OCR
developed at IIIT Hyderabad, line-level segmentation, and document images to
form the database of the system. The database is indexed using Elasticsearch,
which is an open-source distributed Java package.

###  3.1 Functionalities

The IRE system designed by us performs retrieval tasks on multiple levels :

  * •

Query Level Search
The task is performed by entering the keyword/text in the search bar. There
are multiple levels within the query level search like:

    * –

Single Query Search: Search based on one keyword search

    * –

Multiple Query Search: Search based on more than one keywords

    * –

Exact Query Search : Search based on retrieving exact entered query (by using
double quotes)

    * –

Query Processing: Removal of stopwords and implementation of tokenization and
stemming.

  * •

Relevance and Visualization

    * –

Relevance: Based on the number of occurrence of a particular keyword in the
database and number of hits for a particular book from users.

    * –

Visualization: Highlighted query outputs within the resulting books using
segmentation method.

    * –

Pages with figures and varying font styles and sizes retrievable.

  * •

Language Level Search
This level of search is completely based on the language chosen to retrieve
the query.

    * –

Transliteration: A method of mapping from one system of writing to another
based on phonetic similarity.

    * –

Single language search

    * –

Multilingual Search: Based on the search in more than one language and
retrieval of multilingual books.

  * •

Search Engine Based

    * –

Primary Filters: Filtering based on the languages (Hindi, Tamil and Telugu)
and the other filters including content based filter (default), ISBN Code,
Author name and Book title.

    * –

Secondary Filters: Filtering based on Genre and Source provider of the book.

###  3.2 Dataset

Table 1 shows the dataset and its details. Currently, the IRE system supports
around 1.5 million documents. The detailed statistics are given in the
following table.

Language | Hindi | Tamil | Telugu
---|---|---|---
Books | 2287 | 2030 | 2574
Pages | 5,18,500 | 5,18,000 | 5,07,070
Table 1: Statistics of the dataset

###  3.3 Design

The OCR Search Engine is structured on the following basis :

  1. 1.

User End : Home, Search page, View, See Results About.

  2. 2.

Server End: Login, Database Status, Manual Updates to Database.

The following subsection represents the brief description of these ends.

####  3.3.1 User

The user side is the client-side for retrieving the query entered by the user.
The Home page is the launching page where the users can choose the filters for
their search choices. This homepage presents the total number of documents
that this IRE system supports. Users can select the filters, whether it be
language (Hindi, Tamil, Telugu) or search criteria (title, ISBN, author,
default: content). These will fetch the results based on the content present
within the book and use text output generated from the Indic OCR.

Figure 3: Example of the metadata provided for one of the Hindi books

The other filters present are termed ’secondary filters’ that include genre
and the source if the book is extracted from the metadata provided. One of the
examples of the metadata provided is given.

Figure 4: Clicking ’See results about’ opens up a pop up with highlighted
lines in a document where the query is present

The search page displays all the retrieved results. Every result consists of
the cover page of the book, author, title, language, ISBN, short abstract (if
available), an option to View the book and See results about, clicking on
which a pop-up appears that has all the pages of a book with the searched
query present in them. As the current OCR system uses line-level segmentation,
the line in which the query is present is highlighted for making it easier for
the users to search it.

Figure 5: Clicking ”View” will open a new taskbar where users can view and
read the entire book.

####  3.3.2 Server End

The Admin can Login inside the IRE system and assign various roles to the
other admins, whether they can alter the entries in the database or just
monitor them. Database Status can be viewed as a graph that gives the total
number of pages and the total number of books added within the entire database
over a period. Featured datasets lists the datasets present within the system
(currently by the National Digital Library Data).

Manual Updates to the database are also possible. The Admin can edit/delete
the books’ details, including Abstract, ISBN and Author.

###  3.4 System Implementation

The backend of the IRE system is built over the top of Django/python, and the
frontend uses HTML/CSS and Javascript. The pages are indexed using
Elasticsearch. The dynamic components of these IRE systems include the Search
results, filters, visualization where the query searched is highlighted, admin
authentication, and manually updating the database.

The admin/supervisor has the right to completely modify/delete/edit the
dataset/database manually/automatically depending on his/her needs. This will
not affect the already available dataset/database present in the system.

Workflow: Users can view the retrieved results (retrieved books w.r.t. to the
query searched), the highlighted queries, meta details, and the complete
book(s). The download options are currently unavailable. The users do not need
to log in to use the system, and this makes the search and the generated
results accessible for everyone to view and read.

##  4 Future Work

These are the few proposed ideas that can be implemented over the top of the
existing system that has been built and discussed here.

####  4.0.1 Abstract Generation

Abstracts are necessary when a user cannot dedicate his time reading the
complete book. It allows a user to understand the summary of the book, thus
enable a user to filter out books of his/her interest. This becomes more
relevant in the case of Indic languages where there are religious texts in
different languages; for instance, it might be the case that the user would
not want to read the entire drama book in Hindi but the gist of a particular
page. State-of-the-art attention-based models [20, 23, 19] can be used to
generate a fluent, concise, and understandable summary of the complete book.
This can even be used to create summaries of chapters or pages within the
book, which will make the user experience more smooth and increase the
system’s usage. Relevant keywords, frequency within the document, and
readability will contribute to the abstract generation. This feature can
further be enhanced by adding an audio option, i.e., Text-To-Speech (TTS),
which will recite the abstract of the book to the user.

####  4.0.2 Semantic and Context Based IRE

Usually, the searched queries are short or lack of intent behind the search.
These queries might generate ambiguous results that the users do not expect.
The current system designs fetch results using text processing techniques like
tokenization and lemmatization in the Indic query entered or by extracting the
exact matching query. However, it does not generate similar results or
semantically rich results. For example, suppose the query ’Ram aur unke putra’
is entered. In that case, the IRE system should be able to identify the
semantics and relationship between the components ’Ram’ and ’Putra’ and
generate relevant results containing details about ’Luv’ and ’Kush’[24, 17,
4].

By using natural language processing technologies, word similarity techniques,
ontology techniques, etc., to understand the nature of the words and queries,
the search results can be improved. Controlled vocabularies, dictionaries,
thesaurus, and taxonomies are some of the basic ontology techniques that are
used during the annotation. In text-based retrieval, these techniques improve
the extraction of the information easily. The other method that can be used to
achieve semantic Information retrieval is treating words as entities or
concepts. In concept indexing, instances(entities) and abstract
ideas(concepts) are identified within a text document and are linked to the
ontological concepts. Methods like WordNet[13], OpenCyc[12] and ConceptNet[9]
can be used to achieve some results in the case of Indic languages.

Additionally, the search results in the current system can be improved by
using Query Expansion(QE)[2][3]. QE is performed by broadening the query by
introducing additional keywords within the originally entered query. These
usually include abbreviations, synonyms, antonyms, hyponyms, and hypernyms.
Unsupervised or Supervised methods can be used to achieve this. Doc2Vec and
Word2Vec can be used to capture the surface and semantic level similarity and
perform Query Expansion.

####  4.0.3 System Based

Multiple features can be added to the system to enhance it. Machine
Translation(MT)[22, 15, 5, 7, 14] is one of the features that can be added. A
user can type the query in one language and retrieve the results based on the
meaning of that query in another language. For example, ”Mahabharat” searched
in ”Hindi” should fetch ”Mahabharatha” results from the languages Tamil and
Telugu too.

Image and Audio based search methods can also be added where the entered query
can either be audio or image. The audio search method is Speech to Text
conversion, where a user can speak, and the query will automatically be
translated and entered into the search bar. Image-based/Document Information
Retrieval[16, 21] is another search method that can be implemented. In this
work [6] by VGG group, the retrieval is performed over the object category on
British Library data.

Additionally, adding the most popular queries searched by the users over time
and generation of related articles, magazines, as well as books concerning the
book clicked by users are the few features that can be added over the top of
the currently existing OCR Search Engine developed at CVIT, to make it more
accessible and handy during usage.

###  4.1 Conclusion

The end goal is to develop and deploy an enhanced Indic IRE system that can
exploit the large corpora provided by the NDLI, and the output generated by
the Indic OCR grew at CVIT. This system will improve the user search
experience for the Indic languages and increase the popularity of reading
digitized books amongst the general populace. This system is currently being
developed by adding more functionalities and newer languages like Bangla and
Sanskrit.

## References

  * [1] James Allan, J. Aslam, Nicholas Belkin, Chris Buckley, J. Callan, B. Croft, S. Dumais, Norbert Fuhr, Donna Harman, D.J. Harper, Djoerd Hiemstra, Thomas Hofmann, Wessel Kraaij, J. Lafferty, V. Lavrenko, David Lewis, Liz Liddy, R. Manmatha, and C.X. Zhai.  Challenges in information retrieval and language modeling: report of a workshop held at the center for intelligent information retrieval, university of massachusetts amherst. september 2002.  ACM SIGIR Forum, 37, 01 2003.
  * [2] Hiteshwar Kumar Azad and Akshay Deepak.  Query expansion techniques for information retrieval: a survey.  CoRR, abs/1708.00247, 2017.
  * [3] Rafael E. Banchs, Monojit Choudhury, Paolo Rosso, Parth Gupta, and Kalika Bali.  Query expansion for mixed-script information retrieval.  pages 677–686. ACM - Association for Computing Machinery, July 2014.
  * [4] Youssef Bassil and Paul Semaan.  Semantic-sensitive web information retrieval model for HTML documents.  CoRR, abs/1204.0186, 2012.
  * [5] Himanshu Choudhary, Aditya Kumar Pathak, Rajiv Ratan Saha, and Ponnurangam Kumaraguru.  Neural machine translation for English-Tamil.  In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 770–775, Belgium, Brussels, Oct. 2018. Association for Computational Linguistics.
  * [6] Elliot J. Crowley and Andrew Zisserman.  In search of art.  In Lourdes Agapito, Michael M. Bronstein, and Carsten Rother, editors, Computer Vision - ECCV 2014 Workshops, pages 54–70, Cham, 2015\. Springer International Publishing.
  * [7] Ayan Das, Pranay Yerra, Ken Kumar, and Sudeshna Sarkar.  A study of attention-based neural machine translation model on Indian languages.  In Proceedings of the 6th Workshop on South and Southeast Asian Natural Language Processing (WSSANLP2016), pages 163–172, Osaka, Japan, Dec. 2016. The COLING 2016 Organizing Committee.
  * [8] Deepayan Das, Jerin Philip, Minesh Mathew, and C. V. Jawahar.  A cost efficient approach to correct OCR errors in large document collections.  CoRR, abs/1905.11739, 2019.
  * [9] Ming-Hung Hsu, Ming-Feng Tsai, and Hsin-Hsi Chen.  Query expansion with conceptnet and wordnet: An intrinsic comparison.  In Proceedings of the Third Asia Conference on Information Retrieval Technology, AIRS’06, page 1–13, Berlin, Heidelberg, 2006. Springer-Verlag.
  * [10] Jagadeesh Jagarlamudi and A. Kumaran.  Cross-lingual information retrieval system for indian languages.  volume 5152, pages 80–87, 09 2007.
  * [11] M. Mathew, A. K. Singh, and C. V. Jawahar.  Multilingual ocr for indic scripts.  In 2016 12th IAPR Workshop on Document Analysis Systems (DAS), pages 186–191, April 2016.
  * [12] Cynthia Matuszek, John Cabral, Michael Witbrock, and John DeOliveira.  An introduction to the syntax and content of cyc.  pages 44–49, 01 2006.
  * [13] Vuong M. Ngo, Tru H. Cao, and Tuan M. V. Le.  Wordnet-based information retrieval using common hypernyms and combined features.  CoRR, abs/1807.05574, 2018.
  * [14] Jerin Philip, Vinay P. Namboodiri, and C. V. Jawahar.  A baseline neural machine translation system for indian languages.  CoRR, abs/1907.12437, 2019.
  * [15] Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya.  Multilingual unsupervised NMT using shared encoder and language-specific decoders.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3083–3089, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [16] S. Senda, M. Minoh, and K. Ikeda.  Document image retrieval system using character candidates generated by character recognition process.  In Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR ’93), pages 541–546, Oct 1993.
  * [17] Rossitza Setchi, Qiao Tang, and Ivan Stankov.  Semantic-based information retrieval in support of concept design.  Advanced Engineering Informatics, 25(2):131–146, 2011.
  * [18] Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  CoRR, abs/1612.03975, 2016.
  * [19] Tomek Strzalkowski, Gees C Stein, G Bowden Wise, and Amit Bagga.  Towards the next generation information retrieval.  In RIAO, pages 1196–1207, 2000.
  * [20] Lu Wang and Wang Ling.  Neural network-based abstract generation for opinions and arguments.  CoRR, abs/1606.02785, 2016.
  * [21] K. L. Wiggers, A. S. Britto, L. Heutte, A. L. Koerich, and L. E. S. Oliveira.  Document image retrieval using deep features.  In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8, July 2018.
  * [22] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  CoRR, abs/1609.08144, 2016.
  * [23] Pengcheng Yin and Graham Neubig.  Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation.  arXiv preprint arXiv:1810.02720, 2018.
  * [24] Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck.  Generating clarifying questions for information retrieval.  In The Web Conference 2020 (formerly WWW conference), April 2020\.