1088 System or Tool inkscapelatex=false

#  Almanac: An API for Recommending Text Annotations For Time-Series Charts
Using News Headlines

Terrell Ibanez
Stanford University
Tableau Research e-mail<EMAIL_ADDRESS>Vidya Setlur
Tableau Research  e-mail<EMAIL_ADDRESS>Maneesh Agrawala
Stanford University e-mail<EMAIL_ADDRESS>
###### Abstract

Authors often add text annotations to charts to provide additional context for
visually prominent features such as peaks, valleys, and trends. However,
writing annotations that provide contextual information, such as descriptions
of temporal events, often requires considerable manual effort. To address this
problem, we introduce Almanac, a JavaScript API that recommends annotations
sourced from the New York Times Archive of news headlines. Almanac consists of
two independent parts: (1) a prominence feature detector and (2) a contextual
annotation recommender. We demonstrate the utility of the API using D3.js and
Vega-Lite to annotate a variety of time-series charts covering many different
data domains. Preliminary user feedback shows that Almanac is useful to
support the authoring of charts with more descriptive annotations.

###### keywords:

Time trends, visual saliency, causality, explanation.

A time-series chart depicting the number of acres burned in various California
Wildfires between 2013 and 2020. We used the Almanac API to identify the
prominent visual features of this chart (i.e. the peaks) and then annotate
them with headlines from the New York Times. The headlines correctly name the
fire responsible for each labeled peak and link to the corresponding article,
thereby providing explanatory context for the peak. In this case, we used the
top-ranked headline identified by Almanac, but chart authors can manually
choose lower-ranked headlines as well. The one peak left unlabeled did not
return any suitable headline from the New York Times Archive because they did
not cover the wildfire responsible for it.

Human-centered computingVisualizationVisualization techniquesTreemaps; Human-
centered computingVisualizationVisualization design and evaluation methods

##  1 Introduction

Text annotations play an important role in communicating insights about the
data depicted in charts. Chart authors often include annotations to draw the
readers’ attention to specific visual features (e.g., peaks, valleys, and
trends) and to provide additional context [21, 24]. Such text is especially
effective when it provides external information not directly present in the
chart that explains the visually prominent features [15]. For example, Figure
Almanac: An API for Recommending Text Annotations For Time-Series Charts Using
News Headlines shows a time series of acres burned in California wildfires
from 2013 to 2020 [3], and the text annotations name the fire responsible for
each peak while providing additional information about the damage they caused.

Yet, authoring such contextual annotations for the visually prominent features
in a chart requires considerable manual effort. Authors have to identify the
prominent features and examine a variety of external sources (e.g., news
websites) to find relevant information. While researchers have developed
prototype tools for automating chart annotation, the implementations have been
fairly limited. For example, Contextifier [12] is limited to a set of 11
stocks and corresponding financial news articles for the years 2010-2012,
limiting its applicability to more general data sets.

In this work, we present Almanac, a JavaScript API for recommending contextual
annotations from the New York Times (NYT) Archive [29], to any time-series
data visualization that is designed to fit into the workflow of web-based
chart design. Almanac provides two main components; (1) A prominence feature
detector that identifies visually prominent peaks, valleys, and trends in an
input time-series chart. While our built-in detector uses a topological
persistence algorithm [10] to identify these features, the API is designed so
that users can easily swap in other detectors or manually specify the
prominent features to be annotated. (2) A contextual annotation recommender
that takes as input a time-series chart and a visually prominent feature
(specified as a point or time range) and suggests contextually relevant
headlines. We demonstrate the utility of Almanac by annotating a variety of
time-series data sets in D3.js [4], Vega-Lite [25], Observable Notebooks [23]
and Tableau [28].

##  2 Related Work

Figure 1: Chart authors use the Almanac API in two steps. Given a time-series
chart (left) of stock price data for Tesla Inc. [13], the author applies the
getChartFeatures command to return the most prominent peaks and valleys in the
chart (left, marked in red). Authors can manually mark point and trend
features as well. The author then selects the most prominent peak and runs
Almanac’s getAnnotations command to obtain a set of contextual headlines
(middle top). By default, the highest-ranked headline is chosen as the
annotation, but the author can manually examine the other headlines to see if
a better headline is available. The author uses the top-ranked default
headlines to create the final chart (right).

Prior research has explored how authors add annotations to charts guiding a
reader’s attention to visual features in the chart and explaining what the
underlying data means with additional context [26, 11]. Kong and Agrawala
developed techniques for analyzing charts to recover visually salient features
of the data-encoding marks (e.g., min, max, mean values). Users could then
interactively add graphical and text annotations or overlays to facilitate
chart reading [20, 19].

Kandogan [14] introduced just-in-time descriptive analytics by employing
statistics to generate annotations for clusters and outliers. Autotator is a
semi-automatic chart annotation system that provides suggestions for three
annotation tasks, i.e., (1) labeling a chart type, (2) annotating bounding
boxes, and (3) associating a quantity [16]. Our work focuses on providing
external context rather than just annotating a chart with low-level
statistical information. Several projects have also aimed to automatically
generate visualizations for a text article [6] and personalize articles with
text and visualizations based on information about the specific reader [1, 17,
2, 8].

Closest to our work is Contextifier [12], which uses news headlines to provide
external contextual annotations for line charts. Their implementation takes as
input, a financial news article and generates an annotated price chart for the
most frequently mentioned stock in the article. They consider linguistic
relevance, the number of article views, and the visual saliency of chart peaks
to identify the headlines and chart features to annotate. But their
implementation is limited to a relatively small corpus of stocks and financial
articles covering 2010-2012. In contrast, by working with a large corpus of
2,425,425 NYT headlines from 1980 to 2022, our API can handle more general
data sets and covers a longer period of time. Finally, unlike our work,
Contextifier does not provide a JavaScript API that can be incorporated into
web-based visualizations.

.

##  3 Almanac JavaScript API

The Almanac API comprises two main components: a prominence feature detector
(Section 3.1) and a contextual annotation recommender (Section 3.2). While
both components are exposed to chart authors as JavaScript function calls
getChartFeatures and getAnnotations respectively, the prominence feature
detector is implemented in JavaScript and the annotation recommender is
implemented as a Python REST API. We have architected the two components to be
completely independent of one another to facilitate reuse in contexts beyond
chart annotation. The primary input to both components is a time-series data
set $T$, an associated granularity $T_{G}$ (e.g., year, month, day) that
specifies the sample rate of the time series, and a set of keywords, $T_{W}$
that describe the overall data domain of the chart (e.g., “california
wildfires”, “covid cases”).

Almanac’s recommender component includes a built-in annotation database
consisting of NYT headlines that we curated using the NYT Archive API [29]. We
considered this as our built-in database because the NYT is a general and
broad news source covering a wide variety of events and topics and hence would
be more likely to provide relevant annotation for a variety of time-series
data sets. We used the NYT API to obtain all of the metadata, including the
headline, publish date, article type (e.g., news, opinion), and lede paragraph
for NYT articles published between January 1, 1980 and December 31, 2022. We
removed duplicates and filtered the database to include only articles that
cover news events. The resulting database contains metadata for 2,425,425
articles.

###  3.1 Prominence Feature Detector

Almanac’s prominence feature detector takes a time-series data set as input
and outputs a ranked ordering of the peaks (or valleys) in the data based on
their prominence. In JavaScript, a chart author invokes the Almanac feature
detector as,

$P=\texttt{getChartFeatures}(T)$

where $T$ is a time-series data set, $P$ is a set of peak (or valley features
$p_{i}$ and $i$ is the prominence rank of feature $p_{i}$. The input time
series $T$ is associated with a granularity $T_{G}$ (e.g. year, month, day)
that specifies the sample rate of the time series. Each output feature $p_{i}$
can be a single point in time (corresponding to a peak or valley) or interval
in time (corresponding to a trend or slope).

Figure 2: Charts depicting times-series data of Mass Shooting Deaths (a) and
Presidential Approval Ratings (b). Almanac API identifies prominence features
(peaks for a, peaks and valleys for b) and recommends NYT headlines. All
annotations shown are the top-ranked. Note that some of the peaks and valleys
Almanac deemed prominent, are unlabeled because the annotation recommender did
not find a suitable headline.

Figure 3: Charts depicting times-series data of Cable TV News coverage of
Afghanistan (a) and Simpsons Viewership (b). We used the Almanac API to
identify prominence features (peaks) and to recommend NYT headlines for them.
All annotations shown are the top-ranked. Note that some of the peaks Almanac
deemed prominent are unlabeled because the annotation recommender did not find
a suitable headline for them.

Almanac provides a feature detector that measures the prominence of peaks (or
valleys) in the data as a proxy for their visual salience. The detector
computes the prominence of peaks using a topological data analysis technique
[10, 18], which uses a watershed model to determine how long a peak remains
independent when the surrounding relief is metaphorically drained of water.

###  3.2 Contextual Annotation Recommender

Almanac’s contextual annotation recommender uses keyword anomalies to surface
relevant headlines. The keywords appear within a time range associated with a
visual feature but do not appear in ranges associated with other visual
features, allowing us to identify keywords associated with events, often
relating to cause or effect. While there are multiple methods of determining
relevancy, Almanac implements relevance as a time-based variant of Term
Frequency - Inverse Document Frequency (TF-IDF) [22]. The API finds
annotations that are most uniquely relevant to a visually prominent chart
feature $p$ compared to a set of other chart features $Q$.

The input parameters to the contextual annotation recommender are (1) a
visually prominent feature $p$, (2) a set of other features $Q$, and (3) the
time-series data $T$. The annotation recommender outputs a ranked set of
contextual headline annotations $H$ for feature $p$.

$H=\texttt{getAnnotations}(p,Q,T)$

to obtain a ranked set of headlines $H$ for feature $p$, where for each
headline $h_{i}\in H$, $i$ is its relevance rank. Note that $p$ and $Q$ can
consist of features identified by Almanac’s visual feature detector (Section
3.1) or may be specified manually by the chart author.

Almanac’s annotation recommender is designed to find NYT headlines that are
most uniquely relevant to feature $p$ compared to the other features in $Q$.
The finest granularity of publication date for our NYT headlines is the day of
publication. So, the annotation recommender starts by constructing a time
range $r$ in days for each feature in $\\{p\\}\cup Q$, using the sample
frequency granularity $T_{G}$ associated with the input time-series data $T$.
The set of such time ranges is $R$, and $r_{p}\in R$ is the range associated
with feature $p$.

Next, for each feature time range $r\in R$, the recommender filters the
annotation database to the NYT articles published within the range and
includes headlines or lede paragraphs containing descriptive keywords $T_{W}$
associated with the time-series data $T$. For each feature time range $r$, we
obtain a set of headlines $H_{r}$ that are generally relevant to the time-
series data. The recommender then applies TF-IDF [22] to score each headline
$h\in H_{r_{p}}$ corresponding to feature $p$ by summing the TF-IDF score for
each word.

To apply TF-IDF in this context, the recommender treats the set of headlines
$H_{r}$ associated with each feature time range $r$ as a document, while the
corpus consists of the set of documents $\mathcal{H}_{R}=\bigcup_{r\in
R}H_{r}$ corresponding to all of the feature time ranges. Specifically,
Almanac scores each headline $h\in H_{r_{p}}$ corresponding to $p$ by summing
the TF-IDF score for each of its words; i.e., for each word (or term) $t$ in
such a headline, the term frequency $\operatorname{tf}$ is computed as:

$\operatorname{tf}{(t,H_{r_{p}})}=\frac{f_{t,H_{r_{p}}}}{\sum_{t^{\prime}\in
H_{r_{p}}}f_{t^{\prime},H_{r_{p}}}}.$ (1)

This measures the relative frequency $t$ that appears in the headlines
corresponding to $p$. The denominator is the number of terms that appear in
these headlines. The inverse document frequency, $\operatorname{idf}$ for term
$t$ is computed as:

$\operatorname{idf}{(t,\mathcal{H}_{R})}=\log\left(\frac{|\mathcal{H}_{R}|}{|\\{H_{r}\in\mathcal{H}_{R}:t\in
H_{r}\\}|}\right)$ (2)

where the numerator is the total number of documents. Since there is one
document (or set of headlines) for each feature in $\\{p\\}\cup Q$, the
numerator is equivalent to the number of features in $\\{p\\}\cup Q$. The
denominator is the number of documents that contain the term $t$. The final
TF-IDF score for $t$ is then:

$\operatorname{tf-
idf}{(t,H_{r_{p}},\mathcal{H}_{R})}=\operatorname{tf}{(t,H_{r_{p}})}\cdot\operatorname{idf}{(t,\mathcal{H}_{R})}.$
(3)

Almanac treats the TF-IDF score for each headline $h\in H_{r_{p}}$ as its
rank, with the highest scoring headline ranked first, and returns the ranked
set of headlines as the recommended annotations $H$ for $p$.

###  3.3 Using the Almanac API

Almanac can be applied to any chart that is developed using web-based tools,
including D3.js [4] and Observable notebooks
(https://observablehq.com/@ibnzterrell/almanac-observable-demo). Figure 1
illustrates how a chart author might apply the Almanac API to construct an
annotated chart. We have also experimented with incorporating Almanac as a
dashboard extension to annotate Tableau charts [28]. In the context of
Tableau, users can either manually click on prominent features in the chart or
rely on Almanac to automatically identify features prior to displaying the
recommended annotations. Almanac is publicly available on Github
(https://github.com/ibnzterrell/Almanac) and a video showing these various
applications using the API can be found in the supplemental material.

##  4 Results and Future Work

We demonstrate the utility of the Almanac API with various D3.js charts
displaying the top-ranked annotations recommended by Almanac, as shown in
Figures Almanac: An API for Recommending Text Annotations For Time-Series
Charts Using News Headlines, 2, and 3. In all of these cases, the API was used
to identify peaks (and in some cases, also identify valleys) and then generate
annotation recommendations for those features. In all cases, we show the
default, top-ranked annotation headline returned by the annotation
recommender. However, if the recommender cannot find a suitable headline, the
feature initially goes unlabeled. Unlabeled features can then be manually
labeled. These examples show how the headlines annotations can provide useful
context for prominent features in charts depicting various types of data.
Additional annotated line chart examples and datasets can be found in the
supplementary material.

Figure Almanac: An API for Recommending Text Annotations For Time-Series
Charts Using News Headlines shows how the top-ranked annotations name
California wildfires [3]. Prominent features in the chart are annotated with
text containing relevant names and locations of these wildfires, providing
useful context in understanding the effects of the fires. For example, the
highest peak on July 2018 displays the annotation, “Carr Fire in California
Claims Another Victim, Bringing Death Toll to 6.” Other recommended
annotations for the various prominent features in the chart include, “Wildfire
Near Yosemite Destroys Berkeley’s Family Campground” in July 2013 and
“California “Wildfires Updates: 48 Dead in Camp Fire, Toll Expected to Rise,”
referring to the deadly fire that occurred in August 2018.

Similarly, Almanac provides headlines context related to the location and
issues surrounding the mass shootings in Figure 2a. The annotation, “Police
Yet to Determine Motive in Virginia Tech Shooting”  for example, indicates
that a mass shooting took place at Virginia Tech in April 2007. The peak
corresponding to the Orlando shooting in 2016 displays the annotation,
“Orlando Shooting Reignites Gun Control Debate in Congress,” naming the
location and some of the national concerns surrounding it.

Figure 2b shows a chart example of the U.S. presidential approval rating
between the years 2009 and 2019 [7]. Almanac provides causal explanations for
the reader to better understand the context of the fluctuations in
presidential approval ratings. For example, a peak in President Obama’s
approval rating on May 26, 2008, shows an annotation, “Obama Chooses Sotomayor
for Supreme Court Nominee,” indicating that perhaps the public viewed the
nomination of the first Hispanic woman to the Supreme Court positively.
Another peak annotated with, “For Obama, Big Rise in Poll Numbers After Bin
Laden Raid” suggests that the event was viewed favorably by the public, and
Obama’s approval ratings went up.

Figure 3a shows the amount of coverage of Afghanistan on cable TV news
channels between 2010 and 2022 [9]. The annotation for the peak in Apr 2021
“Biden saying it is ‘time to end America’s longest war,’ declares troops will
be out of Afghanistan by Sept. 11” explains that the U.S. is planning to leave
the country, which gives context for the peak in coverage of the country.
There are cases, however, where the API generates annotations that may not be
very relevant to the events occurring at these prominent features. For
example, the recommended annotation for Aug 22, 2017, “Trump’s Request for
India’s Help in Afghanistan Rattles Pakistan” is only tangentially relevant to
the events occurring in that country.

Automatic annotations can also be useful for understanding information
pertaining to an attribute of interest. Figure 3b shows data on viewership of
The Simpsons TV show [5]. The annotations provide information related to the
show. The annotation, “Actors on ‘The Simpsons’ Reach a Deal With Fox”
explains that in May 2004, efforts by actors on the show to win new contracts
finally came through just as the viewership was steadily declining. An
annotation for a spike in viewership for November 2007, “Ratings: Sunday Means
Cartoons (and ‘60 Minutes’)” indicates that the show was a popular animated
comedy line-up during the Sunday TV program.

While Almanac provides an API for annotating time-series charts with NYT
headlines for a variety of examples, future work should consider incorporating
other annotation sources such as Google search results, Wikipedia Events [30],
or domain-specific corpora. Recent work has explored the problem of rendering
charts at different sizes while preserving their prominent features [27].
Almanac’s prominence feature detector command could be used in automating
responsive chart design and for different target displays. Almanac’s
annotation recommender not only identifies relevant headlines but also scores
those headlines and the terms using our time-based variant of TF-IDF. This
metadata could be useful to find additional articles related to the event
topic and thereby support the creation of data narratives beyond annotation
recommendations.

##  5 Conclusion

To summarize, we introduce Almanac, a JavaScript API that identifies visually
prominent features in a time-series chart and recommends annotations sourced
from the NYT corpus of news headlines to provide additional contextual
information to the prominent features. We demonstrate the utility of Almanac
by annotating various D3.js and Vega-Lite time-series charts from a variety of
data sources. By serving as an API-level tool that can be integrated with
other JavaScript libraries, Almanac can facilitate the creation of well-
annotated charts across the Web.

## References

  * [1] E. Adar, C. Gearig, A. Balasubramanian, and J. Hullman.  Persalog: Personalization of news article content.  In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, p. 3188–3200. Association for Computing Machinery, New York, NY, USA, 2017.
  * [2] R. Alhalaseh, M. Munezero, M. Leinonen, L. Leppänen, J. Avikainen, and H. Toivonen.  Towards data-driven generation of visualizations for automatically generated news articles.  In Proceedings of the 22nd International Academic Mindtrek Conference, Mindtrek ’18, p. 100–109. Association for Computing Machinery, New York, NY, USA, 2018.
  * [3] Ares.  California wildfires (2013-2020).  https://www.kaggle.com/datasets/ananthu017/california-wildfire-incidents-20132020, 2020\.
  * [4] M. Bostock, V. Ogievetsky, and J. Heer.  D3 Data-Driven Documents.  IEEE Transactions on Visualization and Computer Graphics, 17(12):2301–2309, Dec. 2011. doi: 10 . 1109/TVCG . 2011 . 185
  * [5] J. Brown.  Simpsons viewership (1989-2016).  https://www.kaggle.com/datasets/jonbown/simpsons-episodes-2016, 2022\.
  * [6] T. Gao, J. R. Hullman, E. Adar, B. Hecht, and N. Diakopoulos.  NewsViews: an automated pipeline for creating custom geovisualizations for news.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3005–3014. ACM. doi: 10 . 1145/2556288 . 2557228
  * [7] G&R.  Presidential approval.  https://www.gallup.com/analytics/213617/gallup-analytics.aspx, 2018\.
  * [8] F. Hohman, M. Conlen, J. Heer, and D. H. P. Chau.  Communicating with interactive articles.  Distill, 2020.
  * [9] J. Hong, W. Crichton, H. Zhang, D. Y. Fu, J. Ritchie, J. Barenholtz, B. Hannel, X. Yao, M. Murray, G. Moriba, M. Agrawala, and K. Fatahalian.  Analysis of Faces in a Decade of US Cable TV News.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, p. 3011–3021. Association for Computing Machinery, New York, NY, USA, 2021.
  * [10] S. Huber.  Persistent homology in data science.  In 3rd Int. Data Sci. Conf. (iDSC ’20), Data Science – Analytics and Applications. Dornbirn, Austria (virtual), May 2020.
  * [11] J. Hullman and N. Diakopoulos.  Visualization rhetoric: Framing effects in narrative visualization.  IEEE Transactions on Visualization and Computer Graphics, 17(12):2231–2240, 2011.
  * [12] J. Hullman, N. Diakopoulos, and E. Adar.  Contextifier: Automatic generation of annotated stock visualizations.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’13, p. 2707–2716. Association for Computing Machinery, New York, NY, USA, 2013. doi: 10 . 1145/2470654 . 2481374
  * [13] A. V. Inc.  Alpha vantage stock api.  https://www.alphavantage.co/documentation/, 2022.
  * [14] E. Kandogan.  Just-in-time annotation of clusters, outliers, and trends in point-based data visualizations.  In 2012 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 73–82, 2012.
  * [15] D. H. Kim, V. Setlur, and M. Agrawala.  Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Charts.  Association for Computing Machinery, New York, NY, USA, 2021.
  * [16] J. Kim, J. Jo, and J. Seo.  Autotator: Semi-automatic approach for accelerating the chart image annotation process.  In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces, ISS ’19, p. 315–318. Association for Computing Machinery, New York, NY, USA, 2019. doi: 10 . 1145/3343055 . 3360741
  * [17] Y.-S. Kim, J. Hullman, and M. Agrawala.  Generating personalized spatial analogies for distances and areas.  In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, pp. 38–48. ACM, New York, NY, USA, 2016.
  * [18] A. Kirmse and J. de Ferranti.  Calculating the prominence and isolation of every mountain in the world.  Progress in Physical Geography: Earth and Environment, 41(6):788–802, 2017.
  * [19] N. Kong and M. Agrawala.  Perceptual interpretation of ink annotations on line charts.  In Proceedings of the 22nd annual ACM symposium on User interface software and technology, pp. 233–236. ACM, 2009.
  * [20] N. Kong and M. Agrawala.  Graphical overlays: Using layered elements to aid chart reading.  IEEE Transactions on Visualization and Computer Graphics, 18(12):2631–2638, 2012.
  * [21] A. Lundgard and A. Satyanarayan.  Accessible visualization via natural language descriptions: A four-level model of semantic content.  IEEE Transactions on Visualization and Computer Graphics, 28:1073–1083, 2022.
  * [22] C. Manning, P. Raghavan, and H. Schuetze.  Introduction to Information Retrieval.  p. 581, 2009.
  * [23] Observable.  Observable notebooks.  https://observablehq.com/, 2022.
  * [24] D. Ren, M. Brehmer, B. Lee, T. Höllerer, and E. K. Choe.  Chartaccent: Annotation for data-driven storytelling.  2017 IEEE Pacific Visualization Symposium (PacificVis), pp. 230–239, 2017.
  * [25] A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer.  Vega-Lite: A Grammar of Interactive Graphics.  p. 10.
  * [26] E. Segel and J. Heer.  Narrative visualization: Telling stories with data.  IEEE Transactions on Visualization and Computer Graphics, 16(6):1139–1148, 2010. doi: 10 . 1109/TVCG . 2010 . 179
  * [27] V. Setlur and H. Chung.  Semantic resizing of charts through generalization: A case study with line charts.  In 2021 IEEE Visualization Conference, IEEE VIS 2021 - Short Papers, New Orleans, LA, USA, October 24-29, 2021, pp. 1–5. IEEE, 2021\.
  * [28] Tableau.  Tableau dashboard extension.  https://tableau.github.io/extensions-api/docs/trex_overview.html.
  * [29] N. Y. Times.  New york times archive developer’s api.  https://developer.nytimes.com/docs/archive-product/1/overview, 2022\.
  * [30] Wikipedia.  Wikipedia events.  https://en.wikipedia.org/wiki/Category%3aEvents, 2022.


# A Systematic Literature Review on Client Selection in Federated Learning

Carl Smestad  Norwegian University of Science and TechnologyTrondheimNorway
<EMAIL_ADDRESS>and  Jingyue Li  Norwegian University of Science and
TechnologyTrondheimNorway<EMAIL_ADDRESS>
###### Abstract.

With the arising concerns of privacy within machine learning, federated
learning (FL) was invented in 2017, in which the clients, such as mobile
devices, train a model and send the update to the centralized server. Choosing
clients randomly for FL can harm learning performance due to different
reasons. Many studies have proposed approaches to address the challenges of
client selection of FL. However, no systematic literature review (SLR) on this
topic existed. This SLR investigates the state of the art of client selection
in FL and answers the challenges, solutions, and metrics to evaluate the
solutions. We systematically reviewed 47 primary studies. The main challenges
found in client selection are heterogeneity, resource allocation,
communication costs, and fairness. The client selection schemes aim to improve
the original random selection algorithm by focusing on one or several of the
aforementioned challenges. The most common metric used is testing accuracy
versus communication rounds, as testing accuracy measures the successfulness
of the learning and preferably in as few communication rounds as possible, as
they are very expensive. Although several possible improvements can be made
with the current state of client selection, the most beneficial ones are
evaluating the impact of unsuccessful clients and gaining a more theoretical
understanding of the impact of fairness in FL.

systematic literature review, software metric, federated learning, client
selection, neural network

††copyright: none††ccs: Computing methodologies Distributed artificial
intelligence††ccs: Computing methodologies Neural networks††ccs: Computing
methodologies Distributed algorithms

##  1\. Introduction

Machine learning (ML) has increased in popularity in recent years amongst
businesses and research. Also, cellphones and tablets are the primary
computing devices for many people (Anderson, 2015). These devices are equipped
with powerful sensors such as cameras, microphones, and GPS, resulting in a
vast amount of private data. With this increased concern regarding personal-
and data privacy, a new paradigm for machine learning arose named
decentralized learning, with the most prominent technique being federated
learning (FL).

FL was introduced in 2017 by McMahan et al. (2017), which is a decentralized
ML paradigm that leaves the training data distributed on mobile devices and
learns a shared model by aggregating locally-computed updates (Ma et al.,
2022). Instead of sending private data to a centralized server (CS), the
clients compute or train a model on their device and send the update to the
centralized server. The server randomly selects a fixed-size subset of clients
and provides them with an initial global model before they train and send the
updates. As the client devices have different data, randomly selecting them
might lead to several challenges. As the FL models are meant to be trained on
smartphones and IoT devices, the expensive cost of communication must be
considered by reducing the number of communication rounds and reducing the
size of the transmitted messages (Li et al., 2020a). Another culprit of FL is
that it is performed synchronously, which implies that one round of training
is finished when every edge device in the network has sent its model. This
results in an effect known as the straggler effect, where the network is as
fast as the slowest edge (Qu et al., 2022). Assuming there is an algorithm
that selects the fastest clients and there are no slow clients inducing the
straggler effect, and the communication cost and communication rounds are at a
minuscule level, it would be natural to assume that this algorithm would be
ideal and that not much more could be done in order to improve it. However,
client selection is much more complicated than that.

To ensure good learning, one must also consider the data heterogeneity,
resource allocation, and fairness between the clients. Likely, the selected
clients do not have the same data distribution, which might lead to heavy
biases within the learning. Thus, choosing the ”best” clients is an integral
part of a well-functioning federated learning network, but it is certainly not
trivial. There are many issues to consider for defining the best client
selection algorithm.

As pointed out by McMahan et al. (2017), only a fraction of clients are
selected for efficiency, as their experiments show diminishing returns for
adding more clients beyond a certain point. If the algorithm chooses to
include every client as opposed to only a subset, there might be a lot of
included clients who do not add value to the training while increasing the
cost of communication. Perhaps some of the clients will not even finish
training, which will make the entire round of learning fail. Therefore, only a
subset of clients must be included during client selection.

Hence, we are motivated to examine how studies have tried to improve client
selection, as many problems can be addressed by better strategies than random
selection. We focus on answering the following research questions (RQs):

  * •

RQ1: What are the main challenges in client selection?

  * •

RQ2: How are clients selected in federated learning?

  * •

RQ3: Which metrics are important for measuring client selection?

  * •

RQ4: What can be improved with the current client selection?

To answer the research questions, a systematic literature review (SLR) was
conducted by following the guidelines by Kitchenham and Charters (2007) and
Wohlin (2014). One iteration of backward and forward-snowballing was performed
on a set of six papers, resulting in 47 primary studies to review after the
quality assessment and study selection. The contributions of this SLR are as
follows.

  * •

It summarizes the main challenges in terms of client selection for FL. The
main challenges are heterogeneity, resource allocation, communication costs,
and fairness.

  * •

It summarizes the important metrics for measuring client selection in regard
to the main challenges. The most commonly used metrics are testing accuracy
and communication rounds.

  * •

It discusses possible future work within the field of client selection for FL.

The rest of the paper is organized as follows. The related work is presented
in section 2. The research methodology and implementation are presented in
section 3. Section 4 shows the results of this SLR, and section 5 discusses
the results. Lastly, section 6 concludes the study and proposes future work.

##  2\. Related Work

There are several literature reviews and surveys related to FL. Hou et al.
(2021) performed an SLR of blockchain-based FL and specialized in the
architectures and applications. They identified four security issues in FL
which motivate the use of blockchain. The study mentioned the Internet of
Things (IoT), medicine, and the Internet of Vehicles (IoV) as promising fields
for application but did not mention client selection.

Pfitzner et al. (2021) conducted an SLR of FL in a medical context. They
focused on the areas that were promising for digital health applications.
Antunes et al. (2022) did an SLR of FL for healthcare and focused on the
architecture and remaining issues regarding applying FL to electronic health
records (EHR). Both Pfitzner et al. (2021) and Antunes et al. (2022) focused
on the security perspective and did not summarize client selection issues. Lo
et al. (2021) performed an SLR of FL from a software engineering perspective.
They focused on what FL is, the different applications, general challenges,
and how they are addressed. The five most common challenges were communication
efficiency, statistical heterogeneity, system heterogeneity, data security,
and client device security. The study noticed that client selection is mostly
server-based but did not discuss it further. Liu et al. (2020) conducted an
SLR of FL but from a model quality perspective. The study presents several
algorithms types, such as neural networks, decision trees, etc., with
corresponding client-side algorithms but does not consider client selection.
Shaheen et al. (2022) investigated the applications, challenges and research
trends of FL. The study underwent 105 research studies and discovered that the
most promising application is within the healthcare domain. They reported data
imbalance, system heterogeneity, expensive communication, privacy concerns,
statistical heterogeneity, and resource allocation as the main challenges of
implementing FL. However, they did not relate any of these challenges to
client selection. Witt et al. (2022) wrote an SLR of FL from the
incentivization methods perspective. This study also discusses blockchain as a
possible improvement but does not mention client selection outside the scope
of blockchain. Hosseinzadeh et al. (2022a) did an SLR of FL with emphasis on
IoT, focusing on the evaluation factors and the future and open challenges of
FL-based IoT. The study mentions a possible client selection method but does
not focus on the topic. Lo et al. (2022) reviewed the different architectural
patterns to design FL systems. The study reports 15 architectural patterns,
where one of which is the client selector. The study provides a high-level
overview of possible solutions, such as resource-based, data-based, and
performance-based client selection, as well as some of the benefits and
drawbacks of the pattern. Abreha et al. (2022) systematically surveyed FL in
edge computing. The survey reports the main challenges as communication cost,
reliability, privacy, and administrative policies. It also discusses client
selection to a small degree by mentioning existing studies on the topic. Ali
et al. (2021) conducted an SLR of incentive-driven FL and the associated
security challenges. Some incentive mechanisms include auction theory and
blockchain but do not touch on the topic of client selection and possibly how
to incentivize clients. Ma et al. (2022) reviewed the state-of-the-art in
solving non-
independant and identically distributed (Non-IID) data in FL and addressed
future trends for the topic. When the datasets are not independent and
identically distributed, it leads to less correlation and dependencies because
samples of the datasets do not have the same probability distribution. Non-IID
data is one of the largest challenges in FL, and the study discusses ways to
improve it through, e.g., data enhancements and data selection. One of these
methods is client selection, but the survey does not go more into depth than
linking to relevant papers.

##  3\. Research Design and Implementation

To summarize the state of the art of client selection of FL and to answer our
research questions, we performed a systematic literature review based upon the
guidelines (Kitchenham and Charters, 2007) and (Wohlin, 2014).

###  3.1. Search Strategy

Generally, the SLR approach for generating a search strategy is to break down
the research questions into searchable terms and generate a list of synonyms,
abbreviations, and alternative spellings. As there exist a vast amount of
studies on the topic of FL, this process became unmanageable. Thus, the
strategy used in this paper is based on the guidelines for snowballing in SLR
by Wohlin (2014), as shown in Figure 1, which includes the following main
steps:

  * •

Step 1: Generate a start set of studies (including only papers that will be a
part of the final analysis)

  * •

Step 2: Perform backward- and forward snowballing

  * •

Step 3: Decide to include or exclude the study

  * •

Step 4: Iterate until finding no new papers

Figure 1. Illustration of snowballing in SLR (Wohlin, 2014)

To start the snowballing procedure, a starting set was needed. Google Scholar
was used to generate this starting set by using relevant terms such as
”Federated Learning” and ”Client Selection in Federated Learning.” The results
are listed below.

  * •

”Communication-Efficient Learning of Deep Networks from Decentralized Data”
(McMahan et al., 2017)

  * •

”Client Selection for Federated Learning with Heterogeneous Resources in
Mobile Edge” (Nishio and Yonetani, 2019)

  * •

”Client selection and bandwidth allocation in wireless federated learning
networks: A long-term perspective” (Xu and Wang, 2021)

  * •

”Federated Learning in a Medical Context: A Systematic Literature Review”
(Pfitzner et al., 2021)

  * •

”A Systematic Literature Review of Blockchain-based Federated Learning:
Architectures, Applications and Issues” (Hou et al., 2021)

  * •

”A state-of-the-art survey on solving non-IID data in Federated Learning” (Ma
et al., 2022)

When starting to perform forward and backward snowballing on the starting set,
it was apparent that there were too many papers to add as (McMahan et al.,
2017) is the first paper on federated learning and is cited by almost every
relevant paper in the field. The paper provided a definition name for devices
in FL, namely ”clients.” By investigating several studies, it was clear that,
despite FL being a young field within machine learning, a consensus existed on
using the term ”Client Selection” for choosing the appropriate devices. Thus,
a substring search on Google Scholar with the ”cited by” feature was conducted
with that substring to choose the most relevant studies.

###  3.2. Study Selection and Quality Assessment

We defined including and exclusion criteria to identify primary studies. For a
paper to be included, it has to fulfil all the following inclusion criteria:

  * •

Written in English

  * •

Published after 2017 because 2017 is the origin of federated learning appeared
in (McMahan et al., 2017)

  * •

Discusses client selection in Federated Learning

  * •

Peer-reviewed

According to (Higgins et al., 2019), quality can be seen as to which extent
the study minimizes bias and maximizes internal and external validity. Table 1
shows the different quality assessment criteria for empirical and non-
empirical sources (Mohanani et al., 2020). For each selected paper, we
assessed its quality according to the quality assessment criteria and awarded
one point for yes and zero points for no. We awarded half a point if it was
uncertain whether or not the study fulfilled the criterion. Then an average
was generated for each paper. A paper has to have an average of 0.5 or more to
be accepted as the primary study.

By applying the selection and quality assessment criteria, a total of 47
papers were chosen as primary studies for data extraction and synthesis.

Table 1. Quality assessment criteria based on (Mohanani et al., 2020) Quality Criteria | Empirical | Non-empirical |
---|---|---|---
Was the motivation for the study provided? | X | X |
Is the relevance to the industry discussed? | X | X |
Are the most important sources linked to or discussed? | X | X |
Is the aim (e.g., objectives, research goal) reported? | X | X |
Was the research method or design described? | X |  |
Were any threats to validity clearly stated? | X |  |

###  3.3. Data Synthesis

Data synthesis involves collating and summarizing the results of the primary
studies (Kitchenham and Charters, 2007). SLRs within IT and software
engineering are generally qualitative in nature. Based on the overview of data
synthesis provided by (van den Berg et al., 2013), we synthesize the data in a
spreadsheet, where the common themes, patterns, and finding between the
extracted information can be viewed.

For each RQ, relevant data were extracted and put into their respective
columns according to the research question. Lastly, a list was manually
generated based on the challenges and themes that were created for answering
the research questions. The data synthesis process was recorded and available
at
111https://docs.google.com/spreadsheets/d/1jIGpbkOcXazFRcR_Rds0mTshX0NDCIXU9Wgw3SECAiw/edit?usp=sharing.

##  4\. Research Results

This section presents the results of each research question.

###  4.1. RQ1: What are the main challenges in client selection?

Results show that 23 studies tried to improve upon heterogeneity, 13 studies
revolved around resource allocation, eight studies focused on communication
costs, and three studies had fairness as the main challenge. The distribution
of the challenges can be seen in Figure 2. Several studies report more than
one challenge, but it has been assigned to the challenge it focuses mostly on.

Figure 2. Distribution of challenges reported from the primary studies

###  4.2. Heterogeneity

In FL, the training is executed on the client’s local devices. This will
result in differences between the clients as they will have different datasets
and availability. This is the most common challenge found in FL, and McMahan
et al. (2017) reported heterogeneity as the main challenge. Almost half of the
primary studies tried to improve it through different measures.

Ma et al. (2022) conducted a state-of-the-art survey on solving non-IID data
in FL and concluded that data heterogeneity could be divided into the
following categories: feature distribution skew, label distribution skew, same
label (different features), same feature (different labels) and quantity skew.
Cho et al. (2022) report that heterogeneity also might arise due to partial
client participation, as only a small fraction of client nodes participate in
each round of training. If the client selection algorithm selects an improper
subset of clients with poor-quality data, this will result in an inefficient
trained model (Saha et al., 2022). Ma et al. (2021) reported label
distribution skew as one of the most significant parameters which lead to
performance degradation, while Rai et al. (2022) reported skewed data as one
of the most critical factors. Zhang et al. (2021b) reported that heterogeneity
/ Non-IID might bring the biases of some clients into the model training and
cause accuracy degradation. This claim is supported by Zhang et al. (2021a),
who claim an urgent need for client selection strategies that promise data
unbiasedness in FL. Li et al. (2022b) analyzed the limitations of the state-
of-the-art client selection in regard to heterogeneity and concluded that due
to under-exploited statistical- and system efficiency, not all the model
updates would contribute to the model training equally. As various clients
have diverse data sizes and importance, uploading unimportant updates
significantly degrades the system’s efficiency. According to Li et al.
(2022a), a significant problem with utilizing FL with IoT is that the local
data of sensors are constantly changing. This will have a similar effect as
device failures and might lead to skewed distributed data, which leads to
model degradation. There might also exist label noise on some clients, which
exists naturally. This will lead to unnecessary information being exchanged
(Yang et al., 2022).

To summarize, the key findings for the challenge of heterogeneity are as
follows.

  * •

48.93% of the studies reported heterogeneity as the main challenge for FL.

  * •

It might result in an inefficient trained model, performance- and accuracy-
degradation.

  * •

Heterogeneity might increase biases and unnecessary exchange of information.

###  4.3. Resource Allocation

Resource allocation was the second most common problem in the primary studies.
This is due to several reasons, but the main one is due to the fact that the
training process becomes inefficient when some clients have limited
computational resources (Nishio and Yonetani, 2019). Xu and Wang (2021) state
that a considerable challenge in resource allocation is that learning rounds
are temporally interdependent and have varying significance toward the final
learning outcome. According to Yu et al. (2022), it is unnecessary to select
more clients than needed, and it is beneficial to have fewer clients. Still,
the challenge consists of the trade-off between the number of clients, energy
consumption, and resource allocation. Furthermore, within hierarchical
federated learning (HFL), unique challenges exist, such as clients sometimes
being inaccessible to the edge servers. Due to differences in resources and
hardware specifications, the ”straggler effect” is bound to happen (Qu et al.,
2022). Zhang et al. (2021c) stated that clients are constrained by personal
energy and computation that may reduce the efficiency of ML training tasks.
This is because the training and transmission of large models are very energy-
consuming and might be difficult on low-energy edge devices. During training,
there might be changes in client resources due to volatility of client
population, client data, and training status (Shi et al., 2022). The topic of
energy consumption within FL is important, as training and transmission of
large models are energy-consuming, while edge devices generally have little
energy. Zeng et al. (2020) propose a client selection policy of giving the
lowest priority to clients with poor communication capacity and a bad channel.

To summarize, the key findings for the challenge of resource allocation are as
follows.

  * •

27.65% of the studies reported resource allocation as the main challenge of
FL.

  * •

The training process becomes inefficient when some clients have limited
computational resources.

  * •

Training and transmission of large models are very energy-consuming and
difficult for low-energy devices.

###  4.4. Communication costs

The third most common problem was the communication costs in FL. The
communication cost is essential as every time the global model updates, it
needs to receive the local aggregation of all the selected clients. According
to Tan et al. (2022), the communication power required to reach convergence
makes up a large portion of the cost. One of the challenges is that a client
with low computing power might not return the local model update on time,
leading to a long convergence time (Ko et al., 2021). Studies (Hosseinzadeh et
al., 2022b; Li et al., 2022c) state that the trade-off between communication
costs and accuracy is a challenge. Asad et al. (2022) state that another
challenge is the long distance between the different clients and the global
server, which results in increased bandwidth usage.

By default, FL is done synchronously. This implies that a round of
communication / global model updates is only executed once every client has
uploaded their model. This leads to an effect known as the straggler effect,
where the system is only as fast as the slowest link (Zhu et al., 2022). This
issue is also addressed by Qu et al. (2022).

Another fundamental challenge with communication costs is the energy usage of
clients in FL. As vast amounts of data are generated from mobile and edge
devices, these devices are energy-restricted. It is imperative to improve the
energy efficiency of the systems (Zeng et al., 2020).

According to Deng et al. (2022), clients’ hardware conditions and data
resources can vary significantly, which might lead to negative performance.

To summarize, the key findings for the challenge of communication costs are as
follows.

  * •

17.02% of the studies reported communication costs as the main challenge of
FL.

  * •

Clients with low power or slow will lead to long convergence time. As FL is
done synchronously, this implies that the learning is as fast as the slowest
client.

  * •

The possibly long distance between clients and servers will result in
increased bandwidth usage.

###  4.5. Fairness

The last common problem encountered was fairness. Only three studies reported
it as the main challenge which they tried to solve. However, fairness is a
researched topic within several similar fields, such as Resource Allocation
(RA) and ML. In the context of resource allocation, the problem is defined as
allocating a scarce shared resource among many users. For machine learning, it
is typically defined as the protection of some specific attribute(s) by, e.g.,
preprocessing the data to remove information about the protected attribute
(Feldman, 2015).

In the context of FL, if the client selection algorithm always selects the
fastest devices, it might boost the training process. However, as stated by
Huang et al. (2021) ” But clients with low priority are simply being deprived
of chances to participate at the same time, which we refer to it as an unfair
selection.” It might result in undesirable effects, such as omitting some
portions of data. Also, if there are less data involved, data diversity will
not be guaranteed and might hurt the performance of model training. Jee Cho et
al. (2020) state that by focusing on improving fairness, the uniformity of
performance across clients will be improved as well. Li et al. (2020b) define
fairness in FL as follows:

> Definition 1 (Fairness of performance distribution). For trained models w
> and $\tilde{w}$, $w$ provides a more fair solution to the federated learning
> objective (1) than model $\tilde{w}$, if the performance of model w on the m
> devices, $\\{a1,\dots,a_{m}\\}$, is more uniform than the performance of
> model $\tilde{w}$ on the m devices.

> Note: Decoupling is the main benefit of FL. The FL algorithms may involve
> hundreds to millions of remote devices learning locally by minimizing the
> objective function $f(w)$ (1) (Li et al., 2020b):
>
> (1) $\textrm{min}_{w}f(w)=\sum_{k=1}^{m}p_{k}F_{k}(w)$
>
> where m is the total number of devices, $p_{k}\geq 0$, $\sum_{k}p_{k}=1$,
> and the local objective $F_{k}$’s can be defined by empirical risks over
> local data.

Through this definition, it becomes apparent that learned models which might
be biased towards devices with large numbers of data points or commonly
occurring devices are unfair.

According to Ma et al. (2022), differences in data distribution and
uncertainty in data quality are challenging in FL and data selection might
exacerbate the unfairness of FL. There are several different methods for
prioritizing clients. If one selects all the ”fast” devices, it might result
in faster training but will deprive slower clients of the chance to
participate. If the selection is one-sided, it will bring negative side
effects, such as neutralizing some portions of data (Huang et al., 2021). In
addition, clients may not provide honest results through various attacks, such
as Byzantine attacks, which minimizes the effect of actual results of honest
clients and reduces fairness (Wan et al., 2022).

To summarize, the key findings for the challenge of fairness are as follows.

  * •

6.38% of the studies reported fairness as the main challenge of FL.

  * •

Selecting only the fastest clients might result in an unfair selection, as
slower clients are deprived of the chance to participate.

  * •

An unfair selection might lead to heavy biases as some portions of the data
are neutralized.

###  4.6. RQ2: How are clients selected in federated learning

The different solutions are presented in this subsection and divided into
their respective challenges. A summary of the findings is shown in Table 2.

####  4.6.1. Heterogeneity

The most common approach to address this issue is to try to select a subset of
clients who together give a more homogenous dataset (Cho et al., 2022). Ma et
al. (2022) performed a state-of-the-art survey on solving non-IID data in FL
and mentioned (Wang and Kantarci, 2020) as a possible solution through client
selection. They proposed selecting clients with small data heterogeneity based
on Thompson sampling. Abdulrahman et al. (2021) suggested a similar algorithm
of selecting a subset of clients who together form a homogeneous subset. Zhang
et al. (2021b) proposed to measure the degrees of non-IID data present in each
client and then select the clients with the lowest degrees. Li et al. (2022b)
and Saha et al. (2022) had similar ideas but suggested a more holistic
approach by also including the system heterogeneity (e.g., resources) as well.
Lin et al. (2022) propose to dynamically update the selection weights
according to the impact of the client’s data.

Clustered Federated Learning (CFL) was introduced as an efficient scheme to
balance out the non-IID data, and Albaseer et al. (2021) suggest leveraging
the devices’ heterogeneity to schedule them based on round latency and
bandwidth to select clients. According to Li et al. (2022a), this type of
approach works well within IoT due to the advantage of naturally clustered
factory devices. Lee et al. (2022) also find clusters of clients who together
have near IID data through being distribution-aware. In order to address the
issue of label distribution skew, Ma et al. (2021) suggested a method where
you check the similarity between the aggregated data distribution of the
selected clients and compare it to the global data distribution. Rai et al.
(2022) suggest giving each client an irrelevance score which improves the data
distribution skewness. Cao et al. (2022) have an interesting approach to
clustering clients by grouping them according to classes of data and then
randomly selecting one client within every group. Another promising approach
suggested by Balakrishnan et al. (2022) is to introduce diversity into client
selection by measuring how a subset of clients can represent the whole when
aggregated on the server for each communication round.

Generally, the studies try to keep an unbiased client selection in order to
promote fairness. However, Cho et al. (2022) report that biasing the client
selection towards choosing clients with higher local losses resulted in an
improvement in the partial client participation problem. Abdulrahman et al.
(2021) suggested another approach to the same problem but suggested a
multicriteria-based approach to predict if they were capable of performing the
FL task.

Other studies, such as (Zhang et al., 2021a), suggest strengthening client
selection with cryptographic methods such as homomorphic encryption (HE). Pang
et al. (2022) bring forward the idea of selecting clients at different global
iterations to guarantee the completion of the FL job. Lastly, Guo et al.
(2021) take into account both model weight divergence and local model training
loss for selecting clients.

####  4.6.2. Resource Allocation

In order to improve the effect of some clients having limited resources,
Nishio and Yonetani (2019) suggest an algorithm that manages clients based on
their resource conditions. Thus, allowing as many client updates as possible.
Xu and Wang (2021) create an algorithm that utilizes bandwidth allocation
under long-term client energy constraints by using available wireless channel
information in order to improve resource allocation. To deal with the resource
allocation problem, Yu et al. (2022) suggest maximizing the number of clients
while minimizing the energy consumption by the clients by allocating a set
amount of resources in terms of CPU and transmission power.

Within HFL, Qu et al. (2022) propose a client selection scheme with a network
operator that learns the number of successful participating clients while
dealing with a limited resource budget. Similarly, (Deng et al., 2022; Kuang
et al., 2021; Liu et al., 2022) suggested evaluating the learning quality of
clients on a limited resource budget and then selecting the best clients. Shi
et al. (2022) suggest that clients should be selected by considering and
quantifying factors such as the relative impact of clients’ data and resource
differences and then selecting the clients with the most significant score.

Another method to deal with resource allocation is to focus on minimizing
energy consumption and training delays in order to encourage more clients to
participate in model updating. This may be done through reinforcement learning
that learns to select the best subset of clients (Zhang et al., 2021c). Du et
al. (2022) propose an algorithm that utilizes fuzzy logic by considering the
number of local data, computing capability, and network resources of each
client.

####  4.6.3. Communication Costs

As communication cost is a vital challenge in FL, many attempts have been
executed in order to improve it. Ko et al. (2021) developed a joint client
selection algorithm that selects appropriate devices and allocates suitable
amounts of resources to reduce convergence time due to high communication
costs. Hosseinzadeh et al. (2022b) suggested a distributed client selection
algorithm where the client devices participate in aggregation, resulting in
lower communication costs while maintaining the low loss. Li et al. (2022c)
had a similar approach where they selected a subset of clients to participate
in each round of training, and the remaining clients did not have to do any
training, resulting in both lower computing and communication resources.

Another proposed solution is proposed by Asad et al. (2022), where there is a
3-way hierarchical framework to improve communication efficiency. It creates a
cluster head that is responsible for communication with the global server, and
local devices communicate with the cluster head. This will lead to model
downloading and uploading requiring less bandwidth due to the short distances
from the source to the destination. To tackle the energy consumption
challenge, Zeng et al. (2020) suggested only selecting the clients who provide
significant information with each round. This would enable them to select
fewer clients and end up with lower total energy consumption. In order to omit
the ”straggler effect” introduced through synchronous FL, Zhu et al. (2022)
suggest an asynchronous approach where the server did not have to wait for all
clients to be finished with their training. Tan et al. (2022) proposed to
utilize stochastic integer programming that selects clients in a reputation-
aware manner.

####  4.6.4. Fairness

Huang et al. (2021) promote a fairness-guaranteed client selection algorithm.
They conclude that the final accuracy may increase by focusing on fairness but
might sacrifice training efficiency. Whereas Jee Cho et al. (2020) suggest
improving fairness through biased client selection by selecting the ones with
higher local loss. Wan et al. (2022) propose to select the most honest and
useful clients by utilizing a multi-armed bandit approach, resulting in
dishonest clients being filtered out.

Table 2. Solutions compared to challenges Challenge | Solution(s)
---|---
Heterogeneity |  | \- Select subset of client to make up homogeneous dataset
---
\- Measure degrees of non-IID data and select lowest values
\- Balance out non-IID data through clustered FL
\- Give clients an irrelevance score and base selection of that
\- Select a subset of clients who represent the entire set
\- Utilize cryptography and weight divergence
Resource Allocation |  | \- Base selection on resource conditions
---
\- Maximize the amount of clients by minimizing energy consumption
\- Encourage clients to participate in model updating
\- Utilize fuzzy logic by considering several resource factors
Communication Costs |  | \- Joint client selection algorithm to reduce convergence time
---
\- Distributed client selection where the clients decide to participate
\- Only active clients should perform training
\- 3-way hierarchical framework to improve efficiency
\- Select the client with the most significant information each round
\- Asynchronous FL
Fairness |  | \- Fairness-guaranteed client selection algorithm
---
\- Improve fairness through biased client selection
\- Select honest clients

###  4.7. RQ3: Which metrics are important for measuring client selection?

The relevant metrics regarding client selection entirely depend on the problem
the study is trying to improve upon. The different key metrics for each of the
main challenges in client selection are summarized in Table 3.

####  4.7.1. Heterogeneity

The most common metric used is measuring the test accuracy against the number
of communication rounds. Out of the 20 studies which reported heterogeneity as
the biggest challenge, 14 used this metric to measure the success of their
client selection. This metric was also utilized by the original FL paper
(McMahan et al., 2017) and is directly comparable to the standard within
regular machine learning, where ”Test Accuracy vs. Epoch” is very commonly
seen. The main difference stems from FL having many clients send their model
updates to a global server and then aggregate them. In that regard, a
communication round corresponds to one epoch of the global server.

Studies (Cao et al., 2022; Li et al., 2022a) included a similar metric: the
number of communication rounds up to a given threshold accuracy. This
approach’s main benefit is that it focuses more on minimizing the number of
communication rounds, which are very costly in FL. Lastly, Abdulrahman et al.
(2021) looked into how many selected clients are able to finish training
without dropping out.

####  4.7.2. Resource Allocation

For the challenge of resource allocation, the most common metric seen is also
”Testing Accuracy vs. Communication Rounds.” This is as expected, as it
directly measures how well the FL-algorithm performs.

Some studies supplement it with other metrics such as energy, delay, and
client consumption (Zhang et al., 2021c). For mobile edge computing (MEC)
systems, the energy is the basis of the client training model, and delay
determines the iteration speed and convergence speed of the global model.

####  4.7.3. Communication costs

As already stated in section 4.4, the cost of communication between clients
and the global server is one of the most expensive parts of FL. Thus,
utilizing the right metrics to validate the reduced cost is vital.

The typical ”Testing Accuracy vs. Communication Rounds” is commonly seen in
the studies, as higher testing accuracy in fewer communication rounds will
lead to lower costs. Another beneficial metric is convergence time and
latency, reported by Ko et al. (2021), as reducing the time spent in
communication will lead to lower costs. Furthermore, Tan et al. (2022)
introduced the cost of hiring clients as an essential metric, as it was simply
overlooked in existing studies and contributed a large part of the overall
costs.

####  4.7.4. Fairness

Other than the already discussed ”testing accuracy vs. communication rounds”
metric, Huang et al. (2021) utilized different metrics for measuring improved
fairness. For instance, they included metrics such as the availability of the
client and mathematically measured the long-term fairness through constraints.

Table 3. Metrics compared to challenges Challenge | Metric(s)
---|---
Heterogeneity |  | \- Testing accuracy vs communication rounds
---
\- Communication rounds until threshold accuracy
\- Number of selected client able to finish training
Resource Allocation |  | \- Testing accuracy vs communication rounds
---
\- Energy, delay, and client consumption
Communication Costs |  | \- Testing accuracy vs communication rounds
---
\- Convergence time vs latency
\- Cost of hiring clients
Fairness |  | \- Testing accuracy vs communication rounds
---
\- Availability of clients
\- Long-term fairness constraints

###  4.8. RQ4: What can be improved with the current client selection?

There are a lot of improvements that can be made with the current client
selection. As decentralized learning is still pretty young, there is room for
improvement within all discussed challenges in this SLR.

####  4.8.1. Heterogeneity

For hierarchical federated learning (HFL), Albaseer et al. (2021) suggested
looking into finding the optimal thresholds for splitting clusters of clients,
which certainly would improve the communication efficiency of the learning
network.

####  4.8.2. Resource Allocation

The primary studies reported several measures for possible future work which
seem exciting and beneficial for the current client selection schemes. Xu and
Wang (2021) stated that selecting clients as late as possible improves the
efficiency of the client selection, but there is a lack of theoretical and
practical research on the topic.

The most commonly suggested improvement was reported by three of the studies
(Shi et al., 2022; Lin et al., 2022; Rai et al., 2022). They suggested looking
into the effect of unsuccessful clients (or free-riders) and how to quantify
the impact. These types of clients bring a lot of overhead costs into the
learning network, and exploring the effects and solutions to those would
undoubtedly improve the current client selection. None of the primary studies
focused mainly on the effect of unsuccessful clients. However, studies (Huang
et al., 2022; Shi et al., 2022) focused on optimizing client selection for
volatile FL. This volatility stems from the dynamic of clients’ data and the
unreliable nature of clients (e.g., unintentional shutdown and network
instability). Therefore, some work has been done on the topic, but there is
certainly a gap that may be improved more. Those studies focus much more on
the client’s ability to enter and leave training rather than the effect of
unsuccessful clients.

####  4.8.3. Communication Costs

Ko et al. (2021) discussed the possibility of creating an incentive mechanism
to encourage more computing power to FL. So far, there are no incentives for
the client devices to allocate more resources to learning than necessary.
Thus, giving them some sort of incentive mechanism would increase
computational power and improve the problem of resource allocation. Perhaps it
would make it easier to create a more homogenous resource distribution amongst
the clients.

####  4.8.4. Fairness

Even though only two studies reported fairness as the main challenge, studies,
such as (Jee Cho et al., 2020; Zhu et al., 2022; Cho et al., 2022), mention it
as a possibly important factor that could promise a higher accuracy. Others
mentioned it as a possible future direction for their work. For instance, Shi
et al. (2022) reported that fairness might play an essential role in FL
training and that studying it in a volatile context would be beneficial. As
already discussed in section 4.6, studies already focus on fairness in client
selection, but there is still a knowledge gap within the topic. Huang et al.
(2021) looked into the trade-off between fairness and training accuracy but
concluded that they could not quantify the relationship and that looking
further into analyzing the fairness factor for FL would be worthy of
investigation.

##  5\. Discussion

This section discusses how the SLR compares to the related work as well as the
limitations of the study.

###  5.1. Comparison to Related Work

To our knowledge, there currently does not exist any SLR focusing solely on
client selection. The previous work has focused either on general FL
challenges or the application of FL. Therefore, the main benefit of this SLR
is its focus on FL from the perspective of client selection. However, there
are a lot of similarities between the related work and this review, as they
all encompass the challenges within FL. The value of this review to the
industry is as a reference for the different client selection techniques and
how they impact overall learning. There is also value in viewing possible
future directions for client selection when looking into what can be improved.

This review has found a couple of areas that researchers may look more into
from the perspective of client selection. Firstly, there are a vast amount of
different client selection schemes proposed for FL, which all claim to
outperform the state-of-the-art of random selection. It would be beneficial to
compare these selection schemes with possible application cases in order to
form an improved state-of-the-art solution for client selection. Secondly, the
topic of fairness is not thoroughly explored. Several studies mention fairness
as an important factor, but there does not exist much research on the topic of
exploring the trade-offs and benefits of focusing on it.

Although FL is a relatively new field within machine learning, it already
shows promising prospects within several application domains, such as
healthcare, natural language processing, smart cities and IoT. For certain
industries, it might be more trivial to implement a well-functioning system as
the developers know the types of devices on which the algorithm will be
implemented, but this is not the case for applications such as IoT and edge
computing. In those fields, the developers do not necessarily know much about
the client devices which will perform the learning, thus making it much more
difficult to tackle the several challenges reported by the related work and
found in this SLR.

Client selection is an integral part of a well-functioning FL system, as it
may be utilized to improve the challenges of heterogeneity, resource
allocation, communication costs, and fairness. Despite the previous- and
related work conducted on the topic, there is no de facto standard for the
client selection algorithm within any application of FL. Even within a subset
of any challenge, such as the issue of clients dropping out during training,
multiple possible solutions exist, such as asynchronous FL, partial
aggregation of dropped-out clients, and resource-aware FL. Within each
category, there exist many algorithms to tackle the challenge through client
selection, which shows the importance of exploring the topic further and
possibly finding the best approach.

For academia and industry, this SLR may assist in several ways. Firstly, it
can be used as a reference guide for the most prominent existing challenges
and their consequences for learning. Secondly, for each given challenge, the
SLR presents several different possible existing solutions to tackle it. This
is especially valuable to the industry when deciding to implement an FL system
and deciding whether or not their ecosystem is well-suited for it. The SLR
also provides guidelines to mitigate some of the challenges.

###  5.2. Limitations

Although the guidelines for systematic reviews by (Kitchenham and Charters,
2007) were followed, several points may have been improved. We might miss some
primary studies in the study search stage because there were many studies on
the topic of FL. For instance, performing the forward snowballing procedure on
the original FL-paper McMahan et al. (2017) resulted in around 7000 studies.
Even though there is plenty of academic research on the topic, we did not look
into any grey literature as a possible source. There certainly may exist many
exciting discussions and ideas on FL which are not discussed in academic
journals but in blogs and newspapers. We might exclude papers that are
relevant to the study during the paper selection process. To mitigate this
risk, the papers’ inclusion and exclusion are cross-checked and agreed upon by
both authors.

##  6\. Conclusions and Future Work

We performed an SLR and summarized the challenges, solutions, and metrics for
evaluating the solutions and possible future work of client selection in FL.
Information from 47 primary studies is analyzed and synthesized. This study is
the only SLR, as far as the author is aware, focusing solely on client
selection in FL.

The SLR delights several possible future research challenges we want to focus
on. The most beneficial ones regard the impact of unsuccessful clients or
fairness. Improving one of those challenges could benefit FL, as the training
efficiency might increase, and the communication costs would be reduced. The
communication cost is also one of the most significant problems in FL. Thus,
improving it would be beneficial.

## References

  * (1)
  * Abdulrahman et al. (2021) Sawsan Abdulrahman, Hanine Tout, Azzam Mourad, and Chamseddine Talhi. 2021.  FedMCCS: Multicriteria Client Selection Model for Optimal IoT Federated Learning.  _IEEE Internet of Things Journal_ 8, 6 (March 2021), 4723–4735.  https://doi.org/10.1109/JIOT.2020.3028742 Conference Name: IEEE Internet of Things Journal.
  * Abreha et al. (2022) Haftay Gebreslasie Abreha, Mohammad Hayajneh, and Mohamed Adel Serhani. 2022.  Federated Learning in Edge Computing: A Systematic Survey.  _Sensors_ 22, 2 (Jan. 2022), 450\.  https://doi.org/10.3390/s22020450 Number: 2 Publisher: Multidisciplinary Digital Publishing Institute.
  * Albaseer et al. (2021) Abdullatif Albaseer, Mohamed Abdallah, Ala Al-Fuqaha, and Aiman Erbad. 2021.  Client Selection Approach in Support of Clustered Federated Learning over Wireless Edge Networks. In _2021 IEEE Global Communications Conference (GLOBECOM)_. 1–6.  https://doi.org/10.1109/GLOBECOM46510.2021.9685938
  * Ali et al. (2021) Asad Ali, Inaam Ilahi, Adnan Qayyum, Ihab Mohammed, Ala Al-Fuqaha, and Junaid Qadir. 2021\.  Incentive-Driven Federated Learning and Associated Security Challenges: A Systematic Review.  (2021), 30.
  * Anderson (2015) Monica Anderson. 2015\.  _Technology device ownership: 2015_.  Report. Pew Research Center.  https://apo.org.au/node/58353
  * Antunes et al. (2022) Rodolfo Stoffel Antunes, Cristiano André da Costa, Arne Küderle, Imrana Abdullahi Yari, and Björn Eskofier. 2022.  Federated Learning for Healthcare: Systematic Review and Architecture Proposal.  _ACM Transactions on Intelligent Systems and Technology_ 13, 4 (May 2022), 54:1–54:23.  https://doi.org/10.1145/3501813
  * Asad et al. (2022) Muhammad Asad, Ahmed Moustafa, Fethi A. Rabhi, and Muhammad Aslam. 2022.  THF: 3-Way Hierarchical Framework for Efficient Client Selection and Resource Management in Federated Learning.  _IEEE Internet of Things Journal_ 9, 13 (July 2022), 11085–11097.  https://doi.org/10.1109/JIOT.2021.3126828 Conference Name: IEEE Internet of Things Journal.
  * Balakrishnan et al. (2022) Ravikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat, Virginia Smith, and Jeffrey Bilmes. 2022.  DIVERSE CLIENT SELECTION FOR FEDERATED LEARNING VIA SUBMODULAR MAXIMIZATION.  (2022), 18.
  * Cao et al. (2022) Mei Cao, Yujie Zhang, Zezhong Ma, and Mengying Zhao. 2022\.  C2S: Class-aware client selection for effective aggregation in federated learning.  _High-Confidence Computing_ 2, 3 (Sept. 2022), 100068.  https://doi.org/10.1016/j.hcc.2022.100068
  * Cho et al. (2022) Yae Jee Cho, Jianyu Wang, and Gauri Joshi. 2022.  Towards Understanding Biased Client Selection in Federated Learning. In _Proceedings of The 25th International Conference on Artificial Intelligence and Statistics_. PMLR, 10351–10375.  https://proceedings.mlr.press/v151/jee-cho22a.html ISSN: 2640-3498.
  * Deng et al. (2022) Yongheng Deng, Feng Lyu, Ju Ren, Huaqing Wu, Yuezhi Zhou, Yaoxue Zhang, and Xuemin Shen. 2022.  AUCTION: Automated and Quality-Aware Client Selection Framework for Efficient Federated Learning.  _IEEE Transactions on Parallel and Distributed Systems_ 33, 8 (Aug. 2022), 1996–2009.  https://doi.org/10.1109/TPDS.2021.3134647 Conference Name: IEEE Transactions on Parallel and Distributed Systems.
  * Du et al. (2022) Zhaoyang Du, Celimuge Wu, Tsutomu Yoshinage, Lei Zhong, and Yusheng Ji. 2022.  On-device federated learning with fuzzy logic based client selection. In _Proceedings of the Conference on Research in Adaptive and Convergent Systems_ _(RACS ’22)_. Association for Computing Machinery, New York, NY, USA, 64–70.  https://doi.org/10.1145/3538641.3561490
  * Feldman (2015) Michael Feldman. 2015\.  _Computational Fairness: Preventing Machine-Learned Discrimination_.  Thesis.  https://scholarship.tricolib.brynmawr.edu/handle/10066/17628 Accepted: 2016-01-19T17:37:36Z.
  * Guo et al. (2021) Yingya Guo, Kai Huang, and Jianshan Chen. 2021.  WCL: Client Selection in Federated Learning with a Combination of Model Weight Divergence and Client Training Loss for Internet Traffic Classification.  _Wireless Communications and Mobile Computing_ 2021 (Dec. 2021), e3381998.  https://doi.org/10.1155/2021/3381998 Publisher: Hindawi.
  * Higgins et al. (2019) Julian P. T. Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, and Vivian A. Welch. 2019\.  _Cochrane Handbook for Systematic Reviews of Interventions_.  John Wiley & Sons.  Google-Books-ID: cTqyDwAAQBAJ.
  * Hosseinzadeh et al. (2022a) Mehdi Hosseinzadeh, Atefeh Hemmati, and Amir Masoud Rahmani. 2022a.  Federated learning-based IoT: A systematic literature review.  _International Journal of Communication Systems_ 35, 11 (2022), e5185.  https://doi.org/10.1002/dac.5185 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/dac.5185.
  * Hosseinzadeh et al. (2022b) Minoo Hosseinzadeh, Nathaniel Hudson, Sam Heshmati, and Hana Khamfroush. 2022b.  Communication-Loss Trade-Off in Federated Learning: A Distributed Client Selection Algorithm. In _2022 IEEE 19th Annual Consumer Communications & Networking Conference (CCNC)_. 1–6.  https://doi.org/10.1109/CCNC49033.2022.9700601 ISSN: 2331-9860.
  * Hou et al. (2021) Dongkun Hou, Jie Zhang, Ka Lok Man, Jieming Ma, and Zitian Peng. 2021.  A Systematic Literature Review of Blockchain-based Federated Learning: Architectures, Applications and Issues. In _2021 2nd Information Communication Technologies Conference (ICTC)_. 302–307.  https://doi.org/10.1109/ICTC51749.2021.9441499
  * Huang et al. (2022) Tiansheng Huang, Weiwei Lin, Li Shen, Keqin Li, and Albert Y. Zomaya. 2022.  Stochastic Client Selection for Federated Learning With Volatile Clients.  _IEEE Internet of Things Journal_ 9, 20 (Oct. 2022), 20055–20070.  https://doi.org/10.1109/JIOT.2022.3172113 Conference Name: IEEE Internet of Things Journal.
  * Huang et al. (2021) Tiansheng Huang, Weiwei Lin, Wentai Wu, Ligang He, Keqin Li, and Albert Y. Zomaya. 2021\.  An Efficiency-Boosting Client Selection Scheme for Federated Learning With Fairness Guarantee.  _IEEE Transactions on Parallel and Distributed Systems_ 32, 7 (July 2021), 1552–1564.  https://doi.org/10.1109/TPDS.2020.3040887 Conference Name: IEEE Transactions on Parallel and Distributed Systems.
  * Jee Cho et al. (2020) Yae Jee Cho, Samarth Gupta, Gauri Joshi, and Osman Yağan. 2020\.  Bandit-based Communication-Efficient Client Selection Strategies for Federated Learning. In _2020 54th Asilomar Conference on Signals, Systems, and Computers_. 1066–1069.  https://doi.org/10.1109/IEEECONF51394.2020.9443523 ISSN: 2576-2303.
  * Kitchenham and Charters (2007) Barbara Kitchenham and Stuart Charters. 2007.  Guidelines for performing Systematic Literature Reviews in Software Engineering.  2 (Jan. 2007).
  * Ko et al. (2021) Haneul Ko, Jaewook Lee, Sangwon Seo, Sangheon Pack, and Victor C. M. Leung. 2021.  Joint Client Selection and Bandwidth Allocation Algorithm for Federated Learning.  _IEEE Transactions on Mobile Computing_ (2021), 1–1.  https://doi.org/10.1109/TMC.2021.3136611 Conference Name: IEEE Transactions on Mobile Computing.
  * Kuang et al. (2021) Junqian Kuang, Miao Yang, Hongbin Zhu, and Hua Qian. 2021\.  Client Selection with Bandwidth Allocation in Federated Learning. In _2021 IEEE Global Communications Conference (GLOBECOM)_. 01–06.  https://doi.org/10.1109/GLOBECOM46510.2021.9685090
  * Lee et al. (2022) Jaewook Lee, Haneul Ko, Sangwon Seo, and Sangheon Pack. 2022\.  Data Distribution-Aware Online Client Selection Algorithm for Federated Learning in Heterogeneous Networks.  _IEEE Transactions on Vehicular Technology_ (2022), 1–11.  https://doi.org/10.1109/TVT.2022.3205307 Conference Name: IEEE Transactions on Vehicular Technology.
  * Li et al. (2022b) Chenning Li, Zeng Xiao, Mi Zhang, and Zhichao Cao. 2022b.  PyramidFL | Proceedings of the 28th Annual International Conference on Mobile Computing And Networking.  https://dl.acm.org/doi/10.1145/3495243.3517017 Archive Location: world.
  * Li et al. (2022c) Pengfei Li, Yunfeng Zhao, Liandong Chen, Kai Cheng, Chuyue Xie, Xiaofei Wang, and Qinghua Hu. 2022c.  Uncertainty Measured Active Client Selection for Federated Learning in Smart Grid. In _2022 IEEE International Conference on Smart Internet of Things (SmartIoT)_. 148–153.  https://doi.org/10.1109/SmartIoT55134.2022.00032 ISSN: 2770-2677.
  * Li et al. (2020a) Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020a.  Federated Learning: Challenges, Methods, and Future Directions.  _IEEE Signal Processing Magazine_ 37, 3 (May 2020), 50–60.  https://doi.org/10.1109/MSP.2020.2975749 Conference Name: IEEE Signal Processing Magazine.
  * Li et al. (2020b) Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. 2020b.  Fair Resource Allocation in Federated Learning.  https://doi.org/10.48550/arXiv.1905.10497 arXiv:1905.10497 [cs, stat].
  * Li et al. (2022a) Zonghang Li, Yihong He, Hongfang Yu, Jiawen Kang, Xiaoping Li, Zenglin Xu, and Dusit Niyato. 2022a.  Data Heterogeneity-Robust Federated Learning via Group Client Selection in Industrial IoT.  _IEEE Internet of Things Journal_ 9, 18 (Sept. 2022), 17844–17857.  https://doi.org/10.1109/JIOT.2022.3161943 Conference Name: IEEE Internet of Things Journal.
  * Lin et al. (2022) Weiwei Lin, Yinhai Xu, Bo Liu, Dongdong Li, Tiansheng Huang, and Fang Shi. 2022\.  Contribution-based Federated Learning client selection.  _International Journal of Intelligent Systems_ 37, 10 (2022), 7235–7260.  https://doi.org/10.1002/int.22879 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/int.22879.
  * Liu et al. (2022) Tingting Liu, Haibo Zhou, Jun Li, Feng Shu, and Zhu Han. 2022.  Uplink and Downlink Decoupled 5G/B5G Vehicular Networks: A Federated Learning Assisted Client Selection Method.  _IEEE Transactions on Vehicular Technology_ (2022), 1–13.  https://doi.org/10.1109/TVT.2022.3207916 Conference Name: IEEE Transactions on Vehicular Technology.
  * Liu et al. (2020) Yi Liu, Li Zhang, Ning Ge, and Guanghao Li. 2020\.  A Systematic Literature Review on Federated Learning: From A Model Quality Perspective.  https://doi.org/10.48550/arXiv.2012.01973 arXiv:2012.01973 [cs].
  * Lo et al. (2021) Sin Kit Lo, Qinghua Lu, Chen Wang, Hye-Young Paik, and Liming Zhu. 2021.  A Systematic Literature Review on Federated Machine Learning: From a Software Engineering Perspective.  _Comput. Surveys_ 54, 5 (May 2021), 95:1–95:39.  https://doi.org/10.1145/3450288
  * Lo et al. (2022) Sin Kit Lo, Qinghua Lu, Liming Zhu, Hye-Young Paik, Xiwei Xu, and Chen Wang. 2022\.  Architectural patterns for the design of federated learning systems.  _Journal of Systems and Software_ 191 (Sept. 2022), 111357\.  https://doi.org/10.1016/j.jss.2022.111357
  * Ma et al. (2021) Jiahua Ma, Xinghua Sun, Wenchao Xia, Xijun Wang, Xiang Chen, and Hongbo Zhu. 2021\.  Client Selection Based on Label Quantity Information for Federated Learning. In _2021 IEEE 32nd Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_. 1–6.  https://doi.org/10.1109/PIMRC50174.2021.9569487 ISSN: 2166-9589.
  * Ma et al. (2022) Xiaodong Ma, Jia Zhu, Zhihao Lin, Shanxuan Chen, and Yangjie Qin. 2022.  A state-of-the-art survey on solving non-IID data in Federated Learning.  _Future Generation Computer Systems_ 135 (Oct. 2022), 244–258.  https://doi.org/10.1016/j.future.2022.05.003
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017\.  Communication-Efficient Learning of Deep Networks from Decentralized Data. In _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_. PMLR, 1273–1282.  https://proceedings.mlr.press/v54/mcmahan17a.html ISSN: 2640-3498.
  * Mohanani et al. (2020) Rahul Mohanani, Iflaah Salman, Burak Turhan, Pilar Rodríguez, and Paul Ralph. 2020.  Cognitive Biases in Software Engineering: A Systematic Mapping Study.  _IEEE Transactions on Software Engineering_ 46, 12 (Dec. 2020), 1318–1339.  https://doi.org/10.1109/TSE.2018.2877759 Conference Name: IEEE Transactions on Software Engineering.
  * Nishio and Yonetani (2019) Takayuki Nishio and Ryo Yonetani. 2019.  Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge. In _ICC 2019 - 2019 IEEE International Conference on Communications (ICC)_. 1–7.  https://doi.org/10.1109/ICC.2019.8761315 ISSN: 1938-1883.
  * Pang et al. (2022) Jinlong Pang, Jieling Yu, Ruiting Zhou, and John C.S. Lui. 2022\.  An Incentive Auction for Heterogeneous Client Selection in Federated Learning.  _IEEE Transactions on Mobile Computing_ (2022), 1–17.  https://doi.org/10.1109/TMC.2022.3182876 Conference Name: IEEE Transactions on Mobile Computing.
  * Pfitzner et al. (2021) Bjarne Pfitzner, Nico Steckhan, and Bert Arnrich. 2021\.  Federated Learning in a Medical Context: A Systematic Literature Review.  _ACM Transactions on Internet Technology_ 21, 2 (June 2021), 50:1–50:31.  https://doi.org/10.1145/3412357
  * Qu et al. (2022) Zhe Qu, Rui Duan, Lixing Chen, Jie Xu, Zhuo Lu, and Yao Liu. 2022\.  Context-Aware Online Client Selection for Hierarchical Federated Learning.  _IEEE Transactions on Parallel and Distributed Systems_ 33, 12 (Dec. 2022), 4353–4367.  https://doi.org/10.1109/TPDS.2022.3186960 Conference Name: IEEE Transactions on Parallel and Distributed Systems.
  * Rai et al. (2022) Sumit Rai, Arti Kumari, and Dilip K. Prasad. 2022.  Client Selection in Federated Learning under Imperfections in Environment.  _AI_ 3, 1 (March 2022), 124–145.  https://doi.org/10.3390/ai3010008 Number: 1 Publisher: Multidisciplinary Digital Publishing Institute.
  * Saha et al. (2022) Rituparna Saha, Sudip Misra, Aishwariya Chakraborty, Chandranath Chatterjee, and Pallav Kumar Deb. 2022.  Data-Centric Client Selection for Federated Learning over Distributed Edge Networks.  _IEEE Transactions on Parallel and Distributed Systems_ (2022), 1–12.  https://doi.org/10.1109/TPDS.2022.3217271 Conference Name: IEEE Transactions on Parallel and Distributed Systems.
  * Shaheen et al. (2022) Momina Shaheen, Muhammad Shoaib Farooq, Tariq Umer, and Byung-Seo Kim. 2022.  Applications of Federated Learning; Taxonomy, Challenges, and Research Trends.  _Electronics_ 11, 4 (Jan. 2022), 670\.  https://doi.org/10.3390/electronics11040670 Number: 4 Publisher: Multidisciplinary Digital Publishing Institute.
  * Shi et al. (2022) Fang Shi, Chunchao Hu, Weiwei Lin, Lisheng Fan, Tiansheng Huang, and Wentai Wu. 2022\.  VFedCS: Optimizing Client Selection for Volatile Federated Learning.  _IEEE Internet of Things Journal_ (2022), 1–1.  https://doi.org/10.1109/JIOT.2022.3195073 Conference Name: IEEE Internet of Things Journal.
  * Tan et al. (2022) Xavier Tan, Wei Chong Ng, Wei Yang Bryan Lim, Zehui Xiong, Dusit Niyato, and Han Yu. 2022\.  Reputation-Aware Federated Learning Client Selection based on Stochastic Integer Programming.  _IEEE Transactions on Big Data_ (2022), 1–12.  https://doi.org/10.1109/TBDATA.2022.3191332 Conference Name: IEEE Transactions on Big Data.
  * van den Berg et al. (2013) Tobias van den Berg, Martijn W. Heymans, Stephanie S. Leone, David Vergouw, Jill A. Hayden, Arianne P. Verhagen, and Henrica CW de Vet. 2013.  Overview of data-synthesis in systematic reviews of studies on outcome prediction models.  _BMC Medical Research Methodology_ 13, 1 (Dec. 2013), 1–10.  https://doi.org/10.1186/1471-2288-13-42 Number: 1 Publisher: BioMed Central.
  * Wan et al. (2022) Wei Wan, Shengshan Hu, Jianrong Lu, Leo Yu Zhang, Hai Jin, and Yuanyuan He. 2022\.  Shielding Federated Learning: Robust Aggregation with Adaptive Client Selection.  https://doi.org/10.48550/arXiv.2204.13256 arXiv:2204.13256 [cs].
  * Wang and Kantarci (2020) Yuwei Wang and Burak Kantarci. 2020.  A Novel Reputation-aware Client Selection Scheme for Federated Learning within Mobile Environments. In _2020 IEEE 25th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD)_. 1–6.  https://doi.org/10.1109/CAMAD50429.2020.9209263 ISSN: 2378-4873.
  * Witt et al. (2022) Leon Witt, Mathis Heyer, Kentaroh Toyoda, Wojciech Samek, and Dan Li. 2022.  Decentral and Incentivized Federated Learning Frameworks: A Systematic Literature Review.  https://doi.org/10.48550/arXiv.2205.07855 arXiv:2205.07855 [cs].
  * Wohlin (2014) Claes Wohlin. 2014\.  Guidelines for snowballing in systematic literature studies and a replication in software engineering. In _Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering - EASE ’14_. ACM Press, London, England, United Kingdom, 1–10.  https://doi.org/10.1145/2601248.2601268
  * Xu and Wang (2021) Jie Xu and Heqiang Wang. 2021.  Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspective.  _IEEE Transactions on Wireless Communications_ 20, 2 (Feb. 2021), 1188–1200.  https://doi.org/10.1109/TWC.2020.3031503 Conference Name: IEEE Transactions on Wireless Communications.
  * Yang et al. (2022) Miao Yang, Hua Qian, Ximin Wang, Yong Zhou, and Hongbin Zhu. 2022.  Client Selection for Federated Learning With Label Noise.  _IEEE Transactions on Vehicular Technology_ 71, 2 (Feb. 2022), 2193–2197.  https://doi.org/10.1109/TVT.2021.3131852 Conference Name: IEEE Transactions on Vehicular Technology.
  * Yu et al. (2022) Liangkun Yu, Rana Albelaihi, Xiang Sun, Nirwan Ansari, and Michael Devetsikiotis. 2022\.  Jointly Optimizing Client Selection and Resource Management in Wireless Federated Learning for Internet of Things.  _IEEE Internet of Things Journal_ 9, 6 (March 2022), 4385–4395.  https://doi.org/10.1109/JIOT.2021.3103715 Conference Name: IEEE Internet of Things Journal.
  * Zeng et al. (2020) Qunsong Zeng, Yuqing Du, Kaibin Huang, and Kin K. Leung. 2020\.  Energy-Efficient Radio Resource Allocation for Federated Edge Learning. In _2020 IEEE International Conference on Communications Workshops (ICC Workshops)_. 1–6.  https://doi.org/10.1109/ICCWorkshops49005.2020.9145118 ISSN: 2474-9133.
  * Zhang et al. (2021c) Hangjia Zhang, Zhijun Xie, Roozbeh Zarei, Tao Wu, and Kewei Chen. 2021c.  Adaptive Client Selection in Resource Constrained Federated Learning Systems: A Deep Reinforcement Learning Approach.  _IEEE Access_ 9 (2021), 98423–98432.  https://doi.org/10.1109/ACCESS.2021.3095915 Conference Name: IEEE Access.
  * Zhang et al. (2021a) Shulai Zhang, Zirui Li, Quan Chen, Wenli Zheng, Jingwen Leng, and Minyi Guo. 2021a.  Dubhe: Towards Data Unbiasedness with Homomorphic Encryption in Federated Learning Client Selection. In _50th International Conference on Parallel Processing_. ACM, Lemont IL USA, 1–10.  https://doi.org/10.1145/3472456.3473513
  * Zhang et al. (2021b) Wenyu Zhang, Xiumin Wang, Pan Zhou, Weiwei Wu, and Xinglin Zhang. 2021b.  Client Selection for Federated Learning With Non-IID Data in Mobile Edge Computing.  _IEEE Access_ 9 (2021), 24462–24474.  https://doi.org/10.1109/ACCESS.2021.3056919 Conference Name: IEEE Access.
  * Zhu et al. (2022) Hongbin Zhu, Miao Yang, Junqian Kuang, Hua Qian, and Yong Zhou. 2022.  Client Selection for Asynchronous Federated Learning with Fairness Consideration. In _2022 IEEE International Conference on Communications Workshops (ICC Workshops)_. 800–805.  https://doi.org/10.1109/ICCWorkshops53468.2022.9814669 ISSN: 2694-2941.