11institutetext: The Open University (United Kingdom) 11email:
<EMAIL_ADDRESS>University of Bologna
(Italy) 22email<EMAIL_ADDRESS>Consiglio Nazionale
delle Ricerche (CNR) 33email<EMAIL_ADDRESS>
# Facade-X:
an opinionated approach to SPARQL anything
Submitted to https://2021-eu.semantics.cc (Accepted, May 2021)

Enrico Daga 11 0000-0002-3184-5407 Luigi Asprino 22 0000-0003-1907-0677 Paul
Mulholland 11 0000-0001-6598-0757 Aldo Gangemi 33

###### Abstract

The Semantic Web research community understood since its beginning how crucial
it is to equip practitioners with methods to transform non-RDF resources into
RDF. Proposals focus on either engineering content transformations or
accessing non-RDF resources with SPARQL. Existing solutions require users to
learn specific mapping languages (e.g. RML), to know how to query and
manipulate a variety of source formats (e.g. XPATH, JSON-Path), or to combine
multiple languages (e.g. SPARQL Generate). In this paper, we explore an
alternative solution and contribute a general-purpose meta-model for
converting non-RDF resources into RDF: Facade-X. Our approach can be
implemented by overriding the SERVICE operator and does not require to extend
the SPARQL syntax. We compare our approach with the state of art methods RML
and SPARQL Generate and show how our solution has lower learning demands and
cognitive complexity, and it is cheaper to implement and maintain, while
having comparable extensibility and efficiency.

###### Keywords:

SPARQL Meta-model Re-engineering

##  1 Introduction

Knowledge graphs have nowadays a key role in domains such as enterprise data
integration and cultural heritage. However, domain applications typically deal
with heterogeneous data objects. Therefore, ontology engineers develop
knowledge graph construction pipelines that include the transformation of
different types of content into RDF. Typically, this is achieved by using
tools that act as mediators between the data sources and the needed format and
data model [12]. Alternatively, dedicated software components implement ad-hoc
transformations from custom formats to a multiplicity of ontologies relevant
to the domain [4]. We place our research under the context of the EU H2020
SPICE project, which aims at developing a linked data infrastructure for
integrating and leveraging museum collections using multiple ontologies
covering sophisticated aspects of citizen engagement initiatives 111SPICE
Project: https://spice-h2020.eu. Museum collections come in a variety of data
objects, spanning from public websites to open data sets. These include
metadata summaries as CSVs, record details as JSON files, and binary objects
(e.g. artwork images), among others. The semantic lifting of such a variety of
resources can be a serious bottleneck for the project activities. Several
languages have been developed to either engineer content transformation (e.g.
RML) or extending the SPARQL query language to access non-RDF resources (e.g.
SPARQL Generate). However, existing solutions require Semantic Web
practitioners to learn a mapping language, or even combine multiple languages,
for example requiring to use XPath for XML transformations. In addition, these
require Semantic Web practitioners to know the details of the original format
(e.g. XML) as well as the target domain ontology.

In this paper, we don’t propose a new language. Instead, we aim at reducing
the effort of Semantic Web practitioners in dealing with heterogeneous data
sources by providing a generic, domain-independent meta-model as a facade to
wrap the original resource and to make it query-able as-if it was RDF.
Specifically, we contribute a meta-model and associated algorithm for
accessing non-RDF resources as RDF: Facade-X. Our approach can be implemented
by overriding the SERVICE operator and does not require to extend the SPARQL
syntax. We compare our approach with the state of art methods RML and SPARQL
Generate, and show how our solution has lower learning demands and cognitive
complexity, and it is cheaper to implement and maintain, while having
comparable extensibility and efficiency (in our naive implementation).

In the next section we analyse the key requirements, building also on the work
of [14]. In Section 3 we describe our approach for adopting facades for re-
engineering resources into RDF and give a formal definition of Facade-X.
Section 4 is dedicated to the prototype implementation of the approach in a
software named SPARQL Anything. Related work is discussed in Section 5. We
compare our approach with state of art methods (RML and SPARQL Generate) in
Section 6, before concluding our paper in Section 7.

##  2 Requirements

The motivation for researching novel ways to transform non-RDF resources into
RDF comes from the scenarios under development in the EU H2020 project SPICE:
Social Cohesion, Participation, and Inclusion for Cultural Engagement. In this
project, a consortium of eleven partners collaborate in developing novel ways
for engaging with cultural heritage, relying on a linked data network of
resources from museums, social media, and businesses active in the cultural
industry. However, the majority of resources involved are not exposed as
Linked Data but are released, for example, as CSV, XML, JSON files, or
combinations of these formats. In addition, the research activity aims at the
design of task-oriented ontologies, producing multiple semantic viewpoints on
the resources and their metadata. It is clear how the effort required for
transforming resources could constitute a significant cost to the project. In
the absence of a strategy to cope with this diversity, content transformation
may result in duplication of effort and become a serious bottleneck. Table 1
provides a summary of the requirements.

Table 1: Requirements Requirement |  Description
---|---
Transform |  Transform several sources having heterogeneous formats
Query |  Query resources having heterogeneous formats
Binary |  Support the transformation of binary formats
Embed |  Support the embedding of content in RDF
Metadata |  Support the extraction of metadata embedded in files
Low learning demands |  Minimise the tools and languages that need to be learned
Low complexity |  Minimise complexity of the queries
Meaningful abstraction |  Enable focus on data structures rather than implementation details
Explorability |  Enable data exploration without premature commitment to a mapping, in the absence of a domain ontology.
Workflow |  Integrate with a typical Semantic Web engineering workflow
Adaptable |  Be generic but flexible and adaptable
Sustainable |  Inform into a software that is easy to implement, maintain, and does not have evident efficiency drawbacks
Extendable |  Support the addition of an open set of formats

The main requirement is the ability to support users in transforming existing
non-RDF resources having heterogeneous formats (Transform). In addition, the
solution should be able to support cases in which practitioners only need to
interrogate the content (Query). A valid approach should be able to cope with
binary resources as well as textual formats (Binary). In the cultural heritage
domain, metadata files are typically associated to repositories of binary
content such as images in various formats. Applications may need to transfer
data and metadata in a single operation, embedding the binary content in a
data value (Embed) and extracting metadata (Metadata) from the file (from EXIF
annotations).

We consider requirements related to usability and adoption. The approach
should ideally limit the number of new languages and tools that need to be
learned in order to transform and use non-RDF resources (Low learning
demands). This can be expected to both encourage adoption and reduce the
learning curve for new users. The code that the user is required to develop in
order to access the resources should be as simple as possible (Low
complexity). The approach should provide the user with a meaningful level of
abstraction, enabling them to focus on the the structure of the data (e.g.
data rows and hierarchies) rather than the details of how the structure has
been implemented (Meaningful abstraction). The approach should support an
exploratory way of working in which the user does not have to prematurely
commit to a domain ontology before they come to understand the data
representation that they require (Explorability). The resulting technology
should be easily combined with typical Semantic Web engineering workflow
(Workflow). This requirement, already mentioned in [14], is interpreted
considering that the solution should rely as much as possible on already
existing technologies typically used by our domain users. The approach should
allow for a technical solution that is generic but easily Adaptable to user
tasks, for example, supporting symbol manipulation, variable assignments, and
data type manipulation.

Finally, we look into requirements of software engineering. The approach
should be Sustainable and inform a software that is easy to implement on top
of existing Semantic Web technologies, easy to maintain, and does not have
efficiency drawbacks compared to alternative state of the art solutions.
Ultimately, the system should be easy to extend (Extendable) to support an
open ended set of formats.

##  3 An opinionated approach

We introduce a novel approach to interrogate non-RDF resources with SPARQL.
Our opinion is that the task of transforming resources into RDF should be
decoupled in two very different operations: (a) re-engineering, and (b)
remodelling. We define re-engineering as the task of transforming resources
minimising domain considerations, focusing on the meta-model. Instead,
remodelling is the transformation of domain knowledge, where the original
domain model is re-framed into a new one, whose main objective is to add
semantics. From this perspective, we propose to solve the re-engineering
problem automatically and delegating the remodelling to the RDF-aware user.
How to use RDF to access heterogeneous source formats? We rely on the notion
of facade as "an object that serves as a front-facing interface masking more
complex underlying or structural code"222The Facade Design Pattern:
https://en.wikipedia.org/wiki/Facade_pattern (accessed 15/12/2020).. Applied
to our problem, a facade acts as a generic meta-model allowing (a) to inform
the development of transformers from an open ended set of formats, and (b) to
generate RDF content in a consistent and predictable way. In what follows, we
describe a generic approach that can be used to develop facade-based
connectors to heterogeneous file formats. After that, we introduce Facade-X,
which is the first of these interfaces, and describe how our facade maps to
RDF. Finally, we design a method to inject facades into SPARQL engines. To
support the reader, we introduce a guide scenario reusing the data of the Tate
Gallery collection, published on GitHub333Tate Gallery collection metadata:
https://github.com/tategallery/collection.. The repository contains CSV tables
with metadata of artworks and artists and a set of JSON files with details
about each catalogue record, for example, with the hierarchy of archive
subjects. The file artwork_data.csv includes metadata of the artworks in the
collection such as id, artist, artistId, title, year, medium, and references
two external resources: a JSON file with the artwork subjects headings and a
link to a JPG thumbnail image. Our objective is to serve this content to the
Semantic Web practitioners for exploration and reuse.

###  3.1 Resources, data sources, and facades

In this section we give a formal definition of the three components of our
approach: resources, data sources, and facades. In addition, we describe a
generic algorithm for applying these concepts for re-engineering resources in
RDF.

We consider a resource anything accessible from a URL and distinguish it from
its content, that we name data source. The file artwork_data.csv444Available
at
https://raw.githubusercontent.com/tategallery/collection/master/artwork_data.csv
and the image $N04858\\_8.jpg$ are resources and the CSV and JPG content are
data sources. We assume that a resource contains at least one data source555In
principle, a resource may include multiple data sources, for example, an Excel
spreadsheet may include several sheets.. A data source can be named with the
URL or have a different name666Although resources and data sources can be
named with the same string (URL), we consider them different entities in our
model.. We introduce the following predicates and associated axioms, in
predicate logic:

⬇

1$Resource(r)$ $DataSource(ds)$ $Name(n)$ $includes(r,ds)$ $hasName(ds,n)$

2$\forall~{}r.Resource(r)\to\exists~{}ds.includes(r,ds)\land~{}DataSource(ds)$

3$\forall~{}ds.DataSource(ds)\to\exists~{}n.hasName(ds,n)\land~{}Name(n)$

4$\forall~{}ds.DataSource(ds)\to\exists~{}r.Resource(r)\land~{}includes(r,ds)$

5$\forall~{}ds_{1}~{}\forall~{}ds_{2}~{}\forall~{}n~{}\forall~{}r.$

6
$includes(r,ds_{1})\land~{}includes(r,ds_{2})\land~{}hasName(ds_{1},n)\land~{}hasName(ds_{2},n)$

7 $\to~{}ds_{1}=ds_{2}$

8$\forall~{}ds~{}\forall~{}n_{1}~{}\forall~{}n_{2}.hasName(ds,n_{1})\land~{}hasName(ds,n_{2})\to~{}n_{1}=n_{2}$

In addition, we refer to two additional concepts: RDF Graph and RDF Dataset,
as specified by RDF 1.1 [3]. We now describe a generic algorithm for applying
facades to resources and obtain RDF datasets capable of answering a certain
query. Let $Q$ be the set of all possible queries, $G$ the set of all possible
graphs, $N$ the set of all possible graph names, $R$ the set of all possible
resources and $DS$ the set of all data sources. We define:  (i) $D$as a
collection of named graphs (i.e. $D\subseteq N\times G$);  (ii) $A$(i.e. the
algorithm) as a function that given a resource ($r\in R$), a facade ($f\in
F$), and a query ($q\in Q$), returns a collection of named graphs (i.e. one of
the possible subsets of $N\times G$);  (iii) $F$is a set of functions where
each $f\in F$ associates a data source ($ds\in DS$) and a query ($q\in Q$)
with a graph $g\in G$.  $A$ and $F$ can be formally defined as follows:

$\displaystyle A:R\times F\times Q\to 2^{N\times G}$ $\displaystyle
F=\\{f|f:DS\times Q\to G\\}$

Additionally, given a query $q\in Q$, a resource $r\in R$ and its data sources
$ds\in DS$, we define:  (i) $g_{ds,q}^{*}\in G$as the graph which contains the
minimal (optimal) set of triples required to answer $q$ on $ds$;  (ii)
$D_{r,q}^{*}=\\{(n,g_{ds,q}^{*})|includes(r,ds)\allowbreak\text{ and
}\allowbreak n\in N\text{ and }g_{ds,q}^{*}\in G$} as the collection of
minimal set of triples required to answer $q$ on $r$.  It is worth noticing
that given a query and a resource neither $A$ nor any $f\in F$ has to return
an optimal response (i.e. $D_{r,q}^{*}$ and $g_{ds,q}^{*}$), but they can
return any super set of the optimum (i.e. any $g\in G\text{ such that
}g_{ds,q}^{*}\subseteq g$). We don’t make any commitment on the underlying
implementation of the facade with respect to the resource/data sources, apart
from assuming that the resulting dataset will be sufficient, but not
necessarily optimal, for answering the query.

###  3.2 Facade-X

We base the design of Facade-X on the distinction between containers and
values. Specifically, we define a container as a set of uniquely identifiable
slots, each one of them including either another container or a data value.
Slot identifiers (keys) can be either XSD strings ($StringKey$) or XSD
positive integers ($NumberKey$). The predicate $Key$ is a reification of
either an integer or a string, while the predicate $Value$ reifies a string
only. Containers can optionally be qualified by a type. In Facade-X, data
sources are referred to as root containers. We specify our facade in predicate
logic as follows:

⬇

1$Root(c0)$ $Container(c1)$ $Slot(s1)$ $Key(n)$

2$StringKey(n)$ $NumberKey(n)$ $Value(v1)$ $Type(t)$

3$\forall k.StringKey(k)\rightarrow Key(k)$

4$\forall k.NumberKey(k)\rightarrow Key(k)$

5$\neg\exists k.NumberKey(k)\land StringKey(k)$

6$\forall~{}c.Root(c)\rightarrow Container(c)$

In addition, we define relations between the model components, including
definitions of domain and range:

⬇

1$hasSlot(c,s)$ $hasType(c,t)$ $hasKey(s,k)$

2$hasContainer(s,c)$ $hasValue(s,v)$

3$\forall(x,y).hasSlot(x,y)\rightarrow Container(x)\land Slot(y)$

4$\forall(x,y).hasType(x,y)\rightarrow Container(x)\land Type(y)$

5$\forall(x,y).hasKey(x,y)\rightarrow Slot(x)\land Key(y)$

6$\forall(x,y).hasContainer(x,y)\rightarrow Slot(x)\land Container(y)$

7$\forall(x,y).hasValue(x,y)\rightarrow Slot(x)\land Value(y)$

We define a set of axioms describing additional properties of the meta-model.
Only containers can have a type (but they don’t have to), and there can only
be one root container. A slot can have either one container or one value and
cannot have both. A slot can be member of one container only and slots of a
container are uniquely identified by their key:

⬇

1$\forall(x,y).Root(x)\land Root(y)\rightarrow x=y$

2$\neg\exists(x,y,z).hasContainer(x,y)\land hasValue(x,z)$

3$\forall(x,y,z).hasContainer(x,y)\land hasContainer(x,z)\rightarrow y=z$

4$\forall(x,y,z).hasValue(x,y)\land hasValue(x,z)\rightarrow y=z$

5$\forall(x,y,z).hasSlot(x,y)\land hasSlot(z,y)\rightarrow x=z$

6$\forall(c,s1,s2p,n).hasSlot(c,s1)\land hasSlot(c,s2)\land hasKey(s1,n)\land
hasKey(s2,n)\rightarrow s1=s2$

The data from our guide scenario can be represented as follows:

⬇

1$Root(ds)$

2$StringKey(id)$ $StringKey(artist)$ $StringKey(artistId)$ $StringKey(title)$

3$hasSlot(ds,s1)$ $hasKey(s1,IntegerKey(1))$ $hasContainer(s1,r1)$

4 $hasSlot(r1,r1s1)$ $hasKey(r1s1,id)$ $hasValue(r1s1,"1035")$

5 $hasSlot(r1,r1s2)$ $hasKey(r1s2,artist)$ $hasValue(r1s2,"Blake~{}Robert")$

6 $hasSlot(r1,r1s3)$ $hasKey(r1s3,artistId)$ $hasValue(r1s3,"38")$

7 $hasSlot(r1,r1s4)$ $hasKey(r1s4,title)$
$hasValue(r1s4,"A~{}Figure~{}Bowing~{}...")$ [...]

Finally, we define mapping rules to RDF, where properties are built using
string keys and resources can be either blank nodes or named IRIs777The system
may allow users to define their own namespace, or reuse the name of the $ds$,
and leave to the underlying machinery to mint IRIs.:

⬇

1$Root(ds)\stackrel{{\scriptstyle
f}}{{\to}}Triple(Resource(ds),rdf:type,fx:Root)$

2$hasSlot(c,s)\land hasKey(s,k)\land StringKey(k)\land hasContainer(s,c1)$

3 $\overset{f}{\to}Triple(Resource(c),Property(k),Resource(c1))$

4$hasSlot(c,s)\land hasKey(s,k)\land IntegerKey(k)\land hasContainer(s,c1)$

5
$\overset{f}{\to}Triple(Resource(c),ContainerMembershipProperty(k),Resource(c1))$

6$hasSlot(c,s)\land hasKey(s,k)\land StringKey(k)\land hasValue(s,v)$

7 $\overset{f}{\to}Triple(Resource(c),Property(k),Literal(v))$

8$hasSlot(c,s)\land hasKey(s,k)\land IntegerKey(k)\land hasValue(s,v)$

9
$\overset{f}{\to}Triple(Resource(c),ContainerMembershipProperty(k),Literal(v))$

10$hasType(c,t)\overset{f}{\to}Triple(Resource(c),rdf:type,Resource(t))$

Our model maps into an RDF that mixes lists, type statements, and key-value
pairs. Recent work suggests good practices for developing lists in RDF that
are efficient to query [5, 6], favouring container membership properties over
nested structures to represent lists. We define two namespaces, one for the
primitive entity Root and another for minting properties from keys888Not all
strings are valid IRI local names. Implementations will need to apply
heuristics to cope with corner cases in CSV or JSON keys.. The above mappings
produce the following Facade-X RDF, from our example scenario:

⬇

1@prefix fx: <http://sparql.xyz/facade-x/ns/>.

2@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>.

3@base <http://sparql.xyz/facade-x/data/>.

4[] a fx:Root ;

5 rdf:_1 [:id "1034"; :artist "Blake Robert"; :artistId "38"; ...

6 rdf:_2 [:id "16216"; :artist "Williams Terrick" :artistId "2149"; ...

7 rdf:_3 [:id "12086"; :artist "Pissarro Lucien" :artistId "1777"; ...

8 ...

###  3.3 Using facades in SPARQL

The algorithm in Section 3.1 requires as input a URL and returns an RDF
dataset as output. We propose to overload the SPARQL SERVICE operator by
defining a custom URI-schema, based on the protocol x-sparql-anything:, which
is intended to behave as a virtual remote endpoint. The related URI-schema
supports an open-ended set of parameters specified by the facade
implementations available. Options are embedded as key-value pairs, separated
by comma. Implementations are expected to either guess the source type from
the resource locator or to obtain an indication of the type from the URI
schema, for example, with an option "mime-type":

⬇

x-sparql-anything:mime-type=application/json; charset=UTF-8,location=$\dots$

Following our example scenario, users can write a query and select metadata
from the CSV file, as well as embed the content of remote JPG thumbnails in
the RDF. Multiple SERVICE clauses may integrate data from more files, for
example, the JSON with details about artwork subjects. We leave the content of
the CONSTRUCT section to be filled by the ontology engineer:

⬇

1PREFIX fx: <http://sparql.xyz/facade-x/ns/>

2PREFIX xyz: <http://sparql.xyz/facade-x/data/>

3PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

4CONSTRUCT {

5 [...] # Amazing ontology here

6} WHERE {

7 BIND (IRI(CONCAT(STR(tate:), "artwork-", ?id )) AS ?artwork) .

8 BIND (IRI(CONCAT(STR(tate:), "artist-", ?artistId )) AS ?artist) .

9 SERVICE <x-sparql-
anything:csv.headers=true,location=file:./artwork_data.csv> {

10 [] xyz:id ?id ; xyz:artist ?artistLabel ;

11 xyz:accessionId ?accId ; xyz:artistId ?artistId ;

12 xyz:title ?title; xyz:medium ?medium ;

13 xyz:year ?year ; xyz:thumbnailUrl ?thumbnail .

14 }

15 # JPEG Thumbnail from the Web

16 BIND (IRI(CONCAT("x-sparql-anything:location=", ?thumbnail )) AS ?embedJPG
).

17 SERVICE ?embedJPG { [] rdf:_1 ?imageInBase64 }.

18 # JSON File with subjects

19 BIND (IRI(CONCAT("x-sparql-anything:file:./artworks/", ?accId )) AS
?subJSON ).

20 SERVICE ?subJSON { [ xyz:id ?subjectId ; xyz:name ?subjectName ] }.

21}

##  4 Implementation to SPARQL Anything

In this section we describe SPARQL Anything which is meant to provide a proof-
of-concept of our approach. SPARQL Anything implements a stack of transformers
mapped to media types and file extensions. The framework allows the addition
of an open-ended set of transformers as Java classes. During execution, a
query manager intercepts usage of the SERVICE operator and in case the
endpoint URI has the x-sparql-anything protocol, it parses the URI extracting
the resource locator and parameters. Default parameters are: mime-type,
locator, namespace (to be used when defining RDF resources), and root (to use
as the IRI of the root RDF resource, instead of a blank node), and metadata.
SPARQL Anything will project an RDF dataset during query execution including
the data content and optionally a graph named
http://sparql.xyz/facade-x/data/metadata, including file metadata extracted
from image files (also in Facade-X). Specific formats may support specific
parameters. For example, the Text triplifier supports a regular expression to
be used by a tokenizer that splits the content in a list of strings (defaults
to the space character). Similarly, the CSV triplifier allows to specify
whether to use the first row as headers or only use column indexes. More
information on the currently supported formats can be found in the project
page999http://github.org/sparql-anything/sparql-anything..

We validated the generality of Facade-X as a meta-model with relation to the
triplifiers currently implemented in SPARQL Anything. We already considered
CSV in the guide example. The following JSON example, also derived from the
Tate Gallery open data, can be mapped to our model as in the associated
listing.

⬇

1{ "fc": "Kazimir Malevich",

2 "id": 1561,

3 "places": [

4 { "name": "Ukrayina", "type": "nation" },

5 { "name": "Moskva, Rossiya", "type": "inhabited_place" }

6 ],

7 "url": "http://www.tate.org.uk/art/artists/kazimir-malevich-1561" }

8

9$Root(malevic)$

10$StringKey(fc)StringKey(id)StringKey(places)StringKey(name)$

11$StringKey(type)StringKey(url)$

12$NumberKey(1)NumberKey(2)$

13$hasSlot(malevic,s_{fc})\land hasKey(s_{fc},fc)\land
hasValue(s_{fc},"KazimirMalevich")$

14$hasSlot(malevic,s_{id})\land hasKey(s_{id},id)\land hasValue(s_{id},1561)$

15$hasSlot(malevic,s_{places})\land hasKey(s_{places},places)\land
hasContainer(s_{places},c_{places})$

16$hasSlot(c_{places},s_{place/1})\land hasKey(s_{place/1},1)\land
hasContainer(s_{place/1},ukraina)$

17$hasSlot(ukraina,s_{ukr/name})\land hasKey(s_{ukr/name},name)\land$

18 $hasValue(s_{ukr/name},"Ukrayina")$

19$hasSlot(ukraina,s_{ukr/type})\land hasKey(s_{ukr/type},type)\land
hasValue(s_{ukr/type},"nation")$

20$hasSlot(c_{places},s_{place/2})\land hasKey(s{place/2},2)\land
hasContainer(s_{place/2},moskva)$

21$hasSlot(moskva,s_{mos/name})\land hasKey(s_{mos/name},name)\land$

22 $hasValue(s_{mos/name},"Moskva,Rossiya")$

23$hasSlot(moskva,s_{mos/type})\land hasKey(s_{mos/type},type)\land$

24 $hasValue(s_{mos/type},"inhabited\\_place")$

Finally, we show how an excerpt from a catalogue record in XML, can be
interpreted with our Facade-X (this also applies to HTML):

⬇

1 <OGT hint="OGGETTO">

2 <OGTD hint="Definizione">reperti antropologici ...</OGTD>

3 <OGTT hint="Tipologia">reperto osteo-dentario</OGTT>

4 ...

5

6$Root(ogt)\land hasType(record,"OGT")$

7$Container(ogtd)\land hasType(ogtd,"OGTD")$

8$Container(ogtt)\land hasType(ogtd,"OGTT")$

9$StringKey(hint)$

10$NumberKey(1)NumberKey(2)$

11$hasSlot(ogt,s_{1})\land hasKey(s_{1},1)\land hasValue(s_{1},ogtd)$

12$hasSlot(ogt,s_{2})\land hasKey(s_{2},2)\land hasValue(s_{2},ogtt)$

13$hasSlot(ogt,s_{hint})\land hasKey(s_{hint},hint)\land
hasValue(s_{hint},"OGGETTO")$

14...

##  5 Related work

Related work includes semantic web approaches to content re-engineering,
approaches to extending the functionalities of SPARQL, and research on end-
user development and human interaction with data.

In ontology engineering, non-ontological resource re-engineering refers to the
process of taking an existing resource and transforming it into an ontology
[19]. These family of approaches integrate resource transformation within the
methodology, where domain knowledge plays a central role. Triplify [2] is one
of the first tools aiming at converting sources into RDF in a domain
independent way. The approach is based on mapping HTTP URIs to ad-hoc database
queries, and rewriting the output of the SQL query into RDF. Other tools are
based on the W3C Direct Mapping recommendation [18] for relational databases.
Systems are available for automatically transforming data sources of several
formats into RDF (Any23101010http://any23.apache.org/,
JSON2RDF111111https://github.com/AtomGraph/JSON2RDF,
CSV2RDF121212http://clarkparsia.github.io/csv2rdf/ to name a few). A recent
survey lists systems to lift tabular data [9]. While these tools have a
similar goal (i.e. enabling the user to access the content of a data source as
if it was in RDF), the (meta)model used for generating the RDF data highly
depends on the input format. All these approaches are not interested in the
requirement of providing a common useful abstraction to heterogeneous formats.
A long history of mapping languages for transforming heterogeneous files into
RDF can be considered superseded by RML [8], including a number of approaches
for ETL-based transformations [1]. We consider RML as representative of
general data integration approaches such as OBDA [22]. This family of
solutions are based on a set of declarative mappings. The mapping languages
incorporate format-specific query languages (e.g. SQL or XPath) and require
the practitioner to have deep knowledge not only of the input data model but
also of standard methods used for its processing. Recent work acknowledges how
these languages are built with machine-processability in mind [13] and how
defining or even understanding the rules is not trivial to users.

We survey approaches to extend SPARQL. A standard method for extending SPARQL
is by providing custom functions131313ARQ provides a library of custom
functions for supporting aggregates such as computing a standard deviation of
a collection of values. ARQ functions:
https://jena.apache.org/documentation/query/extension.html (accessed
15/12/2020)., or by using so-called magic properties. This approach defines
custom predicates to be used for instructing specific behaviour at query
execution. SPARQL-Generate [14] introduces a novel approach for performing
data transformation from heterogeneous sources into RDF by extending the
SPARQL syntax with new operators [14]: GENERATE, SOURCE, and ITERATOR. Custom
functions perform ad-hoc operations on the supported formats, for example,
relying on XPath or JSONPath. Other approaches extend SPARQL without changes
to the standard syntax. BASIL [7] allows to define parametric queries by
enforcing a convention in SPARQL variable names. As a result, SPARQL query
templates can be processed with standard query parsers. SPARQL Micro-service
[16] provides a framework that, on the basis of API mapping specification,
wraps web APIs in SPARQL endpoints and uses JSON-LD profile to translate the
JSON responses of the API into RDF. In this paper, we follow a similar,
minimalist approach and extend SPARQL by overriding the behaviour of the
SERVICE operator. We compare our proposal with SPARQL Generate and RML in
detail in the evaluation section.

Motivationauthor=Paul]If short of space maybe this could be removed. I could
just reference a couple of key papers related to abstraction and explorability
in the requirements section for our work resides in research on end-user
development and human interaction with data. End-user development is defined
by [15] as "methods, techniques, and tools that allow users of software
systems, who are acting as non-professional software developers, at some point
to create, modify or extend a software artefact". Many SPARQL users fall into
the category of end-user developer. In a survey of SPARQL users, [20] found
that although 58% came from the computer science and IT domain, other SPARQL
users came from non-IT areas, including social sciences and the humanities.
Findings in this area [17] suggest that the data with which users work is more
often primarily list-based and/or hierarchical rather than tabular. For
example, [11] proposes an alternative formulation to spreadsheets in which
data is represented as list-of-lists, rather than tables. Our proposal goes in
this direction and accounts for recent findings in end-user development
research.

##  6 Evaluation

We conduct a comparative evaluation of SPARQL Anything with respect to the
state of art methods RML and SPARQL Generate. First, we analyse in a
quantitative way the cognitive complexity of the frameworks. Second, we
conduct a performance analysis of the reference implementations. Finally, we
discuss the approaches in relation to the requirements elicited in Section 2.
Competency questions, queries, experimental data, and code used for the
experiment are available on the GitHub repository of the SPARQL Anything
project141414https://github.com/spice-h2020/SPARQL
Anything/tree/main/experiment.

####  6.0.1 Cognitive Complexity Comparison.

We present a quantitative analysis on the cognitive complexity of SPARQL
Anything, SPARQL Generate and RML frameworks. One effective measure of
complexity is the number of distinct items or variables that need to be
combined within a query or expression [10]. Such a measure of complexity has
previously been used to explain difficulties in the comprehensibility of
Description Logic statements [21]. Specifically, we counted the number of
tokens needed for expressing a set of competency questions. We selected four
JSON files from the case studies of the SPICE project where each file contains
the metadata of artworks of a collection. Each file is organised as a JSON
array containing a list of JSON objects (one for each artwork). This simple
data structure avoids favouring one approach over the others. Then, an
analysis of the schema of the selected resources allowed us to define a set of
12 competency questions (CQs) that were then specified as SPARQL queries or
mapping rules according to the language of each framework, in particular:  (i)
8 CQs (named q1-q8), aimed at retrieving data from the sources, were specified
as SELECT queries (according to SPARQL Anything and SPARQL Generate);  (ii) 4
CQs (named q9-q11), meant for transforming the source data to RDF, were
expressed as CONSTRUCT queries (according to SPARQL Anything and SPARQL
Generate) or as mapping rules complying with RML. These queries/rules intend
to generate a blank node for each artwork and to attach the artwork’s metadata
as dataproperties of the node.  Finally, we tokenized the queries (by using
"(){},;\n\t\r␣ as token delimiters) and we computed the total number of tokens
and the number of distinct tokens needed for each queries. By observing the
average number of tokens per query we can conclude that RML is very verbose
(109.75 tokens) with respect to SPARQL Anything (26.25 tokens) and SPARQL
Generate (30.75 tokens) whose verbosity is similar (they differ of the
$\sim$6.5%). However, the average number of distinct tokens per query shows
that SPARQL Anything requires less cognitive load than other frameworks. In
fact, while SPARQL Anything required 18.25 distinct tokens, SPARQL Generate
needed 25.5 distinct tokens ($\sim$39.72% more) and RML 45.25 distinct tokens
($\sim$150% more).

####  6.0.2 Performance Comparison.

We assessed the performance of three frameworks in generating RDF data. All of
the tests described below were run three times and the average time among the
three executions is reported. The tests were executed on a MacBook Pro 2020
(CPU: i7 2.3 GHz, RAM: 32GB). Figure 1(a) shows the time needed for evaluating
the SELECT queries q1-q8 and for generating the RDF triples according to the
CONSTRUCT queries/mapping rules q9-q12. The three frameworks have comparable
performance. We also measured the performance in transforming input of
increasing size. To do so, we repeatedly concatenated the data sources in
order to obtain a JSON array containing 1M JSON objects and we cut this array
at length 10, 100, 1K, 10K and 100K. We ran the query/mapping q12 on these
files and we measured the execution time shown in Figure 1(b). We observe that
for inputs with size smaller than 100K the three frameworks have equivalent
performance. With larger inputs, SPARQL Anything is slightly slower than the
others. The reason is that, in our naive implementation, the data source is
completely transformed and loaded into a RDF dataset in-memory, before the
query is evaluated. However, implementations could stream the triples during
query execution, or transform the optimal triple set for the query solution,
thus achieving better performance on large input. However, we leave this
optimisations to future work.

(a) Execution time per query.

(b) Execution time with increasing input size.

Figure 1: Analysis of the the execution time. Table 2: Formats supported by RML, SPARQL Generate, and SPARQL Anything. | JSON | CSV | HTML | Bin. | XML | RDB | Text | Embed | Meta. | Spread.
---|---|---|---|---|---|---|---|---|---|---
RML |  |  |  |  |  |  |  |  |  |
SPARQL-Generate |  |  |  |  |  |  |  |  |  |
SPARQL Anything |  |  |  |  |  |  |  |  |  |

####  6.0.3 Requirements satisfaction and discussion

We discuss the requirements introduced in Section 2.

Transform, Binary, Embed, and Metadata. All the frameworks support users in
transforming heterogeneous formats with few differences (a comparison is
provided in Table 2). Currently, SPARQL Anything and SPARQL-Generate cover the
largest set of input formats. SPARQL-Generate however does not support
embedding content (Embed) and extracting metadata from files (Metadata). Both
features are not supported by RML, which doesn’t support plain text as well.
SPARQL Anything allows users to query spreadsheets, but it is not able to
handle relational databases yet151515However, relational tables can be mapped
using an approach similar to CSV and spreadsheets tables. A dedicated
component is currently being developed.. SPARQL Anything is the only tool
supporting the extraction of metadata and the embedding of binary content.

Query. In terms of query support, while RML requires data to be transformed
first and then uploaded to a SPARQL triple store, SPARQL Anything and SPARQL-
Generate enable users to query resources directly.

Low learning demands. SPARQL Generate uses an extension to SPARQL 1.1 to
transform source formats into RDF. RML provides an extension to the R2RML
vocabulary in order to map source formats into RDF. Therefore either a SPARQL
extension or a new mapping language has to be learned to perform the
translation. In the case of Facade-X, no new language has to be learned as
data can be queried using existing SPARQL 1.1 constructs.

Low complexity. Complexity can be measured as the number of distinct items or
variables that need to be combined with the query. In experiments, Facade-X is
found to perform favourably in comparison to SPARQL Generate and RML.

Meaningful abstraction. Differently from RML and SPARQL-Generate, which
require users to be knowledgeable of the source formats and their query
languages (e.g. XPath, JSONPath etc.), Facade-X users can access a resource as
if it was an RDF dataset, hence the complexity of the non-RDF languages is
completely hidden to them. The cost for this solution is limited to the users
which are required to explore the facade that is generated and tweak the
configuration via the Facade-X IRI schema.

Explorability. With SPARQL Generate and RML, the user needs to commit to a
particular mapping or transformation of the source data into RDF. However, the
data representation required to carry out a knowledge intensive task often
emerges from working with data and cannot be wholly specified in advance (this
is a crucial requirement of our project SPICE). By distinguishing the
processes of re-engineering and re-modelling, Facade-X enables the user to
avoid prematurely committing to a mapping and rather focus on querying the
data within SPARQL, in a domain-independent way.

Workflow. All the technologies considered can in principle be integrated with
a typical Semantic Web engineering workflow. However, while we cannot assume
that Semantic Web experts have knowledge of RML, XPath, and SPARQL Generate,
we can definitely expect knowledge of SPARQL.

Adaptable. All technologies provide a flexible set of methods for data
manipulation, sparql.aything relying on plain SPARQL. We make the assumption
that SPARQL itself is enough for manipulating variables, content types, and
RDF structures. It is an interesting, open research question to investigate
content manipulation patterns in the various languages and compare their
ability to meet user requirements.

Extendable and Sustainable. Our approach can be implemented within existing
SPARQL query processors with minimal development effort. Extending SPARQL
Anything requires to write a component that exposes a data source format as
Facade-X. Facade-X does not need to be encoded in the software but serves as a
reference for mapping an open ended set of formats. In contrast, extending
SPARQL Generate and RML requires extending the user toolkit to handle the
specificity of the formats, exposing to users new functions for querying,
filtering, traversing, and so on. In addition, our approach leads to a more
sustainable codebase. To give evidence of this statement, we use the tool
cloc161616cloc: https://github.com/AlDanial/cloc (accessed 15/12/2020). to
count the lines of Java code required to implement the core module of SPARQL
Generate in Apache Jena (without considering format-specific
extensions171717For SPARQL Generate, we only considered the code included in
the submodule sparql-generate-jena.) and the RML implementation in
Java181818RMLMapper: https://github.com/RMLio/rmlmapper-java.. SPARQL Generate
and RML require developing and maintaining 12280 and 7951 lines of Java code,
respectively. We developed the prototype implementation of SPARQL Anything
with 3842 lines of Java code, including all the currently supported
transformers.

##  7 Conclusions

In this paper, we presented an opinionated approach for making non-RDF
resources query-able with SPARQL. We contributed a general approach to apply
facades to content re-engineering and a specific instance of this approach,
Facade-X, which defines a general meta-model akin to a list-of-lists. We
compared our approach with the state of art methods RML and SPARQL Generate
and demonstrated how our solution has lower learning demands and cognitive
complexity, and it is cheaper to implement and maintain, while having
comparable extensibility. Next, we will extend the range of supported formats
of SPARQL Anything, including relational databases, Microsoft Office files,
and binary content other then images, and develop new strategies for
performance optimisation. Moreover, we will perform a user study for
investigating the cognitive implications of using Facade-X as a meta-model
with respect to arbitrary RDF, and compare the tools in terms of expressivity
and ability to meet user requirements. Finally, other facades can be designed
as well. It is an interesting research question to investigate content
manipulation patterns in alternative facades and evaluate their benefit for
content exploration and transformation.

## References

  * [1] Arenas-Guerrero, J., Scrocca, M., Iglesias-Molina, A., Toledo, J., Pozo-Gilo, L., Dona, D., Corcho, O., Chaves-Fraga, D.: Knowledge graph construction: An etl system-based overview. ESWC 2021 Workshop KGCW (Submission) (2021)
  * [2] Auer, S., Dietzold, S., Lehmann, J., Hellmann, S., Aumueller, D.: Triplify: light-weight linked data publication from relational databases. In: Proceedings of the 18th international conference on World wide web. pp. 621–630 (2009)
  * [3] Cyganiak, R., Wood, D., Lanthaler, M.: RDF 1.1 concepts and abstract syntax. W3C recommendation, W3C (Feb 2014), https://www.w3.org/TR/2014/REC-rdf11-concepts-20140225/
  * [4] Daga, E., d’Aquin, M., Adamou, A., Brown, S.: The open university linked data–data. open. ac. uk. Semantic Web 7(2), 183–191 (2016)
  * [5] Daga, E., Meroño-Peñuela, A., Motta, E.: Modelling and querying lists in RDF. A pragmatic study. In: Proceedings of the QuWeDa 2019: 3rd Workshop on Querying and Benchmarking the Web of Data co-located with 18th International Semantic Web Conference (ISWC 2019), Auckland, New Zealand, October 26-30, 2019. CEUR-WS.org (2019)
  * [6] Daga, E., Meroño-Peñuela, A., Motta, E.: Sequential linked data: the state of affairs. Semantic Web (2021)
  * [7] Daga, E., Panziera, L., Pedrinaci, C.: A basilar approach for building web apis on top of sparql endpoints. In: CEUR Workshop Proceedings. vol. 1359, pp. 22–32 (2015)
  * [8] Dimou, A., Vander Sande, M., Colpaert, P., Verborgh, R., Mannens, E., Van de Walle, R.: Rml: a generic language for integrated rdf mappings of heterogeneous data. In: 7th Workshop on Linked Data on the Web (2014)
  * [9] Fiorelli, M., Stellato, A.: Lifting tabular data to rdf: A survey. Metadata and Semantic Research 1355, 85 (2021)
  * [10] Halford, G.S., Andrews, G.: : The development of deductive reasoning: How important is complexity? Thinking & Reasoning 10(2), 123–145 (2004)
  * [11] Hall, A.G.: The Lish: A Data Model for Grid Free Spreadsheets. Ph.D. thesis, The Open University (2019)
  * [12] Haslhofer, B., Isaac, A.: data. europeana. eu: The europeana linked open data pilot. In: International Conference on Dublin Core and Metadata Applications. pp. 94–104 (2011)
  * [13] Heyvaert, P., De Meester, B., Dimou, A., Verborgh, R.: Declarative rules for linked data generation at your fingertips! In: European Semantic Web Conference. pp. 213–217. Springer (2018)
  * [14] Lefrançois, M., Zimmermann, A., Bakerally, N.: A sparql extension for generating rdf from heterogeneous formats. In: European Semantic Web Conference. pp. 35–50. Springer (2017)
  * [15] Lieberman, H., Paternò, F., Klann, M., Wulf, V.: End-user development: An emerging paradigm. In: End user development, pp. 1–8. Springer (2006)
  * [16] Michel, F., Faron-Zucker, C., Corby, O., Gandon, F.: Enabling automatic discovery and querying of web apis at web scale using linked data standards. In: Companion Proceedings of The 2019 World Wide Web Conference. pp. 883–892 (2019)
  * [17] Panko, R.R., Aurigemma, S.: Revising the panko–halverson taxonomy of spreadsheet errors. Decision Support Systems 49(2), 235–244 (2010)
  * [18] Prud’hommeaux, E., Arenas, M., Bertails, A., Sequeda, J.: A direct mapping of relational data to RDF. W3C recommendation, W3C (Sep 2012), https://www.w3.org/TR/2012/REC-rdb-direct-mapping-20120927/
  * [19] Villazón-Terrazas, B.C., Suárez-Figueroa, M., Gómez-Pérez, A.: A pattern-based method for re-engineering non-ontological resources into ontologies. International Journal on Semantic Web and Information Systems (IJSWIS) 6(4), 27–63 (2010)
  * [20] Warren, P., Mulholland, P.: Using sparql–the practitioners’ viewpoint. In: European Knowledge Acquisition Workshop. pp. 485–500. Springer (2018)
  * [21] Warren, P., Mulholland, P., Collins, T., Motta, E.: Making sense of description logics. In: Proceedings of the 11th International Conference on Semantic Systems. pp. 49–56 (2015)
  * [22] Xiao, G., Calvanese, D., Kontchakov, R., Lembo, D., Poggi, A., Rosati, R., Zakharyaschev, M.: Ontology-based data access: A survey. International Joint Conferences on Artificial Intelligence (2018)


# FedNST: Federated Noisy Student Training for Automatic Speech Recognition

###### Abstract

Federated Learning (FL) enables training state-of-the-art Automatic Speech
Recognition (ASR) models on user devices (clients) in distributed systems,
hence preventing transmission of raw user data to a central server. A key
challenge facing practical adoption of FL for ASR is obtaining ground-truth
labels on the clients. Existing approaches rely on clients to manually
transcribe their speech, which is impractical for obtaining large training
corpora. A promising alternative is using semi-/self-supervised learning
approaches to leverage unlabelled user data. To this end, we propose a new
Federated ASR method called FedNST for noisy student training of distributed
ASR models with private unlabelled user data. We explore various facets of
FedNST, such as training models with different proportions of unlabelled and
labelled data, and evaluate the proposed approach on 1173 simulated clients.
Evaluating FedNST on LibriSpeech, where 960 hours of speech data is split
equally into server (labelled) and client (unlabelled) data, showed a 22.5%
relative word error rate reduction (WERR) over a supervised baseline trained
only on server data.

Index Terms: Federated Learning, Speech Recognition, Semi-supervised Learning.

##  1 Introduction

Significant improvements in Automatic Speech Recognition (ASR) have been
achieved through the development of End-to-End (E2E) Attention-based models
[1, 2] and semi/self-supervised learning [3, 4], allowing for utilization of
ever-increasing training corpora. Acquiring speech data for applications such
as voice assistants requires transferring sensitive user data to the cloud,
leading to privacy compromises [5, 6, 7, 8]. Federated Learning (FL) [9, 10]
offers a solution by training models on user devices (clients) without sharing
private data with the server. Briefly, an FL algorithm involves: (1) selecting
a group of clients, (2) transmitting a global model to clients, (3) training
the global model on local user data, (4) transmitting  gradients/weights back
to the server, (5) aggregating gradients/weights, and (6) repeating steps 1-5
until convergence.

Various Federated ASR methods have been proposed to train ASR models in FL
systems [11, 12, 13, 14]. Specific challenges arising from data heterogeneity
(speech characteristics, amount of data, acoustic environments etc.) are
addressed via client-dependent data transformations [14] and imposing upper
limits on the number of client samples [13]. Improvements to distributed
optimization of models, such as Word Error Rate (WER) based aggregation [12]
and a hierarchical gradient weighting scheme [11] have been proposed. FL for
realistic ASR is presented in [12], showing feasibility with the French and
Italian CommonVoice subsets [15], comprising of thousands of challenging
speakers. The above approaches all assume availability of labelled data on
clients participating in FL. In real-world ASR applications, however, manual
annotation of user data on clients is infeasible.

Recent semi- and self-supervised central training methods achieve state-of-
the-art (SotA) accuracy, but require storing data at a central location [16,
2, 17]. A natural question then arises: can we adopt vanilla methods to
leverage unlabelled user data for federated ASR?

Semi-/self-supervised learning methods have been explored in FL systems for
image and audio classification tasks [18, 19, 20, 21, 22]. However, these
methods cannot be directly used to tackle the above problem, since the
proposed objectives are not applicable for sequence-to-sequence learning.

A semi-supervised FL method proposed for ASR [23] involves uploading
unlabelled speech data from clients to an external (cloud) storage. An ASR
model is then trained on this data using a federation of few model trainers
with high computational resources. The proposed set-up is useful for a cross-
silo setup, but it is not applicable to a cross-device setup, comprising a
federation of thousands of user devices with low computational resources.

To this end, we propose a new method called Federated Noisy Student Training
(FedNST), leveraging unlabelled speech data from clients to improve ASR models
by adapting Noisy Student Training (NST) [24] for FL. Our work explores a
challenging scenario: each client holds and trains a model exclusively on its
own unlabelled speech data, leading to a heterogeneous data distribution, and
the number of participating clients exceeds one thousand, resulting in a
cross-device scenario.

The contributions of this work are as follows:

  * •

To our best knowledge, this is the first work which aims to leverage private
unlabelled speech data distributed amongst thousands of clients to improve
accuracy of end-to-end ASR models in FL systems. For this purpose, we propose
a new method called FedNST, employing noisy student training for federated ASR
models, which achieves 22.5% WERR over training with only labelled data.

  * •

We elucidate the change in WER of ASR models from central training to cross-
device Federated Learning regimes with FedNST, achieving a marginal 2.2%
relative difference from fully-centralized NST in a comparable setup.

##  2 Background

End-to-End (E2E) ASR can be viewed as translating a sequence of input audio
frames $\mathbf{x}=(x_{1},\ldots,x_{T})$, into a sequence of corresponding
labels $\mathbf{y}=(y_{1},\ldots,y_{L})$.

Modern E2E ASR models consist of an encoder-decoder architecture trained using
a weighted sum of the sequence-to-sequence (Seq2Seq) [25] objective
$\mathcal{L}_{\text{Seq2Seq}}$ and the Connectionist Temporal Classification
(CTC) [26] objective $\mathcal{L}_{\text{CTC}}$:

$\mathcal{L}=\nu\mathcal{L}_{\text{CTC}}+(1-\nu)\mathcal{L}_{\text{Seq2Seq}}\vspace{0cm}$
(1)

where $\nu\in[0,1]$. The CTC objective helps with convergence during the early
stages of the training and is more robust to noisy conditions while the
attention-based Seq2Seq objective helps in understanding long range
dependencies [27]. Recent advancements in E2E ASR models introduce self-
attention to better capture intra-sequence relationships [1].

Noisy Student Training (NST) for ASR, a semi-supervised learning algorithm
originally proposed for image classification [24], has been recently shown to
significantly improve ASR performance [4]. Briefly, NST [4] involves: (1)
training an initial model $\theta_{0}$ on a labelled dataset
$\mathcal{D}_{L}$; (2) integrating $\theta_{0}$ with a language model; (3)
generating pseudo-labels for an unlabelled dataset $\mathcal{D}_{U}$ with
$\theta_{0}$ (here data filtering and balancing may be applied); (4) training
$\theta_{0}$ on a mix of $\mathcal{D}_{L}$ and $\mathcal{D}_{U}$ to generate
$\theta_{1}$; (5) repeating steps 1-4 until convergence.

##  3 Proposed Approach

Federated Noisy Student Training (FedNST) considers a scenario where a corpus
of labeled data $\mathcal{D}_{L}$ is available on a central server. Each
client $i\in\mathcal{C}$, where $\mathcal{C}$ is the set of all clients, has
an unlabelled speech dataset $\mathcal{D}^{i}_{U}$. The aim of FedNST is then
to leverage the unlabelled datasets $\mathcal{D}^{i}_{U},\forall
i\in\mathcal{C},$ to improve accuracy of an ASR model trained on
$\mathcal{D}_{L}$ while preserving user privacy, i.e., without sending user
data to the server.

Algorithm 1 FedNST
$S$ indicates a server variable and $C$ indicates a client variable.

Input: $\theta_{0}$, $\textsc{ClientOpt}_{C,S}$, ServerOpt, $\eta_{C,S}$,
$\eta$, $T$, $E$

for each client $i\in\mathcal{C}$ in parallel do

$\textsc{PseudoLabel}(\theta_{0})$ $\triangleright$ generates
$\mathcal{D}^{i}_{\hat{U}}$

for $t=0,\ldots,T-1$ do

$\eta^{t}_{C}=\eta^{t-1}_{C}\mu^{t/\lambda}$ $\triangleright$ Init.
$\eta^{-1}_{C}=\eta_{C}$

Sample a subset $\mathcal{S}_{t}\subseteq\mathcal{C}$ of clients

for each client $i\in\mathcal{S}_{t}$ in parallel do

$\theta^{t}_{i}=\theta_{t},\eta^{t}_{i}=\eta^{t}_{C}$ $\triangleright$ receive
$\theta_{t},\eta^{t}_{C}$ from server

Retrieve: $\mathcal{D}^{i}_{\hat{U}}$

for $e=0,\dots,E-1$ do

for $b\in\mathcal{B}_{i}\sim\mathcal{D}^{i}_{\hat{U}}$ do

$g_{i}=\nabla\mathcal{L}_{i}(\theta_{i}^{t};b)$

$\theta_{i}^{t}=\textsc{ClientOpt}_{C}(\theta_{i}^{t},g_{i},\eta_{i}^{t},e|\mathcal{B}_{i}|+b)$

$\Delta_{i}^{t}=\theta^{t}_{i}-\theta_{t}$,
$n_{i}=|\mathcal{D}^{i}_{\hat{U}}|$ $\triangleright$ send $\Delta_{i}^{t}$,
$n_{i}$ to server

$n=\sum_{i\in\mathcal{S}_{t}}n_{i}$

$\Delta^{t}_{C}=\sum_{i\in\mathcal{S}}\frac{n_{i}}{n}\Delta_{i}^{t}$

$\theta^{t}_{S}=\theta_{t}$

for $b\in\mathcal{B}_{S}\sim\mathcal{D}_{L}$ do

$g_{S}=\nabla\mathcal{L}_{S}(\theta^{t}_{S};b)$

$\theta^{t}_{S}=\textsc{ClientOpt}_{S}(\theta^{t}_{S},g_{S},\eta_{S},b)$

$\Delta^{t}_{S}=\theta^{t}_{S}-\theta_{t}$

$\Delta_{t}=\alpha\Delta^{t}_{S}+(1-\alpha)\Delta^{t}_{C}$

$\theta_{t+1}=\textsc{ServerOpt}(x_{t},-\Delta_{t},\eta,t)$

###  3.1 Federated Noisy Student Training

Algorithm 1 describes training models using FedNST which requires an initial
ASR model trained using some labelled data. This requirement is supported by
several studies [28, 29, 30] which discuss the complexity of training modern
SotA architectures from scratch with FL.

Thus, a baseline ASR model $\theta_{0}$ is first trained at a central server
using $\mathcal{D}_{L}$ and the standard training procedure described in
Section 2. Next, the trained model $\theta_{0}$ is transferred to all
participating clients $\mathcal{C}$. The clients use $\theta_{0}$ to pseudo-
label their data $\mathcal{D}^{i}_{U}$ via PseudoLabel (Algorithm 2). A
language model (LM) $\phi$ is integrated into the ASR model and a beam search
[31] is used to improve the quality of the transcripts.

With participating clients having their pseudo-labelled data, at each training
round $t$, the server transfers it’s latest model $\theta_{t}$ to a randomly
sampled fraction of all participating clients
$\mathcal{S}_{t}\subseteq\mathcal{C}$. Next, each $\textit{client
}i\in\mathcal{S}_{t}$ trains on their pseudo-labelled data, generating a new
model, $\theta_{i}$ (see Algorithm 1). The updated models are then transferred
back to the server and aggregated to generate a new global model
$\theta_{t+1}$. This process is repeated for $T$ rounds or until convergence.
The optimization procedure, summarized in Algorithm 1, is described in detail
in Section 3.2.

We explore design choices for FedNST which balance model performance and
training cost on two fronts: when to pseudo-label and whether to re-use
labelled server data.

First, instead of sending model $\theta_{0}$ to all participating clients at
the beginning for the purpose of pseudo-labelling, we propose the following
alternatives:

  1. 1.

Producing the pseudo-labels using $\theta_{t}$ which is the model sent by FL
server every round (evaluated in Section 5.4)

  2. 2.

Clients participating in a given FL round receive $\theta_{0}$ for pseudo-
labelling and $\theta_{t}$ for training.

Second, the standard NST [4] procedure described in Section 2 involves mixing
the labelled and pseudo-labelled datasets before performing the FL training
step. Meanwhile, in supervised FL it is common to start with a model pre-
trained on the server with some labelled data, and during the FL process to
use only the client data for model updates [11, 30]. In this work, we perform
an empirical study to determine the optimal strategies for data mixing with
FedNST– only aggregating client updates, or incorporating the supervised pre-
training data on the server.

Algorithm 2 $\boldsymbol{\textsc{PseudoLabel}}(\theta)$ \- for every client
$i\in\mathcal{C}$.

Input from server: $\theta$

Retrieve: $\phi$, $\mathcal{D}^{i}_{U}$

Initialize: $\mathcal{D}^{i}_{\hat{U}}=\varnothing$

for $j=1,2,\ldots,|\mathcal{D}^{i}_{U}|$ do

$\mathbf{\hat{y}}^{i}_{j}=f(\mathbf{x}^{i}_{j};\theta;\phi)$

$\mathcal{D}^{i}_{\hat{U}}\leftarrow(\mathbf{x}^{i}_{j},\mathbf{\hat{y}}^{i}_{j})$

Store $\mathcal{D}^{i}_{\hat{U}}$ on client $i$ for future retrieval

###  3.2 Federated Optimization for FedNST

The optimization procedure described in Algorithm 1, adopts the notation
introduced in [32]. $\mathcal{L}(\cdot)$ denotes the ASR loss defined in (1).
ClientOpt and ServerOpt are the optimizers [32] that perform a single step of
gradient descent based on the weights $\theta$, gradients $g$ (or pseudo-
gradients $-\Delta$), learning rate $\eta$ and optimization step-count. All
clients locally train on their pseudo-labelled data using
$\textsc{ClientOpt}_{C}$ without sending this data to the server. The
optimizer $\textsc{ClientOpt}_{S}$ is used to train on the labelled data on
the server.

For each round $t$ in FL, the same model $\theta_{t}$ is trained in parallel
on both the server and on a set of randomly sampled clients $\mathcal{S}_{t}$.
The resulting weights difference $\Delta^{t}_{i}=\theta_{i}^{t}-\theta_{t}$ is
uploaded to the server by each participating client $i$ after their local
training for $E$ epochs. This is aggregated by the server to give
$\Delta^{t}_{C}$. Similarly, a weights difference $\Delta^{t}_{S}$ is produced
after server training. The hyper-parameter $\alpha$ performs a weighted
average of $\Delta^{t}_{C}$ and $\Delta^{t}_{S}$ to produce pseudo-gradients
$-\Delta_{t}$. Next, $-\Delta_{t}$ is passed to ServerOpt to produce a set of
weights for the next round $\theta_{t+1}$. This process is repeated for a pre-
determined number of rounds, $T$, or until convergence of models as measured
via WER on a test set located at the server.

##  4 Methodology

Evaluation Datasets:  We evaluate our proposed method on the LibriSpeech (LS)
dataset [33]. Following the setup described in [11], we divide the dataset
into two equal subsets, $\mathcal{A}$ and $\mathcal{B}$, each comprising 480h.
The two sets are disjoint, such that all data arising from a single speaker is
assigned to only one set.

In the experimental analyses, we explore two scenarios:

  1. 1.

Set $\mathcal{A}$ is used only for initial model pre-training, i.e., an FL
step is performed using only set $\mathcal{B}$ (no data mixing).

  2. 2.

Set $\mathcal{A}$ is mixed with $\mathcal{B}$ during federated training of
models as discussed in Algorithm 1.

Experimental Setup:  We use an end-to-end ASR model architecture [34] with a
Conformer (S) encoder [1], a transformer decoder and a joint CTC+Seq2Seq
objective. For Algorithm 1, we set $T=1000$, ${E=1}$, ${\eta_{C,S}=0.1}$,
${\eta=1.0}$, $\alpha=0.5$ and $\frac{|\mathcal{S}_{t}|}{|\mathcal{C}|}=7\%$.
We set an equivalent number of epochs for the centralized experiments. After
training, the model with the lowest WER on the dev-clean is selected. Results
reported on the dev-clean, test-clean and test-other sets of LS are obtained
by fusing the ASR model with a pre-trained LM and using a beam search of size
10. For the federated experiments, we use a FL simulation platform with design
characteristics similar to those described in [11].

Batch Normalization for FedNST: As reported in other works [35, 36, 37], we
find that using standard Batch Normalization (BN) for FL gives rise to
convergence issues due to data heterogeneity. To address this issue, we
replace all BN layers with a modified version of static Batch Normalization
(sBN) [36]. sBN does not keep track of running statistics, i.e., the moving-
average mean and variance associated with BN layers, during FL training. At
the end of FL training, it queries all clients sequentially to produce global
BN statistics which can then be used to evaluate the trained model.This post-
processing step has computational and privacy concerns [36] and it makes it
difficult to perform model evaluation during training.

Our modification is to simply re-use the running statistics from $\theta_{0}$
– the model trained using the supervised data corpus $\mathcal{D}_{L}$ located
at the server. The behaviour of our modified sBN layers during FL training is
the same as original BN: normalization using batch mean and variance, followed
by re-scale and shift using the trainable parameters $\gamma$ and $\beta$.
However, it is worth noting that this approach assumes that the client and
server data arise from the same data distribution i.e. we expect running
statistics before and after FL to be the same.

Federated Optimization in FedNST: By using the FedOpt formulation for
federated optimization [32], we examined the use of adaptive (FedAvgM and
FedAdam) and standard (FedAvg [10]) server-side optimizers for FL. We found
that the differences in accuracy between these optimizers were limited
(results not shown due to limited space), and hence choose the computationally
more efficient FedAvg in our study.

For the client-side optimizer, our default configuration [34] uses Adam to
train the model. Our experiments indicate that using Adam as a local optimizer
performs worse than using SGD because a cross-device FL system expects
stateless clients, whereas Adam has stateful parameters. This was also found
to be the case in [11]. Thus, we use the SGD optimizer for clients. We set
$\alpha$ defined in Algorithm 1 to $0.5$ for all experiments as we found this
to be the best value.

Learning Rate (LR) Decay for Clients in FedNST: When data is distributed
across clients (e.g. 1 client per speaker), we regard such a dataset to be
non-iid. Li et al. [38] reported that the LR of federated optimizers must
decay to guarantee convergence when training with non-iid data. Thus, we use
LR decay to seek a better global minima in analyses. Among various LR
reduction methods [39], we choose to experiment with decaying only the LR of
clients at the start of each round and keeping LR fixed for local training. We
use exponential decay as described in Algorithm 1 where an LR $\eta^{t}_{C}$
is shared among all clients, and updated at each round. We experimented with
various values for $\mu$ (decay rate) and $\lambda$ (decay steps) defining the
LR decay curve. Training curves used for selecting a suitable value for
$\lambda$ are shown in Figure 1. We found $\lambda=1000$ to perform the best,
and used this for all subsequent experiments. Therefore, for our setup, using
a weak decay is more effective than strong decay.

$\displaystyle{250}$$\displaystyle{500}$$\displaystyle{750}$$\displaystyle{1000}$$5.2$$5.35$$5.5$RoundsWER$\displaystyle{0}$$\displaystyle{500}$$\displaystyle{1000}$$\displaystyle{10^{-5}}$$\displaystyle{10^{-3}}$$\displaystyle{10^{-1}}$RoundsLRno
decay$\lambda$ = 125$\lambda$ = 250$\lambda$ = 500$\lambda$ =
1000$\lambda=2000$

Figure 1:  WER of models on the dev-clean (greedy search and no LM) (left) and
client LR (right) for various $\lambda$ values. Plot on the left is smoothed
using exponential moving average with weight 0.8 and only the last 750 rounds
are shown. Plot on the right is logarithmically scaled along the y-axis. Table
1: Description of data splits used for experiments.

Split |  | Hrs
---
| No. Spks
---
Supervision | Location
$\mathcal{A}$ | 480 | 1165 | Labelled | Server
$\mathcal{B}$ | 480 | 1173 | Unlabelled | Clients

##  5 Experimental Analyses and Results

###  5.1 Comparison with SotA Methods

In Table 2, we compare WER of models trained using centralized supervised
learning (SL), supervised Federated Learning (SFL), NST, and our proposed
FedNST.

The results given in the first row (indicated by $\textsc{SL}_{\text{seed}}$)
are obtained through training models on $\mathcal{A}$ (labelled). The results
given in the second row (indicated by SL) are obtained by training models on
$\mathcal{A}\cup\mathcal{B}$ (both labelled). Here, SL is the lower-bound or
best possible WER. We present 2.59 $\%$ WER on test-clean with 960h of
labelled data which is close to SotA for a comparable sized model [1]. The
last three rows present, NST, SFL and FedNST, trained over a combination of
$\mathcal{A}$ and $\mathcal{B}$, starting from the pre-trained model
$\textsc{SL}_{\text{seed}}$. All FedNST experiments use $\textsc{SL}_{seed}$
as $\theta_{0}$ from Table 2 unless stated otherwise. For simplicity, in our
study, both NST and FedNST models are trained for a single generation and
without any data filtering or balancing methods defined in [4].

FedNST achieves comparable WER over NST (2.2% relative difference for test-
clean), without the need of sending user data to a central server. We further
observe a relative WER increase of 8.5% when comparing SFL with FedNST. This
is still smaller than the 22.5% relative WER reduction achieved by FedNST over
$\textsc{SL}_{seed}$. These results provide strong motivations to use FedNST
in a real-world scenario.

Table 2:  Comparison of WER (%) of models trained under different labelled
data regimes.

Method | L | U | test-clean | test-other
---|---|---|---|---
$\text{{SL}}_{\text{seed}}$ | $\mathcal{A}$ | $\varnothing$ | 4.27 | 8.99
SL | $\mathcal{A}\cup\mathcal{B}$ | $\varnothing$ | 2.59 | 6.30
NST | $\mathcal{A}$ | $\mathcal{B}$ | 3.24 | 7.96
SFL | $\mathcal{A}\cup\mathcal{B}$ | $\varnothing$ | 3.05 | 7.57
FedNST | $\mathcal{A}$ | $\mathcal{B}$ | 3.31 | 8.07

Table 3: An analysis of the relationship between the duration of labelled data
$|\mathcal{A}|$ in hours and WER (%) of models.

$|\mathcal{A}|$ | $\text{{SL}}_{\text{seed}}$ | FedNST
---|---|---
test-clean | test-other | test-clean | test-other
120 | 9.44 | 17.96 | 7.89 | 15.86
240 | 6.96 | 12.91 | 5.30 | 12.01
360 | 5.52 | 11.02 | 4.69 | 10.01
480 | 4.27 | 8.99 | 3.31 | 8.07

###  5.2 Analyses with Different Pseudo-Labelled Data Regimes

Intuitively, models trained with fully supervised data perform better than
those trained with self-training methods (e.g. NST) on partially labelled
data. This is due to the noise induced by inaccurately predicted labels. We
explore how changing the proportion of labelled and pseudo-labelled data
affects a model’s performance. The purpose of this analysis is to: (A)
quantify the hypothetical gain in performance if ground-truth labels were
available for all clients, and (B) empirically verify if $\mathcal{A}$ is
useful during federated training. The results are depicted in Figure 2.

Figure 2 presents two analyses: Figure 2 (a) varies the percentage of clients
with entirely labelled data against clients which pseudo-label their data
(i.e., applying in Algorithm 2). Figure 2 (b) varies percentage of labelled
samples per client, i.e., ratio of samples with ground-truth against samples
that are pseudo-labelled for each client. Each plot in Figure 2 presents two
sets of graphs showing results from (i) test-clean and (ii) test-other,
respectively. Within each set, graphs representing
$\mathcal{A}\cup\mathcal{B}$ (blue) and $\mathcal{B}$ (red) show the impact of
dataset mixing in FL. Percentage of labelled clients and labelled samples per
client is thereafter referred to as ‘x%-labelled’ for brevity.

Figures 2 (a) and (b) show similar trends, since in both cases the cumulative
hours of labelled vs. unlabelled data obtained from all clients are roughly
the same. With reference to (A), we find that 0%-labelled leads to a relative
WER increase of 8.5% compared to 100%-labelled for the
$\mathcal{A}\cup\mathcal{B}$ setup. With reference to (B), training models
with FedNST using only $\mathcal{B}$ does not cause significant catastrophic
forgetting. Instead, the model retains its ability to further learn from new
data. We find that when using 0%-labelled, re-using $\mathcal{A}$ (i.e., data-
mixing), produces 0.5 WER improvement. However, this improvement reduces to
nearly 0 when at least 25% data is labelled.

We hypothesize that the benefits seen with $\mathcal{A}\cup\mathcal{B}$ at
0%-labelled come from $\mathcal{A}$ acting as a form of regularisation when
most of client data are pseudo-labelled. Exploring continual learning
techniques for this problem is a promising future direction. It is worth
noting the surprisingly small difference in WER seen with and without mixing
labelled and unlabelled data – this is likely due to the nature of LS rather
than a general observation for ASR datasets (as discussed in [11, 30]).

$7.5$$8$$8.5$(a)$\mathcal{A}\cup\mathcal{B}$$\mathcal{B}$test-cleantest-
other$7.5$$8$$8.5$(b)$0$$50$$100$$3$$3.25$$3.5$Labelled clients ($\%$)WER
($\%$)$0$$50$$100$$3$$3.25$$3.5$ Labelled samples
per client ($\%$)

Figure 2: Reduction of WER as the percentage of labelled clients (a) or
samples per client (b) increases. Relative WER increases from labelled (100%)
to pseudo-labelled (0%) data regime.

###  5.3 Analyses with Different Labelled Data Regimes

We explore the effect of changing the amount of labelled data $\mathcal{A}$ to
produce $\text{{SL}}_{\text{seed}}$ and further with pseudo-labelled
$\mathcal{B}$ with FedNST in Table 3. We used
${|\mathcal{A}|\in\\{120,240,360,480\\}}$ hours and $|B|=480\text{ hours}$ for
all experiments. As expected, decreasing available labelled data for pre-
training shows higher WER compared to FedNST results in Table 2. Table 3 also
points to a clear correlation between the WER of the initial pre-trained model
and that of FedNST.

###  5.4 Analyses with Varying Pseudo-Labelling Frequency

We ran an experiment to test our alternative pseudo-labelling strategy in
which PseudoLabel is performed every round $t$ for all clients in
$\mathcal{S}_{t}$. We provide the results in Table 4. Although this strategy
performs very similarly to our original approach in terms of WER, the pseudo-
labelling step is highly computationally expensive, increasing the overall FL
experiment time by 10x. We leave optimizing on this front as future work.

Table 4: Comparison of pseudo-labelling strategies, using labelled
$\mathcal{A}$ and pseudo-labelled $\mathcal{B}$ for 50 rounds.

Labelling | test-clean | test-other | Wall-clock/FL Round (Mins)
---|---|---|---
Once | 3.31 | 8.07 | 1.74
Every round | 3.34 | 8.29 | 18.58

##  6 Conclusions

We have proposed a new method called FedNST for semi-supervised training of
ASR models in FL systems. FedNST performs noisy student training to leverage
private unlabelled user data and improves the accuracy of models in low-
labelled data regimes using FL. Evaluating FedNST on real-world ASR use-cases
using the LibriSpeech dataset with over 1000 simulated FL clients showed 22.5%
relative WERR over a supervised baseline trained only with labelled data
available at the server. Our analyses showed that FedNST achieves a WER
comparable to fully centralized NST and to supervised training while incurring
no extra communication overhead compared to FedAvg. In the future, we plan to
employ FedNST on more challenging datasets, e.g., CommonVoice [15], and to
incorporate other methods to learn from unlabelled data, such as Wav2Vec2.0
[17].

## References

  * [1] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in _Interspeech_. ISCA, 2020, pp. 5036–5040.
  * [2] Y. Zhang, J. Qin, D. S. Park, W. Han, C. Chiu, R. Pang, Q. V. Le, and Y. Wu, “Pushing the limits of semi-supervised learning for automatic speech recognition,” _CoRR_ , vol. abs/2010.10504, 2020.
  * [3] J. E. van Engelen and H. H. Hoos, “A survey on semi-supervised learning,” _Mach. Learn._ , vol. 109, no. 2, pp. 373–440, 2020.
  * [4] D. S. Park, Y. Zhang, Y. Jia, W. Han, C. Chiu, B. Li, Y. Wu, and Q. V. Le, “Improved noisy student training for automatic speech recognition,” in _INTERSPEECH_. ISCA, 2020, pp. 2817–2821.
  * [5] B. W. Schuller, S. Steidl, A. Batliner, E. Nöth, A. Vinciarelli, F. Burkhardt, R. van Son, F. Weninger, F. Eyben, T. Bocklet, G. Mohammadi, and B. Weiss, “A survey on perceived speaker traits: Personality, likability, pathology, and the first challenge,” _Comput. Speech Lang._ , vol. 29, no. 1, pp. 100–131, 2015.
  * [6] Y. Gu, X. Li, S. Chen, J. Zhang, and I. Marsic, “Speech intention classification with multimodal deep learning,” in _Canadian Conference on AI_ , ser. Lecture Notes in Computer Science, vol. 10233, 2017, pp. 260–271.
  * [7] M. Kotti and C. Kotropoulos, “Gender classification in two emotional speech databases,” in _ICPR_. IEEE Computer Society, 2008, pp. 1–4.
  * [8] B. M. L. Srivastava, A. Bellet, M. Tommasi, and E. Vincent, “Privacy-preserving adversarial representation learning in ASR: reality or illusion?” in _Interspeech_. ISCA, 2019, pp. 3700–3704.
  * [9] J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” _CoRR_ , vol. abs/1610.05492, 2016.
  * [10] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _AISTATS_ , ser. Proceedings of Machine Learning Research, vol. 54. PMLR, 2017, pp. 1273–1282.
  * [11] D. Dimitriadis, K. Kumatani, R. Gmyr, Y. Gaur, and S. E. Eskimez, “A federated approach in training acoustic models,” in _Interspeech_. ISCA, 2020, pp. 981–985.
  * [12] Y. Gao, T. Parcollet, J. Fernández-Marqués, P. P. B. de Gusmao, D. J. Beutel, and N. D. Lane, “End-to-end speech recognition from federated acoustic models,” _CoRR_ , vol. abs/2104.14297, 2021.
  * [13] D. Guliani, F. Beaufays, and G. Motta, “Training speech recognition models with federated learning: A quality/cost framework,” in _ICASSP_. IEEE, 2021, pp. 3080–3084.
  * [14] X. Cui, S. Lu, and B. Kingsbury, “Federated acoustic modeling for automatic speech recognition,” in _ICASSP_. IEEE, 2021, pp. 6748–6752.
  * [15] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in _LREC_. European Language Resources Association, 2020, pp. 4218–4222.
  * [16] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in _NeurIPS_ , 2020\.
  * [17] Y. Chung, Y. Zhang, W. Han, C. Chiu, J. Qin, R. Pang, and Y. Wu, “w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in _ASRU_. IEEE, 2021, pp. 244–250.
  * [18] J. Kahn, A. Lee, and A. Hannun, “Self-training for end-to-end speech recognition,” in _ICASSP_. IEEE, 2020, pp. 7084–7088.
  * [19] W. Jeong, J. Yoon, E. Yang, and S. J. Hwang, “Federated semi-supervised learning with inter-client consistency & disjoint learning,” in _ICLR_. OpenReview.net, 2021.
  * [20] H. Lin, J. Lou, L. Xiong, and C. Shahabi, “Semifed: Semi-supervised federated learning with consistency and pseudo-labeling,” _CoRR_ , vol. abs/2108.09412, 2021.
  * [21] Z. Zhang, S. Ma, J. Nie, Y. Wu, Q. Yan, X. Xu, and D. Niyato, “Semi-supervised federated learning with non-iid data: Algorithm and system design,” _CoRR_ , vol. abs/2110.13388, 2021.
  * [22] W. Zhuang, Y. Wen, and S. Zhang, “Divergence-aware federated self-supervised learning,” in _ICLR_. OpenReview.net, 2022.
  * [23] K. Nandury, A. Mohan, and F. Weber, “Cross-silo federated training in the cloud with diversity scaling and semi-supervised learning,” in _ICASSP_. IEEE, 2021, pp. 3085–3089.
  * [24] Q. Xie, M. Luong, E. H. Hovy, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” in _CVPR_. Computer Vision Foundation / IEEE, 2020, pp. 10 684–10 695.
  * [25] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in _ICASSP_. IEEE, 2016, pp. 4945–4949.
  * [26] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in _ICML_ , ser. JMLR Workshop and Conference Proceedings, vol. 32. JMLR.org, 2014, pp. 1764–1772.
  * [27] S. Kim, T. Hori, and S. Watanabe, “Joint ctc-attention based end-to-end speech recognition using multi-task learning,” in _ICASSP_. IEEE, 2017, pp. 4835–4839.
  * [28] A. Rosenberg, K. Audhkhasi, A. Sethy, B. Ramabhadran, and M. Picheny, “End-to-end speech recognition and keyword search on low-resource languages,” in _ICASSP_. IEEE, 2017, pp. 5280–5284.
  * [29] S. Bansal, H. Kamper, K. Livescu, A. Lopez, and S. Goldwater, “Pre-training on high-resource speech recognition improves low-resource speech-to-text translation,” in _NAACL-HLT (1)_. Association for Computational Linguistics, 2019, pp. 58–68.
  * [30] Y. Gao, T. Parcollet, J. Fernández-Marqués, P. P. B. de Gusmao, D. J. Beutel, and N. D. Lane, “End-to-end speech recognition from federated acoustic models,” _CoRR_ , vol. abs/2104.14297, 2021.
  * [31] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _NIPS_ , 2014, pp. 3104–3112.
  * [32] S. J. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” in _ICLR_. OpenReview.net, 2021.
  * [33] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an ASR corpus based on public domain audio books,” in _ICASSP_ , 2015, pp. 5206–5210.
  * [34] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J. Chou, S. Yeh, S. Fu, C. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “Speechbrain: A general-purpose speech toolkit,” _CoRR_ , vol. abs/2106.04624, 2021.
  * [35] K. Hsieh, A. Phanishayee, O. Mutlu, and P. B. Gibbons, “The non-iid data quagmire of decentralized machine learning,” in _ICML_ , ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 4387–4398.
  * [36] E. Diao, J. Ding, and V. Tarokh, “Heterofl: Computation and communication efficient federated learning for heterogeneous clients,” in _ICLR_. OpenReview.net, 2021.
  * [37] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou, “Fedbn: Federated learning on non-iid features via local batch normalization,” in _ICLR_. OpenReview.net, 2021.
  * [38] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” in _ICLR_. OpenReview.net, 2020.
  * [39] Z. Charles and J. Konečný, “On the outsized importance of learning rates in local update methods,” _CoRR_ , vol. abs/2007.00878, 2020.