11institutetext:  Lero – the Irish Software Research Centre ,
National University of Ireland, 11email<EMAIL_ADDRESS>
# Blockchain as an Enabler for Transfer Learning in Smart Environments

Amin Anjomshoaa  0000-0001-6277-742X Edward Curry  0000-0001-8236-6433

###### Abstract

The knowledge, embodied in machine learning models for intelligent systems, is
commonly associated with time-consuming and costly processes such as large-
scale data collection, data labelling, network training, and fine-tuning of
models. Sharing and reuse of these elaborated models between intelligent
systems deployed in a different environment, which is known as transfer
learning, would facilitate the adoption of services for the users and
accelerates the uptake of intelligent systems in environments such as smart
building and smart city applications. In this context, the communication and
knowledge exchange between AI-enabled environments depend on complicated
networks of systems, system of systems, digital assets, and their chain of
dependencies that hardly follows the centralized schema of traditional
information systems. Rather, it requires an adaptive decentralized system
architecture that is empowered by features such as data provenance, workflow
transparency, and validation of process participants. In this research, we
propose a decentralized and adaptive software framework based on blockchain
and knowledge graph technologies that supports the knowledge exchange and
interoperability between IoT-enabled environments, in a transparent and
trustworthy way.

###### Keywords:

Smart Environment Blockchain Machine Learning Knowledge Graph.

##  1 Introduction

Internet of Things (IoT) and cognitive computing are the two major enabling
technologies in smart environment solutions such as smart home, smart
building, and smart office use cases. Thanks to the proliferation of
networking and communication trends and since the emergence of IoT, deploying
the IoT infrastructure for sensing and monitoring of spaces is rather a
straightforward practice. However, cognitive knowledge cannot be explicitly
defined and is acquired by observation and experimentation. An example of this
type of cognitive knowledge is machine learning models that are taught to
undertake complex tasks and to create explicit and actionable knowledge from
descriptive knowledge.

The IoT-enabled systems are typically built following a three-layer model that
consists of: (i) a sensing layer, which acquires the observation of interest
from the environment; (ii) a cognitive layer, which is concerned with context
acquisition, modeling, and reasoning; and (iii) an actuate layer that triggers
an action or invokes a service according to some predefined logic. The main
effort required for automatic change management in a private IoT space is
hidden in the cognitive layer. Unlike the sense and actuate layers which
typically undertake straightforward tasks, the creation and configuration of
the context layer is a complicated task which is not necessarily based on a
syntactic or deterministic model. Instead, it needs a deep understanding of
incoming events as well as the context of those events. Currently, human
intervention is necessary to interpret the incoming data from the sensor layer
and add the missing semantics about the events and their context.

Figure 1: Three-layer model of an IoT-based system including sensing, context,
and actuation layers.

In order to benefit the services of our IoT-enabled environments, we need to
calibrate and configure them based on space-specific properties and semantics.
These personalized configurations are commonly captured in dedicated
information resources such as service compositions, sense-actuate cycles, and
well-trained machine-learning models. In this context, the smart space
services, embodied in machine learning models, are commonly associated with
time-consuming and costly processes such as large-scale data collection, data
labeling, network training, and fine-tuning models. Increasing access to the
cognitive knowledge held by these models unlocks the value of transfer
learning in smart environment and enables sustainable and autonomous growth of
IoT-enabled environment.

In the context of smart built environments, machine learning methods benefit
from the huge amount of data in IoT-enabled spaces in their training
processes. Such models could yield significant profit over the lifetime of a
building but due to the high cost of training and the need of large amounts of
training and testing data sets, the uptake of machine learning applications in
built environments is rather slow. For instance, in a smart environment a
speech recognition model could be trained based on a rather big and elaborated
dataset. However, the model might not be used in different spaces as it should
be retrained to cope with the surrounding noise of the target space. This
gives smart environments an incentive to reuse and re-purpose the existing
models via techniques such as transfer learning where models trained on one
task are re-purposed for a second related task.

Currently, machine learning processes are commonly built for intra-
organization purposes and tailored towards specific use cases with the
assumption of integrated model repositories and feature pools. In order to
unleash the power of these resources, they need to be shared and reused in a
trustworthy and transparent way which requires an adaptive decentralized
system architecture that is empowered by features such as data provenance,
workflow transparency, and validation of process participants.

In order to make the machine learning processes for intelligent systems and
their digital assets the first-class citizens of a decentralized and peer-to-
peer ecosystem, the following challenges should be addressed:

  1. 1.

Integrity: As we move beyond the borders of ML model’s source environments
(e.g. space, building, or city), we would need to fortify the integrity and
traceability of adopted digital assets such as training datasets or
transferred models.

  2. 2.

Transparency and trust: in a decentralized process, the transparency and
trustworthiness of actors and digital assets are of great importance. Although
the transparency and trustworthiness are the main drivers behind the shift to
blockchain technology, but issues such as communication between private and
public blockchains and inter-organizational trust networks needs to be
investigated via a holistic approach that takes the specific characteristics
of machine learning pipelines into account.

  3. 3.

Explainable machine learnig chains: the transfer learning processes are
commonly undertaken by a human who is able to understand the implicit
semantics of environments, features, as well as the relevant information
resources and services. Therefore, self-explaining and machine-understandable
digital assets (e.g. features and models) are the key requirements to enable
transfer learning beyond the organizational borders.

In this research, we propose a decentralized and adaptive software framework
based on blockchain and knowledge graph technologies for reliable and
decentralized transfer of machine learning digital assets between AI-enabled
smart environments such as smart buildings and smart city, in a transparent
and trustworthy way. The proposed approach in this research follows the
principles of Inter-Space Learning (ISL) framework [1] that was presented ion
our previous work. Using ISL framework, smart environment resources are
processed, aligned, and reconciled to create a uniform and interconnected
information space that facilitates provisioning of the Inter-space learning.
This research an adaptive intelligent system architecture and advances the ISL
framework by:

  1. 1.

Establishing a decentralized feature space that provides context for machine
learning processes and makes their digital assets explainable and
interoperable for transfer learning purposes. This will be achieved by
creating and maintaining a knowledge graph that is aware of both machine
learning features, their semantics, and the models trained based on those
features. The knowledge graph includes also the evaluation indicators of
machine learning models which facilitate the sharing and reuse of models
beyond the organization boundaries.

  2. 2.

Integration of blockchains into machine learning pipelines in order to
safeguard the data integrity and trustworthiness in cross-organization
transfer learning scenarios. To this end, we exploit blockchain technology to
capture the interaction between machine learning digital assets including raw
data, features, and models.

  3. 3.

Creating a collaborative and adoptive framework based on public and private
blockchains where participants can continually share, reuse, and evaluate
machine learning assets in a reliable and incentive-oriented manner. As such,
the participants will be able to benefit from publicly-traceable ML pipelines
and integrate them in their secure intra-organization processes.

###  1.1 Building Industry as a Motivational Use Case

Improvement of the energy performance of buildings is a major contributor for
achieving the goals of the EU 2030 climate and energy framework
111https://ec.europa.eu/energy/sites/ener/files/documents/1.final_report.pdf.
By 2050 at least 75% of today’s buildings will still exist and as a result,
renovation of existing buildings and making them energy-efficient will have
significant economic and environmental impacts. The majority of legacy
buildings were mainly built without considering the energy performance
requirements and there is a high potential of saving in operational energy
costs. Furthermore, by renovating the legacy buildings, the embodied energy of
new constructions which makes up a major part of the total energy use in
buildings, could be avoided.

Current renovation processes commonly extend the existing buildings with
additional intelligence in order to optimize the operational costs. In this
context, the major enabling technologies are IoT and cognitive computing.
While deploying IoT infrastructure during the renovation processes is rather a
straightforward practice, the implementation of cognitive components that
benefit from generated data of IoT infrastructure is a complicated task that
should be elaborated based on the specific attributes of tasks and buildings.

In smart buildings, such cognitive components can be realized via machine
learning models that are taught to undertake complex tasks. The high costs of
creating such models which require large amounts of training and testing
datasets, gives smart environments the incentive to share and reuse the
existing models. This research aims to investigate the potential of
communication and reuse of cognitive knowledge between smart environments
using blockchain technology.

##  2 Knowledge Graph

In our previous work, we introduced the Inter-Space Learning Framework (ISL)
[1] Which uses knowledge graph as an efficient medium for communicating the
semantics of smart environments and bridging the information gap among IoT
infrastructure, spaces, and machine learning processes.

As such the digital assets of machine learning processes are presented in an
explainable and interoperable way which will facilitate the transfer of
cognitive components. The proposed knowledge graph in ISL is aware of IoT
infrastructure of target environments as well as machine learning features,
their semantics, and the models trained based on those features. The knowledge
graph includes also the evaluation indicators of machine learning models which
facilitate the sharing and reuse of models beyond the organization boundaries.

As depicted in Figure 2, machine learning processes can be characterized based
on their input data, output model, and model evaluation measures such as Mean
Absolute Error (MAE) or Mean Squared Error (MSE) of learning processes.
Furthermore, each model is dedicated to a specific task defined by ML-Schema
which helps spaces to find the relevant models shared by other spaces. As
such, the models are presented in a self-explainable and interpretable way to
both human users as well as space agents (machines).

The training process depicted in Figure 2 undertakes a transfer learning
process. In this specific case, the process includes a fine-tuning process
which takes the trained model of another room (model of Room-2) as input and
retrain it based on a small fine-tuning dataset of target room (Room 3).

Figure 2: Definition of a model in Knowledge Graph

The ISL frameworks envisions communication between multiple ISL nodes where
each node represents the underlying knowledge of corresponding smart
environment captured in the knowledge graph using a common schema of building
assets (i.e. semantics of sensors, services, etc.). Based on this
configuration users will be able to query services that match the
characteristics of a given space (i.e., IoT devices such as sensors and
actuators). As shown in Figure 3, in a model query scenario, user can match
the IoT specification of target space with the offered models and retrieve the
feasible models. For instance, in case of room occupancy detection based on
CO2 sensors, the target room should be equipped with a CO2 sensor and all
retrieved models should also accept CO2 data as input. In the next step, the
retrieved models can be ranked based on quality indicators which are also
available in knowledge graph and are offered by the ISL framework. As an extra
step and if some training data is available in the target space, the model can
be fine-tuned for a better performance.

Figure 3: Model query and reuse process.

##  3 Proposed Approach

In order to realize this requirements, the ISL resources needs to be shared
and advertised on blockchain by ISL nodes. These resources are usually not
deterministic which contradicts with the nature of blockchain. The technical
solution to address this issue is to use an Oracle Smart Contract which is
responsible to retrieve and verify external data for blockchain and smart
contracts through methods such as web APIs or data feeds.

In our case, we would need an oracle to interact with external ISL nodes that
are willing to share and reuse machine learning models and their corresponding
assets. In our implementation, the oracle smart contract is created by the
network owner and manages list of trustworthy ISL nodes. It also allows the
trustworthy ISL nodes to add their resources and maintains a registry of
shared resources.

In this section, we formally define the knowledge sharing framework and
outline how we use the blockchain technology to facilitate the communication
and knowledge exchange between ISL nodes.

Let $\mathcal{N}$ be the set of ISL nodes representing the digital twins of
smart environments that are supposed to take part in transfer learning use
cases. Each ISL node $\alpha\in\mathcal{N}$ is then characterized by its
knowledge graph that we denote by $\mathcal{G}_{\alpha}$ and captures the
description of machine learning assets such as training datasets and ML
models. We define a local dataset as a dataset that is captured in
$\mathcal{G}_{\alpha}$ and owned by the corresponding ISL node.

We denote the collection of local datasets introduced in knowledge graph
$\mathcal{G}_{\alpha}$ by $\mathcal{D}|_{G_{\alpha}}$. Additionally, the
subset of local datasets that are shared by the ISL node $\alpha$ are denoted
by $\mathcal{D}^{|}_{G_{\alpha}}$ and we refer to them as blockchain-enabled
datasets. Further, we denote the set of all blockchain-enabled datasets as
$\mathcal{D}^{*}$ and define it as follows:

$\mathcal{D}^{*}=\\{\mathcal{D}^{*}|_{G_{n}}:n\in\mathcal{N}\\}$

Consequently, the models that are introduced in $\mathcal{G}_{\alpha}$ can be
trained using the available local or blockchain-enabled datasets to achieve a
specific task. The set of all models introduced by the knowledge graph
$\mathcal{G}_{\alpha}$ is defined as:

$\mathcal{M}|_{G_{\alpha}}=\\{M_{i}\langle
task,dataset,model\\_uri,base\\_model\rangle:i=1,2,\ldots,m\\}$

Where task is an instance of class Task defined in the knowledge graph;
model_uri is the identifier of the target model file; and dataset is selected
from $\mathcal{D}|_{G_{\alpha}}\cup\mathcal{D}^{*}$. Furthermore, in case of
models that are trained from scratch, no base model is needed and we set the
base-model to null. However, in a transfer learning case, the new model reuses
an existing base model that either resides on the same ISL node as the new
model, or comes from the pool of shared models from other ISL nodes. Similar
to shared datasets, we denote the subset of local models that are shared by
the ISL node $\alpha$ as $\mathcal{M}^{*}|_{G_{\alpha}}$. We also denote the
set of all blockchain-enabled models as $\mathcal{M}^{*}$ and define it as
follows:

$\mathcal{M}^{*}=\\{\mathcal{M}^{*}|_{G_{n}}:n\in\mathcal{N}\\}$

So, the base_model in a transfer learning case is selected from
$\mathcal{M}|_{G_{\alpha}}\cup\mathcal{M}^{*}$. As such, each model
$\mathcal{M}_{n}$ depends on its training/fine-tuning dataset
$\mathcal{D}_{n}$ and its base_model (if any) $\mathcal{M}_{n-1}$. Then the
dependency chain of model $\mathcal{M}_{n}$ that include a number of
blockchain-enabled models and datasets, can be constructed as follows:

$\varnothing\xrightarrow{\mathcal{D}_{0}}\mathcal{M}_{0}\xrightarrow{\mathcal{D}_{1}}\mathcal{M}_{1}\rightarrow{\ldots}\xrightarrow{\mathcal{D}_{n}}\mathcal{M}_{n}$

At the root of all dependency chains, we have a model created based on a null
model and using a specific dataset. This indicates that this model is created
from scratch trained based on the specified dataset. Following this
presentation, the collection of all dependency chains can be captured as a
directed acyclic graph where vertices represent models and edges represent the
corresponding dataset. Furthermore, all vertices, except for the root vertex,
have in-degree one. An example of such dependency graph is depicted in Figure
4 where shared models are shown as red boxes. It’s important to note that all
assets of a shared model including ancestor models and the datasets used for
training/fine-tuning in the dependency chain are required to be shard as well.
In this way, one can easily trace back all the used assets of each model and
investigate the authenticity of models and datasets.

Figure 4: Example of model dependency graph.

As depicted in Figure 5, the blockchain is used as shared medium between
service providers and service consumers on ISL network to store the metadata
of machine learning assets.

Figure 5: Process of sharing and consuming of cognitive resources between
smart dataspaces.

Our proposed blockchain solution, includes an Oracle smart contract that keeps
track of sharing resources and their dependency chains. More specifically, the
Oracle at its basic level is defined as follows:

$Oracle=\langle\mathcal{D}^{*},\mathcal{M}^{*},task\\_index\rangle$

Where task_index maps task instances to models and facilitates searching for
appropriate models.

The resource providers register their models and dataset via the Oracle smart
contract as shown in Figure 6. To this end, the network owner needs to
introduce the ISL node to blockchain by registering them to the oracle smart
contract. Then, registered ISL nodes will be able to add the metadata of their
resources to the oracle smart contract. In return, they receive a unique
identifier that will be added to knowledge graph for further queries.

Figure 6: Sharing resources on blockchain

On the other hand, as shown in Figure 7, the resource consumers can query the
Oracle smart contract for the machine learning models that fulfill a specific
task. Then, the consumer can negotiate the corresponding resource owners (ISL
nodes) for finding the qualified resources. In the next step, the consumer
will pay for the resource and receives the resource address and access token
to retrieve the requested resource.

Figure 7: Consuming blockchain resources.

##  4 Implementation

As proof of concept, we have implemented a prototype based on the proposed
methodology. In this section, we provide an overview of the technical
architecture which is depicted in Figure 8 extends our previous implementation
of ISL framework for knowledge sharing between smart environments [1].

At the core of ISL framework, we have the knowledge graphs of ISL nodes that
include the description of spaces, IoT configurations, and machine learning
assets. The knowledge graphs are maintained by each ISL node and stored using
Cayley222https://github.com/cayleygraph, an open-source database for Linked
Data. The node implementation offers APIs for communication between ISL nodes
as well as a web interface for displaying and visualizing the knowledge graph
assets and entities. In the new implementation, we leverage the existing
knowledge graphs in order to store the relevant information of blockchain
environment and blockchain-enabled assets. This includes the information of
blockchain accounts as well as the blockchain addresses of shared resources
such as datasets and models.

Figure 8: Architecture of implemented prototype.

In order to make the ISL nodes blockchain-enabled, the ISL nodes web
application is upgraded to a decentralized application(dApp) implemented on
Ethereum test network. We have used Ganache
333https://www.trufflesuite.com/ganache environment to develop and test the
required smart contracts. To this end, each ISL nodes is equipped with a
unique account address that is used to identify the corresponding ISL node on
the blockchain. The implementation includes the following two smart contracts
that are implemented in Solidity programming language [3]:

  * •

ISL smart contract which provides an interface for interaction between ISL
nodes and organizes transactions such as acquiring resources (e.g.; datasets
and models) and performing payment transactions if required.

  * •

Oracle smart contract which is responsible for governance of participating ISL
nodes and manages dedicated lists of shared resources. This smart contract
also includes the account number of the oracle owner who has the authority to
register new ISL nodes. As such, only trusted ISL nodes will be allowed to
contribute their resources or acquire the shared resources from other trusted
ISL nodes.

In this architecture, each ISL node stores its resources off-chain in a local
file repositories which is only accessible to the ISL node itself. However,
when the resource is shared on blockchain, we would need to provide access to
the resource files to external ISL nodes. To this end, we use InterPlanetary
File System (IPFS) which is a protocol and peer-to-peer network for storing
and sharing data in a distributed file system. In our case, whenever an ISL
node shares a resource such as dataset of model on blockchain, the underlying
files will be stored on IPFS. Then, the unique content-address which is
assigned by IPFS will be added to both the local knowledge graph and the
corresponding list in oracle smart contract. Since the IPFS provides a
universally unique address for each resource, the resources cannot be shared
twice. As such, the ownership rights of resources can be checked and preserved
by the oracle smart contract.

Figure 9 demonstrates the representation of a blockchain-enabled dataset in
the knowledge graph which includes the URI of the local dataset file, the URI
of shared dataset on IPFS, and the unique transaction identifier which is
assigned by the oracle smart contract.

Figure 9: Structure of blockchain-enabled resources in knowledge graph.

##  5 Related Work

There is a handful of methodologies and platforms that facilitate knowledge
management and data sharing for machine learning scenarios. In this section,
we explore a number of existing solution and research work that are expanded
on the fields of machine learning, blockchain, and data sharing frameworks.

###  5.1 Blockchain-based Machine Learning Approaches

Machine learning solutions are commonly designed to be used within
organization boundaries. However due to the costly approached for creating
elaborated models, it makes sense to share and reuse the machine learning
assets beyond organization borders. In this context we would need a
trustworthy and sustainable approach to make the machine learning models
accessible to external entities.

Recently, several research works have used blockchain as share and reuse
medium in machine learning use cases. For instance, DInEMMo [11] proposes a
solution that combines machine learning and blockchain frameworks to create a
marketplace for machine learning models that enables to reward both model
creators as well as actors that contribute to enhance existing models. In
[13], authors present BlockDeepNet, a Blockchain-based secure deep learning
solution that combines deep learning and blockchain technologies to support
secure collaborative learning in IoT environments. the work in [8] proposes a
framework for participants to collaboratively build a dataset and use smart
contracts to host a continuously updated model. This work also presents
financial and non-financial (gamified) incentive structures for providing good
data and includes a prototype implemented on Ethereum blockchain.

Another study on secure and resilient machine learning adoption for making
blockchain-based smart applications has been presented in [15] where
traditional machine learning techniques are applied to analyze the attacks on
a blockchain-based network. The study also discusses how these technologies
can be applied in applications such as Unmanned Aerial Vehicle (UAV), Smart
Grid (SG), healthcare, and smart cities. The work in [17] proposes a solution
for data trading based on smart contract using blockchain to protect the
rights and interests of the data owner, and off-chain machine learning methods
for addressing the data availability controversies.

The work in [10] have proposed a protocol based on the anonymous and
distributed nature of smart contracts, and the problem-solving aspect of
machine learning which helps users obtain machine learning models for a given
fee. The protocol does not require trust and works completely on a
decentralized blockchain. The work in [6] presents a configurable framework
for training a model and collecting data on blockchain.

Decentralized machine learning protocol (DML) [4] project focuses on on-device
machine learning, blockchain and federated learning technologies. It unleashes
untapped data usage without extraction and idle processing power of user
computers for machine learning. Algorithms will be crowdsourced from a
developer community through the marketplace resulting innovation from
periphery. The GNY’s decentralized machine learning solution [5] aims to bring
a secure, collaborative platform for machine learning, data capture and
analysis to the Lisk blockchain project.

###  5.2 Artifact Sharing in Machine Learning Environments

Machine learning solution consist of multiple sequential steps that range from
data processing and feature engineering to model training and deployment.
There is a handful of existing solutions and research work for managing the
life cycle of machine learning processes and pipelines.

ModelDB [16] is an open source model management system that offers services
for saving models and their associated metadata. It also includes services for
measuring the performance of machine learning models and querying models.
ModelHUB [12] is another platform for managing the lifecycle of deep learning
models that provides services for model storage and keeps track of associated
metadata and accuracy scores. Although these solutions facilitate the
communication between team members and support pipelines for creating and
managing machine learning workflows, however such solutions does not cross the
organization borders and it is not easy to share the artifacts in a reliable
and trustworthy way.

Another project in this context is TensorFlow Extended (TFX) [2] which a
solution built around is the Google’s TensorFlow, an end-to-end open source
platform for machine learning. By integrating various machine learning
artifacts into one platform, it provides a standardize way to manage and
access machine learning assets and artifacts which facilitates the platform
configuration, and reduce the time to production significantly, while
providing platform stability.

Michelangelo [7] platform is the Uber’s solution for building, deploying, and
operating machine learning workflows in a collaborative environment and covers
areas such as data management, model training, and model deployment,
evaluation, and monitoring. An integral part of the Michelangelo platform is a
sub-system called Gallery [14] that manages the lifecycle of machine learning
models by orchestrating the flow of models across different stages in the
lifecycle.

There are also other solutions that are more focused on data pipelines and
feature management aspects. For instance, Hopsworks Feature Store [9] provides
an open data management ecosystem for machine learning features, including the
feature engineering code and the feature data.

##  6 Conclusions and Future Work

Increasing access to machine learning artifacts held by various organizations
will accelerate the uptake of use cases and solutions for smart environments
such as smart home and smart buildings. Due to increasing number of IoT-
enabled environments and the costs associated with machine learning processes,
sharing and reuse of machine learning assets and artifacts beyond organization
borders are plausible solutions. In this paper, we proposed a decentralized
and adaptive software framework based on blockchain and knowledge graph
technologies that supports the knowledge exchange and interoperability between
smart environments, in a transparent and trustworthy way. The proposed
architecture can be used in ways that increase communication between smart
environments, enable sustainable growth of IoT solutions, and drive sharing of
cognitive knowledge via transfer learning. In this context, self-explaining
and machine-understandable digital assets (e.g., features and models) are the
key requirements to enable transfer learning beyond the organizational
borders. Also, blockchain technology and smart contracts foster transparent
and trustworthy sharing of machine learning assets and advance cross-
organization communication between smart environments.

##  7 Acknowledgement

This work was supported with the financial support of the Science Foundation
Ireland grant 13/RC/2094 and co-funded under the European Regional Development
Fund through the Southern & Eastern Regional Operational Programme to Lero -
the Irish Software Research Centre (www.lero.ie).

## References

  * [1] Anjomshoaa, A., Curry, E.: Transfer learning in smart environments. Machine Learning and Knowledge Extraction 3(2), 318–332 (2021)
  * [2] Baylor, D., Breck, E., Cheng, H.T., Fiedel, N., Foo, C.Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al.: Tfx: A tensorflow-based production-scale machine learning platform. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 1387–1395 (2017)
  * [3] Dannen, C.: Introducing Ethereum and solidity, vol. 318. Springer (2017)
  * [4] DML, D.: Decentralized machine learning protocol (dml), https://decentralizedml.com/, [Online; accessed 15-July-2021]
  * [5] GNY: Gny decentralized machine learning), https://www.gny.io/, [Online; accessed 15-July-2021]
  * [6] Harris, J.D., Waggoner, B.: Decentralized and collaborative ai on blockchain. In: 2019 IEEE International Conference on Blockchain (Blockchain). pp. 368–375. IEEE (2019)
  * [7] Hermann, J., Del Balso, M.: Meet michelangelo: Uber’s machine learning platform. URL https://eng. uber. com/michelangelo (2017)
  * [8] Justin, D., Harris, B.: Decentralized & collaborative ai on blockchain. In: Proceedings of the 2019 IEEE International Conference on Blockchain (Blockchain), Atlanta, GA, USA. pp. 14–17 (2019)
  * [9] Kakantousis, T., Kouzoupis, A., Buso, F., Berthou, G., Dowling, J., Haridi, S.: Horizontally scalable ml pipelines with a feature store. In: Proc. 2nd SysML Conf., Palo Alto, USA (2019)
  * [10] Kurtulmus, A.B., Daniel, K.: Trustless machine learning contracts; evaluating and exchanging machine learning models on the ethereum blockchain. arXiv preprint arXiv:1802.10185 (2018)
  * [11] Marathe, A., Narayanan, K., Gupta, A., Manoj, P.: Dinemmo: Decentralized incentivization for enterprise marketplace models. In: 2018 IEEE 25th International Conference on High Performance Computing Workshops (HiPCW). pp. 95–100. IEEE (2018)
  * [12] Miao, H., Li, A., Davis, L.S., Deshpande, A.: Modelhub: Deep learning lifecycle management. In: 2017 IEEE 33rd International Conference on Data Engineering (ICDE). pp. 1393–1394. IEEE (2017)
  * [13] Rathore, S., Pan, Y., Park, J.H.: Blockdeepnet: A blockchain-based secure deep learning for iot network. Sustainability 11(14), 3974 (Jul 2019). https://doi.org/10.3390/su11143974, http://dx.doi.org/10.3390/su11143974
  * [14] Sun, C., Azari, N., Turakhia, C.: Gallery: A machine learning model management system at uber. In: EDBT. pp. 474–485 (2020)
  * [15] Tanwar, S., Bhatia, Q., Patel, P., Kumari, A., Singh, P.K., Hong, W.C.: Machine learning adoption in blockchain-based smart applications: The challenges, and a way forward. IEEE Access 8, 474–488 (2019)
  * [16] Vartak, M., Subramanyam, H., Lee, W.E., Viswanathan, S., Husnoo, S., Madden, S., Zaharia, M.: Modeldb: a system for machine learning model management. In: Proceedings of the Workshop on Human-In-the-Loop Data Analytics. pp. 1–3 (2016)
  * [17] Xiong, W., Xiong, L.: Smart contract based data trading mode using blockchain and machine learning. IEEE Access 7, 102331–102344 (2019)


[datatype=bibtex, overwrite=true] [fieldsource=booktitle,
match=.*Interspeech.*, replace=Proc. Interspeech] [fieldsource=journal,
match=.*INTERSPEECH.*, replace=Proc. Interspeech] [fieldsource=booktitle,
match=.*ICASSP.*, replace=Proc. ICASSP] [fieldsource=booktitle,
match=.*icassp_inpress.*, replace=Proc. ICASSP (in press)]
[fieldsource=booktitle, match=.*Acoustics,.*Speech.*and.*Signal.*Processing.*,
replace=Proc. ICASSP] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Learning.*Representations.*,
replace=Proc. ICLR] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Computational.*Linguistics.*,
replace=Proc. COLING] [fieldsource=booktitle,
match=.*SIGdial.*Meeting.*on.*Discourse.*and.*Dialogue.*, replace=Proc.
SIGDIAL] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Machine.*Learning.*, replace=Proc.
ICML] [fieldsource=booktitle,
match=.*North.*American.*Chapter.*of.*the.*Association.*for.*Computational.*Linguistics:.*Human.*Language.*Technologies.*,
replace=Proc. NAACL] [fieldsource=booktitle,
match=.*Empirical.*Methods.*in.*Natural.*Language.*Processing.*, replace=Proc.
EMNLP] [fieldsource=booktitle,
match=.*Association.*for.*Computational.*Linguistics.*, replace=Proc. ACL]
[fieldsource=booktitle,
match=.*Automatic.*Speech.*Recognition.*and.*Understanding.*, replace=Proc.
ASRU] [fieldsource=booktitle, match=.*Spoken.*Language.*Technology.*,
replace=Proc. SLT] [fieldsource=booktitle,
match=.*Speech.*Synthesis.*Workshop.*, replace=Proc. SSW]
[fieldsource=booktitle, match=.*workshop.*on.*speech.*synthesis.*,
replace=Proc. SSW] [fieldsource=booktitle,
match=.*Advances.*in.*neural.*information.*processing.*, replace=Proc.
NeurIPS] [fieldsource=booktitle,
match=.*Advances.*in.*Neural.*Information.*Processing.*, replace=Proc.
NeurIPS] [fieldsource=booktitle, match=.*Workshop.*on.* Applications.* of.*
Signal.*Processing.*to.*Audio.*and.*Acoustics.*, replace=Proc. WASPAA]
[fieldsource=booktitle, match=.*International.*Spoken.*Lang.*Tran.*,
replace=Proc. IWSLT] [fieldsource=booktitle,
match=.*conf.*empirical.*natural.*language.*, replace=Proc. EMNLP]
[fieldsource=publisher, match=.+, replace=] [fieldsource=month, match=.+,
replace=] [fieldsource=location, match=.+, replace=] [fieldsource=address,
match=.+, replace=] [fieldsource=organization, match=.+, replace=]
captionUnknown document class Charles University, Czechia

#  Long-Form End-to-End Speech Translation via Latent Alignment Segmentation

###### Abstract

Current simultaneous speech translation models can process audio only up to a
few seconds long. Contemporary datasets provide an oracle segmentation into
sentences based on human-annotated transcripts and translations. However, the
segmentation into sentences is not available in the real world. Current speech
segmentation approaches either offer poor segmentation quality or have to
trade latency for quality. In this paper, we propose a novel segmentation
approach for a low-latency end-to-end speech translation. We leverage the
existing speech translation encoder-decoder architecture with ST CTC and show
that it can perform the segmentation task without supervision or additional
parameters. To the best of our knowledge, our method is the first that allows
an actual end-to-end simultaneous speech translation, as the same model is
used for translation and segmentation at the same time. On a diverse set of
language pairs and in- and out-of-domain data, we show that the proposed
approach achieves state-of-the-art quality at no additional computational
cost.

Index Terms—  segmentation, long-form, simultaneous, speech translation,
latent alignment

##  1 Introduction

Simultaneous speech translation (SST) is the task of translating speech in one
language into target-language text before the speaker finishes the utterance.
Traditionally, SST has relied predominantly on cascaded systems that decompose
the task into multiple subtasks, including automatic speech recognition (ASR),
punctuation restoration (PR), and machine translation (MT) [1, 2, 3]. However,
recent advancements in deep learning and the availability of abundant training
data [4, 5] have led to a significant paradigm shift towards end-to-end (E2E)
models. Despite the recent popularity of end-to-end SST within the research
community, most research focuses on the “short-form” setting, which assumes
that the speech input is already pre-segmented into sentences. Critically,
this assumption poses an obstacle to deployment in the “wild”, where speeches
consist of several sentences — a “long-form” regime.

In the traditional cascaded approach, most segmentation methods relied on
punctuation predicted by the inverse text normalization [6, 7, 8]. However,
such an approach is impossible in the end-to-end models, as the intermediate
transcript is unavailable. The E2E approach must, therefore, rely on speech-
based segmentation methods. Typical choices are fixed-based segmentation,
i.e., segmentation into chunks of equal length, or paused-based methods based
on voice activity detection (VAD) [9, 10]. However, these segmentation
approaches harm the resulting translation quality, as the translation task is
sensitive to poor segmentation and generally prefers a segmentation obeying
sentence boundaries [10]. Recent work [11, 12] tries to predict sentence
boundaries directly. However, their use in the simultaneous regime imposes
further translation delay and requires additional computational resources.

This paper proposes a novel segmentation approach that leverages a popular
attention-based encoder-decoder architecture with ST CTC loss [13, 14]. We
perform the sentence segmentation on the fly using the punctuation from the
translation and speech-to-translation alignment from ST CTC. Without any
external segmentation model, we show that models trained for translation only
can also be used for segmentation as well. In extensive experiments on TED
talks and parliamentary speeches, we show that:

  * •

Translation models can segment speech based on the punctuation included in the
translation without any special or additional training.

  * •

Provided segmentation quality is equivalent to or better than the current
state-of-the-art segmentation methods based on large pre-trained models.

  * •

The proposed approach does not introduce any additional latency and does not
need any additional computational resources.

##  2 Background

This section introduces the most essential concepts of long-form simultaneous
speech translation.

### Incremental vs. Re-Translation SST

SST models can be either re-translation or incremental. Re-translation models
[15, 16] typically run their decoding every time they get a new portion of the
speech. Critically, a _re-translation model can revise its translation_ output
as more speech input is read. This design arguably makes it more difficult for
the user to process the translation. On the other hand, because the model can
revisit its translations, the final translation quality matches the offline
translation quality.

Incremental models [17, 18] differ from re-translation models in that they can
only append new words to the end of the partial translation but never change
the previous words. For the user, the _translation changes only by
incrementally getting longer_ ; none of the previously displayed outputs are
ever modified. The incremental approach is required for certain applications
(e.g., speech-to-speech translation) and can be considered easier to follow
from the user’s perspective. From the long-form perspective, re-translation
allows for a substantially lower latency: Imagine that punctuation prediction
needs a 5-second look-ahead buffer for reliable work. In a re-translation
approach, we can emit the expected translation of the 5 seconds, later fixing
any punctuation mistakes. The incremental approach has to be much more
conservative and delay any output until the punctuation is certain because it
has no option to correct itself. In this work, we follow the incremental
approach.111IWSLT shared tasks [19, 20, 21, 22] also follow the incremental
SST approach.

### Audio Segmentation Methods

The simplest audio segmentation method, fixed-length segmentation, splits
audio based on length while disregarding any information contained in the
audio. More advanced strategies rely on acoustic information, typically voice
activity detection (VAD). VAD concentrates solely on the presence of the
speech and disregards sentence boundaries. This usually results in sub-optimal
segmentation [23, 11, 12] as humans place pauses inside sentences, not
necessarily between them (e.g., hesitations before words with high information
content, [24]). To address this, SHAS segmentation classifier [11] is directly
trained to segment audio into sentences. The model consists of a robust pre-
trained multi-lingual model XLS-R [25], an extra Transformer [26] layer and a
classification layer. For each speech frame, SHAS outputs the probability of
whether it should be included in the segment.

To improve the quality of the VAD-based methods, offline divide-and-conquer
(DAC) [27] and simultaneous (SIM) [23] consider the presence of speech and
also the length of the resulting segments. DAC method recursively splits the
audio on the longest pause until all segments are shorter than some pre-
defined maximum length. The SIM method allows simultaneous segmentation (i.e.,
without seeing the entire recording) by segmenting on the longest pause
between minimum and maximum segment length. If no pause is detected, the
segmentation occurs on the maximum length.

### Simultaneous Speech Translation Models with Latent Alignments

A popular architecture for modeling speech translation is the attention-based
encoder-decoder (AED) architecture. AED’s advantage is the powerful cross-
attention mechanism [28, 29] that allows the decoder to “attend” any portion
of the source. While having overall good performance, AED models tend to
hallucinate, especially in the low-latency regime [21, 30, 31]. To remedy
this, an auxiliary CTC [32] directly predicting translation (ST CTC), was
explored [13, 14]. ST CTC provides extra regularization during training,
resulting in faster and better convergence. The ST CTC output can also be used
during decoding to re-score the hypotheses produced by the AED decoder
[33].222Other authors use CTC with source language transcriptions, i.e., ASR
CTC. However, ASR CTC cannot be used to improve the translation quality during
the inference. We note that, unlike AED, CTC does not use cross-attention to
attend the entire source speech and instead directly classifies each source-
speech frame with a translation token or blank (i.e., no translation). Since
each speech frame is classified with a translation or blank, this can be seen
as an explicit latent alignment between the source speech and target
translation. Any word reordering needed between the source and target
languages in ST CTC happens in the encoding phase at the level of speech
frames, leading to a worse quality of ST CTC alone.

##  3 Method

Our method aims to provide segmentation of the source sound by relying on the
punctuation that was automatically created on the target side by the speech-
to-text model. We start from ST CTC, which classifies each source speech frame
with target translation, including punctuation symbols. The ST CTC output thus
directly links target-side punctuation to time positions in the source.
However, we must consider that the ST CTC translations are typically worse
than the AED translations (e.g., [14] report an average translation quality
difference of 4 BLEU points). Also, the latent alignments of [14] are a mere
modeling tool rather than a goal product. We therefore ask two questions: Q1:
Are the latent alignments reliable? Q2: Are the ST CTC punctuation predictions
good enough? To answer these questions, we propose the following two simple
methods:

### Greedy Approach

The first approach, the “greedy” approach, relies solely on the ST CTC
predictions. For each speech frame, the greedy approach takes the translation
label with the highest probability and looks if the label is a sentence
punctuation symbol (i.e., “. ! ?”). If so, the frame is labeled as a segment
boundary. First, the translation of the current segment is finalized using the
standard incremental beam search, and a new sentence is started. The approach
is summarized in Algorithm 1.

### Align Approach

As pointed out, the ST CTC translations are typically worse than the AED
translations. Hence, the second approach, dubbed “align”, uses the AED
predictions, and the ST CTC is used only for the alignment. Specifically, the
SST model provides a simultaneous translation using the standard incremental
beam search. Once a sentence punctuation symbol (i.e., “. ! ?”) is detected in
the translation, we use ST CTC to find the alignment of the punctuation in the
source speech. Because we assume that the incremental beam search uses CTC re-
scoring that computes CTC prefix probabilities [34], we extract the alignment
as the frame with the highest CTC prefix probability, where the prefix is the
generated sentence, including the sentence punctuation. This way, we obtain
the alignment with one pass over the source frames. For technical details on
efficient implementation of the CTC prefix probability, follow [33]. The align
approach is summarized in Algorithm 2.

Input : Streaming speech (split to small blocks), ST model (encoder, ctc,
decoder)

Output : Partial hypotheses

1 for  _each streaming speech block $B$_ do

2 $H$ $\leftarrow$ encoder($B$)

$L\leftarrow$ ctc($H$)

$\triangleright$ CTC lattice; time$\times$(vocab + 1); +1 for blank

$\triangleright$ last frame $t^{\text{seg}}$ such that it’s greedy label $v$
is a punctuation

3
$t^{\text{seg}}\leftarrow\max_{t}\\{t\mid(\arg\max_{v}L_{t,v})\in\\{\text{``.''},\text{``!''},\text{``?''}\\}\\}$

4 if  _$t^{\text{seg}}\neq\emptyset$ and $t^{\text{seg}}\geq$ min_len_ then

5 $H\leftarrow H_{1:t^{\text{seg}}}$

6 prepend $B_{t^{\text{seg}}:|B|}$ to next segment

7

8 return incremental-beam-search($H,L$)

9

List of algs 1 Proposed greedy segmentation approach.

|  | MuST-C (in-domain) | Europarl-ST (out-of-domain)
---|---|---|---
Type | Segm. method | EN$\rightarrow$DE | EN$\rightarrow$FR | EN$\rightarrow$RU | EN$\rightarrow$ZH | EN$\rightarrow$DE | EN$\rightarrow$FR
| BLEU$\uparrow$ | LAAL$\downarrow$ | BLEU$\uparrow$ | LAAL$\downarrow$ | BLEU$\uparrow$ | LAAL$\downarrow$ | BLEU$\uparrow$ | LAAL$\downarrow$ | BLEU$\uparrow$ | LAAL$\downarrow$ | BLEU$\uparrow$ | LAAL$\downarrow$
| Oracle | 25.4 | 1750 | 33.6 | 2091 | 16.2 | 1819 | 21.0 | 1858 | 17.5 | 2043 | 15.8 | 2691
Offline | SHAS+DAC | 24.8 | 1421 | 32.4 | 2273 | 16.0 | 1466 | 20.8 | 1248 | 16.8 | 1450 | 15.1 | 2177
High latency | SHAS+SIM-Q | 25.0 | 5378 | 33.8 | 5733 | 16.0 | 2701 | 20.9 | 3295 | 16.9 | 4833 | 16.4 | 5134
| Fixed-length | 22.8 | 1339 | 31.3 | 3207 | 14.7 | 1418 | 19.6 | 1092 | 14.0 | 392 | 12.2 | 1952
| SHAS+SIM-L | 23.6 | 1582 | 31.3 | 2411 | 15.5 | 1687 | 20.4 | 1581 | 16.1 | 1622 | 14.9 | 2661
| Greedy (ours) |  24.2 | 1533 |  31.9 | 2421 |  16.0 | 1648 |  20.8 | 1553 |  16.7 | 1612 | 15.1 | 2506
Low latency | Align (ours) | 24.0 | 1547 | 31.7 | 2423 |  15.9 | 1638 |  20.9 | 1568 |  16.8 | 1614 | 14.9 | 2529

Table 1: Systems better than the other low-latency baselines in bold.
Underlined and dotted-underlined scores are significantly different from other
low-latency baselines with $p\text{-value}<0.01$ and $p\text{-value}<0.05$,
respectively. Offline segmentation methods have only theoretical latency, as
the segmentation is done offline before the translation. The latency LAAL is
in milliseconds.

##  4 Experimental Setup

### Data

In our experiments, we use the English $\rightarrow$ German, English
$\rightarrow$ French, English $\rightarrow$ Chinese, and English $\rightarrow$
Russian language pairs of the MuST-C [35] data set. We use the training and
validation sets during the training and tuning of the hyper-parameters for the
segmentation algorithms. Finally, we use the tst-COMMON split to report the
final results. Additionally, we use the test split of Europarl-ST [36] to
report out-of-domain results.

### Models

All models are attention-based encoder-decoder models. To accommodate the
simultaneous regime, we adopt a blockwise encoder [37], but any unidirectional
encoder would work. We pre-process the audio with 80-dimensional filter banks.
We build a unigram [38] vocabulary with a size of 4000 for all language pairs.
All models use a block size of 40 (1.6 s). The encoder has 12 layers, and the
decoder has six layers. The model dimension is 256, and the feed-forward
dimension is 2048 with four attention heads. To improve the training speed, we
initialize the encoder with weights pre-trained on the ASR task of the MuST-C
dataset. Further, we employ ST CTC [13, 14] after the encoder with weight 0.3
during training and decoding. As a regularization, we use speed perturbation
(at 0.9, 1.0, and 1.1 speeds), and to improve the long-form performance, we
also include concatenation of two consecutive segments from the training data.
Finally, we use checkpoint averaging for the last ten epochs. We use the
ESPNet-ST toolkit [39]

Input : Streaming Speech (split to small blocks), ST model (encoder, ctc,
decoder)

Output : A set of partial hypotheses and scores

10 for  _each streaming speech block $B$_ do

11 $H$ $\leftarrow$ encoder($B$)

$L\leftarrow$ ctc($H$)

$\triangleright$ CTC lattice; time$\times$(vocab + 1); +1 for blank

12 $Y\leftarrow$ incremental-beam-search($H,L$)

$\triangleright$ index $l^{\text{seg}}$ of last label that is a punctuation

13 $l^{\text{seg}}\leftarrow\max_{l}\\{l\mid
Y_{l}\in\\{\text{``.''},\text{``!''},\text{``?''}\\}\\}$

14

15 if  _$l^{\text{seg}}\neq\emptyset$_ then

$\triangleright$ time $t$ with maximal CTC prefix probability of
$Y_{1:l^{\text{seg}}}$

16 $b^{\text{seg}}\leftarrow\arg\max_{t}$ ctc-prefix-
prob($Y_{1:l^{\text{seg}}},L_{1:t}$)

17 if  _$b^{\text{seg}}\geq$ min_len_ then

18 $Y\leftarrow H_{1:l^{\text{seg}}}$

19 prepend $B_{b^{\text{seg}}+1:|B|}$ to next segment

20

21 return Y

22

List of algs 2 Proposed alignment segmentation.

### Evaluation

All models are evaluated using Simuleval [40] toolkit. We adopt incremental
blockwise decoding [37, 31] with CTC incremental policy [30]. In all our
experiments, we use beam search with size 6. For the long-form evaluation, we
adopt the evaluation protocol suggested by [41]: instead of reporting quality
and latency on the document level, we align the hypothesis to the reference
using a re-implementation of mwerSegmenter333For Chinese, we align on the
character level instead of word level. We also tokenize the inputs before the
alignment process. [42], followed by re-segmentation into sentences based on
the reference punctuation. The quality and latency metrics are then computed
on the re-segmented utterances. For the translation quality, we report
detokenized case-sensitive BLEU [43], and for the latency, we report length-
aware average lagging (LAAL) [44, 45].

### Baselines

We use the development set to tune all hyper-parameters of the baselines. We
tuned all parameters for each language pair separately. Fixed-length
segmentation is tuned on interval (4, 34) seconds (s). For SHAS+DAC, we tune
the maximum length between 4 and 72 s. Both proposed methods and SHAS-SIM have
the minimum length between 2 and 32 s. The maximum length for SHAS+SIM was
tuned relative to the minimum length on interval (1, 7) s. Because this
interval influences the quality-latency tradeoff, we tune one system for
latency (denoted SHAS+SIM-L) and another for quality (SHAS+SIM-Q). We found
the value of approx. 2.5 s as best for SHAS+SIM-L and 7 s for SHAS+SIM-Q.

##  5 Results

We present the result in Table 1. On in- and out-of-domain data, both proposed
methods (greedy and align) outperform all low-latency baselines (fixed-length
and SHAS+SIM-L) except for out-of-domain English-to-French, where the proposed
align ties with SHAS+SIM-L. On average, the proposed align approach
outperforms fixed-length by 1.6 BLEU and SHAS+SIM-L by 0.4 BLEU, and the
proposed greedy approach outperforms fixed-length by 1.7 BLEU and SHAS+SIM-L
by 0.5 BLEU across all language pairs. This answers our question Q1 — the
latent alignments are reliable for the segmentation task. We attribute the
worse quality of SHAS+SIM-L compared to the proposed methods to the SIM
algorithm that forces the segmentation between minimum and maximum length.
I.e., when the SHAS model does not detect any sentence boundary in this
interval, SIM segments on the maximum length. In the low-latency SHAS+SIM-L,
this interval is approx. 2.5 s. Considering that the average sentence length
in the MuST-C test set is 5.8 s, this inevitably leads to incorrect
segmentation of some sentences. On the English-to-German MuST-C test set, this
occurred 203 times out of 941 segments predicted by SHAS+SIM-L in 4.7 hours,
i.e., 0.6 forced sentence segmentations per minute.

Unsurprisingly, the offline SHAS+DAC performs better than the low-latency
systems. However, on average, the proposed low-latency greedy is only 0.2 BLEU
worse than the offline SHAS+DAC. Interestingly, the high-latency SHAS+SIM-Q is
better than the offline SHAS+DAC. This is probably due to the considerable
delay introduced by the 7-second interval in the SHAS+SIM-Q. Since the
translation model has to wait for 7 s, a large portion of each sentence is
translated in an offline regime.

Counterintuitively, the greedy approach outperforms the align approach
slightly (only 0.1-0.2 BLEU). Because the CTC translation quality is worse
than that of the AED [14], we would expect the align approach to reach a
better quality. A possible answer might be a mismatch between the ST CTC and
AED predictions that leads to a slightly poorer alignment. This answers our
question Q2 — the ST CTC punctuation predictions are suitable for the
segmentation.

Segm. method | Segm. param.$\downarrow$ | Total param.$\downarrow$ | RTF$\downarrow$ | LAAL$\downarrow$ | BLEU$\uparrow$
---|---|---|---|---|---
Fixed-length | 0 | 45 M | 0.46 | 1339 | 22.8
SHAS+SIM-L | 208 M | 253 M | 0.61 | 1582 | 23.6
Greedy (ours) | 0 | 45 M | 0.42 | 1533 | 24.2
Align (ours) | 0 | 45 M | 0.41 | 1547 | 24.0

Table 2: Performance comparison of low latency segmentation methods on
English-to-German MuST-C test set. Real-time factor (RTF) measured on Intel
i7-10700 using a single thread. Better values in bold. Total param. is the
total number of parameters, including the translation model.

In Table 2, we compare the computational complexity of the low latency
systems. The proposed segmentation methods, like the fixed-length method, do
not introduce new segmentation parameters. The proposed methods have about 30
% lower real-time factor (RTF) than the SHAS+SIM-L, as they do not have to
evaluate the additional segmentation model. Interestingly, the fixed-length
method has a slightly higher RTF. The probable cause is the quadratic
complexity of the AED decoder and the length of an average segment proposed by
the segmentation methods: the fixed-length method uses 20 s (was found to
maximize the translation quality on the development set) and the proposed
align method produces segments of an average length of 8.5 s.

##  6 Conclusion

In this paper, we presented two simple speech segmentation methods introducing
new state-of-the-art performance to simultaneous speech segmentation. A
thorough evaluation on in- and out-of-domain data shows that the proposed
methods offer the best quality with the same latency and have the smallest
computational footprint. To the best of our knowledge, our methods are the
first that allow an actual end-to-end simultaneous speech translation, as they
use the translation model for the joint translation and segmentation without
explicitly modeling the segmentation. In future research, we will explore the
properties of latent alignments, including latent alignments from other
architectures.

##  7 REFERENCES

## References

  * [1] L. Osterholtz et al.  “Testing generality in JANUS: a multi-lingual speech translation system”  In _Proc. ICASSP_ 1, 1992, pp. 209–212 vol.1  DOI: 10.1109/ICASSP.1992.225935
  * [2] Christian Fügen, Alex Waibel and Muntsin Kolss  “Simultaneous translation of lectures and speeches”  In _Machine translation_ 21, 2007, pp. 209–252
  * [3] Ondřej Bojar et al.  “ELITR Multilingual Live Subtitling: Demo and Strategy”  In _Proc. ACL_ , 2021, pp. 271–277  DOI: 10.18653/v1/2021.eacl-demos.32
  * [4] Kar-Han Tan and Boon Pang Lim  “The artificial intelligence renaissance: deep learning and the road to human-level machine intelligence”  In _APSIPA Transactions on Signal and Information Processing_ 7, 2018, pp. e6
  * [5] Matthias Sperber and Matthias Paulik  “Speech Translation and the End-to-End Promise: Taking Stock of Where We Are”  In _Proc. ACL_ , 2020, pp. 7409–7421  DOI: 10.18653/v1/2020.acl-main.661
  * [6] Wei Lu and Hwee Tou Ng  “Better punctuation prediction with dynamic conditional random fields”  In _Proc. EMNLP_ , 2010, pp. 177–186
  * [7] Eunah Cho et al.  “Punctuation insertion for real-time spoken language translation”  In _Proc. IWSLT_ , 2015
  * [8] Eunah Cho, Jan Niehues and Alex Waibel  “NMT-Based Segmentation and Punctuation Insertion for Real-Time Spoken Language Translation.”  In _Proc. Interspeech_ , 2017, pp. 2645–2649
  * [9] Jongseo Sohn, Nam Soo Kim and Wonyong Sung  “A statistical model-based voice activity detection”  In _IEEE signal processing letters_ 6.1, 1999, pp. 1–3
  * [10] Mark Sinclair et al.  “A semi-markov model for speech segmentation with an utterance-break prior”  In _Fifteenth Annual Conference of the International Speech Communication Association_ , 2014
  * [11] Ioannis Tsiamas et al.  “SHAS: Approaching optimal Segmentation for End-to-End Speech Translation”  In _arXiv preprint arXiv:2202.04774_ , 2022
  * [12] Ryo Fukuda, Katsuhito Sudoh and Satoshi Nakamura  “Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translation”  In _arXiv preprint arXiv:2203.15479_ , 2022
  * [13] Keqi Deng et al.  “Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation”  In _arXiv preprint arXiv:2204.08920_ , 2022
  * [14] Brian Yan et al.  “Ctc alignments improve autoregressive translation”  In _arXiv preprint arXiv:2210.05200_ , 2022
  * [15] Jan Niehues et al.  “Dynamic Transcription for Low-Latency Speech Translation”  In _Proc. Interspeech_ , 2016, pp. 2513–2517  DOI: 10.21437/Interspeech.2016-154
  * [16] J. Niehues et al.  “Low-latency neural speech translation”  In _Proc. Interspeech_ , 2018, pp. 1293–1297  DOI: 10.21437/Interspeech.2018-1055
  * [17] Kyunghyun Cho and Masha Esipova  “Can neural machine translation do simultaneous translation?”  In _arXiv preprint arXiv:1606.02012_ , 2016
  * [18] Fahim Dalvi et al.  “Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation”  In _Proc. ACL_ , 2018, pp. 493–499  DOI: 10.18653/v1/N18-2079
  * [19] Ebrahim Ansari et al.  “FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN”  In _Proc. IWSLT_ , 2020, pp. 1–34  DOI: 10.18653/v1/2020.iwslt-1.1
  * [20] Antonios Anastasopoulos et al.  “FINDINGS OF THE IWSLT 2021 EVALUATION CAMPAIGN”  In _Proc. IWSLT_ , 2021, pp. 1–29  DOI: 10.18653/v1/2021.iwslt-1.1
  * [21] Antonios Anastasopoulos et al.  “FINDINGS OF THE IWSLT 2022 EVALUATION CAMPAIGN”  In _Proc. IWSLT_ , 2022
  * [22] Milind Agarwal et al.  “FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN”  In _Proc. IWSLT_ , 2023, pp. 1–61  URL: https://aclanthology.org/2023.iwslt-1.1
  * [23] Marco Gaido et al.  “Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation”  In _Proceedings of The Fourth International Conference on Natural Language and Speech Processing (ICNLSP 2021)_ , 2021, pp. 55–62
  * [24] Frieda Goldman-Eisler  “Speech production and the predictability of words in context”  In _Quarterly Journal of Experimental Psychology_ 10.2, 1958, pp. 96–106
  * [25] Arun Babu et al.  “XLS-R: Self-supervised cross-lingual speech representation learning at scale”  In _arXiv preprint arXiv:2111.09296_ , 2021
  * [26] Ashish Vaswani et al.  “Attention is all you need”  In _Proc. NeurIPS_ , 2017, pp. 5998–6008
  * [27] Tomasz Potapczyk and Pawel Przybysz  “SRPOL’s System for the IWSLT 2020 End-to-End Speech Translation Task”  In _IWSLT 2020_ , 2020, pp. 89
  * [28] Ilya Sutskever, Oriol Vinyals and Quoc V Le  “Sequence to sequence learning with neural networks”  In _Advances in neural information processing systems_ 27, 2014
  * [29] Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio  “Neural machine translation by jointly learning to align and translate”  In _arXiv preprint arXiv:1409.0473_ , 2014
  * [30] Peter Polák et al.  “Towards Efficient Simultaneous Speech Translation: CUNI-KIT System for Simultaneous Track at IWSLT 2023”  In _Proc. IWSLT_ , 2023, pp. 389–396  DOI: 10.18653/v1/2023.iwslt-1.37
  * [31] Peter Polák et al.  “Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff”  In _Proc. INTERSPEECH 2023_ , 2023, pp. 3979–3983  DOI: 10.21437/Interspeech.2023-2225
  * [32] Alex Graves et al.  “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks”  In _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 369–376
  * [33] Shinji Watanabe et al.  “Hybrid CTC/attention architecture for end-to-end speech recognition”  In _IEEE Journal of Selected Topics in Signal Processing_ 11.8, 2017, pp. 1240–1253
  * [34] Alex Graves  “Supervised sequence labelling with recurrent neural networks.”, 2008
  * [35] Roldano Cattoni et al.  “MuST-C: A multilingual corpus for end-to-end speech translation”  In _Computer Speech & Language_ 66, 2021, pp. 101155  DOI: https://doi.org/10.1016/j.csl.2020.101155
  * [36] Javier Iranzo-Sánchez et al.  “Europarl-st: A multilingual corpus for speech translation of parliamentary debates”  In _Proc. ICASSP_ , 2020, pp. 8229–8233
  * [37] Emiru Tsunoo, Yosuke Kashiwagi and Shinji Watanabe  “Streaming transformer asr with blockwise synchronous beam search”  In _Proc. SLT_ , 2021, pp. 22–29
  * [38] Taku Kudo  “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates”  In _Proc. ACL_ , 2018, pp. 66–75  DOI: 10.18653/v1/P18-1007
  * [39] Brian Yan et al.  “ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit”  In _arXiv preprint arXiv:2304.04596_ , 2023
  * [40] Xutai Ma et al.  “SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation”  In _Proc. EMNLP_ , 2020, pp. 144–150
  * [41] Javier Iranzo-Sánchez, Jorge Civera Saiz and Alfons Juan  “Stream-level Latency Evaluation for Simultaneous Machine Translation”  In _Proc. ACL_ , 2021, pp. 664–670
  * [42] Evgeny Matusov et al.  “Evaluating machine translation output with automatic sentence segmentation”  In _Proc. IWSLT_ , 2005
  * [43] Matt Post  “A Call for Clarity in Reporting BLEU Scores”  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , 2018, pp. 186–191
  * [44] Peter Polák et al.  “CUNI-KIT System for Smultaneous Speech Translation Task at IWSLT 2022”  In _Proc. IWSLT_ , 2022
  * [45] Sara Papi et al.  “Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation”  In _Proceedings of the Third Workshop on Automatic Simultaneous Translation_ , 2022, pp. 12–17  DOI: 10.18653/v1/2022.autosimtrans-1.2