# Review of Technologies for Ion Therapy Accelerators

H. X. Q. Norman1,2<EMAIL_ADDRESS>A. F.
Steinberg1,2<EMAIL_ADDRESS>R. B. Appleby1
University of Manchester  Manchester  UK
H. L. Owen1 STFC Daresbury  Warrington  UK
E. Benedetto  M. Sapinski  SEEIIST Association  Geneva  Switzerland
S. L. Sheehy  University of Melbourne  Victoria  Australia
1also at Cockcroft Institute, Warrington, UK  2also at University of
Melbourne, Victoria, Australia

###### Abstract

Cancer therapy using protons and heavier ions such as carbon has demonstrated
advantages over other radiotherapy treatments. To bring about the next
generation of clinical facilities, the requirements are likely to reduce the
footprint, obtain beam intensities above $1\text{\times}{10}^{10}$ particles
per spill, and achieve faster extraction for more rapid, flexible treatment.
This review follows the technical development of ion therapy, discussing how
machine parameters have evolved, as well as trends emerging in technologies
for novel treatments such as FLASH. To conclude, the future prospects of ion
therapy accelerators are evaluated.

##  1 Introduction

In recent years, using hadrons for radiotherapy has become more widely
recognised for significant benefits with the sparing of healthy tissue [1].
Carbon ions can be more beneficial than light ions when treating radio-
resistant and deep-seated tumours, however the adoption of carbon ion therapy
is hampered by the footprint of the accelerator and gantry, which are
presently much larger than conventional radiotherapy devices [2]. This has
been the focus for improvement over the past 20+ years [3].

##  2 Current and Developing Beam Delivery Systems

There have been many accelerators built for hadron therapy. A diagram of the
progress of representative machines is shown in Fig. 1. The general trend is
that the size of machines has decreased, whereas the number of particles per
beam spill rises as is required for new treatments. In addition, details of
specific designs are given in the main text and briefly summarised in Table 1
(with further detail in the Appendix).

Figure 1: Progress in the design of ion therapy accelerators over time. Accelerators with relatively few particles per spill rely on technologies that allow for a higher repetition (cycle) rate to produce the same overall current, such as the use of static magnetic fields for PAMELA. Table 1: Summary of parameters of ion therapy machines. Some also offer proton therapy, detail not given here. Proposals that were never constructed are given in italics. For a full version with extended detail including citations, see Table 2. Name | Location | Active | Extraction | Circ. (m) | Main Tech. | Species | Extracted | Particles
---|---|---|---|---|---|---|---|---
|  | Years | Method |  |  |  | KE (MeV/u) | per Spill
HIMAC | Japan | 1994 - | Resonant |  $130$(*$2$ | Synchrotron | He |  $100$-$800$ | $1.2\text{\times}{10}^{10}$
|  |  |  | rings) |  | C |  $100$-$800$ | $2.0\text{\times}{10}^{9}$
GSI | Germany | 1998 - 2008 | Resonant | $216.7$ | Synchrotron | C |  $80$-$430$ | $1.0\text{\times}{10}^{8}$
HIBMC | Japan | 2003 - | Resonant | $94$ | Synchrotron | C |  $70$-$320$ | $2.0\text{\times}{10}^{9}$
iRCMS | Proposal | — | — | $60$ | Synchrotron | C |  $96$-$450$ | $2.7\text{\times}{10}^{7}$
HIT | Germany | 2009 - | RF-KO | $65$ | Synchrotron | He |  $51$-$221$ | $1.0\text{\times}{10}^{10}$
|  |  |  |  |  | C |  $88$-$430$ | $1.0\text{\times}{10}^{9}$
|  |  |  |  |  | O |  $103$-$430$ | $5.0\text{\times}{10}^{8}$
GHMC | Japan | 2009 - | RF-KO | $63$ | Synchrotron | C |  $140$-$400$ | $1.3\text{\times}{10}^{9}$
HIRFL-CSR | China | 2009 - | Fast: Kicker | $161$ | Synchrotron | C |  $100$-$430$ | $5.0\text{\times}{10}^{8}$
|  |  | Slow: Resonant |  |  |  |  |
PAMELA | Proposal | — | Kicker | $58$ | FFA | C |  $110$-$440$ | $5.0\text{\times}{10}^{6}$
CNAO | Italy | 2010 - | RF-KO | $77.6$ | Synchrotron | C |  $120$-$400$ | $1.5\text{\times}{10}^{9}$
MIT | Germany | 2015 - | RF-KO | $65$ | Synchrotron | C |  $85$-$430$ | $7.0\text{\times}{10}^{8}$
ARCHADE | Proposal | — | Deflector | $21$ | Cyclotron | C | $400$ | —
KHIMA | Proposal | — | Resonant | $75$ | Synchrotron | C |  $110$-$430$ | $1.4\text{\times}{10}^{9}$
HIMM Wuwei | China | 2019 - | RF-KO | $56.2$ | Synchrotron | C |  $120$-$400$ | $4.0\text{\times}{10}^{9}$
Quantum | Proposal | — | — | $28$ | Laser and | C |  $56$-$430$ | $1.0\text{\times}{10}^{8}$
Scalpel |  |  |  |  | Synchrotron |  |  |
NIMMS | Proposal | — | Fast: Kicker | NC: $76$ | Synchrotron | He |  $60$-$250$ | $8.2\text{\times}{10}^{10}$
|  |  | Slow: RF-KO | DBA: $55$ |  | C |  $100$-$430$ | $2.0\text{\times}{10}^{10}$
|  |  |  | SC: $27$ |  | O |  $100$-$430$ | $1.4\text{\times}{10}^{10}$

###  2.1 Synchrotrons

The design choice for most facilities is based on a synchrotron over a
cyclotron, as it is capable of acceleration of particles with higher magnetic
rigidity than protons, and the beam can be extracted over a wide range of
energies, avoiding losses due to energy modulation [3].

The first hadron therapy treatments were delivered at the Bevelac [4]. Hadron
therapy was also delivered at GSI from $1997$ to $2007$; treatment there was
taken over by HIT [5].

The first accelerator purpose-built for heavy ion therapy was the HIMAC at
NIRS in Japan, which began treatment in
$\mathrm{1}\mathrm{9}\mathrm{9}\mathrm{4}$ [6]. The synchrotron has two
separate, identical rings, each able to deliver ions from
$100$$-$$800\text{\,}\mathrm{M}\mathrm{eV}\mathrm{/}\mathrm{u}$. Several
similar synchrotrons were then developed to achieve the same clinical
requirements with a smaller machine and lower costs. One option was the PATRO
project, resulting in the HIBMC where carbon ion treatment started in $2003$
[7]. Both machines deliver to smaller energy ranges than HIMAC, reducing the
circumference. A more compact alternative at NIRS led to GHMC (12C only),
which used the same focusing structure but increased the number of dipoles to
three per cell [8]. The GHMC design has been the basis for the most recent
generation of Japanese ion therapy centres.

In Europe, HIT has offered full 3D raster scanning since treatments began in
2009, delivering protons, carbon and heavier ions [9]. This was supplemented
by the Proton-Ion Medical Machine Study (PIMMS), a collaboration to provide a
strong baseline for future cancer therapy synchrotrons [10]. The lattice uses
triplet focusing, where the dispersion-free straights are used for injection,
extraction, and acceleration by RF; the extraction method uses a betatron
core. Two machines were subsequently built from the PIMMS design at CNAO
(Italy) and MedAustron (Austria).

The successor to PIMMS, the Next Ion Medical Machine Study (NIMMS), is
currently ongoing, alongside the SEEIIST ion therapy centre proposal [11, 12].
The aim is to offer a range of ions with an order of magnitude more particles
per spill, while also being smaller than its predecessors. It is expected to
offer slow extraction with multiple energies per spill, and fast extraction at
microsecond timescales. This requires extensive R&D, with three main
proposals: a compact synchrotron using normal-conducting (NC) magnets; an even
smaller synchrotron using superconducting (SC) magnets; and a ‘full linac’
design. Of these, the linac option requires the most R&D [12]. The NC
synchrotron has two options: 1) a modification of the PIMMS design with
improved injection and extraction, or 2) a novel double bend achromat (DBA)
design with a smaller circumference [13]. The SC proposal [14] follows ideas
evolved at NIRS [15], using large-angle magnets with canted cosine theta (CCT)
geometry [16] and nested alternating gradient (AG) coils [17]. Although this
option is more compact, development is likely to take considerable time.

There has also been some interest in an ion Rapid Cycling Medical Synchrotron
(iRCMS), which allows for more rapid energy variation assuming single energy
extraction [18]. Other than the higher magnet ramp rate, the iRCMS would have
similar parameters to other synchrotron options.

###  2.2 Alternatives to Synchrotrons

Though synchrotrons are the current workhorse for heavy ion therapy, there are
alternatives that could be implemented. For example, a therapeutic heavy ion
cyclotron has been proposed at ARCHADE in France [19, 20], potentially
beginning treatments in $2023$. Developments in high gradient cavities have
made linacs more promising. As well as NIMMS, designs have also been presented
by AVO-ADAM [21](p+) and ANL (p+, 12C); the latter is still in its design
phase [22]. A ‘bent’ full-linac for carbon ions has also been designed [23].
CABOTO, an NC ‘cyclinac’, is a fast-cycling linac design, but has not yet been
built due to the requirement of multiple high frequency klystron systems [24].

The time-independent magnets of Fixed Field Accelerators (FFAs) allow for a
higher cycle rate, which could increase the number of particles delivered
without requiring more particles per spill. However, FFAs tend to be larger
than equivalent synchrotron counterparts, and magnet designs can become
complicated. Multiple proposals for hadron therapy FFAs exist: the most-
developed of these was PAMELA [25], which required two rings to accelerate
protons and carbon ions over their full energy ranges (see Table 1), but had a
much higher cycle rate (up to $1\text{\,}\mathrm{kHz}$) than equivalent
synchrotrons (<$1\text{\,}\mathrm{Hz}$). Although designs continue to advance,
none have been constructed for treatment, and R&D is required in areas such as
beam stability and extraction.

A laser-hybrid accelerator known as ‘Quantum Scalpel’ [26] is under
development by Japanese industries working with QST-NIRS. This proposes a
laser accelerator for low energies, and a SC synchrotron as the second
acceleration stage. In the first stage, a Petawatt laser is incident on a thin
target, producing an ion beam with low mean energy but broad energy spread
[27]. LhARA is a similar proposal [28].

###  2.3 Gantries

The gantry is one of the key components of therapeutic beam delivery systems.
It bends and focuses the beam in the plane perpendicular to a patient to
deliver a precise dose to the treatment volume [29, 2], with a typical
momentum acceptance of ($\pm\leavevmode\nobreak\ 0.5-1\leavevmode\nobreak\
\%$)$\Delta{p}/{p}$ [30]. The majority of proton gantries use NC magnets. As
carbon ions have three times the rigidity of protons, a suitable gantry’s size
and magnet weight increases dramatically; the first carbon ion gantry (at HIT)
weighs $600\text{\,}\mathrm{t}$ and measures $25\text{\,}\mathrm{m}$ [31],
making installation and integration a challenge. Though it can transport
fully-stripped carbon ions in the range
$\mathrm{4}\mathrm{8}$-$430\text{\,}\mathrm{M}\mathrm{eV}\mathrm{/}\mathrm{u}$
[32], an alternative solution is required to shrink the gantry and increase
momentum acceptance to deliver flexible, efficient treatment. One option is to
use SC magnets: HIMAC employs six $2.88\text{\,}\mathrm{T}$ combined-function
SC magnets, reducing gantry mass to $<$ $300\text{\,}\mathrm{t}$ and the
length to $13\text{\,}\mathrm{m}$ [15]; a developing design will incorporate
$5\text{\,}\mathrm{T}$ magnets, reducing the gantry radius to $<$
$5\text{\,}\mathrm{m}$ [15]. A recent design by TERA/CERN proposes a
$35\text{\,}\mathrm{t}$ gantry[14], utilising
$90\text{\,}\mathrm{\SIUnitSymbolDegree}$, $4\text{\,}\mathrm{T}$ AG-CCT
dipoles [33] to reduce the radius to $5\text{\,}\mathrm{m}$. These magnets are
similar to those used for the NIMMS SC synchrotron.

Another path is to modify the gantry configuration. An example is the
‘GaToroid’ (in its R&D phase), which replaces the rotating beam transfer line
with a large toroidal field, combined with a ‘vector’ (steering) magnet to
bend the beams from several directions towards the patient isocentre [34, 35].
One could also use high momentum acceptance FFA-style magnets [36], though no
such gantries have yet been constructed. A final solution may be a mounted
gantry, similar to one constructed by Mevion (p+) that rotates around the
patient isocentre [37].

##  3 Research Directions

The main targets of hadron therapy R&D are: cost and size reduction [3];
improved reliability; new treatment capability.

###  3.1 Lattice Design

To reduce the size of accelerators and gantries, the options are either to
change the structure and/or use SC technology. An example of the former is the
NIMMS baseline lattice, which could be shrunk by using a DBA. Further
reductions are limited by the necessity of including long drifts for
extraction, while also providing enough space for the required bending
elements. Conversely, the adoption of SC magnets uses similar beam optics, but
the larger fields allow for tighter bending radii, significantly reducing the
overall size.

###  3.2 Magnet Design

The use of SC magnets in accelerator and gantry design can increase the
maximum field strength. Using combined-function magnets also reduces the total
number of magnets. This can lead to cost reductions for the mechanical support
structures and construction, but must be evaluated against the price of the
cooling system and materials used for the magnets [3]. To provide the most
benefit, the system’s magnetic composition (SC or hybrid NC/SC), configuration
(e.g. CCT) and material (e.g. Nb-Ti) must be considered. CCT Nb-Ti magnets are
being developed for future machines [12].

###  3.3 Extraction Methods

Most ion therapy facilities now achieve slow extraction by excitation of a
third-order resonance in combination with RF-knockout (RF-KO), as it can be
used to extract multiple beam energies over the course of a single cycle [38].
This allows different energy layers to be delivered rapidly, however the
energy switching time is too long for treatments such as FLASH [39]. Some also
use a kicker for fast extraction, delivering the entire stored beam in a
single turn. The higher dose rates this provides may be useful for FLASH, but
will be insufficient if rapid energy variation is also required.

One consideration for future extraction methods is the possibility to deliver
multiple ions in a single treatment. This may be desirable to achieve a more
conformal dose than with a single species [40], or for novel imaging methods
such as particle tomography and range verification [41]. For multi-ion
treatments, the issue is similar to that of rapid energy switching, and it is
not yet clear what technologies will provide the required advances. A proposed
SC gantry may be capable of performing proton tomography, so long as it has
sufficiently large momentum acceptance [42].

###  3.4 Pre-Acceleration

Almost all current ion therapy accelerators use a linac to accelerate ions
before injection, although some facilities (in China) opt for a cyclotron
[43]. Future pre-accelerators may need to transmit larger currents and reach
higher energies to avoid space-charge effects in high intensity beams, such as
those required for FLASH therapy [39].

The main objectives for new pre-accelerator systems are to reduce cost and
size, while maintaining or increasing beam current. These depend on the ion
source used; many facilities use the Supernanogan [44], an ECR ion source,
although higher current options such as AISHa [45] or TwinEBIS [46] may come
into greater use. An EBIS can produce small emittance beams, allowing for
higher current through multi-turn injection, but cannot yet match the ECR in
reliability and intensity. The linac itself will likely move to higher
gradients, but the low duty cycle of injectors for medical synchrotrons makes
SC options less favourable [12].

Novel ion production and pre-acceleration methods may eventually take over
from linacs. In particular, laser-accelerator methods discussed earlier may
provide beams of sufficient energy and intensity for clinical use. However,
these technologies require further development; improvements on traditional
methods will likely be used for the next generation of ion therapy machines.

## References

  * [1] Wayne D Newhauser  “The physics of proton therapy”  In _Physics in Medicine & Biology_ 60.8 IOP Publishing, 2015
  * [2] “Ion Beam Therapy”  Springer Berlin Heidelberg, 2012
  * [3] Alexander Gerbershagen  “The advantages and challenges of superconducting magnets in particle therapy”  In _Superconductor Science and Technology_ 29 Institute of Physics Publishing, 2016
  * [4] Joseph R Castro  “Heavy ion therapy: BEVALAC epoch”, 1993
  * [5] D Schardt  “Tumor Therapy with Heavy Ions at GSI Darmstadt”  URL: https://www-pub.iaea.org/MTCD/publications/PDF/P1251-cd/papers/53.pdf
  * [6] S Yamada  “COMMISSIONING AND PERFORMANCE OF THE HIMAC MEDICAL ACCELERATOR”  In _IEEE_ , 1996
  * [7] Masao Murakami  “Current status of the HIBMC and results of representative diseases”  In _AIP Conference Proceedings_ 1153, 2009
  * [8] Koji Noda  “New accelerator facility for carbon-ion cancer-therapy”  In _J. Radiat. Res._ 48, 2007
  * [9] D. Ondreka  “The Heidelberg Ion Therapy (HIT) accelerator coming into operation”  In _European Physical Society Accelerator Group, Proc. of EPAC_ , 2008
  * [10] P J Bryant  “Proton-Ion Medical Machine Study (PIMMS), Part 1”, 1999
  * [11] Sanja Damjanovic  “South East European International Institute for Sustainable Technologies (SEEIIST)”  In _Frontiers in Physics_ 8, 2021
  * [12] E. Benedetto  “Comparison of accelerator designs for an ion therapy and research facility”  In _(Unpublished)_ , 2021
  * [13] X. Zhang  “Current and future synchrotron designs for carbon ion therapy”  In _AIP Conference Proceedings_ American Institute of Physics Inc., 2021
  * [14] E. Benedetto  “A Carbon-Ion Superconducting Gantry and a Synchrotron Based on Canted Cosine Theta Magnets”, 2021  arXiv:2105.04205 [physics.med-ph]
  * [15] Yoshiyuki Iwata  “Development of Carbon-Ion Radiotherapy Facilities at NIRS”  In _IEEE Trans. Appl. Supercond._ IEEE, 2018
  * [16] C.. Goodzeit  “The double-helix dipole - A novel approach to accelerator magnet design”  In _IEEE Trans. Appl. Supercond._ , 2003
  * [17] Weishi Wan  “Alternating-gradient canted cosine theta superconducting magnets for future compact proton gantries”  In _PRAB_ 18 American Physical Society, 2015
  * [18] T Satogata  “Ions in a Rapid Cycling Medical Synchrotron”, 2006
  * [19] J-L Revol  “Highlights of Accelerator Activities in France on behalf of the Accelerator Division of the French Physics Society”  In _1st International Particle Accelerator Conference (IPAC’10)_ , 2010
  * [20] Jacques Balosso  “The ARCHADE Ion Cyclotron” Accessed: 2020-05-17, 2020  URL: https://indico.cern.ch/event/839930/contributions/3523612/
  * [21] A Degiovanni  “LIGHT: A LINEAR ACCELERATOR FOR PROTON THERAPY”  In _NAPAC2016_ , 2016
  * [22] B. Mustapha  “Prospects for an advanced heavy ion therapy center in the Chicago area”  In _AIP Conf. Proc._ 2160 American Institute of Physics Inc., 2019
  * [23] Vittorio Bencini  “DESIGN OF A NOVEL LINEAR ACCELERATOR FOR CARBON ION THERAPY”, 2020
  * [24] Silvia Verdú-Andrés  In _J. Radiat. Res._ , 2013
  * [25] K J Peach  “Conceptual Design of a Nonscaling Fixed Field Alternating Gradient Accelerator for Protons and Carbon Ions for Charged Particle Therapy”  In _PRAB_ 16, 2013
  * [26] Toshiyuki Shirai  “Design of quantum scalpel for the new heavy ion radiotherapy”  In _15th Annual Meeting of Particle Accelerator Society of Japan_ , 2018
  * [27] Estuo Noda  “Direct Injection of Laser-Accelerated Ions into a Superconducting Synchrotronon II”  In _16th Annual Meeting of Particle Accelerator Society of Japan_ , 2019
  * [28] Galen Aymar  “LhARA: The Laser-hybrid Accelerator for Radiobiological Applications”  In _Frontiers in Physics_ 8, 2020
  * [29] Hywel Owen  “Technologies for Delivery of Proton and Ion Beams for Radiotherapy”  In _International Journal of Modern Physics A_ 29, 2014
  * [30] V Rizzoglio  “Uncertainty Quantification Analysis and Optimization for Proton Therapy Beam Lines”  In _Physica Medica_ 75 Elsevier, 2020
  * [31] Hermann Fuchs  “Magnetic field effects on particle beams and their implications for dose calculation in MR-guided particle therapy”  In _Medical physics_ 44, 2017
  * [32] M. Galonska  “The HIT gantry: From commissioning to operation”  In _Proc. of IPAC13_ , 2013
  * [33] Lucas Brouwer  “An achromatic gantry for proton therapy with fixed-field superconducting magnets”  In _International Journal of Modern Physics A_ 34 World Scientific Publishing Co. Pte Ltd, 2019
  * [34] L Bottura  “GaToroid : A Novel Toroidal Gantry for Hadron Therapy”  In _Nucl. Instrum. Methods Phys. Res. A_ 983 Elsevier Ltd, 2020
  * [35] Enrico Felcini  “Magnetic Design of a Superconducting Toroidal Gantry for Hadron Therapy”  In _IEEE Trans. Appl. Supercond._ 30, 2020
  * [36] D Trbojevic  “Innovative Superconducting Non Scaling Fixed Field Alternating Gradient Isocentric Gantry for Carbon Cancer Therapy”  In _Proc. of IPAC2011, San Sebastián, Spain_ , 2011
  * [37] Tianyu Zhao  “Commissioning and initial experience with the first clinical gantry-mounted proton therapy system”  In _Journal of Applied Clinical Medical Physics_ 17, 2016
  * [38] S Savazzi  “Implementation of RF-KO Extraction At CNAO”, 2019
  * [39] Simon Jolly  “Technical challenges for FLASH Proton Therapy”  In _Physica Medica_ 78 Elsevier, 2020
  * [40] Andrea Mairani  “Development and Validation of Single Field Multi-Ion Particle Therapy Treatments”  In _Int. J. Radiat. Oncol. Biol. Phys._ 106 Elsevier Inc., 2020
  * [41] Davide Mazzucconi  “Mixed particle beam for simultaneous treatment and online range verification in carbon ion therapy: Proof-of-concept study”  In _Medical Physics_ 45, 2018
  * [42] E Oponowicz  “Superconducting Gantry Design for Proton Tomography”  In _Proc. of IPAC17_ , 2017
  * [43] Y.. Yuan  “Status of the HIRFL-CSR complex”  In _Nucl. Instrum. Methods Phys. Res. B_ 317, 2013
  * [44] Pantechnik  “Turn-key bench with Supernanogan”  URL: www.pantechnik.com
  * [45] L. Celona  “Experimental characterization of the AISHa ion source”  In _Rev. Sci. Instrum._ 90 AIP Publishing, LLC, 2019
  * [46] F. Wenander  “The TwinEBIS setup: Machine description”  In _Nucl. Instrum. Methods Phys. Res. A_ 856 Elsevier B.V., 2017
  * [47] A. Kitagawa  “Review on heavy ion radiotherapy facilities and related ion sources (invited)”  In _Rev. Sci. Instrum._ 81, 2010
  * [48] “Particle Therapy Co-Operative Group” Accessed: 2020-05-17  URL: https://www.ptcog.ch/
  * [49] B Franczak  “SIS Parameter List”, 1987
  * [50] H. Eickhoff  “Accelerator Aspects of the Cancer Therapy Project at the GSI Darmstadt”
  * [51] Yoshio Hishikawa  “The cancer treatment system at hyogo ion beam medical center (HIBMC)”  In _Journal of JASTRO_ 14, 2002
  * [52] Kenji Mishima  “Survey and Alignment of the Medical Accelerator PATRO at HIBMC”  In _7th International Workshop on Accelerator Alignment_ , 2002
  * [53] A Itano  “OPERATION OF MEDICAL ACCELERATOR PATRO AT HYOGO ION BEAM MEDICAL CENTER”
  * [54] D Trbojevic  “Lattice Design of a Rapid Cyclig Medical Synchrotron for Carbon/Proton Therapy”  In _Proc. of IPAC2011, San Sebastián, Spain_ , 2011
  * [55] Stephanie E. Combs  “Heidelberg Ion Therapy Center (HIT): Initial clinical experience in the first 80 patients”  In _Acta Oncologica_ 49, 2010
  * [56] Andreas Peters  “The accelerator facility of the Heidelberg Ion-Beam Therapy Centre (HIT)”  In _Challenges and Goals for Accelerators in the XXI Century_ , 2016
  * [57] A Dolinskii  “The Synchrotron of the Dedicated Ion Beam Facility for Cancer Therapy, Proposed for the Clinic in Heidelberg”  In _EPAC 2000_ , 2000
  * [58] Tatsuya Ohno  “Carbon ion radiotherapy at the Gunma university heavy ion medical center: New facility set-up”  In _Cancers_ 3, 2011
  * [59] Kota Torikai  “REPORT OF HEAVY-ION MEDICAL CENTER IN GUNMA UNIVERSITY”  In _7th Annual Meeting of Particle Accelerator Society of Japan_ , 2010
  * [60] Y J Yuan  “STATUS OF HIRFL-CSR PROJECT”  In _COOL 09_ , 2009
  * [61] J.W. Xia  “The heavy ion cooler-storage-ring project (HIRFL-CSR) at Lanzhou”  In _Nucl. Instrum. Methods Phys. Res. A_ 488, 2002
  * [62] Sandro Rossi  “The National Centre for Oncological Hadrontherapy (CNAO): Status and perspectives”  In _Physica Medica_ 31 Elsevier Ltd, 2015
  * [63] Sandro Rossi  “Developments in proton and light-ion therapy”  In _EPAC 2006_ , 2006
  * [64] M.. Angoletta  “CERN PSB beam tests of CNAO synchrotron’s digital LLRF”  In _EPAC 2008_ , 2008
  * [65] P J Bryant  “Proton-Ion Medical Machine Study (PIMMS), Part 2”, 2000
  * [66] H Rohdjeß  “STATUS OF THE SIEMENS PARTICLE THERAPY* ACCELERATORS”  In _Proc. of PAC09_ , 2009
  * [67] U. Scheeler  “Recommissioning of the Marburg Ion-Beam Therapy Centre (MIT) accelerator facility”  In _IPAC 2016 - The 7th International Particle Accelerator Conference_ , 2016
  * [68] S.. Møller  “A novel proton and light ion synchrotron for particle therapy”  In _EPAC 2006_ , 2006
  * [69] SP Moller  “A synchrotron based particle therapy accelerator”  In _2007 IEEE Particle Accelerator Conference (PAC)_ , 2007  IEEE
  * [70] Y. Jongen et al.  “Compact superconducting cyclotron C400 for hadron therapy”  In _Nucl. Instrum. Methods Phys. Res. A_ 624 Elsevier, 2010
  * [71] Heejoong Yim et al.  “Design of the KHIMA synchrotron”  In _Journal of the Korean Physical Society_ 67 Korean Physical Society, 2015
  * [72] Hyun Wook Kim, Bong Hwan Hong and Joonsun Kang  “Design Study of Main Dipole Magnets for Korea Heavy Ion Medical Accelerator Synchrotron”  In _IEEE Trans. Appl. Supercond._ 26 Institute of ElectricalElectronics Engineers Inc., 2016
  * [73] J. Shi  “Heavy ion medical machine (HIMM) slow extraction commissioning”  In _Nucl. Instrum. Methods Phys. Res. A_ 918 Elsevier Ltd, 2019
  * [74] J.. Yang  “Design of a compact structure cancer therapy synchrotron”  In _Nucl. Instrum. Methods Phys. Res. A_ 756, 2014
  * [75] Xiaoying Zhang  “The main dipole magnets design and test of HIMM project”  In _IEEE Transactions on Applied Superconductivity_ 26.4 IEEE, 2016
  * [76] Koji Noda  “Review of ion therapy machine and future perspective”, 2019
  * [77] Sho Kudo  “Construction of SAGA HIMAT for carbon ion cancer therapy”  In _AIP Conference Proceedings_ 1525, 2013
  * [78] Yuko Nakayama  “The Ion-Beam Radiation Oncology Center in Kanagawa (i-ROCK) Carbon Ion Facility at the Kanagawa Cancer Center”  In _International Journal of Particle Therapy_ 2, 2015
  * [79] M.T.F. Pivi  “STATUS OF THE CARBON COMMISSIONING AND ROADMAP PROJECTS OF THE MEDAUSTRON ION THERAPY CENTER ACCELERATOR”  In _IPAC2019_ , 2019
  * [80] Guo-Liang Jiang  “How to Start A Particle Therapy Center”, 2014  URL: http://ptcog.ch/archive/conference_p&t&v/PTCOG53/PresentationsEW/PTCOG53_Jiang.pdf

##  4 Appendix

In Table 2 below, we give a full version of the ion therapy machine summary.
This includes additional information on each accelerator, as well as some
other facilities based on the same base machine design, and full citations for
all the information.

Table 2: Parameters of ion therapy machines. Entries that are proposals but never constructed are given in italics. As well as the sources given in the table, additional information is available from [47] and [48]. Name | Location | Active | Main Tech. | Pre-Acc | Extraction | Rep. Rate | Circ. (m) | B${}_{\text{max}}$ (T) | RF Freq. | V${}_{\text{cavity}}$ | Offer | Species | Extracted | Particles | Sources
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  | Years |  |  | Method | (Hz) |  |  | Range | Max (kV) | Protons |  | KE (MeV/u) | per Spill |
HIMAC | Japan | 1994 - | Synchrotron | Linac | Resonant | 1.5 |  $130$(*$2$ | 1.5 | 1.0-8.0 | 10 | No | He |  $100$-$800$ | $1.2\text{\times}{10}^{10}$ | [6]
|  |  |  |  |  |  | rings) |  |  |  |  | C |  $100$-$800$ | $2.0\text{\times}{10}^{9}$ |
GSI | Germany | 1998 - 2008 | Synchrotron | Linac | Resonant | 0.3 | $216.7$ | 1.8 | 0.85-6.0 | 16 | Yes | C |  $80$-$430$ | $1.0\text{\times}{10}^{8}$ | [49, 5, 50]
HIBMC | Japan | 2003 - | Synchrotron | Linac | Resonant | 0.5 | $94$ | 1.38 | 0.99-6.42 | 5.2 | Yes | C |  $70$-$320$ | $2.0\text{\times}{10}^{9}$ | [7, 51, 52, 53]
iRCMS | Proposal | — | Synchrotron | Linac | — | 30 | $60$ | 1.33 | 0.65-3.57 | — | No | C |  $96$-$450$ | $2.7\text{\times}{10}^{7}$ | [54, 18]
HIT | Germany | 2009 - | Synchrotron | Linac | RF-KO | 0.15 | $65$ | 1.53 | 1.0-7.0 | 2 | Yes | He |  $51$-$221$ | $1.0\text{\times}{10}^{10}$ | [55, 9, 56, 57]
|  |  |  |  |  |  |  |  |  |  |  | C |  $88$-$430$ | $1.0\text{\times}{10}^{9}$ |
|  |  |  |  |  |  |  |  |  |  |  | O |  $103$-$430$ | $5.0\text{\times}{10}^{8}$ |
GHMC1 | Japan | 2009 - | Synchrotron | Linac | RF-KO | 0.36 | $63$ | 1.48 | 0.9-7.0 | 2 | No | C |  $140$-$400$ | $1.3\text{\times}{10}^{9}$ | [58, 59, 8]
HIRFL- | China | 2009 - | Synchrotron | Cyclotron | Fast: Kicker | 0.06 | $161$ | 1.4 | 0.24-1.7 | 20 | No | C |  $100$-$430$ | $5.0\text{\times}{10}^{8}$ | [60, 43, 61]
CSR |  |  |  |  | Slow: Resonant |  |  |  |  |  |  |  |  |  |
PAMELA | Proposal | — | FFA | Cyclotron | Kicker | 1000 | $58$ | 4.25 | 19.2-45.6 | 15 | Yes | C |  $110$-$440$ | $5.0\text{\times}{10}^{6}$ | [25]
CNAO2 | Italy | 2010 - | Synchrotron | Linac | RF-KO | 0.4 | $77.6$ | 1.5 | 0.47-2.76 | 5 | Yes | C |  $120$-$400$ | $1.5\text{\times}{10}^{9}$ | [62, 63, 64, 65]
MIT3 | Germany | 2015 - | Synchrotron | Linac | RF-KO | 1 | $65$ | 1.43 | 1.0-7.0 | 2.5 | Yes | C |  $85$-$430$ | $7.0\text{\times}{10}^{8}$ | [66, 67, 68, 69]
ARCHADE | Proposal | — | Cyclotron | — | Deflector | — | $21$ | 4.5 | 75 | 80 | No | C | $400$ | — | [70, 20]
KHIMA | Proposal | — | Synchrotron | Linac | Resonant | 0.3 | $75$ | 1.5 | 0.49-5.83 | 0.79 | Yes | C |  $110$-$430$ | $1.4\text{\times}{10}^{9}$ | [71, 72]
HIMM4 | China | 2019 - | Synchrotron | Cyclotron | RF-KO | 0.31 | $56.2$ | 1.66 | 0.6-3.9 | 5 | No | C |  $120$-$400$ | $4.0\text{\times}{10}^{9}$ | [73, 74, 75]
Quantum | Proposal | — | Laser and | Laser | — | — | $28$ | 4.0 | 10 | — | No | C |  $56$-$430$ | $1.0\text{\times}{10}^{8}$ | [27, 26, 76]
Scalpel |  |  | Synchrotron |  |  |  |  |  |  |  |  |  |  |  |
NIMMS | Proposal | — | Synchrotron | Linac | Fast: Kicker | — | NC: $76$ | 1.5 | — | — | Yes | He |  $60$-$250$ | $8.2\text{\times}{10}^{10}$ | [12, 14, 13]
|  |  |  |  | Slow: RF-KO |  | DBA: $55$ | 1.5 |  |  |  | C |  $100$-$430$ | $2.0\text{\times}{10}^{10}$ |
|  |  |  |  |  |  | SC: $27$ | 4.5 |  |  |  | O |  $100$-$430$ | $1.4\text{\times}{10}^{10}$ |
1 Many facilities follow the GHMC design, mainly in Japan [15]. These include
SAGA-HIMAT [77], i-ROCK [78] and Yamagata University Hospital
2 CNAO and MedAustron [79] share near identical parameters. Though it once
used a betatron core for extraction, CNAO now uses RF-KO
3 MIT and SPHIC [80] are essentially the same design [69]
$4$ The same design is being followed for both HIMM Wuwei and HIMM Lanzhou


[affiliation=1,2]PeikunChen [affiliation=2]SiningSun
[affiliation=2]ChanghaoShan [affiliation=2]QingYang [affiliation=1*]LeiXie

# Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech
Units: A Pilot Study

###### Abstract

Unified speech-text models like SpeechGPT, VioLA, and AudioPaLM have shown
impressive performance across various speech-related tasks, especially in
Automatic Speech Recognition (ASR). These models typically adopt a unified
method to model discrete speech and text tokens, followed by training a
decoder-only transformer. However, they are all designed for non-streaming ASR
tasks, where the entire speech utterance is needed during decoding. Hence, we
introduce a decoder-only model exclusively designed for streaming recognition,
incorporating a dedicated boundary token to facilitate streaming recognition
and employing causal attention masking during the training phase. Furthermore,
we introduce right-chunk attention and various data augmentation techniques to
improve the model's contextual modeling abilities. While achieving streaming
speech recognition, experiments on the AISHELL-1 and -2 datasets demonstrate
the competitive performance of our streaming approach with non-streaming
decoder-only counterparts. The code we used for this work can be found
here111https://github.com/chenpk00/IS2024_stream_decoder_only_asr.

###### keywords:

streaming automatic speech recognition, discrete-token, decoder-only
Transformer

##  1 Introduction

Recently, large language models (LLMs) [1, 2, 3, 4] have made great progress
in various natural language processing (NLP) tasks. Methods based on LLMs are
also leading a revolution in other research fields, such as computing vision
[5, 6, 7] and speech [8, 9, 10, 11]. Thanks to the powerful understanding of
language and generalization ability of an LLM, transferring LLM pretrained
from massive amounts of text data to speech recognition tasks can also bring
significant word error rate reduction [12, 13]. As for the model architecture,
most of the LLMs are based on decoder-only transformers [14], which simplifies
the model structure used for ASR tasks. Therefore, LLM-based ASR is becoming a
hot research topic in the field.

Currently, there are two primary approaches to integrating the speech modality
with LLMs in LLM-based ASR. One approach involves directly integrating
continuous features with text embedding through a trainable adaptor [15, 16,
17]. This kind of approach introduces additional acoustic encoders and models
speech and text separately. Conversely, the other approach is to treat speech
representation as textual tokens and employ a decoder-only model to optimize
multi-modal tasks effectively. For instance, VioLA [8] converts continuous
speech signals to discrete codec codes via EnCodec [18] and unifies several
speech-related tasks into a conditional language modeling task. SpeechGPT [9]
employs LLaMA [2] as its foundational framework, utilizing $k$-means
clustering derived from Hubert [19] to tokenize speech signal. Similarly,
AudioPaLM [10] utilizes PaLM-2 [20] as its underlying architecture and
extracts discrete tokens from the encoder of Universal Speech Model [21]. Such
a unified modeling approach has been demonstrated to effectively improve the
ASR performance. The decoder-only transformer model with unified discrete
input provides a new paradigm for various speech-related tasks, including but
not limited to speech recognition and speech synthesis.

Previous works on decoder-only ASR tasks mainly focused on non-streaming
scenarios [8, 10]. However, real-time streaming recognition can give faster
recognition results and a better user experience in real-world applications.
Many works have been proposed to build a faster and better streaming ASR
system in the past decades based on various end-to-end speech recognition
frameworks [22, 23]. However, the exploration of streaming decoder-only speech
recognition is very limited. As decoder-only-based ASR model performance
improves and the number of model parameters increases, streaming low-latency
inference becomes a challenging task.

This paper presents a pilot study on the streaming decoder-only transformer
ASR model. Current non-streaming decoder-only transformer ASR models learn to
predict text tokens autoregressively using the whole speech utterance [24].
For the streaming version, it is necessary to emit text tokens with minimal
delay as the corresponding speech segment is received. To this end, we
investigate two approaches based on the speech-to-text alignment obtained by a
GMM-HMM model [25]. Specifically, the first approach, Text Token Insertion
(TTI), inserts the corresponding text tokens into the speech token sequences
directly under the guide of speech-to-text alignment during training. By
contrast, in Boundary Token Insertion (BTI), special ``boundary tokens" are
inserted into the speech token sequences in the same way with text tokens
added at the end, effectively decoupling the speech and text modalities. Upon
triggering a boundary token, the corresponding text token can be generated
through a one-step inference process autoregressively. Meanwhile, we introduce
right-chunk attention and various data augmentation techniques to improve the
streaming model's contextual modeling ability. We also explore the efficacy of
leveraging an off-the-shelf text LLM to initialize our streaming ASR model.

(a) Non-streaming model

(b) TTI streaming model

(c) BTI streaming model

Figure 1: The comparison of different methods for training discrete-token-
based decoder-only Transformer for ASR. (a) non-streaming model: decoding
after receiving the whole speech token; (b) Text token insertion (TTI)
streaming model: inserts text tokens into speech token sequences directly
under the guide of speech-to-text alignment; (3) Boundary token insertion
(BTI) streaming model: insert ``boundary tokens" into the discrete speech
token sequence.

Experiments show that our proposed streaming decoder-only model can obtain
5.9% and 7.2% character error rate (CER) on two Chinese Mandarin corpora,
AISHELL-1 [26] and AISHELL-2 [27], respectively, showing competitive
performance with the non-streaming counterpart. Our results also show that the
streaming decoder-only Transformer ASR model can benefit from the
initialization from an off-the-shelf text LLM, such as Qwen [4].

##  2 Proposed Method

###  2.1 Streaming decoder-only model architecture

Figure 1 illustrates three types of decoder-only models designed for ASR tasks
based on discrete speech token input. Among them, Figure 1 (a) represents a
non-streaming model proposed in [24], whereas Figures 1 (b) and (c) depict two
variations of streaming frameworks. Both models (b) and (c) necessitate force
alignment between speech and text. As shown in equation (1), given discrete
speech token sequence $x=(x_{1},...,x_{t},...x_{T})$ and the correspoing text
token sequence $y=(y1,...,y_{L})$, streaming ASR model is optimized by
maximizing the conditional probability, where $t_{y_{i}+\Delta}$ is the time
of emitting text token $y_{i}$, $\Delta$ is a constant, which means how many
right context tokens can be used, $x_{\leq
t_{y_{i}}+\Delta}=({x_{1},...,x_{t_{y_{i}}+\Delta}})$ and $\theta$ is the
trainable model parameter.

$p(y|x;\theta)={\textstyle\prod_{1}^{L}}p\left(y_{i}\mid x_{\leq
t_{y_{i}+\Delta}},\theta\right).$ (1)

In Figure 1 (b), the Text Token Insertion (TTI) approach showcases the
interleaving of discrete speech and text tokens, with text tokens inserted
into speech tokens at the end of the corresponding speech segment.
Mathematically, it equals to optimize equation (1) directly. However, the mix
of text and speech tokens complicates the use of beam search during decoding.
During inference, speech tokens can be treated as conditions. Given the
interleaved nature of text and speech tokens, triggering a text token during
beam search necessitates caching all historical hidden states (e.g., key and
value of self-attention) for each search path.

Figure 1 (c) illustrates the Boundary Token Insertion (BTI) approach, where a
special token, instead of a text token, is inserted into the speech token
sequence, effectively decoupling the text and speech token sequences. In
contrast to Figure 1 (b), this process can be viewed as comprising two stages:
the first stage involves determining the boundary position, while the second
stage entails predicting the corresponding specific text token conditioned on
the history of speech tokens. Equation (2) provides a formal definition. Here,
a hidden variable $b$ is introduced, where $b=(b_{1},...,b_{T})\in{0,1}^{T}$
represents one of the possible boundary paths, and $\beta$ denotes the set of
all possible paths. However, in practice, optimizing by summing over all
possible paths is computationally challenging. Therefore, we opt to
approximate this optimization problem by selecting the most probable path
$b_{p}$.

$\begin{split}p\left(y\mid x,\theta\right)&={\sum_{b\in\beta}}p\left(y,b\mid
x,\theta\right)\\\
&={\sum_{b\in\beta}}{\textstyle\prod_{1}^{L}}p\left(y_{i},b\mid x\leq
t_{y_{i}+\Delta},\theta\right)\\\
&\approx{\textstyle\prod_{1}^{L}}p\left(y_{i}\mid
b_{p},x_{\leq}t_{y_{i}+\Delta},\theta\right)p\left(b_{p}\mid
x_{\leq}t_{y_{i}+\Delta},\theta\right),\\\ \end{split}$ (2)

(a) global attention

(b) causal attention

(c) right-chunk attention

Figure 2: Example diagram of different attention mechanisms. The green blocks
indicate the part of the LLM. The yellow triangle indicates the part of the
attention area. (a) global attention (b) causal attention; (c) right-chunk
attention.

###  2.2 Right-chunk attention

The previous non-streaming decoder-only ASR models predict the text tokens
aggressively with the entire discrete speech tokens. Illustrated in Figure 2
(a), the speech tokens primarily attend to each other, while the text tokens
focus on all discrete speech tokens and preceding text tokens.

To achieve streaming speech recognition, we introduce a causal attention
mechanism. As shown in the formula (3), a casual mask is applied during self-
attention calculation, where $M$ is the mask.

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QKM}{\sqrt{d_{k}}}\right){V},$
(3)

We divide attention mask $M$ into two parts, denoted as $M_{speech}$ and
$M_{text}$, representing the attention mask used for speech and text tokens,
respectively. Equations (4) and (5) provide the definitions of
$M_{\text{speech}}$ and $M_{\text{text}}$.

$M_{speech}(i,j)=\left\\{\begin{array}[]{ll}\text{True }&i\leq j\\\
\text{False }&\text{ otherwise }\end{array}\right.,$ (4)

where $0\leq i\leq T+L$ because of the insertion of boundary tokens in the
speech. Meanwhile, as shown in the figure 1 (b), $mask_{text}$ can be
represented by

$\begin{split}M_{text}(i,j)=\left\\{\begin{array}[]{ll}\text{True }&j\leq
t_{y_{i}}\\\ &or\ T+L<i\leq j\leq T+2L\\\ \text{False }&\text{otherwise
}\end{array}\right.,\end{split}$ (5)

where $M_{text}$ consists of two yellow triangles. At each time step, the
current text focuses on both the preceding speech and text.

Unlike non-streaming ASR models, streaming ASR models face limitations in
considering global information, leading to weakened contextual modeling
capabilities. To address this issue, we integrate the right-chunk attention
mechanism, illustrated in Figure 2 (c). Unlike causal attention, right-chunk
attention in the text portion enables capturing more speech information.

$\begin{split}M_{text}(i,j)=\left\\{\begin{array}[]{ll}\text{True }&j\leq
t_{y_{i}+\Delta}\\\ &or\ T+L<i\leq j\leq T+2L\\\ \text{False }&\text{otherwise
}\end{array}\right.,\end{split}$ (6)

where $\Delta$ means how many speech tokens in the right context can be used.
Larger $\Delta$ also means a higher latency.

###  2.3 Data Pre-processing

Another significant improvement lies in the pre-processing steps. Throughout
the training and decoding stages, we observed a tendency for the model to more
readily overfit discrete features compared to continuous features, thus
significantly impacting the final results. To address this concern, we
employed data pre-processing, which plays a crucial role in model training and
can substantially enhance data diversity and quantity. To improve the model's
robustness, we implemented the following strategies:

Speed Perturbation: Adjusting the audio speed increases the variety of
discrete speech tokens, allowing the text segment to match a wider range of
token combinations. In our study, we applied speed perturbation by altering
the audio speed to 0.9 and 1.1 times the original speed.

Trigger Shift: Leveraging alignment biases to enhance the robustness of
trigger tokens, we randomly shift trigger tokens by 1-4 frames with a
probability of 30% during the training phase.

Time Masking: Time masking is applied to input tokens other than trigger
tokens, encompassing both speech and text tokens, by substituting each token
with a special padding token with a probability of 0.3.

Random De-duplication: Employing a randomized de-duplication approach reduces
computational complexity while amplifying data diversity. Concurrently, during
the decoding phase, we implement global de-duplication to further alleviate
computational overhead.

Label Smoothing: Discretized speech tokens manifest intersections within
clustering, resulting in losses during the speech discretization process. To
address this issue, we adopt label smoothing [28],

$\mathcal{L}_{\text{LS}}=\sum_{t=1}^{T_{s}}D_{\text{KL}}(q^{\prime}(x_{t}|x_{<t})||p(x_{t}|x_{<t};\theta)),$
(7)

where $q^{\prime}(x_{t}|x_{<t})$ is a soft label by label smoothing instead of
a one-hot label, and $D_{\text{kl}}$ is the Kullback-Leibler divergence.

##  3 Experiments

###  3.1 Experimental Setup

Our implementation is based on Wenet [29], an open-source toolkit for End-to-
End (E2E) speech recognition. Our ASR models employ a decoder-only
architecture based on the transformer. We have established two model
configurations: a small model (70 million parameters) without Large Language
Model (LLM) initialization, compared with the previous non-streaming AEDs
model, and a large model (310 million parameters) used to verify the
feasibility of off-the-shelf LLM initialization. In future endeavors, we
intend to incorporate LLMs with significantly more parameters for validation,
such as models with 2 billion or larger parameters.

Dataset: In this study, we conduct experiments on two commonly used Chinese
Mandarin corpora, 178-hour AISHELL-1 [26] and 1000-hour AISHELL-2 [27]. We
report the character error rates (CER) of various models.

Discrete Speech Tokens: In this paper, we draw inspiration from a recent study
[30] employing Canonical Correlation Analysis (CCA) to evaluate the similarity
between layer representations and word labels. For the Chinese corpus, we
select Chinese HuBERT large222https://huggingface.co/TencentGameMate/chinese-
hubert-large. Subsequently, we choose layer 21 from the large models, as it
demonstrates the highest CCA similarities with word labels. The number of
K-Means clusters is set to 2,000, consistent with the previous method [24,
31].

Model Configuration: The small decoder-only transformer without LLM init
comprises 8 blocks, each with 8 self-attention heads, with an attention
dimension of 512, and a feed-forward network (FFN) with an intermediate hidden
dimension of 1024. To explore the effectiveness of using off-the-shelf LLM
init, we also adopt Qwen[4] as the backbone of our decoder-only Transformer
model for discrete-token-based ASR systems. We use the
Qwen2-0.5B333https://huggingface.co/Qwen/Qwen1.5-0.5B model (transformer with
310M parameters), which consists of 24 layers with the hidden size 1024, 16
attention heads, max sequence length of 1024. Note that we do not use the Qwen
text tokenizer because it can cause token sparsity problems, which means there
is not enough training data for some tokens. Instead, we directly discretize
text into Chinese characters. There are a total of 7000 model units with 5000
commonly used Chinese char and 2000 speech tokens. In all experiments, we set
the dropout rate to 0.1. Commonly during the training phase, we dynamically
set $\Delta$ to be equal to the length of the speech segment corresponding to
the next text token. During ASR decoding, we set the beam size to 10 and did
not utilize language models in our experiments.

Table 1: CERs (%) on AISHELL-1 for different methods.

ID | Feature | Model type | Streaming | CER
---|---|---|---|---
dev | test
B1 | SSL 2 | encoder-decoder | ✗ | 3.8 | 4.0
B2 | Fbank [32] | encoder-decoder | ✗ | 4.2 | 4.5
B3 | Discrete [31] | encoder-decoder | ✗ | 4.6 | 4.9
Small
S1 | Discrete | decoder-only | ✗ | 5.9 | 6.2
S2 | Discrete (TTI) | decoder-only | ✓ | 9.4 | 9.8
S3 | Discrete (BTI) | decoder-only | ✓ | 6.1 | 6.4
Large (w/ Qwen-0.5B init)
L1 | Discrete | decoder-only | ✗ | 5.2 | 5.5
L2 | Discrete (TTI) | decoder-only | ✓ | 9.2 | 9.5
L3 | Discrete (BTI) | decoder-only | ✓ | 5.6 | 5.9

###  3.2 Main Results

Table 1 presents a CER on AISHELL-1. The first group in Table 1 presents
results from previous studies, including semi-supervised learning (SSL) [19],
E-Branchformer [32] based ASR model with continuous Fbank feature or discrete
speech token [31] as input. The second group displays the results of our small
model trained with our proposed two streaming methods using random
initialization. In comparison to S3, the substitution errors of S2 have
notably increased due to TTI's design of text token insertion, which weakens
its contextual modeling ability. It is evident that better model performance
was achieved with S3 by decoupling boundary prediction and text prediction
through BTI. The third group shows the results of the large model initializing
with Qwen-0.5B LLM. With LLM initialization, we first train a non-streaming
decoder-only model, which is model L1. Then we fine-tune the model L1 with our
proposed streaming methods TTI and BTI, resulting in two streaming versions,
L2 and L3. We observe that the large model with LLM initialization achieves a
5.9% CER on the AISHELL-1 test set. Upon comparing L2 with S2, it is noted
that LLM initialization yields a relatively minor CER reduction for the TTI
approach. This is attributed to the differing nature of interleaving speech
and text compared to LLMs, resulting in only a relative CER benefit of 3.1%
(9.8%$\to$9.5%). Conversely, employing BTI results in a 9.2% (6.5%$\to$5.9%)
relative CER reduction on the test set compared to model S3. Notably, BTI not
only exhibits superior CER performance but also demonstrates better
adaptability to LLMs. Hence, in subsequent experiments, we employ the BTI
approach.

###  3.3 Ablation Study

Table 2 shows the impact of right-chunk attention and various data
augmentation based on model S3. Notably, the right-chunk attention has the
most significant impact on the overall CER. Without right-chunk attention, the
CER increases from 6.5% to 7.8% on the AISHELL-1 test set. The absence of
right-chunk attention results in more substitution errors due to the limited
contextual information available.

Among the five data pre-processing methods, label smoothing is the most
effective method, resulting in a relative 9.7% CER reduction. We observe that
when modeling speech and text tokens in a unified manner, predicting the next
speech token is considerably easier than predicting the next text token.
Consequently, the model tends to overfit during speech token prediction. Label
smoothing effectively mitigates this overfitting issue. Additionally, other
methods also contribute to some reduction in CER, ranging from 4.4% to 7.2%.

Table 2: Ablation of each component’s impact on CERs (%). Method | CER
---|---
dev | test
BTI (ours) | 6.1 | 6.4
w/o right-chunk attention | 7.4 | 7.8
w/o speed perturb | 6.5 | 6.9
w/o trigger shift | 6.4 | 6.7
w/o time mask | 6.4 | 6.8
w/o random de-duplication | 6.3 | 6.7
w/o label smoothing | 6.8 | 7.2

###  3.4 Results on AISHELL-2

In Table 3, we present the performance on the AISHELL-2 corpus without speed
perturbation. The top lines list the conventional FBank-based ASR system and
the Hubert-Large discrete token-based ASR models. It is evident that using
discrete token input with a decoder-only model yields slightly inferior
performance compared to the encoder-decoder model (6.6% vs 6.9%). Meanwhile,
the recognition accuracy of streaming models decreased by 4.1% (6.9%$\to$7.2%)
compared to non-streaming results. We find that training ASR models using the
discrete units on large-scale data can be quite efficient. We believe that as
the amount of data and model parameters increase, the decoder-only model can
completely surpass the traditional encoder-decoder model.

Table 3: CERs (%) on AISHELL-2 for different methods.

ID | Feature | Model type | Streaming | CER
---|---|---|---|---
B4 | Fbank444https://github.com/wenet-e2e/wenet | encoder-decoder | ✗ | 6.2
B5 | Discrete | encoder-decoder | ✗ | 6.6
Large (w/ Qwen-0.5B init) |
L4 | Discrete | decoder-only | ✗ | 6.9
L5 | Discrete (BTI) | decoder-only | ✓ | 7.2

##  4 Conclusion and Future Work

In this work, we present a pilot study on the streaming decoder-only ASR with
discrete speech units. We explore two approaches to achieving streaming
decoder-only ASR: Text Token Insertion (TTI) and Boundary Token Insertion
(BTI). Experimental results on AISHELL-1 and -2 show that the BTI method
yields significantly better performance and competitive CER with the non-
streaming decoder-only model. With the initialization of pretrained LLM, the
performance of our proposed streaming decoder-only model can be further
improved. As a pilot study, there remains considerable work to be explored in
the follow-up. We will conduct experiments on more languages, larger datasets,
and large-scale models. Note that in this study, we use HuBERT as our speech
tokenizer, we will also compare more speech tokenizers in the future.

## References

  * [1] OpenAI, ``GPT-4 technical report,'' _CoRR_ , vol. abs/2303.08774, 2023.
  * [2] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, ``Llama: Open and efficient foundation language models,'' _CoRR_ , vol. abs/2302.13971, 2023.
  * [3] Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, ``GLM: general language model pretraining with autoregressive blank infilling,'' in _the 60th Annual Meeting of the Association for ComputationalLinguistics,ACL 2022_. Association for Computational Linguistics, 2022, pp. 320–335.
  * [4] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, and et al., ``Qwen technical report,'' _CoRR_ , vol. abs/2309.16609, 2023.
  * [5] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas, ``Videogpt: Video generation using VQ-VAE and transformers,'' _CoRR_ , vol. abs/2104.10157, 2021.
  * [6] J. Y. Koh, R. Salakhutdinov, and D. Fried, ``Grounding language models to images for multimodal inputs and outputs,'' in _International Conference on Machine Learning ,ICML 2023_. PMLR, 2023, pp. 17 283–17 300.
  * [7] L. Yu, Y. Cheng, Z. Wang, V. Kumar, W. Macherey, Y. Huang, D. A. Ross, I. Essa, Y. Bisk, M. Yang, K. P. Murphy, A. G. Hauptmann, and L. Jiang, ``SPAE: semantic pyramid autoencoder for multimodal generation with frozen llms,'' in _Neural Information Processing Systems,NeurIP_ , 2023.
  * [8] T. Wang, L. Zhou, Z. Zhang, Y. Wu, S. Liu, Y. Gaur, Z. Chen, J. Li, and F. Wei, ``Viola: Unified codec language models for speech recognition, synthesis, and translation,'' _CoRR_ , vol. abs/2305.16107, 2023.
  * [9] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu, ``Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,'' in _EMNLP 2023_. ACL, 2023, pp. 15 757–15 773.
  * [10] P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. de Chaumont Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, H. Muckenhirn, D. Padfield, and et al., ``Audiopalm: A large language model that can speak and listen,'' _CoRR_ , vol. abs/2306.12925, 2023.
  * [11] J. Wang, Z. Du, Q. Chen, Y. Chu, Z. Gao, Z. Li, K. Hu, X. Zhou, J. Xu, Z. Ma, W. Wang, S. Zheng, C. Zhou, Z. Yan, and S. Zhang, ``Lauragpt: Listen, attend, understand, and regenerate audio with GPT,'' _CoRR_ , vol. abs/2310.04673, 2023.
  * [12] Y. Hu, C. Chen, C. H. Yang, R. Li, C. Zhang, P. Chen, and E. S. Chng, ``Large language models are efficient learners of noise-robust speech recognition,'' _CoRR_ , vol. abs/2401.10446, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.10446
  * [13] C. Chen, R. Li, Y. Hu, S. M. Siniscalchi, P. Chen, E. S. Chng, and C. H. Yang, ``It's never too late: Fusing acoustic information into large language models for automatic speech recognition,'' _CoRR_ , vol. abs/2402.05457, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2402.05457
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _Neural Information Processing Systems (NeurIPS)_ , 2017, pp. 5998–6008.
  * [15] Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, ``Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,'' _CoRR_ , vol. abs/2311.07919, 2023.
  * [16] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, ``SALMONN: towards generic hearing abilities for large language models,'' _CoRR_ , vol. abs/2310.13289, 2023.
  * [17] J. Wu, Y. Gaur, Z. Chen, L. Zhou, Y. Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y. Wu, ``On decoder-only architecture for speech-to-text and large language model integration,'' in _IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023_. IEEE, 2023, pp. 1–8.
  * [18] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, ``High fidelity neural audio compression,'' _CoRR_ , vol. abs/2210.13438, 2022.
  * [19] W. Hsu, B. Bolte, Y. H. Tsai, and et al., ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE ACM Trans. Audio Speech Lang. Process._ , vol. 29, pp. 3451–3460, 2021.
  * [20] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Cheng, C. Cherry, L. Gonzalez, and et al., ``Palm 2 technical report,'' _CoRR_ , vol. abs/2305.10403, 2023.
  * [21] Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, Z. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghani, J. Riesa, G. Perng, H. Soltau, T. Strohman, F. Beaufays, Y. Wu, and et al., ``Google USM: scaling automatic speech recognition beyond 100 languages,'' _CoRR_ , vol. abs/2303.01037, 2023.
  * [22] S. Arora, G. Saon, S. Watanabe, and B. Kingsbury, ``Semi-autoregressive streaming ASR with label context,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024_. IEEE, 2024.
  * [23] Q. Li, B. Li, D. Hwang, T. N. Sainath, and P. M. Mengibar, ``Modular domain adaptation for conformer-based streaming ASR,'' _CoRR_ , vol. abs/2305.13408, 2023.
  * [24] Q. Chen, W. Wang, Q. Zhang, S. Zheng, S. Zhang, C. Deng, Y. Ma, H. Yu, J. Liu, and C. Zhang, ``Loss masking is not needed in decoder-only transformer for discrete-token based ASR,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024_. IEEE, 2024.
  * [25] L. R. Rabiner, ``A tutorial on hidden markov models and selected applications in speech recognition,'' _Proc. IEEE_ , vol. 77, no. 2, pp. 257–286, 1989.
  * [26] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, ``AISHELL-1: an open-source mandarin speech corpus and a speech recognition baseline,'' in _O-COCOSDA 2017_. IEEE, 2017, pp. 1–5.
  * [27] J. Du, X. Na, X. Liu, and H. Bu, ``AISHELL-2: transforming mandarin ASR research into industrial scale,'' _CoRR_ , vol. abs/1808.10583, 2018.
  * [28] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ``Rethinking the inception architecture for computer vision,'' in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016_. IEEE Computer Society, 2016, pp. 2818–2826.
  * [29] B. Zhang, D. Wu, Z. Peng, X. Song, Z. Yao, H. Lv, L. Xie, C. Yang, F. Pan, and J. Niu, ``Wenet 2.0: More productive end-to-end speech recognition toolkit,'' in _Interspeech 2022_. ISCA, 2022, pp. 1661–1665.
  * [30] A. Pasad, B. Shi, and K. Livescu, ``Comparative layer-wise analysis of self-supervised speech models,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023_. IEEE, 2023, pp. 1–5.
  * [31] X. Chang, B. Yan, K. Choi, J. Jung, Y. Lu, S. Maiti, R. S. Sharma, J. Shi, J. Tian, S. Watanabe, Y. Fujita, T. Maekaku, P. Guo, Y. Cheng, P. Denisov, K. Saijo, and H. Wang, ``Exploring speech recognition, translation, and understanding with discrete speech units: A comparative study,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing,ICASSP 2024_. IEEE, pp. 11 481–11 485.
  * [32] K. Kim, F. Wu, Y. Peng, J. Pan, P. Sridhar, K. J. Han, and S. Watanabe, ``E-branchformer: Branchformer with enhanced merging for speech recognition,'' in _IEEE Spoken Language Technology Workshop,SLT 2022_. IEEE, 2022, pp. 84–91.


# Blockchain-based Collaborated Federated Learning for Improved Security,
Privacy and Reliability

Amir Afaq1, Zeeshan Ahmed1 ID ,Noman Haider2 ID , Muhammad Imran1 ID 1 School
of Engineering, Information Technology and Physical Sciences, Federation
University, Brisbane, Australia.
2 College of Engineering and Science, Victoria University, Sydney, NSW, 2000,
Australia.
Email: {zeeshanahmed, noman90<EMAIL_ADDRESS>
###### Abstract

Federated Learning (FL) provides privacy preservation by allowing the model
training at edge devices without the need of sending the data from edge to
centralized server. FL has distributed the implementation of ML. Another
variant of FL which is well suited for the Internet of Things (IoT) is known
as Collaborated Federated Learning (CFL), which does not require an edge
device to have a direct link to the model aggregator. Instead, the devices can
connect to the central model aggregator via other devices using them as
relays. Although, FL and CFL protect the privacy of edge devices but raises
security challenges for a centralized server that performs model aggregation.
The centralized server is prone to malfunction, backdoor attacks, model
corruption, adversarial attacks and external attacks. Moreover, edge device to
centralized server data exchange is not required in FL and CFL, but model
parameters are sent from the model aggregator (global model) to edge devices
(local model), which is still prone to cyber-attacks. These security and
privacy concerns can be potentially addressed by Blockchain technology. The
blockchain is a decentralized and consensus-based chain where devices can
share consensus ledgers with increased reliability and security, thus
significantly reducing the cyberattacks on an exchange of information. In this
work, we will investigate the efficacy of blockchain-based decentralized
exchange of model parameters and relevant information among edge devices and
from a centralized server to edge devices. Moreover, we will be conducting the
feasibility analysis for blockchain-based CFL models for different application
scenarios like the internet of vehicles, and the internet of things. The
proposed study aims to improve the security, reliability and privacy
preservation by the use of blockchain-powered CFL.

###### Index Terms:

Blockchain, federated learning, decentralized learning, collaborated federated
learning, privacy, security, model poisoning, backdoor attacks, privacy-
preservation, model corruption.

##  I Introduction

The recent technological advancements in information and communication
technologies have completely transformed every human life, however, this
growth has also raised some serious security and privacy challenges. Machine
Learning (ML) is envisioned to be a promising solution to address the security
and privacy challenges and vulnerabilities in the existing and future digital
infrastructure. Moreover, the changing landscape of information and
communication systems (ICS) has brought forth unique security, privacy and
reliability challenges [1]. For instance, because of COVID-19, there has been
600 % increase in cybercrime where complex phishing attacks have been used and
many have fallen victim to it [2]. The malware attacks have seen tremendous
growth in last ten years along with 812.67 million malware attacks in 2018
alone. Same is the case with the malware attacks which rose 350% in 2018
worldwide [2]. Hence, our ICS are more prone to cyber attacks than ever.
Malicious actors also are using machine-powered sophisticated attacks to beat
the defense mechanism in place and any such breaches have proven to be
catastrophic for the victim network. Moreover, the privacy concerns are also
ever-high because of data breaches, theft and data leaks at top-tier digital
media companies. Many of such breaches resulted into sever consequences and
proven to be fatal as well in some cases like Ashley Madison data breach and
cyber attacks on hospitals [3].

###  I-A Machine learning for Security

The recent advancement in artificial intelligence (AI) and Machine Learning
(ML) has the capability protect the digital infrastructure against
sophisticated and ever-evolving cyber attacks [4]. Because of robust, dynamic
and real-time adaptive nature of ML models, they are well-suited as counter-
measures for machine-powered attacks from threat actors. The AI and ML has
open gateways in the domains of security, privacy, threat detection for ICS
and helped in developing effective methods for the protection of ICS. The
concept of using ML and AI in security and privacy is not new but the
importance of these ML and AI methods have been highlighted by the evolution
of deep learning algorithms. Most of the methods before the development of
deep learning were dedicated to model the attack patterns with certain
characteristic which are not robust in nature, but with AI and ML, it is
expected that systems will become more resilient towards new sophisticated
attacks with changing characteristics [5].

The ML is showing the positive impact in information security field by
adopting ML algorithms to address the security and privacy issues. The
information security industry is generating more and more data which open them
to advance threats and AI could be a powerful antidote for these threats. The
first generation of ML solutions are focused on scrutinizing data, detect
threats and assist human in remediation plan. The second generation of ML
solution will make the systems more autonomous and leave the critical support
issues to humans. There have been different types of ML model implementations
based on where the model will be trained and deployed. This seggregation is
evolved because of the privacy issues when transferring ML model or its
parameters and training data between the central ML model and edge devices.
The edge devices can be of different types. For instance, the most resource
constrained yet high in number are Internet of Things (IoTs). Similarly, the
ML models may also need to communicate with other use cases like Internet of
Health Things (IoHTs), Internet of Vehicles (IoVs), Industrial Internet of
Things (IIoTs) and Unmanned Air Vehicles (UAVs). Based on how the ML model is
implemented and information is communicated between ML controllers and edge
devices, different ML implementation types are explained in next section along
with privacy and security issues raising from these implementations.

Figure 1: Centralized machine learning model implementation from central
controller to edge devices and the possible vulnerabilities.

##  II Machine learning Implementation Types

The ML models can be implemented in different ways. Among these include
centralized implementation of global ML models, and distributed implementation
of ML known as Federated Learning (FL). There is also Collaborated FL which
does not require a direction link between the edge devices and central model
aggregator. In this section, we will explain these three types of
implementation, security and privacy concerns in these implementations.

###  II-A Centralized ML

In this type of ML implementation, the model is selected at centralizer ML
server and all the edge devices need to transmit their private data to the
central server for ML model training. The central server and controller trains
the ML model from the data sent by the edge devices. Once the model is trained
and ready of implementation, the central controller than sends the model to
the edge devices for implementation. There are multiple security and privacy
issues with centralized ML implementation,

  1. 1.

The private data of edge devices can be very sensitive and prone to theft
during the transmission. Especially, data-sensitive applications and use cases
like IoHTs and IoTs may have serious privacy concerns.

  2. 2.

The trained ML model can be intercepted and modified during the transmission
from central server to the edge devices. Such malicious attack may cause
severe consequences as the edge devices may not even detect that the ML model
has been compromised.

  3. 3.

The ML model in itself is pron adversarial type attacks and the compromised
data can be fed when intercepted during transmission from edge devices to
corrupt the ML model.

  4. 4.

The central model could also also be a single point of failure and attacking,
risking the entire system.

  5. 5.

Another concern for resource-constrained edge devices in IoT, and IoHTs setups
is that it will be challenging to transmit their data while having limited
battery powers.

Although centralized ML has privacy issues, but it is well-suit for private
environments where entire end-to-end communication link is protected. The
centralized ML model implementation scenario and exchange of data and model
parameters along with their privacy issues is highlighted in Fig. 1. More
detailed insights on threat surface and attack scope in different parts of ML
architecture can be found in [6]. Because of the privacy issues in centralized
ML implementation, Google proposed an idea of distributed implementations of
ML model known as Federated Learning (FL) [7].

###  II-B Federated Learning

The FL is a distributed implementation of centralized ML. The primary
advantage of FL is privacy-preservation for edge devices because edge devices
does not have to transmit their private data to central controller. The edge
devices have their own local ML model while global ML model resides in central
controller. Edge devices use their private data and train their local model
and then send their model parameters to the central controller which then
develops aggregated ML model (shared optimal solution) from all of the
received models from the edge. The aggregated model is finalized at the
central controller and model updates along with the model parameters are sent
to the edge devices. Edge devices can also request for the updates from the
central controller. Hence, this type of distributed ML model implementation
preserves the privacy of private data of edge devices as they don’t need to
transmit it to the central controller [8]. Moreover, this also significantly
reduces the network bandwidth which is consumed for the transmission of
private data from edge to ML server. This reduces the data theft and
associated risks for edge devices. The FL architecture along with its
advantages and security issues is presented in Fig. 2.

Figure 2: The distributed implementation of ML model using FL along with its
privacy advantages and risks.

###  II-C Collaborated Federated Learning

In FL, all the edge devices need to have a link or connection with the central
controller. This may not be possible for some devices in remote locations or
for the devices relying on the sink or relays for their data transmission.
Authors in [9] proposed concept of collaborated FL (CFL) in wireless networks
where edge devices can collaborate with one-another and use them as relays to
forward their model updates, trained models and communication with the model
aggregate. This CFL extends the distributed ML for remote devices which do not
have a direct connection with the model aggregator. The CFL has more practical
applications and extends the privacy-related benefits of ML by making it
scalable. The working mechanism of CFL is presented in Fig. 3.

Figure 3: The collaborated federated learning model.

###  II-D Security and privacy issues in FL and CFL

The FL and CFL improves the privacy related issues and extends the advantages
of ML modelling to the edge. The private data is protected for the edge device
as they don’t need to transmit it, however the ML model parameters are still
being exchanged between model aggregator and edge devices. If this exchange is
not secure, consensus-based, and fully secure, then this could still be a
major security and privacy risk. The man-in-the-middle could still modify the
model exchange parameters, hence making the FL and CFL implementation prone to
an external attacks. Hence, there should be a secure, decentralized and
consensus-based ML model sharing mechanism which should be trusted by all the
devices is needed.

Figure 4: Decentralized and distributed blockchain mechanism which uses chain
of blocks shared among all the participating devices.

##  III Blockchain

Blockchain is a decentralized, secure and consensus based information sharing
technology which comes with its inherent security and privacy benefits.
Devices in blockchain use blocks to add new transactions and share it with
other peers using blocks or ledgers, hence all known as distributed ledger
technologies (DLT). All the devices in peer-to-peer (P2P) distributed networks
are able to verify the integrity of blocks and transactions. Each and every
block is securely connected with it’s previous and next blocks with strong
cryptographic chain which can be verified by all the devices using the
blockchain. Hence, making it very secure and hard to tamper with as multiple
devices have accurate records of block and tampered blocks can easily be
identified and removed by consensus mechanism. The realization of blockchain
mechanism is shown in Fig. 4.

Blockchain technology has proven to be potential candidate in privacy-
sensitive applications [10]. Based on access to certain blockchain network,
applications of block chain can mainly be categorized in to public, and
private blockchains. Some of the important performance and security advantages
of using blockchain are shown in Fig. 5.

Figure 5: Advantages of using blockchain.

##  IV Blockchain powered FL

The security and privacy challenges related to ML model updates, request and
data sharing in FL and CFL can potentially be addressed by using blockchain
consensus based mechanism. Devices in private network implementing their local
ML models with model updates from global ML model can use blockchain DLT to
strengthen the privacy against external attacks. Block chain for FL has been
considered a reliable, safe and secure mechanism for devices to share the ML
model updates and parameters [11]. On-device blockchain based federated
learning model (BlockFL) has been studied in the literature [12]. Blockchain
powered FL (BlockFL) can also be used for secure information sharing among
IoVs [13, 14].

##  V Research Challenges

Most of the existing works in the literature have considered BlockFL for
privacy-preservation in consensus based data sharing applications. To the best
of our knowledge, none of the existing works are using BlockFL for detection,
investigation, and analysis of model-poisoning attacks, and backdoor attacks.
In this proposed research study, we want to investigate the feasibility of
BlockFl for,

  1. 1.

Detection, investigation and analysis of malicious ML model updates.

  2. 2.

Design and modelling of cyber risk assessment based on different types of
adversarial, backdoor and model poisoning attacks.

  3. 3.

Reliability and latency analysis of using blockchain.

  4. 4.

Customized and on-demand data aggregation and protection strategy.

  5. 5.

Different use-cases and applications.

###  V-A Blockchain powered CFL

Based on the research gap we have identified, we will be investigating the use
of blockchain for collaborated FL. To the best of our knowledge, none of the
existing works use BlockCFL mechanism for model updates among the edge
devices. Considering the scalability of CFL and more wider range of possible
application, using blockchain for model updates is expected to open some more
future research directions.

##  VI Conclusion and Future Work

In this work, we have studied the blockchain and federated machine learning
for safe, secure and reliable data transfer. Based on our research, we have
identified that blockchain can complement the performance advantages of
collaborated federated learning and could open new research directions for the
future. In the proposed research work, we intend to use block chain for CFL
(BlockCFL) and study it’s performance advantages in the context of security,
reliability and privacy. Moreover, in future work, we are also working on
developing the taxanomy of the existing works and their performance advantages
and shortcomings.

## References

  * [1] R. Tourani, S. Misra, T. Mick, and G. Panwar, “Security, privacy, and access control in information-centric networking: A survey,” IEEE communications surveys & tutorials, vol. 20, no. 1, pp. 566–600, 2017.
  * [2] “2021 cyber security statistics.” https://purplesec.us/resources/cyber-security-statistics/.  Accessed: 2021-05-07.
  * [3] J. A. Lewis, Assessing the risks of cyber terrorism, cyber war and other cyber threats.  Center for Strategic & International Studies Washington, DC, 2002.
  * [4] T. C. Truong, I. Zelinka, J. Plucar, M. Čandík, and V. Šulc, “Artificial intelligence and cybersecurity: Past, presence, and future,” in Artificial Intelligence and Evolutionary Computations in Engineering Systems, pp. 351–363, Springer, 2020.
  * [5] R. Calderon, “The benefits of artificial intelligence in cybersecurity,” Economic Crime Forensics Capstones, vol. 36, 2019.
  * [6] N. Papernot, P. McDaniel, A. Sinha, and M. Wellman, “Towards the science of security and privacy in machine learning,” arXiv preprint arXiv:1611.03814, 2016.
  * [7] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konečnỳ, S. Mazzocchi, H. B. McMahan, et al., “Towards federated learning at scale: System design,” arXiv preprint arXiv:1902.01046, 2019.
  * [8] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019.
  * [9] M. Chen, H. V. Poor, W. Saad, and S. Cui, “Wireless communications for collaborative federated learning,” IEEE Communications Magazine, vol. 58, no. 12, pp. 48–54, 2020.
  * [10] A. Dorri, S. S. Kanhere, R. Jurdak, and P. Gauravaram, “Blockchain for iot security and privacy: The case study of a smart home,” in 2017 IEEE international conference on pervasive computing and communications workshops (PerCom workshops), pp. 618–623, IEEE, 2017.
  * [11] Y. Lu, X. Huang, Y. Dai, S. Maharjan, and Y. Zhang, “Blockchain and federated learning for privacy-preserved data sharing in industrial iot,” IEEE Transactions on Industrial Informatics, vol. 16, no. 6, pp. 4177–4186, 2019\.
  * [12] H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Blockchained on-device federated learning,” IEEE Communications Letters, vol. 24, no. 6, pp. 1279–1283, 2019.
  * [13] Y. Lu, X. Huang, K. Zhang, S. Maharjan, and Y. Zhang, “Blockchain empowered asynchronous federated learning for secure data sharing in internet of vehicles,” IEEE Transactions on Vehicular Technology, vol. 69, no. 4, pp. 4298–4311, 2020.
  * [14] S. R. Pokhrel and J. Choi, “Federated learning with blockchain for autonomous vehicles: Analysis and design challenges,” IEEE Transactions on Communications, vol. 68, no. 8, pp. 4734–4746, 2020.