# Energy-Efficient Federated Learning in Cooperative Communication within
Factory Subnetworks

Hamid Reza Hashempour, Gilberto Berardinelli, Ramoni Adeogun, Shashi Raj
Pandey This work was supported by the EU Horizon 2020 research and innovation
programme: Hamid Reza Hashempour, Gilberto Berardinelli and Ramoni Adeogun
under the “6G-SHINE” project (Grant 101095738) and Shashi Raj Pandey under the
“6G-XCEL” project (Grant 101139194). Department of Electronic Systems, Aalborg
University, Denmark
E-mail: {hrh, gb, ra<EMAIL_ADDRESS>
###### Abstract

This paper investigates energy-efficient transmission protocols in relay-
assisted federated learning (FL) setup within industrial subnetworks,
considering latency and power constraints. In the subnetworks, devices
collaborate to train a global model by transmitting their local models at the
edge-enabled primary access (pAP) directly or via secondary access points
(sAPs), which act as relays to optimize the training latency. We begin by
formulating the energy efficiency problem for our proposed transmission
protocol. Given its non-convex nature, we decompose it to minimize
computational and transmission energy separately. First, we introduce an
algorithm that categorizes devices into single-hop and two-hop groups to
decrease transmission energy and then selects associated sAPs. Subsequently,
we optimize the transmit power, aiming to maximize energy efficiency. To that
end, we propose a Sequential Parametric Convex Approximation (SPCA) method to
configure system parameters jointly. Simulation results show a 5% improvement
in convergence, significantly reduced outage, and at least a twofold savings
in total energy achieved by our proposed algorithm compared to single-hop
transmission.

###### Index Terms:

Federated learning, energy efficiency, subnetworks, Industrial Internet of
Things (IIoT).

##  I Introduction

The Industrial Internet of Things (IIoT) holds significant promise for
transforming industrial operations and driving future industrial
advancements[1]. Artificial intelligence (AI) has emerged as a cornerstone in
realizing intelligent IIoT applications, often requiring centralized data
collection and processing. However, in many practical scenarios, achieving
centralized processing poses challenges due to the scale of modern IIoT
networks and the requirements to maintain industrial data confidentiality[1].
Federated Learning (FL) has emerged as a compelling privacy-preserving
collaborative AI paradigm, particularly suited for intelligent IIoT networks.
FL coordinates multiple IIoT devices and machines for AI training at the
network edge without the need to share the actual data.[1, 2, 3, 4]. This is
done as follows. The FL server periodically selects users to participate in
each training round. These selected users compute training loss, update
weights, and transmit local models to the server. Upon receiving these models,
the server aggregates them and iterates the process until convergence. Thus,
FL enables distributed AI training while preserving data privacy and
scalability in IIoT environments [5, 6].

Short-range, low-power in-X subnetworks, intended for deployment in entities
such as robots, sensors, and production modules to replace wired control
infrastructure in IIoT environments, are currently under investigation by both
industry and academia[7]. IIoT devices within these subnetworks utilize an FL-
like approach for taming collective intelligence through collaborative
learning while transmitting data to an edge server. However, challenges like
signal blockages and fading, particularly from metallic machinery, as well as
device mobility, lead to delays in training/inferencing in the subnetwork,
conflicting with strict latency and reliability requirements.

Recent studies suggest that relay technology can address these challenges by
improving wireless communication system coverage and reliability without
additional power consumption [8]. Nevertheless, battery-powered IoT devices
used in FL have to carefully balance energy expenditures between local
training demands and the need to upload model updates to a central server.
This adds significant resource consumption for the devices, alleviated
otherwise through relays; thus, it necessitates efficient battery power
management at the device. Integrating relays into the system complicates the
interdependencies among various parameters, so a joint consideration of
wireless resource allocation and computing capacity is needed.

In this study, we focus on energy-efficient communication in FL with relays
within industrial subnetworks to address the above challenges. Our
contributions are summarized below.

  * $\bullet$

Firstly, we formulate the energy efficiency problem for our relay-assisted
Time Division Multiple Access (TDMA) transmission protocol. Given the high
coupling between variables, we partition the problem into minimizing
computational and transmission energy.

  * $\bullet$

Next, we devise a method to schedule sensors into single-hop and two-hop
transmission modes, each served by a corresponding relay based on their
Channel State Information (CSI), to maximize the rate.

  * $\bullet$

We then tackle the critical challenge of minimizing total transmit energy
given the timing constraints, i.e., tight cycle times for transmissions. To
address the inherent non-convexity of this optimization problem, we employ a
promising technique called Sequential Parametric Convex Approximation (SPCA).

To the best of our knowledge, previous literature has not delved into
integrating a relay-assisted TDMA transmission protocol with stringent time
constraints FL training within factory subnetworks.

The remainder of this paper is structured as follows: Section II introduces
the system model and the proposed communication protocol, while Section III
presents the proposed method for energy minimization. In Section IV,
simulation results are presented to validate the effectiveness of our
approach. Finally, conclusions are summarized in Section V.

##  II System Model

In an industrial subnetwork, $N$ sensors connect wirelessly to $K+1$ access
points (APs). One AP serves as the primary AP (pAP) and controls actuators,
while the remaining $K$ APs are secondary APs (sAPs) with only radio
capabilities. The sAPs form set $\mathcal{K}$, and sensors are in set
$\mathcal{N}$. Sensors can communicate directly with the pAP (single-hop) or
both pAP and sAP (two-hop cooperative). Devices scheduled for single-hop
transmission are in set $\mathcal{N}_{1h}$, and those for two-hop transmission
are in set $\mathcal{N}_{2h}$. APs and devices collaborate on an FL algorithm
over wireless networks for data analysis and inference. The FL model trained
by each device’s dataset is termed the local FL model, while the model
generated by the pAP using inputs from all devices is the global FL model.

###  II-A FL Model

In FL, users train a statistical model locally on the dataset stored on their
mobile devices. To enhance model performance, the aim of device $n$ is to
minimize the training loss by optimizing the weight parameter $\mathbf{w}_{n}$
concerning its local dataset $\mathcal{D}_{n}$ as follows:

$\displaystyle\displaystyle\min_{\mathbf{w}_{n}\in\mathbb{R}}F_{n}(\mathbf{w}_{n})=\sum_{j\in\mathcal{D}_{n}}l(\mathbf{w}_{n},x_{j},y_{j}).$
(1)

In this study, the Mean Squared Error (MSE) is employed to calculate the loss
function, with logistic regression used for model learning. After local
training at device $n$, the weight vector $\mathbf{w}_{n}$ is transmitted to
the pAP for model aggregation. A global model is then constructed as defined
in [9], characterized by

$\displaystyle\mathbf{\bar{w}}$
$\displaystyle=\dfrac{\sum_{n\in\mathcal{N}}D_{n}\mathbf{w}_{n}}{\sum_{n\in\mathcal{N}}D_{n}},$
(2)

where $D_{n}$ is the size of local data samples. Since the data is
distributed, solving (1) directly is generally challenging. Therefore, FL
typically employs an iterative algorithm to train a global model from the
devices. Specifically, in each round, the $n$th device computes the training
loss and then updates the weights using gradient descent as

$\displaystyle\mathbf{v}_{n}$ $\displaystyle=\mathbf{w}_{n}-\mu\nabla
F_{n}(\mathbf{w}_{n}),$ (3)

where $\mathbf{v}_{n}$ represents the updated model parameter of the $n$th
device, and $\mu$ denotes the learning rate. Following this, the updated local
models from multiple devices are collected and aggregated at the pAP.

###  II-B Wireless Communication Model

Fig. 1 shows a subnetwork example with 5 sensors and 3 APs. Sensors A, B, and
C operate in two-hop cooperative mode, while sensors D and E use single-hop
mode. In cooperative mode, the pAP receives packets from both the cooperative
link and a direct link, enabling it to combine energy received from sensors
and sAPs effectively.

All APs are assumed to be time-synchronized and use a shared frequency band.
Sensors are allocated time resources in a TDMA manner to prevent intra-cell
interference. This study concentrates solely on the uplink (UL), where each
sensor ($n=1,\cdots,N$) transmits a packet of $s$ bits to the pAP/sAPs over a
bandwidth of $W$ Hz. The pAP has to receive all sensor packets within a time
slot of $T$ seconds.

We assume the pAP knows all channel responses between sensors and the pAP/sAP
and between sAPs and the pAP. This information is obtained during a training
phase, during which channel responses are estimated at the pAP and sAPs
through transmission of reference sequences, with the latter reported to the
pAP. Channel responses are expected to remain stable in static sensor
scenarios. Thus, in practice, the training operation can be periodically
repeated at a relaxed frequency. The pAP scheduler utilizes complete CSI from
all subnetwork links to determine the subset of devices for single-hop and
two-hop transmission, their transmission rates, time resource allocation, and
transmit power.

Figure 1: System model for FL with multiple APs within a subnetwork.

We posit that communication reliability in a subnetwork can be improved by the
presence of sAPs and their forwarding capabilities, necessitating a tailored
communication protocol.

Our proposed protocol is illustrated in Fig. 2. The UL time slot is divided
into three sub-slots of variable duration, corresponding to the first phase of
the two-hop transmissions, the single-hop transmissions, and the second phase
of the two-hop transmission. Sensors belonging to the $\mathcal{N}_{2h}$ set
then use the first phase sub-slot to transmit their packet, which is received
by both the pAP and sAP. We assume that each sensor in $\mathcal{N}_{2h}$ is
only served by the sAP for which it experiences the most advantageous channel
conditions. The sAP uses the second phase sub-slot to forward the received
message from the sensors to the pAP. We assume that the sAP acts as a Decode-
and-Forward (DF) relay. As mentioned above, the pAP can combine energy from
the reception of the signal transmitted by sensors in $\mathcal{N}_{2h}$
during the first phase transmission with the energy received by the sAPs
before decoding. Transmissions by sensors scheduled in the single-hop sub-slot
are only received by the pAP.

Figure 2: An implementation of an FL algorithm via proposed cooperative TDMA
protocol

Below, we outline the signal model for single-hop and two-hop DF relaying. For
notation, $\mathbf{h}_{c}\in\mathbb{C}^{K\times 1}$ represents the vector of
channel responses between the pAP and each of $K$ sAPs. The channel from the
pAP to $N$ devices is denoted as $\mathbf{h}_{p}\in\mathbb{C}^{N\times 1}$,
while the matrix $\mathbf{H}_{s}\in\mathbb{C}^{N\times K}$ represents the
channels from $N$ devices to $K$ sAPs.

####  II-B1 Signal model for single-hop transmission

The Signal-to-Noise Ratio (SNR) of the $n$th device in the direct link to the
pAP is expressed as

$\displaystyle g_{n}^{d}$
$\displaystyle=\dfrac{P_{n}|\mathbf{h}_{p}(n)|^{2}}{\sigma_{0}^{2}},\ \
\forall n\in\mathcal{N}_{1h}.$ (4)

$\sigma_{0}$ represents the standard deviation of the noise, and $P_{n}$
denotes the transmit power of device $n$. The attainable information rate of
the $n$th device in the direct link to the pAP is expressed as

$\displaystyle r_{n}^{d}$
$\displaystyle=\mathrm{log}_{2}\left(1+g_{n}^{d}\right),\ \ \forall
n\in\mathcal{N}_{1h}.$ (5)

####  II-B2 Signal Model for DF relaying

We examine the cooperative link, where transmissions from devices are first
received by sAPs, decoded, and then retransmitted to the pAP. We propose a
relaying scheme where each device is assigned the strongest sAP among all
available ones. This strategy, termed the ‘$1$ of $K$’ relaying method,
provides a flexible framework for improving communication reliability and
extending coverage. We define the index of the strongest sAP for the $n$th
device as $k^{*}_{n}$, where $k^{*}_{n}\in\mathcal{K}$. The SNR and achievable
rate of the $n$th device at the $k^{*}_{n}$th sAP are given, respectively, by

$\displaystyle g_{n,k^{*}_{n}}$
$\displaystyle=\dfrac{P_{n}|\mathbf{H}_{s}(n,k^{*}_{n})|^{2}}{\sigma_{0}^{2}},\
\ \forall n\in\mathcal{N}_{2h},\ \forall k^{*}_{n}\in\mathcal{K}$ (6)
$\displaystyle r_{n,k^{*}_{n}}^{(1)}$
$\displaystyle=\mathrm{log}_{2}\left(1+g_{n,k^{*}_{n}}\right),\ \ \forall
n\in\mathcal{N}_{2h},\ \forall k^{*}_{n}\in\mathcal{K}$ (7)

where the superscript ${(1)}$ indicates the first phase in the two-hop
cooperative transmission. Assuming that the sAP re-encodes and transmits the
received signal to pAP, the SNR and the achievable rate from the $k^{*}_{n}$th
sAP to the pAP are respectively given by

$\displaystyle g_{k^{*}_{n},p}$
$\displaystyle=\dfrac{P_{k^{*}_{n}}^{s}|\mathbf{h}_{c}(k^{*}_{n})|^{2}}{\sigma_{0}^{2}},\
\forall n\in\mathcal{N}_{2h},\ \forall k^{*}_{n}\in\mathcal{K},$ (8)
$\displaystyle r^{(2)}_{n}$
$\displaystyle=\mathrm{log}_{2}\left(1+g_{k^{*}_{n},p}+g_{n}^{d}\right),\
\forall n\in\mathcal{N}_{2h},\ \forall k^{*}_{n}\in\mathcal{K},$ (9)

where the superscript ${(2)}$ denotes the second phase in the two-hop
cooperative transmission. Also, $P_{k^{*}_{n}}^{s}$ is the transmit power of
the $k^{*}_{n}$th sAP. Equation (9) accounts for energy combining of the two
transmission phases at the pAP receiver.

###  II-C Processing and Transmission Model

The FL process between devices and their serving pAP is depicted in Fig. 1.
This procedure comprises three steps at each iteration: local computation at
each device (with multiple local iterations), transmission of the local FL
model for each device, and aggregation and broadcast of results at the pAP.
Each device computes its local FL model using its local dataset and the
received global FL model during the local computation step.

####  II-C1 Local Computation

Let $f_{n}$ denote the computation capacity of device $n$, quantified by the
number of CPU cycles per second. The computation time required at device $n$
for data processing is given by

$\displaystyle\tau_{n}=\dfrac{I_{n}C_{n}D_{n}}{f_{n}},\ \forall
n\in\mathcal{N}.$ (10)

Here, $C_{n}$ (cycles/sample) represents the number of CPU cycles needed to
compute one sample of data at device $n$, and $I_{n}$ stands for the number of
local iterations at device $n$. As per [10], the energy consumption for
performing a total of $C_{n}D_{n}$ CPU cycles at device $n$ is

$\displaystyle E_{n1}^{L}=\kappa C_{n}D_{n}f_{n}^{2},$ (11)

where $\kappa$ denotes the effective switched capacitance dependent on the
chip architecture. To compute the local FL model, device $n$ must perform
$C_{n}D_{n}$ CPU cycles across $I_{n}$ local iterations, resulting in the
total local computation energy at device $n$ as

$\displaystyle E_{n}^{L}=I_{n}E_{n1}^{L}=\kappa I_{n}C_{n}D_{n}f_{n}^{2}.$
(12)

####  II-C2 Wireless Transmission

In this phase, device $n$ must transmit the local FL model to the pAP. Since
the dimensions of the local FL model remain constant for all devices, each
device’s data size for upload remains consistent, denoted as $s$. Considering
time division multiplexing over a bandwidth $W$, a packet of $s$ bits for the
$n$th device can be transmitted within $\dfrac{s}{Wr_{n}^{d}}$ time.
Transmitting packets from all single-hop devices in a TDMA manner results in a
total transmission time

$\displaystyle T_{1h}$
$\displaystyle=\sum_{n\in\mathcal{N}_{1h}}\dfrac{s}{Wr_{n}^{d}}.$ (13)

To execute DF, the signal from device $n$ must be accurately decoded by the
strongest sAP and then re-encoded into a new message. The duration of over-
the-air time required for successful transmission of a packet containing $s$
bits by device $n$ is therefore
$\dfrac{s}{Wr_{n,k^{*}_{n}}^{(1)}}+\dfrac{s}{Wr_{n}^{(2)}},\ \forall
n\in\mathcal{N}_{2h}$. Let’s denote the total transmission time for all
devices in the first and second phases of the two-hop method as

$\displaystyle T_{2h}^{(1)}$
$\displaystyle=\sum_{n\in\mathcal{N}_{2h}}\dfrac{s}{Wr_{n,k^{*}_{n}}^{(1)}},$
(14) $\displaystyle T_{2h}^{(2)}$
$\displaystyle=\sum_{n\in\mathcal{N}_{2h}}\dfrac{s}{Wr_{n}^{(2)}}.$ (15)

The total transmission time in both single-hop and two-hop cases is then
calculated as

$\displaystyle T^{\mathrm{UL}}$
$\displaystyle=T_{1h}+T_{2h}^{(1)}+T_{2h}^{(2)},$ (16)

To transmit data of size $s$ within a time duration $T^{UL}$, the wireless
transmit energy will be given by

$\displaystyle E^{T}$
$\displaystyle=\dfrac{s}{W}\left[\sum_{n\in\mathcal{N}_{1h}}\dfrac{P_{n}}{r_{n}^{d}}+\sum_{n\in\mathcal{N}_{2h}}\left(\dfrac{P_{n}}{r_{n,k^{*}_{n}}^{(1)}}+\dfrac{P_{k^{*}_{n}}^{s}}{r_{n}^{(2)}}\right)\right].$
(17)

Considering the FL model outlined above, each user’s energy consumption
comprises both local computation energy $E_{n}^{L}$ and wireless transmission
energy $E^{T}$. Let’s denote the number of global iterations as $I_{0}$. Then,
the total energy consumption of all users participating in FL will be

$\displaystyle E$
$\displaystyle=I_{0}\left(E^{T}+\sum_{n\in\mathcal{N}}E_{n}^{L}\right)$ (18)

Each device $n\in\mathcal{N}$ possesses a local dataset. Hereafter, the total
time $T^{c}$ required to execute the FL algorithm is referred to as the
completion time. The completion time of each user includes both local
computation time and transmission time, as illustrated in Fig. 2. Based on
(10), the completion time $T_{n}^{c}$ of device $n$ will be

$\displaystyle T_{n}^{c}$
$\displaystyle=I_{0}\left(\tau_{n}+T^{\mathrm{UL}}\right)$ (19)

Let $T^{\mathrm{th}}$ be the maximum allowable time for training the entire FL
algorithm, then we have $T_{n}^{c}\leq T^{\mathrm{th}}$. Thus, the system
outage probability of the FL is given by

$\displaystyle
P_{out}=\dfrac{1}{N}\sum_{n\in\mathcal{N}}\mathrm{Pr}[T_{n}^{c}>T^{\mathrm{th}}].$
(20)

##  III Proposed Method for Energy Minimization

In this section, we establish the energy minimization problem for FL. Given
the nonconvex nature of the problem, obtaining the globally optimal solution
is challenging. Therefore, we propose a low-complex iterative algorithm to
address the energy minimization problem.

###  III-A Problem Formulation

Our objective is to minimize the total energy consumption of all users while
adhering to a latency constraint. This energy-efficient optimization problem
can be formulated as follows:

$\displaystyle\min_{\mathbf{P},\mathbf{f},\mathcal{N}_{1h},\mathcal{N}_{2h},\mathcal{D}}E,$
(21a) $\displaystyle\mathrm{s.t.}\ $ $\displaystyle
I_{0}\left(\dfrac{I_{n}C_{n}D_{n}}{f_{n}}+T^{\mathrm{UL}}\right)\leq
T^{\mathrm{th}},$ (21b) $\displaystyle 0\leq P_{n}\leq P_{\max},\ \forall
n\in\mathcal{N}$ (21c) $\displaystyle 0\leq P_{k^{*}_{n}}^{s}\leq P_{\max},\
\forall n\in\mathcal{N}_{2h}$ (21d) $\displaystyle 0\leq f_{n}\leq
f_{n}^{\max},\ \forall n\in\mathcal{N}$ (21e)

where
$\mathbf{P}=[P_{1},P_{2},\ldots,P_{N},P_{k^{*}_{1}}^{s},P_{k^{*}_{2}}^{s},\ldots,P_{k^{*}_{N_{2h}}}^{s}]$
represents the vector of transmit powers for all devices and sAPs cooperating
in transmission. $P_{\text{max}}$ and $f_{n}^{\max}$ represent the maximum
allowable transmission power and the maximum computational capacity of device
$n$, respectively. Constraint (21b) addresses the requirement for low latency,
while (21c) and (21d) set the power limits.

The formulated energy efficiency problem poses a challenge due to its non-
convexity and the strong coupling among decision variables. Therefore, we
initially decouple the problem into the computing resource management problem
and the transmit energy efficiency problem.

###  III-B Designing the Device CPU Frequency

Initially, we optimize the frequency once the number of training iterations
required to achieve a specific accuracy is known. Minimizing the total energy
consumption across all devices is equivalent to minimizing the individual
energy consumption of each device. For each device, $E_{n}^{L}$ is an
increasing function with respect to $f_{n}$. The time constraint (21b) of
Problem (21) implies that each device should operate at the lowest frequency
$f_{n}^{*}$ permitted by the delay constraint. Thus, we obtain

$\displaystyle
f_{n}^{*}=\dfrac{I_{n}C_{n}D_{n}}{T^{\mathrm{th}}/I_{0}-T^{\mathrm{UL}}}.$
(22)

###  III-C Proposed Method for Relay Selection and Transmit Power Control

After minimizing the computation energy $E_{n}^{L}$, the formulated problem
transforms into the energy transmission efficiency problem, expressed as
follows:

$\displaystyle\min_{\mathbf{P},\mathcal{N}_{1h},\mathcal{N}_{2h},\mathcal{D}}\left[\sum_{n\in\mathcal{N}_{1h}}\dfrac{P_{n}}{r_{n}^{d}}+\sum_{n\in\mathcal{N}_{2h}}\left(\dfrac{P_{n}}{r_{n,k^{*}_{n}}^{(1)}}+\dfrac{P_{k^{*}_{n}}^{s}}{r_{n}^{(2)}}\right)\right],$
(23a) $\displaystyle\mathrm{s.t.}\ $ $\displaystyle T^{\mathrm{UL}}\leq
T^{\prime},$ (23b) $\displaystyle 0\leq P_{n}\leq P_{\max},\ \forall
n\in\mathcal{N}$ (23c) $\displaystyle 0\leq P_{k^{*}_{n}}^{s}\leq P_{\max},\
\forall n\in\mathcal{N}_{2h}$ (23d)

where $T^{\prime}$ is the transmission time requirement obtained from (21b).
Considering the complexity of jointly selecting relays and minimizing
transmission energy, our approach initially identifies the optimal
transmission link before minimizing the transmit energy. Assuming constant
power, we reformulate the objective to identify the link that maximizes the
transmission rate, thereby reducing total delay. Following Laneman et. al.,
[11], the average rate in DF relaying is given by:
$1/2\min\left\\{r_{n,k^{*}_{n}}^{(1)},r_{n}^{(2)}\right\\}$. Assuming constant
and equal power for $P_{n}$ and $P_{k^{*}_{n}}^{s}$, we calculate the minimum
channel gain over both transmission phases for all potential relays, selecting
the link with the highest gain., i.e., $\forall n\in\mathcal{N},\forall
k\in\mathcal{K}$ we have

$\displaystyle\mathbf{G}^{2h}(n,k)\triangleq\frac{1}{2}\min\left\\{|\mathbf{h}_{c}(k)|^{2},|\mathbf{H}_{s}(n,k)|^{2}\right\\},$
(24) $\displaystyle\mathbf{g}^{2h}(n)\triangleq\max_{k}\mathbf{G}^{2h}(n,k).$
(25)

This metric is then compared with the direct channel gain
$|\mathbf{h}_{p}(n)|^{2}$ to determine whether direct or cooperative
transmission is more advantageous. If two-hop transmission is deemed
preferable, the sAP with the maximum channel gain will be selected for
cooperative transmission. The detailed procedure is outlined in Algorithm 1.

Algorithm 1 Algorithm for classification of devices and relay selection

1:Input: The channel gains $\mathbf{h}_{c}(k)$, $\mathbf{H}_{s}(n,k)$, and
$\mathbf{h}_{p}(n)$ for all $N$ devices and $K$ sAPs.

2:for $n=1:N$ do

3: calculate $\mathbf{g}^{2h}(n)$ from (25)

4: if $|\mathbf{h}_{p}(n)|^{2}\geq\mathbf{g}^{2h}(n)$ then

5: $n\rightarrow\mathcal{N}_{1h}$

6: else

7: $n\rightarrow\mathcal{N}_{2h}$

8: $\arg\max_{k}\\{\mathbf{G}^{2h}(n,:)\\}\rightarrow k^{*}_{n}$

9: end if

10:end for

11:Output: $\mathcal{N}_{1h}$, $\mathcal{N}_{2h}$, $\mathcal{K}^{*}$.

Given the sets $\mathcal{N}_{1h}$, $\mathcal{N}_{2h}$, and $\mathcal{D}$, we
proceed to minimize power consumption. This is achieved by simplifying the
optimization problem outlined in (23) as follows:

$\displaystyle\min_{\mathbf{P}}\left[\sum_{n\in\mathcal{N}_{1h}}\dfrac{P_{n}}{r_{n}^{d}}+\sum_{n\in\mathcal{N}_{2h}}\left(\dfrac{P_{n}}{r_{n,k^{*}_{n}}^{(1)}}+\dfrac{P_{k^{*}_{n}}^{s}}{r_{n}^{(2)}}\right)\right],$
(26a) $\displaystyle\mathrm{s.t.}\ $
$\displaystyle\sum_{n\in\mathcal{N}_{1h}}\dfrac{s}{W\mathrm{log}_{2}\left(1+{g}_{n}^{d}\right)}$
(26b)
$\displaystyle+\sum_{n\in\mathcal{N}_{2h}}\dfrac{s}{W\displaystyle\mathrm{log}_{2}\left(1+{g}_{n,k^{*}_{n}}\right)}$
$\displaystyle+\sum_{n\in\mathcal{N}_{2h}}\dfrac{s}{W\mathrm{log}_{2}\left(1+{g}_{k^{*}_{n},p}+{g}_{n}^{d}\right)}\leq
T^{\prime},$ $\displaystyle\eqref{P1-c}-\eqref{P1-d},$ (26c)

where (26b) refers to the total time constraint.

Problem (26) is hard to solve because of the non-convex constraint (26b), and
thus, finding the global optimum is typically intractable due to the non-
convex nature of the problem. To address this, we employ Sequential Parametric
Convex Approximation (SPCA), where a sequence of convex programs iteratively
approximates the problem. At each iteration, the non-convex constraint is
replaced by a convex surrogate, serving as an approximation. Consequently, we
can express (26) as follows:

$\displaystyle\min_{\mathbf{P},\mathbf{\omega},\mathbf{t}}\ E_{t}$ (27a)
$\displaystyle\mathrm{s.t.}\
\left[\sum_{n\in\mathcal{N}_{1h}}\dfrac{1}{t_{n}^{(1)}}+\sum_{n\in\mathcal{N}_{2h}}\left(\dfrac{1}{t_{n}^{(2)}}+\dfrac{1}{t_{n}^{(3)}}\right)\right]\leq
E_{t},$ (27b) $\displaystyle\dfrac{(\omega_{n}^{d})^{2}}{P_{n}}\geq
t_{n}^{(1)},\ \forall n\in\mathcal{N}_{1h}$ (27c)
$\displaystyle\gamma_{n}^{d}\geq(\omega_{n}^{d})^{2},\ \forall
n\in\mathcal{N}_{1h}$ (27d)
$\displaystyle\dfrac{(\omega_{n}^{(1)})^{2}}{P_{n}}\geq t_{n}^{(2)},\ \forall
n\in\mathcal{N}_{2h}$ (27e)
$\displaystyle\gamma_{n,k^{*}_{n}}^{(1)}\geq(\omega_{n}^{(1)})^{2},\ \forall
n\in\mathcal{N}_{2h}$ (27f)
$\displaystyle\dfrac{(\omega_{n}^{(2)})^{2}}{P_{k^{*}_{n}}^{s}}\geq
t_{n}^{(3)},\ \forall n\in\mathcal{N}_{2h}$ (27g)
$\displaystyle\gamma_{n}^{(2)}\geq(\omega_{n}^{(2)})^{2},\ \forall
n\in\mathcal{N}_{2h}$ (27h)
$\displaystyle\sum_{n\in\mathcal{N}_{1h}}\dfrac{1}{\gamma_{n}^{d}}+\sum_{n\in\mathcal{N}_{2h}}\left(\dfrac{1}{\gamma_{n,k^{*}_{n}}^{(1)}}+\dfrac{1}{\gamma_{n}^{(2)}}\right)\leq\dfrac{T^{\prime}W}{s},$
(27i)
$\displaystyle\mathrm{log}_{2}\left(1+g_{n}^{d}\right)\geq\gamma_{n}^{d},\ \
\forall n\in\mathcal{N}_{1h},$ (27j)
$\displaystyle\mathrm{log}_{2}\left(1+g_{n,k^{*}_{n}}\right)\geq\gamma_{n,k^{*}_{n}}^{(1)},\
\forall n\in\mathcal{N}_{2h},$ (27k)
$\displaystyle\mathrm{log}_{2}\left(1+g_{k^{*}_{n},p}+{g}_{n}^{d}\right)\geq\gamma_{n}^{(2)},\
\forall n\in\mathcal{N}_{2h},$ (27l) $\displaystyle\eqref{P1-c}-\eqref{P1-d}$
(27m)

where $E_{t}$ is the energy efficiency metric, $t_{n}^{(1)}$, $t_{n}^{(2)}$,
$t_{n}^{(3)}$, $\omega_{n}^{d}$, $\omega_{n}^{(1)}$, $\omega_{n}^{(2)}$,
$\gamma_{n}^{d}$, $\gamma_{n,k^{*}_{n}}^{(1)}$, and $\gamma_{n}^{(2)}$ are
auxiliary variables to approximate the non-convex terms with convex
counterparts. It can be perceived that $\gamma_{n}^{d}$,
$\gamma_{n,k^{*}_{n}}^{(1)}$, and $\gamma_{n}^{(2)}$ play the roles of lower
bound for $\mathrm{log}_{2}\left(1+{g}_{n}^{d}\right)$,
$\mathrm{log}_{2}\left(1+{g}_{n,k^{*}_{n}}\right)$, and
$\mathrm{log}_{2}\left(1+{g}_{k^{*}_{n},p}+{g}_{n}^{d}\right)$, respectively.
Increasing the lower-bound values and simultaneously reducing the upper bounds
will boost the left side of the constraints, which are needed here, so that
the constraints (27i)-(27l) would be active at the optimum. The (27i) is
convex since it is a linear combination of three quadratic terms over linear
functions that is convex [12]. The left side of (27c), (27e), and (27g) are
noncovex. To get rid of nonconvexity, we define the function
$\Omega^{[i]}(\omega,z)$, as the first-order lower approximation of them as
follows:

$\displaystyle\dfrac{\omega^{2}}{z}\geq\dfrac{2\omega^{[i]}}{z^{[i]}}\omega-(\dfrac{\omega^{[i]}}{z^{[i]}})^{2}z\triangleq\Omega^{[i]}(\omega,z),$
(28)

where $(\omega^{[i]},z^{[i]})$ are the values of the variables $(\omega,z)$ at
the output of the $i$th iteration. Affine approximations of constraints
(27j)-(27l), are given by

$\displaystyle 1+\rho_{n}-2^{\gamma_{n}^{d}}\geq 0,\ \forall
n\in\mathcal{N}_{1h},$ (29a)
$\displaystyle\rho_{n}\leq\dfrac{P_{n}|\mathbf{h}_{p}(n)|^{2}}{\sigma_{0}^{2}},\
\forall n\in\mathcal{N}_{1h},$ (29b) $\displaystyle
1+\psi_{n}-2^{\gamma_{n,k^{*}_{n}}^{(1)}}\geq 0,\ \forall
n\in\mathcal{N}_{2h},$ (29c)
$\displaystyle\psi_{n}\leq\dfrac{P_{n}\displaystyle|\mathbf{H}_{s}(n,k^{*}_{n})|^{2}}{\sigma_{0}^{2}},\
\forall n\in\mathcal{N}_{2h}$ (29d) $\displaystyle
1+\zeta_{n}-2^{\gamma_{n}^{(2)}}\geq 0,\ \forall n\in\mathcal{N}_{2h},$ (29e)
$\displaystyle\zeta_{n}\leq\dfrac{P_{k^{*}_{n}}^{s}|\mathbf{h}_{c}(k^{*}_{n})|^{2}+P_{n}|\mathbf{h}_{p}(n)|^{2}}{\sigma_{0}^{2}},\forall
n\in\mathcal{N}_{2h},$ (29f)

where $\rho_{n}$, $\psi_{n}$, and $\zeta_{n}$, are auxiliary variables.

Thus, by replacing constraints (27j)–(27l) with (29), and using
$\Omega^{[i]}(\omega,z)$ for approximating the left side of (27c), (27e), and
(27g), the optimization problem (27) transforms into a standard convex
semidefinite programming (SDP). This can be solved using numerical solvers,
such as the SDP tool in CVX [12].

##  IV Simulation Results

In our simulations, we consider a $3\times 3$ $\rm m^{2}$ subnetwork with 20
sensors representing a production module in a factory. Sensors and APs are
distributed uniformly within the subnetwork. Other key simulation parameters
are in Table I, except when noted otherwise. The wireless channels between
devices and APs, and between sAPs and pAPs, are modeled with independent
frequency-flat Rayleigh fading. The path loss model follows the factory and
open-plan building channel model from [13]. We use a shadow fading model with
a standard deviation of 7 dB, as per [14]. We test cooperative schemes with
different numbers of sAPs, including ‘1h’ and ‘1 of $K$’. The FL algorithm is
simulated using the Matlab Machine Learning Toolbox for handwritten digit
recognition. For this task, each user trains a feedforward neural network
(FNN) with 50 neurons using the MNIST dataset[15]. The loss function is cross-
entropy.

TABLE I: Federated learning parameters setup

Parameter description |  Value
---|---
Number of sAPs, $K$ |  Variant based on the scheme
transmit data size $s$ |  10 kbits
Local sample data size $D_{n}$ |  200 samples
Maximum completion time, $T^{\mathrm{th}}$ |  6 s
Uplink delay requirements $T^{\prime}$ |  4 ms
Bandwidth |  100 MHz
Carrier frequency |  10 GHz
$P_{\max}$ |  23 dBm
Power spectral density of the AWGN |  -174 dBm/Hz
The effective switched capacitance, $\kappa$ [10] |  $10^{-28}$
maximum computation capacity, $f^{\max}$ |  1 GHz
The number of CPU cycles, $C_{n}$ [10] |  $[1,2]\times 10^{4}$ (cycles/sample)

Fig. LABEL:fig3 illustrates the identification accuracy as a function of the
number of iterations for various transmission protocols. Because of the delay
constraint defined in (23), single-hop transmission may not always meet this
requirement and is therefore excluded from training in some scenarios. As a
result, our proposed cooperative communication method demonstrates superior
performance compared to single-hop transmission. Fig. LABEL:fig4 shows that
system outage performance improves as $P_{\max}$ increases. This is because
higher transmit power at users and relays reduce latency during model upload.
Additionally, having more sAPs reduces system outage probability by providing
more options to select the strongest transmission link.

(a)

(b)

Figure 3: (LABEL:fig3) Identification accuracy against the number of
iterations. (LABEL:fig4) Outage probability versus the maximum transmit power
per device and sAPs.

In Figure 4, we fixed the target accuracy ($I_{0}=130$, $I_{n}=1$) and plotted
total energy against the number of devices $N$ (ranging from 10 to 50),
comparing different communication schemes. The results indicate that
cooperative transmission outperforms single-hop transmission. Notably, the ‘1
of 4’ scheme uses at least two times less energy than single-hop transmission.

Figure 4: Total energy versus number of devices $N$ for different
communication schemes with fixed target accuracy.

##  V Conclusion

In this paper, we considered an energy-efficient federated learning framework
where devices upload their locally trained models in single-hop or two-hop
transmission. We proposed an efficient parameter optimization algorithm to
jointly optimize system parameters, such as the operating frequency of each
device, the single-hop and two-hop devices and the associated relays, and the
transmit power. The proposed SPCA algorithm can reasonably manage the energy
resources by balancing communication and local training costs. Our experiments
demonstrate that using four sAPs in federated learning can reduce total energy
consumption by at least half.

## References

  * [1] D. C. Nguyen et al., “Federated Learning for Industrial Internet of Things in Future Industries,” IEEE Wireless Communications, vol. 28, no. 6, pp. 192-199, December 2021
  * [2] E. Sisinni et al., “Industrial Internet of Things: Challenges, Opportunities, and Directions,” IEEE Trans. Industrial Informatics, vol. 14, no. 11, Nov. 2018, pp. 4724–34.
  * [3] Y. Qu et al., “A Blockchained Federated Learning Framework for Cognitive Computing in Industry 4.0 Networks,” IEEE Trans. Industrial Informatics, vol. 17, no. 4, April 2021, pp. 2964–73.
  * [4] M. Hao et al., “Efficient and Privacy-Enhanced Federated Learning for Industrial Artificial Intelligence,” IEEE Trans. Industrial Informatics, vol. 16, no. 10, Oct. 2020, pp. 6532–42.
  * [5] Z. Zhao et al., “Federated learning with non-IID data in wireless networks,” IEEE Trans. Wireless Commun., vol. 21, no. 3, pp. 1927–1942, Mar. 2022.
  * [6] Y. Zhan et al., “A survey of incentive mechanism design for federated learning,” IEEE Trans. Emerg. Top. Comput., vol. 10, no. 2, pp. 1035–1044, Mar. 2022
  * [7] G. Berardinelli et al., “Extreme Communication in 6G: Vision and Challenges for ’in-X’ Subnetworks,” IEEE Open Journal of the Communications Society, vol. 2, pp. 2516-2535, 2021
  * [8] X. Li, R. Fan et al., “Energy efficient resource allocation for mobile edge computing with multiple relays,” IEEE Internet Things J., vol. 9, no. 13, pp. 10732–10750, Jul. 2022.
  * [9] B. McMahan et al., “Communication-efficient learning of deep networks from decentralized data,” Proc. Int. Conf. Artif. Intell. Stat., Apr. 2017, pp. 1273–1282.
  * [10] Z. Yang et al., “Energy efficient federated learning over wireless communication networks,”  Trans. Wireless Commun., vol. 20, no. 3, pp. 1935–1949, Mar. 2021.
  * [11] J. N. Laneman et al.,“Cooperative diversity in wireless networks: Efficient protocols and outage behavior,” IEEE Transactions on Information Theory, vol. 50, no. 12, pp. 3062-3080, Dec. 2004
  * [12] M. Grant and S. Boyd, “CVX: Matlab software for disciplined convex programming, version 2.1,” http://cvxr.com/cvx, Mar. 2014.
  * [13] S. R. Khosravirad, H. Viswanathan, and W. Yu, “Exploiting Diversity for Ultra-Reliable and Low-Latency Wireless Control,” IEEE Trans. Wireless Commun., vol. 20, no. 1, pp. 316-331, Jan. 2021.
  * [14] 3GPP TR 38.901, v17.0.0, “Technical Specification Group Radio Access Network; Study on channel model for frequencies from 0.5 to 100 GHz,” 2022.
  * [15] Y. LeCun. The MNIST Database of Handwritten Digits. Accessed: Sep. 2020. [Online]. Available: http://yann.lecun.com/exdb/mnist/


# Exploring Retraining-Free Speech Recognition
for Intra-sentential Code-Switching

###### Abstract

In this paper, we present our initial efforts for building a code-switching
(CS) speech recognition system leveraging existing acoustic models (AMs) and
language models (LMs), i.e., no training required, and specifically targeting
intra-sentential switching. To achieve such an ambitious goal, new mechanisms
for foreign pronunciation generation and language model (LM) enrichment have
been devised. Specifically, we have designed an automatic approach to obtain
high quality pronunciation of foreign language (FL) words in the native
language (NL) phoneme set using existing acoustic phone decoders and an LSTM-
based grapheme-to-phoneme (G2P) model. Improved accented pronunciations have
thus been obtained by learning foreign pronunciations directly from data.
Furthermore, a code-switching LM was deployed by converting the original NL LM
into a CS LM using translated word pairs and borrowing statistics for the NL
LM. Experimental evidence clearly demonstrates that our approach better deals
with accented foreign pronunciations than techniques based on human labeling.
Moreover, our best system achieves a 55.5% relative word error rate reduction
from 34.4%, obtained with a conventional monolingual ASR system, to 15.3% on
an intra-sentential CS task without harming the monolingual recognition
accuracy.

Index Terms: multilingual speech recognition, code-switching, grapheme-to-
phoneme

##  1 Introduction

Code-switching (CS) is usually referred to as the situation where a speaker
alternates between different languages within a single conversation, e.g., [1,
2, 3, 4]. Code-switching can be broadly divided into two groups [5]: inter-
sentential switching - the alternation is between sentences (also called
extra-sentential), and intra-sentential switching - the alternation is within
sentences (it can also include intra-word). With the rapidly growing of
bilingual/multilingual population, CS is no longer a phenomenon relevant for
minority languages, which are affected by majority languages, but it also
concerns majority languages influenced by _lingua francas_ , such as English
and French, as properly pointed out in [6]. Despite the recent significant
advances witnessed in the field of automatic speech recognition (ASR) [7], ASR
systems have unfortunately still limited capability in tackling the code-
switching problem, especially intra-sentential switching. Major challenges for
CS speech recognition include different phone sets among languages, and
insufficient intra-sentential code-switching training data.

In recent years, we have witnessed an increasing research effort to address
the CS problem, and available approaches can mainly be divided into two
groups: robust acoustic modeling, and language modeling. For example, a global
phoneme set is constructed by merging phoneme sets from different languages,
and a unified multilingual acoustic model (AM) is trained in [8, 9, 10].
However, the goal of those studies on cross-language AMs is to develop ASR
systems for new languages for which scarce or no training material is
available. In the CS scenario, the native language (NL) AM should be used at
the runtime, and the recognition accuracy and speed for the NL ASR should not
be sacrificed. In [11], the authors compared four ASR approaches to tackle the
CS task under the constraint that the NL AM is to be used at runtime. In those
approaches, the lexicon in the foreign languages (FLs) are represented by the
NL phoneme set through phone/senone mapping/merging using either knowledge-
based or data-driven mapping techniques, and the NL acoustic model (AM) is
employed to carry out recognition. It was shown that top recognition results
were obtained by mapping each senone in the FL’s AM to the senone in the NL’s
AM with the minimum Kullback-Leibler divergence. From a language model (LM)
point of view, a statistical machine translation (SMT) based text generation
technique was used in [12] to artificially augment the amount of training data
to estimate more robust LMs. In [13], a multi-lingual LM is trained by
combining monolingual LMs into a probabilistic finite state grammar. More
recently, syntactic and semantic features were integrated into a factored
language model to combat data scarcity in code-switching speech [14].
Recurrent neural networks (RNNs) and factored language models are instead
combined in [15] for improved CS performance. In [16], the authors
successfully used similar word pairs to borrow information from available
(already trained) monolingual LMs and boost foreign low-frequency words. Since
code-switching deals with more than a language, it is natural to exploite
language identification, e.g., [17, 18, 19], or language diarization [6] to
select among monolingual speech recognizers. Language identification was also
employed in [12] to be combined with AM scores for better recognition
performance. However, those approaches suffers from errors propagated by the
language identification/diarization module to the ASR block.

In the afore-mentioned approaches, either ad-hoc acoustic and/or language
models have to be built to specifically handle the CS task, or multiple
language-dependent recognizers preceded by a language detection step have to
be employed. In this paper, we demonstrate that leveraging existing
monolingual ASR resources is possible to achieve effective intra-sentential
code-switching without retraining any of the ASR models, namely the acoustic
and language model, without sacrificing the recognition accuracy of the NL ASR
system. We attain such a challenging goal by devising new mechanisms for
foreign pronunciation generation, and LM enrichment. Specifically, we generate
a high quality foreign pronunciation lexicon by obtaining the pronunciation of
FL words in the NL phoneme set through acoustic phone decoding, and grapheme-
to-phoneme conversion (G2P). Differently from [11], where phone/senone mapping
and merging was the focus, we adopt data-driven techniques that learn foreign
pronunciations directly from data, which in turn allows us to better deal with
accented pronunciations. The experimental evidence interestingly demonstrates
that data-driven acoustic phone decoding and G2P approaches perform favorably
with respect to manual labeling. Moreover, we have obtained an enriched CS LM
taking inspiration from [16]. Specifically, we have converted the initial NL
LM into a CS LM by borrowing statistics/information from translated word
pairs, which allows us to avoid LM retraining. Comparing to the state-of-the-
art approach in [12], the proposed method doesn’t need to construct a new
phone set and retrain the AM, and also saves the efforts to retrain the LM.
The retained LM by collecting/generating new CS data as in [12] can easily
harm the recognition performance on NL. Experimental evidence on introducing
FL words to a NL system with vocabulary size in the hundreds of thousands
demonstrate that our best system is able to provide 55.5% relative word error
rate reduction (from 34.4% to 15.3%) on a code-switching testing set without
harming the performance on general NL test sets.

##  2 Foreign Pronunciation Generation

NL refers the language whose phoneme set and AM are used in recognition. All
other languages are indicated as FLs. For each FL word in a CS utterances, a
native pronunciation representation in the NL’s phoneme set is generated.

###  2.1 Linguist Manual Labeling

Manual labeling by experts can be the most straightforward way for this kind
of pronunciation generation task. Linguists with knowledge of the NL and a FL
can be hired to directly produce the lexicon of FL words with NL’s phoneme
set. The strength of this approach is pronunciations could be always
reasonable for human perception; the shortcoming could be that linguists tend
to label the FL words with the standard pronunciation, but native people
always has some extent of accent when they speak foreign words in a CS
scenario.

###  2.2 Phone Decoding

Phone decoding is a data-driven method to obtain pronunciations for the FL
words. When the audio segments containing FL words, a NL ASR system can be
used to decode those segments and thus obtain the possible phoneme sequences
of the FL words. The audio segments mainly come from two sources: foreign
people speaking FL words, and native people speaking FL words.

####  2.2.1 Phoneme Confusion Network

In harvesting audio segments for FL words, a particular word usually
corresponds to more than one audio segment. The decoded phoneme sequences for
those audio segments may vary a lot, and the process to obtain a good phoneme
representation by summarizing all decoded phoneme sequences becomes crucial.
In this paper, we adopt a ROVER-like [20] method based on phoneme confusion
networks (CNs) to accomplish the summarization process. For example, let’s
suppose that there exist three (3) audio segments containing the English word
“always,” and the phone sequences generated with a Chinese ASR system are:

  * •

OU W EI Z

  * •

OU W I Z

  * •

OU W EI S

Then the corresponding phoneme confusion network is displayed in Figure 1,
where the numbers in parentheses represent the number of times a phoneme
appears. From the phoneme confusion network in Figure 1, we can conclude that
the best phoneme sequence (with most votes) for “always” is “OU W EI Z,” and
the second best could either be “OU W EI S,” or “OU W I Z.” For each FL words,
we can retain an N-best representation.

Figure 1: Phoneme confusion network for the word “always”

####  2.2.2 Phone Decoding on FL words spoken by Foreign people (PDFF)

In this paper, we suppose we have an already-trained monolingual speech
recognition system for each language and there are plenty of corresponding
training utterances. To harvest audio segments for foreign words, we use the
FL speech recognition system to performance force alignment on its training
utterances, by doing so, we can obtain the start/end time stamps of each word
in those utterance. Then the audio segments containing concerned words are
decoded with NL speech recognition system generating possible phoneme
sequences for those words.

The advantage of this approach is that large amount of training data can be
utilized to generate a lot of possible pronunciation representations and
summaries them to get the best guesses, but a major concern is that those
foreign words are spoken by foreign people, so to some extend, we are actually
obtaining “standard” pronunciations while what we really care about is native
people speaking foreign words with possible accents. To obtain such
pronunciation guessing, we try to make use of the NL training data.

####  2.2.3 Phone Decoding on FL words spoken by Native people (PDFN)

Differently from FL training data, there are much fewer examples in NL
training data that contain foreign words. Since we don’t have native
pronunciation in the native phoneme set for those words, it is not possible to
obtain forced-alignments and find the corresponding audio segments (for those
words) as shown in Section 2.2.2. However, foreign words appearing in the NL
training data are the actual CS words, and although limited amount, these
words can still be fundamental to find more relative pronunciation guesses for
CS speech recognition. To obtain the audio segments of those foreign words in
NL training datasets, we first utilize the PDFF system (see Section 2.2.2) to
obtain an initial lexicon, and we then perform forced-alignment with such an
initial lexicon to find the location of those words. After finding those audio
segments, we can follow Section 2.2.2 to generate the pronunciation guesses.

###  2.3 Grapheme-to-Phoneme Conversion

Grapheme-to-Phoneme conversion(G2P) is the process of converting the written
form of a word to its pronunciation. In this paper, we use a Long Short-Term
Memory (LSTM) [21] based G2P model to translate the FL words from written form
to pronunciation representation in NL’s phoneme set. Our G2P model adopts an
encoder-decoder architecture with attention mechanism [22] which is different
from the LSTM G2P model proposed in [23]. There are two sources of data we
used for the G2P model training:

  * (a)

Standard NL lexicon containing linguist pronunciation labeling of both NL
words and FL words. The amount of FL words is limited in this scenario;

  * (b)

Phone decoding results on FL words spoken by native people (i.e., PDFN in
Section 2.2.3)

The training data (b) is added for the purpose of capturing the accents of NL
speakers when they speak FL words. To obtain training data (b), we went
through the following steps:

  * 1

Collect audio recordings for the 5000 most popular FL words, which are spoken
by NL speakers.

  * 2

Use G2P trained with data (a) to generate 4 pronunciations from written form
for each FL word used in Step1.

  * 3

Use a NL phone decoder to generate additional 4 pronunciations for each FL
word provided the audio collected in Step1.

  * 4

The combined 8 pronunciations from Step2 and Step3 are rescored using the NL
acoustic model, and the top 4 pronunciations with the highest acoustic model
scores are retained for each FL word.

Then (a) and (b) are combined to form the final training data for the LSTM G2P
model training.

##  3 Language Model Enriching

With the FL word pronunciation problem tackled, we still have to fix the LM
part to deploy a robust CS system. LM training is a crucial problem in CS
speech recognition, and the chief difficulty is in finding enough intra-
sentential CS training data to build a reliable CS LM. In fact, the CS
phenomenon does not happen as frequently in written form. However, it should
not be difficult to convince ourselves that the lack of abundant CS text
material is not a real concern, since the performance of the NL ASR system
would be harmed if too much CS data is used to train the LM. Therefore, we
started seeking into possible ways to overcome improve the LM in a CS system
without collecting ad-hoc CS LM data, and we the idea proposed in [16] to be a
sound solution to construct a CS LM. The idea is to borrow
statistics/information in an already-trained LM, and we adopt and extend this
idea in the present work.

In code-switching, we usually replaces a native word with the same meaning
word (translated counterpart) in foreign language. For example, the
monolingual Chinese utterance (meaning in English: “We play basketball”)

  * •

我们 打 篮球

can be spoken as

  * •

我们 打 basketball

in code-switching scenarios. The code-switched words are a translated word
pair “篮球–basketball”. Here “basketball” has the same meaning with “ 篮球 ”. In
this example, when we construct the CS LM, we add the English (FL) word
“basketball” into Chinese (NL) LM by copying the statistics/information of its
translated counterpart “ 篮球 ”. By doing so, we borrow statistics/information
of the word “篮球” in the NL LM for the added FL word “basketball”. The effort
of collecting/generating CS data and retraining LM can thereby be avoided
using such an elegant statistics-borrowing method.

Modern ASR systems usually use weighted finite-state transducers (WFST) [24]
to present the LM as well as other components. We therefore show how to
convert an NL LM into a CS LM in the context of WFST. In the LM WFST graph G,
each transition/edge corresponds to a word, and the weight of the edge
indicates the word’s probability. Figure 2 is a toy example of a native LM’s G
graph. When we convert this toy NL LM into a CS LM concerning only the word
“basketball” as follows: (i) Obtain the Chinese counterpart word “篮球” of
“basketball,” (ii) Find the edge in the WFST corresponding to the word “篮球,”
and (iii) Insert a parallel edge with the same weight into the WFST but
carrying the the foreign word “basketbal.” The above steps will result in the
CS G graph shown in Figure 3.

Figure 2: NL LM G Figure 3: CS LM G’ converted from NL LM G in Figure 2 by
adding FL word “basketball” Table 1: Lexicons used in the experimental
evaluation

Lexicons Configurations and Methods L0 original NL lexicon (no FL words) L1
224 FL words by linguist labeling in Section 2.1 L2f 1155 FL words by PDFF in
Section 2.2.2 L2n 224 FL words by PDFN in Section 2.2.3 L3a 1155 FL words by
G2P with data (a) in Section 2.3 L3ab 1155 FL words by G2P with data (a) and
(b) in Section 2.3

Table 2: LMs with different FL vocabularies (see Section 3)

LMs Configurations G0 original NL LM (no FL words) G1 CS LM with 224 FL words
G2 CS LM with 1155 FL words

##  4 Experimental Setup & Results

Our approach is validated using Chinese as the NL, and English as the FL.

###  4.1 Experimental Setup

Acoustic Model: The NL Chinese acoustic model is a Feed-Forward deep neural
network (FFDNN) trained with filter bank features from 160 hours Chinese
speech using cross-entropy and bMMI objective function. The FL English
acoustic model utilize similar setup.

Lexicon: Table 1 is the lexicons we obtained by methods proposed in Section 2.
The 1155 FL words are selected by their appearing frequencies in the English
training data and also the probability occurring during Chinese-English code-
switching. The 224 FL words is a subset of the 1155 words which actually occur
in the Chinese training data.

Code-Switching LM: Table 2 shows the code-switching LMs obtained following
Section 3. We refer to the original Chinese LM as G0, the CS LM contains the
224 English word as G1, and the CS LM contains all the 1155 words as G2.

Testing sets: There are two test sets. The code-switching test set contains
107 CS utterances. Each of the utterances contains 1 or more intra-sentential
code-switchings. A typical one is like “ 我们 打 basketball”. The FL words
included in the CS test set is mainly a subset of the 224 English words afore-
mentioend. The general test set contains 9677 utterances, most of which are
pure Chinese utterances.

Evaluation Metric: Experimental results are reported in terms of a slightly
modified version of the conventional word error rate (WER) metric, namely each
Chinese character is treated as a word in the contest of CS speech
recognition.

Table 3: WERs (in %) on CS test set by using different lexicon and LM
configurations. Word error rates reductions (WERRs) are given in parentheses.

1 Configuration WER on CS 2 L0+G0 (baseline) 34.4% 4 (L0+L2f)+G1 24.3% (29.4%)
3 (L0+L1)+G1 24.3% (29.4%) 5 (L0+L2n)+G1 20.7% (39.9%) 6 (L0+L3a)+G1 21.6%
(37.2%) 7 (L0+L3ab)+G1 18.8% (45.3%) 8 (L0+L1+L2f+L2n+L3ab)+G1 15.3% (55.5%) 9
(L0+L1+L2f+L2n+L3ab)+G2 15.3% (55.5%)

Table 4: WERs (in %) on the general test set.

Configuration WER on general L0+G0 (baseline) 14.5% (L0+L1+L2f+L2n+L3ab)+G1
14.5% (L0+L1+L2f+L2n+L3ab)+G2 14.6%

###  4.2 Experimental Results

Table 3 shows the CS performance of several systems having different lexicon
and LM configurations. The CS system employing the original NL lexicon, namely
L0, and the original LM, namely G0, is referred to as the baseline system. By
comparing the second and third row in in Table 3, we can easily argue that the
proposed PDFF for generating the lexicon of FL words (see Section 2.2.2) along
with enriched LM using 224 FL words (G1) delivers significant performance
improvement over the baseline system. Indeed, the WER is reduced from 34.4%
down to 24.3%, which is equivalent to a word error rate reduction (WERR) of
29.4%. The monolingual Chinese ASR system (baseline) cannot conversely
properly recognize any foreign words, which are treated as out of vocabulary
words. Moreover, we can observed that the proposed PDFF approach is
equivalent, in terms of recognition performance, to the manual labeling of an
expert (compare third and fourth rows). The latter might allow us to conclude
that the data-driven method - without taking into account speaker’s accents
yet, is comparable to an expensive human-driven solution. By taking into
account the speaker’s accent using the L2n lexicon, we can further reduce the
WER down to 20.7%, which represents a WERR of 39.9%, outperforming both human
labeling, and the PDFN approach that tries to only generate “standard”
pronunciations.

If an LSTM-based G2P approach trained on both the standard NL lexicon covering
manually labeled NL and FL words (i.e., set (a) in Section 2.3), and the phone
decoding results (i.e., set (b) in Section 2.3), a WER of 18.8% is attained,
as displayed in the seventh row. By comparing the latter result with that
shown in the sixth row in Table 3, which corresponds to G2P train on the set
(a) only, we can see that taking speaker’s accent into account has a
beneficial effect on the recognition performance. Merging together all the
pronunciation guesses from different sources brings the WER is down to 15.3%,
as shown in the eighth row in Table 3, which accounts for a WERR of 55.5%. The
latter outcome seems to imply that those pronunciation generation methods are
complementary to each other. Finally, we noticed that increasing the amount of
FL words in the proposed CS LM does not provide any further improvement, but
it does not harm the recognition performance on CS test set either. It is also
important to verify that the addition of ad-hoc CS lexicons and LMs does not
cause a drop in baseline ASR performance on pure Chinese utterances. To this
end, we ran a new set of experiments on a more general data set (see Section
4.1). The corresponding results are reported the results in Table 4, and these
results demonstrate that the performance degradation on a more general ASR
task is ignorable.

Finally, the effect of tuning the weight-copying scale in LM enriching is
given in Table 5. A scale factor is multiplied by the original NL edge’s
weight before being copied in the added parallel FL edge. A value greater than
1 implies a boost of the FL edge, whereas a value smaller than 1 suppresses
the FL edge. From Table 5, we can see that boosting FL edges improves the
recognition performance on the CS test, but it also causes a degradation on
the general test set. Suppressing FL edges won’t harm the performance on the
general set but causes a degradation on the CS set. That can easily be
understood by bearing in mind that errors on the CS test set are mainly caused
by unrecognized FL words; therefore, better results can be obtained by
boosting the FL edges. However, there are nearly no FL words in a more general
test set, and that explains the drop in performance.

Table 5: Effects on tuning the weight-copying scale with configuration of
(L0+L1+L2f+L2n+L3ab)+G2.

Scale WER on general WER on CS 1 14.5% 15.3% 1.5 14.9% 12.8% 0.667 14.5% 18.6%

##  5 Conclusion

In this work, we build a intra-sentential code-switching ASR prototype by
leveraging existing monolingual systems without either acoustic, or language
model retraining. Our approach focuses on data-driven pronunciation generation
for FL words and takes into account speakers’ accent. Moreover, we also devise
a simply CS LM generation approach by enriching existing monolingual LM
leveraging word pairs. The significant accuracy improvement on CS test data
together with retention of LM accuracy on a more general speech task
highlights the effectiveness of our idea. A major limitation of the proposed
approach concerns the LM enriching method, which may not handle the word
reordering problem. To clear ideas, let’s consider the English word “tomorrow”
that corresponds to the Chinese word “明天”. In “I will go to school tomorrow”
the adverb “tomorrow” appears at the end of the sentence, but the same
sentence in Chinese, namely “我 明天 去 学校“” would places “明天” in the middle of
the sentence. In a CS sentence having “tomorrow” as the FL word, ‘tomorrow”
would rarely appear in the middle of the sentence, as it should in Chinese,
and that may make the use of the statistics borrowed from the NL LM for the CS
LM not accurate. We will try to tackle this issue by combing the proposed
statistic-borrowing and the traditional data augmentation methods.

## References

  * [1] S. Poplack, “Sometimes ill start a sentence in spanish y termino en espa$\tilde{n}$ol: toward a typology of code-switching1,” _Linguistics_ , vol. 18, pp. 581–618, 1980.
  * [2] D. Sankoff and S. Poplack, “A formal grammar for code-switching,” _Research on Language & Social Interaction_, vol. 14, pp. 3–45, 1981.
  * [3] L. Milroy and P. Muysken, _One speaker, two languages: Cross-disciplinary perspectives on code-switching_. Cambridge University Press, 1995.
  * [4] P. Auer, _Code-switching in conversation: Language, interaction and identity_. Routledge, 2013.
  * [5] C. Myers-Scotton, _A formal grammar for code-switching_. Oxford: Clarendon Press, 1993.
  * [6] E. Yilmaz, M. McLaren, H. V. den Heuvel, and D. A. V. Leeuwen, “Language diarization fro semi-supervised bilingual acoustic model training,” in _Proc. ASRU_. ISCA, 2017, pp. 91–96.
  * [7] G. Hinton, L. Deng, D. Yu, G. Dahl, A. r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition,” _Signal Processing Magazine_ , 2012.
  * [8] S. Yu, S. Zhang, and B. Xu, “Chinese-english bilingual phone modeling for cross-language speech recognition,” in _Proc. ICASSP_ , vol. 1. IEEE, 2004, pp. I–917.
  * [9] C.-L. Huang and C.-H. Wu, “Generation of phonetic units for mixed-language speech recognition based on acoustic and contextual analysis,” _IEEE Trans. Computers_ , vol. 56, no. 9, pp. 1225–1233, 2007.
  * [10] H. Lin, L. Deng, D. Yu, Y. Gong, A. Acero, and C.-H. Lee, “A study on multilingual acoustic modeling for large vocabulary asr,” in _Proc. ICASSP_. IEEE, 2009, pp. 4333–4336.
  * [11] D. Yu, L. Deng, P. Liu, J. Wu, Y. Gong, and A. Acero, “Cross-lingual speech recognition under runtime resource constraints,” in _Proc. ICASSP_. IEEE, 2009, pp. 4193–4196.
  * [12] N. T. Vu, D.-C. Lyu, J. Weiner, D. Telaar, T. Schlippe, F. Blaicher, E.-S. Chng, T. Schultz, and H. Li, “A first speech recognition system for mandarin-english code-switch conversational speech,” in _Proc. ICASSP_. IEEE, 2012, pp. 4889–4892.
  * [13] F. Weng, H. Bratt, L. Neumeyer, and A. Stolcke, “A study of multilingual speech recognition.” in _Proc. EUROSPEECH_ , vol. 1997, 1997, pp. 359–362.
  * [14] H. H. Adel, N. T. Vu, K. Kirchhoff, D. Telaar, and T. Schultz, “Syntactic and semantic features for code-switching factored language models,” _IEEE Trans. Audio, Speech, and Language Processing_ , vol. 23, no. 3, pp. 431–440, 2015\.
  * [15] H. Adel, D. Telaar, N. T. Vu, K. Kirchhoff, and T. Schultz, “Combining recurrent neural networks and factored language models during decoding of code-switching speech.” in _Proc. INTERSPEECH_ , 2014, pp. 1415–1419.
  * [16] X. Ma, X. Wang, D. Wang, and Z. Zhang, “Recognize foreign low-frequency words with similar pairs,” in _Proc. INTERSPEECH_ , vol. 2015.
  * [17] A. Waibel, H. Soltau, T. Schultz, T. Schaaf, and F. Metze, “Multilingual speech recognition,” in _Verbmobil: Foundations of Speech-to-Speech Translation_. Springer, 2000, pp. 33–45.
  * [18] H. Lin, J.-T. Huang, F. Beaufays, B. Strope, and Y.-H. Sung, “Recognition of multilingual speech in mobile applications,” in _Proc. ICASSP_. IEEE, 2012, pp. 4881–4884.
  * [19] J. Gonzalez-Dominguez, D. Eustis, I. Lopez-Moreno, A. Senior, F. Beaufays, and P. J. Moreno, “A real-time end-to-end multilingual speech recognition architecture,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 9, no. 4, pp. 749–759, 2015.
  * [20] J. G. Fiscus, “A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (rover),” in _Proc. ASRU_. IEEE, 1997, pp. 347–354.
  * [21] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [22] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” 2015.
  * [23] K. Rao, F. Peng, H. Sak, and F. Beaufays, “Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks,” in _Proc. ICASSP_. IEEE, 2015, pp. 4225–4229.
  * [24] M. Mohri, F. Pereira, and M. Riley, “Weighted finite-state transducers in speech recognition,” _Computer Speech & Language_, vol. 16, no. 1, pp. 69–88, 2002.