# Improved long-form speech recognition by jointly modeling the primary and
non-primary speakers

###### Abstract

ASR models often suffer from a long-form deletion problem where the model
predicts sequential blanks instead of words when transcribing a lengthy audio
(in the order of minutes or hours). From the perspective of a user or
downstream system consuming the ASR results, this behavior can be perceived as
the model “being stuck”, and potentially make the product hard to use. One of
the culprits for long-form deletion is training-test data mismatch, which can
happen even when the model is trained on diverse and large-scale data
collected from multiple application domains. In this work, we introduce a
novel technique to simultaneously model different groups of speakers in the
audio along with the standard transcript tokens. Speakers are grouped as
primary and non-primary, which connects the application domains and
significantly alleviates the long-form deletion problem. This improved model
neither needs any additional training data nor incurs additional training or
inference cost.

Index Terms—  speech recognition, end-to-end-models, long-form deletion,
multi-domain

##  1 Introduction

End-to-end automatic speech recognition (ASR) systems [1, 2, 3] are capable of
handling different scenarios (e.g. a few seconds to hours in length, clean
audio vs noisy audio with background speech or music) and different downstream
applications such as voice assistants (Google, Siri, Alexa, etc.) and
captioning YouTube videos. These systems are trained with data from different
sources and noise conditions for robust performance [4, 5]. One problem for
such models is long-form deletion — sometimes the model introduces high
deletion errors when the input audio is long. In such cases, the user may
perceive the system as being stuck (no words are emitted intermittently),
which can significantly hurt the user retention rate of the end product.
Alternatively, if the transcript from ASR is consumed by downstream systems
for semantic understanding, the missing words can induce cascading errors.

There have been many efforts in the literature to understand the cause of
long-form deletions and attempt to improve general long-form quality. Some
studies indicate that training-test data mismatch (due to acoustics, noise,
audio length differences) is one cause [6, 7, 8]. The long-form deletion
problems could be introduced in both models with recurrent structure (e.g.
LSTM-based ASR [6, 7]) and non-recurrent structure (e.g. conformer based ASR
[8]). [9, 10, 11] attempts to use previous context as additional input to the
model to improve long-form performance. [12] indicates that model
architectures with global attention do not scale well to long-form audio and
propose smaller attention windows for better scalability. But they do not
explicitly target the long-form deletion problem directly. Although improved
model architectures can potentially alleviate the problem, we still sometimes
see that the problem exists. This paper looks at an alternative approach to
fix it.

In this work, we demonstrate that even a model without global attention and
trained with vast amount of diverse data can have significant long-form
deletion problem. We create a targeted testset and metric that shows the long-
form deletion clearly and analyze when the model exhibits the problem. We
identify that a model trained on data from different applications can exhibit
the long-form deletion problem for some domains more prominently than others.
Different application domains expect different types of speakers to be
transcribed. For example, for a voice assistant (e.g. Google, Siri), ASR is
expected to only transcribe the target user talking to the device (i.e. the
primary speaker) and ignore other speakers. However, when captioning a YouTube
video, ASR is expected to transcribe all speakers. These expectations are
reflected in the ground truth annotations of both training and test data. If
data from these domains are directly mixed together without special handling,
the conflicting expectations from different domains will cause confusions
during the model training when non-primary speakers are present — to
transcribe or not to transcribe, that is the question. These conflicting
training goals ultimately limit the model from learning cross-domain
capabilities, thus leading to long-form deletion errors.

To address the above mentioned problem, we simultaneously model the speakers
along with the ASR transcript tokens to create better synergy across domains.
We propose to expand the ASR output vocabulary with two novel tokens: <end-
primary> and <end-others>, and show how to train and run inference with ASR.

There are many approaches to model the speakers in an audio segment, such as
speaker recognition, speaker diarization, speaker turn detection, speaker-
attributed ASR, and speech separation. We will briefly review these work in
Section 2.3. But in summary, our work differs from those approaches by
explicitly identifying the primary speaker, transcribes audio in a single
inference pass without additional steps. Our work also differs with the end
goal being long-form deletion improvement as opposed to improving speaker
detection or speaker turn accuracy.

We investigate the Word Error Rate (WER), endpointing latency (EP) on multiple
testsets. Additionally, we measure the number of times 25 words are deleted in
sequence on the targeted testset for long-form deletion. With the proposed
technique, we show that the WER and EP are neutral compared to baseline and
the long-form deletion metric improves by 55%.

##  2 Method

###  2.1 Modeling primariness of speakers

A single end-to-end (E2E) ASR model serving different use cases is generally
preferable to separate models, which could be more expensive to train,
maintain and serve. Modern ASR systems are capable of training on large
amounts of diverse data and serve multiple downstream applications e.g. [5].
One way to group these applications would be:

  1. 1.

Short domain: User making a short request (few seconds) with voice assistant
(e.g. Alexa or Siri) or a short query in search bar (like Google voice
search).

  2. 2.

Dictation domain: User dictating a (potentially) long query (like composing an
email or message by speaking instead of typing).

  3. 3.

Caption domain: Generating captions (like for YouTube videos, live-streams).

For Short and Dictation, there is one unique speaker who should be considered
as the user and the ASR model is expected to transcribe only their speech. If
there is any background noise (e.g. TV noise, music) or other speakers, the
model should ignore them. For Caption, the goal is to transcribe all the
speech (e.g. a podcast with multiple participants). We would like to serve all
these domains (Short, Dictation and Caption) with a single E2E ASR model.
However, the different expected output for different domains in terms of whose
speech should be transcribed is a potential challenge for the model. We aim to
simplify this by explicitly modeling the different speakers in a given audio
segment. We group the speakers into Primary and Non-Primary. For Short and
Dictation, Primary is the target user and Non-Primary is all other speakers /
background noise. This distinction between primary and non-primary is
necessary, since for Short and Dictation the ASR model is expected to not
transcribe the non-primary speakers. The two groups can be modeled by the
addition of two new tokens to the ASR output vocabulary:

  1. 1.

<end-primary> \- Used to indicate when the primary speaker is done speaking.
Either there is a speaker turn after this event and other speakers begin
speaking; or this is the end of the audio.

  2. 2.

<end-others> \- Used to indicate when all other speakers are done speaking.
Either there is a speaker turn after this event and primary speaker speaks
again; or this is the end of the audio.

For instance, if the primary speaker says “Play music on … no cancel” and
another background speaker says “but we need to leave” during the utterance,
the expected ASR output would be “Play music on <end-primary> but we need to
leave <end-others> no cancel <end-primary>”. With this protocol, two APIs can
be exposed to serve all application domains. For Short and Dictation domains,
the ASR output (containing all speakers transcript and speaker-tag tokens) can
be processed to remove non-primary speakers transcript. And for the Caption
domain, the ASR output can be processed to contain all speakers’ transcript.
This is shown in Figure 1. Now, the ASR output is uniform for all domains
(Short, Dictation and Caption) i.e. transcribe all speakers and output the
speaker-tag tokens appropriately.

Fig. 1: Primary speaker text in blue, others text in yellow

We call these new tokens “speaker-tags”. Note that this does not explicitly
handle overlapping speech by both primary and a non-primary speaker. This is a
potential limitation that can be an area of research for future work. It’s
also worth noting that we can alternatively use <begin-primary> and <begin-
others> to indicate when a speaker starts speaking (instead of finishes
speaking). By definition, these would be equivalent to speaker-tags but we do
not pursue this as emitting tokens earlier rather than later could be more
difficult for the model.

###  2.2 Relabeling training data with speaker-tags

To create training data in the above format, we introduce a technique that can
relabel existing data without manual examination. The technique utilizes two
teacher models:

  1. 1.

600M parameter bidirectional teacher model trained with Short and Dictation
domain data only with the architecture and procedure described in [13]. In
particular, this model is trained only on supervised data from domains which
only preserve the primary speaker transcript. We call this “Teacher-Primary”.

  2. 2.

183.5M parameter bidirectional model trained on short-segments of YouTube data
as described in [14] and further finetuned with 1k hours of supervised YouTube
data. In particular, this model is trained on data from domains where all
(primary, secondary, background noise) speech has to be transcribed. We call
this “Teacher-All”.

For a given audio segment, suppose we know the transcript for just the primary
speaker (say Trans-Primary) and the whole transcript (say Trans-All). We do a
sub-sequence match between the two to add the two speaker tag tokens at
appropriate boundaries as shown in Figure 2.

Fig. 2: Technique based on sub-sequence match between transcripts. Trans-P is
Trans-Primary and Trans-A is Trans-All

For Short and Dictation training data, the available ground truth transcript
would be Trans-Primary. Trans-All can be obtained by running the Teacher-All.
Similarly, for Caption data, the available ground truth transcript would be
Trans-All. Trans-Primary can be obtained by running the Teacher-Primary.
Although Caption audio data may not have a clear primary speaker, the primary
speaker can be identified implicitly based on the matching.

For this process to succeed, there must be exactly one sub-sequence match
between Trans-Primary and Trans-All. This may not happen for several reasons:

  1. 1.

Teacher models make mistakes and human-transcription could be incorrect or
inconsistent resulting in no sub-sequence match.

  2. 2.

Normalization mismatches due to capitalization, punctuation and related
differences e.g. “where‘s the Eiffel-tower located” vs “Where is the Eiffel
tower located ?”.

  3. 3.

There might be multiple valid sub-sequence matches. e.g. Trans-Primary is “how
tall is Barack Obama” and Trans-All is “how tall is it is the end Barack
Obama” - where the word “is” can be matched in multiple ways leading to
different output transcripts.

For robust data relabeling, all punctuation and capitalization is denormalized
before matching. Additionally, any one word differences are not considered as
a mismatch (e.g. “how tall is Barack Obama” and “how tall is a Barack Obama”).
For other cases with no or multiple matches after these adjustments, the
original transcript truth is used directly without any speaker-tag tokens
inserted. Additionally, some training data could be very long and is segmented
into smaller chunks for convenience (like YT-T data in [15]). This audio
segmentation boundary is arbitrary and not a speaker-tag transition boundary.
When relabeling such segmented data, the last speaker-tag token at the end
will be removed (e.g. “Welcome to the show <end-primary> thank you <end-
others> how are <end-primary>” becomes “Welcome to the show <end-primary>
thank you <end-others> how are”).

###  2.3 Related work

There are several technologies in the literature to model multi-speaker /
background noise scenarios, but they are not ideal here. Speaker recognition
[16, 17, 18] requires target voice enrollment [19], which needs a separate
step before ASR and it is not feasible for all applications (e.g. not all
users will enroll their voice prior to using the product). Speaker diarization
[20, 21, 22, 23] and speaker-attributed ASR [24] anonymously label speakers,
but do not explicitly identify who is the primary speaker. Speaker turn
detection [25, 26] would mark the boundaries between two speaker’s speech,
however it is also insufficient to distinguish the primary speaker. Speech
separation [27] usually suffers from “permutation-invariance” (correctly
ordering speakers across segments) and high computational cost due to running
ASR on multiple separated channels, in addition to not identifying the primary
speaker. Our work will explicitly identify the primary speaker (needed for
Short and Dictation domains) and does not need any additional step outside
ASR.

The method proposed here with introducing new tokens for separating speakers
has some parallels to other work e.g. [28, 29]. However, our work differs on a
few fronts: how speakers are grouped into primary vs non-primary and our
particular focus on long-form deletion problem.

###  2.4 RNN-T ASR and speaker-tags

As shown in Fig. 3, our E2E ASR models have the RNN-T architecture, where the
label sequence can be augmented with the speaker-tag tokens. At each time step
$t$, the model would receive a new acoustic frame $x_{t}$ and output a
probability distribution over $y_{t}\in V\cup\\{\texttt{<end-
primary>},\texttt{<end-others>}\\}$, where $V$ being the wordpiece vocabulary
and a blank symbol.

Fig. 3: E2E model architecture

The RNN-T network consists of three components: encoder, prediction network
and joint network. The encoder is analogous to an acoustic model and the
prediction network is analogous to a language model. At each time step, the
encoder receives acoustic features and outputs acoustic encodings. The
recurrent prediction network obtains the embedding vectors from $N$ previous
(non-blank) tokens and produces the text encodings. The encodings from the
encoder and prediction network are then fed into the joint network, which
fuses the acoustic and text encodings together to compute the output logits.
For the baseline model, the prediction network is conditioned only on the
wordpiece tokens. But for the new proposed model, the prediction network is
conditioned on the speaker-tag tokens as well.

During inference, we perform beam search decoding over the wordpiece tokens
and the speaker-tag tokens:

$y*=\operatorname*{arg\,max}_{y}\log
P_{\text{asr}}(y|x_{t-k},\cdots,x_{t},y_{u-N},\ldots,y_{u})$ (1)

where $\mathbf{x}_{t-k},...\mathbf{x}_{t}$ represents acoustic observations
received by the encoder; $\mathbf{y}_{u-N},...\mathbf{y}_{u}$ stands for the
sequence of wordpieces and speaker-tags; $k$ and $N$ are the context window
sizes for the encoder and prediction network, respectively. By the definition
of speaker-tags, the <end-primary> and <end-others> should come in alternate
order. But as per Eq. 1, this is not explicitly enforced. It’s possible that
the model emits the same speaker-tag token repeatedly e.g. “why is the <end-
primary> sky blue <end-primary> welcome home <end-others>”. But during post
processing, each segment preceding the <end-primary> token can be added as the
primary speaker transcript i.e. the above example is equivalent to “why is the
sky blue <end-primary> welcome home <end-others>”. It’s possible to enforce
alternating the two speaker-tag tokens during beam search. But we do not purse
that option as the above post processing is sufficient. Note that the training
data will not have such repeated tokens.

##  3 Experimental Setup

###  3.1 Datasets and Metrics

The training data consists a mix of three application domains: Short,
Dictation and Caption. The Short training data has about 500M utterances,
totalling 500k hours. The Dictation set has about 10M utterances, totalling
23k hours. The Caption training data has about 250M utterances, totalling 600k
hours. Dictation training data is small in scale compared to Caption and
Short. Additionally, Dictation data is generally clean but Caption data is
noisy (music, background speech or noise, multiple-speakers). All the data is
either human-transcribed or machine-transcribed for semi-supervised training
[13, 14]. Multi-style training (MTR) [30] is also used for increasing the data
diversity. For experiments with speaker-tags, the transcripts are modified as
described in Section 2.2.

For evaluation, one test set is used for each domain. The Short testset has 9k
utterances, totalling 12 hours. The Dictation testset has 17k utterances,
totalling 39 hours. The Caption testset has 20k utterances, totalling 380
hours. On these testsets, the Word Error Rate (WER) is measured. The
endpointer median latency (EP50) and 90th percentile latency (EP90) are also
measured for the Short testset. Endpointer latency is defined as the time it
takes for the microphone to close after the user finishes speaking as
determined by forced alignment [31, 32]. This measurement is only required for
Short domain since Dictation and Caption domains can have arbitrarily long
audio input (order of minutes / hours) and the end of the audio is typically
decided by the user or the end of the video file or other systems. A special
<end-of-speech> token is included in the wordpiece vocabulary for endpointer.

###  3.2 Long-form deletion testset and metric

To the best of our knowledge, no existing literate proposes a dedicated
testset or metrics for measuring the long-form deletion problem directly.
Typically, Deletion WER is measured on a generic testset as a proxy, but this
does not explicitly check for continuous deletions in sequence. One
contribution here is to build a novel testset and metrics to measure the long-
form deletion problem reliably. From user feedback and bug reports, the long-
form deletion problem seem to occur prominently when there is static or
background noise in the audio. Based on this observation, we create a testset
by augmenting hand-transcribed long audio segments with bursty noise using
approaches described in [30]. Bursty noise is a type of noise that only spans
a finite time range in the entire audio, like a step function. The modified
audio is transcribed by an ASR model and we check for long sequential deletion
errors after the end of noise. The process is also shown in Figure 4.

For this process, we choose to check for deletions after noise disappears
since with noise it might be harder to separate the speech. This could be a
different problem altogether e.g. [33]. For this reason, we check for errors
outside the noise range. Only variants with 25 sequential word deletions after
the end of the noise are chosen for the testset.

After this process, the testset consists of 70 long audio queries totalling 20
hours. For this testset, the WER and the number of times 25 long sequential
deletions (i.e. 25 words are sequentially deleted by the model) are measured.

Fig. 4: We measure long-form deletions by counting consecutive deleted words
after a bursty noise.

###  3.3 Model description

Experiments are run with a Conformer transducer model [34]. The encoder is
based on the cascaded architecture [35, 36] with 7 causal Conformer blocks
without future context followed by 10 non-causal Conformer blocks with 900 ms
of future acoustic context. The first two causal blocks do not have multi-head
self-attention (MHSA) modules. The model dimensions are 512 and 640 for causal
and non-causal encoders. All Conformer blocks have 8 attention heads in the
MHSA modules. Convolution kernel size in Conv modules is 15; FFN module
dimension is 4x the model dimension. In total, the causal encoder has 47M
parameters and the non-causal encoder has 99M parameters. The output of both
encoders are projected to 384 dimensions. The joint network combines features
from the encoder and embedding from prediction network [37] with 640 units.
Each decoder has 9.5M parameters. During training, the encoders are co-trained
with equal weights. The model has FastEmit loss [38] to encourage lower user
perceived latency and better endpointer performance. The output vocabulary has
4096 wordpieces and tokens, two of which are speaker-tag tokens.

Models are trained with the Lingvo toolkit [39] for 450k steps on 8x8 tensor
processing units (TPUs) with a batch size of 4096.

##  4 Experiments and Results

The E2E ASR model is trained at a large scale with diverse data from different
application domains and is serving different use cases. A baseline model
without speaker-tags can potentially make mistakes since the domains have
different expectations (Short needs endpointer prediction; Short and Dictation
expects only primary speaker transcription vs Caption expects all speech
transcribed). To improve performance, a domain-id encoded by a 4-dimensional
one-hot vector can be appended to the acoustic features [5]. The domain-id
distinguishes Short vs Dictation vs Caption domains. As shown in Table 1,
baseline model B1 (with domain-id input) outperforms the baseline model B0
(without domain-id input) on WER measurement significantly. Note that the
results for B1 are with each domain correspondingly. The Short testset is
evaluated with Short domain-id; the Dictation testset is evaluated with
Dictation domain-id; and the Caption testset is evaluated with Caption domain-
id. This provides the best performance and using any other combination (e.g.
Caption domain-id for the Short testset) leads to much worse performance (over
20% regression) and is not feasible.

Table 1: WER of different models on Short, Dictation (Dt.) and Caption (Cp.) testsets. Results are without endpointer. Exp Id | Model | Short | Dt. | Cp.
---|---|---|---|---
B0 | Base no domain-id input | 5.6 | 4.2 | 16.9
B1 | Base with domain-id input | 5.6 | 3.6 | 15.0
E1 | speaker-tags model | 5.6 | 3.6 | 15.3

Table 2 however shows that with the addition of domain-id input, the model
exhibits huge long-form deletions with the dictation domain-id. The same model
evaluated with Caption domain-id has much better performance. This indicates
that domain-id input can limit the cross-domain learning from training data.
Caption training data is very rich (with different noise conditions, music,
speakers etc) unlike Dictation training data. What the model learned from
Caption training data does not translate well for Dictation domain-id causing
worse long-form deletion. Note that the Short domain only has short audio (few
seconds) and measuring long-form deletion would be meaningless.

Table 2: Long-form deletion results. We report WER together with deletion / insertion / substitution (D/I/S). We also report the number of times 25 consecutive words are deleted (# of 25 del.). Exp | Setup | WER (D/I/S) | # of
---|---|---|---
Id |  |  | 25 del.
B0 | Base no domain-id | 39.9 (30.1/1.2/8.6) | 30
B1-c | B1 caption-id | 30.6 (17.9/1.5/11.1) | 25
B1-d | B1 dictation-id | 68.7 (64.8/0.4/3.4) | 62
E1 | speaker-tags model | 37.5 (27.0/1.4/9.1) | 28

Domain-id input is critical to serve different applications with good WER
quality (Table 1), however it makes the long-form deletion issue more
pronounced (Table 2). Ideally, we can train a model with no additional domain-
id input for better cross-domain learning without hurting the WER performance
on any domain. By unifying the output format for ASR with speaker-tags, this
is now feasible. E1 in Table 1 and Table 2 shows the performance of a model
trained with speaker-tags and without domain-id as as input. Comparing E1 to
the baseline B1, the WER on Short, Dictation and Caption is neutral or has
very little regression. But E1 has substantially less long-form deletions than
B1-d. The speaker-tags model alleviates the long-form deletion problem
significantly.

These results indicate that the long-form deletion is not a problem when the
ASR model is only trained with data from the Caption domain (similar to
baseline B1-c in Table 2). It becomes a problem when the ASR model is trained
with data from other domains (similar to baseline B1-d in Table 2). In domains
like Short and Dictation, the ground truth is annotated in a way such that all
non-primary speaker signals are ignored on purpose, and tends to end early;
while in the Caption domain, the ground truth annotation attempts to cover all
speech signals. The conflicting goals from different domains caused confusions
during the model training, thus leading to long-form deletion errors when non-
primary speakers are present. By explicitly modeling the primary and non-
primary speakers in E1, we removed the confusion during the training, and
fulfilled the domain-specific requirements as a post-processing step.

Next we examine the endpointer behavior (which is required only for Short
domain). In Table 3, the EP50 and EP90 for E1 model with speaker-tags is about
two times higher than the baseline. The model struggles to learn two
conditions in the end - first emit a speaker-tag token then emit the <end-of-
speech> token. To correct this, we assume that <end-primary> is also the
endpointer signal. i.e. when the model emits the <end-primary> token, it’s
considered to be also emitting <end-of-speech> in parallel. E2 shows the
results for such a model where the endpointer latency is significantly better
and is on-par with the baseline. By merging the two tokens, the model is
unable to handle cases where other speakers speak in the middle of the audio.
During post processing, as soon as the first <end-primary> token is emitted,
mic is closed. For example, if the ground truth is “turn on the lights <end-
primary> where is the book <end-others> in the bedroom <end-primary>”, by the
time the model outputs “turn on the lights <end-primary>”, the mic will be
closed and the model will fail to transcribe “in the bedroom”. Although this
is a limitation, such cases are very rare in Short and so, we see no WER
degradation in practice. E2 has the same WER on Caption, Dictation (endpointer
is needed only for Short) and the same long-form deletion performance as E1.

Table 3: Endpointer results. Note: unlike Table 1, here endpointer is enabled so WER is different for Short. Exp | Setup | Short | EP50 | EP90
---|---|---|---|---
Id |  | WER |  |
B1 | Baseline | 6.2 | 260 | 830
E1 | speaker-tags model | 6.3 | 590 | 1625
E2 | E1 + merge tokens | 6.2 | 287 | 770

Overall, E2 model with speaker-tags has substantially better long-form
deletion performance compared to the baseline evaluated on dictation domain-id
and performs almost as well as the baseline on all other testsets and metrics.
Additionally, this model is capable of separating the output transcript by
speaker groups and potentially unlocks new use cases (for instance, if a
downstream application needs to switch between primary and non-primary speaker
transcripts).

##  5 Conclusions

In this work, we proposed a technique to simultaneously model the primary and
non-primary speakers along with the transcript tokens. This is achieved by
introducing two new tokens into the wordpiece vocabulary, and relabeling the
training data with two teacher models. The proposed approach enabled better
cross-domain learning for large scale ASR training without additional training
data. The improved cross domain learning reduces the training-test data
mismatch, thereby significantly alleviating the long-form deletion problem.

##  6 Acknowledgements

The authors would like to acknowledge the following people for many
discussions that helped us shape this work: Weiran Wang, Arun Narayanan,
Yanzhang He, Chao Zhang, Han Lu and Trevor Strohman, affiliated with Google
LLC.

## References

  * [1] A. Graves and N. Jaitly,  “Towards end-to-end speech recognition with recurrent neural networks,”  in International Conference on Machine Learning. 2014, pp. 1764–1772, PMLR.
  * [2] W. Chan, D. Park, C. Lee, et al.,  “SpeechStew: Simply mix all available speech recognition data to train one large neural network,”  arXiv preprint arXiv:2104.02133, 2021.
  * [3] K. Rao, H. Sak, and R. Prabhavalkar,  “Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer,”  in Proc. ASRU. IEEE, 2017, pp. 193–199.
  * [4] A. Narayanan, A. Misra, K.C. Sim, et al.,  “Toward domain-invariant speech recognition via large scale training,”  in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 441–447.
  * [5] T.N. Sainath, Y. He, B. Li, et al.,  “A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,”  in Proc. ICASSP. IEEE, 2020, pp. 6059–6063.
  * [6] A. Narayanan, R. Prabhavalkar, C.-C. Chiu, et al.,  “Recognizing long-form speech using streaming end-to-end models,”  in Proc. ASRU. IEEE, 2019, pp. 920–927.
  * [7] C.-C. Chiu, W. Han, Y. Zhang, R. Pang, et al.,  “A comparison of end-to-end models for long-form speech recognition,”  in Proc. ASRU. IEEE, 2019, pp. 889–896.
  * [8] Z. Lu, Y. Pan, T. Doutre, et al.,  “Input length matters: Improving RNN-T and MWER training for long-form telephony speech recognition,”  arXiv preprint arXiv:2110.03841, 2021.
  * [9] Suyoun Kim and Florian Metze,  “Dialog-context aware end-to-end speech recognition,”  in Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 434–440.
  * [10] S. Kim, S. Dalmia, and F. Metze,  “Gated embeddings in end-to-end speech recognition for conversational-context fusion,”  arXiv preprint arXiv:1906.11604, 2019.
  * [11] T. Hori, N. Moritz, C. Hori, and J.L. Roux,  “Advanced long-context end-to-end speech recognition using context-expanded transformers,”  arXiv preprint arXiv:2104.09426, 2021.
  * [12] A. Zeyer, R. Schmitt, W. Zhou, et al.,  “Monotonic segmental attention for automatic speech recognition,”  in Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 229–236.
  * [13] D. Hwang, K. C. Sim, Z. Huo, and T. Strohman,  “Pseudo label is better than human label,”  arXiv preprint arXiv:2203.12668, 2022.
  * [14] C-C. Chiu, A. Narayanan, W. Han, et al.,  “RNN-T models fail to generalize to out-of-domain audio: Causes and solutions,”  in Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 873–880.
  * [15] Y. Zhang, D. S. Park, W. Han, et al.,  “BigSSL: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1519–1532, 2022.
  * [16] L. Wan, Q. Wang, A. Papir, and I.L. Moreno,  “Generalized end-to-end loss for speaker verification,”  in Proc. ICASSP. IEEE, 2018, pp. 4879–4883.
  * [17] C. Li, X. Ma, B. Jiang, et al.,  “Deep speaker: an end-to-end neural speaker embedding system,”  arXiv preprint arXiv:1705.02304, 2017.
  * [18] D. Snyder, D. Garcia-Romero, G. Sell, et al.,  “X-vectors: Robust DNN embeddings for speaker recognition,”  in Proc. ICASSP. IEEE, 2018, pp. 5329–5333.
  * [19] Quan Wang and Ignacio Lopez Moreno,  “Version control of speaker recognition systems,”  arXiv preprint arXiv:2007.12069, 2020.
  * [20] X. Anguera, S. Bozonnet, N. Evans, et al.,  “Speaker diarization: A review of recent research,”  IEEE Transactions on audio, speech, and language processing, vol. 20, no. 2, pp. 356–370, 2012.
  * [21] T.J. Park, N. Kanda, D. Dimitriadis, et al.,  “A review of speaker diarization: Recent advances with deep learning,”  Computer Speech & Language, vol. 72, pp. 101317, 2022.
  * [22] Chao Zhang and Quan Wang,  “Speaker diarization: A journey from unsupervised to supervised approaches,” Odyssey: The Speaker and Language Recognition Workshop, 2022,  Tutorial session.
  * [23] Quan Wang, Carlton Downey, Li Wan, Philip Andrew Mansfield, and Ignacio Lopz Moreno,  “Speaker diarization with lstm,”  in Proc. ICASSP. IEEE, 2018, pp. 5239–5243.
  * [24] Naoyuki Kanda, Xuankai Chang, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Zhuo Chen, and Takuya Yoshioka,  “Investigation of end-to-end speaker-attributed asr for continuous multi-talker recordings,”  in Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 809–816.
  * [25] W. Xia, H. Lu, Q. Wang, et al.,  “Turn-to-diarize: Online speaker diarization constrained by transformer transducer speaker turn detection,”  in Proc. ICASSP. IEEE, 2022, pp. 8077–8081.
  * [26] G. Zhao, Q. Wang, H. Lu, et al.,  “Augmenting transformer-transducer based speaker change detection with token-level training loss,”  in Proc. ICASSP. IEEE, 2023, pp. 1–5.
  * [27] J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe,  “Deep clustering: Discriminative embeddings for segmentation and separation,”  in Proc. ICASSP. IEEE, 2016, pp. 31–35.
  * [28] R. Masumura, D. Okamura, N. Makishima, et al.,  “Unified autoregressive modeling for joint end-to-end multi-talker overlapped speech recognition and speaker attribute estimation,”  arXiv preprint arXiv:2107.01549, 2021.
  * [29] N. Kanda, Y. Gaur, X. Wang, et al.,  “Serialized output training for end-to-end overlapped speech recognition,”  arXiv preprint arXiv:2003.12687, 2020.
  * [30] Chanwoo Kim, Ananya Misra, Kean Chin, Thad Hughes, Arun Narayanan, Tara N. Sainath, and Michiel Bacchiani,  “Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in Google Home,”  in Proc. Interspeech 2017, 2017, pp. 379–383.
  * [31] R. Maas, A. Rastrow, C. Ma, et al.,  “Combining acoustic embeddings and decoding features for end-of-utterance detection in real-time far-field speech recognition systems,”  in Proc. ICASSP. IEEE, 2018, pp. 5544–5548.
  * [32] S-Y. Chang, R. Prabhavalkar, Y. He, et al.,  “Joint endpointing and decoding with end-to-end models,”  in Proc. ICASSP. IEEE, 2019, pp. 5626–5630.
  * [33] T. O’Malley, A. Narayanan, Q. Wang, et al.,  “A conformer-based ASR frontend for joint acoustic echo cancellation, speech enhancement and speech separation,”  in Proc. ASRU. IEEE, 2021, pp. 304–311.
  * [34] A. Gulati, J. Qin, C-C. Chiu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [35] A. Narayanan, T. N. Sainath, R. Pang, et al.,  “Cascaded encoders for unifying streaming and non-streaming ASR,”  in Proc. ICASSP. IEEE, 2021, pp. 5629–5633.
  * [36] T. N. Sainath, Y. He, A. Narayanan, et al.,  “Improving the latency and quality of cascaded encoders,”  in Proc. ICASSP. IEEE, 2022, pp. 8112–8116.
  * [37] R. Botros, T. N. Sainath, R. David, et al.,  “Tied & reduced RNN-T decoder,”  arXiv preprint arXiv:2109.07513, 2021.
  * [38] J. Yu, C-C. Chiu, B. Li, et al.,  “Fastemit: Low-latency streaming ASR with sequence-level emission regularization,”  in Proc. ICASSP. IEEE, 2021, pp. 6004–6008.
  * [39] J. Shen, P. Nguyen, Y. Wu, et al.,  “Lingvo: a modular and scalable framework for sequence-to-sequence modeling,”  arXiv preprint arXiv:1902.08295, 2019.


# Artificial Neural Network for Resource Allocation in Laser-based Optical
wireless Networks  ††thanks: This work has been supported by the Engineering
and Physical Sciences Research Council (EPSRC), in part by the INTERNET
project under Grant EP/H040536/1, and in part by the STAR project under Grant
EP/K016873/1 and in part by the TOWS project under Grant EP/S016570/1. All
data are provided in full in the results section of this paper.

Ahmad Adnan Qidan, Taisir El-Gorashi1, Jaafar M. H. Elmirghani  School of
Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United
Kingdom
Email: {a.a.qidan, t.e.h.elgorashi<EMAIL_ADDRESS>
###### Abstract

Optical wireless communication offers unprecedented communication speeds that
can support the massive use of the Internet on a daily basis. In indoor
environments, optical wireless networks are usually multi-user multiple-input
multiple-output (MU-MIMO) systems, where a high number of optical access
points (APs) is required to ensure coverage. In this work, a laser-based
optical wireless network is considered for serving multiple users. Moreover,
blind inference alignment (BIA) is implemented to achieve a high degree of
freedom (DoF) without the need for channel state information (CSI) at
transmitters, which is difficult to provide in such wireless networks. Then,
an objective function is defined to allocate the resources of the network
taking into consideration the requirements of users and the available
resources. This optimization problem can be solved through exhaustive search
or distributed algorithms. However, a practical algorithm that provides
immediate solutions in real time scenarios is required. In this context, an
artificial neural network (ANN) model is derived in order to obtain a sub-
optimal solution with low computational time. The implementation of the ANN
model involves three important steps, dataset generation, offline training,
and real time application. The results show that the trained ANN model
provides a significant solution close to the optimal one.

##  I Introduction

In recent years, researches have investigated exploiting the enormous
unlicensed-bandwidth of the optical band to support the increase in user
demands. In this context, optical wireless networks are investigated using
light emitting diode (LED) for providing illumination and data transmission.
Despite the high achievable data rates in these networks compared to radio
frequency (RF) wireless networks, the low modulation speed of LEDs limits the
potential of considering optical wireless notworks in the next generation (6G)
of wireless networks. Infrared (IR) Lasers, namely Vertical-Cavity Surface
Emitting (VCSEL) sources, are used as transmitters in [1919, AA19901111] under
eye safety constraints. The features of VCSELs including their high modulation
speed compared with LED, low power consumption and low cost make them a strong
candidate to achieve unprecedented rates that fulfill the escalating demands
of users.

In laser-based wireless networks, deploying a high number of optical access
points (APs) is needed to ensure coverage. Therefore, these networks are
naturally considered as multi-user multiple-input multiple-output (MU-MIMO)
systems. In this sense, transmit precoding schemes such as zero forcing (ZF)
[QHQHALMH] are implemented to align multi-user interference with the need for
channel state information (CSI) at transmitters. It is worth mentioning that
the characteristics of the optical signal limit the performance of ZF where a
DC bias current must be applied to ensure the non-negativity of the
transmitted signal. In [MMAL17, 8636954] a reconfigurable optical detector is
proposed with the aim of implementing blind interference alignment (BIA),
which serves multiple users simultaneously without CSI at transmitters
following a certain methodology. It is shown that the precoding matrices of
BIA are given by positive values, and it is more suitable for optical wireless
networks compared to ZF.

Resource management plays an important role in maximizing the overall sum rate
of the network. In [AA19901111], an objective function is defined to allocate
the resources based on the connectivity of users. In [9521837], an
optimization problem is formulated to maximize the sum rate of users by
allocating their requirements of resources. In general, such optimization
problems have high complexity that increases considerably with the size of the
network. Therefore, distributed algorithms using Lagrangian multipliers are
used in order to reduce the complexity, while providing sub-optimal solutions
[AA19901111, 9521837, FRLHANZO]. However, a distributed algorithm requires an
iterative algorithm, affecting its accuracy in obtaining solutions with
minimum errors in real time scenarios due to the fact that users might change
their requirements, locations, etc., while the algorithm performs resource
allocation. Recently, artificial neural networks (ANNs) have received massive
interest as an alternative technique to solve the optimization problems in
high complexity problems in low computational time [sun20, luca19], where an
ANN model can be trained over a dataset in an offline phase, so that accurate
solutions can be obtained instantaneously in real time scenarios.

In contrast to the work in [9521837], in this paper, an optimization problem
is formulated in a laser-based wireless network to satisfy the requirements of
users and maximize the utility function of the sum rate using a certain ANN
model. We first derive the achievable user rate considering BIA as a
transmission scheme. Then, the optimization problem is formulated under
several constraints of user-requirements and AP-capacity limitations. This
problem can be solved using exhaustive search to provide an optimal solution
with high complexity. Finally, an ANN model is defined with the aim of
providing a sub-optimal solution with low complexity. The ANN model is
implemented in three steps: Dataset generation, offline training, and real
time scenario. The results show that the proposed ANN model provides a
solution close to the optimal one. Moreover, BIA is more suitable for optical
wireless network than ZF.

Figure 1:  Reconfigurable optical detector with $M$ photodiodes.

##  II System Model

We consider a laser-based wireless network composed of $L$,
$l=\\{1,\dots,L\\}$, APs serving $K$, $k=\\{1,\dots,K\\}$, users distributed
on the receiving plane. Each user is equipped with a reconfigurable detector,
which consists of $M$ multiple photodiodes as shown in Fig. 1, providing a
wide field of view (FoV) to ensure the connectivity to most of the available
APs. It is worth mentioning that this detector has the ability to provide
linearly independent channel responses where each photodiode has a distinct
direction $\mathbf{\widehat{n}}$, more details are in [MMAL17, 8636954]. In
this sense, the received signal of a generic user $k$ at photodiode $m$, $m\in
M$, is given by

$y^{[k,l]}[n]=\mathbf{h}^{[k]}(m^{[k]}[n])^{T}\mathbf{x}[n]+z^{[k]}[n],$ (1)

where $\mathbf{h}^{[k]}(m^{[k]}[n])^{T}\in\mathbb{R}_{+}^{L\times 1}$,
$m^{[k]}[n]$ is a preset mode selected by photodiode $m$ at time slot $n$,
$\mathbf{x}$ is the transmitted signal and $z^{[k,l]}$ is real valued additive
white Gaussian noise with zero mean and variance given by the sum of shot
noise, thermal noise and the intensity noise of VCSEL. In this work, CSI is
avoided, and all APs are connected to a central unit (CU), which controls the
resources of the network. Moreover, users can send their resources
requirements through a WiFi link.

###  II-A Transmitter

The optical channel between user $k$ and AP $l$ at photodiode $m$ can be
expressed as

$h^{[k,l]}(m)=h_{\mathrm{LoS}}^{[k,l]}(m)+h_{\mathrm{diff}}(f)e^{-j2\pi
f\Delta T},$ (2)

where $h_{\mathrm{LoS}}^{[k,l]}(m)$ denotes Line-of-Sight (LoS) components of
the direct link, $h_{\mathrm{\mathrm{diff}}}$ is the diffuse channel (Non-LoS)
and $\Delta T$ is the delay between LoS and diffuse components. Each VCSEL
illuminates a small and confined area, and therefore, the diffuse component
can be neglected, for the sake of simplicity, where most of the received power
is due to LoS components [AA19901111].

The transmitted power of VCSEL can be determined based on the beam waist
$W_{0}$, the wavelength $\lambda$ and the distance between the ceiling and the
receiving plane $d$. Moreover, the beam profile of the VCSEL transmitter is
Gaussian ignoring higher weak modes, and its intensity is defined as a
function of the radial distance $r$ from the center of the beam spot and the
distance $d$, i.e., $I(r,d)$. In this context, considering the transmitted
power of VCSEL $l$, ${P_{t,l}}$, and its beam radius, $W_{d}$, at distance
$d$, the received power by user $k$ at photodiode $m$ located right below
VCSEL $l$ can be expressed as

$\begin{split}&P_{m,l}=\\\ &\int_{0}^{A_{m}/2\pi}I(r,d)2\pi
rdr=P_{t,l}\left[1-\mathrm{exp}\left(-2\left(\frac{A_{m}}{2\pi
W_{d}}\right)^{2}\right)\right],\end{split}$ (3)

where $A_{m}=\frac{A_{rec}}{M}$ is the area of photodiode $m$, assuming the
whole area of the reconfigurable detector is $A_{rec}$.

##  III Blind Multiple Access scheme

In [Gou11, 8636954], BIA is proposed for interference management in RF and
optical wireless networks, respectively, without the need for CSI at
transmitters, . Basically, in BIA, a transmission block referred to as
supersymbol is generated, which consists of two blocks, Block 1 and Block 2.
In this section, the construction of the supersymbol is presented first for a
toy example, and then, the achievable user rate is derived for the general
case.

Figure 2: The supersymbol of BIA for $L=2$ and $K=3$. Each color represents a
preset mode.

Let us consider a case in which $L=2$ APs that serve $K=3$ users, each user is
equipped with a reconfigurable photodetector. In BIA-based interference
management, all users receive their information over Block 1 of the
supersymbol generating interference among them. However, orthogonal
transmission is carried out over Block 2 providing enough dimensions for each
user to measure and cancel the interference received over Block 1. To
guarantee this methodology for the case considered, Block 1 comprises one time
slot, while three time slots belong to Block 2, forming the supersymbol of BIA
as shown in Fig.2. In this context, the transmitted signal is given by

$\mathbf{X}=\begin{bmatrix}\mathbf{I}\\\ \mathbf{I}\\\ \mathbf{0}\\\
\mathbf{0}\end{bmatrix}\mathbf{u}^{[1]}+\begin{bmatrix}\mathbf{I}\\\
\mathbf{0}\\\ \mathbf{I}\\\
\mathbf{0}\end{bmatrix}\mathbf{u}^{[2]}+\begin{bmatrix}\mathbf{I}\\\
\mathbf{0}\\\ \mathbf{0}\\\ \mathbf{I}\end{bmatrix}\mathbf{u}^{[3]},$ (4)

where $\mathbf{u}^{[k]}=\begin{bmatrix}u_{1}^{[k]}&u_{2}^{[k]}\end{bmatrix}$
is the symbol transmitted to user $k$. From equation (4), orthogonal
transmission among the users can be observed over the second, third and fourth
time slots. Focusing on user 1, without loss of generality, the interference
because of the transmission to the other users over Block 1, i.e., the first
time slot, is measured over time slots 3 and 4 of Block 2, and then, it can be
subtracted afterwords at the cost of increasing noise. As a consequence, user
1 decodes 2 DoF of $\mathbf{u}^{[1]}$ transmitted over $\\{1,2\\}$ time slots.
Similarly, users 2 and 3 can decode 2 DoF of $\mathbf{u}^{[2]}$ and
$\mathbf{u}^{[3]}$, transmitted over $\\{1,3\\}$ and $\\{1,4\\}$ time slots,
respectively. Therefore, the sum DoF equals to $6/3$ DoF for the case
considered, which is higher than orthogonal transmission schemes such as TDMA.

For the general case where $L$ APs serve $K$ users, the BIA supersymbol
comprises $(L-1)^{K}+K(L-1)^{K-1}$ time slots, more mathematical details are
provided in [Gou11, 8636954]. In this sense, the achievable rate of user $k$
is

$r^{[k]}=\frac{1}{L+K-1}\log_{2}\left(\mathbf{I}+P_{\rm{str}}\mathbf{H}^{[k]}{\mathbf{H}^{[k]}}^{H}{\mathbf{R}_{z}}^{-1}\right),$
(5)

where $P_{\rm{str}}$ is the power allocated to each stream,
$\mathbf{H}^{[k]}=\begin{bmatrix}\mathbf{h}^{[k]}(1)&\dots&\mathbf{h}^{[k]}(L)\end{bmatrix}^{T}\in\mathbb{R}^{L\times
L}$ is the channel matrix of user $k$, and
$\mathbf{R}_{z}=\begin{bmatrix}K\mathbf{I}_{L-1}&0\\\ 0&1\end{bmatrix}$ is the
covariance matrix of noise.

##  IV Problem Formulation

Resource allocation schemes enhance the performance of wireless networks in
terms of resource utilization and the maximization of the overall sum rate. In
particular, the resources of a network can be allocated uniformly among users
avoiding complexity. However, this way might not satisfy the requirements of
users. In this sense, a utility function-based optimization problem is
formulated with the aim of maximizing the sum rate of the users by allocating
their resource demands taking into consideration the capacity limitations of
the APs, as in the following

$\displaystyle\max_{e}$ $\displaystyle\sum_{l\in L}\,\sum_{k\in
K}\log\left(1+\xi_{k}~{}e^{[k,l]}r^{[k,l]}\right)$ (6) s.t.
$\displaystyle\sum_{k\in K}e^{[k,l]}\leq\rho_{l},\,\forall l\in L$
$\displaystyle\sum_{l\in L}e^{[k,l]}\leq e^{[k]}_{\max},\,\forall k\in K$
$\displaystyle\sum_{l\in L}e^{[k,l]}\geq e^{[k]}_{\min},\,\forall k\in K,$

where $U(\cdot)=\log(\cdot)$ 111The objective function is considered in the
form of $\log\left(1+\xi_{k}~{}e^{[k,l]}r^{[k,l]}\right)$ in order to avoid
$U(\cdot)=-\infty$ if $e^{[k,l]}=0$. is a logarithmic function, which achieves
proportional fairness among the users. Moreover, $e^{[k,l]}$ are the resources
allocated from AP $l$ to user $k$, $\xi_{k}>0$ is the scalability to flow
$e^{[k,l]}$ [9521837], $r^{[k,l]}$ is the achievable user rate and $\rho_{l}$
is the capacity constraint of AP $l$. The first constraint satisfies that AP
$l$ is not overloaded, and the second and third constraints ensure that each
user receives its requirements of resources located within a certain range,
where $e^{[k]}_{\max}$ and $e^{[k]}_{\min}$ are the maximum and minimum
resources required by user $k$, respectively. This optimization problem can be
solved through an exhaustive search method, which involves high complexity. In
this sense, full dual decomposition method via the Lagrangian multiplier can
be considered to solve this problem providing sub-optimal resource allocation
with less complexity, more details are provided in [AA19901111, 9521837,
FRLHANZO]. The Lagrangian function of (6) is

$\displaystyle g\left(e,\rho,\lambda,\eta_{1},\eta_{2}\right)=\sum_{l\in
L}\,\sum_{k\in K}\log\left(1+\xi_{k}~{}e^{[k,l]}r^{[k,l]}\right)$ (7)
$\displaystyle+\underbrace{\sum_{l\in L}\lambda_{l}\left(\rho_{l}-\sum_{k\in
K}e^{[k,l]}\right)}_{\text{constraint 1 in
\eqref{OP2}}}+\underbrace{\sum_{k\in
K}\eta_{k,1}\left(e^{[k]}_{\max}-\sum_{l\in
L}e^{[k,l]}\right)}_{\text{constraint 2 in \eqref{OP2}}}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+\underbrace{\sum_{k\in
K}\eta_{k,2}\left(\sum_{l\in
L}e^{[k,l]}-e^{[k]}_{\min}\right)}_{\text{constraint 3 in~{}\eqref{OP2}}},$

where $\lambda_{l}$, $\eta_{1}$ and $\eta_{2}$ are multipliers associated with
the first, second and third constraints in (6), respectively. It is worth
mentioning that solving (LABEL:eq:lag) requires an iterative algorithm, which
is not practical to use in providing instantaneous estimations of resources in
real time scenarios. In the following, an ANN model is presented to avoid the
complexity of the optimization problem in (6), while providing a sub-optimal
solution.

##  V Artificial Neural Network

Figure 3: An ANN model with an input layer, $D$ hidden layers and two output
layers.

Generally, an ANN model consists of a series of multiple layers, each layer is
composed of artificial neurons that preform a certain function. An example of
the ANN model is presented in Fig.3, for the sake of simplicity. As shown, it
has an input layer, multiple hidden layers and an output layer. The input
layer usually corresponds to information received from the environment with a
bias denoted as $b_{bia}$, on which mathematical calculations are applied to
provide intended solutions. The hidden layer is composed of $D$,
$d=\\{1,\dots,D\\},$ multiple layers, and each artificial layer $d$ has a
total of $C$ artificial neurons where each neuron is designed with the ability
to perform certain mathematical calculations on its input, so that its output
can be fed into the next layer. The output layer of the ANN model consists of
two artificial neurons with outputs determined based on information that is
fed from the hidden layer $D$. It is worth pointing out that in this work, the
ANN model must learn the requirements of the users and the available
resources, and then, the output layer gives the resources allocated to each
user and the capacity limitation of each AP.

Focusing on the $d$ th layer, the input and output of this layer are given by

$\mathbf{s}_{i,d}=[s_{1,d-1}\dots,s_{C,d-1}]^{T},$ (8)

and

$\mathbf{s}_{o,d}=[s_{1,d}\dots,s_{C,d}]^{T},$ (9)

respectively. It is easy to notice that the input of the $d$ th layer is the
exact output $\mathbf{s}_{o,d-1}$ of layer $d-1$. Moreover, the output
$s_{c,d}$, $c\in C$, is calculated as in the following

$s_{c,d}=\kappa_{c,d}[\alpha(\mathbf{s}_{o,d-1},\mathbf{w}_{c,d},b_{bia_{c,d}})],$
(10)

where $\kappa[.]$ is an activation function, $\alpha(.,.,.)$ is a linear
transformation function, $\mathbf{w}_{c,d}$ is the the weight vector of neuron
$n$ in layer $d$-th, and $b_{bia_{c,d}}$ is a scalar bias. Additionally, the
linear transformation function $\alpha(.,.,.)$ of the ANN model is given by

$\alpha(\mathbf{s}_{o,d-1},\mathbf{w}_{c,d},b_{bia_{c,d}})=\mathbf{w}_{c,d}\circledast\mathbf{s}_{o,d-1}+b_{bia_{c,d}},$
(11)

where $\circledast$ is the convolution operator where a convolutional neural
network (CNN) is considered in this work due to its relatively high speed and
efficiency. According to the process above, the outputs of the ANN model in
Fig.3, $v$ and $j$, are given by

$v=\kappa_{v,o}[\alpha(\mathbf{s}_{o,D},\mathbf{w}_{v,o},b_{bia_{v,o}})],$
(12)

and

$j=\kappa_{j,o}[\alpha(\mathbf{s}_{o,D},\mathbf{w}_{j,o},b_{bia_{j,o}})],$
(13)

respectively. At this point, we aim to find the mapping $f(\mathcal{V};.)$ of
the ANN model that guarantees allocating the resources based on the
requirements of the users and the available resources, where $\mathcal{V}$ is
defined as a set of weight terms working as bridges among the vital parameters
of the optimization problem that maximize the sum rate of the network. The
optimalaity of $\mathcal{V}$ can be ensured following a training process
presented in the next sub-sections.

###  V-A Dataset generation

A dataset obtained from solving the main problem can be used for training the
ANN model to choose the accurate set of parameters, i.e., $\mathcal{V^{*}}$,
that provide a sub-optimal solution in a real time scenario. Our system model
is composed of a number of APs serving multiple users distributed on the
receiving plane. Each user sends its resource requirements through a WiFi link
or low data rate diffuse optical wireless link to all the available APs.

In this context, our aim is to get a set of data points given by $N$, each $n$
corresponds to $K$ users sending their resources requirements located within
the range $e^{[k]}_{min}\leq e^{[k]}\leq e^{[k]}_{max}$ to $L$ APs. It is
worth mentioning that the resources allocated for a certain user $e^{[k]}$ is
given by $\sum^{L}_{l=1}e^{[k,l]}$, and its value differs from one user $k$ to
another $k^{\prime}$ according to the activity of a user at a given time,
i.e., $e^{[k]}\neq e^{[k^{\prime}]},k\neq k^{\prime}$. Moreover, the value of
$e^{[k,l]}$ allocated to user $k$ from AP $l$ is determined in accordance to
the capacity limitation of that AP, and it is different from the value of
$e^{[k,l^{\prime}]}$ allocated to the same user from another AP $l\neq
l^{\prime}$ .

The resources required by each user are based on satisfying its requirements
as well as maximizing the sum rate of the users. That is, each AP $l$ solves
the following equation independently to maximize its own utility function

$\max_{e}\\{\sum_{k\in K}\log(1+e^{[k,l]}r^{[k,l]})-\\\ \lambda_{l}\sum_{k\in
K}e^{[k,l]}-\sum_{k\in K}(\eta_{k,1}-\eta_{k,2})e^{[k,l]}\\}.$ (14)

The multipliers $\lambda$, $\eta_{1}$ and $\eta_{2}$ work corresponding to the
capacity limitation and user-requirements constraints, respectively (see (6)
and (LABEL:eq:lag)). To solve this problem, the optimal resources allocated to
each user by an AP are determined for fixed values of the multipliers by
applying the Karush - Kuhn - Tucker (KKT) conditions [dfdf]. Then, an updating
process for the multipliers is preformed to satisfy the requirements of users
by increasing the resources allocated to each user towards the maximum value
$e^{[k]}_{max}$ if there are sufficient resources, otherwise the multipliers
are updated to decrease the resources allocated to the minimum value
$e^{[k]}_{min}$ [9521837], i.e.,

$\lambda_{l}(i+1)=\left[\lambda_{l}(i)-\Omega_{\lambda}(i)\left(\rho_{l}-\sum_{k\in
K}e^{*[k,l]}\right)\right]^{+},$ (15)
$\eta_{k,1}(i+1)=\left[\eta_{k,1}(i)-\Omega_{\eta_{1}}(i)\left(e^{[k]}_{max}-\sum\limits_{l=1}^{L}e^{*[k,l]}\right)\right]^{+},$
(16)
$\eta_{k,2}(i+1)=\left[\eta_{k,2}(i)-\Omega_{\eta_{2}}(i)\left(\sum\limits_{l=1}^{L}e^{*[k,l]}-e^{[k]}_{min}\right)\right]^{+},$
(17)

respectively, where $i$th denotes the iteration of the gradient algorithm,
$[.]^{+}$ is a projection on the positive orthant to account for considering
the fact that we have $\lambda,\eta_{1},\eta_{2}\geq 0$. Furthermore,
$\Omega_{j}(i)$ , $j\in\\{\lambda,\eta_{1},\eta_{2}\\}$, is the step size at a
given $i-$th iteration that is taken in the direction of the negative gradient
for the multipliers $\lambda$, $\eta_{1}$ and $\eta_{2}$. After running the
optimization problem for different user activities, the maximized rates are
recorded for training the ANN model.

###  V-B ANN Implementation

The implementation of the ANN model can avoid the high complexity of solving
the optimization problem in (6). Notice that, generating the dataset for
learning purposes involves also high complexity. However, it is an offline
process, and then, the results can be recorded for use in satisfying the
requirements of users instantly in real time scenarios.

####  V-B1 Offline phase

The ANN model must be trained over the dataset generated to find the unknown
mapping between user-requirements on one side and resource allocation and the
AP capacity limitations on the other side. In other words, the set of optimal
weight terms $\mathcal{V^{*}}$ must be found in order to make the input and
output of the ANN model relevant. In particular, the output layer of the ANN
model applied to solve our problem in (6) estimates
$\widehat{\mathbf{e}}=[\widehat{{e}}^{[1]},\dots,\widehat{{e}}^{[k]},\dots,\widehat{{e}}^{[K]}]$,
where $\widehat{{e}}^{[k]}=\sum^{L}_{l=1}\widehat{e}^{[k,l]}$ is the
estimation of the optimal resource allocated to user $k$, which is given by
${e^{*[k]}}=\sum^{L}_{l=1}e^{*[k,l]}$. Notice that, the ANN model estimates
also the capacity limitations of the APs due to the fact that the optimal
resource allocation is determined under the first constraint in the original
optimization problem (see equation (6)).

Let us focus on having a training dataset that contains $N$ data points. At
data point $n$, $n=[1,\dots,N]$, the optimal resource allocation is given by
${\mathbf{e^{*}}(n)}$ for the training input denoted as ${\mathbf{v}}(n)$,
while the estimation for this data point is given by
$\widehat{\mathbf{e}}(n)$. In this sense, we train the ANN model to choose the
optimal set of weight terms that minimizes a certain loss function between the
optimal and estimated resources, i.e.,

$\\\
\min_{\mathbf{W}}\frac{1}{N}\sum^{N}_{n=1}\ell(\widehat{\mathbf{e}}(n),\mathbf{e^{*}}(n)).$
(18)

where $\ell(.,.)$ is the mean-square-error (MSE) function. By solving equation
(18), the ANN is trained to find sub-optimal resource allocation for several
data points even if these are not included in the training dataset. In a real
time scenario, the requirements of the users might change from one time to
another, and by sending the new requirements to the ANN model at a given time,
an instantaneous solution can be provided with low complexity.

####  V-B2 Real time phase

In the offline phase, the ANN model is trained to find the optimal weight
terms. Subsequently, the ANN model is deployed at the APs to perform resource
allocation in an online phase where each user requires a certain amount of
resources based on its activity. The requirements of the users and the
available resources are fed into the ANN model in order to determine the
resources allocated to each user from the whole set of the APs. Notice that,
the overall resources allocated for user $k$ must satisfy the condition
$e^{[k]}_{min}\leq e^{[k]}\leq e^{[k]}_{max}$. Therefore, during the process
of calculating the resources of each user, if one AP $l$ has sufficient
resources compared to the other APs, that AP allocates more resources to user
$k$ in order to increase the resources $e^{[k]}$ towards the maximum value
$e^{[k]}_{max}$, and therefore, maximizing the sum rate of the users,
otherwise the resources $e^{[k]}$ decease to the minimum value
$e^{[k]}_{min}$. Finally, the APs and users update their multipliers according
to the outputs of the ANN model (see equations (15), (16) and (17)), so that,
if any user is not satisfied with its allocated resources, the AP with low
multiplier allocates more resources to that user. Moreover, if any user
changes its activity, a new set of user-requirements must be fed into the ANN
model for new resource allocation among users.

TABLE I: Simulation Parameters Parameter | Value
---|---
VCSEL Bandwidth | 5 GHz
VCSEL Wavelength | 830 nm
VCSEL beam waist |  $10-30~{}\mu$m
Physical area of the photodiode | 15 $\text{mm}^{2}$
Receiver FOV | 45 deg
Detector responsivity | 0.53 A/W
Gain of optical filter | 1.0
Laser noise |  $-155~{}dB/H$z
ANN model | CNN
Number of hidden layers | $D=3$
Dataset size |  $N=10^{4}$, $N=5000$
Training |  $90\%$ of $N$
Validation |  $10\%$ of $N$
Figure 4: Training and validation of the ANN model for two different sizes of
dataset.

##  VI PERFORMANCE EVALUATIONS

An indoor environment with 5m$\times$ 5m$\times$ 3m dimensions is considered
to test the performance of the proposed ANN model. On the ceiling, $L=16$
VCSELs are deployed with a uniform distribution to form an array of VCSELs
serving $K=10$ active users with different requirements at a given time. These
users are distributed randomly on a receiving plane with a 2.15m distance from
the ceiling, and each user is equipped with a reconfigurable detector that
consists of $M$ photodiodes, and has the ability to provide $L$ preset modes
in order to apply BIA successfully, mores details are in [AA19901111,
8636954]. All the other simulation parameters are listed in Table 1.

The accuracy of the ANN model is shown in Fig. 4, in terms of training and
validation losses versus a set of epochs. The MSE of the training loss over a
$N=5000$ dataset size is 0.6 at epoch 5, and it starts decreasing with the
number of epochs, while the validation loss is 0.55 for the same dataset size
at epoch 5. On the other hand, the accuracy of the ANN model increases with
the size of the dataset, where the MSE losses of the training and validation
processes over a $N=10000$ dataset size are $0.5$ and $0.45$ at epoch 5,
respectively. Notice that, the ANN model for both dadaset sizes is not
overfitting, and an acceptable solution can be obtained even if the ANN model
is fed with information that are not included in the dataset used for the
training process. Therefore, the ANN model is validated to provide sub-optimal
solutions in real time scenarios where the requirements of users and the
available resources of the network might change multiple times in a few
seconds.

In Fig. 5, the sum rate is depicted against the beam waist of the VCSEL,
considering two different sizes of the dataset used for training the ANN
model. It can be seen that the ANN model trained over $N=10000$ achieves a
solution with high accuracy compared with $N=5000$ in all the scenarios
considered. Compared with solving the main problem in (6), the ANN model
provides a significant sum rate close to the optimal solution, and therefore,
using the proposed ANN model is beneficial in solving optimization problems of
high complexity in real time. Moreover, allocating the resources based on the
proposed utility function results in a higher sum rate than simply dividing
the resources of the network among the users regardless of their demand, where
some users might waste the resources allocated due to their low rate
applications used at a given time. The figure further shows that increasing
the beam waist of the VCSEL enhances the sum rate achieved for all the
optimization techniques, which is due to the fact that the received power
increases with increase in the beam waist where the transmitted power is
considerably focused towards the users as the illuminated area of the VCSEL
gets more confined.

Figure 5: Achievable sum rates of BIA for the optimization techniques
considered versus different values of the VCSEL beam waist $W0$.

In Fig. 6, the cumulative distribution function (CDF) is shown for the sum
rate of BIA compared with ZF. The performance of BIA is superior to ZF in both
resource allocation scenarios using the ANN model and the uniform scheme. It
is worth mentioning that BIA satisfies the non-negativity of the transmitted
signal naturally due to its positive precoding matrix given by 0 and 1, and
therefore, applying a DC bias current, which might cause clipping distortion
to the transmitted signal, is avoided. In contrast, the performance of ZF is
limited due to the characteristics of the optical channel where the negative
values of the transmitted signal must be avoided strictly by applying a DC
bias current [AA19901111, 8636954]. Finally, the ANN model trained over an
$N=10000$ dataset size achieves higher sum rate than the uniform resource
allocation scheme.

Figure 6: CDF of the sum rate for the optimization problem solved using the
ANN mode compared to uniform scheme.

##  VII CONCLUSIONs

In this paper, an optimization problem is formulated in a laser-based wireless
network with an objective function that aims to allocate resources based on
the requirements of users. This problem can be solved by exhaustive search,
which involves high complexity, or by distributed algorithms via Lagrangian
multipliers, which requires an iterative algorithm that might consume time.
Therefore, an ANN model is introduced to solve the problem in a real time
scenario with low computational time. We first generate a dataset from solving
the main problem in an offline phase, and then, the ANN model is trained over
this dataset to choose an optimal set of weights that minimizes a certain loss
function. After that, the trained ANN is implemented in a real time scenario
to allocate resources among users considering their requirements and the
available resources of the network. The results show the optimality of the ANN
model where an instantaneous sub-optimal solution close to the optimal one is
provided.