Yochai Blau, Rohan Agrawal, Lior Madmony, Gary Wang, Andrew Rosenberg, Zhehuai
Chen, Zorik Gekhman, Genady Beryozkin, Parisa Haghani, Bhuvana Ramabhadran

# Using Text Injection to Improve Recognition of Personal Identifiers in
Speech

###### Abstract

Accurate recognition of specific categories, such as persons' names, dates or
other identifiers is critical in many Automatic Speech Recognition (ASR)
applications. As these categories represent personal information, ethical use
of this data including collection, transcription, training and evaluation
demands special care. One way of ensuring the security and privacy of
individuals is to redact or eliminate Personally Identifiable Information
(PII) from collection altogether. However, this results in ASR models that
tend to have lower recognition accuracy of these categories. We use text-
injection to improve the recognition of PII categories by including fake
textual substitutes of PII categories in the training data using a text
injection method. We demonstrate substantial improvement to Recall of Names
and Dates in medical notes while improving overall WER. For alphanumeric digit
sequences we show improvements to Character Error Rate and Sentence Accuracy.

Index Terms: conformers, E2E models, Medical ASR, Text-Injection Training,
protecting PHI, de-identification, PII

##  1 Introduction

While a lot of speech is publicly broadcast or shared online to a vast
audience, speech recognition applications frequently interact with more
private communications such as dictation, call-center conversations or
conversing with a digital assistant. Automatic speech recognition (ASR)
systems are trained using transcribed speech, and operate best when their
training data matches the context in which they are used. This raises a
question of how to train ASR to work well on private communications, while
being sensitive to the complexities of collecting and transcribing training
material from private contexts.

We engage with this question in the context of healthcare. Accurate
transcription of medical speech is key to a growing set of applications,
including dictation of clinical notes and voice assistance. However, ethical
collection and transcription of training data for medical ASR requires care.
Medical data is exceptionally sensitive and respecting patients' privacy in
any collection effort is essential. De-identification, a process which
includes the redaction of PII tokens, is a common practice to protect users'
private data. In the process, the audio segment is replaced by silence and the
text is either removed or replaced by a special markup tag. De-identification
is used extensively for healthcare data and is often required by regulations
such as the US HIPAA Act [1].

In contexts such as call-center applications, speech utterances are shorter
and may be in response to prompts like ``Please confirm your date of birth''
or ``Please say your user id''. In this case, de-identification results in
completely eliminating the utterance. In both domains, using such de-
identified datasets for training ASR models makes recognition of these classes
of PII terms operate at a higher error rate than the surrounding speech.

Prior work has focused around replacing PII in speech with arbitrary
synthesized speech from the same category [2] (for example, named entities,
number sequences) or ensuring that these PII tokens/sequences are adequately
covered by a language model, as done in [3]. Text injection has recently
emerged as an effective way to leverage text-only data for ASR training in a
single model [4, 5, 6, 7] without an LM. Techniques such as the one described
in [4] have shown promise for zero-transcribed speech adaptation. In [8], Wang
& Kastner et al. demonstrated that text-injection can effectively address
domain transfer on diverse SpeechStew corpora [9] with little to no in-domain
speech. Motivated by this, we propose to generate a text dataset that contains
fake terms instead of the redacted PII and use text-injection during model
training. This eliminates the need for any sensitive transcribed training data
and allows effective use of de-identified medical datasets.

This paper addresses three specific data challenges when training a model for
private contexts with the use of text-injection in ASR training.

  * •

Domain adaptation: Transcribed speech from private contexts is much harder to
come by than public contexts. We show that text-injection along with a small
amount of transcribed speech can effectively perform domain adaptation to
Medical ASR resulting in a 11.5 reduction to WER and improvements of 1% to
Names and 11% to Dates (Section 5.1).

  * •

Redaction: Available in-domain data has redacted PII from the transcript with
corresponding speech replaced by silence. To mitigate this, we train on
generated, spoken text similar to the PII, which allows for substantial
improvement to sensitive term recognition without using any individual's
protected data; compared to training on redacted speech data, Names and Dates
recall is improved by 8% and 13% (Section 5.1).

  * •

Elimination: De-identification may eliminate utterances that contain sensitive
terms. For example, short utterances containing alphanumeric and digit
sequences are eliminated. We show that text-injection can substantially
improve recognition of these classes of potentially sensitive information with
a full-sequence Sentence Accuracy improvement of 3.2% (and CER improvement of
1.4%) (Section 5.2).

##  2 Related Work

The complexity of recognizing personal identifiers in medical speech has
recently been studied in [2]. This work shows that PII recognition accuracy
declines when training an end-to-end model on de-identified data. The authors
propose to generate artificial audio/text pairs with synthetic identifiers,
and show that training on such data restores most of the performance
degradation. Artificial audio/text pairs are generated in a two-step process:
first, PII in the text is replaced with random data. Then, corresponding audio
is generated or spliced with matching fragments. Generating or manipulating
the audio is more demanding then replacing identifiers in the text and also
requires accurate word-level timestamps. In our work we take this concept one
step further, by improving PII recognition with text replacements alone, and
_without_ any audio generation or manipulation. Other work has studied medical
domain ASR [3, 10, 11, 12], yet these have not reported findings on PII
recognition accuracy and the effect of training on de-identified data.

Regarding text-injection in ASR training, this work builds on the architecture
described in [4]. This is described in detail in Section 3. However, there is
a set of related approaches and architectures that similarly use separate
speech and text encoders that feed into a shared encoder [5, 6, 7].
Additionally, a similar thread of work trains on speech and text data operates
by first converting the text data to speech via TTS [13, 14, 15, 16, 17].
These approaches, in effect, use speech as an intermediate representation to
perform text injection.

##  3 Text-Injection and ASR Details

The text-injection model used in this work includes a speech encoder, a text
encoder with a learned duration model, a shared encoder, decoder and an
alignment decoder following [4]. The speech encoder consists of 4 causal
conformer layers. The shared encoder consists of 3 causal conformer layers and
10 non-causal conformer layers, each with model dimension of 512. The text
encoder contains 2 conformer layers and 4 lightweight convolutional upsampling
layers [18]. HAT decoders [19] with $v^{2}$ embeddings [20] are used in both
decoder and an alignment decoder with the distinction of the former produces
word-piece models as text outputs and the latter uses phonemes as model units
to get speech-text alignments. The overall model contains 165M parameters,
with an additional 58M in the text encoder which is only used during training.

Text-injection training involves speech-text and text-only training paths in a
curriculum fashion. Initially, speech-text training is used to minimize a
consistency loss, a HAT decoder loss and an alignment decoder loss, for the
duration model. Consistency loss ensures that we learn corresponding mappings
from text to speech embedding. The duration model is used to up-sample text
embedding before being fed to the shared encoder. After beginning training
with speech-text loss, we enable text-only loss to be able to perform text-
injection training. The text-only training step involves minimizing an aligned
masked-language model loss. Further details of text injection architecture and
training can be found in [4].

In all experiments, we train a text-injection model on supervised only data
for 10k steps first so that the speech and text encoders generate consistent
features. We continue training with target domain text or targeted terms as
described in the following sections. While training with text alone is
feasible after the initial supervised training, we find that including
supervised data during training helps to stabilize the model behavior.

##  4 Data Description

###  4.1 Medical domain datasets

The Medical Audio dataset consists of dictations of clinical notes by
healthcare professionals from a variety of medical specialties. During the
transcription process any potentially identifying information was redacted.
The corresponding audio segment was replaced by silence and a special markup
was placed in the transcript to indicate the type of the data that was
removed. Examples of the markup tags include PATIENT_NAME,
MEDICAL_PROFESSIONAL_NAME, AGE and DATE. The resulting dataset was certified
to comply with de-identification requirements of the US HIPAA Privacy Rule
[1].

The Medical Text dataset consists of the transcriptions from the Medical Audio
dataset (without the audio recordings). As mentioned above, these
transcriptions include markup tags for PIIs such as names, addresses, dates,
etc. that were redacted. In this dataset, we replace four types of markup tags
(names, identifying numbers, dates and ages) with fake random data. For
example, any name tag is replaced by a random name from a real-world
distribution, each redacted digit is replaced with a random digit between 0-9,
and so on. Note that replacing redacted PII information in text is quite easy
and straightforward. However, splicing synthetic speech within speech
recordings where the PII was silenced in a natural manner is challenging and
error-prone, especially in the case of fast-paced speech recorded in a noisy
environment as it is in the Medical Audio dataset.

The Synthetic Notes dataset is a collection of fake dictations, recorded by
clinicians in a noise-free environment and consists of 14.5 hours of audio.
The dataset consists of synthetic hospital visit summaries which include fake
identifiers such as names, IDs, dates, etc. The dataset creation process
started with creation of fake hospital visit summaries by trained clinicians.
Later, a separate group of clinicians simulated dictations of hospital notes
based on these visit summaries, which were then independently transcribed.

The Synthetic Names dataset is a 14-hours long dataset that consists of short
phrases (average length 7.6s) that resemble phrases in the dictation of
clinical notes and include references to patient or clinician names. First,
medical specialists created 550 textual templates with placeholders for names,
ensuring diversity of name formats. Each template was recorded in a quiet
environment by 17 different speakers, with placeholders substituted by names,
each time randomly drawn from a real-world distribution. All name occurrences
are tagged, which allows to compute a recall metric for names alone (along
with more global metrics such as WER).

Table 1: Dataset descriptions. Name |  Size (hours) |  Type |  Description
---|---|---|---
Training data
Captions |  377K |  Speech |  Speech from videos and corresponding captions.
Medical Audio |  5K |  Speech |  Clinicians dictating notes. All private information is redacted.
Medical Text |  - |  Text |  The transcriptions from the Medical Audio dataset, where redacted identifiers are replaced with random data.
Testing data
Synthetic Notes |  14.5 |  Speech |  Synthetic hospital notes dictated by clinicians, including (fake) identifiers.
Synthetic Names |  14 |  Speech |  Synthetic short phrases from clinical notes, including (fake) names (patient/clinician/family member/etc.).

###  4.2 Alphanumeric Sequence Identifier Datasets

For personal identifier text-injection train data, we use 100M sequences each
of alphanumeric sequences and text sequences. The length of these sequences is
sampled from a skewed normal distribution $X(\mu=10,\,\sigma=5)+3$. For digit
sequences, characters from 0-9 are sampled uniformly. For alphanumeric
sequences, characters from the alphanumeric set 0-9,a-z are sampled uniformly.
10% of sequences are chosen at random, and character repeats of length 2, 3 or
4 are added. This is done because character or digit repetitions are a
challenging use case for an ASR system.

The spoken digit sequence test set consists of 2146 utterances (average length
4.3s) with randomly generated digit sequences, spoken by voice actors.

The spoken alphanumeric sequence test set consists of 1,992 utterances
(average length 5.2s) with randomly generated alphanumeric sequences, spoken
by voice actors. The voice actors in both sequence test sets have acted out
hesitations and pauses while speaking these sequences.

The following TTS datasets contain utterances synthesized by a commercial
American English TTS system sampled from 6 voices. In this work, we use the
TTS datasets as test data, and the spoken sequence data as either test or
training material (Section 5.2).

TTS digit sequence test set consists of 1991 utterances (average length 5.4s)
with randomly generated digit sequences.

The TTS alphanumeric sequence test set consists of 2000 utterances (average
length 5.7s) with randomly generated alphanumeric sequences.

The TTS digit sequence repetition consists of 3188 utterances (average length
5.4s) with randomly generated digit sequences which have at minimum, 1
consecutive digit repetition. 1001 utterances have 2, 3 and 4 consecutive
repetitions and 185 utterances have more than 4 repetitions.

The TTS alphanumeric sequence repetition consists of 2179 utterances (average
length 5.7s) with randomly generated alphanumeric sequences which have at
minimum, 1 consecutive character repetition. 1001 utterances have 2, 3
consecutive repetitions and 177 utterances have more than 4 repetitions.

###  4.3 General datasets

The base model (B1) is trained with close to 300k hours of multidomain
utterances including YouTube, Telephony and Dictation for US English as
described in [22]. Multi-condition training [23], random 8kHz down-sampling
[24] are applied on the training data. Importantly, any utterance that
includes alphanumeric and digit sequences are eliminated from transcribed
training data, as they may represent personal identifiers. This data is only
included in initial checkpoint training, described as A1/B1 in Section 5, and
is not further used during finetuning.

Along with the medical domain data described above, we also include 377k hours
of YouTube Captions data as described in [25] to maintain strong general
purpose ASR performance. While this material may incidentally include medical
content, no particular targeting, selection or filtering for in-domain
material was performed.

##  5 Recognizing Redacted and Eliminated Terms

###  5.1 Identifiers in medical speech

All models used the architecture described in Section 3, and use a base
checkpoint, B1, trained on multidomain data (Section 4.3). The A1 model was
trained on the (non-medical) Captions dataset, in which medical terms are
relatively uncommon. This shows clear domain-transfer effects; the WER on the
medical-term rich Synthetic Notes dataset is high, with a WER of 14.6% (Table
2).

Adapting this model to the medical domain is done by training on a mixture of
Captions and Medical Audio datasets with a 90%/10% ratio, resulting in model
A2. The Synthetic Notes WER drops significantly, and we observe higher recall
for medical entities such as conditions and medications. Yet, for other
entities such as names and dates the recall metrics degrade. This is a direct
consequence of names, dates and other identifiers being redacted from the
Medical Audio dataset.

To improve error rates on PII, model T1 is trained on the same mixture of
speech data as for A2, with the Medical Text dataset injected during training.
This textual data contains identifiers such as names and dates, leading to a
8%/13% boost in names/dates recall, respectively. In Table 3 we show two cases
were model T1, which has been exposed to medical text with (fake) identifiers,
performs better in transcribing examples from the Synthetic Names dataset. We
also observe that the introduction of medical text further improves the
overall WER from 3.3 to 3.1, demonstrating the value of additional in-domain
text to address domain transfer as well as targeted terms.

Table 2: Word error rate for Synthetic Notes and medical-term/name/date recall for Synthetic Names. |  Datasets |  Synthetic Notes WER |  Medical term recall |  Names recall |  Dates recall
---|---|---|---|---|---
B1 |  Multidomain (MD) |  14.6 |  82.6% |  60% |  66%
A1 |  MD, Captions |  14.7 |  82.1% |  59% |  72%
A2 |  MD, Captions
Medical Audio |  3.3 |  97.4% |  53% |  64%
T1 |  MD, Captions
Medical Audio
Medical Text (injected) |  3.1 |  97.5% |  61% |  77%
Table 3: Examples of name mistranscriptions from Synthetic Names corpus. Note: All examples are fake descriptions of hypothetical scenarios.  Model |  Transcription
---|---
Truth |  Scarlett Kathleen Ibarra was admitted to the floors
A2 |  Scarlet Caffeine Ebara was admitted to the floors
T1 |  Scarlett Kathleen Ibarra was admitted to the floors
Truth |  Oliver Barry Matthews is adamant that the patient had no past surgeries
A2 |  All of our bearing math uses adamant that the patient had no past surgeries
T1 |  Oliver Barry Matthews is adamant that the patient had no past surgeries

The training data used for models A1, A2 and T1 above, includes a large
general-domain multidomain and Captions dataset. This dataset includes
identifying entities such as names and dates to some extent. To isolate the
effect of PII redaction, we train a cold-start model A3 only on the Medical
Audio dataset (where all identifiers are redacted). The results of this
experiment are in Table 4. All error metrics degrade, but particularly
noticeable is the names recall which drops to 1%, as the model has not seen
_any_ names during training. Yet performance can be improved by injecting text
which includes fake identifiers, even without a single audio example of such
entities, as seen for model T2. While the metrics indicate that text injection
alone cannot match the performance of T1, the recall of Names and Dates are
improved by a factor of 1800% and 207%, respectively. Moreover, the
introduction of medical text improves the WER performance of the model from
6.1 to 4.1.

Table 4: Ablation study: training only on medical data. |  Datasets |  Synthetic Notes WER |  Medical term recall |  Names recall |  Dates recall
---|---|---|---|---|---
A3 |  Medical Audio |  6.1 |  96.2% |  1% |  28%
T2 |  Medical Audio
Medical Text (injected) |  4.1 |  97.2% |  18% |  58%

###  5.2 Recognizing Alphanumeric Sequence Identifiers

The architecture of the base ASR model B1 is the same RNN-T model with a
cascade encoder and HAT decoder with $v^{2}$ embeddings in Section 5.1. The
baseline text injection model, M1 was trained with the same paired text data
as B1, and additional base text injection data (not related to target domain)
consists of 100B anonymized sentences across different domains. Note that this
text injection model is trained in arbitrary text to improve core ASR
performance rather than targeting any particular domain or type of lexical
content.

We aim to improve recognition of alphanumeric and digit sequence identifiers.
Thus, we perform a round of text injection training based on the initial model
described above, specifically, the personal identifier text data (cf. Section
4.2). Results of these experiments measured by character error rate (CER) and
Sentence Accuracy (SACC) in Tables 5 and 6. Character error rate is more fine
grained, by being able to measure individual errors. However, when recognizing
an identifier it is crucial to recognize the full sequence; this is measured
by SACC.

Table 5: CER on alphanumeric and digit test sets. Dataset |  B1: Base model |  M1: B1 + base text injection |  M2: M1 +
in-domain text inj. |  M3: M2+
Spoken datasets
---|---|---|---|---
Spoken digit |  3.1 |  2.4 |  2 |  -
TTS digit rep. |  15.2 |  15.7 |  14.5 |  17.0
TTS digit |  1.7 |  1 |  1.1 |  0.8
Spoken alphanum |  8.6 |  7.3 |  6.8 |  -
TTS alphanum rep. |  12.1 |  11.9 |  9.3 |  8.4
TTS alphanum |  9.2 |  8.5 |  8 |  6.2
average CER |  8.3 |  7.8 |  6.9 |  -
Table 6: SACC on alphanumeric and digit test sets. Dataset |  B1: Base model |  M1: B1 + base text injection |  M2: M1 +
in-domain text inj. |  M3: M2+
Spoken datasets
---|---|---|---|---
Spoken digit |  93.8 |  94.7 |  95.2 |  -
TTS digit rep. |  67.4 |  69.1 |  70.4 |  64.3
TTS digit |  96.8 |  96.6 |  97.1 |  96.8
Spoken alphanum |  73.6 |  75.5 |  76.5 |  -
TTS alphanum rep. |  54.2 |  53.5 |  63.9 |  63.6
TTS alphanum |  66.0 |  67.4 |  68.1 |  72.4
average SACC |  75.3 |  76.1 |  78.5 |  -

We use CER as a metric instead of word error rate for evaluation of personal
identifier recognition because CER is a better fit for these sequences; an
utterance may contain a single ``word'' like A1B2C3. Normalization helps to
ignore formatting issues with respect to such sequences. As can be seen from
Table 5, even the addition of base text injection data which is not related to
target domain (M1) improves CER from 8.3 to 7.8, a reduction of 6% from B1.
Additional training on domain specific text injection data (M2), further
reduces normalized CER to 6.9, a reduction of 14.1% from M1. Correspondingly
when considering SACC, after adding base text injection data (M1), we see an
increase from 75.3 to 76.1, a modest increase of 1.1% from B1. SACC further
increases to 78.5 after training on domain specific text injection data (M2),
an increase of 3.1% from M1. In M3, we add Spoken sequence data (4138
utterances) to our training corpus. The motivation for this experiment is to
show the effect of adding a small amount of paired speech-text data in
addition to text-injection training can further help. This further reduces CER
on alphanumeric test sets, but gives mixed results in digit test sets.

Catastrophic forgetting is a concern when fine-tuning a pretrained model on
material from a narrow domain, as we are doing here with text-injection of
sequences. To assess the impact of this we assess performance on a Short
Utterance test set representative of spoken search queries, and TTS test sets
focusing on rare proper names from 5 different domains (Table 7). While we see
a minor regression between M1 and M2, the performance is still substantially
better than B1, the base model with no text-injection. This suggests that this
regression can be mitigated with tuning of a ratio and schedule of the text-
injection training.

Table 7: WER on broad domain test sets. Dataset |  B1: Base model |  M1: B1 + base text injection |  M2: M1 +
in-domain text inj.
---|---|---|---
Short Utterances |  6.7 |  5.6 |  5.7
TTS rare terms 1 |  14.5 |  11.3 |  11.9
TTS rare terms 2 |  9.8 |  8.6 |  8.9
TTS rare terms 3 |  37.9 |  33.5 |  34.4
TTS rare terms 4 |  21.9 |  17.9 |  18.8
TTS rare terms 5 |  24.9 |  20.7 |  21.4

##  6 Conclusions

Due to the sensitive nature of medical speech and personal identifiers,
substantial care must be taken in the collection, transcription and use of
this data, often requiring the removal of personal identifiers from the data.
Models trained on such data have higher error rates when recognizing such
identifiers. In this paper, we demonstrate that text-injection is an effective
approach to improving ASR performance of models trained on de-identified
speech, including improved recognition of personal identifiers. Text-injection
can leverage text where speech is unavailable, and synthetic text is simple to
construct, frequently indistinguishable from real transcripts, and not
associated with any person's private communication. In Medical ASR where names
and dates are redacted from speech and transcripts, we find core WER improves
from 3.3 to 3.1, and recall of names and dates are improved by 8% and 13%
respectively. For short utterances containing alphanumeric and digit sequences
which can be used as personal identifiers in a broad range of contexts, we
improve SACC by an average of 3.2% while CER is reduced from 8.3% to 6.9%

## References

  * [1] ``The Health Insurance Portability and Accountability Act of 1996 (HIPAA),'' https://www.hhs.gov/hipaa/index.html, 1996.
  * [2] M. Flechl, S.-C. Yin, J. Park, and P. Skala, ``End-to-end speech recognition modeling from de-identified data,'' in _Proc. Interspeech_ , 2022, pp. 1382–1386.
  * [3] E. Edwards, W. Salloum, G. P. Finley, J. Fone, G. Cardiff, M. Miller, and D. Suendermann-Oeft, ``Medical speech recognition: Reaching parity with humans,'' in _Speech and Computer_ , 2017, pp. 512–524.
  * [4] Z. Chen, Y. Zhang, A. Rosenberg, B. Ramabhadran, P. J. Moreno, A. Bapna, and H. Zen, ``MAESTRO: Matched Speech Text Representations through Modality Matching,'' in _Proc. Interspeech_ , 2022, pp. 4093–4097.
  * [5] A. Bapna, C. Cherry, Y. Zhang, Y. Jia, M. Johnson, Y. Cheng, S. Khanuja, J. Riesa, and A. Conneau, ``mslam: Massively multilingual joint pre-training for speech and text,'' _arXiv preprint arXiv:2202.01374_ , 2022.
  * [6] T. N. Sainath, R. Prabhavalkar, A. Bapna, Y. Zhang, Z. Huo, Z. Chen, B. Li, W. Wang, and T. Strohman, ``Joist: A joint speech and text streaming model for asr,'' in _2022 IEEE Spoken Language Technology Workshop (SLT)_ , pp. 52–59.
  * [7] S. Thomas, B. Kingsbury, G. Saon, and H.-K. J. Kuo, ``Integrating text inputs for training and adapting rnn transducer asr models,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 8127–8131.
  * [8] G. Wang, K. Kastner, A. Bapna, Z. Chen, A. Rosenberg, B. Ramabhadran, and Y. Zhang, ``Understanding shared speech-text representations,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023.
  * [9] W. Chan, D. Park, C. Lee, Y. Zhang, Q. Le, and M. Norouzi, ``Speechstew: Simply mix all available speech recognition data to train one large neural network,'' _arXiv preprint arXiv:2104.02133_ , 2021.
  * [10] C.-C. Chiu, A. Tripathi, K. Chou, C. Co, N. Jaitly, D. Jaunzeikare, A. Kannan, P. Nguyen, H. Sak, A. Sankar, J. Tansuwan, N. Wan, Y. Wu, and X. Zhang, ``Speech Recognition for Medical Conversations,'' in _Proc. Interspeech_ , 2018, pp. 2972–2976.
  * [11] A. Mani, S. Palaskar, N. V. Meripo, S. Konam, and F. Metze, ``Asr error correction and domain adaptation using machine translation,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 6344–6348.
  * [12] L. Zhou, S. V. Blackley, L. Kowalski, R. Doan, W. W. Acker, A. B. Landman, E. Kontrient, D. Mack, M. Meteer, D. W. Bates, and F. R. Goss, ``Analysis of errors in dictated clinical documents assisted by speech recognition software and professional transcriptionists,'' _JAMA network open_ , vol. 1, no. 3, pp. e180 530–e180 530, 2018.
  * [13] A. Rosenberg, Y. Zhang, B. Ramabhadran, Y. Jia, P. Moreno, Y. Wu, and Z. Wu, ``Speech recognition with augmented synthesized speech,'' in _IEEE automatic speech recognition and understanding workshop (ASRU)_ , 2019, pp. 996–1002.
  * [14] Z. Chen, Y. Zhang, A. Rosenberg, B. Ramabhadran, P. Moreno, and G. Wang, ``Tts4pretrain 2.0: Advancing the use of text and speech in asr pretraining with consistency and contrastive losses,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022.
  * [15] A. Fazel, W. Yang, Y. Liu, R. Barra-Chicote, Y. Meng, R. Maas, and J. Droppo, ``SynthASR: Unlocking Synthetic Data for Speech Recognition,'' in _Proc. Interspeech_ , 2021, pp. 896–900.
  * [16] X. Zheng, Y. Liu, D. Gunceler, and D. Willett, ``Using synthetic audio to improve the recognition of out-of-vocabulary words in end-to-end asr systems,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 5674–5678.
  * [17] G. Wang, A. Rosenberg, Z. Chen, Y. Zhang, B. Ramabhadran, Y. Wu, and P. Moreno, ``Improving speech recognition using consistent predictions on synthesized speech,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 7029–7033.
  * [18] I. Elias, H. Zen, J. Shen, Y. Zhang, Y. Jia, R. J. Weiss, and Y. Wu, ``Parallel tacotron: Non-autoregressive and controllable tts,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 5709–5713.
  * [19] E. Variani, D. Rybach, C. Allauzen, and M. Riley, ``Hybrid autoregressive transducer (hat),'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020.
  * [20] M. Ghodsi, X. Liu, J. Apfel, R. Cabrera, and E. Weinstein, ``Rnn-transducer with stateless prediction network,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 7049–7053.
  * [21] ``Our principles - Google AI,'' https://ai.google/principles/, 2023, [Online; accessed 01-March-2023].
  * [22] A. Narayanan, R. Prabhavalkar, C.-C. Chiu, D. Rybach, T. N. Sainath, and T. Strohman, ``Recognizing long-form speech using streaming end-to-end models,'' in _IEEE automatic speech recognition and understanding workshop (ASRU)_ , 2019, pp. 920–927.
  * [23] C. Kim, A. Misra, K. Chin, T. Hughes, A. Narayanan, T. N. Sainath, and M. Bacchiani, ``Generation of Large-Scale Simulated Utterances in Virtual Rooms to Train Deep-Neural Networks for Far-Field Speech Recognition in Google Home,'' in _Proc. Interspeech_ , 2017, pp. 379–383.
  * [24] J. Li, D. Yu, J.-T. Huang, and Y. Gong, ``Improving wideband speech recognition using mixed-bandwidth training data in cd-dnn-hmm,'' in _IEEE Spoken Language Technology Workshop (SLT)_ , 2012, pp. 131–136.
  * [25] H. Liao, E. McDermott, and A. Senior, ``Large scale deep neural network acoustic modeling with semi-supervised training data for youtube video transcription,'' in _IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2013, pp. 368–373.


#  Cosmological time and the constants of nature

João Magueijo  Theoretical Physics Group, Imperial College, Prince Consort Rd,
London, SW7 2BZ, United Kingdom<EMAIL_ADDRESS>
###### Abstract

We propose that cosmological time is effectively the conjugate of the
constants of nature. Different definitions of time arise, with the most
relevant related to the constant controlling the dynamics in each epoch. The
Hamiltonian constraint then becomes a Schrodinger equation. In the connection
representation, it is solved by monochromatic plane waves moving in a space
generalizing the Chern-Simons functional. Normalizable superpositions exist
and for factorizable coherent states we recover the classical limit and a
seamless handover between potentially disparate times. There is also a rich
structure of alternative states, including entangled constants, opening up the
doors to new phenomenology.

The seemingly unrelated problems of time in Quantum Gravity Isham ; Kuch  and
of the values of the constants of nature Barrowtip  are two foundational
nuisances in physics occasionally dismissed as metaphysical. And yet, without
a physical definition of time it is hard to see how Quantum Gravity could ever
connect with the real world. Perhaps for this reason, the subject is often
relegated to the depths of the “Planck epoch”, as if our classical Universe
should not appear as a limit of Quantum Cosmology, possibly with perennial
corrections and novelties, and so direct observational consequences. The fact
that we appear to be entering a baffling phase of accelerated expansion
accelexp , possibly due to the cosmological “constant” $\Lambda$ (with all its
fine-tuning headaches weinberg ; padilla ) might be taken as a hint. Could the
two puzzles be closely related?

In this Letter we propose adopting as effective cosmological clocks the
conjugates of the constants of nature. For this to be possible we have to
partly “deconstantize” them, demoting constants from parameters “set in stone”
to phase space variables which happen to be constant as a result of the
equations of motion (thus, leaving room for an explanation of their values).
This proposal is intuitive: it is only natural that the conjugate (and quantum
complementary) of something that does not vary—a constant—should be the
essence of change itself, and so a measure of relational, physical time (as
opposed to coordinate time). It is also not entirely new: in the context of
$\Lambda$ it resonates with unimodular gravity unimod1 ; unimod , where the
conjugate of $\Lambda$ was identified with a time variable (which turned out
to be Misner’s volume time misner ).

But why would $\Lambda$ be special? A clock is crafted with what is at hand.
Depending on the constant(s) dominating the dynamics in each region of phase
space, we should employ different times. We are thus led to emergent times
variously conjugate to $\Lambda$, the gravitational constant, $G_{N}$, or even
the speed of light, $c$, among others. This is usually regarded as a curse:
either there is no time in General Relativity, or, if we attempt to define
one, we are spoilt for choice. In this Letter we embrace a democracy of times
as a blessing, with the adjustment of clocks across different “time zones” to
be seen as a physical feature of our world. Semiclassical states will be
found, for which the handover between clocks is trivial. But there are also
alternative states, which might be of great predictive relevance, considering
that the Universe is currently busy passing on the baton from a matter (or
$G_{N}$) clock to a $\Lambda$ clock.

We focus on constants, $\alpha$, for which the timeless Hamiltonian constraint
$H=0$ can be approximately written as:

${\cal H}=H_{0}-\alpha=0$ (1)

in some phase space regions. We illustrate our approach in minisuperspace, but
generalizations can be built gens , using unimodular gravity as the blueprint
unimod ; gielen  (but also non-local theories pad ; lomb ). We use the
Einstein-Cartan formulation of Relativity, but our construction can be applied
to other theories of gravity. Then our starting point is the action:

$S=\frac{3V_{c}}{8\pi G_{0}}\int
dt\bigg{(}a^{2}\dot{b}-Na\left[-(b^{2}+kc^{2})+\sum_{i}\frac{m_{i}}{a^{1+3w_{i}}}\right]\bigg{)},$
(2)

where $G_{0}$ is a fixed value for $G_{N}$ (should $G_{N}$ be up for
“deconstantization”), $a$ is the expansion factor, $b$ is the connection
variable ($b=\dot{a}/N$ on-shell), $k=0,\pm 1$ is the normalized spatial
curvature, $N$ is the lapse function and $V_{c}$ is the comoving volume of the
region under study (or the whole manifold, if compact). The summation term can
accommodate a large number of models, the specifics of which will not be
relevant for the rest of this Letter, but which we list here nonetheless. For
a geometrical $\Lambda$ we have $m_{i}=\Lambda/3$ and equation of state
$w_{i}=-1$. For dust and radiation we have $w_{i}=0,1/3$, and can set
$m_{i}=C_{i}8\pi G_{0}/3$, where $C_{i}$ are conserved quantities gielen . We
can also set $m_{i}=C_{i}8\pi G_{N}/3$, and deconstantize the gravitational
coupling $G_{N}$ (as opposed to the $G_{0}$ controlling the non-commutation
properties of geometrical operators). For a massless scalar field, in
particular, we would then have $m_{i}\propto\Pi_{\phi}/G_{N}$ and $w_{i}=1$.
Finally, and more speculatively, both the curvature $k$ and $\Lambda$ could be
seen as components with $m_{i}=-kc^{2}$ and $w_{i}=-1/3$, and
$m_{i}=\Lambda_{0}c^{2}/3$ and $w_{i}=-1$, with $c$ the target constant.

In this Letter we develop the quantum theory in the connection representation
(but metric versions exist gielen , with the two equivalent if it is accepted
that $a^{2}\in(-\infty,\infty)$ CSHHV ). Then, the Hamiltonian constraint can
be more easily solved by putting (1) in the approximate form:

${\cal H}=h_{i}(b)a^{2}-\alpha_{i}=0,$ (3)

in each region/epoch where one $i$ in (2) dominates (we will examine each $i$
separately at first; no summation over $i$ implied where repeated). As we
shall see, this amounts to a canonical transformation selecting “linearizing”
variables (to use the terminology of DSR ). From (2) we see that:

$\displaystyle h_{i}(b)$ $\displaystyle=$
$\displaystyle(b^{2}+k)^{\frac{2}{1+3w_{i}}}$ (4) $\displaystyle\alpha_{i}$
$\displaystyle=$ $\displaystyle m_{i}^{\frac{2}{1+3w_{i}}},$ (5)

(with $w_{i}=-1/3$ to be treated separately elsewhere). The central proposal
in this Letter is to promote the $\alpha_{i}$ to phase space variables with
conjugate momenta $p_{i}$ to be identified as a time variable $T_{i}\equiv
p_{i}$:

$S\rightarrow S+\frac{3V_{c}}{8\pi G_{0}}\int dt\dot{\alpha}_{i}p_{i},$ (6)

(the pre-factor introduced for later convenience). In this perspective, the
constancy of the “constants” is not preset, but rather arises from the
equation of motion:

$\dot{\alpha}_{i}=\\{\alpha_{i},H\\}=0$ (7)

resulting from Poisson bracket:

$\\{\alpha_{i},p_{i}\\}=\frac{8\pi G_{0}}{3V_{c}}$ (8)

and the fact that $H$ is independent of $p_{i}$ (in itself an expression of
the “time”-independence of the Hamiltonian, at a more fundamental level).
Classically, the $\alpha_{i}$ values are just integration constants,
generalizing a well-known situation in unimodular gravity unimod1 ; unimod
and not only Hawking3f . But more importantly, given (8), the quantum theory
now reveals an effective Schrodinger equation

$\left[H_{0}(b)-i\frac{l_{P}^{2}}{3V_{c}}\frac{\partial}{\partial
T_{i}}\right]\psi(b,T_{i})=0$ (9)

(with fixed $l_{P}=\sqrt{8\pi G_{0}\hbar}$) in lieu of the timeless Wheeler-
DeWitt (WDW) equation. It is only when we impose “fixed constants”, i.e. the
monochromatic ansatz:

$\psi(b,T_{i};\alpha_{i})=\psi_{s}(b;\alpha_{i})\exp{\left[-i\frac{3V_{c}}{l_{P}^{2}}\alpha_{i}T_{i}\right]},$
(10)

that we find that the “spatial” $\psi_{s}$ must satisfy the original timeless
WDW equation. Furthermore, since:

$\\{b,a^{2}\\}=\frac{8\pi
G_{0}}{3V_{c}}\implies\hat{a}^{2}=-i\frac{l_{P}^{2}}{3V_{c}}\frac{\partial}{\partial
b}$ (11)

we see that (with $H_{0}\approx h_{i}(b)a^{2}$) the WDW equation, with
suitable ordering, becomes:

$\left(-i\frac{l_{P}^{2}}{3V_{c}}\frac{\partial}{\partial
X_{i}}-\alpha_{i}\right)\psi_{s}=0.$ (12)

with:

$X_{i}(b)=\int\frac{db}{h_{i}(b)}.$ (13)

The “spatial” solutions are therefore plane-waves in $X_{i}$:

$\displaystyle\psi_{s}(b;\alpha_{i})$ $\displaystyle=$ $\displaystyle{\cal
A}(\alpha_{i})\exp{\left[i\frac{3V_{c}}{l_{P}^{2}}\alpha_{i}X_{i}(b)\right]}$
(14)

where for $w_{i}=-1$ we can recognize the Chern-Simons state CS ; jackiw ;
kodama . In this sense Eq. (13) is a generalization of the Chern-Simons
functional. The full monochromatic solutions are:

$\displaystyle\psi(b,T_{i};\alpha_{i})$ $\displaystyle=$ $\displaystyle{\cal
A}(\alpha_{i})\exp{\left[i\frac{3V_{c}}{l_{P}^{2}}\alpha_{i}(X_{i}(b)-T_{i})\right]}.$
(15)

These are not plane-waves in $b$ or in the original $m_{i}$, indeed in terms
of them the waves have non-linear dispersion. The $X_{i}$ and $\alpha_{i}$ are
“linearizing” variables DSR ; DSR1 , in terms of which the solutions become
plane waves moving with constant speed in superspace.

By demoting the $\alpha_{i}$ to circumstantial constants we gain more than a
time variable in the quantum theory: we enlarge the space of solutions.
Instead of being restricted to (15) we are allowed superpositions:

$\psi(b,T_{i})=\int d\alpha_{i}{\cal
A}(\alpha_{i})\exp{\left[i\frac{3V_{c}}{l_{P}^{2}}\alpha_{i}(X_{i}(b)-T_{i})\right]}.$
(16)

Among these, we highlight coherent/squeezed states, with ${\cal
A}(\alpha_{i})=\sqrt{{\bf N}(\alpha_{i0},\sigma_{i})}$ (where $\bf N$ denotes
a normal distribution) resulting in:

$\displaystyle\psi_{S}(b,T_{i})$ $\displaystyle=$ $\displaystyle{\cal
N}\psi(b,T_{i};\alpha_{i0})\exp\left[-\frac{\sigma_{i}^{2}(X_{i}-T_{i})^{2}}{(l_{P}^{2}/3V_{c})^{2}}\right]$
(17)

where $\sigma_{i}$ can be dialled anywhere between the limiting cases of a
plane wave (${\cal A}\propto\delta(\alpha-\alpha_{i0})$) and the “light-ray”
$\psi\propto\delta(T_{i}-X_{i}$) following from ${\cal A}\propto{\rm const}$
(which is actually required in quasi-topological theories GB ; MZ , and
expresses the conformal constraint). Note the Heisenberg uncertainty relation
between each $\alpha$ and its associated time:

$\sigma_{T}\sigma_{\alpha}\geq\frac{l_{P}^{2}}{6V_{c}}.$ (18)

Coherent states $\psi_{C}(b,T_{i})$ are states (17) with:

$\sigma^{2}_{i}=\frac{l_{P}^{2}}{6V_{c}},$ (19)

saturating (18) and equally spreading the uncertainties.

In fact, we do not even need to construct general states from monochromatic
waves. Eq. (9) can be rewritten as:

$\left(\frac{\partial}{\partial X_{i}}-\frac{\partial}{\partial
T_{i}}\right)\psi=0,$ (20)

(suggesting the conserved current, $j^{0}=j^{1}=|\psi|^{2}$, with the
concomitant probability interpretation). Then, the general solution is of the
form:

$\psi(b)=F(T_{i}-X_{i}).$ (21)

By setting $F$, for example, a Gaussian, or any other such localized function,
we thus have an expression for a solitonic wave, showing no dispersion in $X$.
All of these solutions are normalizable with the “naive” inner product. No
longer do we need to blame the trivial inner product for the non-
normalizability of the monochromatic solutions.

Having a time variable and this larger space of solutions is of paramount
importance for making contact with reality. Plane waves imply a uniform
distribution in $b$, hardly a prediction, but they are also not immediately
physical. We need both a time variable and the ability to superpose plane
waves into normalized peaked distributions to recover something minimally
physical. In fact the semi-classical limit is recovered for the coherent
states $\psi_{C}(b,T_{i})$, for which the second Hamilton equation:

$\dot{T}_{i}=\\{T_{i},H\\}=-\frac{1+3w_{i}}{2}Na^{-3w_{i}}\alpha_{i}^{\frac{3w_{i}-1}{2}}$
(22)

is true not only on average (an expression of Ehrenfest’s theorem), but with
minimal and balanced uncertainties in the complementary $\alpha_{i}$ and
$T_{i}$ appearing on the two sides of (22). Their peak, then, follows the
classical trajectory since it is easy to check that $\dot{T}_{i}=\dot{X}_{i}$,
with (22) and (13), is equivalent to the Einstein equations111Note the
difference of interpretation with regards to Chern-Simons time Chopin  for
$w_{i}=-1$. Here $X_{i}=\Im(Y_{CS})$ is not a time, but a spatial variable.
Time, instead, is the conjugate of $\alpha=3/\Lambda$. The two can be loosely
confused only because if $\psi$ is peaked, its peak moves along the outgoing
“light-ray” $T_{i}=X_{i}$..

But now we have an embarrassment of riches. Let $\bm{\alpha}$ be a vector
representing the whole set of relevant constants and $\bm{T}$ their
conjugates. The various components of $\bm{T}$ are a priori independent
variables, so we have a plethora of times instead of a single one. True, (22)
implies that classically and on-shell they are all related by a lapse function
redefinition. Indeed, we see that for $\Lambda$, radiation and dust, $T_{i}$
is proportional to volume time misner , conformal time and cosmological proper
time, respectively. Quantum mechanically, however, the situation is more
complex.

The implication in this Letter is that there is not one “Schrodinger” equation
(our examples being merely limiting cases) but a PDE in multiple times running
concurrently, obtained by taking the Hamiltonian in (2) and applying the
replacements:

$H\left[b,a^{2};\bm{\alpha}\rightarrow
i\frac{l_{P}^{2}}{3V_{c}}\frac{\partial}{\partial\bm{T}}\right]\psi=0.$ (23)

Its general solutions are:

$\psi(b)=\int d\bm{\alpha}{\cal
A}(\bm{\alpha})\exp{\left[-i\frac{3V_{c}}{l_{P}^{2}}\bm{\alpha}\bm{T}\right]}\psi_{s}(b;\bm{\alpha}),$
(24)

where $\psi_{s}(b;\bm{\alpha})$ solves the WDW equation with constant
$\bm{\alpha}$. However, for a Hamiltonian carving up phase space into regions
dominated by a single constant, the readjustment of quantum clocks across such
regions is seamless if we assume coherent states in all $\alpha_{i}$ and
factorization:

${\cal A}(\bm{\alpha})=\prod_{i}\sqrt{{\bf N}(\alpha_{0i},l_{P}^{2}/6V_{c})}.$
(25)

Then, the $\psi_{s}(b;\bm{\alpha})$ is a piecewise plane wave in the
$X_{i}(b)$ associated with each dominant $\alpha_{i}$. Each piece engages with
the phase associated with the corresponding $T_{i}$ (hence, the approximate
single-time Schrodinger equation), producing a wave-packet describing the
correct classical limit, as above. If ${\cal A}(\bm{\alpha})$ factorizes, all
the other times factorize too, and stop describing the $X$ evolution. We end
up with a classical limit in terms of different $X$ and $T$ in each region,
but classically they are all equivalent.

The semiclassical limit is reassuring, but our proposal also permits a wild
west of alternative states, wherein lies its true novelty. For example, we
could set up a state made up of the superposition of coherent packets centred
on different ${\bm{\alpha}}_{0}$ and shifted by various ${\bm{T}}_{0}$:

$\psi_{E}=\sum_{{\bm{\alpha}}_{0},{\bm{T}}_{0}}\psi_{C}(b,{\bm{T}}-{\bm{T}}_{0};{\bm{\alpha}}_{0}).$
(26)

Such a single state provides the perfect realization of an “ensemble” of
Universes, with all combinations of constants realized, and even different
realizations (shifted in $\bm{T}$) for the same set of constants. Could this
provide a solid basis for an explanation of the values of the constants
Barrowtip ?

But we could also consider non-semiclassical states. The possibility that
${\cal A}(\bm{\alpha})$ does not factorize, so that the constants are in an
entangled state, adds a layer of complexity to such an ensemble. Moreover
entangled states would leave a memory of non-dominant times at any epoch,
since they would not factor out in (24), as they do in the argument above
(with phenomenology currently under investigation future ). We could also
consider hybrid states, for example a factorizable state which is coherent in
$G_{N}$ (associated with radiation and matter domination), but squeezed or
even plane-wave-like in $\Lambda$ (and so “smudged” in $\Lambda$-time). Given
the prospect of imminent $\Lambda$-domination accelexp , this should raise an
eschatological alarm. Could Quantum Cosmology be in the future, and the future
be timeless future ?

Even avoiding such extremes, new phenomenology may be inevitable. Note that
the fact that the general solution (21) contains outgoing-only waves is a
reflection of the “horizon problem” (recall that on-shell $b=\dot{a}/N$, i.e.
the inverse of the comoving Hubble scale). Indeed $\dot{T}=\dot{X}$ implies
that a positive $b$ must decrease if $w>-1/3$, and increase otherwise. We
therefore have a “bounce” in connection space each time the dominant equation
of state crosses $w=-1/3$. Thus, from the point of view of our proposal, we
are living in “interesting times” for two reasons. First, components with
different $w$ currently have comparable densities, implying that the Universe
is in the process of switching clocks (from a dust/$G_{N}$ clock to a
$\Lambda$ one). Second, the change in $w$ is such that we must have just
emerged from a bounce in $b$, something that can only have happened on another
occasion if there was a period of primordial inflation.

The full implications will be studied elsewhere future , but here we
illustrate the issues with a toy model involving $\Lambda$ and radiation.
Setting $g=b^{2}+k$, we have $g^{2}\geq g_{0}^{2}=\frac{4}{3}\Lambda m_{r}$,
and with $\alpha=3/\Lambda$ in (1) we find after some algebra plane waves in:

$X(b;\Lambda m_{r},\pm)=\int
db\frac{1}{2}\left(g\pm\sqrt{{g^{2}-\frac{4}{3}\Lambda m_{r}}}\right)$ (27)

where the $+/-$ refer to $\Lambda$/radiation domination. It is not hard to
then reproduce the argument presented above for a seamless handover of times
when $g\gg g_{0}$ for the two branches. However, for $g\approx g_{0}$, even
with (25), we see that (24) does not factorize (due to the $X$ dependence on
$\Lambda m_{r}$), with possible observational implications.

Other, more speculative novelties may exist. Even in regions where one
ingredient $i$ dominates, there may be more than one constant affecting the
dynamics. By construction, or by judicious choice of the state, it may be that
only one of the associated times is relevant. But there are also more complex
states juggling several times in the same region of phase space. This is
particularly relevant for scalar fields in models with deconstantized $G_{N}$.
Another possible peculiarity involves models with bounces in $b$, even well
away from the bounce. These models will see the wave packet pass a given $b$
more than once. Classically this happens at different “times”, but quantum
mechanically we have the inevitable interference terms. Could there be cross-
talk between different times due to them? Even more speculatively, what would
the situation be in an ensemble of the form (26), where a given $b$ might be
crossed by different “realizations” moving in opposite directions?

We close by briefly commenting on work akin to ours. We stress that our
proposal is related, but not a twin of the suggestion that some constants may
be complementary to each other LeeGL ; eigenconstant . Here, the complementary
of a constant is, rather, a hypothetical momentum, which turns out to be a
measure of time. Our proposal is also orthogonal to the idea of ebb and flow
in Chern-Simons time ebb , since here we interpreted the latter as a spatial
variable. Our formulation of unimodular gravity and its generalization to
other constants is also related to, but different from the ideas in vikman  (a
full formulation, leading to the spatial constancy of $\alpha_{i}$ beyond
minisuperspace is deferred to gens ). On the other hand we may make direct
contact with the suggestion that the laws of physics could be evolving (e.g.
selftaught ) and that time could be dynamical (e.g. marina , or even rovelli
). So as to introduce time variability in the laws of physics we must know
what time is. Our definition, therefore, has immediate implications upon this
important discussion at the core of physics.

Acknowledgments. I would like to thank Stephon Alexander, Steffen Gielen,
Chris Isham, Lee Smolin and Tom Zlosnik for discussions and advice. This work
was supported by the STFC Consolidated Grant ST/L00044X/1.

## References

  * (1) C. J. Isham, “Canonical quantum gravity and the problem of time,” NATO Sci. Ser. C 409, 157-287 (1993).
  * (2) K. Kuchar, “Time and interpretations of quantum gravity,” in Proceedings of the 4th Canadian Conference on General Relativity and Relativistic Astrophysics’, World Scientic, Singapore, 1992.
  * (3) J. D. Barrow and F. J. Tipler, “The Anthropic Cosmological Principle,” Oxford University Press, 1988.
  * (4) P. J. E. Peebles and B. Ratra, Rev. Mod. Phys. 75, 559-606 (2003).
  * (5) S. Weinberg, Rev. Mod. Phys. 61, 1-23 (1989).
  * (6) A. Padilla, “Lectures on the Cosmological Constant Problem,” [arXiv:1502.05296 [hep-th]].
  * (7) C. Isham and J. Magueijo, in preparation.
  * (8) W. G. Unruh, Phys. Rev. D40, 1048 (1989).
  * (9) M. Henneaux and C. Teitelboim, Physics Letters B 222, 195 (1989).
  * (10) C. W. Misner, Phys. Rev. 186, 1328 (1969); Phys. Rev.1 86, 1319 (1969).
  * (11) S. Gielen and L. Menéndez-Pidal, Class. Quant. Grav. 37, no.20, 205018 (2020).
  * (12) N. Kaloper and A. Padilla, Phys. Rev. Lett. 112, no.9, 091304 (2014).
  * (13) L. Lombriser, Phys. Lett. B 797, 134804 (2019).
  * (14) J. Magueijo, Phys. Rev. D 102, no.4, 044034 (2020).
  * (15) G. Amelino-Camelia, Int. J. Mod. Phys. D 11, 35-60 (2002); J. Magueijo and L. Smolin, Phys. Rev. Lett. 88, 190403 (2002).
  * (16) S.W. Hawking, Physics Letters B, 134, 403 (1984).
  * (17) D. Kimberly, J. Magueijo and J. Medeiros, Phys. Rev. D 70, 084007 (2004).
  * (18) S. Alexander, M. Cortês, A. R. Liddle, J. Magueijo, R. Sims and L. Smolin, Phys. Rev. D 100, no.8, 083506 (2019); Phys. Rev. D 100, no.8, 083507 (2019).
  * (19) J. Magueijo and T. Złośnik, Phys. Rev. D 100, no.8, 084036 (2019).
  * (20) S. S. Chern and J. Simons, Ann. Math. 99, 48 (1974); G. V. Dunne, “Aspects of Chern-Simons theory,” [arXiv:hep-th/9902115 [hep-th]].
  * (21) R. Jackiw, Conf. Proc. C 8306271, 221-331 (1983) MIT-CTP-1108.
  * (22) H. Kodama, Phys. Rev. D 42, 2548-2565 (1990); J. Magueijo, [arXiv:2012.05847 [gr-qc]].
  * (23) L. Smolin and C. Soo, Nucl. Phys. B 449, 289-316 (1995).
  * (24) J. Magueijo et al, “The phenomenology of multi-time”, in preparation.
  * (25) L. Smolin, Class. Quant. Grav. 33, no.2, 025011 (2016).
  * (26) J. D. Barrow and J. Magueijo, Phys. Rev. D 99, no.2, 023509 (2019).
  * (27) J. Magueijo and L. Smolin, Universe 5, 84 (2019).
  * (28) P. Jiroušek, K. Shimada, A. Vikman and M. Yamaguchi, JCAP 04, 028 (2021).
  * (29) S. Alexander, W. J. Cunningham, J. Lanier, L. Smolin, S. Stanojevic, M. W. Toomey and D. Wecker, “The Autodidactic Universe,” [arXiv:2104.03902 [hep-th]].
  * (30) M. Cortês and L. Smolin, Phys. Rev. D 90, no.8, 084007 (2014); Phys. Rev. D 90, no.4, 044035 (2014).
  * (31) C. Rovelli and M. Smerlak, Class. Quant. Grav. 28, 075007 (2011).


# Predictive Temporal Attention on Event-based Video Stream for Energy-
efficient Situation Awareness

Yiming Bu, Jiayang Liu and Qinru Qiu
Department of EECS, Syracuse University, Syracuse, USA
{ybu104, jliu206<EMAIL_ADDRESS>*This work is partially supported by the
National Science Foundation I/UCRC ASIC (Alternative Sustainable and
Intelligent Computing) Center (CNS-1822165).

###### Abstract

The Dynamic Vision Sensor (DVS) is an innovative technology that efficiently
captures and encodes visual information in an event-driven manner. By
combining it with event-driven neuromorphic processing, the sparsity in DVS
camera output can result in high energy efficiency. However, similar to many
embedded systems, the off-chip communication between the camera and processor
presents a bottleneck in terms of power consumption. Inspired by the
predictive coding model and expectation suppression phenomenon found in human
brain, we propose a temporal attention mechanism to throttle the camera output
and pay attention to it only when the visual events cannot be well predicted.
The predictive attention not only reduces power consumption in the sensor-
processor interface but also effectively decreases the computational workload
by filtering out noisy events. We demonstrate that the predictive attention
can reduce $46.7\%$ of data communication between the camera and the processor
and reduce $43.8\%$ computation activities in the processor.

##  I INTRODUCTION

While human eyes are sensitive to color information, certain animals like
birds and insects excel in perceiving objects in rapid motion[1]. Taking
inspiration from this phenomenon, event cameras are designed to detect changes
in light intensity. Among them, the Dynamic Vision Sensor (DVS) is a
representative event-based sensor that naturally detects moving objects and
filters out redundant information automatically[2]. Unlike conventional RGB
cameras that capture image frames consisting of the light intensity of every
pixel, DVS cameras measure per-pixel brightness changes and output events
asynchronously. In the remainder of this paper, we refer to these events as
visual events. The DVS cameras outperform their traditional counterparts in
several aspects[2]. For instance, the microsecond temporal resolution of DVS
cameras allows them to avoid motion blur, and their high dynamic range (140dB
compared to the 60dB of conventional cameras) enables effective functioning in
low illumination conditions. They are particularly appealing for their low
energy dissipation and are commonly used in energy-constrained systems where
situation awareness is required. The visual events generated by the DVS camera
are typically sparse. When processed by event-driven neuromorphic computing
models, this sparsity allows for further reduction in computation energy.

Existing neuromorphic computing utilizes Spiking Neural Network (SNN) models,
which are known as the third generation of neural networks. Spiking neurons
communicate and process information through discrete electrical impulses
referred to as spikes. Information is encoded using the rate or interval of
these spikes. Similarly to biological neurons, SNNs are inherently
asynchronous, and process information in an event-driven manner [1]. SNNs have
been successfully applied to various tasks [3][4][5][6], demonstrating a
potential in achieving high accuracy with low power consumption. They have
been utilized for classification [7][3] and object detection [8][9] of DVS
camera output, enabling real-time visual analysis.

A DVS camera is connected to the back-end processor via a USB port. State of
the art neuromorphic processors such as IBM TrueNorth [10] and Intel Loihi
[11][12] exhibit power consumption well below 1 Watt. The power consumption of
a DVS camera can also be as low as 30mW[13]. The energy efficiency of data
communication over USB channel ranges from approximately $3.1\times 10^{-8}$
to $1.4\times 10^{-7}$ Joules per bit [14]. With a 128x128 image resolution, a
conventional camera capturing 30 frames per second generates 107,520 pixels
per second. Assuming that only 10% of the pixels experience brightness change
and it requires 3 bytes to describe an event, the DVS camera will generate
approximately 258kb data per second. Consequently the transmission power
required to move the data amounts to approximately 8 mW to 36 mW. Compared to
the power consumption of the DVS camera and back-end processor, the power
consumption of their data interface is not negligible.

It is widely accepted that human brain functions as a predictive
machine[15][16]. The predictive coding model [17] proposes that the brain
consistently generates and updates a ”mental model” of the environment to
anticipate input signals from the senses. The predictive behavior contributes
to the exceptional energy efficiency of the human brain. Neuroscience
experiments have demonstrated that neural activity decreases when a stimulus
is expected to occur [18]. In other words, when the environment is
predictable, the brain suppresses sensory inputs and relies on prior knowledge
to infer information about the surrounding world. This phenomenon is known as
expectation suppression.

Inspired by the predictive coding and expectation suppression, we propose a
predictive attention model to control the output of the DVS camera. The model
consists of two main components: a predictor that anticipates future visual
events and an attention generator that produces a control signal to regulate
the output of the DVS sensor. The attention should be guided by the high-level
goal of the application. In this study, our objective is to maintain a
situation awareness, so attention is generated to ensure the accuracy of
predicted visual events.

To implement the visual event predictor, we employ an SNN-ANN hybrid auto-
encoder. The SNN-based encoder models the feedforward neural pathway, while
the artificial neural network (ANN) based decoder represents the feedback
neural pathway. Depending on the attention level, the predictor receives
inputs from either the DVS camera or its own predictions. When attention is
high, the sensed events are utilized, whereas when attention is low, the
predicted events are used. The predicted visual events can be seen as an
internal belief about the world, therefore, we consider improving the
prediction accuracy as equivalent to enhancing situation awareness.

The conventional approach to assess the quality of video prediction is by
calculating the mean-square error (MSE) between the predicted frame and the
ground truth frame [19] [20]. However, this metric is insufficient for
measuring the similarity between two sets of visual events. In this study, we
introduce a novel metric called event similarity (Esim) and demonstrate that
the Esim score demonstrates a stronger correlation with human visual
perception compared to MSE.

Our contributions can be summarized as follows:

  * •

We present an attention-assisted hybrid architecture that combines SNN and ANN
for continuous visual event prediction. Compared to an ANN-based prediction
model, our hybrid model on average improves the quality of prediction by
$14.5\%$.

  * •

We have developed a novel evaluation metric to assess the similarity between
two sets of visual events. Unlike metrics such as MSE, our Esim measurement
provides enhanced robustness against random noises and focuses more on the
differences in salient features.

  * •

The proposed attention model effectively gates $46.7\%$ of the communication
between the DVS camera and the back-end processor, while maintaining an
acceptable level of situation awareness. Compared to random gating at the same
rate, a system utilizing attention-based gating achieves a $81.1\%$ higher
level of situation awareness.

  * •

Moreover, by incorporating the predicted visual event, which contains
significantly less noise and emphasizes salient features, we can reduce
computation in the SNN-based encoder network by $43.8\%$, leading to more
energy savings.

##  II BACKGROUND AND RELATED WORKS

###  II-A DVS data representation

A DVS camera generates an event when the logarithmically-scaled change in
light intensity exceeds a certain threshold:

$\begin{cases}pos\ event\ \ log(I_{t+1})-log(I_{t})>th.\\\ neg\ event\ \
log(I_{t})-log(I_{t+1})>th.\end{cases}$ (1)

The event recording process is pixel-specific, where each pixel is evaluated
independently of others. Each event consists of a 2D location of the event
$(x,y)$, a timestamp $t$, and a polarity $p$ indicating whether it is a
positive (light intensity increase) or negative event (light intensity
decrease). Therefore, an event is recorded as a four-element tuple,
$e=\left\\{x,y,t,p\right\\}$, which is commonly known as Address Event
Representation (AER).

Typically, visual events are grouped into event frames. Two grouping
strategies are often used, i.e., dividing frames based on equally time
interval or equal number of events. In this study, we employ the first
strategy. Each event frame is represented as a 2-dimensional ternary array,
$F\in B^{H\times W}$, where $B\in\left\\{0,1,-1\right\\}$ corresponds to no
event, positive or negative events respectively. The dimension $H\times W$
represents the camera resolution. All events within the same frame are
generated at approximately the same time, so their individual timestamps are
disregarded during processing.

###  II-B Prediction Models

Video prediction has been extensively studied in the field of deep learning.
The objective is to generate a sequence of future frames, denoted as
$\mathbf{Y}=(\hat{Y}_{t+1},\hat{Y}_{t+2},...,\hat{Y}_{t+m})$ given a sequence
of $n$ preceding frames $\mathbf{X}=(X_{t-n},...,X_{t-1},X_{t})$ [21].
Training the video sequence model is self-supervised, as it does not require
additional label information.

To predict future frames, a model needs to possess a strong understanding of
the spatiotemporal correlations within a video. Models such as recurrent
neural networks (RNN) and Long Short-Term Memory (LSTM) [22][23][24][25] are
commonly used. While the outcome of a prediction model could be stochastic ,
the majority of works provide deterministic predictions [21]. The MSE metric
is often used to evaluate the prediction quality. However, when dealing with
event-based data, this method exhibits limitations due to the extreme sparsity
of hot pixels (i.e., pixels with positive or negative events). Under the MSE
metric, most frames appear ”similar” to each other since large regions are
often devoid of events. To the best of our knowledge, no prior works have been
designed specifically to predict event frames from a DVS camera.

###  II-C Attention mechanism

The attention mechanism has demonstrated significant success in machine
learning, leading to improved model performance [26][27][28] and reduced
computation resources [29][28]. Generally, there are three types of attention
mechanisms. Spatial-wise attention coupled with channel-wise attention is
commonly used in convolutional neural network models[28], while recurrent
neural networks often employ temporal attention[30]. Although relatively new,
the attention mechanism has also been successfully applied in SNNs. For
example, in [31], SNN spatial attention is used for object recognition, while
[32] focuses on discovering more informative period of events in a redundant
DVS event stream using a squeeze-and-excitation[26] method. However, the soft
attention employed in these works does not help reduce computation complexity.
In contrast, our model generates hard attention, ensuring that the input is
queried only when necessary, thus reducing communication and computation
overhead.

##  III METHOD

The overall architecture of the proposed visual event attention system
comprises two main components: an autoencoder-based visual event predictor and
an attention generator. The back-end processor buffers incoming visual events
and combines them into event frames. The predictor takes an input event frame
and predicts the next event frame. The attention generator evaluates the
quality of the prediction and generates a binary gating signal to either block
or unblock the camera output. The autoencoder takes a combination of the
camera-sensed events and the predicted events as its input.

Figure 1: Overall visual predictive attention system architecture.

###  III-A Visual Event Predictor

Figure 2: Visual event predictor.

The architecture of the visual event predictor is illustrated in Fig 2. The
predictor consists of three components: an SNN-based encoder, a sequence of
residual blocks, and a decoder for next frame generation. Each layer of the
encoder is a convolutional network of LIF neurons. The encoder extracts
temporal and spatial features from the incoming sequence of event frames. The
residual blocks are ANN-based convolutional layer with skip structure. The
decoder consists of multiple layers of ANN-based deconvolutional networks for
generating the next frame. Feedforward skip connections are inserted between
corresponding layers in the encoder and decoder. The output dimensions of each
layer are labeled beside the blocks. While the SNN encoder leverages the
sparsity in the DVS camera output, the ANN-based forecasting and frame
generation provide more flexibility and higher trainability as the neurons
output continuous values. Through experimentation, we discovered that the skip
connection is crucial for achieving accurate predictions. Without the
feedforward skip information, the sparse activities within the network often
result in an all-zero output. Each pixel in the predicated frame may be in 3
possible states, i.e. positive, negative or no-event, a softmax layer is used
at the output to predict the probability of the three possible states.
Finally, events are sampled based on the probability.

We employ leaky-integrate-and-fire (LIF) neuron in the SNN encoder. The LIF
neuron maintains a membrane potential $u$ through leaky integration of input
spikes. When the membrane potential exceeds the threshold, the neuron
generates an output spike and resets the membrane potential. The dynamics of
the LIF neuron are described by Equations 2 and 3. Here $u^{t,n}$ represents
the membrane potential of a neuron in layer $n$ at time $t$, $y^{t,n}$ denotes
the neuron’s output, and $V_{th}$ denotes the threshold of the neuron’s
membrane potential. In a convolutional-based SNN, each neuron is solely
connected to and receives signals from other neurons within the receptive
fields $F$ from the previous layer.

$u^{t,n}=\displaystyle\sum\limits_{j\in F}w_{j}^{n}y_{j}^{t,n-1}+\tau
u^{t-1,n}(1-y^{t-1,n})$ (2)

$y^{t,n}=\begin{cases}1\text{ if }u^{t,n}>V_{th}\\\ 0\text{
otherwise}\end{cases}$ (3)

The predictor is trained using Cross Entropy(CE) loss:

$\mathcal{L}=-\frac{1}{H\cdot W}\displaystyle\sum\limits_{i=0}^{H\cdot
W}\sum\limits_{j=0}^{2}{y}_{i,j}\log\widehat{y}_{i,j},$ (4)

where $\widehat{y}_{i,j}$ and ${y}_{i,j}$ represent the estimated and ground
truth probability of pixel $i$ having event $j$.

The activation function of the spiking neuron (Equation 3) is non-
differentiable, making it unsuitable for direct application of conventional
backpropagation for training. A commonly used approach is to employ a
surrogate function that approximates the gradient of the spikes’ activation
function. In this study, we utilize the arctangent function and its derivative
as surrogates for calculating the derivative of the SNN activation function.
The expressions for these surrogates are provided below, where $x$ and $y$
epresent the input and output of the activation function, respectively:

$y=\frac{1}{\pi}\arctan(\frac{\pi}{2}\alpha x)+\frac{1}{2}$ (5)
$\frac{\partial y}{\partial x}=\frac{\alpha}{2(1+(\frac{\pi}{2}\alpha
x)^{2})}$ (6)

when $\alpha$ becomes larger, the function resembles a activation function for
spiking neuron, for our experiment, we set $\alpha=2$.

Fig 3 presents an example of the predicted frame and the ground truth frame.
In the image, the red and blue dots represent positive and negative events,
respectively. Interestingly, we observed that the predicted frame exhibits
significantly less noise compared to the original frame. This can be
attributed to the fact that noise events are random and lack any discernible
pattern, making them inherently unpredictable.

(a) sensor input

(b) model prediction

Figure 3: The prediction can reduce the noise and reflect a region activity

###  III-B Measuring Event Similarity

A reliable measure of similarity between two event frames should accurately
capture the level of overlap in their visual events. Taking inspiration from
the Intersection over Union (IOU) metric commonly used for evaluating object
bounding boxes, we propose a novel metric called Event Similarity (Esim).
Given two event frames, denoted as $F_{1}$ and $F_{2}$ each comprising
positive and negative visual events, we begin by defining the intersection of
$F_{1}$ and $F_{2}$ as $F_{1}\bigcap
F_{2}=\left\\{e=\left\\{x,y,p\right\\}\mid e\in F_{1},e\in F_{2}\right\\}$.
Furthermore, we define the union of $F_{1}$ and $F_{2}$ as $F_{1}\bigcup
F_{2}=\left\\{e=\left\\{x,y,p\right\\}\mid e\in F_{1}\parallel\in
F_{2}\right\\}$. The intersection of two event frames only includes events
that are present in both frames, while the union contains events from either
of the frames. Esim is defined by the following equation:

$Esim(F_{1},F_{2})=\frac{\mid F_{1}\cap F_{2}\mid}{\mid F_{1}\cup F_{2}\mid}$
(7)

where $\mid\cdot\mid$ is $\ell{1}$ norm. Formula denotes the ratio of common
visual events over the number of total visual events in these two frames.

Using Esim to assess the quality of predictions by comparing them to the
ground truth frame has certain limitations. Firstly, if the predicted event
shifts from its ground truth location, regardless of the extent of the shift,
there is no intersection between the two and the Esim score will be 0.
Furthermore, due to its high sensitivity, the output of a DVS camera often
contains a considerable number of noise events, sometimes even surpassing the
number of informative events. Consequently, discrepancies with noise events
can significantly diminish the Esim score of the predicted frame.

The aforementioned limitations arise from the fact that Esim relies on exact
event matching at a per-pixel level, which is unnecessary. To introduce some
tolerance towards random noise and small shifts, we compare the events within
a small region surrounding the pixel. Similar to using polarized filter for
image noise reduction, we define the polarity intensity($PI$) of a
${h}\times{w}$ region centered at locate $(x,y)$ as

$PI(x,y)=\frac{|E_{pos}|-|E_{neg}|}{h\cdot w}$ (8)

where $|E_{pos}|$ and $|E_{neg}|$ represent the number positive and negative
events within the specified region. Each pixel is polarized by checking the
polarity intensity of its surrounding area against a threshold using the
following equation:

$P^{\prime}(x,y)=\begin{cases}1&PI(x,y)>th\\\ -1&PI(x,y)<-th\\\
0&PI(x,y)<\lvert th\rvert\end{cases}$ (9)

The Esim calculated based on the array with polarized value of the pixel is
refer to as Region Esim. Region Esim with a $n\times{n}$ polarization size is
denoted as Esim$n$. The threshold can be tuned under different scenario(we use
0.2501, which proves to be fine value for our experiment for Esim2 and Esim4).
Because one frame consists of random noise while the other does not. Despite
featuring similar events related to ball movement, when using Esim, their
similarity is nearly zero. However, when employing Region Esim, their
similarity score exceeds to a reasonable level.

We compared Esim with Mean Square Similarity (MSS), which is mathematically
defined as $1-MSE$. The reference frame and the comparison frames are provided
in Fig. 4, with the white box indicating the location of the ball in the
reference frame. The metric scores are reported in Table I. It is evident that
MSS fails to accurately depict the magnitude of the displacement of the ball’s
location. In contrast, Esim-based metrics is more suitable in measuring the
similarity between the two event frames.

Figure 4: The prediction can reduce the noise and reflect a region activity. The left most figure is the reference, others are figures where the ball was moved to the right by a percentage of radius(the black box denotes the reference position). TABLE I: Metric Score on Noised Shifted Ball offset |  $0\%$ |  $5\%$ |  $10\%$ |  $25\%$
---|---|---|---|---
1-MSE |  0.9949 |  0.9898 |  0.9848 |  0.9798
Esim |  0.599 |  0.3321 |  0.1438 |  0.0022
Esim2 |  0.9719 |  0.5230 |  0.2433 |  0.0000
Esim4 |  0.9882 |  0.5980 |  0.3412 |  0.0000

###  III-C Prediction Evaluator and attention generator

Video prediction exhibits robust spatial and temporal consistency. By feeding
the predicted frame back into the autoencoder’s input, we can continuously
generate a sequence of future frames, extending beyond just the immediate next
frame. However, as this iterative process continues, errors accumulate,
leading to a degradation in the prediction quality. At this time, it becomes
crucial to ”take a look” (or pay attention) to the sensor stream once again in
order to perceive the actual information and rectify the prediction errors.
The challenge lies in determining when to look.

As discussed in previous section, the region Esim can be used to measure the
similarity between the predicted and the ground truth frame. However, since
the system lacks access to the ground truth without ”looking” at the camera
output, we must estimate the Region Esim score based on the predicted frame.
Fortunately, this is feasible because a poorly generated event frame usually
is noisy and the events do not form a clear contour of the object. By
examining the event distribution within the predicted frame, we can make a
rough estimation of how closely the prediction resembles the ground truth.
Moreover, the quality of predictions often deteriorates significantly when the
object changes its current trajectory. By observing the sequence of event
frames, we can make a general prediction of when such a change is likely to
occur.

In this work, we developed a neural network based evaluator to predict the
Region Esim score of the predicted frame. The architecture of the evaluator is
illustrated in figure 5. The evaluator architecture is a typical SNN feature
extractor. It takes channel-wise concatenation of the first frame and the
current prediction as input and the concatenation was continuously fed into
the evaluator for 10 time steps and then outputs a confidence of current
prediction. The training of the evaluator is same as and independent from the
predictor.

Figure 5: Esim estimation model

##  IV Experiment Result

###  IV-A Experimental Setup

Our experiments were conducted on four datasets: BouncingBall, MovingMnist,
Visevent[33], and DVS128[34]. All of them contain sequences of event frames
from a DVS camera. Both BouncingBall and MovingMnist are synthetic datasets.
BouncingBall offers resolutions of 64x64 or 256x256 and features three balls
moving in straight path and bouncing off boundaries or each other upon
collision. MovingMnist, on the other hand, has a resolution of 64x64 and
showcases two MNIST digits bouncing within the scene. Visevent is specifically
designed for object tracking of pedestrians and vehicles in street scenarios.
DVS128 captures the upper body’s moving gestures of a man, with a resolution
of $128\times 128$. For each dataset, we use 50% of the sequences as training
set and the remaining as the testing set. During the training, the learning
rate is set to $1e-3$, the batch size to $8$, and a total of $2000$ epochs
were trained .

###  IV-B Predictor Performance

TABLE II:  Predictor Performance Dataset |  frame size |  SNN |  ANN
---|---|---|---
BouncingBall64 |  (64,64) |  0.801 |  0.7289
BouncingBall256 |  (256,256) |  0.589 |  0.5124
MovingMnist |  (64,64) |  0.691 |  0.6702
DVSGesture |  (128,128) |  0.502 |  0.3918
Visevent |  (256,256) |  0.317 |  0.2726

We tested our predictor on aforementioned datasets and reported the average
Esim4 score in Table II. We also trained an ANN predictor as a baseline. It
has a similar architecture except that an LSTM-based encoder is used to
replace the SNN encoder. However, due to the very sparse input, the accuracy
of ANN prediction is much lower than SNN. In addition, the ANN in general does
not support event-driven operation, hence will have more computation workload.

The predictor also demonstrates its ability to significantly reduce noise in
the input. To evaluate its noise tolerance, we manually added Gaussian noise
to the test set and passed the noisy frames through the predictor. We
introduced a term called ”relative Esim” to quantify the prediction’s
robustness. The relative Esim is calculated as the ratio of the Esim scores
between predictions with noisy input and predictions with clean input. By
gradually increasing the noise level, we observed the corresponding changes in
the relative Esim. The trends for the four datasets are depicted in Fig 6. As
we can see, the predictor can tolerate about from $47.3\%$ (bouncingball64) to
$52.1\%$ (visevent) noise, while maintain a prediction that is $90\%$ similar
to the prediction with clean input. In other words, the predictor is capable
of filtering out from $32.0\%$(bouncingball64) to $43.8\%$ (visevent)noisy
events. This can be turned into reduced computation in the event-driven SNN
encoder.

Figure 6: Relative performance reduction as noise increases.

###  IV-C Attention Directed Situation Awareness

We further apply the evaluator to continuously estimate the Esim score of the
prediction. If the score falls below a threshold, the camera output is
attended, and a sensed frame is used to replace the poorly predicted frame
while the prediction continues. Fig 7 shows an example of predicted frames.
The 3th frame is replaced by the sensed information.

Figure 7: Attention assisted sequence prediction of MovingMnist (The red box
indicates the input of next time step.)

Fig 8 illustrates how the prediction quality varies throughout the attention
process. The black lines represent the actual Esim score of the prediction
after comparing it with the ground truth, while the grey lines represent the
Esim score estimated by the evaluator. The red dot indicates the predicted
Esim right after the sensor is attended, and the green line represents the
threshold. As we can see that the evaluator gives quite good estimation of the
Esim score of the prediction. Each time the sensor input is received, the
prediction quality is reinforced. We can also see that, when the predictor
maintaing an acceptable relative performance, with the proposed attention
mechanism, the sensor processor interface can be gated for about
76.7%(bouncingball64) of time on average, leading to energy reduction in data
communication. Even in the most complex scenario(visevent), the gating rate
can still be up to 46.7% . Due to the fact that the prediction contains
significantly fewer noise events compared to the sensor input, when the sensor
is gated, the encoder experiences lower computational activity. This
observation aligns with the neurological observation that neurons exhibit
reduced activity when attention levels are low.

Figure 8: Evaluator assisted Situation Awareness. Figure 9: Average score
comparison to random gated sensor and periodically gated sensor.

We further show that the attention is carefully timed to ensure situation
awareness. To evaluate this, we compared our attention mechanism with a
randomly and a periodicallytimed attention mechanism using the same attention
rate. By adjusting the evaluation threshold, we varied the rough attention
rate, which represents the percentage of time that the sensor is open, ranging
from 90% to 10%. We used the quality of the prediction as a metric to measure
situation awareness. The results are presented in Fig. 9. It is evident that,
on average, our attention model provides 81% better awareness compared to the
random attention mechanism.

###  IV-D Conclusion

In this work, we have designed a generative model for predicting the next
frame using inputs from a DVS camera. This model combines the strengths of a
deep SNN encoder and an ANN decoder. We have introduced Esim as an evaluation
metric to assess the quality of the predictions. An evaluator is developed to
estimate the Esim score from predicted event frames and controls the output of
the DVS camera based on the attention level. We have compared the performance
of the predictive model with both the proposed attention mechanism and a
random attention mechanism. The results demonstrate that the proposed
attention mechanism achieves superior situation awareness. Additionally, the
predicted frames contain significantly less noise, resulting in reduced
computational requirements. Overall, the proposed predictive attention
mechanism offers a promising approach to decrease both communication and
computational energy consumption.

## Acknowledgement

This work is partially supported by the National Science Foundation I/UCRC
ASIC (Alternative Sustainable and Intelligent Computing) Center (CNS-1822165).

## References

  * [1] M. F. Land, “Motion and vision: why animals move their eyes,” _Journal of Comparative Physiology A_ , vol. 185, pp. 341–352, 1999.
  * [2] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis _et al._ , “Event-based vision: A survey,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 1, pp. 154–180, 2020.
  * [3] M. Yao, H. Gao, G. Zhao, D. Wang, Y. Lin, Z. Yang, and G. Li, “Temporal-wise attention spiking neural networks for event streams classification,” _CoRR_ , vol. abs/2107.11711, 2021. [Online]. Available: https://arxiv.org/abs/2107.11711
  * [4] D. Roy, P. Panda, and K. Roy, “Synthesizing images from spatio-temporal representations using spike-based backpropagation,” _CoRR_ , vol. abs/1906.08861, 2019. [Online]. Available: http://arxiv.org/abs/1906.08861
  * [5] H. Kamata, Y. Mukuta, and T. Harada, “Fully spiking variational autoencoder,” _CoRR_ , vol. abs/2110.00375, 2021. [Online]. Available: https://arxiv.org/abs/2110.00375
  * [6] Y. Hu, J. Binas, D. Neil, S. Liu, and T. Delbrück, “DDD20 end-to-end event camera driving dataset: Fusing frames and events with deep learning for improved steering prediction,” _CoRR_ , vol. abs/2005.08605, 2020. [Online]. Available: https://arxiv.org/abs/2005.08605
  * [7] X. She, S. Dash, and S. Mukhopadhyay, “Sequence approximation using feedforward spiking neural network for spatiotemporal learning: Theory and optimization methods,” in _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022. [Online]. Available: https://openreview.net/forum?id=bp-LJ4y_XC
  * [8] H. Liu, D. P. Moeys, G. Das, D. Neil, S.-C. Liu, and T. Delbrück, “Combined frame- and event-based detection and tracking,” in _2016 IEEE International Symposium on Circuits and Systems (ISCAS)_ , 2016, pp. 2511–2514.
  * [9] D. Tedaldi, G. Gallego, E. Mueggler, and D. Scaramuzza, “Feature detection and tracking with the dynamic and active-pixel vision sensor (davis),” in _2016 Second International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP)_ , 2016, pp. 1–7.
  * [10] F. Akopyan, J. Sawada, A. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla, N. Imam, Y. Nakamura, P. Datta, G.-J. Nam _et al._ , “Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip,” _IEEE transactions on computer-aided design of integrated circuits and systems_ , vol. 34, no. 10, pp. 1537–1557, 2015.
  * [11] M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, Y. Liao, C.-K. Lin, A. Lines, R. Liu, D. Mathaikutty, S. McCoy, A. Paul, J. Tse, G. Venkataramanan, Y.-H. Weng, A. Wild, Y. Yang, and H. Wang, “Loihi: A neuromorphic manycore processor with on-chip learning,” _IEEE Micro_ , vol. 38, no. 1, pp. 82–99, 2018.
  * [12] G. Orchard, E. P. Frady, D. B. D. Rubin, S. Sanborn, S. B. Shrestha, F. T. Sommer, and M. Davies, “Efficient neuromorphic signal processing with loihi 2,” _CoRR_ , vol. abs/2111.03746, 2021. [Online]. Available: https://arxiv.org/abs/2111.03746
  * [13] M. Zaffar, S. Ehsan, R. Stolkin, and K. D. McDonald-Maier, “Sensors, SLAM and long-term autonomy: A review,” _CoRR_ , vol. abs/1807.01605, 2018. [Online]. Available: http://arxiv.org/abs/1807.01605
  * [14] S. Lurye, “Surges in mobile energy consumption during USB charging and data exchange – Securelist,” 7 2016. [Online]. Available: https://securelist.com/surges-in-mobile-energy-consumption-during-usb-charging-and-data-exchange/75297/
  * [15] A. Clark, “Whatever next? predictive brains, situated agents, and the future of cognitive science,” _Behavioral and brain sciences_ , vol. 36, no. 3, pp. 181–204, 2013.
  * [16] T. P. Lillicrap, A. Santoro, L. Marris, C. J. Akerman, and G. Hinton, “Backpropagation and the brain,” _Nature Reviews Neuroscience_ , vol. 21, no. 6, pp. 335–346, 2020.
  * [17] Y. Huang and R. P. Rao, “Predictive coding,” _Wiley Interdisciplinary Reviews: Cognitive Science_ , vol. 2, no. 5, pp. 580–593, 2011.
  * [18] K. S. Walsh and D. P. McGovern, “Expectation suppression dampens sensory representations of predicted stimuli,” _Journal of Neuroscience_ , vol. 38, no. 50, pp. 10 592–10 594, 2018.
  * [19] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [20] Y. Zhou, H. Dong, and A. El Saddik, “Deep learning in next-frame prediction: A benchmark review,” _IEEE Access_ , vol. 8, pp. 69 273–69 283, 2020.
  * [21] S. Oprea, P. Martinez-Gonzalez, A. Garcia-Garcia, J. A. Castro-Vargas, S. Orts-Escolano, J. Garcia-Rodriguez, and A. Argyros, “A review on deep learning techniques for video prediction,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 6, pp. 2806–2826, 2022.
  * [22] C. Lee, A. K. Kosta, A. Z. Zhu, K. Chaney, K. Daniilidis, and K. Roy, “Spike-flownet: event-based optical flow estimation with energy-efficient hybrid neural networks,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16_. Springer, 2020, pp. 366–382.
  * [23] W. Lotter, G. Kreiman, and D. D. Cox, “Deep predictive coding networks for video prediction and unsupervised learning,” _CoRR_ , vol. abs/1605.08104, 2016. [Online]. Available: http://arxiv.org/abs/1605.08104
  * [24] Y. Wang, Z. Gao, M. Long, J. Wang, and P. S. Yu, “Predrnn++: Towards A resolution of the deep-in-time dilemma in spatiotemporal predictive learning,” _CoRR_ , vol. abs/1804.06300, 2018. [Online]. Available: http://arxiv.org/abs/1804.06300
  * [25] R. Villegas, J. Yang, Y. Zou, S. Sohn, X. Lin, and H. Lee, “Learning to generate long-term future via hierarchical prediction,” _CoRR_ , vol. abs/1704.05831, 2017. [Online]. Available: http://arxiv.org/abs/1704.05831
  * [26] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” _CoRR_ , vol. abs/1709.01507, 2017. [Online]. Available: http://arxiv.org/abs/1709.01507
  * [27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _CoRR_ , vol. abs/1706.03762, 2017. [Online]. Available: http://arxiv.org/abs/1706.03762
  * [28] S. Woo, J. Park, J. Lee, and I. S. Kweon, “CBAM: convolutional block attention module,” _CoRR_ , vol. abs/1807.06521, 2018. [Online]. Available: http://arxiv.org/abs/1807.06521
  * [29] A. Papadopoulos, P. Korus, and N. D. Memon, “Hard-attention for scalable image classification,” _CoRR_ , vol. abs/2102.10212, 2021. [Online]. Available: https://arxiv.org/abs/2102.10212
  * [30] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” _CoRR_ , vol. abs/1409.3215, 2014. [Online]. Available: http://arxiv.org/abs/1409.3215
  * [31] M. Cannici, M. Ciccone, A. Romanoni, and M. Matteucci, “Attention mechanisms for object recognition with event-based cameras,” _CoRR_ , vol. abs/1807.09480, 2018. [Online]. Available: http://arxiv.org/abs/1807.09480
  * [32] M. Yao, H. Gao, G. Zhao, D. Wang, Y. Lin, Z. Yang, and G. Li, “Temporal-wise attention spiking neural networks for event streams classification,” _CoRR_ , vol. abs/2107.11711, 2021. [Online]. Available: https://arxiv.org/abs/2107.11711
  * [33] X. Wang, J. Li, L. Zhu, Z. Zhang, Z. Chen, X. Li, Y. Wang, Y. Tian, and F. Wu, “Visevent: Reliable object tracking via collaboration of frame and event flows,” _arXiv preprint arXiv:2108.05015_ , 2021.
  * [34] A. Amir, B. Taba, D. Berg, T. Melano, J. McKinstry, C. Di Nolfo, T. Nayak, A. Andreopoulos, G. Garreau, M. Mendoza _et al._ , “A low power, fully event-based gesture recognition system,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7243–7252.