# Learning a Neural Diff for Speech Models

###### Abstract

As more speech processing applications execute locally on edge devices, a set
of resource constraints must be considered. In this work we address one of
these constraints, namely over-the-network data budgets for transferring
models from server to device. We present neural update approaches for release
of subsequent speech model generations abiding by a data budget. We detail two
architecture-agnostic methods which learn compact representations for
transmission to devices. We experimentally validate our techniques with
results on two tasks (automatic speech recognition and spoken language
understanding) on open source data sets by demonstrating when applied in
succession, our budgeted updates outperform comparable model compression
baselines by significant margins.

Index Terms: automatic speech recognition, edge ML, incremental updates,
communication cost

##  1 Introduction

00footnotetext: The first two authors contributed equally.

With the expanding capabilities of smart devices and speakers, a recent trend
has emerged to enable local speech processing for virtual assistants such as
Amazon Alexa and Google Assistant [1, 2]. While local execution of speech
models yields benefits such as reduced latency and enhanced reliability [3],
the edge-first architecture poses a series of challenges traditionally not
faced for a server-centric one. Prior work has emphasized the two most
manifest constraints: limited compute resources which impacts latency [4, 5]
and memory constraints for storage of large models locally [6]. However,
another challenge for on-device speech which has yet to be studied to the same
degree is imposing over-the-network, or over-the-air (OTA), data budgets. One
of the primary benefits of a server-based speech architecture is the ease of
deployment for new model generations or improvements which can occur near
instantaneously and at low cost. In industry settings, having regular updates
remains a critical component of the machine learning flywheel: enhanced model
releases deliver more accuracy and functionality, which drives user adoption
of the technology, which in turn leads to more data for building superior
models. Meanwhile, users are able to experience the added benefits or
functionality immediately when new model generations are deployed to the
server. When speech models are executed locally, however, an individual who
has multiple speech enabled devices (e.g. phone, smart speaker, vehicle) would
require a download for each when a model is available. Speech models can be
large while an individual’s network data plan might be restrictive (e.g. 1 GB
of data monthly before higher data rates apply). Particularly relevant for
remote or developing regions, where network availability can be limited,
smaller artifacts can also provide faster download times.

To complicate matters, another trend which has complimented that of on-device
technology is the adoption of end-to-end neural architectures for speech [7,
8, 9, 10]. While often superior in performance, fully neural speech
architectures can have a large footprint (120+ million parameters [11]) which
are expensive with respect to data transfer. Furthermore, by their end-to-end
nature, these architectures blur the roles of model components when compared
to their traditionally decomposed counterparts. For example, a conventional
DNN-HMM ASR system with a disjoint acoustic model, pronunciation model, and
language model might only require an update to a segment of just a single
subsystem (e.g. adjusting a weighted finite state transducer). Or, consider a
two-phase spoken language understanding (SLU) system: ASR followed by natural
language understanding (NLU) where only the NLU component might require an
update for a new functionality release. In such cases only the new NLU module,
which can be highly compressed [12, 13], would need to be transferred. Thus,
the shift to end-to-end neural models motivates the need to develop holistic
approaches for efficient edge model update methods addressing data budget
constraints. In this paper we consider the problem of incrementally updating
models for evolving and expanding training data sets with minimal OTA transfer
costs. We refer to this here as _neural diff learning_ , analogous to a source
code diff for a software patch.

Communication efficient model updating has been studied in several related
contexts, one of the most prominent being with respect to data-parallel
distributed model training. There is a rich literature on effective gradient
compression schemes [14, 15, 16, 17, 18] which are designed to limit the I/O
bottleneck of communication and aggregation of gradient information applied to
a model in training at each step of gradient descent. In contrast, with our
setting we aim to train just a single, communication efficient artifact to
apply to the prior model. Our setting is also closely related to the modern
continual learning (CL) and federated learning (FL) class of problems which
both have broad bodies of work [19, 20, 21, 22]. CL typically addresses the
problem of never-ending model adaptation over an evolving data stream with a
particular focus on preventing _catastrophic forgetting_ of learnings acquired
over no longer accessible data [23, 24]. FL draws on ideas from both the CL
and distributed training spaces to enable data-private, decentralized
learning. Among a plethora of other challenges, FL often requires techniques
for efficient two-way communication and synchronization for all client-devices
in order to be practical at scale [25]. In our study here, we will retain a
growing data set server-side and focus on server-to-device communication,
though our techniques draw inspiration from these fields, particularly the
regularization based methods of CL [23].

Our contributions include formalizing our neural diffing problem; presenting
two architecture agnostic approaches for minimizing communication cost for
model updates; and applying and evaluating these techniques in speech settings
for two tasks, ASR and SLU classification, on publicly available data sets.

##  2 Problem Setting

To encapsulate the notion of growing and evolving training data for a
production voice-assistant system, we define a series of $T$ data sets
$D_{1}\subset D_{2}\subset\cdots\subset D_{T}$. Each data set corresponds to a
model $\mathcal{F}$ in sequence of model generations
$\mathcal{F}_{1},\mathcal{F}_{2},\dots,\mathcal{F}_{T}$ which are built
successively on the data sets and deployed at regular intervals.

###  2.1 Objectives

In a traditionally unconstrained setting, we simply aim to train the best
possible model over the given data with respect to a loss function
$\mathcal{L}$. In this scenario each model generation is defined approximately
as

$\mathcal{F}_{t}^{*}\approx\operatorname*{argmin}_{\mathcal{F}}\sum_{(x,y)\in
D_{t}}\mathcal{L}\left(\mathcal{F}_{t}(x),y\right).$

For convenience we will hereafter denote
$\mathcal{L}_{D}\left(\mathcal{F}\right)$ as an abbreviation for the loss of a
model $\mathcal{F}$ over some dataset $D$.

In the constrained setting we consider in this work, our goal now, broadly
construed, is to train and deploy a series of models
$\mathcal{F}_{1},\dots,\mathcal{F}_{T}$ where at each generation $t$ we aim to
minimize the predictive performance regret under the OTA budget when compared
to the unconstrained ideal $\mathcal{F}_{t}^{*}$. Namely,

$\text{regret}_{t}=\mathcal{L}_{D_{t}}\left(\mathcal{F}_{t}\right)-\mathcal{L}_{D_{t}}\left(\mathcal{F}_{t}^{*}\right)$

subject to the following constraint

$\Delta(\mathcal{F}_{t},\mathcal{F}_{t+1})\leq B,$

where $\Delta$ measures the diff size between subsequent model generations and
$B$ is the OTA update budget per generation. Last, we assume the device comes
prepackaged with an initial generation model $\mathcal{F}_{0}$ out of the box
or at initial application download.

###  2.2 Data Set Evolution

Our data sets consist of $(x,y)$ pairs where $x$ is a feature vector (e.g.
audio features) and $y$ is a label for the instance (e.g. $y\in\left[C\right]$
for a classification task such as speech commands, $y\in\Sigma^{\ell}$ for
sequence labeling such as open vocabulary speech recognition). We consider the
evolution of the data set sequence under two settings.

####  2.2.1 Setting 1

With this evolution there is a global distribution $\mathcal{D}$ from where
data at all times is sampled. However, as the training data becomes more
abundant, i.e. $|D_{t}|\leq|D_{t+1}|$, the task and associated loss becomes
more precisely defined. We expect the model to improve performance on its task
at each generation because it simply is exposed to more training data.

####  2.2.2 Setting 2

In this scenario, the evolution of the dataset shifts the task. Thus each
generation has $D_{t}\sim\mathcal{D}_{t}$ but
$\mathcal{D}_{t}\neq\mathcal{D}_{t+1}$. These different distributions could
simply shift the sampling frequency of certain examples (e.g. a trending word
or phrase) or be the introduction of a new $y$ class label (e.g. a new command
now enabled with the release of a new functionality). While still
$|D_{t}|\leq|D_{t+1}|$, there is not necessarily an expectation that
$\mathcal{L}_{D_{t}}\left(\mathcal{F}_{t}^{*}\right)\geq\mathcal{L}_{D_{t+1}}\left(\mathcal{F}_{t+1}^{*}\right)$
because the task difficulty could be different during the evolution.

##  3 Diffing Approach

Our general approach to updating models using a low OTA budget will be to
directly learn a small diff (or patch) to apply to the previous model
generation over the expanded training data. We do this by anchoring the
previous model generation, then learning the diff object $\mathcal{G}_{t+1}$
as free weights

$\mathcal{F}_{t+1}\approx\mathcal{F}_{t}\oplus\operatorname*{argmin}_{|\mathcal{G}_{t+1}|\leq
B}\mathcal{L}_{D_{t+1}}\left(\mathcal{F}_{t}\oplus\mathcal{G}_{t+1}\right)$

where $\oplus$ is an operator expressing the application of the learned diff
and $|\mathcal{G}|$ designates the size of a diff $\mathcal{G}$. As a result,
only $\mathcal{G}_{t+1}$ needs to be transferred to the local device where it
can then be applied to $\mathcal{F}_{t}$.

In this work, our implementation will view $\mathcal{G}$ as a series of matrix
masks each corresponding to a weight matrix of $\mathcal{F}$. Each mask is
applied through a simple additive or multiplicative operation (see below) to
its matching anchored weight matrix during the forward pass of training.

The masks will be subject to a size budget (not dimension budget) bounding
their footprint. Below we detail two generic ways to learn budgeted diffs, one
leveraging sparsity and another using weight hashing. In a traditional model
compression setting, these techniques have limits before noticeable accuracy
degradation appears; however, when applied to diff objects instead of original
model weights, our motivating intuition is that the diff objects can be
compressed to a much higher degree by exploiting and reusing the information
content of the prior model generation. As a result, the methods can
potentially surpass the observed raw model compression barriers.

###  3.1 Sparse Diffing

Many works have demonstrated that deep neural networks can be trained
effectively using sparse representations [26, 6, 27, 28, 29]. Sparsity is
attractive not only for reducing a model’s inference operations but also its
memory footprint. In this work we apply sparsity to the additive weight masks
of $\mathcal{G}$. Specifically, we use the pruning approach of [28] which
presents a method to train to any desired sparsity by gradually “zero-ing out
weights” of smallest magnitude during the training process. For a target
sparsity $s_{f}$, initial pruning step $r_{0}$, and final pruning step
$r_{f}$, we prune the $s_{r}$ fraction of diff weights of least magnitude
according to schedule

$s_{r}=s_{f}\cdot\left(1-\left(1-\frac{r-r_{0}}{r_{f}-r_{0}}\right)^{3}\right).$

Since we prune diff weights and not model weights, the updated model on the
device remains fully dense after the application of the diff. A major strength
of this approach is that as mask weights are pruned, the resulting model
weights are not “zeroed-out” as they would be when sparse pruning a model
directly. Rather, the weights are simply substituted with the prior
generation’s weights. So while prior works show that a model performance
begins to degrade at a particular level of sparsity ([6, 29] show $\sim 50$%
for their speech models), we circumvent this compression barrier by retaining
a fully dense model for inference. Furthermore, the sparse diffing process has
the ability to dynamically adjust sparsity structure generation over
generation (see Figure 1). While a subset of non-zero diff weights are used
for a particular update, an entirely different set may be selected during the
next generation. Last, observe that the approach remains generic. It makes no
assumptions about the roles of model components and does not restrict which
weights of the neural network can be updated.

\begin{overpic}[width=390.25534pt]{diffing_sparse}
\put(33.8,106.5){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{G}_{1}$}}}
\put(105.0,106.25){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{F}_{0}$}}}
\put(135.0,106.5){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{G}_{1}$}}}
\put(163.75,106.25){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{F}_{1}$}}}
\par\put(33.8,60.0){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{G}_{2}$}}}
\put(105.0,59.75){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{F}_{1}$}}}
\put(135.0,60.0){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{G}_{2}$}}}
\put(163.75,59.75){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{F}_{2}$}}}
\par\put(33.8,13.75){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{G}_{3}$}}}
\put(105.0,13.5){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{F}_{2}$}}}
\put(135.0,13.75){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{G}_{3}$}}}
\put(163.75,13.5){\parbox{0.0pt}{\scalebox{0.85}{$\mathcal{F}_{3}$}}}
\end{overpic} Figure 1: Sparse diffing procedure. Each model $\mathcal{F}_{t}$
has learned sparse weight masks $\mathcal{G}_{t+1}$ applied to arrive at the
next generation model $\mathcal{F}_{t+1}$.

###  3.2 Hash Diffing

Weight hashing is another technique which has been shown effective in
compressing large neural networks [30]. Here we adapt the technique for the
diffing setting by using it to compress the weight masks of a diff object.
Given a weight matrix $W_{t}$ of $\mathcal{F}_{t}$ we train an array of diff
weights $A_{t+1}$ so that

$W_{t+1}^{(i,j)}=W_{t}^{(i,j)}+|W_{t}^{(i,j)}|\sum_{k=1}^{K}A_{t+1}\left[h_{k}(i,j)\right],$

where $W_{t+1}^{(i,j)}$ is the $(i,j)$th entry of the updated weights for
model generation $\mathcal{F}_{t+1}$ and $h_{k}$ is one of $K$ hash functions
randomly hashing weight positions into locations of $A$. The size of $A$ is
chosen to match one’s budget $B$, which is much smaller than the size of $W$.
As a result, there is a high rate of collisions leading to heavy randomized
weight sharing. Because of the high collision rate, we add the scaling factor
$|W_{t}^{(i,j)}|$ 111This is a unique aspect in the diffing case over the
original method because we leverage predefined weights from prior generations.
This factor essentially changes what would be an additive update operation
into a multiplicative one, with the added benefit that when expressed this way
allows the trainable weight distribution to be centered at zero (i.e. identity
is 0 instead of 1). While not strictly necessary, we found empirically that
magnitude scaling was more effective than plain additive weight updates
because it is much easier to shift a pair of colliding weights proportionally
the same amount rather than absolutely (e.g. consider a large weight colliding
with a smaller one).

While weight hashing is a compelling method capable of delivering significant
compression, it does not appear to be the common method of choice for
representing neural models for real-time inference applications on-device.
Among other reasons, the limited adoption can be attributed to the random
access patterns in memory the hash functions produce when accessing weights,
while modern system hardware, on the other hand, is designed to support
efficient spatial locality for bulk matrix operations at high speed.
Interestingly, in this setting where the hashing is used only to represent the
diff object during transmission before it is decompressed and applied to the
previous model on device, we do not encounter this issue. In addition, as [31,
30] suggest, the use of multiple hash functions ($K>1$) can provide extra
modeling capacity to the compression but come at the cost of multiple memory
accesses and addition operations to reconstruct a single weight. Again, in the
diffing instance, this reconstruction work can occur offline before being
applied to the dense model used at inference time. Last, generation over
generation, we have the ability to reseed our hash functions to prevent the
same weight positions from always colliding. This adaptation is analogous to
the sparse method selecting different non-zero diff weights each generation.

##  4 Experimental Results

Table 1: Comparison of an ideal model’s performance baseline (BL) against
baseline model compression (BLC) and incremental diff application (Diff) under
different OTA budgets compression ratios (CR) at each model generation.

| LibriSpeech (WER) | Fluent Speech (Classification Acc.)
---|---|---
|  |  |  Sparse |  Hash |  |  |  Sparse |  Hash |  Static
CR |  |  1 |  10 |  10 |  |  1 |  20 |  40 |  20 |  40 |  inf
| hours | BL | BLC | Diff | BLC | Diff | # dom. | BL | BLC | Diff | BLC | Diff | BLC | Diff | BLC | Diff | $\text{BL}_{\infty}$
Gen 0 | 480 | 9.93 | - | - | - | - | 3 | 97.5 | - | - | - | - | - | - | - | - | -
Gen 1 | 576 | 9.37 | 9.79 | 9.28 | 11.39 | 9.10 | 4 | 97.2 | 94.1 | 96.5 | 88.5 | 96.6 | 95.3 | 96.6 | 91.2 | 95.6 | 87.8
Gen 2 | 672 | 8.65 | 8.81 | 8.38 | 11.02 | 8.23 | 5 | 96.8 | 93.8 | 96.6 | 88.2 | 96.3 | 95.1 | 96.0 | 91.3 | 93.8 | 77.7
Gen 3 | 768 | 8.01 | 8.63 | 8.05 | 10.55 | 8.15 | 6 | 97.0 | 93.7 | 96.4 | 87.5 | 95.0 | 94.3 | 94.9 | 88.7 | 92.9 | 73.1
Gen 4 | 864 | 7.61 | 8.54 | 7.81 | 9.67 | 8.00 | - |  |  |  |  |  |  |  |  |  |
Gen 5 | 960 | 7.46 | 8.14 | 7.52 | 8.98 | 7.89 | - |  |  |  |  |  |  |  |  |  |

We evaluated our neural diffing approach on two speech tasks each
encapsulating a data evolution scenario described in Section 1.

###  4.1 Speech Recognition

For our first set of experiments, our objective is a classic speech
recognition task. The goal will be to deploy a series of ASR models, each
trained over more data than the previous to simulate a classic ML flywheel
setting where as more users interact with a product, the data set continues to
expand. We capture Setting 1 with this experiment, where the underlying ASR
task remains constant but training data becomes more abundant at each model
release. We use the LibriSpeech data set, which is a speech corpus of read
English audio books [32]. The data set provides $\sim 960$ hours of training
data and we use its “clean” test set to report our word error rate (WER). Our
generation 0 model was trained on $480$ hours of data and then we increment
the size of the training data with $96$ additional hours (10%) at each
generation.

Our model is a basic unidirectional LSTM RNN-T architecture with 6
transcription network layers and 2 prediction network layers. Each LSTM has
1024 units, the prediction network uses a dropout rate of $0.3$, and both
networks apply a final projection layer of size 640. We apply a single joint
layer with tanh activation and use a 2500 word-piece vocabulary with a 256
dimensional input embedding. The resulting model has 64.1M parameters. Audio
feature extraction uses 64 dimension log-filterbank energy with feature frames
downsampled by 3 and stacked with a stride size of two to produce 30
millisecond frames. We also apply SpecAugment masking [33] on training data.
Decoding is conducted using standard RNN-T beam search with a width of 16.

For each generation and compression method, we built baseline models (BLC) to
demonstrate the added benefit of using diffing as opposed to attempting to
compress a fresh model trained from scratch at the same compression rate. We
applied sparsity and hashing methods at a compression factor of $10\times$
(i.e. objects will be $90\%$ sparse or hash to a $90\%$ smaller array). For
sparsifying our baseline models, we prune over steps 20k-35k while for diff
objects we prune earlier from 1k-11k. For our hashing, each weight matrix of
the models are hashed to weight arrays $\nicefrac{{1}}{{10}}$ th their
original size with three hash functions, and as mentioned in Section 3.2, we
seed the hash functions differently at each generation. All diff weights are
initialized to zero at the start of training across experiments. The results
are shown in Table 1 where we also include unconstrained baselines trained
from scratch (BL).

One observes that both the sparse and hash diffs outperform their comparable
compressed baselines BLC generation over generation. However, the sparse
diffing method consistently maintains nearly matching WERs with BL at each
generation (e.g. 7.46 BL vs 7.52 Diff at 960 hours) whereas, hash diffing
performance begins to diverge, resulting in a final WER degradation of 8%
relative against the unconstrained ideal BL. Also note that for our setup Hash
BLC had difficulty training to satisfactory levels.

###  4.2 SLU Classification

For our second set of experiments, we demonstrate the effectiveness of our
diffing approach on a speech command SLU task. We use the publicly available
Fluent Speech Commands (FSC) [34] dataset to conduct these experiments. FCS
contains $\sim 30$k utterances from $97$ unique speakers and is decomposed
into train, dev, and test sets with $\sim 23$k, $3$k, and $4$k utterances
respectively. Each FCS utterance is one of $248$ phrasings, each of which is
associated with a domain of intent: Lighting, Heat, Volume, Bring, Language
Change, or Music.

The modeling objective for the experiment is one of classification where we
aim to correctly identify each utterance as one of a predfined set of
supported phrases. To emulate Setting 2, where the underlying task changes at
each generation, we evolve the system by behaving as if the local assistant
edge device is being enabled with new functionality releases which need speech
support. We do this with three successive updates. Namely, our generation 0
model begins by only being trained on and classifying phrases among the
Lighting, Heat and Volume domains. At generation 2, we introduce the Bring
domain data and add support for its phrases. Likewise at generation 3, we add
in Language Change commands, and finally generation 4 introduces the Music
domain phrases. At each generation our accuracy metric is evaluated on the
subset of test data encompassing only those supported domains up to that
point.

Our model architecture here will be based on a contrastive approach with a
speech encoder and text encoder, much like the approach of [35]. The speech
encoder and text encoder jointly learn to embed representations from their
modality of input into a shared space where corresponding utterance acoustic
and text phrase embeddings are close with respect to the $L_{2}$ distance
while nonmatching are well separated. We apply the _triplet loss_ with online
batch mining [36] where batches contain audio utterances and the full set of
supported text phrases. At runtime, nearest neighbor search is conducted from
the speech encoder embedding against the reference embeddings of the supported
phrases generated by the text encoder. Our speech encoder is a $4$-layer LSTM
with $128$ units with a dense projection layer with $128$ units. We also use
the same audio feature extraction as performed with LibriSpeech data. Our text
encoder consists of a $2$-layer LSTM with $128$ units again with a projection
output layer, and it also has $64$-dimension input embeddings over a
vocabulary of 4500 word pieces. The resulting architecture has 1.1M
parameters. The recurrent layers are trained with a dropout rate of $0.2$, and
we used the Adam optimizer coupled with a simple hold and decay learning
rate222An exception was the hash diffs which were altered to hold a higher
rate for longer in some instances and converged before decay.. Our sparsity
schedule for all sparse training operates over steps 2k-12k.

For this set of experiments, we ran with two different levels of
compression/data budgets, $20\times$ and $40\times$. Because of the relative
simplicity of the task, we can apply these higher compression levels, but
also, they serve to magnify the effect of the diffing approaches over their
baselines. We also included a static baseline ($\text{BL}_{\infty}$) where we
do not update the model at all to show the performance drop at each step. The
results are shown in Table 1 using phrase classification accuracy as the
measure for model performance. As with the LibriSpeech experiments, we found
the sparse methods for diffing maintain high accuracy at each generation with
the most extreme $40\times$ compression only degrading accuracy by 2% absolute
with respect to BL, a 7.5% improvement over its baseline BLC. While we see
moderate success with hash diffing over baselines BLC, it was not as
pronounced as sparsity. It is also important to note that for this task,
hashing achieved better accuracy results for simply compressing the baseline
model over sparsity (e.g. $95.1$ vs. $93.8$ for 5 domains) . Furthermore,
during the training process we found hash diffing much more challenging to
tune, a lack of consistency across updates, and a more expensive training
process (both higher compute time per step and more steps to converge). The
significance of these findings highlight that while a compression method might
be a promising technique to directly shrink the footprint of a model trained
from scratch, the method might not directly translate into success at
compressing a diff.

##  5 Conclusion

We address the problem of limited network OTA budgets for transferring
generations of speech models from server to device. We detail two architecture
agnostic diff methods based on sparsity and hashing, which learn compact
representations to increment a model locally. We show these methods can
compress to diff objects to high degrees, up to $40\times$ in some cases,
without significant impact on predictive performance. We find that not all
compression techniques adapt equally well for diffing, and in future work we
would like to see more approaches proposed with additional update mechanisms
introduced beyond basic additive and multiplicative schemes.

## References

  * [1] J. Li, H. Hu, and Y. Gong, “Improving RNN transducer modeling for end-to-end speech recognition,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 114–121, 2019.
  * [2] I. McGraw, R. Prabhavalkar, R. Alvarez, M. G. Arenas, K. Rao, D. Rybach, O. Alsharif, H. Sak, A. Gruenstein, F. Beaufays, and C. Parada, “Personalized speech recognition on mobile devices,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 5955–5959, 2016.
  * [3] T. N. Sainath and E. al., “A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6059–6063, 2020.
  * [4] Y. He and E. al., “Streaming end-to-end speech recognition for mobile devices,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019.
  * [5] J. Macoskey, G. P. Strimel, and A. Rastrow, “Bifocal neural asr : exploiting keyword spotting for inference optimization,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [6] R. Pang, T. N. Sainath, R. Prabhavalkar, S. Gupta, Y. Wu, S. Zhang, and C. C. Chiu, “Compression of end-to-end models,” _Proc. Interspeech_ , pp. 27–31, 2018.
  * [7] A. Graves, “Sequence transduction with recurrent neural networks,” _International Conference on Machine Learning (ICML)_ , 2012.
  * [8] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” _31st International Conference on Machine Learning (ICML)_ , pp. 3771–3779, 2014.
  * [9] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, attend and spell,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4960–4964, 2016.
  * [10] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. Mcdermott, S. Koo, and S. Kumar, “Transformer transducer: a streamable speech recognition model with transformer encoders and RNN-T loss,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 7829–7833, 2020.
  * [11] A. Narayanan, R. Prabhavalkar, C.-c. Chiu, D. Rybach, T. N. Sainath, and T. Strohman, “Recognizing long-form speech using streaming end-to-end models,” _IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , 2019.
  * [12] G. P. Strimel, K. M. Sathyendra, and S. Peshterliev, “Statistical model compression for small-footprint natural language understanding,” _Proc. Interspeech_ , 2018.
  * [13] K. M. Sathyendra, S. Choudhary, and L. Nicolich-henkin, “Extreme model compression for on-device natural language understanding,” _The 28th International Conference on Computational Linguistics (COLING)_ , 2020.
  * [14] F. Seide, H. Fu, J. Droppo, G. Li, D. Yu, D. Street, and H. District, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,” _Proc. Interspeech_ , pp. 1058–1062, 2014.
  * [15] N. Strom, “Scalable distributed DNN training using commodity GPU cloud computing,” _Proc. Interspeech_ , pp. 1488–1492, 2015.
  * [16] D. Alistarh, D. Grubic, J. Z. Li, R. Tomioka, and M. Vojnovic, “QSGD: communication-efficient SGD via gradient quantization and encoding,” _31st Conference on Neural Information Processing Systems (NIPS)_ , 2017.
  * [17] L. Yujun, S. Han, H. Mao, Y. Wang, and W. J. Dally, “Deep gradient compression: reducing the communication bandwidth for distributed training,” _In Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * [18] R. Spring, A. Kyrillidis, V. Mohan, and A. Shrivastava, “Compressing gradient optimizers via count-sketches,” _Proceedings of the 36th International Conference on Machine Learning (ICML)_ , pp. 5946–5955, 2019.
  * [19] M. D. Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “Continual learning : a comparative study on how to defy forgetting in classification tasks,” _arXiv preprint arXiv:1909.08383_ , 2019.
  * [20] S. Sadhu and H. Hermansky, “Continual learning in automatic speech recognition,” _Proc. Interspeech_ , pp. 1246–1250, 2020.
  * [21] H. B. Mcmahan and D. Ramage, “Communication-efficient learning of deep networks from decentralized data,” _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)_ , vol. 54, pp. 1273–1282, 2017.
  * [22] D. Dimitriadis, K. Kumatani, R. Gmyr, Y. Gaur, S. E. Eskimez, and O. M. Way, “A federated approach in training acoustic models,” _Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH)_ , pp. 981–985, 2020.
  * [23] Z. Li and D. Hoiem, “Learning without forgetting,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 12, pp. 2935–2947, 2018.
  * [24] M. Farajtabar and A. Mott, “Orthogonal gradient descent for continual learning,” _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , vol. 108, pp. 3762–3773, 2020.
  * [25] P. Kairouz, H. B. Mcmahan, Z. Charles, S. E. Rouayheb, Z. Garrett, P. B. Gibbons, A. Ozg, H. Qi, W. Song, J. Wang, and F. X. Yu, “Advances and open problems in federated learning,” _arXiv preprint arXiv:1912.04977_ , 2019\.
  * [26] B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Penksy, “Sparse convolutional neural networks,” _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 806–814, 2015.
  * [27] C. Louizos, M. Welling, and K. Kingma, “Learning sparse neural networks through l0 regularization,” _In Proceedings of the International Conference on Learning Representations (ICLR)_ , pp. 1389–1397, 2018.
  * [28] M. H. Zhu and S. Gupta, “To prune, or not to prune: exploring the efficacy of pruning for model compression,” _Proceedings of the International Conference on Learning Representations (ICLR) Workshop_ , 2018.
  * [29] K. Zhen, H. D. Nguyen, F.-J. Chang, A. Mouchtaris, A. Rastrow, and ., “Sparsification via compressed sensing for automatic speech recognition,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [30] W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, “Compressing neural networks with the hashing trick,” _Proceedings of the International Conference on Machine Learning (ICML)_ , pp. 2285–2294, 2015.
  * [31] Q. Shi, J. Petterson, G. Dror, J. Langford, A. Smola, and S. Vishwanathan, “Hash kernels for structured data,” _Journal of Machine Learning Research_ , vol. 10, pp. 2615–2637, 2009.
  * [32] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , pp. 5206–5210, 2015.
  * [33] D. S. Park, Y. Zhang, C. C. Chiu, Y. Chen, B. Li, W. Chan, Q. V. Le, and Y. Wu, “Specaugment on large scale datasets,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6879–6883, 2020.
  * [34] L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio, “Speech model pre-training for end-to-end spoken language understanding,” _Proc. Interspeech_ , pp. 814–818, 2019.
  * [35] B. Agrawal, M. Muller, M. Radfar, S. Choudhary, A. Mouchtaris, and S. Kunzmann, “Tie your embeddings down: cross-modal latent spaces for end-to-end spoken language understanding,” _arXiv preprint arXiv:2011.09044._ , 2020.
  * [36] A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” _arXiv preprint arXiv:1703.07737_ , 2017.


* [165] Biao Yuan, Zengde Deng, Na Geng, Yujie Chen, and Haoyuan Hu.  Practice summary: Cainiao optimizes the fulfillment routes of parcels.  INFORMS Journal on Applied Analytics, 2023.
  * [166] Jing Yuan, Yu Zheng, and Xing Xie.  Discovering regions of different functions in a city using human mobility and pois.  In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 186–194, 2012.
  * [167] Jing Yuan, Yu Zheng, Chengyang Zhang, Wenlei Xie, Xing Xie, Guangzhong Sun, and Yan Huang.  T-drive: driving directions based on taxi trajectories.  In Proceedings of the 18th SIGSPATIAL International conference on advances in geographic information systems, pages 99–108, 2010.
  * [168] Yuan Yuan, Jingtao Ding, Huandong Wang, Depeng Jin, and Yong Li.  Activity trajectory generation via modeling spatiotemporal dynamics.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4752–4762, 2022.
  * [169] Yuan Yuan, Jingtao Ding, Huandong Wang, Depeng Jin, and Yong Li.  Activity trajectory generation via modeling spatiotemporal dynamics.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4752–4762, 2022.
  * [170] Yuan Yuan, Huandong Wang, Jingtao Ding, Depeng Jin, and Yong Li.  Learning to simulate daily activities via modeling dynamic human needs.  arXiv preprint arXiv:2302.10897, 2023.
  * [171] Yuan Yuan, Huandong Wang, Jingtao Ding, Depeng Jin, and Yong Li.  Learning to simulate daily activities via modeling dynamic human needs.  In Proceedings of the ACM Web Conference 2023, WWW ’23, page 906–916, New York, NY, USA, 2023. Association for Computing Machinery.
  * [172] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al.  Glm-130b: An open bilingual pre-trained model.  In The Eleventh International Conference on Learning Representations, 2023.
  * [173] Guozhen Zhang, Zihan Yu, Depeng Jin, and Yong Li.  Physics-infused machine learning for crowd simulation.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2439–2449, 2022.
  * [174] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan.  Building cooperative embodied agents modularly with large language models.  arXiv preprint arXiv:2307.02485, 2023.
  * [175] Jun Zhang, Wenxuan Ao, Depeng Jin, Li Liu, and Yong Li.  A city-level high-performance spatio-temporal mobility simulation system.  2023\.
  * [176] Jun Zhang, Depeng Jin, and Yong Li.  Mirage: an efficient and extensible city simulation framework (systems paper).  In Proceedings of the 30th International Conference on Advances in Geographic Information Systems, pages 1–4, 2022.
  * [177] Junbo Zhang, Yu Zheng, and Dekang Qi.  Deep spatio-temporal residual networks for citywide crowd flows prediction.  In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  * [178] Mingyang Zhang, Haohao Fu, Yong Li, and Sheng Chen.  Understanding urban dynamics from massive mobile traffic data.  IEEE Transactions on Big Data, 5(2):266–278, 2017.
  * [179] Siyao Zhang, Daocheng Fu, Zhao Zhang, Bin Yu, and Pinlong Cai.  Trafficgpt: Viewing, processing and interacting with traffic foundation models.  arXiv preprint arXiv:2309.06719, 2023.
  * [180] Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu.  Evaluating the performance of large language models on gaokao benchmark.  arXiv preprint arXiv:2305.12474, 2023.
  * [181] Yifan Zhang, Cheng Wei, Shangyou Wu, Zhengting He, and Wenhao Yu.  Geogpt: Understanding and processing geospatial tasks through an autonomous gpt.  arXiv preprint arXiv:2307.07930, 2023.
  * [182] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  arXiv preprint arXiv:2303.18223, 2023.
  * [183] Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C Parkes, and Richard Socher.  The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning.  Science advances, 8(18):eabk2607, 2022.
  * [184] Yu Zheng, Licia Capra, Ouri Wolfson, and Hai Yang.  Urban computing: concepts, methodologies, and applications.  ACM Transactions on Intelligent Systems and Technology (TIST), 5(3):1–55, 2014.
  * [185] Yu Zheng, Quannan Li, Yukun Chen, Xing Xie, and Wei-Ying Ma.  Understanding mobility based on gps data.  In Proceedings of the 10th international conference on Ubiquitous computing, pages 312–321, 2008.
  * [186] Yu Zheng, Yuming Lin, Liang Zhao, Tinghai Wu, Depeng Jin, and Yong Li.  Spatial planning of urban communities via deep reinforcement learning.  Nature Computational Science, pages 1–15, 2023.
  * [187] Yu Zheng, Yanchi Liu, Jing Yuan, and Xing Xie.  Urban computing with taxicabs.  In Proceedings of the 13th international conference on Ubiquitous computing, pages 89–98, 2011.
  * [188] Yu Zheng, Xing Xie, Wei-Ying Ma, et al.  Geolife: A collaborative social networking service among user, location and trajectory.  IEEE Data Eng. Bull., 33(2):32–39, 2010.
  * [189] Yu Zheng, Lizhu Zhang, Xing Xie, and Wei-Ying Ma.  Mining interesting locations and travel sequences from gps trajectories.  In Proceedings of the 18th international conference on World wide web, pages 791–800, 2009.
  * [190] Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al.  Sotopia: Interactive evaluation for social intelligence in language agents.  arXiv preprint arXiv:2310.11667, 2023.
  * [191] Yu Zhou and Chuncheng Liu.  The logic and innovation of building digital twin city in xiong’an new area.  Urban Development Studies, 25(10):60–67, 2018.
  * [192] Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang.  A survey on model compression for large language models.  arXiv preprint arXiv:2308.07633, 2023.
  * [193] Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al.  Mindstorms in natural language-based societies of mind.  arXiv preprint arXiv:2305.17066, 2023.