# Age-specific transmission dynamics of SARS-CoV-2 during the first two years
of the pandemic

Otilia Boldea  Department of Econometrics and OR and CentER, Tilburg School of
Economics and Management, Tilburg University, Tilburg, The Netherlands  PhD
Department of Econometrics and OR and CentER, Tilburg School of Economics and
Management, Tilburg University, Tilburg, The Netherlands  Amir Alipoor
Department of Econometrics and OR and CentER, Tilburg School of Economics and
Management, Tilburg University, Tilburg, The Netherlands  MSc  Department of
Econometrics and OR and CentER, Tilburg School of Economics and Management,
Tilburg University, Tilburg, The Netherlands  Sen Pei  Department of
Environmental Health Sciences, Mailman School of Public Health, Columbia
University, New York, USA  PhD  Department of Environmental Health Sciences,
Mailman School of Public Health, Columbia University, New York, USA  Jeffrey
Shaman  Department of Environmental Health Sciences, Mailman School of Public
Health, Columbia University, New York, USA  Columbia Climate School, Columbia
University, New York, USA  PhD  Department of Environmental Health Sciences,
Mailman School of Public Health, Columbia University, New York, USA  Columbia
Climate School, Columbia University, New York, USA  Ganna Rozhnova  Julius
Center for Health Sciences and Primary Care, University Medical Center
Utrecht, Utrecht University, Utrecht, The Netherlands  BioISI—Biosystems &
Integrative Sciences Institute, Faculdade de Ciências, Universidade de Lisboa,
Lisbon, Portugal  Center for Complex Systems Studies (CCSS), Utrecht
University, Utrecht, The Netherlands  PhD111Corresponding authors:
Dr. Ganna Rozhnova
Julius Center for Health Sciences and Primary Care
University Medical Center Utrecht
P.O. Box 85500 Utrecht, The Netherlands
Email<EMAIL_ADDRESS>phone: +31 631117965

Dr. Otilia Boldea
Department of Econometrics and OR, and CentER
Tilburg School of Economics and Management, Tilburg University
P.O. Box 90153, Tilburg, The Netherlands
E-mail<EMAIL_ADDRESS>phone: +31 134663219 Julius Center for
Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht
University, Utrecht, The Netherlands  BioISI—Biosystems & Integrative Sciences
Institute, Faculdade de Ciências, Universidade de Lisboa, Lisbon, Portugal
Center for Complex Systems Studies (CCSS), Utrecht University, Utrecht, The
Netherlands

###### Abstract

During its first two years, the SARS-CoV-2 pandemic manifested as multiple
waves shaped by complex interactions between variants of concern, non-
pharmaceutical interventions, and the immunological landscape of the
population. Understanding how the age-specific epidemiology of SARS-CoV-2 has
evolved throughout the pandemic is crucial for informing policy decisions. We
developed an inference-based modelling approach to reconstruct the burden of
true infections and hospital admissions in children, adolescents and adults
over the seven waves of four variants (wild-type, Alpha, Delta, Omicron BA.1)
during the first two years of the pandemic, using the Netherlands as the
motivating example. We find that reported cases are a considerable
underestimate and a generally poor predictor of true infection burden,
especially because case reporting differs by age. The contribution of children
and adolescents to total infection and hospitalization burden increased with
successive variants and was largest during the Omicron BA.1 period. Before the
Delta period, almost all infections were primary infections occurring in naive
individuals. During the Delta and Omicron BA.1 periods, primary infections
were common in children but relatively rare in adults who experienced either
re-infections or breakthrough infections. Our approach can be used to
understand age-specific epidemiology through successive waves in other
countries where random community surveys uncovering true SARS-CoV-2 dynamics
are absent but basic surveillance and statistics data are available.

## Introduction

During the pandemic, the dynamics of SARS-CoV-2 demonstrated a complex spatio-
temporal pattern with multiple waves [1] and pronounced differences in the
age-specific burden of confirmed cases and hospitalizations [2, 3]. A notable
example is the age distribution of reported cases with much lower number of
cases among younger individuals in European countries, including the
Netherlands, reported in the first wave of the wild-type variant than in the
spike of infections caused by the Omicron BA.1 variant of concern (VoC) [4].

Understanding how the age-specific epidemiology of SARS-CoV-2 has changed
during the pandemic [5] is crucial for informing public health policy. For
instance, information about how infection burden varies by age and time and
the contribution of different age groups to transmission [6, 7] may inform
non-pharmaceutical interventions [8, 9, 10, 11, 12] like school- and non-
school-based measures [13], while understanding of the age-specific
hospitalization burden [14] underpins prioritization of vaccination [15, 16].
Some of this information can be provided by surveillance and serological
surveys. However, full reconstruction of the age-specific burden of infections
and hospitalizations in a country is complicated by several factors. Firstly,
under-reporting of cases is age-specific and time-varying due to peculiarities
of surveillance systems and testing policies, which varied across ages and
time as the pandemic progressed. This fact coupled with the evidence of
asymptomatic infection [17] undermines the ability of surveillance to capture
the true burden of infections among different age groups. Few countries
conduct random community PCR testing [18]. Secondly, nationally representative
serological surveys provide information on which subpopulations carry
antibodies to SARS-CoV-2 and thus could help to characterize prior infection
burden [19, 20]. However, due to the waning of immunity after vaccination and
infection, the immunological landscape of the population has become
increasingly complex [21]. Implementing representative serosurveys that
estimate population immunity among different age groups characterized by
varying numbers of prior infections before and after vaccination is difficult,
costly, and time-consuming. Thirdly, additional factors including VoCs [22],
non-pharmaceutical interventions [8, 9, 10, 11, 12], changes in population
immunity after vaccination or infection, and seasonality in transmission [23,
24, 25] complicate estimation of the age-specific burden.

Modelling studies have provided important insights into temporal changes in
the epidemiology of SARS-CoV-2 in specific countries [13, 26, 27, 28, 29, 24,
30, 20]. Some of these were hypothesis-generating and not rigorously validated
against all available evidence [28, 29, 24]. Other studies did not reconstruct
age-specific epidemiology [27] or were limited to specific periods of the
pandemic such as the first wave [13, 31, 32, 33, 34] or periods of dominance
for the Alpha [26, 20] and Delta variants [30, 35]. As data accumulate, formal
evaluations based on mathematical models calibrated to different types of
observational data [27, 13, 30] are crucial for reconstructing the burden of
infections and hospitalizations over long periods of time.

Here we reconstruct the epidemiology of SARS-CoV-2 in the Netherlands over the
first 23 months of the pandemic, a period including wild-type, Alpha, Delta
and Omicron BA.1 waves, using an age- and regionally stratified transmission
model fitted to various data sources (see Supplementary Materials). Our
fitting approach is a Bayesian evidence synthesis [36, 37, 38, 13, 26] based
on an ensemble adjustment Kalman filter [39, 40, 27] combined with
surveillance and national statistics data typically available from individual
countries (hospital admissions, serological surveys, PCR testing data, genetic
VoC data, vaccination coverage data, social contact matrices, demographic
data, regional train and Google mobility data) with the Netherlands used as
the motivational example.

To account for control measures targeted at elementary and secondary
schoolchildren versus the rest of the population [13] and for age-dependent
transmission effects [41, 42, 43, 44, 17, 14, 3], the population is stratified
into young children (0 to 9 years old), adolescents (10 to 19 years old) and
adults (above 19 years old) in twelve Dutch provinces. The model structure is
rooted in current knowledge of SARS-CoV-2, which suggests waning of immunity
after infection or vaccination [21], potential changes in susceptibility,
infectivity and severity of re-infections and breakthrough infections [29,
28], and seasonality in transmission [23, 24, 25]. The regional stratification
is augmented with real-world mobility across the provinces under the
assumption that infected individuals with undocumented infection may travel to
and infect susceptible individuals in other provinces whereas infected
individuals with confirmed infection stay in their province of origin.
Previous studies indicate that epidemic models with transmission dynamics
coupled across locations can improve the identifiability of epidemiological
parameters [45, 27].

## Results

### Time-dependent burden of hospital admissions

Figure 1: Estimated hospital admissions. Total (A) and age-specific national
daily hospital admissions in adults (B), adolescents (C) and children (D). The
colored lines represent the estimated posterior means for adults (pink),
adolescents (turquoise), children (yellow), and all ages (purple). The gray-
shaded regions correspond to posterior 95% credible intervals defined as the
2.5% and 97.5% quantiles from 300 posterior ensemble values. The dots are
daily hospital admission data used for fitting the model. Data 1 refers to the
National Intensive Care Evaluation data obtained from the National Institute
for Public Health and the Environment (RIVM); Data 2 and 3 refer to the RIVM
Dashboard data (see Table 1 and Supplementary Materials, Section 1). The
dashed and solid vertical lines indicate when each VoC corresponded to 5% and
50% of samples in the genetic variant data, respectively.

Our parsimonious model allows for robust estimation of key time-varying
epidemiological parameters compatible with national and regional epidemiology
of SARS-CoV-2 in pre- and post-vaccination periods (Supplementary Tables A1—A6
and Figures A1—A4; see also Section “Parameter identifiability and sensitivity
analyses”). The model reproduces well national and regional hospital
admissions in young children, adolescents and adults during all waves that
occurred from the first official case on February 27, 2020 until January 31,
2022 (Figure 1, Supplementary Figures A5—A7). At the national level, the total
estimated mean number of hospital admissions is 84,674 (95% CrI 82,729—86,151)
(Supplementary Table A7) spread over seven waves in the wild-type, Alpha,
Delta and Omicron BA.1 periods, with the last Omicron wave starting but not
yet ending at the end of the study period, January 31, 2022 (Figure 1 A). Of
these, two and four waves occurred before and, respectively, after the start
of vaccination, and one wave peaked in January 2021 around the onset of the
vaccination program. As expected, the pattern of hospital admissions in adults
largely mirrors that of total national admissions due to much higher
probability of clinical disease and hospitalization in this subpopulation
compared to adolescents and children [14, 3, 13, 26] (Figure 1 B). In our
analyses, we quantified the burden per period of VoCs, defined as the time
when a VoC reached a frequency of 5% in the genetic variant data described in
Table 1, until another VoC reaches the same frequency. The burden of hospital
admissions in adults progressively decreased over the periods of successive
VoCs, in line with the expansion of the primary vaccination series and of the
first booster campaign (Supplementary Table A7). The estimated cumulative mean
number of hospital admissions for adults was thus largest for the wild-type
(30,099, 95% CrI 29,126—30,784) and smallest for the incomplete Omicron BA.1
period until January 31, 2022 (8,352, 95% CrI 7,895—8,816) (Supplementary
Table A7). The picture is different for adolescents and children for whom the
hospitalization burden stayed lower than 5 hospital admissions per day for
either group and did not demonstrate a pronounced pattern until October 2021
(Figure 1 C and D). Unlike in adults, the estimated cumulative mean number of
hospital admissions in these two subpopulations was largest in the Omicron
BA.1 and Delta periods (Supplementary Table A7; for adolescents, 292, 95% CrI
239—346 and 266, 95% CrI 173—453; for children, 433, 95% CrI 333—586 and 388,
95% CrI 258—693).

### Time-dependent burden of confirmed cases and seroprevalence

Figure 2: Estimated seroprevalence and confirmed cases. Age-specific national
seroprevalence due to infection (A) and national confirmed daily cases in
adults (B), adolescents (C) and children (D). Inset: the first wave
characterized by low case reporting. The colored lines represent the estimated
means for adults (pink), adolescents (turquoise) and children (yellow). The
gray-shaded regions correspond to posterior 95% credible intervals defined as
the 2.5% and 97.5% quantiles from 300 posterior ensemble values. The black
dots are seroprevalence (A) and daily confirmed cases (B, C, D) data used for
fitting the model (see Table 1 and Supplementary Materials, Section 1). The
dashed and solid vertical lines indicate when each VoC corresponded to 5% and
50% of samples in the data, respectively. For comparison, the scale of the
y-axis is the same in B, C and D.

Figure 3: Age distribution of estimated total infections and hospital
admissions by VoC. Age-specific fraction of confirmed and unconfirmed cases in
the total national infections (A) and hospital admissions (B) during each VoC
period. The bars and the black lines show the means and 95% credible intervals
obtained from 300 posterior ensemble values.

The model further reproduces the age-specific seroprevalence and confirmed
cases nationally (Figure 2 B-D) and regionally (Figure 2, Supplementary
Figures A8—A16). In the context of our study, the seroprevalence is the
fraction of the population who have antibodies due to (any) infection. The
level of seroprevalence is thus determined by how fast antibodies decay and
how fast new infections happen. Confirmed cases refer to the part of
infections captured by testing surveillance. The estimated national
seroprevalence steadily increased in all age groups reaching 84% (95% CrI
82%—86%) in adults, 90% (95% CrI 87%—93%) in adolescents and 64% (95% CrI
60%—68%) in children by 31 January 2022 (Figure 2 A and Supplementary Table
A8). In contrast with the magnitude of the first wave of hospital admissions
in spring 2020, the case reporting in this period was overall low, even more
so among children and adolescents (insets Figure 2 B, C and D). Throughout
most of the study period that covers the next five waves, less than 1 case per
1,000 adults was reported daily, but this sharply increased during the Omicron
BA.1 wave to 4 cases per 1,000 adults by the end of January 2022 (Figure 2 B).
The pattern of reported cases was similar in adolescents with less than 1 case
per 1,000 adolescents daily during the first five waves; however,
substantially more Omicron BA.1 cases were reported in this age group compared
to adults during the sixth and seventh waves, skyrocketing from 1 to 12 daily
cases per 1,000 adolescents in January 2022 (Figure 2 C). More cases were
documented in children than in adults in the Omicron BA.1 wave too, i.e.,
about 6 versus 4 cases per 1,000 individuals per day at the peak, while case
reporting in children was much lower than in adults during the rest of the
study period (Figure 2 D). The observed increase in reported cases in
adolescents and children compared to adults may indicate a higher burden of
infections in these subpopulations during the Omicron BA.1 period and a change
in the age-specific distribution of total national cases which we verify
further below.

### Age distribution of total infections and hospital admissions per VoC

Our model estimates temporal changes in the age-specific contact rates as a
result of implemented control measures (Supplementary Materials, Section 2.4).
These changes combined with varying mobility patterns of reported and
unreported cases across provinces allow estimation of the age-specific
distribution of reported and unreported cases in the total national infections
during each VoC period (Figure 3 A and Supplementary Tables A9—A10). The
fraction of total infections in children among total estimated national
infections (light and dark yellow bars) steadily increased from 1.4% in the
wild-type period to 12.9% in the Omicron BA.1 period, of which about 66% were
unreported in total. The same increasing contribution to total infections per
successive VoC periods is estimated for adolescents (light and dark turquoise
bars) who comprised between 10.1% and 20.4% of all national infections for the
wild-type and Omicron BA.1 periods, respectively. The majority of all cases in
adolescents were not reported to surveillance, with the estimated fraction of
unreported infections 60% in total for all VoCs. The contribution of adults to
total infections (light and dark pink bars) decreased from 88.5% to 66.8%
through the study period while the fraction of reported adult cases increased
from approximately 23.5% during the wild-type period to 32.5% during the
Omicron BA.1 period. In terms of the age-specific contribution to total
hospital admissions per VoC, the adult population suffered the largest burden
during all VoC periods with a minor fraction of hospital admissions attributed
to children and adolescents during the circulation of Delta and Omicron BA.1
(Figure 3 B and Supplementary Table A11).

Figure 4: Burden stratification by immune status and VoC. (A) Fraction of
primary and breakthrough infections or re-infections in the total population
of each age group. (B) Hospital admissions after primary and breakthrough
infections or re-infections per 1,000 individuals in each age group. The bars
and the black lines show the means and 95% credible intervals obtained from
300 posterior ensemble values.

### Age-specific burden stratification by immune status and VoC

We further analyzed the age-specific burden of total infections and hospital
admissions stratified by immune status and period of VoC (Figure 4 and
Supplementary Tables A12—A13). For this, we distinguished infections in fully
susceptible (naive) individuals (primary infections) from infections in
individuals who were vaccinated (breakthrough infections) or lost immunity
after primary infection (re-infections) (Figure 4 A). Hospital admission
burden was stratified into hospital admissions after primary infection versus
after re-infection or breakthrough infection despite vaccination (Figure 4 B).
An estimated 21.1% (95% CrI 19.7%—23.0%) and 20.6% (95% CrI 20.2%—21.4%) of
adults (pink bars) had primary infection during the wild-type and Alpha
periods, respectively, while re-infections or breakthrough infections were
experienced in only 3.3% of adults for both periods combined. This pattern
reversed during the Delta and Omicron periods as re-infections or breakthrough
infections occurred in 16.0% (95% CrI 15.7%—16.7%) and 23.1% (95% CrI
22.7%—23.6%) of adults, respectively, while primary infections were
experienced by only about 4% of adults during each period. Similar dynamics
are estimated for the burden of hospital admissions in adults; namely, there
were more hospital admissions due to Delta and Omicron BA.1 breakthrough
infections/re-infections than due to primary infections by these variants
(Figure 4 B). We stress that larger hospitalization burden in adults
associated with breakthrough infections/re-infections than with primary
infections during the Delta and Omicron BA.1 periods is a consequence of high
vaccination coverage in adults while the hospitalization probability after
vaccination or prior infection is very low compared to that in adults without
any immunity. In the extreme case of a fully vaccinated adult population, all
hospitalization burden would be attributed to breakthrough infections, and
this would be much smaller than the burden the totally naive population
experienced at the beginning of the pandemic. The burden of primary infections
in adolescents (turquoise bars) was only slightly lower than that of adults
for the wild-type (16.6% of the age group, 95% CrI 15.7%—18.4%) and Alpha
(19.1% of the age group, 95% CrI 18.0%—21.3%) VoC. However, unlike for adults,
primary infections in adolescents were more common than breakthrough
infections or re-infections during the Delta and Omicron BA.1 VoCs, too (13.7%
and 22.6% of the age group, respectively), owing possibly to the delayed
vaccination schedule for adolescents. The burden in adolescents was especially
high during the Omicron BA.1 wave when an estimated 43.6% of all adolescents
were infected. In part because young children were the last to start
vaccination, the burden in this group increased with time and was mainly due
to primary infections which, in turn, led to a slight increase of
hospitalizations (yellow bars, Figure 4 A and B).

### Parameter identifiability and sensitivity analyses

The posterior distribution of most parameters tighten after the first wave
(Supplementary Materials, Section 6.1), indicating that the data are
informative in recovering model parameters. To further verify system
identifiability in the first wave, we generated one synthetic outbreak and
verified that the generated infection and hospitalization data follow similar
trends to the original data. To assess parameter identifiability in the first
wave, we considered higher, lower and time-varying case detection rates. For
each parameter combination, $100$ synthetic outbreaks were generated, each of
which was used as data to re-estimate the model parameters. Across the three
model configurations, the “true” parameters were either within the range of
the posterior mean densities, or the differences were not too large
(Supplementary Materials, Section 6.2).

Sensitivity analyses were conducted for the model parameters that were not
estimated but rather calibrated (see Methods and Supplementary Materials,
Section 7). The results were validated against Dutch seroprevalence estimates,
some of which were not used in the estimation. The selected values for
calibrated parameters yielded results that are closer to the Dutch
seroprevalence estimates than alternative calibrations.

## Discussion

We developed an inference-based modelling approach to reconstruct how the age-
specific epidemiology of SARS-CoV-2 changed through time and was shaped by the
complex interaction between VoCs, non-pharmaceutical interventions, and the
immunological landscape of the population. Our study provides several insights
into changes in the age-specific burden of infections and hospitalizations
over seven waves with different VoCs during the first two years of the
pandemic in the Netherlands. Firstly, cases reported to surveillance were a
considerable underestimate of total infections, as the majority of infections
in all age groups remained unidentified despite overall improvement of case
reporting through time. Reported cases are a generally poor predictor of true
incidence of infection, especially because reporting differs substantially by
age. Secondly, the contribution of children and adolescents to total infection
and hospitalization burden increased with successive VoCs and was largest
during the Omicron BA.1 period. This increase demonstrates that despite being
recognized as less virulent than previous VoCs [46], variants such as Omicron
BA.1 can impose substantial burdens on adolescents and children and additional
pressure on healthcare systems. Thirdly, we observed a shift in the pattern of
infections and hospitalizations by immune status and age. Before Delta, almost
all infections were primary infections occurring in naive individuals.
However, during the Delta and Omicron BA.1 periods primary infections were
common in children who were infected less frequently early on and for whom
vaccination roll-out lagged behind that of other age groups, but were
relatively rare in adults who experienced either re-infections or breakthrough
infections.

A similar shift in the dynamics was reported in a study based on identified
case data from the UK [47]. However, unlike in [47] where re-infections
dramatically increased during the Omicron wave, in our study the shift in
primary infections versus re-infections and breakthrough infections is
estimated to have begun during the Delta period. The discrepancy between the
two studies could be attributed to the population stratification, used in this
study. For computational efficiency, our model distinguishes fully naive
individuals from those with immunity induced by either vaccine or natural
infection, therefore re-infections and breakthrough infections are grouped.

To our knowledge, this is the first and most comprehensive application of a
computationally efficient inference-based modeling approach that provides
estimates of infection and hospitalization burden by age, region, VoC, and
immune status [13, 48, 49, 50, 51, 52, 53, 31]. The system is applied over
seven waves during the first two pandemic years in the Netherlands. While we
focus on the Dutch case, our framework can be applied to other countries given
the common surveillance and national statistics data typically available for
most countries. Compared to other studies [27, 28, 24, 54], we separately
model contacts in elementary schools, secondary schools, and the rest of the
population to enable more reliable disentanglement of the roles of children,
adolescents, and adults in transmission. We estimate contact changes due to
non-pharmaceutical interventions outside the school environment based on proxy
mobility measures [55] — Google and train mobility at transit stations. From
this proxy, we observe that most measures are taken up by the population ahead
of official implementation. We also estimate the speed of behavioral changes
and the number of age-specific contacts after each intervention [26, 13, 56].
This is in contrast to other inference-based studies (e.g. [30]) that assume
that age-related mixing patterns remain constant over time. Retrospective
analyses of the impact on transmission and burden in different age groups of
vaccination and boosting and of non-pharmaceutical interventions targeted at
elementary and secondary schoolchildren, or at the rest of the population,
will be a focus of future work.

Our model is fitted to regional reported cases, regional hospital admissions,
and national seroprevalence. We find that the first two data sources, combined
with modeling movement of unreported infections across regions using mobility
data, are crucial for identifying unreported infections with our inference
method. The national seroprevalence data played a secondary role in
reconstructing transmission dynamics due to the low frequency of serosurveys
(i.e., four surveys during 23 months in our case) and the fact that our
underlying transmission model is stochastic, unlike other studies [13, 26].
Our approach could therefore be applicable to countries where seroprevalence
estimates are not available at a high enough frequency. Incorporation of other
data into our model, particularly, publicly available SARS-CoV-2 measurements
in sewage water, can help to address another problem that is increasingly
apparent, i.e., drastic changes in COVID-19 surveillance in many European
countries including the Netherlands. This model extension could provide
projections of transmission dynamics after large-scale PCR testing stopped.
Our model could also be extended to quantify SARS-CoV-2 infection and COVID-19
hospitalization burdens in vulnerable populations with pre-existing chronic
conditions [57] and to determine which interventions are required to protect
them.

Our model has limitations. Firstly, not testing children and adolescents in
the early stages of the pandemic makes the identification of their age-
specific parameters (i.e., susceptibility) weaker in that initial period.
Later on, as testing capacity was expanded to younger individuals, we observe
good identification of all age-specific parameters. Secondly, we did not model
older age categories separately. It is technically straightforward to stratify
the adult population into smaller age categories relevant for estimating the
burden in older ages. However, older individuals travel less which means that
our metapopulation model with mobility may not capture their case detection
rates, and other data might be needed to estimate their age-specific
parameters. Thirdly, our choice of stratification into primary versus
breakthrough infections and re-infections is justified for the time period
when a large proportion of the population did not yet have any immunity to
SARS-CoV-2. As SARS-CoV-2 transitions from pandemicity to endemicity, primary
infections will be experienced only by very young children born into the
population [29, 58]. For later periods, our model could be modified to
differentiate between several immunity classes such as individuals with
primary vaccination series and various boosters, prior infections, and hybrid
immunity [59].

In conclusion, we developed an inference-based transmission model that
estimates how the age-specific epidemiology of SARS-CoV-2 changes over time.
This approach is relevant for countries in which random community surveys
uncovering true SARS-CoV-2 dynamics are absent but basic surveillance and
statistics data are available. The findings of our study on the burden of
infections and hospitalizations in children, adolescents and adults are
important for informing public health policy on non-pharmaceutical
interventions and vaccination.

## Methods

### Overview

The transmission model was calibrated using surveillance and national
statistics data (PCR testing data, hospital admissions, serological surveys,
demographic data, regional train and Google mobility data, vaccination
coverage data, genetic VoC data, social contact matrices) for the Netherlands.
Parameter estimates were obtained from the model fit to (i) age- and province-
stratified SARS-CoV-2 case notification data during the period from February
27, 2020 until January 31, 2022; (ii) age- and province-stratified COVID-19
hospital admission data for the same period; and (iii) cross-sectional age-
stratified national seroprevalence data from four serosurveys assessed on
April 3, June 4 and September 20, 2020, and February 11, 2021 (median dates).
Additional data for the model input were: (iv) population by age and province
on January 1, 2020; (v) daily commuters across twelve provinces computed from
Dutch national train data during the period from February 1, 2020 until
September 30, 2022, and Google mobility data from February 5, 2020 until
January 31, 2022; (vi) daily full vaccinations per province and age category
during the period from January 31, 2021 until January 31, 2022; (vii) weekly
boosters administered during the period from November 21, 2021 until January
31, 2022; (viii) weekly genetic VoC data during the period from December 1,
2020 until January 31, 2022; and (ix) school and non-school contact matrices
from three surveys: before the pandemic, April 2020 and June 2020. All model
analyses were performed in MATLAB, versions 2021A and 2022A. Data cleaning was
performed in STATA SE v16. The code for fitting the model over the entire
sample period runs in approximately 30 minutes on a Windows 10 Dell laptop
with an Intel Core i5 processor and without parallelization.

### Data

Table 1 gives an overview of the data, notation and sources. More details on
how each dataset was constructed for use in the model fitting are given in the
Supplementary Materials, Section 1.

Table 1: Overview of the data used in the model fitting.

Data | Notation∗ |  Source
---|---|---
Population of age group $k$ | $N_{ik}$ |  CBS, January 1, 2020
in province $i$ |  |  https://opendata.cbs.nl/statline/portal.html?_la=en&_catalog=CBS&tableId=37259eng&_theme=1135
Daily reported cases | $Inew_{ik}^{obs}(t)$ |  RIVM Dashboard
|  |  https://data.rivm.nl/covid-19/COVID-19_casus_landelijk.csv
Daily hospital admissions |  |
Dataset 1 | $Hnew_{ik,(1)}^{obs}(t)$ |  RIVM data, from February 27, 2020 until September 30, 2021
Dataset 2 | $Hnew_{i,(2)}^{obs}(t)$ |  RIVM Dashboard, from October 1, 2021 onward
|  |  https://data.rivm.nl/covid-19/COVID-19_ziekenhuisopnames.csv
|  |  https://data.rivm.nl/covid-19/COVID-19_ziekenhuisopnames_tm_03102021.csv
Dataset 3 | $Hnew_{k,(3)}^{obs}(t)$ |  RIVM Dashboard, from October 1, 2021 onward
|  |  https://data.rivm.nl/covid-19/COVID-19_ziekenhuis_ic_opnames_per_leeftijdsgroep.csv
|  |  https://data.rivm.nl/covid-19/COVID-19_ziekenhuis_ic_opnames_per_leeftijdsgroep_tm_03102021.csv
Seroprevalence data |  |  obtained from RIVM PIENTER Corona Study [60, 13] and
Serosurvey rounds | $h=1,2,3,4$ |  https://www.rivm.nl/en/pienter-corona-study/results
Median inclusion time | $t_{h}^{ser}$ |
Fraction seropositive | $ser_{hk}$ |
Sample size | $n_{hk}$ |
Train mobility | $M_{ij}(t)$ |  computed from the Dutch Railways (NS) data,
between provinces $i$ and $j$ |  |  from February 27, 2020 until September 30, 2021; extrapolated until January 31, 2022 using Google mobility data
Fraction commuters | $w_{ij}$ |  CBS, year 2019
from province $j$ to $i$ |  |  https://www.cbs.nl/nl-nl/cijfers/detail/83628NED
Google mobility | $y_{g}$ |  https://www.google.com/covid19/mobility/
|  |  from February 7, 2020 onward
Daily vaccinations | $V_{ik}(t)$ |  constructed from RIVM Dashboard
|  |  https://data.rivm.nl/covid-19/COVID-19_vaccinatiegraad_per_gemeente_per_week_leeftijd.csv
|  |  and RIVM data
|  |  https://www.rivm.nl/en/covid-19-vaccination/archive-covid-19-vaccination-figures-2021
Booster transition function | $g_{B,k}(t),k=1,2$ |  fitted to RIVM data
|  |  https://www.rivm.nl/en/covid-19-vaccination/archive-covid-19-vaccination-figures-2022
Variants transition function | $g_{\ell}(t),\ell\in\\{\alpha,\delta,o\\}$ |  fitted to RIVM Dashboard data
|  |  https://data.rivm.nl/covid-19/COVID-19_varianten.csv
Contact matrices |  |  computed from [42, 61]
Elementary school contacts | $c_{ikk^{*},ES}(t)$ |  and additional information on school closures and vacations
Secondary school contacts | $c_{ikk^{*},SS}(t)$ |  and additional information on school closures and vacations
All-setting contacts |  $c_{kk^{*}}^{(r)}$, $r=1,2,3$ |  calculated from [61], $r$ refers to survey rounds
Non-school contacts |  $c_{kk^{*},NS}^{(r)}$, $r=1,2,3$ |  calculated from [42, 61], $r$ refers to survey rounds

∗The indices $i,j=1,\ldots,12$ denote the province of the Netherlands. The
indices $k,k^{*}$ denote the age group, namely $k,k^{*}=1$ — adults ($>19$
years old), $k,k^{*}=2$ — adolescents ($10-19$ years old), and $k,k^{*}=3$ —
children ($0-9$ years old). The index $h=1,2,3,4$ denotes the serosurvey
round. The index $l=\alpha,\delta,o$ denotes Alpha, Delta, and Omicron VoCs.

### Transmission model

Figure 5: Schematic of the metapopulation transmission model. (A) Disease
dynamics. (B) Mobility dynamics. (C) Overview of the parameters used in the
model diagram. For simplicity of presentation, the age-specific indices are
not shown in the schematic.

We developed a stochastic compartmental metapopulation model describing SARS-
CoV-2 transmission in the population of the Netherlands stratified by
province, disease status and age. A schematic depicting disease and mobility
dynamics, as well as an overview of the main model parameters are shown in
Figure 5.

Disease dynamics The fully naive susceptible individuals ($S^{1}$) in each age
group can become latently infected ($E^{1}$) with the age-specific force of
infection $\beta^{1}$. After an average latent period $Z$, the latently
infected individuals become infectious (primary infection, $I^{1}$). Out of
the total daily new infectious cases, an age-specific fraction $\alpha$ is
reported to surveillance ($I^{R,1}$), and the $(1-\alpha)$ fraction is
unreported ($I^{U,1}$). Both reported and unreported infectious individuals
may recover without hospitalization ($R^{1}$) after an average infectious
period $D$, but only reported infectious individuals may be hospitalized
($H^{1}$) with the age-specific rate $\gamma^{1}$. Hospitalized individuals
are discharged after an average age-specific hospitalization period
$1/\delta$. After a primary infection, individuals lose immunity and become
partially susceptible ($S^{2}$) with the rate $\eta^{1}$. Disease progression
for partially susceptible individuals is similar to that for fully susceptible
individuals. However, partially susceptible individuals are reinfected with
the age-specific force of infection $\beta^{2}$, individuals reported with re-
infection are hospitalized with the age-specific rate $\gamma^{2}$, and the
immunity after re-infection is lost with the rate $\eta^{2}$. Upon losing
immunity after re-infection, the individuals return to the same partially
susceptible compartment ($S^{2}$). Vaccination of individuals with the age-
specific rate $v$ occurs in all disease stages except for those with primary
reported infection or hospitalization. We used a simplified approach whereby
susceptible individuals after vaccination and individuals recovered after
primary infection whose immunity has waned are grouped together in one class
of partially susceptible individuals ($S^{2}$). Therefore, re-infections and
breakthrough infections are grouped, too. Vaccination or prior infection has
three effects: (i) lower susceptibility to re-infection and breakthrough
infection that affects $\beta^{2}$, (ii) lower infectivity of re-infection and
breakthrough infection (not shown in the diagram), and (iii) lower
hospitalization rate after re-infection and breakthrough infection
$\gamma^{2}$. The model assumes a constant population size during the study
period.

Mobility dynamics The model assumes the unreported infected travel and infect
other individuals in other regions, and that susceptible individuals can
travel and become exposed in another region, while reported infected do not
travel. Due to heterogeneity in population size and mobility across regions,
this assumption equips the model with additional dynamics for the unreported
infected compared to reported infected. The identification of unreported cases
is possible as long as the case detection rate is the same across regions, and
there are sufficient observations available across regions or time [62, 27,
63]. These two conditions are met in our study. Firstly, the case detection
rate is assumed common across regions due to the ability to test at any
location available across the country. Secondly, we estimate the model at a
daily frequency for all twelve Dutch provinces.

A full description of the model and the model equations are reported in
Supplementary Materials, Section 2.

### Parameter inference

The model is fitted to infection, hospitalization, and seroprevalence data
using the ensemble adjustment Kalman filter [39]. This method allows inference
on a high-dimensional system of observed and unobserved variables and
parameters in a computationally efficient way. First, a large number of
ensembles are drawn from priors on all parameters and state variables. The
latter then evolve according to a stochastic version of the metapopulation
model (Figure 5). For the filter updates, each ensemble member is individually
propagated forward by means of a closed form approximation akin to the normal
distribution but with an additional bias correction. The updates are computed
sequentially based on the data in each province and its neighbours, allowing
for a large number of state variables and parameters to be updated. Further
details on the estimation and the algorithm can be found in Sections 3.3 and
3.4 of the Supplementary Materials.

A small number of parameters that could not be identified were calibrated.
These calibrations are described and motivated in Supplementary Materials,
Section 3.1. The remaining parameters were estimated, and the choice of priors
is detailed and motivated in the Supplementary Materials, Section 3.2. The
parameter posteriors and their time-evolution over variant periods are
discussed in Section 4 of the Supplementary Materials.

### Model outcomes

The exact calculations behind Figures 1—4 are explained in Supplementary
Materials, Sections 5.1—5.4. The data plotted in Figures 3 and 4 can be found
in Supplementary Tables A9 and A11—A13, respectively.

### Details on the parameter identifiability and sensitivity analyses

To verify system identifiability for the first wave, we fixed the parameters
at their estimated posteriors in the original sample (observations from
February 27, 2020 until March 30, 2020, see Supplementary Table A14),
generated one synthetic outbreak and re-estimated the model parameters on the
synthetic data. In addition to showing that the synthetic data are similar to
the original data and are fitted well by our method, we also verified that the
fitted and model seroprevalence match (Supplementary Materials, Section 6.1,
Table A15 and Figures A17—A20). To assess parameter identifiability for the
first wave, we considered three parameter configurations: (i) same as
described above, with case detection rates equal to the original sample
posterior means; (ii) larger case detection rates; and (iii) case detection
rates as in (i) until March 30, 2020, and as in (ii) until April 30, 2020. For
each parameter combination, $100$ synthetic outbreaks were generated, each of
which was used as data to re-estimate the model parameters. Across the three
model configurations, the “true” parameters were either within the range of
the posterior mean densities or the differences were not too large
(Supplementary Materials, Section 6.2, Figures A21—A23). In case (iii), the
filter also approximated well the increase of case detection rates, even
though this increase was not modelled. This suggests that changes in testing
capacity or recommendations, and other unmodelled parameter changes can be
captured by our model inference technique even in periods of high uncertainty,
when data are less informative (Supplementary Figure A24).

For our sensitivity analyses, we re-estimated the model over the entire sample
period: (i) without the serosurvey data; (ii) with lower and higher immunity
waning rates after primary infection; (iii) with lower and higher immunity
waning rates at the start of Omicron BA.1 period after both primary and
breakthrough infection/re-infection; (iv) with higher transmissibility of
Omicron BA.1; and (v) without seasonality. We found that in most cases, except
(iii), our calibrations lead to closer matches to available seroprevalence
estimates, independently of whether these were used in the model estimation or
not (Supplementary Materials, Section 7 and Tables A16—A19). In cases
(iii)-(iv), the results are robust to whether immunity wanes on average after
two, three or four months for the Omicron BA.1 variant, and to whether Omicron
BA.1 variant is 40% or 60% more transmissible compared to the Delta variant.

## Reporting summary

Further information on research design is available in the Nature Research
Reporting Summary submitted together with this manuscript.

## Data availability

All output datasets generated during this study will be made publicly
available in the designated GitHub repository
(https://github.com/oboldea/COVID_age_regions) prior to acceptance of the
manuscript. The datasets used as input in this study are listed in Table 1.
All input datasets except for seroprevalence data and train mobility data will
be made publicly available in the designated GitHub repository
(https://github.com/oboldea/COVID_age_regions) prior to acceptance of the
manuscript. Requests for seroprevalence data should be addressed to Fiona van
der Klis<EMAIL_ADDRESS>and Eric Vos<EMAIL_ADDRESS>from
the National Institute for Public Health and the Environment, Bilthoven, The
Netherlands. Requests for train mobility data should be addressed to Jan
Banninga<EMAIL_ADDRESS>from the Dutch Railways (Nederlandse Spoorwegen
— NS). All datasets are available to reviewers and editors upon request.

## Code availability

All codes reproducing the results of this study will be made publicly
available in the designated GitHub repository
(https://github.com/oboldea/COVID_age_regions) prior to acceptance of the
manuscript. The codes are available to reviewers and editors upon request.

## References

  * [1] World Health Organization. Coronavirus (COVID-19) Dashboard; 2022.  Available from: https://covid19.who.int/.
  * [2] Variation in the COVID-19 infection-fatality ratio by age, time, and geography during the pre-vaccine era: a systematic analysis.  The Lancet. 2022;399(10334):1469–1488.  doi:10.1016/S0140-6736(21)02867-1.
  * [3] O’Driscoll M, Dos Santos GR, Wang L, Cummings DAT, Azman AS, Paireau J, et al.  Age-specific mortality and immunity patterns of SARS-CoV-2.  Nature. 2021;590(7844):140–145.  doi:https://doi.org/10.1038/s41586-020-2918-0.
  * [4] Coronavirus dashboard; 2020.  Available from: https://coronadashboard.government.nl/.
  * [5] Koelle K, Martin MA, Antia R, Lopman B, Dean NE.  The changing epidemiology of SARS-CoV-2.  Science. 2022;375(6585):1116–1121.  doi:10.1126/science.abm4915.
  * [6] Monod M, Blenkinsop A, Xi X, Hebert D, Bershan S, Tietze S, et al.  Age groups that sustain resurging COVID-19 epidemics in the United States.  Science. 2021;371(6536):eabe8372.  doi:10.1126/science.abe8372.
  * [7] Davies NG, Klepac P, Liu Y, Prem K, Jit M, Pearson CAB, et al.  Age-dependent effects in the transmission and control of COVID-19 epidemics.  Nature Medicine. 2020;26(8):1205–1211.  doi:10.1038/s41591-020-0962-9.
  * [8] Perra N.  Non-pharmaceutical interventions during the COVID-19 pandemic: A review.  Physics Reports. 2021;913:1–52.  doi:https://doi.org/10.1016/j.physrep.2021.02.001.
  * [9] Liu Y, Morgenstern C, Kelly J, Lowe R, Group CCW, Jit M.  The impact of non-pharmaceutical interventions on SARS-CoV-2 transmission across 130 countries and territories.  BMC medicine. 2021;19(1):40–40.  doi:10.1186/s12916-020-01872-8.
  * [10] Sharma M, Mindermann S, Rogers-Smith C, Leech G, Snodin B, Ahuja J, et al.  Understanding the effectiveness of government interventions against the resurgence of COVID-19 in Europe.  Nature communications. 2021;12(1):5820–5820.  doi:10.1038/s41467-021-26013-4.
  * [11] Brauner JM, Mindermann S, Sharma M, Johnston D, Salvatier J, Gavenčiak T, et al.  Inferring the effectiveness of government interventions against COVID-19.  Science. 2021;371(6531):eabd9338.  doi:10.1126/science.abd9338.
  * [12] Li Y, Campbell H, Kulkarni D, Harpur A, Nundy M, Wang X, et al.  The temporal association of introducing and lifting non-pharmaceutical interventions with the time-varying reproduction number (R) of SARS-CoV-2: a modelling study across 131 countries.  The Lancet Infectious Diseases. 2021;21(2):193–202.  doi:10.1016/S1473-3099(20)30785-4.
  * [13] Rozhnova G, van Dorp CH, Bruijning-Verhagen P, Bootsma M, van de Wijgert J, Bonten M, et al.  Model-based evaluation of school and non-school-related measures to control the COVID-19 pandemic.  Nature Communications. 2021;12(1):1614.  doi:10.1038/s41467-021-21899-6.
  * [14] Flook M, Jackson C, Vasileiou E, Simpson CR, Muckian MD, Agrawal U, et al.  Informing the public health response to COVID-19: a systematic review of risk factors for disease, severity, and mortality.  BMC infectious diseases. 2021;21(1):342–342.  doi:10.1186/s12879-021-05992-1.
  * [15] Bubar KM, Reinholt K, Kissler SM, Lipsitch M, Cobey S, Grad YH, et al.  Model-informed COVID-19 vaccine prioritization strategies by age and serostatus.  Science. 2021;371(6532):916–921.  doi:10.1126/science.abe6959.
  * [16] Matrajt L, Eaton J, Leung T, Brown ER.  Vaccine optimization for COVID-19: Who to vaccinate first?  Science Advances. 2021;7(6):eabf1374.  doi:10.1126/sciadv.abf1374.
  * [17] Sah P, Fitzpatrick MC, Zimmer CF, Abdollahi E, Juden-Kelly L, Moghadas SM, et al.  Asymptomatic SARS-CoV-2 infection: A systematic review and meta-analysis.  Proceedings of the National Academy of Sciences. 2021;118(34):e2109229118.  doi:10.1073/pnas.2109229118.
  * [18] Office for National Statistics. Coronavirus (COVID-19) Infection Survey, UK; 2022\.  Available from: https://www.ons.gov.uk/surveys/informationforhouseholdsandindividuals/householdandindividualsurveys/covid19infectionsurvey.
  * [19] Shioda K, Lau MSY, Kraay ANM, Nelson KN, Siegler AJ, Sullivan PS, et al.  Estimating the Cumulative Incidence of SARS-CoV-2 Infection and the Infection Fatality Ratio in Light of Waning Antibodies.  Epidemiology. 2021;32(4).
  * [20] Hozé N, Paireau J, Lapidus N, Tran Kiem C, Salje H, Severi G, et al.  Monitoring the proportion of the population infected by SARS-CoV-2 using age-stratified hospitalisation and serological data: a modelling study.  The Lancet Public Health. 2021;6(6):e408–e415.  doi:10.1016/S2468-2667(21)00064-5.
  * [21] Milne G, Hames T, Scotton C, Gent N, Johnsen A, Anderson RM, et al.  Does infection with or vaccination against SARS-CoV-2 lead to lasting immunity?  The Lancet Respiratory Medicine. 2021;9(12):1450–1466.  doi:10.1016/S2213-2600(21)00407-0.
  * [22] World Health Organization. Tracking SARS-CoV-2 variants; 2022.  Available from: https://www.who.int/en/activities/tracking-SARS-CoV-2-variants/.
  * [23] Townsend JP, Lamb AD, Hassler HB, Sah P, Nishio AA, Nguyen C, et al.  Projecting the seasonality of endemic COVID-19.  medRxiv. 2022;doi:10.1101/2022.01.26.22269905.
  * [24] Kissler SM, Tedijanto C, Goldstein E, Grad YH, Lipsitch M.  Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period.  Science. 2020;368(6493):860–868.  doi:10.1126/science.abb5793.
  * [25] Neher RA, Dyrdak R, Druelle V, Hodcroft EB, Albert J.  Potential impact of seasonal forcing on a SARS-CoV-2 pandemic.  Swiss Med Wkly. 2020;150.  doi:10.4414/smw.2020.20224.
  * [26] Viana J, van Dorp CH, Nunes A, Gomes MC, van Boven M, Kretzschmar ME, et al.  Controlling the pandemic during the SARS-CoV-2 vaccination rollout.  Nature Communications. 2021;12(1):3674.  doi:10.1038/s41467-021-23938-8.
  * [27] Pei S, Yamana TK, Kandula S, Galanti M, Shaman J.  Burden and characteristics of COVID-19 in the United States during 2020\.  Nature. 2021;598(7880):338–341.  doi:10.1038/s41586-021-03914-4.
  * [28] Saad-Roy CM, Wagner CE, Baker RE, Morris SE, Farrar J, Graham AL, et al.  Immune life history, vaccination, and the dynamics of SARS-CoV-2 over the next 5 years.  Science. 2020;370(6518):811–818.  doi:10.1126/science.abd7343.
  * [29] Lavine JS, Bjornstad ON, Antia R.  Immunological characteristics govern the transition of COVID-19 to endemicity.  Science. 2021;371(6530):741–745.  doi:10.1126/science.abe6522.
  * [30] Sonabend R, Whittles LK, Imai N, Perez-Guzman PN, Knock ES, Rawson T, et al.  Non-pharmaceutical interventions, vaccination, and the SARS-CoV-2 delta variant in England: a mathematical modelling study.  The Lancet. 2021;398(10313):1825–1835.  doi:10.1016/S0140-6736(21)02276-5.
  * [31] Dekker MM, Coffeng LE, Pijpers FP, Panja D, de Vlas SJ.  Reducing societal impacts of SARS-CoV-2 interventions through subnational implementation.  medRxiv. 2022;doi:10.1101/2022.03.31.22273222.
  * [32] Gatto M, Bertuzzo E, Mari L, Miccoli S, Carraro L, Casagrandi R, et al.  Spread and dynamics of the COVID-19 epidemic in Italy: Effects of emergency containment measures.  Proceedings of the National Academy of Sciences. 2020;117(19):10484–10491.  doi:10.1073/pnas.2004978117.
  * [33] Bertuzzo E, Mari L, Pasetto D, Miccoli S, Casagrandi R, Gatto M, et al.  The geography of COVID-19 spread in Italy and implications for the relaxation of confinement measures.  Nature Communications. 2020;11(1):4264.  doi:10.1038/s41467-020-18050-2.
  * [34] Giordano G, Blanchini F, Bruno R, Colaneri P, Di Filippo A, Di Matteo A, et al.  Modelling the COVID-19 epidemic and implementation of population-wide interventions in Italy.  Nature Med. 2020;26:855–860.  doi:10.1038/s41591-020-0883-7.
  * [35] Bosetti P, Tran Kiem C, Andronico A, Colizza V, Yazdanpanah Y, Fontanet A, et al.  Epidemiology and control of SARS-CoV-2 epidemics in partially vaccinated populations: a modeling study applied to France.  BMC Medicine. 2022;20(1):33.  doi:10.1186/s12916-022-02235-1.
  * [36] Presanis AM, De Angelis D, Goubar A, Gill ON, Ades AE.  Bayesian evidence synthesis for a transmission dynamic model for HIV among men who have sex with men.  Biostatistics. 2011;12(4):666–681.  doi:10.1093/biostatistics/kxr006.
  * [37] Birrell PJ, Ketsetzis G, Gay NJ, Cooper BS, Presanis AM, Harris RJ, et al.  Bayesian modeling to unmask and predict influenza A/H1N1pdm dynamics in London.  Proceedings of the National Academy of Sciences. 2011;108(45):18238–18243.  doi:10.1073/pnas.1103002108.
  * [38] Rozhnova G, Kretzschmar ME, van der Klis F, van Baarle D, Korndewal M, Vossen AC, et al.  Short- and long-term impact of vaccination against cytomegalovirus: a modeling study.  BMC Medicine. 2020;18.  doi:https://doi.org/10.1186/s12916-020-01629-3.
  * [39] Anderson JL.  An Ensemble Adjustment Kalman Filter for Data Assimilation.  Monthly Weather Review. 2001;129(12):2884 – 2903.  doi:10.1175/1520-0493(2001)129¡2884:AEAKFF¿2.0.CO;2.
  * [40] Pei S, Kandula S, Shaman J.  Differential effects of intervention timing on COVID-19 spread in the United States.  Science Advances. 2020;6(49):eabd6370.  doi:10.1126/sciadv.abd6370.
  * [41] Mossong J, Hens N, Jit M, Beutels P, Auranen K, Mikolajczyk R, et al.  Social contacts and mixing patterns relevant to the spread of infectious diseases.  PLOS Medicine. 2008;5(3):1–1.  doi:10.1371/journal.pmed.0050074.
  * [42] Mistry D, Litvinova M, Pastore Y Piontti A, Chinazzi M, Fumanelli L, Gomes MFC, et al.  Inferring high-resolution human mixing patterns for disease modeling.  Nature Communications. 2021;12(1):323–323.  doi:10.1038/s41467-020-20544-y.
  * [43] Goldstein E, Lipsitch M, Cevik M.  On the effect of age on the transmission of SARS-CoV-2 in households, schools and the community.  The Journal of Infectious Diseases. 2020;doi:10.1093/infdis/jiaa691.
  * [44] Viner RM, Mytton OT, Bonell C, Melendez-Torres GJ, Ward J, Hudson L, et al.  Susceptibility to SARS-CoV-2 Infection Among Children and Adolescents Compared With Adults: A Systematic Review and Meta-analysis.  JAMA Pediatrics. 2021;175(2):143–156.  doi:10.1001/jamapediatrics.2020.4573.
  * [45] Li R, Pei S, Chen B, Song Y, Zhang T, Yang W, et al.  Substantial undocumented infection facilitates the rapid dissemination of novel coronavirus (SARS-CoV-2).  Science. 2020;368(6490):489–493.  doi:10.1126/science.abb3221.
  * [46] European Centre for Disease Prevention and Control. SARS-CoV-2 variants of concern as of 14 October 2022.; 2022.  Available from: https://www.ecdc.europa.eu/en/covid-19/variants-concern.
  * [47] Keeling MJ.  Patterns of reported infection and reinfection of SARS-CoV-2 in England.  Journal of Theoretical Biology;556:111299.
  * [48] Miura F, Leung KY, Klinkenberg D, Ainslie KEC, Wallinga J.  Optimal vaccine allocation for COVID-19 in the Netherlands: A data-driven prioritization.  PLOS Computational Biology. 2021;17(12):1–13.  doi:10.1371/journal.pcbi.1009697.
  * [49] Ainslie KEC, Backer J, de Boer P, van Hoek AJ, Klinkenberg D, Altes HK, et al.  The impact of vaccinating adolescents and children on COVID-19 disease outcomes.  medRxiv. 2021;doi:10.1101/2021.10.21.21265318.
  * [50] van Boven M, Hetebrij WA, Swart AM, Nagelkerke E, van der Beek RFHJ, Stouten S, et al.  Modelling patterns of SARS-CoV-2 circulation in the Netherlands, August 2020-February 2022, revealed by a nationwide sewage surveillance program.  medRxiv. 2022;doi:10.1101/2022.05.25.22275569.
  * [51] Schoot Uiterkamp MHH, Gösgens M, Heesterbeek H, van der Hofstad R, Litvak N.  The role of inter-regional mobility in forecasting SARS-CoV-2 transmission.  Journal of The Royal Society Interface. 2022;19(193):20220486.  doi:10.1098/rsif.2022.0486.
  * [52] Gösgens M, Hendriks T, Boon M, Steenbakkers W, Heesterbeek H, van der Hofstad R, et al.  Trade-offs between mobility restrictions and transmission of SARS-CoV-2.  Journal of The Royal Society Interface. 2021;18(175):20200936.  doi:10.1098/rsif.2020.0936.
  * [53] de Vlas SJ, Coffeng LE.  Achieving herd immunity against COVID-19 at the country level by the exit strategy of a phased lift of control.  Scientific Reports. 2021;11(1):4445.  doi:10.1038/s41598-021-83492-7.
  * [54] Brand SPC, Ojal J, Aziza R, Were V, Okiro EA, Kombe IK, et al.  COVID-19 transmission dynamics underlying epidemic waves in Kenya.  Science. 2021;374(6570):989–994.  doi:10.1126/science.abk0414.
  * [55] Kishore N, Taylor AR, Jacob PE, Vembar N, Cohen T, Buckee CO, et al.  Evaluating the reliability of mobility metrics from aggregated mobile phone data as proxies for SARS-CoV-2 transmission in the USA: a population-based study.  The Lancet Digital Health. 2022;4(1):e27–e36.  doi:10.1016/S2589-7500(21)00214-4.
  * [56] Boldea O, Cornea-Madeira A, Madeira J.  Disentangling the effect of measures, variants and vaccines on SARS-CoV-2 Infections in England: A dynamic intensity model.  medRxiv. 2022;doi:10.1101/2022.03.09.22272165.
  * [57] European Centre for Disease Prevention and Control. Core protocol for ECDC studies of COVID-19 vaccine effectiveness against hospitalisation with Severe Acute Respiratory Infection laboratory confirmed with SARS-CoV-2, version 1.0.; 2021.  Available from: https://www.ecdc.europa.eu/en/publications-data/core-protocol-ecdc-studies-covid-19-vaccine-effectiveness-against-hospitalisation.
  * [58] Cohen LE, Spiro DJ, Viboud C.  Projecting the SARS-CoV-2 transition from pandemicity to endemicity: Epidemiological and immunological considerations.  PLOS Pathogens. 2022;18(6):1–21.  doi:10.1371/journal.ppat.1010591.
  * [59] Bobrovitz N, Ware H, Ma X, Li Z, Hosseini R, Cao C, et al.  Protective effectiveness of prior SARS-CoV-2 infection and hybrid immunity against Omicron infection and severe disease: a systematic review and meta-regression.  medRxiv. 2022;doi:10.1101/2022.10.02.22280610.
  * [60] Vos ERA, den Hartog G, Schepp RM, Kaaijk P, van Vliet J, Helm K, et al.  Nationwide seroprevalence of SARS-CoV-2 and identification of risk factors in the general population of the Netherlands during the first epidemic wave.  Journal of Epidemiology & Community Health. 2020;doi:10.1136/jech-2020-215678.
  * [61] Backer JA, Mollema L, Vos ER, Klinkenberg D, van der Klis FR, de Melker HE, et al.  Impact of physical distancing measures against COVID-19 on contacts and mixing patterns: repeated cross-sectional surveys, the Netherlands, 2016-17, April 2020 and June 2020.  Eurosurveillance. 2021;26(8).  doi:10.2807/1560-7917.ES.2021.26.8.2000994.
  * [62] Li Q, Guan X, Wu P, Wang X, Zhou L, Tong Y, et al.  Early transmission dynamics in Wuhan, China, of novel coronavirus-infected pneumonia.  New England Journal of Medicine. 2020;doi:10.1056/NEJMoa2001316.
  * [63] Hortaçsu A, Liu J, Schwieg T.  Estimating the fraction of unreported infections in epidemics with a known epicenter: An application to COVID-19.  Journal of Econometrics. 2021;220(1):106–129.  doi:10.1016/j.jeconom.2020.07.047.

## Acknowledgements

G.R. was supported by the VERDI project (101045989), funded by the European
Union. Views and opinions expressed are however those of the author(s) only
and do not necessarily reflect those of the European Union or the Health and
Digital Executive Agency. Neither the European Union nor the granting
authority can be held responsible for them. G.R. and S.P. were supported by
the Fundação para a Ciência e a Tecnologia project 2022.01448.PTDC. O.B. and
A.A. were supported by the Tilburg University Fund. J.S. and S.P. were
supported by US NIAID grant R01AI163023 and CDC contract 75D30122C14289. We
thank Prof. Marc Bonten (University Medical Center Utrecht), Dr. Susan van den
Hof (National Institute for Public Health and the Environment), Prof. Dennis
Huisman (Erasmus University Rotterdam) and Jan Banninga (Nederlandse
Spoorwegen) for their substantial help in obtaining the data for this study.
We also thank Dr. Bettina Siflinger (Tilburg University) for the Tilburg
CentER LISS Panel Survey questions added to support our study, Dr. Marino van
Zelst (Wageningen University) and Prof. Jaap Abbring (Tilburg University) for
useful discussions on data availability and modelling.

## Author contributions

O.B. and G.R. developed the modeling study, with important feedback from all
other authors. O.B. and G.R. obtained the data. O.B. and A.A. designed and
wrote the computer code, with substantial feedback from S.P. G.R. wrote the
manuscript, with substantial feedback from O.B., S.P., and J.S. O.B. wrote the
Supplementary Materials, with substantial feedback from all other authors.

## Competing interests

J.S. and Columbia University declare partial ownership of SK Analytics. The
other authors declare no competing interests.

## Additional information

Supplementary Materials contain details of the data, the model, inference,
system identifiability, sensitivity analyses, Supplementary Figures, and
Tables. Supplementary Materials are publicly available in the designated
GitHub repository (https://github.com/oboldea/COVID_age_regions).

## Correspondence

Correspondence and material requests should be addressed to Dr. Ganna
Rozhnova, Julius Center for Health Sciences and Primary Care, University
Medical Center Utrecht, P.O. Box 85500 Utrecht, The Netherlands, email:
<EMAIL_ADDRESS>and Dr. Otilia Boldea, Department of Econometrics and
OR, and CentER, Tilburg School of Economics and Management, Tilburg
University, P.O. Box 90153, Tilburg, The Netherlands, email:
<EMAIL_ADDRESS>


2008):

$\left<\Delta E_{h}\right>=\frac{1}{6}I_{\rm tid}r_{\rm hm}^{2}$ (12)

where $I_{\rm tid}$ is the tidal heating parameter, given by

$I_{\rm
tid}=\sum_{i,j}\left(\int\mathbf{T}^{ij}dt\right)^{2}\left(1+\left(\frac{\tau}{t_{\rm
dyn}}\right)^{2}\right)^{-3/2}$ (13)

where the sum is over all components $i$, $j$, of the tidal tensor defined in
Equation (4). The second term in the parenthesis is the adiabatic correction,
accounting for the ability of the clusters to successfully re-absorb injected
energy. We use the power-law adiabatic correction from (Gnedin & Ostriker,
1999), which depends on the ratio of the length of the tidal shock $\tau$ to
the dynamical time of the cluster $t_{\rm dyn}$. Once we have the average
energy injected by every tidal shock, we can convert this to a typical mass
loss experienced by the cluster due to its subsequent expansion. We use the
approximate relation from Gieles et al. (2006):

$\frac{\Delta M}{M_{0}}\approx 0.22\frac{\left<\Delta E_{h}\right>}{E_{0}}$
(14)

where $M_{0}$ and $E_{0}$ are the initial mass and energy of the cluster.

As an order-of-magnitude check, we assume that once the cumulative mass loss
given by Equation (14) exceeds the total mass, the cluster has been disrupted.
For every cluster in our evolved catalog, we identify tidal shocks as
occurring whenever any component of the tidal tensor is one standard deviation
away from the median value of that component as measured over the entire
cluster lifetime (Figure 1 shows the typical amount by which the effective
tidal tensor changes). The integral in Equation (13) is then calculated by
directly summing the component of each shock times the $\Delta t$ between the
FIRE snapshots that Equation (4) was extracted from. In practice, the vast
majority of shocks only appear in a single snapshot, so we multiply each
component by the spacing between snapshots ($\approx$20-30 Myr).

After applying this procedure to our catalog, we found virtually no
contribution to our final cluster population arising from our neglect of tidal
shocking. In fact, according to this procedure, only one of our 895 clusters
should have been destroyed by tidal shocking, and even in that case–a
particularly massive ($\sim 3\times 10^{6}M_{\odot}$) and large ($r_{\rm
hm}=26$ pc) cluster–it was destroyed by tidal shocking at the same time CMC
recorded it as being destroyed by tidal stripping. In other words, the strong
tidal fields responsible for shocking the cluster are also responsible for the
increase in mass loss through the tidal boundary.

From this, one might conclude that our implementation of tidal stripping in
CMC naturally culls large clusters at the same rate as tidal shocking would,
since they are both based in the rapid variation of the galactic tidal field,
allowing us to reproduce the correct distribution of clusters in the mass-
radius region as Gnedin & Ostriker (1997). However, such optimism would be
premature. As previously mentioned, we have extracted our tidal tensors from
snapshots of the m12i galaxy that are typically taken every 20-30 Myr. But the
typical dynamical time of any of our clusters during a tidal shocks is a
significantly shorter ($\sim$ 1 Myr), and as a result, each tidal shock is
reduced by a factor of $10^{4}$ from the adiabatic correction to Equation
(13).

What this actually means is that the 20 Myr interval between snapshots is
insufficient to resolve the influence of tidal shocking on the star cluster
population of a single galaxy. Work to improve the extraction of the
instantaneous tidal boundaries in zoom in simulations, as well as comparisons
to direct $N$-body models capable of resolving the influence of tidal shocking
on the cluster survival, are currently underway.


11institutetext: School of Computer and Information Technology, Beijing
Jiaotong University, Beijing, China 22institutetext: Institute of Automation,
Chinese Academy of Science, Beijing, China 33institutetext: Department of
Automation, Tsinghua University, Beijing, China

# TST: Time-Sparse Transducer for Automatic Speech Recognition

Xiaohui Zhang 1122 Mangui Liang 11 Zhengkun Tian 22 Jiangyan Yi 22 Jianhua Tao
33

###### Abstract

End-to-end model, especially Recurrent Neural Network Transducer (RNN-T), has
achieved great success in speech recognition. However, transducer requires a
great memory footprint and computing time when processing a long decoding
sequence. To solve this problem, we propose a model named time-sparse
transducer, which introduces a time-sparse mechanism into transducer. In this
mechanism, we obtain the intermediate representations by reducing the time
resolution of the hidden states. Then the weighted average algorithm is used
to combine these representations into sparse hidden states followed by the
decoder. All the experiments are conducted on a Mandarin dataset AISHELL-1.
Compared with RNN-T, the character error rate of the time-sparse transducer is
close to RNN-T and the real-time factor is 50.00% of the original. By
adjusting the time resolution, the time-sparse transducer can also reduce the
real-time factor to 16.54% of the original at the expense of a 4.94% loss of
precision.

###### Keywords:

speech recognition human-computer interaction computational paralinguistics

##  1 Introduction

In recent years, significant advancements have been made in end-to-end speech
recognition models, including the connectionist temporal classification (CTC)
[1, 2, 3], attention-based sequence-to-sequence models (AED) [4, 5, 6, 7], and
recurrent neural network transducer (RNN-T) [8, 9, 10, 11, 12, 13]. The CTC
algorithm, employed by many models, performs frame-level decoding by
converting speech sequences to corresponding label sequences. However, this
method relies on the assumption of conditional independence among speech
frames, making it unable to effectively model the dependencies between
outputs. On the other hand, the RNN-T leverages its recurrent structure to
overcome the conditional independence assumption and optimizes acoustic and
language components jointly through the introduction of language and joint
networks. Consequently, the RNN-T has found success in online automatic speech
recognition (ASR) systems [14, 15].

Despite its advantages, the RNN-T imposes a higher memory demand compared to
AED and CTC methods [10, 16]. During the forward-backward pass of the RNN-T,
posteriors are calculated at each point within the grid composed of the
encoder and prediction network. Computing a long decoding sequence in this
grid consumes more memory and time than the aforementioned methods, making the
vocabulary less dependent on training/inference speech and more reliant on
sequence length [17, 18, 19]. Therefore, reducing memory consumption and
improving computing speed are crucial for deploying RNN-T models on low-
resource devices [20, 21, 22].

This paper proposes a model, named the time-sparse transducer (TST), designed
to address the memory cost and computing time consumption of the RNN-T. Our
approach consists of a convolutional front end, an acoustic encoder, a time-
sparse mechanism, and a decoder. The encoder maps input acoustic frames into
high-level representations, while the decoder, analogous to a conventional
language model, combines these representations to produce a distribution over
the output target through a softmax layer. The prediction network and joint
network collectively form the decoder. The time-sparse mechanism reduces the
time resolution by decomposing the encoder’s hidden states into intermediate
encoded representations using a sliding pooling window. These representations
are then combined into sparse hidden states using a weighted average
algorithm, and subsequently fed to the joint network. As a result, the
sequence length of the sparse hidden states outputted by the time-sparse
mechanism is significantly smaller than that of the encoder. This compression
in the length of hidden states effectively reduces the GPU memory footprint
and computing time. Additionally, introducing an attention mechanism [5]
during the combination of intermediate encoded representations enhances the
coefficients of representations with valuable information and suppresses noisy
representations. Furthermore, the attention coefficients contribute to a lower
character error rate (CER) for our model compared to the RNN-T baseline, as
demonstrated through experiments conducted on the AISHELL-1 dataset.

The remaining sections of this paper are organized as follows: Section 2
provides an overview of the RNN-T method, while Section 3 describes the
structure of the time-sparse mechanism and highlights key considerations
during the generation of sparse representations. This section also presents
the strategies for generating the weighted average coefficients. In Section 4,
we detail the experiments conducted and their respective results. Finally, in
Section 5, we present our conclusions, summarizing the key aspects and effects
of the TST model.

##  2 Background

Our proposed approach is based on the Recurrent Neural Network Transducer
(RNN-T) model. In this section, we provide an overview of the RNN-T structure,
training strategy, and decoding process [23].

The RNN-T consists of two distinct networks: the acoustic encoder and the
prediction network, which are connected through the joint network. The
acoustic encoder maps an input frame $\textbf{x}_{t}$ to a hidden state vector
$\textbf{h}_{t}$. The linguistic state vector $\textbf{g}_{u}$ is generated by
appending the prediction ”non-blank” symbol from the previous time step to the
prediction network. The joint network is a feed-forward network that combines
the hidden state vector $\textbf{h}_{t}$ and the linguistic state vector
$\textbf{g}_{u}$ as follows:

$\textbf{z}_{t,u}=f_{Activate}(W\textbf{h}_{t}+V\textbf{g}_{u}+b)$ (1)

Here, $W$ and $V$ are weight matrices, $b$ is a bias vector, and
$f_{Activate}$ represents an activation function such as Tanh or ReLU. The
output $\textbf{z}_{t,u}$ is then linearly transformed:

$\textbf{m}_{t,u}={\rm{Linear}}(\textbf{z}_{t,u})$ (2)

To obtain the posterior probability distribution $p(k|t,u)$ over the next
output symbol, a softmax function is applied to $\textbf{m}_{t,u}$:

$p(k|t,u)=\rm{Softmax}(\textbf{m}_{t,u})$ (3)

The probability distribution is computed at each point in the grid formed by
the prediction network and the acoustic encoder. The RNN-T model employs the
forward-backward algorithm to sum the probabilities of all possible paths.
However, this processing approach results in significant memory requirements
[10]. The loss function for RNN-T is defined as the negative log-likelihood of
the target sequence $y*$:

$\mathcal{L}_{RNN-T}=-{\rm{ln}}(P(y*|x))$ (4)

In terms of inference, the RNN-T performs frame-by-frame computation, which
can be slow when processing long sequences. The decoder employs beam search
and greedy search methods to identify the most likely sequence as the output
of the network [8].

Overall, the RNN-T model exhibits a distinctive architecture, comprising an
acoustic encoder, prediction network, and joint network. The forward-backward
algorithm is employed to compute probabilities, while the negative log-
likelihood serves as the loss function. The inference process can be time-
consuming for long sequences, and decoding methods like beam search and greedy
search are employed to obtain the output sequence.

Figure 1: Illustration of the structure of our proposed model time-sparse
transducer. Compared with the RNN-T, a sparse block in time is introduced
between the encoder and the joint network as shown in the figure. All
intermediate representations are generated by sliding a window, which is shown
in the Time-Sparse Block of this figure, on hidden states with two
hyperparameters: window length and stride. Both of them can be set before the
training process and would make observed disparity on the reasoning output.
After that, the time-sparse block combines the encoded representations through
the weighted average algorithm and feeds them into the joint network. The
sequence length of the sparse hidden state is reduced after this process.

##  3 Methodology

Our proposed method, Time-Sparse Transformer (TST), is based on the
decomposition of encoded hidden states using a time-sparse mechanism. The
time-sparse mechanism consists of two components: a decomposition of hidden
states based on a sliding pooling window and a combination process of sparse
hidden states based on the weighted average algorithm. In this section, we
provide detailed explanations of each component.

###  3.1 The decomposition based on sliding pooling window

We first reduce the time resolution by decomposing the encoded hidden states
into intermediate encoded representations by the TST algorithm. In the time-
sparse mechanism, the intermediate representations are generated by sliding a
window on hidden states with fixed window length and stride. Through this
process, the information carried by the output of the encoder can be spread
over various encoded representations $\mathbf{r_{k}}$ with smaller time
resolution than hidden states as Equation 5:

$[\mathbf{r_{1}},\mathbf{r_{2}},\mathbf{r_{3}},...\mathbf{r_{k}},...\mathbf{r_{n}}]=f_{win}(\mathbf{x},input,length,stride)$
(5)

where the $\mathbf{x}$ and $\mathbf{r_{k}}$ are the input and intermediate
encoded representation output by decomposition respectively and $n$ is the
number of intermediate representations. The window can overlap partly during
sliding to ensure continuous information between each representation. The
pooling process is primarily affected by two factors, namely, the window
length affecting the size of the sliding window, and the window stride
affecting the size of the overlap between windows. The time resolution of the
intermediate encoded representation decreases as the window length and stride
size increase. So a large window length and stride size will cause the loss of
detailed information but retain more global information when the encoder
output is decomposed. After the decomposition, the intermediate encoded
representations can be calculated with less computational effort in a shorter
time due to the smaller sequence length than encoded hidden states.
Conversely, setting a smaller window length and stride size can retain more
detailed information, but also generate representations with greater sequence
length than setting smaller sequence length and size, thus increasing the
processing time and memory requirements of the time-sparse mechanism.

###  3.2 The combination based on weighted average algorithm

After decomposing the encoded hidden states by sliding window, we combine the
encoded representations through the weighted average algorithm and feed them
into the joint network. In this process, the weighted average algorithm does
not need to change the sequence length of the input and output, so that the
time resolution of the sparse hidden state fed into the joint network by TST
is much smaller than that of the hidden state fed into the joint network by
RNN-T.

####  3.2.1 Combine with absolute average

In weighted average combination, an intuitive way is that all weight
coefficients are initialized to be identical, as shown in Equation 6.

$\textbf{h}^{{}^{\prime}}_{t}=\sum_{k=1}^{n}\frac{1}{n}\cdot\textbf{r}_{k}$
(6)

This method reduces the time it consumes to calculate the coefficient but
ignores the difference in the information carried by intermediate encoded
representations.

####  3.2.2 Combine with learnable coefficients

In addition to the absolute average, the weight coefficient can also be
initialized with a set of random coefficients and jointly optimized with other
parameters through the RNN-T loss function during model training [24]. This
process is illuminated as

$\textbf{h}^{{}^{\prime}}_{t}=\sum_{k=1}^{n}w_{k}\cdot\textbf{r}_{k}$ (7)

where $w_{k}$ is the learnable coefficient. The precision of the TST can be
improved by increasing the coefficient of the intermediate encoded
representation with the information that has a positive impact on the
prediction and suppressing the coefficient of the representation that carries
noise information.

####  3.2.3 Combine with attention mechanism

Inspired by the self-attention mechanism [5], we introduce the attention
mechanism to calculate the weighted average coefficients. The attention
weights $\alpha_{k}$ are computed as follows:

${\rm{e}}_{k}=\rm{Linear}(\textbf{r})$ (8) $\alpha_{k}={\rm{Softmax}(e}_{k})$
(9) $\textbf{h}^{{}^{\prime}}_{t}=\sum_{k=1}^{n}\alpha_{k}\cdot\textbf{r}_{k}$
(10)

In this case, the attention mechanism allows the model to focus more on
positive information. By calculating the attention weights, TST ensures that
the intermediate encoded representations with significant contributions are
given higher coefficients in the combination process.

In conclusion, the decomposition of the encoded hidden states and the
combination of intermediate encoded representations occur only between the
encoder and the joint network. From the decoder’s perspective, the features
inputted to the decoder in TST are indistinguishable from those in RNN-T.
Therefore, any decoder used for RNN-T can be employed for TST.

##  4 Experiments and Results

###  4.1 Experimental setup

All of our experiments are conducted on a public Mandarin speech corpus
AISHELL-1 [25]. We use 80-dimension Mel-filter Bank coefficients (FBank)
features with 3-dim pitch features computed on 25 ms window with 10 ms shift,
which is known to be effective in Mandarin speech recognition. The 4234
characters (including a padding symbol <PAD>, an unknown token <UNK>, a begin-
of-sequence token <BOS> and an end-of-sequence token <EOS>) are chosen as
modeling units.

For the baseline RNN-T model, the front-end convolutional block followed by
the encoder consists of two 2D-Convolution layers with a ReLU activation,
stride size 2, channels 384, kernel size 3, and output size 384. The acoustic
encoder consists of 12 transformer blocks with 4 heads in multi-head
attention. The feed-forward size of the encoder is 384 and the hidden size is
768. We utilize three types of decoders. The first is a transformer decoder
with 6 blocks and 4 heads in multi-head attention [26, 27, 6]; The second is a
state-less decoder [28], and the third is an RNN decoder with 2-layer Long
Short-Term Memory (LSTM) model. The configuration of TST is the same as RNN-T,
except for introducing a time-sparse mechanism between the RNN-T encoder and
joint network. Experiments are conducted on the sliding pooling window length
from 1 to 10 with fixed window stride 1 and stride from 1 to 10 with window
length 10. The generation strategies of intermediate encoded representation
coefficients are absolute equality (AE), optimization of the random
initialization coefficients through the RNN-T loss function (LC), and
calculation through the self-attention mechanism (SA). We utilize the CER to
evaluate the accuracy of different models and the real-time factor (RTF) to
evaluate the inference speed.

Figure 2: Illumination of the GPU memory utilization and time consumption of
RNN-T and TST on batch data. Both experiments are conducted on a Tesla K80
GPU.

###  4.2 Results

####  4.2.1 The comparison of GPU consumption between RNN-T and TST

We conducted a comparative analysis of GPU memory utilization and time
consumption between RNN-T and TST, as depicted in Figure 2. The primary
objective was to evaluate the impact of the time-sparse mechanism on these
performance metrics. Our results revealed that the utilization of GPU memory
decreased from 9% to 7.1% when utilizing TST instead of RNN-T. Additionally,
the time consumption exhibited a noticeable improvement, reducing from 299.856
ms to 213.387 ms when processing batch data. These findings clearly
demonstrate that TST achieves lower GPU occupancy and faster computation speed
compared to RNN-T. By incorporating the time-sparse mechanism into RNN-T, we
effectively reduce the sequence length of the hidden states received by the
prediction network. Consequently, this reduction in sequence length leads to
decreased GPU occupancy and significantly reduces the time required for
subsequent computations.

Table 1: Comparison of our TST with the RNN-T after the downsampling process along time. To illustrate the disparity between before and after time sparing, we also show the performance of the commonly used RNN-T as the baseline. The time resolutions of both TST and downsampling RNN-T are reduced to 1/10 that of the RNN-T. Model | RNN-T | Downsampling | TST
---|---|---|---
AE | LC | SA
CER (%) | 7.824 | 43.926 | 15.784 | 13.411 | 12.760
RTF | 0.122 | 0.019 | 0.021 | 0.021 | 0.021

####  4.2.2 The influence of downsampling in the encoded hidden state of TST
and RNN-T

In this section, we evaluate the performance of TST and RNN-T when the time
resolution is reduced. Table 1 presents the results, highlighting that there
is no significant difference in terms of Real-Time Factor (RTF) between TST
and RNN-T. Both models achieve a similar RTF reduction, approximately to 1/6
of the original value, when the time resolution is decreased to 1/10. However,
it is important to note that downsampling the data along the time axis leads
to a substantial loss in accuracy, amounting to 36.102%.

Regarding our TST, it incorporates a time-sparse mechanism, enhanced by a
self-attention block, which effectively mitigates the accuracy loss.
Specifically, TST achieves a remarkable reduction in accuracy loss to 4.936%.
The introduction of the time-sparse mechanism, with its self-attention
component, enables TST to better preserve the relevant information in the
sparse representations, thus significantly minimizing the adverse effects of
downsampling on accuracy.

Table 2: Comparison of TST with different sliding pooling window lengths. The window stride of all experiments is 1. Window Length | AE | LC | SA
---|---|---|---
CER(%) | RTF | CER(%) | RTF | CER(%) | RTF
10 | 9.454 | 0.160 | 9.213 | 0.161 | 8.714 | 0.158
8 | 9.225 | 0.162 | 8.521 | 0.163 | 8.231 | 0.159
6 | 9.013 | 0.160 | 8.244 | 0.163 | 7.942 | 0.160
4 | 8.562 | 0.161 | 7.685 | 0.162 | 7.418 | 0.161
2 | 8.259 | 0.162 | 7.812 | 0.164 | 7.533 | 0.159
1 | 8.205 | 0.162 | 8.139 | 0.163 | 7.824 | 0.159
Table 3: Comparison of TST with different sliding pooling window strides. The window length of all experiments is 10. Window Stride | AE | LC | SA
---|---|---|---
CER(%) | RTF | CER(%) | RTF | CER(%) | RTF
10 | 15.784 | 0.021 | 13.411 | 0.022 | 12.760 | 0.021
8 | 13.598 | 0.039 | 11.531 | 0.039 | 11.320 | 0.039
6 | 11.347 | 0.050 | 10.338 | 0.050 | 9.672 | 0.049
4 | 10.442 | 0.061 | 9.866 | 0.061 | 8.831 | 0.060
2 | 10.215 | 0.093 | 9.734 | 0.093 | 8.799 | 0.093
1 | 9.454 | 0.160 | 9.213 | 0.161 | 8.714 | 0.158

####  4.2.3 The influence of the sliding pooling window length and stride on
the model performance

This section presents a comparative analysis of models employing different
sliding pooling window lengths and stride sizes. The experimental results, as
depicted in Tables 2 and 3, clearly demonstrate that reducing the window
length and stride size improves the accuracy of the models. When a large
window with a large step size slides over the encoded hidden state, it leads
to the loss of more detailed information. Conversely, a small window with a
small step size preserves more information but results in a longer sequence
length. The outcomes reveal that, for TST, the weighted average combination
approach effectively reduces the CER when the window length is less than 4,
accomplishing this by increasing the weight assigned to positive information
while suppressing the weight assigned to noise. Moreover, employing the SA
yields the lowest CER among the tested approaches. However, it should be noted
that an increase in window length diminishes the information content, thereby
decreasing the accuracy of TST. Notably, the tables demonstrate that the RTF
remains largely unaffected by changes in window length but decreases as the
stride size increases. This observation highlights that the inference speed of
TST primarily depends on the decoding sequence length rather than the scale of
intermediate encoded representation.

####  4.2.4 The experimental results of different window types

This section primarily investigates the impact of different strategies
employed for generating weighted average coefficients within the Time-Sparse
Transformer (TST) framework. Notably, these strategies have no bearing on the
decoding sequence length, thereby ensuring consistent computational speed for
TST. The experimental findings presented in Table 1, 2, and 3 provide
compelling evidence regarding the efficacy of the SA strategy, which yields
the lowest CER on the AISHELL-1 dataset. Comparing the LC approach with the SA
strategy, it becomes evident that the inclusion of the attention mechanism
significantly aids the model in learning the weighted average coefficients.
Specifically, the self-attention mechanism strengthens the coefficient
associated with intermediate encoded representations that convey pertinent
information while simultaneously diminishing the coefficient assigned to
representations containing noise. Consequently, the resulting sparse hidden
state exhibits enhanced recognition capabilities pertinent to the target task.
These results shed light on the discriminative power and adaptability provided
by the self-attention mechanism within the TST framework, reinforcing its
effectiveness for optimizing speech recognition performance.

Table 4: The Result of RNN-T and TST with different decoders. Both window length and window stride are 4. The window type is SA. Model |  CER(%) |  RTF
---|---|---
RNNT-T |  7.979 |  0.508
TST-T (Ours) |  7.744 |  0.285
RNNT-S |  7.824 |  0.122
TST-S (Ours) |  7.528 |  0.061
RNNT-R |  11.454 |  0.397
TST-R (Ours) |  10.193 |  0.222
Table 5: The results of RNN-T and TST with different decoders. Both window length and window stride are 10. The window type is SA. Model |  CER(%) |  RTF
---|---|---
RNNT-T |  7.979 |  0.508
TST-T (Ours) |  12.934 |  0.084
RNNT-S |  7.824 |  0.122
TST-S (Ours) |  12.760 |  0.021
RNNT-R |  11.454 |  0.397
TST-R (Ours) |  15.968 |  0.089

####  4.2.5 The experimental results of sliding pooling window with different
decoders

This section focuses on comparing the performance of TST and RNN-T with
different decoders, namely RNN-T with Transformer decoder (RNNT-T), RNN-T with
State-Less decoder (RNNT-S), RNN-T with RNN decoder (RNNT-R), TST with
Transformer decoder (TST-T), TST with State-Less decoder (TST-S), and TST with
RNN decoder (TST-R). Experiments were conducted to evaluate their performance
in terms of CER and recognition speed.

Table 4 presents the results, indicating that TST consistently achieves a
lower CER and higher recognition speed compared to RNN-T in all three types of
decoder. Specifically, when comparing TST-S with RNNT-S, it was found that the
CER of TST-S is very close to that of RNN-T, while the RTF is reduced to 50%
of RNNT-S. These findings highlight that TST-S achieves comparable accuracy to
RNN-T while significantly improving computational efficiency.

Furthermore, Table 5 provides insights when the time resolution is reduced to
1/10. The CERs of models with different decoders are observed to increase, but
the RTFs are reduced to 16.535% to 22.418% of the original values. Notably,
among all the experiments, TST-S achieves the highest accuracy and its RTF is
reduced to 17.213% of that of RNNT-S. These results demonstrate that TST
offers the potential to improve both recognition accuracy and computational
efficiency across various decoder types. Moreover, when the time resolution is
decreased (see Table 4), TST maintains superior performance in terms of
accuracy and computational efficiency compared to RNN-T.

##  5 Conclusion

In this study, we have introduced a novel model called the time-sparse
transducer, which incorporates a time-sparse mechanism between the recurrent
neural network (RNN) transducer encoder and the prediction network. Our
proposed model offers several advantages compared to conventional transducers,
including reduced memory consumption and accelerated computation through the
reduction of time resolution in the encoded hidden state. Through our
experimentation on the AISHELL-1 dataset, we have observed that the
incorporation of the time-sparse mechanism in the prediction phase leads to a
notable decrease in the character error rate. This finding underscores the
efficacy of the time-sparse transducer in enhancing recognition accuracy and
reducing inference time consumption when employed in conjunction with various
decoders. While this study primarily focused on the development and evaluation
of the time-sparse transducer, there are avenues for future research that
warrant exploration. Specifically, we intend to investigate optimizing
strategies for weighted average coefficients on diverse speech corpora. Such
exploration will enable us to better understand the implications and
generalized ability of our proposed model across different linguistic
contexts.

## References

  * [1] Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on Machine learning. pp. 369–376 (2006)
  * [2] Graves, A., Jaitly, N.: Towards end-to-end speech recognition with recurrent neural networks. In: International conference on machine learning. pp. 1764–1772. PMLR (2014)
  * [3] Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., Chen, G., et al.: Deep speech 2: End-to-end speech recognition in english and mandarin. In: International conference on machine learning. pp. 173–182. PMLR (2016)
  * [4] Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
  * [5] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [6] Dong, L., Xu, S., Xu, B.: Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5884–5888. IEEE (2018)
  * [7] Kim, S., Hori, T., Watanabe, S.: Joint ctc-attention based end-to-end speech recognition using multi-task learning. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 4835–4839. IEEE (2017)
  * [8] Graves, A.: Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711 (2012)
  * [9] Graves, A., Mohamed, A.r., Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE international conference on acoustics, speech and signal processing. pp. 6645–6649. Ieee (2013)
  * [10] Rao, K., Sak, H., Prabhavalkar, R.: Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer. In: 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). pp. 193–199. IEEE (2017)
  * [11] He, Y., Sainath, T.N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al.: Streaming end-to-end speech recognition for mobile devices. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6381–6385. IEEE (2019)
  * [12] Han, W., Zhang, Z., Zhang, Y., Yu, J., Chiu, C.C., Qin, J., Gulati, A., Pang, R., Wu, Y.: Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. ArXiv abs/2005.03191 (2020)
  * [13] Zhang, Q., Lu, H., Sak, H., Tripathi, A., McDermott, E., Koo, S., Kumar, S.: Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 7829–7833 (2020). https://doi.org/10.1109/ICASSP40776.2020.9053896
  * [14] Kannan, A., Datta, A., Sainath, T.N., Weinstein, E., Ramabhadran, B., Wu, Y., Bapna, A., Chen, Z., Lee, S.: Large-scale multilingual speech recognition with a streaming end-to-end model. arXiv preprint arXiv:1909.05330 (2019)
  * [15] Variani, E., Rybach, D., Allauzen, C., Riley, M.: Hybrid autoregressive transducer (hat). In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6139–6143 (2020). https://doi.org/10.1109/ICASSP40776.2020.9053600
  * [16] Li, J., Zhao, R., Hu, H., Gong, Y.: Improving rnn transducer modeling for end-to-end speech recognition. In: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). pp. 114–121. IEEE (2019)
  * [17] Venkatesh, G., Valliappan, A., Mahadeokar, J., Shangguan, Y., Fuegen, C., Seltzer, M.L., Chandra, V.: Memory-efficient speech recognition on smart devices. In: ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 8368–8372 (2021). https://doi.org/10.1109/ICASSP39728.2021.9414502
  * [18] Han, Y., Zhang, C., Li, X., Liu, Y., Wu, X.: Query-based composition for large-scale language model in lvcsr. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 4898–4902 (2014). https://doi.org/10.1109/ICASSP.2014.6854533
  * [19] Zhang, Y., Sun, S., Ma, L.: Tiny transducer: A highly-efficient speech recognition model on edge devices. In: ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6024–6028 (2021). https://doi.org/10.1109/ICASSP39728.2021.9413854
  * [20] Chen, X., Wu, Y., Wang, Z., Liu, S., Li, J.: Developing real-time streaming transformer transducer for speech recognition on large-scale dataset. In: ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5904–5908 (2021). https://doi.org/10.1109/ICASSP39728.2021.9413535
  * [21] Li, J., Zhao, R., Hu, H., Gong, Y.: Improving rnn transducer modeling for end-to-end speech recognition. In: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). pp. 114–121 (2019). https://doi.org/10.1109/ASRU46091.2019.9003906
  * [22] Kim, C., Gowda, D., Lee, D., Kim, J., Kumar, A., Kim, S., Garg, A., Han, C.: A review of on-device fully neural end-to-end automatic speech recognition algorithms. In: 2020 54th Asilomar Conference on Signals, Systems, and Computers. pp. 277–283 (2020). https://doi.org/10.1109/IEEECONF51394.2020.9443456
  * [23] Tian, Z., Yi, J., Tao, J., Zhang, S., Wen, Z.: Hybrid autoregressive and non-autoregressive transformer models for speech recognition. IEEE Signal Processing Letters 29, 762–766 (2022)
  * [24] Ostmeyer, J., Cowell, L.: Machine learning on sequential data using a recurrent weighted average. Neurocomputing 331, 281–288 (2019)
  * [25] Bu, H., Du, J., Na, X., Wu, B., Zheng, H.: Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In: 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment (O-COCOSDA). pp. 1–5. IEEE (2017)
  * [26] Moritz, N., Hori, T., Le, J.: Streaming automatic speech recognition with the transformer model. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6074–6078. IEEE (2020)
  * [27] Tian, Z., Yi, J., Tao, J., Bai, Y., Zhang, S., Wen, Z.: Spike-triggered non-autoregressive transformer for end-to-end speech recognition. arXiv preprint arXiv:2005.07903 (2020)
  * [28] Ghodsi, M., Liu, X., Apfel, J., Cabrera, R., Weinstein, E.: Rnn-transducer with stateless prediction network. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 7049–7053. IEEE (2020)