# Dynamic modeling of the Italians’ attitude towards Covid-19

Emanuele Aliverti, Massimilano Russo Department of Economics, University Ca’
Foscari Venezia<EMAIL_ADDRESS>Division of Pharmacoepidemiology
and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital
and Harvard Medical School<EMAIL_ADDRESS>
###### Abstract

We analyze repeated cross-sectional survey data collected by the Institute of
Global Health Innovation, to characterize the perception and behavior of the
Italian population during the Covid-19 pandemic, focusing on the period that
spans from April 2020 to July 2021. To accomplish this goal, we propose a
Bayesian dynamic latent-class regression model, that accounts for the effect
of sampling bias including survey weights into the likelihood function.
According to the proposed approach, attitudes towards covid-19 are described
via ideal behaviors that are fixed over time, corresponding to different
degrees of compliance with spread-preventive measures. The overall tendency
toward a specific profile dynamically changes across survey waves via a latent
Gaussian process regression, that adjusts for subject-specific covariates. We
illustrate the evolution of Italians’ behaviors during the pandemic, providing
insights on how the proportion of ideal behaviors has varied during the phases
of the lockdown, while measuring the effect of age, sex, region and employment
of the respondents on the attitude toward covid-19.

Keywords: Bayesian inference; categorical data; dynamic modeling; repeated
cross-sectional data; survey weights.

##  1 Introduction

The outbreak of the covid-19 pandemic has impacted our world, with more than
$250$ million infected people, and more than $5$ million deaths at December
2021. 1 From an economic perspective, the covid-19 outbreak and the national
measures to contain the spread of the disease lead to severe economic
recessions in many sectors, such as tourism, accommodation, and food services.
2 Psychological and social effects on the population are less immediate to
quantify, but some preliminary results suggest that the pandemic has affected
also these aspects of human life. 3, 4, 5

In this article, we study the evolution of behaviors in compliance with some
measures that prevent the spread of covid-19 (see Table 1). We focus on the
Italian population, that represents an interesting case study, because Italy
was the first European country to introduce a national lockdown to limit the
spread of covid-19, imposing behavioral changes in the population. The adopted
measures are likely to have had a psychological and social effect
proportionate with their severity and duration. In fact, some early reports,
based on online surveys, suggest an increased level of distress, anxiety, and
fear. 6, 7, 8 Monitoring these aspects, quantifying their evolution over time,
and characterizing their impact on individuals is of primary importance to
evaluate the wellbeing of a population. Similarly, it is of interest to
evaluate the compliance with covid-19 policies, that can largely depend on
personal status. 9

Variations of personal routines and practices have been documented; for
example, the reduction of social activities and gatherings and the changes in
mobility patterns, such as the increase of work-from-home practices and the
reduction of public transport use. 10, 11 These behaviors reflect the
compliance with the government regulations, as well as the internalization of
different recommendations to reduce the spread of the virus, that changed day-
to-day life. However, compliance with spread reducing measures depends upon
personal conditions, psychological status, and many other personal factors, 12
and it is likely to change during different stages of the pandemic.

During the first phase of the pandemic (up to June 2020), there is evidence
that the Italian population has scrupulously followed government measures 13,
14; however, to the best of our knowledge, no analysis has been performed to
assess if such compliance is constant over time, across socio-demographic
groups, or proportional to the severity of the measures adopted by the Italian
government throughout the pandemic. To quantify these aspects, we analyze
survey data provided by the Institute of Global Health Innovation (ighi) at
the Imperial College of London, in collaboration with the company YouGov, 15
described in Section 3.

We describe the attitude of the Italian population toward covid-19 throughout
the pandemic with a dynamic Bayesian latent class regression model. 16, 17
Such models assume that the population is composed of $H$ groups corresponding
to ideal behaviors, that can represent different attitudes towards covid-19.
At any given time, each subject composing the population is associated with
one of these ideal profiles, and the categorical variables characterizing
behaviors are modeled as conditionally independent given the profile
memberships. Latent class models are conceptually simple and have been used as
a building block for several methods to analyze categorical variables; for
example, in problems including survey weights, 18, 19 when interest is on
characterizing temporal dependence across contingency tables 20 or differences
among groups of subjects. 21 See also Chapters 9 and 11.5 of Handbook of
mixture analysis22 for further references.

The paper is structured as follows: Section 2 contextualize our contribution
within the current literature, while Section 3 describes the survey data
analyzed in this article; Section 4 introduces the proposed dynamic Bayesian
latent-class regression model, and Section 5 illustrates the results and
empirical findings. Finally, Section 6 provides a brief discussion.

##  2 Related work

During the past two years, there has been a broad interest in modeling the
evolution of the covid-19 pandemic. Indeed, incidence data such as the number
of positive individuals, hospitalizations, and intensive care unit admissions
have been systematically collected and released to the public on regular
basis; for example, in Italy they have been released daily by the Department
of “Protezione Civile” (https://github.com/pcm-dpc/COVID-19). Accurate
modeling of these data has proven useful to measure the current state of the
pandemic, to develop strategies based on empirical evidence, and to evaluate
the success of different policies adopted by the governments. In this context,
Girardi et al.23 developed a robust non-linear regression to model and predict
the contagion dynamics of covid-19 in Italy. Alaimo Di Loro et al. 24
introduced a novel generalized linear model based on Richards’ curves,
obtaining accurate short-term forecasts of incidence indicators. Girardi et
al.25 also proposed a change-point growth model that is able to capture
subsequent pandemic waves, while Scrucca 26 developed a real-time index that
summarizes the current state of the pandemic. We also refer to27, 28, 29, 30
for additional modeling strategies and analysis of Italian incidence data.
Similar analysis have been developed at the European 31, 32, 33 and global
level. 34, 35, 36

Less attention has been devoted to study the impact of covid-19 on individual
attitudes, analyzing personal behavior and their interactions with compliance
with preventive measures. For example, Daoust et al., 37 studied the different
level of compliance of Canadian provinces, using composite indicators, and
Krekel et al. 38 studied the associations between happiness and level of
compliance with government regulations. Behavioral data have also been used as
important predictors of the number of covid-19 cases. 39 In Italy, Duradoni et
al. 40 studied the psychological profile of people that were compliant with
the government regulations a month after the lockdown started, while Guazzini
et al. 41 tested changes in the psychological adaptation across the first two
waves of the pandemic. In this context, we consider the evolution of the
compliance of Italian population, quantifying variations of this compliance
over time, across socio-demographic groups, and studying its associations with
the severity of the measures adopted by the Italian government throughout the
pandemic.

##  3 Data description

Table 1:  List of analyzed survey items with code, label, and description.
Subjects can respond to each item with “Not at all”, “Rarely”, “Sometimes”,
“Frequently” and “Always”, according to their level of agreement with each
survey item.

Survey Code Label Description i12_health_1 ih1 Worn a face mask outside your
home (e.g. when on public transport going to a supermarket or going to a main
road) i12_health_2 ih2 Washed hands with soap and water i12_health_3 ih3 Used
hand sanitiser i12_health_4 ih4 Covered your nose and mouth when sneezing or
coughing i12_health_5 ih5 Avoided contact with people who have symptoms or you
think may have been exposed to the coronavirus i12_health_6 ih6 Avoided going
out in general i12_health_7 ih7 Avoided going to hospital or other healthcare
settings i12_health_8 ih8 Avoided taking public transport i12_health_11 ih11
Avoided having guests to your home i12_health_12 ih12 Avoided small social
gatherings (not more than 2 people) i12_health_13 ih13 Avoided medium-sized
social gatherings (between 3 and 10 people) i12_health_14 ih14 Avoided large-
sized social gatherings (more than 10 people) i12_health_15 ih15 Avoided
crowded areas i12_health_16 ih16 Avoided going to shops

We analyze repeated cross-sectional survey data provided by the Institute of
Global Health Innovation (ighi) at the Imperial College of London, in
collaboration with the company YouGov. 15 Data are publicly available for
research purposes at the repository https://github.com/YouGov-
Data/covid-19-tracker, along with a brief description of the collected
variables. This survey aims to investigate how different populations responded
to covid-19, gathering self-reported data on several aspects of the pandemic,
including objective measurements, such as testing results and observed
symptoms, and subjective measurements, such as daily behaviors.

We are interested in subjective measurements describing the compliance with
national preventive regulations. This measurements include questions on self-
isolation, avoidance of social gatherings, frequency of hand washing, use of
hand sanitizer and contact with other people, among many others. The complete
list of variables used in this study is reported in Table 1, which includes
$14$ out of the $20$ subjective measurements available in the survey. The
removed items were considered uninformative for the population behavior in
relation with the covid-19 measures adopted by the Italian government. For
example, one removed question asked whether children living in the same
household were going to school; however, schools have been closed for most of
the pandemic and students could only attend lectures remotely.

Publicly available data focus on $T=37$ survey waves conducted from April 2020
to July 2021. Notably, the survey waves are not administered at regular time-
intervals; for example, $4$ survey waves were submitted in April and May 2020,
and only $1$ in July and August 2020; see Table 2 for details on the waves
administration dates. Without loss of generality, we will indicate this time
grid with $\mathcal{T}=\\{t_{1},\dots,t_{T}\\}$. In each wave
$t\in\mathcal{T}$, cross-sectional data were collected for a representative
sample of $1000$ statistical units, indexed as $i=1,\dots,n_{t}=n$. Each unit
is associated with a sampling weight $w_{i}(t)$ and a vector of covariates
$\mathbf{x}_{i}(t)=[x_{i1}(t),\dots,x_{im}(t)]^{\intercal}$, including age,
sex, region of residence (“North-West”, “North-East”, “Center”, “South”,
“Islands”) and employment status (“Full-time employment”, “Part-time
employment”, “Not working”, “Student” and “Retired”). Note that unit $i$ at
wave $t$ indicates a different subject than unit $i$ at wave $t^{\prime}\neq
t$; for this reason, we include the wave index $t$ in the covariate vector as
$\mathbf{x}_{i}(t)$, even if the covariates $\mathbf{x}_{i}(t)$ can be
considered fixed over time. Refer to Jones 15 for further information.

This data collection scheme is commonly referred to as “repeated cross-
sectional” or “pseudo-panel” in the literature. We refer to Verbeek 42 for an
extensive discussion on the advantages and limitations of repeated cross-
section data compared to longitudinal data. In longitudinal data, temporal
variations can be modeled, for example, relying on Hidden Markov Models; refer
to Latent Markov models for longitudinal data 43 and references therein for
more details. These approaches model the transitions across ideal behaviors
for each individual, and allow to include the effects of covariates directly
in the transition probabilities that characterize the time dynamic. In our
application, we cannot characterize individual trajectories due to the cross-
sectional nature of the survey data. This is in fact a major limitation of
cross-sectional data along with their inability to monitor changes in personal
attitudes. On the other end, repeated cross-sectional data allow to
effectively model average tendencies of a population across time and, since
they are less effected by attrition, they typically present a larger sample
sizes than longitudinal data. 42 Therefore in Section 4, we describe a dynamic
regression model for repeated cross-sectional data that can characterize
average variations of the attitude of Italian population during the pandemic.

Table 2:  Calendar date of the 37 analyzed survey waves.

Wave 1 2 3 4 5 6 7 8 Date 2 Apr 2020 8 Apr 2020 16 Apr 2020 26 Apr 2020 1 May
2020 7 May 2020 13 May 2020 29 May 2020 Wave 9 10 11 12 13 14 15 16 Date 10
Jun 2020 25 Jun 2020 8 Jul 2020 23 Jul 2020 7 Aug 2020 19 Aug 2020 3 Sep 2020
12 Sep 2020 Wave 17 18 19 20 21 22 23 24 Date 2 Oct 2020 15 Oct 2020 28 Oct
2020 11 Nov 2020 16 Dec 2020 6 Jan 2021 13 Jan 2021 27 Jan 2021 Wave 25 26 27
28 29 30 31 32 Date 10 Feb 2021 24 Feb 2021 10 Mar 2021 24 Mar 2021 7 Apr 2021
21 Apr 2021 5 May 2021 19 May 2021 Wave 33 34 35 36 37 Date 2 Jun 2021 16 Jun
2021 30 Jun 2021 14 Jul 2021 28 Jul 2021

##  4 Methods

###  4.1 Model specification and interpretation

Let $\mathbf{y}_{i}(t)=(y_{i1}(t),\dots,y_{ip}(t))^{\intercal}$ denote the
responses of subject $i$ to the survey items outlined in Table 1 during survey
wave $t$; without loss of generality, we encode these responses as
$y_{ij}(t)\in\\{1,\dots,d\\}$ for $j=1,\ldots,p$, with $d=5$ and $p=14$. We
indicate with ${\mathbf{y}(t)=\\{\mathbf{y}_{i}(t),\,i=1,\ldots,n\\}}$ the
observed responses for the $n$ interviewed subjects during the survey wave
$t$. Our main goal is to study the evolution of the population’s behaviors
over time and during different stages of the covid-19 pandemic. To accomplish
this goal, we specify a low-dimensional dynamic model for multivariate
categorical data, that characterizes the time-varying probability mass
function of $\mathbf{y}_{i}(t)$ in terms of a set of static latent classes and
dynamic class-memberships. Since at each survey wave we observe $n$
statistical units chosen according to a survey design, we adjust the proposed
model likelihood to obtain an estimate of the population parameters.

In order to describe the evolution of $\mathbf{y}(t)$ over time, we assume
that the population can be divided into $H$ ideal behaviors (or profiles) that
express different attitudes toward covid-19. These ideal profiles can
correspond to different behavioral patterns, such as people that rigidly
observed all rules and directives to avoid the spread of the disease, or
people that kept behaving as usual, ignoring the emergency. The interpretation
and the structure of the ideal behaviors is considered fixed over time, while
their proportion can change dynamically. For example, it is likely that,
before seeing the effect of covid-19, several people were not concerned about
using public transport or having guests at home. However, as the disease
spread and the effects became more evident, these inclinations potentially
reversed for a subset of individuals.

$\nu_{ih}(t)$ $\mathbf{x}_{i}(t)$
$z_{i}(t)$$\boldsymbol{\beta}_{h}$$\eta_{h}(t)$$\mathbf{y}_{i}(t)$
$\boldsymbol{\psi}_{h}^{(j)}$$h=1,\ldots,H$$j=1,\ldots,p$$h=1,\ldots,H$ Figure
1: Graphical representation of the mechanism to generate data
$\mathbf{y}_{i}(t)$ from model (1).

This evolution is modeled relying on a dynamic latent-class regression model,
graphically described in Figure 1. For each wave $t$, subject $i$ is
associated to one of the $H$ profiles via the discrete latent variable
$z_{i}(t)\in\\{1,\dots,H\\}$ with probability
$\nu_{ih}(t)=\mbox{pr}(z_{i}(t)=h\mid\mathbf{x}_{i}(t))$ that depends on the
observed covariate vector $\mathbf{x}_{i}(t)$. Given the profile membership
$z_{i}(t)=h$, the probability that subject $i$ responds with $c_{j}$ to the
survey item $j$ is denoted as
${\psi^{(j)}_{hc_{j}}=\mbox{pr}(y_{ij}(t)=c_{j}\mid z_{i}(t)=h)}$, with
$c_{j}\in\\{1,\dots,d\\}$ and $j=1,\dots,p$. Therefore, we assume that the
profiles’ attributes are constant over time, and can be characterized by the
conditional probabilities of responding with a certain category to the
different items, namely
${\boldsymbol{\Psi}_{h}=\\{\psi^{(j)}_{hc_{j}},c_{j}=1,\dots,d,\,j=1,\dots,p\\}}$.
Within each mixture component, the observed survey items are modeled as a
realization from a product of independent multinomial distributions. And,
under the considered conditional independence assumption, the dependence
structure of the survey items is entirely induced via marginalization of the
mixture weights.

Potentially, we could rely on alternative model specifications, exploiting the
ordering of the survey items categories. For example, we could let the
difference of probabilities across adjacent categories for each item to be
constant, effectively reducing the number of parameters in the model. However,
this formulation implies that the distributions of the survey items are
stochastically ordered. This assumption might be non-trivial to check
empirically, in particular if several items are modeled jointly; therefore we
prefer to follow the common practice of using a latent class model for
multivariate ordered categorical data, since this model is robust to
violations of ordering assumptions.44

The profile-specific membership varies across the survey waves and according
to subject specific covariates. Specifically, the evolution of the profile
memberships is modeled trough the probability vector
${\boldsymbol{\nu}_{i}(t)=(\nu_{i1}(t),\dots,\nu_{iH}(t))^{\intercal}}$, with
$\nu_{ih}(t)$ denoting the probability that subject $i$ of survey wave $t$
belongs to the $h$-th profile and $\sum_{h=1}^{H}\nu_{ih}(t)=1$. This
parameter is decomposed into two quantities: a profile-specific component
$\eta_{h}(t)$ and a subject-specific effect
$\boldsymbol{\beta}_{h}^{\intercal}\mathbf{x}_{i}(t)$ that linearly depends on
their observed covariates, with
$\boldsymbol{\beta}_{h}=(\beta_{1h},\dots,\beta_{mh})^{\intercal}$. The
dynamic component $\eta_{h}(t)$ characterizes the temporal evolution of the
profiles’ memberships, while the linear term for the effect of subject-
specific covariates.

Isolating the individual and the profile-specific component—which is shared
across the population—is particularly relevant in our setting, where we
analyze repeated cross-sectional data, including different subjects across
survey waves. Indeed, this structure allows to estimate the shared dynamic
component $\eta_{h}(t)$, adjusting for the different composition of the cross-
sectional population, and measuring the effect of demographic information on
the probability of belonging to a certain profile.

According to these specifications, the model depicted in Figure 1 can be
formalized as pr(y_i1(t) = c_1, …, y_ip(t) = c_p ∣x_i (t)) &= ∑_h=1^Hν_ih(t)
∏_j=1^pψ^(j)_hc_j,
log(νih(t)νi1(t)) = η_h(t) + β_h^⊺x_i(t). Using the first latent profile as a
reference, we let $\eta_{1}(t)=0$ and $\beta_{k1}=0$ for $k=1,\dots,m$ and
interpret each $\beta_{kh}$ as the effect of the $k$-th covariate on the log-
odds of belonging to profile $h$, instead of the first one, as in multinomial
logit regression e.g., 45. The conditional independence assumption among the
$p$ categorical variables, and the inclusion of the temporal component in the
mixture weights, leads to substantial dimensionality reduction in the number
of model parameters, while incorporating borrowing of information across
survey waves. The effect of of time in the model (1) can be made explicit as
follows. Let $\mathbf{X}$ denote an $(\sum_{t\in\mathcal{T}}n_{t}\times
m)$-dimensional design matrix obtained stacking the vectors
${[\mathbf{x}_{i}(t),i=1,\dots,n_{t},\,t\in\mathcal{T}]^{\intercal}}$ by rows;
i.e., the first $n_{1}$ rows of $\mathbf{X}$ correspond to the observation of
the $n_{1}$ individuals interviewed during first survey wave $t=t_{1}$, the
rows from $n_{1}+1$ to $n_{2}$ correspond to the individuals interviewed at
the second survey wave $t=t_{2}$, and so on. In matrix form, the linear
predictor of the second line of Equation (1) can be expressed as

$\mathbf{I}\boldsymbol{\eta}_{h}+\mathbf{X}\boldsymbol{\beta}_{h},$

where
$\mathbf{I}=\mbox{diag}(\boldsymbol{1}_{n_{1}},\dots,\boldsymbol{1}_{n_{T}})$
is a $(\sum_{t\in\mathcal{T}}n_{t}\times T)$-dimensional matrix which
identifies the survey wave of each observation, and $\boldsymbol{1}_{n}$ is a
vector of ones with length $n$; similarly,
$\boldsymbol{\eta}_{h}=[\eta_{h}(t_{1}),\dots,\eta_{h}(t_{T})]$ is a
$T$-dimensional vector containing the values of the dynamic intercepts. This
specification illustrates that time variations in the mean composition of each
profile are characterized non-parametrically, since the model includes a
different value of the intercept for each survey wave, instead of assuming a
parametric relationship between consecutive values of $\eta_{h}(t)$.

Following Equation (1), the likelihood contribution for subject $i$ in the
survey wave $t$ can be expressed as

$\mathbf{p}(\mathbf{y}_{i}(t)\mid\boldsymbol{\nu}_{i}(t),\mathbf{x}_{i}(t),\boldsymbol{\Psi})=\sum_{h=1}^{H}\nu_{ih}(t)\prod_{j=1}^{p}\prod_{c_{j}=1}^{d}\left[\psi^{(j)}_{hc_{j}}\right]^{\mathds{1}[y_{ij}(t)=c_{j}]},$
(1)

where $\mathds{1}[A]$ denotes the indicator function for the event $A$. To
mitigate the effect of potential bias due to the survey design, we follow the
approach described by Vermunt and Magidson 18 and Patterson et al., 19 and
incorporate the survey weights into the model via

$L(\boldsymbol{\nu}(t),\boldsymbol{\Psi})\propto\prod_{i=1}^{n}\left[\mathbf{p}(\mathbf{y}_{i}(t)\mid\boldsymbol{\nu}_{i}(t),\mathbf{x}_{i}(t),\boldsymbol{\Psi})\right]^{w_{i}(t)},$
(2)

exponentiating each likelihood contribution in (1) to the corresponding survey
weight $w_{i}(t)$. The pseudo-likelihood in (2) is used as building-block of
several likelihood-based procedures that include survey weights, 46, 47, 48
and recently in some Bayesian methods. 49 Alternatively, in a Bayesian
setting, one can use the survey weights to approximate the population
generative mechanism, and inferring characteristics of the non-sampled units
50 or, when available, use strong prior information to correct for the
sampling bias. 51 However, a weighted likelihood approach is conceptually
simpler and computationally more efficient when the sampling mechanism is
unknown, as in our application. 49

###  4.2 Prior specification and posterior computation

We consider a Bayesian approach to inference, using the pseudo-likelihood
outlined in Equation (2). An advantage of this approach is that prior
regularization can avoid convergence issues of maximization algorithms, such
as Expectation-Maximization (em), when used in latent-class regression.44, 52
Also, a Bayesian approach simplifies modeling of temporal dependencies across
survey waves leveraging a hierarchical dynamic model. To effectively estimate
the dynamic intercepts $\boldsymbol{\eta}_{h}$ inducing borrowing of
information across survey waves, we assume that the variations in profile
composition are smooth over time, and leverage a Gaussian Process (gp) prior
for the joint distribution of $\boldsymbol{\eta}_{h}$ over $\mathcal{T}$;
refer, for example, to Gaussian Processes for Machine Learning53 for an
introduction to gp. Recalling that the first group is fixed as a reference,
this prior assumes for each latent group $h=2,\dots,H$

$\eta_{h}(\cdot)\sim\mbox{{gp}}(0,\mathbf{C}_{\vartheta_{h}}),\quad\mbox{with}\quad\mathbf{C}_{\vartheta_{h}}(t,t^{\prime})=\vartheta_{1h}\exp\left(-\frac{(t-t^{\prime})^{2}}{2\vartheta_{2h}}\right)+\vartheta_{3h}\mathds{1}[t=t^{\prime}],$
(3)

denoting a squared-exponential covariance function and
$\vartheta_{1h}\in\mathbb{R}^{+},\vartheta_{2h}\in\mathbb{R}^{+}$ and
$\vartheta_{3h}\in\mathbb{R}^{+}$ corresponding to the variance, length-scale
and noise variance parameters, respectively. 53 The squared exponential
function favors smooth transitions across time, with the parameters
$\boldsymbol{\vartheta}_{h}=\\{\vartheta_{1h},\vartheta_{2h},\vartheta_{3h}\\}$
controlling the overall structure of the covariance function. Since the
covariance is parametrized as a function of the squared time lags across all
pairs of time points $(t,t^{\prime})$, it accounts for the unequal spacing
effectively, inducing higher correlation among closer time points; indeed,
continuous stochastic processes such as the gp are appropriate for time series
defined over continuous domains, where the spacing across time points is
arbitrary, see for example Chapter 6 of Analysis of financial time series.54

Prior specification is completed selecting: independent log-normal
distributions with log-mean $0$ and log-standard deviation $10$ for the
components of $\boldsymbol{\vartheta}_{h}$, standard Gaussian distribution for
the coefficients $\beta_{lh}$, and symmetric Dirichlet distributions for the
profile-specific conditional probabilities $\psi^{(j)}_{hc_{j}}$, letting ϑ_qh
&∼logN(0,10), q=1, …, 3, h=2,…,H,
β_kh ∼N(0,1), k=1, …, m, h=2,…,H,
(ψ^(j)_hc_1, …, ψ^(j)_hd) ∼Dirichlet(1, …, 1), j = 1,…,p, h = 1, …, H, and
recalling that $\beta_{hk}=0$ for identifiability.

We approximate the posterior distribution of the model parameters via Markov
Chain Monte Carlo (mcmc). Specifically, we rely on a Hamilton Monte-Carlo
algorithm, efficiently implemented in the r package rstan, 55 including the
weighted likelihood specification of Equation (2).

##  5 Modeling attitude towards Covid-19

To select the number of latent profiles, we evaluated the model performance in
predicting the $14$ survey items in Table 1 for various values of $H$, relying
on a $4$-folds cross-validation to estimate out-of-sample accuracy. The data
provide evidence that $H=3$ latent profiles provide the best fit to the data.
Refer to the Appendix A for further details on model selection. Posterior
inference for the selected model relies on $4000$ iterations collected after a
burn-in of $1000$. Convergence was assessed via graphical inspection of the
trace-plots, auto-correlation function and convergence diagnostics. All chains
mixed well, with an effective sample size larger than $3500$ for all
parameters. We did not observe label switching across the chains; see Appendix
B for further details. Simulating the $5000$ draws from the posterior required
approximately $4$ hours on a laptop with an Intel i7-7700HQ cpu and $16$GB of
ram.

###  5.1 Latent profiles description

In this Section, we describe the composition of the three considered latent
profiles (or groups) and their response pattern $\boldsymbol{\Psi}_{h}$, for
$h=1,2,3$, estimated via posterior mean and reported in Figure 2.

Figure 2: Posterior estimates for the profile-specific parameters
$\boldsymbol{\Psi}_{h}$. The color gradient of the cells varies according to
the values of the estimated probabilities, with lighter shades corresponding
to smaller values. The number in each cell is the posterior mean of
$\boldsymbol{\Psi}_{h}$ multiplied by $100$. For each item, the white text
corresponds to the response with higher posterior probability.

The first profile is composed of individuals that scrupulously followed
spread-preventive measures. Individuals in this group, with a probability of
approximately $90\%$, always wore masks outside home (ih1), washed their hands
frequently (ih2), and avoided taking public transport (ih8). With probability
of $45\%$ they avoided going out in general (ih6). Additionally, with a
probability of $84\%$ they avoided going to an hospital or health care
institution (ih7). Also, they avoided any small gathering (ih12) with
probability of $79\%$, and avoided having guests to their home (ih11) with
probability of $88\%$ . We will refer to this group as “meticulous” in the
rest of the article.

The second profile is composed of individuals that moderately followed
preventative measure. Compared to the previous group, a smaller fraction of
individuals avoided going out (ih6) ($5\%$ probability of responding
“Always”). In particular, subjects in this group completely avoided going to
shops (ih16) with probability $5\%$ (compared to the $43\%$ of the meticulous
group). A larger fraction of individuals considered small gathering as safe;
in fact, the probability of completely avoiding small gatherings (ih12) is of
$14\%$, compared with the $79\%$, of the meticulous profile. Large gatherings
(ih14) are avoided also in this group, with a probability of replying “Always”
equal to $75\%$ . Compared to the meticulous, the probability of taking public
transport (ih8) decreased from $92\%$ to $62\%$, and the probability of having
guests at their home (ih11) decreased from $88\%$ to $33\%$. Finally, the
probability of avoiding to go to an hospital (ih7) decreased from $84\%$ to
$48\%$. We will refer to this group as “moderate”.

The third profile is composed of individuals with a more lenient attitude
towards covid-19 measures. In this profile the probability of always wearing a
mask outside home (ih1) is $64\%$ compared to $94\%$ and $90\%$ for meticulous
and moderate; in particular, the probability of using a mask outside home
“Sometimes” or “Rarely” is $15\%$. The probability of responding “Always” to
“Avoided going out in general” (ih6) is $2\%$ and $49\%$ for the response “Not
at all”. This profile is characterized by more permissive behaviors also for
gatherings. For example, the probability of responding “Not at all” to items
referring to avoiding gatherings is $36\%$, $24\%$ and $17\%$ for small,
medium and large gatherings, respectively (items ih12, ih13 and ih14).
Notably, the probability of responding “Always” to “avoiding crowded places”
(ih15) is only $15\%$, compared to $56\%$ and $95\%$ of the moderate and
meticulous profile. We refer to this group as “permissive”.

To summarize, the three estimated latent profiles can be interpreted in terms
of level of compliance with covid-19 preventive measures. Some behaviors are
similar in the three profiles; for example the modal class for “Avoided going
to the hospital or other healthcare setting” (ih7) is “Always” in all the
profiles, even if the distribution of the responses is different across
groups. Other behaviors switch across profiles; for example in the item
“Avoiding going out in general” (ih6), we observe modal category “Always”,
“Sometimes” and “Not at all” for the meticulous, moderate, and permissive
group, respectively.

###  5.2 Effects of covariates

Figure 3 illustrates the effects of the covariates on the log-odds of being
assigned to the moderate or permissive group against the meticulous group,
that is used as baseline. Our empirical findings suggest that older
respondents are more likely to be in the meticulous group rather than in the
moderate or permissive ones, since the coefficient for age is negative. For
example, the odds of a subject belonging to profile $2$ (moderate) instead of
$1$ (meticolous) decrease $\exp(-0.08)=0.92$ times per each $5$ years of age,
while for profile $3$ (permissive) this estimate is $\exp(-0.159)=0.852$.
Males are more likely to be associated with the permissive groups compared to
females; the estimated odds of belonging to profile $2$ instead of $1$ are
$\exp(0.328)=1.388$ times the estimated odds for females. The odds for profile
$3$ instead of $1$ are even larger, with an estimate for males that is
$\exp(0.78)=2.18$ times the estimated odds for females. We also observe a
clear regional effect: individuals living in the south of Italy or in the
Italian Islands report higher probabilities to be associated with the
meticulous group; for example, the odds of belonging to profile $3$ instead of
$1$ for subjects in south Italy and Islands are $\exp(-0.676)=0.508$ and
$\exp(-0.556)=0.573$ times the estimated odds for North-West, respectively.
Lastly, students, retired and people that are not currently working show
higher probabilities to be associated with the meticulous group rather than
moderate or permissive ones. These results are in line with what reported in
Carlucci et al. 9 and references therein, and suggests a more cautious
behavior for younger individuals, females, living in south-Italy or Islands
and without a full-time occupation.

Figure 3: Posterior distributions of the regression coefficients
$\boldsymbol{\beta}_{h}$ representing the effect of subject characteristics on
the profile memberships.

###  5.3 Evolution of the attitude towards covid-19

Figure 4: Proportion of Italians associated with the three profiles described
in Section 5.1. Dots indicate the observed waves and grey shaded areas the
$95\%$ credible interval. Dashed lines correspond to important dates of
Italian covid-19 policy and official announcements: June 3 2020 the end of the
first lockdown; October 13 2020 the beginning of the second lockdown; December
27 2020 the European vaccine day; and, March 13 2021 the day of the
publication of the national vaccine plan.

To characterize the evolution of the attitudes towards covid-19, we rely on
the properties of the gp specification (3), providing daily predictions of the
proportion of Italians associated with the three profile described in Section
5.1. The mean parameters of the gp is approximated predicting the probability
to belong to each of the three considered profiles on a new set of locations
$\mathcal{T}^{\star}$ that is a large grid of equally spaced points between
the first and last survey wave. These predictions are mapped to the proportion
of Italians associated to each profile described in Section 5.1, at any given
time, by setting the values of the covariates in Equation (1) at their
population averages. Posterior estimates are displayed in Figure 4.

Each panel in Figure 4 is divided in five areas, separated by dashed lines
that correspond to important dates of Italian covid-19 policy and official
announcements. The first phase in Italian lockdown lasted from February 21
2020 to June 3 2020, and included several preventative measures such as
avoiding leaving the house for non-essential reasons. In the second phase
(June 3 2020–October 13 2020) shops, bars, and restaurants were open to the
public, although appropriate social distance was still required. The third
phase (October 13 2020–December 27 2020) correspond to the period of the
second Italian lockdown, after covid-19 cases increased over Europe, till the
European vaccine day (27 December 2020); this date correspond to the beginning
of vaccination policy in Europe. The fourth period spans from December 27 2020
to March 13 2021, corresponding to the publication of the Italian vaccination
plan. Finally, the fifth phase after March 13 2021 correspond to the larger
scale diffusion of the vaccine.

According to our analysis, in the interval from April 2 to May 18, about
$86\%$ of the Italian population observed a meticulous behavior, following
most of covid-19 preventive measures. In the same period, the proportion of
individuals in the moderate profile was $11\%$, and only $3\%$ for the
indulgent group. This is in agreement with what reported by Graffigna et al.
13 and Barari et al. 14, although with different methodologies and data.

On May 18 2020, the prime minister Conte introduced an easing of the lockdown
restrictions, allowing most businesses to open to the public. Commuting across
Regions was still banned until the official announcement of the second phase,
on June 3. In this period, we notice a rapid variation in the composition of
the profiles, with the proportions at June 3 corresponding to $55\%$, $36\%$,
and $9\%$ for the meticulous, moderate, and permissive group, respectively.

In the period ranging from July to October 2020 the proportion of Italians in
the three profiles is essentially constant across time, with values close to
$17\%$, $54\%$ and $29\%$, respectively. In this phase, the number of
confirmed cases increased (see Figure 5), and half of the Italian population
has a moderate attitude towards covid-19. Also, almost one third of the
population is in the permissive group. However, it is worth noting that most
of their behaviors, such as not wearing a mask outside, were allowed in this
period.

After a new lockdown was enforced on October 13 2020, we observe significant
changes in the proportions of Italians associated with each profile. For
example, the proportion of Italians in the meticulous group in the last three
observed waves of the third period (October 28, November 11, and December 16
2020) correspond to $32\%$, $48\%$, and $46\%$. For the moderate group we
observe values of $56\%$, $44\%$, and $45\%$ while the permissive drop to
$12\%$, $8\%$, and $8\%$. It is worth noting that despite the reported number
of cases of covid-19 on November 11 was comparable to the early phase of the
pandemic, only half of the population shows a meticulous attitude toward
covid-19, compared to $86\%$ of the first lockdown. The permissive group also
presents an higher proportion: $8\%$, compared to $3\%$ of the first phase.
These results suggest that, although the increase in the number of cases was
comparable between November and April 2020, the behaviors and the reactions of
the population were different, with the second phase characterized by less
meticulous behaviors than the previous lockdown. A possible explanation
involves different level of awareness on the disease compared to the first
lockdown. For example, in the first period of the pandemic, it was recommended
to clean streets and surfaces with disinfectant to avoid the infection; this
practice was later flagged as an exaggerate behavior. 56

The profiles composition stabilizes after the vaccine day on the 27 December
2020, with averages for the period December 27 2020–March 13 2021 of $50\%$,
$42\%$ and $8\%$ for the meticulous, moderates, and permissive groups,
respectively. This composition remained essentially constant until vaccine
became largely availability to the public. In the last phase (after March 13
2021) we can see how the proportion of Italians following a meticulous
behavior steeply declined, with a value of $18\%$ for the last wave.
Contemporary, the proportion of permissive increased with a value at last wave
of $32\%$. The proportion of moderates stabilized around $50\%$ in this phase.
This evolution suggests confidence in the vaccine effectiveness.

Figure 5: Daily variations in the total number of covid-19 cases in Italy in
log-scale. Dashed lines correspond to important dates of Italian covid-19
policy and official announcements: June 3 2020 the end of the first lockdown;
October 13 2020 the beginning of the second lockdown; December 27 2020 the
European vaccine day; and, March 13 2021 the day of the publication of the
national vaccine plan. The data used for the plot can be downloaded at
https://lab24.ilsole24ore.com/coronavirus/.

##  6 Discussion

We analyzed Italian attitude towards covid-19 leveraging a dynamic Bayesian
latent class regression model for survey data. The estimated latent profiles
can be interpreted as different degrees of precaution that determine the
compliance with the national rules. At the population level, the proportion of
Italians associated with each of these profiles follows the various phases of
the lockdown. This suggests that Italians have followed the national rules.

There are several potential future directions for this work. When more recent
data will be released, it would be interesting to analyze how the composition
of the profiles changes in relation to the diffusion of new covid-19 variants,
such as Omicron. A further research direction would be to simultaneously study
the evolution of attitude towards covid-19 in several countries, comparing
different nations and highlighting the main differences. However, such an
extension is not immediate, since the interpretation of the profiles in
different country might be substantially different and highly influenced by
cultural aspects.

## Appendix

##  Appendix A Model selection

The number of latent profiles $H$ is selected via $4$-folds cross validations.
Specifically, for each wave $t\in\mathcal{T}$ we divided data into $4$ folds
of equal size, using in turn $3$ folds for estimating the model and the
remaining one for evaluating its performance. Since subjects are different
across each wave, folds can be constructed with random sampling, independently
across waves. We simulate $4000$ samples from the posterior after a burn-in of
$1000$, letting the number of groups $H\in\\{2,\dots,8\\}$ for each cross-
validation fold. Predictive probabilities for each individual and modalities
are computed as in Equation (1), using Monte-Carlo integration, while the
final out-of-sample predictions have been obtained selecting the modality with
largest probability. We compare observed and predicted values in terms of
accuracy, averaging the results in the four cross-validation folds.

Results are reported in Table 3, divided for the $14$ response items. Accuracy
increases moving from a model with $H=2$ latent profiles to a model with $H=3$
profiles for most items ($10$ out of $14$), while the remaining items are
predicted with the same accuracy by the two models. The prediction accuracy
stabilizes with the model with $H=3$ and does not increase further with $H$.
Therefore, model with $H=3$ is the most parsimonious model with best fit to
the data, and should be selected.

Table 3: Accuracy, multiplied by $100$. Largest values are highlighted in
bold-face, and correspond to the best model. In case of multiple maxima, the
smallest model is preferred and the corresponding accuracy is highlighted in
bold-face.

Item label $H=2$ $H=3$ $H=4$ $H=5$ $H=6$ $H=7$ $H=8$ ih1 78 78 78 78 78 78 78
ih2 61 62 62 62 62 62 62 ih3 57 58 58 58 58 58 58 ih4 67 67 67 67 67 67 67 ih5
65 66 66 66 66 66 66 ih6 56 57 57 57 57 57 57 ih7 58 59 59 59 59 59 59 ih8 78
78 78 78 78 78 78 ih11 61 62 62 62 62 62 62 ih12 57 58 58 58 58 58 58 ih13 67
67 67 67 67 67 67 ih14 65 66 66 66 66 66 66 ih15 56 57 57 57 57 57 57 ih16 59
59 59 59 59 59 59

##  Appendix B Convergence diagnostic

Figure 6 shows traceplots of the marginal posterior distributions for some
parameters of the model described in Section 4. The chains exhibit good
mixing, showing no jumps between profiles, that would indicate label
switching; traceplots for the remaining model parameters behave similarly.

Figure 6: Traceplots of the posterior distribution of some illustrative
parameters. Light gray lines denote the cumulative means of the parameters at
each iteration.

## References

  * 1 World Health Organization (WHO) . Coronavirus disease (covid-19). 2021.  https://covid19.who.int/, checked on 28/12/2021.
  * 2 Papadimitriou E, Cseres-Gergelyne Blasko Z. Economic sectors at risk due to COVID-19 disruptions: will men and women in the EU be affected similarly?. EUR 30327 EN 2020\.
  * 3 Cullen W, Gulati G, Kelly B. Mental health in the Covid-19 pandemic. QJM: An International Journal of Medicine 2020; 113(5): 311–312.
  * 4 Balkhi F, Nasir A, Zehra A, Riaz R. Psychological and behavioral response to the coronavirus (COVID-19) pandemic. Cureus 2020; 12(5).
  * 5 American Psychological Association (APA) . Patients with Depression and Anxiety Surge as Psychologists Respond to the Coronavirus Pandemic. 2020.  Checked on 23/02/2021.
  * 6 Forte G, Favieri F, Tambelli R, Casagrande M. The Enemy Which Sealed the World: Effects of COVID-19 Diffusion on the Psychological State of the Italian Population. Journal of Clinical Medicine 2020; 9(6).
  * 7 Moccia L, Janiri D, Pepe M, et al. Affective temperament, attachment style, and the psychological impact of the COVID-19 outbreak: an early report on the Italian general population. Brain, Behavior, and Immunity 2020; 87: 75-79.
  * 8 Motta Zanin G, Gentile E, Parisi A, Spasiano D. A preliminary evaluation of the public risk perception related to the COVID-19 health emergency in Italy. International journal of environmental research and public health 2020; 17(9): 3024.
  * 9 Carlucci L, D’Ambrosio I, Balsamo M. Demographic and Attitudinal Factors of Adherence to Quarantine Guidelines During COVID-19: The Italian Model. Frontiers in Psychology 2020; 11: 2702.
  * 10 Google llc . Google COVID-19 Community Mobility Reports. 2021.  https://www.google.com/covid19/mobility/, accessed on 17/02/2021.
  * 11 Bavadekar S, Dai A, Davis J, et al. Google COVID-19 Search Trends Symptoms Dataset: Anonymization Process Description (version 1.0). arXiv preprint arXiv:2009.01265 2020\.
  * 12 Wolff W, Martarelli CS, Schüler J, Bieleke M. High Boredom Proneness and Low Trait Self-Control Impair Adherence to Social Distancing Guidelines during the COVID-19 Pandemic. International Journal of Environmental Research and Public Health 2020; 17(15).
  * 13 Graffigna G, Barello S, Savarese M, et al. Measuring Italian citizens’ engagement in the first wave of the COVID-19 pandemic containment measures: A cross-sectional study. PLOS ONE 2020; 15(9): 1-22.
  * 14 Barari S, Caria S, Davola A, et al. Evaluating COVID-19 public health messaging in Italy: Self-reported compliance and growing mental health concerns. MedRxiv 2020\.
  * 15 Jones SP. Imperial College London YouGov Covid Data Hub, v1. 0. Imperial College London Big Data Analytical Unit and YouGov Plc 2020\.
  * 16 Lazarsfeld PF. The logical and mathematical foundation of latent structure analysis. Studies in Social Psychology in World War II Vol. IV: Measurement and Prediction 1950: 362–412.
  * 17 Dayton CM, Macready GB. Concomitant-variable latent-class models. Journal of the American statistical association 1988; 83(401): 173–178.
  * 18 Vermunt JK, Magidson J. Latent class analysis with sampling weights: A maximum-likelihood approach. Sociological methods & research 2007; 36(1): 87–111.
  * 19 Patterson BH, Dayton CM, Graubard BI. Latent class analysis of complex sample survey data: application to dietary data. Journal of the American Statistical Association 2002; 97(459): 721–741.
  * 20 Kunihama T, Dunson DB. Bayesian modeling of temporal dependence in large sparse contingency tables. Journal of the American Statistical Association 2013; 108(504): 1324–1338.
  * 21 Russo M, Durante D, Scarpa B. Bayesian inference on group differences in multivariate categorical data. Computational Statistics & Data Analysis 2018; 126: 136 - 149.
  * 22 Fruhwirth-Schnatter S, Celeux G, Robert CP. Handbook of mixture analysis.  CRC press .  2019\.
  * 23 Girardi P, Greco L, Mameli V, et al. Robust inference for non-linear regression models from the Tsallis score: Application to coronavirus disease 2019 contagion in Italy. Stat 2020; 9(1): e309.  e309 sta4.309.
  * 24 Alaimo Di Loro P, Divino F, Farcomeni A, et al. Nowcasting COVID-19 incidence indicators during the Italian first outbreak. Statistics in Medicine 2021; 40(16): 3843-3864.
  * 25 Girardi P, Greco L, Ventura L. Misspecified modeling of subsequent waves during COVID-19 outbreak: A change-point growth model. Biometrical Journal 2022; 64(3): 523-538.
  * 26 Scrucca L. A COVINDEX based on a GAM beta regression model with an application to the COVID-19 pandemic in Italy. Statistical Methods & Applications 2022: 1–20.
  * 27 Sebastiani G, Massa M, Riboli E. Covid-19 epidemic in Italy: evolution, projections and impact of government measures. European journal of epidemiology 2020; 35(4): 341–345.
  * 28 Farcomeni A, Maruotti A, Divino F, Jona-Lasinio G, Lovison G. An ensemble approach to short-term forecast of COVID-19 intensive care occupancy in Italian regions. Biometrical Journal 2021; 63(3): 503-513.
  * 29 Celani A, Giudici P. Endemic–epidemic models to understand COVID-19 spatio-temporal evolution. Spatial Statistics 2021: 100528.
  * 30 D’Urso P, De Giovanni L, Vitale V. A D-vine copula-based quantile regression model with spatial dependence for COVID-19 infection rate in Italy. Spatial statistics 2022: 100586.
  * 31 Cabras S. A Bayesian-Deep Learning Model for Estimating COVID-19 Evolution in Spain. Mathematics 2021; 9(22).
  * 32 Padellini T, Jersakova R, Diggle PJ, et al. Time varying association between deprivation, ethnicity and SARS-CoV-2 infections in England: A population-based ecological study. The Lancet Regional Health - Europe 2022; 15: 100322.  doi: https://doi.org/10.1016/j.lanepe.2022.100322
  * 33 Selinger C, Choisy M, Alizon S. Predicting COVID-19 incidence in French hospitals using human contact network analytics. International Journal of Infectious Diseases 2021; 111: 100-107.
  * 34 ihme covid-19 forecasting team . Modeling COVID-19 scenarios for the United States. Nature medicine 2020\.
  * 35 Reddy T, Shkedy Z, Rensburg J. vC, et al. Short-term real-time prediction of total number of reported COVID-19 cases and deaths in South Africa: a data driven approach. BMC medical research methodology 2021; 21(1): 1–11.
  * 36 Roosa K, Lee Y, Luo R, et al. Real-time forecasts of the COVID-19 epidemic in China from February 5th to February 24th, 2020. Infectious Disease Modelling 2020; 5: 256–263.
  * 37 Daoust JF, Belanger E, Dassonneville R, Lachapelle E, Nadeau R. Two Solitudes or All in the Same Boat? Citizens’ Levels of Compliance with COVID-19 Preventive Measures in Quebec and the Rest of Canada. Citizens’ Levels of Compliance with COVID-19 Preventive Measures in Quebec and the Rest of Canada (December 4, 2020) 2020\.
  * 38 Krekel C, Swanke S, De Neve JE, Fancourt D. Are happier people more compliant? Global evidence from three large-scale surveys during Covid-19 lockdowns. 2020\.
  * 39 Tripathy D, Camorlinga SG. Prediction of COVID-19 Cases based on Human Behavior using DNN Regressor for Canada. In: ; 2021: 1-6
  * 40 Duradoni M, Fiorenza M, Guazzini A. When Italians Follow the Rules against COVID Infection: A Psychological Profile for Compliance. COVID 2021; 1(1): 246–262.
  * 41 Guazzini A, Pesce A, Marotta L, Duradoni M. Through the Second Wave: Analysis of the Psychological and Perceptive Changes in the Italian Population during the COVID-19 Pandemic. International Journal of Environmental Research and Public Health 2022; 19(3).
  * 42 Verbeek M. Pseudo-panels and repeated cross-sections. In: Springer. 2008 (pp. 369–383).
  * 43 Bartolucci F, Farcomeni A, Pennoni F. Latent Markov models for longitudinal data.  CRC Press .  2012\.
  * 44 Linzer DA, Lewis JB, others . poLCA: An R package for polytomous variable latent class analysis. Journal of statistical software 2011; 42(10): 1–29.
  * 45 Azzalini A, Scarpa B. Data analysis and data mining: An introduction.  OUP USA .  2012\.
  * 46 Godambe VP, Thompson ME. Parameters of Superpopulation and Survey Population: Their Relationships and Estimation. International Statistical Review / Revue Internationale de Statistique 1986; 54(2): 127–138.
  * 47 Rabe-Hesketh S, Skrondal A. Multilevel modelling of complex survey data. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2006; 169(4): 805-827.
  * 48 Skinner C, Mason B. Weighting in the regression analysis of survey data with a cross-national application. Canadian Journal of Statistics 2012; 40(4): 697-711.
  * 49 Gunawan D, Panagiotelis A, Griffiths W, Chotikapanich D. Bayesian weighted inference from surveys. Australian & New Zealand Journal of Statistics 2020; 62(1): 71-94.
  * 50 Si Y, Pillai NS, Gelman A. Bayesian Nonparametric Weighted Sampling Inference. Bayesian Analysis 2015; 10(3): 605 – 625.
  * 51 Gao Y, Kennedy L, Simpson D, Gelman A. Improving Multilevel Regression and Poststratification with Structured Priors. Bayesian Analysis 2021; -1(-1): 1 – 26.
  * 52 Durante D, Canale A, Rigon T. A nested expectation–maximization algorithm for latent class models with covariates. Statistics & Probability Letters 2019; 146(C): 97-103.
  * 53 Rasmussen C, Williams C. Gaussian Processes for Machine Learning.  Adaptive Computation and Machine LearningCambridge, MA, USA: MIT Press .  2006\.
  * 54 Tsay RS. Analysis of financial time series.  John wiley & sons .  2005\.
  * 55 Stan Development Team . RStan: the R interface to Stan. 2020.  R package version 2.21.2.
  * 56 Goldman E. Exaggerated risk of transmission of COVID-19 by fomites. The Lancet Infectious Diseases 2020; 20(8): 892–893.


# Music Source Separation with Band-Split RoPE Transformer

###### Abstract

Music source separation (MSS) aims to separate a music recording into multiple
musically distinct stems, such as vocals, bass, drums, and more. Recently,
deep learning approaches such as convolutional neural networks (CNNs) and
recurrent neural networks (RNNs) have been used, but the improvement is still
limited. In this paper, we propose a novel frequency-domain approach based on
a Band-Split RoPE Transformer (called BS-RoFormer). BS-RoFormer relies on a
band-split module to project the input complex spectrogram into subband-level
representations, and then arranges a stack of hierarchical Transformers to
model the inner-band as well as inter-band sequences for multi-band mask
estimation. To facilitate training the model for MSS, we propose to use the
Rotary Position Embedding (RoPE). The BS-RoFormer system trained on MUSDB18HQ
and 500 extra songs ranked the first place in the MSS track of Sound Demixing
Challenge (SDX’23). Benchmarking a smaller version of BS-RoFormer on
MUSDB18HQ, we achieve state-of-the-art result without extra training data,
with 9.80 dB of average SDR.

Index Terms—  Music source separation, band-split, rotary position embedding,
Transformer, BS-RoFormer, SDX’23.

##  1 Introduction

Music source separation (MSS) [1, 2] is a task of separating a music recording
into musically distinct sources. As defined in the 2015 Signal Separation
Evaluation Campaign (SiSEC) [3], researchers are focused on the 4-stem
setting: vocals, bass, drums, and other. The MUSDB18 dataset [4] has been used
to benchmark the performance. MSS is considered to be a more challenging audio
separation task given the complexity of input signals (i.e., stereo, 44.1k Hz
sampling rate) and more sources to separate (i.e., 4 or more stems). MSS can
benefit various downstream MIR tasks, such as vocal pitch estimation [5],
music transcription [6], and so on. MSS may also enable applications such as
karaoke-version generation, intelligent music editing and remixing [7], and
other music inspired derivative works.

In recent years, many deep learning approaches have been proposed to tackle
the MSS problem. They are typically categorized into frequency-domain and
time-domain approaches. Frequency-domain approaches reply on Fourier transform
to derive a time-frequency (T-F) representation for input. Then, models such
as fully connected neural networks [8], convolutional neural networks (CNNs)
[9, 10, 11], and recurrent neural networks (RNNs) [12] are applied. On the
other hand, time-domain approaches such as Wave-U-Net [13], ConvTasNet [14],
and Demucs [15] build their neural networks directly on the waveform input.
Recently, Hybrid Transformer Demucs (HTDemucs) [16] proposes to use a cross-
domain Transformer to combine the frequency- and time-domain models.

Speaking of frequency-domain approaches, most existing models do not make
assumptions on weighting the frequency bins based on prior knowledge. Models
like CNNs use the same kernels across all frequency bins for all target
sources, expecting the models can learn the band-pass mechanism from the data
with raw frequency bins. However, this might not be efficient, since different
frequency bands may have different patterns that preferably characterize
different instrument sources. Recently, band-split mechanism is introduced at
the front-end in Band-split RNN (BSRNN) [17] to force the model to learn band-
wise features. BSRNN splits the T-F representation into non-overlapping
subbands and arranges a stack of interleaved RNNs to simultaneously model the
inner-band and inter-band sequences. This design has demonstrated state-of-
the-art result on Musdb18. However, its use of RNN could be still sub-optimal,
given that Transformer [18] has constantly shown the superiority in many
relevant tasks of sequential data modeling.

The idea of modeling the T-F representation of music audio using interleaved
Transformers has been explored recently [19, 20]. For example, SpecTNT [19]
consists of two Transformer encoders in a hierarchical order to model the
frequency and time sequences respectively, and then uses the frequency class
tokens to connect between the two Transformers. This or similar ideas have
shown state-of-the-art performance in various music transcription related
tasks [20, 21, 22], but have not been studied for MSS. In this paper, we
propose a novel MSS approach based on Band-Split RoPE Transformer (termed as
_BS-RoFormer_) that originates from combining the ideas of band-split and
hierarchical Transformer architecture. Making it work, however, is non-
trivial, mainly because the model is large, and training it effectively is
very memory and time consuming. We introduce the use of Rotary Position
Embedding (RoPE) [23] in Transformer significantly improves the performance.
Other efforts such as checkpointing, mixed precision, and flash attention [24]
could also help the training efficiency. We submitted the BS-RoFormer system
to Sound Demixing Challenge 2023
(SDX’23)111https://www.aicrowd.com/challenges/sound-demixing-challenge-2023/,
the Music Separation track. Our system ranked the first place and outperformed
the second best by a large margin in SDR. In ablation study, we demonstrate
the importance of RoPE and that a smaller BS-RoFormer model trained solely on
MUSDB18HQ can achieve state-of-the-art performance compared to existing
models.

Fig. 1: The framework of the Band-Split RoFormer system.

##  2 System Overview

Fig. 1 depicts the system. Let $x\in\mathbb{R}^{C\times L}$ denote the audio
waveform of the input mixture, where $C$ and $L$ are the numbers of channels
and audio samples, respectively. Our frequency-domain MSS system uses the
complex spectrogram as input, where $x$ is transformed into a time-frequency
(T-F) representation $X\in\mathbb{C}^{C\times T\times F}$ by a short-time
Fourier transform (STFT), where $T$ and $F$ are the the numbers of frames and
frequency bins, respectively. Let $f_{\theta}$ denote a T-F neural network
with a set of learnable parameters $\theta$, the output of $f_{\theta}$ can be
linear spectrums, ideal binary masks (IBMs), ideal ratio mask (IRMs), or
complex IRMs (cIRMs) [25]. We adopt the cIRMs, denoted as
$\hat{M}\in\mathbb{C}^{C\times T\times F}$. In this work, the goal of
$f_{\theta}$ is to estimate the cIRMs: $\hat{M}=f_{\theta}(X)$, so that both
the magnitude and phase signals of the target source are attained.

Then, the separated complex spectrogram $\hat{Y}\in\mathbb{C}^{C\times T\times
F}$ is obtained by multiplying the cIRM by the input complex spectrogram:
$\hat{Y}=\hat{M}\odot X$. Finally, an inverse STFT (iSTFT) is applied to
$\hat{Y}$ to recover the separated signal $\hat{y}$ in the time-domain. Mean
absolute error (MAE) loss between the reference $y$ and output $\hat{y}$ is
used to train $f_{\theta}$. Specifically, the objective loss includes both the
time-domain MAE and the multi-resolution complex spectrogram MAE [26]:

$\text{loss}=||y-\hat{y}||+\sum_{s=0}^{S-1}||Y^{(s)}-\hat{Y}^{(s)}||,$ (1)

where $S=5$ multi-resolution STFTs are used with the window sizes of
$[4096,2048,1024,512,256]$ and a fixed hop size of $147$, which is equivalent
to 300 frames per second.

##  3 Band-Split RoPE Transformer

BS-RoFormer consists of a band-split module, RoPE Transformer blocks, and a
multi-band mask estimation module.

###  3.1 Band-Split Module

In BSRNN [17], it has shown that the band-split strategy can help the
frequency-domain approach. One rationale is that the input mixture audio $X$
is a full-band signal, so splitting the bands based on a prior setting can
foster the model to purify the learned representations at different bands,
gaining the robustness against cross-band vagueness. Following [17], we split
$X$ into $N$ uneven non-overlapping subbands along the frequency axis and
apply individual multi-layer perceptions (MLPs) to each subband. We denote the
output of each subband as $X_{n}\in\mathbb{C}^{C\times T\times F_{n}}$, where
$F_{n}$ is the number of frequency bins in the $n$-th subband. All subbands
$X_{n}$ constitute the entire complex spectrum $X$, and there is
$\sum_{n=1}^{N}F_{n}=F$.

Each MLP consists of a RMSNorm layer [27] followed by a linear layer. The
RMSNorm layer regularizes the summed inputs to a neuron in one layer according
to root mean square. RMSNorm is an efficient replacement to the LayerNorm
normalization [27]. The $n$-th linear layer contains a learnable matrix with a
shape of $(C\times F_{n})\times D$ and a learnable bias with a shape of $D$,
where $D$ is the number of features. The transformed output of each subband is
denoted as $H_{n}^{0}$ with a shape of $T\times D$. We stack all
$H_{n}^{0},n=1,\dots,N$, along the subband axis to obtain a stacked $H^{0}$
with a shape of $T\times N\times D$ as the input to the RoPE Transformer
blocks.

###  3.2 RoPE Transformer Blocks

Suppose there are $L$ blocks in RoPE Transformer blocks, each block’s output
is denoted as $H_{n}^{l}\in\mathbb{R}^{T\times N\times D}$. The band-split
output $H_{n}^{0}$ serves as the input for the 1-st Transformer block.
Different from conventional Transformer encoder that applies self-attention to
one-dimensional sequence (e.g., time), our Transformer structure is
hierarchical, where interleaved Transformer layers are in turn applied to the
time and frequency axes in a Transformer block. The former Transformer, called
_time-Transformer_ , models the inner-band (local) temporal sequence, while
the latter Transformer, called _subband-Transformer_ , handles the inter-band
(global) spectral sequence to ensure the information across bands is
exchangeable. The process of a Transformer block is presented in Algorithm 1.
For time-Transformer, all mini-batches (with a batch size of $B$) and subbands
are stacked; for subband-Transformer, all mini-batches and frames are stacked.
All Transformer layers use the same architecture. Fig. 2 depicts a Transformer
layer, which consists of an attention module (with rotary position embedding)
and a feedforward module.

Algorithm 1 Hierarchical Transformer for a time-frequency input, with
parameters $B$: batch size, $T$: frames number, $N$: subband number, and $D$:
latent dimension.

1:Input: $H^{l}$ with a shape of $B\times T\times N\times D$

2:Permute $H^{l}$ to $(B\times N)\times T\times D$

3:Apply a Transformer along $T$.

4:Rearrange the above output to $(B\times T)\times N\times D$

5:Apply a Transformer along $N$.

6:Output: Permute the above output to $H^{l+1}$ with a shape of $B\times
T\times N\times D$.

####  3.2.1 Attention Module with Rotary Position Embedding

In an attention module, we first rearrange the input to a shape as needed (see
Algorithm 1), followed by a RMSNorm [27]. Then, we apply the query, key, and
value layers to predict the query, key, and value [18]: $Q=H^{l}W_{q}$,
$K=H^{l}W_{k}$, and $V=H^{l}W_{v}$, where $Q$, $K$, and $V$ have the shapes of
$D\times Z$, and $Z$ is the latent dimension. Learnable matrices $W_{q}$,
$W_{k}$, and $W_{v}$ have the shapes of $D\times Z$. The $Q$, $K$, and $V$ are
split into multiple heads. For positional encoding, we propose to use the
rotary position embedding (RoPE) [23], which are applied as:

$\begin{split}&\hat{Q}=\text{Rot}(Q)\\\ &\hat{K}=\text{Rot}(K)\end{split}$ (2)

where $\text{Rot}(\cdot)$ is the RoPE encoder [23] shared throughout the
entire Transformer blocks. The time-Transformer and subband-Transformer have
their own RoPE encoders that apply rotation matrices on each embedding in $Q$
and $K$ based on its position of the corresponding sequence. Then, we apply
the attention operation by using the processed query, key, and value as
follows [18]:

$\text{output}=Dropout\left(Softmax(\frac{\hat{Q}\hat{K}^{T}}{\sqrt{Z/h}})\right)V,$
(3)

where $h$ is the number of heads. Equation (3) is the core part that requires
the most computation. To speed up, we employ the FlashAttention technique
[24]. After the attention module, a fully-connected layer with Dropout is
used. A residual addition is applied between the input and output of the
attention module.

We believe the use of RoPE is crucial in this work. Given the proposed
hierarchical Transformer structure, the positional embeddings for the time-
and subband-Transformers should be robust to the alternating self-attention
operations between the time and subband axes. Directly adding learnable
absolute positional embeddings to encode the positions may fail to maintain
the scale of norm after repetitive transposed self-attention processes. Since
RoPE encodes the relative position by multiplying the context representations
with a rotation matrix, we argue this would help preserve the positional
information for both time and subband sequences.

####  3.2.2 Feedforward Module

The feedforward module consists of a RMSNorm layer, a fully connected layer
with GeLU activation [28], and a dropout. Then, one more fully connected layer
with dropout is applied. Similarly, we apply a residual addition between the
input and output of the feedforward module.

Fig. 2: Diagram of a Transformer layer. This example is a time-Transformer
along the temporal sequence and $B^{\prime}=B\times N$.

###  3.3 Multi-band Mask Estimation Module

We denote the outputs from the RoPE Transformer blocks as
$H_{n}^{L},n=1,\dots,N$. Similar to the band-split module, we apply $N$
individual MLP layers to each subband $H_{n}^{L}$. Each MLP layer consists of
a RMSNorm layer, a fully connected layer followed by a Tanh activation, and a
fully connected layer followed by a gated linear unit (GLU) layer [29]. The
$n$-th MLP layer outputs the subband mask $\hat{M}_{n}$ with a shape of
$(2\times C)\times T\times F_{n}$, which contains a mask of real values and a
mask of imaginary values. All the outputs are concatenated along the frequency
axis to derive a cIRM $\hat{M}\in\mathbb{C}^{C\times T\times F}$.

##  4 Experiments

This section first presents our implementation details of the winning system
in Sound Demixing Challenge 2023 (SDX’23). Then, we describe the ablation
study on the a smaller version of BS-RoFormer and the effectiveness of Rotary
position embedding.

###  4.1 Dataset

The Musdb18HQ dataset [4] contains 100 and 50 songs for training and
evaluation, respectively. All recordings are stereo with a sampling rate of
44.1k Hz. Each recording contains four stems: vocals, bass, drums, and other.
For MDX’23 submission, we also use an “In-House” dataset containing 500 songs:
450 songs are added to the training set, and 50 songs are for validation. Each
song has four stems with a sampling rate of 44.1k Hz following [4].

###  4.2 Evaluation Metrics

We adopt the signal-to-distortion ratio (SDR) [30] implemented by museval [31]
as the evaluation. The SDR score is calculated by:

$SDR(y,\hat{y})=10\log_{10}\frac{||y||^{2}}{||\hat{s}-s||^{2}}$ (4)

Higher SDR indicates better separation quality.

###  4.3 Data Augmentation

Our model takes a segment of 8-seconds waveform for input and output. For
better training efficiency, we maintain a dynamic pool of 8-seconds stem-level
segments in the memory. At each training step, the dataloader samples a batch
of random 4-stem tuples from the pool. Each stem audio is processed with a
random gain in a range of $\pm 3$ dB and has a chance of 10% to be replaced
with a silence waveform. The four stems of a tuple, which can originate from
different songs, are mixed by linear addition to generate a training example.
This random-mixing strategy will produce examples that are not musically
aligned.

The dynamic pool has a size (e.g., 512 segments for each stem) larger than the
batch size to facilitate wider diversity for a batch, and is updated in a
first-in-first-out manner at each training step to remain its size. To add
samples to the pool, we crop an 8-seconds segment of 4 stems at a random
moment from a full-length song. A stem-level segment must satisfy a loudness
level larger than -50 dB.

###  4.4 Implementation Details

Band-split Module. We apply a Hann window size of 2048 and a hop size of 10 ms
for STFT to compute the complex spectrogram of 8-seconds long input. We use
the following band-split scheme: 2 bins per band for frequencies under 1000
Hz, 4 bins per band between 1000 Hz and 2000 Hz, 12 bins per band between 2000
Hz and 4000 Hz, 24 bins per band between 4000 Hz and 8000 Hz, 48 bins per band
between 8000 Hz and 16000 Hz, and the remaining bins beyond 16000 Hz equally
divided into two bands. This results in a total of 62 bands. All bands are
non-overlapping. We derive this setting referring to [17]. In our pilot study,
slightly varying the band-slplit setting does not show significant difference
in results.

Configuration. We use $D$ = 384 for feature dimension, $L$ = 12 for the number
of Transformer blocks, 8 heads for each Transformer, and a dropout rate of
0.1. The multi-band mask estimation module utilizes MLPs with a hidden layer
dimension of $4D$. The resulting model has 93.4M parameters. We use Pytorch-
Lightning 2.0 and Exponential Moving Averaging (EMA) with a decay 0.999. For
training, we adopt the AdamW optimizer [32] with a learning rate (LR) $5\times
10^{-4}$, and reduce LR by 0.9 every 40k steps. To optimize the GPU memory
usage, we employ the checkpointing technique as well as the mixed precision,
where the STFT and iSTFT modules use FP32 and all the others use FP16.

We trained three separation models respectively for vocals, bass, and drums
using In-House and the Musdb18HQ training set. For the “other” stem, we
subtracted the vocals, bass, and drums signals from the input mixture in the
time domain. For each model, the training process lasted for 4 weeks using 16
Nvidia A100-80GB GPUs with a total batch size of 128 (i.e., 8 for each GPU).
The model checkpoint with the best validation result was selected.

Enframe & Deframe. We use a hop size of 4 seconds for segment enframing. Two
deframing methods are studied: “truncate&concat” (TC) and “overlap&average”
(OA). In TC, we discard the front and rear 2-seconds signals of each segment-
level output and concatenate all truncated segments. In OA, each output
segment has a 4-second overlap by the next segment at the rear, and the
overlapped part is averaged.

The $\star$ symbol indicates the proposed BS-RoFormer.  System Vocals Bass
Drums Other Mean 1\. SAMI-ByteDance⋆ 11.36 11.15 10.27 7.08 9.97 2\. ZFTurbo
[33] 10.51 9.94 9.53 7.05 9.26 3\. kimberley_jensen 10.40 10.06 9.47 6.80 9.18
4\. kuielab [34] 10.01 9.72 9.43 6.72 8.97 5\. alina_porechina 9.07 9.92 9.29
6.23 8.63 (Baseline) BSRNN [17] 7.98 5.63 6.53 4.43 6.14

Table 1: SDX’23 Leaderboard C final results (in global SDR).

###  4.5 SDX’23 MSS Results

SDX’23 MSS track featured three leaderboards, A: label noise, B: bleeding, and
C: standard music separation. The first two leaderboards set some requirements
about robustness, so the submitted systems can be trained only on the provided
dataset. Our system participated the standard music separation track, where
the submitted systems can be trained on any data with no limitation, so it is
more desirable. Table 1 presents the final results on the organizer’s private
set for the top-5 systems of SDX’23 MSS leaderboard C [35]. We used TC
deframing for the submission.

Our system (SAMI-ByteDance) outperforms the second best (ZFTurbo) by a large
margin (0.71 dB on average), demonstrating the effectiveness of the proposed
BF-RoFormer. When comparing the results of vocals, bass, and drums (since we
did not train the “other” separator), the average SDR difference is 0.93 dB.
In terms of listening experience, we found our model outputs are highly
accurate, meaning they contain less residues from the background. However, the
resulting spectrogram may look sharp and less foggy compared to that generated
by existing CNN-based models, and this might not be a favorable feature to
some audiences. According to [35], our model outputs gained more preference
from musicians and educators than from music producers in the listening test
of SDX23.

The ${\dagger}$ symbol indicates MSS systems trained with extra data.
The $\ddagger$ symbol indicates the results are evaluated on non-HQ version.
Vocals Bass Drums Other Avg. Conv-TasNet [14]‡ 6.81 5.66 6.08 4.37 5.73
Spleeter [36]†‡ 6.86 5.51 6.71 4.55 5.91 ResUNet [10]‡ 8.98 6.04 6.62 5.29
6.73 HDemucs [37]† 8.13 8.76 8.24 5.59 7.68 BSRNN [17] 10.01 7.22 9.01 6.70
8.24 BSRNN [17]† 10.47 8.16 10.15 7.08 8.97 Sparse HT Demucs [16]† 9.37 10.47
10.83 6.41 9.27 BS-Transformer ($L$=6, OA) 9.15 6.11 3.08 4.77 5.78 BS-
RoFormer ($L$=6, TC) 10.68 11.28 9.41 7.68 9.76 BS-RoFormer ($L$=6, OA) 10.66
11.31 9.49 7.73 9.80 BS-RoFormer ($L$=12, OA)† 12.72 13.32 12.91 9.01 11.99

Table 2: Median SDRs of different MSS models.

###  4.6 Ablation Study

We investigate the effects of three aspects: 1) the number of Transformer
blocks; 2) RoPE or absolute positional encoding; 3) TC or OA deframing
methods. To this end, we implement three smaller variants of the proposed
model with $L$=6. In one variant, we remove the RoPE and add learnable
absolute positional embeddings in the attention module, and call it “BS-
Transformer,” since it uses a standard Transformer. We train a dedicated
separation model for the “other” stem, except “BS-RoFormer ($L$=12, OA),”
which is our MDX’23 submission. The numbers of parameters for BS-RoFormer and
BS-Transformer with $L$=6 are 72.2M and 72.5M, respectively. Models with $L$=6
are trained solely on the Musdb18HQ training set using 16 Nvidia V100-32GB
GPUs. We do not use the In-House dataset for ablation study. The effective
batch size is 64 (i.e., 4 for each GPU) using accumulate_grad_batches=2.

Table 2 presents the comparison between our proposed models and existing
models. We report the median SDR across the median SDRs over all 1 second
chunks of each test song in Musdb18HQ following prior works. First, BS-
RoFormer with $L$=6 is still very competitive and can achieve state-of-the-art
performance compared to models trained without extra training data. “BS-
RoFormer† ($L$=12, OA)” outperforms all existing models by a large margin
(over 2 dB on average). Second, BS-Transformer without RoPE does not seem to
work given the low SDRs, demonstrating that RoPE is crucial in our proposed
architecture as discussed in Section 3.2.1. According to our observations, the
training progress of BS-Transformer is very slow, and it still remains low
SDRs after two weeks of training on Musdb18HQ. Instead, BS-RoFormer models
with $L$=6 get converged within a week. Lastly, the OA deframing shows better
performance than TC except vocals. Qualitatively, OA offers smoother song-
level quality that can improve the overall listening experience.

##  5 Conclusion

We have presented the BS-RoFormer model, which is based on a novel
hierarchical RoPE Transformer architecture. Its outstanding performance may
shed the light on the development of next generation MSS systems. For future
work, we would focus on improving the qualitative performance. The sharp sound
quality may be improved by introducing overlapping band projection at the
front-end.

## References

  * [1] Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, D. FitzGerald, and B. Pardo,  “An overview of lead and accompaniment separation in music,”  IEEE/ACM Trans. Audio Speech Lang. Process., vol. 26, no. 8, pp. 1307–1335, 2018.
  * [2] Y. Mitsufuji, G. Fabbro, S. Uhlich, F.-R. Stöter, A. Défossez, M. Kim, W. Choi, C.-Y. Yu, and K.-W. Cheuk,  “Music demixing challenge 2021,”  Frontiers in Signal Processing, 2022.
  * [3] A. Liutkus, F.-R. Stöter, Z. Rafii, D. Kitamura, B. Rivet, N. Ito, N. Ono, and J. Fontecave,  “The 2016 signal separation evaluation campaign,”  in 13th International Conference on Latent Variable Analysis and Signal Separation, 2017, pp. 323–332.
  * [4] Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner,  “The MUSDB18 corpus for music separation,” Dec. 2017,  https://doi.org/10.5281/zenodo.1117372.
  * [5] T. Nakano, K. Yoshii, Y. Wu, R. Nishikimi, K. W. E. Lin, and M. Goto,  “Joint singing pitch estimation and voice separation based on a neural harmonic structure renderer,”  in IEEE WASPAA, 2019, pp. 160–164.
  * [6] L. Lin, Q. Kong, J. Jiang, and G. Xia,  “A unified model for zero-shot music source separation, transcription and synthesis,”  in ISMIR, 2021.
  * [7] M. Moon,  “TikTok-owner ByteDance debuts Ripple music creation app,” https://www.engadget.com/tiktok-owner-bytedance-debuts-ripple-music-creation-app-130023602.html.
  * [8] E. M. Grais, M. U. Sen, and H. Erdogan,  “Deep neural networks for single channel source separation,”  in IEEE ICASSP, 2014, pp. 3734–3738.
  * [9] P. Chandna, M. Miron, J. Janer, and E. Gómez,  “Monoaural audio source separation using deep convolutional neural networks,”  in Latent Variable Analysis and Signal Separation (LVA/ICA), 2017, pp. 258–266.
  * [10] Q. Kong, Y. Cao, H. Liu, K. Choi, and Y. Wang,  “Decoupling magnitude and phase estimation with deep resunet for music source separation,”  in ISMIR, 2021.
  * [11] A. Jansson, E. Humphrey, N. Montecchio, R. Bittner, A. Kumar, and T. Weyde,  “Singing voice separation with deep U-Net convolutional networks,”  in ISMIR, 2017.
  * [12] S. Uhlich, M. Porcu, F. Giron, M. Enenkl, T. Kemp, N. Takahashi, and Y. Mitsufuji,  “Improving music source separation based on deep neural networks through data augmentation and network blending,”  in IEEE ICASSP, 2017, pp. 261–265.
  * [13] D. Stoller, S. Ewert, and S. Dixon,  “Wave-U-Net: A multi-scale neural network for end-to-end audio source separation,”  arXiv preprint arXiv:1806.03185, 2018.
  * [14] Y. Luo and N. Mesgarani,  “Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,”  IEEE/ACM Trans. Audio Speech Lang. Process., vol. 27, no. 8, pp. 1256–1266, 2019.
  * [15] A. Défossez, N. Usunier, L. Bottou, and F. Bach,  “Music source separation in the waveform domain,”  arXiv preprint arXiv:1911.13254, 2019.
  * [16] S. Rouard, F. Massa, and A. Défossez,  “Hybrid transformers for music source separation,”  in IEEE ICASSP, 2023.
  * [17] Y. Luo and J. Yu,  “Music Source Separation With Band-Split RNN,”  IEEE/ACM Trans. Audio Speech Lang. Process., vol. 31, pp. 1893–1901, 2023.
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,  “Attention is all you need,”  NeurIPS, vol. 30, 2017.
  * [19] W.-T. Lu, J.-C. Wang, M. Won, K. Choi, and X. Song,  “SpecTNT: A time-frequency transformer for music audio,”  in ISMIR, 2021.
  * [20] W.-T. Lu, J.-C. Wang, and Y.-N. Hung,  “Multitrack music transcription with a time-frequency perceiver,”  in IEEE ICASSP, 2023.
  * [21] Y.-N. Hung, J.-C. Wang, X. Song, W.-T. Lu, and M. Won,  “Modeling beats and downbeats with a time-frequency transformer,”  in IEEE ICASSP, 2022, pp. 401–405.
  * [22] J.-C. Wang, Y.-N. Hung, and J. B. Smith,  “To catch a chorus, verse, intro, or anything else: Analyzing a song with structural functions,”  in IEEE ICASSP, 2022, pp. 416–420.
  * [23] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu,  “Roformer: Enhanced transformer with rotary position embedding,”  arXiv preprint arXiv:2104.09864, 2021.
  * [24] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré,  “FlashAttention: Fast and memory-efficient exact attention with io-awareness,”  in NeurIPS, 2022.
  * [25] D. Wang and J. Chen,  “Supervised speech separation based on deep learning: An overview,”  IEEE/ACM Trans. Audio Speech Lang. Process., vol. 26, no. 10, pp. 1702–1726, 2018.
  * [26] E. Gusó, J. Pons, S. Pascual, and J. Serrà,  “On loss functions and evaluation metrics for music source separation,”  in IEEE ICASSP, 2022, pp. 306–310.
  * [27] B. Zhang and R. Sennrich,  “Root mean square layer normalization,”  NeurIPS, vol. 32, 2019.
  * [28] D. Hendrycks and K. Gimpel,  “Gaussian error linear units (GeLUs),”  arXiv preprint arXiv:1606.08415, 2016.
  * [29] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier,  “Language modeling with gated convolutional networks,”  in ICML, 2017, pp. 933–941.
  * [30] E. Vincent, R. Gribonval, and C. Févotte,  “Performance measurement in blind audio source separation,”  IEEE Trans. Audio Speech Lang. Process., vol. 14, no. 4, pp. 1462–1469, 2006.
  * [31] F.-R. Stöter, A. Liutkus, and N. Ito,  “The 2018 signal separation evaluation campaign,”  in LVA/ICA, 2018, pp. 293–305.
  * [32] I. Loshchilov and F. Hutter,  “Decoupled weight decay regularization,”  in ICLR, 2019.
  * [33] R. Solovyev, A. Stempkovskiy, and T. Habruseva,  “Benchmarks and leaderboards for sound demixing tasks,”  arXiv preprint arXiv:2305.07489, 2023.
  * [34] M. Kim and J. H. Lee,  “Sound demixing challenge 2023–music demixing track technical report,”  arXiv preprint arXiv:2306.09382, 2023.
  * [35] G. Fabbro et al.,  “The sound demixing challenge 2023–music demixing track,”  arXiv preprint arXiv:2308.06979, 2023.
  * [36] R. Hennequin, A. Khlif, F. Voituret, and M. Moussallam,  “Spleeter: a fast and efficient music source separation tool with pre-trained models,”  Journal of Open Source Software, vol. 5, no. 50, pp. 2154, 2020\.
  * [37] A. Défossez,  “Hybrid spectrogram and waveform source separation,”  arXiv preprint arXiv:2111.03600, 2021.