# The Glamorisation of Unpaid Labour:
AI and its Influencers

Nana Mgbechikwere Nwachukwu
Berkman Klein Center for Internet and Society at Harvard
<EMAIL_ADDRESS>
Jennafer Shae Roberts
Accel AI Institute
<EMAIL_ADDRESS>
Laura N Montoya
Accel AI Institute
<EMAIL_ADDRESS>

###### Abstract

To harness the true potential of Artificial Intelligence (AI) for
sustainability and societal betterment, we need to move away from the goals of
racing to mimic human behaviour and prioritising corporate interests, where
workers are continuously exploited. The unpaid labour and societal harms which
are generated from Digital Value Networks (DVNs) used by companies producing
AI needs to be regulated. Unethical data collection and data labelling
practices have serious consequences, as evidenced by the case studies reviewed
in this short paper, such as with influencer marketing. This paper addresses
important neglected areas of study in worker and user data and labeling
exploitation practices, where ethical AI could be impactful.

##  1 Introduction

Microtask marketplaces stimulate an environment where companies and academics
may outsource tedious jobs for AI training while paying less to data workers
who earn just enough to justify the flexibility and convenience of the
platform.[7][30] Human Intelligence Tasks (HITs) may include work that is
deemed too complex for computers, or where algorithms are still prone to low
accuracy. Common HITs for AI training include data scraping, data labeling,
content moderation, transcription, data generation, survey completion, and
data entry.[1] Scientists are questioning whether it is ethical to continue
the use of gig-pay microtask marketplaces for crowdsourced data collection and
processing with consideration of the workers’ exploitation,[13][16] low wages
offered per task,[7] and exposure to stressful or potentially hazardous
environments.[30] While scholars in the U.S. must get approval from an
Institutional Review Board (IRB) to conduct research involving human subjects,
including crowd workers, companies hiring on these platforms and those
academics outside the US are not held to such standards.[31] The IRB follows
the Bellmont Principles, a set of ethical guidelines for use in research in
order to ensure the protection of human subjects. The Bellmont principles for
research involving human subjects include respect for persons, beneficence,
and justice.[3] These principles are not recognized during Business Process
Outsourcing (BPO) in other countries. For example, BPO by French Startups for
HIT labour in Madagascar to enhance AI capabilities through data labelling and
annotation allows for long-term crowdsourced subcontracting with little regard
for workers’ rights.[29]

####

Amazon Mechanical Turk, launched in 1995 is estimated to be the largest global
source for outsourced work with over half a million registered
participants.[8] Workers are often paid pennies per task, resulting in
earnings far below the minimum wage in many developed countries.[7] However,
this rate can still be higher than average wages in some parts of the Global
South, leading to a form of economic exploitation. Task valuation is often set
by the task requester, leading to significant pay discrepancies. Tasks that
require a high level of skill or a large time investment may not pay
proportionally. Pay rates can vary significantly from task to task and
requester to requester, with no standard pay rate or policy. Workers have
limited rights and protections, and have little to no recourse in the event of
disputes. A recent survey of 2,026 United States Mechanical Turk workers whose
demographics and education level mirrored the US population, found that it is
possible to earn $30.05 per hour working full-time on the platform, with the
medium being $10.26 per hour.[7] While these platforms provide valuable income
opportunities for many individuals in Africa, Latin America, and India, the
broader economic implications, including the potential for exploitation and
perpetuating cycles of poverty, are significant and problematic.

####

The gains for companies that employ through these Digital Value Networks
(DVNs) often include unpaid labour in the form of time spent by crowd-workers
to maintain access and reputation in the marketplace, time spent applying for
micro-tasks, improving skills through certification, and non-payment for
completed work that is denied by the client with no repercussions.[24] DVNs
disproportionately bias crowd-workers in the Global North due to geographic
location, burdening those in the Global South with more unpaid labour. Unpaid
labour is also common in marketing by companies through the use of social
media influencers. Often neither the influencer nor those being influenced are
aware of the transactional nature of the exchange, that their data is being
collected for profit, or the extensive long term social harms they are
perpetuating through this medium at scale. Social media gave oxygen to the
influencer economy as a form of marketing that leverages the popularity and
reach of influential individuals to promote products and services.[2] Notably,
these marketing campaigns often involve digitally brushed images to present an
idealised representation of reality.

####

This tendency to portray a perfect aesthetic online has raised concerns about
authenticity and the fostering of unrealistic consumer expectations.[20]
Despite this, the influencer marketing industry has grown rapidly, with
marketers increasingly recognising its potential for driving brand awareness
and consumer engagement.

##  2 Harms of Influencer Marketing

The rise of generative AI technologies, which can autonomously create high-
quality content, is altering the landscape of influencer marketing. Generative
AI, such as Remini and Open AI’s Dall-E combined with ChaptGPT-4, can create
authentic-looking images, videos, and written content, essentially becoming
virtual influencers.[10]

####

With this transition, the authenticity concern could take a new form:
distinguishing real from AI-generated influencers and maintaining the
authenticity of reality by human influencers. Remini AI, a cutting-edge
generative AI enterprise from Italy, is making waves across ’Black Twitter’.
The firm offers various services, such as unblurring and sharpening images,
noise reduction, restoring old photos, enlarging images, fixing colours,
enhancing faces, and improving backgrounds. Leveraging these capabilities,
Remini AI pledges to create diverse renditions of users’ images from the user-
provided images.

####

Remini is just one player in an increasingly crowded field of generative AI
enterprises. What’s captivating, though, is the emerging trend among Instagram
influencers. They are using generative AI technologies like Remini to create
virtually perfect images. These enhanced images amplify their brand and
improve their visibility on Instagram’s ’Explore’ pages. The evolution of this
industry will be influenced by how society and regulatory bodies respond to
these advancements.

###  2.1 Behaviours that Contribute to Harm

Certain behaviours contribute to potential harm in AI influencer marketing on
platforms such as Twitter or Instagram. Influencers frequently disseminate
compelling narratives, emphasising the competitive advantage of AI utilisation
while neglecting potential risks. This technique primarily revolves around
content creation and audience building, with little regard for potential
consequences.

####

On Instagram, transparency deficiencies are a prevalent issue in influencer
marketing. Influencers often use AI enhancements for their photos without
disclosing this manipulation, particularly when promoting beauty products.
This can set unrealistic expectations and potentially jeopardise followers
attempting to emulate these artificially enhanced images.[15] This failure to
disclose, especially in the context of ’glow’ products advocating skin
lightening, can lead to harmful social outcomes.[4]

####

Moreover, influencers often endorse AI tools without adequately revealing
their financial motivations, creating a false impression of impartiality.[6]
These undisclosed, financially-driven endorsements present serious ethical
implications, as they may mislead consumers into adopting services or tools
that they would not have chosen if privy to the underlying motivations. The
question of who takes responsibility for ethical issues surrounding AI has
been constantly debated across different sectors.[26] However, in 2022,
reality television star Kim Kardashian was penalised $1.26 million for
neglecting to reveal her collaboration with EthereumMax while promoting them
to her extensive online following.[27] While laws and sanctions exist for the
influence of investments and affiliate marketing without disclosure, laws have
not kept up with the marketing of AI products or the encouragement of data
sharing for profit.

####

Despite their enthusiastic endorsement, many influencers often need more
comprehensive knowledge of the AI tools they promote. This could expose their
followers to unseen risks associated with these tools, including privacy
concerns.[28] Thus, inadvertently promoting potentially hazardous tools and
companies’ varying data handling practices could put consumers at significant
privacy risk.

####

Apart from privacy concerns, significant cultural and societal detriments
exist too. Although digital natives might understand these images to be
enhanced or manipulated, digital immigrants may struggle to grasp this
concept, leading to a distorted perception of products, individuals, and their
capabilities.

###  2.2 Driven by Capitalism

Capitalist motives primarily fuel AI-edited images. The entities behind these
products are typically tech companies and digital marketers who recognise the
commercial potential of visually enhanced content.[20] By promoting
’perfected’ visuals, these stakeholders can drive consumer engagement,
stimulate desire, and ultimately encourage purchase behaviours.

####

Technology firms monetise by selling AI-powered software and applications
designed for image augmentation. These solutions employ advanced machine
learning and deep learning models to transform visual content, empowering
users to digitally modify their appearances.[5] While some argue that having
an abundance of images representing diverse individuals can benefit society,
it remains to be seen whether individuals are adequately informed and
consenting to their images being used for further research. Often, such
consent is buried within lengthy and complex terms of service agreements. What
is known, is that followers of these influencers are not sharing in the reward
of the profit generated from their data, resulting in more unpaid labour and
exploitation.

##  3 Ethical Data Collection, Responsible AI Development, and the Path
Forward

Recent scholarly research[25][17] advocates for the simultaneous presence of
workers and AI systems. This coexistence requires businesses to adopt a
forward-looking approach to integrating AI, being mindful of their treatment
of employees.[14] The objective is to create a harmonious coexistence between
human workers and AI technologies in work environments, prioritising ethical
concerns and the welfare of workers.[32] Acknowledging the historical burdens
of colonialism and oppression, we must halt exploitative systems and
transition toward adaptive solutions that benefit all of humanity and the
planet. The status quo is unsustainable. If integrated and directed
appropriately, AI can play a pivotal role in driving positive change. In an
ideal scenario, workers and workplace AI complement each other by leveraging
their respective strengths.[18][22] This coexistence allows workers to harness
the precision, numerical capabilities, and pattern recognition of AI
applications.[11] By training AI to handle repetitive tasks accurately,
workers can redirect their human expertise toward complex decision-making and
critical analysis.[19][23][17] The synergy between human intelligence and AI
technologies fosters increased efficiency, productivity, and innovation in the
workplace.[32]

#### Transparency and Accountability:

Despite efforts to train AI systems to be explainable and transparent, a
significant concern persists regarding AI solution providers and their
willingness to disclose algorithmic details[9] and to inform people when AI is
being used, or when they are communicating with an AI and not a person.
Relying on corporations to self-regulate is impractical. Global systems
require comprehensive regulations to prevent exploitation. Companies that
brand themselves as "ethical" cannot claim this status if they perpetuate
exploitative practices, particularly by relying on labour from the Global
South. To achieve sustainable development, the focus should shift away from
prioritising corporate profit and fast-paced development, and instead,
emphasise the respect and humane treatment of workers and prioritise
environmental protection for future generations.

#### Addressing Current Issues in Data Collection and Labelling:

The issues associated with data collection and labelling, rooted in oppression
and exploitation, are pervasive throughout industries. Companies operating in
this field, including those mentioned in this paper, must be subject to
regulation. It is crucial to enforce and protect the rights of workers and all
individuals who share their data online. Many people remain unaware that their
data is being collected and the potential dangers that come with it. This
paper aims to raise awareness about the harms of such practices, where data is
exploited by AI and the corporations behind it, and sometimes even misused by
scammers.

#### Promoting Transparency in Data Collection:

Transparency in data collection is paramount. Individuals should be informed
when their data is being collected, used, and shared. This necessitates clear
communication from AI influencers when they collaborate with companies, and
full disclosure of sponsored content in advertisements. People need to know
when they are providing data, and they must understand the purpose behind its
collection and usage.

#### Protecting Workers in Data Collection and Labelling:

Supposing that workers’ engagement with workplace AI is aimed at compensating
for AI’s shortcomings,[17] the AI that gains advantages from interacting with
workers, rather than workers directly benefiting from these interactions. Such
a situation may not align with the AI Principles of the Organisation for
Economic Co-operation and Development (OECD), which emphasise that AI should
primarily benefit people.[21][32] Workers involved in data collection and
labelling should receive fair compensation in addition to fair treatment.
Additionally, they should have a comprehensive understanding of the work they
are undertaking and the implications of the data they handle. Their rights as
employees must be safeguarded, and measures should be in place to protect them
from exploitative practices. A truly empowering approach in workplaces
involves fostering a symbiotic relationship between workers and intelligent
systems to overcome limitations.[17] However, it is essential for companies to
prioritise workers’ rights by ensuring that the workforce possesses the
necessary skills for this symbiosis.[12]

##  4 Conclusion

Unethical data collection and labelling practices pose significant risks to
individuals and society at large. To progress ethically, we must prioritise
transparency, accountability, and fair treatment of workers. By regulating
companies and promoting responsible AI development, we can pave the way for a
sustainable future that benefits everyone. It is time for a collective effort
to address these challenges and steer AI technology toward a positive and
inclusive future. By upholding workers’ rights and providing them with the
required skills, we can create a collaborative and mutually beneficial
environment where both humans and AI technologies thrive.

While this short review has shed some light on pertinent issues around ethical
considerations for AI-driven digital value networks, an in-depth investigation
is warranted in future research to fully understand the implications and
potential regulation of the practices outlined in this piece.

## Acknowledgments and Disclosure of Funding

We acknowledge the Berkman Klein Center for Internet and Society at Harvard
for hosting the AI and Agency in the Global South working group which brought
these authors together.

## References

  * [1] Joel Ross et al.  “Who are the crowdworkers?”  In _CHI ’10 Extended Abstracts on Human Factors in Computing Systems_ New York, NY, USA: ACM, 2010, pp. 2863–2872  DOI: 10.1145/1753846.1753873
  * [2] Karen Freberg, Kristin Graham, Karen McGaughey and Laura A. Freberg  “Who are the social media influencers? A study of public perceptions of personality”  In _Public Relations Review_ 37.1, 2011, pp. 90–92  DOI: 10.1016/j.pubrev.2010.11.001
  * [3] Education Department of Health, Welfare and National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research  “The Belmont Report. Ethical principles and guidelines for the protection of human subjects of research.”  In _The Journal of the American College of Dentists_ 81.3, 2014, pp. 4–13
  * [4] Jasmine Fardouly, Phillippa C. Diedrichs, Lenny R. Vartanian and Emma Halliwell  “Social comparisons on social media: The impact of Facebook on young women’s body image concerns and mood”  In _Body Image_ 13, 2015, pp. 38–45  DOI: 10.1016/j.bodyim.2014.12.002
  * [5] Raz Schwartz and Germaine R Halegoua  “The spatial self: Location-based identity performance on social media”  In _New Media & Society_ 17.10, 2015, pp. 1643–1660  DOI: 10.1177/1461444814531364
  * [6] Crystal Abidin  “Visibility labour: Engaging with Influencers’ fashion brands and #OOTD advertorial campaigns on Instagram”  In _Media International Australia_ 161.1, 2016, pp. 86–100  DOI: 10.1177/1329878X16665177
  * [7] Paul Hitlin  “Research in the Crowdsourcing Age, a Case Study”  In _PEW RESEARCH CENTER_ , 2016
  * [8] Matthew Pittman and Kim Sheehan  “Amazon’s Mechanical Turk a Digital Sweatshop? Transparency and Accountability in Crowdsourced Online Research”  In _Journal of Media Ethics_ 31.4, 2016, pp. 260–262  DOI: 10.1080/23736992.2016.1228811
  * [9] Thomas H. Davenport  “Can We Solve AI’s ‘Trust Problem’?”  In _MIT Sloan Manag. Rev_ 60, 2018  URL: https://sloanreview.mit.edu/article/can-we-solve-ais-trust-problem/
  * [10] Ming-Hui Huang and Roland T. Rust  “Artificial Intelligence in Service”  In _Journal of Service Research_ 21.2, 2018, pp. 155–172  DOI: 10.1177/1094670517752459
  * [11] Frieda Klotz  “How AI Can Amplify Human Competencies”  In _MIT Sloan Manag. Rev_ 60, 2018  URL: https://sloanreview.mit.edu/article/how-ai-can-amplify-human-competencies/
  * [12] Maria José Sousa and Daniela Wilks  “Sustainable Skills for the World of Work in the Digital Age”  In _Systems Research and Behavioral Science_ 35.4, 2018, pp. 399–405  DOI: 10.1002/sres.2540
  * [13] E Damer  “Stop using MTurk for research”  In _Prolific_ , 2019  URL: https://blog.prolific.co/%20stop-using-mturk-for-research/
  * [14] Lan Li, Gang Li and Shui F. Chan  “Corporate responsibility for employees and service innovation performance in manufacturing transformation”  In _Career Development International_ 24.6, 2019, pp. 580–595  DOI: 10.1108/CDI-04-2018-0109
  * [15] Caitlin McBride et al.  “Digital Manipulation of Images of Models’ Appearance in Advertising: Strategies for Action Through Law and Corporate Social Responsibility Incentives to Protect Public Health”  In _American Journal of Law & Medicine_ 45.1, 2019, pp. 7–31  DOI: 10.1177/0098858819849990
  * [16] A Newman  “I found work on an Amazon website. I made 97 cents an hour.”  In _The New York Times_ , 2019
  * [17] H. Wilson and Paul R. Daugherty  “Creating the Symbiotic AI Workforce of the Future”  In _MIT Sloan Manag. Rev_ 61, 2019  URL: https://sloanreview.mit.edu/article/creating-the-symbiotic-ai-workforce-of-the-future/
  * [18] Alexander P. Henkel, Stefano Bromuri, Deniz Iren and Visara Urovi  “Half human, half machine – augmenting service employees with AI for interpersonal emotion regulation”  In _Journal of Service Management_ 31.2, 2020, pp. 247–265  DOI: 10.1108/JOSM-05-2019-0160
  * [19] Naomi Aoki  “The importance of the assurance that “humans are still in the decision loop” for public trust in artificial intelligence: Evidence from an online experiment”  In _Computers in Human Behavior_ 114, 2021, pp. 106572  DOI: 10.1016/j.chb.2020.106572
  * [20] Yogesh K. Dwivedi et al.  “Setting the future of digital and social media marketing research: Perspectives and research propositions”  In _International Journal of Information Management_ 59, 2021, pp. 102168  DOI: 10.1016/j.ijinfomgt.2020.102168
  * [21] OECD  “State of implementation of the OECD AI Principles: insights from national AI policies”, 2021
  * [22] Sebastian Raisch and Sebastian Krakowski  “Artificial Intelligence and Management: The Automation–Augmentation Paradox”  In _Academy of Management Review_ 46.1, 2021, pp. 192–210  DOI: 10.5465/amr.2018.0072
  * [23] Yash Raj Shrestha, Vaibhav Krishna and Georg Krogh  “Augmenting organizational decision-making with deep learning algorithms: Principles, promises, and challenges”  In _Journal of Business Research_ 123, 2021, pp. 588–603  DOI: 10.1016/j.jbusres.2020.09.068
  * [24] Kelle Howson et al.  “Unpaid labour and territorial extraction in digital value networks”  In _Global Networks_ , 2022  DOI: 10.1111/glob.12407
  * [25] Akanksha Jaiswal, C. Arun and Arup Varma  “Rebooting employees: upskilling for artificial intelligence in multinational corporations”  In _The International Journal of Human Resource Management_ 33.6, 2022, pp. 1179–1208  DOI: 10.1080/09585192.2021.1891114
  * [26] Nithesh Naik et al.  “Legal and Ethical Consideration in Artificial Intelligence in Healthcare: Who Takes Responsibility?”  In _Frontiers in Surgery_ 9, 2022  DOI: 10.3389/fsurg.2022.862322
  * [27] Securities and Exchange Commission  “SEC Charges Kim Kardashian for Unlawfully Touting Crypto Security”  In _SEC_ , 2022  URL: https://www.sec.gov/news/press-release/2022-183
  * [28] Liselot Hudders and Chen Lou  “The rosy world of influencer marketing? Its bright and dark sides, and future research recommendations”  In _International Journal of Advertising_ 42.1, 2023, pp. 151–161  DOI: 10.1080/02650487.2022.2137318
  * [29] Clément Le Ludec, Maxime Cornet and Antonio A Casilli  “The problem with annotation. Human labour and outsourcing between France and Madagascar”  In _Big Data & Society_ 10.2, 2023  DOI: 10.1177/20539517231188723
  * [30] Aaron J. Moss et al.  “Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages”  In _Behavior Research Methods_ , 2023  DOI: 10.3758/s13428-022-02005-0
  * [31] H Xia  “Two ethical dilemmas in conducting human subjects research on Amazon Mechanical Turk (MTurk)”  In _Information Matters_ , 2023  URL: https://informationmatters.org/2023/03/two-ethical-dilemmas-in-conducting-human-subjects-researchonamazon-mechanical-turk-mturk/
  * [32] Araz Zirar, Syed Imran Ali and Nazrul Islam  “Worker and workplace Artificial Intelligence (AI) coexistence: Emerging themes and research agenda”  In _Technovation_ 124 Elsevier Ltd, 2023  DOI: 10.1016/j.technovation.2023.102747


2023 Jun.

AND AES Fellow

# Non-Exponential Reverberation Modeling Using Dark Velvet Noise

JON FAGERSTRÖM  1 SEBASTIAN J. SCHLECHT  1,2 VESA VÄLIMÄKI  1
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS> 1Acoustics Lab, Department of Information and
Communications Engineering, Aalto University, Espoo, Finland
2Media Lab, Department of Art and Media, Aalto University, Espoo, Finland

###### Abstract

Previous research on late-reverberation modeling has mainly focused on
exponentially decaying room impulse responses, whereas methods for accurately
modeling non-exponential reverberation remain challenging. This paper extends
the previously proposed basic dark-velvet-noise reverberation algorithm and
proposes a parametrization scheme for modeling late reverberation with
arbitrary temporal energy decay. Each pulse in the velvet-noise sequence is
routed to a single dictionary filter that is selected from a set of filters
based on weighted probabilities. The probabilities control the spectral
evolution of the late-reverberation model and are optimized to fit a target
impulse response via non-negative least-squares optimization. In this way, the
frequency-dependent energy decay of a target late-reverberation impulse
response can be fitted with mean and maximum T60 errors of $4$% and $8$%,
respectively, requiring about $50$% less coloration filters than a previously
proposed filtered velvet-noise algorithm. Furthermore, the extended dark-
velvet-noise reverberation algorithm allows the modeled impulse response to be
gated, the frequency-dependent reverberation time to be modified, and the
model’s spectral evolution and broadband decay to be decoupled. The proposed
method is suitable for the parametric late-reverberation synthesis of various
acoustic environments, especially spaces that exhibit a non-exponential
energy-decay, motivating its use in musical audio and virtual reality.

##  1 INTRODUCTION

Artificial reverberation algorithms have been developed since the 1960s,
starting with Schroeder’s original algorithm [1, 2]. Schroeder’s algorithm, as
well as many that followed, are based on the assumption that the late
reverberation part of a room impulse response (IR) can be modeled with
exponentially decaying filtered white noise [3, 4, 5, 2]. However, non-
exponentially decaying reverberation can be observed in forests [6, 7], in
coupled rooms [8, 9, 10], and in the famous gated reverb sound from the 1980s
[11].

A room IR can be divided into three perceptually motivated parts, the direct
sound, the early reflections, and the late reverberation [12]. The study
presented in this paper focuses on the modeling of the late-reverberation
part. In this paper, we propose a novel artificial reverberator capable of
modeling target late reverberation with arbitrary energy decay and spectral
evolution.

A notable branch of artificial reverberation algorithms is based on pseudo-
random noise. Rubak and Johansen proposed using sparse random noise to model
exponentially decaying Gaussian noise [4, 13]. However, the resulting sparse
finite-impulse-response (FIR) filters placed inside a feedback loop would
still need over 10,000 filter coefficients to produce a smooth reverberation
IR, i.e. reverberation IR that does not sound rough. Later, Karjalainen and
Järveläinen introduced velvet noise [5], which becomes smooth broadband noise
with a pulse density of 1500 to 2000 pulses/s [5, 14]. To reduce computational
costs, several velvet-noise-based algorithms employ a feedback structure,
limiting them to generating only exponential decay [5, 15, 16].

It is important to make a distinction between the psychoacoustic quantity
roughness, measured in asper [17], and the term ”temporal roughness” used in
this work to describe the perceived quality of sparse noise. The former is
defined as the auditory sensation caused by amplitude-modulated pure tones,
with modulation frequencies within the range of $15–300$ Hz [17]. The latter
is only loosely defined in previous literature as the sensation when a sparse
noise sequence is not perceived as sounding smooth [5, 14]. As Meyer-Kahlen et
al. [18] pointed out, the random assignment of the pulses of a sparse noise
sequence can be interpreted as pseudo-random amplitude modulation.

The feedback delay network (FDN) [19] is a generalization of the comb-filter-
based reverberator, which is still actively studied today [20, 21, 22, 23].
Hybrid reverberators combining an FDN and velvet noise have also been
proposed, which place the velvet-noise filters at the inputs and outputs [24]
or within the feedback matrix of an FDN [22] to increase the echo density.
However, in its basic form, the FDN can produce only an exponential decay.
Combining two FDNs with different parameters allows for generating various
non-exponential attenuation patterns, such as fade-in control or two-stage
decay [25, 26, 27, 10]. An extended method based on the FDN has been proposed
to synthesize double-slope decays of coupled rooms [23, 9]. However, no FDN-
based method is capable of synthesizing reverberation, which has an arbitrary
and non-exponential energy decay.

Karjalainen and Järveläinen proposed a modal reverberator structure for
modeling late reverberation [28], and the idea was later refined by Abel et
al. [29]. The modal reverberator is implemented with a parallel combination of
mode filters, whose resonant frequencies and damping coefficients are tuned to
match those of the target space. The number of modes to produce high-quality
reverberation is suggested to be between $1000$ and $2000$ modes, based on
informal listening. Wells recommends the use of a much larger number of modes
[30]. The modal reverberators are best suited for exponentially decaying
reverberation. However, implementing two-stage decays and fade-ins is also
possible by tuning the damping of a portion of the mode filters [26].
Recently, modal synthesis has been proposed for the resynthesis of denoised
anisotropic late reverberation with multi-slope decays by Hold et al. [31].
The multi-slope decays, however, are still a linear combination of exponential
decays.

Holm et al. introduced an FIR-filter-based algorithm called the filtered
velvet-noise (FVN) reverberator [32] that Välimäki et al. refined later [33].
The FVN models a target late-reverberation IR with concatenated filtered-noise
segments of different lengths. The filters are designed based on the time-
frequency analysis of a target IR. The variable-length windowing mitigates
audible transitions between the consecutive filters. The FIR-based FVN
structure allows manipulating the IR by changing the lengths of the VN block,
e.g., lengthening or shortening the decay time [33].

In this paper, our previous work on the dark-velvet-noise (DVN) reverberator
[34] is extended to fit its IR to a measured target late-reverberation IR. An
extension to the original DVN algorithm is introduced, which replaces its
recursive running-sum filters with arbitrary dictionary filters. Furthermore,
the uniform probability for a pulse to be connected to a certain filter is set
as a free parameter. The proposed method fits the extended DVN model to a
target response via non-negative least-squares (NNLS) optimization [35], which
is applied for the first time to model reverberation. The resulting model is
parametric and facilitates various perceptually relevant modifications.
Additionally, we investigate the temporal roughness properties of the proposed
extended DVN and propose a scheme for mitigating it.

Our proposed extended DVN model IR is subjected to an objective evaluation:
First, we compare the original target IRs in terms of spectro-temporal fit and
reverberation time (T60) estimation in the case where the target IR has an
exponential energy decay, with IRs synthesized with optimized extended DVN
model instances. We demonstrate that the new method provides a good objective
fit in parametrizing two distinctly different target reverberation IRs, those
of a concert hall and an outdoor space coupled to a cave opening. Next, we
compare the proposed method to the previously proposed FVN method [33], and
formulate the FVN method as a special case of the proposed method.

The rest of this paper is organized as follows. Sec. 2 summarizes the
previously proposed DVN algorithm [34]. Sec. 3 proposes the novel extension of
the DVN structure. Sec. refsec:modeling discusses the proposed reverberation
modeling scheme, step by step, including the NNLS optimization scheme. Sec. 5
presents an objective performance evaluation when using the proposed method
for modeling a target IR, and discusses the ﬂexibility of the extended DVN in
generating parametric modiﬁcations of the modeled IR. Sect. 6 concludes the
paper.

##  2 DARK VELVET NOISE

This section provides the relevant background on the previously proposed DVN
algorithm [34]. The basics of velvet noise and dark velvet noise serve as the
basis for developing the proposed method for late-reverberation synthesis.

Original velvet noise [5] is a sparse pseudo-random noise, which consists of
sparsely placed unit impulses with uniformly distributed signs. The main
design parameter of a velvet-noise sequence is its pulse density $\rho$ in
pulses/s. Based on the desired pulse density, the grid size $T_{\mathrm{d}}$
can be computed as,

$T_{\mathrm{d}}=\frac{f_{\mathrm{s}}}{\rho},$ (1)

where $f_{\textrm{s}}$ is the sample rate in Hz. Within each grid segment, a
single unit impulse occurs.

Whereas the original velvet noise has a white magnitude spectrum [36], DVN is
an extension of velvet noise that has a lowpass spectrum [34]. The lowpass
spectrum is achieved by a random modulation of the pulse width ${w}$ along the
DVN sequence, i.e., the unit impulses of the original velvet noise are
replaced by square pulses of varying width. In practice, the pulse-width
modulation is implemented using a discrete set of recursive running-sum
filters, one for each required pulse width [34]. The DVN sequence is given as

$h(n)=\begin{cases}s(m)&\mathrm{for}\,\,k(m)\leq n<k(m)+w(m),\\\
0&\mathrm{otherwise,}\end{cases}$ (2)

where $n$ is the sample index, $m$ is the pulse index, $s(m)$ is the sign of
the $m$th pulse, and $k(m)$ is the location of the $m$th pulse. When
$w(m)\equiv 1$, Eq. (2) gives the original velvet noise sequence where each
pulse is a unit impulse.

The pulse locations of DVN are computed as

$k(m)=\lfloor mT_{\mathrm{d}}+r_{1}(m)(T_{\mathrm{d}}-w(m))\rceil,$ (3)

where $\lfloor\cdot\rceil$ is the rounding operator and $r_{1}(m)$ is a
uniform random number between $0$ and $1$. This formulation assures that the
pulses do not overlap each other, given that $1\leq w(m)\leq T_{\mathrm{d}}$.

The pulse widths of DVN (in samples) are computed as

$w(m)=\lfloor r_{2}(m)(w_{\text{max}}-w_{\text{min}})+w_{\text{min}}\rceil,$
(4)

where $r_{2}(m)$ is an uniform random number between $0$ and $1$, and
$w_{\text{max}}$ and $w_{\text{min}}$ are the maximum and minimum pulse
widths, respectively. The sign of each pulse is computed by [5]

$s(m)=2\,\lfloor r_{3}(m)\rceil-1,$ (5)

where $r_{3}(m)$ is a uniform random number between $0$ and $1$.

The first 8 ms ($f_{\textrm{s}}=48$ kHz 111We used the sample rate of
$f_{\textrm{s}}=48$ kHz throughout this work.) of an example basic DVN
sequence, and the power spectral density (PSD) of the corresponding infinitely
long DVN sequence are shown in Fig. 1(a) and Fig. 1(b), respectively.

(a)

(b)

Figure 1: (a) Example of a basic DVN sequence, and (b) its PSD normalized to 0
dB. The dashed lines show the grid spacing $mT_{\mathrm{d}}$ at the sample
rate $f_{\textrm{s}}=48$ kHz.

##  3 EXTENDED DARK VELVET NOISE

In this section, an extension to the previously proposed DVN [34] is
presented. The proposed extension replaces the previously used recursive
running-sum filters with arbitrary dictionary filters, whose probabilities are
set as a free parameter. Additionally, the temporal roughness of the proposed
extension is discussed.

###  3.1 Generalization of Dark Velvet Noise

As the first step of the generalization, we replace the recursive running-sum
filters with arbitrary dictionary filters $\bm{F}(z)$, allowing the generation
of colored noise of desired spectral shape. Fig. 2 shows the block diagram of
the proposed extended DVN convolution structure with $Q$ dictionary filters
$\bm{F}(z)$, and $M\gg Q$ pulses. The decay-envelope gains $g(m)$ parametrize
the broadband energy decay. Finally, in contrast to the basic DVN, we set the
uniform probabilities of each dictionary filter as a free parameter.

Figure 2: Structure of the proposed extended DVN convolution with $Q$
dictionary filters and $M$ pulses. Each pulse is multiplied by a gain $g(m)$
and routed to the input of one dictionary filter, as indicated by the matrix.
Each dot in the matrix represents a connection path.

The filter probabilities for a single pulse are denoted by the vector

$\displaystyle\begin{aligned}
\bm{p}=\begin{bmatrix}p_{1},p_{2},\ldots,p_{Q}\end{bmatrix}^{\textrm{T}}\geq
0,\,\mathrm{with}\,\sum_{q=1}^{Q}p_{q}=1,\end{aligned}$ (6)

where $[\cdot]^{\textrm{T}}$ is the transpose operation.

A list of filter indices for each pulse is determined based on the pulse-
filter probabilities $\bm{p}$ as

$\phi(m)=f(\bm{p}(m))\in\\{1,2,...,Q\\},$ (7)

where $f(\cdot)$ is any function that selects the pulse filter based on the
probability $\bm{p}$. The resolved list of filter indices $\phi$ is visualized
in matrix form in Fig. 2, where the one-hot column vectors show the filter
selection for each pulse.

Eq. (2) can be reformulated so that the velvet-noise sequence is split into
$Q$ sub-sequences. Each sub-sequence includes the pulses routed to one of the
dictionary filters, corresponding to a single row of the connection matrix of
Fig. 2. The $q$th sub-sequence is then given as

$v_{q}(n)=\begin{cases}s(m){g(m)}&\mathrm{for}\,\,n=k(m)\land\phi(m)=q,\\\
0&\mathrm{otherwise.}\end{cases}$ (8)

The transfer function of the extended DVN can now be written as

$H(z)=\sum_{q=1}^{Q}V_{q}(z)F_{q}(z),$ (9)

where $V_{q}(z)$ is the velvet-noise subsequence routed to the $q$th
dictionary filter $F_{q}(z)$. The time-dependent PSD of the extended DVN
sequence is described by the weighted mean magnitude response of the
dictionary filters:

$|H(m,\omega)|=\sum_{q=1}^{Q}|F_{q}(\omega)|p_{q}(m),$ (10)

where $|F_{q}(\omega)|$ is the magnitude response of the $q$th dictionary
filter, and the probabilities $p_{q}$ are defined in Eq. (6). The PSD of the
extended DVN sequence, Eq. (10), is independent of the pulse density $\rho$,
and the equation holds true as the occurrences of dictionary filters are
uncorrelated due to the randomized selection of the filter and the placement
of pulses.

Figs. 3(a) and 3(b) show an example of an extended DVN sequence and its PSD,
respectively. The pulse locations are the same as in the basic DVN sequence in
Fig. 1(a). Note that the pulses are no longer located within the grid segments
of the basic DVN, and are smeared in time.

(a)

(b)

Figure 3: (a) Extended DVN IR with the underlying pulse locations and signs
indicated by asterisks. (b) The corresponding PSD, computed using Eq. (10) and
normalized to 0 dB. The used dictionary filters correspond to the ones
presented in Fig. 4(a).

###  3.2 Mitigating Temporal Roughness

In this work, and in previous studies on velvet noise [5, 14], temporal
roughness describes the perceived quality of sparse noise. For the original
velvet noise, which consists of randomly placed unit impulses, the perceived
temporal roughness is simply inversely proportional to its pulse density
$\rho$. Based on informal listening experiments the basic DVN method, which
uses the recursive running-sum filters, shows similar behaviour in terms of
temporal roughness [34].

In this work, however, we used high-order filters with a potentially steep
spectral decay. In combination with the naive implementation of the pulse-
filter selection (see Eq. (7),) this could lead to perceivable temporal
roughness. In particular, roughness issues may originate from the random
switching between filters that have large energy differences within certain
frequency bands, as seen in the spectrogram of Fig. 4(b).

To mitigate issues related to temporal roughness, we propose a two-step
solution. First, the filter energies are normalized to minimize the temporal
roughness, by ensuring the broadband filter energy does not fluctuate when
switching between different filters. Next, a greedy filter assignment is
proposed. The naive pulse filter selection is based on a weighted uniform
random number and is given as

$\phi(m)=\operatorname*{arg\,max}_{q}\\{r_{q}(m)p_{q}(m)\\},$ (11)

where $r_{q}(m)$ is a uniform random number in the range $[0,1]$.

We propose the following greedy filter assignment instead:

$\phi(m)=\operatorname*{arg\,max}_{q}\\{(\tau_{q}(m)+\epsilon
r_{q}(m))p_{q}(m)\\},$ (12)

where $\epsilon$ is a free parameter controlling the amount of randomization
and $\tau_{q}$ is a sample index (i.e., time) when the $q$th dictionary filter
was last selected. The value $\tau_{q}$ is updated sequentially based on the
previously selected pulse filter with

$\tau_{q}(m+1)=\begin{cases}0&\mathrm{for}\,\,\phi(m)=q,\\\
\tau_{q}(m)+1&\mathrm{otherwise.}\end{cases}$ (13)

When the $q$th dictionary filter stays inactive, its weight $\tau_{q}$ grows,
thus making it more likely for the greedy assignment to pick that filter.

(a)

(b)

(c)

Figure 4: (a) Magnitude responses of the example second-order dictionary
filters $F_{q}(z)$ with $q=1,...,Q=10$, and the spectrograms of the resulting
extended DVN sequence with (b) a totally random and (c) greedy filter
assignment.

The roughness problem is best visualized by synthesizing stationary noise with
uniform probabilities $p_{q}\equiv 1/Q$, i.e., a pulse routing to any of the
defined dictionary filters with uniform probability. Fig. 4(a) shows the
magnitude responses of an example set of ten second-order dictionary filters.
However, the spectrogram in Fig. 4(b) highlights the frequency-dependent
roughness problem that remains after the normalization when the naive pulse
filter selection of Eq. (11) is used. We computed the spectrograms using FFT
of length $2048$, with a $256$-sample Hann window with $50$% overlap. The
frequency axis in Fig. 4(b) was limited from $1$ kHz to $20$ kHz for better
visualization of the sparsity, which creates audible temporal roughness in the
noise sequence. The naive filter assignment is based on uniform probabilities,
and thus the resulting sub-sequences defined by Eq. (8) resemble totally
random noise, which was shown to sound rougher than velvet noise [5, 14].

Fig. 4(c) shows the spectrogram of the extended DVN sequence generated based
on the same uniform probabilities, but now applying the greedy filter
assignment of Eq. (12) instead of the random assignment. The resulting
extended DVN sequence is visually and audibly much smoother. However, there is
a trade-off since the greedy assignment creates some periodicity in the
sequence. Note that the PSD of the sequences in Fig. 4(b) and Fig. 4(c) is the
same. The noise sequences in Fig. 4(b) and Fig. 4(c) can be listened to on the
companion web page of this paper222https://github.com/Ion3rik/dark-velvet-
noise-reverb.

##  4 REVERBERATION MODELING

The proposed framework for parametrizing a target late-reverberation IR using
the novel extended DVN structure is presented in this section. The previously
proposed FVN method is shown to be a special case of the extended DVN
algorithm.

###  4.1 Preprocessing and Analysis

To parameterize a late-reverberation IR with the extended DVN algorithm, we
used the short-time Fourier transform (STFT) to obtain a time-frequency
representation of the target IR. The late-reverberation IR of the Promenadi
Hall [37], Pori, Finland, serves as an example target IR throughout this
section. The T60 of the target IR, evaluated at octave bands between $125$ Hz
and $8\,000$ Hz, ranges from $2.7$ s at the lowest band, to $1.2$ s at the
highest band [37].

In the following, the number of STFT time frames is denoted with $T$. We used
an FFT length of $2048$. The design choice in the analysis step is to choose a
suitable frame size and overlap amount to capture the temporal and spectral
changes in sufficient detail. The choice may vary depending on the
characteristics of the target IR. Typically, larger rooms with longer decay
times will tend to have smooth exponentially decaying time envelopes for which
a shorter window size yields good results. Small rooms or outdoor spaces with
short reverb time tend to have a more detailed envelope that affects heavily
the timbre of the IR, thus requiring a denser framing for the analysis to
yield a more detailed decay envelope for the extended DVN model.

Figure 5: Example magnitude responses of the pre-filter and the complementary
post-filter of an extended DVN model instance tuned to the Promenadi Hall IR.

As a preprocessing step, we applied linear prediction (LP) on the first time
frame of the STFT representation of the target IR. The solution of the LP
gives the allpole post-filter $1/R(z)$ coefficients. The coefficients of the
LP filters were obtained using Matlab’s lpc function. A similar LP modeling
approach was applied previously by Holm et al. [32]. We used a filter order of
10 in this case for LP modeling since this was found to be sufficient for
capturing the lowpass characteristic of late reverberation in a previous study
[33].

However, the low-order LP filter cannot model any possible low-frequency roll-
off present in the target IR. Thus, in this work, we fitted an additional
first-order direct-current-blocker (DC-blocker) filter [38] to the target IR
to improve the model’s spectral accuracy at low frequencies. The inverse
filter of the allpole filter, i.e., the pre-filter, was applied to the whole
target IR to whiten it. The whitening ensures that the target IR starts with a
flat magnitude spectrum. Thus the post-filter takes care of matching the
initial coloration of the target IR, whereas the dictionary filters
concentrate on modeling the time-dependent relative spectral change.

The magnitude response of the post-filter is shown in Fig. 5 (solid line). The
response is the combined response of the tenth-order allpole filter and the
first-order DC-blocker filter. In this example, the effect of the DC blocker
is visible as a slight cut below $100$ Hz. The corresponding pre-filter
magnitude response of the target IR is shown as a dashed line in Fig. 5.

###  4.2 Dictionary-Filter Design

The main design question was to define the dictionary filters to be used in
the extended DVN structure shown in Fig. 2. In this work, the dictionary
filters were obtained directly from the analysis stage, reminiscent of the FVN
algorithm [33], by approximating each analysis frame response with a second-
order allpole filter. The coefficients of the second-order allpole filter were
estimated using an LP. The single tenth-order pre-filter allows applying
second-order filters in the dictionary instead of the tenth-order filters used
in the previously proposed FVN method [33].

Figure 6: Normalized magnitude responses of the dictionary filters used for
modeling the late-reverberation IR of the Promenadi Hall. Here, the dictionary
consists of ten second-order allpole filters extracted from the analyzed
target reverberation. The filter energies are normalized to one.

Applying a subset of the analyzed filters that were logarithmically spaced in
time was found to yield good results. Due to the logarithmic spacing, closely
spaced frame filters are picked from the beginning of the analyzed IR, and
more sparsely spaced filters towards the end. The justification for the
logarithmic spacing was inspired by the results of Välimäki et al. [33]. They
observed that the magnitude spectrum of late reverberation typically varies
rapidly at the beginning of the IR and slower towards the end of the IR [33].

Fig. 6 shows example magnitude responses of the set of dictionary filters used
to model the Promenadi Hall late-reverberation IR. In this example, the target
IR was analyzed in $50$ frames, and the logarithmically spaced subset of ten
dictionary filters included the filters $1,2,3,5,7,10,15,23,34$, and $50$. The
filter energies were normalized to one. Note that the first dictionary filter
in Fig. 6, which is estimated from the first frame of the pre-whitened target
IR, has a practically flat magnitude response and can be omitted in the
implementation without causing an audible effect.

###  4.3 Solving the Filter Probabilities

After designing the dictionary filters, their magnitude responses need to be
fit to the target magnitude response by solving an NNLS problem. NNLS is a
constrained version of the least-squares optimization problem, which is a
convex problem with linear constraints. [35].

The dictionary filter magnitude responses are denoted by the matrix
$|\bm{F}(\omega_{k})|$, where the columns contain the magnitude response of
each dictionary filter at discrete frequencies. The NNLS solution yields the
activation vector $\bm{z}$. The values of coefficients $\bm{z}$ are
constrained to be positive or zero. The optimization problem has the form

$\min_{\bm{z}}\norm{\absolutevalue{\bm{F}(\omega_{k})}\bm{z}-\absolutevalue{\bm{h}(\omega_{k})}}^{2}_{2},\>\textrm{subject
to}\>\bm{z}\geq 0,$ (14)

where $\omega_{k}$ are the discrete frequencies and $|\bm{h}(\omega_{k})|$ is
the target magnitude response. In this work, Matlab’s lsqnonneg function was
used to solve the optimization problem. To compose the activation matrix for
the whole time-frequency representation of the target IR, the optimization
problem, cf. Eq. (14), was solved for each time step separately. The
activation matrix has the form

$\displaystyle\begin{aligned}
\bm{Z}=\begin{bmatrix}\bm{z}(1),&\bm{z}(2),&\dots,&\bm{z}(T)\\\
\end{bmatrix},\end{aligned}$ (15)

where $\bm{z}(t)$ are the column vectors containing the solved activations for
each analysis frame $t$.

Due to the formalization of the proposed extended DVN structure, we normalized
the solved activation matrix $\bm{Z}$ to obtain the probability of each
dictionary filter for each time step. The normalization gains are computed as

$\gamma(t)=\norm{\bm{z}(t)}_{1}=\sum_{i=1}^{Q}\absolutevalue{\bm{z}(t)}.$ (16)

The probability matrix is then given as

$\bm{P}=\bm{Z}\odot\frac{1}{\vec{\gamma}},$ (17)

where $\odot$ is the Hadamard product and
$\vec{\gamma}=[\gamma(1),\gamma(2),\dots,\gamma(T)]^{\textrm{T}}$ After the
normalization, each column vector $\bm{p}(t)$ of the probability matrix
fulfills Eq. (6).

Fig. 7(a) shows the probability matrix $\bm{P}$ obtained via normalization of
the activation matrix solution for the target IR. Dark areas indicate the most
probable filter routings for each analysis time frame. The general trend shows
that the higher probabilities shift from the first dictionary filter towards
the last dictionary filter.

(a)

(b)

Figure 7: (a) Probability matrix $\bm{P}$ and (b) the corresponding resolved
pulse filters for modeling the Promenadi Hall late-reverberation IR.

###  4.4 Sparse Synthesis with Dark Velvet Noise

The first step of the synthesis is to generate a velvet-noise sequence which
has the same length as the target IR with the desired pulse density $\rho$. In
this work, we used a time-dependent density starting from $\rho=2000$ pulses/s
and linearly decreasing towards $\rho=500$. The decreasing density allows
maintaining a low computational load of the algorithm without introducing
temporal roughness to the reverberation [33]. The pulse locations and signs of
the sequence were computed using Eqs. (3) and (5), respectively.

Conveniently, the normalization gains $\gamma(t)$ defined in Eq. (16) describe
the broadband energy decay envelope of the target IR. Fig. 8 shows the
normalization gains $\gamma(t)$ corresponding to the probability matrix
$\bm{P}$ of Fig. 7. On a logarithmic scale, the normalization gains
$\gamma(t)$ decrease approximately linearly over time, except for their
beginning and end—a trend which corresponds to the exponential decay of the
target IR.

Figure 8: Normalization gains $\gamma(t)$ per frame.

As the synthesis requires a discrete number of pulses, we interpolated the
probability matrix $\bm{P}$ and the normalization gains $\vec{\gamma}$ to
match the number of pulses $M$. We used the pre-computed pulse locations $k$
as query points, and applied linear interpolation. The interpolated
normalization gains are denoted by $\widehat{\vec{\gamma}}$. The per-pulse
decay gains of the extended DVN convolution structure shown in Fig. 2 are
obtained as

$g(m)=\widehat{\gamma}(m)\sqrt{T_{\mathrm{d}}(m)},$ (18)

where the multiplication with the square root of the grid size
$T_{\mathrm{d}}(m)$ compensates for the change in energy due to the time-
dependent density.

Finally, the pulse filters are resolved using Eq. (12), based on the
interpolated probability matrix. Fig. 7(b) shows the resolved pulse filters
computed for the probability matrix of Fig. 7(a). High concentrations of
pulses (see Fig. 7(b),) are likely routed to filters with high probabilities
(see Fig. 7(a),) whereas areas of low probability, i.e., the areas of lighter
color, show sparser patterns of filter selection.

###  4.5 Comparison to Filtered Velvet Noise

The reverberation modeling framework presented above resembles the previously
proposed FVN algorithm [32, 33]. In this section, a comparison between the two
methods is provided.

In the FVN algorithm, the target IR is segmented into non-overlapping frames
of different lengths, and an allpole LP filter is extracted from each segment
[33]. In the extended DVN method, the extracted LP filters can be used as both
the target and dictionary responses. In this special case, the solution of the
NNLS problem returns a diagonal probability matrix, since the best fit is
obtained by simply activating the extracted segment filters in order.

(a)

(b)

(c)

Figure 9: First ten frames of the (a) probability matrix $\bm{P}$ and the
corresponding resolved pulse filters (b) with filter interpolation, resembling
uniformly segmented FVN, and (c) with probability interpolation, unique to the
extended DVN. The presented dictionary filters are the LP allpole target
filters of each frame.

Fig. 9(a) shows the first ten segments of the probability matrix $\bm{P}$
obtained using the matched dictionary filters extracted from the target IR.
Fig. 9(b) shows the corresponding filter assignment for each pulse when the
filters are assigned using the per-frame probabilities and the resulting
filter list is interpolated to yield a filter routing for each pulse. In this
configuration, the extended DVN essentially implements the FVN algorithm,
where the filter is switched at the segment boundaries. However, the constant
segment length results in periodic switching of the dictionary filter, which
can cause audible disturbance in the synthesized IR [33]. The switching
periodicity was mitigated in the FVN algorithm by using a non-uniform segment
length.

The extended DVN presents another option for the filter assignment, where the
probability matrix is first interpolated, and the filters are then assigned
per pulse based on the interpolated probability matrix. Fig. 9(c) shows the
corresponding filter assignment for each pulse. In this case, the switching is
not discrete at each segment boundary. Instead, a smoother mix is obtained
between the consecutive filters. Informal listening experiments suggested that
the mixing mitigates the problem of periodic disturbances even when using
uniform segmentation. Sound examples of the noise sequences shown in Fig. 9(b)
and Fig. 9(c) are available online2.

##  5 EVALUATION

(a)

(b)

(c)

(d)

(e)

(f)

Figure 10:  The left-column sub-figures show the spectrograms of the Promedi
Hall and their modeled counterparts: (a) Target IR, and estimated IRs based on
(c) the FVN model and (e) the extended DVN model. The T60 estimates of the
target IR (dashed) and the modeled IRs (solid) are overlayed on the
spectrograms, indicating a better fit using the proposed method. The right-
column sub-figures present (b) the target IR, (d) the FVN model instance IR,
and (f) the extended DVN model instance IR. The direct part and the early
reflections of the IR, both of which are not modeled, are shown in gray.

In this section, an objective evaluation of the extended DVN reverberation
algorithm is presented. We applied the method to model the late-reverberation
part of two target IRs from different spaces. The early IR parts containing
the direct sound and the early reflections were not modeled but directly
adopted from the original data. The two spaces have distinctive acoustical
characteristics and were selected to provide challenging conditions to test
the generalizability of the proposed reverberation algorithm.

###  5.1 Target Impulse Responses

The first of the two target IRs is the high-quality measurement of the IR of
the Promenadi concert hall in Pori, Finland, conducted by Merimaa et al. [37].
We modeled the late part of the IR after $110$ ms as a test case to allow a
direct comparison with the previously proposed advanced FVN model [33].
Välimäki et al. [33] determined the late reverberation of the Pori Hall IR to
start after $110$ ms based on preliminary testing. The second target IR has
been recorded in Creswell Crags [39], where the IR has a strong second echo.
The specific IRs we chose from the two datasets are “s1_r3_o.wav” and
“8_r_grundymouth_s_grundypath.wav”.

###  5.2 Modeling Concert-Hall Reverberation

In this section, we compare IRs synthesized with our extended DVN and the
previously proposed FVN [33] objectively to the target IR of the Promenadi
Hall. The objective accuracy was analyzed in terms of spectrogram comparison
and T60 estimation. We parametrized the extended DVN model of the Promenadi
Hall using the configuration described in Sec. 4.

Figs. 10(b) and 10(f) show the target IR of the Promenadi Hall and the IR
synthesized with an optimized instance of the extended DVN model,
respectively. In Fig. 10, the early parts of the IR are shown in gray and the
late reverberation in black. The target IR envelope is visually highly similar
to the envelope of the extended DVN model instance IR. The temporal resolution
of the extended DVN model envelope is determined by the STFT window length,
which we selected to be $85$ ms with $50$% overlap. Lengthening the STFT
window would result in a smoother envelope. The previously proposed advanced
FVN [33] model intance IR, shown in Fig. 10(d), has a slightly longer decay
compared to the target IR envelope.

The estimated T60 curves are overlayed on the spectrograms in Fig. 10, where
the solid line shows the T60 of the models and the dashed line shows the
target T60. The mean and maximum T60 errors measured in the frequency range of
20–16 000 Hz of the FVN model instance are $14$% and $28$%, respectively. The
extended DVN model instance in Fig. 10(e) follows the frequency-dependent
decay of the target IR well across all frequencies with mean and maximum T60
errors of $4$% and $8$%, respectively.

Overall, the extended DVN model instance achieved a better spectro-temporal
fit than the FVN model instance using only ten dictionary filters, whereas the
advanced FVN model uses 20 coloration filters. This implies that the proposed
method is both more accurate and more efficient in modeling a late-
reverberation IR than the previously proposed FVN method.

###  5.3 Modeling Non-exponential Reverberation

We evaluate objectively the accuracy of the synthesized IR of the extended DVN
model instance in modeling the Cresswell Crags target IR. The target IR and
its spectrogram are shown in Fig. 11(b) and Fig. 11(a), respectively. There is
no other parametric method that can successfully model target IRs containing a
single echo which results in two exponential energy decays. The analysis
presented in this section is intended to demonstrate the variable capabilities
of the proposed method.

The extended DVN model uses ten second-order allpole dictionary filters, a
single $12$th-order post-filter, as well as two cascaded first-order DC
blockers. We increased the order of the post-filter components to obtain a
better spectral fit to the target IR. The analysis methods are identical to
the ones used in Sec. 5.2, except for the T60 analysis, which is not a
meaningful metric for the target IR in question. For this example, the
extended DVN was used to synthesize the entire target IR except for the direct
sound (first $1$ ms), since the target IR does not show any prominent early
reflections.

(a)

(b)

(c)

(d)

Figure 11: Spectrogram of the Creswell Crags (a) target IR and (c) the
extended DVN model. Energy (in dB) of the Cresswell Crags (b) target IR and
(d) the corresponding extended DVN model. The late and early reflections,
after the direct sound ($1$ ms), are modeled.

Fig. 11(c) shows the spectrogram of the extended DVN model instance IR. The
DVN model provides an accurate approximation of the overall shape of the
target spectrogram. However, a slightly longer decay is seen in Fig. 11(c)
around $1$ kHz. The largest difference between the target and model instance
spectrograms is seen at low frequencies, at 100 Hz and just above it, where
the DC blocker filters out the low-frequency noise visible in the target
spectrogram of Fig. 11(a).

The IR of the extended DVN model instance in Fig. 11(d) now features a more
detailed energy-decay envelope compared to that of the concert hall
reverberation DVN model instance in Fig. 10(f). This is due to the smaller
frame length ($5.3$ ms) with $50$% overlap for the STFT analysis. The smaller
frame length and hop size are beneficial in modeling the more complex double-
stage decay of the Crasswell Crags target reverberation. The envelope shape of
the extended DVN model instance in Fig. 11(d) follows well that of the target
IR in Fig. 11(b). In summary, this design example demonstrates the ability of
the proposed method to model a non-exponential IR.

###  5.4 Modification of the Proposed Model

One of the benefits of the parametric extended DVN is its flexibility
concerning modification possibilities of the synthesized IR. The direct
modification of IRs used in convolution reverberation has been proposed by
Canfield-Dafilou and Abel [40] to change the perceived room size of the
original IR. Modifications such as time-stretching and temporal envelope
modification have been demonstrated already using the previously proposed FVN
algorithm [32, 33].

The extended DVN lends itself just as well to time stretching and envelope
modifications. Furthermore, the achievable level of detail in envelope
modifications with the extended DVN algorithm is vaster than with the
previously proposed FVN algorithm, since the former method relies on pulses
scaled by individual gains. The choice of envelope resolution is a trade-off
between computational cost and envelope detail. Time stretching with the
extended DVN is implemented by changing the length of the generated velvet-
noise sequence and interpolating the probability matrix $\bm{P}$ to match the
modified pulse locations. In the following examples, we highlight three more
modifications achievable with the extended DVN algorithm.

Fig. 12 shows the spectrograms of three modifications of the Promenadi Hall
extended DVN model instance. Fig. 12(a) illustrates the effect of the gated
reverb modification made to the extended DVN model instance IR. Since the
extended DVN is implemented as an FIR structure, generating a gated reverb is
achieved simply by truncating part of the delay-line coefficients from the end
of the model instance IR.

In Fig. 12(b), we slowed down the spectral change relative to the original
model instance of Fig. 10(e). With the proposed method, the relative spectral
change of the late reverberation can be adjusted by manipulating the
probability matrix $\bm{P}$, cf. Eq. (17). The slowing down of the decay was
achieved by taking a subset of columns from the beginning of the original
probability matrix, i.e., the matrix

$\displaystyle\begin{aligned}
\bm{P}_{\alpha}=\begin{bmatrix}\bm{p}(1)&\bm{p}(2)&\dots\bm{p}(\hat{T)}\\\
\end{bmatrix},\end{aligned}$ (19)

where $\hat{T}=\lfloor T\alpha\rceil$ and $0\leq\alpha\leq 1$. We then
interpolated the new matrix $\bm{P}_{\alpha}$ to fit the original pulse
locations. In Fig. 12(b), the resulting T60 estimate is overlayed on the
spectrogram, where the dashed line shows the original target T60 and the solid
line shows the modified T60, where the high frequencies ring considerably
longer than in the target.

The final modification example is the time-reversed reverb effect, which has
two versions. Since the spectral change is parametrized by the probability
matrix $\bm{P}$ and the energy decay with $g(m)$ of Eq. (18), they are
decoupled. In Fig. 12(c), we have time reversed the filter list $\phi$, cf.
Eq. (7), of the extended DVN model instance. The resulting IR retains its
overall energy decay, but the spectral change is now from dark to bright.
Implementing the opposite version of the two flip operations is also a
possibility. Flipping the decay envelope while using the original filter list
results in an IR whose energy rises while retaining the original spectral
change, as shown in Fig. 12(d). Sound examples of the modifications presented
in Fig. 12 are provided together with the Matlab application that can recreate
them2.

(a)

(b)

(c)

(d)

Figure 12:  Spectrograms of the modified Promenadi Hall extended DVN model
instance with (a) gated reverb, (b) modified T60 (compared to the original in
Fig. 10(e)), (c) reversed spectral evolution, and (d) reversed decay. The
modifications are applied only to the late part, which is modeled, except for
the reversed decay, where the early parts are also reversed and placed after
the modeled part. The T60 estimates of the original target IR (dashed) and the
modified model IR (solid) are overlayed on top of the spectrogram in (b).

The brightening effect and the gated effect can be implemented also with
recursive algorithms given that an accurate model is first obtained. However,
the gated reverb requires either a second parallel recursive algorithm to be
run to cancel the remaining part of the IR [25] or, alternatively, an
additional noise gate to mute the output of the reverberator. On the other
hand, the reversed energy decay is impossible to implement in a recursive
manner, since it requires growth of energy over time, which would result in
instability in a recursive implementation.

##  6 CONCLUSION

In this paper, an extension of the previously proposed DVN algorithm was
presented. We developed a parametric reverberator based on the extended DVN
structure utilizing sparse velvet-noise convolution. By replacing the square
pulses of the previously proposed DVN method with arbitrary dictionary filters
and setting the probability of each dictionary filter as a free parameter, the
extended DVN model can be fitted to a target IR via NNLS optimization. We
additionally showed that a previously proposed FVN reverberation algorithm
[33] is a special case of the proposed extended DVN reverberator.

We evaluated the proposed method objectively and demonstrated its capability
to accurately model the late reverberation of two distinctly different spaces,
a large concert hall and a coupled space in which sound decays in a non-
exponential manner. No previously proposed parametric reverberation algorithm
exists for modeling the latter target IR. We assessed the spectro-temporal fit
of optimized instances of the proposed extended DVN model to the two target
IRs objectively, and the proposed method produced an accurate model in both
cases. For the concert hall target IR, the extended DVN yielded a better
spectro-temporal fit than the previously proposed FVN algorithm, when using
half the number of filters. We demonstrated the method’s capability by
providing practically relevant examples of model IR modification, such as
slowing down the decay in a frequency-dependent way and gating the model IR.

All in all, the proposed extended DVN reverberator is applicable to
synthesizing late reverberation of various spaces while providing perceptually
meaningful parametric control. Future work could investigate a more general
approach to specify a broad set of dictionary filters and finding a sparse
solution, instead of using the empirically designed filters.

##  7 ACKNOWLEDGMENT

This research is part of the activities of the Nordic Sound and Music
Computing Network—NordicSMC (NordForsk project no. 86892).

## References

  * [1] M. R. Schoeder, “Natural Sounding Artificial Reverberation,” _J. Audio Eng. Soc._ , vol. 10, no. 3, pp. 219–223 (1962 Jul.).
  * [2] V. Välimäki, J. D. Parker, L. Savioja, J. O. Smith, and J. S. Abel, “Fifty Years of Artificial Reverberation,” _IEEE Trans. Audio Speech Lang. Process._ , vol. 20, no. 5, pp. 1421–1448 (2012 Jul.), https://doi.org/10.1109/TASL.2012.2189567.
  * [3] J. A. Moorer, “About This Reverberation Business,” _Comput. Music J._ , vol. 3, no. 2, pp. 13–28 (1979 Jun.), https://doi.org/10.2307/3680280.
  * [4] P. Rubak and L. G. Johansen, “Artificial Reverberation Based on a Pseudo-Random Impulse Response: Part I,” presented at the _104th Convention of the Audio Engineering Society_ (1998 May), paper 4725.
  * [5] M. Karjalainen and H. Järveläinen, “Reverberation Modeling Using Velvet Noise,” in _Proceedings of the 30th AES International Conference on Intelligent Audio_ (Saariselkä, Finland) (2007 Mar.).
  * [6] K. Spratt and J. S. Abel, “A Digital Reverberator Modeled After the Scattering of Acoustic Waves by Trees in a Forrest,” presented at the _125th Convention of the Audio Engineering Society_ (2008 Oct.), paper 7650.
  * [7] F. Stevens, D. T. Murphy, L. Savioja, and V. Välimäki, “Modeling Sparsely Reflecting Outdoor Acoustic Scenes Using the Waveguide Web,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , vol. 25, no. 8, pp. 1566–1578 (2017 Aug.), https://doi.org/10.1109/TASLP.2017.2699424.
  * [8] C. F. Eyring, “Reverberation Time Measurements in Coupled Rooms,” _J. Acoust. Soc. Am._ , vol. 3, no. 2, pp. 181–206 (1931 Oct.).
  * [9] O. Das and J. S. Abel, “Grouped Feedback Delay Networks for Modeling of Coupled Spaces,” _J. Audio Eng. Soc._ , vol. 69, no. 7/8, pp. 486–496 (2021 Jul.), https://doi.org/10.17743/jaes.2021.0026.
  * [10] C. Kirsch, T. Wendt, S. Van De Par, H. Hu, and S. D. Ewert, “Computationally-Efficient Simulation of Late Reverberation for Inhomogeneous Boundary Conditions and Coupled Rooms,” _J. Audio Eng. Soc._ , vol. 71, no. 4, pp. 186–201 (2023 Apr.), https://doi.org/10.17743/jaes.2022.0053.
  * [11] R. Fink, M. Latour, and Z. Wallmark, _The Relentless Pursuit of Tone: Timbre in Popular Music_ (Oxford University Press, 2018).
  * [12] T. Rossing, R. Moore, and P. Wheeler, _The Science of Sound, 3rd ed._ , chap. Auditorium Acoustics, pp. 525–545 (Addison-Wesley, London, U.K.) (2002).
  * [13] P. Rubak and L. G. Johansen, “Artificial Reverberation Based on a Pseudo-Random Impulse Response: Part II,” presented at the _106th Convention of the Audio Enginering Society_ (1999 May), paper 4900.
  * [14] V. Välimäki, H.-M. Lehtonen, and M. Takanen, “A perceptual Study on Velvet Noise and Its Variants at Different Pulse Densities,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , vol. 21, no. 7, pp. 1481–1488 (2013 Jul.), https://doi.org/10.1109/TASL.2013.2255281.
  * [15] K. Lee, J. Abel, V. Välimäki, T. Stilson, and D. P. Berners, “The Switched Convolution Reverberator,” _J. Audio Eng. Soc._ , vol. 60, no. 4, pp. 227–236 (2012 Apr.).
  * [16] V. Välimäki and K. Prawda, “Late-Reverberation Synthesis Using Interleaved Velvet-Noise Sequences,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , vol. 29, p. 1149–1160 (2021 Feb.), https://doi.org/10.1109/TASLP.2021.3060165.
  * [17] H. Fastl and E. Zwicker, _Psychoacoustics – Facts and Models_ (Springer, 2007).
  * [18] N. Meyer-Kahlen, S. J. Schlecht, and T. Lokki, “Perceptual Roughness of Spatially Assigned Sparse Noise for Rendering Reverberation,” _J. Acoust. Soc. Am._ , vol. 150, no. 5, pp. 3521–3531 (2021 Nov.), https://doi.org/10.1121/10.0007048.
  * [19] J.-M. Jot, “An Analysis/Synthesis Approach to Real-time Artificial Reverberation,” in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing_ , vol. 2, pp. 221–224 (1992 Mar.), https://doi.org/10.1109/ICASSP.1992.226080.
  * [20] B. Alary, A. Politis, S. Schlecht, and V. Välimäki, “Directional Feedback Delay Network,” _J. Audio Eng. Soc._ , vol. 67, no. 10, pp. 752–762 (2019 Oct.), https://doi.org/10.17743/jaes.2019.0026.
  * [21] K. Prawda, V. Välimäki, and S. J. Schlecht, “Improved Reverberation Time Control for Feedback Delay Networks,” in _Proceedings of the International Conference on Digital Audio Effects (DAFx)_ (Birmingham, UK) (2019 Sep.).
  * [22] S. J. Schlecht and E. A. P. Habets, “Scattering in Feedback Delay Networks,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , vol. 28, p. 1915–1924 (2020 Jun.), https://doi.org/10.1109/TASLP.2020.3001395.
  * [23] O. Das, J. S. Abel, and E. K. Canfield-Dafilou, “Delay Network Architectures for Room and Coupled Space Modeling,” in _Proceedings of the International Conference on Digital Audio Effects (DAFx)_ , pp. 234–241 (Vienna, Austria) (2020 Sep.).
  * [24] J. Fagerström, B. Alary, S. J. Schlecht, and V. Välimäki, “Velvet-Noise Feedback Delay Network,” in _Proceedings of the International Conference on Digital Audio Effects (DAFx)_ , pp. 219–226 (Vienna, Austria) (2020 Sep.).
  * [25] E. Piirilä, T. Lokki, and V. Välimäki, “Digital Signal Processing Techniques for Non-exponentially Decaying Reverberation,” in _Proceedings of the COST-G6 Workshop on Digital Audio Effects_ , pp. 21–24 (Barcelona, Spain) (1998 Nov.).
  * [26] K.-S. Lee and J. S. Abel, “A Reverberator with Two-Stage Decay and Onset Time Controls,” presented at the _Audio Engineering Society 129th Convention_ (2010 Nov.), paper 10287.
  * [27] N. Meyer-Kahlen, S. J. Schlecht, and T. Lokki, “Fade-in Control for Feedback Delay Networks,” in _Proceedings of the International Conference on Digital Audio Effects (DAFx)_ , pp. 227–233 (Vienna, Austria) (2020 Sep.).
  * [28] M. Karjalainen and H. Järveläinen, “More About This Reverberation Science: Perceptually Good Late Reverberation,” presented at the _111th Convention of the Audio Engineering Society_ (2001 Sep.), paper 5415.
  * [29] J. S. Abel, S. Coffin, and K. Spratt, “A Modal Architecture for Artificial Reverberation with Application to Room Acoustics Modeling,” presented at the _137th Convention of the Audio Engineering Society_ (2014 Oct.), paper 9208.
  * [30] J. J. Wells, “Modal Decompositions of Impulse Responses for Parametric Interaction,” _J. Audio Eng. Soc._ , vol. 69, no. 7/8, pp. 530–541 (2021 Jul.), https://doi.org/10.17743/jaes.2021.0027.
  * [31] C. Hold, T. McKenzie, G. Götz, S. J. Schlecht, and V. Pulkki, “Resynthesis of Spatial Room Impulse Response Tails with Anisotropic Multi-Slope Decays,” _J. Audio Eng. Soc._ , vol. 70, no. 6, pp. 526–538 (2022 Jun.).
  * [32] B. Holm-Rasmussen, H.-M. Lehtonen, and V. Välimäki, “A New Reverberator Based on Variable Sparsity Convolution,” in _Proceedings of the International Conference on Digital Audio Effects (DAFx)_ , pp. 344–350 (Maynooth, Ireland) (2013 Sep.).
  * [33] V. Välimäki, B. Holm-Rasmussen, B. Alary, and H.-M. Lehtonen, “Late Reverberation Synthesis Using Filtered Velvet Noise,” _Appl. Sci._ , vol. 7, no. 5 (2017 May), https://doi.org/10.3390/app7050483.
  * [34] J. Fagerström, N. Meyer-Kahlen, S. J. Schlecht, and V. Välimäki, “Dark Velvet Noise,” in _Proceedings of the International Conference on Digital Audio Effects (DAFx)_ , pp. 192–199 (Vienna, Austria) (2022 Sep.).
  * [35] C. L. Lawson and R. J. Hanson, “Linear Least Squares with Linear Inequality Constraints,” in _Solving Least-Squares Problems_ , p. 161 (Prentice Hall, Upper Saddle River, NJ, 1974).
  * [36] N. Meyer-Kahlen, S. J. Schlecht, and V. Välimäki, “Colours of Velvet Noise,” _Electron. Lett._ , vol. 58, no. 12, pp. 495–497 (2022 Jun.), https://doi.org/10.1049/ell2.12501.
  * [37] J. Merimaa, T. Peltonen, and T. Lokki, _Concert Hall Impulse Responses—Pori, Finland_ (http://legacy.spa.aalto.fi/projects/poririrs/, accessed June 1, 2023).
  * [38] J. Pekonen and V. Välimäki, “Filter-Based Alias Reduction for Digital Classical Waveform Synthesis,” in _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing_ , pp. 133–136 (Las Vegas, NV) (2008 May), https://doi.org/10.1109/ICASSP.2008.4517564.
  * [39] D. T. Murphy and S. Shelley, “OpenAIR: An Interactive Auralization Web Resource and Database,” presented at the _129th Convention of the Audio Engineering Society_ (2010 Nov.), paper 8226.
  * [40] E. K. Canfield-Dafilou and J. S. Abel, “Resizing Rooms in Convolution, Delay Network, and Modal Reverberators,” in _Proceedings of the International Conference on Digital Audio Effects (DAFx)_ , pp. 229–236 (Aveiro, Portugal) (2018 Sep.).

Jon Fagerströmfagerstrom.jpegJon Fagerström received his M.Sc. degree in
electrical engineering, majoring in acoustics and audio technology, from Aalto
University, Espoo, Finland, in 2020. He is currently working toward a doctoral
degree at the Acoustics Lab, Aalto University. His research interests include
sparse-noise modeling, decorrelation filters, artificial reverberation, and
reverberation perception.

Sebastian J. Schlechtsebastian.jpgSebastian J. Schlecht is a Professor of
Practice for Sound in Virtual Reality at the Acoustics Lab, Department of
Information and Communications Engineering and Media Labs, Department of Art
and Media, of Aalto University, Finland. He received the Diploma in Applied
Mathematics from the University of Trier, Germany, in 2010 and an M.Sc. degree
in Digital Music Processing from the School of Electronic Engineering and
Computer Science at Queen Mary University of London, UK, in 2011. In 2017, he
received a Doctoral degree at the International Audio Laboratories Erlangen,
Germany, on artificial spatial reverberation and reverberation enhancement
systems. From 2012 to 2019, Dr. Schlecht was also external research and
development consultant and lead developer of the 3D Reverb algorithm at the
Fraunhofer IIS, Erlangen, Germany.

Vesa VälimäkiVesa_Valimaki.jpgVesa Välimäki received his D.Sc. degree in
electrical engineering from the Helsinki University of Technology, Espoo,
Finland, in 1995. In 1996, he was a Postdoctoral Researcher at the University
of Westminster, London, UK. In 2001–2002, he was a Professor of signal
processing at the Pori unit of the Tampere University of Technology, Finland.
In 2008–2009, he was a Visiting Scholar at Stanford University. He is a Full
Professor of audio signal processing and the Vice Dean for Research in
electrical engineering at Aalto University. He is a Fellow of the Institute of
Electrical and Electronics Engineers (IEEE). In 2015–2020, he was a Senior
Area Editor of the IEEE/ACM Transactions on Audio, Speech, and Language
Processing. Since 2020, Prof. Välimäki has been the Editor-in-Chief of the
Journal of the Audio Engineering Society.


* [8] N. Brambilla, S. Eidelman, C. Hanhart, A. Nefediev, C. P. Shen, C. E. Thomas, A. Vairo and C. Z. Yuan, The $XYZ$ states: experimental and theoretical status and perspectives, Phys. Rept. 873, 1-154 (2020).
  * [9] H. X. Chen, W. Chen, X. Liu, Y. R. Liu and S. L. Zhu, An updated review of the new hadron states, Rept. Prog. Phys. 86, 026201 (2023).
  * [10] H. X. Chen, W. Chen, X. Liu and S. L. Zhu, The hidden-charm pentaquark and tetraquark states, Phys. Rept. 639, 1-121 (2016).
  * [11] Y. R. Liu, H. X. Chen, W. Chen, X. Liu and S. L. Zhu, Pentaquark and Tetraquark states, Prog. Part. Nucl. Phys. 107, 237-320 (2019).
  * [12] S. L. Olsen, T. Skwarnicki and D. Zieminska, Nonstandard heavy mesons and baryons: Experimental evidence, Rev. Mod. Phys. 90, 015003 (2018).
  * [13] R. F. Lebed, R. E. Mitchell and E. S. Swanson, Heavy-Quark QCD Exotica, Prog. Part. Nucl. Phys. 93, 143-194 (2017).
  * [14] A. Esposito, A. Pilloni and A. D. Polosa, Multiquark Resonances, Phys. Rept. 668, 1-97 (2017).
  * [15] A. Ali, J. S. Lange and S. Stone, Exotics: Heavy Pentaquarks and Tetraquarks, Prog. Part. Nucl. Phys. 97, 123-198 (2017).
  * [16] X. K. Dong, F. K. Guo and B. S. Zou, A survey of heavy–heavy hadronic molecules, Commun. Theor. Phys. 73, no.12, 125201 (2021).
  * [17] E. Eichten, K. Gottfried, T. Kinoshita, J. B. Kogut, K. D. Lane and T. M. Yan, The Spectrum of Charmonium, Phys. Rev. Lett. 34, 369-372 (1975) [erratum: Phys. Rev. Lett. 36, 1276 (1976)].
  * [18] E. Eichten, K. Gottfried, T. Kinoshita, K. D. Lane and T. M. Yan, Charmonium: The Model, Phys. Rev. D 17, 3090 (1978) [erratum: Phys. Rev. D 21, 313 (1980)].
  * [19] S. Godfrey and N. Isgur, Mesons in a Relativized Quark Model with Chromodynamics, Phys. Rev. D 32, 189-231 (1985).
  * [20] T. Barnes, S. Godfrey and E. S. Swanson, Higher charmonia, Phys. Rev. D 72, 054026 (2005).
  * [21] D. Ebert, R. N. Faustov and V. O. Galkin, Spectroscopy and Regge trajectories of heavy quarkonia and $B_{c}$ mesons, Eur. Phys. J. C 71, 1825 (2011).
  * [22] K. Heikkila, S. Ono and N. A. Tornqvist, Heavy $c\bar{c}$ and $b\bar{b}$ quarkonium states and unitarity effects, Phys. Rev. D 29, 110 (1984) [erratum: Phys. Rev. D 29, 2136 (1984)].
  * [23] Y. S. Kalashnikova, Coupled-channel model for charmonium levels and an option for $X(3872)$, Phys. Rev. D 72, 034010 (2005).
  * [24] T. Barnes and E. S. Swanson, Hadron loops: General theorems and application to charmonium, Phys. Rev. C 77, 055206 (2008) [arXiv:0711.2080 [hep-ph]].
  * [25] M. R. Pennington and D. J. Wilson, Decay channels and charmonium mass-shifts, Phys. Rev. D 76, 077502 (2007).
  * [26] B. Q. Li, C. Meng and K. T. Chao, Coupled-Channel and Screening Effects in Charmonium Spectrum, Phys. Rev. D 80, 014012 (2009) [arXiv:0904.4068 [hep-ph]].
  * [27] J. Ferretti, G. Galatà and E. Santopinto, Interpretation of the X(3872) as a charmonium state plus an extra component due to the coupling to the meson-meson continuum, Phys. Rev. C 88, 015207 (2013) [arXiv:1302.6857 [hep-ph]].
  * [28] Z. Y. Zhou and Z. Xiao, Comprehending heavy charmonia and their decays by hadron loop effects, Eur. Phys. J. A 50, 165 (2014).
  * [29] Y. Lu, M. N. Anwar and B. S. Zou, $X(4260)$ Revisited: A Coupled Channel Perspective, Phys. Rev. D 96, 114022 (2017).
  * [30] J. Ferretti and E. Santopinto, Threshold corrections of $\chi_{c}(2P)$ and $\chi_{b}(3P)$ states and $J/\psi\rho$ and $J/\psi\omega$ transitions of the $\chi(3872)$ in a coupled-channel model, Phys. Lett. B 789, 550-555 (2019).
  * [31] H. F. Fu and L. Jiang, Coupled-channel-induced $S{-}D$ mixing of Charmonia and testing possible assignments for $Y$(4260) and $Y$(4360), Eur. Phys. J. C 79, 460 (2019).
  * [32] P. G. Ortega, D. R. Entem and F. Fernández, Unquenching the Quark Model in a Nonperturbative Scheme, Adv. High Energy Phys. 2019, 3465159 (2019).
  * [33] J. Ferretti, E. Santopinto, M. N. Anwar and Y. Lu, Quark structure of the $\chi_{\mathrm{c}}(3P)$ and $X(4274)$ resonances and their strong and radiative decays, Eur. Phys. J. C 80, 464 (2020).
  * [34] J. Ferretti and E. Santopinto, Quark Structure of the $X(4500)$, $X(4700)$ and $\chi_{c}(4P,5P)$ States, Front. in Phys. 9, 76 (2021).
  * [35] M. N. Anwar and Y. Lu, Heavy quark spin partners of the Y(4260) in coupled-channel formalism, Phys. Rev. D 104, 094006 (2021).
  * [36] M. X. Duan and X. Liu, Where are 3P and higher P-wave states in the charmonium family?, Phys. Rev. D 104, 074010 (2021).
  * [37] R. Bruschini and P. González, Diabatic description of charmoniumlike mesons II: mass corrections and strong decay widths, Phys. Rev. D 103, 074009 (2021).
  * [38] S. Kanwal, F. Akram, B. Masud and E. S. Swanson, Charmonium spectrum in an unquenched quark model, Eur. Phys. J. A 58, 219 (2022).
  * [39] X. Chen, Y. Tan and Y. Yang, Charmonium mass shifts in an unquenched quark model, Eur. Phys. J. Plus 138, 653 (2023).
  * [40] Y. B. Ding, K. T. Chao and D. H. Qin, Screened $Q\bar{Q}$ potential and spectrum of heavy quarkonium, Chin. Phys. Lett. 10, 460-463 (1993).
  * [41] Y. B. Dong, Color screening potential and dissolution of heavy quarkonium, Int. J. Mod. Phys. E 5, 643-648 (1996).
  * [42] B. Q. Li and K. T. Chao, Higher Charmonia and X,Y,Z states with Screened Potential, Phys. Rev. D 79, 094004 (2009).
  * [43] W. J. Deng, H. Liu, L. C. Gui and X. H. Zhong, Charmonium spectrum and their electromagnetic transitions with higher multipole contributions, Phys. Rev. D 95, 034026 (2017).
  * [44] J. Z. Wang, D. Y. Chen, X. Liu and T. Matsuki, Constructing $J/\psi$ family with updated data of charmoniumlike $Y$ states, Phys. Rev. D 99, 114003 (2019).
  * [45] J. J. Dudek, R. G. Edwards, N. Mathur and D. G. Richards, Charmonium excited state spectrum in lattice QCD, Phys. Rev. D 77, 034501 (2008).
  * [46] L. Liu et al. [Hadron Spectrum], Excited and exotic charmonium spectroscopy from lattice QCD, JHEP 07, 126 (2012).
  * [47] C. DeTar, Charmonium Spectroscopy from Lattice QCD, Int. J. Mod. Phys. Conf. Ser. 02, 31-35 (2011).
  * [48] G. S. Bali, S. Collins and C. Ehmann, Charmonium spectroscopy and mixing with light quark and open charm states from $n_{F}$=2 lattice QCD, Phys. Rev. D 84, 094506 (2011).
  * [49] C. B. Lang, L. Leskovec, D. Mohler and S. Prelovsek, Vector and scalar charmonium resonances with lattice QCD, JHEP 09, 089 (2015).
  * [50] D. J. Wilson, C. E. Thomas, J. J. Dudek and R. G. Edwards, Scalar and tensor charmonium resonances in coupled-channel scattering from QCD, [arXiv:2309.14070 [hep-lat]].
  * [51] D. J. Wilson, C. E. Thomas, J. J. Dudek and R. G. Edwards, Charmonium $\chi_{c0}$ and $\chi_{c2}$ resonances in coupled-channel scattering from lattice QCD, [arXiv:2309.14071 [hep-lat]].
  * [52] R. H. Ni, J. J. Wu and X. H. Zhong, Unified unquenched quark model for heavy-light mesons with chiral dynamics, [arXiv:2312.04765 [hep-ph]].
  * [53] L. Micu, Decay rates of meson resonances in a quark model, Nucl. Phys. B 10, 521-526 (1969).
  * [54] A. Le Yaouanc, L. Oliver, O. Pene and J. C. Raynal, Naive quark pair creation model of strong interaction vertices, Phys. Rev. D 8, 2223-2234 (1973).
  * [55] A. Le Yaouanc, L. Oliver, O. Pene and J. C. Raynal, Naive quark pair creation model and baryon decays, Phys. Rev. D 9, 1415-1419 (1974).
  * [56] D. Morel and S. Capstick, Baryon meson loop effects on the spectrum of nonstrange baryons, [arXiv:nucl-th/0204014 [nucl-th]].
  * [57] B. Silvestre- Brac and C. Gignoux, Unitary effects in spin orbit splitting of P wave baryons, Phys. Rev. D 43, 3699-3708 (1991).
  * [58] P. G. Ortega, J. Segovia, D. R. Entem and F. Fernandez, Molecular components in P-wave charmed-strange mesons, Phys. Rev. D 94, no.7, 074037 (2016).
  * [59] P. G. Ortega, J. Segovia, D. R. Entem and F. Fernández, Threshold effects in P-wave bottom-strange mesons, Phys. Rev. D 95, no.3, 034010 (2017).
  * [60] Z. Yang, G. J. Wang, J. J. Wu, M. Oka and S. L. Zhu, Novel Coupled Channel Framework Connecting the Quark Model and Lattice QCD for the Near-threshold Ds States, Phys. Rev. Lett. 128, 112001 (2022).
  * [61] Z. Yang, G. J. Wang, J. J. Wu, M. Oka and S. L. Zhu, The investigations of the P-wave Bs states combining quark model and lattice QCD in the coupled channel framework, JHEP 01, 058 (2023).
  * [62] H. H. Zhong, R. H. Ni, M. Y. Chen, X. H. Zhong and J. J. Xie, Further study of $\Omega^{*}|1P_{3/2^{-}}\rangle$ within a chiral quark model, Chin. Phys. C 47, 063104 (2023).
  * [63] S. Godfrey, Spectroscopy of $B_{c}$ mesons in the relativized quark model, Phys. Rev. D 70, 054017 (2004).
  * [64] Q. Li, M. S. Liu, L. S. Lu, Q. F. Lü, L. C. Gui and X. H. Zhong, Excited bottom-charmed mesons in a nonrelativistic quark model, Phys. Rev. D 99, 096020 (2019).
  * [65] E. Hiyama, Y. Kino and M. Kamimura, Gaussian expansion method for few-body systems, Prog. Part. Nucl. Phys. 51, 223-307 (2003).
  * [66] Z. Y. Zhou and Z. Xiao, Hadron loops effect on mass shifts of the charmed and charmed-strange spectra, Phys. Rev. D 84, 034023 (2011).
  * [67] R. H. Ni, Q. Li and X. H. Zhong, Mass spectra and strong decays of charmed and charmed-strange mesons, Phys. Rev. D 105, 056006 (2022).
  * [68] Q. li, R. H. Ni and X. H. Zhong, Towards establishing an abundant $B$ and $B_{s}$ spectrum up to the second orbital excitations, Phys. Rev. D 103, 116010 (2021).
  * [69] W. J. Deng, H. Liu, L. C. Gui and X. H. Zhong, Spectrum and electromagnetic transitions of bottomonium, Phys. Rev. D 95, 074002 (2017).
  * [70] Q. Li, L. C. Gui, M. S. Liu, Q. F. Lü and X. H. Zhong, Mass spectrum and strong decays of strangeonium in a constituent quark model, Chin. Phys. C 45, 023116 (2021).
  * [71] B. Aubert et al. [BaBar], Exclusive Initial-State-Radiation Production of the $D\bar{D}$, $D^{*}\bar{D}$, and $D^{*}\bar{D}^{*}$ Systems, Phys. Rev. D 79, 092001 (2009).
  * [72] K. Abe et al. [Belle], Measurement of the near-threshold $e^{+}e^{-}\to D^{(*)\pm}D^{(*)\mp}$ cross section using initial-state radiation, Phys. Rev. Lett. 98, 092001 (2007).
  * [73] J. Z. Wang and X. Liu, Identifying a characterized energy level structure of higher charmonium well matched to the peak structures in $e^{+}e^{-}\to\pi^{+}D^{0}D^{*-}$, [arXiv:2306.14695 [hep-ph]].
  * [74] J. Z. Wang and X. Liu, Confirming the existence of a new higher charmonium $\psi(4500)$ by the newly released data of $e^{+}e^{-}\to K^{+}K^{-}J/\psi$, Phys. Rev. D 107, 054016 (2023).
  * [75] L. C. Gui, L. S. Lu, Q. F. Lü, X. H. Zhong and Q. Zhao, Strong decays of higher charmonium states into open-charm meson pairs, Phys. Rev. D 98, 016010 (2018).
  * [76] X. L. Wang et al. [Belle], Measurement of $e^{+}e^{-}\to\pi^{+}\pi^{-}\psi(2S)$ via Initial State Radiation at Belle, Phys. Rev. D 91, 112007 (2015).
  * [77] M. Ablikim et al. [BESIII], Observation of a vector charmoniumlike state at 4.7 ${\rm GeV}/c^{2}$ and search for $Z_{cs}$ in $e^{+}e^{-}\to K^{+}K^{-}J/\psi$, Phys. Rev. Lett. 131, 211902 (2023).
  * [78] G. J. Ding, J. J. Zhu and M. L. Yan, Canonical Charmonium Interpretation for Y(4360) and Y(4660), Phys. Rev. D 77, 014033 (2008).
  * [79] J. Segovia, A. M. Yasser, D. R. Entem and F. Fernandez, $J^{PC}=1^{--}$ hidden charm resonances, Phys. Rev. D 78, 114033 (2008).
  * [80] Z. Zhao, K. Xu, A. Limphirat, W. Sreethawong, N. Tagsinsit, A. Kaewsnod, X. Liu, K. Khosonthongkee, S. Cheedket and Y. Yan, Mass spectrum of $1^{--}$ heavy quarkonium, [arXiv:2304.06243 [hep-ph]].
  * [81] M. X. Duan, S. Q. Luo, X. Liu and T. Matsuki, Possibility of charmoniumlike state $X(3915)$ as $\chi_{c0}(2P)$ state, Phys. Rev. D 101, 054029 (2020).
  * [82] X. Liu, Z. G. Luo and Z. F. Sun, X(3915) and X(4350) as new members in P-wave charmonium family, Phys. Rev. Lett. 104, 122001 (2010).
  * [83] R. Aaij et al. [LHCb], Amplitude analysis of the $B^{+}\to D^{+}D^{-}K^{+}$ decay, Phys. Rev. D 102, 112003 (2020).
  * [84] S. Uehara et al. [Belle], Observation of a charmonium-like enhancement in the $\gamma\gamma to\omega J/\psi$ process, Phys. Rev. Lett. 104, 092001 (2010).
  * [85] K. Abe et al. [Belle], Observation of a near-threshold $\omega J/\psi$ mass enhancement in exclusive $B\to K\omega J/\psi$ decays, Phys. Rev. Lett. 94, 182002 (2005).
  * [86] J. P. Lees et al. [BaBar], Study of $X(3915)\to J/\psi\omega$ in two-photon collisions, Phys. Rev. D 86, 072002 (2012).
  * [87] B. Aubert et al. [BaBar], Observation of $Y(3940)$ $\to J/\psi\omega$ in $B\to J/\psi\omega K$ at BABAR, Phys. Rev. Lett. 101, 082001 (2008).
  * [88] F. K. Guo and U. G. Meissner, Where is the $\chi_{c0}(2P)$?, Phys. Rev. D 86, 091501 (2012).
  * [89] K. Chilikin et al. [Belle], Observation of an alternative $\chi_{c0}(2P)$ candidate in $e^{+}e^{-}\rightarrow J/\psi D\bar{D}$, Phys. Rev. D 95, 112003 (2017).
  * [90] Y. F. Wang, U. G. Meißner, D. Rönchen and C. W. Shen, On the nature of the $N^{*}$ and $\Delta$ resonances via coupled-channel dynamics, [arXiv:2307.06799 [nucl-th]].
  * [91] V. Baru, J. Haidenbauer, C. Hanhart, Y. Kalashnikova and A. E. Kudryavtsev, Evidence that the $a_{0}(980)$ and $f_{0}(980)$ are not elementary particles, Phys. Lett. B 586, 53-61 (2004).
  * [92] Y. S. Kalashnikova and A. V. Nefediev, Nature of X(3872) from data, Phys. Rev. D 80, 074004 (2009).
  * [93] F. Giacosa, M. Piotrowska and S. Coito, $X(3872)$ as virtual companion pole of the charm–anticharm state $\chi_{c1}(2P)$, Int. J. Mod. Phys. A 34, 1950173 (2019).
  * [94] P. G. Ortega, J. Segovia, D. R. Entem and F. Fernandez, Coupled channel approach to the structure of the X(3872), Phys. Rev. D 81, 054023 (2010).
  * [95] Z. Y. Zhou and Z. Xiao, Understanding $X(3862)$, $X(3872)$, and $X(3930)$ in a Friedrichs-model-like scheme, Phys. Rev. D 96, 054031 (2017) [erratum: Phys. Rev. D 96, 099905 (2017)].
  * [96] S. Coito, G. Rupp and E. van Beveren, $X(3872)$ is not a true molecule, Eur. Phys. J. C 73, 2351 (2013).
  * [97] C. Meng and K. T. Chao, Decays of the $X(3872)$ and $\chi_{c1}(2P)$ charmonium, Phys. Rev. D 75, 114002 (2007).
  * [98] C. Meng, J. J. Sanz-Cillero, M. Shi, D. L. Yao and H. Q. Zheng, Refined analysis on the X(3872) resonance, Phys. Rev. D 92, no.3, 034020 (2015).
  * [99] J. Ferretti, G. Galatà and E. Santopinto, Quark structure of the $X(3872)$ and $\chi_{b}(3P)$ resonances, Phys. Rev. D 90, no.5, 054010 (2014).
  * [100] Z. Y. Zhou and Z. Xiao, Comprehending Isospin breaking effects of $X(3872)$ in a Friedrichs-model-like scheme, Phys. Rev. D 97, 034011 (2018).
  * [101] G. J. Wang, Z. Yang, J. J. Wu, M. Oka and S. L. Zhu, New insight into the exotic states strongly coupled with the $D\bar{D}^{*}$ from the $T^{+}_{cc}$, [arXiv:2306.12406 [hep-ph]].
  * [102] T. Aaltonen et al. [CDF], Observation of the $Y(4140)$ Structure in the $J/\psi\phi$ Mass Spectrum in $B^{\pm}\to J/\psi\phi K^{\pm}$ Decays, Mod. Phys. Lett. A 32, 1750139 (2017).
  * [103] R. Aaij et al. [LHCb], Observation of New Resonances Decaying to $J/\psi K^{+}$\+ and $J/\psi\phi$, Phys. Rev. Lett. 127, 082001 (2021).
  * [104] R. Aaij et al. [LHCb], Observation of $J/\psi\phi$ structures consistent with exotic states from amplitude analysis of $B^{+}\to J/\psi\phi K^{+}$ decays, Phys. Rev. Lett. 118, 022003 (2017).
  * [105] R. Aaij et al. [LHCb], Amplitude analysis of $B^{+}\to J/\psi\phi K^{+}$ decays, Phys. Rev. D 95, 012002 (2017).
  * [106] Z. H. Wang and G. L. Wang, Two-body strong decays of the 2P and 3P charmonium states, Phys. Rev. D 106, no.5, 054037 (2022).
  * [107] Q. F. Lü and Y. B. Dong, X(4140) , X(4274) , X(4500) , and X(4700) in the relativized quark model, Phys. Rev. D 94, no.7, 074007 (2016).
  * [108] C. P. Shen et al. [Belle], Evidence for a new resonance and search for the $Y(4140)$ in the $\gamma\gamma\to\phi J/\psi$ process, Phys. Rev. Lett. 104, 112004 (2010).
  * [109] R. Aaij et al. [LHCb], First observation of the $B^{+}\to D_{s}^{+}D_{s}^{-}K^{+}$ decay, Phys. Rev. D 108, 034012 (2023).
  * [110] R. Aaij et al. [LHCb], Observation of a Resonant Structure near the Ds+Ds- Threshold in the $B^{+}\to D_{s}^{+}D_{s}^{-}K^{+}$ Decay, Phys. Rev. Lett. 131, 071901 (2023).
  * [111] N. R. Soni, B. R. Joshi, R. P. Shah, H. R. Chauhan and J. N. Pandya, $Q\bar{Q}$ ( $Q\in\\{b,c\\}$ ) spectroscopy using the Cornell potential, Eur. Phys. J. C 78, 592 (2018).
  * [112] R. Chaturvedi and A. K. Rai, Charmonium spectroscopy motivated by general features of pNRQCD, Int. J. Theor. Phys. 59, 3508-3532 (2020).
  * [113] M. A. Sultan, N. Akbar, B. Masud and F. Akram, Higher Hybrid Charmonia in an Extended Potential Model, Phys. Rev. D 90, 054001 (2014).
  * [114] H. Mansour and A. Gamal, Meson spectra using Nikiforov-Uvarov method, Results Phys. 33, 105203 (2022).
  * [115] Z. Y. Fang, Y. R. Wang and C. Q. Pang, $Q\bar{Q}(Q\in\\{b,c\\})$ Spectroscopy Using the Modified Rovibrational Model, Phys. Part. Nucl. Lett. 20, 589-597 (2023).
  * [116] L. Cao, Y. C. Yang and H. Chen, Charmonium states in QCD-inspired quark potential model using Gaussian expansion method, Few Body Syst. 53, 327-342 (2012).
  * [117] R. Aaij et al. [LHCb], Study of $B^{0}_{s}\to J\psi\pi^{+}\pi^{-}K^{+}K^{-}$ decays, JHEP 02, 024 (2021) [erratum: JHEP 04, 170 (2021)].
  * [118] M. Ablikim et al. [BESIII], Search for new decay modes of the $\psi_{2}(3823)$ and the process $e^{+}e^{-}\rightarrow\pi^{0}\pi^{0}\psi_{2}(3823)$, Phys. Rev. D 103, L091102 (2021).
  * [119] E. J. Eichten, K. Lane and C. Quigg, B Meson Gateways to Missing Charmonium Levels, Phys. Rev. Lett. 89, 162002 (2002).
  * [120] H. Xu, X. Liu and T. Matsuki, Understanding $B^{-}\rightarrow X(3823)K^{-}$ via rescattering mechanism and predicting $B^{-}\to\eta_{c2}(^{1}D_{2})/\psi_{3}(^{3}D_{3})K^{-}$, Phys. Rev. D 94, no.3, 034005 (2016).
  * [121] Y. Fan, Z. G. He, Y. Q. Ma and K. T. Chao, Predictions of Light Hadronic Decays of Heavy Quarkonium $1D_{2}$ States in NRQCD, Phys. Rev. D 80, 014001 (2009).
  * [122] E. J. Eichten, K. Lane and C. Quigg, New states above charm threshold, Phys. Rev. D 73, 014014 (2006) [erratum: Phys. Rev. D 73, 079903 (2006)].
  * [123] J. Segovia, D. R. Entem and F. Fernandez, Strong charmonium decays in a microscopic model, Nucl. Phys. A 915, 125-141 (2013).
  * [124] P. Pakhlov et al. [Belle], Production of New Charmoniumlike States in $e^{+}e^{-}\to J/\psi D^{(*)}\bar{D}^{(*)}$ at $\sqrt{s}\approx 10.6$ GeV, Phys. Rev. Lett. 100, 202001 (2008).
  * [125] V. Kher and A. K. Rai, Spectroscopy and decay properties of charmonium, Chin. Phys. C 42, 083101 (2018).
  * [126] M. Shah, A. Parmar and P. C. Vinodkumar, Leptonic and Digamma decay Properties of S-wave quarkonia states, Phys. Rev. D 86, 034015 (2012).