# Crowdotic: A Privacy-Preserving Hospital Waiting Room Crowd Density
Estimation with Non-speech Audio

Forsad Al Hossain  Manning College of Information & Computer Sciences,
University of Massachusetts AmherstAmherstMA01003USA<EMAIL_ADDRESS>,
Tanjid Hasan Tonmoy  Halıcıoğlu Data Science Institute, University of
California, San DiegoSan DiegoCA92093USA<EMAIL_ADDRESS>,  Andrew A. Lover
School of Public Health & Health Sciences, University of Massachusetts
AmherstAmherstMA01003USA<EMAIL_ADDRESS>,  George A. Corey  University
Health Services, University of Massachusetts AmherstAmherstMA01003USA
<EMAIL_ADDRESS>,  Mohammad Arif Ul Alam  Miner School of Computer &
Information Sciences, University of Massachusetts LowellLowellMA01854USA
<EMAIL_ADDRESS>and  Tauhidur Rahman  Halıcıoğlu Data Science
Institute, University of California, San DiegoSan DiegoCA92093USA
<EMAIL_ADDRESS>
###### Abstract.

Privacy-preserving crowd density analysis finds application across a wide
range of scenarios, substantially enhancing smart building operation and
management while upholding privacy expectations in various spaces. We propose
a non-speech audio-based approach for crowd analytics, leveraging a
transformer-based model. Our results demonstrate that non-speech audio alone
can be used to conduct such analysis with remarkable accuracy. To the best of
our knowledge, this is the first time when non-speech audio signals are
proposed for predicting occupancy. As far as we know, there has been no other
similar approach of its kind prior to this. To accomplish this, we deployed
our sensor-based platform in the waiting room of a large hospital with IRB
approval over a period of several months to capture non-speech audio and
thermal images for the training and evaluation of our models. The proposed
non-speech-based approach outperformed the thermal camera-based model and all
other baselines. In addition to demonstrating superior performance without
utilizing speech audio, we conduct further analysis using differential privacy
techniques to provide additional privacy guarantees. Overall, our work
demonstrates the viability of employing non-speech audio data for accurate
occupancy estimation, while also ensuring the exclusion of speech-related
content and providing robust privacy protections through differential privacy
guarantees.

occupancy estimation, neural networks, audio processing, machine learning

##  1\. Introduction

Estimating the number of people present within a specific area over a period
of time has many applications. These include management of public spaces and
events (Saleh et al., 2015), surveillance and security (Rahmalan et al.,
2006), preventing the spread of infectious diseases (Muthunagai et al., 2023).
Quantification of crowd density empowers relevant authorities to make well-
informed decisions, enabling smart building management to optimize safety,
enhance comfort, minimize energy usage, and efficiently allocate resources for
the management of diverse indoor and outdoor spaces. In addition to ensuring
the accuracy and reliability, it is crucial to address the potential concerns
related to the intrusiveness and privacy implications of the technology used
for such crowd analytic system. To instill public trust and confidence in the
smart building management framework, it is crucial to address potential
privacy concerns, particularly in indoor spaces where privacy expectations are
higher. Achieving such highly accurate and privacy preserving system requires
the thoughtful selection of suitable sensing modalities and robust data
processing methods, and minimizing the amount of information that is retained
for future analysis. This emphasis on privacy-preserving practices contributes
to the broader goal of integrating crowd analytics into smart buildings while
respecting individuals’ privacy rights.

Most of the current research and and deployed methods primarily utilize
computer vision-based techniques for the task (Fiandeiro et al., 2023; Fan et
al., 2022). However, there remains significant concern regarding the use of
different types of cameras and vision sensors, as they may fail to conform to
the privacy expectation of users. Additionally, these techniques have several
technical limitations such as susceptibility to occlusions, limited field of
view and lighting conditions. Thus many research works investigated the use of
other sensing modalities and their fusion for this task. These modalities
include radio frequency (RF) (Shahbazian and Trubitsyna, 2023), acoustics
(Williams et al., 2023), environmental sensors (e.g. $\textit{CO}_{2}$,
temperature, light, motion) (Viani et al., 2014; Chen et al., 2020). However,
all of them have been shown to be location/environment dependent
($\textit{CO}_{2}$, thermal, motion, vision, rf), with some also posing
potential privacy concerns (vision, audio).

The acoustic signatures generated by crowds contain rich information that can
be used to extract useful information, such as occupancy. However, one of the
main challenges in using audio-based analysis is ensuring user privacy since
the audio may include speech. In order to ensure privacy, it is necessary to
record and analyze the audio in a way that makes it impossible to discern any
speech content and prevents the identity of the speakers from being revealed.
In this work, we deploy a sensor-based platform with an on-device machine
model to filter speech and only capture non-speech audio. Our model is able to
distinguish speech and non-speech audio with high accuracy (Al Hossain et al.,
2020) and we only use the non-speech signal for our analysis. We demonstrate
that it is possible to reliably estimate crowd statistics in a coarse time
scale using only non-speech, ensuring privacy. We use a microphone array to
capture the audio, which captures acoustic information in multiple channels
and angle of arrival information. Compared to vision-based estimation using a
thermal camera, we obtain improved accuracy with non-speech audio.

We run a three-month-long study by deploying the platform to capture non-
speech audio in the waiting room of a large hospital. We collected a large-
scale dataset of non-speech audio in a realistic environment. We show that it
is possible to estimate the occupancy with a high degree of accuracy using
only non-speech audio. We compared the results from our non-speech-based
models to the thermal camera-based model. Our results showed that the non-
speech audio-based models outperformed the thermal-based model, especially for
shorter time windows. We used a transformer-based model for the non-speech-
based occupancy estimation. With this method, we have successfully designed an
occupancy estimation system that harnesses the power of modern deep-learning
techniques without requiring complex on-device deep learning models. This
approach prioritizes privacy by excluding speech content and implementing
differential privacy techniques. As a result, our system strikes a balance
between accuracy, efficiency, and safeguarding sensitive information.

The main contributions in this paper are as follows-

  1. (1)

We propose a privacy-preserving non-speech audio-based crowd estimation
approach and show that it is possible to obtain high levels of accuracy using
only non-speech audio. To the best of our knowledge, this is the first-ever
attempt to accomplish occupancy counting from non-verbal audio.

  2. (2)

We collected a large dataset of non-speech audio by deploying our sensor-based
platform in the waiting room of a large hospital and show that our model works
robustly with real-life audio data.

  3. (3)

We compare our non-speech approach with thermal camera-based occupancy
estimation and other baseline approaches and obtain better results.

  4. (4)

In addition to using the privacy-preserving non-speech modality for analysis,
we further enhance privacy preservation by employing differential privacy
techniques.

##  2\. Related work

Multiple applicable methods can be applied for the estimation of occupancy in
a particular location. Traditional methods use a variety of sensing methods,
including vision-based sensors, $\textit{CO}_{2}$ detection, EM waves, PIR
sensors, sounds; including indirect methods such as computer activity or
energy consumption. Some of the most popular methods are :

###  2.1. $\textit{CO}_{2}$ based occupancy counting

As every occupant in a place exhales carbon-di-oxide, the concentration of
carbon-di-oxide can be used as an indicator of occupancy. Although it showed
highly accurate results for occupancy detection, to estimate occupancy, the
results are in general poor. To estimate occupancy either physics-based models
(Ansanay-Alex, 2013), (Ebadat et al., 2015), (Amayri et al., 2016) or data-
driven methods (Basu et al., 2015) (Jiang et al., 2016), (Candanedo and
Feldheim, 2016), (Zhou et al., 2020) besides CO2 or even a hybrid approach
utilizing both physics-based and data-based models (Kampezidou et al., 2021)
are utilized. Usually, physics-based models suffer from errors due to real-
world complexities not accounted for in the ideal physics equation, including
ventilation, exhaust fans, air conditioner, or frequency of door opening or
closing. On the other hand, data-driven methods either perform poorly compared
to physics-based models due to a lack of relevant data or require multiple
sensor integration (thermal, light, humidity, etc.), which can be costly.

###  2.2. RF signal based occupancy counting

Different electromagnetic spectrum bands with different properties are usually
used for RF-based occupancy counting. Works in (Weppner and Lukowicz, 2013),
(Lymberopoulos et al., 2015), (Zou et al., 2016) tries to measure the
occupancy based on the devices carried by the users. Several other works used
device-free schemes to estimate occupancy from wifi signals, either using RSSI
information (Depatla et al., 2015) or by using CSI property (Zou et al.,
2017b), (Cheng and Chang, 2017). However, these methods are not well suited
for different environments and get easily affected by environmental factors.

Another popular method for people is to use an IR-UWB radar ((Choi et al.,
2017), (Muric et al., 2019), (Yang et al., 2019), (Guruacharya et al., 2020)).
This kind of system usually consists of one or multiple pairs of transmitter-
receiver pairs. The transmitter sends an impulse, and the antenna receives the
reflected signal. The same signal gets reflected from different objects at
different distances. The difference between two successive frames indicates
movement, which can be indicative of human presence. However, as it detects
movement, there is a good chance of getting interfered with by any movement,
which complicates the detection and counting of humans.

###  2.3. Sound-wave/audio-based occupancy estimation

The usual audio-based occupancy estimation method utilizes speaker-diarization
techniques ( (Chen et al., 2017), (Uziel et al., 2013), (Valle, 2016)) to
distinguish individual speakers to estimate the occupancy. However, the
downside of such approaches is that it is assumed that everyone present in the
room is talking, which might not be true in all settings, for example, say in
a waiting room or a bus stop or in general, any public place where there is no
social connection between different people occupying the area. Also, these
techniques are privacy insensitive when the data needs to be sent to a backend
to further processing.

Another active sound-based method for counting occupancy uses ultrasonic-
chirps to detection motion from people ((Shih and Rowe, 2015), (Shih et al.,
2016)). In this approach, a transmitter outputs a chirp, and a microphone
listens for reverberation to detection. This technique is similar to IR-UWB
based RF methods and, similar to them, also suffer from distinguishing real
humans from other motion sources like air-conditioning or exhaust fans.

###  2.4. Video-based method

Video-based occupancy measurements systems use cameras to count and track the
number of people in indoor settings. These methods either utilizes RGB video
stream (Liu et al., 2015), (Huang and Hao, 2020), (Conti et al., 2014) , (Zou
et al., 2017a), (Benezeth et al., 2011), (Mutis et al., 2020) or thermal
camera based methods (Yuan et al., 2019), (Metwaly et al., 2019). There are
also other group of methods that measures occupancy by counting and tracking
the people that crosses the entrance and exit of indoor locations [(Kuutti et
al., 2014), (Sangogboye et al., 2017), (Petersen et al., 2016), (Aftab et al.,
2017), (Garaza and Pedrasa, 2016)] to estimate occupancy of a location.

All vision-based methods, while being highly accurate when the camera field of
view covers the whole location. However, they become unreliable when the
location cannot be monitored with a single camera. In those settings, multiple
cameras and algorithms to merge various video streams are needed to count the
total number of people. So it becomes more and more complicated and costly to
monitor space with compartments such as cubicles or walls. Also, a place might
have multiple exits and entrance locations and monitoring each entrance exit
requires multiple camera streams. Besides that, camera-based occupancy
counting is highly undesirable from a privacy perspective. Video contains
numerous unnecessary pieces of information that can be used for action
recognition or person identification. As the state-of-the-art methods for
people counting from video require computation-heavy deep learning models
counting the number of people, the data usually needs to go to a backend to
effectively merge and compute the count of people from the video streams.
Indeed, because of the above-mentioned concerns, there has been a movement to
ban video-based surveillance (fac, [n. d.])

###  2.5. Multimodal fusion-based method

In addition to using single sensing modality, many works focus of fusing
multiple sensing modalities. Thse approaches include fusion of environmental
sensors such as (Sayed et al., 2023; Colace et al., 2022), and inclusion of
image and audio modalities (Tan et al., 2022). These methods similar
limitations associated with each sensing modality as discussed and also
require higher overhead due to increased points of failure and need for
synchronization.

###  2.6. Approaches for privacy preservation

In addition to utilizing less intrusive sensing modalities such as PIR sensors
(Stamatescu and Chitu, 2021), approaches such as federated learning has been
used to ensure privacy for crowd estimation tasks (Khan et al., 2022). Other
approaches include perturbing audio data to prevent identification of speech
and speaker (Williams et al., 2023). However, these approaches have several
limitations such as the use of simulated data that may not capture the
challenging conditions associated with realistic data and fail to preserve
privacy aspects such as speaker identity.

##  3\. Dataset

###  3.1. Data collection platform

We collected data for our non-speech dataset from an on-the-edge device
consisting of a thermal camera, a microphone array and a neural computing
stick attached to a Raspberry Pi platform. Following is a short description of
the components of the platform, illustrated in Figure 1:

  * •

Microphone array: We used a ReSpeaker Microphone array (ReSpeaker, 2022) to
collect audio data. The ReSpeaker microphone array has four microphone arrays
that collect audio streams. Then it produces a cleaner single-channel audio
stream using on-device audio beamforming.

  * •

Thermal camera: A SeekCompact Pro (SEE, 2022) thermal camera was used to take
thermal images. It has a resolution of 320x240 pixels.

  * •

Neural Computing Stick: An Intel Neural Computing Stick (int, 2022) was used
to accelerate deep learning computations on the device.

  * •

Raspberry pi: A Raspberry Pi was used to manage all the other sensors and
store the data on a hard drive.

Figure 1. Front view and the components of the sensor platform.

We have an on-device speech recognition model to eliminate any speech data
from the audio stream. We built our speech-recognizer using TIMIT dataset
(Garofolo, 1993) as a speech data source. This dataset contains speech 630
American English speakers with all eight major dialects in the USA. Each
speaker speaks eight major dialects. We also utilized a custom-labeled google
audio set (Gemmeke et al., 2017) as a negative example source of speech. We
use both of these datasets to develop a custom CNN-based speech-detection
model. If our speech recognizer model determines that an audio snippet has a
probability of being speech content greater than $0.5$, we do not store that
snippet. We only retrain audio snippets with a speech probability less than
0.5, and we encrypt the data before storing.

All of our collected data were stored on a local hard drive after a two-stage
encryption scheme. In the first stage, the original data gets encrypted with a
randomly generated AES key. Then, in the second stage, that randomly-generated
AES key was encrypted and stored with a public key. It is only possible to
decrypt the stored AES key using a private key that is only available to the
researchers.

###  3.2. Deployment

Figure 2. Waiting rooms and the placement of our sensor system

We deployed our device to collect data in a large public university’s
University Health Services (UHS) clinic building. The university has a
student, faculty, and staff size of more than 30,000 people, and the UHS
facility serves the basic health needs of all of the people in that
university.

We deployed our device in the main waiting room for that university health
services building. We started deploying our device on December 10, 2018 and
finished our deployment on July 12, 2019. In the end, we collected around 4
million audio seconds of data from our deployment

##  4\. Audio Processing and Modeling

###  4.1. Multi-Channel Audio Processing

We used single-channel audio captured by the Respeaker microphone by on-device
beamforming (http://wiki.seeedstudio.com/ReSpeaker_Mic_Array_v2.0/, [n. d.]).
This single channel audio stream was produced by applying 4-channel
Generalized Cross-Correlation Phase Transformation (GCC-PHAT) algorithm (Knapp
and Carter, 1976). In this algorithm, at first, the time delay of arrival
(TDOA) is estimated by computing cross-correlation between different multi-
channel audio streams captured by multiple microphones. With this information,
audio from different channels are beam-formed to produce a single-channel
audio stream.

###  4.2. Handling Missing Audio Segments due to Presence of Speech

Our system explored two different schemes for audio processing, as illustrated
in Figure 3.

####  4.2.1. Scheme 1

In this scheme, the transformer model processes the audio stream data in
1-second chunks continuously. To remove speech-related information, a speech
detection model is employed. Chunks identified as containing speech (with a
probability greater than 0.5) are discarded. Non-speech audio chunks are
accumulated until the count reaches 60. Afterwards, these 60 non-speech data
points are fed into the transformer model as a sequence, enabling occupancy
estimation.

####  4.2.2. Scheme 2

In this scheme, a different approach was used for speech and non-speech
chunks. For every second of audio data, if the chunk is identified as speech,
the input spectrogram is replaced with all-zero values. In that case, only the
speech probability obtained from the speech activity detection model is
utilized as an input. On the other hand, for non-speech chunks, both the input
spectrogram and the corresponding speech probability are used as inputs to the
transformer model.

For example: Consider an audio stream from a hospital waiting room area with
the following sequence: [n, n, s, n, s, n, n, n], where n denotes non-speech
chunks, and s denotes audio chunks with speech. Using Scheme 1, the chunks
identified as speech (with probabilities above 0.5) are discarded, resulting
in the following sequence of non-speech chunks: [n, n, n, n, n, n]. In
comparison, using Scheme 2, this sequence will be converted to [n, n, z, n, z,
n, n, n] where z represents all zero spectrograms. Besides these spectrograms,
the speech probabilities are also fed into the model using this scheme.

(a) Scheme 1: construct spectogram using only non-speech segments (duration of
60 seconds) , speech probability information is not used

(b) Scheme 2: construct spectogram using a cobination of non-speech segments
and zero for segments detected as speech (duration of 60 seconds), speech
probability is used as model input

Figure 3. Schemes for dealing with missing audio due to speech segments

###  4.3. Audio Embedding

To preprocess spectrograms for transformer models, we investigated two CNN
encoder models: TRILL (Shor et al., 2020) and VGGish (Hershey et al., 2017).

TRILL: TRILL is a deep-learning model that can be used for extracting
representation from non-semantic audio. Before feeding our data to the
transformer model, for each second of audio, we extract 512-dimensional
embedding for the audio snippet. This embedding is used as a representation of
the non-speech audio snippet.

VGGish: VGGish is another deep-learning model that can be used to extract
128-dimensional representation from audio. However, unlike the TRILL model, we
attached VGGish network to the input layers of our transformer and trained the
whole model in an end to end fashion. This particular method ensures that
besides optimizing the transformer model, we also optimize the VGGish model
for generating suitable representation of audio.

In both cases, when the speech probabilities are used as input to the models
(4.2.2), we append the speech probabilities, and the end of these 512 (for
TRILL) and 128 (for VGGish) dimensional representations before feeding the
embedding representation sequence to the transformer model.

(a)

(b)

Figure 4. Overview of transformer model: (a) Multi-head attention encoder (b)
Our four-layer transformer model

###  4.4. Transformer model

A transformer(Vaswani et al., 2017) is a time-series model that consists of
several multi-head self-attention layers. A single head in a layer takes a
sequence of $T$ embeddings of dimension $d_{emb}$. For our case $d_{emb}=128$
and projects each of the elements in the sequence to query ($Q$), key ($K$),
and value ($V$) embeddings. For a single embedding $e$, the query, key, and
value can be computed as follows: $Q=W_{Q}e$, $K=W_{K}e$, and $V=W_{V}e$.
Here, $W_{Q}$, $W_{K}$, and $W_{V}$ are projection matrices. For our
particular setting, we project 128-dimensional embeddings to a 16-dimensional
space.

We calculate the values ($V$) by aggregating them using the attention score
(computed using queries and keys) as defined in the following equation:

$\overline{V}=\sum_{j=1}^{T}\left(\frac{\exp(q_{i}^{T}k_{j})}{\sum_{t=1}^{T}\exp(q_{i}^{T}k_{t})}\right)V_{j}$

From each head, for each embedding position, we produce these 16-dimensional
embeddings. Finally, we concatenate the embeddings from 8 different heads,
resulting in a 128-dimensional embedding output. The original 128-dimensional
embedding values are added to this output via a residual connection(He et al.,
2016). The subsequent encoder layer takes this output after applying a ReLU
activation function to it.

For our models, we used four such encoder layers. The output from the final
layer goes through a linear layer to produce T outputs, which are our
prediction values for occupancy in each second.

Intuitively, each transformer encoder layer model head finds the relationship
between the input sequence elements passed from the previous layer. The
value(V) represents an extracted feature from each of the elements of the
sequence. These features are aggregated according to the self-attention score
to create a richer set of representations for the sequence. With multiple
heads, different sets of features are extracted. The final concatenated
embeddings represent the aggregated version of all the features for that
element in the sequence.

Transformer-based models can take variable length sequence input similar to
RNN and LSTM, and has several key advantages such as faster training and
inference in comparison (Vaswani et al., 2017). Other advantages such as the
use of residual connections, not requiring backpropagation through time enable
transformer models to be more efficient.

###  4.5. Model Training

####  4.5.1. Ground Truth Collection

In this section, we outline the process of ground truth data collection for
our study. We extract ground truth data from the entry-exit data of the
hospital waiting room, obtained on a daily basis. To accurately determine the
number of individuals present in the waiting room at any given time, we
created a people counting time series dataset for each day.

The time series dataset was constructed by initializing the patient count to 0
at the beginning of each day. The count was incremented as individuals entered
the waiting room and reduced when they left. This approach provided a reliable
ground truth dataset, allowing accurate assessment of the occupancy in the
hospital waiting room at any given time.

Table 1. Mean Average Error (MAE), Root Mean Squared Error (RMSE), Correlation Coefficient ($\rho$) between the occupancy prediction and ground truth for different models Model | Modality | Model Descripition | MAE | RMSE | $\rho$
---|---|---|---|---|---
1 | Baseline(Average prediction) | Predict mean | 3.73 | 4.54 | 0.0
2 | Thermal image | Faster RCNN | 3.90 | 4.26 | 0.11
3 | Speech probabilities | Random Forest | 3.23 | 4.16 | 0.34
4 | 1 channel(PHAT) Non-speech audio | Trill-embedding + Transformer | 2.18 | 3.00 | 0.74
5 | 1 channel(PHAT) Non-speech audio | VGGish-CNN + Transformer | 2.23 | 3.00 | 0.73
6 | Specch probability + 1 channel(PHAT) Non-speech audio | VGGish-CNN + Transformer | 2.46 | 3.21 | 0.69
7 | Specch probability + 1 channel(PHAT) Non-speech audio | Trill + Transformer | 2.63 | 3.21 | 0.63

####  4.5.2. Model Setup and Training

Our model is based on a 4-layer transformer encoder. We explore two different
approaches for embedding: TRILL-based and the VGGish-based model.

For the TRILL-based model, we extract a 128-dimensional embedding and utilize
it as input for the transformer model. On the other hand, for the VGGish-based
model, we incorporate the VGGish model as the input for the transformer
embeddings. We train this model in conjunction with our transformer model.

At each layer of the transformer, we employ 8 transformer heads to process the
time series data. This allows for effective information processing at each
layer. In the final layer, a linear layer converts the time series into a
series of single numbers. As a result, for a 60-second audio input, we obtain
an output of size $60\times 1$. We define this output as representing the
occupancy at any given time within the 60-second input.

To evaluate the performance of our model, we employ the Mean Squared Error
(MSE) loss function, which measures the discrepancy between the ground truth
and the output. During the training phase, we use the Adam Optimizer with a
learning rate of $0.001$. We train the model for 30 epochs, allowing it to
learn and optimize its performance over time.

##  5\. Results

We develop a baseline model which predicts the mean occupancy, and train
models using other data modalities, thermal image, and speech probability (no
speech audio used) for every minute. We train the models with our collected
waiting room data and leave out the last ten days of data for testing. We also
perform cross validation by leaving out the data for entire months.

###  5.1. Performance using different data modalities

Table 2. Results for different cross-validation months in terms of MAE (Mean Average Error), RMSE(Root Mean Squared Error), Correlation coefficient($\rho$) | MAE | RMSE | $\rho$
---|---|---|---
Month | February | March | April | February | March | April | February | March | April
Model | Modality |  |  |  |  |  |  |  |  |
1 | Baseline | 3.18 | 2.68 | 3.93 | 4.02 | 3.21 | 4.74 | 0.00 | 0.00 | 0.00
2 | Thermal image | 3.34 | 2.94 | 3.57 | 4.39 | 3.44 | 4.49 | 0.14 | 0.19 | 0.14
3 | Speech probability | 2.62 | 2.81 | 3.61 | 3.62 | 3.41 | 4.75 | 0.50 | 0.32 | 0.32
- | Speech probability + loudness | 2.50 | 2.32 | 3.58 | 3.50 | 3.35 | 4.71 | 0.53 | 0.34 | 0.41
4 | Non-speech Audio (TRILL) | 1.82 | 2.00 | 2.65 | 2.65 | 2.70 | 3.48 | 0.72 | 0.61 | 0.69
5 | Non-speech Audio (VGGish pretrained) | 1.86 | 2.21 | 2.87 | 3.01 | 2.86 | 3.80 | 0.61 | 0.56 | 0.61
6 | Speech probability + VGGish | 1.95 | 2.13 | 2.82 | 2.90 | 2.77 | 3.88 | 0.70 | 0.63 | 0.68
7 | Speech probability + Trill | 1.95 | 2.14 | 2.96 | 2.90 | 2.80 | 3.89 | 0.70 | 0.54 | 0.61

Table 1 shows the performance of different minute-level waiting room occupancy
prediction models trained with different modalities including thermal camera,
speech and non-speech. The first key takeaway is that the non-speech audio-
based models outperform the thermal-based and speech-probability based models.
While the thermal camera-based Faster RCNN model (Ren et al., 2015) achieves a
Person correlation coefficient ($\rho$) of 0.11 and a Mean Average Error (MAE)
of 3.90, the non-speech audio based Transformer model with Trill embedding
(scheme 1) achieves significantly better results with a $\rho$ of 0.74 and MAE
of 2.18. Similarly, the non-speech audio based Transformer model with Trill
embedding outperforms model that leverages speech probability information by a
significant margin (i.e. $\rho$ of 0.34 for speech probabilities vs. $\rho$ of
0.74 for non-speech). Overall, this results clearly highlight that non-speech-
based models contain information about waiting room occupancy which can be
used to develop a privacy-aware occupancy system that solely relies on non-
speech audio (and avoids analyzing any speech audio). This type of waiting
room occupancy estimation system is particularly well-suited for
clinical/hospital environment, as well as other public spaces that have higher
level of privacy expectations.

We experimented with different schemes (section 4.2.1, 4.2.2) and embeddings
as we develop our non-speech audio-based model. As can be seen in Table 1,
scheme 1 that uses 60 seconds of non-speech audio performs better than scheme
2 that leverages the extra information of speech probability (Note that only
speech probability was used and no speech audio signal was used). For example,
Model 4 with non-speech audio data achieves a $\rho$ of 0.74 and MAE of 2.18.
Compared to this, Model 7 that uses both speech probability and non-speech
audio as input to the same model architecture of Model 4 achieves a slightly
lower performance with $\rho$ of 0.69 and MAE of 2.46. Overall, with the
addition of speech probability in scheme 2, we get a slightly worse
performance. It shows that the results of both of the schemes are comparable
and after using non-speech sound adding speech probabilities does not contain
that much information.

(a)

(b)

Figure 5. (a) Plot showing average RMSE by occupancy (b) Total samples(in
seconds) in the dataset by occupancy

(a)

(b)

Figure 6. (a) Plot showing average RMSE by occupancy (b) Total samples(in
seconds) in the dataset by occupancy

In Figure 6, we can observe that the majority of the collected real life data
consists of non-speech snippets for the entire minute. Having a substantial
amount of non-speech data available enhances the performance of our models.
Conversely, when the amount of non-speech snippets is lower, the performance
remains relatively unchanged and does not drop. In Figure 5, we observe that
we have more data for lower occupancy setting and our model has lowest error
for such scenarios. The performance of our model decreases slightly for higher
occupancy setting by negligible amount.

###  5.2. Time scale aggregation

We aggregate the predictions from our minute-level occupancy models and
compare the correlation coefficient with the ground truth data in Table 3. It
can be seen that $\rho$ improves if a longer aggregation window is used for
all modalities. However, the key takeaway is that non-speech audio-based model
outperforms the models using other modalities by substantial margins for time
aggregated results ($\rho\geq 0.90$ for non speech audio compared to
$\rho<0.25$ using thermal image and $\rho<0.50$ using speech probability).

Table 3. Correlation coefficient of average aggregation results for different modalities for different timescales Data | 30 min $\rho$ | 1 hour $\rho$ | 2 hour $\rho$
---|---|---|---
Thermal image | 0.23 | 0.24 | 0.24
Speech probability | 0.44 | 0.47 | 0.48
Non-speech Audio | 0.90 | 0.91 | 0.93

###  5.3. Cross validation

We also conducted leave-one-month-out cross validation, where the entire data
from a specific month was used as test data. The results are shown in Table 2.
Consistent with the results in the previous section, the cross-validation
results demonstrate superior performance using the non-speech modality. Using
scheme 1, non-speech audio using TRILL embedding obtains the best performance
for all three of the months compared to using other modalities and audio
processing schemes in terms of MAE, RMSE and $\rho$. Using scheme 2, we also
obtain better performance than using the other modalities.

(a)

(b)

Figure 7. (a) Plot showing distribution of visits by patients (b) Plot showing
distribution of total duration of visits by patients

##  6\. Privacy preserving Inference

###  6.1. Information linkage attacks and data privacy

Linkage attacks, exemplified by the case of Netflix (Netflix_wired, [n. d.]),
pose a significant security threat by linking new data with existing public
data, potentially compromising individuals’ privacy. To combat this risk,
differential privacy has emerged as a powerful framework. It provides a
rigorous mathematical guarantee to protect individual privacy while enabling
meaningful analysis of sensitive data. By injecting controlled noise into data
analysis, differential privacy prevents specific information extraction and
safeguards privacy against linkage attacks. It strikes a balance between
privacy preservation and data utility, allowing valuable insights while
upholding individuals’ privacy rights. Adopting differential privacy
techniques is crucial to address privacy concerns and ensure responsible use
of data in the face of evolving security challenges.

As our system processes non-speech audio data, we can implement differential
privacy by adding noise to the embeddings generated by our CNN model (which is
then later fed into the transformer model). As the transformer model can be
computationally heavy, while the CNN features can be computed in real-time on-
device, this methodology ensures that we only send differentially private
information to the backend for further processing.

###  6.2. Implementation of Differential Privacy guarantees for our system

To ensure our system outputs data with differential privacy requirements, we
use Laplace mechanism to add noise to the output of our CNN based audio
embeddging generation system (Dwork and Roth, 2014). To do that, at first we
define the sensitivity of any function $f$ as

$\Delta f=\max_{\begin{subarray}{c}x,y\in\mathbb{N}^{k}\\\ {\left\lVert
x-y\right\rVert_{1}=1}\end{subarray}}\left\lVert f(x)-f(y)\right\rVert_{1}$

where $f:\mathbb{N}^{|{\mathcal{X}}|}\rightarrow\mathbb{R}^{k}$. With this,
definition, one can prove that, with the Laplace mechanism on the output of
$f(x)$ defined as:

$\mathcal{M}_{L}(f(x))=f(x)+(Y_{1},Y_{2}...Y_{k})$

where each of $Y_{i}$ are i.i.d variables drawn from Laplace distribution of
$L(\frac{\Delta f}{\epsilon})=\frac{\epsilon}{2\Delta
f}\exp(-\frac{|t|\epsilon}{\Delta f})$; is an ($\epsilon$ ,0) differential
private algorithm.

Based on this analysis; for our modeling scenario if we extract $d$
dimensional embeddings from $t$ timesteps (from $t$ seconds of audio clips);
our embedding generation function will produce $t\times d$ dimensional output.
If we clip the output embedding for each $i$th second as:

$\tilde{e_{i}}=\frac{e_{i}}{\max(1,\frac{\left\lVert
e_{i}\right\rVert_{1}}{C})}$

Where $C$ is the clipping parameter; which is a hyperparameter. From this
formulation we can conclude that:

$\left\lVert\tilde{e}\right\rVert_{1}\leq C$

So, for two $t\times d$ embedding sets where in one embedding set the
embedding in some particular second $i$ is hidden (which means not using that
particular second of audio); say $e_{-i}$ by setting those positions with
zeros compared to another embedding set where the embedding from $i$th second
is present ($e$), we can express the relation as: $e=e_{-i}+e_{i}$

where $e_{i}$ a zero vector of size $t\times d$ except from positions
$(i-1)*d+1$ to $i*d$; where it contains clipped embeddings from $i$th second.

So from the above, we can conclude:

$\left\lVert e-e_{-i}\right\rVert_{1}=\left\lVert e_{i}\right\rVert_{1}\leq C$

Thus the embedding space has a sensitivity of $C$. From that, we can conclude
for per clip, $(\epsilon,0)$ differential privacy, all we need to do is
perturb the embeddings with Laplace noise sampled from $L(\frac{C}{\epsilon})$
and add it to our embedding. This procedure ensures adding each second of
audio and sending it to a remote backend (where the transformer-based
occupancy detection model might run) has an at-most $1+\epsilon$ diminishing
effect return on any future utility (Dwork and Roth, 2014).

In this procedure, if a user decides to stay in the place $T$ seconds, the
total differential privacy loss will be $\epsilon T$ because of the
composition theorem of differential privacy mentioned at (Dwork and Roth,
2014).

###  6.3. Experimental Results for our differential privacy formulation

To make our models compatible with differential privacy, we first trained our
model where CNN embeddings were clipped as described in 6.2. We also added
noise according to the Laplace mechanism during the training to make the model
prediction more robust with differential privacy noise. Table 4 shows the
effect on the occupancy estimation algorithm for different $\epsilon$
differential privacy targets:

Table 4. Results for epsilon-delta differential privacy settings Mechanism | $\epsilon$ | MAE | RMSE | $\rho$
---|---|---|---|---
Laplace | 5 | 2.74 | 3.80 | 0.69
Laplace | 2 | 2.87 | 3.97 | 0.66
Laplace | 1 | 3.03 | 4.18 | 0.62
Laplace | 0.5 | 3.18 | 4.34 | 0.55
Laplace | 0.25 | 3.60 | 4.80 | 0.41
Laplace | 0.10 | 4.12 | 5.31 | 0.19

From table 4, we can see that our algorithm even performs well for $\epsilon$
values as small as 1. For the other values ¿ 0.5, we can see that the MAE
remains consistently low compared to the baseline. With these small $\epsilon$
values, we can ensure that using our methodology reduces the possibility of
future linkage attacks for each additional audio snippet we use for occupancy
detection with our model.

##  7\. Discussion and Conclusion

In this work, we demonstrate the feasibility of estimating occupancy using
non-speech audio, presenting superior results compared to other modalities,
including thermal cameras. Compared to techniques that aim to preserve privacy
while still using speech data, our results show that non-speech audio contains
sufficient information to estimate occupancy reliably. Our non-speech-based
method not only protects speech content but also renders speaker
identification extremely difficult. Our analysis with differential privacy
techniques shows that we can provide additional privacy guarantees on top of
the privacy preserving data modality that we use.

Our approach incorporates a transformer-based model which has proven to be
effective across various tasks and domains. By leveraging this non-speech
based method, we enable the application of such models even in situations
where deploying such models on edge-devices might not be feasible.

One notable advantage of employing non-speech audio lies in the richness of
information that is contained, resulting in more reliable occupancy estimates.
While alternative modalities, such as $\textit{CO}_{2}$ sensor and PIR
sensors,may be effective in detecting the presence of humans, they often fall
short in providing accurate occupancy information, as evident from the
existing literature.

Furthermore, our analysis incorporates differential privacy techniques, adding
an additional layer of privacy guarantees to the already privacy-preserving
data modality we use. By incorporating differential privacy safeguards, we
have not only ensured the present security of our system but also fortified it
against potential future linkage attacks. This robust approach guarantees the
protection of sensitive data, reinforcing the trustworthiness of our system
for long-term use.

One of the limitations of our current approach is that we have only deployed
our sensors in a hospital waiting room. To confirm the general applicability
of our method, further investigations and evaluations in other scenarios may
be required. We argue that our method is highly suitable for similar indoor
environments with higher privacy expectation.

Our findings have a significant impact on the design and implementation of
smart technology for homes, public spaces, and workplaces. Our approach could
help to instill public trust by providing privacy protection while providing
valuable insight beyond occupancy such as syndromic surveillance (Al Hossain
et al., 2020).

To conclude, our study demonstrates the feasibility and advantages of
estimating occupancy using non-speech audio. Our approach surpasses other
modalities in terms of privacy preservation and accuracy of occupancy
estimation.

## References

  * (1)
  * (2) . [n. d.].  Netflix Movie review contest.  ([n. d.]).  https://www.wired.com/2010/03/netflix-cancels-contest/?utm_source=WIR_REG_GATE.
  * fac ([n. d.]) [n. d.].  San Fransisco facial recognition ban.  https://www.nytimes.com/2019/05/14/us/facial-recognition-ban-san-francisco.html. ([n. d.]).
  * int (2022) 2022\.  Intel Neural Compute Stick 2.  (2022).  https://software.intel.com/en-us/neural-compute-stick
  * SEE (2022) 2022\.  Seek CompactPRO: Thermal Camera.  (2022).  https://www.thermal.com/compact-series.html/
  * Aftab et al. (2017) Muhammad Aftab, Chien Chen, Chi-Kin Chau, and Talal Rahwan. 2017.  Automatic HVAC control with real-time occupancy recognition and simulation-guided model predictive control in low-cost embedded system.  _Energy and Buildings_ 154 (2017), 141–156.
  * Al Hossain et al. (2020) Forsad Al Hossain, Andrew A. Lover, George A. Corey, Nicholas G. Reich, and Tauhidur Rahman. 2020.  FluSense: A Contactless Syndromic Surveillance Platform for Influenza-Like Illness in Hospital Waiting Areas.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 4, 1, Article 1 (March 2020), 28 pages.  https://doi.org/10.1145/3381014
  * Amayri et al. (2016) Manar Amayri, Abhay Arora, Stephane Ploix, Sanghamitra Bandhyopadyay, Quoc-Dung Ngo, and Venkata Ramana Badarla. 2016.  Estimating occupancy in heterogeneous sensor environment.  _Energy and Buildings_ 129 (2016), 46–58.
  * Ansanay-Alex (2013) Guillaume Ansanay-Alex. 2013.  Estimating occupancy using indoor carbon dioxide concentrations only in an office building: a method and qualitative assessment. In _REHVA World Congress on Energy efficient, smart and healthy buildings (CLIMA)_. 1–8.
  * Basu et al. (2015) Chandrayee Basu, Christian Koehler, Kamalika Das, and Anind K. Dey. 2015.  PerCCS: Person-Count from Carbon Dioxide Using Sparse Non-Negative Matrix Factorization. In _Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing_ _(UbiComp ’15)_. Association for Computing Machinery, New York, NY, USA, 987–998.  https://doi.org/10.1145/2750858.2807525
  * Benezeth et al. (2011) Yannick Benezeth, Hélène Laurent, Bruno Emile, and Christophe Rosenberger. 2011.  Towards a sensor for detecting human presence and characterizing activity.  _Energy and Buildings_ 43, 2-3 (2011), 305–314.
  * Candanedo and Feldheim (2016) Luis M Candanedo and Véronique Feldheim. 2016.  Accurate occupancy detection of an office room from light, temperature, humidity and CO2 measurements using statistical learning models.  _Energy and Buildings_ 112 (2016), 28–39.
  * Chen et al. (2017) Siyuan Chen, J. Epps, E. Ambikairajah, and P. Le. 2017.  An Investigation of Crowd Speech for Room Occupancy Estimation. In _INTERSPEECH_.
  * Chen et al. (2020) Zhenghua Chen, Chaoyang Jiang, Mustafa K Masood, Yeng Chai Soh, Min Wu, and Xiaoli Li. 2020.  Deep learning for building occupancy estimation using environmental sensors.  _Deep Learning: Algorithms and Applications_ (2020), 335–357.
  * Cheng and Chang (2017) Yen-Kai Cheng and Ronald Y. Chang. 2017.  Device-Free Indoor People Counting Using Wi-Fi Channel State Information for Internet of Things. In _GLOBECOM 2017 - 2017 IEEE Global Communications Conference_. 1–6.  https://doi.org/10.1109/GLOCOM.2017.8254522
  * Choi et al. (2017) Jeong Woo Choi, Dae Hyeon Yim, and Sung Ho Cho. 2017.  People Counting Based on an IR-UWB Radar Sensor.  _IEEE Sensors Journal_ 17, 17 (2017), 5717–5727.  https://doi.org/10.1109/JSEN.2017.2723766
  * Colace et al. (2022) Simone Colace, Sara Laurita, Giandomenico Spezzano, and Andrea Vinci. 2022.  Room Occupancy Prediction Leveraging LSTM: An Approach for Cognitive and Self-Adapting Buildings.  In _IoT Edge Solutions for Cognitive Buildings_. Springer, 197–219.
  * Conti et al. (2014) Francesco Conti, Antonio Pullini, and Luca Benini. 2014.  Brain-inspired classroom occupancy monitoring on a low-power mobile platform. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_. 610–615.
  * Depatla et al. (2015) Saandeep Depatla, Arjun Muralidharan, and Yasamin Mostofi. 2015.  Occupancy Estimation Using Only WiFi Power Measurements.  _IEEE Journal on Selected Areas in Communications_ 33, 7 (2015), 1381–1393.  https://doi.org/10.1109/JSAC.2015.2430272
  * Dwork and Roth (2014) Cynthia Dwork and Aaron Roth. 2014.  The Algorithmic Foundations of Differential Privacy.  _Found. Trends Theor. Comput. Sci._ 9, 3–4 (Aug. 2014), 211–407.  https://doi.org/10.1561/0400000042
  * Ebadat et al. (2015) Afrooz Ebadat, Giulio Bottegal, Damiano Varagnolo, Bo Wahlberg, and Karl H Johansson. 2015.  Regularized deconvolution-based approaches for estimating room occupancies.  _IEEE Transactions on Automation Science and Engineering_ 12, 4 (2015), 1157–1168.
  * Fan et al. (2022) Zizhu Fan, Hong Zhang, Zheng Zhang, Guangming Lu, Yudong Zhang, and Yaowei Wang. 2022.  A survey of crowd counting and density estimation based on convolutional neural network.  _Neurocomputing_ 472 (2022), 224–251.
  * Fiandeiro et al. (2023) Miguel Fiandeiro, Thanh Thi Nguyen, Hanting Wong, and Edbert B Hsu. 2023.  Modernized crowd counting strategies for mass gatherings—a review.  _Journal of Acute Medicine_ 13, 1 (2023), 4.
  * Garaza and Pedrasa (2016) Christian Rabbi A Garaza and Jhoanna Rhodette Pedrasa. 2016.  Development of a doorway occupancy counter based on thermal array sensing technology. In _2016 IEEE Region 10 Conference (TENCON)_. IEEE, 3506–3510.
  * Garofolo (1993) John S Garofolo. 1993.  Timit acoustic phonetic continuous speech corpus.  _Linguistic Data Consortium, 1993_ (1993).
  * Gemmeke et al. (2017) Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017.  Audio Set: An ontology and human-labeled dataset for audio events. In _Proc. IEEE ICASSP 2017_. New Orleans, LA.
  * Guruacharya et al. (2020) Sudarshan Guruacharya, Batu K. Chalise, and Braham Himed. 2020.  Energy Distribution of Multiple Target Signal With Application to Target Counting.  _IEEE Signal Processing Letters_ 27 (2020), 431–435.
  * He et al. (2016) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep Residual Learning for Image Recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2016), 770–778.
  * Hershey et al. (2017) Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. 2017.  CNN Architectures for Large-Scale Audio Classification.  In _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_.  https://arxiv.org/abs/1609.09430
  * Huang and Hao (2020) Qian Huang and Kangli Hao. 2020.  Development of CNN-based visual recognition air conditioner for smart buildings.  _J. Inf. Technol. Constr._ 25 (2020), 361–373.
  * Jiang et al. (2016) Chaoyang Jiang, Mustafa K Masood, Yeng Chai Soh, and Hua Li. 2016.  Indoor occupancy estimation from carbon dioxide concentration.  _Energy and Buildings_ 131 (2016), 132–141.
  * Kampezidou et al. (2021) Styliani I. Kampezidou, Archan Ray, S. Duncan, M. Balchanos, and D. Mavris. 2021.  Real-time occupancy detection with physics-informed pattern-recognition machines based on limited CO2 and temperature sensors.  _Energy and Buildings_ (2021), 110863.
  * Khan et al. (2022) Irfanullah Khan, Antonio Guerrieri, Giandomenico Spezzano, and Andrea Vinci. 2022.  Occupancy Prediction in Buildings: An approach leveraging LSTM and Federated Learning. In _2022 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech)_. 1–7.
  * Knapp and Carter (1976) C. Knapp and G. Carter. 1976.  The generalized correlation method for estimation of time delay.  _IEEE Transactions on Acoustics, Speech, and Signal Processing_ 24, 4 (1976), 320–327.  https://doi.org/10.1109/TASSP.1976.1162830
  * Kuutti et al. (2014) Jussi Kuutti, Kim H Blomqvist, and Raimo E Sepponen. 2014.  Evaluation of visitor counting technologies and their energy saving potential through demand-controlled ventilation.  _Energies_ 7, 3 (2014), 1685–1705.
  * Liu et al. (2015) Jun Liu, Ye Liu, Guyue Zhang, Peiru Zhu, and Yan Qiu Chen. 2015.  Detecting and tracking people in real time with RGB-D camera.  _Pattern Recognition Letters_ 53 (2015), 16–23.
  * Lymberopoulos et al. (2015) Dimitrios Lymberopoulos, Jie Liu, Xue Yang, Romit Roy Choudhury, Vlado Handziski, and Souvik Sen. 2015.  A Realistic Evaluation and Comparison of Indoor Location Technologies: Experiences and Lessons Learned. In _Proceedings of the 14th International Conference on Information Processing in Sensor Networks_ _(IPSN ’15)_. Association for Computing Machinery, New York, NY, USA, 178–189.  https://doi.org/10.1145/2737095.2737726
  * Metwaly et al. (2019) Aly Metwaly, Jorge Peña Queralta, Victor Kathan Sarker, Tuan Nguyen Gia, Omar Nasir, and Tomi Westerlund. 2019.  Edge computing with embedded ai: Thermal image analysis for occupancy estimation in intelligent buildings. In _Proc of the INTelligent Embedded Systems Architectures and Applications Workshop 2019_. 1–6.
  * Muric et al. (2019) Admir Muric, Christos Anestis Georgiadis, Fisayo Caleb Sangogboye, and Mikkel Baun Kjærgaard. 2019.  Practical IR-UWB-Based Occupant Counting Evaluated in Multiple Field Settings. In _Proceedings of the 1st ACM International Workshop on Device-Free Human Sensing_ _(DFHS’19)_. Association for Computing Machinery, New York, NY, USA, 48–51.  https://doi.org/10.1145/3360773.3360885
  * Muthunagai et al. (2023) SU Muthunagai, MS Girija, R Iyswarya, S Poorani, and R Anitha. 2023.  Crowd Density Estimation Using Neural Network for COVID-19 and Future Pandemics.  In _System Design for Epidemics Using Machine Learning and Deep Learning_. Springer, 257–270.
  * Mutis et al. (2020) Ivan Mutis, Abhijeet Ambekar, and Virat Joshi. 2020.  Real-time space occupancy sensing and human motion analysis using deep learning for indoor air quality control.  _Automation in Construction_ 116 (2020), 103237.
  * Petersen et al. (2016) Steffen Petersen, Theis Heidmann Pedersen, Kasper Ubbe Nielsen, and Michael Dahl Knudsen. 2016.  Establishing an image-based ground truth for validation of sensor data-based room occupancy detection.  _Energy and Buildings_ 130 (2016), 787–793.
  * Rahmalan et al. (2006) Hidayah Rahmalan, Mark S Nixon, and John N Carter. 2006.  On crowd density estimation for surveillance.  (2006).
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In _Advances in Neural Information Processing Systems_ , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc.
  * ReSpeaker (2022) ReSpeaker. 2022.  ReSpeaker: Microphone Array V2.  (2022).  http://wiki.seeedstudio.com/ReSpeaker_Mic_Array_v2.0/
  * Saleh et al. (2015) Sami Abdulla Mohsen Saleh, Shahrel Azmin Suandi, and Haidi Ibrahim. 2015.  Recent survey on crowd density estimation and counting for visual surveillance.  _Engineering Applications of Artificial Intelligence_ 41 (2015), 103–114.
  * Sangogboye et al. (2017) Fisayo Caleb Sangogboye, Krzysztof Arendt, Ashok Singh, Christian T Veje, Mikkel Baun Kjærgaard, and Bo Nørregaard Jørgensen. 2017.  Performance comparison of occupancy count estimation and prediction with common versus dedicated sensors for building model predictive control. In _Building Simulation_ , Vol. 10. Springer, 829–843.
  * Sayed et al. (2023) Aya Nabil Sayed, Faycal Bensaali, Yassine Himeur, and Mahdi Houchati. 2023.  Edge-Based Real-Time Occupancy Detection System through a Non-Intrusive Sensing System.  _Energies_ 16, 5 (2023).  https://doi.org/10.3390/en16052388
  * Shahbazian and Trubitsyna (2023) Reza Shahbazian and Irina Trubitsyna. 2023.  Human Sensing by Using Radio Frequency Signals: A Survey on Occupancy and Activity Detection.  _IEEE Access_ 11 (2023), 40878–40904.  https://doi.org/10.1109/ACCESS.2023.3269843
  * Shih et al. (2016) Oliver Shih, Patrick Lazik, and Anthony Rowe. 2016.  AURES: A Wide-Band Ultrasonic Occupancy Sensing Platform. In _Proceedings of the 3rd ACM International Conference on Systems for Energy-Efficient Built Environments_ _(BuildSys ’16)_. Association for Computing Machinery, New York, NY, USA, 157–166.  https://doi.org/10.1145/2993422.2993580
  * Shih and Rowe (2015) Oliver Shih and Anthony Rowe. 2015.  Occupancy Estimation Using Ultrasonic Chirps. In _Proceedings of the ACM/IEEE Sixth International Conference on Cyber-Physical Systems_ _(ICCPS ’15)_. Association for Computing Machinery, New York, NY, USA, 149–158.  https://doi.org/10.1145/2735960.2735969
  * Shor et al. (2020) Joel Shor, Aren Jansen, Ronnie Zvi Maor, Oran Lang, Omry Tuval, Félix de Chaumont Quitry, Marco Tagliasacchi, Ira Shavitt, and Dotan Emanuel. 2020.  Towards Learning a Universal Non-Semantic Representation of Speech.  https://www.isca-speech.org/archive/Interspeech_2020/abstracts/1242.html
  * Stamatescu and Chitu (2021) Grigore Stamatescu and Claudia Chitu. 2021.  Privacy-Preserving Sensing and Two-Stage Building Occupancy Prediction using Random Forest Learning.  _Journal of Sensors_ 2021 (2021), 1–12.
  * Tan et al. (2022) Sin Yong Tan, Margarite Jacoby, Homagni Saha, Anthony Florita, Gregor Henze, and Soumik Sarkar. 2022.  Multimodal sensor fusion framework for residential building occupancy detection.  _Energy and Buildings_ 258 (2022), 111828.  https://doi.org/10.1016/j.enbuild.2021.111828
  * http://wiki.seeedstudio.com/ReSpeaker_Mic_Array_v2.0/ ([n. d.]) http://wiki.seeedstudio.com/ReSpeaker_Mic_Array_v2.0/. [n. d.].  ([n. d.]).  ReSpeaker: Microphone Array V2.
  * Uziel et al. (2013) Sebastian Uziel, Thomas Elste, Wolfram Kattanek, Danilo Hollosi, Stephan Gerlach, and Stefan Goetze. 2013.  Networked embedded acoustic processing system for smart building applications. In _2013 Conference on Design and Architectures for Signal and Image Processing_. 349–350.
  * Valle (2016) Rafael Valle. 2016.  ABROA: Audio-based room-occupancy analysis using Gaussian mixtures and Hidden Markov models. In _2016 Future Technologies Conference (FTC)_. 1270–1273.  https://doi.org/10.1109/FTC.2016.7821763
  * Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need.  _ArXiv_ abs/1706.03762 (2017).
  * Viani et al. (2014) Federico Viani, Alessandro Polo, Fabrizio Robol, Giacomo Oliveri, Paolo Rocca, and Andrea Massa. 2014.  Crowd detection and occupancy estimation through indirect environmental measurements. In _The 8th European Conference on Antennas and Propagation (EuCAP 2014)_. 2127–2130.  https://doi.org/10.1109/EuCAP.2014.6902229
  * Weppner and Lukowicz (2013) Jens Weppner and Paul Lukowicz. 2013.  Bluetooth based collaborative crowd density estimation with mobile phones. In _2013 IEEE International Conference on Pervasive Computing and Communications (PerCom)_. 193–200.  https://doi.org/10.1109/PerCom.2013.6526732
  * Williams et al. (2023) Jennifer Williams, Vahid Yazdanpanah, and Sebastian Stein. 2023.  Privacy-Preserving Occupancy Estimation. In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 1–5.
  * Yang et al. (2019) Xiuzhu Yang, Wenfeng Yin, Lei Li, and Lin Zhang. 2019.  Dense People Counting Using IR-UWB Radar With a Hybrid Feature Extraction Method.  _IEEE Geoscience and Remote Sensing Letters_ 16, 1 (2019), 30–34.  https://doi.org/10.1109/LGRS.2018.2869287
  * Yuan et al. (2019) Yazhou Yuan, Xin Li, Zhixin Liu, and Xinping Guan. 2019.  Occupancy estimation in buildings based on infrared array sensors detection.  _IEEE Sensors Journal_ 20, 2 (2019), 1043–1053.
  * Zhou et al. (2020) Yaping Zhou, Jiayu Chen, Zhun Jerry Yu, Jun Li, Gongsheng Huang, Fariborz Haghighat, and Guoqiang Zhang. 2020.  A novel model based on multi-grained cascade forests with wavelet denoising for indoor occupancy estimation.  _Building and Environment_ 167 (2020), 106461.
  * Zou et al. (2016) Han Zou, Baoqi Huang, Xiaoxuan Lu, Hao Jiang, and Lihua Xie. 2016.  Standardizing location fingerprints across heterogeneous mobile devices for indoor localization. In _2016 IEEE Wireless Communications and Networking Conference_. IEEE, 1–6.
  * Zou et al. (2017b) Han Zou, Yuxun Zhou, Jianfei Yang, Weixi Gu, Lihua Xie, and Costas Spanos. 2017b.  FreeCount: Device-Free Crowd Counting with Commodity WiFi. In _GLOBECOM 2017 - 2017 IEEE Global Communications Conference_. 1–6.  https://doi.org/10.1109/GLOCOM.2017.8255034
  * Zou et al. (2017a) Jianhong Zou, Qianchuan Zhao, W. Yang, and Fulin Wang. 2017a.  Occupancy detection in the office by analyzing surveillance videos and its application to building energy conservation.  _Energy and Buildings_ 152 (2017), 385–398.