# rWiFiSLAM: Effective WiFi Ranging based SLAM System in Ambient Environments

Bo Wei1∗, Mingcen Gao2, Chengwen Luo3, Sen Wang4, Jin Zhang3 * Corresponding
Author1Bo Wei is with School of Computing and Communications, Lancaster
University, Lancaster, UK<EMAIL_ADDRESS>Gao is with Google,
San Francisco, USA<EMAIL_ADDRESS>Luo and Jin Zhang are with
College of Computer Science and Software Engineering, Shenzhen University,
Shenzhen, China<EMAIL_ADDRESS><EMAIL_ADDRESS>Wang is with
Department of Electrical and Electronic Engineering, Imperial College London,
London, UK<EMAIL_ADDRESS>paper is funded by Lancaster University
Start-up Funding and Lancaster Security.

###### Abstract

In this paper, we propose rWiFiSLAM, an indoor localisation system based on
WiFi ranging measurements. Indoor localisation techniques play an important
role in mobile robots when they cannot access good quality GPS signals in
indoor environments. Indoor localisation also has many other applications,
such as rescue, smart buildings, etc. Inertial Measurement Units (IMU) have
been used for Pedestrian Dead Reckoning (PDR) to provide localisation services
in the indoor environment as it does not rely on any other signals. Although
PDR is a promising technique, it still suffers from unavoidable noise and bias
from IMUs in mobile devices. Loop closure is necessary for these scenarios. In
this paper, we design an efficient loop closure mechanism based on WiFi
ranging measurements along with IMU measurements in a robust pose graph SLAM
framework for indoor localisation. One novelty of the proposed method is that
we remove the requirement of the full knowledge of the WiFi access point
locations, which makes our proposed method feasible for new and/or dynamic
environments. We evaluate our designed system in real environments and show
the proposed method can achieve sub-meter localisation accuracy and improve
the localisation performance by more than 90% compared with the IMU based PDR.

##  I INTRODUCTION

Localisation is a fundamental research topic with a wide range of
applications, such as mobile robotics, self-driving cars, and smart buildings.
Localisation also plays an important role in mobile systems, such as mobile
phones, to support many services. Global Positioning System (GPS) is a common
localisation solution, where devices with GPS receive signals from satellites
to localise themselves. However, due to the weak GPS signal perceived in the
indoor environment, the GPS localisation accuracy significantly drops, which
cannot provide sufficiently good performance.

Figure 1: Example of the proposed system: using IMU and WiFi RTT measurements
for indoor localisation

To solve these challenges, many indoor localisation solutions have been
designed for mobile devices [1, 2, 3]. Inertial Measurement Units (IMU) are
common sensors integrated into mobile devices as motion sensors and leveraged
for Pedestrian Dead Reckoning (PDR). Although it can provide promising
localisation performance, the IMU sensor bias generates unavoidable
accumulated trajectory drift. In order to correct the drift in IMU-based PDR
solutions, supplementary sensors, such as cameras [1], mmWave radar [4], etc.,
are used. Since cameras and mmWave radar can capture high-quality and high-
resolution features of the surrounding environment, they can achieve good and
stable localisation performance. However, cameras and mmWave radar cost a high
volume of energy, which is not suitable for long-term usage on mobile devices.
Other radio signal based solutions have been designed, such as Ultra-wide Band
(UWB) based [3] and magneto-inductive device based [5], but they are not
commonly equipped within mobile devices.

WiFi, as a common communication technique, has been widely integrated into
mobile systems. Received Signal Strength Indicator (RSSI) from WiFi has been
used with IMU sensors for indoor localisation, such as [2, 6, 7]. The
localisation performances of these systems are promising, but they need heavy
signal processing methods to mitigate the WiFi signal variance caused by
multi-path effect [8]. Although RSSI has a certain relationship with the
distance between a device and a WiFi access point, the RSSI variance in a
certain location is too huge for accurate localisation. This is because RSSI
was not designed for localisation purposes, and multi-path effect is very
common in the indoor environment. Recently, a new protocol IEEE 802.11mc with
WiFi Round Trip Time (RTT) feature has been designed for ranging between an
RTT capable WiFi access point and a mobile device [9, 10]. With the deployment
of multiple APs with the WiFI RTT (at least 3 access points for 2D
localisation and 4 access points for 3D localisation), a mobile system can
locate itself in the area of interest. However, since trilateration is usually
used on the ranging results between a mobile device and APs, the exact
locations of APs are required. This condition requires frequent
reconfiguration and the information exchanged in a new environment or a
dynamic environment, such as when WiFi routers are moved. The configuration,
i.e., surveying and updating the map of WiFi access points, is labour
expensive, especially in a large and/or dynamic environment, not to mention
localisation in an uncooperative environment. Additionally, due to multiple
path effects in an indoor environment [11], current localisation solutions
only relying on WiFI RTT ranging results may decrease the indoor localisation
performance.

In this paper, we propose a SLAM system based on data fusion on WiFi RTT and
inertial measurements, which achieves good accuracy in indoor localisation.
Different from existing research, we do not need the prior knowledge of the
locations of WiFi access points and the dependence on accurate ranging
results. Instead, the ranging measurements are used as observations directly
for clustering. As discussed, the IMU only based PDR can provide an accurate
trajectory in a relatively short term, but with unavoidable long term drift.
In our paper, we propose to use clusters from WiFi ranging measurements for
loop closure and the PDR drift correction. One novelty of the system is the
removal of the configuration requirement of the deployment of WiFi access
points, which enables the proposed system suitable for a dynamic environment,
such as home environment, where APs are not fixed and/or easily moved, or a
new environment, where mobile devices do not contain the location information
of WiFi access points. Furthermore, we consider the false positives of loop
closures and use a robust pose graph SLAM framework to provide high quality
indoor localisation performance.

To summarise, the contributions of this paper are as follows:

  * •

We design an indoor localisation system based on the observations of WiFi
ranging measurements within a pose graph SLAM framework. Our designed system
removes the prior knowledge of the locations of WiFi access points and the
dependence on accurate ranging results.

  * •

We design an effective clustering method for loop closure along with a robust
SLAM framework to tolerate false positives of loop closing constraints and
provide accurate localisation performance.

  * •

We have conducted extensive experiments in a real environment, which shows our
proposed method has good localisation performance.

The rest part of this paper is organised as follows. Section II discusses the
related works, and we show the overview of the designed system in Section III.
Section IV details the proposed accurate indoor localisation system. Next, in
Section V, extensive experiments are performed in a real environment, which
shows a good and robust localisation performance of the proposed system.
Finally, we conclude the paper in Section VI.

##  II RELATED WORKS

In this section, we will discuss the related works regarding indoor
localisation systems. As a ubiquitous signal, WiFi has been studied for indoor
localisation. [6, 2, 12] used WiFi received signal strength along with IMUs
for indoor localisation. However, due to the multipath effect [8], the use of
received signal strength cannot maintain stable quality for indoor
localisation. The wireless communication research community has also been
aware of the importance of WiFi for indoor localisation, standard IEEE
802.11mc has been designed for ranging [9] and WiFi routers and mobile devices
have started to integrate the new standard [10]. The common methods using WiFi
RTT measurements are based on multilateration, i.e., localisation based on
estimate distances between the mobile device and WiFi access points. The data
fusion of WiFi RTT and IMU measurements has also been studied [13, 14].
Different from it, our proposed method does not rely on accurate ranging
estimate from WiFi RTT measurements. Because the multiple path effect still
exists, albeit having better performance compared with the use of received
strength, it is not easy to estimate the ranging quality of WiFi RTT. Instead,
we focus on the localisation of the mobile device rather than other factors,
such as the location estimate of WiFi access points. We use WiFi RTT
measurements directly as observations for loop closure and apply the pose
GraphSLAM to obtain an accurate trajectory.

Other wireless signals have also been used for localisation. Bluetooth, as
another common wireless communication technique, was also employed to indicate
the locations [15]. However, due to the short communication range of
Bluetooth, a large volume of Bluetooth anchor points are needed for
localisation purposes. UWB is another wireless communication technique. As UWB
is designed for indoor localisation, it can achieve centimetre-level accuracy
[16]. Along with IMU measurements, it can deliver a high-precision context
awareness system, including localisation[3]. The basic principle of UWB for
localisation is also multilateration based on the ranging between mobile
devices and WiFi access points. One fallback for UWB based applications is
that UWB is not as common as WiFi devices, which needs specialised devices and
extra installation.

Additional to wireless technologies, other sensors with high resolutions are
used for localisation, such as laser based [17], mmWave radar based [4],
camera based [1]. Although they can provide very high accuracy localisation
performance, their high energy consumption is one constraint for the large-
scale application in mobile devices. Researchers have also investigated other
signals for localisation purposes, such as magnetic field [5, 18, 19],
photovoltage [20], etc.

There also exist radio based device free localisation methods, i.e., users do
not need to carry any devices and the system localises the user in the area of
interest based on the radio interference by human beings [8].

Localisation is also the core technology for self-driving cars [21], which
uses multiple sensors, such as GPS, IMU, Lidar, and cameras. With the
assistance of a powerful computer, self-driving cars can achieve good
localisation performance. Different from localisation techniques used in self-
driving cars, our paper focuses on the use of readily available sensors on
mobile phones, i.e., WiFi and IMU, for indoor localisation.

Figure 2: System overview of the proposed indoor localisation system:
rWiFiSLAM

##  III SYSTEM OVERVIEW

In this section, we overview the whole structure of the system rWiFiSLAM, and
the flowchart is shown in Figure 2. The user, carrying a mobile phone, walks
in an area of interest with the deployment of WiFi access points. The mobile
phone, equipped with our designed system, can localise itself. To enable the
indoor localisation system, the mobile system continuously collects
acceleration, angular velocity measurements and magnetic field measurements
from its own IMU, which provides an initial trajectory estimate. WiFi RTT
measurements are collected simultaneously, and a clustering method will be
applied on WiFi RTT measurements for the loop closure detection and adjusting
the biased initial trajectory. A robust pose graph SLAM will then be applied
on both initial trajectory and WiFi RTT constraints, which provides an
accurate indoor localisation performance.

##  IV METHOD

In this section, we will show the details of the proposed indoor localisation
method, which includes Pedestrian Dead Reckoning, an efficient loop closure
method based on WiFi RTT measurements and a robust SLAM.

###  IV-A Pedestrian Dead Reckoning

The first step of the system is to provide an initial trajectory based on
inertial navigation. A robust Pedestrian Dead Reckoning (PDR) method is
firstly applied on the collected acceleration, angular velocity and magnetic
field measurements from the accelerometer, gyroscope and magnetometer.
Theoretically, an IMU in a mobile device, containing an accelerometer, a
gyroscope and a magnetometer, can estimate the carrier’s trajectory based on
acceleration and gyroscope. However, the low specification IMU in mobile
devices cannot provide sufficiently accurate measurements because of the noise
in the accelerometer and long term drift in the gyroscope. To solve this
problem, the whole trajectory estimate is split into multiple step motion
estimates. This is based on the fact that the velocity is 0 when both feet are
on the ground during walking and the long term bias can be mitigated at a
certain level by dividing the long term estimate into smaller chunks.
Therefore, the PDR can be conducted by the following two steps, (1) performing
step detection and (2) estimating the length $L$ and attitude $\alpha$ of each
step. In our system, acceleration magnitudes are used for step detection. To
be specific, the norm of $i$th 3D acceleration measurement
$(a_{xi},a_{yi},a_{zi})$ is calculated as
$a_{i}=(a_{xi}^{2}+a_{yi}^{2}+a_{zi}^{2})^{\frac{1}{2}}$. A threshold of
standard deviations within a window size $w$ is set to detect the step
(relatively steady and the corresponding standard deviation is below the
threshold) and moving (the corresponding standard deviation is above the
threshold). Once the steps are detected, the step length is estimated using
Weinberg algorithm [22] by using vertical movement of the hip. It is
empirically shown in [22, 23] that the $j$th step length can be calculated in
Equation 1.

$L_{j}=h\times(\text{max}(\widetilde{a}(j))-\text{min}(\widetilde{a}(j)))^{1/4}$
(1)

,where $a(j)$ represents the vectors of the magnitude of acceleration
measurements (i.e., the norm of 3D acceleration measurements) between $j$th
step and ($j$-1)th step after a low pass filter to remove the measurement
noise. The maximum and minimal values of this vector are obtained by using
max() and min(), respectively. $h$ is a constant. In the meanwhile, the
attitude $\alpha$ of each step is estimated by the angular velocity in the
global reference. Please note that the raw data from IMUs are based on the IMU
reference system. Therefore, the measurements of angular velocity from the IMU
reference system need to be transformed to the global reference with the
assistance of the magnetometer 111More details of the reference transformation
can be found in [24]..

Figure 3: Examples of clustering for loop closures Figure 4: Examples of the
use of robust SLAM for localisation: (a) Ground truth of trajectory in which
user walks in a cycle; (b) Raw trajectory estimate using IMU is biased; (c)
Two loop closures detected. One is a true positive, and the other one is a
false positive; (d) The estimated trajectory is “collapsed” when using
traditional SLAM; (e) Good trajectory estimate when using robust SLAM

###  IV-B Efficient WiFi RTT Measurement Clustering for Loop Closure

Along with trajectory from the PDR method, WiFi RTT measurements are used as
observations directly for the loop closure. WiFi RTT was initially designed
for indoor localisation based on its ranging capability between a mobile
device and a WiFi access point. The common solution using WiFi RTT
measurements is trilateration, i.e., the location estimation based on the
locations of WiFi access points and the corresponding WiFi RTT ranging
measurements. However, due to multipath effect, the ranging from WiFi RTT may
not be sufficiently accurate [11]. Different from existing solutions, we do
not rely on the knowledge of locations of WiFi access points or accurate
ranging results. Instead, we use the WiFi RTT measurements as observations to
directly identify loop closures, based on the fact that the mobile device can
perceive similar WiFi RTT measurements at the same place, as shown in Figure
3. These similar observations will be used to indicate loop closures, i.e.,
the user revisits the same place.

Since the locations of WiFi access points are not known in our setting, it is
impossible to calculate the exact locations based on WiFi RTT ranging
measurements. Therefore, we cannot estimate the relevant locations between two
observations except if they are in the vicinity. When a mobile phone revisits
the same place, the observations from all WiFi RTT measurements are similar.
Our loop closure methods are based on the similarity of the observations
containing all the measurements from WiFi access points.

To enable loop closure, we design an efficient clustering method for WiFi RTT
observations. The $i$th WiFi RTT observation is denoted as
$w_{i}=(w_{1i},w_{2i},...,w_{ki},...,w_{ni})$, where $n$ is the number of WiFi
access points and $w_{ki}$ is the WiFi RTT measurement between the mobile
device and the $k$th WiFi access point in this $i$th observation. Once the
$i$th WiFi RTT observation $w_{i}$ is collected, it is used for clustering
with previous WiFi RTT observations, i.e., $\\{w_{1},w_{2},...,w_{i-1}\\}$, by
using a Euclidean distance $d_{i,k}=||w_{i}-w_{k}||_{2}$. When $d_{i,k}$ is
below a threshold, the corresponding points when $i$th and $k$th WiFi RTT
measurements in the raw estimate trajectory are labelled as a constraint for
loop closure, which means they are candidates of places in the vicinity.
However, due to the multipath effect, WiFi RTT measurements are not
necessarily accurate, and many false positives of loop closures exist during
the clustering method. To solve this, we use a robust SLAM framework to
achieve global optimisation.

###  IV-C Robust SLAM Framework

In this section, we demonstrate the used pose graph SLAM framework in detail.
The traditional pose graph SLAM [25, 26] conducts optimisation based on all
the constraints equally by solving the following optimisation problem in
Equation 2.

$\operatorname*{argmin}_{A}\sum_{i\in L}r_{i}^{T}M_{i}r_{i}$ (2)

, where $A$ is the estimated trajectory for the localisation purpose, and $L$
is the constraint set, including IMU raw trajectory and loop closures. $M$ is
the information matrix of all constraints, and $r$ represents the
corresponding error terms. When applying the traditional SLAM framework to our
application, the false positives of constraints from WiFi RTT observations are
equally treated, which will lead to unnecessary loop closures and decrease
indoor localisation performance significantly. In the example in Figure 2(d),
the estimated trajectory is collapsed due to the false positive of loop
closures.

The robust pose graph SLAM optimiser solves this problem by introducing
scaling factors for loop closures. Equation 3 shows the optimiser of the used
robust pose graph SLAM.

$\operatorname*{argmin}_{A}\underbrace{\sum_{i\in
L_{traj}}r_{i}^{T}M_{i}r_{i}}_{\text{(Trajectory
Constraints)}}+\underbrace{\sum_{i\in
L_{loop}}s_{i}^{2}r_{i}^{T}M_{i}r_{i}}_{\text{(Loop Closures)}}$ (3)

When using the robust pose graph SLAM optimiser, constraints from raw IMU
based trajectories and loop closures. Constraints from raw IMU trajectories
are used as that in the traditional graph SLAM, while the consideration of
loop closures involves a scaling factor $s$ defined in 4.

$s_{i}=\operatorname*{min}(1,\frac{2C}{C+\Theta_{i}^{2}})$ (4)

, where $C$ is a free parameter, and $\Theta$ is also the error term of loop
constraints [27]. The error terms for loop closures depend on the used
threshold for the loop closure detection. The use of the scaling factor $s$
can eliminate the impact of false positives of loop closure by providing less
weight to them and relying more on the constraints from IMU based trajectories
when conducting the optimisation. Please refer to [25, 26, 28, 27] for more
technical details regarding the traditional pose graph SLAM and robust pose
graph SLAM.

(a) Google Pixel 6 Mobile Phone

(b) WiFi Access Points

Figure 5: Hardware: (a) Google Pixel 6 Mobile Phone carried by the user in the
designed system. IMU measurements and WiFi RTT measurements are collected
using this mobile phone for localisation purpose.

(a) Ground Truth

(b) Trajectory Only Using IMU

(c) Trajectory with Traditional SLAM

(d) Trajectory Using Proposed Method

(e) CDF of localisation errors

Figure 6: Performances: (a) Ground Truth; (b) Trajectory Only Using IMU; (c)
Trajectory with Traditional SLAM; (d) Trajectory Using Proposed Method; (e)
CDF of localisation errors

##  V EVALUATION

In this section, we will show the details of the used hardware, performance
metric and the evaluation of the proposed system in real environments.

###  V-A Hardware

Figure 5 shows the hardware used in the evaluation. We use one Google Nest
WiFi router and three Google Nest WiFi points [29] as the WiFi access points.
The mobile device we use is Google Pixel 6. Other devices supporting IEEE
802.11mc can also be used for the same purpose. The full list of devices
supporting IEEE 802.11mc can be found in [10]. WiFi access points are deployed
on one floor of a house. There is no specific configuration for Google Nest
WiFi routers and Nest WiFi points other than enabling WiFi RTT functions using
the official Google WiFi App [30]. For Google Pixel 6, the sampling rate of
the IMU is set as 100 Hz, and the sampling rate of WiFi RTT is empirically set
as 5 Hz. A higher sampling rate could be set in the mobile device, but that
will lead to packet loss because of the burden of processing the ranging
requests in the router end222One WiFi ranging measurement is averaged over a
sequence of the ranging requests in the lower level, but the API to access
each measurement is not available in the current Android Version Android 12
(version 8.4). .

###  V-B Performance Metric

Our proposed system will focus on 2D localisation in the indoor environment.
In this section, we use the following metrics to evaluate the proposed indoor
localisation system.

  * •

Trajectories: Trajectories using the proposed method, IMU based PDR and that
using traditional SLAM [25] as well as ground truth will be shown using
figures, which will provide an intuitive image of the good performance of the
proposed method.

  * •

Root mean square of errors of the trajectory:
$RMSE=\sqrt{\Sigma^{i_{e}}_{i=i_{s}}{e(i)^{2}}/{(i_{e}-i_{s})}}$, where
$i_{s}$ and $i_{e}$ are the start point and end point of the whole trajectory,
and $e(i)$ the error at the time point $i$. $e(i)=||l(i)-g(t)||_{2}$, where
$l(i)$ and $g(t)$ are estimated locations and ground truth in 2D,
respectively.

  * •

Cumulative distribution function (CDF) of localisation errors: The CDF of
localisation errors demonstrates the frequency that a localisation error is
less than or equal to a corresponding error.

  * •

Localisation error of the end point: As the proposed localisation method is
based on the trajectory, a good quality localisation system should not have a
huge cumulative error in the end.

###  V-C Experiment in Real Environment

To evaluate the proposed system, the user walks on one floor in a house within
10 m $\times$ 5 m area. Figure 6(a) shows the ground truth of the trajectory.
The situation is similar to walking in a house, where there is furniture, so
that people can only walk in a certain area. The ground truth is obtained by
manually adding loop closures to key points, such as narrow corridors where
people have to walk by.

In this experiment, the total trajectory path is 308.8 m. Figure 6(b) shows
the trajectory only using IMU. Compared with Figure 6(a), there are two
observations: (1) the shape of the trajectory is similar to the ground truth;
(2) the trajectory has accumulative bias compared with the ground truth. This
confirms that it is feasible to use an IMU based PDR method for the indoor
localisation system, and a secondary mechanism, such as the loop closure
method, is required to calibrate the raw trajectory. After considering WiFi
RTT based loop closures, false positives are introduced, i.e., due to the
noise and multipath effect, distant points could be clustered falsely for
further loop closure. When using the traditional pose graph SLAM [25], as
shown in Figure 6(c), it cannot provide accurate localisation performance.
Different from traditional SLAM, the robust pose graph SLAM, as discussed,
uses various weights for loop closures during the optimisation process, so the
false positives of loop closures can be given less weight. Figure 6(d) shows
the trajectory using the proposed method, which clearly shows its good
performance and outperforms the IMU based PDR method. The trajectory in Figure
6(d) also confirms that the robust SLAM has the capability to handle the false
positives of loop closures and obvious enhancement compared with the
trajectory using traditional SLAM shown in Figure 6(c).

When looking at RMSE of the trajectory, the use of IMU PDR can achieve 2.67 m.
When using traditional SLAM, the RMSE of the trajectory is 5.95 m. When using
the proposed method, the RMSE of the trajectory can reach 0.13 m, which has
significantly been increased compared with the use of IMU and traditional
SLAM. The good performance of the proposed method can be further confirmed by
the localisation error of the end point. When using IMU based PDR, the
localisation error of the end point is 9.84 m. The use of the traditional SLAM
and our proposed method can achieve nearly 0 m. The good performance in this
metric for the proposed method and traditional SLAM is due to loop closures
near the end point. Although the use of traditional SLAM can achieve good
performance in this metric, it can still be clearly seen in Figure 6(c) that,
for the most part of its estimate trajectory, the use of the traditional SLAM
cannot achieve satisfactory localisation performance.

Figure 6(e) shows the CDF of errors using different methods. It demonstrates
that the 90th percentile tracking errors for the IMU based PDR, traditional
SLAM and the proposed method are 9.12 m and 4.46 m, and 0.21 m, respectively.
This means, when using the CDF of localisation errors as a metric, the
proposed method can have 97.6% and 95.2% enhancement compared with the PDR and
traditional SLAM, respectively.

##  VI CONCLUSIONS

In this paper, we use WiFi RTT measurements as observations for loop closure.
A robust pose graph SLAM system is applied on both types of constraints, i.e.,
raw trajectories from IMU measurements and loop closures from WiFi RTT
measurements. An efficient clustering method for WiFi RTT measurements has
been proposed for loop closure, which, different from traditional methods,
does not rely on the knowledge of the locations of WiFi access points and
accurate ranging results. The robust pose graph SLAM can also help handle
false positives of loop closures, which provides a good indoor localisation
performance. To evaluate our proposed method, we have conducted experiments in
a real indoor environment, which can achieve 0.13 m accuracy and clearly shows
the proposed method outperforms the IMU based PDR and the localisation using
traditional SLAM by more than 90%.

## References

  * [1] E. S. Jones and S. Soatto, “Visual-inertial navigation, mapping and localization: A scalable real-time causal approach,” _The International Journal of Robotics Research_ , jan 2011. [Online]. Available: http://ijr.sagepub.com/cgi/doi/10.1177/0278364910388963
  * [2] J. Huang, D. Millman, M. Quigley, D. Stavens, S. Thrun, and A. Aggarwal, “Efficient, generalized indoor WiFi GraphSLAM,” _Proceedings - IEEE International Conference on Robotics and Automation_ , pp. 1038–1043, 2011.
  * [3] D. Roetenberg, H. Luinge, and P. Slycke, “Xsens mvn: full 6dof human motion tracking using miniature inertial sensors,” _Xsens Motion Technologies BV, Tech. Rep_ , 2009.
  * [4] C. X. Lu, S. Rosa, P. Zhao, B. Wang, C. Chen, J. A. Stankovic, N. Trigoni, and A. Markham, “See through smoke: robust indoor mapping with low-cost mmwave radar,” in _Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services_ , 2020, pp. 14–27.
  * [5] B. Wei, N. Trigoni, and A. Markham, “imag: Accurate and rapidly deployable inertial magneto-inductive localisation,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 99–106.
  * [6] B. Ferris, D. Fox, and N. Lawrence, “WiFi-SLAM using Gaussian process latent variable models,” pp. 2480–2485, 2007.
  * [7] P. Mirowski, T. K. Ho, Saehoon Yi, and M. MacDonald, “SignalSLAM: Simultaneous localization and mapping with mixed WiFi, Bluetooth, LTE and magnetic signals,” in _International Conference on Indoor Positioning and Indoor Navigation_. IEEE, oct 2013, pp. 1–10. [Online]. Available: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6817853
  * [8] B. Wei, A. Varshney, N. Patwari, W. Hu, T. Voigt, and C. T. Chou, “drti: Directional radio tomographic imaging,” in _Proceedings of the 14th International Conference on Information Processing in Sensor Networks_. ACM, 2015, pp. 166–177.
  * [9] “Ieee 802.11tm wireless local area networks.” [Online]. Available: http://www.ieee802.org/11/
  * [10] “Wi-fi rtt (ieee 802.11mc): Android open source project.” [Online]. Available: https://source.android.com/devices/tech/connect/wifi-rtt
  * [11] M. Ibrahim, H. Liu, M. Jawahar, V. Nguyen, M. Gruteser, R. Howard, B. Yu, and F. Bai, “Verification: Accuracy evaluation of wifi fine time measurements on an open platform,” in _Proceedings of the 24th Annual International Conference on Mobile Computing and Networking_ , 2018, pp. 417–427.
  * [12] R. Liu, S. H. Marakkalage, M. Padmal, T. Shaganan, C. Yuen, Y. L. Guan, and U.-X. Tan, “Collaborative slam based on wifi fingerprint similarity and motion information,” _IEEE Internet of Things Journal_ , vol. 7, no. 3, pp. 1826–1840, 2019.
  * [13] G. Guo, R. Chen, F. Ye, Z. Liu, S. Xu, L. Huang, Z. Li, and L. Qian, “A robust integration platform of wi-fi rtt, rss signal and mems-imu for locating commercial smartphone indoors,” _IEEE Internet of Things Journal_ , 2022\.
  * [14] C. Gentner and D. Avram, “Wifi-rtt-slam: Simultaneously estimating the positions of mobile devices and wifi-rtt access points,” in _Proceedings of the 34th International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS+ 2021)_ , 2021, pp. 3142–3148.
  * [15] Z. D. Tekler, R. Low, B. Gunay, R. K. Andersen, and L. Blessing, “A scalable bluetooth low energy approach to identify occupancy patterns and profiles in office spaces,” _Building and Environment_ , vol. 171, p. 106681, 2020.
  * [16] “Dacawave TREK1000 Evaluation Kit,” https://www.decawave.com/product/trek1000-evaluation-kit/, accessed: 2019-10-23.
  * [17] “A Laser-Aided Inertial Navigation System (L-INS) for human localization in unknown indoor environments,” _Proceedings - IEEE International Conference on Robotics and Automation_ , pp. 5376–5382, 2010.
  * [18] B. Wei, N. Trigoni, and A. Markham, “imag+: An accurate and rapidly deployable inertial magneto-inductive slam system,” _IEEE Transactions on Mobile Computing_ , 2021.
  * [19] S. Wang, H. Wen, R. Clark, and N. Trigoni, “Keyframe based large-scale indoor localisation using geomagnetic field and motion pattern,” in _2016 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2016, pp. 1910–1917.
  * [20] B. Wei, W. Xu, C. Luo, G. Zoppi, D. Ma, and S. Wang, “Solarslam: Battery-free loop closure for indoor localisation,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 4485–4490.
  * [21] W.-C. Ma, I. Tartavull, I. A. Bârsan, S. Wang, M. Bai, G. Mattyus, N. Homayounfar, S. K. Lakshmikanth, A. Pokrovsky, and R. Urtasun, “Exploiting sparse semantic hd maps for self-driving vehicle localization,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 5304–5311.
  * [22] H. Weinberg, “Using the adxl202 in pedometer and personal navigation applications,” _Analog Devices AN-602 application note_ , vol. 2, no. 2, pp. 1–6, 2002.
  * [23] A. R. J. Ruiz, “Pedestrian dead-reckoning (pdr) tutorial,” in _Proceedings of Eighth International Conference on Indoor Positioning IPIN 2017_ , 2017.
  * [24] O. J. Woodman, “An introduction to inertial navigation,” University of Cambridge, Computer Laboratory, Tech. Rep., 2007.
  * [25] S. Thrun and M. Montemerlo, “The GraphSLAM algorithm with applications to large-scale mapping of urban structures,” _International Journal on Robotics Research_ , vol. 25, no. 5/6, pp. 403–430, 2005.
  * [26] G. Grisetti, R. Kümmerle, C. Stachniss, U. Frese, and C. Hertzberg, “Hierarchical optimization on manifolds for online 2d and 3d mapping,” in _2010 IEEE International Conference on Robotics and Automation_. IEEE, 2010, pp. 273–278.
  * [27] P. Agarwal, G. D. Tipaldi, L. Spinello, C. Stachniss, and W. Burgard, “Robust map optimization using dynamic covariance scaling,” in _2013 IEEE International Conference on Robotics and Automation_. Citeseer, 2013, pp. 62–69.
  * [28] R. Kummerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “G2o: A general framework for graph optimization,” in _2011 IEEE International Conference on Robotics and Automation_. IEEE, may 2011, pp. 3607–3613. [Online]. Available: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5979949
  * [29] “Google wi-fi systems.” [Online]. Available: https://store.google.com/product/google_wifi?hl=en-GB
  * [30] “Wifirttscan app.” [Online]. Available: https://play.google.com/store/apps/details?id=com.google.android.apps.location.rtt.wifirttscan&hl=en&gl=US


# Advancements in Repetitive Action Counting: Joint-Based PoseRAC Model With
Improved Performance

Haodong Chen, Ming C. Leu
Department of Mechanical and Aerospace Engineering
Missouri University of Science and Technology
<EMAIL_ADDRESS>
Md Moniruzzaman, Zhaozheng Yin
Department of Biomedical Informatics and Department of Computer Science
Stony Brook University
Solmaz Hajmohammadi, Zhuoqing Chang
Peloton Interactive

###### Abstract

Repetitive counting (RepCount) is critical in various applications, such as
fitness tracking and rehabilitation. Previous methods have relied on the
estimation of red-green-and-blue (RGB) frames and body pose landmarks to
identify the number of action repetitions, but these methods suffer from a
number of issues, including the inability to stably handle changes in camera
viewpoints, over-counting, under-counting, difficulty in distinguishing
between sub-actions, inaccuracy in recognizing salient poses, etc. In this
paper, based on the work done by [1], we integrate joint angles with body pose
landmarks to address these challenges and achieve better results than the
state-of-the-art RepCount methods, with a Mean Absolute Error (MAE) of 0.211
and an Off-By-One (OBO) counting accuracy of 0.599 on the RepCount data set
[2]. Comprehensive experimental results demonstrate the effectiveness and
robustness of our method.

##  1 Introduction

Repetitive counting (RepCount) is essential for monitoring physical
activity[3], assessing exercise performance [4], and guiding the
rehabilitation process [5]. Accurate and reliable RepCount algorithms are
critical for providing accurate feedback to users, enabling objective
assessments, and facilitating personalized fitness programs [6, 7].

Existing RepCount methods, such as the method in [2], rely on the red-green-
and-blue (RGB) frame inputs, which provide a solution for RepCount but cannot
independently isolate and recognize periodic movement. Some other studies,
such as those by [8] et al. and [9] et al., have achieved better performance
by using contextual information in repetitive actions. In addition, [1]
proposed a pose saliency transformer for RepCount, which achieved state-of-
the-art performance that has been maintained until our improvement. In this
paper, we integrate joint angles with the pose landmark model proposed by [1]
to address the RepCount problem, and our method addresses several existing
issues in RepCount, including the inability to stably deal with changes in
camera viewpoints, over-counting, under-counting, difficulty in distinguishing
between sub-actions, and inaccuracy in recognizing salient poses. By
addressing these issues, our model is able to perform a more accurate and
robust counting of repetitive actions on the RepCount data set proposed by
[2].

###  1.1 Related Works

In our work, we adopt and improve the RepCount method proposed in [1], which
introduces a new mechanism called Pose Saliency Representation (PSR). This
mechanism uses the two most salient poses to represent the action, providing a
more efficient alternative to the common RGB frame-based representation.
Traditional RGB frame-based methods require complex computations to obtain
high-level semantic information from intra-frame spatial and inter-frame
temporal information. In contrast, as shown in Fig. 1, the PSR utilizes only
two salient poses to capture the features of each action, thus establishing a
unique mapping between salient poses and RepCount.

Figure 1: Pose saliency annotation. Instead of annotating the start and end,
the two most salient poses were annotated in [1]

Based on the PSR, a Pose Saliency Transformer for Repetitive Action Counting
(PoseRAC) is proposed in [1]. Their PoseRAC consists of three key components:
(1) pose extraction, which uses state-of-the-art algorithms, i.e., BlazePose
[10], to extract poses from all frames; (2) a core model, which maps each
extracted pose to an action category; and (3) a lightweight action trigger
model, which performs video-level RepCount using the actions in each frame.

In work done by [1], RepCount relies heavily on human pose estimation, which
uses landmark detection to identify the joint positions of human skeletons.
However, these methods often suffer from several drawbacks, such as the
inability to stably deal with changes in camera viewpoints, over-counting,
under-counting, difficulty in distinguishing between sub-actions, inaccuracy
in recognizing salient poses, etc.

To overcome these limitations, we propose a novel approach that integrates 5
joint angles with body poses for the RepCount analysis, which improves
performance and addresses the aforementioned challenges.

###  1.2 Contribution

The contribution of this paper is as follows:

  * •

We analyze different combinations of joint angles and body pose landmarks in
solving the RepCount problem.

  * •

We improve the RepCount performance in dealing with changes in camera
viewpoints, over-counting, under-counting, difficulty in distinguishing
between sub-actions, inaccuracy in recognizing salient poses, etc.

  * •

Our experimental results obtain better performance than the state-of-the-art
results on the public data set (RepCount). [2].

The structure of the rest content is as follows: we first introduce our
approach in Section Sec. 2. Next, we provide experimental results in Section
Sec. 3. Finally, we give conclusions in Section Sec. 4.

##  2 Pose and Joint Angle Annotation

As shown in Fig. 2, the 33 pose landmarks are extracted by the Google
Mediapipe BlazePose model [10]. Five joint angles are extracted based on the
landmarks shown in Fig. 2, which are elbow, shoulder, hip, knee, and ankle
angles.

Figure 2: BlazePose landmarks and five joint angles

###  2.1 Data Set Annotation Correction

After analyzing the original data set RepCount [2], we noticed that the ground
truth of the data stu4_5.mp4 in the test data set was incorrectly labeled as
51, whereas the correct ground truth label should be 5, and we corrected the
label annotation.

###  2.2 RepCount Visualization - Density Map

After pose mapping using the Swin Transformer in [11], we can obtain the score
for each frame and generate a density map as shown in Fig. 3. Higher values
indicate a higher similarity to the salient pose I, while lower values
indicate a higher match to the salient pose II. The action-trigger mechanism
is used to compute the time at which two salient poses appear in sequence in
an action category, where a specific upper and lower limit is set to
differentiate the scores of the two salient poses, thus clustering the non-
salient poses in the middle and easily categorizing the salient poses at both
ends [1, 12, 13, 14].

Figure 3: Action-trigger mechanism using density map [1].

##  3 Experiments and Results

###  3.1 Experiment Setup

The experimental platform is a workstation with an Ubuntu 16.04 system
equipped with an Intel Xeon Gold 6226R CPU, an NVIDIA GeForce RTX 3090
graphics card, and 64343M RAM. The model was trained on one GPU in 20 minutes
for 20 epochs.

###  3.2 Evaluation of Different Scenarios Using Joint Angles

We evaluate different scenarios using joint angles in solving the RepCount
problem, including:

  * •

using the five left joint angles, i.e., [left $A_{e}$, left $A_{s}$, left
$A_{h}$, left $A_{k}$, left $A_{a}$ ]

  * •

using five left and right joint angles, i.e., [left $A_{e}$, right $A_{e}$,
left $A_{s}$, right $A_{s}$, left $A_{h}$, right $A_{h}$, left $A_{k}$, right
$A_{k}$, left $A_{a}$, right $A_{a}$]

  * •

using average values of the five left and right joint angles, i.e., [average
$A_{e}$, average $A_{s}$, average $A_{h}$, average $A_{k}$, average $A_{a}$]

The $A_{e}$, $A_{s}$, $A_{h}$, $A_{k}$, and $A_{a}$ represent the elbow,
shoulder, hip, knee, and ankle angles, respectively. We consider the single-
side joint angles because all actions in the RepCount data set are symmetric.

In this paper, we adopt the main evaluation metrics used in previous work [8,
9, 2, 1], i.e., Mean Absolute Error (MAE) and Off-By-One (OBO) counting
accuracy. MAE represents the normalized absolute error between ground truth
and prediction, while OBO measures the repetitive count rate of the entire
data set. They can be defined as:

$MAE=\frac{1}{N}\sum_{i=1}^{N}\frac{\left|\widetilde{c}_{i}-c_{i}\right|}{\widetilde{c}_{i}}$
(1)
$OBO=\frac{1}{N}\sum_{i=1}^{N}\left[\left|\widetilde{c}_{i}-c_{i}\right|\leq
1\right]$ (2)

where $\widetilde{c}_{i}$ is the ground truth count, $c_{i}$ is the prediction
count, and $N$ is the number of videos.

The comparison results for the different scenarios are shown in Table 1. We
find that the case using landmarks and joint angles performed better than the
case using only landmarks. In the cases using landmarks and joint angles, the
best performance is obtained by using landmarks and the average values of the
five left and right joint angles, in which the MAE $\approx$ 0.211 and OBO
$\approx$ 0.599. Table 2 shows that on the RepCount data set, our method
consistently outperforms previous methods for both evaluation metrics, with an
MAE metric of 0.211 compared to the 0.236 of the PoseRAC and an OBO metric of
0.599 compared to the 0.559 of the PoseRAC.

Different Cases | MAE | OBO
---|---|---
Only landmarks [1] | 0.236 | 0.559
Landmarks + five left joint angles | 0.227 | 0.571
Landmarks + five left and right joint angles | 0.213 | 0.587
Landmarks + average values of the five left and right joint angles | 0.211 | 0.599
Table 1: Comparison of different cases using landmarks and joint angles. Methods | RepCount
---|---
MAE | OBO
Video-level | Zhang et al. [9] | 0.879 | 0.155
Huang et al. [15] | 0.526 | 0.160
TransRAC [2] | 0.443 | 0.291
Pose-level | PoseRAC [1] | 0.236 | 0.559
PoseRAC + joint angles (Ours) | 0.211 | 0.599
Table 2: Performance comparison on RepCount(-pose).

###  3.3 Visualization Comparison of Models: Landmark-Only vs. Landmarks +
Joint Angles

To validate the effectiveness of our proposed method, we visually analyze the
output density maps obtained using two models: the model using only the
landmarks and the model integrating the 5 average joint angles with the
landmarks. The density map represents the density of a particular human pose
in an input video sample. Higher values (close to 1) indicate a higher
similarity to salient pose I, while lower values (close to 0) indicate a
higher match to salient pose II. The density map provides insight into the
distribution of the two salient poses throughout the video. Our experiments
focus on visualizing and comparing the following issues in RepCount: inability
to stably deal with changes in camera viewpoints, over-counting, under-
counting, difficulty in distinguishing sub-actions, inaccuracy in recognizing
salient poses, etc.

Inability to stably deal with changes in camera viewpoints: As shown in Fig.
4, we observe that the density map integrating the 5 average joint angles with
the landmarks exhibits a more accurate capture of salient poses when the
camera viewpoint changes. Specifically, the density values for the salient
pose I remain consistently high (close to 1) after changing the camera
viewpoint. In contrast, the density map using only the landmarks shows a
significant drop in density values after the camera viewpoint changes from the
front view to the side view, with density values $\approx$ 0 in the 440-800
frame range. This difference suggests that compared to using only the
landmarks, using both the landmarks and 5 average joint angles enables the
model not to miss certain salient poses during camera viewpoint changes,
thereby maintaining the accuracy of RepCount.

Figure 4: Density maps: landmarks-only vs. landmarks + joint angles -
addressing the inability issue to stably deal with changes in camera
viewpoints

Over-counting: As shown in Fig. 5, the density map generated from the
landmarks alone tends to produce over-counts when a subject attempts to
perform the Pull Up action but fails to complete it due to fatigue. The over
counts are generated due to the subject’s slight movement around the salient
pose where the subject’s arms are extended. In such cases, a landmark-only
model may misinterpret these attempts as valid transitions between two salient
poses, resulting in overcounts in RepCount. However, incorporating the 5
average joint angles allows the model to recognize salient poses more
accurately. As shown in Fig. 5, when the subject attempts to perform a Pull Up
action but fails to complete it, this phenomenon is reflected in the density
map obtained by integrating the 5 average joint angles with the landmarks,
where the density value is lower than that for the salient pose I, i.e., the
subject fully completes the Pull Up action with the arms bent. Integrating the
5 average joint angles helps the model recognize when a subject’s attempts are
unsuccessful, thus avoiding over-counting these partial or incomplete
attempts.

Figure 5: Density maps: landmarks-only vs. landmarks + joint angles -
addressing the over-counting issue

Under-counting: As shown in Fig. 6, when a subject endeavors to perform a Side
Raise action continuously, the density map derived exclusively from the
landmarks fluctuates irregularly between 0 and 1 and shows multiple peaks
between 0.5 and 0.75, rather than regularly fluctuating between 0 (salient
posture II) and 1 (salient posture I). However, the density map integrating
the 5 average joint angles with the landmarks shows density values regularly
fluctuating between 0 and 1, representing the subject’s movements between the
two salient poses. Also, compared to the density map using only the landmarks,
which cannot accurately identify the salient pose I in the 200-400 and 610-800
frame ranges, the density map obtained using both the landmarks and 5 average
joint angles can accurately identify the salient pose I and provide the
correct density values $\approx$ 1 for the salient pose I. This sample
suggests that integrating the 5 average joint angles with the landmarks in
RepCount makes the model more sensitive to salient poses than using only the
landmarks, thus improving performance when dealing with the under-counting
issue.

Figure 6: Density maps: landmarks-only vs. landmarks + joint angles -
addressing the under-counting issue

Difficulty in distinguishing sub-actions: As shown in Fig. 7, an additional
jump action is present at the end of the Pommel Horse action, which has a
lower limb feature similar to the regular Pommel Horse action and is a sub-
action of the Pommel Horse. The density map obtained using only the landmarks
treats this jump action as a valid count for Pommel Horse, with a peak close
to 1 at around frame 340 in Fig. 7, which suggests that an additional count is
computed using the landmark-only model, and illustrates the difficulty in
distinguishing sub-actions in RepCount using the landmark-only model. However,
the density map integrating the 5 average joint angles with the landmarks
effectively identifies the jump action is not a valid count for Pommel Horse,
resulting in a lower density value $\approx$ 0 for this irrelevant sub-action,
which in turn provides a more accurate RepCount.

Figure 7: Density maps: landmarks-only vs. landmarks + joint angles -
addressing the difficulty in distinguishing sub-actions issue

Inaccuracy in recognizing salient poses: As shown in Fig. 8(a), it is
difficult for a model using only the landmarks to provide consistent and
reliable density values for the salient pose I (straightened arms) in the
RepCount of the action Bench Press. Instability occurs when the subject is
hindered by factors such as fatigue or slight movements during the execution
of the salient pose I, resulting in drops in the density values of the
continuous salient pose I, which should be close to 1. However, by integrating
the 5 average joint angles with the landmarks, the model consistently and
accurately identifies the salient pose I (straightened arms) with a stable
density value of 1. A similar performance is shown in Fig. 8(b), where the
subject performs the Jump Jack action on the right side of the camera view.
The density map obtained using only the landmarks does not accurately
differentiate between the two salient poses, resulting in small-scale
fluctuations between 0 and 0.25 in the beginning half of the density map,
while the correct density value for the salient pose I should be close to 1.
However, the density map obtained using both the landmarks and 5 average joint
angles provides a clear RepCount of the Jump Jack action, with the density
values fluctuating uniformly between 0 and approximately 1.

(a) Inaccuracy in recognizing salient poses in Bench Press

(b) Inaccuracy in recognizing salient poses in Jump Jack

Figure 8: Density maps using landmarks-only and landmarks + joint angles -
addressing the inaccuracy issue in recognizing salient poses

These experimental results confirm the advantages of our proposed approach
integrating the 5 average joint angles with the landmarks in solving the
following issues in RepCount: inability to stably deal with changes in camera
viewpoints, over-counting, under-counting, difficulty in distinguishing sub-
actions, and inaccuracy in recognizing salient poses, making our method a
robust and effective approach for the RepCount task.

In addition to the improvements demonstrated above in the regular RepCount
cases, we have observed that integrating the 5 average joint angles with the
landmarks can lead to effective and robust RepCount in video samples with
various video effects, such as instantaneous brightness changes, zoom shifts,
etc. Fig. 9 illustrates that the density map using both the landmarks and 5
average joint angles provides more accurate results in RepCount than only
using the landmarks.

Figure 9: Density maps: landmarks-only vs. landmarks + joint angles -
addressing the video effect issue

##  4 Conclusion

In summary, this paper integrates the 5 average joint angles and body
landmarks in solving the Repetitive Counting (RepCount) problem. Our method
significantly improves the performance of RepCount and provides the following
improvements: i) Accurate performance in handling camera viewpoint variations.
ii) Solving the over-counting and under-counting problems. iii) Improving the
recognition of sub-actions. iv) Performing more accurate salient pose
recognition. Our method obtains a mean absolute error (MAE) of 0.211 and an
off-by-one (OBO) counting accuracy of 0.599. Comprehensive experimental
results demonstrate the effectiveness and robustness of our proposed method.
Overall, our results outperform previous state-of-the-art PoseRAC methods and
point the way to future research in the RepCount problem area.

## References

  * [1] Ziyu Yao, Xuxin Cheng, and Yuexian Zou.  Poserac: Pose saliency transformer for repetitive action counting.  arXiv preprint arXiv:2303.08450, 2023.
  * [2] Huazhang Hu, Sixun Dong, Yiqun Zhao, Dongze Lian, Zhengxin Li, and Shenghua Gao.  Transrac: Encoding multi-scale temporal correlation with transformers for repetitive action counting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19013–19022, 2022.
  * [3] Katie-Jane Brickwood, Greig Watson, Jane O’Brien, Andrew D Williams, et al.  Consumer-based wearable activity trackers increase physical activity participation: systematic review and meta-analysis.  JMIR mHealth and uHealth, 7(4):e11819, 2019.
  * [4] Bill Foran.  High-performance sports conditioning.  Human kinetics, 2001.
  * [5] Chenguang Shen, Bo-Jhang Ho, and Mani Srivastava.  Milift: Efficient smartwatch-based workout tracking using automatic segmentation.  IEEE Transactions on Mobile Computing, 17(7):1609–1622, 2017.
  * [6] A Branzan Albu, Robert Bergevin, and Sébastien Quirion.  Generic temporal segmentation of cyclic human motion.  Pattern Recognition, 41(1):6–21, 2008.
  * [7] Alexia Briassouli and Narendra Ahuja.  Extraction and analysis of multiple periodic motions in video sequences.  IEEE transactions on pattern analysis and machine intelligence, 29(7):1244–1261, 2007.
  * [8] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  Counting out time: Class agnostic video repetition counting in the wild.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10387–10396, 2020.
  * [9] Huaidong Zhang, Xuemiao Xu, Guoqiang Han, and Shengfeng He.  Context-aware and scale-insensitive temporal repetition counting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 670–678, 2020.
  * [10] Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann.  Blazepose: On-device real-time body pose tracking.  arXiv preprint arXiv:2006.10204, 2020.
  * [11] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202–3211, 2022.
  * [12] Haodong Chen, Niloofar Zendehdel, Ming C Leu, and Zhaozheng Yin.  Fine-grained activity classification in assembly based on multi-visual modalities.  Journal of Intelligent Manufacturing, pages 1–19, 2023.
  * [13] Daniel Onoro-Rubio and Roberto J López-Sastre.  Towards perspective-free object counting with deep learning.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14, pages 615–629. Springer, 2016.
  * [14] G Sreenu and Saleem Durai.  Intelligent video surveillance: a review through deep learning techniques for crowd analysis.  Journal of Big Data, 6(1):1–27, 2019.
  * [15] Yifei Huang, Yusuke Sugano, and Yoichi Sato.  Improving action segmentation via graph-based temporal reasoning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14024–14034, 2020.


# MOSRA: Joint Mean Opinion Score and Room Acoustics
Speech Quality Assessment

###### Abstract

The acoustic environment can degrade speech quality during communication
(e.g., video call, remote presentation, outside voice recording), and its
impact is often unknown. Objective metrics for speech quality have proven
challenging to develop given the multi-dimensionality of factors that affect
speech quality and the difficulty of collecting labeled data. Hypothesizing
the impact of acoustics on speech quality, this paper presents MOSRA: a non-
intrusive multi-dimensional speech quality metric that can predict room
acoustics parameters (SNR, STI, T60, DRR, and C50) alongside the overall mean
opinion score (MOS) for speech quality. By explicitly optimizing the model to
learn these room acoustics parameters, we can extract more informative
features and improve the generalization for the MOS task when the training
data is limited. Furthermore, we also show that this joint training method
enhances the blind estimation of room acoustics, improving the performance of
current state-of-the-art models. An additional side-effect of this joint
prediction is the improvement in the explainability of the predictions, which
is a valuable feature for many applications.

Index Terms: Speech quality assessment, joint learning, room acoustics

##  1 Introduction

Speech quality is becoming increasingly important in daily life with the rise
of video-conferencing solutions, digital content creation, collaborative
remote gaming, and other audio applications. However, it has been
traditionally challenging to measure it, especially when no clean reference is
available. One reason is that many audio dimensions can influence speech
quality. For example, it can be affected by reverberant rooms [1], codecs [2],
network losses [3], or additive noises [4], among others.

The mean opinion score (MOS) belongs to the primary evaluation metrics of
interest. The main method to assess MOS on speech quality is subjective
listening tests, as described by the ITU-T P.800 [5]. The obvious drawback is
that it is time-consuming and expensive to carry out subjective tests. More
recently, crowd-sourcing has been used to evaluate speech samples [6], but
still, it is impractical and costly. Hence the search for objective metrics
for speech quality. The main industry standards used in the last years include
PESQ [7], POLQA [8] or 3-QUEST [9]. All of them have been validated for
specific use cases and might not cover the entire range of distortions present
in today’s applications [10]. In addition, they are intrusive methods, meaning
that they require a clean reference signal to make quality predictions.

Development of non-intrusive metrics for speech quality has been a hot
research topic in recent years, especially with the rise of deep learning.
Indeed, there is now a large variety of models (CNNs [10, 11] , BLSTM [12],
CNN-LSTM [13], CNN-Self Attention [14], CNN-BLSTM-Self Attention [15]) and
input features for the models (raw audio [11], log-mel-spectrograms [13, 14],
MFCCs [10, 16]).

Training these models generally requires a considerable effort in data
collection. Many researchers have tried to use joint-learning to tackle this
problem of lower data availability. The strategy is to learn speech quality
labels jointly with other audio dimensions, hoping that the model will learn
to extract more informative features from the audio without requiring as much
data. In NISQA [14], the model was trained to predict four perceptual speech
quality dimensions (noisiness, coloration, discontinuity, loudness), in
addition to the speech quality itself. An additional advantage of this method
is the improved explainability of the model. MetricNet [17] explores adding a
signal reconstruction branch that aims to reconstruct the clean signal from
the degraded signal for which we are predicting the MOS score. Another multi-
task model learns to jointly predict four objective speech quality and
intelligibility scores (PESQ, ESTOI, HASQI, SDR) [15]. Finally, joint speech
intelligibility prediction and scene classification is presented in [18].

Room acoustics is potentially one of the main factors affecting speech
quality. A different line of research thus has proven possible to predict room
acoustics parameters blindly from audio samples. For example, [19] trains a
Convolutional Recurrent Neural Network to jointly and blindly estimate
reverberation time (T60), clarity (C50 and C80), and direct-to-reverberant
ratio (DRR) for both music and speech. In [20] the authors expand the model to
also predict the speech transmission index (STI) and signal-to-noise ratio
(SNR).

Therefore, we hypothesize that training a model which is explicitly optimized
to capture these multi-dimensional factors will improve the performance of the
quality predictions, especially when a small amount of quality data is
available. Equivalently, we also hypothesize that adding quality MOS will also
improve the prediction of room acoustics parameters. Finally, having a joint
model improves the explainability of the predictions, which is an essential
feature for many applications.

This work combines speech quality and room acoustics data and trains a model
to predict them jointly. We call this model MOSRA (MOS + Room Acoustics). We
evaluate its performance against current state-of-the-art methods on non-
intrusive speech quality metrics and blind room acoustics predictors. The
paper is structured as follows. The next section introduces our model in
detail. Section 3 describes the evaluation datasets that we use and the
baselines against which we compare our model. Finally, sections 4 and 5 show
the results and their discussion, respectively.

##  2 Methods

Figure 1: Overview of the MOSRA model Figure 2: Overview of the CNN that
extracts features from individual segments

This section introduces our proposed MOSRA model, the architecture of which is
shown in Fig. 1. It is designed to be able to learn and predict several tasks
jointly, which, as we will later discuss, can improve the features that are
extracted in the shared parts of the model. The different stages of the model
are described in detail below.

###  2.1 Shared feature extractor

The input features used are Mel-spectrogram segments inspired by [14]. These
are obtained by first computing the Mel-spectrogram of the input audio file
with 48 Mel bands, an FFT window size of 20ms, and a hop size of 10ms. The
maximum Mel band frequency is set to 20kHz, which means that full band audio
is supported. The spectrogram is then separated into overlapping frames, each
with a width of 150ms and a hop size of 40ms.

A CNN, described in Fig. 2, is then used to extract features from individual
segments. Its architecture is inspired by [13] but has additional blocks to
increase its capacity.

The features extracted from each frame are then passed as input to a
Transformer encoder [21] to extract temporal dependencies between the segments
across time. The encoder has $M=2$ layers, $h=1$ head, intermediate fully-
connected layers of dimension $d_{ff}=64$, and accepts inputs with
$d_{\text{model}}=64$ features.

###  2.2 Prediction heads

The outputs from the shared feature extractor are then fed to individual
heads, each of which is entrusted with predicting for a single task.

Each head first has its own Transformer encoder to further extract temporal
dependencies that are of interest for each specific task. Each encoder has
$M=1$ layer, $h=1$ head, $d_{ff}=32$, and $d_{\text{model}}=32$.

The encoder’s outputs are then passed to an attention pooling block, as
described in [14], and then finally to a fully-connected layer which outputs
the final prediction.

###  2.3 Tasks

We jointly train the model to predict MOS, SNR, and the following room
acoustics estimators: T60, STI, DRR, and C50. Room acoustics are some of the
main factors that can affect speech quality, and there isn’t sufficient data
with MOS labels that covers the effects of room acoustics on quality. This
gives us the incentive to try to learn these tasks jointly to improve the
model’s ability to generalize to speech samples that are fairly affected by
acoustics.

###  2.4 Loss function

For each task, we first calculate its MSE loss:

$\text{MSE}=\sum_{i=1}^{n}(y_{pred}-y_{true})^{2}$ (1)

The overall loss is then a weighted sum of the per-task losses:

$\text{Loss}_{\text{room
acoustics}}=\text{MSE}_{\text{SNR}}+\text{MSE}_{\text{T60}}+\text{MSE}_{\text{STI}}+\text{MSE}_{\text{DRR}}+\text{MSE}_{\text{C50}}$
(2)

$\text{Loss}_{\text{overall}}=\lambda_{1}\text{MSE}_{\text{MOS}}+\lambda_{2}\text{Loss}_{\text{room
acoustics}}$ (3)

Since the room acoustics estimators have very different scales, we normalize
their labels and network outputs by calculating the mean and standard
deviation of each estimator across the dataset. We note that, in this study,
we did not explore tuning individual weights for each estimator, but it would
be worth exploring in the future, along with adding additional estimators and
trying to find the best selection of dimensions.

The task weights $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters that are
used, in a sense, to give different levels of priority for the different tasks
[22]. Here, we give priority to the MOS task, giving it a weight
$\lambda_{1}=2$, while scaling down the rest of the tasks with a weight of
$\lambda_{2}=\frac{1}{5}$. The intuition behind the $\lambda_{2}$ value was to
give the five room acoustics tasks as much weight as the single MOS task since
the latter is the main one we are interested in. $\lambda_{1}$ was set to $2$,
following some experiments that showed that increasing the weight of the MOS
task improved its performance without affecting the other tasks.

###  2.5 Training

The model was trained using the ADAM [23] optimizer, a learning rate
$\alpha=5\mathrm{e}{-4}$, a batch size of 32, and early stopping with patience
set to 15.

We use separate datasets for the MOS and room acoustics characterization tasks
in this work. Therefore, given that we would like the model to learn all the
tasks jointly, we opt to interleave MOS data, and room acoustics data [22].
This means that the model performs a forward pass on a batch of MOS data every
iteration and then performs another forward pass on a batch of room acoustics
data. The MOS results from the first pass and the room acoustics results from
the second one are then used to calculate the loss for that iteration.

Finally, the MSE loss for the MOS task is the metric used for early stopping.
Indeed, when training on multiple tasks, the tasks can converge at different
speeds [22], which makes it unfeasible most of the time to find an optimal
stopping point for all tasks. Therefore, we give priority to the MOS task in
this case and stop training when its validation performance stops improving.

##  3 Experimental setup

We evaluate models that are trained on two different sets of data. First, we
discuss models trained on what we will refer to as the Smaller Data, which has
a relatively small amount of speech quality labels. This constraint gave us
the incentive to experiment with joint learning and to assess whether it will
lead to models that generalize better. We then further evaluate models that
are trained on the Challenge Data, which was released for the
ConferencingSpeech 2022111https://tea-lab.qq.com/conferencingspeech-2022
challenge [24] and which significantly increases the amount of speech quality
labels available. We consequently assess the effect that training the model
with the extra data has on performance. We describe the different datasets and
baselines below.

###  3.1 Datasets

####  3.1.1 Smaller Data

The smaller set of data consists of the following:

For the MOS task, the NISQA [14] training datasets, consisting of 11,020
speech files that are between 6 and 12 seconds long, are used for training.
The NISQA validation sets are further used for validation when training the
model. We finally test our models on the NISQA test datasets, and on the
internal ReverbSpeechQuality dataset which was described in [10]. The latter
contains a wide range of reverberations that are applied to the speech. This
focus on room acoustics-related effects on quality distinguishes it from other
datasets which focus more on degradations that are caused by issues with the
communication channel.

For the room acoustics characterization tasks, we prepare and use a dataset as
described in [19, 20]. The training set consists of 80k samples with SNR, STI,
RT60, C50, C80, and DRR labels, while the validation and test sets consist of
10k samples each.

####  3.1.2 Challenge Data

For the challenge, two additional training datasets were open sourced. The
Tencent Corpus contains 10k speech samples without reverberation, and 4k
samples that include reverberation. 80% of it is used for training, and the
rest is used as a test set for the challenge. The PSTN [25] training dataset
consists of around 62k samples, 95% of which is used for training while the
rest is also used as a challenge test set. Additionally, the TUB evaluation
set, consisting of 433 samples, is used to assess models for the challenge.

Our model is optimized to work with audio files that have a sample rate of
48kHz. Therefore, any data that is used which has a smaller sampling rate is
upsampled before being fed to the model.

###  3.2 Baseline systems

We compare our model to several baselines. We first discuss the models trained
on the smaller set of data and then move on to others that are trained on the
challenge data.

####  3.2.1 Smaller Data

First, for the MOS estimation task, we compare the performance difference
between the multi-dimensional MOSRA model, and the same architecture trained
to predict MOS only. We then compare the multi-dimensional model with the
open-source multi-dimensional NISQA model from [14].

For the room acoustics characterization tasks, we compare our model with the
state of the art “universal room acoustics estimator” (URAE) model presented
in [20].

####  3.2.2 Challenge Data

The baseline used for the challenge consists of a variation of the NISQA model
trained on the challenge data for 50 epochs. We thus compare it with MOSRA
models trained on the challenge data. We further compare them with POLQA [8],
which is the current ITU-T recommendation and is intrusive.

###  3.3 Performance metrics

To evaluate the MOS performance of each model, we calculate the Pearson
Correlation Coefficient and RMSE after mapping with a third-order polynomial
function, which are included in the ITU-T P.1401 recommendation [26]. They are
computed for each dataset independently due to the subjective nature of the
tests.

To evaluate the performance on the room acoustics characterization tasks, we
use the RMSE.

Table 1: PCC ($\uparrow$) and RMSE after third-order mapping ($\downarrow$)
comparison of MOSRA models trained on the Smaller Data

Dataset | # Files |  | MOSRA
---
MOS only
| MOSRA
---
Multi-dim
|  | PCC | RMSE | PCC | RMSE
NISQA_TEST_LIVETALK | 232 | 0.63 | 0.73 | 0.68 | 0.69
NISQA_TEST_FOR | 240 | 0.85 | 0.45 | 0.87 | 0.42
NISQA_TEST_P501 | 240 | 0.9 | 0.44 | 0.89 | 0.47
ReverbSpeechQuality | 4032 | 0.77 | 0.59 | 0.85 | 0.48

##  4 Results

###  4.1 Speech Quality

####  4.1.1 Smaller Data

Table 1 first shows the comparison between the multi-dimensional MOSRA model
and a MOS only variation, both trained on the smaller set of data. We can
observe that the multi-dimensional model outperforms the single-task model in
all but one of the cases, most significantly in the ReverbSpeechQuality test.

Table 2: PCC ($\uparrow$) and RMSE after third-order mapping ($\downarrow$)
comparison of MOSRA and NISQA trained on the Smaller Data

Dataset | # Files |  | NISQA
---
Multi-dim
| MOSRA
---
Multi-dim
|  | PCC | RMSE | PCC | RMSE
NISQA_TEST_LIVETALK | 232 | 0.74 | 0.63 | 0.68 | 0.69
NISQA_TEST_FOR | 240 | 0.88 | 0.41 | 0.87 | 0.42
NISQA_TEST_P501 | 240 | 0.92 | 0.4 | 0.89 | 0.47
ReverbSpeechQuality | 4032 | 0.77 | 0.58 | 0.85 | 0.48

Table 3: RMSE ($\downarrow$) performance comparison on the room acoustics
characterization tasks

Model |  | SNR
---
[dB]
STI |  | DRR
---
[dB]
| T60
---
[s]
| C50
---
[dB]
URAE | 3.36 | 0.08 | 4.6 | 0.46 | 9.60
MOSRA Room acoustics | 2.63 | 0.073 | 4.21 | 0.45 | 8.84
MOSRA Multi-dim | 3.22 | 0.072 | 4.15 | 0.44 | 8.62

We also compare the multi-dimensional MOSRA with the multi-dimensional NISQA
model in Table 2. We observe that NISQA performs better than MOSRA on the
NISQA test datasets, while MOSRA performs better on the ReverbSpeechQuality
dataset.

####  4.1.2 Challenge Data

Table 4: PCC ($\uparrow$) and RMSE after third-order mapping ($\downarrow$) comparison of models trained on the Challenge data Dataset | # Files | POLQA |  | Challenge
---
Baseline
| MOSRA
---
MOS only
| MOSRA
---
Multi-dim
|  | PCC | RMSE | PCC | RMSE | PCC | RMSE | PCC | RMSE
NISQA_TEST_LIVETALK | 232 | n/a | n/a | 0.43 | 0.85 | 0.76 | 0.61 | 0.74 | 0.63
NISQA_TEST_FOR | 240 | 0.85 | 0.45 | 0.70 | 0.62 | 0.85 | 0.46 | 0.86 | 0.44
NISQA_TEST_P501 | 240 | 0.88 | 0.49 | 0.63 | 0.78 | 0.86 | 0.48 | 0.9 | 0.45
ReverbSpeechQuality | 4032 | 0.78 | 0.53 | 0.82 | 0.51 | 0.89 | 0.41 | 0.88 | 0.43
Tencent Eval | 2898 |  |  | 0.881 | 0.55 |  |  | 0.941 | 0.393
TUB Eval | 433 |  |  | 0.348 | 0.649 |  |  | 0.535 | 0.589
PSTN Eval | 1039 |  |  | 0.361 | 0.293 |  |  | 0.534 | 0.271

Table 4 first shows the results for the MOSRA models that are trained on the
challenge data. In this case, we can no longer observe a clear difference
between the two models, as they perform similarly on all test sets.

We also compare MOSRA with the challenge baseline. The baseline is clearly
outperformed by the MOSRA models. We note that for the challenge evaluation
sets, we can only present the results for the baseline and our Multi-dim
submission.

Finally, POLQA is also outperformed by the MOSRA models. The
NISQA_TEST_LIVETALK dataset does not have clean reference files alongside the
degraded ones, meaning that POLQA cannot perform predictions for this set.

###  4.2 Room Acoustics Characterization

Table 3 shows the results for the room acoustics estimation tasks. MOSRA
consistently outperforms the URAE model on the five tasks. Further, the model
trained to only predict the room acoustics tasks is better at predicting SNR,
while it is outperformed by the multi-dimensional model on the four other
tasks.

##  5 Discussion & conclusions

Training our model jointly on MOS and room acoustics characterization tasks
significantly helped the model’s performance when the subjective quality data
was scarce, notably when predicting for samples that are greatly affected by
room acoustics-related degradations such as heavy reverberation.

However, that advantage was not as clear when the models were trained on the
much larger amounts of speech quality data that were released for the
challenge. This might either be because the extra data allowed the model to
extract the features learned from the room acoustics characterization tasks,
or because the model was not designed with enough capacity to handle the
substantial data increase. Indeed, the model’s size (410k parameters) is
relatively small and was designed with the smaller set of data in mind.
Nonetheless, we still observe a clear improvement over the baseline on the
challenge evaluation datasets. At the same time, the model’s small size and
architecture make it time and space efficient, and therefore easy to deploy in
many practical settings. On average, it needs 120ms and 30MB to process an 8s
long clip on a single threaded Intel Core i7 clocked at 3.6 GHz.

Further, the model’s performance on the room acoustics characterization tasks
successfully improves on the state of the art, showing the versatility of this
architecture. The results also confirmed our hypothesis that jointly training
the model to predict MOS and the room acoustics tasks would improve its
performance all-round. Indeed, the model’s shared feature extractor seems to
benefit from all the extra data that it’s trained on thanks to joint learning,
and is consequently able to produce features that contain more useful
information.

Finally, having the additional tasks adds a layer of explainability to the
model’s predictions, which is an important feature in many use cases.

## References

  * [1] Y. Zhao, Z.-Q. Wang, and D. Wang, “Two-stage deep learning for noisy-reverberant speech enhancement,” _IEEE/ACM transactions on audio, speech, and language processing_ , vol. 27, no. 1, pp. 53–62, 2018.
  * [2] Y. Zango, R. Le Bouquin Jeannès, and C. Quinquis, “Modeling speech and audio codecs reverberation artifact,” in _2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO)_ , 2012, pp. 2070–2074.
  * [3] L. Sun and E. Ifeachor, “Perceived speech quality prediction for voice over ip-based networks,” in _2002 IEEE International Conference on Communications. Conference Proceedings. ICC 2002 (Cat. No.02CH37333)_ , vol. 4, 2002, pp. 2573–2577 vol.4.
  * [4] K. Arehart, J. Kates, and M. Anderson, “Effects of noise, nonlinear processing, and linear filtering on perceived speech quality,” _Ear and Hearing_ , vol. 31, no. 3, pp. 420–436, 2010.
  * [5] ITU-T Recommendation P.800, “Methods for objective and subjective assessment of quality,” 1996.
  * [6] ITU-T Recommendation P.808, “Subjective evaluation of speech quality with a crowdsourcing approach,” 2021.
  * [7] ITU-T Recommendation P.862, “Perceptual evaluation of speech quality (pesq): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs,” 2001.
  * [8] ITU-T Recommendation P.863, “Perceptual objective listening quality prediction,” 2018.
  * [9] ETSI EG 202 393-3, “Speech processing, transmission and quality aspects (STQ); speech quality performance in the presence of background noise, Part 3: Background noise transmission - Objective test methods,” 2008.
  * [10] N. Nessler, M. Cernak, P. Prandoni, and P. Mainar, “Non-Intrusive Speech Quality Assessment with Transfer Learning and Subject-Specific Scaling,” in _Proc. Interspeech 2021_ , 2021, pp. 2406–2410.
  * [11] A. A. Catellier and S. D. Voran, “Wawenets: A no-reference convolutional waveform-based approach to estimating narrowband and wideband speech quality,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 331–335.
  * [12] S.-W. Fu, Y. Tsao, H.-T. Hwang, and H.-M. Wang, “Quality-net: An end-to-end non-intrusive speech quality assessment model based on blstm,” _arXiv preprint arXiv:1808.05344_ , 2018.
  * [13] G. Mittag and S. Möller, “Non-intrusive speech quality assessment for super-wideband speech communication networks,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 7125–7129.
  * [14] G. Mittag, B. Naderi, A. Chehadi, and S. Möller, “Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,” _Interspeech 2021_ , Aug 2021. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2021-299
  * [15] X. Dong and D. S. Williamson, “An attention enhanced multi-task model for objective speech assessment in real-world environments,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 911–915.
  * [16] A. R. Avila, H. Gamper, C. Reddy, R. Cutler, I. Tashev, and J. Gehrke, “Non-intrusive speech quality assessment using neural networks,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 631–635.
  * [17] M. Yu, C. Zhang, Y. Xu, S.-X. Zhang, and D. Yu, “MetricNet: Towards Improved Modeling For Non-Intrusive Speech Quality Assessment,” in _Proc. Interspeech 2021_ , 2021, pp. 2142–2146.
  * [18] L. Marcinek, M. Stone, R. Millman, and P. Gaydecki, “N-mttl si model: Non-intrusive multi-task transfer learning-based speech intelligibility prediction model with scenery classification,” in _Interspeech_ , Sep. 2021\.
  * [19] P. Callens and M. Cernak, “Joint blind room acoustic characterization from speech and music signals using convolutional recurrent neural networks,” _arXiv preprint arXiv:2010.11167_ , 2020.
  * [20] P. S. López, P. Callens, and M. Cernak, “A universal deep room acoustics estimator,” in _2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_. IEEE, 2021, pp. 356–360.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _CoRR_ , vol. abs/1706.03762, 2017. [Online]. Available: http://arxiv.org/abs/1706.03762
  * [22] R. Caruana, “Multitask learning,” _Machine Learning_ , vol. 28, no. 1, pp. 41–75, Jul 1997. [Online]. Available: https://doi.org/10.1023/A:1007379606734
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [24] G. Yi, W. Xiao, Y. Xiao, B. Naderi, S. Moller, G. Mittag, R. Cutler, Z. Zhang, D. S. Williamson, F. Chen, F. Yang, and S. Shang, “Conferencingspeech 2022 challenge evaluation plan,” 2022.
  * [25] G. Mittag, R. Cutler, Y. Hosseinkashi, M. Revow, S. Srinivasan, N. Chande, and R. Aichner, “DNN No-Reference PSTN Speech Quality Prediction,” in _Proc. Interspeech 2020_ , 2020.
  * [26] ITU-T Recommendation P.1401, “Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models,” 2020.