# Learning Multiplicative Interactions with Bayesian
Neural Networks for Visual-Inertial Odometry

Kashmira Shinde  Jongseok Lee  Matthias Humt  Aydin Sezgin  Rudolph Triebel

# Supplementary Material

Kashmira Shinde  Jongseok Lee  Matthias Humt  Aydin Sezgin  Rudolph Triebel

###### Abstract

This paper presents an end-to-end multi-modal learning approach for monocular
Visual-Inertial Odometry (VIO), which is specifically designed to exploit
sensor complementarity in the light of sensor degradation scenarios. The
proposed network makes use of a multi-head self-attention mechanism that
learns multiplicative interactions between multiple streams of information.
Another design feature of our approach is the incorporation of the model
uncertainty using scalable Laplace Approximation. We evaluate the performance
of the proposed approach by comparing it against the end-to-end state-of-the-
art methods on the KITTI dataset and show that it achieves superior
performance. Importantly, our work thereby provides an empirical evidence that
learning multiplicative interactions can result in a powerful inductive bias
for increased robustness to sensor failures.

Deep Learning, Sensor Fusion, Multi-head Attention, Uncertainty Estimation,
Laplace Approximation

##  1 Introduction

For safety critical applications such as autonomous driving, it is crucial for
learning approaches to effectively rely on multiple sources of information. In
case of a localization problem as an example, learning from complementary
sensor types such as cameras and IMUs can bring improved robustness and
redundancy when there exists sensor corruptions and failures. Existing methods
for learning based VIO rely on different strategies to fuse visual and
inertial data, which include so-called FC-fusion networks (Clark et al.,
2017b; Han et al., 2019) and soft/hard feature selection networks (Chen et
al., 2019; Almalioglu et al., 2019). Broadly speaking, these strategies
typically learn additive interactions for integrating multiple streams of
information, i.e. , given the features u and v from different modalities, the
additive interactions are modeled as
$f(\textbf{u},\textbf{v})=\textbf{W}\texttt{concat}(\textbf{u},\textbf{v})$
where $\texttt{concat}(\textbf{u},\textbf{v})$ is the concatenation of the two
features, and W are the learned fusion network parameters.

Instead, we present an end-to-end trainable deep neural network for monocular
VIO with the focus on how to better exploit sensor complementarity.
Specifically, the designed network architecture (i) learns multiplicative
interactions (Jayakumar et al., 2020) for the fusion network, which is based
on multi-head self-attention layers (section 2.2), and (ii) treat the network
weights as probability distribution using scalable Laplace Approximation (LA)
(MacKay, 1992), resulting in Bayesian Neural Networks (Section 2.5). In our
experiments, we compete with the current methods that learn additive
interactions against the proposed network (section 3.1) and further evaluate
the obtained uncertainty (section 3.2). As a result, we demonstrate improved
performance in terms of accuracy and robustness. We believe that these results
imply that a more elaborated fusion might bring performance improvements for
the end-to-end VIO.

Figure 1: Architecture of the proposed approach. $\sigma^{2}$ denotes the
variance, i.e., uncertainty of the poses.

##  2 Method

We now describe the proposed architecture in figure 1.

###  2.1 Visual-Inertial Feature Extractors

Two sequential images, i.e., uv1, uv2 are stacked together and fed to the
optical flow feature extractor (FlowNet Dosovitskiy et al. (2015) or even Sun
et al. (2018)) gvision. It learns geometric feature representation to be able
to generalise well in unfamiliar environments as opposed to learning
appearance representation. The visual features bv obtained from the final
layer of FlowNet can be expressed as:

bv
$\displaystyle=g_{\text{vision}}(\textbf{u}\textit{\textsubscript{v1}},\textbf{u}\textit{\textsubscript{v2}}).$
(1)

Raw inertial measurements ui falling between two consequent image frames,
i.e., $x,y,z$ components of linear and angular acceleration taken together
forming a 6 dimensional vector are passed to an encoder ginertial. It is
composed of a bidirectional LSTM (Long Short-Term Memory) as done by (Chen et
al., 2018) and operates at a rate at which IMU receives data. The inertial
features bi can be expressed as:

bi $\displaystyle=g_{\text{inertial}}(\textbf{u}\textit{\textsubscript{i}}).$
(2)

###  2.2 Feature Fusion Mechanism

The fusion block combines both the inertial and visual feature streams
generated from the respective feature extractors. Here, a widely used fusion
method is to either simply concatenate the two feature streams into a single
feature space (Clark et al., 2017b) or re-weight them (Chen et al., 2019),
which are so-called additive interactions. In other words, given two different
input modalities u and v, the used fusion blocks rely on rather simple
functions: ($\texttt{concat}(\textbf{u},\textbf{v})$) or a linear layer
$f(\textbf{u},\textbf{v})=\textbf{W}\texttt{concat}(\textbf{u},\textbf{v})+\textbf{a}$.

In contrast to this, we consider so-called multiplicative interactions (MI).
For deep neural networks, MI can be expressed by the following function
(Jayakumar et al., 2020):

$\displaystyle f(\textbf{u},\textbf{v})$
$\displaystyle=\textbf{v}^{\top}\mathcal{W}\textbf{u}+\textbf{v}^{\top}\textbf{W}_{v}+\textbf{W}_{u}\textbf{u}+\textbf{a},$
(3)

wherein the weight matrices $\textbf{W}_{u}$ and $\textbf{W}_{v}$, vector a
and 3D weight tensor $\mathcal{W}$ are the learned parameters. A major
difference to additive interaction is the product term
$\textbf{v}^{\top}\mathcal{W}\textbf{u}$. Here, one might argue that neural
networks are universal approximators, and thus, adding more costly module may
not affect the approximation power of neural networks. Yet, the hypothesis
space or the sets of learnable functions, the compactness of the estimator as
well as learnability can increase as proved by Jayakumar et al. (2020). When
learned correctly, MI can help in preferential selection of hypotheses, and
thus may result in an improved inductive bias.

Motivated by this recent result, we proposed to use a more elaborated fusion
strategy that can learn MI. To this end, we propose a fusion block based on
multi-head self-attention (denoted by $f_{\text{attn}}$) (Vaswani et al.,
2017) which relates different parts of the same sequence to learn a suitable
combined feature representation. An advantage over other MI modules is that
training can be often easier (e.g. alternatives such as gating networks are
often notoriously difficult to train (Shazeer et al., 2017)). In this module,
the concatenated visual and inertial features of dimension $d_{k}$ are passed
to the attention function as query $\mathbf{q}$ and a key-value pair
($\mathbf{k}$,$\mathbf{v}$). In order to learn distinct representations of the
input, $n$ such attention heads perform attention function $\mathcal{A}$ in
parallel for a model of dimension $N$. The output from each head
$\mathbf{h_{i}}$ is then concatenated to produce final attention outputs for a
given concatenated feature representation:

$f_{\text{attn}}(\mathbf{q},\mathbf{k},\mathbf{v})=\mathbf{h}\mathbf{W^{h}},$
(4)

where each head $\mathbf{h_{i}}$ for $i=1,...,n$ is given by

$\displaystyle\mathbf{h_{i}}$
$\displaystyle=\mathcal{A}(\mathbf{q}\mathbf{W_{i}^{q}},\mathbf{k}\mathbf{W_{i}^{k}},\mathbf{v}\mathbf{W_{i}^{v}})$
$\displaystyle\mathcal{A}(\mathbf{q},\mathbf{k},\mathbf{v})$
$\displaystyle=\texttt{softmax}\left(\frac{\mathbf{q}\mathbf{k}^{\top}}{\sqrt{d_{k}}}\right)\mathbf{v},$

where $\mathbf{h}=\texttt{concat}(\mathbf{h_{1}},...,\mathbf{h_{n}})$ and
learned attention weights $\mathbf{W_{i}^{q}}\in\mathbb{R}^{N\times d_{k}}$ ,
$\mathbf{W_{i}^{k}}\in\mathbb{R}^{N\times d_{k}}$ ,
$\mathbf{W_{i}^{v}}\in\mathbb{R}^{N\times d_{k}}$ and
$\mathbf{W^{h}}\in\mathbb{R}^{nd_{k}\times N}$. The attention outputs y
obtained from the visual bv and inertial bi feature vectors which are passed
to pose regression module can be expressed as:

y
$\displaystyle=f_{\text{attn}}(\textbf{b}\textit{\textsubscript{v}},\textbf{b}\textit{\textsubscript{i}}).$
(5)

We note that, while the used module is more costly than other simpler
variants, we did not find training more difficult. Furthermore, we use the
attention mechanism only in the fusion block, and so, the added number of
parameters has smaller contribution to the total number of parameters.

###  2.3 Sequential Modeling and Pose Regression

Next, we describe modeling sequential dependence for ego-motion estimation. In
order to learn complex model dynamics (motion model) and to derive connections
between sequential features, the core LSTM is used. The resultant
representation from the feature fusion block yt at time $t$ is fed to the core
LSTM along with its hidden states from the previous time step
$\textbf{h}_{t-1}$. The fully connected layer does the pose regression and the
output is a 6D camera pose $\textbf{x}_{t}$:

$\displaystyle\textbf{x}_{t}$
$\displaystyle=f_{\text{lstm}}(\textbf{y}\textit{\textsubscript{t}},\textbf{h}_{t-1}).$
(6)

###  2.4 Loss Function

The loss as in Wang et al. (2017) constitutes of the summed up mean squared
error (MSE) between predicted poses $\hat{\textbf{x}_{t}}$ =
($\hat{\textbf{z}}\textit{\textsubscript{t}}$,$\hat{\mathbf{\psi}}\textit{\textsubscript{t}}$)
and ground truth $\textbf{x}_{t}$ =
(zt,$\mathbf{\psi\textit{\textsubscript{t}}}$) (Geiger et al., 2012) where zt
denotes position and $\mathbf{\psi\textit{\textsubscript{t}}}$ the orientation
at time $t$,

$\ell=\frac{1}{M}\sum_{i=1}^{M}\sum_{t=1}^{k}\left\|\hat{\textbf{z}\textit{\textsubscript{t}}}-\textbf{z}\textit{\textsubscript{t}}\right\|^{2}_{2}+\beta\left\|\hat{\mathbf{\psi_{t}}}-\mathbf{\psi_{t}}\right\|^{2}_{2},$
(7)

where $\beta$ is a scale factor to balance the position and orientation
elements for $M$ mini-batch samples. This scaling factor is to balance the
magnitude so that the both translation and rotation are given the similar
level of importance.

###  2.5 Uncertainty estimation

We estimate the predictive uncertainty of our method using a LA similar to
Ritter et al. (2018); Lee et al. (2020b), but only make use of the diagonal
approximation to the Fisher information matrix. Just like MC-dropout (Gal &
Ghahramani, 2016) it is a practical method for uncertainty estimation in deep
learning without the need to change or retrain the model, but without the
reliance on dropout layers.

LA employs information from the inverse curvature (Hessian) of the loss for
the covariance matrix of a multivariate Gaussian as an approximation to the
model’s weight posterior distribution. We collect the output of multiple
stochastic forward passes obtained through sampling from this posterior and
perform Bayesian model averaging using Monte Carlo integration to make
predictions. Note that we do not consider aleatoric uncertainty as we focus on
the influence of model uncertainty on deep sensor fusion. More details can be
found in the supplementary material.

##  3 Experimental Results

In this section, we evaluate our approach in comparison to the baselines -
DeepVO (Wang et al., 2017), VINet (Clark et al., 2017b) and VIO Soft (Chen et
al., 2019). Implementation details can be found in supplementary material.

###  3.1 Analysis of Robustness to Sensor Degradation

(a) Seq 10 (b) Seq 10 with vision deg.

Figure 2: Predicted trajectories of the KITTI dataset comparing MHA and
baselines with ground truth (in green). (a) for non-degraded data. (b) for
degraded vision data (occlusion, noise+blur, missing images) for sequence 10.
Best viewed in colour. Table 1: Comparison of approaches for various sensor
corruptions are reported along with nominal case. For complexity comparison,
the total number of parameters in the networks and fusion blocks are shown.
Per sequence results can be found in supplementary material.

Model |  #Total Param |  #Fusion param | Nominal | Inertial | Vision | All
---|---|---|---|---|---|---
$t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$
DeepVO |  149,518,326 |  N/A |  18.07 |  5.88 |  N/A |  N/A |  Fails |  Fails |  N/A |  N/A
VINet |  294,828,726 |  31,020 |  13.98 |  4.66 |  16.31 |  5.28 |  15.46 |  4.97 |  15.87 |  5.14
VIO Soft |  58,969,014 |  262,656 |  13.65 |  4.83 |  14.05 |  4.94 |  14.59 |  4.58 |  15.48 |  4.63
MHA |  60,019,638 |  1,313,280 |  11.42 |  3.94 |  12.48 |  3.93 |  12.61 |  3.98 |  12.47 |  3.79

  * •

$t_{rel}(\%)$: translational RMSE, $r_{rel}$(°/100m): rotational RMSE.

We first compare the baselines with our approach (referred to as MHA) for
nominal case where dataset has no degradation. In many real world scenarios,
it is plausible for sensors to be imperfect thereby resulting in corrupted
data. For a camera, these corruptions can be in the form of occlusions, noisy
and blurry images as well as missing image frames. The IMU can have bias in
gyroscope data as well as white noise in accelerometer data. To determine the
behaviour of the approaches in such scenarios and evaluate the influence of
the modalities, we prepare three datasets - a dataset with degraded inertial
data, a vision degraded dataset (consisting of image occlusions, blur and
missing data) and all sensor dataset (both vision and inertial data are
degraded).

Table LABEL:table:nom shows mean values of the sequences while its expanded
version can be found in supplementary material. We use the metric provided by
Shamwell et al. (2018) \- average translation RMSE $t_{rel}(\%)$ and average
rotational RMSE $r_{rel}$(°/100m) computed on sub-sequences of length 100m -
800m. In our experiments, we find that our method makes predictions with lower
RMSE on average, when compared the considered baselines across the sequences.
For inertial degradation, it can be seen that as the sensor fusion strategy
becomes elaborate for each approach, the robustness to IMU degradation
increases. For low to moderate intensity of degradation, the performance of
all methods remains similar to nominal case, suggesting visual features are
more dominant than inertial ones. For vision degradation case as the dataset
is severely degraded, from Figure 2 we see that the performance of DeepVO
degrades immensely and tracking is lost completely. Interestingly, when vision
degrades, the translation error increases for all the approaches, but the
rotational error remains comparable to the nominal case. This suggests that
the visual features might contribute to determining translation and inertial
contribute to rotation, and furthermore, inertial features might become more
reliable in case of strong visual degradation.

To summarize, exploiting the combination of sensors can be beneficial for
odometry in comparison to use of a single sensor (DeepVO). Among all three
sensor fusion approaches, our approach being more expressive than VINet and
VIO Soft, performs better than them on average in terms of accuracy (lower
translational and orientation error) in our experiments. Our method appears to
be more robust to sensor corruptions and tends to diverge less as compared to
the other two approaches. We further report total number of parameters in a
network, and also in the fusion block. As expected, our approach shows an
increased cost for the fusion parameters, but requires not as much total
network parameters to obtain an improved performance, suggesting the
competitiveness of MI w.r.t performance vs cost trade-off.

###  3.2 Results of the Bayesian Framework

(a) Nominal (b) Degraded

Figure 3: Uncertainty in some poses of trajectory for Sequence 10.

Figure 4: Box plots of uncertainty vs. errors for translation along z
direction and pitch angle $\theta$ respectively for Sequence 10.

Next, we evaluate uncertainty estimates in the form of variance in mean
predicted poses, which is depicted with the grey shaded region around the
trajectory in Figure 3. From Figure 3b it can be seen that increased variance
expresses the models’ uncertainty about the point. To further evaluate the
behavioral effects of uncertainty, some more representative results can be
seen in Figure 4. The errors and uncertainties are shown for nominal (green)
and occlusion degradation (blue) cases. For Sequence 10 it was observed that
the pose error was large along Z-axis as compared to others. A large
uncertainty can be seen along this axis which suggests that the error
fluctuations can be better captured by it. The box plots show steady increase
in uncertainty along with error (strong correlation with error). Higher
uncertainty is observed for degraded case in all plots. Lastly, end-to-end
pose regression approaches (Kendall et al., 2015) such as ours, are often used
in the hybrid methods such as (Yang et al., 2020) that effectively leverage
the domain knowledge within the learner. In this sense, purely a multi-modal
learning approaches can also be useful for such advanced methods beyond end-
to-end strategy. For future works, further evaluations on other public
datasets (Sun et al., 2019; Caesar et al., 2019) and other applications of VIO
(Lutz et al., 2020; Lee et al., 2020a) can be envisioned.

##  4 Conclusion

We introduced a learning based method for VIO, in which, sensor fusion was
carried out using multi-head self-attention. Our approach was evaluated for
the scenarios with imperfect sensor data, and we demonstrated that robustness
improves with a more expressive fusion strategy. Employing Laplace
Approximation, we made our model capable of making predictions with a notion
of uncertainty, and demonstrated that the errors in the pose and their
confidence can be made correlated for end-to-end VIO based approaches.

## Acknowledgements

We thank the anonymous reviewers and the area chairs for their time and
thoughtful comments. The authors acknowledge the support by the Helmholtz
Association’s Initiative and Networking Fund (INF) under the Helmholtz AI
platform grant agreement (contract number ID ZT-I-PF-5-1), the project ARCHES
(contract number ZT-0033) and the EU-project AUTOPILOT (contract number
731993).

## References

  * Almalioglu et al. (2019) Almalioglu, Y., Turan, M., Sari, A. E., Saputra, M. R. U., de Gusmão, P. P., Markham, A., and Trigoni, N.  Selfvio: Self-supervised deep monocular visual-inertial odometry and depth estimation.  _arXiv preprint arXiv:1911.09968_ , 2019.
  * Caesar et al. (2019) Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O.  nuscenes: A multimodal dataset for autonomous driving.  _arXiv preprint arXiv:1903.11027_ , 2019.
  * Chen et al. (2018) Chen, C., Lu, X., Markham, A., and Trigoni, N.  Ionet: Learning to cure the curse of drift in inertial odometry.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018\.
  * Chen et al. (2019) Chen, C., Rosa, S., Miao, Y., Lu, C. X., Wu, W., Markham, A., and Trigoni, N.  Selective sensor fusion for neural visual-inertial odometry.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Clark et al. (2017a) Clark, R., Wang, S., Markham, A., Trigoni, N., and Wen, H.  Vidloc: A deep spatio-temporal model for 6-dof video-clip relocalization.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017a.
  * Clark et al. (2017b) Clark, R., Wang, S., Wen, H., Markham, A., and Trigoni, N.  Vinet: Visual-inertial odometry as a sequence-to-sequence learning problem.  In _The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17)_ , 2017b.
  * Couzinié-Devy et al. (2013) Couzinié-Devy, F., Sun, J., Alahari, K., and Ponce, J.  Learning to estimate and remove non-uniform image blur.  In _2013 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1075–1082, 2013.
  * Dosovitskiy et al. (2015) Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., and Brox, T.  Flownet: Learning optical flow with convolutional networks.  In _The IEEE International Conference on Computer Vision (ICCV)_ , December 2015.
  * El-Sheimy et al. (2007) El-Sheimy, N., Hou, H., and Niu, X.  Analysis and modeling of inertial sensors using allan variance.  _IEEE Transactions on instrumentation and measurement_ , 57(1):140–149, 2007.
  * Gal & Ghahramani (2016) Gal, Y. and Ghahramani, Z.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pp. 1050–1059, 2016.
  * Geiger et al. (2012) Geiger, A., Lenz, P., and Urtasun, R.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * Geiger et al. (2013) Geiger, A., Lenz, P., Stiller, C., and Urtasun, R.  Vision meets robotics: The kitti dataset.  _The International Journal of Robotics Research_ , 32(11):1231–1237, 2013.
  * Han et al. (2019) Han, L., Lin, Y., Du, G., and Lian, S.  Deepvio: Self-supervised deep learning of monocular visual inertial odometry using 3d geometric constraints.  _arXiv preprint arXiv:1906.11435_ , 2019.
  * Jayakumar et al. (2020) Jayakumar, S. M., Czarnecki, W. M., Menick, J., Schwarz, J., Rae, J., Osindero, S., Teh, Y. W., Harley, T., and Pascanu, R.  Multiplicative interactions and where to find them.  In _International Conference on Learning Representations_ , 2020.
  * Kendall & Cipolla (2016) Kendall, A. and Cipolla, R.  Modelling uncertainty in deep learning for camera relocalization.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 4762–4769, May 2016.  doi: 10.1109/ICRA.2016.7487679.
  * Kendall & Cipolla (2017) Kendall, A. and Cipolla, R.  Geometric loss functions for camera pose regression with deep learning.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * Kendall et al. (2015) Kendall, A., Grimes, M., and Cipolla, R.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In _The IEEE International Conference on Computer Vision (ICCV)_ , December 2015.
  * Lee et al. (2020a) Lee, J., Balachandran, R., Sarkisov, Y. S., De Stefano, M., Coelho, A., Shinde, K., Kim, M. J., Triebel, R., and Kondak, K.  Visual-inertial telepresence for aerial manipulation.  _arXiv preprint arXiv:2003.11509_ , 2020a.
  * Lee et al. (2020b) Lee, J., Humt, M., Feng, J., and Triebel, R.  Estimating model uncertainty of neural networks in sparse information form.  _arXiv preprint arXiv:2006.11631_ , 2020b.
  * Li et al. (2018) Li, R., Wang, S., Long, Z., and Gu, D.  Undeepvo: Monocular visual odometry through unsupervised deep learning.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 7286–7291, May 2018.  doi: 10.1109/ICRA.2018.8461251.
  * Lutz et al. (2020) Lutz, P., Müller, M. G., Maier, M., Stoneman, S., Tomić, T., von Bargen, I., Schuster, M. J., Steidle, F., Wedler, A., Stürzl, W., et al.  Ardea—an mav with skills for future planetary missions.  _Journal of Field Robotics_ , 37(4):515–551, 2020\.
  * MacKay (1992) MacKay, D. J.  A practical bayesian framework for backpropagation networks.  _Neural computation_ , 4(3):448–472, 1992.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.  Pytorch: An imperative style, high-performance deep learning library.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 32_ , pp. 8024–8035. Curran Associates, Inc., 2019.
  * Ritter et al. (2018) Ritter, H., Botev, A., and Barber, D.  A scalable laplace approximation for neural networks.  In _6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings_ , volume 6. International Conference on Representation Learning, 2018.
  * Shamwell et al. (2018) Shamwell, E. J., Leung, S., and Nothwang, W. D.  Vision-aided absolute trajectory estimation using an unsupervised deep network with online error correction.  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 2524–2531, Oct 2018.
  * Shazeer et al. (2017) Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  In _5th International Conference on Learning Representations, ICLR 2017_. OpenReview.net, 2017.
  * Sun et al. (2018) Sun, D., Yang, X., Liu, M.-Y., and Kautz, J.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8934–8943, 2018.
  * Sun et al. (2019) Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., Vasudevan, V., Han, W., Ngiam, J., Zhao, H., Timofeev, A., Ettinger, S., Krivokon, M., Gao, A., Joshi, A., Zhang, Y., Shlens, J., Chen, Z., and Anguelov, D.  Scalability in perception for autonomous driving: Waymo open dataset, 2019\.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  In _Advances in neural information processing systems_ , pp. 5998–6008, 2017.
  * Walch et al. (2017) Walch, F., Hazirbas, C., Leal-Taixe, L., Sattler, T., Hilsenbeck, S., and Cremers, D.  Image-based localization using lstms for structured feature correlation.  In _The IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * Wang et al. (2017) Wang, S., Clark, R., Wen, H., and Trigoni, N.  Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 2043–2050, 2017.
  * Wang et al. (2018) Wang, S., Clark, R., Wen, H., and Trigoni, N.  End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks.  _The International Journal of Robotics Research_ , 37(4-5):513–542, 2018.
  * Wang et al. (2015) Wang, T.-C., Efros, A. A., and Ramamoorthi, R.  Occlusion-aware depth estimation using light-field cameras.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 3487–3495, 2015.
  * Yang et al. (2020) Yang, N., Stumberg, L. v., Wang, R., and Cremers, D.  D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1281–1292, 2020.
  * Yin & Shi (2018) Yin, Z. and Shi, J.  Geonet: Unsupervised learning of dense depth, optical flow and camera pose.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * Zhan et al. (2018) Zhan, H., Garg, R., Saroj Weerasekera, C., Li, K., Agarwal, H., and Reid, I.  Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * Zhou et al. (2017) Zhou, T., Brown, M., Snavely, N., and Lowe, D. G.  Unsupervised learning of depth and ego-motion from video.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.

##  1 Related Work

We consider Deep learning (DL) based approaches for visual localization.
Amongst many, we review the most related work as follows. Wang et al. (2017)
describes an end-to-end trainable approach without using modules from
conventional VO pipeline. One of the first deep learning approaches that used
CNNs for estimation of 6 DoF poses is PoseNet (Kendall et al., 2015). Some
works have shown improved performance by making certain modifications to
PoseNet, for e.g. in Kendall & Cipolla (2017) by changing the loss function to
incorporate geometric information. Addition of LSTM after CNN similar to Wang
et al. (2017) is made to the network in (Walch et al., 2017; Clark et al.,
2017a) and the network is further improved by making the CNN part Bayesian to
estimate uncertainty of localization in Kendall & Cipolla (2016).
Probabilistic approach has also been explored in Wang et al. (2018). Several
approaches(Li et al., 2018; Zhan et al., 2018; Zhou et al., 2017; Yin & Shi,
2018) have used unsupervised learning approach for depth estimation and VO.

In learning based methods for VIO like VINet (Clark et al., 2017b), wherein
the two features streams are simply concatenated and fed into another LSTM for
pose regression. It is one of the early end-to-end frameworks for VIO trained
in a supervised manner. The work in Chen et al. (2019), replaces the simple
fusion strategy used in Clark et al. (2017b) with a more sophisticated one.
Two approaches are suggested - soft and hard fusion which do the fusion either
deterministic-ally or stocastically. VIOLearner (Shamwell et al., 2018) is an
unsupervised scaled trajectory estimation and online error correction work.
However, the necessity of depth information to recover absolute scale may make
it difficult to use as it may not always be available. Self-supervised
approaches like DeepVIO (Han et al., 2019) use optical flow and pre-integrated
IMU network with a status update module to update its status, and SelfVIO
(Almalioglu et al., 2019) does monocular VIO and depth reconstruction using
adversarial training.

##  2 Implementation details

###  2.1 Dataset

The KITTI odometry benchmark (Geiger et al., 2012, 2013) for odometry
evaluations consists of 22 image sequences. Images are acquired at the rate of
10 Hz and saved in png format. OXTS RT 3003 is a GPS/IMU localization system
that captures IMU data at 100 Hz (found in raw KITTI dataset) and also
determines the ground truth data for this dataset. The ground truth is also
acquired at 10 Hz. Sequences 00-10 have corresponding ground truth data
associated with them for training whereas sequences 11-22 are without
corresponding ground truth for evaluation purposes. We use the sequences 00,
01, 02, 05, 08, 09 for training because they are comparatively longer and
sequences 04, 06, 07, 10 for testing. Sequence 03 is omitted due to missing
raw data file. Images and raw IMU data are manually synchronized. In order to
generate more training data, the sequences are segmented into trajectories of
varying lengths which in turn help in avoiding overfitting by core LSTM. Here
the varying sequence length is treated as a hyperparameter.

###  2.2 Architecture

The network is implemented using PyTorch framework (Paszke et al., 2019) and
trained on an NVIDIA GTX 1080 GPU. The Adam optimizer with learning rate of
5$\mathrm{e}^{-5}$ is used to train the network for up to 80 epochs with a
batch size of 8. The scale factor $\beta$ for MSE is chosen to be 1000 as
improved orientation estimates are obtained at this value. Pre-trained
FlowNetS model (Dosovitskiy et al., 2015) is used for the CNN and transfer
learning is done from our implementation of the VINet model for both feature
extractors to reduce time required for training and convergence.
Regularization techniques like batch-normalization and dropout have been used.
For the FlowNet adaptation, the CNN has total 17 layers, each convolutional
layer followed by ReLU activation except for last convolutional layer. Images
have been resized to $184\times 608$. The IMU-LSTM and the core LSTM both are
chosen to be bidirectional with two layers with 15 and 1000 hidden states
respectively.

###  2.3 Baselines

For all the baselines, as open source code was not available, and thus, the
implementations might differ. The implementation details are as follows. The
same configuration for FlowNet and LSTM has been followed for the baselines
since a performance improvement was found. specific choice stems For DeepVO,
Adagrad optimizer with a learning rate of 5$\mathrm{e}^{-4}$ is used, the
network is trained for 250 epochs. VINet is trained using Adam optimizer with
learning rate of 1$\mathrm{e}^{-4}$. We have removed the SE(3) composition
layer used in original work and replaced it with a fully-connected layer. VIO
Soft is trained using Adam optimizer with a learning rate of
1$\mathrm{e}^{-5}$.

##  3 Degradation datasets

The sensor corruptions tell a lot about network robustness. Robustness of the
network to such degradations provides an insight into the underlying sensor
fusion mechanism. It points towards the resilience of the network to sensor
corruption and consequent failure, highlighting the complementary nature of
the sensor modalities in face of adversity. It also gives an insight into how
different fusion strategies give relative importance to visual and inertial
features.

For this reason, we prepare two groups of datasets by introducing various
types of sensor degradation as done in (Chen et al., 2019). They are as under:

###  3.1 Visual-Inertial degradation

  * •

Part occlusion: We cutout a part of dimension $200\times 200$ pixels from an
image at a random location by overlaying a mask of the same size on it. Such
situation can occur when camera view is obstructed by objects very close to it
or due to dust (Wang et al., 2015).

  * •

Noise: We add salt and pepper noise along with Gaussian blur to the images.
This can occur due to substantial horizontal motion of the camera, changing
light conditions or due to defocusing (Couzinié-Devy et al., 2013).

  * •

Missing frames: Some images are removed at random. This happens when the
sensor temporarily gets disconnected or while passing through low-lit area
like a tunnel.

  * •

Noise: IMU has inherent errors due to biases in accelerometers and gyroscopes
and gyro drift. In addition to the already existing noise we add white noise
to accelerometer and bias to gyroscope. This can happen due to temperature
variations giving rise to white noise and random walk (El-Sheimy et al.,
2007).

  * •

Missing frames: Random removal of IMU frames between two image frames is done.
This is plausible due to packet loss from bus or sensor instability.

##  4 Additional results for sensor degradation

Tables LABEL:table:normal, LABEL:table:imu, LABEL:table:allv and
LABEL:table:allvi are the detailed version of Table LABEL:table:nom comparing
the metric for sequences 04, 06, 09 and 10. Figure 5 and Figure 6 substantiate
the analysis presented in Section 3.1.

Table 2: Comparison metric for nominal case Seq | DeepVO | VINet | VIO Soft | MHA
---|---|---|---|---
$t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$
04 |  6.87 |  2.48 |  11.30 |  2.63 |  10.84 |  2.42 |  8.90 |  1.58
06 |  26.52 |  8.90 |  15.33 |  5.25 |  17.75 |  6.91 |  15.59 |  5.56
09 |  18.63 |  4.68 |  6.95 |  1.79 |  6.98 |  2.26 |  8.38 |  2.80
10 |  23.1 |  4.33 |  22.34 |  8.98 |  19.03 |  7.74 |  12.82 |  5.80
Mean |  18.07 |  5.88 |  13.98 |  4.66 |  13.65 |  4.83 |  11.42 |  3.94
Table 3: Comparison metric for IMU degradation case Seq | DeepVO | VINet | VIO Soft | MHA
---|---|---|---|---
$t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$
04 |  N/A |  N/A |  12.56 |  3.02 |  10.563 |  2.95 |  9.30 |  1.60
06 |  N/A |  N/A |  16.71 |  5.31 |  18.87 |  7.16 |  17.04 |  5.82
09 |  N/A |  N/A |  10.47 |  3.50 |  7.71 |  2.33 |  7.40 |  2.13
10 |  N/A |  N/A |  25.48 |  9.28 |  19.07 |  7.33 |  16.18 |  6.15
Mean |  - |  - |  16.31 |  5.28 |  14.05 |  4.94 |  12.48 |  3.93
Table 4: Comparison metric for all vision degradation case Seq | DeepVO | VINet | VIO Soft | MHA
---|---|---|---|---
$t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$
04 |  F |  F |  12.29 |  2.83 |  11.82 |  2.43 |  9.61 |  1.90
06 |  A |  A |  14.80 |  5.08 |  19.01 |  6.78 |  16.75 |  5.67
09 |  I |  I |  10.11 |  3.39 |  7.44 |  2.20 |  10.22 |  2.84
10 |  L |  L |  24.67 |  8.55 |  20.06 |  6.92 |  13.86 |  5.53
Mean |  - |  - |  15.46 |  4.97 |  14.59 |  4.58 |  12.61 |  3.98
Table 5: Comparison metric for all sensor degradation case Seq | DeepVO | VINet | VIO Soft | MHA
---|---|---|---|---
$t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$ |  $t_{rel}$ |  $r_{rel}$
04 |  N/A |  N/A |  13.59 |  3.09 |  12.98 |  2.61 |  10.67 |  2.03
06 |  N/A |  N/A |  15.00 |  4.98 |  16.94 |  5.91 |  15.38 |  5.17
09 |  N/A |  N/A |  10.39 |  3.50 |  9.34 |  2.71 |  9.93 |  2.66
10 |  N/A |  N/A |  24.49 |  9.00 |  22.65 |  7.28 |  13.88 |  5.30
Mean |  - |  - |  15.87 |  5.14 |  15.48 |  4.63 |  12.47 |  3.79

##  5 Laplace Approximation

Given model parameters $\theta$ and data $\mathcal{D}$, the log posterior can
be captured approximately by a second order Taylor expansion around the
maximum a posteriori (MAP) estimate,

$\displaystyle\log p(\theta|\mathcal{D})$ $\displaystyle\approx\log
p(\theta_{MAP}|\mathcal{D})$
$\displaystyle-\frac{1}{2}(\theta-\theta_{MAP})^{T}H(\theta-\theta_{MAP}),$

where $H$ is the matrix of second order derivatives (Hessian) w.r.t. the
negative log posterior. Notably, the right hand side is of Gaussian functional
form when applying the exponential, thus it can be used to sample parameter
configurations:

$\theta\sim\mathcal{N}(\theta_{MAP},H^{-1}).$

Because the Hessian of a deep neural network is infeasible to compute and
invert, approximations to the Fisher information matrix (Fisher), like
Kronecker factored or diagonal approximations, are used instead.

Approximate Bayesian inference can be performed using Monte Carlo integration,
by sampling multiple parameter configurations from the approximate posterior
and collecting the model output for each of them given an unseen datum
$\mathbf{\tilde{x}}$:

$\displaystyle p(\mathbf{\tilde{y}}|\mathbf{\tilde{x}},\mathcal{D})$
$\displaystyle=\int
p(\mathbf{\tilde{y}}|\mathbf{\tilde{x}},\theta)p(\theta|\mathcal{D})\mathrm{d}\theta$
$\displaystyle\approx\frac{1}{T}\sum_{t=1}^{T}p(\mathbf{\tilde{y}}|\mathbf{\tilde{x}},\theta_{t}).$

The mean is used to make predictions while for regression, the variance is a
measure of the predictive uncertainty. We make use of $T=30$ posterior samples
in all our experiments. Because of numerical instability, overestimation of
variance and wrong attribution of probability mass, the posterior
approximation needs to be regularized in practice. We make use of the
following simple regularization scheme:

$NF+\tau I,$

where $F$ is the approximation to the Fisher and $N$ and $\tau$ are treated as
hyperparameters to be optimized on a validation set. We made use of our own
implementation of LA for Bayesian inference in PyTorch which we will make
publicly available in https://github.com/DLR-RM.

(a) IMU deg. (b) All sensor deg.

Figure 5: Predicted trajectories for (a) IMU degradation (b) all sensor
degradation for Sequence 10.

(a) Nominal (b) IMU deg. (c) Vision deg. (d) All sensor deg.

Figure 6: Predicted trajectories with corresponding datasets for Sequence 06.


# Multi-scale Speaker Diarization with Dynamic Scale Weighting

###### Abstract

Speaker diarization systems are challenged by a trade-off between the temporal
resolution and the fidelity of the speaker representation. By obtaining a
superior temporal resolution with an enhanced accuracy, a multi-scale approach
is a way to cope with such a trade-off. In this paper, we propose a more
advanced multi-scale diarization system based on a multi-scale diarization
decoder. There are two main contributions in this study that significantly
improve the diarization performance. First, we use multi-scale clustering as
an initialization to estimate the number of speakers and obtain the average
speaker representation vector for each speaker and each scale. Next, we
propose the use of 1-D convolutional neural networks that dynamically
determine the importance of each scale at each time step. To handle a variable
number of speakers and overlapping speech, the proposed system can estimate
the number of existing speakers. Our proposed system achieves a state-of-art
performance on the CALLHOME and AMI MixHeadset datasets, with 3.92% and 1.05%
diarization error rates, respectively.

Index Terms: speaker diarization, multi-scale

##  1 Introduction

Speaker diarization is a task of partitioning an input audio stream into
speaker-homogeneous segments allowing audio segments to be associated with
speaker labels. Speaker diarization requires decisions on relatively short
segments ranging from a few tenths of a second to several seconds. In speaker
embedding extraction, to obtain high-quality speaker representation vectors,
the temporal resolution should be sacrificed by taking a long speech segment.
Thus, speaker diarization systems always face a trade-off between two
quantities, i.e., the temporal resolution and quality of the representation.
In the early versions of speaker diarization, the speaker homogeneous
variable-length Mel-frequency cepstral coefficients (MFCCs) segments were
generated through the detection of speaker change points [1].

Owing to the increased popularity of i-vectors [2] and x-vectors [3] in the
field of speaker diarization, a uniform segmentation approach has been widely
used, in which a fixed speaker segment length is applied to extract the
speaker representation vector per segment. Because fixed-length segments lead
to stable speaker representations by controlling the length factor in speaker
embedding extraction, speaker diarization systems based on a uniform
segmentation method [2, 4] have shown a competitive performance. However, the
uniform segmentation approach has innate limitations in terms of the temporal
resolution because reducing the segment length leads to a decrease in accuracy
[5]. In a uniform segmentation setting, the shortest temporal resolution is
limited to the hop-length during the segmentation process. Moreover, without a
post processing approach, because each segment is assigned to only one
speaker, the clustering approaches lack an overlap-aware diarization
capability.

Recent speaker diarization systems such as an end-to-end approach [6] or
target-speaker voice activity detection (TS-VAD) [7] employ feature frame-
level speaker labels. These systems are based on sequence models such as long
short-term memory (LSTM) [7, 8] or a Transformer-style encoder-decoder [6].
Frame-level sequence model based systems enable a far superior temporal
resolution of approximately 0.01 s. Another significant benefit of these
methods is overlap-aware diarization where the neural network model outputs
more than one speaker label output. The multi-dimensional sigmoid output
enables an overlap detection without employing an additional resegmentation
module on top of the diarization system.

However, sequence model based methods also have a few drawbacks. First,
because these sequence models are trained with a fixed number of speakers,
sequence model based diarization systems often lack the capability to handle a
flexible number of speakers [7] or estimate their numbers [6]. Second,
according to the recent results of a diarization challenge [9, 10], the
performances of sequence model based end-to-end approaches [6, 8, 11] still
lag behind that of a state-of-art modular diarization [7, 12]. The hidden
Markov model (HMM) based clustering approach proposed in [12] does not have
the aforementioned downsides, having the capabilities of speaker counting and
overlap detection and achieving a state-of-art performance. Thus, we compared
our proposed system with the system proposed in [12] on the same datasets.

Figure 1: Data-flow of the proposed multi-scale speaker diarization system.

Our proposed multi-scale diarization decoder (MSDD) tackles the problems
inherent to previous studies. The proposed MSDD was designed to support the
following features: overlap-aware diarization, an improved temporal
resolution, and a flexible number of speakers. As shown in Fig.1, the proposed
system comprises three components overall, i.e., a pretrained speaker
embedding extractor (TitaNet, [13]), a multi-scale speaker clustering module,
and the MSDD model.

Although we apply a multi-scale clustering idea from a previous study [5], we
do not estimate the static session level multi-scale weights. Instead, we
propose the MSDD approach, which takes advantage of the initialized clusters
by comparing the extracted speaker embeddings with the cluster-average speaker
representation vectors. The weight of each scale at each time step is
determined through a scale weighting mechanism where the scale weights are
calculated from a 1-D convolutional neural network (CNN) applied to the input
speaker embeddings and the cluster-average embedding based on the clustering
results. Finally, based on the weighted cosine similarity vectors, an LSTM-
based sequence model estimates thelabels of each speaker. As described in [7],
using the clustering result as an initialization allows the proposed system to
be free from permutations issue [6] and turns the diarization problem into
binary classification problem at each time step. Unlike the systems described
in [6, 7] we only train and test two-speaker models to handle a flexible
number of speakers. Therefore, the speaker labels are predicted by taking the
average of the sigmoid outputs from the multiple pairs. The 2-D dimensional
output enables an overlap-aware diarization for the given input stream.

In the experimental result section, we demonstrate that the proposed multi-
scale approach is more accurate than our previous single scale (SS),
clustering based diarization system. Furthermore, we show that our proposed
system achieves new state-of-art results on the CALLHOME and AMI MixHeadset
datasets.

Figure 2: Cosine similarity values from each speaker and each scale are
weighted by the scale weights to form a weighted cosine similarity vector.

##  2 Multi-scale Clustering

###  2.1 Multi-scale segmentation

The largest difference between a conventional single scale clustering method
and multi-scale method is in the way in which the speech stream is segmented.
The proposed system uses the same multi-scale embedding extraction and
clustering mechanism as in [5], except that the scale weights are computed
differently in this work. We employ a uniform segmentation scheme that
originally appeared in [2, 4], and such segmentation is applied for multiple
scales. Fig. 2 shows an example of a multi-scale segmentation. As shown in
Fig. 2, we refer to the finest scale, i.e., 0.5 s, as the base scale because
the speaker label estimation is applied at the finest scale. We denote the
number of scales as $K$.

After segmentation is applied for all scales, grouping among the segments at
each scale is then applied. The grouping process is conducted by assigning the
segments from each scale for the corresponding base scale segment. The
segments from the lower temporal resolution scales (in this example, 2.0, 1.5,
and 1.0 s in length) are selected and grouped by measuring the distance
between the centers of the segments and choosing the closest ones. By grouping
the segments as in Fig. 2, each group will generate one weighted cosine
similarity value when a weighted affinity matrix is calculated. A detailed
description of multi-scale clustering can be found in [5]. We use the speaker
embedding extractor model proposed in [13], which generates 192-dimensional
embedding.

###  2.2 Clustering for initialization

Because we previously proposed a multi-scale system in [5], we employ the
auto-tuning spectral clustering approach proposed in [14], which is referred
to as normalized maximum eigengap spectral clustering (NME-SC). From the
clustering result, we obtain the estimated cluster label per each base scale
segment and the estimated number of speakers $S$.

Figure 3: Scale weights calculated from a 1-D CNN. The 1-D filter captures the
context from the input embeddings and cluster-average embeddings.

Figure 4: Example plot of target labels, prediction, and scale weights (K=5).
Note that scale weights vary at each time step.

Based on the clustering results, we take the average of all the speaker
embeddings obtained from the initial clustering result as in Eq. (1). We
obtain cluster-average embedding vectors $\textbf{v}_{k}^{s}$ from the initial
clustering result as follows:

$\textbf{v}_{k}^{s}=\frac{1}{N_{k}}\sum_{i=1}^{N_{k}}\textbf{v}_{i,k}^{s},$
(1)

where $k$ is the scale index, $N_{k}$ is the number of $k$-th scale embeddings
during the session and $s$ is the cluster index (speaker index) from the
clustering result.

##  3 Multi-scale Diarization Decoder

###  3.1 1-D CNN for Dynamic Scale Weights

To dynamically adjust the scale weights during the inference phase, the
proposed diarization system should look into the input speaker and reference
embeddings simultaneously. Fig. 3 shows how scale weights are calculated in
the case of $K$$=$$4$ scales for $S$$=$$2$ speakers. We stack the embeddings
from the input signal and the average embeddings from the clustering results
of two speakers ($s$$=$$1$,$2$) in the following manner:

$\vspace{-1.0ex}\textbf{D}_{i}=[\textbf{u}_{i,0};\text{\ldots};\textbf{u}_{i,K-1};\textbf{v}_{0}^{1};\text{\ldots}\textbf{v}_{\text{$K$-$1$}}^{1};\textbf{v}_{0}^{2};\text{\ldots};\textbf{v}_{K-1}^{2}]^{\intercal},$
(2)

where $\textbf{u}_{i,k}$ are column-wise input speaker embeddings, and
$\textbf{v}_{k}^{s}$ are cluster-average speaker embeddings, as described in
Section 3.2. Thus, the CNN input becomes a $3K\times N_{e}$ matrix
$\textbf{D}_{i}$, where $N_{e}$ is the embedding dimension, and $K$ is the
number of scales.

To estimate the scale weight, we propose 1-D CNN with 1-D filters. The
motivation behind using a 1-D filter is to compare the embedding vectors bin-
by-bin such that the learned weights in the filters focus only on the
difference between other embeddings for each bin. Two CNN layers are followed
by two linear layers and the softmax layer, i.e.,

$w_{k}=\frac{e^{f(z_{k})}}{\sum_{k=0}^{K-1}e^{f(z_{k})}},$ (3)

where $k$ is the scale index, and $f(z_{k})$ is the output of the linear layer
(see Fig. 3). The scale weight values $w_{k}$ in Eq. (3) are multiplied with
the cosine similarity in an element-wise manner and create the context vectors
for each speaker $\textbf{c}_{i}^{s}$. We concatenate these two speaker-
specific context vectors for each $i$-th time step into the global context
vector $\mathbf{C}_{i}=[\textbf{c}_{i}^{1};\textbf{c}_{i}^{2}]$ of length $2K$
(see Fig. 2).

###  3.2 Scale Weighting Mechanism

Although clustering based diarization shows a competitive performance, it
lacks the ability to assign multiple speakers at the same time step, which is
needed for overlap-aware diarization. In addition, the clustering-based multi-
scale diarization requires a separate training step for multi-scale weights or
a high-dimensional grid search. This negatively affects the accuracy of the
model for unseen domains during training. Moreover, the performance of a
clustering-based multi-scale diarization system is heavily dependent on the
scale weights, as shown in [5].

Hence, we derived a scale weighting mechanism that dynamically calculates the
scale weights at every time step when a trained sequence model estimates the
speaker labels. In the scale weighting system, during the training process,
the scale weights are multiplied to the cosine similarity values to create a
context vector that is fed to the sequence model (LSTM). The scale weighting
mechanism is described in Fig. 2. Let $\textbf{u}_{i,k}$ be the $k$-th scale
embedding of the input speech signal and $i$ be the time-step index. In
addition, $\textbf{v}_{k}^{s}$ is the $k$-th scale average embedding vector of
cluster $s$ from the initial clustering result in Eq. (1), and let ${w}_{k,i}$
be the $k$-th scale weight at the $i$-th time-step index. The scale weight
$w_{k,i}$ is applied to the cosine distance between the inputs
$\textbf{v}_{k}^{s}$ and $\textbf{u}_{i,k}$, i.e.,

$\textbf{c}_{i}^{s}[k]=w_{k}\cdot\frac{\textbf{v}_{k}^{s}\cdot\textbf{u}_{i,k}}{\|\textbf{v}_{k}^{s}\|\|\textbf{u}_{i,k}\|},$
(4)

where $\textbf{c}_{i}^{s}$ is the context vector for the $s$-th speaker (see
Fig.5).

###  3.3 Sequence model for speaker label estimation

The weighted cosine similarity vectors are fed to the sequence model. In our
proposed system, we employ a two-layer Bi-LSTM. Fig. 5 shows how the context
vector is fed to the LSTM for estimating the speaker-wise label. Note that the
output layer goes through the sigmoid layer such that the output value ranges
from zero to 1 independently from the neighboring values. We use the binary
cross-entropy loss to train the model. The binary ground truth label is
generated by calculating the overlap between the ground truth speaker
timestamps and the base scale segment: If the overlap is greater than 50%, the
segment is assigned a value of 1 (see Fig.4).

The speaker label inference for the sessions with more than two speakers is
obtained by selecting pairs from the estimated number of speakers and
averaging the prediction for the corresponding speaker from the two-speaker
models. For example, in the three speaker cases in which speakers A, B, and C
exist, we average the two results for A from (A, B) and (A, C). Letting S be
the total number of speakers in a session, then the proposed model infers
$\binom{S}{2}$ number of speakers and then take average of all $S$$-$$1$
output pairs from a certain speaker through the following equation:

$p(s,i)=\frac{1}{\text{$S$ $-$ $1$}}\sum_{q\in
U\text{$-$}\\{s\\}}\sigma(\mathbf{z})^{s,(s,q)}_{i}$ (5)

where $U$ is a complete set of the speakers, and
$\sigma(\mathbf{z})^{s,(s,q)}_{i}$ is the sigmoid output of the $i$-th base
scale segment of the $s$-th speaker, which is extracted from the pairing of
the $s$-th and $q$-th speakers. We use a threshold value $T$ on the average
sigmoid value $p(s,i)$ to obtain the final speaker label output for each base
scale segment. If both sigmoid values are below the threshold $T$ for the
given time step $i$, we then choose the label from the initial clustering
result.

Figure 5: Sequence modeling using LSTM for generating speaker labels.

##  4 Experimental Results

###  4.1 Models and training

We train the meeting and telephonic models (3.4M parameters) separately using
the same dataset but different parameters. Table 1 shows the length of sclae
and the speaker embedding extractor models. In this study, we use TitaNet-M
(13.4M parameters) and TitaNet-L (25.3M parameters) models [13]. The shift
lengths for each scale are half of each scale window length. The scale
information in Table 1 is applied to both the neural network model and the
clustering as initialization. The initial scale weights are set using the
following equation:

$w_{k}=r-\frac{(r-1)}{K-1}k,$ (6)

where the weights are linearly increased or decreased from the parameter value
$r$ and the base scale (the highest index) is always assigned a value of
$1.0$. In this way, the initial clustering scale weight vector
$\mathbf{w}=[w_{0},w_{1},\dots,w_{K-1}]$ is formed. Note that the overall
scale of the scale weights $w_{k}$ for clustering does not affect the result
because the weighted sum of the cosine similarity values are min-max
normalized during the clustering process [14]. The proposed system requires
parameter tuning on $r$ in Eq. (6) for clustering and threshold $T$ described
in Section 3.3 for the MSDD.

###  4.2 Evaluation setup

An evaluation of a speaker diarization system depends on numerous conditions
regarding the use of the development set, whether information on the oracle
number of speakers is used, and the diarization error rate (DER) evaluation
conditions such as collar and overlap inclusion. We only report the results
based on the estimated number of speakers and oracle voice activity detection
(VAD). We report two different DER evaluation settings following the name of
the setups in [12]: (1) Forgiving, $\text{DER}_{A}$ , 0.25-s collar and
ignored overlap speech; and (2) Full, $\text{DER}_{B}$, 0-s collar and
included overlap speech.

###  4.3 Datasets

We use following datasets:

  * •

CH(NS): NIST-SRE-2000 (LDC2001S97) is the most popular diarization evaluation
dataset and referred to as CALLHOME in the speaker diarization papers. To make
a fair comparison with other studies and our previous research, a two-fold
cross validation is used for tuning the parameters using the split appeared in
[15, 16].

  * •

CH109: Call Home American English Speech (CHAES, LDC97S42) is a corpus that
contains only English telephonic speech data. We evaluate 109 sessions (CH109)
which have two speakers per session. The parameters are optimized on the
remaining 11 sessions in the CHAES dataset.

  * •

AMI-MH: AMI meeting corpus [17] is a meeting speech dataset containing up to
five speakers. For the evaluation, we use annotation files and the file lists
of splits provided by [18] from the MixHeadset part. All parameters are
optimized on the dev set.

  * •

The Fisher corpus: The Fisher corpus contains 10 minute English conversations.
We use Fisher corpus [19] to train the proposed model using the two-speaker
setting. We use our own random split that has 10,000 sessions for training and
1,500 sessions for validation.

In terms of the hyperparameters, we use 256 nodes of hidden layer units for
both hidden layers in the scale weight generation and the LSTM. We use 16
filters for the 1-D CNN described in 3.1. F1 score is used for stopping the
training or select the model. The results reported herein can be reproduced
using the NeMo111https://github.com/NVIDIA/NeMo open-source toolkit [20].

Table 1: Scale length and quantity for each model. Model | $K$ | Scale (Segment) Length | Emb.
---|---|---|---
Domain | Model
Telephonic | 5 | [1.5, 1.25, 1.0, 0.75, 0.5] | TitaNet-M
Meeting | 6 | [3.0, 2.5, 2.0, 1.5, 1.0, 0.5] | TitaNet-L
|  |  |

###  4.4 Diarization performance

Table 2 shows the performance of the previously published speaker diarization
performance on the same dataset. In Table 2, note that $\text{DER}_{A}$ does
not reflect the overlap detection capability. SS-Clus is our previous approach
[13], where SS clustering-based diarization systems with TitaNet models [13]
were used. The Equal-$\mathbf{w}$-Clus system uses equal weights, which means
$w_{k}$$=$$1$ in Eq. (6) for all elements in the scale weight vector
$\mathbf{w}$. Accordingly, Equal-$\mathbf{w}$-MSDD is the result obtained from
the MSDD based on the initializing clustering with an equal scale weight. By
contrast, Opt-$\mathbf{w}$-Clus indicates a clustering-based diarization
result with the optimized scale weight vector $\mathbf{w}$ on each development
set, and Opt-$\mathbf{w}$-MSDD refers to the system based on the clustering
result Opt-$\mathbf{w}$-Clus. Whereas Opt-$\mathbf{w}$-MSDD shows an overall
better result, we want to emphasize the importance of the
Equal-$\mathbf{w}$-MSDD result because Equal-$\mathbf{w}$-MSDD does not
require parameter tuning for the initializing clustering.

Table 2: DER results from previous studies and the proposed methods. The
number of speakers $N_{s}$ is estimated within the range
$1$$\leq$$N_{s}$$\leq$$8$.

|  | Telephonic | Meeting
---|---|---|---
Systems | Eval. | CH | CH | AMI-MH
Setup | (NS) | 109 | dev | test
Park et al. [5] | $\text{DER}_{A}$ | 6.46 | 2.04* | - | 3.32
Aronow- et al. [21] | $\text{DER}_{A}$ | 5.1 | - | - | -
Dawala- et al. [22] | $\text{DER}_{A}$ | - | - | 2.43 | 4.03
Landini et al. [12] | $\text{DER}_{A}$ | 4.42 | - | 2.14 | 2.17
$\text{DER}_{B}$ | 21.77 | - | 22.98 | 22.85
SS-Clus [13] | $\text{DER}_{A}$ | 5.38 | 1.42 | - | 1.89
Equal-$\mathbf{w}$-MS-Clus | $\text{DER}_{A}$ | 4.57 | 1.19 | 1.34 | 1.06
Equal-$\mathbf{w}$-MSDD | $\text{DER}_{A}$ | 4.25 | 0.69 | 1.34 | 1.05
$\text{DER}_{B}$ | 20.39 | 10.94 | 22.21 | 21.18
Opt-$\mathbf{w}$-MS-Clus | $\text{DER}_{A}$ | 4.18 | 0.70 | 1.30 | 1.06
Opt-$\mathbf{w}$-MSDD | $\text{DER}_{A}$ | 3.92 | 0.73 | 1.30 | 1.06
$\text{DER}_{B}$ | 20.14 | 10.82 | 22.20 | 21.19
|  |  |  |  |

* This is eval set in the CHAES dataset.

##  5 Discussions

Although our proposed system shows a competitive performance, there is still
room for improvement, which requires further investigation. First, although
the trained model can adjust the scale weight on the fly, to obtain improved
results, we need to have separate models for both the meeting and telephonic
data. Nevertheless, the proposed MSDD model does not show a significantly
improved performance on the AMI datasets. We believe this can be overcome by
including datasets from many different domains. Second, the temporal
resolution is still limited to the shift length (0.25 s) of the finest scale.
We observed that reducing the base scale to below 0.5 s only results in a
larger error, which can also be an area of improvement.

Despite the aforementioned downsides, we found the following benefit of the
MSDD. First, the proposed MSDD approach achieves similar or better results
compared to clustering-based diarization results. Second, compared to the
clustering based multi-scale diarization, the proposed system can achieve a
more stable performance for unknown domains by using an equal scale weight. As
investigated in previous multi-scale studies [5], the performance of
clustering-based diarization results is heavily dependent on multi-scale
weights. However, in terms of the proposed diarization decoder, the averaged
speaker embeddings achieve a relatively minor effect from the scale weights
used for the clustering. In particular, the smaller dependency on the tuned
parameters is a benefit when the diarization system is deployed under real-
life scenarios. Third, the proposed system does not rely on an iterative
procedure, unlike the system in [7] where the speaker representation is
extracted again after the first prediction. Thus, the proposed system can be
applied to streaming diarization systems where we first apply clustering for a
relatively short amount of time and then predict the diarization result on the
incoming buffer in an incremental manner.

##  6 Conclusions

In this paper, we proposed a multi-scale diarization decoder with a scale
weighting mechanism. The proposed system has the following benefits: First,
this is the first study applying a multi-scale weighting concept with sequence
model (LSTM) based speaker label estimation. Thus, the multi-scale diarization
system enables overlap-aware diarization, which cannot be achieved with
traditional clustering-based diarization systems. Moreover, because the
decoder is based on a clustering-based initialization, the diarization system
can deal with a flexible number of speakers. Finally, we showed a superior
diarization performance compared to the previous published results. There are
two future areas of research regarding the proposed system. First, we plan to
implement a streaming version of the proposed system by implementing
diarization decoder based on window-wise clustering. Second, the end-to-end
optimization from speaker embedding extractor to diarization decoder can be
investigated.

## References

  * [1] S. Chen, P. Gopalakrishnan _et al._ , “Speaker, environment and channel change detection and clustering via the bayesian information criterion,” in _Proc. DARPA broadcast news transcription and understanding workshop_ , vol. 8. Virginia, USA, 1998, pp. 127–132.
  * [2] M. Senoussaoui, P. Kenny, T. Stafylakis, and P. Dumouchel, “A study of the cosine distance-based mean shift for telephone speech diarization,” _IEEE/ACM TASLP_ , vol. 22, no. 1, pp. 217–227, 2013.
  * [3] G. Sell, D. Snyder, A. McCree, D. Garcia-Romero, J. Villalba, M. Maciejewski, V. Manohar, N. Dehak, D. Povey, S. Watanabe _et al._ , “Diarization is hard: Some experiences and lessons learned for the JHU team in the inaugural DIHARD challenge.” in _Proc. INTERSPEECH_ , Sep. 2018, pp. 2808–2812.
  * [4] P. Kenny, D. Reynolds, and F. Castaldo, “Diarization of telephone conversations using factor analysis,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 4, no. 6, pp. 1059–1070, 2010.
  * [5] T. J. Park, M. Kumar, and S. Narayanan, “Multi-scale speaker diarization with neural affinity score fusion,” in _Proc. ICASSP_. IEEE, 2021, pp. 7173–7177.
  * [6] Y. Fujita, N. Kanda, S. Horiguchi, Y. Xue, K. Nagamatsu, and S. Watanabe, “End-to-end neural speaker diarization with self-attention,” in _2019 ASRU_. IEEE, 2019, pp. 296–303.
  * [7] I. Medennikov, M. Korenevsky, T. Prisyach, Y. Khokhlov, M. Korenevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny _et al._ , “Target-speaker voice activity detection: A novel approach for multi-speaker diarization in a dinner party scenario,” _Proc. INTERSPEECH_ , pp. 274–278, 2020.
  * [8] S. Horiguchi, Y. Fujita, S. Watanabe, Y. Xue, and K. Nagamatsu, “End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,” _Proc. INTERSPEECH_ , pp. 269–273, 2020.
  * [9] S. Watanabe, M. Mandel, J. Barker, E. Vincent, A. Arora, X. Chang, S. Khudanpur, V. Manohar, D. Povey, D. Raj _et al._ , “Chime-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings,” in _Proc. CHiME_ , 2020, pp. 1–7.
  * [10] N. Ryant, K. Church, C. Cieri, J. Du, S. Ganapathy, and M. Liberman, “Third dihard challenge evaluation plan,” _arXiv preprint arXiv:2006.05815_ , 2020\.
  * [11] K. Kinoshita, M. Delcroix, and N. Tawara, “Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds,” in _Proc. ICASSP_. IEEE, 2021, pp. 7198–7202.
  * [12] F. Landini, J. Profant, M. Diez, and L. Burget, “Bayesian hmm clustering of x-vector sequences (vbx) in speaker diarization: theory, implementation and analysis on standard tasks,” _Computer Speech & Language_, vol. 71, p. 101254, 2022.
  * [13] N. R. Koluguri, T. Park, and B. Ginsburg, “Titanet: Neural model for speaker representation with 1d depth-wise separable convolutions and global context,” _arXiv preprint arXiv:2110.04410_ , 2021.
  * [14] T. J. Park, K. J. Han, M. Kumar, and S. Narayanan, “Auto-tuning spectral clustering for speaker diarization using normalized maximum eigengap,” _IEEE Signal Processing Letters_ , vol. 27, pp. 381–385, 2019.
  * [15] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in _Proc. ICASSP_ , Apr. 2018, pp. 5329–5333.
  * [16] D. Snyder, “Callhome diarization recipe using x-vectors,” Github, May 4, 2018\. [Online]. Available: https://david-ryan-snyder.github.io/2018/05/04/model_callhome_diarization_v2.html, [Accessed Mar. 21, 2022].
  * [17] I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos _et al._ , “The ami meeting corpus,” in _Proceedings of the 5th international conference on methods and techniques in behavioral research_ , vol. 88. Citeseer, 2005, p. 100.
  * [18] H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, “Pyannote.audio: neural building blocks for speaker diarization,” in _Proc. ICASSP_. IEEE, 2020, pp. 7124–7128.
  * [19] C. Cieri, D. Miller, and K. Walker, “The fisher corpus: A resource for the next generations of speech-to-text.” in _Proc. LREC_ , vol. 4, 2004, pp. 69–71.
  * [20] O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook _et al._ , “Nvidia nemo,” https://github.com/NVIDIA/NeMo, [Accessed Mar. 21, 2022].
  * [21] H. Aronowitz, W. Zhu, M. Suzuki, G. Kurata, and R. Hoory, “New advances in speaker diarization.” in _Proc. INTERSPEECH_ , 2020, pp. 279–283.
  * [22] N. Dawalatabad, M. Ravanelli, F. Grondin, J. Thienpondt, B. Desplanques, and H. Na, “Ecapa-tdnn embeddings for speaker diarization,” _arXiv preprint arXiv:2104.01466_ , 2021.