* [6] A. Ferrero, F. Gazzola and T. Weth, _On a fourth order Steklov eigenvalue problem_ , Analysis (Munich) 25 (2005), no. 4, 315–332.
  * [7] S. Gallot and D. Meyer, _Opérateur de courbure et laplacien des formes différentielles d’une variété riemannienne_ , J. Math. Pures Appl. 54 (1975), 259–284.
  * [8] F. Gazzola, H.-C. Grunau and G. Sweers, Polyharmonic boundary value problems. Positivity preserving and nonlinear higher order elliptic equations in bounded domains, Lecture notes in Mathematics, Springer, 1991.
  * [9] P. Guérini and A. Savo, Eigenvalue and gap estimates for the Laplacian acting on $p$-forms, Trans. Amer. Math. Soc. 356 (2004), no. 1, 319–344.
  * [10] A. Hassannezhad and A. Siffert, A note on Kuttler–Sigillito’s inequalities, Ann. Math. Québec 44 (2020), 125–147.
  * [11] M.H. Karpukhin, The Steklov Problem on Differential Forms, Canad. J. Math. 71 (2019), no. 2, 417–435.
  * [12] A. Kasue, Ricci curvature, geodesics and some geometric properties of Riemannian manifolds with boundary, J. Math. Soc. Japan 35 (1983), 117–131.
  * [13] S. Kouzayha, Estimations géométriques de fonctionnelles spectrales pour le laplacien avec condition de Robin au bord, PhD thesis, Université de Tours (2019).
  * [14] S. Kumaresan and J. Prajapat, Serrin’s result for hyperbolic space and sphere, Duke Math. J. 91 (1998), no. 1, 17–28.
  * [15] J.R. Kuttler, Remarks on a Stekloff eigenvalue problem, SIAM J. Numer. Anal. 9 (1972), 1–5.
  * [16] J.R. Kuttler and V.G. Sigillito, Inequalities for membrane and Stekloff eigenvalues, J. Math. Anal. Appl. 23 (1968), 148–160.
  * [17] J.-L. Lions, E. Magenes, _Problèmes aux limites non homogènes et applications_ , Vol. 1, Travaux et Recherches Mathématiques, No. 17, Dunod, 1968.
  * [18] R. Molzon, Symmetry and overdetermined boundary value problems, Forum Math. 3 (1991), no. 2, 143–156.
  * [19] L.E. Payne, Some isoperimetric inequalities for harmonic functions, SIAM J. Math. Anal. 1 (1970), 354–359.
  * [20] S. Raulot and A. Savo, _A Reilly formula and eigenvalue estimates for differential forms_ , J. Geom. Anal. 21 (2011), no. 3, 620–640.
  * [21] S. Raulot and A. Savo, On the first eigenvalue of the Dirichlet-to-Neumann operator on forms, J. Funct. Anal. 262 (2012), no. 3, 889–914.
  * [22] S. Raulot and A. Savo, _Sharp bounds for the first eigenvalue of a fourth-order Steklov problem_ , J. Geom. Anal. 25 (2015), no. 3, 1602–1619.
  * [23] A. Savo, On the lowest eigenvalue of the Hodge Laplacian on compact, negatively curved domains, Ann. Global Anal. Geom. 35 (2009), no. 1, 39–62.
  * [24] G. Schwarz, _Hodge decomposition-A method for solving boundary value problems_ , Lecture notes in Mathematics, Springer, 1995.
  * [25] J. Serrin, _A symmetry problem in potential theory_ , Arch. Rational Mech. Anal. 43 (1971), 304–318.
  * [26] Q. Wang and C. Xia, _Sharp bounds for the first non-zero Stekloff eigenvalues_ , J. Funct. Anal. 257 (2009), no. 8, 2635–2644.
  * [27] H.F. Weinberger, _Remark on the preceding paper of Serrin_ , Arch. Rational Mech. Anal. 43 (1971), 319–320.


11institutetext: IMP Lab - D.I.A. - University of Parma - Parma, Italy
11email: {leonardo.rossi, akbar.karimi<EMAIL_ADDRESS>
implab.ce.unipr.it/

# Improving Localization for Semi-Supervised Object Detection

Leonardo Rossi  0000-0002-9316-595X Akbar Karimi  0000-0002-5132-2435 Andrea
Prati  0000-0002-1211-529X

###### Abstract

Nowadays, Semi-Supervised Object Detection (SSOD) is a hot topic, since, while
it is rather easy to collect images for creating a new dataset, labeling them
is still an expensive and time-consuming task. One of the successful methods
to take advantage of raw images on a Semi-Supervised Learning (SSL) setting is
the Mean Teacher technique [17], where the operations of pseudo-labeling by
the Teacher and the Knowledge Transfer from the Student to the Teacher take
place simultaneously. However, the pseudo-labeling by thresholding is not the
best solution since the confidence value is not strictly related to the
prediction uncertainty, not permitting to safely filter predictions. In this
paper, we introduce an additional classification task for bounding box
localization to improve the filtering of the predicted bounding boxes and
obtain higher quality on Student training. Furthermore, we empirically prove
that bounding box regression on the unsupervised part can equally contribute
to the training as much as category classification. Our experiments show that
our IL-net (Improving Localization net) increases SSOD performance by 1.14% AP
on COCO dataset in limited-annotation regime. The code is available at
https://github.com/IMPLabUniPr/unbiased-teacher/tree/ilnet.

###### Keywords:

Object Detection Multi-Task Learning Teacher-Student Technique Semi-Supervised
Learning Semi-Supervised Object Detection.

##  1 Introduction

Supervised learning usually requires a large amount of annotated training data
which can be expensive and time-consuming to produce. Semi-Supervised Learning
(SSL), on the other hand, addresses this issue by taking advantage of large
unlabeled data accompanied by a small labeled dataset. Usually, in an SSL
setting, we have two models that act as Teacher and Student. The latter is
trained in a supervised way, drawing the needed ground truths from the
dataset, if exist, and from the Teacher’s predictions, otherwise.

This approach can be beneficial in many machine learning tasks including
object detection, whose final result is a list of bounding boxes (bboxes) and
the corresponding classes. In this task, the network is devoted to finding the
position of the localized objects in an image, as well as identifying which
class they belong to. Hereinafter, we will refer to this specific application
as Semi-Supervised Object Detection (SSOD). In [10], authors have collected
many interesting ideas applied to SSOD. Among them, the one that seems more
promising is the Mean Teacher [17], which uses the Exponential Moving Average
(EMA) as a knowledge transfer technique from the Student to the Teacher. In
particular, while the Teacher produces pseudo-labels on unlabeled and weakly
augmented images to obtain more reliable labels, the Student is trained using
the pseudo-labels as ground-truth on the same images, but strongly augmented
to differentiate the training. In weak augmentation, only a random horizontal
flip is applied, while in strong augmentation, other transformations such as
randomly adding color jittering, grayscale, Gaussian blur, and cutout patches
are also used.

Although they obtain good performances with this pseudo-labeling technique,
the Student model is still influenced by the Teacher’s erroneous predictions.
The quality of these predictions is difficult to control by applying only a
simple confidence thresholding. For this reason, there is a need for an
additional way to strengthen the region proposals and reduce the number of
erroneous predictions by the Teacher. In the proposed architecture, we
introduce a new task used for the classification of the bboxes, with the aim
of distinguishing good quality ones from the others. This new score exploits
the information complementary to the class score already used in these
networks, allowing a different level of filtering. In addition, we show how to
take advantage of the regression tasks on the unsupervised learning part.
Usually, they are excluded in the unsupervised training phase. The
justification is that the classification score is not able to filter the
potentially incorrect bboxes [5, 10]. A well-known problem during training is
the Objective Imbalance [12], which is characterized by the difficulty in
balancing the different loss contributions. In our hypothesis, this is the
case for the unsupervised training part. In order to obtain a positive effect
from regression losses, an adequate balance of the contribution of these two
losses (regression and category classification) is a possible solution to the
above-mentioned problem. In this way, we prevent the regression losses on
unsupervised dataset from dominating the training, a phenomenon that greatly
amplifies the error introduced by inaccurate Teacher predictions.

The main contributions of this paper are the following:

  * •

a new bounding box IoU (Intersection over Union) classification task to filter
out errors on pseudo-labels produced by the Teacher;

  * •

the introduction of a regression task on the unlabeled dataset which can help
the network to learn better;

  * •

an exhaustive ablation study on all the components of the architecture.

##  2 Related Work

Semi-Supervised Learning for Object Detection. In [14], the authors proposed
the $\Pi$-Model which is an ensemble of the predictions of the model at
different epochs under multiple regularizations and input augmentation
conditions. The predictions for the unlabeled images are merged together to
form a better predictor. In [4], the authors proposed Consistency-based Semi-
supervised Learning for Object Detection (CSD), which uses consistency
constraints as a self-training task to obtain a better training of the
network. In [15], an SSL framework with Self-Training (via pseudo label) and
the Augmentation driven Consistency regularization (STAC) is introduced,
exploiting weak data augmentation for model training and strong data
augmentation for pseudo-labeling. Several works [10, 16, 17] proposed to use
the Mean Teacher model, applying EMA to the weights instead of the
predictions, facilitating knowledge transfer from Student to the Teacher model
at each iteration. For one-stage object detection models, authors of [11] use
an Expectation-Maximization approach, generating pseudo-labels in the
expectation step and training the model on them in the maximization step,
optimizing for classification in each iteration and for localization in each
epoch. In [19], the authors propose a Soft Teacher mechanism where the
classification loss of each unlabeled box is weighted by the Teacher
classification score, also using a box jittering approach to select the most
reliable pseudo-bboxes. In [6], authors utilize SelectiveNet to properly
filter pseudo-bboxes trained after the Teacher. In [20], the authors propose
two models, one of which generates a proposal list of bounding boxes and the
second one refines these proposals, using the average class probability and
the weighted average of the bounding box coordinates.

Bounding Box Intersection over Union (BBox IoU). In [3], the authors added a
new branch on top of the Fast R-CNN model to estimate standard deviation of
each bounding box and use it at Non-maximum Suppression (NMS) level to give
more weight to the less uncertain ones. In [5, 21], the authors added a new
branch on top of the Faster R-CNN to regress bounding box IoU and to multiply
this value with the classification score to compute final score of the
suppression criterion of the NMS. In the same direction, Fitness NMS was
proposed in [18] to correct the detection score for better selecting bounding
boxes which maximize their estimated IoU with the ground-truth.

##  3 Teacher-Student Learning Architecture

In SSOD, we have two datasets. The first set
$\mathbf{D_{s}}=\\{x_{i}^{s},y_{i}^{s}\\}_{i=1}^{N_{s}}$, typically smaller,
contains $N_{s}$ images $\mathbf{x^{s}}$ with the corresponding labels
$\mathbf{y^{s}}$, and the second set
$\mathbf{D_{u}}=\\{x_{i}^{u}\\}_{i=1}^{N_{u}}$ contains $N_{u}$ images
$\mathbf{x^{u}}$ without labels. Similarly to [10], our architecture is
composed of two identical models where one behaves as Teacher and the other as
the Student. At each iteration of the semi-supervised training, the Student is
trained with one batch of images coming from $\mathbf{D_{s}}$ (supervised
training) and one from $\mathbf{D_{u}}$ (unsupervised training), using the
pseudo-labels generated by the Teacher as ground-truth and filtered by a
threshold. Then, the Student transfers the learned knowledge to the Teacher
using the EMA applied to the weights.

In our model, the total loss $L$ for the Student is composed of two terms,
which come from the the supervised $L_{sup}$ and unsupervised $L_{unsup}$
training:

$\begin{split}L&=L_{sup}+L_{unsup}\\\
L_{sup}&=\sum_{i}L_{cls}+L_{reg}+\boldsymbol{L^{IoU}}\\\
L_{unsup}&=\sum_{i}\alpha L_{cls}+\boldsymbol{\beta
L_{reg}}+\boldsymbol{\gamma L^{IoU}}\\\
L_{cls}&=L_{cls}^{rpn}(x_{i}^{s},y_{i}^{s})+L_{cls}^{roi}(x_{i}^{s},y_{i}^{s})\\\
L_{reg}&=L_{reg}^{rpn}(x_{i}^{s},y_{i}^{s})+L_{reg}^{roi}(x_{i}^{s},y_{i}^{s})\\\
\end{split}$ (1)

where $L_{cls}$ contains the sum of Region Proposal Network (RPN) and Region
of Interest (RoI) classification losses, while $L_{reg}$ contains the sum of
RPN and RoI regression losses. The $L^{IoU}$ loss will be defined in
Subsection 3.2. $\alpha$, $\beta$ and $\gamma$ represent how much weight each
component of the unsupervised training has. The new terms introduced in this
paper, w.r.t. the ones defined in [10], are shown in bold in the above
equations and will be described in the following.

###  3.1 Bounding Box Regression on Unsupervised Training

The training consists of two phases, the burn up stage and Teacher-Student
Mutual Learning stage. In the first stage, the Student is trained only on the
labeled dataset, following the standard supervised training procedure. Then,
for the second stage, at each iteration, the Student is trained on two batches
at the same time, one coming from the labeled dataset and the other coming
from the unlabeled dataset. For the latter, the baseline [10] trains only the
RPN and RoI head classifier and disregards regression. The authors justify
this choice noticing that the classification confidence thresholds are not
able to filter the incorrect bounding box regression. Our hypothesis is that
training also on pseudo bounding box regression could help the Student as long
as the task is correctly weighted with respect to the others. Category
classification and regression tasks are learned in parallel. Given that the
classification performance improves during the training, we can also expect
the bounding box regression to behave the same way. In other words, while the
training proceeds, the average quality of the pseudo-labels increases and so
does the classification confidence value.

(a) count pseudo bboxes

(b) count class errors

(c) count class errors per IoU

(d) Unsupervised $L_{cls}$ and $L_{reg}$

Figure 1: The pseudo bounding boxes generated during training: (1(a)) IoU
distribution of pseudo-bboxes, (1(b)) distribution of pseudo-bboxes when the
predicted class is wrong. (1(c)) number of pseudo-bboxes per IoU collected
every 5000 iterations. (1(d)) Unsupervised classification and regression
losses comparison during the training. Better seen in color.

During the Student training, although pseudo bounding boxes are filtered with
a threshold on classification confidence score, we still have bounding boxes
of any IoU quality. This problem is related to the uncertainty on prediction.
Figure 1(a) visualizes the IoU distribution quality with respect to the
ground-truth of the Teacher filtered predictions. We can notice that the
number of pseudo bounding boxes with IoU less than the threshold (0.6 in our
experiments) remains almost constant during the entire training, unlike the
others which slowly increase. In Figure 1(b), we show only the filtered pseudo
bounding boxes that the Teacher has wrongly classified and split by their IoU.
In Figure 1(c), the same data are shown using a different graph. We can notice
that the number of wrongly classified pseudo-bboxes decreases exponentially
with the increase of the quality, and this trend remains the same during the
training. The bounding boxes with low quality ($IoU<0.6$) represent 45% of the
total pseudo-bboxes and more than 90% of classification errors. This means
that the low-quality IoU bboxes contain almost all the classification errors.

By looking at Figure 1(d), we can see that the unsupervised regression loss
(from RPN and RoI heads) represents between 20% and 30% of the total loss.
Conversely, the unsupervised classification loss (from RPN and RoI heads)
accounts only for 15% (at most) of the total loss. This means that an error in
pseudo-labels has almost three times (see blue line in Figure 1(d)) more
weight on regression branch than on classification branch. To avoid the
amplification of the errors, an appropriate value for $\beta$ is chosen, by
under-weighting the regression contribution and making it comparable with the
classification one.

###  3.2 Bounding Box IoU Classification Task

As noted by [5, 21], adding a new task related to the IoU prediction could
help the network to train better. Our model learns to predict the bounding box
quality, using this value to further filter pseudo-labels in conjunction with
the classification score. Differently from them, we train the model to make a
binary classification instead of a regression since it is sufficient (and
easier to learn) for the purpose of filtering.

Figure 2: Faster R-CNN architecture with our branch in red.

Fig. 2 illustrates the architecture of a two-stage object detection model such
as Faster R-CNN [13] with our additional branch highlighted in red. For each
positive bounding box (i.e., the ones recognized as foreground), it
concatenates the output of the shared layers ($x_{i}^{sh}$, with the size of
1024), all the classification scores ($s_{i}$, with the size of the number of
classes plus the background) and all bounding box regression deltas ($d_{i}$,
with the size of 4). All these features are passed to the IoU cls branch (also
called $f(\cdot)$ in the following), which outputs a vector with the same size
as $s_{i}$ (i.e., one for each class). This will return the $i^{th}$ IoU
classification, called IoU score $q^{IoU}$, corresponding to the predicted
bbox. The $f(\cdot)$ branch consists of two fully-connected layers with an ELU
[1] activation function between them and a sigmoid activation function at the
end.

The loss $L_{i}^{IoU}$ for the new branch of IoU classification, conditioned
on the class, is defined as:

$\begin{split}q_{i}^{IoU}&=f(concat(x_{i}^{sh},s_{i},d_{i}))\\\
L_{i}^{IoU}&=FL(q_{i}^{IoU},\>t_{i})\\\ \end{split}$ (2)

where the FL function is the Focal Loss [8] with its own $\gamma$ value equal
to 1.5 and $t_{i}$ represents the binary target label. For the $i^{th}$
bounding box, the branch output $q_{i}^{IoU}\in[0,1]$ is a single value which
predicts if it is a high- or low-quality bounding box. The target $t_{i}$ and
its IoU are defined as follows:

$\begin{split}IoU_{i}&=\max{\textrm{IoU}\left(b_{i}^{t},g\right)}\>\>\>\>\>\forall
b_{i}\in\mathbf{B}|IoU\left(b_{i},g\right)\geqslant u\\\
t_{i}&=\begin{cases}1&IoU_{i}>\mu\\\ 0&\textrm{otherwise}\\\
\end{cases}\end{split}$ (3)

where $\mathbf{B}$ is the list of proposals coming from the RPN, $b_{i}$ are
the $i^{th}$ predicted bounding boxes, $g$ is the ground-truth bounding box,
$u$ is the minimum IoU threshold to consider the bounding $b_{i}$ as positive
example (typically set to 0.5) and $\mu$ (set to 0.75) is the minimum IoU
threshold to be classified as high-quality.

In the evaluation phase, the Teacher’s filtering of the predicted bounding
boxes with low confidence is preceded by the IoU classification filtering,
which uses a new IoU inference threshold $\theta$ (set to 0.4).

##  4 Experiments

Dataset. We perform our tests on the MS COCO 2017 dataset [9]. The training
dataset consists of more than 117,000 images and 80 different classes of
objects, where 10% of the labeled images are used.

Evaluation Metrics. All the tests are done on COCO minival 2017 validation
dataset, which contains 5000 images. We report mean Average Precision (mAP)
and $AP_{50}$ and $AP_{75}$ with 0.5 and 0.75 minimum IoU thresholds, and
$AP_{s}$, $AP_{m}$ and $AP_{l}$ for small, medium and large objects,
respectively.

Implementation details. All the values are obtained running the training with
the same hardware and hyper-parameters. When available, the original code
released by the authors is used. Our code is developed on top of the Unbiased
Teacher [10] source code. We perform the training on a single machine with 6
Tesla P100 GPUs with 12GB of memory. The train lasts 180,000 iterations with a
batch size of 2 images per GPU for the supervised part and 2 images for the
unsupervised part, with $\alpha$ set to 4. We use the Stochastic Gradient
Descent (SGD) optimization algorithm with a learning rate of 0.0075, a weight
decay of 0.0001, and a momentum of 0.9. The learning rate decays at iteration
179990 and 179995. We use the Faster R-CNN with FPN [7] and the ResNet 50 [2]
backbone for the teacher and student models, initialized by pre-trained on
ImageNet, and the same augmentation of Unbiased Teacher [10].

(a) Unsup regression loss weights.

(b) Classification vs Regression.

(c) Filtering with bbox IoU score.

(d) Count filtered pseudo-bboxes.

Figure 3: Ablation studies: (3(a)) weights for unsupervised regression loss on RPN and RoI. (3(b)) classification vs regression loss on bbox IoU branch. (3(c)) filtering bbox on inference with bbox IoU classification score. (3(d)) count bboxes filtered by inference threshold $\mu$ during training.  # | $\beta$ | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{s}$ | $AP_{m}$ | $AP_{l}$
---|---|---|---|---|---|---|---
0 | 0.5 | 31.775 | 51.450 | 34.190 | 16.952 | 34.384 | 41.549
1 | 1.0 | 31.947 | 51.530 | 34.270 | 16.949 | 34.900 | 41.306
2 | 2.0 | 31.754 | 51.078 | 34.143 | 16.691 | 35.008 | 41.670
3 | 4.0 | 30.445 | 49.387 | 32.727 | 15.044 | 33.200 | 40.209
Table 1: Performance varying the weight loss $\beta$ on unsupervised regression losses. # | Method | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{s}$ | $AP_{m}$ | $AP_{l}$
---|---|---|---|---|---|---|---
1 | UT | 31.027 | 50.757 | 33.056 | 17.014 | 33.684 | 40.322
2 | Ours (with filter) | 31.604 | 51.181 | 33.962 | 16.816 | 34.283 | 40.809
3 | Ours (w/out filter) | 31.509 | 51.118 | 33.564 | 16.848 | 34.684 | 40.582
Table 2: Performance comparison with original Unbiased Teacher (UT) model: (2) Training with BBox IoU branch with and (3) w/out pseudo-labels filtering. # | $\mu$ | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{s}$ | $AP_{m}$ | $AP_{l}$
---|---|---|---|---|---|---|---
1 | 0.5 | 31.199 | 51.009 | 33.047 | 16.187 | 34.000 | 40.180
2 | 0.6 | 31.128 | 50.785 | 33.268 | 17.102 | 33.805 | 39.932
3 | 0.7 | 31.461 | 51.319 | 33.637 | 16.714 | 34.217 | 40.100
4 | 0.75 | 31.604 | 51.181 | 33.962 | 16.816 | 34.283 | 40.809
5 | 0.8 | 31.336 | 50.601 | 33.707 | 16.327 | 34.180 | 40.476
6 | 0.9 | 27.125 | 43.515 | 28.800 | 12.815 | 29.486 | 36.034
Table 3: Performance using BBox IoU classification branch with inference threshold $\theta$ fixed to 0.5 and varying training threshold $\mu$. # | $\theta$ | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{s}$ | $AP_{m}$ | $AP_{l}$
---|---|---|---|---|---|---|---
0 | 0.3 | 31.404 | 51.205 | 33.792 | 16.273 | 34.542 | 40.851
1 | 0.4 | 31.630 | 51.185 | 34.044 | 17.387 | 34.494 | 40.784
2 | 0.5 | 31.604 | 51.181 | 33.962 | 16.816 | 34.283 | 40.809
3 | 0.6 | 31.158 | 50.227 | 33.418 | 16.203 | 33.687 | 40.323
4 | 0.7 | 30.649 | 49.216 | 33.138 | 16.532 | 33.409 | 39.542
Table 4: Performance using BBox IoU classification branch with training threshold $\mu$ fixed to 0.75 and varying inference threshold $\theta$. # | $L_{reg}^{unsup}$ | $x^{sh}$ | scores | deltas | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{s}$ | $AP_{m}$ | $AP_{l}$
---|---|---|---|---|---|---|---|---|---|---
1 |  |  |  |  | 31.027 | 50.757 | 33.056 | 17.014 | 33.684 | 40.322
2 | ✓ |  |  |  | 31.947 | 51.530 | 34.270 | 16.949 | 34.900 | 41.306
3 | ✓ | ✓ |  |  | 31.754 | 51.189 | 34.032 | 16.850 | 34.657 | 41.320
4 | ✓ | ✓ | ✓ |  | 32.166 | 51.772 | 34.765 | 16.647 | 34.999 | 41.870
5 | ✓ | ✓ | ✓ | ✓ | 31.923 | 51.464 | 34.070 | 16.202 | 35.197 | 41.368
6 |  | ✓ | ✓ | ✓ | 31.630 | 51.185 | 34.044 | 17.387 | 34.494 | 40.784
Table 5: Study on unsupervised regression losses and IoU classification loss.

###  4.1 Ablation study

Unsupervised regression loss. In this experiment, we empirically show that we
can also use regression losses on RPN and RoI head for the unsupervised part.
We test different weights for the constant $\beta$ in the loss formula (see
eq. 1). In Figure 3(a) and Table 1, we can see that greatly amplifying the
contribution can be deleterious, becoming counterproductive in the case of
$\beta$ equal to 4.

Bounding box IoU branch loss type. Our proposal involves a new IoU
classification task, trained with a binary cross-entropy function. In this
experiment, we test how performance changes in case our new branch learns a
regression task instead of a classification, using a smooth L1 loss. In this
case, the ground-truth is represented by the real IoU value between the bbox
and the ground-truth. In Figure 3(b), we can see that classification branch is
a bit more stable and reaches a slightly higher performance.

With and without filtering bboxes. The bbox IoU branch learns to recognize
high quality bounding boxes and, as the default behavior, also to pre-filter
Teacher’s pseudo-bboxes depending on our new threshold score. In Figure 3(c)
and in Table 2 (rows #2 and #3), we see that our new branch contributes to the
increase of the general performance (+0.48% mAP). Then, another small
improvement is given by the filtering phase, increasing the performance by
+0.1% mAP.

Bounding box training threshold $\mu$. In this experiment, we test the bbox
IoU classification branch, setting inference threshold $\theta$ to 0.5 and
varying the training threshold $\mu$. From Table 3, it is clear that the
choice of a correct threshold greatly influences the performance. On the one
hand, if the threshold is too low, it does not help the network to learn more
descriptive feature maps. On the other hand, if it is too high, the risk to
wrongly filter out the bounding boxes will increase. As we can see in Fig.
3(d), with the increase of IoU threshold $\mu$, the number of teacher pseudo-
bboxes filtered during the training increases exponentially. This is likely
due to an imbalance in training, where the higher the threshold, the fewer
high-quality examples are available. The best value is in the middle between
the threshold $u$ (0.5) and the IoU maximum value 1.0.

Bounding box inference filter threshold $\theta$. In this experiment, we test
our bbox IoU branch, setting the training threshold $\mu$ to 0.75 (best value
previously found) and varying the inference threshold $\theta$. In Table 4, we
see that the best value for this threshold is in the middle as expected
because the branch is trained to reply 1 if the bbox is good enough and 0
otherwise.

IL-net: Improving Localization net. Finally, we test the full architecture IL-
net, composed of the unsupervised regression losses and the new IoU
classification branch. Since, using both of them, the contribution of the new
branch is absorbed by the loss of unsupervised regression (see rows 2 and 5 in
Table 5), we performed an ablation study reducing the values in input to the
new branch (see eq. 2). This analysis has allowed us to highlight that by
removing the contribution of the deltas, we can increase the general
performance. This behavior could be explained by the fact that the deltas are
optimized from both losses ($L_{usup}^{reg}$ and $L^{IoU}$), causing the
conflict as a result.

##  5 Conclusions

In this paper, we proposed two new architectural enhancements with respect to
the network proposed in [10]: a new bounding box IoU classification task to
filter out errors on pseudo-labels produced by the Teacher and the
introduction of the unsupervised regression losses. For the former, we
introduced a lightweight branch to predict the bounding box IoU quality. For
the latter, we demonstrated how to successfully integrate it in the training,
balancing the training tasks. Our new model called IL-net, which contains
both, increases the general SSOD performance by a 1.14% AP on COCO dataset in
a limited annotation regime.

### Acknowledgments

This research benefits from the HPC (High Performance Computing) facility of
the University of Parma, Italy.

## References

  * [1] Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015)
  * [2] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [3] He, Y., Zhu, C., Wang, J., Savvides, M., Zhang, X.: Bounding box regression with uncertainty for accurate object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2888–2897 (2019)
  * [4] Jeong, J., Lee, S., Kim, J., Kwak, N.: Consistency-based semi-supervised learning for object detection. Advances in neural information processing systems 32, 10759–10768 (2019)
  * [5] Jiang, B., Luo, R., Mao, J., Xiao, T., Jiang, Y.: Acquisition of localization confidence for accurate object detection. In: Proceedings of the European conference on computer vision (ECCV). pp. 784–799 (2018)
  * [6] Li, Y., Huang, D., Qin, D., Wang, L., Gong, B.: Improving object detection with selective self-supervised self-training. In: European Conference on Computer Vision. pp. 589–607. Springer (2020)
  * [7] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2117–2125 (2017)
  * [8] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980–2988 (2017)
  * [9] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [10] Liu, Y.C., Ma, C.Y., He, Z., Kuo, C.W., Chen, K., Zhang, P., Wu, B., Kira, Z., Vajda, P.: Unbiased teacher for semi-supervised object detection. arXiv preprint arXiv:2102.09480 (2021)
  * [11] Nguyen, N.V., Rigaud, C., Burie, J.C.: Semi-supervised object detection with unlabeled data. In: VISIGRAPP (5: VISAPP). pp. 289–296 (2019)
  * [12] Oksuz, K., Cam, B.C., Kalkan, S., Akbas, E.: Imbalance problems in object detection: A review. IEEE transactions on pattern analysis and machine intelligence (2020)
  * [13] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28, 91–99 (2015)
  * [14] Samuli, L., Timo, A.: Temporal ensembling for semi-supervised learning. In: International Conference on Learning Representations (ICLR). vol. 4, p. 6 (2017)
  * [15] Sohn, K., Zhang, Z., Li, C.L., Zhang, H., Lee, C.Y., Pfister, T.: A simple semi-supervised learning framework for object detection. arXiv preprint arXiv:2005.04757 (2020)
  * [16] Tang, Y., Chen, W., Luo, Y., Zhang, Y.: Humble teachers teach better students for semi-supervised object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3132–3141 (2021)
  * [17] Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In: NIPS (2017)
  * [18] Tychsen-Smith, L., Petersson, L.: Improving object localization with fitness nms and bounded iou loss. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6877–6885 (2018)
  * [19] Xu, M., Zhang, Z., Hu, H., Wang, J., Wang, L., Wei, F., Bai, X., Liu, Z.: End-to-end semi-supervised object detection with soft teacher. arXiv preprint arXiv:2106.09018 (2021)
  * [20] Zhou, Q., Yu, C., Wang, Z., Qian, Q., Li, H.: Instant-teaching: An end-to-end semi-supervised object detection framework. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4081–4090 (2021)
  * [21] Zhu, L., Xie, Z., Liu, L., Tao, B., Tao, W.: Iou-uniform r-cnn: Breaking through the limitations of rpn. Pattern Recognition 112, 107816 (2021)


[affiliation=1,2]GuillemBonafos [affiliation=3,4]ClaraBourot
[affiliation=3]PierrePudlo [affiliation=3]Jean-MarcFreyermuth
[affiliation=3]LaurenceReboul [affiliation=5]SamuelTronçon
[affiliation=4]ArnaudRey

# Dirichlet process mixture model based on topologically augmented signal
representation for clustering infant vocalizations

###### Abstract

Based on audio recordings made once a month during the first 12 months of a
child's life, we propose a new method for clustering this set of
vocalizations. We use a topologically augmented representation of the
vocalizations, employing two persistence diagrams for each vocalization: one
computed on the surface of its spectrogram and one on the Takens' embeddings
of the vocalization. A synthetic persistent variable is derived for each
diagram and added to the MFCCs (Mel-frequency cepstral coefficients). Using
this representation, we fit a non-parametric Bayesian mixture model with a
Dirichlet process prior to model the number of components. This procedure
leads to a novel data-driven categorization of vocal productions. Our findings
reveal the presence of 8 clusters of vocalizations, allowing us to compare
their temporal distribution and acoustic profiles in the first 12 months of
life.

###### keywords:

clustering, Bayesian non-parametric, Dirichlet process, mixture model,
topologically-augmented machine learning, TDA, babbling, language development,
vocalizations

##  1 Introduction

During the first year of life, the vocal productions of human infants undergo
a developmental trajectory, actively exploring their acoustic environment
through behaviors such as crying, cooing, and babbling. Infants adapt the
evolution and diversification of their vocalizations to a target language [1]
and typically produce their first word by the end of the first year [2].
Monitoring these pre-language vocalizations is of great importance, not only
for gaining a deeper comprehension of the distinct phases of language
development but also for predictive insights into various disorders [3, 4].
The use of advanced storage and recording tools allows for the creation of
extensive new databases. When combined with innovative statistical analysis
techniques, these tools contribute to a deeper exploration of the early stages
of language development [5, 6].

In this study, we worked with a database that includes vocalizations
automatically extracted from long-form audio recordings of a child at home.
Recordings were done over her first year, spanning from 0 to 12 months, with
three days of recordings per month. The outcome is a longitudinal vocalization
database, capturing vocalizations in a real-life setting and diverse contexts.

Our objective is to propose a novel method to categorize vocal productions,
without predefining the number of categories, but rather estimating them from
the data. To achieve this, we employ a non-parametric Bayesian model,
specifically a Dirichlet process mixture model. The clustering process is
grounded in a topologically augmented representation of the signal, allowing
the incorporation of additional information pertaining to the topology of the
vocalizations.

For clustering, we need to represent vocalizations in a low-dimensional space.
Topological Data Analysis (TDA), which has demonstrated its efficacy across
various domains [7, 8, 9], is a promising candidate for enhancing the current
representation of infant vocalizations. Its stability-theoretic properties
make it particularly valuable for the examination of natural signals [10]. The
integration of topological information can provide valuable additional
information for a more nuanced description of these vocal productions.

In the subsequent sections, we provide an overview of the database in Section
2. The computation of the augmented topological representation and the
clustering model is detailed in Section 3. Section 4 outlines the clustering
results, followed by a comprehensive discussion in Section 5. Finally, we draw
conclusions in Section 6.

##  2 Data

The dataset comprises vocalizations of a child, spanning from birth to the
child's first birthday. Each vocalization is represented by a stereo-channel
audio signal sampled at $44.1\text{\,}\mathrm{kHz}$ in PCM format. Extracted
from longer audio files, the signals are converted to mono by averaging both
channels and then rescaled the pulse modulation signal to a range of $-1$ to
$1$.

These vocalizations originate from long-form audio recordings made by the
parents of a female French child at home, at regular intervals, over a one-
year period. Ethical approval was obtained, along with a declaration of
conformity for experimental research involving humans, allowing for the
recording of human baby vocalizations. Parents, equipped with a portable
microphone located near the child, recorded audio samples three days a month,
capturing various moments throughout the day and night. Following the
methodology outlined in [11], we automatically extracted all the segments
labeled as baby vocalizations from these continuous recordings, resulting in a
dataset of 1924 vocalizations. Unfortunately, due to legal constraints, we are
unable to publicly share the data. Vocalizations lasting more than
$10\text{\,}\mathrm{s}$ were excluded, yielding a final set of 1851
vocalizations with an average duration of $2.51\text{\,}\mathrm{s}$.

Table 1 provides the distribution of vocalizations detected over the first
year. It's noteworthy that vocalizations are not available for every month;
specifically, we lack data for the first, fourth, fifth, and tenth months. The
absence of vocalizations for these months stems from the inability to conduct
recordings during these periods or the absence of detected vocalizations in
the recordings.

Table 1: Number of vocalizations per month in the long-form audio recordings, as well as the mean and the standard deviation of the duration of the vocalizations produced per month Month | Count | Mean duration | Standard deviation
---|---|---|---
$2$ | $667$ | $2.09\text{\,}\mathrm{s}$ | $1.57\text{\,}\mathrm{s}$
$3$ | $139$ | $2.71\text{\,}\mathrm{s}$ | $2.00\text{\,}\mathrm{s}$
$6$ | $132$ | $3.12\text{\,}\mathrm{s}$ | $2.12\text{\,}\mathrm{s}$
$7$ | $154$ | $2.81\text{\,}\mathrm{s}$ | $1.94\text{\,}\mathrm{s}$
$8$ | $159$ | $2.74\text{\,}\mathrm{s}$ | $2.15\text{\,}\mathrm{s}$
$9$ | $285$ | $2.75\text{\,}\mathrm{s}$ | $2.03\text{\,}\mathrm{s}$
$11$ | $212$ | $2.59\text{\,}\mathrm{s}$ | $1.91\text{\,}\mathrm{s}$
$12$ | $98$ | $2.62\text{\,}\mathrm{s}$ | $2.09\text{\,}\mathrm{s}$

##  3 Modelling

###  3.1 Topologically augmented signal representation

We give here the technical details for reproducibility, we refer the reader to
[12] for more details. TDA assumes that data has a shape [13]. We recover this
shape through a filtration, a nested sequence of simplicial complexes [14]. We
then derive from the filtration the persistent homology, which serves as a
topological descriptor of the data. In our case, we represent each
vocalization by using two different objects: the surface of its spectrogram
and its Takens' embeddings. Depending on the object, we then adapt the
filtration to compute the persistent homology.

For each audio recording, we first compute the spectrogram using a Gaussian
window of $11.6\text{\,}\mathrm{ms}$ and a $90\%$ overlap. The spectrogram
$S(t,\omega)=|F(t,\omega)|^{2}$, where $F(t,\omega)$ denotes the Short Time
Fourier Transform, defines a surface in $\mathbb{R}^{3}$, with dimensions
representing time $t$, frequency $\omega$ and amplitude $S$. We apply a
sublevel set filtration to compute the persistent homology of the spectrogram,
i.e., for $f:S\rightarrow\mathbb{R}$, we compute a nested sequence of
topological spaces $S_{r}=f^{-1}(-\infty,r]$ for increasing value of $r$.

Second, we compute the Takens' embeddings, which embed a time series into a
$D$ dimensional Euclidean space using time-delay [15]. We estimate the time
delay parameter $\tau$ such that $\text{AMI}(\tau)<1/e$, where AMI is the
Average Mutual Information. We estimate the embedding dimension $D$ using
Cao's algorithm [16]. We reduce the dimension $D$ to 3 for all embeddings
using UMAP [17], ensuring uniformity across all embeddings. This yields the
vocalization representation as a point cloud
$\mathcal{P}_{D}=\\{p_{1},...p_{D}\\}\subset\mathbb{R}^{D}$ where
$p_{i}=(x_{i},x_{i+\tau},x_{i+2\tau},...,x_{i+(D-1)\tau}).$ We apply an Alpha
filtration to compute the persistent homology of the embeddings, involving a
nested family of Alpha complex
$Alpha(r)=\\{\sigma\subseteq\mathcal{P}|\bigcap_{p\in\sigma}R_{x}(r)\neq\emptyset\\},$
where $R_{x}(r)$ is the intersection of each Euclidean ball with its
corresponding Voronoi cell, for increasing value of $r$.

For both objects, we have an increasing sequence of topological spaces. We
compute the homology at all scales, i.e., for all $r$ of the sequence. We
resume in a persistence diagram the persistence homology of the object, where
a point in a diagram has two coordinates, the value $r$ of its birth and the
value $r$ of its death. Persistent homology then yields a multiscale
topological description of the object [18].

Persistence diagrams cannot be used directly for statistical analysis. We
therefore extract information from the diagrams by computing a set of
variables: persistent entropy [19], $p$-norm of the diagram [20], persistent
Betti number [21], and descriptors of the vector collecting the lifetime of
the points of the diagram following [22, 23]. From this set of variables for
each diagram, we compute a synthetic persistent variable using PCA. The first
principal component of the PCA is retained, explaining $27.79\%$ of the
variance of the set of variables from the persistence diagram of the
spectrogram surface, and $65.94\%$ of the variance of the set of variables
from the persistence diagram of Takens' embeddings.

In addition to the topological features, we compute Mel Frequency Cepstral
Coefficients (MFCC), classical frequency descriptors of human speech analysis
[24]. We compute twelve coefficients, with a window length of
$25\text{\,}\mathrm{ms}$ and an overlap of $40$%. We take the average of the
twelve coefficients to ensure a consistent number of MFCC for all
vocalizations.

The resulting topologically augmented representation of vocalizations
comprises fourteen dimensions: 12 MFCC and 2 synthetic persistent variables,
with one summarizing the persistence diagram computed on the surface of the
spectrogram and another summarizing the persistence diagram computed on
Takens' embeddings.

###  3.2 Nonparametric Bayesian modelling for clustering

We aim to determine the number of clusters in the dataset $X$ using a
Dirichlet process mixture model. The mixture model is defined as
$p(\bm{x})=\int_{\Theta}f(\bm{x};\theta)dG(\theta)$, where $\Theta$ is the
parameter space, $f$ is a $p$-dimensional Gaussian kernel. Consequently,
$\theta=(\bm{\mu},\bm{\Sigma})$, and $\Theta=\mathbb{R}^{p}\times S_{+}^{p}$,
where $S_{+}^{p}$ is the space of semi-definite positive $p\times p$ matrices.
We are interested in determining the number $K$ of mixture components and the
assignment of vocalizations to these components. To learn the complexity of
the model on the data, we set a Dirichlet process as a prior on $G$ and define
the model as follows:

$\begin{split}\bm{x}_{i}|\theta_{i}&\sim f(\bm{x}_{i};\theta_{i})\qquad
i=1,\ldots,N\\\ \theta_{i}|G&\sim G\\\ G&\sim DP(\alpha,G_{0}),\end{split}$
(1)

where $DP$ is a Dirichlet process. See [25] for further reading. We choose the
concentration parameter $\alpha$ such that
$\mathbb{E}[K|n,\alpha]=\sum_{i=1}^{n}\frac{\alpha}{\alpha+i-1}$ [26] to favor
values around $K=5$, following expectations based on prior knowledge [27].

For conjugacy, we put a normal-inverse Wishart prior on the base measure
$G_{0}$, with $\Sigma_{j}\sim IW(\nu_{0},\Sigma_{0})$ and
$\mu_{j}|\Sigma_{j}\sim\mathcal{N}(\bm{m}_{0},\Sigma_{0}/k_{0})$. We use non-
informative priors for the degree of freedom of the inverse Wishart,
$\nu_{0}=p$, and hyperpriors
$\bm{m}_{0}\sim\mathcal{N}(\bm{m}_{1},\bm{S}_{1})$,
$k_{0}\sim\mathcal{G}amma(\tau_{1},\xi_{1})$, and $\Sigma_{0}\sim
W(\nu_{1},\bm{\Sigma}_{1})$. We follow an empirical Bayes procedure to
calibrate the hyperparameters on the dataset: $\bm{m}_{1}$ is the mean of each
dimension of $X$, $\bm{S}_{1}$ is the variance-covariance matrix,
$\tau_{1}=\xi_{1}=1$, $\nu_{1}=p+2$, and $\bm{\Sigma}_{1}=\bm{S}_{1}/2$.

We utilize the collapsed Gibbs sampler of [28], based on the Chinese
Restaurant Process representation, to sample the indicator variable
$z=\\{z_{i}\\}_{i=1}^{N}$, which assigns each vocalization to a latent cluster
by marginalizing mixture weights and parameters. This assignment gives us the
clustering, and we run the MCMC with 10,000 iterations, discarding the first
4,000 as burn-in. As a Bayesian model, the posterior provides a distribution
on possible clusterings rather than a single point estimate. Following [29],
we select the best clustering by specifying a loss function of the true
clustering. The loss function used is the Variation of Information and the
estimate is the one that minimizes the posterior expected loss.

###  3.3 Acoustics differences between clusters

After obtaining the partition, we proceed to compare the different clusters by
computing various acoustic descriptors. Subsequently, we utilize these
descriptors as input for a multinomial logit model, where the cluster serves
as the response variable. We estimate one model per cluster, treating each
cluster as the referential group. For each of the eight models, we assess the
statistical differences in each acoustic descriptor between the referential
group and the other clusters. This analysis provides insights into the
distinctive acoustic characteristics associated with each cluster, helping to
characterize and differentiate them based on the selected features.

##  4 Data analysis

###  4.1 Partition

Our model identifies 8 distinct clusters. Initially, we detected 9 clusters,
but one of them comprised only 5 records, and none of these records included
baby vocalizations. This cluster essentially served as a "garbage cluster,"
grouping false positives that remained in the dataset. The Dirichlet process
mixture model, leveraging our topologically augmented representation,
effectively groups together recordings that differ from the rest of the
dataset. It automatically recognizes and segregates a "garbage" class, helping
eliminate false positives.

Table 2 provides a summary of the cluster distribution by month of the year,
indicating the proportion of production for each cluster during each month.
This breakdown offers insights into how vocalization patterns vary across
different clusters and months.

Table 2: Proportion (percentage) of production for each cluster during the year | Month
---|---
Cluster | 2 | 3 | 6 | 7 | 8 | 9 | 11 | 12
1 | 15 | 16 | 2 | 9 | 11 | 22 | 19 | 6
2 | 11 | 4 | 1 | 7 | 16 | 33 | 20 | 8
3 | 65 | 2 | 2 | 2 | 7 | 15 | 4 | 3
4 | 31 | 23 | 0 | 8 | 0 | 31 | 8 | 0
5 | 83 | 1 | 4 | 2 | 3 | 1 | 6 | 1
6 | 24 | 11 | 4 | 7 | 16 | 4 | 24 | 9
7 | 22 | 10 | 20 | 16 | 8 | 6 | 10 | 8
8 | 0 | 2 | 3 | 15 | 2 | 73 | 3 | 3

###  4.2 Comparison of clusters

The insights from Table 2 reveal distinctive temporal patterns among the
clusters. First, Cluster 2 is characterized by late vocalizations in the first
year, with a substantial portion (a third) produced in the ninth month and a
notable increase in production during the eighth month. About 20% of its
production occurs during the eleventh month, indicating that more than half of
the cluster's production takes place after the ninth month. Similar to Cluster
2, Cluster 8 represents late vocalizations, primarily produced from the ninth
month onwards. In contrast to Clusters 2 and 8, Cluster 5 comprises
vocalizations produced predominantly in the first months of life, with over
80% occurring during this period. This cluster encapsulates early
vocalizations, which decrease as the child learns to produce other types of
vocalizations. Like Cluster 5, Cluster 3 also contains vocalizations primarily
produced in the first two months, constituting 65% of its production. Clusters
4 and 7 exhibit a skewed distribution towards the first few months of life,
with the majority of vocalizations produced in the first 6 months, and even
just the first three months for Cluster 4. However, there is comparatively
more vocal production from these clusters over the rest of the year than
Clusters 3 and 5. Clusters 1 and 6 stand out for being produced throughout the
entire year, indicating a more consistent vocalization pattern across the
different months.

These temporal variations in vocalization patterns highlight the diversity of
the clusters and the developmental changes in vocal behavior over the course
of the first year of life, that we illustrate in Figure 1. Whereas we have the
proportion of vocalization of each cluster per month (i.e., it sums to one per
cluster) in Table 2, we plot in Figure 1 the proportion of vocalization of
each cluster at each month (i.e., it sums to one per month).

Figure 1: Proportion of monthly production of vocalization per cluster.
Parents did not record during three months, yet the gap.

We present a summary of median acoustic descriptors per cluster in Table 3.
Utilizing these descriptors, we employed multinomial logit models to estimate
acoustic differences among the identified clusters. To enhance clarity, we
provide a concise overview of the main results and distinctions between
clusters. Specifically, we highlight instances where the parameter associated
with a descriptor is statistically different from zero for the majority of
other levels (i.e., from other clusters).

Clusters 2 and 8, characterized as late vocalizations, exhibit differences
from other clusters. Cluster 2 differs in its proportion of voiced frames,
entropy level, and $F_{3}$. Cluster 8, on the other hand, varies in spectral
centroid level, entropy level, $F_{2}$, and $F_{3}$.

Clusters 3 and 5, representing earlier vocalizations, also show distinctions
from other clusters. Cluster 3 differs in its proportion of voiced frames,
spectral centroid level, loudness, and $F_{3}$. Cluster 5 exhibits differences
in its proportion of voiced frames, spectral centroid level, entropy level,
Harmonics-to-Noise Ratio, loudness, Frequency Modulation, and $F_{3}$.

Cluster 4 stands out from others due to differences in its proportion of
voiced frames and entropy level. Cluster 7 differs in the proportion of voiced
frames, spectral centroid level, loudness, $F_{2}$, and $F_{3}$. Cluster 1
exhibits distinctions in the proportion of voiced frames, spectral centroid
level, Frequency Modulation, loudness, roughness, $F_{1}$, and $F_{3}$.
Lastly, Cluster 6 varies in its proportion of voiced frames, spectral centroid
level, and $F_{3}$.

Table 3: Acoustics descriptors of each cluster. We report the median for each cluster. Unit of Pitch, Formants, Spectral Centroid and FM is , Loudness is in , HNR is in , Duration is in sec, Voiced and Roughness are proportions. Cluster | Duration | Pitch | F1 | F2 | F3 | Voiced | SC | Entropy | HNR | FM | Loudness | Roughness
---|---|---|---|---|---|---|---|---|---|---|---|---
1 | $1.4$ | $301$ | $725$ | $2374$ | $4640$ | $46$ | $3084$ | $0.33$ | $9.57$ | $6.23$ | $12.4$ | $22.7$
2 | $2.6$ | $334$ | $832$ | $2041$ | $3757$ | $48$ | $2596$ | $0.24$ | $12.5$ | $5.46$ | $13.0$ | $20.5$
3 | $1.6$ | $338$ | $1534$ | $3571$ | $6000$ | $17$ | $2268$ | $0.27$ | $10.7$ | $5.46$ | $7.15$ | $23.1$
4 | $6.6$ | $342$ | $878$ | $2610$ | $4804$ | $16$ | $1870$ | $0.35$ | $10.5$ | $5.00$ | $11.2$ | $23.3$
5 | $1.4$ | $316$ | $913$ | $3056$ | $5685$ | $30$ | $3769$ | $0.41$ | $8.41$ | $6.16$ | $10.3$ | $23.8$
6 | $2.8$ | $382$ | $925$ | $2546$ | $4244$ | $43$ | $3549$ | $0.36$ | $8.23$ | $5.46$ | $13.7$ | $22.6$
7 | $2.2$ | $346$ | $856$ | $2117$ | $3724$ | $28$ | $3252$ | $0.39$ | $10.4$ | $5.44$ | $17.1$ | $21.2$
8 | $1.6$ | $381$ | $1034$ | $2384$ | $4065$ | $23$ | $8528$ | $0.48$ | $9.29$ | $5.46$ | $12.8$ | $23.8$

##  5 Discussion

We conducted an analysis on a unique dataset comprising 1851 vocalizations
from a baby, spanning her birth to her first birthday. These vocalizations
were extracted from longitudinal recordings captured at home, devoid of
external interactions. Employing an innovative topologically augmented signal
representation, we adapted an unsupervised strategy to cluster the
vocalizations based on this representation.

Remarkably, certain clusters of vocalizations only emerge after a specific
period, while others exhibit a decreasing production trend. Clusters 3 and 5,
prominently generated post-birth, experience minimal production throughout the
remainder of the year. It is plausible to hypothesize that these clusters
represent the initial vocalization classes, serving as a foundation for
subsequent vocalization categories.

In contrast, clusters 2 and 8 materialize towards the end of the year,
predominantly around the ninth month, with cluster 8 showing distinctive
formant characteristics. These late-emerging clusters coincide with the
child's increased diversification of vocal productions and heightened
babbling, a phenomenon documented in the literature, peaking between the ninth
and tenth months [30, 31].

From a language development perspective, aligning with the concept of
calibration [1], the child undergoes a learning process during the initial
months, gradually mastering her vocal apparatus to produce sounds resembling
the phonemes of her native language. Notably, an early vocalization cluster
such as cluster 5 also exhibits nonlinear phenomena, indicative of strong
vocal tension [32].

Cluster 5, primarily produced in the initial two months, stands out with a
notably high entropy level. On the contrary, Cluster 2, produced later in the
year, distinguishes itself from other clusters with lower entropy. This
suggests an improvement in the child's motor control of the buco-phonatory
apparatus over the course of the year, resulting in vocalizations with lower
entropy compared to earlier productions. The identified clusters, with diverse
temporal distributions, highlight variations in precocity among vocal
productions.

The incorporation of topological information in signal representation proves
effective in clustering vocalizations based on various acoustic parameters.
Notably, we observe no differences in pitch between clusters. Given that we
have only one child in this database and that pitch is a good individual
marker [33], our model does not rely on this feature for clustering.

However, our approach exhibits limitations. First, our model treats all
vocalizations as exchangeable, neglecting time dependence. To do this, we need
to consider the temporal aspect by employing a non-parametric regression [34],
enabling the incorporation of covariates.

Moreover, while the current topological representation aids in identifying
clusters with distinct acoustic profiles, refinement is needed. Synthetic
persistent variables are constructed to mitigate the curse of dimensionality,
resulting in information loss, particularly for the persistent homology of the
spectrogram. Exploring methods for constructing a lower-dimensional signal
representation that incorporates topological information, such as [35, 36] is
a worthwhile avenue.

In terms of modeling, incorporating expert knowledge and refining priors,
especially the choice of $\alpha$, could enhance the clustering process.
Adjusting $\alpha$ might impact the final clustering outcome, and its initial
computation based on expecting five clusters could be refined based on the
diversity of vocalizations observed in literature during a baby's first year
[30, 37].

Finally, the current analysis, although valuable for its longitudinal and
ecological nature, pertains to a single child. The results, while insightful,
cannot propose a new categorization of vocal productions. Future research
should deepen the analysis by including more children. Introducing hierarchy
in subsequent analyses could facilitate comparisons of vocal productions and
their evolution over the first year of life, considering integrated
covariates.

##  6 Conclusion

In conclusion, we investigated a novel database comprising vocalizations
extracted from long-form audio recordings of a child from birth to her first
birthday. This dataset offers a unique perspective, capturing vocalizations in
an uncontrolled, longitudinal setting without interaction, allowing for the
exploration of new inquiries.

We introduced an innovative approach to analyze this database, aiming to
identify distinct clusters of vocalizations produced by the child. Employing
an unsupervised methodology, we utilized a Dirichlet process mixture model
without specifying the number of classes beforehand. By incorporating
topological information into the signal representation, we successfully
identified 8 vocalization classes throughout the year.

Acknowledging the outlined limitations, the detected clusters exhibited
varying production proportions over time. Furthermore, our topologically
augmented representation facilitated the identification of clusters with
diverse acoustic profiles, illustrating the child's evolving motor control of
her buco-phonatory apparatus.

## References

  * [1] S. M. ter Haar, A. A. Fernandez, M. Gratier, M. Knörnschild, C. Levelt, R. K. Moore, M. Vellema, X. Wang, and D. K. Oller, ``Cross-species parallels in babbling: Animals and algorithms,'' _Philosophical Transactions of the Royal Society B: Biological Sciences_ , vol. 376, no. 1836, p. 20200239, Oct. 2021.
  * [2] P. K. Kuhl, ``Early language acquisition: Cracking the speech code,'' _Nature Reviews Neuroscience_ , vol. 5, no. 11, pp. 831–843, Nov. 2004.
  * [3] D. K. Oller, R. E. Eilers, A. R. Neal, and H. K. Schwartz, ``Precursors to speech in infancy: The prediction of speech and language disorders,'' _Journal of Communication Disorders_ , vol. 32, no. 4, pp. 223–245, Jul. 1999.
  * [4] K. D. Bartl-Pokorny, F. B. Pokorny, D. Garrido, B. W. Schuller, D. Zhang, and P. B. Marschik, ``Vocalisation Repertoire at the End of the First Year of Life: An Exploratory Comparison of Rett Syndrome and Typical Development,'' _Journal of Developmental and Physical Disabilities_ , Mar. 2022.
  * [5] D. K. Oller, P. Niyogi, S. Gray, J. A. Richards, J. Gilkerson, D. Xu, U. Yapanel, and S. F. Warren, ``Automated vocal analysis of naturalistic recordings from children with autism, language delay, and typical development,'' _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 107, no. 30, pp. 13 354–13 359, Jul. 2010.
  * [6] M. Milling, F. B. Pokorny, K. D. Bartl-Pokorny, and B. W. Schuller, ``Is Speech the New Blood? Recent Progress in AI-Based Disease Detection From Audio in a Nutshell,'' _Frontiers in Digital Health_ , vol. 4, p. 886615, May 2022.
  * [7] A. Salch, A. Regalski, H. Abdallah, R. Suryadevara, M. J. Catanzaro, and V. A. Diwadkar, ``From mathematics to medicine: A practical primer on topological data analysis (TDA) and the development of related analytic tools for the functional discovery of latent structure in fMRI data,'' _PLOS ONE_ , vol. 16, no. 8, p. e0255859, Aug. 2021.
  * [8] Y. Cao, S. Zhang, F. Yan, W. Li, F. Sun, and H. Sun, ``Unsupervised Environmental Sound Classification Based On Topological Persistence,'' in _2019 IEEE International Conference on Signal, Information and Data Processing (ICSIDP)_ , Dec. 2019, pp. 1–5.
  * [9] F. Hensel, M. Moor, and B. Rieck, ``A Survey of Topological Machine Learning Methods,'' _Frontiers in Artificial Intelligence_ , vol. 4, p. 681108, May 2021.
  * [10] D. Cohen-Steiner, H. Edelsbrunner, and J. Harer, ``Stability of Persistence Diagrams,'' _Discrete & Computational Geometry_, vol. 37, no. 1, pp. 103–120, Jan. 2007.
  * [11] G. Bonafos, P. Pudlo, J.-M. Freyermuth, T. Legou, J. Fagot, S. Tronçon, and A. Rey, ``Detecting human and non-human vocal productions in large scale audio recordings,'' Feb. 2023.
  * [12] F. Chazal and B. Michel, ``An Introduction to Topological Data Analysis: Fundamental and Practical Aspects for Data Scientists,'' _Frontiers in Artificial Intelligence_ , vol. 4, 2021.
  * [13] G. Carlsson, ``Topology and data,'' _Bulletin of the American Mathematical Society_ , vol. 46, no. 2, pp. 255–308, 2009.
  * [14] A. Zomorodian and G. Carlsson, ``Computing Persistent Homology,'' _Discrete & Computational Geometry_, vol. 33, no. 2, pp. 249–274, Feb. 2005.
  * [15] F. Takens, ``Detecting strange attractors in turbulence,'' in _Dynamical Systems and Turbulence, Warwick 1980_ , D. Rand and L.-S. Young, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 1981, vol. 898, pp. 366–381.
  * [16] L. Cao, ``Practical method for determining the minimum embedding dimension of a scalar time series,'' _Physica D: Nonlinear Phenomena_ , vol. 110, no. 1, pp. 43–50, Dec. 1997.
  * [17] L. McInnes, J. Healy, and J. Melville, ``UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction,'' Sep. 2020.
  * [18] L. Wasserman, ``Topological Data Analysis,'' _Annual Review of Statistics and Its Application_ , vol. 5, no. 1, pp. 501–532, 2018.
  * [19] N. Atienza, R. Gonzalez-Diaz, and M. Rucco, ``Persistent entropy for separating topological features from noise in vietoris-rips complexes,'' _Journal of Intelligent Information Systems_ , vol. 52, no. 3, pp. 637–655, Jun. 2019.
  * [20] D. Cohen-Steiner, H. Edelsbrunner, J. Harer, and Y. Mileyko, ``Lipschitz Functions Have L p -Stable Persistence,'' _Foundations of Computational Mathematics_ , vol. 10, no. 2, pp. 127–139, Apr. 2010.
  * [21] H. Edelsbrunner and J. Harer, _Computational Topology: An Introduction_. AMS Press, 2009.
  * [22] T. Fireaizen, S. Ron, and O. Bobrowski, ``Alarm Sound Detection Using Topological Signal Processing,'' in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. Singapore, Singapore: IEEE, May 2022, pp. 211–215.
  * [23] C. M. Pereira and R. F. de Mello, ``Persistent homology for time series and spatial data clustering,'' _Expert Systems with Applications_ , vol. 42, no. 15-16, pp. 6026–6038, Sep. 2015.
  * [24] J. Sueur, _Sound Analysis and Synthesis with R_ , 1st ed. New York, NY: Springer Berlin Heidelberg, 2018.
  * [25] Y. W. Teh, ``Dirichlet processes,'' in _Encyclopedia of Machine Learning_. Springer, 2010.
  * [26] R. M. Dorazio, ``On selecting a prior for the precision parameter of Dirichlet process mixture models,'' _Journal of Statistical Planning and Inference_ , vol. 139, no. 9, pp. 3384–3390, Sep. 2009.
  * [27] M. Cychosz, A. Seidl, E. Bergelson, M. Casillas, G. Baudet, A. S. Warlaumont, C. Scaff, L. Yankowitz, and A. Cristia, ``BabbleCor: A Crosslinguistic Corpus of Babble Development in Five Languages,'' Oct. 2019.
  * [28] R. M. Neal, ``Markov Chain Sampling Methods for Dirichlet Process Mixture Models,'' _Journal of Computational and Graphical Statistics_ , vol. 9, no. 2, pp. 249–265, 2000.
  * [29] S. Wade and Z. Ghahramani, ``Bayesian Cluster Analysis: Point Estimation and Credible Balls (with Discussion),'' _Bayesian Analysis_ , vol. 13, no. 2, pp. 559–626, Jun. 2018.
  * [30] D. K. Oller, E. H. Buder, H. L. Ramsdell, A. S. Warlaumont, L. Chorna, and R. Bakeman, ``Functional flexibility of infant vocalization and the emergence of language,'' _Proceedings of the National Academy of Sciences_ , vol. 110, no. 16, pp. 6318–6323, Apr. 2013.
  * [31] M. Cychosz, A. Cristia, E. Bergelson, M. Casillas, G. Baudet, A. S. Warlaumont, C. Scaff, L. Yankowitz, and A. Seidl, ``Vocal development in a large-scale crosslinguistic corpus,'' _Developmental Science_ , vol. 24, no. 5, Sep. 2021.
  * [32] A. Koutseff, D. Reby, O. Martin, F. Levrero, H. Patural, and N. Mathevon, ``The acoustic space of pain: cries as indicators of distress recovering dynamics in pre-verbal infants,'' _Bioacoustics_ , vol. 27, no. 4, pp. 313–325, Oct. 2018. [Online]. Available: https://www.tandfonline.com/doi/full/10.1080/09524622.2017.1344931
  * [33] M. Lockhart-Bouron, A. Anikin, K. Pisanski, S. Corvin, C. Cornec, L. Papet, F. Levréro, C. Fauchon, H. Patural, D. Reby, and N. Mathevon, ``Infant cries convey both stable and dynamic information about age and identity,'' _Communications Psychology_ , vol. 1, no. 1, pp. 1–15, Oct. 2023.
  * [34] F. A. Quintana, P. Müller, A. Jara, and S. N. MacEachern, ``The Dependent Dirichlet Process and Related Models,'' _Statistical Science_ , vol. 37, no. 1, pp. 24–41, Feb. 2022.
  * [35] M. Moor, M. Horn, B. Rieck, and K. Borgwardt, ``Topological Autoencoders,'' in _arXiv:1906.00722 [Cs, Math, Stat]_ , vol. PMLR 119, 2020, pp. 7045–7054.
  * [36] I. Trofimov, D. Cherniavskii, E. Tulchinskii, N. Balabin, E. Burnaev, and S. Barannikov, ``Learning topology-preserving data representations,'' in _The Eleventh International Conference on Learning Representations_ , Feb. 2023.
  * [37] Y. Jhang, ``Emergence of Functional Flexibility in Infant Vocalizations of the First 3 Months,'' _Frontiers in Psychology_ , vol. 8, p. 11, 2017.


Robotic manipulation stands as a largely unsolved problem despite significant advances in robotics and machine learning in recent years.
One of the key challenges in manipulation is the exploration of the dynamics of the environment when there is continuous contact between the objects being manipulated.
This paper proposes a model-based active exploration approach that enables efficient learning in sparse-reward robotic manipulation tasks.
The proposed method estimates an information gain objective using an ensemble of probabilistic models and deploys model predictive control (MPC) to plan actions online that maximize the expected reward while also performing directed exploration.
We evaluate our proposed algorithm in simulation and on a real robot, trained from scratch with our method, on a challenging ball pushing task on tilted tables, where the target ball position is not known to the agent a-priori.
Our real-world robot experiment serves as a fundamental application of active exploration in model-based reinforcement learning of complex robotic manipulation tasks.
Project page <https://sites.google.com/view/aerm>.

§ INTRODUCTION

Our active exploration strategy evaluated on a challenging
task in simulation (left) and on the real robot (right).
The agent needs to learn the dynamics of the task and identify a sparse reward model in order to bring the ball to a target location.
The robot learns to solve the task online in the real setting from scratch.

A common view in cognitive science is that the evolution of dexterous manipulation capabilities was one of the major driving factors in the development of the human mind <cit.> and the success of humankind in general <cit.>.
Performing manipulation is cognitively highly demanding, forcing the agent to reason not only about the impact of its actions on itself, but also on the environment.
This inherent complexity leaves autonomous robotic manipulation a largely unsolved problem, despite significant advances in robotics and machine learning in the last decades <cit.>.

One of the central challenges of manipulation is the uncertainty about the environment.
When an object is manipulated, its physical properties are rarely known in advance.
Instead, they must be inferred from observations and touch.
To deal with such inference problems effectively, humans have developed various active haptic exploration strategies <cit.>.

Prominent approaches in robotic manipulation span from motion planning methods <cit.> to imitation <cit.> and reinforcement learning <cit.>.
Motion planning usually suffers from ill-defined task descriptions that combined with uninformed prior trajectory distribution lead to suboptimal behaviors.
Learning-based methods, on the other hand, overfit single solutions, and collapse to low-entropy behaviors that fail to generalize to unseen variations of the same task.
We believe that for robots to reach human-level manipulation skills, they must actively explore and adapt to new instances of a task.

We define active exploration as the directed search of the agent, during the learning process, for unvisited state-action pairs that would maximize the agent's performance.
In this work, we draw inspiration from the ai <cit.> field of studies to propose an active exploration framework for model-based reinforcement learning of challenging robotic manipulation tasks.
In our problem formulation, we consider a fully observable environment with unknown world model whose dynamics we need to learn.

Unlike other approaches, that either couple model-free and model-based reinforcement learning to achieve better model learning through costly maximum entropy exploration <cit.>, or introduce intrinsic signals related to the learned model variance to promote exploration <cit.>, we take an information-theoretic perspective, under the umbrella of ai.
Starting from a common framework for model learning, namely, using an ensemble of neural networks that allow the estimation of epistemic uncertainty, we propose to use an information-seeking mpc, that systematically explores in environments with sparse rewards.

Our information-seeking mpc tries to select actions that maximize the agent's information gain, balancing between highly exploratory actions when the dynamics model is unknown and the task performance when the agent has better confidence about its knowledge of the world model.
We provide a thorough theoretical analysis regarding the implementation of our active exploration framework in model-based reinforcement learning.
We evaluate our algorithmic contribution in simulated tasks of increased difficulty, where a 6 degrees of freedom (dof) robotic arm aims to solve a task where it has to push a ball on a tilted table to reach a goal position (<ref>).
This task, though seemingly simple, introduces many challenges, as the robot has to balance the ball at the tip of its end-effector, and push the ball over a tilted table to reach an unknown for the agent goal.
Moreover, we provide proof of concept results on a real robotic system that learns online, and we provide our insights regarding real-world robot learning with challenging dynamics.

To summarize our contributions in this work,

* we derive a novel algorithm for information-seeking model-based reinforcement learning,

* we investigate different measures of curiosity and information seeking strategies that can promote exploration for better dynamics model learning in simulated ball-pushing tasks with different difficulty levels, and

* we demonstrate a real-world execution of our actively exploring model-based controller for 7-dof manipulator learning to push a ball over a tilted table.

§ RELATED WORK

This field of research attempts to solve rl problems by predicting actions as an a posteriori estimate of trajectory rollouts given a prior distribution of actions  <cit.>.
Model-based rl learns the transition dynamics of the Markov decision process (MDP) and solves an optimal control problem, usually employing mpc.
On the one hand, model-based rl is a promising way for learning reactive robot control strategies, benefiting from the integrated planner that can additionally integrate constraints about the problem.
Applications of model-based rl can be found for robot in-hand manipulation <cit.>, human-robot interaction <cit.> and robot manipulation skill learning <cit.>.

On the other hand, learning the dynamics of the environment, especially in robotics, is a very challenging problem.
Real-world dynamics, particularly the dynamics of complex manipulation tasks, like the task of interest in this paper, are characterized by multimodality that function approximators cannot easily capture.
A prominent approach in the last years for model-based rl is the use of an ensemble of neural networks that can learn from different instances of the collected dataset to capture the epistemic and aleatoric uncertainty of the probabilistic model, thus, attempting to learn more accurate models <cit.>.
These probabilistic ensembles are coupled with trajectory sampling (PETS) <cit.> via the cross-entropy method to estimate the best actions to apply to the environment.
However, PETS suffers from the local exploration in the model rollouts that do not drive the agent to unknown states in the environment, therefore, it learns suboptimal dynamics models.
Following methods attempt to couple model-free exploration with model learning, as in model-based policy optimization (MBPO) <cit.>, but the purpose is to accelerate policy learning by utilizing model-based approximate samples.
While this method is faster in terms of convergence compared to its pure model-free competitor sac, it still needs an impractical amount of samples to learn a good control policy.
MoPAC <cit.> improved over MBPO by employing model-predictive rollouts in the approximate MDP that is learned through the model, incentivizing the agent to explore areas of the state-space where model predictions are inefficient.
A parallel line of works aims to learn world models from images, i.e., for observations, using a variational autoencoder to encode the image features and a probabilistic model to encode the dynamics with cem for planning, like PlaNet  <cit.>.
Unlike PETS and PlaNet, which greedily select the actions that the cem predicts to yield the highest reward, we propose a novel framework for model-based rl using an information-seeking objective in our mpc, that balances exploration-exploitation during learning, by promoting maximization of the information again when the model is still suboptimal and optimizes for the end-task once the agent is confident about its knowledge about the world dynamics.

Exploration for efficient model learning is an open research topic, with intrinsic motivation being a well-known method for exploration towards model learning <cit.>.
However, intrinsic rewards usually rely on hand-crafted metrics of learning progress, making them difficult to apply in a wide range of tasks <cit.>.
In the era of the deep probabilistic ensembles, a self-supervised way of encouraging exploration of states that will improve model learning are methods based on ensemble disagreement, where the variance in the predictions of the different ensemble models is used as intrinsic reward for a model-free policy <cit.>.
This policy, thus, learns to collect data in areas of the state-space where the disagreement between the predictions of the ensemble models is higher.
However, advantages of these methods compared to MBPO, MoPAC and other model-based methods without explicit exploration bonus <cit.> are not well-established.
Even if disagreement promotes exploration in early stages, it is more beneficial to vision-based rl settings, where the variance between the ensembles can be high due to the high image reconstruction errors, but its benefit to trajectory-based MDPs is incremental, as the ensembles are bounded by the statistics of the marginal state distribution of the dataset.
Other methods for inducing curiosity in rl rely on prediction error <cit.>, epistemic uncertainty <cit.> or state visitation counts <cit.> as reward signals during roll-outs.
Contrary to these methods that introduce heuristic intrinsic rewards or rely on a model-free policy for model learning, our method employs a principled information-theoretic approach that can be traced back to early works on Bayesian experiment design <cit.>.
Specifically, we utilize mpc to trade off between actions that yield a high expected extrinsic reward and actions that maximize the expected information gain of the observed states for our model in real time.
The use of an expected information gain term inside mpc allows our agent to plan even beyond the space of states it has visited so far to obtain observations that it expects to be beneficial for model learning.

§ ACTIVE EXPLORATION FOR MODEL LEARNING

We assume that the environment is fully observable, governed by unknown dynamics $\PP{\vs_\tau}[\vs_{\tau - 1}, \va_\tau]$, and provides the agent with an a-priori unknown reward $\PP{\vr_\tau}[\vs_{\tau}, \va_\tau]$ in every time step.
Here, $\vs_{\tau} \in \R^{N_\vs}$ denotes the state of the environment at time $\tau$, $\va_\tau \in \R^{N_\va}$ the action the agent can take, and $\vr_\tau \in \R$ the reward it is receiving.
The agent's objective is the maximization of the cumulative reward over a fixed time horizon of $T$ discrete time steps, that is

\begin{align}
\max_{\pi} \quad \Ex{\PP{\mathbf{\vr}_{1:T}}[\pi]}{\sum_{\tau = 1}^T \vr_{\tau}}
\end{align}

where $\pr{\pi}{\va_\tau}[\vs_{\tau - 1}]$ is the agent's policy.

Since the real dynamics and reward distributions are unknown, the agent maintains approximations $\p{\vs_\tau}[\va_\tau, \vs_{\tau -1}, \theta]$ and $\p{r_\tau}[\vs_{\tau}, \va_\tau, \theta]$ of them, where $\theta$ are the model parameters.
Hence, the agent assumes the following generative model of the environment:

\begin{align}
\label{eq:model_mdp}
\begin{split}
&\p{\mathbf{\vs}_{0:T}, \mathbf{\va}_{1:T}, \mathbf{\vr}_{1:T}, \theta}
\p[k]{\vs_0}
\p{\theta}
\prod_{\tau = 1}^T
\Big(
\p{\vr_\tau}[\vs_{\tau}, \va_\tau, \theta] \rhookswarrow
\\
&\qquad\p{\vs_\tau}[\vs_{\tau - 1}, \va_\tau, \theta]
\pr{\pi}{\va_\tau}[\vs_{\tau - 1}] \Big)
\end{split}
\end{align}

where $\p[k]{\theta}$ is the agent's belief over the correct model parameters in episode $k$.

Instead of greedily optimizing the expected reward directly, we propose to optimize the sum of the expected reward and an intrinsic term that encourages the agent to make observations informative w.r.t. its model.
Hence, at time $t$, we choose $\pi$ such that it optimizes the problem

\begin{align}
\max_{\pi} \;
\Ex{\p{\mathbf{\vr}_{t+1:t+H}}[\pi]}{\sum_{\tau = t + 1}^{t + H} \vr_{\tau}}
\beta \text{I} \left( \pi, \vs_t \right)
\label{eq:obj}
\end{align}

where $H \in \mathbb{N}$ is the planning horizon, $\beta \in \R$ a weighting factor, and $\text{I} \left( \pi, \vs_t \right)$ the intrinsic term.

In this work, we choose the intrinsic term to be the information gain between the model parameters and the expected states and rewards:

\begin{align}
\text{I} \left( \pi, \vs_t \right)
\text{MI} \left( \left(\mathbf{\vs}, \mathbf{\vr} \right), \theta \mid \pi, \vs_t \right)
\\
\Ex{\p{\mathbf{\vs}, \mathbf{\vr}}[\pi, \vs_t]}{
\kl{\p{\theta}[\mathbf{\vs}, \mathbf{\vr}, \pi, \vs_t]}{\p{\theta}}
\end{align}

where $\mathbf{\vs} \coloneqq \mathbf{\vs}_{t+1:t+H}$ and $\mathbf{\vr} \coloneqq \mathbf{\vr}_{t+1:t+H}$ is the sequence of states and rewards up to the planning horizon.
Note that in the literature, the expected information gain is also known as mi, which is why we denoted it accordingly in the equation above.

The expected information gain can be seen as a measure of how much the agent expects to learn about the environment by following policy $\pi$ in state $s_t$.
Specifically, this measure becomes maximal if the agent expects to make observations that will change its belief about the correct choice of model parameters drastically.
Hence, an agent maximizing this term will be curious about its environment and explore it systematically, even in the total absence of extrinsic reward.
In combination with the expected reward, we obtain an agent that is acting both information-seeking and goal-directed, with the trade-off being explicitly controlled by the weighting factor $\beta$.

The optimization of the objective can now be performed by any planner that is capable of handling continuous action spaces.
In this work, we use a variant of the cem to find an open loop sequence of actions $\mathbf{\va}_{t+1:T}$ that maximizes the objective.

§.§ Approximation of the planning objective

A major challenge in computing the joint objective is that neither the expected reward nor the intrinsic term can be computed in closed form.
While the expected reward can straightforwardly be approximated via mc <cit.>, the intrinsic term is known to be notoriously difficult to compute <cit.>.
Thus, instead of maximizing mi directly, many methods maximize a variational lower bound of it <cit.>.
However, due to the high-dimensional nature of $\theta$, these approaches are too expensive to be executed during planning in real time.

Hence, instead we propose to use a nmc estimator that reuses samples from the outer estimator in the inner estimator:

\begin{align}
\label{eq:meth_nmc_est_reuse}
&\text{MI} \left( \left(\mathbf{\vs}, \mathbf{\vr} \right), \theta \mid \pi, \vs_t \right)
\\
\Ex{\p{\theta}}{
\Ex{\p{\mathbf{\vs}, \mathbf{\vr}}[\pi, \vs_t, \theta]}{
\ln \p{\mathbf{\vs}, \mathbf{\vr}}[\pi, \vs_t, \theta]
- \ln \p{\mathbf{\vs}, \mathbf{\vr}}
\\
\underbrace{
\frac{1}{n}
\sum_{i=1}^{n}
( \ln \p{\vs^{i}, \vr^i}[\theta^{i}]
\ln
\underbrace{
\frac{1}{n}
\sum_{\substack{k=1\\k\neq i}}^{n}
\p{\vs^{i}, \vr^i}[\theta^{k}]
}_{\text{inner estimator}}
)}_{\text{outer estimator}}
\end{align}

\begin{equation}
\theta^{i} \sim \p{\theta}, \quad \left(\mathbf{\vs}^i, \mathbf{\vr}^i \right) \sim \p{\mathbf{\vs}, \mathbf{\vr}}[\theta^{i}], \quad \forall i \in \left\{ 1, \dots, n \right\}.
\end{equation}

Although using the same samples $\theta_1, \dots, \theta_n$ in the inner estimator as in the outer estimator violates the iid assumption, we found this reuse of samples can substantially increase the sample efficiency.

§.§ Choice of model

We approximate the dynamics and the reward models with Gaussian distributions where the mean and covariance are given by neural networks with weights $\theta$

\begin{align}
\label{eq:model_def}
\begin{split}
\p{\vs_\tau}[\va_\tau, \vs_{\tau -1}, \theta]
\mathcal{N} \left( \vs_\tau \mid \mu^{\vs}_\theta \left( \vs_{\tau - 1}, \va_\tau \right), \Sigma^{\vs}_\theta \left( \vs_{\tau - 1}, \va_\tau \right) \right)
\\
\p{r_\tau}[\vs_{\tau}, \va_\tau, \theta]
\mathcal{N} \left( r_\tau \mid \mu^r_\theta \left( \vs_\tau, \va_\tau \right), \sigma^r_\theta \left( \vs_\tau, \va_\tau \right) \right).
\end{split}
\end{align}

There are multiple options for representing distributions over neural network parameters.
Common choices are particle-based representations <cit.>, Gaussian distributions with diagonal covariance matrix <cit.> or a combination of both <cit.>.
Since our approximation of the planning objective only requires samples of $\theta$, we choose to represent $\p[k]{\theta}$ by a set of particles $\theta_1, \dots, \theta_n$, making our model a neural network ensemble.

§.§ Lautum Information

title=Cosine similarity over sample count,
legend style=nodes=scale=0.8, transform shape,
reverse legend,
legend pos=south east,
legend cell align=left,
xlabel = Total number of samples,
ylabel = Cosine similarity]

Comparison of the cosine similarity between the sample-reusing nmc approximations of mi and li to their respective exact values.
For each number of samples, we conducted 1000 experiments on randomly generated discrete generative models $\p{\vs, \theta}[\pi]$.
To assess the approximation quality independently of scale, we compute the cosine similarity between the approximated and exact information vectors $\left( \text{I} \left( \vs, \theta \mid \pi_1 \right), \dots, \left( \vs, \theta \mid \pi_m \right) \right)^T$, where I is either mi or li and $m$ is the number of policies in the model.
Number of samples refers to the sum of $\vs$ and $\theta$ samples.
Note that the optimal value possible under cosine similarity is 1.

An alternative to our choice of the intrinsic term is to use the reverse kl divergence, yielding the li <cit.>, which in our case is defined as

\begin{align}
\text{I} \left( \pi, \vs_t \right)
\text{LI} \left( \left(\mathbf{\vs}, \mathbf{\vr} \right), \theta \mid \pi, \vs_t \right)
\\
\Ex{\p{\mathbf{\vs}, \mathbf{\vr}}[\pi, \vs_t]}{
\kl{\p{\theta}}{\p{\theta}[\mathbf{\vs}, \mathbf{\vr}, \pi, \vs_t]}
\end{align}

li, similar to mi, measures how much information we are expected to gain about $\theta$ by observing $\left(\mathbf{\vs}, \mathbf{\vr} \right)$. However, it does it in a different way and leads to a different result, similar to how reverse KL leads to mode-seeking behavior and the forward KL to the moment matching behavior.

To gain an intuition about the difference between mi and li, it is useful to consider which policy $\pi$ maximizes each of them.
For li, the prior $\p{\theta}$ is in the numerator in the kl divergence term,
therefore li becomes maximal for policies that are expected to produce observations which make a-priori likely $\theta$ have a low probability in the posterior.
In a sense, li encourages the agent to seek out observations that disprove the optimality of those $\theta$ that the agent assigned a high prior probability to.
Analogously, mi encourages the agent to make observations that cause $\theta$ with a low prior probability to have a high posterior probability.

One advantage of li is the independence of $\left(\mathbf{\vs}, \mathbf{\vr} \right)$ and $\theta$ in the outer expectations

\begin{equation}
\text{LI} =
\Ex{\p{\theta}}{
\Ex{\p{\mathbf{\vs}, \mathbf{\vr}}[\pi, \vs_t]}{
\ln \p{\mathbf{\vs}, \mathbf{\vr}} - \ln \p{\mathbf{\vs}, \mathbf{\vr}}[\pi, \vs_t, \theta]
\end{equation}

which allows for a more efficient reuse of samples.
Contrary to the mi approximation, when approximating the inner expectation $\Ex{\p{\mathbf{\vs}, \mathbf{\vr}}[\pi, \vs_t]}{.}$, we can reuse the same samples $\left(\mathbf{\vs}^i, \mathbf{\vr}^i \right)$ for all samples $\theta^j$ from the outer expectation.
Hence, the resulting nmc approximation of li is given as

\begin{equation}
\label{eq:meth_li_reuse_nmc}
\text{LI}
\approx
\frac{1}{n}
\sum_{i=1}^{n}
\ln
\left(
\frac{1}{n}
\sum_{\substack{k=1\\k\neq i}}^{n}
\p{\mathbf{\vs}^i, \mathbf{\vr}^i}[\theta^{k}]
\right)
\frac{1}{n}
\sum_{\substack{j=1\\j\neq i}}^{n}
\ln \p{\mathbf{\vs}^i, \mathbf{\vr}^i}[\theta^{j}].
\end{equation}

An empirical comparison of the stochastic estimators of mi and  li is shown in <ref>,
which suggest a clear advantage in sample efficiency of the li approximation in comparison to the mi approximation.
Theoretical analysis of these estimators may be of interest for future work.
The practical influence of the choice of the information gain term is evaluated in <ref>.
To our knowledge, li has not been used in the context of exploration before.

Further implementation details as well as the link to our code can be found on our project page:

§ CONCLUSION

In this paper, we developed an active exploration method that is capable of solving complex robotic manipulation tasks.
Our main algorithmic contribution is the introduction of an information-seeking strategy in model-based reinforcement learning, that balances between exploration of new states in the environment to improve the dynamics model and task performance.
We evaluated our method on two simulated and one real-world tasks, all designed to be particularly hard to explore.
Throughout our experiments, we showed that our method induces systematic exploratory behavior and learns to solve a manipulation task without dense extrinsic reward, but is driven by its own curiosity.
Considering that none of the baselines were able to solve these problems, we conclude that the information-seeking behavior of our agents is beneficial for solving challenging exploration problems with sparse rewards, suitable for learning complex manipulation tasks in the real world.

In the future, we plan to incorporate tactile sensors into our setup.
Tactile sensors would allow one to obtain more detailed feedback about the objects being manipulated.
For example, the spin of the ball was not observed in our experiments, although it provides a useful signal for the manipulation task.
Considering that humans deploy a variety of active haptic exploration strategies during manipulation, research on robotic active tactile exploration might bring us closer to human-level manipulation skills.
One of the main limitations of our method that prevents it from being used on more dynamic tasks is that it is computationally comparably heavy and therefore limited to relatively slow tasks.
Hence, an exciting future research direction is to tackle this issue with hierarchical controllers by combining our planning module with a learned low-level balancing controller running at a high frequency.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

journaltitleThe American Journal of Psychology
titleThe significance of the human hand in the evolution of mind

Wiley Online Library

journaltitleJournal of Anatomy
titleEvolution of the human hand: the role of throwing and clubbing

journaltitleJournal of Machine Learning Research
titleA review of robot learning for manipulation: Challenges, representations, and algorithms

Taylor & Francis

journaltitleLaterality: Asymmetries of Body, Brain and Cognition
titleManual exploratory procedures and asymmetries for a haptic search task: A comparison between capuchins (Cebus apella) and humans

journaltitlePerception of space and motion
titleDynamic touch

booktitleProc. IEEE Conf. on Robotics and Automation, Scottsdale, AZ, 1989
titleOn motion planning for dexterous manipulation, part I: the problem formulation

journaltitleInternational Journal of Intelligent Robotics and Applications
titleSurvey of imitation learning for robotic manipulation

booktitleConference on Robot Learning
titleScalable deep reinforcement learning for vision-based robotic manipulation

MIT Press

titleActive inference: the free energy principle in mind, brain, and behavior

journaltitlearXiv preprint arXiv:1906.08253
titleWhen to trust your model: Model-based policy optimization

booktitle2021 IEEE International Conference on Robotics and Automation (ICRA)
titleModel predictive actor-critic: accelerating robot skill acquisition with deep reinforcement learning

journaltitlearXiv preprint arXiv:1912.01603
titleDream to control: Learning behaviors by latent imagination

booktitleInternational conference on machine learning
titleCuriosity-driven exploration by self-supervised prediction

booktitleInternational conference on machine learning
titleSelf-supervised exploration via disagreement

booktitleProceedings of the 28th International Conference on machine learning (ICML-11)
titlePILCO: A model-based and data-efficient approach to policy search

journaltitleJournal of Intelligent & Robotic Systems
titleModel-based reinforcement learning variable impedance control for human-robot collaboration

journaltitleIEEE Robotics and Automation Letters
titleLearn to adapt to human walking: A model-based reinforcement learning approach for a robotic assistant rollator

journaltitlearXiv preprint arXiv:1802.09081
titleTemporal difference models: Model-free deep rl for model-based control

booktitleInternational conference on machine learning
titleCombining model-based and model-free updates for trajectory-centric reinforcement learning

journaltitlearXiv preprint arXiv:1805.12114
titleDeep reinforcement learning in a handful of trials using probabilistic dynamics models

booktitleInternational Conference on Machine Learning
titleLearning latent dynamics for planning from pixels

journaltitleIEEE Transactions on Autonomous Mental Development
titleIntrinsically motivated reinforcement learning: An evolutionary perspective

journaltitleArtificial Intelligence
titleIntrinsically motivated model learning for developing curious robots

journaltitleAdvances in neural information processing systems
titleExploration in model-based reinforcement learning by empirically estimating learning progress

booktitleInternational conference on machine learning
titleModel-based active exploration

booktitleInternational Conference on Machine Learning
titlePlanning to explore via self-supervised world models

titleCurious iLQR: Resolving Uncertainty in Model-based RL

titleGo-Explore: a New Approach for Hard-Exploration Problems

Institute of Mathematical Statistics

journaltitleThe Annals of Mathematical Statistics
titleOn a measure of the information provided by an experiment

journaltitleAdvances in neural information processing systems
titleThe im algorithm: a variational approach to information maximization

journaltitlearXiv preprint arXiv:1903.05480
titleVariational bayesian optimal experimental design

journaltitlearXiv preprint arXiv:1801.04062
titleMine: mutual information neural estimation

booktitleInternational Conference on Machine Learning
titleOn variational bounds of mutual information

journaltitleJournal of artificial intelligence research
titlePopular ensemble methods: An empirical study

journaltitlearXiv preprint arXiv:2002.12636
titleReinforcement learning through active inference

journaltitleNeural networks
titleBayesian approach for neural networks—review and case studies

journaltitlearXiv preprint arXiv:2007.06823
titleHands-on Bayesian Neural Networks–a Tutorial for Deep Learning Users

journaltitleIEEE transactions on information theory
titleLautum information

booktitleInternational conference on machine learning
titleSoft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

titleA Survey of Exploration Methods in Reinforcement Learning

journaltitleBiological cybernetics
titleAction and behavior: a free-energy formulation

Taylor & Francis

journaltitleCognitive neuroscience
titleActive inference and epistemic value

journaltitlearXiv preprint arXiv:1606.01540
titleOpenai gym

titledm_control: Software and Tasks for Continuous Control

journaltitleIEEE Transactions on Big Data
titleBillion-scale similarity search with GPUs

journaltitlearXiv preprint arXiv:1608.04471
titleStein variational gradient descent: A general purpose bayesian inference algorithm

journaltitlearXiv preprint arXiv:1412.6980
titleAdam: A method for stochastic optimization