Marco Gaido†, Sara Papi†‡, Matteo Negri†, Marco Turchi⋆∗

# Joint Speech Translation and Named Entity Recognition

###### Abstract

Modern automatic translation systems aim at supporting the users by providing
contextual knowledge. In this framework, a critical task is the output
enrichment with information regarding the mentioned entities. This is
currently achieved by processing the generated translations with named entity
recognition (NER) tools and retrieving their description from knowledge bases.
In light of the recent promising results shown by direct speech translation
(ST) models and the known weaknesses of cascades (error propagation and
additional latency), in this paper we propose multitask models that jointly
perform ST and NER, and compare them with a cascade baseline. Experimental
results on three language pairs (en-es/fr/it) show that our models
significantly outperform the cascade on the NER task (by 0.4-1.0 F1), without
degradation in terms of translation quality, and with the same computational
efficiency of a plain direct ST model.

Index Terms: augmented translation, speech translation, named entity
recognition, direct, multi-task

##  1 Introduction

Drawing inspiration from augmented reality, where real-world vision is
complemented with overlaid relevant information, ``augmented translation'' [1]
is an emerging research line aimed to enrich automatically-generated
translations with semantic information by highlighting named entities (NEs)
and key concepts (the focus of this work) and eventually linking them to
external knowledge bases (an aspect we do not cover here). On one side, this
can ease, speed up, and improve the generation of fluent and high-quality
translations by professional translators and post-editors; on the other, it
provides end users with additional information that may be needed to fully
understand a sentence, especially in highly specialized
domains.111https://intelligent-information.blog/en/augmented-translation-puts-
translators-back-in-the-center/

Current solutions rely on a cascade architecture comprising a text-to-text
machine translation (MT) system whose output is fed to a NE recognition (NER)
model [2]. No work has instead explored its application to speech-to-text
translation (ST), and the possibility of jointly performing the ST and NER
tasks with a single model, despite positive signals from related fields.
Indeed, in the task of NER from speech, the traditional cascade approach –
composed of an automatic speech recognition (ASR) system followed by an NER
model – has been recently challenged by the competitiveness of direct models
that perform the two tasks jointly [3, 4, 5, 6]. Similarly, in MT, multitask
models that jointly perform MT and NER have been shown to improve NE accuracy
without degrading translation quality [7].

In light of this and the competitive results of direct ST models [8, 9]
compared to the conventional ASR+MT pipeline [10, 11], in this paper we
address two research questions: (1) Is the current cascade of an ST system
(either direct ST or ASR+MT) followed by an NER tool better than performing
the two tasks with a single model? (2) What are the effects on NE accuracy and
translation quality of using a single multi-task model?

To answer these questions, we explore different methods to jointly perform ST
and NER.222Code available at: https://github.com/hlt-mt/FBK-fairseq. Our
experiments on three language pairs show that joint models significantly
outperform the ST+NER cascade by 0.4-1.0 F1 in the NER task while being on par
in terms of translation quality. Such improvement is achieved without
introducing any computational overhead with respect to a plain ST model,
making our solution remarkably more efficient than the cascade approach. This
is directly reflected in the computational-aware latency in simultaneous ST
scenarios, where our best model jointly performs ST and NER with the same
latency (and quality) as an ST-only model.

##  2 Joint NER and ST

The easiest way to extract the NEs from a translation consists in applying an
NER model on the output of the ST model. Henceforth, we refer to this approach
as cascade, and we consider it as a baseline for comparison against our
systems that jointly perform the two tasks with a single model. Our solutions
– inline, and parallel – are described below:

Figure 1:  Architecture of the inline solution. The additional tokens
generated in the output are highlighted in green, and are passed to the
decoder as all the other previous output tokens.

Inline (Fig. 1). The vocabulary of the direct ST model is extended with tags
that represent the start (e.g., <LOC>) and end (e.g., </LOC>) of the NE
categories to be recognized which, in our case, are 18,333The categories are
those defined in the OntoNotes annotation [12]. for a total of 36 entries.
These tags are treated as all other tokens (subwords): they are predicted in
the output sequence, and – together with the other tokens – fed to the decoder
as previous output tokens, informing it about the NE categories. This solution
does not require architectural changes to the ST model but introduces
additional overhead, especially at inference time, as the higher number of
tokens to generate (due to the additional start/end NE tags) leads to an
increase in the number of forward passes on the autoregressive decoder.

Parallel (Fig. 2). At each time step, two linear layers process in parallel
the output of the last decoder layer: one maps the vectors to the vocabulary
space to predict the next token as in standard ST models; the other maps the
same vectors to the NE-category space to predict the NE category to which the
token belongs, if any, or O (i.e. OTHER), if the token is not part of a NE.
Although the second linear layer introduces additional parameters to train,
its computational cost is negligible compared to that of the whole decoder.
Moreover, this solution avoids the supplementary decoder forward passes
required by the inline method. However, the potential drawback in comparison
with the inline solution is that it cannot exploit information about the NE
categories predicted for the previously generated tokens during translation.
As we posit that this lack of information may cause performance degradation,
we propose a variant of this method in which the embeddings of the previous
output tokens are summed with learned embeddings of their corresponding NE
categories.444The beginning-of-sentence (bos) token is considered of O
category. This change requires only 19 additional embeddings to learn (one for
each NE category, plus O) – a negligible number compared to the target
vocabulary size – and a sum, hence producing no significant computational
overhead. We refer to this variant as Parallel + NE emb.

##  3 Experimental Settings

Models. All our ST models are fed with 80 features extracted from the audio
every 10ms with sample windows of 25ms. These sequences of features are
processed by two 1D convolutional layers that reduce the sequence length by a
factor of 4, before passing them to a 12-layers Conformer encoder [13], and a
6-layer autoregressive Transformer decoder [14]. We use 512 features with 1024
hidden neurons in the FFN for both the encoder and decoder. The target
vocabulary is created with 8,000 BPE [15] merge rules. As a result, our models
have 116M parameters that we optimize with label-smoothed cross-entropy loss
[16] (0.1 smoothing factor) and an auxiliary CTC [17] loss on the output of
8th encoder layer with the transcript as the target to improve model
convergence [18]. Moreover, we adopt CTC compression [19] to reduce the input
dimension and speed up both training and inference. As optimizer, we use Adam
[20], and the learning rate is initially increased for 20k steps up to 0.005
and then it decreases with the inverse squared root policy. We train on 4 K80
GPUs with 10k tokens per mini-batch and 8 as update frequency. The training
stops after 5 epochs without loss decrease on the validation set, and average
5 checkpoints around the best. At inference time, we decode using beam search
with 5 as beam size. The ASR model of our ASR+MT+NER pipelines is trained on
the same data and with the same method described for the ST models. We rely on
a multilingual BERT-based
model,555http://docs.deeppavlov.ai/en/master/features/models/bert.html openly-
available in DeepPavlov [21], as NER system and, on the 1.3B-parameters
distilled NLLB [22] as MT model.

Data and Evaluation Metrics. All models are trained on MuST-C [23] and
Europarl-ST [24]. To train the joint NER and ST models, we automatically
annotated the NEs on the target translations with the same NER tool used in
our cascade approach, obtaining parallel training data with speech and the
corresponding annotated translations without any manual intervention.
Translation quality is evaluated with
SacreBLEU666case:mixed$|$eff:no$|$tok:13a$|$smooth:exp$|$version:2.0.0 [25] on
the Europarl-ST test set. Regarding NEs, instead, we measure three aspects on
NEuRoparl-ST benchmark [26]: 1) the generation of the correct translation, 2)
the recognition (or identification) of the NEs in the generated text, and 3)
the classification with the correct NE category. First, we use NE accuracy
(case-insensitive, for the sake of comparison with previous work) to assess
the ability to translate NEs. Second, we compute F1 to measure the ability in
recognizing NEs, although F1 is also influenced by the NE translation quality,
as it is computed by considering as correct only those NEs that are accurately
translated and identified, but disregarding their category classification. As
such, NEs that are poorly translated and recognized by a model penalize both
recall and precision. The strict F1 definition mirrors the users' perception:
in augmented ST, while unrecognized NEs are only a lack of help to the users,
recognized but incorrect NEs are more harmful as they would distract them with
unrelated and potentially misleading content. Lastly, we use classification
accuracy to measure the percentage of NEs assigned to the correct category.

Figure 2:  Architecture of the parallel solution. The introduced linear layer (in green) is processed token-by-token in parallel with the other linear layer. In the \+ NE emb. variant (yellow dotted area), the previous tags are converted into embeddings that are summed to those of the corresponding previous tokens. Model | en-es | en-fr | en-it
---|---|---|---
BLEU | NE Acc | F1 | Cat. Acc. | BLEU | NE Acc | F1 | Cat. Acc. | BLEU | NE Acc | F1 | Cat. Acc.
Prev. work [26] | 37.7 | 71.4 | - | - | 30.1 | 67.3 | - | - | 26.0 | 67.3 | - | -
Cascade (ASR+MT+NER) | 37.3 | 71.0 | 47.4 | 89.9 | 37.6 | 68.8 | 44.6 | 90.2 | 26.2 | 64.8 | 42.0 | 87.5
Cascade (ST+NER) | 37.9 | 71.9 | 49.1 | 89.8 | 36.2 | 69.2 | 44.8 | 90.2 | 28.3 | 66.5 | 44.5 | 88.8
Inline | 37.9 | 72.2 | 49.5†‡ | 90.1 | 36.3 | 69.6 | 45.6†‡ | 90.2 | 28.3 | 66.9 | 45.5†‡ | 89.4
Parallel | 38.1 | 71.9 | 48.1 | 89.5 | 36.1 | 69.0 | 44.5 | 90.6 | 28.4 | 67.5 | 43.9 | 89.1
\+ NE emb. | 38.0 | 72.1 | 49.5†‡ | 89.9 | 36.1 | 69.3 |  45.5†‡ | 90.4 | 28.2 | 67.3 |  45.4†‡ | 89.1
Table 2:  BLEU ($\uparrow$), case-insensitive NE accuracy ($\uparrow$), F1
($\uparrow$), and category classification accuracy (Cat. Acc., $\uparrow$) of
previous work, our cascades (ASR+MT+NER and ST+NER) and the proposed joint
ST&NER models. All results are the average of three runs. † indicates
statistically significant improvements over ST+NER, and ‡ over parallel. A
result is considered statistically significant if we can reject with 95%
confidence the null hypothesis that the considered mean is not higher than the
mean of the baseline with Student's t-test [27].

##  4 Results

Translation Quality (Overall and at NE Level). First, to ensure the soundness
of our experimental settings and, in turn, of our analysis, we compare our
base direct ST model with recent works on Europarl-ST (Table 1). As shown by
the results, our systems outperform, to the best of our knowledge, all
recently published scores on the same benchmark. This confirms the strength of
our models and the reliability of our results.

Model | en-es | en-fr | en-it
---|---|---|---
ASR+MT [24] | 28.0 | 23.4 | -
NPDA-kNN-ST [28] | 29.0 | 27.7 | 20.5
STR+KD [29] | - | 29.3 | -
NEuRoparl-ST [26] | 37.7 | 30.1 | 26.0
Triangle Multi [30] | 37.4 | 35.4 | 28.2
Ours | 37.9 | 36.2 | 28.3
Table 1: BLEU ($\uparrow$) of our direct ST system in comparison with previous
ST works on Europarl-ST.

In Table 2, instead, we compare our cascade ST+NER and ASR+MT+NER baselines,
the joint ST&NER inline and parallel methods, and the only previous work [26]
that reports scores (NE accuracy) on the NEuRoparl-ST benchmark (using a
direct ST system trained on a large amount of data). We can notice that, even
though trained on fewer data, the direct ST models of our cascade ST+NER
baselines compare favourably with the previous work not only in terms of
translation quality (BLEU), but also in NE accuracy. In particular, the NE
accuracy of our cascade ST+NER baseline is superior on average on the three
language pairs, as the gains in en-es (+0.5) and en-fr (+1.9) are only
partially balanced by the small drop in en-it (-0.8). In addition, the full
cascade model (ASR+MT+NER) – despite leveraging NLLB, which is trained on a
large amount of data, and the good ASR performance (12.5 WER, slightly better
than the ASR trained on thousands of hours presented in [26]) – is inferior to
the ST+NER baseline on all metrics, with the only exception of the en-fr BLEU.
This further demonstrates the strength of our cascade ST+NER baseline.

Focusing on the comparison of the cascade and joint methods in performing the
ST and NER tasks, we notice that the performance of both inline and parallel
models are close in terms of translation quality, both generic (BLEU) and
specific to NEs (NE accuracy), compared to the ST+NER baseline. The small
differences among the scores of the various methods (up to 0.2 BLEU and up to
0.6 NE accuracy) are not consistent across language directions and are never
statistically significant, thus being ascribable to fluctuations due to the
inherent randomness of neural methods. We can conclude that the additional NER
task does not bring any improvement to ST in terms of NE translation (in
contrast with previous findings for MT [7]) but also does not degrade
translation quality, as it could have happened since part of the model is
dedicated to the additional task.

NE Recognition. When we consider the F1 metric, instead, the results highlight
the differences between the various approaches. Our joint NER and ST
approaches beat the cascade by a statistically significant margin on all
language pairs (0.4-1.0 F1). This is surprising if we consider that the
training data of the joint methods was generated with the NER system of the
cascade approach, and highlights the strength of direct multitask systems.
Among the joint solutions, the inline and parallel + NE emb. significantly
outperform the parallel method, proving the importance of feeding the decoder
with information about the NE category predicted for the previously generated
tokens. The difference between inline and parallel + NE emb., however, is very
small (0.1, if any) and not statistically significant. These two methods can
therefore be considered on par.

Figure 3:  Confusion matrix over the 15 NE categories with at least one NE
correctly translated and recognized for the parallel + NE emb. system on en-
es. On the y-axis, there are the true labels, while on the x-axis the
predicted labels. The numbers are percentages computed on the y-axis.

NE Classification. Lastly, all systems (joint and cascade) show a good ability
in NE category classification. The accuracy differences range between 0.6 and
0.3, are not coherent across language pairs and are never statistically
significant. Not only their overall performance is on par, but also their
confusion matrices over the NE categories are basically the same on all
language pairs. As an example, Fig. 3 reports the confusion matrix of the
parallel + NE emb. model for en-es. The classification accuracy is high
(87.95-100%) for all categories but three: facilities (FAC), events (EVENT),
and names of laws (LAW). FAC and EVENT are very rare (19 and 9 occurrences in
the test set), while LAW is more frequent (141 occurrences), thus representing
the main source of classification errors. The root of this difficulty may lay
in the nature of law names, which have high variability, are long, and
frequent only in specific domains. At last, another common source of errors is
classifying GPE as location names, which is unsurprising as their
categorization highly depends on the context in which they occur (e.g. Europe
as a continent is a LOC, but in politics it can be a GPE).

##  5 Efficiency in Simultaneous ST

One known advantage of direct systems over cascade ones is their lower overall
computational cost since they need a forward pass on only one model instead of
two. For this reason, in applications where the computational cost is
particularly critical, such as in simultaneous ST (SimulST), where it directly
affects the output latency, direct ST systems obtain a significantly better
latency-quality trade-off than ASR+MT solutions [11, 31].

However, in our case of ST and NER, the computational cost is not only
determined by the choice of a cascade or a full direct system, but also by
which of the two joint solutions is used. Indeed, the number of decoding steps
(i.e. forward passes on the autoregressive decoder) required by the inline and
parallel systems is different: the former method has to predict the start and
end NE tags, requiring on average 7% more decoding steps on the Europarl-ST
test set compared to a plain ST model and to the parallel systems, which do
not introduce additional decoding steps. For this reason, we conclude our work
by comparing the two best models (inline and parallel + NE emb) in the
simultaneous setting using the popular wait-k [32] policy.

The wait-k policy consists in initially waiting for a predefined number of
words ($k$) before starting to alternate between WRITE (emit a word) and READ
(wait for more input audio) actions. Since the source is speech, the
information about the number of words is not already present in the input,
therefore a word detection strategy is applied to determine how many words
have been pronounced at each time step. Here, we use an adaptive word
detection strategy [33, 34] that estimates the number of words in an audio
segment by counting them in the transcripts predicted by the CTC module
trained on the encoded audio. The choice of this method is motivated by its
favorable performance compared to other word detection strategies [35]. The
wait-k policy is directly applied to offline-trained models without the need
for any adaptation for the task, as this approach has been demonstrated to be
competitive with the one adopting models specifically trained to work in
simultaneous [36].

$2{,}200$$2{,}400$$2{,}600$$2{,}800$$3{,}000$$3{,}200$$3{,}400$$18$$20$$22$$24$$26$$28$$30$$32$LAAL
(ms)BLEUInlineParallel + NE emb.Only ST

$2{,}200$$2{,}400$$2{,}600$$2{,}800$$3{,}000$$3{,}200$$3{,}400$$0.36$$0.38$$0.4$$0.42$$0.44$$0.46$LAAL
(ms)F1

Figure 4:  BLEU- and F1-LAAL curves for en-es of the inline and parallel + NE
emb solutions (we also include an ST-only system as reference). Each point
corresponds to a different value of $k=\\{1,2,3\\}$ and is the average over
three models.

Evaluating the performance in simultaneous allows us to estimate the overhead
introduced by the additional decoding steps of the inline model compared to
the parallel + NE emb. one. In Fig. 4, we report the BLEU- and F1-latency
curves computed on the outputs obtained by running the SimulEval tool [37] on
the two joint NER&ST models for en-es (for the sake of brevity, we do not
report the curves for en-fr and en-it that show the same trends). We also
report the BLEU-latency curve of the direct ST-only model as a reference,
while we do not show the cascade ST+NER as the computational cost (and hence,
latency) is significantly higher. Latency is measured through computational-
aware length-adaptive average lagging (LAAL) [38]. The $k$ value of the wait-k
policy is varied from 1 to 3 ($k=\\{1,2,3\\}$), in order to reach different
latency regimes.

The curves show that the parallel + NE emb model has the same latency and
quality of an ST-only model, despite the additional NER task to perform. The
inline solution, instead, has similar quality but features a (slightly)
increased latency, because of its higher computational cost. However, since
the computational cost only accounts for a fraction of the latency ($\sim$53%
of the computational-aware LAAL is due to the wait time of the wait-k policy),
and the computational difference is not large ($\sim$5%), the gap between the
two models is limited.

All in all, we can conclude that the inline model introduces a computational
overhead that depends on the number of NEs detected in an utterance. On our
test set, with 1,267 sentences, 30.6K words, and 1,638 NEs, we estimated as 5%
its computational overhead in time compared to a base direct ST model and to
our parallel + NE emb. solution. In light of the similar quality of inline and
parallel + NE emb. systems, this difference – which may be larger in domains
where NEs are more frequent, as news or molecular biology [39] – makes the
parallel + NE emb. method our best solution overall.

##  6 Conclusions

We presented the first multitask models jointly performing speech translation
and named entity recognition. First, we showed the importance of properly
feeding information about the previously predicted NE tags, as done in the
inline and parallel + NE emb. models. Second, and most importantly, we showed
that our joint solutions consistently outperform a cascade system on the NER
task (by 0.4-1.0 F1), while being on par in terms of translation quality.
Lastly, we evaluated the computational efficiency of our methods and
demonstrated that the parallel + NE emb. system, which does not introduce
noticeable overhead with respect to a plain ST model, is more efficient than
the inline method, besides being on par in terms of translation and NER
quality. As such, it represents the most attractive solution to jointly
perform ST and NER, especially in the simultaneous scenario where its
computational-aware latency is the same as a single model performing the ST
task only.

##  7 Acknowledgment

This work is part of the project Smarter Interpreting (https://smarter-
interpreting.eu/) financed by CDTI Neotec funds. We acknowledge the support of
the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR
program funded by the NextGenerationEU.

## References

  * [1] A. Lommel, ``Augmented translation: A new approach to combining human and machine capabilities,'' in _Proc. of the 13th AMTA_ , Boston, MA, 2018, pp. 5–12.
  * [2] A. D. De Palma, ``Augmented translation powers up language services,'' _Common Sense Advisory Research_ , 2017.
  * [3] S. Ghannay, A. Caubrière, Y. Estève, N. Camelin, E. Simonnet, A. Laurent, and E. Morin, ``End-to-end named entity and semantic concept extraction from speech,'' in _Proc. of 2018 IEEE SLT Workshop_ , 2018, pp. 692–699.
  * [4] H. Yadav, S. Ghosh, Y. Yu, and R. R. Shah, ``End-to-End Named Entity Recognition from English Speech,'' in _Proc. of Interspeech 2020_ , 2020, pp. 4268–4272.
  * [5] A. Caubrière, S. Rosset, Y. Estève, A. Laurent, and E. Morin, ``Where are we in Named Entity Recognition from Speech?'' in _Proc. of the 12th LREC_ , Marseille, France, 2020, pp. 4514–4520.
  * [6] B. Chen, G. Xu, X. Wang, P. Xie, M. Zhang, and F. Huang, ``Aishell-ner: Named entity recognition from chinese speech,'' in _Proc. of ICASSP 2022_ , 2022, pp. 8352–8356.
  * [7] S. Xie, Y. Xia, L. Wu, Y. Huang, Y. Fan, and T. Qin, ``End-to-end entity-aware neural machine translation,'' _Mach. Learn._ , vol. 111, no. 3, p. 1181–1203, mar 2022.
  * [8] A. Bérard, O. Pietquin, C. Servan, and L. Besacier, ``Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation,'' in _NIPS Workshop on end-to-end learning for speech and audio processing_ , Barcelona, Spain, 2016.
  * [9] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and Z. Chen, ``Sequence-to-Sequence Models Can Directly Translate Foreign Speech,'' in _Proc. of Interspeech 2017_ , Stockholm, Sweden, Aug. 2017, pp. 2625–2629.
  * [10] L. Bentivogli, M. Cettolo, M. Gaido, A. Karakanta, A. Martinelli, M. Negri, and M. Turchi, ``Cascade versus direct speech translation: Do the differences still make a difference?'' in _Proc. of the 59th ACL_ , Online, 2021, pp. 2873–2887.
  * [11] A. Anastasopoulos _et al._ , ``FINDINGS OF THE IWSLT 2021 EVALUATION CAMPAIGN,'' in _Proc. of the 18th IWSLT 2021_ , Bangkok, Thailand (online), 2021, pp. 1–29.
  * [12] R. Weischedel _et al._ , ``OntoNotes Release 5.0,'' 2012. [Online]. Available: https://catalog.ldc.upenn.edu/docs/LDC2013T19/OntoNotes-Release-5.0.pdf
  * [13] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, ``Conformer: Convolution-augmented Transformer for Speech Recognition,'' in _Proc. of Interspeech 2020_ , 2020, pp. 5036–5040.
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is All You Need,'' in _Proc. of NIPS 30_ , Long Beach, California, 2017, pp. 5998–6008.
  * [15] R. Sennrich, B. Haddow, and A. Birch, ``Neural Machine Translation of Rare Words with Subword Units,'' in _Proc. of the 54th ACL_ , Berlin, Germany, Aug. 2016, pp. 1715–1725.
  * [16] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ``Rethinking the Inception Architecture for Computer Vision,'' in _Proc. of 2016 IEEE CVPR_ , Las Vegas, Nevada, United States, 2016, pp. 2818–2826.
  * [17] A. Graves, S. Fernández, F. J. Gomez, and J. Schmidhuber, ``Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks,'' in _Proc. of the 23rd ICML_ , Pittsburgh, Pennsylvania, 2006, pp. 369–376.
  * [18] P. Bahar, T. Bieschke, and H. Ney, ``A Comparative Study on End-to-End Speech to Text Translation,'' in _Proc. of 2019 IEEE ASRU_ , 2019, pp. 792–799.
  * [19] M. Gaido, M. Cettolo, M. Negri, and M. Turchi, ``CTC-based compression for direct speech translation,'' in _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , Online, Apr. 2021, pp. 690–696.
  * [20] D. P. Kingma and J. Ba, ``Adam: A method for stochastic optimization,'' in _Proc. of the 3rd ICLR_ , San Diego, USA, 2015.
  * [21] M. Burtsev _et al._ , ``DeepPavlov: Open-Source Library for Dialogue Systems,'' in _Proc. of ACL 2018, System Demonstrations_ , Melbourne, Australia, 2018, pp. 122–127.
  * [22] M. R. Costa-jussà, J. Cross, O. Çelebi _et al._ , ``No language left behind: Scaling human-centered machine translation,'' _arXiv preprint arXiv:2207.04672_ , 2022.
  * [23] R. Cattoni, M. A. Di Gangi, L. Bentivogli, M. Negri, and M. Turchi, ``Must-c: A multilingual corpus for end-to-end speech translation,'' _Computer Speech & Language_, vol. 66, pp. 101–155, 2021.
  * [24] J. Iranzo-Sánchez, J. A. Silvestre-Cerdà, J. Jorge, N. Roselló, A. Giménez, A. Sanchis, J. Civera, and A. Juan, ``Europarl-st: A multilingual corpus for speech translation of parliamentary debates,'' in _Proc. of ICASSP 2020_ , 2020, pp. 8229–8233.
  * [25] M. Post, ``A Call for Clarity in Reporting BLEU Scores,'' in _Proc. of the 3rd WMT_ , Belgium, Brussels, 2018, pp. 186–191.
  * [26] M. Gaido, S. Rodríguez, M. Negri, L. Bentivogli, and M. Turchi, ``Is ``moby dick'' a Whale or a Bird? Named Entities and Terminology in Speech Translation,'' in _Proc. of the 2021 EMNLP_ , Punta Cana, Dominican Republic, 2021, pp. 1707–1716.
  * [27] Student, ``The probable error of a mean,'' _Biometrika_ , vol. 6, no. 1, pp. 1–25, 1908.
  * [28] Y. Du, W. Wang, Z. Zhang, B. Chen, T. Xu, J. Xie, and E. Chen, ``Non-parametric domain adaptation for end-to-end speech translation,'' in _Proceedings of the 2022 EMNLP_ , Abu Dhabi, United Arab Emirates, Dec. 2022, pp. 306–320.
  * [29] T. K. Lam, S. Schamoni, and S. Riezler, ``Sample, translate, recombine: Leveraging audio alignments for data augmentation in end-to-end speech translation,'' in _Proc. of the 60th ACL_ , Dublin, Ireland, May 2022, pp. 245–254.
  * [30] M. Gaido, M. Negri, and M. Turchi, ``Who Are We Talking About? Handling Person Names in Speech Translation,'' in _Proc. of the 19th IWSLT_ , Dublin, Ireland, May 2022, pp. 62–73.
  * [31] A. Anastasopoulos _et al._ , ``Findings of the IWSLT 2022 evaluation campaign,'' in _Proc. of IWSLT 2022_ , Dublin, Ireland, May 2022, pp. 98–157.
  * [32] X. Ma, J. Pino, and P. Koehn, ``SimulMT to SimulST: Adapting simultaneous text translation to end-to-end simultaneous speech translation,'' in _Proc. of the 1st AACL-IJCNLP_ , Suzhou, China, 2020, pp. 582–587.
  * [33] Y. Ren, J. Liu, X. Tan, C. Zhang, T. Qin, Z. Zhao, and T.-Y. Liu, ``SimulSpeech: End-to-end simultaneous speech to text translation,'' in _Proc. of the 58th ACL_ , Online, Jul. 2020, pp. 3787–3796.
  * [34] X. Zeng, L. Li, and Q. Liu, ``RealTranS: End-to-end simultaneous speech translation with convolutional weighted-shrinking transformer,'' in _Findings of ACL-IJCNLP 2021_ , Online, Aug. 2021, pp. 2461–2474.
  * [35] B. Zheng, K. Liu, R. Zheng, M. Ma, H. Liu, and L. Huang, ``Simultaneous translation policies: From fixed to adaptive,'' in _Proc. of the 58th ACL_ , Online, Jul. 2020, pp. 2847–2853.
  * [36] S. Papi, M. Gaido, M. Negri, and M. Turchi, ``Does simultaneous speech translation need simultaneous models?'' in _Findings of EMNLP 2022_ , Abu Dhabi, United Arab Emirates, Dec. 2022.
  * [37] X. Ma, M. J. Dousti, C. Wang, J. Gu, and J. Pino, ``SIMULEVAL: An evaluation toolkit for simultaneous translation,'' in _Proc. of EMNLP 2020_ , Online, Oct. 2020.
  * [38] S. Papi, M. Gaido, M. Negri, and M. Turchi, ``Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation,'' in _Proc. of the 3rd Workshop on Automatic Simultaneous Translation_ , Online, 2022, pp. 12–17.
  * [39] C. Nobata, N. Collier, and J. Tsujii, ``Comparison between tagged corpora for the named entity task,'' in _The Workshop on Comparing Corpora_ , Hong Kong, China, 2000, pp. 20–27.


# Using the Projected Belief Network at High Dimensions

Paul M. Baggenstoss  Fraunhofer FKIE, Fraunhoferstrasse 20
53343 Wachtberg, Germany
Email<EMAIL_ADDRESS>Frank Kurth  Fraunhofer FKIE,
Fraunhoferstrasse 20
53343 Wachtberg, Germany
Email<EMAIL_ADDRESS>
###### Abstract

The projected belief network (PBN) is a layered generative network (LGN) with
tractable likelihood function, and is based on a feed-forward neural network
(FFNN). There are two versions of the PBN: stochastic and deterministic
(D-PBN), and each has theoretical advantages over other LGNs. However,
implementation of the PBN requires an iterative algorithm that includes the
inversion of a symmetric matrix of size $M\times M$ in each layer, where $M$
is the layer output dimension. This, and the fact that the network must be
always dimension-reducing in each layer, can limit the types of problems where
the PBN can be applied. In this paper, we describe techniques to avoid or
mitigate these restrictions and use the PBN effectively at high dimension. We
apply the discriminatively aligned PBN (PBN-DA) to classifying and auto-
encoding high-dimensional spectrograms of acoustic events. We also present the
discriminatively aligned D-PBN for the first time.

##  I Introduction

###  I-A Motivation: Advantages of PBN

The projected belief network (PBN) is a layered generative network (LGN) with
tractable likelihood function, and is based on a feed-forward neural network
(FFNN). There are two versions of the PBN: stochastic and deterministic, and
each has theoretical advantages over other layered generative networks.

It has recently been shown that the information at the output of a dimension-
reducing transformation can be maximized using probability density function
projection (PDF projection) [1]. PDF projection estimates the distribution of
the input data, simultaneously with a dimension-reducing transformation that
extracts the latent variables [2, 3]. The method is more general than other
methods of non-linear dimension reduction (NLDR) [4, 5, 6, 7, 8]. Implementing
PDF projection in a neural network architecture is called projected belief
network (PBN) [9, 10, 11, 12]. As a result, the PBN is the most direct and
general way to apply NLDR in a neural network architecture.

The LF of other widely used LGNs can only be obtained by intractable
integration (marginalizing) over the hidden variables. They must rely on a
surrogate cost function in order to approximate LF training. Examples are
contrastive divergence (CD) to train restricted Boltzmann machines [13, 14],
and Kullback Leibler divergence to train variational auto-encoder (VAE) [15],
or an adversarial discriminative network to train generative adversarial
networks (GAN) [16]. On the other hand, since the PBN generates data by
manifold sampling, it posesses a tractable likelihood function (LF) that
allows direct gradient-based training.

The deterministic PBN (D-PBN) can be used as an auto-encoder [11, 9] and has
theoretical advantage over conventional auto-encoders. While other auto-
encoders use an empirical reconstruction network, the deterministic PBN
reconstructs input data by backing up (back-projecting) through the same feed-
forward neural network (FFNN) that was used to extract the features. In each
layer, it selects the conditional mean estimate of the layer input based on a
maximum entropy prior, a type of optimal estimator.

Another advantage of the PBN and D-PBN is that they are based on a FFNN.
Therefore, a single FFNN can be simultaneously a generative and a
discriminative network [12]. This offers the most direct way to combine the
advantages of both network types. A number of variations of this concept have
been proposed. The PBN or D-PBN cost functions can be used as a regularization
for discriminative neural networks [12], or the opposite: discriminative cost
function can be used to “align” a PBN to decision boundaries to create better-
performing generative models [17]. We will use this approach in this paper to
test the PBN and D-PBN at high dimensions.

###  I-B Discriminative Alignment of PBN

It was shown in [17] that a generative classifier (a PBN) can compete with
state of the art discriminative classifiers. This seems to contradict the
widely-held belief that the generative task is much harder, and unnecessary
for classifying [18]. However, generative models are useful in their own
right, and can be applied in some tasks where discriminative classifiers
cannot [19]. A generative classifier than can perform as well as a
discriminative classifier, is highly desireable.

Sand box analogy. Discriminative alignment can be understood in terms of the
conceptual image of a sand box enclosed by a wooden frame. The frame can be
thought of as the discriminative task, separating the probability mass (the
sand) from the other classes at the decision boundaries. The generative model
can be thought of as heaping the sand at the places where data is more likely.
The two tasks can be achieved simultaneously using discriminative alignment of
a PBN.

Despite the benefits it offers, the PBN is bound by restrictions that also
affect the discriminative model that it may share : (a) dimension-reducing
layers, (b) no max-pooling in convolutional layers, (c) no dropout
regularization, and (d) no batch normalization. On the positive side, L2
regularization can be used, and the generative cost function can be seen as a
form of regularization in itself [12], and one can avoid the restriction on
dimension-reducing layers using the method in Section III-E.

###  I-C Computational Challenge of PBN

Despite these theoretical advantages, implementation of the PBN requires in
each layer the inversion of a $M\times M$ symmetric matrix where $M$ is the
layer output dimension. In convolutional layers, $M$ can be extremely large
because it is the product of the number of kernels with the size of the
feature maps. Depending on the type of PBN layer (i.e depending in the type of
maximum entropy prior distribution), this matrix may need to be inverted for
each sample in a mini-batch.

###  I-D Paper Organization and Contributions

We begin by a mathematical review of PBN and D-PBN in Section II, explaining
the computational challenge of PBN at high dimensions, then suggest new
approaches to applying PBN at high dimensions in Section III. Experimental
results are provided in Section IV including a novel variation of PBN, called
D-PBN-DA in Section IV-H.

##  II The PBN: Review

###  II-A PDF Projection

A PBN is just a layer-wise application of PDF projection. In probability
density function (PDF) projection, one defines an $N\times 1$ input vector
${\bf x}$, and a transformation ${\bf z}=T({\bf x})$ producing the $M\times 1$
output vector ${\bf z}$, where $M<N$. Suppose we have data ${\bf x}$ drawn
from some distribution $p({\bf x})$ with support on
$\mathbb{X}\subseteq\mathbb{R}^{N}$, that we’d like to estimate. Although
$p({\bf x})$ is unknown, we know $g({\bf z})$, which is an estimate of the
feature distribution of ${\bf z}$, where ${\bf x}$ is drawn from $p({\bf x})$
and ${\bf z}=T({\bf x})$, and has support on
$\mathbb{Z}\subseteq\mathbb{R}^{M}$. To estimate $p({\bf x})$ based on knowing
$g({\bf z})$, we first define a reference (or prior) distribution $p_{0}({\bf
x})$ for which we know $p_{0}({\bf z})$, the mapping of $p_{0}({\bf x})$
through $T({\bf x})$. Then, the PDF projection theorem [2] states that the
function

$G({\bf x};T,p_{0},g)=\frac{p_{0}({\bf x})}{p_{0}({\bf z})}\;g({\bf z}),$ (1)

where ${\bf z}=T({\bf x})$, is a PDF on $\mathbb{X}$ (it integrates to 1) and
is among the class of PDFs which map to distribution $g({\bf z})$ through
$T({\bf x})$. Simply stated, $G({\bf x};T,p_{0},g)$ is an estimate of $p({\bf
x})$ based on prior distribution $p_{0}({\bf x})$ and feature PDF estimate
$g({\bf z})$.

The biggest challenge on PDF projection is the derivation of $p_{0}({\bf z})$.
When $p_{0}({\bf x})$ is a canonical reference distribution, the feature
distribution $p_{0}({\bf z})$ is known in closed form for many types of
transformations [20]. For an even broader class of reference distributions and
transformations, $p_{0}({\bf z})$ is not known, but the moment generating
function (MGF) is known exactly. In these cases, we can invert the MGF using
the saddle point approximation (see eq. (16) in [20]). We note that the term
“approximation” is misleading because the error can be neglected for all
practical purposes [17].

###  II-B The PBN

For complex transformations, such as a neural network, $p_{0}({\bf z})$ may be
difficult or impossible to derive, so we apply PDF projection recursively
(layer-wise). This results in a projected belief network (PBN) [21], where the
likelihood function (LF) $G({\bf x};T,p_{0},g)$ becomes a product of terms,
each term being the application of (1) to a layer (see eq. (1) in [12]).

###  II-C Finding the Saddle Point

Each term in the LF of a PBN requires computing the saddle point. Let ${\bf
x}\subseteq\mathbb{R}^{N}$ be the input of a layer, where $N$ is the
dimension. Let ${\bf z}\subseteq\mathbb{R}^{M}$ be the output of the linear
transformation given by ${\bf z}={\bf W}^{\prime}{\bf x}$. This is the
dimension-reducing linear transformation used in each layer of a neural
network before application of the activation function. Note that this equation
holds both for dense (fully connected) and convolutional layers. Although
convolutional layers are implemented by efficient convolution, there exists at
least theoretically an $N\times M$ matrix ${\bf W}$.

The saddle point $\hat{{\bf h}}$ is the $M\times 1$ vector ${\bf h}$ that
solves

${\bf W}^{\prime}\lambda\left(\mbox{\boldmath$\alpha$}_{0}+{\bf W}{\bf
h}\right)={\bf z},$ (2)

where $N\times 1$ constant vector $\mbox{\boldmath$\alpha$}_{0}$ and element-
wise activation function $\lambda(\;)$ depend on the chosen MaxEnt prior [9].

Equation (2) is solved by an iterative algorithm (Newton-Raphson) that
involves inverting the matrix ${\bf C}(\hat{{\bf h}})={\bf
W}^{\prime}\Lambda{\bf W}$, where $\Lambda$ is the $N\times N$ diagonal matrix
with diagonal
$\lambda^{\prime}(\alpha_{1}),\lambda^{\prime}(\alpha_{2}),\ldots\lambda^{\prime}(\alpha_{N})$,
where $\lambda^{\prime}(\alpha)$ is the value of the first derivative of
$\lambda\left(\alpha\right)$ with repect to $\alpha$, and
$\mbox{\boldmath$\alpha$}=\mbox{\boldmath$\alpha$}_{0}+{\bf W}\hat{{\bf h}}$.

###  II-D Using the Saddle Point

The deterministic PBN (D-PBN) reconstructs ${\bf x}$ from ${\bf z}$ and the
saddle point as follows [9]:

$\hat{{\bf x}}=\lambda\left(\mbox{\boldmath$\alpha$}_{0}+{\bf W}\hat{{\bf
h}}\right)$ (3)

A multi-layer D-PBN is created by propagating the estimate backward in the
network [9, 11]. The stochastic PBN uses the saddle point to compute terms in
the LF. Specifically, $p_{0}({\bf z})$ is written using equation (16) in [20],
where the saddle point is written $\hat{\mbox{\boldmath$\lambda$}}$ and
$p_{0}({\bf z})$ is written $p_{z}({\bf u})$.

##  III Dimension Mitigation

The computational challenges of using PBN at high dimensions, outlined in
Section I-C, can be mitigated using the following approaches.

###  III-A Seamless Streaming and Reconstruction

Large data samples that have a time dimension, such as audio recordings can be
broken into smaller segments and processed by a sliding window. There is an
implicit assumption of independence between segments here that can be
approximately true if the segments are large enough. As long as the sliding
window is wide enough to capture enough temporal information, the generative
model can provide meaningful and significant dimension reduction with high
information content. The collection of output features, i.e. the “stream” of
output features can be used to reconstruct the complete event, and further
feature dimension reduction can be done on the assembled stream so that
temporal information spanning larger time durations can be extracted
downstream.

For seamless reconstruction, the sliding windows must overlap by exactly the
right amount, so that the assembled output stream is no different than what
would have been created by the same processing on the entire event. Any
temporal processing, i.e. temporal convolution, must be done without edge
effects. For this reason, convolutions in convolutional network layers must be
done without zero-padding.

###  III-B Gaussian Layers

A very significant speed-up is afforded by Gaussian layers. In a Gaussian
layer, the Gaussian MaxEnt prior is assumed, and $\lambda(x)=x$ is the linear
activation, meaning that ${\bf C}(\hat{{\bf h}})={\bf C}={\bf W}^{\prime}{\bf
W}$ is independent of $\hat{{\bf h}}$ and $\mbox{\boldmath$\alpha$}_{0}={\bf
0}$, so (2) can be solved in closed form, i.e. $\hat{{\bf h}}=({\bf
W}^{\prime}{\bf W})^{-1}{\bf z},$ and needs to be done only once per mini-
batch. Note that for convolutional layers, all operations are done by
convolution, and matrix ${\bf W}$ never needs to be actually created.

Using Gaussian layers, however, assumes that the layer input has support on
$\mathbb{R}^{N}$, which means that activation functions should be avoided in
the layer that precedes a Gaussian layer to attain a good PDF estimate using
PDF projection. Using linear activations in the first few layers, and then
non-linear activations in middle and low-dimensional layers may be an
effective compromise.

###  III-C Gaussian Layer Groups (GLG)

For very large hidden variable dimensions, using a Gaussian layer may be
prohibitive - even allocating memory for ${\bf C}$ may be difficult. In these
cases, it is possible to use several Gaussian layers in a row, ending in a
manageable output dimension, then grouping the linear transformations so that
they are representable by a single linear transformation with low output
dimension $M$, and small ${\bf C}$. After the GLG, non-linear activation
functions can be used. As long as $M$ at the group output is large enough,
significant functional approximation power is afforded by the network.

###  III-D Dimension-Preserving Layers

If a layer input and output have the same dimension, it is a dimension-
preserving (1:1) transformation layer and computing the saddle point is not
required. It is only necessary to solve for the determinant of the Jacobian
matrix of the transformation as a whole.

###  III-E Expand-Contract Groups (ECG)

To afford even greater functional approximation power, it is possible to
expand the hidden variable dimension using a series of high-dimensional
network layers, then reduce the dimension back down to what it was at the
start. This group of layers, seen as a unit, can be analyzed as a dimension-
preserving (1:1) transformation, as if it was a single dimension-preserving
layer.

###  III-F Direct Saddle Point Estimation

One can find $\hat{{\bf h}}$, the solution to (2), by direct inversion using a
neural network by training on sample pairs ${\bf z}$, $\hat{{\bf h}}$. Error
in the result is removed by a few iterations of the Newton-Raphson algorithm.

##  IV Experimental Results

###  IV-A Experimental Goal: Discriminative Alignment of PBN

The goal of our first experiment was to show that discriminative alignment,
which was demonstrated previously on input data data dimensions of 196 and 900
[17], can be used on a different data set with much higher dimension.
Discriminative alignment is explained in Section I-B and further details are
given in Section IV-F.

###  IV-B Data and Feature Extraction

The Office Sounds database [22, 23] contains twenty-four signal classes
containing 102 samples of each class, a total of 2448 example sounds. Most of
the sounds are created by dropping common objects or operating office tools
such as scissors or staplers. All time-series are 16128 samples long (1/2
second in duration at 32000 Hz). We selected six of the twenty-four classes
that had the highest inter-class errors. These classes were “skit”, “sciss”,
“pret”, “pens”, “paper”, and “jing” and are seen on the top row of Figure 1.
There were 102 samples of each class, so 612 samples total. The emphasis is to
achieve very high correct classification performance on acoustic data with few
training samples. The 612 events were divided into training and testing sets
of 306 each. We conducted two-fold hold-out in which we trained on half of the
data and tested using the other half, then repeated the experiment with the
data switched. To extract features, we segmented each time-series sample into
size-384 windows with 2/3 overlap, producing 126 segments per time series. We
then took the FFT, magnitude-squared, then the log. The features were
therefore $126\times 193$, or dimension 24,318.

###  IV-C Network, Algorithm, and Hardware

The network consisted of seven layers:

  1. 1.

Convolutional with 6 ($9\times 10$) kernels, with ($3\times 3$) downsampling,
linear activation,

  2. 2.

Convolutional with 36 ($7\times 8$) kernels, with ($3\times 3$) downsampling,
linear activation,

  3. 3.

Convolutional with 96 ($8\times 7$) kernels, with ($2\times 3$) downsampling,
linear activation,

  4. 4.

Dense with 512 neurons, linear activation.

  5. 5.

Dense with 256 neurons, TG activation.

  6. 6.

Dense with 128 neurons, TG activation.

  7. 7.

Classifier layer, dense with 6 neurons.

The “TG” activation is the truncated Gaussian activation [24], similar in
behavior to softplus, not unlike leaky Relu, but continuous. All convolutions
used the “valid” border mode, so no zero-padding was used and the output
feature maps are smaller than the input map, even without down-sampling. The
output feature maps of the five layers have shapes (6, 40, 62), (36, 12, 19),
(96, 3, 5), 512, 256, 128, and 6, with total dimensions 14880, 8208, 1440,
512, 256, 128, and 6 respectively.

###  IV-D Dimension Mitigation and Training

Seamless streaming was not neccessary for this data set because the entire
event of dimension 24,318 could be processed. However, note that the first
five layers form a GLG with output dimension 256, so the saddle point can be
solved by inverting a matrix ${\bf C}$ of only $256\times 256$, just once per
mini-batch. Interestingly, the dominant computational load in the network
involves the 6-th layer, which requires inverting a $128\times 128$ matrix
${\bf C}(\hat{{\bf h}})$ on each sample of the mini-batch. The algorithm
maximized the log-likelihood of the PBN. We used direct saddle point
estimation using a symmetric linear transformation, followed by some Newton-
Raphson iterations. Surprisingly, a linear transformation $\hat{{\bf h}}={\bf
A}^{\prime}{\bf A}{\bf z}$, results in a small residual error. We used a mini-
batch size of 306, which was all of the training data. The large batch size
made best use of the GPU. On a Quadro P6000 GPU with 32 bit (single precision)
we were able compute an epoch in 8.5 seconds, or 11 seconds with 64 bit
precision. All experiments were done using the PBN Toolkit, which is a
graphical tool using the Theano framework. Each model required about 500
epochs.

###  IV-E Benchmark Network

For a performance benchmark, we applied conventional deep convolutional neural
network (CNN) with state of the art training methods. We used the same network
structure as the PBN, but the CNN was not bound by the restrictions of the
PBN. Therefore, we used TG activation function in each layer (linear
activation was not used), and we used max-pooling in the convolutional layers
(not down-sampling), dropout, and L2 regularization.

###  IV-F PBN and DNN Training

A separate PBN was trained for each class by maximizing a combined generative-
discriminative target function consisting of (a) the PBN log-likelihood
function, and (b) the negative cross-entropy classifier cost for
discriminating the corrsponding class from all other classes. The first
component was given a weight of one for all samples from the target class, and
zero for all samples from other classes. The second component was weighted by
one for all classes. This strategy caused the network to train as a PBN for
the target class, and as a classifier against all other classes. Data was
classified by evaluating the PBN log-likelihood function for each class
assumption, and choosing the largest. A class-dependent prior density was used
for the output distribution, as explained in [17]. A benchmark DNN was also
trained using the same data partitioning.

###  IV-G Classifier Results

Figure 2 shows the results of the classifier combination experiment in which
the PBN log-likelihood and DNN output were combined with a variable combing
factor. On the far left is seen the DNN performance, and on the far right the
discriminatively aligned PBN (PBN-DA) performance. The two classifiers alone
had exactly the same error probability. In the center of the graph, a deep
drop in classifier error is seen, reaching about half the error of each
classifier individually. Classifier combining works best when (a) the two
classifiers have comparable performance and (b) the classifiers are based on
independent views of the data, what usually occurs when combining generative
and discriminative classifiers. This result reflects what was seen on other
data sets for PBN-DA (See Figures 4 and 5 in [17]).

Figure 1: Original (top) and reconstructed (center and bottom) spectrograms
using D-PBN. There are two samples of each class in the order “skit”, “sciss”,
“pret”, “pens”, “paper”, and “jing”. A description of the sounds is found in
[22]. As can be clearly seen, the two rows of reconstructed samples were
trained on the first and fifth data class. Figure 2: Classifier combination
experiment. At the left side, performance with just DNN, and on the right side
just PBN-DA.

###  IV-H D-PBN-DA

To demonstrate the application of deterministic PBN (D-PBN) auto-encoder at
high dimension, we trained D-PBN on individual classes by reconstructing the
input data from the sixth-layer output (256 dimension) as explained in [9].
There is a 100-fold reduction in dimension. In contrast to previous uses of
D-PBN, we applied a discriminative component to the cost function, exactly as
done for the PBN-DA. Figure 1 shows examples of reconstructed samples. On the
top are a set of 12 original events, two from each class. Below this are seen
the reconstructions from the models trained on the first and fifth class. The
class correspondence can be easily seen because only data from the target
class is well reconstructed. One can classify based on this reconstruction
selectivity by choosing the class with lowest reconstruction error. The
classifier confusion matrix is (in errors out of 612 samples):

     [102  0   0   0   0   0]
     [ 0  90   1   1  10   0]
     [ 0   2  94   4   2   0]
     [ 0   1   1 100   0   0]
     [ 0   1   2   0  99   0]
     [ 0   0   0   0   0  102]

having 25 errors in 612 samples (4.08% error), almost as good as PBN-DA and
DNN. Due to the excellent class selectivity in reconstruction error, the
D-PBN-DA has promising potential applications in open-set classification.

##  V Conclusions

Using some of the dimension mitigation methods proposed in Section III, we
were able to effectively apply PBN at an input data dimension of 24,318, much
higher than the previously demonstrated dimensions of 900 and 196 (See Figures
4 and 5 in [17]). Furthermore, Figure 2 confirms that the advantages of a
discriminatively aligned PBN seen at lower dimensions, (a) PBN-DA working
about as well as a state of the art DNN, (b) the combined performance being
significantly better that either classifier alone, also apply at higher
dimension. In addition, we have demonstrated a novel new auto-encoder
classifier (D-PBN-DA) that performs as well as a state of the art CNN on the
same high-dimensional data set. The data, software, and instruction for
recreating these results are archived at [25].

## References

  * [1] P. M. Baggenstoss and S. Kay, “Nonlinear dimension reduction by pdf estimation,” (Accepted in) IEEE Transactions on Signal Processing, 2022\.
  * [2] P. M. Baggenstoss, “The PDF projection theorem and the class-specific method,” IEEE Trans Signal Processing, pp. 672–685, March 2003.
  * [3] P. M. Baggenstoss, “Maximum entropy PDF design using feature density constraints: Applications in signal processing,” IEEE Trans. Signal Processing, vol. 63, June 2015.
  * [4] M. Rosenblatt, “Remarks on a multivariate transformation,” The Annals of Mathematical Statistics, vol. 23, no. 3, pp. 470–472, 1952.
  * [5] G. Deco and W. Brauer, “Nonlinear higher-order statistical decorrelation by volume-conserving neural architectures,” Neural Netw., vol. 8, p. 525–535, June 1995.
  * [6] G. Deco and D. Obradovic, AN INFORMATION-THEORETIC APPROACH TO NEURAL COMPUTING.  Springer, 1996.
  * [7] A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis.  Wiley, 2001.
  * [8] A. Hyvärinen and P. Pajunen, “Nonlinear independent component analysis: Existence and uniqueness results,” Neural Networks, vol. 12, no. 3, pp. 429–439, 1999.
  * [9] P. M. Baggenstoss, “A neural network based on first principles,” in ICASSP 2020, Barcelona (virtual), (Barcelona, Spain), Sep 2020.
  * [10] P. M. Baggenstoss, “Applications of projected belief networks (pbn),” in Proceedings of EUSIPCO 2019, (La Coruña, Spain), Sep 2019.
  * [11] P. M. Baggenstoss, “Applications of projected belief networks (PBN),” Proceedings of EUSIPCO, A Corunã, Spain, 2019.
  * [12] P. M. Baggenstoss, “The projected belief network classifier: both generative and discriminative,” Proceedings of EUSIPCO, Amsterdam, 2020.
  * [13] M. Welling, M. Rosen-Zvi, and G. Hinton, “Exponential family harmoniums with an application to information retrieval,” Advances in neural information processing systems, 2004.
  * [14] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” in Neural Computation 2006, 2006.
  * [15] D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models,” in Proceedings of the 31st International Conference on Machine Learning (E. P. Xing and T. Jebara, eds.), vol. 32 of Proceedings of Machine Learning Research, (Bejing, China), pp. 1278–1286, PMLR, 22–24 Jun 2014.
  * [16] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems 27 (Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, eds.), pp. 2672–2680, Curran Associates, Inc., 2014.
  * [17] P. M. Baggenstoss, “Discriminative alignment of projected belief networks,” IEEE Signal Processing Letters, Sep 2021.
  * [18] V. Vapnik, The Nature of Statistical Learning.  Springer, 1999.
  * [19] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep Learning.  Cambridge, MA: MIT press, 2016.
  * [20] S. M. Kay, A. H. Nuttall, and P. M. Baggenstoss, “Multidimensional probability density function approximations for detection, classification, and model order selection,” IEEE Transactions on Signal Processing, vol. 49, pp. 2240–2252, Oct 2001.
  * [21] P. M. Baggenstoss, “On the duality between belief networks and feed-forward neural networks,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–11, 2018.
  * [22] P. Baggenstoss, “Office sounds database,” {http://class-specific.com/os}.  Accessed: 2022-02-28.
  * [23] P. M. Baggenstoss, “Acoustic event classification using multi-resolution hmm,” Proceedings of EUSIPCO 2018, Rome, Sep 2018.
  * [24] P. Baggenstoss, “New restricted Boltzmann machines and deep belief networks for audio classification,” 2021 ITG Speech Communication, Kiel (Virtual), 2021.
  * [25] P. Baggenstoss, “PBN Toolkit,” {http://class-specific.com/pbntk}.  Accessed: 2022-02-28.