# An Efficient Multitask Learning Architecture for Affective Vocal Burst
Analysis

Tobias Hallmen  Chair for Human-Centered Artificial Intelligence
Augsburg University, Germany
<EMAIL_ADDRESS>Silvan Mertes  Chair for Human-Centered Artificial
Intelligence
Augsburg University, Germany
<EMAIL_ADDRESS>Dominik Schiller  Chair for Human-Centered Artificial
Intelligence
Augsburg University, Germany
<EMAIL_ADDRESS>Elisabeth André  Chair for Human-Centered
Artificial Intelligence
Augsburg University, Germany
<EMAIL_ADDRESS>
###### Abstract

Affective speech analysis is an ongoing topic of research. A relatively new
problem in this field is the analysis of vocal bursts, which are nonverbal
vocalisations such as laughs or sighs. Current state-of-the-art approaches to
address affective vocal burst analysis are mostly based on wav2vec2 or HuBERT
features. In this paper, we investigate the use of the wav2vec successor
data2vec in combination with a multitask learning pipeline to tackle different
analysis problems at once. To assess the performance of our efficient
multitask learning architecture, we participate in the 2022 ACII Affective
Vocal Burst Challenge, showing that our approach substantially outperforms the
baseline established there in three different subtasks.

###### Index Terms:

Vocal Burst, Affect Recognition, Multitask Learning, data2vec

##  I Introduction

When developing systems to automatically recognise emotions from a given
input, current state-of-the-art models often rely on large amounts of manually
annotated training data. While humans have an innate capability of recognising
a universal set of emotions [10], recent studies show that socio-cultural
factors, such as culture or language impact the perceived authenticity and
accuracy of a recognised emotion [8, 7]. Therefore, it makes sense to take
this information into account when developing machine learning models for
automatic emotion recognition.

A particularly interesting problem arises when differences in culture are not
apparent through linguistic features, but only through prosodic ones. This is
the case, for example, when emotions are not expressed through verbal speech,
but rather through non-verbal vocal expressions, called _Vocal Bursts_. Here,
cultural differences are not immediately obvious. However, it is still
conceivable that humans develop the expression of vocal bursts differently
depending on their social and cultural environment.

The _2022 ACII Affective Vocal Burst Challenge_ introduces the problem of
understanding emotion in vocal bursts, like grunts, laughs or gasps. The
introduced HUME-VB [9] large-scale dataset consists of samples of emotional
human vocalisations annotated with respect to emotion and culture with vocal
bursts originating out of 4 countries. Thus, it is well suited to explore
methods for incorporating socio-cultural aspects into automatic emotion
recognition, which is the aim of this paper. The challenge consists of four
subtasks: (i) In the _High_ task, the intensity of 10 emotions has to be
identified, (ii) in the _Two_ task, the two-dimensional valence/arousal
representations have to be assessed, (iii) in the _Culture_ task, participants
have to predict the intensity of 10 emotions while paying respect to culture-
specific groundtruths, and (iv) in the _Type_ task, the type of expressive
vocal bursts (_Cry, Gasp, Groan, Grunt, Laugh, Pant, Scream, Other_) has to be
recognised.

To address these tasks, we focus on a combination of feature and multitask
learning to develop one coherent model that does not require explicit
modelling of features or context knowledge.111Code available at
github.com/hcmlab/acii-2022-vb-challenge.

The continued success of deep learning approaches to audio recognition tasks
raises the question of whether hand-crafted features are still necessary or
automatic learning of such features is a better way to go [20]. Recently,
Baevski et al. [3] proposed the wav2vec2 framework that is able to learn
meaningful feature representations from the raw waveform of an audio signal
that can outperform state of the art approaches for the task of automatic
speech recognition. Especially for voice emotion recognition, recent work
often relies on such embeddings as a backbone for the respective models [21,
22, 17, 6]. Data2vec [2], a generalised, abstracted derivate of wav2vec2 [3]
is a modality-agnostic approach to learning latent representations in a self-
supervised way. Data2vec, compared to wav2vec2, directly predicts
contextualized latent representations without quantization. For learning audio
representations, data2vec is pretrained on the Librispeech ASR corpus [16],
respectively Libri-light ASR benchmark [13] for the large version, and
released to the
public222github.com/facebookresearch/fairseq/tree/main/examples/data2vec.

Multitask learning for vocal bursts has already been a task in a previous
iteration of the challenge [5]. There, participants were asked to predict the
expression of 10 emotions along with the age and native country of the speaker
at the same time. Sharma et al. [19] approached the task by experimenting with
various encoder front ends as well as handcrafted features. They found that
using the HuBERT model [12], which is closely related to the wav2vec
architecture and training approach as a backbone yielded the best performance.
Purohit [18] et al. compared various embeddings that have been either trained
using self-supervision or directly in a task-dependent manner. They found that
overall, the self-supervised embeddings are outperforming the task-dependent
ones, which, supports the choice of data2vec for our experiments.
Anuchitanukul et al. [1] also rely on wav2vec and HuBERT backbones to extract
embeddings for their multitask training system. They further utilise an
adversarial training approach to disentangle the input representations into
shared, task-specific ones. Their experiments showed that the wav2vec-based
model performs best, but using ensemble techniques to combine multiple
variations of their wav2vec and HuBERT models can achieve even higher
performance. All three works indicate that self-supervision in general and
wav2vec specifically are building a good foundation for the task at hand and
confirming our choice of data2vec as the successor of wav2vec.

##  II Approach

Since vocal bursts are not “speech made out of words”, but rather “speech made
out of vocalised emotions”, the not fine-tuned version of data2vec is used.

To follow data2vec’s agnostic approach, no or as few assumptions as possible
are posed for the downstream supervised fine-tuning. To use all the provided
labels, multi-task learning is applied in a self-learning way [14], which
approximates optimal task weights by learning task uncertainty, including the
non-challenge task _Country_ – classifying the vocal burst’s origin.

In Figure 1, the net architecture is depicted. The raw audio is fed into the
pretrained data2vec model, including its preprocessor. Variable sequence
lengths are zero-padded to the longest seen sequence. Because of the attention
mask, the extracted features vary in length. Therefore, these are mean-pooled
and fed into downstream projection layers. To investigate the question of
whether knowing certain tasks before predicting other tasks is helpful, the
intermediate tasks are separated and their prediction is fed along with the
extracted features to the remaining tasks.

The projection layers reduce the output dimension of data2vec (768 base, 1024
large) to 256, apply GELU [11], then further reduce the dimension down to the
five task’s required dimensions (high: 10, culture: 40, two: 2, type: 8,
country: 4). The tasks’ layers then apply softmax for classification and
compute the loss via cross-entropy using inversely proportional class weights.
For regression, we apply sigmoid and compute the loss using the concordance
correlation coefficient. The losses are then linearly combined through the
learnt optimal weights.

The whole net has a size of 360MB and is trained in a two-stage matter: First,
freeze data2vec and train only the tasks with their projection layers. Second,
unfreeze data2vec and fine-tune the whole net. Both stages are trained for 30
epochs, early stopping with a patience of 2, batch size of 32 for the base
version and 24 for the large version, and optimiser AdamW [15]. For the first
stage, the optimiser uses defaults, for the second one, the learning rate is
initialised at $4\cdot 10^{-5}$ and follows a cosine schedule with a warmup of
1 epoch.

Training is done on a single Nvidia A40 GPU and takes 5 to 6.5 hours for the
base version of data2vec, and 5.5 to 8.5 hours for the large version.

Figure 1: Overview of the net architecture.

##  III Experiments & Results

In our experiments, we varied certain details of the network architecture as
listed below. The results for the validation set are shown in Table I,
additionally to the official baseline results presented in [4]. Note, that the
single experiments were conducted iteratively, carrying the best configuration
of the previous experiments over to the next set of experiments.

_Intermediate Tasks._ Following the challenge’s motivation that vocal bursts
depend on the country of origin and their type to assess the conveyed emotions
like a rater of the same origin would we tried three different variants: (i)
_Country_ and _Type_ as intermediate tasks before predicting the remaining
three (_2/3_). (ii) Only _Country_ as an intermediate task (_1/4_), assuming
the type depends less on the country. (iii) No intermediate tasks, i.e.,
predict all tasks simultaneously (_0/5_). (iii) performed best.

_Task Loss._ To investigate, why the task _Two_ performs so badly, we
experimented with replacing the CCC loss by (i) mean squared error (_MSE_) and
(ii) mean absolute error (_MAE_). (iii) We excluded the artificially most
certain task _Two_ from training (_-Two_). (iii) turned out to be the best
option.

_Weighting._ Since the cross-entropy losses in tasks _Country_ and _Type_ have
inverse proportional class weights and the other tasks do not, we experimented
with (i) removing these class weights from the training (_-CW_), so every
sample’s tasks are unweighted, (ii) adding sample weights inversely
proportional to _Country_ (_+SW_), thereby weighting all tasks, and (iii)
keeping the class weights but removing the sigmoid activation from the last
layer(_-SM_) and clamping the output to $[0,1]$ in the regression tasks _High_
and _Culture_. We could substantially boost the network’s performance by
applying (iii).

_Fine-Tuning Splits._ To investigate if fine-tuning the self-supervised learnt
audio representations using labelled speech helps, we experimented with
different versions of the base network, where each version was fine-tuned
after pretraining using connectionist temporal classification (CTC) loss on a
different amount of LibriSpeech: (i) 10 minutes (_B10m_), (ii) 100 hours
(_B100h_), (iii) 960 hours (_B960h_). Version (iii) performed best.

_Network Size._ At last, we replaced the base network with (i) the _Large_
(_L_) version and (ii) the _Large Fine-tuned_ (_L960h_) version. Also, we
experimented with completely dropping the _Country_ task (_-Country_).
However, no improvement could be observed in any experiment here.

As can be seen in Table I, for the _High_ and _Culture_ tasks, experiment
_B960h_ worked best on the validation set. For the tasks _Type_ and _Country_
, experiment _-SM_ performed best on the validation set. To obtain results on
the test set, predictions from both models (_B960h_ and _-SM_) were submitted
to the challenge organisers. The tasks _Country_ and _Two_ were omitted as
non-challenge, respectively unsuited representations for valence/arousal
estimation. The results on the test set are shown in Table II. As can be seen,
our results outperform the baseline in all three submitted tasks.

TABLE I: Results on the validation split of the A-VB challenge dataset. | High | Culture | Two | Type | Country
---|---|---|---|---|---
Method | CCC | CCC | CCC | UAR | UAR
Baseline | .564 | .436 | .499 | .412 | –
2/3 | .639 | .625 | .252 | .562 | .631
1/4 | .628 | .614 | .245 | .542 | .603
0/5 | .650 | .636 | .254 | .564 | .633
MSE | .624 | .598 | .244 | .553 | .591
MAE | .640 | .573 | .251 | .575 | .655
-Two | .651 | .637 | – | .574 | .645
-CW | .645 | .631 | – | .276 | .632
+SW | .643 | .629 | – | .319 | .644
-SM | .654 | .639 | – | .584 | .657
B10m | .656 | .642 | – | .577 | .655
B100h | .639 | .624 | – | .552 | .634
B960h | .658 | .642 | – | .567 | .647
-Country | .655 | .639 | – | .570 | –
L | .635 | .622 | – | .554 | .603
L960h | .613 | .617 | – | .540 | .539
TABLE II: Results on the test split of the A-VB challenge dataset. | High | Culture | Type
---|---|---|---
Method | CCC | CCC | UAR
Baseline | .569 | .440 | .417
-SM | .685 | .515 | .586
B960h | .679 | .526 | .573

##  IV Discussion

Recapitulating the conducted experiments and considering the results, the
following insights can be drawn:

_Intermediate Tasks._ Determining the country of origin, optionally also the
type of the vocal burst, before assessing the conveyed emotions like a rater
of that country would, may be beneficial to a human rater, so that the
emotional biases can be adjusted, but it is disadvantageous for neural
networks – the extracted features already encompass these biases and need not
be handcrafted into.

_Task Loss._ Varying the task losses did not improve the bad performance on
the task _Two_. Since the baseline shows double the performance here, the
self-supervised learnt audio representations are unsuited for estimating
valence and arousal. Therefore, the model learns to predict a rather constant
output for _Two_ , so that the uncertainty in this task is reduced, minimising
the penalising uncertainty term, but also maximising the task weight in the
computation of the MTL loss. This can be seen in the learnt task weights,
where _Two_ has 18-times (_MSE_), respectively 4-times (_MAE_), higher weight
than the least weighted task (_Country_), overshadowing other better-
performing tasks. Removing _Two_ allows for higher variance and thereby better
predictions in other tasks.

_Weighting._ These experiments investigated different weighting techniques to
counter imbalance in the training data. Removing the inversely proportional
class weights in the calculation of the cross-entropy loss greatly reduces the
performance in _Type_. Extending the cross-entropy weights to the whole
sample, to inversely proportional intra-batch weights depending on _Country_ ,
alleviates this only slightly. Inversely proportional class weights both in
_Type_ and _Country_ to counter class imbalance, combined with the removal of
sigmoid activation, thereby increasing the net’s sensitivity to samples close
to the boundaries of the value ranges, is the best approach. Interestingly,
mostly _Type_ is affected by these changes, which can be seen in its learnt
task weight – the uncertainty almost doubled.

_Fine-Tuning Splits._ This addresses the assumption, that vocal bursts are not
a word-based language, thereby we expect a decline in performance the more the
pretrained net is fine-tuned using CTC on word-based labels. This decline is
only slight for 10 minutes ($0.02\%$), but more so for 100 hours ($10.4\%$) of
data used for fine-tuning. Interestingly, this does not hold for using all the
data (960 hours) in fine-tuning. The predictions do differ, but on average,
these differences cancel out.

_Network Size._ This investigates if the findings carry over if the net’s
feature extractor dimension is increased, or the output dimension is reduced,
i.e. removing a task. Surprisingly, enlarging the feature extractor’s
dimension from 768 to 1024 decreases the performance noticeably. The more so,
if it is fine-tuned on the whole LibriSpeech dataset after pretraining on
Libri-light, where 960 hours resemble $1.6\%$ of data. Since both Libri-
datasets are English-only, the large version of data2vec may overfit to the
English language, hence reducing performance on non-English vocal bursts.
Removing the non-challenge task _Country_ decreases the performance of the
remaining tasks. Thereby, using “irrelevant” data in multi-task learning does
help.

_Test results._ Regarding the test results in Table II, there is still no
clear difference visible between the pretrained and the fine-tuned base
version of data2vec. In the tasks _High_ and _Type_ , the generalisability of
the models hold, achieving similar results on the test set as on the
validation set. Note, that on task _Culture_ , both models decreased by 12
percentage points compared to validation results. Without the test labels it
is not clear why, since the task _High_ performs well while sharing with task
_Culture_ both the emotion-classification type and uncertainty ($+2.3\%$). All
tasks’ uncertainties are in the same order of magnitude, with task _Culture_
as the most uncertain and task _Type_ as the most certain ($-40\%$). Hence,
overfitting to _Culture_ seems unlikely, and it may be a sign of English-only
pretraining and fine-tuning, which averages out for task _High_.

##  V Conclusion & Outlook

We showed that no handcrafted features or assumptions have to be applied to
achieve good results in multiple tasks simultaneously. Per-task ensembling and
large structures are not needed, a single lightweight net for all tasks
simultaneously suffices. Although being pretrained self-supervised only on
English speech, data2vec is able to assess vocal bursts originating out of
different (non-English) countries after fine-tuning it for fiveish hours.

The results may be further improved by enlarging the pretraining dataset by
gathering multi-language samples in the wild and modifying the multi-task loss
by penalising learning a constant prediction. Since the net’s pretraining can
be fine-tuned using CTC, it may be worthwhile to investigate datasets
featuring other labels than words, that are more suited for vocal bursts, e.g.
letters or phonemes.

## Acknowledgements

This work was partially funded by the KodiLL project (FBM2020, Stiftung
Innovation in der Hochschullehre).

## References

  * [1] Atijit Anuchitanukul and Lucia Specia.  Burst2vec: An adversarial multi-task approach for predicting emotion, age, and origin from vocal bursts.  arXiv preprint arXiv:2206.12469, 2022.
  * [2] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli.  Data2vec: A general framework for self-supervised learning in speech, vision and language.  arXiv preprint arXiv:2202.03555, 2022.
  * [3] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  Advances in Neural Information Processing Systems, 33:12449–12460, 2020.
  * [4] Alice Baird, Panagiotis Tzirakis, Anton Batliner, Björn Schuller, Dacher Keltner, and Alan Cowen.  The acii 2022 affective vocal bursts workshop and competition: Understanding a critically understudied modality of emotional expression.  arXiv preprint arXiv:2207.03572v1, 2022.
  * [5] Alice Baird, Panagiotis Tzirakis, Gauthier Gidel, Marco Jiralerspong, Eilif B. Muller, Kory Mathewson, Björn Schuller, Erik Cambria, Dacher Keltner, and Alan Cowen.  The icml 2022 expressive vocalizations workshop and competition: Recognizing, generating, and personalizing vocal bursts.  arXiv preprint arXiv:2205.01780v3, 2022.
  * [6] Li-Wei Chen and Alexander Rudnicky.  Exploring wav2vec 2.0 fine-tuning for improved speech emotion recognition.  arXiv preprint arXiv:2110.06309, 2021.
  * [7] Georgia Chronaki, Michael Wigelsworth, Marc D Pell, and Sonja A Kotz.  The development of cross-cultural recognition of vocal emotion during childhood and adolescence.  Scientific reports, 8(1):1–17, 2018.
  * [8] Gonçalo Cosme, Vânia Tavares, Guilherme Nobre, César Lima, Rui Sá, Pedro Rosa, and Diana Prata.  Cultural differences in vocal emotion recognition: a behavioural and skin conductance study in portugal and guinea-bissau.  Psychological Research, 86(2):597–616, 2022.
  * [9] Alan Cowen, Alice Baird, Panagiotis Tzirakis, Michael Opara, Lauren Kim, Jeff Brooks, and Jacob Metrick.  The hume vocal burst competition dataset (H-VB) — raw data [exvo: updated 02.28.22] [data set].  Zenodo, 2022.
  * [10] Paul Ekman.  An argument for basic emotions.  Cognition & emotion, 6(3-4):169–200, 1992.
  * [11] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [12] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  * [13] Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al.  Libri-light: A benchmark for asr with limited or no supervision.  In 2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 7669–7673. IEEE, 2020.
  * [14] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482–7491, 2018.
  * [15] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [16] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  LibriSpeech: an asr corpus based on public domain audio books.  In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015.
  * [17] Leonardo Pepino, Pablo Riera, and Luciana Ferrer.  Emotion recognition from speech using wav2vec 2.0 embeddings.  arXiv preprint arXiv:2104.03502, 2021.
  * [18] Tilak Purohit, Imen Ben Mahmoud, Bogdan Vlasenko, and Mathew Magimai Doss.  Comparing supervised and self-supervised embedding for exvo multi-task learning track.  arXiv preprint arXiv:2206.11968, 2022.
  * [19] Roshan Sharma, Tyler Vuong, Mark Lindsey, Hira Dhamyal, Rita Singh, and Bhiksha Raj.  Self-supervision and learnable strfs for age, emotion, and country prediction.  arXiv preprint arXiv:2206.12568, 2022.
  * [20] Johannes Wagner, Dominik Schiller, Andreas Seiderer, and Elisabeth André.  Deep learning in paralinguistic recognition tasks: Are hand-crafted features still relevant?  In B. Yegnanarayana, editor, Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 September 2018, pages 147–151. ISCA, 2018.
  * [21] Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn Schuller.  Dawn of the transformer era in speech emotion recognition: Closing the valence gap. arxiv 2022.  arXiv preprint arXiv:2203.07378, 2022.
  * [22] Yingzhi Wang, Abdelmoumene Boumadane, and Abdelwahab Heba.  A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding.  arXiv preprint arXiv:2111.02735, 2021.


# SAGE-HB: Swift Adaptation and Generalization in Massive MIMO Hybrid
Beamforming ††thanks:  The authors are with the Department of Electrical
Engineering, Polytechnique Montreal, Montreal, QC H3T 1J4, Canada. Emails:
{ali.hasanzadeh-karkan, hamed.hojatian, j-f.frigon, francois.leduc-
<EMAIL_ADDRESS>
Ali Hasanzadeh Karkan, Hamed Hojatian, Jean-François Frigon, and François
Leduc-Primeau

###### Abstract

Deep learning (DL)-based solutions have emerged as promising candidates for
beamforming in massive Multiple-Input Multiple-Output (mMIMO) systems.
Nevertheless, it remains challenging to seamlessly adapt these solutions to
practical deployment scenarios, typically necessitating extensive data for
fine-tuning while grappling with domain adaptation and generalization issues.
In response, we propose a novel approach combining Meta-Learning Domain
Generalization (MLDG) with novel data augmentation techniques during fine-
tuning. This approach not only accelerates adaptation to new channel
environments but also significantly reduces the data requirements for fine-
tuning, thereby enhancing the practicality and efficiency of DL-based mMIMO
systems. The proposed approach is validated by simulating the performance of a
backbone model when deployed in a new channel environment, and with different
antenna configurations, path loss, and base station height parameters. Our
proposed approach demonstrates superior zero-shot performance compared to
existing methods and also achieves near-optimal performance with significantly
fewer fine-tuning data samples.

##  I Introduction

The emergence of massive MIMO (mMIMO) technology ushers in a new era of
wireless communication, promising unprecedented gains in spectral efficiency
and network capacity [1]. Hybrid beamforming (HBF) offers a balanced solution,
merging analog and digital beamforming to efficiently deploy mMIMO systems.
Unlike fully digital precoder (FDP), HBF strikes a practical solution,
optimizing signal transmission while overcoming hardware constraints in large-
scale deployments [2].

Alongside traditional beamforming methods, recent advances in deep learning
(DL)-based HBF design, encompassing supervised, unsupervised, and
reinforcement learning paradigms, have drawn significant attention [3, 4, 5,
6, 7, 8]. These DL-based HBF methods not only hold the potential for improved
spectral efficiency but also offer the promise of lower complexity. Supervised
learning excels in precision but lacks adaptability and relies on labeled
data, while unsupervised learning is more adaptive than supervised learning
but is not known for its fast adaptability; it often involves time-complex
fine-tuning that can be ineffective. Reinforcement learning holds promise for
adaptability in dynamic environments but imposes significant computational
complexity and maintaining adaptability, requiring extensive resources and
potentially impractical training times. In practical deployment scenarios, the
need for fine-tuning poses a significant challenge for DL-based approaches.
The data collection not only incurs resource costs but, more critically,
reduces the base station’s operational efficiency. There is thus a strong need
for resource-efficient fine-tuning in DL implementation.

Recent studies have explored transfer learning, and meta-learning techniques
aimed at enhancing the adaptation of pre-trained DNN to the specific
requirements of deployment base station (BS) [9, 10, 11]. Particularly, in
[9], the authors propose two offline adaptive beamforming algorithms based on
deep transfer learning and meta-learning, and an online algorithm to enhance
the adaptation capability of the offline meta-algorithm in realistic non-
stationary environments. The authors in [10] presented a solution for tackling
the problem of mismatch by first training an embedding model to extract
transferable features, and then using support vector regression for fitting.
An iterative meta-learning algorithm is introduced in [11] for maximizing
Weighted Sum Rate (WSR) in MISO downlink channels. In [12], the authors
proposed to use data augmentation techniques to enhance the robustness of the
neural precoding methods for designing beamforming in the MISO system.
However, their methods rely on adding noise to the input CSI, which may not
capture the true diversity and complexity of the new deployment domain. A
shortcoming of many existing studies in transfer learning for wireless
communications is that both the training and deployment scenarios are
simulated using statistical channel models, which may not capture the full
extent of the domain shift.

In this paper, we focus on enhancing the adaptability and generalization
capabilities of DL-based HBF in diverse channel conditions while addressing
the resource-intensive nature of fine-tuning. Our goal is to seamlessly
integrate DL-based HBF into the dynamic realm of mMIMO systems. Inspired by
the success of meta-learning and domain generalization, we aim for swift
adaptation. Additionally, we introduce an innovative data augmentation
technique in unsupervised fine-tuning to reduce the resource demands for
dataset gathering. Based on our evaluations, we demonstrate our method’s swift
adaptation, requiring fewer fine-tuning samples, and its superior performance
over existing DL-based solutions. Notably, it can achieve good performance
even in scenarios with changes in antenna configurations. Finally, the
experimental validation provides insight into the transition from statistical
channel models to real-world scenarios.

##  II System model

Let us consider a multi-user mMIMO system consisting of a BS equipped with
$N_{\sf{T}}$ antennas and $N_{\sf{RF}}$ radio frequency (RF) chains serving
$N_{\sf{U}}$ single antenna users simultaneously. For the downlink
transmission, HBF precoders are employed by the BS. The signal received by
each user can be written as

$\mathbf{y}_{u}=\mathbf{h}_{u}^{\dagger}\mathbf{A}\sum_{\forall
u}\mathbf{w}_{u}x_{u}+\boldsymbol{\eta},$ (1)

where $\mathbf{h}_{u}\in\mathbb{C}^{N_{\sf{T}}\times 1}$ stands for the
channel vector from the user index $u$ to the $N_{\sf{T}}$ antennas at the BS,
$\mathbf{x}=[x_{1},...,x_{u},...,x_{N_{\sf{U}}}]$ is the vectors of
transmitted symbol for all users, normalized to
$\mathop{\mathbb{E}}[\mathbf{x}\mathbf{x}^{\dagger}]=\frac{1}{N_{\sf{U}}}\mathbbm{1}_{N_{\sf{U}}}$,
and $\boldsymbol{\eta}$ is the additive white gaussian noise (AWGN) term with
noise power $\sigma^{2}$. The HBF vectors consist of a digital baseband
precoder
$\mathbf{W}=[\mathbf{w}_{1},...,\mathbf{w}_{u},...,\mathbf{w}_{N_{\sf{U}}}]\in\mathbb{C}^{N_{\sf{RF}}\times
N_{\sf{U}}}$ and analog precoder (AP) which is defined by a $N_{\sf{T}}\times
N_{\sf{RF}}$ matrix $\mathbf{A}\in\mathbb{C}^{N_{\sf{T}}\times N_{\sf{RF}}}$,
where $[\mathbf{A}]_{n,m}=e^{j\varphi_{n,m}}$ is the state of the phase-
shifter (PS) connected between the $n^{\text{th}}$ antenna and $m^{\text{th}}$
RF chain by the phase-shift of $\varphi_{n,m}$. The signal-to-interference-
noise ratio (SINR) of the $u^{\text{th}}$ user is given by

$\displaystyle\text{SINR}(\mathbf{A},\mathbf{w}_{u})$
$\displaystyle=\frac{\big{|}\mathbf{h}^{\dagger}_{u}\mathbf{A}\mathbf{w}_{u}\big{|}^{2}}{\sum_{j\neq
u}\big{|}\mathbf{h}^{\dagger}_{u}\mathbf{A}\mathbf{w}_{j}\big{|}^{2}+\sigma^{2}}\,.$
(2)

For a given hybrid beamformer $(\mathbf{A},\mathbf{W})$, the sum-rate is

$\displaystyle R(\mathbf{A},\mathbf{W})=\sum_{\forall
u}\text{log}_{2}\Bigl{(}1+\text{SINR}(\mathbf{A},\mathbf{w}_{u})\Bigr{)}\,.$
(3)

Then, the HBF design consists of finding the precoder matrices $\mathbf{W}$
and $\mathbf{A}$ that maximize the sum-rate in (3) subject to a maximum
transmission power $P_{\sf{max}}$. More formally, in training backbone model
and deployment scenario, we seek to solve the following optimization problem:

$\displaystyle\underset{\mathbf{A},\mathbf{W}}{\max}$ $\displaystyle
R(\mathbf{A},\mathbf{W})=\underset{\mathbf{A},\mathbf{w}_{u}}{\max}\sum_{\forall
u}\text{log}_{2}\Bigl{(}1+\text{SINR}(\mathbf{A},\mathbf{w}_{u})\Bigr{)},$
(4a) s.t. $\displaystyle\sum_{\forall
u}\mathbf{w}_{u}^{\dagger}\mathbf{A}^{\dagger}\mathbf{A}\mathbf{w}_{u}\leq
P_{\sf{max}}.$ (4b)

###  II-A Problem Definition

In this paper, we study how effectively the DNN, initially trained in a
statistical channel environment, can adapt its HBF strategies to the dynamic
and unpredictable nature of practical wireless communication scenarios.
Training with a statistical channel model offers a practical advantage. By
utilizing this simplified yet representative model, we can train a robust
backbone model in a cloud-based environment with a large available statistical
dataset, detaching the training process from the intricacies of specific
channel conditions. This decoupling allows us to create a flexible and
generalized deep neural network (DNN) in varying channels that is applicable
across numerous BS.

The statistical channel model utilized in this study offers a practical and
computationally efficient tool for training our DNN for HBF in mMIMO systems,
with a specific focus on Line-of-Sight (LOS) scenario similar to the
deployment phase. To generate statistical channel models for the backbone
training, $\mathbf{h}_{u}$ is expressed as [13]

$\mathbf{h}_{u}=\mathbf{a}(\phi,\theta,\mathbf{n})\sqrt{G_{BS}G_{u}}\left(\frac{\ell_{BS}\ell_{u}}{4\pi
x_{u}^{\gamma}}\right)e^{j2\pi\frac{x_{u}}{\lambda_{c}}}\,,$ (5)

where $(G_{BS},\ell_{BS})$ and $(G_{u},\ell_{u})$ are the gains and the height
corresponding with the antennas of the BS and $u$th UE, respectively. The
three-dimensional distance between the BS and $u$th UE is denoted by $x_{u}$
and $\lambda_{c}$ represents the carrier wavelength. The exponent path loss
factor ($\gamma$) is a parameter that describes how the signal power decreases
with distance in a wireless communication system. In addition,
$\mathbf{a}(\phi,\theta,\mathbf{n})$ denotes the antenna array response vector
of the BS, where $\mathbf{n}$ is the base station antenna array dimensions in
each direction $\mathbf{n}=[N_{x},N_{y},N_{z}]$, and $\phi,\theta$ stand for
azimuth and elevation angles of arrival at the BS, respectively. The general
array response vector $\mathbf{a}(\phi,\theta,\mathbf{n})$ is decomposed as

$\displaystyle\mathbf{a}(\phi,\theta,\mathbf{n})=$
$\displaystyle~{}\boldsymbol{\omega}\left(\psi_{z}(\phi,\theta),N_{z}\right)\otimes$
(6)
$\displaystyle~{}\boldsymbol{\omega}\left(\psi_{y}(\phi,\theta),N_{y}\right)\otimes$
$\displaystyle~{}\boldsymbol{\omega}\left(\psi_{x}(\phi,\theta),N_{x}\right),$

with the components given by

$\boldsymbol{\omega}\left(\psi,N\right)=\left[1,~{}\mathrm{e}^{j2\pi\frac{d_{BS}}{\lambda_{c}}\psi},~{}\ldots,~{}\mathrm{e}^{j2\pi\frac{d_{BS}}{\lambda_{c}}\left(N-1\right)\psi}\right]^{T},$
(7)

where $d_{BS}$ is the spacing between antennas at the BS, and for each
direction, $\psi$ is expressed as

$\displaystyle\psi_{x}(\phi,\theta)$ $\displaystyle=\sin(\theta)\cos(\phi)$
(8) $\displaystyle\psi_{y}(\phi,\theta)$
$\displaystyle=\sin(\theta)\sin(\phi)$ $\displaystyle\psi_{z}(\phi,\theta)$
$\displaystyle=\cos(\theta).$

For the testing and deployment phase, we transition to a more complex and
realistic channel model, namely the ray-tracing channel model. We used
deepMIMO channel model [14] for our deployment scenario. Unlike the
statistical model, the ray-tracing model provides a high-fidelity
representation of the wireless channel by simulating the propagation of
electromagnetic waves in a detailed 3D environment. This approach considers
various factors, including reflections, diffractions, and scattering, to
capture the intricacies of real-world propagation conditions accurately. By
training the DNN with a statistical channel model and subjecting the DNN-based
HBF model to the complexities of the ray-tracing realistic channel model, we
aim to uncover crucial insights into its adaptability and robustness in real-
world conditions.

One of the main research challenges of our study is to develop a robust
backbone model characterized by its adaptability to variations in the
environment and BS configurations. In particular, this study emphasizes the
importance of three critical parameters in mMIMO systems: the antenna array
response vector, the height of the base station, and the path loss exponent
factor. The antenna array response vector is a critical component that can
vary based on different antenna heights and possess different numbers of
elements in each direction. For instance, as illustrated in Table I, the
backbone model was initially trained under different antenna array settings
and subsequently deployed in diverse configurations. In this context, it is
important to note that we assume $N_{\sf{T}}=64$, and the subscript $i$ within
$\mathbf{n}_{(i)}$ indicates the directions of the antenna array. For
instance, in the case of $\mathbf{n}_{yz}$, it implies the deployment of 64
antennas along the $y$ and $z$ axes with respective quantities of
$\mathbf{n}_{yz}=[1,8,8]$. The outcomes reveal a significant degradation in
zero-shot111i.e., without fine-tuning performance when the model was deployed
in configurations for which it was not originally tailored (as observed in the
non-diagonal elements of the tables). Therefore, it becomes apparent that the
alteration of the antenna array response vector can lead to a substantial
decline in model performance.

The height of the base station is also another critical parameter in mMIMO
systems, influencing signal propagation and coverage. To underscore its
significance, we conducted experiments highlighting the impact of varying base
station heights during training and deployment. For example, training the
backbone model for a base station with a height of 12m and deploying it in the
ray-tracing scenario with a different height of 6m resulted in a significant
drop of $43.24\%$ in zero-shot performance compared to near-optimal [7]. This
highlights the need to consider base station height variations during training
for improved model robustness and zero-shot performance in diverse deployment
scenarios.

The path loss exponent factor captures the power falloff of the signal
relative to the distance from the transmitter and depends on the carrier
frequency, environment, obstructions, etc. It is an important parameter for
modeling the channel behavior and performance of mMIMO systems in different
scenarios. The path loss exponent factor is used to increase the
generalization of the channel model to unknown environments. By varying this
parameter, different levels of path loss can be simulated and analyzed. This
allows us to evaluate the robustness and adaptability of mMIMO systems under
various propagation conditions.

TABLE I: Impact of Different Antenna Configuration
on Zero-Shot Sum Rate (b/s/Hz) ($-\log\sigma^{2}=13$)

|  | Deployment
---|---|---
|  | $\mathbf{n}_{x}$ | $\mathbf{n}_{y}$ | $\mathbf{n}_{z}$ | $\mathbf{n}_{xy}$ | $\mathbf{n}_{xz}$ | $\mathbf{n}_{yz}$ | $\mathbf{n}_{xyz}$
| $\mathbf{n}_{x}$ | 15.89 | 6.53 | 1.54 | 10.42 | 8.23 | 6.34 | 7.34
| $\mathbf{n}_{y}$ | 8.92 | 15.39 | 1.57 | 8.33 | 7.63 | 7.83 | 6.51
| $\mathbf{n}_{z}$ | 1.23 | 1.36 | 14.45 | 1.54 | 5.84 | 4.53 | 6.34
| $\mathbf{n}_{xy}$ | 11.13 | 8.49 | 1.26 | 14.67 | 4.64 | 5.69 | 3.18
| $\mathbf{n}_{xz}$ | 8.01 | 7.51 | 5.64 | 4.15 | 13.86 | 5.51 | 7.61
| $\mathbf{n}_{yz}$ | 6.21 | 8.14 | 4.16 | 5.77 | 6.36 | 14.24 | 5.76
Backbone | $\mathbf{n}_{xyz}$ | 7.69 | 5.41 | 5.26 | 5.36 | 8.13 | 5.32 | 12.64
near-optimal [7] | 22.36 | 21.26 | 18.64 | 20.91 | 19.18 | 18.54 | 18.37

##  III Sage-HB-Net

This section presents our proposed adaptive deep learning method for mMIMO
HBF, which includes a data augmentation method in the deployment phase that
allows fine-tuning of the model with a small dataset.

###  III-A Backbone Training

Domain generalization, also known as out-of-distribution generalization, is a
technique applied in scenarios where data availability in the deployment
environment is restricted. This limitation may arise due to privacy
considerations or the necessity for real-time learning [15]. On the other
hand, meta-learning, often described as “learning to learn,” introduces a
hierarchical structure in AI. This approach empowers AI systems to learn
rapidly, adapt to changing environments, and apply their knowledge to a
broader array of tasks. Instead of relying on copious data from similar tasks
encountered in the past to tackle a new challenge, meta-learning focuses on
the capacity to acquire the skill of learning itself.

In our work, one objective is to significantly reduce the data dependency of
our model during the deployment phase, while concurrently enhancing its energy
and spectral efficiency. To train such a model, we use the Meta-Learning
Domain Generalization (MLDG) method introduced in [16]. MLDG employs a
training technique with the goal of improving the adaptability of a model to
unseen domains. Within the meta-training, the aim of the model is to reduce
the loss within a specific set of domains, referred to as the training
domains. Simultaneously, it seeks to ensure that the optimization gradients
selected also lead to enhancements in a different subset of domains known as
the testing domains. Leveraging the MLDG algorithm as our foundation, we
intend to formulate a meta-learning and domain generalization algorithm
tailored to adaptive mMIMO HBF. We aim to achieve a superior adaptation and
higher level of zero-shot performance compared with other existing adaptive
methods.

The proposed method is outlined in Algorithm 1, which we now explain in
detail. In the first stage, known as “Backbone-Training,” the algorithm is
initialized with model parameters ($\Theta$) and learning rates for each
updating part ($\alpha,\beta,\epsilon$). It operates on a set of statistical
channel model domains ($\boldsymbol{\mathcal{D}}$) with the following key
steps. The algorithm initiates by randomly dividing available domains into two
subsets: $\boldsymbol{\mathcal{D}}^{train}$ for training and
$\boldsymbol{\mathcal{D}}^{gen}$ for generalization, ensuring adaptability to
new, previously unseen environments. In the meta-training phase, for each
domain within the training set
($\hat{\mathcal{D}}\in\boldsymbol{\mathcal{D}}^{train}$), the algorithm
computes gradients of the loss function with respect to the model parameters
($\nabla_{\Theta}\mathcal{L}(\hat{D};\Theta)$). These gradients represent the
learning progress for each domain, which are accumulated in $L$. Following the
accumulation of gradients, the algorithm updates the model parameters using
$L$ to enhance its performance within the training domains
($\Theta^{\prime}=\Theta-\alpha L$). This step allows the model to better
adapt to the characteristics of the known training environments. In the meta-
testing phase, similar to meta-training, the algorithm computes gradients
($\nabla_{\Theta^{\prime}}\mathcal{L}(\hat{\mathcal{D}};\Theta^{\prime})$) for
the domains in the generalization set
($\hat{\mathcal{D}}\in\mathcal{D}^{gen}$). These gradients are accumulated in
$L^{\prime}$ and are used to evaluate the model’s ability to generalize to
new, unseen environments. Subsequently, the algorithm performs a meta-update,
adjusting the model parameters based on the aggregated gradients from both the
training and generalization domains ($\Theta\leftarrow\Theta-\lambda(L+\beta
L^{\prime})$). This meta-update is crucial for enhancing the model’s
generalization capabilities, making it more adaptable to different
environments. The main objective is to reduce losses in both domains
simultaneously in a synchronized way.

Backbone-Training
Input: Statistical channel model domains $\boldsymbol{\mathcal{D}}$

Initialize : Model parameters $\Theta$, Learning rates $\alpha,\beta,\epsilon$

1

2for  _$e=1$ to $Epochs$_ do

3 Divide $\boldsymbol{\mathcal{D}}\rightarrow\boldsymbol{\mathcal{D}}^{train}$
and $\boldsymbol{\mathcal{D}}^{gen}$

4 $L\leftarrow 0$

5 for  _each $\hat{D}$ in $\boldsymbol{\mathcal{D}}^{train}$ _ do

6 $L\leftarrow
L+\frac{\nabla_{\Theta}\mathcal{L}(\hat{D};\Theta)}{|\boldsymbol{\mathcal{D}}^{train}|}$

7

8 Update $\Theta^{\prime}=\Theta-\alpha L$

9 Meta-Training $L^{\prime}\leftarrow 0$

10 for  _each $\hat{D}$ in $\boldsymbol{\mathcal{D}}^{gen}$ _ do

11 $L^{\prime}\leftarrow
L^{\prime}+\frac{\nabla_{\Theta^{\prime}}\mathcal{L}(\hat{D};\Theta^{\prime})}{{|\boldsymbol{\mathcal{D}}^{gen}|}}$

12

13 Meta-Tesing Meta-update: $\Theta\leftarrow\Theta-\epsilon(L+\beta
L^{\prime})$

14

Fine-Tuning
Input: Flattened dataset $\mathbf{F}$, Augmentation size $m$

Initialize : Model parameters $\Phi\leftarrow\Theta$, Learning rate $\tau$

15 $D^{Aug}\leftarrow\emptyset$

16 for  _$i=1$ to $m$_ do

17 $S\leftarrow$ all-zero matrix of size $N_{T}\times N_{U}$

18 for  _$j=1$ to $N_{U}$_ do

19 $k\sim\mathcal{U}(1,|\mathbf{F}|)$

20 $S[:,j]=F[:,k]$

21

22 $D^{Aug}\leftarrow D^{Aug}\cup\\{S\\}$

23

24 Data Augmentation for  _$e=1$ to $Epochs$_ do

25 $\Phi\leftarrow\Phi-\tau\nabla_{\Phi}\mathcal{L}(\mathcal{D}^{Aug};\Phi)$

Algorithm 1 SAGE-HB-Net

###  III-B Fine Tuning

In the second stage, known as “Fine-Tuning,” the BS is deployed in a specific
new environment and it starts the unsupervised fine-tuning. Firstly, the BS
initializes a new set of model parameters ($\Phi$) with the parameters
downloaded from the backbone model ($\Theta$) in the cloud, as illustrated in
Fig 1. Next, the Data Augmentation step is performed. Let $\mathcal{D}$ be a
dataset, where each element with index $d$ corresponds to a matrix
$\mathbf{H}^{(d)}$ of size $N_{T}\times N_{U}$, and in turn each column of
$\mathbf{H}^{(d)}$ corresponds to a CSI vector $\mathbf{h}_{u}^{(d)}$ for user
$u$. The flattened dataset $\mathbf{F}$ is defined as an $N_{T}\times
N_{U}|\mathcal{D}|$ matrix obtained by stacking the columns of all the matrix
elements $\mathbf{h}^{(i)}$ of $\mathcal{D}$, that is

$\mathbf{F}=\begin{bmatrix}\mathbf{h}_{1,:}^{(1)}&\mathbf{h}_{2,:}^{(1)}&\ldots&\mathbf{h}_{N_{U},:}^{(1)}&\mathbf{h}_{1,:}^{(2)}&\ldots&\mathbf{h}_{N_{U},:}^{(|\mathcal{D}|)}\end{bmatrix}\,.$

The BS randomly selects subsets of users’ CSI ($S$) from $\mathbf{F}$,
effectively creating a new dataset ($D^{Aug}$). Since user positions are
independent and do not affect one another’s channels, by combining different
channels of individual users, we are able to quickly grow the multi-user
dataset and cover a wider range of possible scenarios. The maximum value of
the data augmentation size is $m=\binom{|\mathcal{D}|N_{U}}{N_{U}}$. Finally,
in the parameter update step, the model parameters ($\Phi$) in the BS are
updated using stochastic gradient descent with unsupervised learning. The
gradients of the loss function with respect to the augmented data
($\nabla_{\Phi}\mathcal{L}(\mathcal{D}^{Aug};\Phi)$) guide the parameter
updates. This fine-tuning process ensures that the model is optimized and
ready to perform effectively in the new deployment environment.

Figure 1: SAGE-HB-Net Backbone Training Procedure and Deployment Across
Diverse Base Stations

###  III-C SAGE-HB-Net Architecture

Our DNN architecture for hybrid beamforming is illustrated in Figure 2. This
design is customized to the specific requirements of our problem. Unlike
previous unsupervised learning models for hybrid beamforming that use the real
and imaginary parts of channel data as inputs [5], we find that utilizing the
amplitude and phase of the channels as the DNN inputs yields superior
performance, particularly when dealing with varying antenna configurations.
The architecture comprises three consecutive CNN layers with $128$ channels,
followed by three fully connected layers (FC) with $1024$ neurons. These
components collectively enable the model to effectively capture complex
channel interactions and patterns. Subsequently, we extract the analog
precoder and also real and imaginary parts of the digital precoder as output.
The architecture incorporates ReLU activation functions, 50% dropout layers to
prevent overfitting, and batch normalization for enhanced convergence and
stability in adapting to mMIMO environments.

Figure 2: Proposed DNN architecture for Hybrid Beamforming

##  IV Transfer Learning Numerical Results

In order to benchmark as accurately as possible the adaptability of the DNN
models, we model the deployment scenario using a ray-tracing environment,
while only statistical models are allowed to be used for backbone training.
The performance of SAGE-HB-Net and the baseline solutions was numerically
evaluated using the PyTorch DL framework. In the process of training a
backbone model, a batch size of $1000$ was employed to process data
efficiently. During backbone training, the learning rate parameters are set to
$\alpha=10^{-4}$, $\epsilon=10^{-5}$, and $\beta=1$. We use $\tau=10^{-3}$ as
a fine-tuning learning rate.

###  IV-A Backbone Model Training

Our backbone training dataset is constructed using statistical channel models.
To ensure the model’s adaptability and robustness, we introduce diversity in
our training data by considering various source domains. In practical base
station scenarios, the antenna configuration may vary widely due to spatial
constraints, user distribution, or hardware limitations. In this phase, the
selection of specific antenna configurations serves a strategic purpose aimed
at benchmarking the adaptability of the backbone model. We create a training
set that encourages the model to generalize effectively across diverse antenna
array responses, but we specifically omit configurations that will be tested
at deployment time and, in particular, avoid any two-dimensional
configurations. Excluding certain configurations ensures that the model
encounters scenarios it has not seen before. The chosen configurations are
$\textbf{n}\in\\{[1,1,64],[1,64,1],[64,1,1],[4,4,4]\\}$.

We further expand our source domains by introducing different path loss
exponent values to enhance the model’s ability to capture the nuances of
signal attenuation and propagation in various environments, mirroring real-
world scenarios. Specifically, we consider four path loss exponents
$\gamma\in\\{1.3,1.5,1.7,2.0\\}$. Thus, in total, the source domain set
contains 16 domains.

In addition to the diverse antenna array responses and path loss exponents,
our statistical channel models incorporate integer-varying base station
heights. Specifically, the parameter $\ell_{BS}$ in equation (5) is a random
variable uniformly sampled once from the set $\\{6,7,\dots,12\\}$ for each
element of the dataset, ensuring variability in base station height and
contributing to the model’s exposure to a broad range of scenarios for
enhanced adaptability and robust performance in mMIMO environments with
different base station configurations.

###  IV-B Deployment Scenario

Scenario “O1-$28$ GHz” of the deepMIMO channel model [14] was used in the
generation of a dataset for the evaluation. This scenario involves placing
several users’ locations randomly in streets bounded by constructions. It
involves the third BS with height of 6m and equipped with $N_{\sf{T}}=64$
antennas and $N_{\sf{RF}}=8$ RF chains serving $N_{\sf{U}}=4$ users randomly
located in a dedicated area (in deepMIMO channel model
$\texttt{active\\_user\\_first}=400$ and
$\texttt{active\\_user\\_last}=1400$). This unseen scenario provides a
realistic environment for assessing the adaptability and performance of the
proposed SAGE-HB-Net algorithm in a mMIMO system. Note that all methods were
optimized independently, and we applied the data augmentation method across
all methods. This ensures fairness and comparability in our study. In
addition, all DL-based methods were evaluated on the same network
architecture. The training sets used for backbone training and fine-tuning
were the same across all DL-based methods.

TABLE II: Impact of Antenna Configuration Variability on SAGE-HB-Net’s Zero
Shot sum rate (b/s/Hz), Noise power: $-\log\sigma^{2}=13$

|  | Deployment
---|---|---
|  | $\mathbf{n}_{x}$ | $\mathbf{n}_{y}$ | $\mathbf{n}_{z}$ | $\mathbf{n}_{xy}$ | $\mathbf{n}_{xz}$ | $\mathbf{n}_{yz}$ | $\mathbf{n}_{xyz}$
SAGE-HB-Net | 11.63 | 10.82 | 11.24 | 9.62 | 9.45 | 9.79 | 10.36
near-optimal [7] | 22.36 | 21.26 | 18.64 | 20.91 | 19.18 | 18.54 | 18.37

Table II reports a significant improvement in zero-shot performance for SAGE-
HB-Net compared to the non-generalized models discussed in Table I. In Table
I, the model exhibited a substantial degradation in zero-shot sum rate,
indicating challenges in adapting to diverse antenna array configurations. In
contrast, Table II highlights the enhanced adaptability of SAGE-HB-Net,
consistently achieving high sum rates across various deployment scenarios with
varying antenna array response vectors. Although SAGE-HB-Net achieves
approximately half of the near-optimal sum rate for zero-shot performance, its
ability to maintain robust performance in diverse settings underscores its
effectiveness in practical mMIMO deployments. This improvement emphasizes the
ability of SAGE-HB-Net to address challenges related to antenna configuration
variability, showcasing its potential for real-world applications.

Figure 3: Fine-tuning on DeepMIMO with $\textbf{n}=[1,8,8]$,
$|\mathcal{D}|=10^{6}$, and $-\log\sigma^{2}=13$.

In Figure 3, we present an evaluation of the adaptability of our proposed
solution in comparison to other state-of-the-art adaptive methods. The near-
optimal performance baseline is evaluated by training the DNN of [7] on
DeepMIMO for $200$ epochs. Additionally, PE-AltMin [17] is included in the
comparison to provide an example of the performance achievable with
conventional optimization-based methods. Adaptive methods include Model
Agnostic Meta-Learning (MAML) [9], and Deep-All, which is a training approach
in which we aggregate datasets from all source domains to build the backbone
model. An additional benchmark, “Rnd-initialized,” assesses adaptability to
new deployment scenarios without pre-trained knowledge.

The results show that SAGE-HB-Net achieves substantially higher zero-shot sum-
rates, surpassing the sum-rates of MAML, Deep-All, and Rnd-initialized
approaches by factors of approximately $1.9$, $3.2$, and $11.6$ times,
respectively. Furthermore, the convergence speed of the proposed solution
stands out as another notable advantage. We see that in just $15$ epochs,
SAGE-HB-Net approaches the near-optimal performance. In contrast, MAML
requires around 30 epochs, Deep-All approximately $35$ epochs, and Rnd-
initialized about $45$ epochs to reach the same sum-rate performance. In
unseen environments, SAGE-HB-Net shows enhanced adaptability and efficiency in
harnessing the potential of DL-based beamforming.

Figure 4:  Zero Shot performance for different noise powers on DeepMIMO, with
$\textbf{n}=[1,8,8]$.

We assess the zero-shot sum-rate performance of our proposed method across
various noise power values ($\sigma^{2}$), as depicted in Figure 4. Our
results indicate that our proposed solution outperforms other DL-based
approaches in terms of zero-shot sum-rate performance. It is important to note
that while the initial sum-rate of our method may be lower than that of PE-
AltMin, this observation is specific to the zero-shot sum-rate scenario. As
shown in Figure 3, after a few fine-tuning iterations, the sum-rate result of
all DL-based methods surpasses that of PE-AltMin.

Figure 5: Performance after 20 epochs of fine-tuning on DeepMIMO as a function
of the fine-tuning dataset size, for $\textbf{n}=[1,8,8]$,
$-\log\sigma^{2}=13$, and $m=10^{6}$.

To evaluate the efficacy of our data augmentation technique during fine-
tuning, we compared sum-rate performance against the number of samples needed
for fine-tuning in the deployment scenario, as illustrated in Figure 5.
Remarkably, our approach achieves $92\%$ of near-optimal sum-rate with just
$20$ samples, outperforming MAML, which requires $1000$ samples. Both Deep-All
and Rnd-initialized, even with $10^{6}$ samples, fail to reach the performance
of SAGE-HB-Net achieved with only $20$ samples. Moreover, we notice that the
DNN overfits the training data if we do not use data augmentation for
$|\mathcal{D}|$ is less than $10^{6}$. This shows the importance of data
augmentation for adapting to unknown environments. In the cases when
$|\mathcal{D}|=10$ or $15$, as the maximum value of ($m$) is less than
$10^{6}$, then we sample the user’s CSI with replacement, which acts as a
regularizer and prevents overfitting. It is crucial to note that, despite
leveraging unsupervised learning that solely relies on CSI for DNN training,
acquiring each sample in the deployment scenario demands time, resources, and
energy for channel estimation, highlighting not only the computational
efficiency of our method but also its practical advantage in reducing the time
required for data collection and model fine-tuning in real-world mMIMO
environments.

##  V Conclusion

The promise of DL in mMIMO systems for beamforming faces significant
challenges in adapting to real-world scenarios. These challenges include the
need for extensive data for adaptability, generalization, and fine-tuning
issues. In this paper, we have shown how to configure MLDG for HBF and enable
it to generalize to various antenna configurations, path loss exponent
factors, and BS heights. Our proposed approach combines MLDG with innovative
data augmentation techniques during the fine-tuning process. Our approach not
only accelerates the adaptation of DL-based beamforming solutions to new
channel environments but also dramatically reduces the data requirements for
fine-tuning. This advancement enhances the practicality and efficiency of
conventional DL-based beamforming solutions for mMIMO systems. Our simulation
results provide strong evidence of the effectiveness of our approach.

## Acknowledgement

This work was supported by Ericsson - Global Artificial Intelligence
Accelerator AI-Hub Canada in Montréal and jointly funded by NSERC Alliance
Grant 566589-21 (Ericsson, ECCC, InnovÉÉ).

## References

  * [1] J. G. Andrews, S. Buzzi, W. Choi, S. V. Hanly, A. Lozano, A. C. Soong, and J. C. Zhang, “What will 5G be?” _IEEE J. on Sel. Areas in Commun._ , vol. 32, no. 6, pp. 1065–1082, Jun. 2014.
  * [2] A. F. Molisch, V. V. Ratnam, S. Han, Z. Li, S. L. H. Nguyen, L. Li, and K. Haneda, “Hybrid beamforming for massive MIMO: A survey,” _IEEE Commun. Mag._ , vol. 55, no. 9, pp. 134–141, 2017.
  * [3] H. Huang, G. Gui, H. Sari, and F. Adachi, “Deep learning for super-resolution DOA estimation in massive MIMO systems,” in _2018 IEEE 88th Veh. Technol. Conf. (VTC-Fall)_ , vol. 2018-Augus, 2018, pp. 1–5.
  * [4] A. Alkhateeb, S. Alex, P. Varkey, Y. Li, Q. Qu, and D. Tujkovic, “Deep learning coordinated beamforming for Highly-Mobile millimeter wave systems,” _IEEE Access_ , vol. 6, pp. 37 328–37 348, 2018.
  * [5] H. Hojatian, J. Nadal, J.-F. Frigon, and F. Leduc-Primeau, “Unsupervised deep learning for massive MIMO hybrid beamforming,” _IEEE Trans. on Wireless Commun._ , vol. 20, no. 11, pp. 7086–7099, 2021.
  * [6] A. M. Elbir and A. K. Papazafeiropoulos, “Hybrid Precoding for Multiuser Millimeter Wave Massive MIMO Systems: A Deep Learning Approach,” _IEEE Trans on Veh. Technol._ , vol. 69, no. 1, pp. 552–563, Jan. 2020.
  * [7] H. Hojatian, J. Nadal, J. Frigon, and F. Leduc–Primeau, “Flexible unsupervised learning for massive MIMO subarray hybrid beamforming,” in _2022 IEEE Global Commun. Conference_ , 2022, pp. 3833–3838.
  * [8] Q. Wang, K. Feng, X. Li, and S. Jin, “PrecoderNet: Hybrid beamforming for millimeter wave systems with deep reinforcement learning,” _IEEE Wireless Commun. Letters_ , vol. 9, no. 10, pp. 1677–1681, 2020.
  * [9] Y. Yuan, G. Zheng, K.-K. Wong, B. Ottersten, and Z.-Q. Luo, “Transfer learning and meta learning-based fast downlink beamforming adaptation,” _IEEE Trans. on Wireless Commun._ , vol. 20, no. 3, pp. 1742–1755, 2020.
  * [10] J. Zhang, Y. Yuan, G. Zheng, I. Krikidis, and K.-K. Wong, “Embedding model-based fast meta learning for downlink beamforming adaptation,” _IEEE Trans. on Wireless Commun._ , vol. 21, no. 1, pp. 149–162, 2021.
  * [11] J. Xia and D. Gunduz, “Meta-learning based beamforming design for MISO downlink,” in _2021 IEEE International Symposium on Information Theory (ISIT)_. IEEE, 2021, pp. 2954–2959.
  * [12] S. Zhang, J. Xu, W. Xu, N. Wang, D. W. K. Ng, and X. You, “Data augmentation empowered neural precoding for multiuser MIMO with MMSE model,” _IEEE Commun. Lett._ , vol. 26, no. 5, pp. 1037–1041, 2022.
  * [13] S. Mukherjee and R. Chopra, “Performance analysis of cell-free massive MIMO systems in LoS/NLoS channels,” _IEEE Trans. on Veh. Technol._ , vol. 71, no. 6, pp. 6410–6423, 2022.
  * [14] A. Alkhateeb, “DeepMIMO: A generic deep learning dataset for millimeter wave and massive MIMO applications,” in _Proc. of Inform. Theory and App. Workshop (ITA)_ , San Diego, CA, Feb. 2019, pp. 1–8.
  * [15] Z. Shen, J. Liu, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui, “Towards out-of-distribution generalization: A survey,” _arXiv:2108.13624_ , 2021.
  * [16] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, “Learning to generalize: Meta-learning for domain generalization,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [17] X. Yu, J.-C. Shen, J. Zhang, and K. B. Letaief, “Alternating minimization algorithms for hybrid precoding in millimeter wave MIMO systems,” _IEEE J. of Sel. Topics in Signal Process._ , vol. 10, no. 3, pp. 485–500, 2016.


11institutetext: DFKI GmbH, Alt-Moabit 91c, 10559 Berlin, Germany
22institutetext: 3pc GmbH Neue Kommunikation, Prinzessinnenstraße 1, 10969
Berlin, Germany
Corresponding Author: Georg Rehm – 22email<EMAIL_ADDRESS>
# Towards Discourse Parsing-inspired
Semantic Storytelling

Georg Rehm 11 Karolina Zaczynska 11 Julián Moreno-Schneider 11
Malte Ostendorff 11 Peter Bourgonje 11 Maria Berger 11
Jens Rauenbusch 22 André Schmidt 22 Mikka Wild 22

###### Abstract

Previous work of ours on Semantic Storytelling uses text analytics procedures
including Named Entity Recognition and Event Detection. In this paper, we
outline our longer-term vision on Semantic Storytelling and describe the
current conceptual and technical approach. In the project that drives our
research we develop AI-based technologies that are verified by partners from
industry. One long-term goal is the development of an approach for Semantic
Storytelling that has broad coverage and that is, furthermore, robust. We
provide first results on experiments that involve discourse parsing, applied
to a concrete use case, “Explore the Neighbourhood!”, which is based on a
semi-automatically collected data set with documents about noteworthy people
in one of Berlin’s districts. Though automatically obtaining annotations for
coherence relations from plain text is a non-trivial challenge, our
preliminary results are promising. We envision our approach to be combined
with additional features (NER, coreference resolution, knowledge graphs).

###### Keywords:

Semantic Storytelling Natural Language Processing Discourse Parsing Rhetorical
Structure Theory Penn Discourse TreeBank

##  1 Introduction

Cultural institutions such as museums, archives or libraries often rely on
public funding and therefore need to communicate their value to the public
constantly. One successful way to achieve this goal is to employ storytelling,
which can be defined as creating emotional, interactive narratives in a
digital format. Storytelling enables cultural institutions to make use of
their digitized collections, demonstrating their relevance and reaching out to
new audiences. Due to the extremely large amounts of available digital
content, the curation of stories is typically performed by human knowledge
workers. This calls for automated procedures. Such procedures should 1) label
the content for several types of metadata semi-automatically, allowing for
relevant categorisation. And 2) process the individual content pieces to
present the information contained in them to a knowledge worker in an
intuitive way. Since cultural organisations are often unlikely to be able to
face this challenge on their own, we develop a platform supporting this use
case in the the technology transfer project QURATOR. Our goal are semi-
automatic technologies that keep the human in the loop and allow for fast,
efficient and intuitive exploration of large and highly domain-specific data
sets. Relating events into a schematic structure, i. e., storytelling, and
ordering them, e. g., in terms of topic, locality or causal or temporal
relationships, aid humans in finding meaningful patterns in data [3].

In earlier work, we described approaches to Semantic Storytelling making use
of Named Entity Recognition (NER) and Event Detection [16, 24, 23]. In this
article, we explore ways to present a knowledge worker the semantic structure
between text segments in an incoming text collection, making it possible to
find interesting and surprising connections and information inside texts
regarding a predefined topic. We focus on means of relating text segments to
each other by borrowing from frameworks for the processing of coherence
relations. From Rhetorical Structure Theory (RST) [12] we borrow the idea that
larger sequences of texts (i. e., non-elementary discourse units) are related,
moving beyond the shallow parsing of individual coherence relations. From the
Penn Discourse TreeBank (PDTB) [20] we use the sense inventory and perform a
series of experiments, relating text segments according to the four top-level
classes of the PDTB sense hierarchy. The experiments are centered around the
use case “Explore the Neighbourhood!”. This tool, currently in development, is
an urban exploration app that makes uses of documents on the Berlin district
of Moabit. It allows users to follow stories, created by an editor semi-
automatically, while exploring the district both physically and digitally.

The remainder of this paper is structured as follows. Section 2 reviews
relevant work, in particular, approaches using discourse relations in text.
Section 3 explains the use case in more detail. Section 4 provides a technical
definition, while Section 5 outlines the experiments on the data set we
created. Finally, Section 6 provides a summary and suggests directions for
future work.

##  2 Related Work

The act of storytelling and the resulting stories, can be seen as a strategy
to uncover meaningful patterns in the world around us [3]. At the core of
research on classical narratology, essential to storytelling, is the
uncovering of the rules that underlie this strategy, or at least the ways to
best achieve the goal. Early work on narratology is described in [1], defining
a narrative as a discourse following a plot structure that has a chronological
and logical event order. More recently, [5] applied this definition of plot
structure to (chrono)logically ordered events. Another line of work on
narratology is represented by the work of [21], who analyzes the basic,
irreducible, structural elements of Russian folk tales. More recently, Propp’s
work was used by [33] for their story detection and generation systems. The
same authors, in [32], make use of another field of research related to text
coherence, namely that of the processing of coherence relations. They apply
the work of [29] on hierarchical discourse relations to work out how
paragraphs behave when being used as discourse-structural units in news
articles, with the ultimate goal of understanding the importance and temporal
order of story items. Our work follows a similar approach, but uses PDTB sense
hierarchy labels. The PDTB [20] is an (English) corpus of Wall Street Journal
articles (a subsection of the Penn TreeBank [13]) annotated for individual
discourse relations. We adopt the PDTB sense hierarchy, because it is the
single largest corpus annotated for coherence relations and therefore the
corpus best facilitating machine-learning based approaches. Due to the shallow
nature of the PDTB framework (it only annotates individual relations, without
making commitment to larger text structure, or mutual importance or
relevance), we additionally source from RST [12], particularly the notion of
nuclearity. In RST, a text is divided into _Elementary Discourse Units_ ,
which are joined together, forming either a _mono-nuclear_ relation (with one
unit being the more prominent, important or relevant _nucleus_ and the other,
less prominent unit being the _satellite_) or a _multi-nuclear_ relation. It
is this notion of prominence, or relative importance to the storyline at hand,
that we adopt from RST.

With regard to application-driven approaches, much work has been done on the
final, surface realisation aspect of text generation [7, 8]. An approach
resembling more closely ours is described by [17], who use dependency parsing
in combination with discourse relations to determine sentence relations. In
our approach, however, in addition to finding relevant articles for the user,
we want to classify the type of relation the articles in question have to each
other.

In our own previous work we described tools supporting the processing and
generation of digital content with a strong industry focus, as is equally the
case in the current context of the QURATOR project. The functionality of the
curation technology platform is explained in [23]. [24] presents an example of
this platform applied to the use case of a personal communication archive, i.
e., a collection of approx. 2,800 letters exchanged between the German
architect Erich Mendelsohn and his wife Luise between 1910 and 1953. From
this, we extracted, i. a., named entities, temporal expressions and events,
combined these and used them to track and visualise the movement (across the
globe) of Erich and Luise. Additional prototypes are presented in [15] and
[22].

##  3 Industry Needs and Applications: The “Explore the Neighbourhood!” Use
Case

“Explore the Neighbourhood!” is a concept for a mobile app, which engages
urban explorers in semi-automatically created stories, making use of digitized
cultural collections. Moabit is a district in Berlin and was chosen due to its
rich history and lively present. Such an app could be made available by
museums, cities or municipalities, tourist information offices or local
marketing campaigns. End users might be tourists, pupils studying in or
visiting the neighborhood, or residents. Value is created for all parties by
entertaining and educating users whilst communicating the district’s or
cultural institution’s relevance. The app offers both curated and generated
stories. While in a final concept of “Explore the Neighbourhood!” these
differences might not be noticed by the end user, in the following we will
present each approach separately to describe the concept more precisely. We
plan to fully integrate the approach described in Section 4.

###  3.1 Curated Stories

Upon launching the app a set of interactive stories is offered to the end user
who can influence the story’s direction, depth, and pace. Nevertheless, it
still contains significant plot elements curated by an editor. The curation
process requires the editor to define several storylines in a customised tool,
which contains search capabilities and a recommendation system (Figure 1),
both of which help surface relevant content for each step along a story path.
Such a tool is made possible due to rich metadata which allow queries such as
“poems describing Berlin in a praising tone” (text classification and analysis
detecting locations and sentiment) or “photos showing Kurt Tucholsky next to a
church” (image classification and analysis detecting people and objects, in
this case churches). Figure 1 shows the user interface of such a tool.

Figure 1: The smart authoring environment

Curated stories can be published to the app (Figure 2(a)). Stories may contain
geographical points of interest within Moabit which are connected through an
overall story arch, such as a biography. The exemplary stories depicted in
this article follow the biography of Kurt Tucholsky (Figure 2(b)), a German-
Jewish journalist and writer born in Moabit in 1890. The stories contain
locations, historic photos and maps, scanned original works and editorial
content.

(a) Description of Kommune 1

(b) Kurt Tucholsky’s biography

Figure 2: The app allows the exploration of different aspects of Moabit

The existence of several storylines within a story, as well as several stories
in parallel, allows for connections to be forged. These connections can be
based on common topics, locations, or other parameters that support a
consistent and emotional narrative. Users can follow one path through a story,
choose to dive deeper into certain aspects of it, e. g., Kurt Tucholsky
(Figure 2(b)), change their perspective onto a topic by exploring alternative
stories, or switch to a completely different, yet connected story. The
consumable stories are linked in a network and limited only by the amount of
pieces of information and the size of the network created by the editor, who
can extend it continuously.

###  3.2 Generated Stories

Unlike curated ones, generated stories are created entirely by a storytelling
engine. This is made possible due to a set of well-chosen parameters which
influence the automatic selection and connection of content. These parameters
are defined by several factors:

  * •

a chosen topic (initiated through a keyword or phrase)

  * •

the type of story being told (such as biography or travel guide),

  * •

users’ preferences (such as available time, current sentiment, preferred mode
of travel),

  * •

users’ behavior (such as current location, walking speed, orientation).

Based upon the factors listed above, “Explore the Neighbourhood!”
automatically generates a story by selecting the right content based on its
rich metadata. The end result, which is the story consumed by the users, may
not look so different from editor-curated stories. Nevertheless, since
generating a story happens in real-time, it constantly adapts to users’
choices, which creates a more personal and more interactive experience.

##  4 Semantic Storytelling: Technical Description

One of the goals of our Semantic Storytelling system is to aid knowledge
workers in selecting relevant pieces of content, e. g., the app editor who
wants to curate stories for the app. Following the prototype of the “Explore
the Neighbourhood!” app (Section 3), this section describes the technical
details of the back-end.

Let us assume the following situation. A user is visiting a city and wants
information about a topic $T$ regarding the neighbourhood. The goal of the
curation prototype is, then, to identify and to suggest new content for the
app that can be included in the user’s tour. To do so, we first have to
initialise the topic $T$, for example, as a sentence, keyword or named entity.
Next up, the tool has to identify if, for example, a document in a collection
or a web page is _relevant_ for topic $T$, and, if so, if it is _important_
for $T$. Finally, we identify the _semantic relation_ between incoming texts
and the provided topic $T$, which could be, among others, _background_ ,
_cause_ , _contrast_ , _example_ etc. In the following, we describe these
steps in more detail (Figure 3).

Figure 3: Architecture of the Semantic Storytelling approach

####  4.0.1 Step 1: Determine the Relevance of a Segment for a Topic

The approach starts with a topic $T$, instantiated through a text segment such
as a complete document, a headline or a named entity. To identify content
pieces relevant for $T$, we process incoming textual content, like a self-
contained document collection, a systematically compiled corpus or a knowledge
base.

For each piece of content, we need to decide whether its topic is relevant for
$T$, which can be computed in various ways. We can employ topic modeling (LDA,
LSA) or, without explicitly modeling topics, we can also perform pair-wise
comparisons of document similarity. Document pairs with a high similarity
score are assumed to cover the same topic, therefore, we start with the seed
document $d_{s}$ of which we know that it represents $T$ and measure its
similarity to other candidates. To compute semantic similarity, documents are
represented as numerical vectors. Classical methods like bag-of-words or tf-
idf encode documents as sparse vectors [25], while neural methods (word2vec,
sent2vec, doc2vec, see e. g., [14, 19, 27]) produce dense representations. In
both cases, cosine similarity can be used to compute the similarity of the
document vectors.

####  4.0.2 Step 2: Determine the Importance of a Segment

If we have determined all documents $d$ which are related to $T$, we need to
determine the importance of $d$ (or its segments or sentences) with regard to
$T$. There is no off-the-shelf approach to determine the importance of a
segment with regard to a topic, but various cues and indicators can
potentially be exploited. One way of doing this is to borrow from RST,
especially the notion of nuclearity. Constructing an RST tree involves
decisions with regard to the status of text segments including their discourse
relation to other segments and also regarding their role as a _nucleus_ (the
important core part of a relation) or _satellite_ (the contributing part of a
relation) in the context of a specific discourse relation. Two segments are
assigned either a satellite-nucleus (S-N), nucleus-satellite (N-S) or a
nucleus-nucleus (N-N) structure. This sub-task can be done in isolation [9,
28], or in conjunction with the relation classification task [11]. When
performed iteratively, this pair-wise classification can result in a set of
most important segments regarding $T$. Another way of determining topical
importance is to treat it as a segment-level question answering task. Given a
document $d$ consisting of text segments $(t_{1},t_{2},\dots t_{n})$, the aim
is to find the segment $t_{i}$ that contains the answer to the input question
(i. e., topic $T$). Transformer language models have achieved state-of-the-art
results for question answering [6], suggesting that those model architectures
would be beneficial for storytelling.

####  4.0.3 Step 3: Semantic or Discourse Relation between two Segments

After having established the relevance and relative importance, we proceed
with determining the semantic or discourse relation that exists between the
text segments and topic $T$. Our initial experiments are based on the PDTB due
to its considerably larger size with more than 1.1 million tokens over the
RST-Discourse TreeBank [4] with approx. 200k tokens. We adopt the PDTB’s sense
hierarchy, which comprises four top-level classes, 16 types on the second
level and 23 sub-types on the third. For now, our experiments are based on the
top-level senses, _Temporal_ , _Contingency_ , _Comparison_ , _Expansions_ ,
and an additional label, _None_.

##  5 Experiment for “Explore the Neighbourhood!”

In this section, we describe our first experiments, which aim to explore the
suitability of the approach and helps us gain an understanding of what we can
achieve in the long run. We concentrate on step 3, therefore, we created a
data set of crawled web documents about the Berlin district Moabit, and
implemented initial experiments to classify discourse relations between text
segments inside the data set. We would like to show a comparison with similar
tools, but to the best of our knowledge, there are no similar tools that are
extracting semantic relations through intra-document text segments (using
PDTB).

###  5.1 Data Set

The data set is composed of documents containing information and stories
connected to the district of Moabit in Berlin. We are in the first stages of
developing this data set. In the long term, the idea is to put together a much
larger collection of documents focused on Moabit so that it can be used for
the Semantic Storytelling prototype. We used the focused crawler
Spidey111https://github.com/vikrambajaj22/Spidey-Focused-Web-Crawler, which
returns a list of URLs from websites which are based on a set of predefined
query terms. We manually defined 28 queries about interesting places,
buildings, or persons connected to Moabit. Some of these terms are _Moabit_ ,
_Moabit gentrification_ , _Kleiner Tiergarten_ , _Kulturfabrik Moabit_ ,
_Berlin Central Station_ and _Kurt Tucholsky_. After obtaining the website
URLs, we crawl and boilerplate the content of the pages and their
metadata222We use Newspaper3k, see https://github.com/codelucas/newspaper. The
resulting data set is composed of slightly more than 100 documents that have
been filtered manually in a second step.

###  5.2 Classifiers for Discourse Relation between Text Segments

Our aim is to extract discourse relations from texts and so, being able to
extract relevant content from a text collection and, in the longer run, to
find new storylines composed of semantically related parts of different text
segments taken from the collection. We train a relation sense classifier on
PDTB2 [20] and apply it on two pieces of content. For training, we use the two
arguments of a relation, but at a later point we deploy it using individual
sentences. We argue that the sentence-level is the most appropriate level to
use as input for our classifier (as opposed to the shorter token or phrase
level, or the longer paragraph level) and that the discrepancy between
argument shapes and typical sentence lengths (itself very much dependent on
the domain) is tolerable.

####  5.2.1 Classifier Model

Classifying the discourse relation between sentence pairs requires a semantic
understanding of the sentences. We encode the text as deep contextual
representations with a language model based on the Transformer architecture
[30]. To be precise, the pre-trained language model from DistilBERT [26], a
distilled version of Bidirectional Encoder Representations from Transformers
[6] is used333We use the PyTorch implementation by HuggingFace [31].. BERT
performs well for document classification tasks [18].

To classify the relation between two texts, we employ a Siamese architecture
[2]. In contrast to a classical Siamese model, in which a binary classifier is
employed on the output of the two identical sub-networks, we feed the sub-
network output into a multi-label classifier, as illustrated in Figure 4.

[capbesideposition=right,center,capbesidewidth=0.5]figure[]

Figure 4: The architecture of the Siamese BERT model for the classification of
discourse relations between two text segments $d_{1}$ and $d_{2}$. The output
of the classification layer $\hat{y}$ holds the predicted semantic relation
according to the top-level PDTB2 senses.

Text snippets $d_{1}$ and $d_{2}$ are inputs to the classifier. BERT’s
architecture consists of six hidden layers, each layer consists of 768 units
(66M parameters; DistilBERT). BERT is used in a Siamese fashion such that
$h_{i}=\text{BERT}(d_{i})$ is the encoded representation of text $d_{i}$ where
$h_{i}$ is the last hidden state of the last BERT layer. The final feature
vector $x_{f}$ is a combined concatenation of the text representations:

$x_{f}=[h_{1};h_{2};|h_{1}-h_{2}|;h_{1}*h_{2};\frac{h_{1}+h_{2}}{2}]$ (1)

On top of the concatenation, we implement a Multi-Layer Perceptron (MLP). The
MLP consists of two fully-connected layers, $\mathcal{F}_{f}(\cdot)$ and
$\mathcal{F}_{g}(\cdot)$, where each layer has 100 units and
$\text{ReLU}(\cdot)$ is the activation function. The discourse relation
$\hat{y}$ is classified on the basis of the feature vector $x_{f}$ as follows:

$\hat{y}=\sigma(\mathcal{F}_{f}(\text{ReLU}(\mathcal{F}_{d}(x_{f}))))$ (2)

The logistic softmax function $\sigma(\cdot)$ generates probabilistic multi-
label classifications. The dimension of $\hat{y}$ corresponds to the number of
classification labels, which are the four top-level PDTB2 senses (_Temporal_ ,
_Contingency_ , _Comparison_ , _Expansions_) and one additional dimension
(_None_).

####  5.2.2 Transfer Learning

Our target corpus of texts for “Explore the Neighbourhood!” does not include
any kind of annotated training data. Thus, we cannot use the data set to train
the classifier. Instead, we rely on the PDTB2 data set. Training is performed
with batch size $b=16$, dropout probability $d=0.1$, learning rate
$\eta=2^{-5}$ (Adam optimizer) and 5 training epochs. These hyperparameters
are the ones proposed by [6] for BERT fine-tuning.

PDTB Relation |  Precision |  Recall |  F1-score | Support
---|---|---|---|---
_Comparison_ | 0.50 | 0.47 | 0.48 | 1598
_Contingency_ | 0.38 | 0.65 | 0.48 | 1582
_Expansion_ | 0.50 | 0.79 | 0.61 | 2993
_Temporal_ | 0.51 | 0.55 | 0.53 | 869
_None_ | 0.49 | 0.73 | 0.59 | 1078
Micro avg. | 0.47 | 0.67 | 0.55 | 8120
Macro avg. | 0.48 | 0.64 | 0.54 | 8120
Table 1: Results of training multi-class prediction based on PDTB2 data set in
a 80-20 train-test-split.

The results that are derived from a 80-20 train-test-split are shown in Table
1. For evaluation, we use the multi class metric F1-micro average, which
calculates the metrics globally by counting the total true positives, false
negatives and false positives to compute the average metric. In a multi-class
classification setup, micro-average is preferable if you suspect there might
be class imbalance. In the end, we achieve 0.55 micro average F1. Due to the
fact that we have not implemented features relating to the connective, our
classification performs lower than current state-of-the-art approaches.

###  5.3 First Experiment on Use Case Data Set and Discussion

Given the PTDB2-based classifier, we continue to find discourse relations
within the corpus containing documents for the “Explore the Neighbourhood!”
use case. As a preprocessing step, we first exclude all non-English documents
and group documents by topic based on the query terms for the focused crawler.
Next, we find document pairs among the topic groups (only semantically similar
document pairs are considered). More precisely, documents are represented as
tf-idf vectors and the cosine similarity of a document pair $d_{a}$ and
$d_{b}$ must be above a fixed threshold ($cosine(d_{a},d_{b})>0.15$). Our
classifier is trained to detect sentence-level relations, thus, we also split
the documents into sentences444We use pySBD, see
https://github.com/nipunsadvilkar/pySBD. After excluding all sentences with
less than five words, we end up with 96,796 sentence pairs that are passed to
the classifier.

| Documents | Discourse relations
---|---|---
| Topic | Segment A | Segment B | S. | Co. | Ct. | E. | T. | N.
1 |  Farin Urlaub Moabit |  In April 2012 they released another album “auch” (“also”) |  At the age of 16, Vetter went on a school trip to London, and returned home as a punk with dyed blonde hair. | .51 | .01 | .01 | .04 | .93 | .1
2 |  Uschi Obermaier |  In 1968 and 1969 Obermaier starred in Rudolf Thome’s first two feature films, “Detektive” and “Rote Sonne” (“Red Sun”). |  She played maracas in the band Amon Düül, aka Amon Düül I, on two albums: Collapsing (1970, released by Metronome) and Disaster (1972, released by BASF Records [de]). | .39 | .0 | .0 | .96 | .03 | .02
3 |  AEG turbine factory |  It is an influential and well-known example of industrial architecture. |  However, when it came to AEG’s public image and public perception, the focus remained on Peter Behrens: the famous artist-cum-architect overshadowed the engineer. | .32 | .62 | .16 | .15 | .01 | .06
4 |  Kurt Tucholsky Moabit |  Admittedly, Tucholsky is seldom recognized as a philosopher. |  He saw himself as a left-wing democrat and pacifist and warned against anti-democratic tendencies – above all in politics, the military and justice – and the threat of National Socialism. | .25 | .45 | .28 | .19 | .01 | .08
5 |  Schult-heiss Brewery |  ”Good people drink good beer,” Hunter S Thompson once said, writing about a beverage that is considered to be typically German and is, of course, also popular in Berlin. |  Schultheiss is currently brewing far less beer than at the time of re-unification. | .16 | .32 | .19 | .42 | .01 | .06
Table 2: Manually evaluated examples. On the right, the table shows the
similarity score (S.) between sentences and the prediction scores for each
discourse relation (Co.=Comparison, Ct.=Contingency, E.=Expansion,
T.=Temporal, N.=None)

To get a first impression on the applicability of our approach, and to
motivate our next steps, we manually select five example sentence pairs to
evaluate them qualitatively. The first line of Table 2 shows an example where
the classifier correctly labels the discourse relation as _Temporal_ , most
likely because of the temporal markers included. In the second line, the
approach correctly identifies the discourse relation as an _Expansion_ , i.
e., segment B can be seen as an extension of the biography described in
segment A. Nevertheless, in other examples, the approach is often unable to
handle coreference. The classifier is often not detecting a discourse relation
between two segments, even if those segments reference the same entity, while
one segment uses a pronoun for the entity. By implementing a preprocessing
step with rudimentary coreference resolution we expect the classification to
improve significantly. The classifier predicts the label _Comparison_ often
when specific lexical markers, such as _however_ , _but_ or _while_ , appear
in segment B, like in example 3. Example 4 is an exception, where the
classifier predicts the relation _Comparison_ correctly without needing a
lexical marker, but, generally we observe that this dependency on lexical
features leads to wrong predictions. We see one reason in the fact that the
sentences are taken from different sources, and the lexical markers for the
discourse relation are therefore often missing, also even if semantically it
can be seen as a Comparison. This is the case in example 5, which is wrongly
predicted as an Extension while we interpret it as a Comparison (paraphrased
as ’Even if he is recognized as a philosopher, he saw himself as a political
activist’). On the other hand, in other examples, the lexical markers cause
false positives errors. Hence, future work will extent the number of
preprocessing steps to better group text segments which have the same content
and talk about the same entities, events or topics.

##  6 Conclusions

We describe first experiments in order to apply our Semantic Storytelling
approach to an industrial use case. This use case, “Explore the
Neighbourhood!”, makes it possible to interactively create a city guide with
adjusting interesting stories about a particular district built upon user-
dependent parameters, such as predefined topics, keywords, etc. The basic idea
is to automate storytelling by detecting discourse relations between texts
segments of different sources on the same topic, which makes it possible to be
able to detect and create new storylines extracted from a document collection.
We describe the different steps in order to create a corresponding processing
framework. In the experiment presented here, we focus on the third step of our
approach, the classification of discourse relations between segments. By
focusing more on steps one and two as described in Section 4, we will be able
to improve the results in further experiments. For example, we expect the
classification to improve significantly by using coreference resolution during
preprocessing. One way of improving the coreference resolution would be to
pretrain the classifier on the coreference task first [10]. As data sets are
still limited, we will expand the data set for our needs and create, in the
longer run, annotations to develop a gold standard.

## Acknowledgements

The research presented in this article is funded by the German Federal
Ministry of Education and Research (BMBF) through the project QURATOR
(Unternehmen Region, Wachstumskern, no. 03WKDA1A). http://qurator.ai

## References

  * [1] Bal, M.: Narratology: Introduction to the Theory of Narrative. 1985. Trans. by Christine van Boheemen. Toronto: University of Toronto Press (1985)
  * [2] Bromley, J., Bentz, J., Bottou, L., Guyon, I., Lecun, Y., Moore, C., Eduard Sackinger, Shah, R.: Signature Verification using a Siamese Time Delay Neural Network. International Journal of Pattern Recognition and Artificial Intelligence 7(4) (1993)
  * [3] Bruner, J.: The Narrative Construction of Reality. Critical Inquiry 18(1), 1–21 (1991)
  * [4] Carlson, L., Marcu, D., Okurowski, M.E.: RST Discourse Treebank, ldc2002t07 (2002), https://catalog.ldc.upenn.edu/LDC2002T07
  * [5] Caselli, T., Vossen, P.: The Event StoryLine Corpus: A New Benchmark for Causal and Temporal Relation Extraction. In: Proceedings of the Events and Stories in the News Workshop. pp. 77–86. ACL (2017)
  * [6] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: NAACL-HLT. pp. 4171–4186 (2019)
  * [7] Fan, A., Lewis, M., Dauphin, Y.: Hierarchical Neural Story Generation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 889–898 (2018)
  * [8] Fan, A., Lewis, M., Dauphin, Y.: Strategies for Structuring Story Generation. arXiv preprint arXiv:1902.01109 (2019)
  * [9] Hernault, H., Prendinger, H., duVerle, D.A., Ishizuka, M.: HILDA: A Discourse Parser Using Support Vector Machine Classification. Dialogue & Discourse 1(3), 1–33 (2010)
  * [10] Joshi, M., Levy, O., Zettlemoyer, L., Weld, D.: BERT for Coreference Resolution: Baselines and Analysis. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 5807–5812. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1588, https://www.aclweb.org/anthology/D19-1588
  * [11] Joty, S., Carenini, G., Ng, R.T.: CODRA: A Novel Discriminative Framework for Rhetorical Analysis 41-3 (2015)
  * [12] Mann, W.C., Thompson, S.A.: Rhetorical Structure Theory: Toward a Functional Theory of Text Organization. Text 8, 243–281 (1988)
  * [13] Marcus, M., Kim, G., Marcinkiewicz, M.A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., Schasberger, B.: The Penn Treebank: Annotating Predicate Argument Structure. In: Proceedings of the Workshop on Human Language Technology. pp. 114–119. HLT ’94, ACL (1994)
  * [14] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient Estimation of Word Representations in Vector Space. In: 1st International Conference on Learning Representations (2013)
  * [15] Moreno-Schneider, J., Bourgonje, P., Rehm, G.: Towards User Interfaces for Semantic Storytelling. In: Yamamoto, S. (ed.) Human Interface and the Management of Information: Information, Knowledge and Interaction Design, 19th International Conference, HCI International 2017 (Vancouver, Canada). pp. 403–421. Lecture Notes in Computer Science (LNCS), Springer (2017)
  * [16] Moreno-Schneider, J., Srivastava, A., Bourgonje, P., Wabnitz, D., Rehm, G.: Semantic Storytelling, Cross-lingual Event Detection and Other Semantic Services for a Newsroom Content Curation Dashboard. In: Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism. pp. 68–73. ACL (2017)
  * [17] Nie, A., Bennett, E., Goodman, N.: DisSent: Learning Sentence Representations from Explicit Discourse Relations. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4497–4510. ACL (2019)
  * [18] Ostendorff, M., Bourgonje, P., Berger, M., Moreno-Schneider, J., Rehm, G., Gipp, B.: Enriching BERT with Knowledge Graph Embeddings for Document Classification. In: Proceedings of the GermEval 2019 Workshop (2019)
  * [19] Pagliardini, M., Gupta, P., Jaggi, M.: Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 528–540 (2018)
  * [20] Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A., Webber, B.: The Penn Discourse Treebank 2.0. In: In Proceedings of LREC (2008)
  * [21] Propp, V.Y.: Morphology of the Folktale. Publication … of the Indiana University Research Center in Anthropology, Folklore, and Linguistics, University of Texas Press (1968)
  * [22] Rehm, G., He, J., Schneider, J.M., Nehring, J., Quantz, J.: Designing User Interfaces for Curation Technologies. In: Yamamoto, S. (ed.) Human Interface and the Management of Information: Information, Knowledge and Interaction Design, 19th International Conference, HCI International 2017 (Vancouver, Canada). pp. 388–406. Lecture Notes in Computer Science (LNCS), Springer (2017), part I
  * [23] Rehm, G., Moreno-Schneider, J., Bourgonje, P., Srivastava, A., Fricke, R., Thomsen, J., He, J., Quantz, J., Berger, A., König, L., Räuchle, S., Gerth, J., Wabnitz, D.: Different Types of Automated and Semi-Automated Semantic Storytelling: Curation Technologies for Different Sectors. In: Rehm, G., Declerck, T. (eds.) Language Technologies for the Challenges of the Digital Age: 27th International Conference, GSCL 2017, Berlin, Germany, September 13-14, 2017, Proceedings. pp. 232–247. Lecture Notes in Artificial Intelligence (LNAI), Gesellschaft für Sprachtechnologie und Computerlinguistik e.V., Springer (2018)
  * [24] Rehm, G., Moreno-Schneider, J., Bourgonje, P., Srivastava, A., Nehring, J., Berger, A., König, L., Räuchle, S., Gerth, J.: Event Detection and Semantic Storytelling: Generating a Travelogue from a large Collection of Personal Letters. In: Caselli, T., Miller, B., van Erp, M., Vossen, P., Palmer, M., Hovy, E., Mitamura, T. (eds.) Proceedings of the Events and Stories in the News Workshop. pp. 42–51. ACL (2017)
  * [25] Salton, G., McGill, M.: Introduction to Modern Information Retrieval. Computer Series, McGraw-Hill, New York (1983)
  * [26] Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. CoRR pp. 1–5 (2019)
  * [27] Selivanov, D., Wang, Q.: text2vec: Modern Text Mining Framework for R. Computer software manual](R package version 0.4. 0) (2016)
  * [28] Soricut, R., Marcu, D.: Sentence Level Discourse Parsing using Syntactic and Lexical Information. In: Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics. pp. 228–235 (2003)
  * [29] Van Dijk, T.A.: News as Discourse. Routledge (2013)
  * [30] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention Is All You Need. Advances in Neural Information Processing Systems 30 (Nips), 5998–6008 (jun 2017)
  * [31] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Brew, J.: HuggingFace’s Transformers: State-of-the-art Natural Language Processing (oct 2019)
  * [32] Yarlott, W.V., Cornelio, C., Gao, T., Finlayson, M.: Identifying the Discourse Function of News Article Paragraphs. In: Proceedings of the Workshop Events and Stories in the News 2018. pp. 25–33 (2018)
  * [33] Yarlott, W.V.H., Finlayson, M.A.: ProppML: A Complete Annotation Scheme for Proppian Morphologies. In: CMN. OASICS, vol. 53, pp. 8:1–8:19. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik (2016)