# DCASE 2021 TASK 3: SPECTROTEMPORALLY-ALIGNED FEATURES
FOR POLYPHONIC SOUND EVENT LOCALIZATION AND DETECTION

###### Abstract

Sound event localization and detection consists of two subtasks which are
sound event detection and direction-of-arrival estimation. While sound event
detection mainly relies on time-frequency patterns to distinguish different
sound classes, direction-of-arrival estimation uses magnitude or phase
differences between microphones to estimate source directions. Therefore, it
is often difficult to jointly optimize these two subtasks simultaneously. We
propose a novel feature called spatial cue-augmented log-spectrogram (SALSA)
with exact time-frequency mapping between the signal power and the source
direction-of-arrival. The feature includes multichannel log-spectrograms
stacked along with the estimated direct-to-reverberant ratio and a normalized
version of the principal eigenvector of the spatial covariance matrix at each
time-frequency bin on the spectrograms. Experimental results on the DCASE 2021
dataset for sound event localization and detection with directional
interference showed that the deep learning-based models trained on this new
feature outperformed the DCASE challenge baseline by a large margin. We
combined several models with slightly different architectures that were
trained on the new feature to further improve the system performances for the
DCASE sound event localization and detection challenge.

Index Terms—  DCASE, deep learning, spatial audio, feature extraction, sound
event localization and detection.

##  1 Introduction

Sound event localization and detection (SELD) has many applications in urban
sound sensing [1], wildlife monitoring [2], and surveillance [3]. SELD is the
problem of recognizing the sound class, as well as estimating the
corresponding direction of arrival (DOA), onset, offset of the detected sound
event [4]. Polyphonic SELD refers to cases where there are multiple sound
events overlapping in time. In 2019, the Challenge on Detection and
Classification of Acoustic Scenes and Events (DCASE) introduces a polyphonic
SELD task with only stationary sound sources [5]. The 2020 rendition sees an
introduction of moving sound sources [6]. In 2021, the SELD task additionally
introduces unknown directional interferences that further complicate the task
[7].

SELD is an emerging topic in audio processing. It consists of two subtasks,
which are sound event detection (SED) and DOA estimation (DOAE). These two
subtasks are mature research topics, and there exists a large body of
effective algorithms for SED and DOAE [8, 9]. Over the past few years, the
majority of the methods proposed for SELD have focused on jointly optimizing
SED and DOAE in one network. Hirvonen first formulated SELD as a multi-class
classification task where the number of output classes is equal to the number
of DOAs multiplied by the number of sound classes [10]. In 2018, Adavanne  _et
al._ pioneered a seminal work that used a single-input multiple-output
convolutional recurrent neural network (CRNN) model called SELDnet, which
jointly detects sound events and estimates the corresponding DOAs [4].

Because sound event detection mainly relies on time-frequency (TF) patterns to
distinguish different sound classes while direction-of-arrival estimation
relies on magnitude or phase differences between microphones to estimate
source directions, it is often difficult to jointly optimize these two
subtasks in a single network. To remedy this problem, Cao  _et al._ proposed a
two-stage strategy by training separate SED and DOA models [11], then using
the SED outputs as masks to select DOA outputs. This training scheme
significantly improved the SELD performance over the jointly-trained SELDnet.
Cao  _et al._ later proposed an end-to-end SELD network called Event
Independent Network (EIN) [12, 13] that used soft parameter sharing between
the SED and DOAE encoder branches, and segregated the SELD output into event-
independent tracks. The second version of EIN that used multi-head self-
attention (MHSA) to decode the SELD output is currently the state-of-the-art
solution for on DCASE 2020 evaluation set using a single model [13]. In
another research direction, Shimada  _et al._ proposed a new output format for
SELD called activity-coupled Cartesian DOA (ACCDOA) that required only one
loss function to optimize the SELD network [14]. The authors also proposed to
use a densely connected multi-dilated DenseNet (RD3Net) instead of CRNN to
achieve a better SELD performance. The RD3Net with ACCDOA outputs is currently
the state-of-the-art solution on the DCASE 2020 test set using a single model.
The top-ranked solution for DCASE 2020 SELD challenge synthesized a larger
dataset from the provided data using four different data augmentation methods
and combined many SELDnet-like models with more complex sub-networks into an
ensemble [15, 16].

When SELDnet was first introduced, it was trained on multichannel magnitude
and phase spectrograms [4]. Subsequently, different features such as
multichannel log-mel spectrograms and intensity vectors (IV) for the first-
order ambisonics format (FOA) and generalized cross-correlation with phase
transform (GCC-PHAT) for the microphone array (MIC) format were shown to be
more effective for the SELD task [6, 7, 11, 13, 14, 15, 16, 17]. The
advantages of the log-mel spectrograms over the linear magnitude spectrograms
for deep learning-based SED are lower dimensions and more emphasis on the
lower frequency bands where signal contents are mostly populated.

However, combining IV or GCC-PHAT features with log-mel spectrograms is not
trivial and the implicit DOA information stored in the former features are
often compromised. In order to stack the IVs with log-mel spectrograms,
frequency band compression on the IVs is required. In practice, the IVs are
often passed through the mel filters which merge DOA cues in different narrow
bands into one mel band, making it more difficult to resolve different DOAs in
multi-source scenarios. Nonetheless, although the resolution of the DOA cues
is reduced, the corresponding frequency mapping between log-mel spectrograms
and the IVs are preserved. This frequency correspondence is crucial for
algorithms to associate sound classes and DOAs of multiple sound events, where
signals of different sound sources are often distributed differently along the
frequency dimension. This frequency correspondence, however, has no
counterpart for GCC-PHAT features since the time lags dimension of the GCC-
PHAT features does not have a linear one-to-one mapping with the mel bands of
the log-mel spectrograms. As a result, all of the DOA information is
aggregated at the frame level, making it difficult to assign correct DOAs to
different sound events. In addition, GCC-PHAT features are noisy when there
are multiple sound sources. In order to solve SELD more effectively in multi-
source scenarios with interferences, a better feature is needed for both audio
formats.

In this work, we proposed a novel feature for SELD task called Spatial Cue-
Augmented Log-Spectrogram (SALSA) with exact spectrotemporal mapping between
the signal power and the source direction-of-arrival. The feature includes
multichannel log-spectrograms stacked along with the estimated direct-to-
reverberant ratio and a normalized version of the principal eigenvector of the
spatial covariance matrix at each TF bin on the spectrograms. The principal
eigenvector is normalized such that it represents the inter-channel intensity
difference (IID) for the FOA format, or inter-channel phase difference (IPD)
for the microphone array format. We evaluated the effectiveness of the
proposed feature using the DCASE 2021 SELD dataset with FOA format.
Experimental results showed that the deep learning-based models trained with
SALSA feature outperformed the DCASE 2021 challenge baseline model that was
trained with log-mel spectrograms and IV features by a large margin. We
further combined several SELD models with slightly different architectures
into ensembles to maximize the performance of our submitted systems to the
challenge. The rest of our paper is organized as follows. Section 2 describes
our proposed method with a brief introduction to SALSA features. Section 3
presents the experimental results and discussions, where our submission
strategies are elaborated. Finally, we conclude the paper in Section 4.

##  2 The proposed method

Figure 1: SELD network architecture.

Figure 1 shows the block diagram of our SELD system. The SELD encoder is a
convolutional neural network (CNN) that learns spatial and spectrotemporal
representation from SALSA features. The SELD decoder consists of a temporal
network and fully connected (FC) layers to decode SELD output sequences.
Popular choices for the temporal network are long short-term memory (LSTM),
gated recurrent unit (GRU), and MHSA with positional encoding. We train
different SELD models with different temporal networks and combine them into
ensembles.

###  2.1 Spatial cue-augmented log-spectrogram features (SALSA)

Figure 2: SALSA features of a 16-second audio segment with FOA inputs in a
multi-source scenario. The horizontal axis represents time in seconds, and the
vertical axis represents frequency in kHz.

Figure 2 illustrates SALSA features of a 16-second audio segment with four-
channel inputs in multi-source cases. The first four channels are log
spectrograms. The fifth channel is the estimated direct-to-reverberant ratios
(DRR). The last three channels are a normalized version of the principal
eigenvectors of the spatial covariance matrix at each time-frequency bin on
the spectrograms. The DRR and the eigenvectors are only computed for TF bins
whose energy mainly comes from a single sound source. These bins are called
single-source (SS) TF bins. In order to find these SS TF bins, we apply a
magnitude test and a coherence test on each TF bin [18]. The magnitude test
finds all TF bins whose powers are higher than a noise threshold to mitigate
the effect of background noise. The coherence test finds TF bins whose spatial
covariance matrices are approximately rank-$1$. We use DRR which is the ratio
between the two largest eigenvalues of the covariance matrix as the criterion
for the coherence test. As shown in [18], the principal eigenvector of the
covariance matrix at each SS TF bin is a scaled version of the theoretical
steering vector of the corresponding dominant sound source at that particular
TF bin. We normalize the principal eigenvectors and remove the redundant
element corresponding to the reference microphone. For the FOA format, this
normalized eigenvector corresponds to the IID. The SALSA features are real-
valued for both FOA and MIC formats. We can see in Figure 2 that the last four
spatial-feature channels are visually discriminant for different sources
originating from different directions.

###  2.2 Network architecture

For the SELD encoder, we use the ResNet22 network, which was adopted for audio
tagging application [19], without the last global pooling and fully connected
layer. Unlike the ResNet18 network used in image applications [20], the first
convolution layer of the ResNet22 is a double convolution with kernel size
$(3,3)$ and stride $1$. The downsampling factor of ResNet22 is $16$ on both
time and frequency dimensions. The output embedding of the ResNet22 encoder is
average-pooled along the frequency dimension before fed into the SELD decoder.
For the SELD decoder, we use $2$ layers of either of the three different
temporal networks: bidirectional LSTM, bidirectional GRU, and MHSA with
positional encoding. The hidden size of the LSTM and GRU is $256$. The
dimension of the feedforward network of the MHSA is $1024$. The SELD problem
is formulated as a multi-label multi-class classification task for sound
classes and a regression task for DOAs. We employ $4$ FC layers to produce the
SELD output. The first FC layer, which is followed by a sigmoid activation,
produces the posterior probabilities of the sound classes. The remaining three
FC layers produce Cartesian coordinates of the DOA on a unit sphere. We call
this output format _SEDXYZ_. We also use the newly proposed ACCDOA output
format in our experiments. When the ACCDOA format is used, the contribution of
the classification loss is set to zero.

##  3 Experimental results and discussions

System | # Params. | Validation | Test
---|---|---|---
$\text{ER}_{\leq$20\text{\,}\mathrm{\SIUnitSymbolDegree}$}$ | $\text{F}_{\leq$20\text{\,}\mathrm{\SIUnitSymbolDegree}$}$ | $\text{LE}_{\text{CD}}$ | $\text{LR}_{\text{CD}}$ | $\mathcal{D}_{\text{SELD}}$ | $\text{ER}_{\leq$20\text{\,}\mathrm{\SIUnitSymbolDegree}$}$ | $\text{F}_{\leq$20\text{\,}\mathrm{\SIUnitSymbolDegree}$}$ | $\text{LE}_{\text{CD}}$ | $\text{LR}_{\text{CD}}$ | $\mathcal{D}_{\text{SELD}}$
A | 112.2M | 0.347 | 0.756 | 13.4 | 0.783 | 0.221 | 0.378 | 0.740 | 11.4 | 0.756 | 0.236
B | 83.9M | 0.337 | 0.762 | 13.5 | 0.785 | 0.216 | 0.376 | 0.738 | 11.2 | 0.750 | 0.238
C | 107.8M | 0.334 | 0.760 | 13.2 | 0.775 | 0.218 | 0.372 | 0.737 | 11.2 | 0.741 | 0.239
D | 112.2M | 0.363 | 0.749 | 13.8 | 0.801 | 0.222 | 0.389 | 0.741 | 12.1 | 0.779 | 0.239
Table 1: Evaluation results for submitted systems

We evaluated our proposed SALSA features using the TAU-NIGENS Spatial Sound
Events 2021 Dataset [7]. We compared the performance of different models
trained on this new feature against the challenge baseline.

###  3.1 Dataset

We used only the FOA subset of the dataset for our experiments. The
development split of the dataset consists of $400$, $100$, and $100$ one-
minute audio recordings for the train, validation, and test splits
respectively. There are $12$ target sound classes. The ground truth metadata
provided by the 2021 dataset only includes the labels for sound events
belonging to the target classes. In other words, all directional interferences
are unlabelled. The azimuth and elevation ranges are
$[$-180\text{\,}\mathrm{\SIUnitSymbolDegree}$,$180\text{\,}\mathrm{\SIUnitSymbolDegree}$)$
and
$[$-45\text{\,}\mathrm{\SIUnitSymbolDegree}$,$45\text{\,}\mathrm{\SIUnitSymbolDegree}$]$,
respectively. During development stage, the validation set was used for model
selection while the test set was used for evaluation. During evaluation stage,
all development data were used for training evaluation models.

###  3.2 Evaluation metrics

To evaluate the SELD performance, we used the official evaluation metrics [21]
that were newly introduced in this year DCASE challenge. The new metrics not
only take into account the joint dependencies between SED and DOAE but also
penalize systems that cannot resolve the overlapping of multiple instances of
the same class [21]. A sound event was considered a correct detection only if
it has correct class prediction and its estimated DOA is also less than
$T$\mathrm{\SIUnitSymbolDegree}$$ away from the DOA ground truth, where
$T=$20\text{\,}\mathrm{\SIUnitSymbolDegree}$$ for this competition. The DOAE
metrics are also class-dependent, that is, the detected sound class will also
have to be correct in order for the corresponding localization predictions to
count.

Similar to DCASE 2020, the DCASE 2021 SELD task adopted four evaluation
metrics: location-dependent error rate ($\text{ER}_{\leq
T$\mathrm{\SIUnitSymbolDegree}$}$) and F1 score ($\text{F}_{\leq
T$\mathrm{\SIUnitSymbolDegree}$}$) for SED; and class-dependent localization
error ($\text{LE}_{\text{CD}}$), and localization recall
($\text{LR}_{\text{CD}}$) for DOAE. We also reported an aggregated SELD metric
which was computed as

$\mathcal{D}_{\text{SELD}}=\dfrac{1}{4}\left[\text{ER}_{\leq
T$\mathrm{\SIUnitSymbolDegree}$}+(1-\text{F}_{\leq
T$\mathrm{\SIUnitSymbolDegree}$})+\dfrac{\text{LE}_{\text{CD}}}{$180\text{\,}\mathrm{\SIUnitSymbolDegree}$}+(1-\text{LR}_{\text{CD}})\right].$
(1)

A good SELD system should have low $\text{ER}_{\leq
T$\mathrm{\SIUnitSymbolDegree}$}$, high $\text{F}_{\leq
T$\mathrm{\SIUnitSymbolDegree}$}$, low $\text{LE}_{\text{CD}}$, high
$\text{LR}_{\text{CD}}$, and low aggregated metric
$\mathcal{D}_{\text{SELD}}$.

###  3.3 Hyperparameters and training procedure

We used sampling rate of $24\text{\,}\mathrm{kHz}$, window length of $512$
samples, hop length of
$300\text{\,}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{s}$,
Hann window, and $512$ FFT points. As a result, the input frame rate of SALSA
features was $80\text{\,}\mathrm{f}\mathrm{p}\mathrm{s}$. Since the model
temporally downsampled the input by a factor of $16$, we temporally upsampled
the final outputs by a factor of $2$ to match the label frame rate of
$10\text{\,}\mathrm{f}\mathrm{p}\mathrm{s}$. The loss weights for SED and DOAE
outputs were set to $0.7$ and $0.3$ respectively. Adam optimizer was used to
train all the models. Learning rate was initially set to $0.003$ and gradually
decreased to ${10}^{-4}$. The maximum number of training epochs was $60$. A
threshold of $0.3$ was used to binarize active class predictions in the SED
outputs.

###  3.4 Experimental settings

We trained several SELD models on the new SALSA features. We compared our
models with the DCASE 2021 challenge baseline [4, 14] that was trained on log-
mel spectrograms and IV features. Since the provided dataset is relatively
small, we employed several data augmentation techniques. First, we extended
the spatial augmentation technique [22] that randomly swaps and negates the X,
Y, and Z channels of the FOA format to our SALSA feature. Using the provided
theoretical steering vector for the FOA format, the last three channels of the
SALSA features correspond to Y, Z, and X responses. Therefore, we swapped and
negated the last three spatial channels accordingly. Secondly, we applied
random cutout [23] and SpecAugment [24] on all the channels of the SALSA
features. For SpecAugment, we only applied time masking and frequency masking.
Random cutout produces a rectangular mask on the spectrograms while
SpecAugment produces a cross-shaped mask. Lastly, we also randomly removed
selected SS TF bins on the last 4 channels of the SALSA feature.

We used different input lengths, e.g. $4$ seconds, $8$ seconds, etc., to train
different SELD models. We experimented with three different SELD decoders:
bidirectional LSTM, bidirectional GRU, and MHSA with positional encoding. We
train the majority of these models using the SEDXYZ output format and some
models with the ACCDOA output format. Since these two output formats are both
class-wise format, they can be easily aggregated into ensembles using mean
operation. The disadvantage of the class-wise output format is that they
cannot resolve overlapping same-class events, which accounts for
$10.45\text{\,}\mathrm{\char 37\relax}$ of the total frames in the DCASE 2021
SELD dataset. We chose to use class-wise outputs for the ease of ensemble. To
further improve the performance of each trained model, we applied test-time
augmentation (TTA) during inference. We adapted the 16-pattern spatial
augmentation technique from [22] for TTA, similar to the spatial augmentation
technique that was employed during training. We augmented all the channels of
the SALSA features (except for the DRR channel), estimated the output,
reversed the outputs accordingly, and computed the mean of all the $16$
outputs.

We combined different SELD models into 4 SELD ensembles. We also train several
CRNN models for SED only. We experimented with different combinations of CNN
architectures, such as VGG and ResNet, and recurrent neural network (RNN)
architectures, such as bidirectional GRU and bidirectional LSTM. We combined
SED models with various CNN-RNN combinations into an SED ensemble. This SED
ensemble was then combined with the 4 SELD ensembles to form 4 submission
systems that were submitted to the challenge. When the SED ensemble and an
SELD ensemble were combined, only the SED outputs were averaged, the DOA
outputs of the SELD ensemble were kept intact. All four systems used all six
folds of the development dataset for training.

###  3.5 Experimental results

Table 1 shows the performances on the validation and test splits of our four
submitted systems. System D is similar to System A except for lower SED
threshold on common classes such as _foot step_ , and _alarm_. As a result,
System D scored higher in $\text{LR}_{\text{CD}}$ at the expense of higher
$\text{ER}_{\leq$20\text{\,}\mathrm{\SIUnitSymbolDegree}$}$, and
$\text{LE}_{\text{CD}}$ on both validation and test splits.

Model | # Params. | Test
---|---|---
| $\text{ER}_{\leq$20\text{\,}\mathrm{\SIUnitSymbolDegree}$}$ | $\text{F}_{\leq$20\text{\,}\mathrm{\SIUnitSymbolDegree}$}$ | $\text{LE}_{\text{CD}}$ | $\text{LR}_{\text{CD}}$
Baseline LM/IV | 0.5M | 0.690 | 0.339 | 24.1 | 0.439
GRU LM/IV | 14.2M | 0.650 | 0.483 | 22.0 | 0.626
GRU w/o TTA | 14.2M | 0.426 | 0.686 | 12.1 | 0.683
w/o TTA | 14.2M | 0.404 | 0.702 | 11.0 | 0.674
LSTM w/o TTA | 15.0M | 0.428 | 0.685 | 11.9 | 0.697
w/o TTA | 15.0M | 0.410 | 0.695 | 11.0 | 0.691
MHSA w/o TTA | 16.1M | 0.498 | 0.673 | 14.4 | 0.761
w/o TTA | 16.1M | 0.450 | 0.700 | 13.1 | 0.759
A w/o SED ensem. | 73.6M | 0.377 | 0.734 | 11.0 | 0.740
A w/o SED ensem. | 112.2M | 0.378 | 0.740 | 11.4 | 0.756
Table 2: Evaluation results for ablation studies. Unless ‘LM/IV’ is indicated,
the models are trained on SALSA features. LM/IV stands for log-mel spectrogram
and intensity vector features.

Table 2 compares the performance of different SELD models. The challenge
baseline, denoted as Baseline LM/IV, and our model GRU LM/IV were trained on
the log-mel spectrograms and IV feature. The difference between our SELD model
and the baseline CRNN model were that our model had more parameters than the
baseline, and we used $128$ mel bands instead of $64$ mel bands. The SELD
performance improved with bigger model, GRU LM/IV, especially the
$\text{LR}_{\text{CD}}$ increased from $0.439$ to $0.626$. The rest of our
models in Table 2 were trained with SALSA features. The model GRU LM/IV and
GRU w/o TTA had almost identical network architectures and similar number of
parameters, except that the number of inputs channel to GRU LM/IV was 7, while
the number of input channels to GRU w/o TTA was 8. The GRU w/o TTA model
outperformed GRU LM/IV by a large margin across all the evaluation metrics.
This result demonstrated the effectiveness of our new proposed feature over
the conventional log-mel spectrogram and intensity vector feature for SELD
using FOA format. When comparing the performance of different SELD decoders,
LSTM and GRU achieved similar scores. MHSA scored higher on
$\text{LR}_{\text{CD}}$ than RNN-based decoders but lower on other metrics.
TTA improved the performance especially for SED metrics and DOA error.
Combining several SELD models into an ensemble boosted the final performance.
Interestingly, by combining with an SED ensemble, the performance of the final
ensemble was increased slightly.

##  4 CONCLUSION

In conclusion, we presented a novel spectrotemporally-aligned feature, SALSA,
for training a joint end-to-end SELD network. Experimental results showed our
networks trained on the new feature outperformed the DCASE 2021 Task 3
baseline system by a large margin, demonstrating the effectiveness of SALSA
feature in enabling the deep models to learn useful spatial and
spectrotemporal information for SELD task.

##  5 REFERENCES

## References

  * [1] J. Salamon and J. P. Bello, “Deep convolutional neural networks and data augmentation for environmental sound classification,” _IEEE Signal Process. Lett._ , vol. 24, no. 3, pp. 279–283, March 2017.
  * [2] D. Stowell, M. Wood, Y. Stylianou, and H. Glotin, “Bird detection in audio: A survey and a challenge,” in _Proc. IEEE Int. Workshop Mach. Learn. Signal Process._ , Sep. 2016, pp. 1–6.
  * [3] P. Foggia, N. Petkov, A. Saggese, N. Strisciuglio, and M. Vento, “Audio surveillance of roads: A system for detecting anomalous sounds,” _IEEE Trans. Intell. Transp. Syst._ , vol. 17, no. 1, pp. 279–288, Jan 2016\.
  * [4] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, “Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,” _IEEE J. Sel. Topics Signal Process._ , vol. 13, no. 1, pp. 34–48, March 2019.
  * [5] S. Adavanne, A. Politis, and T. Virtanen, “A multi-room reverberant dataset for sound event localization and detection,” in _Proc. Detect. Classification Acoust. Scenes Events Workshop_ , New York University, NY, USA, October 2019, pp. 10–14.
  * [6] A. Politis, S. Adavanne, and T. Virtanen, “A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection,” _arXiv e-prints: 2006.01919_ , 2020.
  * [7] A. Politis, S. Adavanne, D. Krause, A. Deleforge, P. Srivastava, and T. Virtanen, “A dataset of dynamic reverberant sound scenes with directional interferers for sound event localization and detection,” _arXiv preprint arXiv:2106.06999_ , 2021.
  * [8] E. Cakır, G. Parascandolo, T. Heittola, H. Huttunen, and T. Virtanen, “Convolutional recurrent neural networks for polyphonic sound event detection,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 25, no. 6, pp. 1291–1303, 2017.
  * [9] S. Mohan, M. E. Lockwood, M. L. Kramer, and D. L. Jones, “Localization of multiple acoustic sources with small arrays using a coherence test,” _J. Acoust. Soc. Amer._ , vol. 123, no. 4, pp. 2136–2147, 2008.
  * [10] T. Hirvonen, “Classification of spatial audio location and content using convolutional neural networks,” _J. Audio Eng. Soc._ , 2015.
  * [11] Y. Cao, Q. Kong, T. Iqbal, F. An, W. Wang, and M. D. Plumbley, “Polyphonic sound event detection and localization using a two-stage strategy,” in _Proc. Detect. Classification Acoust. Scenes Events Workshop_ , 2019.
  * [12] Y. Cao, T. Iqbal, Q. Kong, Y. Zhong, W. Wang, and M. D. Plumbley, “Event-independent network for polyphonic sound event localization and detection,” in _Proc. Detect. Classification Acoust. Scenes Events Workshop_ , 2020.
  * [13] Y. Cao, T. Iqbal, Q. Kong, F. An, W. Wang, and M. D. Plumbley, “An improved event-independent network for polyphonic sound event localization and detection,” in _Proc. IEEE Int. Conf. Acoust. Speech, Signal Process._ IEEE, 2021, pp. 885–889.
  * [14] K. Shimada, Y. Koyama, N. Takahashi, S. Takahashi, and Y. Mitsufuji, “ACCDOA: Activity-coupled cartesian direction of arrival representation for sound event localization and detection,” in _Proc. IEEE Int. Conf. Acoust. Speech, Signal Process._ IEEE, 2021, pp. 915–919.
  * [15] Q. Wang, H. Wu, Z. Jing, F. Ma, Y. Fang, Y. Wang, T. Chen, J. Pan, J. Du, and C.-H. Lee, “The USTC-iFlytek system for sound event localization and detection of DCASE2020 challenge,” DCASE2020 Challenge, Tech. Rep., July 2020\.
  * [16] Q. Wang, J. Du, H.-X. Wu, J. Pan, F. Ma, and C.-H. Lee, “A four-stage data augmentation approach to ResNet-Conformer based acoustic modeling for sound event localization and detection,” _arXiv preprint arXiv:2101.02919_ , 2021\.
  * [17] Q. Wang, H. Wu, Z. Jing, F. Ma, Y. Fang, Y. Wang, T. Chen, J. Pan, J. Du, and C.-H. Lee, “A model ensemble approach for sound event localization and detection,” in _Proc. Int. Symp. Chin. Spoken Lang. Process._ , 2021, pp. 1–5.
  * [18] T. N. T. Nguyen, S. K. Zhao, and D. L. Jones, “Robust DOA estimation of multiple speech sources,” in _Proc. IEEE Int. Conf. Acoust. Speech, Signal Process._ IEEE, 2014, pp. 2287–2291.
  * [19] Q. K. Y., Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,” _arXiv preprint arXiv:1912.10211_ , 2019.
  * [20] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE Conf. Comput. Vis. Pattern Recognit._ , 2016, pp. 770–778.
  * [21] A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen, “Overview and evaluation of sound event localization and detection in DCASE 2019,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 29, pp. 684–698, 2020\.
  * [22] L. Mazzon, Y. Koizumi, M. Yasuda, and N. Harada, “First order ambisonics domain spatial augmentation for dnn-based direction of arrival estimation,” in _Proc. Detect. Classification Acoust. Scenes Events Workshop_ , New York University, NY, USA, October 2019, pp. 154–158.
  * [23] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, “Random erasing data augmentation,” _arXiv preprint arXiv:1708.04896_ , 2017.
  * [24] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” _arXiv preprint arXiv:1904.08779_ , 2019.


# Macroeconomic forecasting and sovereign risk assessment using deep learning
techniques

Anastasios Petropoulos Cyprus University of Technology
<EMAIL_ADDRESS>Anastasios Petropoulos1 Vassilis Siakoulis1
Konstantinos P. Panousis1 Loukas Papadoulas2&Sotirios Chatzis1 1Cyprus
University of Technology
2Ethical AI Novelties
{apetropoulos<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this study, we propose a novel approach of nowcasting and forecasting the
macroeconomic status of a country using deep learning techniques. We focus
particularly on the US economy but the methodology can be applied also to
other economies. Specifically US economy has suffered a severe recession from
2008 to 2010 which practically breaks out conventional econometrics model
attempts. Deep learning has the advantage that it models all macro variables
simultaneously taking into account all interdependencies among them and
detecting non-linear patterns which cannot be easily addressed under a
univariate modelling framework. Our empirical results indicate that the deep
learning methods have a superior out-of-sample performance when compared to
traditional econometric techniques such as Bayesian Model Averaging (BMA).
Therefore our results provide a concise view of a more robust method for
assessing sovereign risk which is a crucial component in investment and
monetary decisions.

##  1 Motivation

Deep Learning models have a short history, however, they have found
application in a variety of scientific fields. In particular, Deep Learning
algorithms have dramatically improved the capabilities of performing pattern
recognition (like speech recognition, image recognition) and forecasting, so
that they offer state of the art performance in various scientific fields like
biology, engineering etc. Their structure offers the ability to adjust in
streaming sequences using continuous learning algorithms, and recognize new
and evolving patterns in time series data. In addition, deep learning is
proven to effectively deal with high dimensional data. Their capacity to learn
and adapt to new data can lead to a better predictive performance in financial
time series modelling problems where non-linear relationships and
observational noise often exist.

Furthermore, this new generation of statistical algorithms offers the
necessary flexibility in modelling multivariate time series, as its structure
includes a cascade of many layers with non-linear processing agents. Deep
learning networks base their functionality on the interaction of layers that
simulate the abstraction and composition functionalities of the human brain.
Therefore, via capturing the full spectrum of information contained in
financial datasets, they are capable of exploring in depth the inherent
complexity of the underlying dynamics in big and high dimensional time series
data.

In this study we explore a novel approach to now-casting and forecasting the
macroeconomic status of a country using deep learning techniques. We focus
particularly on the US economy but the methodology can be applied also to
other economies. Specifically US economy has suffered a severe recession from
2008 to 2010 which practically breaks out single time series model attempts.
Our approach can simultaneously simulate a country’s key financial variables
in a holistic way, under a dynamic balance sheet assumption, and by utilizing
deep learning algorithms. Experimental results give strong evidence that deep
learning applied in financial datasets creates a state of the art paradigm,
which is capable to simulate real word scenarios in a holistic and more
efficient way. Deep learning has the advantage that it models all macro
variables simultaneously taking into account all interdependencies among them
and detecting non-linear patterns which cannot be easily addresses under
univariate modelling framework.

The main contribution of this study is that it proposes a holistic framework
for macroeconomic forecasts. Our research analysis lies at the intersection of
computational finance and statistical machine learning, leveraging the unique
properties and capabilities of deep learning networks to increase the
prediction efficacy and minimizing the modelling error. Under the proposed
approach, forecasting of macroeconomic situation can be heavily supported by
artificial intelligence algorithms simulating better the propagation channels
across different parts economy i.e. banking system, consumer confidence and
state interaction. In a nutshell, the proposed innovation lies in the use of
advanced deep learning techniques for the simultaneous projection of
Macroeconomic variables, while benchmarking vs with traditional methods of
econometrics frequently employed in financial practice (i.e. Bayesian Model
averaging).

The remainder of this study is structured as follows. In section 2, we focus
on the related literature review. Section 3, describes the data collection and
processing. In section 4, we provide details regarding the estimation process
of the various alternative models developed. In section 5, we compare the
employed methodologies. Finally, in concluding section 6, we summarize our
findings and identify any potential weaknesses or limitations, while we also
discuss areas for future research extensions.

##  2 Related Work

Macroeconomic Forecasting is a very useful tool effectively utilized in
banking, finance, business, and other areas. Academic studies have established
a close link between sovereign risk and the expected GDP growth of a country.
Vice versa a country’s economic growth exhibits a significant response to
sovereign risk changes driven by the interest rate and capital-flow
channelschen2016relation . Furthermore, the borrowing costs of the economy are
statistically and economically affected by an increase in sovereign risk as
measured through firm’s credit spreads, based on a recent analysis of credit
default swap data in the Eurozonebedendo2015sovereign .

Macroeconomic satellite models are also the cornerstone of bank stress testing
methodologies as they are the pipeline through which macroeconomic scenarios
are converted to tangible risk factors. In this framework macro models are
used either for scenario projections under specific assumptions or dynamic
forecasting. Usually banks develop a group of satellite models each for every
one macro variable based on which projections and mapping to risk factors are
performed. Multi models usually ignore the interrelations among variables
except for the cases where a joint distribution (copula) is employed for
correlation modelling. Underestimating the correlation impact may adversely
affect the validity of stress test results as second order effects deployed
during a crisis period will be ignored and the estimated impact will deviate
from realized losses.

With respect to sovereign risk assessment, the majority of the academic
literature employs conventional econometric and statistical techniques to
tackle the problem of government debt credit risk assessment. These methods
range from simple regression model to classification-regression
treesmanasse2009rules .

The recent developments of Machine and Deep Learning methodologies and the
availability of large scale macroeconomic data have led a number of
researchers to depart from the field of traditional econometric techniques and
employ novel techniques both in forecasting and now-casting macroeconomic
variables. Bontempi et al.Bontempi2013  make an overview of machine learning
techniques in time series forecasting by focusing on three aspects: the
formalization of one-step forecasting problems as supervised learning tasks,
the discussion of local learning techniques as an effective tool for dealing
with temporal data and the role of the forecasting strategy when we move from
one-step to multiple-step forecasting. Atiya et al.ahmed2010empirical ,
perform a large scale comparison study for the major machine learning models
for time series forecasting by applying the models to the monthly M3 time
series competition data. Katris .katris2020prediction  benchmarks machine
learning techniques vs traditional econometric models in the prediction of
Unemployment Rates.

Liaoliao2017machine  used Artificial Neural Network in time-series
forecasting, by combining First order Markov Switching Model and K-means
algorithms and found that machine learning has outperformed the benchmark of
time-series inflation rate forecasting. Medeiros et al.medeiros2021forecasting
showed that ML models with a large number of covariates are systematically
more accurate than the benchmarks in US inflation since they capture better
potential nonlinearities between past key macroeconomic variables and
inflation. Smalterhall2018machine  points that supplied with diverse and
complex data, a machine learning model can outperform simpler time-series
models, with better performance at shorter horizons. In particular, his
results show that a machine learning model can identify turning points in the
unemployment rate earlier than competing methods.

Kaushik et al.kaushik2020forecasting  come up with a multivariate time series
approach to forecast the exchange rate. His results show that Support Vector
Machines and Recurrent Neural Networks outperform widely used traditional
method of econometric forecasting such as Vector Autoregressive Models. In a
similar vein, Chen et al. chen2004regression  employ a two stage model in
exchange rate forecasting where in the first stage, a time series model
generates estimates of the exchange rates whereas in the second stage, General
Regression Neural Network is used to correct the errors of the estimates. Both
empirical and trading simulation experiments suggest that the proposed hybrid
approach not only produces better exchange rate forecasts but also results in
higher investment returns than the single-stage models.

##  3 Data Collection and Model Structures

In order to calibrate a Deep Neural Network to a series of macroeconomic
series we have collected monthly series from macroeconomic and financial
indicators for the US economy from January 1973 to December 2018. The
described dataset comprised of 540 observations and was split into two parts:
An in-sample train dataset, comprising data pertaining to the $65\%$
(1973-2005) and an out-of-sample dataset that comprises the rest $35\%$ of the
observations (2006-2018). The latter sample was used for evaluation of the
performance of all the statistical techniques implemented. In the split of the
sample the 2008-2010 crisis period was deliberately left out in order to
challenge our model capacity in foreseeing an exceptional structural break in
the macroeconomic status. Thus, the developed system is expected to exhibit
more stability through the cycle behaviour. The macroeconomic series
incorporated into the network depict all relevant aspects of the economic
status of a country. Namely we include in the network the following 9
macroeconomic time series:

  * •

GDP: Gross Domestic Product yearly growth

  * •

DEBT: Government Debt as % of GDP

  * •

RRE: Real Estate prices yearly growth

  * •

UNR: Yearly change in Unemployment Rate

  * •

INFLAT: Price inflation i.e. Consumer Price Index yearly growth

  * •

YIELD10Y: 10 year Government bond yield changes

  * •

GOVEXP: Government Expenses as % of GDP

  * •

EXPORT: Exports yearly growth

  * •

STOCKS: Annual return of S&P 500

Our aim is to effectively forecast the evolution of the abovementioned time
series both under a univariate framework by employing a Bayesian Moving
Average model and under a multi-variate framework employing Deep Neural
Networks taking into account simultaneous interdependencies. In the next step
we benchmark the forecasting performance of the univariate vs the multivariate
framework both under a static and dynamic view. We employ an explanatory set
of variables for each of the 9 macro series 1 to 2 yearly lags of both the
dependent and the rest of the other macro series based on the assumption that
macro and financial variables ratios carry all the information necessary to
describe and predict the macroeconomic state of a country. We also include as
independent variables a set of variables, on current and lagged values, which
account for the effect of banking system in the real economy, namely

  * •

DEP: Bank Deposits yearly growth

  * •

LOAN: Commercial Loans yearly growth

  * •

INTLOAN: Interbank loan yearly growth

This is a crucial supplement as the interlinkage between banking sector and
economic activity is both empirically and theoretically justified. Confidence
of the public and the market in the banking sector will lead to liquidity
provision through the disposition of customer deposits and interbank funding
which in turn will be to commercial loans which will fuel economic activity.
On the other hand, low banking confidence leads to deposit outflows and
shrinkage of the interbank market, rendering problematic the financing of the
economy from the banking system. This is the main reason why a crisis in the
banking sector could lead to an economic crisis which in turn could send
feedback effects on the banking system initiating a new second order crisis.
In all, the individual models estimated through Bayesian Model Averaging are
shown below in eq (1).

$\displaystyle\begin{split}\text{DEPVAR}_{t}&=\text{GDP}_{t-1,2}+\text{DEBT}_{t-1,2}+\text{RRE}_{t-1,2}+\text{UNR}_{t-1,2}\\\
&+\text{INFLAT}_{t-1,2}+\text{YIELD10Y}_{t-1,2}+\text{GOVEXP}_{t-1,2}\\\
&+\text{EXPORT}_{t-1,2}+\text{STOCKS}_{t-1,2}+\text{DEP}_{t-0,1,2}\\\
&+\text{LOAN}_{t-0,1,2}+\text{INTLOAN}_{t-0,1,2}+\epsilon\end{split}$ (1)

where $\text{DEPVAR}_{t}$ comprises each one of the following variables
($\text{GDP}_{t}$, $\text{DEBT}_{t}$, $\text{RRE}_{t}$, $\text{UNR}_{t}$,
$\text{INFLAT}_{t}$, $\text{YIELD10Y}_{t}$, $\text{GOVEXP}_{t}$,
$\text{EXPORT}_{t}$, $\text{STOCKS}_{t}$) leading to 9 separate models, one
for each macroeconomic factor.

Modeling separately the macroeconomic variables has the drawback that it does
not take into account the simultaneous interdependencies among variables. For
example it could the case that GDP is not affected solely by the lagged value
of RRE but also by the $\text{RRE}_{t}$. Traditional econometrics handle this
case through the use of SUR (Seemingly Unrelated Regression) models or under a
VAR framework. In our proposed approach for dynamic macro forecasting we
introduce a Deep neural networks architecture as an innovative way to take
into account contemporaneous dependencies among variables.

In all, we identify the main channels of risk propagation in a recurrent form
to account of all the existing evidence of feedback effects in a macroeconomic
system by putting all the components together in a multivariate structure. On
the other hand, the use of classical econometric techniques offers limited
capabilities for simulating complex systems. Our approach accounts for
temporal patterns in the economy providing a dynamic modelling approach. This
is achieved through the multivariate training of deep neural networks which
takes into account the dynamic nature of the economy. The approach proposed is
composed of multivariate input and output layers able to capture the cross
correlation between macroeconomic variables. Training is performed as one big
complex network minimizing estimation errors and double counting effects among
various financial variables.

To account for non-linear relationships that materialize under different
macroeconomic conditions machine learning techniques like deep learning can
provide more efficient estimations. Based on academic literature Deep Neural
networks are capable of simulating real life phenomena where relationships are
complex, so our proposed framework by using multilayer deep networks envisages
capturing the dynamics inherent in the economy. In addition the architecture
of aims to capture the amplification channels leading to structural breaks.

In the general structure of our model lagged macroeconomic indicators along
with current values for Loan Growth, Deposit Growth and Interbank Loan growth
will be inserted in the input layers and their now-casted values will be
produced in the output layer.

##  4 Model Development

###  4.1 Single Variable Forecasting – Bayesian Model Averaging (BMA)

We employ the Bayesian Model Averaging (BMA) methodology for estimating single
equation satellite models which are used for univariate predictions for the
macroeconomic variables. The BMA method accounts better than linear regression
for the uncertainty surrounding the main determinants of risk dynamics. Using
BMA, a pool of equations is generated using a subgroup of determinants
randomly selected. In the next step, a weight is assigned to each model that
reflects their relative forecasting performance. Aggregating all equations
using the corresponding weights produces a posterior model probability,
provided that the number of equations estimated in the first step is large
enough to capture all possible combinations of a predetermined number of
independent variables. Thus Bayesian model averaging addresses model
uncertainty and misspecification better than a simple linear regression
problem.

To further illustrate BMA, suppose a linear model structure with $Y_{t}$ being
the dependent variable, $X_{t}$ the explanatory variables, $\alpha$ constant,
$\beta$ the coefficients, and $\epsilon_{t}$ a normal error term with variance
$\sigma$.

$\displaystyle
Y_{t}=\alpha_{\gamma}+\beta_{\gamma}X_{\gamma,t}+\epsilon_{t},\qquad\epsilon_{t}\sim\mathcal{N}(0,\sigma^{2}I)$
(2)

In an ordinary linear regression problem, the existence of many potential
explanatory variables in a matrix $X_{t}$ renders their correct combination a
quite burdensome task. Including all the variables does not provide a feasible
solution as it can lead to overfitting and multicollinearity especially when
there is a limited number of observations. BMA tackles the problem by
estimating models for all possible combinations of $\\{X\\}$ and constructing
a weighted average over all of them.

Under the assumption that $X$ contains $K$ potential explanatory variables,
BMA estimates $2^{K}$ combinations and thus $2^{K}$ models. Applying Bayes’
theorem, model averaging is based on the posterior model probabilities.

$\displaystyle\begin{split}p(M_{\gamma}\cup Y,X)&=\frac{p(Y\cup
M_{\gamma},X)p(M_{\gamma})}{p(Y\cup X)}\\\ &=\frac{p(Y\cup
M_{\gamma},X)p(M_{\gamma})}{\sum_{s=1}^{2^{K}}p(Y\cup
M_{s},X)p(M_{s})}\end{split}$ (3)

In Equation (3), $p(Y,X)$ denotes the integrated likelihood which is constant
over all models and is thus simply a multiplicative term. Therefore, the
posterior model probability (PMP) is proportional to the integrated likelihood
$p(Y\cup M,X)$ which reflects the probability of the data given the model $M$.
Thus, the corresponding weight assigned to each model is measured by using
$p(M_{\gamma}\cup Y,X)$ in Eq. (3). In equation (3), $p(M)$ denotes the prior
belief of how probable model $M$ is before analyzing the data. Furthermore, to
estimate $p(Y,X)$ integration is performed across all models in the model
space and to estimate the probability $p(Y\cup M,X)$ integration is performed
given model $M$ across all parameter space. By performing renormalization of
the product in equation (3), PMPs can be inferred and subsequently the model’s
weighted posterior distribution for estimator $\beta$ is given by

$\displaystyle p(\beta\cup Y,X)=\sum_{\gamma=1}^{2^{K}}p(\beta\cup
M_{\gamma},Y,X)p(M_{\gamma}\cup X,Y)$ (4)

The priors, posteriors and the marginal likelihood employed in the estimation
are described analytically in Appendix A. In Bayesian Model Averaging
estimation we employ unit information prior (UIP), which sets g=N commonly for
all models. We use also a birth/death MCMC algorithm (20000 draws) due to the
large number of covariates included since using the entire model space would
lead to a large number of iterations. We fix the number of burn-in draws for
the MCMC sampler to 10000. Finally, the models prior employed is the “random
theta” prior by Ley and Steel, who suggest a binomial-beta hyperprior on the a
priori inclusion probability. This has the advantage that is less tight around
prior expected model size (i.e. the average number of included regressors) so
it reflects prior uncertainty about model size more efficiently. In order to
develop all the satellite models for this approach we employ the utilities of
BMS R package.

###  4.2 Multiple Variable Forecasting - Deep Neural Networks

We propose the use of multilayer deep neural networks in order to
simultaneously forecast the basic macroeconomic and financial variables,
capturing the dynamics inherent in the economy by taking into account the
contemporaneous interdependencies among them. Deep learning applications in
the domain of finance is rather limited but it has been an active field of
research in recent years, as it has achieved significant breakthroughs in the
fields of computer vision and language understanding. Specifically, our paper
constitutes one of the first works presented in the literature that considers
application of deep learning to address the challenging task of macroeconomic
prediction.

Deep Neural Networks are built on the basis of nonlinear activation functions
typical choices of which are the logistic sigmoid, hyperbolic tangent, and
rectified linear unit (ReLU). The first two (logistic sigmoid and hyperbolic
tangent) activation functions are closely related as they belong both to the
sigmoid family. The sigmoid family of activation functions has the
disadvantage of saturating the network with large positive or negative values.
To alleviate this problem, practitioners have derived linear activation
functions, like the popular ReLU, which are now the standard choice in deep
learning research.

The activation layers increase the ability and flexibility of a DNN to capture
non-linear relationships in the training dataset but on the other hand, the
huge number of trainable parameters could lead to overfitting. Therefore the
use of simple, effective, and efficient regularization techniques is necessary
to avoid poor out of sample performance. Dropout is the most popular
regularization technique for DNNs. In essence, it consists in randomly
dropping different units of the network on each iteration of the training
algorithm, so in this way, only the parameters related to a subset of the
network units are trained during each iteration. This strategy reduces the
associated network overfitting tendency ensuring that all network parameters
are effectively trained. Inspired by these merits, we employ Dropout DNNs with
ReLU activations to train and deploy feed forward deep neural networks. We
postulated deep networks that are up to five hidden layers deep and comprise
various numbers of neurons.

A supplementary way to improve the fitting efficacy of the trained network is
the incorporation prior information in a Bayesian framework in a similar way
to BMA linear models. Conventional network architectures compute point
estimates of the unknown values without taking into consideration any prior
information and without any uncertainty estimation of the produced values. The
Bayesian treatment of a particular model has been shown to increase its
capacity and potential, while oﬀering a natural way to assess the uncertainty
of the resulting estimates. To this end, we augment the conventional model
architectures of the previous sections by also relying on the Bayesian
framework.

Specifically, we impose a prior Normal distribution over network weights,
seeking to infer their posterior distribution given the data. Since the
marginal likelihood is intractable for the considered architectures, from the
existing Bayesian methods, we rely on approximate inference and specifically
on Variational Inference. Since the true posterior of the model cannot be
computed in Variational Inference, we introduce an auxiliary variational
posterior distribution of a family of distributions and then try and find the
optimal member of the considered family to match the true posterior. The
matching is achieved through the minimization of the Kullback-Leibler (KL)
divergence between the true and the introduced variational posterior111The KL
divergence is a metric of similarity between two distributions and is a non-
negative value; KL is zero, if and only if, the two considered distributions
match exactly.. Minimizing the KL divergence is equal to the maximization of
the Evidence Lower Bound (ELBO), a well-known bound on the marginal likelihood
derived using Jensen’s inequality. Thus, for training the following
architectures, we resort to ELBO maximization.

Non-linear activation functions such as ReLUs are a mathematically convenient
tool for training deep networks but nevertheless, they do not come with strong
biological plausibility. Current research has shown that in real life neurons
with similar functionality and structure tend to group and compete with each
other for their output. To this end, researchers have devoted signiﬁcant
effort to explore this type of competition between neurons and apply it in
existing models. The resulting procedure is referred to as Local Winner-Takes-
All (LWTA) and has been shown to provide competitive, or even better, results
in benchmarks architectures in different domains of Deep Learning
applications. Thus, apart from the conventional ReLU activations of the
previous section, we additionally explore the potency of the LWTA in our work.
The linear units after the affine transformation in each layer are grouped
together and compete from their outputs. This competition is performed in a
probabilistic way, by employing a softmax nonlinearity, obtaining the
probability of activation of each unit in each block.

##  5 Model Validation

In order to assess the robustness of our approach we perform a thorough
validation procedure. The dataset comprised of 540 observations was split into
two parts: An in-sample train dataset, comprising data pertaining to the 65%
(1973-2005) and an out-of-sample dataset that comprises the rest 35% of the
observations (2006-2018). The latter sample was used for evaluation of the
performance of all the statistical techniques implemented. In the split of the
sample, the 2008-2010 crisis period was deliberately left out in order to
challenge our model capacity in foreseeing an exceptional structural break in
the macroeconomic status. Models compared in this benchmark exercise are
Bayesian Model Average (BMA), Deep Learning Model with ReLU activation
function and Drop out terms trained using the MXNET algorithm (MXNET),
Bayesian Deep Learning model using ReLU activation function (Bayesian ReLU)
and Bayesian Deep Learning model using Local Winner-Takes-All activation
function (Bayesian LWTA).

Note that, in order to train the deep learning algorithms in the current
study, the in-sample dataset is further split randomly in train and validation
set. The validation dataset is used to find the best set of hyper parameters
of the models and select the best candidate model for performing the out–of-
sample evaluation.

The validation process is performed following two different strategies, namely
static and rolling forecast. In the static case, all models are estimated only
one time in the train sample (1973-2005) and the whole test sample is used to
produce the out of time (2006-2018) predictions in a one-off way. In the
rolling forecast framework the models are re-estimated in each step of the out
of time sample (2006-2018) and the prediction is produced for one month ahead
each in each step. For example in order to produce the January 2006 forecast
the sample 1973 till December 2005 is used, whereas in the next step the train
sample is enlarged to 1973 till January 2006 and the February 2006 forecast is
generated.

From Table 1 where the errors for each forecasted macro variable are shown we
deduce that Deep Learning algorithms clearly outperform the benchmark BMA
approach, especially in cases of variables such as the government bond yield
(YIELD10Y), the real estate price growth (RRE) and the stock market growth
(STOCKS) which showed a massive structural outbreak during the Subprime crisis
(2008-2010) in the US. The Bayesian Deep Neural Network with Local Winner
Takes All activation function outperforms not only the BMA model but also more
conventional Deep Learning algorithms showing the benefit of applying more
biologically plausible activation functions.

Table 1: Static Forecast Error Metrics for each variable: MSE stands for Mean
Square Error and MAE stands for Mean Absolute Error.

Method | YIELD10Y | UNR | RRE
---|---|---|---
| MSE | MAE | MSE | MAE | MSE | MAE
Satellite Modeling (BMS) | 13.31 | 2.54 | 1.40 | 0.90 | 79.14 | 7.09
Deep Learning (MXNET) | 1.14 | 0.81 | 1.95 | 0.97 | 40.42 | 5.21
Deep Learning (Bayesian ReLU) | 1.76 | 0.93 | 0.95 | 0.73 | 36.78 | 4.83
Deep Learning (Bayesian LWTA) | 1.08 | 0.80 | 0.87 | 0.69 | 28.56 | 4.33
| INFLAT | STOCKS | GDP
| MSE | MAE | MSE | MAE | MSE | MAE
Satellite Modelling(BMA) | 6.73 | 1.90 | 472.83 | 17.32 | 8.80 | 2.42
Deep Learning (MXNET) | 12.61 | 2.95 | 158.55 | 8.69 | 4.55 | 1.74
Deep Learning (Bayesian ReLU) | 10.14 | 2.46 | 172.06 | 9.26 | 4.30 | 1.68
Deep Learning (Bayesian LWTA) | 5.51 | 1.92 | 171.01 | 9.86 | 6.58 | 2.00
| EXPORT | DEBT | GOVEXP
| MSE | MAE | MSE | MAE | MSE | MAE
Satellite Modelling(BMA) | 159.57 | 9.73 | 13.30 | 2.63 | 14.2 | 1.47
Deep Learning (MXNET) | 103.81 | 7.88 | 16.53 | 2.73 | 2.28 | 1.18
Deep Learning (Bayesian ReLU) | 100.36 | 8.10 | 12.10 | 2.41 | 1.31 | 0.84
Deep Learning (Bayesian LWTA) | 93.70 | 8.01 | 18.57 | 2.93 | 2.32 | 1.16

In order to compare the prediction of the two models with actual value of each
variable, the diagrams shown in the Appendix have been created. In particular,
from the graph in Figure 1 , which shows the scaling of the actual value of
YIELD10Y from August 2016 to November 2018, it can be clearly seen that the
Bayesian LWTA follows the trend of the actual value much better than the BMA,
which deviates significantly, given that it does not take into account the
contemporaneous interdependencies among variables in the long run. Of course,
there are cases such as Government Expense and Debt to GDP where Deep Learning
model also overshoots but those are variables heavily dependent on political
decisions which may diverge from the pattern observed especially in crisis
periods. One should also take into account the challenging nature of the
testing sample which includes a structural break (Subprime crisis) that has
not been observed in the train sample, during which important fiscal decisions
were taken from the US government in order to mitigate the crisis
repercussions.

Figure 1: 10 year Government bond yield change- Forecast Comparison. Actual
value vs Bayesian Model Average (BMA) vs Bayesian Neural Network with Local
Winner-Takes-All activation function (Bayesian LWTA).

Focusing on the rolling forecast experiment in Table 2, we notice, as
expected, that errors have reduced significantly vs the static case as more
current information space is available in each step of the forecasting
process. If the static experiment represents a situation of forecasting a
macro variable full path on the basis of a model and a scenario provided, the
rolling forecast experiment simulates a situation of a practitioner that re-
trains a model on a monthly basis so as to have a hint on what will happen in
the next month. From a benchmarking perspective even in the rolling forecast
case Bayesian Deep Learning algorithms clearly outperform the BMA model in all
variables as they better account for contemporaneous relationships across
variables.

By examining Figs. 7 to 12 in Appendix B we notice that both BMA and the best
performing Deep Learning algorithm (Bayesian ReLU) follow closer the trend in
dependent variables evolution but the former in many cases overshoots
rendering the Deep Learning alternative more robust for macroeconomic
projections under a crisis situation.

Table 2: Rolling Forecast Error Metrics for each variable: MSE stands for Mean
Square Error and MAE stands for Mean Absolute Error.

Method | YIELD10Y | UNR | RRE
---|---|---|---
| MSE | MAE | MSE | MAE | MSE | MAE
Satellite Modeling (BMS) | 3.34 | 1.29 | 0.4 | 0.48 | 18.02 | 3.32
Deep Learning (MXNET) | 1.34 | 0.87 | 0.74 | 0.66 | 29.40 | 4.34
Deep Learning (Bayesian ReLU) | 0.50 | 0.49 | 0.09 | 0.23 | 15.50 | 3.08
Deep Learning (Bayesian LWTA) | 1.44 | 0.96 | 0.26 | 0.38 | 22.16 | 3.77
| INFLAT | STOCKS | GDP
| MSE | MAE | MSE | MAE | MSE | MAE
Satellite Modelling(BMA) | 3.07 | 1.30 | 203.02 | 11.43 | 2.59 | 1.26
Deep Learning (MXNET) | 5.40 | 1.82 | 144.25 | 9.29 | 2.59 | 1.28
Deep Learning (Bayesian ReLU) | 2.96 | 1.07 | 151.93 | 7.99 | 0.48 | 0.51
Deep Learning (Bayesian LWTA) | 4.62 | 1.49 | 157.06 | 8.49 | 3.25 | 1.38
| EXPORT | DEBT | GOVEXP
| MSE | MAE | MSE | MAE | MSE | MAE
Satellite Modelling(BMA) | 72.56 | 6.96 | 6.98 | 2.05 | 14.40 | 1.12
Deep Learning (MXNET) | 92.16 | 7.43 | 14.95 | 2.72 | 1.10 | 0.82
Deep Learning (Bayesian ReLU) | 87.93 | 7.26 | 2.26 | 1.13 | 0.40 | 0.50
Deep Learning (Bayesian LWTA) | 102.19 | 8.18 | 2.99 | 1.42 | 1.14 | 0.90

##  6 Conclusions

In this study we explore a novel approach of now-casting and forecasting the
macroeconomic status of a country using deep learning techniques focusing
particularly on the US economy but the methodology can be applied also to
other countries. Deep Learning algorithms have a short history in economics
but this new generation of statistical algorithms offer the necessary
flexibility in modelling multivariate time series, as its structure includes a
cascade of many layers with non- linear processing agents.

In particular, we identify the main channels of risk propagation in a
recurrent form to account of all the existing evidence of feedback effects in
a macroeconomic system by putting all the components together in a
multivariate structure. Our approach takes into account the dynamic nature of
the economy, through the multivariate training of deep neural networks, that
employ multivariate input and output layers which are able to capture the
cross correlation between macroeconomic variables. Training is performed as
one big complex network minimizing estimation errors and double counting
effects among various financial variables. Benchmarking a series of Deep
Learning algorithms versus Bayesian Model regressions on a test sample that
includes a financial turbulent period in the US (2008 – 2012) we find that
Deep Learning models provide better forecast both on a static perspective
(model train in 1973-2005 period and forecast on 2006 – 2018 period) and a
dynamic perspective (initial model train in 1973-2005 period and rolling
forecast with continuous re-training during the 2006 – 2018 period). Examining
both error metrics and relevant plots it is evident that deep learning
algorithms capture better the realized trends, especially in cases where the
absence of linearities and the contemporaneous dependencies cause traditional
modes to overshoot.

This first attempt at employing Deep Learning in macroeconomic time series
forecasting shows that potential benefits may pave the ground for a wider
spectrum of application in economic sciences. Of course, deep learning
techniques even though they better address non-linear patterns they are not a
panacea, especially in such challenging problem as the prediction of the Sub-
prime crisis in the US, but they certainly lie in the correct path. Criticism
could rely on the black box nature of the algorithm which when compared to
traditional econometrics does not provide a clear view of the economic
relationships, but in any case, as it has been proven empirically,the economy
is not dominated by clear linear patterns but from non-linear interactions
which constantly evolve. Especially under the rolling forecast framework where
both techniques follow the realized trend, one could use the results of the 2
techniques (Linear models – Deep Learning models) in a combined way, so that
the linear model provides a first order approximation of the problem at hand
revealing the economic rational and use also the more precise non-linear model
to correct for temporal fluctuations.

## References

  * [1] Nesreen K Ahmed, Amir F Atiya, Neamat El Gayar, and Hisham El-Shishiny.  An empirical comparison of machine learning models for time series forecasting.  Econometric reviews, 29(5-6):594–621, 2010.
  * [2] Mascia Bedendo and Paolo Colla.  Sovereign and corporate credit risk: Evidence from the eurozone.  Journal of corporate Finance, 33:34–52, 2015.
  * [3] Gianluca Bontempi, Souhaib Ben Taieb, and Yann-Aël Le Borgne.  Machine Learning Strategies for Time Series Forecasting, pages 62–77.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.
  * [4] An-Sing Chen and Mark T Leung.  Regression neural network for error correction in foreign exchange forecasting and trading.  Computers & Operations Research, 31(7):1049–1068, 2004.
  * [5] Sheng-Syan Chen, Hsien-Yi Chen, Chong-Chuo Chang, and Shu-Ling Yang.  The relation between sovereign credit rating revisions and economic growth.  Journal of Banking & Finance, 64:90–100, 2016.
  * [6] Aaron Smalter Hall et al.  Machine learning approaches to macroeconomic forecasting.  The Federal Reserve Bank of Kansas City Economic Review, 103(63):2, 2018.
  * [7] Christos Katris.  Prediction of unemployment rates with time series and machine learning techniques.  Computational Economics, 55(2):673–706, 2020.
  * [8] Manav Kaushik and AK Giri.  Forecasting foreign exchange rate: a multivariate comparative analysis between traditional econometric, contemporary machine learning & deep learning techniques.  arXiv preprint arXiv:2002.10247, 2020.
  * [9] Yun Liao et al.  Machine learning in macro-economic series forecasting.  International Journal of Economics and Finance, 9(12):71–76, 2017\.
  * [10] Paolo Manasse and Nouriel Roubini.  “rules of thumb” for sovereign debt crises.  Journal of International Economics, 78(2):192–205, 2009.
  * [11] Marcelo C Medeiros, Gabriel FR Vasconcelos, Álvaro Veiga, and Eduardo Zilberman.  Forecasting inflation in a data-rich environment: the benefits of machine learning methods.  Journal of Business & Economic Statistics, 39(1):98–119, 2021\.

##  Appendix A Appendix

###  A.1 Static Forecast

Figure 2: Unemployment Rate- Forecast Comparison. Actual value vs Bayesian
Model Average (BMA) vs Bayesian Neural Network with Local Winner-Takes-All
activation function (Bayesian LWTA). Figure 3: Real Estate price growth-
Forecast Comparison. Actual value vs Bayesian Model Average (BMA) vs Bayesian
Neural Network with Local Winner-Takes-All activation function (Bayesian
LWTA). Figure 4:  Inflation- Forecast Comparison. Actual value vs Bayesian
Model Average (BMA) vs Bayesian Neural Network with Local Winner-Takes-All
activation function (Bayesian LWTA). Figure 5: Stock market returns- Forecast
Comparison. Actual value vs Bayesian Model Average (BMA) vs Bayesian Neural
Network with Local Winner-Takes-All activation function (Bayesian LWTA).
Figure 6: GDP growth- Forecast Comparison. Actual value vs Bayesian Model
Average (BMA) vs Bayesian Neural Network with Local Winner-Takes-All
activation function (Bayesian LWTA).

###  A.2 Rolling Forecast

Figure 7: 10 year Government bond yield change- Forecast Comparison. Actual
value vs Bayesian Model Average (BMA) vs Bayesian Neural Network with ReLU
activation function (Bayesian ReLU). Figure 8: Unemployment Rate- Forecast
Comparison. Actual value vs Bayesian Model Average (BMA) vs Bayesian Neural
Network with ReLU activation function (Bayesian ReLU). Figure 9: Real Estate
price growth- Forecast Comparison. Actual value vs Bayesian Model Average
(BMA) vs Bayesian Neural Network with ReLU activation function (Bayesian
ReLU). Figure 10:  Inflation- Forecast Comparison. Actual value vs Bayesian
Model Average (BMA) vs Bayesian Neural Network with ReLU activation function
(Bayesian ReLU). Figure 11: Stock market returns- Forecast Comparison. Actual
value vs Bayesian Model Average (BMA) vs Bayesian Neural Network with ReLU
activation function (Bayesian ReLU). Figure 12: GDP growth- Forecast
Comparison. Actual value vs Bayesian Model Average (BMA) vs Bayesian Neural
Network with ReLU activation function (Bayesian ReLU).