series prediction using Gaussian process models is considered in Section VI-A;
adversarial learning using Bayesian deep neural networks is presented in
Section VI-B; and lastly social group clustering and image completion using
unsupervised tensor decompositions are demonstrated in Section VI-C.

###  VI-A Time Series Prediction via GPs

In the following, we present an important signal processing and machine
learning application, namely the _time series prediction_ , using non-
parametric GPs. We will focus on the GP regression models with the family of
sparse spectrum kernels introduced in Section IV-B. To demonstrate the
advantages of the sparsity-promoting GP models over other competing
counterparts, we selected a number of classic time series datasets such as
$\textrm{CO}_{2}$, Electricity, Unemployment777These datasets are available
from the UCL repository as well as a “fresh” real-world 5G wireless traffic
dataset in our tests. Data descriptions are given in Table II.

Table II: Descriptions of the selected datasets. The training data, $\mathcal{D}$, is used for optimizing the hyper-parameters of the learning model, while the test data, $\mathcal{D}_{*}$, is used for evaluating the prediction accuracy. The numbers given in the last two columns are the training sample size and test sample size, respectively. Name |  Data Description | Training $\mathcal{D}$ | Test $\mathcal{D}_{*}$
---|---|---|---
ECG |  Electrocardiography of an ordinary person measured over a period of time | 680 | 20
$\textrm{CO}_{2}$ |  Carbon dioxide concentration observed from 1958 to 2003 | 481 | 20
Electricity |  Monthly average residential electricity usage in Iowa City from 1971 to 1979 | 86 | 20
Employment |  Wisconsin employment status observed from January 1961 to October 1975 | 158 | 20
Hotel |  Monthly hotel occupied rooms collected from 1963 to 1976 | 148 | 20
Passenger |  Passenger miles flown domestic U.K. form July 1962 to May 1972 | 98 | 20
Clay |  Monthly production of clay bricks from January 1956 to August 1995 | 450 | 20
Unemployment |  Monthly U.S. female (16-19 years) unemployment figures from 1948 to 1981 | 380 | 20
5G wireless traffic |  Downlink data usage in a small cell observed in four weeks of 2021 | 607 | 67

####  VI-A1 Classic Datasets

In the sequel, we compare the performance of the sparsity-promoting GP models
using the original SM kernels [35, 36] and the modified GridSM kernel [7] with
that of a classic deep learning based time series prediction model, namely the
long-short-term-memory (LSTM) [68], as well as a canonical statistical model,
namely the autoregressive integrated moving average (ARIMA) model888ARMA model
can be regarded as a special case of a GP model adopting a specific sparse
kernel matrix. [69], from various different aspects. Furthermore, we compare
GP models with recently proposed Transformer-based time series prediction
model, called Informer, which successfully addressed the computation issues
and some inherent limitations of the encoder-decoder architecture in the
original Transformer model. For brevity, we name the first sparse spectrum GP
model (equivalent to using trigonometric basis functions) proposed in [35] as
SSGP, the GP model using the original SM kernel [36] as SMGP, and the most
recent one with the rectified GridSM kernel [7] as GSMGP. Their configurations
can be found in detail in [7].

Table III shows the obtained prediction accuracy of the various methods
quantified in terms of the prediction _mean-squared-error_ (MSE). It is
readily observed that the sparsity-promoting GP models, and in particular SMGP
and GSMGP, outperform all other competitors by far. The following facts need
to be mentioned. For both the classic LSTM and Informer models to achieve good
performance, the time series should, in general, be long so that the
underlying pattern can be learnt during the training phase. The ARIMA model
strongly relies on the optimal configuration of the parameters $(p,d,q)$, and
it is incapable for long term prediction. In contrast, the sparsity-promoting
GP models can automatically fit the underlying data pattern through solving
the hyper-parameters from maximizing the evidence function. We have also shown
in the supplement of [7] that the GSMGP is also superior to the GP models with
elementary kernel (such as the SE kernel, rational quadratic kernel, etc.) as
well as a hybrid of those.

Table III: Comparisons of various time series prediction models in terms of the prediction MSE. Herein, we let both the GSMGP and the SMGP employ $Q=500$ Gaussian mixture modes in their kernels, and the SSGP employs the same amount of trigonometric basis functions. The GSMGP samples $Q$ normalized frequency parameters, $\mu_{i}$, $i=1,2,\cdots,Q$, uniformly from $[0,1/2)$, while set the variance parameter to $\sigma=0.001$. The LSTM model follows a standard setup with one hidden layer and the dimension of the hidden state is set to 30. The Informer model follows the default setup given in the original paper [70]. The $\textrm{ARIMA}(p,d,q)$ model is a standard one with $(p=5,d=1,q=2)$. Name | GSMGP | SSGP | SMGP | LSTM | Informer | ARIMA
---|---|---|---|---|---|---
| MSE | MSE | MSE | MSE | MSE | MSE
ECG | 1.3E-02 | 1.6E-01 | 1.9E-02 | 2.1E-02 | 5.4E-02 | 1.8E-01
CO2 | 1.5E+00 | 2.0E+02 | 1.1E+00 | 2.1E+00 | 8.4E+01 | 4.9E+00
Electricity | 4.7E+03 | 8.2E+03 | 7.5E+03 | 4.7E+03 | 8.3E+03 | 1.2E+04
Employment | 1.1E+02 | 7.7E+01 | 0.7E+02 | 4.3E+02 | 2.0E+03 | 3.9E+02
Hotel | 8.9E+02 | 1.9E+04 | 2.8E+03 | 7.8E+03 | 2.3E+04 | 1.7E+04
Passenger | 1.9E+02 | 6.9E+02 | 1.6E+02 | 1.6E+02 | 1.2E+02 | 4.5E+03
Clay | 1.9E+02 | 5.3E+02 | 3.3E+02 | 2.7E+02 | 1.4E+02 | 3.3E+02
Unemployment | 3.6E+03 | 2.1E+04 | 1.4E+04 | 3.5E+03 | 3.8E+03 | 1.5E+04

Besides the improved prediction performance, the training time of the GSMGP
outperforms the original SMGP by far.999Here, training time refers to the
computational time required for training the learning models introduced in
Section V. For the selected datasets, SMGP requires training time in the
magnitude of $10^{3}$ seconds, while the GSMGP only requires $10^{2}$ seconds.
By reducing the number of Gaussian modes, $Q$, the SMGP is able to reduce its
training time albeit at the cost of sacrificing the fitting performance. On
the other hand, the GSMGP improves its training time by fixing the frequency
and variance parameters of the original SM kernel to known grids, so that the
evidence maximization task enjoys the favorable difference-of-convex structure
that can be efficiently handled by the MM algorithm introduced in Section V-B.
In addition to the reduced training time, the overall optimization performance
(including the convergence speed, chance of being trapped in a bad local
minimum, etc.) and the sparsity level of the solution have been significantly
improved. Detailed comparisons and pictorial illustrations can be found in
[7]. In comparison, the LSTM and ARIMA models require the least training time
in the magnitude of $10^{1}$ seconds on average. However, due to the huge
architecture adopted in the Informer model, the computational time is in the
magnitude of $10^{2}$ seconds on average. As it is readily observed from the
results, the sparsity-promoting property helps reducing the computational time
significantly; more importantly, the sparse solution identifies the most
effective frequency components of the data and, thus, leads to good model
interpretability.

####  VI-A2 Real 5G Dataset

Next, we focus on another favorable advantage of the GP models over their deep
learning counterparts, namely the natural uncertainty region of a point
prediction. We specifically select the real 5G wireless traffic dataset for
visualization purposes due to the high demand of such a wireless application
on a reasonable prediction uncertainty [25, 71]. The dataset was collected in
a small cell of a southern city in China, and it contains the downlink data
volume consumed by the mobile users located in the cell within each hour
during a period of four weeks. Accurate prediction of the future downlink data
consumption is vital to the operators for tuning the transmit power of the
base station and switching on/off it automatically. In this application,
uncertainty information is even more crucial because wrongly reducing the
transmit power may largely influence the mobile users’ surfing experience.

For this 5G wireless traffic prediction example, we constrained ourselves to a
GSMGP with $Q=500$, an SMGP with $Q=500$, a standard GP with a hybrid of 3
elementary kernels (two periodic kernels plus an SE kernel) as was used in
[25], and a classic LSTM deep learning model as described above. We used the
data collected in the first 607 samples for training the models and used the
last 67 samples to test their prediction performance. For comparing their
prediction accuracy, we chose the _mean-absolute-percentage-error_ (MAPE)
measure, which is commonly used for evaluating the wireless traffic prediction
error. The MAPE averaged over multiple test points is given by

$e_{\mathrm{MAPE}}=\frac{1}{n_{*}}\sum_{i=1}^{n_{*}}\left|\frac{y_{i}-\hat{y}_{i}}{y_{i}}\right|\times
100\%.$ (64)

The prediction performance of the above learning models is shown in Fig. 14.
It is readily seen that the GSMGP gives the best point prediction in terms of
the MAPE. Moreover, as we mentioned before, the focus of this example is
primarily on the uncertainty quantification. For GPs, the desired uncertainty
region can be obtained naturally by computing the posterior variances
associated with the test samples. In contrast, the classic LSTM model can only
provide point predictions without any uncertainty quantification. A recent
technique using the so-called deep ensembles [72] can be applied to quantify
the predictive uncertainty of the LSTM model. The common characteristic of
these techniques lies in that one has to train the models for multiple times,
using different configurations (such as different initial guesses, step sizes,
etc). However, such an approach increases substantially the computational load
compared to the Bayesian approach, especially when complex (deterministic)
learning models are involved. When comparing the uncertainty regions of the GP
models, we can observe that the one using a mixture of elementary kernels tend
to be conservative and show the largest uncertainty region. In contrast, both
SMGP and GSMGP provide rather accurate point predictions as well as smaller
uncertainty levels. It is noteworthy that SMGP presents less accurate point
prediction (using its posterior mean) compared to that of GSMGP, but its
uncertainty level is modestly larger that its counterpart. This suggests that,
in this case, SMGP is less favorable because wrong decisions of switching
on/off the BS are more likely made.

The above fitting results clearly demonstrate the advantages of the sparse
spectrum kernel-based GP models; however, the obtained performance depends on
the quality of the initialization. In particular, the method is sensitive to
the initial guess of the SM kernel. According to our experience, a reliable
initial guess can be obtained by fitting a periodogram (namely a nonparametric
approximation of the true spectral density) in the frequency domain. We could
also combine this strategy with the bootstrap technique to generate a number
of candidate initial guesses for avoiding bad local minima. Codes for
implementing the GSM kernel-based GP model are online available from
https://github.com/Paalis/MATLAB_GSM.

Figure 14: Comparison of 5G wireless traffic prediction performance obtained
from different models. Top left: GSMGP model with $Q=500$ fixed grids whose
$e_{\mathrm{MAPE}}=0.28$; Top right: SMGP model with $Q=500$ modes whose
$e_{\mathrm{MAPE}}=0.42$; Bottom left: a standard GP with a hybrid of 3
elementary kernels whose $e_{\mathrm{MAPE}}=0.30$; Bottom right: LSTM model
whose $e_{\mathrm{MAPE}}=1.12$. The gray shaded areas represent the
uncertainty region (computed as the posterior variances) of the GP models.

###  VI-B Adversarial Learning via Bayesian Deep Neural Networks

Despite the widespread success of DNNs, recent investigations have revealed
their high susceptibility to adversarial examples; that is, cleverly crafted
examples whose sole purpose is that of fooling a considered model into
misclassification. Adversarial examples can be constructed using various
approaches, e.g. FSGM[73] and CW[74]. A popular and powerful attack is the
projected gradient descent (PGD)[75] attack. Under this scheme, the adversary
is assumed to have access to the objective function of the target model,
$L(\bm{w},\bm{x},\bm{y})$, where $\bm{w}$ are the model trainable parameters,
$\bm{x}$ the input, and $\bm{y}$ the predicted output variables. On this
basis, the adversary performs an iterative computation; at each iteration,
$t$, the adversary computes an (e.g., $\ell_{\infty}$-bounded) adversarial
perturbation of the training set examples $\bm{x}$, based on a multi-step PGD
procedure that reads:

$\displaystyle\bm{x}^{t+1}=\prod_{\bm{x}+\mathcal{S}}(\bm{x}^{t}+a\
\mathrm{sgn}(\nabla_{\bm{x}}L(\bm{w},\bm{x},y)))$ (65)

where $\mathcal{S}$ is the set of allowed perturbations, that is the
manipulative power of the adversary, e.g. $\ell_{\infty}$-ball around
$\bm{x}$; $\mathrm{sgn}(\cdot)$ denotes the sign function that extracts the
sign of a real number. In this context, even some minor, and many times
imperceptible modifications, can successfully “attack” the model, resulting in
severe performance degradation. This frailness of DNNs casts serious doubt
about their deployment in _safety-critical applications_ , e.g., autonomous
driving [76].

Drawing upon this vulnerability, significant research effort has been recently
devoted towards more reliable and robust DNNs. On this basis, several
adversarial attacks and defenses have been proposed in the literature, e.g.
adversarial training[75, 77, 78]. Among these, lies the stochastic modeling
rationale; its main operating principle is founded upon the introduction of
stochasticity in the considered architecture, e.g., by randomizing the input
data and/or the learning model itself [79, 80, 81]. Clearly, the Bayesian
reasoning, which treats parameters as random entities instead of deterministic
values, seeking to infer an appropriate generative process, seems to offer a
natural stochastic defense framework towards more adversarially robust
networks. It must be emphasized that the Bayesian techniques differ from the
more standard randomized ones, which simply rely on the randomization of
deterministic variables, in the context of the standard deterministic neural
networks. Such techniques can be fairly easily handled and attacked. In
contrast, in the Bayesian framework, the whole modeling and learning is built
upon statistical arguments and the training involves learning of
distributions.

Therefore, in the following, we focus on a recent application of the Bayesian
rationale towards adversarial robustness. Specifically, we present the novel
Bayesian deep network design paradigm proposed in [3] that yields state-of-
the-art performance against powerful gradient-based adversarial attacks, e.g.
PGD [75]. The key aspect of this method is its doubly stochastic nature
stemming from two separate sampling processes relying on Bayesian arguments:
a) the sparsity inducing non-parametric link-wise IBP prior introduced in
Section IV-A3, and b) a stochastic adaptation of the biologically inspired and
competition-based LWTA activation, as discussed in Sections IV-A3 and V-D.

We investigate the potency of LWTA-based networks against adversarial attacks
under an Adversarial Training regime; we employ a PGD adversary [75]. To this
end, we use the well-known WideResNet-34 [82] architecture, considering three
different widen factors: 1, 5, and 10; note from the definition of the
WideResNet-34, the larger the widen factor the larger the network. We focus on
the CIFAR-10 dataset and adopt experimental settings similar to [83]. We use a
batch size of 128 and an initial learning rate of 0.1; we halve the learning
rate at every epoch after the $75$-th epoch. We use a single sample for
prediction. All experiments were performed using a single NVIDIA Quadro P6000.

For evaluating the robustness of this structure, we initially consider the
conventional PGD attack with 20 steps, step size $0.007$ and $\epsilon=8/255$,
which are the two parameters required by the PGD. In Table IV, we compare the
robustness of LWTA-based WideResNet networks against the baseline results of
[83]. As we observe, the Stochastic LWTA-based networks yield significant
improvements in robustness under a traditional PGD attack; they retain
extremely high natural accuracy (up to $\approx 13\%$ better), while
exhibiting a staggering, up to $\approx 32.6\%$, difference in robust accuracy
compared to the exact same architectures employing the conventional ReLU-based
nonlinearities and trained in the same fashion. Natural accuracy refers to the
performance based on non-adversarial examples, while robust accuracy refers to
the case where the network is tested against adversarial examples.

Table IV: Natural and Robust accuracy under a conventional PGD attack with 20
steps and $0.007$ step-size using WideResNet-34 models with different widen
factors. We use the same PGD-based Adversarial Training scheme for all models
[75].

Adversarial Training-PGD
---
|  Natural Accuracy ($\%$) |  Robust Accuracy ($\%$)
Widen Factor | Baseline | Stochastic LWTA | Baseline | Stochastic LWTA
1 | 74.04 | 87.0 | 49.24 | 81.87
5 | 83.95 | 91.88 | 54.36 | 83.4
10 | 85.41 | 92.26 | 55.78 | 84.3

Table V: Robust Accuracy $(\%)$ comparison under the AutoAttack framework.
$\dagger$ denotes models that are trained with additional unlabeled data. The
AutoAttack performance corresponds to the final robust accuracy after
employing all the attacks in AA. Results directly from the AA leaderboard.

Method | AutoAttack
---|---
HE [84] | 53.74
WAR [83] | 54.73
Pre-training [85]$\dagger$ | 54.92
[86]$\dagger$ | 65.88
WAR [83]$\dagger$ | 61.84
Ours (Stochastic-LWTA/PGD/WideResNet-34-1) | 74.71
Ours (Stochastic-LWTA/PGD/WideResNet-34-5) | 81.22
Ours (Stochastic-LWTA/PGD/WideResNet-34-10) | 82.60

Further, to ensure that this approach does not cause the well-known obfuscated
gradient problem [87], stronger parameter-free attacks were adopted using the
newly introduced AutoAttack (AA) framework [88]. AA comprises an ensemble of
four powerful white-box and black-box attacks, e.g., the commonly employed
APGD attack; this is a step-free variant of the standard PGD attack [75],
which avoids the complexity and ambiguity of step-size selection. In addition,
for the entailed $L_{\infty}$ attack, the common $\epsilon=8/255$ value was
used. Thus, in Table V, we compare the LWTA-based networks to several recent
state-of-the-art approaches evaluated on
AA101010https://github.com/fra31/auto-attack. The reported accuracies
correspond to the final reported robust accuracy of the methods after
sequentially performing all the considered AA attacks. Once again, we observe
that these stochastic and sparse networks yield state-of-the-art (SOTA)
robustness against all SOTA methods, with an improvement of $\approx 16.72\%$,
even when compared with methods that employ substantial data augmentation to
increase robustness, e.g. [86]. More results are reported in [3]. All results
vouch for the potency of Stochastic LWTA networks in adversarial settings.

Finally, since the newly proposed networks consist of stochastic components,
i.e., the competitive random sampling procedure to determine the winner in
each LWTA block, the output of the classifier might change at each iteration;
this obstructs the attacker from altering the final decision. To counter such
randomness in the involved computations, in [88] the APGD attack is combined
with an averaging procedure of 20 computations of the gradient at the same
point. This technique is known as Expectation over Transformation (EoT) [87].
Thus, AA was used jointly with EoT for further performance evaluation of the
LWTA-based networks. The corresponding results are presented in Table VI. As
we observe, all of the considered networks retain state-of-the-art robustness
against the powerful AA & EoT attacks. This conspicuously supports the
usefulness of the stochastic LWTA activations towards adversarial robustness.
Further explanations on why this performance is obtained are provided in,
e.g., [2].

Table VI: Robustness against AA combined with 20 iterations of EoT. APGD-DLR corresponds to the APGD attack, using a different loss, i.e., the Difference of Logits Ratio [88]. Widen Factor | Natural Accuracy | APGD | APGD-DLR
---|---|---|---
1 | 87.00 | 79.67 | 76.15
5 | 91.88 | 81.67 | 77.65
10 | 92.26 | 82.55 | 79.00

###  VI-C Unsupervised Learning via Bayesian Tensor Decompositions

In this subsection, we present some recent advances of Bayesian tensor
decompositions in two unsupervised learning applications: _social group
clustering_ and _image completion_. The first application adopts Bayesian
tensor CPD [14, 11], while the second one employs Bayesian tensor TTD [48, 49,
50]. Exploiting the GSM-based sparsity-promoting prior as introduced in
Section IV-C and the effective MF-VI inference as introduced in Section V-C,
the resulting algorithms offer nice features bypassing the need of hyper-
parameters tuning and in dealing with overfitting.

####  VI-C1 Bayesian Tensor CPD for Social Group Clustering

In contrast to matrix decompositions, which are not unique in general (unless
certain constraints are imposed), tensor CPD is provably unique under mild
conditions [42]. This appealing property has made CPD an important tool for
extracting the underlying signals/patterns from the observed data. The
interpretability of CPD model can be further enhanced by incorporating some
side information, e.g., non-negativeness [11], into model learning. Here,
using the ENRON E-mail corpus dataset (a $3$-D tensor with the size $184\times
184\times 44$), we demonstrate how the Bayesian tensor CPD (with non-negative
factor matrices) [11] can be used to simultaneously determine the number of
social groups, cluster people into different groups, and extract interpretable
temporal profiles of different social groups.

The considered dataset records the number of E-mail exchanges between $184$
people within $44$ months. In particular, each entry is the number of E-mails
exchanged between two people within a certain month. The physical meaning of
three tensor dimensions are: the people who sent E-mails, the people who
received E-mails, and the months, respectively. After applying Bayesian tensor
CPD, the automatically determined tensor rank can be interpreted as the number
of underlying social groups. For the first two factor matrices, the physical
meaning for each element is that it quantifies the “score” that a particular
person belongs to a particular E-mail sending and receiving group,
respectively. For the third factor matrix, each column corresponds to the
temporal profile of the associated social group, see discussion in [11].

Figure 15: Bayesian tensor CPD for social group clustering.

Typically, we set the initial number of the social groups (i.e., tensor rank)
large, (e.g., the minimal dimension of the tensor data, $44$, in the above
example), and then run the Bayesian learning algorithm [11] to automatically
determine the number of social groups that best interprets the data. As seen
in Fig. 15, the estimated number of social groups gradually reduces to the
value $4$, indicating four underlying social groups. This is consistent with
the results published in [43, 89], which are obtained via trial-and-error
experiments. The clustering results can be read from the first factor matrix,
which is of size $184\times 4$. Specifically, for each column, only a few
elements have non-zero values, and they can be used to identify the
significance of the corresponding people in this social group. After sorting
the scores of each column in the first factor matrix, the people with top $10$
scores in each social group are shown in Table V of [11]. The clustering
results are well interpretable as illustrated in Fig. 15. For example, the
people in the first group work either in legal department or as lawyers, thus
are clustered together. Moreover, interesting temporal patterns can be
observed from the third factor matrix. It is clear that when the company has
important events such as the change of CEO, crisis breaks and bankruptcy,
distinct peaks appear.111111This information can be acquired via checking the
E-mail content and related research works, e.g., [43, 89]. The E-mail data
analysis results have showcased the appealing advantage of Bayesian SAL in the
context of tensor CPD, that is, the automatic determination of the social
group number. This is important, since it leads to interpretable results on
the group member and the temporal profiles can be naturally obtained.

####  VI-C2 Bayesian Tensor TTD for Image Completion

Color images are naturally $3$-D tensor (with two spatial dimensions and one
RGB dimension). To fully exploit the inherent structures of images, recent
image completion works [90, 91, 92] usually fold an image into a higher
dimensional tensor (e.g., $9$-D tensor), and then apply tensor decompositions
to recover the missing pixels, among which TTD is one of the most important
tools due to its excellent performance. The folding operation is called tensor
augmentation, see details in, e.g., [90, 91, 92]. For a $P$-D tensor, TTD has
$P-1$ hyper-parameters (called TT ranks). Manually tuning different
combinations of these hyper-parameters for overfitting avoidance is time-
consuming. To facilitate this process, recent advances, see e.g., [48, 49,
50], first assume large values for TT-ranks, employ the sparsity-promoting GSM
prior, and use variational inference for effective Bayesian SAL. The resulting
algorithms can automatically learn the most suitable TT-ranks to match the
underlying data pattern [48, 49, 50].

As an illustration, we consider the image completion of $5$ images. Each image
is with size $256\times 256\times 3$, and $80$ percent of its pixels are
randomly removed. After tensor augmentation, they are folded into a $9$-D
tensor with size $16\times 4\times 4\times 4\times 4\times 4\times 4\times
4\times 3$. We assume the initial TT-ranks is set as large as $60$, and apply
the Bayesian TTD algorithm [50] to complete the missing pixels. In comparison,
we present the image completion results from other recent TTD algorithms: TTC-
TV [90], TMAC-TT [91], and STTO [92], with the suggested hyper-parameter
settings in their papers. The widely-used metrics, e.g., the peak signal-to-
noise ratio (PSNR), were reported in Table II of [50], from which it can be
concluded that the Bayesian TTD algorithm achieves the best overall
performance. Particularly, in most cases, the Bayesian TTD algorithm recovers
images with $1$-$5$ dB higher PSNR than other algorithms. This is visually
evident in the recovered images shown in Fig. 16.  In this example, the
Bayesian SAL-based tensor TTD [50] gets rid of the costly hyper-parameter
tuning process for balancing the trade-off between data fitting and the noise
overfitting; it directly learns these hyper-parameters from observations and
shows excellent image restoration performance.

Some suggestions are provided on the real implementations of Bayesian tensor
decomposition algorithms. a) Initialization: To assist the algorithm to avoid
being trapped in a poor local minima, the initial factor matrix is usually set
equal to the singular value decomposition approximation of the matrix, which
is obtained by unfolding the tensor data along a specific dimension, see e.g.,
[10, 11, 50, 14]. b) On-the-fly pruning: To accelerate the learning process
while not affecting the convergence, in each iteration, if some of the columns
in the factor matrices are found to be indistinguishable from an all-zeros
column vector, they can be safely pruned, see, e.g., the discussion in [10,
11]. c) Robustness against strong noise: When the corrupting noise sources are
of large power, it was shown in [10] that slowing the noise precision learning
can increase the robustness of the algorithm. Demo codes of Bayesian tensor
CPD and TTD algorithms are online available from https://github.com/leicheng-
tensor?tab=repositories.

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure 16:  Experimental results for visual comparison on image completion
with $80\%$ missing data. Ground-truth images are in the top row. The second
row includes the images with missing values. The third to the bottom rows
include results from Bayesian TTD, TTC-TV, TMAC-TT, and STTO.

##  VII Concluding Remarks and Future Research

In this article, we have presented an overview of some state-of-the-art
sparsity-promoting priors for both Bayesian linear and non-linear modeling, as
well as parametric and non-parametric models. In particular, these priors are
incorporated into three advanced data analysis tasks, namely, the GP models,
Bayesian deep neural networks, and tensor decomposition models, that can be
applied to a wide spectrum of signal processing and machine learning
applications. Commonly used inference algorithms for estimating the associated
hyper-parameters and the (approximate) posterior distributions have also been
discussed.

To demonstrate the effectiveness of the considered advanced sparsity-promoting
models, we have carefully selected four important use cases, namely the time
series prediction via the GP regression, adversarial learning via Bayesian
deep neural networks, social group clustering and image completion using
tensor decompositions. The reported results indicate that: a) Sparsity-
promoting priors are able to adapt themselves to the given data and enable
automatic model structure selection; b) The resulting sparse solution can
better reveal the underlying (physical) characteristics of a target
system/signal with only a few effective components; c) Sparsity-promoting
priors, acting as the counterparts of the regularizers in optimization-based
methods, can effectively help to avoid data overfitting, especially when the
data size is relatively small; and (4) Sparsity-promoting priors lead to
natural and more reasonable uncertainty quantification which is hard to obtain
via traditional deep learning models.

Despite the rapid development of Bayesian learning and the enumerated
advantages of sparsity-promoting models, still, such models are confronted
with some challenges. Some open research directions are summarized as follows.

  * •

Quality of the posterior/predictive distribution. As we mentioned before, a
unique feature of Bayesian learning models lies in its posterior distribution
that can be used to generate a point prediction and meanwhile provide an
uncertainty quantification of the point prediction. Various recent works [23,
93, 94] indicate that the quality of the posterior distribution that is
derived via the Bayesian deep neural networks and GP models can be
significantly improved by using cold tempering:

$p(\bm{\theta}|\bm{X},\bm{y})\propto\left(p(\bm{y}|\bm{X},\bm{\theta})\cdot
p(\bm{\theta})\right)^{1/T},\quad T<1.$ (66)

Two conjectures lie in the misspecification of the learning model and careless
adoption of an inadequate, unintentionally informative prior[93]. Deeper
analysis of such behavior is highly demanded. It is of great value to verify
either analytically or experimentally if adopting the sparsity-promoting
priors can help to avoid the use of cold tempering. It is also interesting to
investigate the generalization property of the sparsity-promoting Bayesian
learning models.

Another path to improve the quality of posterior/predictive distribution is
through designing more effective inference methods. There is a recent trend to
integrate the strengths of the variational inference [58] and Monte Carlo
sampling [95] in a principled fashion, see, e.g., [96], in order to achieve
the best trade-off between the inference accuracy and the computational
efficiency. Given multiple inference results, Bayesian deep ensembles, e.g.,
[23], were also proposed for improved posterior/predictive distribution. It
will be interesting to investigate how these recent general-purpose advances
can be tailored to the sparsity-aware Bayesian modeling introduced in this
article.

  * •

More emerging applications in complex systems. We have witnessed various
applications of Bayesian learning models, and they will surely continue to
play important role in _large and complex systems_ , such as 6G wireless
communication systems [25] and autonomous systems [76], that are constantly
facing rapid changing environments and critical decision making. Sparsity-
promoting models are flexible enough to adapt themselves (for instance by
nulling irrelevant basis kernels in the GP models) to changing data profile
and provide rather reliable uncertainty quantification with small
computational expense.

  * •

More and tighter interactions of the three data analysis tools. Each of the
three data analysis tool (introduced in this article) has already tapped into
the design of other tools, see, e.g., DNN and GP [16], GP and tensor [97], DNN
and tensor [98], to achieve performance enhancement by borrowing the strengths
of other tools. However, many of these works are not under the framework of
Bayesian SAL, and thus do not possess the associated comparative advantages.
It is promising to investigate how to combine the strengths of the three
popular models, especially under the Bayesian SAL umbrella, to tackle
challenging tasks such as non-linear regression for multi-dimensional and even
heterogeneous data with deep kernels.

  * •

Sparsity-awareness in emerging learning paradigms. Recently, we have witnessed
various new paradigms, including, for instance, federated learning, life-long
learning, meta learning, etc. We strongly believe that by further encoding
sparsity-awareness through Bayesian sparse learning strategies, these emerging
learning paradigms can further improve their learning efficiency over the
learning models that were introduced in this article.

## References

  * [1] S. Theodoridis, _Machine learning: A Bayesian and optimization perspective, 2nd edition_. Academic press, 2020.
  * [2] K. Panousis, S. Chatzis, and S. Theodoridis, “Nonparametric Bayesian deep networks with local competition,” in _Proc. International Conference on Machine Learning (ICML)_ , 2019, pp. 4980–4988.
  * [3] K. Panousis, S. Chatzis, A. Alexos, and S. Theodoridis, “Local competition and stochasticity for adversarial robustness in deep learning,” in _Proc. International Conference on Artificial Intelligence and Statistics (AISTAT)_ , vol. 130, 2021, pp. 3862–3870.
  * [4] K. Panousis, S. Chatzis, and S. Theodoridis, “Stochastic local winner-takes-all networks enable profound adversarial robustness,” in _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , 2022\.
  * [5] C. Louizos, K. Ullrich, and M. Welling, “Bayesian compression for deep learning,” in _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , 2017, pp. 3288–3298.
  * [6] S. Ghosh, J. Yao, and F. Doshi-Velez, “Model selection in Bayesian neural networks via horseshoe priors,” _Journal of Machine Learning Research_ , vol. 20, no. 182, pp. 1–46, 2019.
  * [7] F. Yin, L. Pan, T. Chen, S. Theodoridis, and Z.-Q. Luo, “Linear multiple low-rank kernel based stationary Gaussian processes regression for time series,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 5260–5275, 2020.
  * [8] T. Paananen, J. Piironen, M. R. Andersen, and A. Vehtari, “Variable selection for Gaussian processes via sensitivity analysis of the posterior predictive distribution,” in _Proc. International Conference on Artificial Intelligence and Statistics (AISTAT)_ , 2019, pp. 1743–1752.
  * [9] H. Kim and Y. W. Teh, “Scaling up the automatic statistician: Scalable structure discovery using Gaussian processes,” in _Proc. International Conference on Artificial Intelligence and Statistics (AISTAT)_ , vol. 84, Lanzarote, Spain, 2018, pp. 575–584.
  * [10] L. Cheng, Z. Chen, Q. Shi, Y.-C. Wu, and S. Theodoridis, “Towards flexible sparsity-aware modeling: Automatic tensor rank learning using the generalized hyperbolic prior,” _IEEE Transactions on Signal Processing_ , vol. 1, no. 1, pp. 1–16, 2022.
  * [11] L. Cheng, X. Tong, S. Wang, Y.-C. Wu, and H. V. Poor, “Learning nonnegative factors from tensor data: Probabilistic modeling and inference algorithm,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 1792–1806, 2020.
  * [12] Y. Zhou and Y.-M. Cheung, “Bayesian low-tubal-rank robust tensor factorization with multi-rank determination,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 62–76, 2019.
  * [13] L. Cheng, Y.-C. Wu, and H. V. Poor, “Probabilistic tensor canonical polyadic decomposition with orthogonal factors,” _IEEE Transactions on Signal Processing_ , vol. 65, no. 3, pp. 663–676, 2016.
  * [14] Q. Zhao, L. Zhang, and A. Cichocki, “Bayesian CP factorization of incomplete tensors with automatic rank determination,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 9, pp. 1751–1763, 2015\.
  * [15] Z. Zhang and C. Hawkins, “Variational Bayesian inference for robust streaming tensor factorization and completion,” in _Proc. IEEE International Conference on Data Mining (ICDM)_ , Singapore, 2018, pp. 1446–1451.
  * [16] Y. Dai, T. Zhang, Z. Lin, F. Yin, S. Theodoridis, and S. Cui, “An interpretable and sample efficient deep kernel for Gaussian process,” in _Proc. International Conference on Uncertainty in Artificial Intelligence (UAI)_ , 2020, pp. 759–768.
  * [17] L. Cheng, Y.-C. Wu, and H. V. Poor, “Scaling probabilistic tensor canonical polyadic decomposition to massive data,” _IEEE Transactions on Signal Processing_ , vol. 66, no. 21, pp. 5534–5548, 2018.
  * [18] L. Cheng and Q. Shi, “Towards overfitting avoidance: Tuning-free tensor-aided multi-user channel estimation for 3D massive MIMO communications,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 15, no. 3, pp. 832–846, 2021.
  * [19] C. M. Bishop, _Pattern Recognition and Machine Learning_. Springer, 2006.
  * [20] S. F. Gull, “Bayesian inductive inference and maximum entropy,” in _Maximum-entropy and Bayesian methods in science and engineering_. Springer, 1988, pp. 53–74.
  * [21] G. Schwarz, “Estimating the dimension of a model,” _The annals of statistics_ , pp. 461–464, 1978.
  * [22] D. J. MacKay, “Bayesian interpolation,” _Neural computation_ , vol. 4, no. 3, pp. 415–447, 1992.
  * [23] A. G. Wilson and P. Izmailov, “Bayesian deep learning and a probabilistic perspective of generalization,” in _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * [24] C. E. Rasmussen and C. Williams, _Gaussian Processes for Machine Learning_. MIT Press, 2006.
  * [25] Y. Xu, F. Yin, W. Xu, J. Lin, and S. Cui, “Wireless traffic prediction with scalable Gaussian process: Framework, algorithms, and verification,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 6, pp. 1291–1306, June 2019.
  * [26] D. J. C. MacKay, “Gaussian processes – a replacement for supervised neural networks?” 1997. [Online]. Available: http://www.inference.org.uk/mackay/gp.pdf
  * [27] J. Lee, J. Sohl-dickstein, J. Pennington, R. Novak, S. Schoenholz, and Y. Bahri, “Deep neural networks as Gaussian processes,” in _Proc. of International Conference on Learning Representations (ICLR)_ , Vancouver, BC, Canada, 2018.
  * [28] Y. C. Eldar and G. Kutyniok, _Compressed sensing: theory and applications_. Cambridge University Press, 2012.
  * [29] M. Elad, _Sparse and redundant representations: from theory to applications in signal and image processing_. Springer Science Business Media, 2010.
  * [30] D. F. Andrews and C. L. Mallows, “Scale mixtures of normal distributions,” _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 36, no. 1, pp. 99–102, 1974.
  * [31] D. P. Wipf and B. D. Rao, “Sparse Bayesian learning for basis selection,” _IEEE Transactions on Signal processing_ , vol. 52, no. 8, pp. 2153–2164, 2004.
  * [32] D. J. MacKay, “Probable networks and plausible predictions—a review of practical bayesian methods for supervised neural networks,” _Network: computation in neural systems_ , vol. 6, no. 3, pp. 469–505, 1995.
  * [33] J. Bridle, “Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters,” _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 2, 1989.
  * [34] D. Duvenaud, J. R. Lloyd, R. Grosse, J. B. Tenenbaum, and Z. Ghahramani, “Structure discovery in nonparametric regression through compositional kernel search,” in _Proc. International Conference on Machine Learning (ICML)_ , Atlanta, USA, 2013, pp. 1166–1174.
  * [35] M. Lázaro-Gredilla, J. Quiñonero Candela, C. E. Rasmussen, and A. R. Figueiras-Vidal, “Sparse spectrum Gaussian process regression,” _Journal of Machine Learning Research_ , vol. 11, pp. 1865–1881, August 2010\.
  * [36] A. Wilson and R. P. Adams, “Gaussian process kernels for pattern discovery and extrapolation,” in _Proc. International Conference on Machine Learning (ICML)_ , Atlanta, USA, 2013, pp. 1067–1075.
  * [37] J. Shawe-Taylor and N. Cristianini, _Kernel Methods for Pattern Analysis_. Cambridge University Press, 2004\.
  * [38] A. Rahimi and B. Recht, “Random features for large-scale kernel machines,” in _Proc. International Conference on Neural Information Processing Systems_ , Vancouver, British Columbia, Canada, 2007, pp. 1177–1184.
  * [39] M. W. Seeger, “Bayesian inference and optimal design for the sparse linear model,” _Journal of Machine Learning Research_ , vol. 9, pp. 759—813, June 2008.
  * [40] M. E. Tipping and A. Faul, “Fast marginal likelihood maximisation for sparse Bayesian models,” in _Proc. International Conference on Artificial Intelligence and Statistics (AISTAT)_ , Key West, Florida, USA, 2003, pp. 3–6.
  * [41] N. I. Achieser, _Theory of Approximation_. Dover Publications, Inc., New York, 1992.
  * [42] N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalexakis, and C. Faloutsos, “Tensor decomposition for signal processing and machine learning,” _IEEE Transactions on Signal Processing_ , vol. 65, no. 13, pp. 3551–3582, 2017.
  * [43] E. E. Papalexakis, N. D. Sidiropoulos, and R. Bro, “From k-means to higher-way co-clustering: Multilinear decomposition with sparse latent factors,” _IEEE Transactions on Signal Processing_ , vol. 61, no. 2, pp. 493–506, 2013\.
  * [44] H. Becker, L. Albera, P. Comon, R. Gribonval, F. Wendling, and I. Merlet, “Brain-source imaging: From sparse to tensor models,” _IEEE Signal Processing Magazine_ , vol. 32, no. 6, pp. 100–112, 2015.
  * [45] C. Chatzichristos, E. Kofidis, M. Morante, and S. Theodoridis, “Blind fMRI source unmixing via higher-order tensor decompositions,” _Journal of neuroscience methods_ , vol. 315, pp. 17–47, 2019.
  * [46] H. Chen, F. Ahmad, S. Vorobyov, and F. Porikli, “Tensor decompositions in wireless communications and MIMO radar,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 15, no. 3, pp. 438–453, 2021.
  * [47] Q. Zhao, L. Zhang, and A. Cichocki, “Bayesian sparse tucker models for dimension reduction and tensor completion,” _ArXiv_ , 2015. [Online]. Available: https://arxiv.org/abs/1505.02343
  * [48] C. Hawkins and Z. Zhang, “Bayesian tensorized neural networks with automatic rank selection,” _Neurocomputing_ , vol. 453, pp. 172–180, Sep. 2021.
  * [49] L. Xu, L. Cheng, N. Wong, and Y.-C. Wu, “Probabilistic tensor train decomposition with automatic rank determination from noisy data,” in _2021 IEEE Statistical Signal Processing Workshop (SSP)_. IEEE, 2021, pp. 461–465.
  * [50] ——, “Overfitting avoidance in tensor train factorization and completion: Prior analysis and inference,” in _2021 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2021, pp. 1439–1444.
  * [51] Y. Panagakis, J. Kossaifi, G. G. Chrysos, J. Oldfield, M. A. Nicolaou, A. Anandkumar, and S. Zafeiriou, “Tensor methods in computer vision and deep learning,” _Proceedings of the IEEE_ , vol. 109, no. 5, pp. 863–890, 2021\.
  * [52] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [53] M. Razaviyayn, M. Hong, and Z.-Q. Luo, “A unified convergence analysis of block successive minimization methods for nonsmooth optimization,” _SIAM Journal on Optimization_ , vol. 23, no. 2, pp. 1126–1153, 2013.
  * [54] M. Hong, Z.-Q. Luo, and M. Razaviyayn, “Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems,” _SIAM Journal on Optimization_ , vol. 26, no. 1, pp. 337–364, 2016.
  * [55] Y. Sun, P. Babu, and D. P. Palomar, “Majorization-minimization algorithms in signal processing, communications, and machine learning,” _IEEE Transactions on Signal Processing_ , vol. 65, no. 3, pp. 794–816, 2016.
  * [56] T.-H. Chang, M. Hong, H.-T. Wai, X. Zhang, and S. Lu, “Distributed learning in the nonconvex world: From batch data to streaming and beyond,” _IEEE Signal Processing Magazine_ , vol. 37, no. 3, pp. 26–38, 2020.
  * [57] G. Parisi and R. Shankar, _Statistical Field Theory_. Westview Press, 1988.
  * [58] C. Zhang, J. Bütepage, H. Kjellström, and S. Mandt, “Advances in variational inference,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 8, pp. 2008–2026, 2018.
  * [59] R. Zhang, C. Li, J. Zhang, C. Chen, and A. G. Wilson, “Cyclical stochastic gradient MCMC for Bayesian deep learning,” in _Proc. International Conference on Learning Representations (ICLR)_ , 2020.
  * [60] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in _Proc. International Conference on Computational Statistics (COMPSTAT)_. Paris, France: Springer, 2010, pp. 177–186.
  * [61] G. Lan, _First-order and Stochastic Optimization Methods for Machine Learning_. Springer, 2020.
  * [62] S. Boyd and L. Vandenberghe, _Convex Optimization_. Cambridge University Press, 2004.
  * [63] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” _Found. Trends Mach. Learn._ , vol. 3, no. 1, pp. 1–122, January 2011.
  * [64] O. Shayer, D. Levi, and E. Fetaya, “Learning discrete weights using the local reparameterization trick,” in _Proc. International Conference on Learning Representations (ICLR)_ , Vancouver, BC, Canada, 2018.
  * [65] D. P. Kingma and M. Welling, “Auto-encoding variational bayes.” In International Conference on Learning Representations, 2014.
  * [66] C. J. Maddison, A. Mnih, and Y. W. Teh, “The concrete distribution: A continuous relaxation of discrete random variables.” In International Conference on Learning Representations, 2017.
  * [67] P. Kumaraswamy, “A generalized probability density function for double-bounded random processes,” _Journal of hydrology_ , vol. 46, no. 1-2, pp. 79–88, 1980.
  * [68] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, no. 8, pp. 1735—1780, Nov. 1997.
  * [69] P. J. Brockwell and R. A. Davis, _Time Series: Theory and Methods_. Berlin, Heidelberg: Springer-Verlag, 1986.
  * [70] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in _Proc. AAAI Conference on Artificial Intelligence_ , vol. 35, no. 12, May 2021, pp. 11 106–11 115.
  * [71] Y. Xu, F. Yin, W. Xu, C.-H. Lee, J. Lin, and S. Cui, “Scalable learning paradigms for data-driven wireless communication,” _IEEE Communications Magazine_ , vol. 58, no. 10, pp. 81–87, 2020.
  * [72] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Proc. International Conference on Neural Information Processing Systems (NIPS)_ , 2017, pp. 6405—6416.
  * [73] I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _Proc. International Conference on Learning Representation (ICLR)_ , San Diego, CA, USA, 2015.
  * [74] N. Carlini and D. A. Wagner, “Towards evaluating the robustness of neural networks,” in _Proc. IEEE Symposium on Security and Privacy_. San Jose, CA, USA: IEEE Computer Society, May 2017, pp. 39–57.
  * [75] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” _arXiv preprint arXiv:1706.06083_ , 2017.
  * [76] R. McAllister, Y. Gal, A. Kendall, M. van der Wilk, A. Shah, R. Cipolla, and A. Weller, “Concrete problems for autonomous vehicle safety: Advantages of Bayesian deep learning,” in _Proc. International Joint Conference on Artificial Intelligence (IJCAI)_ , 2017, pp. 4745–4753.
  * [77] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble adversarial training: Attacks and defenses,” in _Proc. International Conference on Learning Representations (ICLR)_ , Vancouver, BC, Canada, 2018.
  * [78] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb, “Learning from simulated and unsupervised images through adversarial training,” in _Proc. IEEE conference on computer vision and pattern recognition (CVPR)_ , Honolulu, HI, USA, 2017, pp. 2107–2116.
  * [79] C. Xie, J. Wang, Z. Zhang, Z. Ren, and A. Yuille, “Mitigating adversarial effects through randomization,” in _Proc. International Conference on Learning Representations (ICLR)_ , Vancouver, BC, Canada, 2018.
  * [80] A. Prakash, N. Moran, S. Garber, A. DiLillo, and J. Storer, “Deflecting adversarial attacks with pixel deflection,” in _Proc. IEEE conference on computer vision and pattern recognition (CVPR)_ , Salt Lake City, UT, USA, 2018, pp. 8571–8580.
  * [81] G. S. Dhillon, K. Azizzadenesheli, Z. C. Lipton, J. Bernstein, J. Kossaifi, A. Khanna, and A. Anandkumar, “Stochastic activation pruning for robust adversarial defense,” in _Proc. International Conference on Learning Representations (ICLR)_ , Vancouver, BC, Canada, 2018.
  * [82] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _Proc. BMVC_ , 2016.
  * [83] B. Wu, J. Chen, D. Cai, X. He, and Q. Gu, “Do wider neural networks really help adversarial robustness?” in _In Proc. NIPS_ , 2021.
  * [84] T. Pang, X. Yang, Y. Dong, K. Xu, J. Zhu, and H. Su, “Boosting adversarial training with hypersphere embedding,” in _Proc. NIPS_ , 2020.
  * [85] D. Hendrycks, K. Lee, and M. Mazeika, “Using pre-training can improve model robustness and uncertainty,” _In Proc. ICML_ , 2019.
  * [86] S. Gowal, C. Qin, J. Uesato, T. Mann, and P. Kohli, “Uncovering the limits of adversarial training against norm-bounded adversarial examples,” _arXiv preprint arXiv:2010.03593_ , 2021.
  * [87] A. Athalye, N. Carlini, and D. A. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in _Proc. ICML_ , 2018.
  * [88] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in _Proc. ICML_ , 2020.
  * [89] B. W. Bader, T. G. Kolda, and R. A. Harshman, “Temporal analysis of social networks using three-way dedicom.” Sandia National Lab.(SNL-NM), Albuquerque, NM (United States); Sandia, Tech. Rep., 2006.
  * [90] C.-Y. Ko, K. Batselier, L. Daniel, W. Yu, and N. Wong, “Fast and accurate tensor completion with total variation regularized tensor trains,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 6918–6931, 2020.
  * [91] J. A. Bengua, H. N. Phien, H. D. Tuan, and M. N. Do, “Efficient tensor completion for color image and video recovery: Low-rank tensor train,” _IEEE Transactions on Image Processing_ , vol. 26, no. 5, pp. 2466–2479, 2017\.
  * [92] L. Yuan, Q. Zhao, and J. Cao, “High-order tensor completion for data recovery via sparse tensor-train optimization,” in _Proc. IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 2018, pp. 1258–1262.
  * [93] F. Wenzel, K. Roth, B. S. Veeling, J. Swiatkowski, L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Jenatton, and S. Nowozin, “How good is the Bayes Posterior in deep neural networks really?” in _Proc. International Conference on Machine Learning (ICML)_ , vol. 119, 2020, pp. 10 248–10 259.
  * [94] B. Adlam, J. Snoek, and S. L. Smith, “Cold posteriors and aleatoric uncertainty,” _CoRR_ , vol. abs/2008.00029, 2020. [Online]. Available: https://arxiv.org/abs/2008.00029
  * [95] E. Angelino, M. J. Johnson, and R. P. Adams, “Patterns of scalable Bayesian inference,” _Foundations and Trends® in Machine Learning_ , vol. 9, no. 2-3, pp. 119–247, 2016.
  * [96] J. Han, F. Ding, X. Liu, L. Torresani, J. Peng, and Q. Liu, “Stein variational inference for discrete distributions,” in _Proc. International Conference on Artificial Intelligence and Statistics (AISTAT)_. PMLR, 2020, pp. 4563–4572.
  * [97] P. Izmailov, A. Novikov, and D. Kropotov, “Scalable Gaussian processes with billions of inducing inputs via tensor train decomposition,” in _Proc. International Conference on Artificial Intelligence and Statistics (AISTAT)_. Playa Blanca, Lanzarote, Canary Islands, Spain: PMLR, 2018, pp. 726–735.
  * [98] A. Tjandra, S. Sakti, and S. Nakamura, “Compressing recurrent neural network with tensor train,” in _Proc. International Joint Conference on Neural Networks (IJCNN)_. Anchorage, AK, USA: IEEE, 2017, pp. 4451–4458.