# Entropy-MCMC: Sampling from Flat Basins with Ease

Bolian Li, Ruqi Zhang
Department of Computer Science, Purdue University, USA
<EMAIL_ADDRESS>

###### Abstract

Bayesian deep learning counts on the quality of posterior distribution
estimation. However, the posterior of deep neural networks is highly multi-
modal in nature, with local modes exhibiting varying generalization
performance. Given a practical budget, sampling from the original posterior
can lead to suboptimal performance, as some samples may become trapped in
“bad” modes and suffer from overfitting. Leveraging the observation that
“good” modes with low generalization error often reside in flat basins of the
energy landscape, we propose to bias sampling on the posterior toward these
flat regions. Specifically, we introduce an auxiliary guiding variable, the
stationary distribution of which resembles a smoothed posterior free from
sharp modes, to lead the MCMC sampler to flat basins. By integrating this
guiding variable with the model parameter, we create a simple joint
distribution that enables efficient sampling with minimal computational
overhead. We prove the convergence of our method and further show that it
converges faster than several existing flatness-aware methods in the strongly
convex setting. Empirical results demonstrate that our method can successfully
sample from flat basins of the posterior, and outperforms all compared
baselines on multiple benchmarks including classification, calibration, and
out-of-distribution detection.

##  1 Introduction

The effectiveness of Bayesian neural networks relies heavily on the quality of
posterior distribution estimation. However, achieving an accurate estimation
of the full posterior is extremely difficult due to its high-dimensional and
highly multi-modal nature (Zhang et al., 2020b; Izmailov et al., 2021).
Moreover, the numerous modes in the energy landscape typically exhibit varying
generalization performance. Flat modes often show superior accuracy and
robustness, whereas sharp modes tend to have high generalization errors
(Hochreiter & Schmidhuber, 1997; Keskar et al., 2017; Bahri et al., 2022).
This connection between the geometry of energy landscape and generalization
has spurred many works in optimization, ranging from theoretical understanding
(Neyshabur et al., 2017; Dinh et al., 2017; Dziugaite & Roy, 2018; Jiang et
al., 2019a) to new optimization algorithms (Mobahi, 2016; Izmailov et al.,
2018; Chaudhari et al., 2019; Foret et al., 2020).

However, most of the existing Bayesian methods are not aware of the flatness
in the energy landscape during posterior inference (Welling & Teh, 2011; Chen
et al., 2014; Ma et al., 2015; Zhang et al., 2020b). Their inference
strategies are usually energy-oriented and could not distinguish between flat
and sharp modes that have the same energy values. This limitation can
significantly undermine their generalization performance, particularly in
practical situations where capturing the full posterior is challenging. In
light of this, we contend that prioritizing the capture of flat modes is
essential when conducting posterior inference for Bayesian neural networks.
This is advantageous for improved generalization as justified by previous
works (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017; Bahri et al.,
2022). It can further be rationalized from a Bayesian marginalization
perspective: within the flat basin, each model configuration occupies a
substantial volume and contributes significantly to a more precise estimation
of the predictive distribution (Bishop, 2006). Moreover, existing flatness-
aware methods often rely on a single solution to represent the entire flat
basin (Chaudhari et al., 2019; Foret et al., 2020), ignoring the fact that the
flat basin contains many high-performing models. Therefore, Bayesian
marginalization can potentially offer significant improvements over flatness-
aware optimization by sampling from the flat basins (Wilson, 2020; Huang et
al., 2020).

Prioritizing flat basins during posterior inference poses an additional
challenge to Bayesian inference. Even for a single point estimation,
explicitly biasing toward the flat basins will introduce substantial
computational overhead, inducing nested loops (Chaudhari et al., 2019;
Dziugaite & Roy, 2018), doubled gradients calculation (Foret et al., 2020;
Möllenhoff & Khan, 2022) or min-max problems (Foret et al., 2020). The
efficiency problem needs to be addressed before any flatness-aware Bayesian
method becomes practical for deep neural networks.

In this paper, we propose an efficient sampling algorithm to explicitly
prioritize flat basins in the energy landscape of deep neural networks.
Specifically, we introduce an auxiliary guiding variable $\bm{\theta}_{a}$
into the Markov chain to pull the model parameter $\bm{\theta}$ toward flat
basins at each updating step (Fig. 1a). $\bm{\theta}_{a}$ is sampled from a
smoothed posterior distribution which eliminates sharp modes based on local
entropy (Baldassi et al., 2016) (Fig. 1b). Our method enjoys a simple joint
distribution of $\bm{\theta}$ and $\bm{\theta}_{a}$, and the computational
overhead is similar to Stochastic gradient Langevin dynamics (SGLD) (Welling &
Teh, 2011). Theoretically, we prove that our method is guaranteed to converge
faster than some common flatness-aware methods (Chaudhari et al., 2019;
Dziugaite & Roy, 2018) in the strongly convex setting. Empirically, we
demonstrate that our method successfully finds flat basins efficiently across
multiple tasks. Our main contributions are summarized as follows:

  * •

We propose Entropy-MCMC (EMCMC) for sampling from flat basins in the energy
landscape of deep neural networks. EMCMC utilizes an auxiliary guiding
variable and a simple joint distribution to efficiently steer the model toward
flat basins.

  * •

We prove the convergence of EMCMC and further show that it converges faster
than several existing flatness-aware methods in the strongly convex setting.

  * •

We provide extensive experimental results to demonstrate the advantages of
EMCMC in sampling from flat basins. EMCMC outperforms all compared baselines
on classification, calibration, and out-of-distribution detection with
comparable overhead akin to SGLD.

(a) Training dynamics at each step (b) Original and smoothed posterior
distributions

Figure 1: Illustration of Entropy-MCMC. (a) shows how the guiding variable
$\bm{{\theta}}_{a}$ pulls $\bm{\theta}$ toward flat basins; (b) shows two
posterior distributions, where $p(\bm{\theta}_{a}|\mathcal{D})$ is a smoothed
distribution transformed from $p(\bm{\theta}|\mathcal{D})$, and only keeps
flat modes. Entropy-MCMC prioritizes flat modes by leveraging the guiding
variable $\bm{\theta}_{a}$ from the smoothed posterior as a form of
regularization.

##  2 Related Works

##### Flatness-aware Optimization.

The concept of flatness in the energy landscape was first studied by
Hochreiter & Schmidhuber (1994), and its connection with generalization was
then empirically discussed by Keskar et al. (2017); Dinh et al. (2017); Jiang
et al. (2019b). In order to pursue flatness for better generalization,
Entropy-SGD (Chaudhari et al., 2019) introduced local entropy to consider the
averaged loss of a region, SAM (Foret et al., 2020) developed a new optimizer
to minimize the worst-case near the current model, bSAM (Möllenhoff & Khan,
2022) further improved SAM with a Bayes optimal convex lower bound, LPF (Bisla
et al., 2022) introduced low-pass filter to actively search flat basins, and
SWA (Izmailov et al., 2018) found that averaging weights along the trajectory
of SGD training can also find flatter modes. Our Entropy-MCMC follows the
local entropy measurement and collects more than a single point to fully
exploit the flat basins.

##### MCMC on Deep Neural Networks.

Markov chain Monte Carlo is a class of general and practical sampling
algorithms (Andrieu et al., 2003), which has been applied to infer Bayesian
neural network posteriors (Neal, 2012). SGMCMC (Welling & Teh, 2011; Ma et
al., 2015) methods use the mini-batching technique to adapt MCMC to deep
neural networks. SGHMC (Chen et al., 2014) exploited the second-order Langevin
dynamics to calibrate the stochastic estimates of HMC gradients. cSGMCMC
(Zhang et al., 2020b) further improves sampling efficiency by leveraging a
cyclical step size schedule. Symmetric Split HMC (Cobb & Jalaian, 2021)
developed a way to apply HMC to deep neural networks without stochastic
gradients. Our Entropy-MCMC builds upon the SGMCMC framework and is designed
to favor the flat basins in the energy landscape during sampling.

##  3 Preliminaries

##### Flatness-aware Optimization.

One common flatness-aware optimization technique is to use the concept of
_local entropy_ , which measures the geometric properties of the energy
landscape (Baldassi et al., 2016; Chaudhari et al., 2019). The local entropy
is computed by:

$\mathcal{F}(\bm{\theta};\eta)=\log{\int_{\bm{\Theta}}\exp{\left\\{-f(\bm{\theta}^{\prime})-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}^{\prime}\|^{2}\right\\}}}d\bm{\theta}^{\prime},$
(1)

where $f(\cdot)$ is the loss function and $\eta$ is a scalar. The local
entropy of a point $\bm{\theta}$ is determined by its neighbors weighted by
their distances, which considers the volume of local modes. Previous
optimization methods minimize $-\mathcal{F}(\bm{\theta};\eta)$ to find the
flat minimum.

##### SGMCMC.

Given a dataset $\mathcal{D}$, a neural network with the parameter
$\bm{\theta}\in\operatorname{\mathbb{R}}^{d}$ and a prior distribution
$p(\bm{\theta})$, we can use Markov chain Monte Carlo (MCMC) to sample from
the posterior $p(\bm{\theta}|\mathcal{D})\propto\exp(-U(\bm{\theta}))$, where
the energy function is $U(\bm{\theta})=-\sum_{x\in\mathcal{D}}\log
p(x|\bm{\theta})-\log p(\bm{\theta})$. However, the computational cost for
MCMC with large-scale data is too high to be practical. SGMCMC tackles this
problem by stochastic gradient $\nabla U_{\bm{\Xi}}$ based on a subset of data
$\bm{\Xi}\subseteq\mathcal{D}$. We use Stochastic Gradient Langevin Dynamics
(SGLD) (Welling & Teh, 2011) in the paper as the backbone MCMC algorithm,
which has the following updating rule:

$\bm{\theta}\leftarrow\bm{\theta}-\alpha\nabla_{\bm{\theta}}U_{\bm{\Xi}}(\bm{\theta})+\sqrt{2\alpha}\cdot\bm{\epsilon},$
(2)

where $\alpha$ is the step size and $\bm{\epsilon}$ is standard Gaussian
noise. Our method can also be implemented by other SGMCMC methods. During
testing, Bayesian marginalization is performed to make predictions based on
the sample set collected during sampling
$\mathcal{S}=\\{\bm{\theta}_{j}\\}_{j=1}^{M}$ and the predictive distribution
is obtained by $p(y|\bm{x},\mathcal{D})=\int
p(y|\bm{x},\bm{\theta})p(\bm{\theta}|\mathcal{D})d\bm{\theta}\approx\sum_{\bm{\theta}\in\mathcal{S}}p(y|\bm{x},\bm{\theta})$.

##  4 Entropy-MCMC

In this section, we present the Entropy-MCMC algorithm. We introduce the
guiding variable $\bm{\theta}_{a}$ obtained from the local entropy in section
4.1 and discuss the sampling strategy in section 4.2.

###  4.1 From Local Entropy to Flat Posterior

While flat basins in the energy landscape have been shown to be of good
generalization (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017; Bahri et
al., 2022), finding such regions is still challenging due to the highly multi-
modal nature of the DNN energy landscape. The updating direction of the model
typically needs extra force to mitigate the risk of converging to sharp modes
(Chaudhari et al., 2019; Foret et al., 2020). To bias sampling to flat basins,
we look into the local entropy (Eq. 1), which can eliminate the sharp modes in
the energy landscape (Chaudhari et al., 2019).

We begin by the original posterior distribution
$p(\bm{\theta}|\mathcal{D})\propto\exp(-f(\bm{\theta}))=\exp\\{\log{p(\mathcal{D}|\bm{\theta})}+\log{p(\bm{\theta})\\}}$,
which contains both sharp and flat modes. By replacing the original loss
function with local entropy, we obtain a smoothed posterior distribution in
terms of a new variable $\bm{\theta}_{a}$:

$p(\bm{\theta}_{a}|\mathcal{D})\propto\exp{\mathcal{F}(\bm{\theta}_{a};\eta)}=\int_{\bm{\Theta}}\exp{\left\\{-f(\bm{\theta})-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}\right\\}}d\bm{\theta}.$
(3)

The effect of local entropy on this new posterior is visualized in Fig. 1b.
The new posterior measures both the depth and flatness of the mode in
$p(\bm{\theta}|\mathcal{D})$ by considering surrounding energy values.
Thereby, $p(\bm{\theta}_{a}|\mathcal{D})$ is expected to primarily capture
flat modes in the energy landscape, which can be used as the desired external
force to revise the updating directions of the model parameter $\bm{\theta}$.

However, the complex integral in Eq. 3 requires marginalization on the model
parameter $\bm{\theta}$, which poses a non-trivial challenge. Previous works
using local entropy usually adopt an inner Markov chain with Monte Carlo (MC)
approximation (Chaudhari et al., 2019; Dziugaite & Roy, 2018), which
sacrifices the accuracy of local entropy computation and induces
computationally expensive nested loops in training. We tackle this challenge
in a simple yet principled manner, eliminating the need for nested loops or
approximation. This is achieved by _coupling_ $\bm{\theta}\sim
p(\bm{\theta}|\mathcal{D})$ and $\bm{\theta}_{a}\sim
p(\bm{\theta}_{a}|\mathcal{D})$ into a joint posterior distribution, which
enjoys a simple form, as discussed in Lemma 1.

######  Lemma 1.

Assume
$\widetilde{\bm{\theta}}=[\bm{\theta}^{T},\bm{\theta}_{a}^{T}]^{T}\in\mathbb{R}^{2d}$
and $\widetilde{\bm{\theta}}$ has the following distribution:

$p(\widetilde{\bm{\theta}}|\mathcal{D})=p(\bm{\theta},\bm{\theta}_{a}|\mathcal{D})\propto\exp\left\\{-f(\bm{\theta})-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}\right\\}.$
(4)

Then the marginal distributions of $\bm{\theta}$ and $\bm{\theta}_{a}$ are the
original posterior $p(\bm{\theta}|\mathcal{D})$ and
$p(\bm{\theta}_{a}|\mathcal{D})$ (Eq. 3).

This joint posterior offers three key advantages: i) by coupling $\bm{\theta}$
and $\bm{\theta}_{a}$, we avoid the intricate integral computation, and thus
remove the requirement of expensive nested training loops and mitigate the MC
approximation error; ii) the joint posterior turns out to be surprisingly
simple, making it easy to sample from both empirically and theoretically
(details discussed in Sections 4.2 and 5); iii) after coupling,
$\bm{\theta}_{a}$ provides additional paths for $\bm{\theta}$ to traverse,
making $\bm{\theta}$ reach flat modes efficiently.

###  4.2 Sampling from Flat Basins

We discuss how to sample from the joint posterior distribution (Eq. 4) in this
section. We adopt SGLD (Welling & Teh, 2011), a simple stochastic gradient
MCMC algorithm that is suitable for deep neural networks, as the backbone of
EMCMC sampling. More advanced MCMC algorithms can also be combined with our
method. The energy function of the joint parameter variable
$\widetilde{\bm{\theta}}$ is
$U(\widetilde{\bm{\theta}})=f(\bm{\theta})+\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}$,
and thus its gradients is given by:

$\nabla_{\widetilde{\bm{\theta}}}U(\widetilde{\bm{\theta}})=\left[\begin{array}[]{c}\nabla_{\bm{\theta}}U(\widetilde{\bm{\theta}})\\\
\nabla_{\bm{\theta}_{a}}U(\widetilde{\bm{\theta}})\end{array}\right]=\left[\begin{array}[]{c}\nabla_{\bm{\theta}}f(\bm{\theta})+\frac{1}{\eta}(\bm{\theta}-\bm{\theta}_{a})\\\
\frac{1}{\eta}(\bm{\theta}_{a}-\bm{\theta})\end{array}\right].$ (5)

For the model parameter $\bm{\theta}$, the original gradient direction
$\nabla_{\bm{\theta}}f(\bm{\theta})$ is revised by
$\frac{1}{\eta}(\bm{\theta}-\bm{\theta}_{a})$ to get the flatness-aware
gradient direction $\nabla_{\bm{\theta}}U(\widetilde{\bm{\theta}})$, as
visualized in Fig. 1a. Importantly, the practical implementation does not
require computing $\nabla_{\bm{\theta}_{a}}U(\widetilde{\bm{\theta}})$ through
back-propagation, as we can utilize the analytical expression presented in Eq.
5. Therefore, despite $\widetilde{\bm{\theta}}$ being in a $2d$ dimension, our
cost of gradient computation is essentially the _same_ as $d$-dimensional
models (e.g., standard SGLD).

With the form of the gradients in Eq. 5, the training procedure of EMCMC is
straightforward using the SGLD updating rule in Eq. 2. The details are
summarized in Algorithm 1. At testing stage, the collected samples
$\mathcal{S}$ are used to approximate the predictive distribution
$p(y|\bm{x},\mathcal{D})\approx\sum_{\bm{\theta}_{s}\in\mathcal{S}}p(y|\bm{x},\bm{\theta}_{s})$.
Our choice of sampling from the joint posterior distribution using SGLD,
rather than a Gibbs-like approach (Gelfand, 2000), is motivated by SGLD’s
ability to simultaneously update both $\bm{\theta}$ and $\bm{\theta}_{a}$,
which is more efficient than alternative updating (see Appendix A for a
detailed discussion). For the sample set $\mathcal{S}$, we collect both
$\bm{\theta}$ and $\bm{\theta}_{a}$ after the burn-in period in order to
obtain more high-quality and diverse samples in a given time budget (see
Appendix C.3 for details).

In summary, thanks to EMCMC’s simple joint distribution, conducting sampling
in EMCMC is straightforward, and its computational cost is comparable to that
of standard SGLD. Despite its algorithmic simplicity and computational
efficiency, EMCMC is guaranteed to bias sampling to flat basins and obtain
samples with enhanced generalization and robustness.

Inputs: The model parameter $\bm{\theta}\in\bm{\Theta}$, guiding variable
$\bm{\theta}_{a}\in\bm{\Theta}$, and dataset
$\mathcal{D}=\\{(\bm{x}_{i},y_{i})\\}_{i=1}^{N}$;

Results: Collected samples $\mathcal{S}\subset\bm{\Theta}$;

$\bm{\theta}_{a}\leftarrow\bm{\theta},\mathcal{S}\leftarrow\emptyset$ ;

/* Initialize */

for  _each iteration_ do

$\bm{\Xi}\leftarrow~{}\text{A mini-batch sampled from}~{}\mathcal{D}$;

$U_{\bm{\Xi}}\leftarrow-\log{p(\bm{\Xi}|\bm{\theta})}-\log{p(\bm{\theta})}+\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}$;

$\bm{\theta}\leftarrow\bm{\theta}-\alpha\nabla_{\bm{\theta}}U_{\bm{\Xi}}+\sqrt{2\alpha}\cdot\bm{\epsilon}_{1}$
;

/* $\bm{\epsilon}_{1},\bm{\epsilon}_{2}\sim\mathcal{N}(\bm{0},\bm{I})$ */

$\bm{\theta}_{a}\leftarrow\bm{\theta}_{a}-\alpha\nabla_{\bm{\theta}_{a}}U_{\bm{\Xi}}+\sqrt{2\alpha}\cdot\bm{\epsilon}_{2}$;

if  _after burn-in_ then

$\mathcal{S}\leftarrow\mathcal{S}\cup\\{\bm{\theta},\bm{\theta}_{a}\\}$ ;

/* Collect samples */

end if

end for

Algorithm 1 Entropy-MCMC

##  5 Theoretical Analysis

In this section, we provide a theoretical analysis on the convergence rate of
Entropy-MCMC and compare it with previous local-entropy-based methods
including Entropy-SGD (Chaudhari et al., 2019) and Entropy-SGLD (Dziugaite &
Roy, 2018) (used as a theoretical tool in the literature rather than a
practical algorithm). We leverage the 2-Wasserstein distance bounds of SGLD,
which assumes the target distribution to be smooth and strongly log-concave
(Dalalyan & Karagulyan, 2019). While the target distribution in this case is
unimodal, it still reveals the superior convergence rate of EMCMC compared
with existing flatness-aware methods. We leave the theoretical analysis on
non-log-concave distributions for future work. Specifically, we have the
following assumptions for the loss function $f(\cdot)$ and stochastic
gradients:

######  Assumption 1.

The loss function $f(\bm{\theta})$ in the original posterior distribution
$\pi=p(\bm{\theta}|\mathcal{D})\propto\exp{(-f(\bm{\theta}))}$ is $M$-smooth
and $m$-strongly convex (i.e.,
$m\bm{I}\preceq\nabla^{2}f(\bm{\theta}^{\prime})\preceq M\bm{I}$).

######  Assumption 2.

The variance of stochastic gradients is bounded by $\mathbb{E}[\|\nabla
f(\bm{\theta})-\nabla f_{\bm{\Xi}}(\bm{\theta})\|^{2}]\leq\sigma^{2}$ for some
constant $\sigma>0$.

To establish the convergence analysis for EMCMC, we first observe that the
smoothness and convexity properties of the joint posterior distribution
$\pi_{\text{joint}}(\bm{\theta},\bm{\theta}_{a})=p(\bm{\theta},\bm{\theta}_{a}|\mathcal{D})$
in Eq. 4 is the same as the original posterior $p(\bm{\theta}|\mathcal{D})$,
which is formally stated in Lemma 2.

######  Lemma 2.

If Assumption 1 holds and $m\leq 1/\eta\leq M$, then the energy function in
the joint posterior distribution
$\pi_{\text{joint}}(\bm{\theta},\bm{\theta}_{a})=p(\bm{\theta},\bm{\theta}_{a}|\mathcal{D})$
is also $M$-smooth and $m$-strongly convex.

With the convergence bound of SGLD established by Dalalyan & Karagulyan
(2019), we derive the convergence bound for EMCMC in Theorem 1.

######  Theorem 1.

Under Assumptions 1 and 2, let $\mu_{0}$ be the initial distribution and
$\mu_{K}$ be the distribution obtained by EMCMC after $K$ iterations. If
$m\leq 1/\eta\leq M$ and the step size $\alpha\leq 2/(m+M)$, the 2-Wasserstein
distance between $\mu_{K}$ and $\pi_{\text{joint}}$ will have the following
upper bound:

$\mathcal{W}_{2}(\mu_{K},\pi_{\text{joint}})\leq(1-\alpha
m)^{K}\cdot\mathcal{W}_{2}(\mu_{0},\pi)+1.65(M/m)(2\alpha
d)^{1/2}+\frac{\sigma^{2}(2\alpha d)^{1/2}}{1.65M+\sigma\sqrt{m}}.$ (6)

Comparing Theorem 1 with the convergence bound of SGLD obtained by Dalalyan &
Karagulyan (2019), the only difference is the doubling of the dimension, from
$d$ to $2d$. Theorem 1 implies that the convergence rate of EMCMC will have at
most a minor slowdown by a constant factor compared to SGLD while ensuring
sampling from flat basins.

In contrast, previous local-entropy-based methods often substantially slow
down the convergence in order to bias toward flat basins. For example,
consider Entropy-SGD (Chaudhari et al., 2019) which minimizes a flattened loss
function
$f_{\text{flat}}(\bm{\theta})=-\mathcal{F}(\bm{\theta};\eta)=-\log{\int_{\bm{\Theta}}\exp{\left\\{-f(\bm{\theta}^{\prime})-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}^{\prime}\|^{2}\right\\}}}d\bm{\theta}^{\prime}$.
We discuss the convergence bound of Entropy-SGD in Theorem 2, which shows how
the presence of the integral (and the nested Markov chain induced by it) slows
down the convergence.

######  Theorem 2.

Consider running Entropy-SGD to minimize the flattened loss function
$f_{\text{flat}}(\bm{\theta})$ under Assumptions 1 and 2. Assume the inner
Markov chain runs $L$ iterations and the 2-Wasserstein distance between the
initial and target distributions is always bounded by $\kappa$. Let
$f_{\text{flat}}^{*}$ represent the global minimum value of
$f_{\text{flat}}(\bm{\theta})$ and
$E_{t}:=\mathbb{E}f_{\text{flat}}(\bm{\theta}_{t})-f_{\text{flat}}^{*}$. If
the step size $\alpha\leq 2/(m+M)$, then we have the following upper bound:

$E_{K}\leq\left(1-\frac{\alpha m}{1+\eta M}\right)^{K}\cdot
E_{0}+\frac{A(1+\eta M)}{2m},$ (7)

where $A^{2}=(1-\alpha
m)^{L}\cdot\kappa+1.65\left(\frac{M+1/\eta}{m+1/\eta}\right)(\alpha
d)^{1/2}+\frac{\sigma^{2}(\alpha
d)^{1/2}}{1.65(M+1/\eta)+\sigma\sqrt{m+1/\eta}}$.

Another example is Entropy-SGLD (Dziugaite & Roy, 2018), a theoretical tool
established to analyze Entropy-SGD. Its main distinction with Entropy-SGD is
the SGLD updating instead of SGD updating in the outer loop. The convergence
bound for Entropy-SGLD is established in Theorem 3.

######  Theorem 3.

Consider running Entropy-SGLD to sample from
$\pi_{\text{flat}}(\bm{\theta})\propto\exp\mathcal{F}(\bm{\theta};\eta)$ under
Assumptions 1 and 2. Assume the inner Markov chain runs $L$ iterations and the
2-Wasserstein distance between initial and target distributions is always
bounded by $\kappa$. Let $\nu_{0}$ be the initial distribution and $\nu_{K}$
be the distribution obtained by Entropy-SGLD after $K$ iterations. If the step
size $\alpha\leq 2/(m+M)$, then:

$\mathcal{W}_{2}(\nu_{K},\pi_{\text{flat}})\leq(1-\alpha
m)^{K}\cdot\mathcal{W}_{2}(\nu_{0},\pi_{\text{flat}})+1.65\left(\frac{1+\eta
M}{1+\eta m}\right)(M/m)(\alpha d)^{1/2}+\frac{A(1+\eta M)}{m},$ (8)

where $A^{2}=(1-\alpha
m)^{L}\cdot\kappa+1.65\left(\frac{M+1/\eta}{m+1/\eta}\right)(\alpha
d)^{1/2}+\frac{\sigma^{2}(\alpha
d)^{1/2}}{1.65(M+1/\eta)+\sigma\sqrt{m+1/\eta}}$ .

Comparing Theorem 1, 2 and 3, we observe that the convergence rates of
Entropy-SGD and Entropy-SGLD algorithms are significantly hindered due to the
presence of the nested Markov chains, which induces a large and complicated
error term $A$. Since $\sigma$ and $\alpha$ are typically very small, the
third term in Theorem 1 will be much smaller than both the third term in
Theorem 3 and the second term in Theorem 2. To summarize, we prove that the
convergence rate of Entropy-MCMC is notably better than that of Entropy-SGD
and Entropy-SGLD in the strongly convex setting.

##  6 Experiments

We conduct comprehensive experiments to show the superiority of EMCMC. Section
6.1 and 6.3 demonstrate that EMCMC can successfully sample from flat basins.
Section 6.2 verifies the fast convergence of EMCMC. Section 6.4 and 6.5
demonstrate the outstanding performance of EMCMC on multiple benchmarks.
Following Zhang et al. (2020b), we adopt a cyclical step size schedule for all
sampling methods. For more implementation details, please refer to Appendix C.

###  6.1 Synthetic Examples

To demonstrate EMCMC’s capability to sample from flat basins, we construct a
two-mode energy landscape
$\frac{1}{2}\mathcal{N}([-2,-1]^{T},0.5\bm{I})+\frac{1}{2}\mathcal{N}([2,1]^{T},\bm{I})$
containing a sharp and a flat mode. To make the case challenging, we set the
initial point at $(-0.2,-0.2)$, the ridge of the two modes, which has no
strong preference for either mode. Fig. 2 shows that the proposed EMCMC finds
the flat basin while SGD and SGLD still prefer the sharp mode due to the
slightly larger gradients coming from the sharp mode. From Fig. 2(c)&(d), we
see that the samples of $\bm{\theta}_{a}$ are always around the flat mode,
showing its ability to eliminate the sharp mode. Although $\bm{\theta}$ visits
the sharp mode in the first few iterations, it subsequently inclines toward
the flat mode, illustrating the influence of gradient revision by the guiding
variable $\bm{\theta}_{a}$. It is noteworthy that the result of EMCMC is
essentially _independent_ of initialization with appropriate values of $\eta$,
since the guiding variable $\bm{\theta}_{a}$ will always steer $\bm{\theta}$
to the flat mode. We show the results for different initialization in Appendix
D.

(a) SGD (b) SGLD (c) EMCMC ($\bm{\theta}$) (d) EMCMC ($\bm{\theta}_{a}$)

Figure 2: Sampling trajectories on a synthetic energy landscape with sharp
(lower left) and flat (top right) modes. The initial point is located at the
ridge of two modes. EMCMC successfully biases toward the flat mode whereas SGD
and SGLD are trapped in the sharp mode.

###  6.2 Logistic Regression

To verify the theoretical results on convergence rates in Section 5, we
conduct logistic regression on MNIST (LeCun, 1998) to compare EMCMC with
Entropy-SGD Chaudhari et al. (2019), SGLD (Welling & Teh, 2011) and Entropy-
SGLD (Dziugaite & Roy, 2018). We follow Maclaurin & Adams (2015) and Zhang et
al. (2020a) to use a subset containing $7$s and $9$s and the resulting
posterior is strongly log-concave, satisfying the assumptions in Section 5.
Fig. 3 shows that EMCMC converges faster than Entropy-SG(L)D, demonstrating
the advantage of using a simple joint distribution without the need for nested
loops or MC approximation, which verifies Theorems 1& 2& 3. Besides, while
EMCMC and SGLD share similar convergence rates, EMCMC achieves better
generalization as shown by its higher test accuracy. This suggests that EMCMC
is potentially beneficial in unimodal distributions under limited budgets due
to finding samples with high volumes.

(a)  (b)

Figure 3: Logistic regression on MNIST in terms of training NLL and testing
accuracy (repeated 10 times). EMCMC converges faster than others, which is
consistent with our theoretical analysis.

###  6.3 Flatness Analysis on Deep Neural Networks

We perform flatness analysis with ResNet18 (He et al., 2016) on CIFAR100
(Krizhevsky, 2009). We use the last sample of SGD, SGLD, and EMCMC (averaged
result from $\bm{\theta}$ and $\bm{\theta}_{a}$) respectively, and each
experiment is repeated 3 times to report the averaged scores.

##### Eigenspectrum of Hessian.

The Hessian matrix of the model parameter measures the second-order gradients
of a local mode on the energy landscape. Smaller eigenvalues of Hessian
indicate a flatter local geometry (Chaudhari et al., 2019; Foret et al.,
2020). Since computing the exact Hessian of deep neural networks is extremely
costly, we use the diagonal Fisher information matrix (Wasserman, 2004) to
approximate its eigenspectrum:

$[\lambda_{1},...,\lambda_{d}]^{T}\approx
diag(\mathcal{I}(\bm{\theta}))=\mathbb{E}\left[\left(\nabla U-\mathbb{E}\nabla
U\right)^{2}\right],$ (9)

where $\lambda_{1},...,\lambda_{d}$ are eigenvalues of the Hessian. Fig. 4
shows the eigenspectra of local modes discovered by different algorithms. The
eigenvalues of EMCMC are much smaller compared with SGD and SGLD, indicating
that the local geometry of EMCMC samples is flatter. The eigenspectrum
comparison verifies the effectiveness of EMCMC to find and sample from flat
basins.

(a) SGD (b) SGLD (c) EMCMC

Figure 4: Eigenspectrum of Hessian matrices of ResNet18 on CIFAR100. $x$-axis:
eigenvalue, $y$-axis: frequency. A nearly all-zero eigenspectrum indicates a
local mode that is flat in all directions. EMCMC successfully finds such flat
modes with significantly smaller eigenvalues.

##### Parameter Space Interpolation.

Another way to measure the flatness of local modes is directly interpolating
their neighborhood in the parameter space (Izmailov et al., 2018). Local modes
located in flat basins are expected to have larger widths and better
generalization performance (Keskar et al., 2017; Chaudhari et al., 2019). The
interpolation begins at $\bm{\theta}$ and ends at $\bm{\theta}_{\epsilon}$ (a
random point near $\bm{\theta}$ or $\bm{\theta}_{\epsilon}=\bm{\theta}_{a}$).
The interpolated point $\bm{\theta}_{\delta}$ is computed by:

$\bm{\theta}_{\delta}=\left(1-\delta/\|\bm{\theta}-\bm{\theta}_{\epsilon}\|\right)\bm{\theta}+\left(\delta/\|\bm{\theta}-\bm{\theta}_{\epsilon}\|\right)\bm{\theta}_{\epsilon},$
(10)

where $\delta$ is the Euclidean distance from $\bm{\theta}$ to
$\bm{\theta}_{\delta}$. Fig. 5a and 5b show the training NLL and testing error
respectively. The neighborhood of EMCMC maintains consistently lower NLL and
errors compared with SGD and SGLD, demonstrating that EMCMC samples are from
flatter modes. Furthermore, Fig. 5c visualizes the interpolation between
$\bm{\theta}$ and $\bm{\theta}_{a}$, revealing that both variables essentially
converge to the same flat mode while maintaining diversity. This justifies the
benefit of collecting both of them as samples to obtain a diverse set of high-
performing samples.

(a) $\bm{\theta}$ $\rightarrow$ Random (b) $\bm{\theta}$ $\rightarrow$ Random
(c) $\bm{\theta}$ $\rightarrow$ $\bm{\theta}_{a}$

Figure 5: Parameter space interpolation of ResNet18 on CIFAR100. Exploring the
neighborhood of local modes from $\bm{\theta}$ to (a)-(b): a random direction
in the parameter space, and (c): $\bm{\theta}_{a}$. (a) and (b) show that
EMCMC has the lowest and the most flat NLL and error curves. (c) shows that
$\bm{\theta}$ and $\bm{\theta}_{a}$ converge to the same flat mode while
maintaining diversity.

###  6.4 Image Classification

We conduct classification experiments on CIFAR (Krizhevsky, 2009), corrupted
CIFAR (Hendrycks & Dietterich, 2019b) and ImageNet (Deng et al., 2009), to
compare EMCMC with both flatness-aware optimization methods (Entropy-SGD
(Chaudhari et al., 2019), SAM (Foret et al., 2020) and bSAM (Möllenhoff &
Khan, 2022)) and MCMC methods (SGLD (Welling & Teh, 2011) and Entropy-SGLD
(Dziugaite & Roy, 2018)). We use ResNet18 and ResNet50 (He et al., 2016) for
CIFAR and ImageNet respectively. All sampling algorithms collect a total of 16
samples for Bayesian marginalization, and all entries are repeated 3 times to
report the mean$\pm$std. Table 1 shows the results on the 3 datasets, in which
EMCMC significantly outperforms all baselines. The classification results
strongly suggest that by sampling from flat basins, Bayesian neural networks
can achieve outstanding performance and EMCMC is an effective and efficient
method to do so.

The results for corrupted CIFAR (Hendrycks & Dietterich, 2019a) are shown in
Table 1b to show the robustness of EMCMC against multiple types of noises. The
results are averaged over all noise types, and the severity level refers to
the strength of noise added to the original data. EMCMC consistently
outperforms all compared baselines across all severity levels, indicating that
samples from flat basins are more robust to noise. The results for individual
noise types are shown in Appendix D.

Table 1: Classification results on (a) CIFAR10/100, (b) corrupted CIFAR and
(c) ImageNet, measured by NLL and accuracy. EMCMC outperforms all compared
baselines.

Method | CIFAR10 | CIFAR100
---|---|---
ACC (%) $\uparrow$ | NLL $\downarrow$ | ACC (%) $\uparrow$ | NLL $\downarrow$
SGD | $94.87\pm 0.04$ | $0.205\pm 0.015$ | $76.49\pm 0.27$ | $0.935\pm 0.021$
Entropy-SGD | $95.11\pm 0.09$ | $0.184\pm 0.020$ | $77.45\pm 0.03$ | $0.895\pm 0.009$
SAM | $95.25\pm 0.12$ | $0.166\pm 0.005$ | $78.41\pm 0.22$ | $0.876\pm 0.007$
bSAM | $95.53\pm 0.09$ | $0.165\pm 0.002$ | $78.92\pm 0.25$ | $0.870\pm 0.005$
SGLD | $95.47\pm 0.11$ | $0.167\pm 0.011$ | $78.79\pm 0.35$ | $0.854\pm 0.031$
Entropy-SGLD | $94.46\pm 0.24$ | $0.194\pm 0.020$ | $77.98\pm 0.39$ | $0.897\pm 0.027$
EMCMC | $95.69\pm 0.06$ | $0.162\pm 0.002$ | $79.16\pm 0.07$ | $0.840\pm 0.004$
(a) CIFAR10 and CIFAR100

Severity | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
SGD | $88.43$ | $82.43$ | $76.20$ | $67.93$ | $55.81$
SGLD | $88.61$ | $82.46$ | $76.49$ | $69.19$ | $56.98$
EMCMC | $88.87$ | $83.27$ | $77.44$ | $70.31$ | $58.17$
(b) Corrupted CIFAR (ACC (%) $\uparrow$)

Metric | NLL $\downarrow$ | Top-1 (%) $\uparrow$ | Top-5 (%) $\uparrow$
---|---|---|---
SGD | $0.960$ | $76.046$ | $92.776$
SGLD | $0.921$ | $76.676$ | $93.174$
EMCMC | $0.895$ | $77.096$ | $93.424$
(c) ImageNet

###  6.5 Uncertainty and OOD Detection

To illustrate how predictive uncertainty estimation benefits from flat basins,
we evaluate EMCMC on out-of-distribution (OOD) detection. We train each model
on CIFAR and quantify uncertainty using the entropy of predictive
distributions (Malinin & Gales, 2018). Then we use the uncertainty to detect
SVHN samples in a joint testing set combined by CIFAR and SVHN (Netzer et al.,
2011). We evaluate each algorithm with Area under ROC Curve (AUC) (McClish,
1989) and Expected Calibration Error (ECE) (Naeini et al., 2015). All other
settings remain the same as the classification experiments. Table 2 shows the
evaluation results, where EMCMC outperforms all baselines, especially on the
ECE metric. This indicates that predictive uncertainty estimation is more
accurate if the samples are from flat basins of the posterior.

Table 2: OOD detection on CIFAR-SVHN. The predictive uncertainty quantified by EMCMC is the best among the compared algorithms. Method | CIFAR10-SVHN | CIFAR100-SVHN
---|---|---
AUC (%) $\uparrow$ | ECE (%) $\downarrow$ | AUC (%) $\uparrow$ | ECE (%) $\downarrow$
SGD | $96.67\pm 0.98$ | $18.09\pm 6.42$ | $74.85\pm 1.69$ | $14.74\pm 2.43$
Entropy-SGD | $98.17\pm 0.73$ | $6.95\pm 4.22$ | $78.89\pm 2.97$ | $9.30\pm 3.50$
SAM | $98.01\pm 0.84$ | $3.93\pm 1.19$ | $78.58\pm 1.39$ | $8.16\pm 2.13$
bSAM | $97.54\pm 0.01$ | $4.31\pm 0.01$ | $79.12\pm 0.01$ | $6.11\pm 0.03$
SGLD | $97.84\pm 0.26$ | $8.79\pm 1.77$ | $78.18\pm 0.72$ | $9.74\pm 1.55$
Entropy-SGLD | $95.89\pm 2.64$ | $16.35\pm 8.76$ | $77.50\pm 2.50$ | $6.85\pm 3.57$
EMCMC | $98.65\pm 0.52$ | $2.93\pm 0.65$ | $79.96\pm 0.52$ | $4.06\pm 0.18$

##  7 Conclusion and Discussion

We propose a practical MCMC algorithm to sample from flat basins of DNN
posterior distributions. Specifically, we introduce a guiding variable based
on the local entropy to steer the MCMC sampler toward flat basins. The joint
distribution of this variable and the model parameter enjoys a simple form
which enables efficient sampling. We prove the fast convergence rate of our
method compared with two existing flatness-aware methods. Comprehensive
experiments demonstrate the superiority of our method, verifying that it can
sample from flat basins and achieve outstanding performance on diverse tasks.
Our method is mathematically simple and computationally efficient, allowing
for adoption as a drop-in replacement for standard sampling methods such as
SGLD.

The results hold promise for both Bayesian methods and deep learning
generalization. On the one hand, we demonstrate that explicitly considering
flatness in Bayesian deep learning can significantly improve generalization,
robustness, and uncertainty estimation, especially under practical
computational constraints. On the other hand, we highlight the value of
marginalizing over flat basins in the energy landscape, as a means to attain
further performance improvements compared to single point optimization
methods.

## References

  * Ajalloeian & Stich (2020) Ahmad Ajalloeian and Sebastian U. Stich.  On the convergence of sgd with biased gradients.  _Journal of Machine Learning Research_ , 2020.  URL https://api.semanticscholar.org/CorpusID:234358812.
  * Andrieu et al. (2003) Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan.  An introduction to mcmc for machine learning.  _Machine learning_ , 50:5–43, 2003.
  * Bahri et al. (2022) Dara Bahri, Hossein Mobahi, and Yi Tay.  Sharpness-aware minimization improves language model generalization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 7360–7371, 2022\.
  * Baldassi et al. (2016) Carlo Baldassi, Christian Borgs, Jennifer T Chayes, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina.  Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes.  _Proceedings of the National Academy of Sciences_ , 113(48):E7655–E7662, 2016.
  * Bishop (2006) Christopher M Bishop.  _Pattern recognition and machine learning_ , volume 4.  Springer, 2006.
  * Bisla et al. (2022) Devansh Bisla, Jing Wang, and Anna Choromanska.  Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 8299–8339. PMLR, 2022.
  * Chaudhari et al. (2019) Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina.  Entropy-sgd: Biasing gradient descent into wide valleys.  _Journal of Statistical Mechanics: Theory and Experiment_ , 2019(12):124018, 2019.
  * Chen et al. (2014) Tianqi Chen, Emily Fox, and Carlos Guestrin.  Stochastic gradient hamiltonian monte carlo.  In _International conference on machine learning_ , pp. 1683–1691. PMLR, 2014.
  * Cobb & Jalaian (2021) Adam D Cobb and Brian Jalaian.  Scaling hamiltonian monte carlo inference for bayesian neural networks with symmetric splitting.  In _Uncertainty in Artificial Intelligence_ , pp. 675–685. PMLR, 2021.
  * Dalalyan & Karagulyan (2019) Arnak S Dalalyan and Avetik Karagulyan.  User-friendly guarantees for the langevin monte carlo with inaccurate gradient.  _Stochastic Processes and their Applications_ , 129(12):5278–5311, 2019.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Dinh et al. (2017) Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio.  Sharp minima can generalize for deep nets.  In _International Conference on Machine Learning_ , pp. 1019–1028. PMLR, 2017.
  * Dziugaite & Roy (2018) Gintare Karolina Dziugaite and Daniel Roy.  Entropy-sgd optimizes the prior of a pac-bayes bound: Generalization properties of entropy-sgd and data-dependent priors.  In _International Conference on Machine Learning_ , pp. 1377–1386. PMLR, 2018.
  * Foret et al. (2020) Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.  Sharpness-aware minimization for efficiently improving generalization.  In _International Conference on Learning Representations_ , 2020.
  * Gelfand (2000) Alan E Gelfand.  Gibbs sampling.  _Journal of the American statistical Association_ , 95(452):1300–1304, 2000.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hendrycks & Dietterich (2018) Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In _International Conference on Learning Representations_ , 2018.
  * Hendrycks & Dietterich (2019a) Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _Proceedings of the International Conference on Learning Representations_ , 2019a.
  * Hendrycks & Dietterich (2019b) Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In _International Conference on Learning Representations_ , 2019b.
  * Hochreiter & Schmidhuber (1994) Sepp Hochreiter and Jürgen Schmidhuber.  Simplifying neural nets by discovering flat minima.  _Advances in neural information processing systems_ , 7, 1994.
  * Hochreiter & Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Flat minima.  _Neural computation_ , 9(1):1–42, 1997.
  * Huang et al. (2020) W Ronny Huang, Zeyad Ali Sami Emam, Micah Goldblum, Liam H Fowl, JK Terry, Furong Huang, and Tom Goldstein.  Understanding generalization through visualizations.  In _”I Can’t Believe It’s Not Better!”NeurIPS workshop_ , 2020.
  * Izmailov et al. (2018) P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and T Garipov.  Averaging weights leads to wider optima and better generalization.  In _34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018_ , pp. 876–885, 2018.
  * Izmailov et al. (2021) Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson.  What are bayesian neural network posteriors really like?  In _International conference on machine learning_ , pp. 4629–4640. PMLR, 2021.
  * Jiang et al. (2019a) Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio.  Fantastic generalization measures and where to find them.  In _International Conference on Learning Representations_ , 2019a.
  * Jiang et al. (2019b) Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio.  Fantastic generalization measures and where to find them.  In _International Conference on Learning Representations_ , 2019b.
  * Keskar et al. (2017) Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy.  On large-batch training for deep learning: Generalization gap and sharp minima.  In _5th International Conference on Learning Representations, ICLR 2017_ , 2017.
  * Krizhevsky (2009) A Krizhevsky.  Learning multiple layers of features from tiny images.  _Master’s thesis, University of Tront_ , 2009.
  * LeCun (1998) Yann LeCun.  The mnist database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ , 1998.
  * Li et al. (2014) Mu Li, Tong Zhang, Yuqiang Chen, and Alexander J Smola.  Efficient mini-batch training for stochastic optimization.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 661–670, 2014.
  * Ma et al. (2015) Yi-An Ma, Tianqi Chen, and Emily Fox.  A complete recipe for stochastic gradient mcmc.  _Advances in neural information processing systems_ , 28, 2015.
  * Maclaurin & Adams (2015) Dougal Maclaurin and Ryan P Adams.  Firefly monte carlo: exact mcmc with subsets of data.  In _Proceedings of the 24th International Conference on Artificial Intelligence_ , pp. 4289–4295, 2015.
  * Malinin & Gales (2018) Andrey Malinin and Mark Gales.  Predictive uncertainty estimation via prior networks.  _Advances in neural information processing systems_ , 31, 2018.
  * McClish (1989) Donna Katzman McClish.  Analyzing a portion of the roc curve.  _Medical decision making_ , 9(3):190–195, 1989\.
  * Mobahi (2016) Hossein Mobahi.  Training recurrent neural networks by diffusion.  _arXiv preprint arXiv:1601.04114_ , 2016.
  * Möllenhoff & Khan (2022) Thomas Möllenhoff and Mohammad Emtiyaz Khan.  Sam as an optimal relaxation of bayes.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 29, 2015.
  * Neal (2012) Radford M Neal.  _Bayesian learning for neural networks_ , volume 118.  Springer Science & Business Media, 2012.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  _NIPS Workshop on Deep Learning and Unsupervised Feature Learning_ , 2011.
  * Neyshabur et al. (2017) Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro.  Exploring generalization in deep learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Singh & Krishnan (2020) Saurabh Singh and Shankar Krishnan.  Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 11237–11246, 2020.
  * Wasserman (2004) Larry Wasserman.  _All of statistics: a concise course in statistical inference_ , volume 26.  Springer, 2004.
  * Welling & Teh (2011) Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In _Proceedings of the 28th international conference on machine learning (ICML-11)_ , pp. 681–688, 2011.
  * Wilson (2020) Andrew Gordon Wilson.  The case for bayesian deep learning.  _arXiv preprint arXiv:2001.10995_ , 2020.
  * Zhang et al. (2020a) Ruqi Zhang, A Feder Cooper, and Christopher M De Sa.  Asymptotically optimal exact minibatch metropolis-hastings.  _Advances in Neural Information Processing Systems_ , 33:19500–19510, 2020a.
  * Zhang et al. (2020b) Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson.  Cyclical stochastic gradient mcmc for bayesian deep learning.  In _International Conference on Learning Representations_ , 2020b.

##  Appendix A Algorithm Details

We list some details of the proposed Entropy-MCMC in this section, to help
understanding our code and reproduction. As discussed in section 4.2, the
updating rule of Entropy-MCMC can be written as:

$\widetilde{\bm{\theta}}\leftarrow\widetilde{\bm{\theta}}-\alpha\nabla_{\widetilde{\bm{\theta}}}U(\widetilde{\bm{\theta}})+\sqrt{2\alpha}\cdot\bm{\epsilon},$
(11)

which is a full-batch version. We will show how to apply modern deep learning
techniques like mini-batching and temperature to the updating policy in the
following section.

###  A.1 Mini-batching

We adopt the standard mini-batching technique in our method, which samples a
subset of data points per iteration (Li et al., 2014). We assume the entire
dataset to be $\mathcal{D}=\\{(\bm{x}_{i},y_{i})\\}_{i=1}^{N}$. Then a batch
sampled from $\mathcal{D}$ is
$\bm{\Xi}=\\{(\bm{x}_{i},y_{i})\\}_{i=1}^{M}\subset\mathcal{D}$ with $M\ll N$.
For the entrie dataset, the loss function is computed by:

$f(\bm{\theta})\propto-\sum_{i=1}^{N}\log{p(y_{i}|\bm{x}_{i},\bm{\theta})}-\log{p(\bm{\theta})},$
(12)

and in order to balance the updating stride per iteration, the loss function
for a mini-batch is:

$f_{\bm{\Xi}}(\bm{\theta})\propto-\frac{N}{M}\sum_{i=1}^{M}\log{p(y_{i}|\bm{x}_{i},\bm{\theta})}-\log{p(\bm{\theta})}.$
(13)

Therefore, if we average the mini-batch loss over all data points, we can
obtain the following form:

$\bar{f}_{\bm{\Xi}}(\bm{\theta})\propto-\frac{1}{M}\sum_{i=1}^{M}\log{p(y_{i}|\bm{x}_{i},\bm{\theta})}-\frac{1}{N}\log{p(\bm{\theta})}.$
(14)

If we regard the averaging process as a modification on the stepsize $\alpha$
(i.e., $\bar{\alpha}=\alpha/N$), we will have the following form for the
updating policy:

$\displaystyle\Delta\widetilde{\bm{\theta}}$
$\displaystyle=-\bar{\alpha}\cdot\nabla_{\widetilde{\bm{\theta}}}\widetilde{U}(\widetilde{\bm{\theta}})+\sqrt{2\bar{\alpha}}\cdot\bm{\epsilon}$
(15)
$\displaystyle=-\bar{\alpha}\cdot\nabla_{\widetilde{\bm{\theta}}}\left[f_{\bm{\Xi}}(\bm{\theta})+\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}-\sqrt{\frac{2}{\bar{\alpha}}}\bm{\epsilon}\odot\widetilde{\bm{\theta}}\right]$
$\displaystyle=-\alpha\cdot\nabla_{\widetilde{\bm{\theta}}}\left[\bar{f}_{\bm{\Xi}}(\bm{\theta})+\frac{1}{2\eta
N}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}-\sqrt{\frac{2}{\alpha
N}}\bm{\epsilon}\odot\widetilde{\bm{\theta}}\right].$

Therefore, the updating rule in Eq.11 can be equivalently written as:

$\widetilde{\bm{\theta}}\leftarrow\widetilde{\bm{\theta}}-\Delta\widetilde{\bm{\theta}}.$
(16)

###  A.2 Data Augmentation and Temperature

We apply data augmentation, which is commonly used in deep neural networks,
and compare all methods with data augmentation in the main text. Here, we
additionally compare the classification results without data augmentation in
Table 3 to demonstrate the effectiveness of EMCMC in this case.

Table 3: Comparison of data augmentation of 3 baselines on CIFAR10. EMCMC outperforms previous methods with and without data augmentation. Augmentation | SGD | SGLD | EMCMC
---|---|---|---
$\times$ | $89.60$ | $89.24$ | $89.87$
✓ | $95.59$ | $95.64$ | $95.79$

Besides, in the updating policy, a noise term is introduced to add randomness
to the sampling process. However, in mini-batch training, the effect of noise
will be amplified so that the stationary distribution of might be far away
from the true posterior distribution (Zhang et al., 2020b). Therefore, we also
introduce a system temperature $T$ to address this problem.

Formally, the posterior distribution is tempered to be
$p(\bm{\theta}|\mathcal{D})\propto\exp(-U(\bm{\theta})/T)$, with an averagely
sharpened energy landscape. Similarly, we can regard the temperature effect as
a new stepsize $\alpha_{T}=\alpha/T$, and the updating policy would be:

$\displaystyle\Delta\widetilde{\bm{\theta}}$
$\displaystyle=-\alpha\cdot\nabla_{\widetilde{\bm{\theta}}}\left[\left(\bar{f}_{\bm{\Xi}}(\bm{\theta})+\frac{1}{2\eta
N}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}\right)/T-\sqrt{\frac{2}{\alpha
N}}\bm{\epsilon}\odot\widetilde{\bm{\theta}}\right]$ (17)
$\displaystyle=-\alpha_{T}\cdot\nabla_{\widetilde{\bm{\theta}}}\left[\bar{f}_{\bm{\Xi}}(\bm{\theta})+\frac{1}{2\eta
N}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}-\sqrt{\frac{2T}{\alpha_{T}N}}\bm{\epsilon}\odot\widetilde{\bm{\theta}}\right].$

In order to empirically determine the best temperature, we compare different
temperature level in table 4, and find that $T=10^{-4}$ is appropriate for
classification task.

Table 4: Comparison of temperature effect on CIFAR10 with data augmentation. $T=10^{-4}$ is best temperature level. $T$ | 1 | 1e-1 | 1e-2 | 1e-3 | 1e-4 | 1e-5
---|---|---|---|---|---|---
Testing ACC (%) $\uparrow$ | $95.30$ | $95.42$ | $95.41$ | $95.48$ | $95.50$ | $95.47$

###  A.3 Gibbs-like Updating Procedure

Instead of jointly updating, we can also choose to alternatively update
$\bm{\theta}$ and $\bm{\theta}_{a}$. The conditional distribution for the
model $\bm{\theta}$ is:

$p(\bm{\theta}|\bm{\theta}_{a},\mathcal{D})=\frac{p(\bm{\theta},\bm{\theta}_{a}|\mathcal{D})}{p(\bm{\theta}_{a}|\mathcal{D})}\propto\frac{1}{Z_{\bm{\theta}_{a}}}\exp{\left\\{-f(\bm{\theta})-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}\right\\}},$
(18)

where $Z_{\bm{\theta}_{a}}=\exp{\mathcal{F}(\bm{\theta}_{a};\eta)}$ is a
constant. While for the guiding variable $\bm{\theta}_{a}$, its conditional
distribution is:

$p(\bm{\theta}_{a}|\bm{\theta},\mathcal{D})=\frac{p(\bm{\theta},\bm{\theta}_{a}|\mathcal{D})}{p(\bm{\theta}|\mathcal{D})}\propto\frac{1}{Z_{\bm{\theta}}}\exp{\left\\{-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}\right\\}},$
(19)

where $Z_{\bm{\theta}}=\exp{(-f(\bm{\theta}))}$ is a constant. Therefore, with
Guassian noise, $\bm{\theta}_{a}$ is equivalently sampled from
$\mathcal{N}(\bm{\theta},\eta\bm{I})$, and the variance $\eta$ controls the
expected distance between $\bm{\theta}$ and $\bm{\theta}_{a}$. To obtain
samples from the joint distribution, we can sample from
$p(\bm{\theta}|\bm{\theta}_{a},\mathcal{D})$ and
$p(\bm{\theta}_{a}|\bm{\theta},\mathcal{D})$ alternatively. The advantage of
doing Gibbs-like updating is that sampling $\bm{\theta}_{a}$ can be done
exactly. Empirically, we observe that joint updating yields superior
performance compared to Gibbs-like updating due to the efficiency of updating
both $\bm{\theta}$ and $\bm{\theta}_{a}$ at the same time.

##  Appendix B Proof of Theorems

###  B.1 Lemma 1

_Proof._ Assume
$\widetilde{\bm{\theta}}=[\bm{\theta}^{T},\bm{\theta}_{a}^{T}]^{T}$ is sampled
from the joint posterior distribution:

$p(\widetilde{\bm{\theta}}|\mathcal{D})=p(\bm{\theta},\bm{\theta}_{a}|\mathcal{D})\propto\exp\left\\{-f(\bm{\theta})-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}\right\\}.$
(20)

Then the marginal distribution respectively for $\bm{\theta}$ and
$\bm{\theta}_{a}$ would be:

$\displaystyle p(\bm{\theta}|\mathcal{D})$
$\displaystyle=\int_{\bm{\Theta}}p(\bm{\theta},\bm{\theta}_{a}|\mathcal{D})d\bm{\theta}_{a}$
(21)
$\displaystyle=\int_{\bm{\Theta}}\exp\left\\{-f(\bm{\theta})-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}\right\\}d\bm{\theta}_{a}$
$\displaystyle=\exp(-f(\bm{\theta}))\int_{\bm{\Theta}}\exp\left\\{-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}\right\\}d\bm{\theta}_{a}$
$\displaystyle=\exp(-f(\bm{\theta})),$

and similarly we have

$\displaystyle p(\bm{\theta}_{a}|\mathcal{D})$
$\displaystyle=\int_{\bm{\Theta}}p(\bm{\theta},\bm{\theta}_{a}|\mathcal{D})d\bm{\theta}$
(22)
$\displaystyle=\int_{\bm{\Theta}}\exp\left\\{-f(\bm{\theta})-\frac{1}{2\eta}\|\bm{\theta}-\bm{\theta}_{a}\|^{2}\right\\}d\bm{\theta}$
$\displaystyle=\exp{\mathcal{F}(\bm{\theta}_{a};\eta)}.$

∎

###  B.2 Lemma 2

###### Proof.

Note that we have

$-\nabla^{2}\log\pi_{\text{joint}}=\begin{bmatrix}\nabla^{2}f(\theta^{\prime})+\frac{1}{\eta}I&-\frac{1}{\eta}I\\\
-\frac{1}{\eta}I&\frac{1}{\eta}I\end{bmatrix},$

and after a row reduction, we get

$\begin{bmatrix}\nabla^{2}f(\theta^{\prime})&0\\\
-\frac{1}{\eta}I&\frac{1}{\eta}I\end{bmatrix}.$

The eigenvalues for this matrix are the eigenvalues of
$\nabla^{2}f(\theta^{\prime})$ and $1/\eta$. By the assumption $m\leq
1/\eta\leq M$, we have

$mI\preceq\begin{bmatrix}\nabla^{2}f(\theta^{\prime})&0\\\
-\frac{1}{\eta}I&\frac{1}{\eta}I\end{bmatrix}\preceq MI,$

which means $-\nabla^{2}\log\pi_{\text{joint}}$ is also a $M$-smooth and
$m$-strongly convex function.

In most scenarios, $m$ often takes on very small values and $M$ tends to be
very large. Therefore, the assumption $m\leq 1/\eta\leq M$ is mild. ∎

###  B.3 Proof of Theorem 1

###### Proof.

The proof relies on Theorem 4 from Dalalyan & Karagulyan (2019). Lemma 2 has
already provided us with the smoothness and strong convexity parameters for
$\pi_{\text{joint}}$. We will now address the bias and variance of stochastic
gradient estimation. The stochastic gradient is given by
$[\nabla\tilde{f}(\theta^{\prime})+\frac{1}{\eta}(\theta^{\prime}-\theta),-\frac{1}{\eta}(\theta^{\prime}-\theta)]^{T}$.
As $\nabla\tilde{f}(\theta^{\prime})$ is unbiased and has a variance of
$\sigma^{2}$, the stochastic gradient in our method is also unbiased and has
variance $\sigma^{2}$. Combining the above results, we are ready to apply
Theorem 4 (Dalalyan & Karagulyan, 2019) and obtain

$W_{2}(\mu_{K},\pi_{\text{joint}})\leq(1-\alpha
m)^{K}W_{2}(\mu_{0},\pi)+1.65(M/m)(2\alpha d)^{1/2}+\frac{\sigma^{2}(2\alpha
d)^{1/2}}{1.65M+\sigma\sqrt{m}}.$

∎

###  B.4 Theorem 3

###### Proof.

Let
$\pi^{\prime}(\theta^{\prime})\propto\exp(-f(\theta^{\prime})-\frac{1}{2\eta}\left\|\theta^{\prime}-\theta\right\|^{2}_{2})$.
It is easy to see that $m+1/\eta\preceq\nabla^{2}(-\log\pi^{\prime})\preceq
M+1/\eta$. Based on Theorem 4 in Dalalyan & Karagulyan (2019), the
2-Wasserstein distance for the inner Markov chain is

$\displaystyle W_{2}(\zeta_{L},\pi^{\prime})$ $\displaystyle\leq(1-\alpha
m)^{L}W_{2}(\zeta_{0},\pi^{\prime})+1.65\left(\frac{M+1/\eta}{m+1/\eta}\right)(\alpha
d)^{1/2}+\frac{\sigma^{2}(\alpha
d)^{1/2}}{1.65(M+1/\eta)+\sigma\sqrt{m+1/\eta}}$ $\displaystyle\leq(1-\alpha
m)^{L}\kappa+1.65\left(\frac{M+1/\eta}{m+1/\eta}\right)(\alpha
d)^{1/2}+\frac{\sigma^{2}(\alpha
d)^{1/2}}{1.65(M+1/\eta)+\sigma\sqrt{m+1/\eta}}$ $\displaystyle:=A^{2}.$

Now we consider the convergence of the outer Markov chain. We denote
$\pi_{\text{flat}}(\theta)\propto\exp{\mathcal{F}(\bm{\theta};\eta)}$. From
Chaudhari et al. (2019), we know that

$\inf_{\theta}\left\|\frac{1}{I+\eta\nabla^{2}f(\theta)}\right\|mI\preceq-\nabla^{2}\log\pi_{\text{flat}}\preceq\sup_{\theta}\left\|\frac{1}{I+\eta\nabla^{2}f(\theta)}\right\|MI.$

Since $mI\preceq\nabla^{2}f(\theta)\preceq MI$, it follows

$\inf_{\theta}\left\|\frac{1}{I+\eta\nabla^{2}f(\theta)}\right\|\geq\frac{1}{1+\eta
M},\hskip
20.00003pt\sup_{\theta}\left\|\frac{1}{I+\eta\nabla^{2}f(\theta)}\right\|\leq\frac{1}{1+\eta
m}.$

Therefore,

$\frac{m}{1+\eta
M}I\preceq-\nabla^{2}\log\pi_{\text{flat}}\preceq\frac{M}{1+\eta m}I.$

The update rule of the outer SGLD is

$\theta=\theta-\alpha/\eta(\theta-\mathbf{E}_{\zeta_{L}}[\theta^{\prime}])+\sqrt{2\alpha}\xi.$

The gradient estimation can be written as
$\theta-\mathbf{E}_{\pi^{\prime}}[\theta^{\prime}]+(\mathbf{E}_{\pi^{\prime}}[\theta^{\prime}]-\mathbf{E}_{\zeta_{L}}[\theta^{\prime}])$
which can be regarded as the true gradient
$\theta-\mathbf{E}_{\pi^{\prime}}[\theta^{\prime}]$ plus some noise
$(\mathbf{E}_{\pi^{\prime}}[\theta^{\prime}]-\mathbf{E}_{\zeta_{L}}[\theta^{\prime}])$.
The bias of the noise can be bounded as follows

$\displaystyle\left\|\mathbf{E}_{\pi^{\prime}}[\theta^{\prime}]-\mathbf{E}_{\zeta_{L}}[\theta^{\prime}]\right\|^{2}_{2}$
$\displaystyle=\left\|\int[\theta^{\prime}_{\pi^{\prime}}-\theta^{\prime}_{\zeta_{L}}]dJ(\theta^{\prime}_{\pi^{\prime}},\theta^{\prime}_{\zeta_{L}})\right\|^{2}_{2}$
$\displaystyle\leq\int\left\|\theta^{\prime}_{\pi^{\prime}}-\theta^{\prime}_{\zeta_{L}}]\right\|^{2}_{2}dJ(\theta^{\prime}_{\pi^{\prime}},\theta^{\prime}_{\zeta_{L}}).$

Since the inequality holds for any $J$, we can take the infimum over all
possible distributions to conclude

$\displaystyle\left\|\mathbf{E}_{\pi^{\prime}}[\theta^{\prime}]-\mathbf{E}_{\zeta_{L}}[\theta^{\prime}]\right\|^{2}_{2}\leq
W_{2}(\zeta_{L},\pi^{\prime}).$

Furthermore, we note that the variance of the noise is zero. Therefore, by
applying Theorem 4 in Dalalyan & Karagulyan (2019) we get

$W_{2}(\nu_{K},\pi_{\text{flat}})\leq(1-\alpha
m)^{K}W_{2}(\nu_{0},\pi_{\text{flat}})+1.65\left(\frac{1+\eta M}{1+\eta
m}\right)(M/m)(\alpha d)^{1/2}+\frac{A(1+\eta M)}{m}.$

∎

###  B.5 Theorem 2

###### Proof.

Compared to Entropy-SGLD, the only difference of Entropy-SGD is to do SGD
update instead of SGLD update in the outer loop. Therefore, the analysis for
the inner Markov chain remains the same as in Theorem 3. To analyze the error
of SGD in the outer loop, we follow the results in Ajalloeian & Stich (2020).
Since the strongly convex parameter for $f_{\text{flat}}$ is $\frac{m}{1+\eta
M}$, by Section 4.2 and Assumption 4 in Ajalloeian & Stich (2020), we know
that

$\displaystyle\frac{1}{2}\left\|\nabla
f_{\text{flat}}(\bm{\theta}_{t})\right\|^{2}$
$\displaystyle\leq\frac{E_{t}-E_{t+1}}{\alpha}+\frac{1}{2}A$
$\displaystyle\Rightarrow\frac{m}{1+\eta M}E_{t}$
$\displaystyle\leq\frac{E_{t}-E_{t+1}}{\alpha}+\frac{1}{2}A$
$\displaystyle\Rightarrow E_{t+1}$ $\displaystyle\leq(1-\frac{\alpha m}{1+\eta
M})E_{t}+\frac{1}{2}\alpha A.$

By unrolling the recursion, we obtain

$E_{K}\leq\left(1-\frac{\alpha m}{1+\eta M}\right)^{K}E_{0}+\frac{A(1+\eta
M)}{2m}.$

∎

##  Appendix C Ablation Studies

We empirically discuss several important hyper-parameters and algorithm
settings in this section, which justifies our choice of their values.

###  C.1 Variance Term

We compare different choices of the variance term $\eta$ to determine the best
value for each dataset. The experimental results are shown in Fig. 6.
Generally, the best $\eta$ for CIFAR100 is about $10^{-2}$ and for CIFAR10 is
about $10^{-3}$, which implies that the energy landscapes of CIFAR10 and
CIFAR100 may be different.

(a) CIFAR10 (b) CIFAR100

Figure 6: Comparison of different variance level for Entropy algorithm.
$\eta\approx 10^{-3}$ is the best on CIFAR10, and $\eta\approx 10^{-2}$ is the
best on CIFAR100.

###  C.2 Step size Schedules

We compare different types of stepsize schedules in Table 6. Specifically, we
assume the initial and final stepsize to be $\alpha_{0}$ and $\alpha_{1}$
respectively. $T$ is the total number of epochs and $t$ is the current epoch.
The detailed descriptions of stepsize schedules are listed in Table 5. The
cyclical stepsize is the best among all stepsize schedules.

Table 5: Formulas or descriptions of different stepsize schedules. Name | Formula/Description
---|---
constant | $\alpha(t)=\alpha_{0}$
linear | $\alpha(t)=\frac{T-t}{T}(\alpha_{0}-\alpha_{1})+\alpha_{1}$
exponential | $\alpha(t)=\alpha_{0}\cdot\left(\alpha_{1}/\alpha_{0}\right)^{t/T}$
step | Remain the same stepsize within one “step”, and decay between “steps”.
cyclical | Follow Eq. 1 in Zhang et al. (2020b).
Table 6: Comparison of stepsize schedules on CIFAR100. The cyclical stepsize is the best for Entropy-MCMC. Learning Rate Schedule | constant | linear | exponential | step | cyclical
---|---|---|---|---|---
Testing ACC (%) $\uparrow$ | $88.04$ | $87.89$ | $87.75$ | $89.59$ | $89.93$

###  C.3 Collecting Samples

Due to the introduction of the auxiliary guiding variable $\bm{\theta}_{a}$,
the composition of sample set $\mathcal{S}$ has multiple choices: only collect
samples of $\bm{\theta}$, only collect samples of $\bm{\theta}_{a}$, collect
both samples. We conduct the comparison of all choices and the results are
reported in Table 7. It shows that using samples from both $\bm{\theta}$ and
$\bm{\theta}_{a}$ gives the best generalization accuracy.

Table 7: Ablation study on the composition of sample set $\mathcal{S}$ on CIFAR10. With samples from both $\bm{\theta}$ and $\bm{\theta}_{a}$, the Bayesian marginalization can achieve the best accuracy. $\bm{\theta}$ | $\bm{\theta}_{a}$ | ACC (%) $\uparrow$
---|---|---
✓ |  | $95.58$
| ✓ | $95.64$
✓ | ✓ | $95.65$

###  C.4 Normalization Layers

During testing, the usage of bath normalization layers (BN) in the model
architecture induces a problem regarding the mini-batch statistics. The mean
and variance of a batch need calculated through at least one forward pass,
which is not applicable for the guiding variable $\bm{\theta}_{a}$ since it is
updated by the distance regularization during training. We try different
solutions for this problem, including one additional forward pass and the
Filter Response Normalization (Singh & Krishnan, 2020). The comparison is
listed in Table 8, where simply adding one additional forward pass during
testing can achieve promising accuracy with negligible computational overhead.

Table 8: Comparison of different normalization layers on CIFAR10. Simply adding one additional forward pass during testing with standard batch normalization is the best solution. Normalization Layer | ACC (%) $\uparrow$ | Time (h)
---|---|---
BN | $95.40$ | $1.8$
BN (one additional forward) | $95.47$ | $1.9$
FRN (Singh & Krishnan, 2020) | $93.92$ | $2.5$

###  C.5 SGD Burn-in

We also try SGD burn-in in our ablation studies, by adding the random noise
term only to the last few epochs to ensure the fast convergence. We evaluate
different settings of SGD burn-in epochs in Table 9. We find that adding 40
burn-in epochs per 50 epochs is the best choice.

Table 9: Comparison of different SGD burn-in epochs on CIFAR10. In a 50-epoch round, using SGD burn-in in the first 40 epochs is the best choice. SGD Burn-in Epoch | 0 | 10 | 20 | 30 | 40 | 47
---|---|---|---|---|---|---
Test ACC (%) $\uparrow$ | $95.61$ | $95.62$ | $95.57$ | $95.67$ | $95.72$ | $95.41$

##  Appendix D Additional Experimental Results

We list the additional experimental results in this section, to demonstrate
the superiority of our method and show some interesting findings.

###  D.1 Additional Synthetic Examples

To demonstrate that EMCMC can bias toward the flat mode under random
initialization, we conduct additional synthetic experiments under two
different initialization settings. Specifically, we set the initial point to
be $(-0.4,-0.4)$ to prefer the sharp mode (Fig. 7) and $(0.0,0.0)$ to prefer
the flat mode (Fig. 8). It is clear that EMCMC can find the flat mode under
all initialization settings, while SGD and SGLD are heavily affected by the
choices of initialization.

(a) SGD (b) SGLD (c) EMCMC ($\bm{\theta}$) (d) EMCMC ($\bm{\theta}_{a}$)

Figure 7: Synthetic Experiments with sharp-mode-biased initialization.

(a) SGD (b) SGLD (c) EMCMC ($\bm{\theta}$) (d) EMCMC ($\bm{\theta}_{a}$)

Figure 8: Synthetic Experiments with flat-mode-biased initialization.

###  D.2 Parameter space interpolation

As the supplement for Fig. 5, we show additional interpolation results to
demonstrate some interesting findings about the model $\bm{\theta}$ and the
auxiliary guiding variable $\bm{\theta}_{a}$. The additional interpolation can
be separated into the following types:

####  D.2.1 toward Random Directions.

We show the interpolation results toward averaged random directions (10 random
directions) in Fig. 9. For the training loss, the auxiliary guiding variable
$\bm{\theta}$ is located at a flatter local region with relatively larger loss
values. While for the testing error, the guiding variable $\bm{\theta}_{a}$
consistently has lower generation errors, which illustrates that the flat
modes are more preferable in terms of generalization.

(a) Random interpolation by training NLL (b) Random interpolation by testing
error

Figure 9: Interpolation toward averaged random directions on CIFAR100,
comparing the model $\bm{\theta}$ and the guiding variable $\bm{\theta}_{a}$.

####  D.2.2 Between Model Parameter and Guiding Variable.

The line between the model parameter $\bm{\theta}$ and the guiding variable
$\bm{\theta}_{a}$ is a special direction in the parameter space. The NLL and
testing error are both much lower than random directions, which is shown in
Fig. 10. Besides, this special direction is biased toward the local region of
$\bm{\theta}_{a}$, with averagely lower testing errors. This finding justifies
the setting of adding $\bm{\theta}_{a}$ to the sample set $\mathcal{S}$, since
the generalization performance of $\bm{\theta}_{a}$ is better.

(a) Interpolation between $\bm{\theta}$ and $\bm{\theta}_{a}$ by training NLL
(b) Interpolation between $\bm{\theta}$ and $\bm{\theta}_{a}$ by testing error

Figure 10: Interpolation between the model $\bm{\theta}$ and the guiding
variable $\bm{\theta}_{a}$ in terms of training NLL and testing error on
CIFAR100.

###  D.3 Classification on Corrupted CIFAR

We list the detailed classification results on corrupted CIFAR (Hendrycks &
Dietterich, 2018) in Fig. 11, where each corruption type is evaluated at a
corresponding subfigure. For the majority of corruption types, our method
ourperforms other baselines under all severity levels, and is superior
especially under sever corruptions.

(a) brightness (b) contrast (c) defocus_blur (d) elastic_transform (e) fog (f)
frost (g) gaussian_blur (h) gaussian_noise (i) glass_blur (j) impulse_noise
(k) jpeg_compression (l) motion_blur (m) pixelate (n) saturate (o) shot_noise
(p) snow (q) spatter (r) speckle_noise (s) zoom_blur

Figure 11: Classification accuracies under different severity levels on
corrupted CIFAR. The results are shown per corruption type. Our method
outperforms the compared baselines on most of corruption types, especially
under high severity levels.


# Are Classes Clusters?

Kees Varekamp
Stanford Center for Professional Development
<EMAIL_ADDRESS>
###### Abstract

Sentence embedding models aim to provide general purpose embeddings for
sentences. Most of the models studied in this paper claim to perform well on
STS tasks – but they do not report on their suitability for clustering.

This paper111Project was completed as part of the XCS224U professional course
looks at four recent sentence embedding models (Universal Sentence Encoder
(Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), LASER (Artetxe
and Schwenk, 2019), and DeCLUTR (Giorgi et al., 2020)). It gives a brief
overview of the ideas behind their implementations.

It then investigates how well topic classes in two text classification
datasets (Amazon Reviews (Ni et al., 2019) and News Category Dataset (Misra,
2018)) map to clusters in their corresponding sentence embedding space. While
the performance of the resulting classification model is far from perfect, it
is better than random.

This is interesting because the classification model has been constructed in
an unsupervised way. The topic classes in these real life topic classification
datasets can be partly reconstructed by clustering the corresponding sentence
embeddings.

Model | Cluster F1 | LogReg F1
---|---|---
Sentence-BERT | 0.22 | 0.53
Universal Sentence Encoder | 0.29 | 0.56
DeCLUTR | 0.22 | 0.56
LASER | 0.10 | 0.42
TfidfVectorizer | 0.07 | 0.55
Table 1: Average F1 scores of unsupervised cluster classifier and logistic
regression classifier. Averages taken over two datasets: Amazon Reviews
dataset and News dataset. Bold is best of column.

##  1 Introduction

Since the success of universal word embeddings such as word2vec (Mikolov et
al., 2013) and GloVE (Pennington et al., 2014) the interest in the sentence
equivalent of such embeddings has been increasing.

Sentence embeddings promise to be useful for many tasks in natural language
processing. They can provide standardized inputs for custom models such as
topic classifiers or sentiment analysis models. But they are even more useful
for tasks that involve computing the semantic similarity between many
sentences.

It is perfectly possible to train a well-performing model on sentence
similarity without any pre-computed sentence embeddings. But as was pointed
out by (Reimers and Gurevych, 2019), you will need to run one inference for
every similarity between two sentences. For a similarity matrix between $n$
sentences you will need to run $n(n-1)/2$ inferences.

And when the model is large (a Transformer based model for example), these
costs add up. The big advantage of universal sentence embedding models is that
the vector representations of the sentences can be pre-computed individually.
If the proximity of two vectors in the embedding space closely matches the
semantic similarity of the two corresponding sentences (and most sentence
embedding models indeed aim for this), the similarity calculation becomes as
simple as the opposite of an L2 difference or a cosine similarity. Only $n$
potentially expensive embeddings need to be pre-computed instead of
$n(n-1)/2$, after which the above similarity matrix can be produced much more
efficiently.

This opens the door to many useful applications that involve semantic search
in one way or another, two of the most prominent ones being k-nearest-
neighbour search and cluster analysis.

This paper focuses on cluster analysis. It will try to answer the following
question: Do topic classes in classification datasets correspond to clusters
in the corresponding embedding space?

It attempts to answer this question in a very straightforward manner: by
creating a classification model based on an unsupervised clustering algorithm.
We use the usual metrics to evaluate the performance of this unsupervised
model. And finally we compare the result to a logistic regression model which
acts as the standard supervised baseline.

It turns out (Table 1) that it is in fact somewhat possible to use clusters in
the embedding space of a topic classification corpus to classify the
corresponding topics. The approach is nowhere near state of the art, but the
fact that it works is interesting and may be useful for unsupervised text
classification applications.

##  2 Related work: SentEval

The canonical benchmark for sentence embedding models is SentEval (Conneau and
Kiela, 2018). SentEval consists of 2 parts: The first part is a series of
benchmarks that are called Downstream tasks - these are tasks that use the
embeddings to perform a series of common NLP benchmark tasks. The second part
is called Probing Tasks. These are a series of tasks that attempt to
reconstruct certain linguistic properties of the original sentence.

The tasks report a variety of scores, but all of them include at least either
accuracy or Spearman’s rank correlation coefficient.

The models in this study have been evaluated on SentEval for comparison with
the Cluster Classifier experiment. Results are in Table 2.

##  3 Data

###  3.1 Amazon Reviews Dataset

The first dataset that we investigate is the Amazon Reviews Dataset (Ni et
al., 2019). The data was downloaded from
https://nijianmo.github.io/amazon/index.html and was sampled down to contain a
1000 examples for each category. There are 29 categories of products in the
dataset (for example ”Books”, ”Appliances”, ”Toys and Games”, etc).

There are titles in the reviews but they are not always equally suitable for
topic classification: They tend to be short and generic (”Great Product”). So
rather than using the titles we pick the first sentence in the review that is
longer than 10 words and shorter than 20 words.

This approach aims to select sentences that are long enough to be
representative of the topic label, and filter out short sentences that may be
too generic (such as ‘I really like this product’). But it also avoids
selecting sentences that are too long for the sentence embedding models to be
able to accurately represent them (the representations will get too diluted).

The approach gives pretty good results for finding meaningful examples. Some
examples:
(Luxury and Beauty): ”It doesnt stay on or protect your hands through
washing.”
(Fashion):”Bought the just as gorgeous rich, red Fireman’s scarf, too.”
(Magazine Subscriptions): ”I enjoy this magazine very much And especially this
one on COLOR.”
(Automotive): ”Great for routine maintenance tasks such as oil changes, etc.”

###  3.2 News Dataset

The second dataset collects news from the Huffington Post (Misra, 2018) and
was downloaded from https://www.kaggle.com/rmisra/news-category-dataset. It
has also been sampled down to contain a 1000 examples of each news category.
There are 40 news categories, for example ”SPORTS”, ”CRIME”, etc.

Each example contains metadata, a short description, a headline and a
category. We choose the headline for our analysis.

Some examples:
(CRIME): ”There Were 2 Mass Shootings In Texas Last Week, But Only 1 On TV”
(POLITICS): ”How The Chinese Exclusion Act Can Help Us Understand Immigration
Politics Today”
(WOMEN): ”The 20 Funniest Tweets From Women This Week”
(TECH): ”Self-Driving Uber In Fatal Accident Had 6 Seconds To React Before
Crash”

##  4 Models

###  4.1 Embedding Models

This section describes the universal sentence embedding models that we will
investigate.

####  4.1.1 Predecessors

A lot of work has been done in this space, and there is not enough room here
to go over all of it. So this summary only includes four of the most recent
models in some detail (USE, SBERT, LASER, and DeCLUTR). It is omitting
important ground work, in particular the following 3 studies and their
resulting models:

  * •

SkipThought (Kiros et al., 2015): Basically “Thou shalt know a word by the
company it keeps”, but extended to sentences. An unsupervised method based on
the idea that sentences that are close to each other are more similar than
sentences that are far away from each other, much like word2vec and glove.

  * •

InferSent (Conneau et al., 2017): max pooling on the hidden states of a biLSTM
trained on the Stanford Natural Language Inference dataset

  * •

QuickThoughts (Logeswaran and Lee, 2018): A non-generative version of
SkipThought – rather than trying to reconstruct neighbouring sentences, it
tries to train a classifier for predicting neighbouring/not neighbouring.

####  4.1.2 Universal Sentence Encoder

Universal Sentence Encoder (USE) (Cer et al., 2018) is a model developed by
Google. The goal of USE is to provide easy-to-use sentence embeddings with
good transfer performance.

There are actually 2 implementations of the encoder:

  * •

A Transformer-based model using (self) attention that mean-pools the word
embeddings to produce sentence embeddings

  * •

A Deep Averaging Network (DAN) that simply averages word embeddings and bi-
gram embeddings and feeds the resulting representation into a straightforward
deep neural network. The DAN trades accuracy for performance, with respect to
the transformer, which scales with the square of the sentence length.

The models are trained on a set of self-supervised tasks and on SNLI

The unsupervised tasks are trained on Wikipedia, web news, web question-answer
pages and discussion forums

One interesting note in the paper mentions that for STS tasks the authors use
the angular distance (the arccos of the cosine similarity) as a distance
measure, because “arccos converts cosine similarity into an angular distance
that obeys the triangle inequality. We find that angular distance performs
better on STS than cosine similarity.”

####  4.1.3 Sentence-BERT

Sentence-BERT (SBERT) (Reimers and Gurevych, 2019) is a BERT based model that
uses a Siamese network approach to fine-tune BERT on a set of tasks in order
to produce useful sentence embeddings.

For SBERT the goal is to produce embeddings that are primarily useful for
computationally and/or combinatorially expensive tasks like clustering and
nearest neighbour search – producing vectors that can be used in optimized
computation libraries such as numpy or sklearn.

SBERT promises semantically meaningful sentence embeddings (similar sentences
are close in vector space).

It tries to accomplish this by pooling the token embeddings while training on
STS tasks but using fixed weights between 2 BERTs, so that the resulting
embeddings for the two input sentences are created using identical weights. It
claims to be better on STS or SentEval than InferSent or USE.

SBERT trains using 3 objectives:

  * •

Classification, by pooling the embeddings of $u$ and $v$ as $[u;v;u-v]$ as
input to a softmax, using cross entropy loss

  * •

Regression (presumably on STS), by using cosim(u,v) using MSE as the loss

  * •

Triplet loss: Given 3 sentences; a (anchor), p (positive example) and n
(negative example), train the model such that $|a-p|<|a-n|$, or in other
words: minimize $|a-p|-|a-n|$.

SBERT is trained on SNLI and MultiNLI. It does well on STS tasks (as expected)

####  4.1.4 LASER

LASER (Artetxe and Schwenk, 2019) is a bit different from the other models in
this summary in that it is primarily interested in universal language agnostic
sentence embeddings. It is trained on a large dataset consisting of 93
languages. It uses a relative simple biLSTM to encode the input text – BPE
tokenized text of any language is fed into the same encoder. This creates a
fixed size language agnostic embedding that is then used in the decoder to
translate the output into a specified language. The paper claims that fixed
length representations are more versatile and compatible than variable length
representations: “For instance, there is not always a one-to-one
correspondence among words in different languages (e.g. a single word of a
morphologically complex language might correspond to several words of a
morphologically simple language), so having a separate vector for each word
might not transfer as well across languages.”

  * •

The embeddings are results of max pooling the hidden states of layers of the
biLSTM of dimensionality 512, concatenating forward and backward hidden state
into sentence representations of dimensionality 1024

  * •

The authors claim that the model does well on

    * –

XNLI (entailment)

    * –

MLDoc (document classification)

    * –

BUCC (finding the same sentence in another language)

    * –

Tatoeba: a new cross language similarity search benchmark

  * •

In the future, the authors would like to improve the result by using a self
attention encoder, pre trained word embeddings, and back translation.

####  4.1.5 DeCLUTR

DeCLUTR (Giorgi et al., 2020) aims to produce useful universal sentence
embeddings by unsupervised learning only. It points out that the best results
so far have been obtained by methods that used at least some supervised
learning, but that it is important to close the gap between supervised and
unsupervised methods for languages and domains for which no supervised data
exists.

  * •

Wants to be good at wide variety of tasks

  * •

Most universal sentence embedders train supervised on SNLI or multiNLI
(entailment, contradiction, neutral). Examples are InferSent, USE, and SBERT.

  * •

The authors describe Skip-thought as an unsupervised generative model that
uses sentence embeddings to predict words in neighbouring sentences. They
mention that the generative nature of the model makes it expensive and surface
focused. QuickThoughts tries to improve on this by classifying context
sentences from non-context sentences rather than generating them.

  * •

The authors describe DeCLUTR’s approach as similar to SBERT, but self-
supervised, and the objective as similar to QuickThoughts, but using segments
rather than whole sentences.

  * •

Like SBERT’s triplet approach, it uses contrastive loss: It tries to minimize
$|a-p|-|a-n|$.

  * •

Like QuickThoughts, it classifies sentences (or rather sentence segments) as
near by or far away.

####  4.1.6 TfidfVectorizer

This is simply the TfidfVectorizer model from scikit-learn (https://scikit-
learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html).
It is a WordVectorizer (bag of words) with a TF-IDF weighting applied to it.
It has been included in the experiments as a baseline for the encoder models.

###  4.2 Classifiers

This section describes the models we will be using for our experiment

####  4.2.1 Cluster Classifier

We test the main hypothesis by using a simple model:

  * •

For every dataset, we cluster the data into as many clusters as there are
classes. We use k-means clustering.

  * •

For every cluster, we allocate the most frequently occurring class within the
cluster as its designated class

  * •

This is the model. Prediction is simply a mapping from the predicted cluster
of an example to the designated class of that cluster

  * •

F1 (macro averaged) will give a measure of how well the model performs

####  4.2.2 Logistic Regression

For establishing a baseline on the classifier models, we use logistic
regression on the embedding data to train a linear model.

##  5 Experiments

###  5.1 Cluster Classifier

The Cluster Classifier was able to retrieve some of the topic classes. Table 1
shows that the F1 scores of the Cluster Classifier are not great, but they are
also not random.

Most of the classes ended up being mapped to by one (or more!) of the
clusters. The cluster classifier also mixed up some classes and missed out on
some other ones altogether.

The full ranking for Cluster Classifier is:

  1. 1.

USE

  2. 2.

DeCLUTR

  3. 3.

SBERT

  4. 4.

LASER

  5. 5.

TfidfVectorizer

###  5.2 Logistic Regression Classifier

The Logistic Regression Classifier did significantly better than the Cluster
Classifier. This is not surprising as it is a supervised model that has been
given the classes in advance.

It is interesting to note that TfidfVectorizer performed reasonably well
(better than SBERT) using this classifier. The full ranking for Logistic
Regression is:

  1. 1.

USE

  2. 2.

DeCLUTR

  3. 3.

TfidfVectorizer

  4. 4.

SBERT

  5. 5.

LASER

####  5.2.1 SentEval

If we count the number of tasks on SentEval for which each model has the
highest score (see Table 2, we end up with the following list:

  1. 1.

DeCLUTR (10 wins)

  2. 2.

SBERT (8 wins)

  3. 3.

LASER (6 wins)

  4. 4.

USE (2 wins)

##  6 Analysis

The embedding models that were trained with STS tasks in mind (SBERT, USE, and
DeCLUTR) scored better than the ones that didn’t (LASER and TfidfVectorizer).
This was to be expected.

But there are also unexpected results. Universal Sentence Encoder performs the
worst on SentEval, yet it manages to obtain by far the best scores in both
Cluster Classifier and Logistic Regression. It is unclear where this
discrepancy comes from. Perhaps it is related to the fact that USE uses
angular distance rather than cosine similarity. But it is difficult to dig
deeper as the paper is light on details about the training data and
procedures. At the very least we can conclude from this that great performance
on STS tasks does not automatically lead to great performance on clustering
tasks.

Over all it is encouraging that Cluster Classifier even partially works. The
datasets used in this experiment are real life datasets. The topic classes
used in them are not created artificially with academic purposes in mind, but
have evolved out of practical considerations in real world scenario’s. Yet the
various sentence embedding models have managed to find a partial overlap.

##  7 Conclusion

Sentence embeddings can be clustered. In real life datasets, the resulting
clusters form at least some overlap with topic classes.

This is interesting because clustering is an unsupervised analysis technique.
It means that sentence embedding clusters can be used for setting up
unsupervised text classification, which is a major task in real world
applications such as for example customer feedback analysis.

## References

  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the Association for Computational Linguistics_ , 7:597–610.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018.  Universal sentence encoder for English.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 169–174, Brussels, Belgium. Association for Computational Linguistics.
  * Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018.  Senteval: An evaluation toolkit for universal sentence representations.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  * Giorgi et al. (2020) John M. Giorgi, Osvald Nitski, Gary D. Bader, and Bo Wang. 2020.  Declutr: Deep contrastive learning for unsupervised textual representations.
  * Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015.  Skip-thought vectors.
  * Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018.  An efficient framework for learning sentence representations.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems (NIPS)_ , pages 3111––3119.
  * Misra (2018) Rishabh Misra. 2018.  News category dataset.
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 188–197, Hong Kong, China. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.

##  Appendix A Supplemental Material

| Result type | SBERT | DeCLUTR | USE | LASER
---|---|---|---|---|---
STS12 | spearman | 0.695 | 0.635 | 0.656 | 0.623
STS13 | spearman | 0.737 | 0.726 | 0.680 | 0.516
STS14 | spearman | 0.763 | 0.717 | 0.715 | 0.670
STS15 | spearman | 0.821 | 0.799 | 0.808 | 0.754
STS16 | spearman | 0.807 | 0.796 | 0.787 | 0.723
MR | accuracy | 79.830 | 84.990 | 75.150 | 74.080
CR | accuracy | 86.780 | 90.010 | 81.780 | 80.530
MPQA | accuracy | 86.630 | 88.330 | 87.150 | 88.210
SUBJ | accuracy | 92.680 | 95.240 | 91.800 | 91.480
SST2 | accuracy | 84.130 | 89.840 | 80.510 | 79.850
SST5 | accuracy | 45.930 | 48.550 | 42.810 | 44.250
TREC | accuracy | 89.200 | 91.800 | 92.200 | 89.200
MRPC | accuracy | 73.860 | 74.030 | 69.620 | 75.190
SICKEntailment | accuracy | 81.290 | 82.100 | 82.360 | 80.980
SICKRelatedness | spearman | 0.802 | 0.786 | 0.789 | 0.790
STSBenchmark | spearman | 0.819 | 0.794 | 0.791 | 0.778
Length | accuracy | 64.430 | 82.360 | 64.910 | 79.660
WordContent | accuracy | 73.970 | 62.150 | 70.130 | 40.380
Depth | accuracy | 31.440 | 34.430 | 27.430 | 39.390
TopConstituents | accuracy | 71.800 | 71.050 | 62.840 | 78.490
BigramShift | accuracy | 76.380 | 87.630 | 59.930 | 67.590
Tense | accuracy | 87.000 | 88.400 | 80.080 | 87.290
SubjNumber | accuracy | 83.320 | 85.800 | 74.550 | 90.220
ObjNumber | accuracy | 81.810 | 83.130 | 72.550 | 88.740
OddManOut | accuracy | 56.500 | 64.210 | 54.030 | 50.790
CoordinationInversion | accuracy | 57.080 | 66.290 | 54.300 | 67.840
Table 2: SentEval scores for SBERT, DeCLUTR, USE, and LASER. SBERT scores very well on STS tasks. LASER scores well on probing tasks (linguistic properties of the input). Bold is best of row. DeCLUTR scores best over all. | Clusters F1 | Clusters Acc | LogReg F1 | LogReg Acc
---|---|---|---|---
SBERT | 0.225 | 0.263 | 0.586 | 0.586
USE | 0.321 | 0.345 | 0.618 | 0.618
DeCLUTR | 0.239 | 0.276 | 0.613 | 0.612
LASER | 0.087 | 0.113 | 0.438 | 0.446
TfidfVectorizer | 0.088 | 0.107 | 0.605 | 0.604
Table 3: F1 macro and accuracy scores for SBERT, DeCLUTR, USE, and LASER on the Amazon Reviews dataset. Bold is best of column. | Clusters F1 | Clusters Acc | LogReg F1 | LogReg Acc
---|---|---|---|---
SBERT | 0.212 | 0.247 | 0.472 | 0.475
USE | 0.256 | 0.286 | 0.511 | 0.515
DeCLUTR | 0.206 | 0.243 | 0.506 | 0.508
LASER | 0.112 | 0.138 | 0.398 | 0.402
TfidfVectorizer | 0.058 | 0.075 | 0.486 | 0.487
Table 4: F1 macro and accuracy scores for SBERT, DeCLUTR, USE, and LASER on
the News dataset. Bold is best (vertically)

  * •

Full results of the SentEval benchmark are available in Table 2.

  * •

Full results of the Cluster Classifier and Logistic Regression on the Amazon
Reviews dataset are available in Table 3.

  * •

Full results of the Cluster Classifier and Logistic Regression on the News
dataset are available in Table 4.