# Bayesian Attention Modules

David S. Hippocampus
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies.

###### Abstract

Attention modules, as simple and effective tools, have not only enabled deep
neural networks to achieve state-of-the-art results in many domains, but also
enhanced their interpretability. Most current models use deterministic
attention modules due to their simplicity and ease of optimization. Stochastic
counterparts, on the other hand, are less popular despite their potential
benefits. The main reason is that stochastic attention often introduces
optimization issues or requires significant model changes. In this paper, we
propose a scalable stochastic version of attention that is easy to implement
and optimize. We construct simplex-constrained attention distributions by
normalizing reparameterizable distributions, making the training process
differentiable. We learn their parameters in a Bayesian framework where a
data-dependent prior is introduced for regularization. We apply the proposed
stochastic attention modules to various attention-based models, with
applications to graph node classification, visual question answering, image
captioning, machine translation, and language understanding. Our experiments
show the proposed method brings consistent improvements over the corresponding
baselines.

##  1 Introduction

Attention modules, aggregating features with weights obtained by aligning
latent states, have become critical components for state-of-the-art neural
network models in various applications, such as natural language processing
[bahdanau2014neural, vaswani2017attention, devlin2018bert, lan2019albert],
computer vision [bello2019attention, ramachandran2019stand], graph analysis
[velivckovic2017graph, lee2019attention], and multi-modal learning
[xu2015show, rennie2017self, yu2019deep, wang2019reinforced]. They have been
proven to be effective in not only being combined with other types of neural
network components, such as recurrent [bahdanau2014neural, xu2015show] and
convolutional units [wang2018non, bello2019attention], but also being used to
build a stand-alone architecture [vaswani2017attention, ramachandran2019stand,
parmar2018image]. Besides boosting the performance, using them also often
helps aid model visualization and enhance interpretability
[bahdanau2014neural, xu2015show].

While the attention mechanism provides useful inductive bias, attention
weights are often treated as deterministic rather than random variables.
Consequently, the only source of randomness lies at the model output layer.
For example, in classification models, the randomness is in the final logistic
regression layer, while in discrete sequence generation, it is in the
conditional categorical output layer. However, a single stochastic output
layer is often insufficient in modeling complex dependencies
[chung2015recurrent]. The idea of augmenting deterministic neural networks
with latent random variables has achieved success in many fields to model
highly structured data, such as speeches [chung2015recurrent,
fraccaro2016sequential, bayer2014learning], natural language sequences
[bowman2015generating, he2018sequence, zhang2016variational], and images
[kingma2013auto, higgins2017beta, blundell2015weight]. Such modification may
not only boost the performance, but also provide better uncertainty estimation
[gal2016dropout, gal2017concrete].

As attention weights can be interpreted as alignment weights, it is intuitive
to connect the attention module with latent alignment models [xu2015show,
deng2018latent, lawson2018learning], where latent alignment variables are
stochastic and the objective becomes a lower bound of the log marginal
likelihood. Making the attention weights stochastic and learning the alignment
distribution in a probabilistic manner brings several potential advantages.
First, adding latent random variable enhances the model’s ability to capture
complicated dependencies in the target data distribution. Second, we are able
to adopt Bayesian inference, where we may build our prior knowledge into prior
regularization on the attention weights and utilize posterior inference to
provide a better basis for model analysis and uncertainty estimation
[deng2018latent, gal2017concrete].

Most current work on stochastic attention focus on hard attention [xu2015show,
shankar2018posterior, deng2018latent], where the attention weights are
discrete random variables sampled from categorical distributions. However,
standard backpropagation no longer applies to the training process of such
models and one often resorts to a REINFORCE gradient estimator
[williams1992simple], which has large variance. Such models generally
underperform their deterministic counterparts, with a few exceptions, where a
careful design of baselines and curriculum learning are required [xu2015show,
deng2018latent, lawson2018learning]. While attending to multiple positions at
one time is intuitively more preferable, probabilistic soft attention is less
explored. bahuleyan2017variational [bahuleyan2017variational] propose to use
the normal distribution to generate the attention weights, which are hence
possibly negative and do not sum to one. deng2018latent [deng2018latent]
consider sampling attention weights from the Dirichlet distribution, which is
not reparameterizable and hence not amenable to gradient descent based
optimization.

In this paper, we propose Bayesian attention modules where the attention
weights are treated as latent random variables, whose distribution parameters
are obtained by aligning keys and queries. We satisfy the simplex constraint
on the attention weights, by normalizing the random variables drawn from
either the Lognormal or Weibull distributions. Both distributions generate
non-negative random numbers that are reparameterizable. In this way, the whole
training process can be made differentiable via the reparameterization trick.
We introduce a contextual prior distribution whose parameters are functions of
keys to impose a Kullback–Leibler (KL) divergence based regularization. To
reduce the variance of gradient estimation, we pick the prior distribution
such that the KL term can be rewritten in a semi-analytic form, $i.e.$, an
expectation of analytic functions.

Compared to previous stochastic attentions, our method is much simpler to
implement, requires surprisingly few modifications to standard deterministic
attention, is stable to train, and maintains good scalability, thereby making
it attractive for large-scale deep learning applications. We evaluate the
proposed stochastic attention module on a broad range of tasks, including
graph node classification, visual question answering, image captioning,
machine translation, and language understanding, where attention plays an
important role. We show that the proposed method consistently outperforms
baseline attention modules and provides better uncertainty estimation.
Further, we conduct a number of ablation studies to reason the effectiveness
of the proposed model.

##  2 Preliminaries on attention modules

In this section, we briefly review the standard deterministic soft attention
modules that have been widely used in various neural networks.

Basic module: Consider $n$ key-value pairs, packed into a key matrix
$K\in\mathbb{R}^{n\times d_{k}}$ and a value matrix $V\in\mathbb{R}^{n\times
d_{v}}$, and $m$ queries packed into $Q\in\mathbb{R}^{m\times d_{k}}$, where
the dimensions of queries and keys are both equal to $d_{k}$. Depending on the
applications, key, value, and query may have different meanings. For example,
in self-attention layers [vaswani2017attention], key, value, and query are all
from the same source, $i.e.$, the output of the previous layer and in this
case $m$ equals to $n$. In encoder-decoder attention layers, the queries come
from the decoder layer, while the keys and values come from the output of the
encoder [vaswani2017attention, xu2015show, bahdanau2014neural]. When attention
is used for multi-modal cases, the queries often come from one modality while
the keys and values come from the other [yu2019deep].

Attention modules make use of keys and queries to obtain deterministic
attention weights $W$, which are used to aggregate values $V$ into output
features $O=WV\in\mathbb{R}^{m\times d_{v}}$. Specifically, $W$ is obtained
through a softmax function across the key dimension as
$W=\text{softmax}(f(Q,K))\in\mathbb{R}^{m\times n}$, so that it is a non-
negative matrix with each row summing to one. Thus if we denote $\Phi=f(Q,K)$,
then

$\displaystyle\textstyle
W_{i,j}=\exp(\Phi_{i,j})/\sum_{j=1}^{n}\exp(\Phi_{i,j}).$ (1)

Intuitively, the scale of element $W_{i,j}$ represents the importance of the
$j$th key to the $i$th query, and the neural network should learn $Q$ and $K$
such that $W$ gives higher weights to more important features. There are many
choices of the alignment score function $f$, including scaled dot-product
[vaswani2017attention, devlin2018bert], additive attention [xu2015show,
rennie2017self, bahdanau2014neural], and several other variations
[luong2015effective, graves2014neural, wang2018non].

Multi-head and multi-layer attention: Multi-head attention is proposed to
attend to information from different representation subspaces
[vaswani2017attention], where queries, keys, and values are linearly projected
$H$ times by $H$ different projection matrices, producing $H$ output values
that are concatenated as the final attention layer output. One may then stack
attention layers by placing one on top of another, leading to deep attention
modules [vaswani2017attention, yu2019deep]. For a deep attention module with
$L$ attention layers and $H$ heads for each layer, the output of the $l$th
layer would be $O^{l}=[W^{l,1}V^{l,1},...,W^{l,H}V^{l,H}],$ where
$W^{l,h}=\text{softmax}(f(Q^{l,h},K^{l,h}))$, $Q^{l,h}=Q^{l}M^{l,h}_{Q}$,
$K^{l,h}=K^{l}M^{l,h}_{K}$, and $V^{l,h}=V^{l}M^{l,h}_{V}$ for $h=1,...,H$,
and the $M$’s are parametric matrices that the neural network needs to learn.
Then the output of the $l$th attention layer, $O^{l}$, is fed into the next
attention layer (possibly after some transformations) and the queries, keys,
and values of the $(l+1)$th layer would be functions of $O^{l}$.

##  3 Bayesian attention modules: a general recipe for stochastic attention

We suggest a general recipe for stochastic attention: (1) treat attention
weights as data-dependent local random variables and learn their distributions
in a Bayesian framework, (2) use normalized reparametrizable distributions to
construct attention distributions over simplex, and (3) use a key-based
contextual prior as regularization.

###  3.1 Learning attention distributions in a Bayesian way

Consider a supervised learning problem with training data
$\mathcal{D}:=\\{\boldsymbol{x}_{i},\boldsymbol{y}_{i}\\}_{i=1}^{N}$, where we
model the conditional probability
$p_{\boldsymbol{\theta}}(\boldsymbol{y}_{i}\,|\,\boldsymbol{x}_{i})$ using a
neural network parameterized by $\boldsymbol{\theta}$, which includes the
attention projections $M$’s. For notational convenience, below we drop the
data index $i$. Using vanilla attention modules, the mapping from
$\boldsymbol{x}$ to the likelihood
$p_{\boldsymbol{\theta}}(\boldsymbol{y}|\boldsymbol{x})$ is deterministic, so
the whole model is differentiable meaning that it is tractable to directly
maximize the likelihood.

Now, we turn the mapping from queries and keys to attention weights $W$
stochastic. Instead of using deterministic weights obtained from queries and
keys to aggregate values, we treat $W=\\{W^{l,h}\\}_{l=1:L,h=1:H}$ as a set of
data-dependent local latent variables sampled from $q_{\boldsymbol{\phi}}$,
which can be parameterized by some functions of queries and keys. Intuitively,
we argue that this distribution can be viewed as a variational distribution
approximating the posterior of local attention weights $W$, under a Bayesian
model, given the data $\boldsymbol{x},\boldsymbol{y}$. Therefore, we can learn
$q_{\boldsymbol{\phi}}$ with amortized variational inference [kingma2013auto].
Note that, unlike deng2018latent and lawson2018learning, we do not enforce
$q_{\boldsymbol{\phi}}$ to be dependent on $\boldsymbol{y}$, which might not
be available during testing. Instead, we use the queries and keys in standard
attention modules to construct $q_{\boldsymbol{\phi}}$, so
$q_{\boldsymbol{\phi}}$ depends on $\boldsymbol{x}$ only or both
$\boldsymbol{x}$ and the part of $\boldsymbol{y}$ that has already been
observed or generated by the model. For example, in visual question answering
or graph node classification, $q_{\boldsymbol{\phi}}$ only depends on input
$\boldsymbol{x}$. While in sequence generation, like image captioning or
machine translation, $q_{\boldsymbol{\phi}}$ could be dependent on
$\boldsymbol{y}$ as the queries come from $\boldsymbol{y}$.

Constructing variational distribution in such a way has several advantages.
First, as we will show in the next section, by utilizing keys and values,
transforming a deterministic attention weights into an attention distribution
becomes straightforward and requires minimal changes to standard attention
models. We can even easily adapt pretrained standard attention models for
variational finetuning (shown in Section 4.5). Otherwise, building an
efficient variational distribution often requires domain knowledge
[deng2018latent] and case by case consideration. Second, due to a similar
structure as standard attention modules, $q_{\boldsymbol{\phi}}$ introduces
little additional memory and computational cost, for which we provide a
complexity analysis in Section 3.4. Third, as keys and values are available
for both training and testing, we can use the variational distribution
$q_{\boldsymbol{\phi}}$ during testing. In contrast, previous works
[deng2018latent, lawson2018learning] enforce $q_{\boldsymbol{\phi}}$ to
include information not available during testing, restricting its usage at the
testing time. Further, as keys and queries depend on the realization of
attention weights in previous layers, this structure naturally allows cross-
layer dependency between attention weights in different layers so that it is
capable of modeling complex distributions.

Consider a Bayesian model, where we have prior $p_{{\boldsymbol{\eta}}}(W)$
and likelihood $p_{\boldsymbol{\theta}}(\boldsymbol{y}\,|\,\boldsymbol{x},W)$
that shares a common structure with vanilla deterministic soft attention. We
learn the distribution $q_{\boldsymbol{\phi}}$ by minimizing
$\mbox{KL}(q_{\boldsymbol{\phi}}(W)||p(W\,|\,\boldsymbol{x},\boldsymbol{y}))$,
the KL divergence from the posterior distribution of $W$ given
$\boldsymbol{x}$ and $\boldsymbol{y}$ to $q_{\boldsymbol{\phi}}$. With
amortized variational inference, it is equivalent to maximizing
$\mathcal{L_{\mathcal{D}}}=\sum_{(\boldsymbol{x},\boldsymbol{y})\in\mathcal{D}}\mathcal{L}({\boldsymbol{x},\boldsymbol{y}})$,
an evidence lower bound (ELBO) [hoffman2013stochastic, blei2017variational,
deng2018latent] of the intractable log marginal likelihood
$\sum_{(\boldsymbol{x},\boldsymbol{y})\in\mathcal{D}}\log
p(\boldsymbol{y}\,|\,\boldsymbol{x})=\sum_{(\boldsymbol{x},\boldsymbol{y})\in\mathcal{D}}\log\int
p_{\boldsymbol{\theta}}(\boldsymbol{y}\,|\,\boldsymbol{x},W)p_{{\boldsymbol{\eta}}}(W)dW$,
where

$\displaystyle\small\textstyle\mathcal{L}({\boldsymbol{x},\boldsymbol{y}}):=\mathbb{E}_{q_{\boldsymbol{\phi}}(W)}\left[\log
p_{\boldsymbol{\theta}}(\boldsymbol{y}\,|\,\boldsymbol{x},W)\right]-\mbox{KL}(q_{\boldsymbol{\phi}}(W)||p_{\boldsymbol{\eta}}(W))=\mathbb{E}_{q_{\boldsymbol{\phi}}(W)}\left[\log\frac{p_{\boldsymbol{\theta}}(\boldsymbol{y}\,|\,\boldsymbol{x},W)p_{\boldsymbol{\eta}}(W)}{q_{\boldsymbol{\phi}}(W)}\right].$

Learning attention distribution $q_{\boldsymbol{\phi}}$ via amortized
variational inference provides a natural regularization for
$q_{\boldsymbol{\phi}}$ from prior $p_{{\boldsymbol{\eta}}}$, where we can
inject our prior beliefs on attention distributions. We will show we can
parameterize the prior distribution with keys, so that the prior distribution
can be data-dependent and encode the importance information of each keys.
Meanwhile, we can update $\boldsymbol{\theta}$ and ${\boldsymbol{\eta}}$ to
maximize the ELBO. As $q_{\boldsymbol{\phi}}$ becomes closer to the posterior,
the ELBO becomes a tighter lower bound.

###  3.2 Reparameterizable attention distributions

A challenge of using stochastic attention weights is to optimize their
distribution parameters. Existing methods [xu2015show, shankar2018posterior,
deng2018latent] construct attention distributions in a way that standard
backpropagation based training no longer applies. Without carefully
customizing a training procedure for each specific task, it is generally hard
to learn such distributions. Below we introduce reparameterizable soft
stochastic attentions that allow effectively optimizing the distribution
parameters in a simple and general way.

Our goal is to construct a reparameterizable attention distribution
$q_{\boldsymbol{\phi}}$ over the simplex, $i.e.$, $W^{l,h}_{i,j}\geq 0$ and
$\sum_{j}W^{l,h}_{i,j}=1$. While the Dirichlet distribution, satisfying the
simplex constraint and encouraging sparsity, appears to be a natural choice,
it is not reparameterizable and hence not amenable to gradient descent based
optimization. Here, we consider satisfying the simplex constraint by
normalizing random variables drawn from non-negative reparameterizable
distributions. In particular, we consider the Weibull and Lognormal
distributions. We choose them mainly because they both lead to optimization
objectives that are simple to optimize, as described below.

Weibull distribution: The Weibull distribution
$S\sim\text{Weibull}(k,\lambda)$ has probability density function (PDF)
$p(S\,|\,k,\lambda)=\frac{k}{\lambda^{k}}S^{k-1}e^{-(S/\lambda)^{k}}$, where
$S\in\mathbb{R}_{+}$. Its expectation is $\lambda\Gamma(1+1/k)$ and variance
is
$\lambda^{2}\big{[}\Gamma\left(1+{2}/{k}\right)-\left(\Gamma\left(1+1/k\right)\right)^{2}\big{]}$.
It is reparameterizable as drawing $S\sim\text{Weibull}(k,\lambda)$ is
equivalent to letting
$S=\tilde{g}(\epsilon):=\lambda(-\log(1-\epsilon))^{1/k},~{}\epsilon\sim\text{Uniform}(0,1)$.
It resembles the gamma distribution, and with $\gamma$ denoted as the
Euler–Mascheroni constant, the KL divergence from the gamma to Weibull
distributions has an analytic expression [zhang2018whai] as

$\small\textstyle\mbox{KL}(\text{Weibull}(k,\lambda)||\text{Gamma}(\alpha,\beta))=\frac{\gamma\alpha}{k}-\alpha\log\lambda+\log
k+\beta\lambda\Gamma(1+\frac{1}{k})-\gamma-1-\alpha\log\beta+\log\Gamma(\alpha).$

Lognormal distribution: The Lognormal distribution
$S\sim\text{Lognormal}(\mu,\sigma^{2})$ has PDF
$p(S\,|\,\mu,\sigma)=\frac{1}{S\sigma\sqrt{2\pi}}\exp\left[-\frac{(\log
S-\mu)^{2}}{2\sigma^{2}}\right]$, where $S\in\mathbb{R}_{+}$. Its expectation
is $\exp(\mu+{{\sigma^{2}}/{2}})$ and variance is
$[\exp(\sigma^{2})-1]\exp(2\mu+\sigma^{2})$. It is also reparameterizable as
drawing $S\sim\text{Lognormal}(\mu,\sigma^{2})$ is equivalent to letting
$S=\tilde{g}(\epsilon)=\exp(\epsilon\sigma+\mu),~{}\epsilon\sim\mathcal{N}(0,1)$.
The KL divergence is analytic as

$\textstyle\mbox{KL}(\text{Lognormal}(\mu_{1},\sigma^{2}_{1})||\text{Lognormal}(\mu_{2},\sigma^{2}_{2}))=\log\frac{\sigma_{2}}{\sigma_{1}}+\frac{\sigma_{1}^{2}+(\mu_{1}-\mu_{2})^{2}}{2\sigma^{2}_{2}}-0.5.$

Sampling $S_{i,j}^{l,h}$ from either the Weibull or Lognormal distribution, we
obtain the simplex-constrained random attention weights $W$ by applying a
normalization function $\bar{g}$ over $S$ as
$W_{i}^{l,h}=\bar{g}(S_{i}^{l,h}):=S_{i}^{l,h}/\sum_{j}S_{i,j}^{l,h}$. Note
$W$ is reparameterizable but often does not have an analytic PDF.

Parameterizing variational attention distributions: To change the
deterministic mapping from the queries and keys to attention weights to a
stochastic one, we use queries and keys to obtain the distribution parameters
of unnormalized attention weights $S$, which further define the variational
distribution of the normalized attention weights $W$.

[color=blue!10]mz: We let
$k_{i,j}^{l,h}=\exp(-\Phi_{i,j}^{l,h})+\exp(\Phi_{i,j}^{l,h})$ and
$\lambda_{i,j}^{l,h}=\exp(\Phi_{i,j}^{l,h})/\Gamma(1+\frac{1}{\exp(-\Phi_{i,j}^{l,h})+\exp(\Phi_{i,j}^{l,h})})$,
which makes $\mathbb{E}[S_{i,j}^{l,h}]=\exp(\Phi_{i,j}^{l,h})$ and
$\mbox{Var}[S_{i,j}^{l,h}]=\exp(2\Phi_{i,j}^{l,h})[\frac{\Gamma(1+\frac{2}{\exp(-\Phi_{i,j}^{l,h})+\exp(\Phi_{i,j}^{l,h})})}{\Gamma^{2}(1+\frac{1}{\exp(-\Phi_{i,j}^{l,h})+\exp(\Phi_{i,j}^{l,h})})}-1]$.
This construction has several advantages: when $\Phi\rightarrow\pm\infty$,
$\mbox{Var}[S]/\mathbb{E}[S]\rightarrow 0$;

[color=blue!10]mz: We let
$k_{i,j}^{l,h}=2^{-52}\exp(-\Phi_{i,j}^{l,h})+\exp(\Phi_{i,j}^{l,h})$ and
$\lambda_{i,j}^{l,h}=\exp(\Phi_{i,j}^{l,h})/\Gamma(1+\frac{1}{2^{-52}\exp(-\Phi_{i,j}^{l,h})+\exp(\Phi_{i,j}^{l,h})})$,
which makes $\mathbb{E}[S_{i,j}^{l,h}]=\exp(\Phi_{i,j}^{l,h})$ and
$\mbox{Var}[S_{i,j}^{l,h}]=\exp(2\Phi_{i,j}^{l,h})[\frac{\Gamma(1+\frac{2}{\exp(\Phi_{i,j}^{l,h})})}{\Gamma^{2}(1+\frac{1}{\exp(\Phi_{i,j}^{l,h})})}-1]$.
This construction has several advantages: when $\Phi\rightarrow\infty$,
$\mbox{Var}[S]/\mathbb{E}[S]\rightarrow 0$, but when $\Phi\rightarrow-\infty$,
$\mbox{Var}[S]/\mathbb{E}[S]$ quickly explodes; The variance to mean ratio
appears to be a monotonically decreasing function of $\phi$. By comparing the
PDFs, we also find that Weibull$(k,\lambda)$ under this parameterization
already approximate $\mbox{Gamma}(e^{\phi},1)$ reasonably very well when
$\phi$ is not too far from zero (they are identical when $\phi=0$), and we can
do a better job by letting $k=ae^{\phi}+b$, where $a$ and $b$ need to be
adjusted.

[color=blue!10]mz: We let $k=f_{k}(\Phi)=\ln(1+\exp(\omega_{k}\Phi+b_{k}))$
and $\lambda=f_{\lambda}(\Phi)=\ln(1+\exp(\omega_{\lambda}\Phi+b_{\lambda}))$

We let in the prior that
$S\sim\mbox{Gamma}(e^{\Phi},\beta)\in\mathbb{R}_{+}^{m\times n}$, which means
we have $W(i,:)=\frac{S(i,:)}{|S(i,:)|}\sim\mbox{Dir}(e^{\Phi(i,:)})$. The
posterior of $S$ or $W$ is intractable, and resort to amortized variational
inference with reparameterizable variational distribution. In particular, we
let

$Q(S)=\mbox{Weibull}({{\bf
K}}+e^{\Phi},\boldsymbol{\Lambda})~{}~{}~{}or~{}~{}~{}Q(S)=\mbox{Weibull}({{\bf
K}},\boldsymbol{\Lambda})$

where

${{\bf K}}=\ln(1+\exp({\boldsymbol{\Omega}}_{1}\Phi+{{\bf
B}}_{1}))\in\mathbb{R}_{+}^{m\times
n},~{}{\boldsymbol{\Omega}}_{1}\in\mathbb{R}^{m\times m},~{}{{\bf
B}}_{1}\in\mathbb{R}^{m\times n}$
$\boldsymbol{\Lambda}=\ln(1+\exp({\boldsymbol{\Omega}}_{2}\Phi+{{\bf
B}}_{2}))\in\mathbb{R}_{+}^{m\times
n},~{}{\boldsymbol{\Omega}}_{2}\in\mathbb{R}^{m\times m},~{}{{\bf
B}}_{2}\in\mathbb{R}^{m\times n}$

We let in the prior that
$S\sim\mbox{Gamma}(e^{\Phi},\beta)\in\mathbb{R}_{+}^{m\times n}$, which means
we have $W(i,:)=\frac{S(i,:)}{|S(i,:)|}\sim\mbox{Dir}(e^{\Phi(i,:)})$. We let

$Q(S)=\prod_{i,j}\mbox{Weibull}(S_{i,j};k_{ij},\lambda_{ij})~{}~{}~{}~{}or~{}~{}~{}~{}Q(S)=\prod_{i,j}\mbox{Weibull}(S_{i,j};k_{ij}+e^{\alpha_{1}}e^{\phi_{i}}+e^{\alpha_{2}}e^{-\phi_{i}}+e^{\alpha_{3}},\lambda_{ij})$

where

$k_{ij}=\ln(1+\exp(w_{1}h_{ij}+b_{1})),~{}h_{ij}=\ln(1+\exp(w_{2}\phi_{ij}+b_{2}))$
$\lambda_{ij}=\beta^{-1}\ln(1+\exp(w_{3}h_{ij}+b_{3})),~{}h_{ij}=\ln(1+\exp(w_{4}\phi_{ij}+b_{4}))$
$or~{}~{}\lambda_{ij}=\frac{e^{\phi_{ij}}}{\beta\Gamma(1+1/k_{ij})}$

We want use $\mbox{Weibull}(f_{k}(\phi),f_{\lambda}(\phi))$ to approximate
$\mbox{Gamma}(e^{\phi},\beta)$

With the Weibull distribution, we treat $k$ as a global hyperparameter and let
$\lambda^{l,h}_{i,j}=\exp(\Phi_{i,j}^{l,h})/\Gamma(1+1/k)$, and like before
$\Phi^{l,h}=f(Q^{l,h},K^{l,h})$. Then, we sample
$S_{i,j}^{l,h}\sim\mbox{Weibull}(k,\lambda^{l,h}_{i,j})$, which is the same as
letting
$S_{i,j}^{l,h}=\exp(\Phi_{i,j}^{l,h})\frac{(-\log(1-\epsilon_{i,j}^{h,l}))^{1/k}}{\Gamma(1+1/k)},~{}\epsilon_{i,j}^{h,l}\sim\mbox{Uniform}(0,1)$
. With the Lognormal distribution, we treat $\sigma$ as a global
hyperparameter and let $\mu_{i,j}^{l,h}=\Phi_{i,j}^{l,h}-\sigma^{2}/2$. Then,
we sample $S_{i,j}^{l,h}\sim\mbox{Lognormal}(\mu_{i,j}^{l,h},\sigma^{2})$,
which is the same as letting
$S_{i,j}^{l,h}=\exp(\Phi_{i,j}^{l,h})\exp(\epsilon_{i,j}^{h,l}\sigma-\sigma^{2}/2),~{}\epsilon_{i,j}^{h,l}\sim\mathcal{N}(0,1)$.
Note our parameterizations ensure that
$\mathbb{E}[S_{i,j}^{l,h}]=\exp(\Phi_{i,j}^{l,h})$. Therefore, if, instead of
sampling $S_{i,j}^{l,h}$ from either distribution, we use its expectation as a
substitute, then the mapping becomes equivalent to that of vanilla soft
attention, whose weights are defined as in (1). In other words, if we let $k$
of the Weibull distribution go to infinity, or $\sigma$ of the Lognormal
distribution go to zero, which means the variance of $S_{i,j}^{l,h}$ goes to
zero and the distribution becomes a point mass concentrated at the
expectation, then the proposed stochastic soft attention reduces to
deterministic soft attention. Therefore, the proposed stochastic soft
attention can be viewed as a generalization of vanilla deterministic soft
attention.

We have now constructed $q_{\boldsymbol{\phi}}$ to be a reparameterizable
distribution
$W=g_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}):=\bar{g}(\tilde{g}_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}))$,
where $\boldsymbol{\epsilon}$ is a collection of $i.i.d.$ random noises with
the same size as $W$. To estimate the gradient of the ELBO, however, we need
either the analytic forms of both $p_{\boldsymbol{\eta}}$ and
$q_{\boldsymbol{\phi}}$, or the analytic form of the KL term, neither of which
are available. In the next section, we show how to work around this issue by
imposing the KL regularization for latent variable $S$ before normalization
and decomposing the joint distribution into a sequence of conditionals.

With the Weibull distribution, we treat
$k^{l,h}_{i,j}=\exp(\Phi_{1}{}_{i,j}^{l,h})+a$ as a global hyperparameter and
let $\lambda^{l,h}_{i,j}=\exp(\Phi_{2}{}_{i,j}^{l,h})/\Gamma(1+1/k)$, and like
before $\Phi_{1}^{l,h}=f_{1}(Q^{l,h},K^{l,h})$ and
$\Phi_{2}^{l,h}=f_{2}(Q^{l,h},K^{l,h})$. Then, we sample
$S_{i,j}^{l,h}\sim\mbox{Weibull}(k,\lambda^{l,h}_{i,j})$, which is the same as
letting
$S_{i,j}^{l,h}=\exp(\Phi_{2}{}_{i,j}^{l,h})\frac{(-\log(1-\epsilon_{i,j}^{h,l}))^{1/k}}{\Gamma(1+1/k)},~{}\epsilon_{i,j}^{h,l}\sim\mbox{Uniform}(b,1-b)$.
With this $\epsilon_{i,j}^{h,l}$, the original Weibull probability density
function (PDF)
$p(S\,|\,k,\lambda)=\frac{k}{\lambda^{k}}S^{k-1}e^{-(S/\lambda)^{k}}$ has
become as
$p(S\,|\,k,\lambda)=\frac{1}{1-2b}\frac{k}{\lambda^{k}}S^{k-1}e^{-(S/\lambda)^{k}}$

Flowing the attention weight into the $k$, the expression of k is
$k^{l,h}_{i,j}=\exp(\Phi_{1}{}_{i,j}^{l,h})+a+S_{i,j}^{l,h}$

Flowing the $k$, the expression of $k$ is
$k^{l,h}_{i,j}=\exp(\Phi_{1}{}_{i,j}^{l,h})+a+k^{l-1,h}_{i,j}$

Flowing the $\lambda$, the expression of $\lambda$ is
$\lambda^{l,h}_{i,j}=\exp(\Phi_{2}{}_{i,j}^{l,h})/\Gamma(1+1/k)S_{i,j}^{l,h}+\lambda^{l-1,h}_{i,j}$

###  3.3 Contextual prior: key-dependent Bayesian regularization

In the ELBO objective, there is a built-in regularization term, $i.e.$, the KL
divergence from the prior distribution $p_{\boldsymbol{\eta}}$ to variational
distribution $q_{\boldsymbol{\phi}}$. To estimate the gradients, we need to
evaluate $q_{\boldsymbol{\phi}}(W)$ and $p_{\boldsymbol{\eta}}(W)$ for given
attention weights $W$. We note that even though the analytic form of
$q_{\boldsymbol{\phi}}(W)$ is not available,
$q_{\boldsymbol{\phi}}(S)=\prod_{l=1}^{L}q_{\boldsymbol{\phi}}(S_{l}\,|\,S_{1:l-1})$
is a product of analytic PDFs (Weibull or Lognormal) for unnormalized weights
$S$, so we rewrite the ELBO in terms of $S$ (we keep using
$q_{\boldsymbol{\phi}},p_{{\boldsymbol{\eta}}}$ for $S$ as the distribution of
$W$ is defined by $S$),

$\displaystyle\textstyle\mathcal{L}(\boldsymbol{x},\boldsymbol{y}):=\mathbb{E}_{q_{\boldsymbol{\phi}}(S)}\left[\log
p_{\boldsymbol{\theta}}(\boldsymbol{y}\,|\,\boldsymbol{x},S)\right]-\mbox{KL}(q_{\boldsymbol{\phi}}(S)||p_{\boldsymbol{\eta}}(S))$
(2)

Note the KL divergence, as shown in Section 3.2, can be made analytic and
hence it is natural to use either the gamma or Lognormal distribution to
construct $p_{\boldsymbol{\eta}}(S)$. Regardless of whether the gamma or
Lognormal is used to construct $p_{\boldsymbol{\eta}}(S)$, due to the
dependencies between different stochastic attention layers, we do not have
analytic expressions for
$\mbox{KL}(q_{\boldsymbol{\phi}}(S)||p_{\boldsymbol{\eta}}(S))$. Fortunately,
as shown in Lemma 1, by decomposing the joint into a sequence of conditionals
and exploiting these analytic KL divergence expressions, we can express each
KL term in a semi-analytic form. The proof is deferred to Appendix A.
According to the Rao-Blackwellization theorem [mcbook], we can reduce the
Monte Carlo estimation variance by plugging in the analytic part.

######  Lemma 1.

The KL divergence from the prior to variational distributions is semi-analytic
as

$\displaystyle\emph{\mbox{KL}}(q_{\boldsymbol{\phi}}(S)||p_{\boldsymbol{\eta}}(S))=\sum\nolimits_{l=1}^{L}\mathbb{E}_{q_{\boldsymbol{\phi}}(S_{1:l-1})}\small\underbrace{\emph{\mbox{KL}}(q_{\boldsymbol{\phi}}(S_{l}|S_{1:l-1})||p_{\boldsymbol{\eta}}(S_{l}|S_{1:l-1}))}_{\text{analytic}}$
(3)

Figure 1: Prior and posterior attention weights visualization for image
captioning.

Key-based contextual prior: Instead of treating the prior as a fixed
distribution independent of the input $\boldsymbol{x}$, here we make prior
depend on the input through keys. The motivation comes from our application in
image captioning. Intuitively, given an image (keys), there should be a global
prior attention distribution over the image, indicating the importance of each
part of the image even before the caption generation process. Based on the
prior distribution, the attention distribution can be updated locally using
the current state of generation (queries) at the each step (see Figure 1).
This intuition can be extended to the general attention framework, where the
prior distribution encodes the global importance of each keys shared by all
queries, while the posterior encodes the local importance of each keys for
each query. To obtain the prior parameters, we take a nonlinear transformation
of key features, followed by a softmax to obtain positive values and enable
interactions between keys. Formally, let
$\Psi^{l,h}=\text{softmax}(F_{2}(F_{NL}(F_{1}(K^{l,h}))))\in\mathbb{R}^{n\times
1}$, where $F_{1}$ is linear mapping from $\mathbb{R}^{d_{k}}$ to a hidden
dimension $\mathbb{R}^{d_{\text{mid}}}$, $F_{2}$ is linear mapping from
$\mathbb{R}^{d_{\text{mid}}}$ to $\mathbb{R}$, and $F_{NL}$ denotes a
nonlinear activation function, such as ReLU [nair2010rectified]. With the
gamma prior, we treat $\beta$ as a hyperparameter and let
$\alpha^{l,h}_{i,j}=\Psi_{i,1}^{l,h}$. With the Lognormal, we treat $\sigma$
as a hyperparameter and let $\mu_{i,j}^{l,h}=\Psi_{i,1}^{l,h}$. Following
previous work [bowman2016generating], we add a weight $\lambda$ to the KL term
and anneal it from a small value to one.

###  3.4 Putting it all together

Combining (2) and (3) and using reparameterization, we have
$\mathcal{L}_{\lambda}(\boldsymbol{x},\boldsymbol{y})=\mathbb{E}_{\boldsymbol{\epsilon}}[\mathcal{L}_{\lambda}(\boldsymbol{x},\boldsymbol{y},\boldsymbol{\epsilon})]$,
where

$\displaystyle\mathcal{L}_{\lambda}(\boldsymbol{x},\boldsymbol{y},\boldsymbol{\epsilon})=\log
p_{\boldsymbol{\theta}}(\boldsymbol{y}\,|\,\boldsymbol{x},\tilde{g}_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}))-{\lambda}\sum\nolimits_{l=1}^{L}\small\underbrace{\mbox{KL}(q_{\boldsymbol{\phi}}(S_{l}\,|\,\tilde{g}_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}_{1:l-1}))||p_{\boldsymbol{\eta}}(S_{l}\,|\,\tilde{g}_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}_{1:l-1})))}_{\textit{analytic}}.$
(4)

To estimate the gradient of
$\mathcal{L}_{\lambda}(\boldsymbol{x},\boldsymbol{y})$ with respect to
$\boldsymbol{\phi},\boldsymbol{\theta},{\boldsymbol{\eta}}$, we compute the
gradient of
$\mathcal{L}_{\lambda}(\boldsymbol{x},\boldsymbol{y},\boldsymbol{\epsilon})$,
which is a Monte Carlo estimator with one sample of $\boldsymbol{\epsilon}$.
This way provides unbiased and low-variance gradient estimates (see the pseudo
code in Algorithm 1 in Appendix).

At the testing stage, to obtain a point estimator, we adopt the common
practice of approximating the posterior means of prediction probabilities by
substituting the latent variables by their posterior expectations
[srivastava2014dropout]. To calibrate estimation uncertainties, we draw
multiple posterior samples, each of which produces one posterior prediction
probability sample.

Complexity analysis: Our framework is computationally and memory efficient due
to parameter sharing between the variational, prior, and likelihood networks.
Extra memory cost comes from the contextual prior network which, for a single
layer and single head attention, is of scale $O(d_{k}d_{\text{mid}})$. This is
insignificant compared to the memory scale of $M^{\prime}s$,
$O(d_{k}d_{v}+d_{v}d_{v})$, as $d_{\text{mid}}$ is as small as $10\ll d_{v}$.
Meanwhile, the additional computations involve the sampling process and
computing the KL term which is of scale $O(mn)$. Computing the contextual
prior is of scale $O(nd_{k}d_{\text{mid}})$. All above is inconsiderable
compared to the computational scale of deterministic attentions,
$O(mnd_{k}d_{v})$.

##  4 Experiments

Our method can be straightforwardly implemented in any attention based models.
To test the general applicability of our method, we conduct experiments on a
wide range of tasks where attention is essential, covering graphs (node
classification), multi-modal domains (visual question answering, image
captioning), and natural language processing (machine translation, language
understanding). A variety of attention types appear in these domains,
including self, encoder-decoder, and guided attentions. In this section, we
summarize the main experimental settings and results, and include the details
in Appendix C. All experiments are conducted on a single Nvidia Tesla V100
GPU.

###  4.1 Attention in graph neural networks

We first adapt our method to graph attention networks (GAT)
[velivckovic2017graph], which leverages deterministic self-attention layers to
process node-features for graph node classification. The graph structure is
encoded in the attention masks in a way that nodes can only attend to their
neighborhoods’ features in the graph. GAT is computationally efficient,
capable of processing graphs of different sizes, and achieves state-of-the-art
results on benchmark graphs. We use the same model and experimental setup as
in GAT [velivckovic2017graph], as summarized in Appendix C.1. We experiment
with three benchmark graphs, including Cora, Citeseer, and Pubmed, for node
classification in a transductive setting, meaning that training and testing
are performed on different nodes of the same graph [yang2016revisiting]. We
include a summary of these datasets in Table 6 in Appendix. For large and
sparse graph datasets like Pubmed, following GAT [velivckovic2017graph], we
implement a sparse version of the proposed method, where sparse tensor
operations are leveraged to limit the memory complexity to be linear in the
number of edges.

Results. The results are summarized in Table 1. We report the results of soft
attention (GAT), and $4$

Table 1: Classification accuracy for graphs.

Attention | Cora | Citeseer | PubMed
---|---|---|---
GAT | 83.00 | 72.50 | 77.26
BAM-LF | 83.24 | 72.86 | 78.30
BAM-LC | 83.34 | 73.04 | 78.76
BAM-WF | 83.48 | 73.18 | 78.50
BAM-WC | 83.81$\pm 0.3$ | 73.52$\pm 0.4$ | 78.82$\pm 0.3$

versions of Bayesian Attention Modules (BAM): Lognormal with fixed prior (LF),
Lognormal with contextual prior (LC), Weibull and fixed prior (WF), and
Weibull and contextual prior (WC). We report the mean classification
accuracies on test nodes over $5$ random runs, and the standard deviations of
BAM-WC. Note the results of GAT are reproduced by running the code provided by
the authors (https://github.com/PetarV-/GAT). Our results demonstrate that
adapting the deterministic soft attention module in GAT to Bayesian attention
significantly improves the performance. Weibull distribution performs better
that Lognormal, and contextual prior outperforms fixed prior.

###  4.2 Attention in visual question answering models

We consider a multi-modal learning task, visual question answering (VQA),
where a model predicts an answer to a question relevant to the content of a
given image. The recently proposed MCAN [yu2019deep] uses self-attention to
learn the fine-grained semantic meaning of both the image and question, and
guided-attention to learn the reasoning between these two modalities. We apply
BAM to MCAN and conduct experiments on the VQA-v2 dataset [goyal2017making],
consisting of human-annotated question-answer pairs for images from the MS-
COCO dataset [lin2014microsoft] (see detailed experiment settings in Appendix
C.2). To investigate the model’s robustness to noise, we also perturb the
input by adding Gaussian noise to the image features
[larochelle2007empirical]. For evaluation, we consider both accuracy and
uncertainty, which is necessary here as some questions are so challenging that
even human annotators might have different answers. We use a hypothesis
testing based Patch Accuracy vs Patch Uncertainty (PAvPU)
[mukhoti2018evaluating] to evaluate the quality of uncertainty estimation,
which reflects whether the model is uncertain about its mistakes. We defer the
details of this metric to Appendix C.2.1.

Table 2: Results of different attentions on VQA.

Metric | Accuracy |  | PAvPU
---|---|---|---
Data | Original | Noisy |  | Original | Noisy
Soft | 66.95 | 61.25 |  | 70.04 | 65.34
BAM-LF | 66.89 | 61.43 |  | 69.92 | 65.48
BAM-LC | 66.93 | 61.58 |  | 70.14 | 65.60
BAM-WF | 66.93 | 61.60 |  | 70.09 | 65.62
BAM-WC |  67.02 $\pm 0.04$ |  62.89 $\pm 0.06$ |  |  71.21 $\pm 0.06$ | 66.75 $\pm 0.08$

Results. In Table 2, we report the accuracy and uncertainty for both original
data and noisy data (see complete results in Table 7). In terms of accuracy,
BAM performs similarly as soft attention on the original dataset, but
outperforms it significantly on the more challenging noisy dataset, showing
that probabilistic soft attention is more robust to noise than deterministic
ones. For uncertainty, as soft attention is deterministic we use the dropout
in the model to obtain uncertainty, while for BAM we use both dropout and
probabilistic attention to obtain uncertainty. We observe that on both
original and noisy dataset, BAM has better uncertainty estimations, meaning it
is more uncertain on its mistakes and more certain on its correct predictions.
We note that we again observe the improvement from using contextual prior and
that Weibull also performs better than Lognormal.

###  4.3 Attention in image captioning models

We further experiment a multi-modal sequence generation task, image
captioning, where probabilistic attention (hard attention) was found to
outperform deterministic ones [xu2015show]. Image captioning models map an
image $\boldsymbol{x}$ to a sentence $\boldsymbol{y}=(y_{1},\ldots,y_{T})$
that summarizes the image information. Encoder-decoder attention is commonly
adopted in state-of-the-art models. During encoding, bottom-up bounding box
features [anderson2018bottom] are extracted from images by a pretrained Faster
R-CNN [ren2015faster]. At each step of decoding, a weighted sum of bounding
box features is injected into the hidden states of an LSTM-based RNN
[hochreiter1997long, mikolov2010recurrent] to generate words. The weights are
computed by aligning the bounding box features (keys) and hidden states from
the last step (queries). We conduct our experiments on MS-COCO
[lin2014microsoft], following the setup of luo2018discriminability. For the
model architecture, we employ an attention-based model (Att2in) of
rennie2017self, which we implement based on the code by
luo2018discriminability and replace the ResNet-encoded features by bounding
box features (see details in Appendix C.3). In all experiments, we use MLE
training alone without reinforcement learning based fine-tuning
[rennie2017self, ranzato2015sequence], which is beyond the scope of this paper
and we leave it as future work. We report four widely used automatic
evaluation metrics, BLEU, CIDEr, ROUGE, and METEOR.

Table 3: Comparing different attention modules on image captioning.

Attention | BLEU-4 | BLEU-3 | BLEU-2 | BLEU-1 | CIDEr | ROUGE | METEOR
---|---|---|---|---|---|---|---
Soft[xu2015show] | 24.3 | 34.4 | 49.2 | 70.7 | - | - | 23.9
Hard[xu2015show] | 25.0 | 35.7 | 50.4 | 71.8 | - | - | 23.0
Soft (Ours) | 32.2 | 43.6 | 58.3 | 74.9 | 104.0 | 54.7 | 26.1
Hard (Ours) | 26.5 | 37.2 | 51.9 | 69.8 | 84.4 | 50.7 | 23.3
Soft+Gaussian | - | - | - | - | - | - | -
BAM-LC | 32.7 | 44.0 | 58.7 | 75.1 | 105.0 | 54.8 | 26.3
BAM-WC | 32.8$\pm 0.1$ | 44.1$\pm 0.1$ | 58.8$\pm 0.1$ | 75.3$\pm 0.1$ |  104.5$\pm 0.1$ | 54.9$\pm 0.1$ |  26.2$\pm 0.1$

Results. We incorporate the results of both deterministic soft attention and
probabilistic hard attention from xu2015show. We also report those results
based on our implementation. Moreover, we experiment on modeling the attention
weights as Gaussian distribution directly as in bahuleyan2017variational
(Soft+Gaussian). Results in Table 3 show that the proposed probabilistic soft
attention module (BAM) consistently outperforms the deterministic soft
attention. In our implementation, we observe that it is difficult to make hard
attention work well due to the high variance of gradients. Also, our
experiment with Soft+Gaussian shows that naively modeling attention weights
with Gaussian distribution would easily lead to NAN results, as it allows the
attention weights to be negative and not sum to $1$. Therefore, it is
desirable to model attention weights simplex constrained distributions. We
also experiment with BAM where the KL term is completely sampled and observe
that the training becomes very unstable and often lead to NAN result.
Therefore, constructing prior and posterior in a way that the KL term is semi-
analytic does bring clear advantages to our model. In Figure 1, we visualize
the prior attention weights and posterior attention weights at each step of
generation, where we can visually see how the posterior attention adapts from
prior across steps. Further, we again observe that BAM-WC ourperforms BAM-LC
and hence for the following tasks, we focus on using BAM-WC.

###  4.4 Attention in neural machine translation

We also experiment with neural machine translation, where we compare with the
variational attention method proposed by deng2018latent. We follow them to set
the base model structure and experimental setting (see in Appendix C.4), and
adapt their deterministic attention to BAM-WC. We compare the BLEU score
[papineni2002bleu] of BAM-WC and several variants of variational attention in
deng2018latent.

Table 4: Results on IWSLT.

Model | BLEU
---|---
Soft Attention | 32.77
Variational Relaxed Attention | 30.05
Variational Attention + Enum | 33.68
Variational Attention + Sample | 33.30
BAM-WC (Ours) | 33.81

Results. As shown in Table 4, BAM-WC outperforms deterministic soft attention
significantly in BLEU score with same-level computational cost. Moreover,
compared to all variants of variational attention [deng2018latent], BAM-WC
achieves better performance with much less training cost, as BAM does not
require the training of a completely separate variational network. It is
interesting to note that in deng2018latent variational relaxed attention
(probabilistic soft attention) underperforms variational hard attention, while
BAM-WC, which is also probabilistic soft attention, can achieve better
results. One of the main reason is that BAM-WC is reparameterizable and has
stable gradients, while deng2018latent use the Dirichlet distribution which is
not reparameterizable so the gradient estimations still have high variances
despite the use of a rejection based sampling and implicit differentiation
[jankowiak2018pathwise]. Also, we note that, compared to deng2018latent, our
method is much more general because we do not need to construct the
variational distribution case by case.

###  4.5 Attention in pretrained language models

Finally, we adapt the proposed method to finetune deterministic self-attention
[vaswani2017attention] based language models pretrained on large corpora. Our
variational distribution parameters use the pretrained parameters from the
deterministic models, and we randomly initialize the parameters for contextual
prior. Then we finetune BAM for downstream tasks. We conduct experiments on
$8$ benckmark datasets from General Language Understanding Evaluation (GLUE)
[wang2018glue] and two versions of Stanford Question Answering Datasets
(SQuAD) [rajpurkar2016squad, rajpurkar2018know]. We leverage the state-of-the-
arts pretrained model, ALBERT [lan2019albert], which is a memory-efficient
version of BERT [devlin2018bert] with parameter sharing and embedding
factorization. Our implementation is based on Huggingface PyTorch Transformer
[wolf2019transformers] and we use the base version of ALBERT following the
same setting [lan2019albert] (summarized in Appendix C.5).

Table 5: Performance of BAM on GLUE and SQuAD benchmarks.

| MRPC | CoLA | RTE | MNLI | QNLI | QQP | SST | STS | SQuAD 1.1 | SQuAD 2.0
---|---|---|---|---|---|---|---|---|---|---
ALBERT-base | 86.5 | 54.5 | 75.8 | 85.1 | 90.9 | 90.8 | 92.4 | 90.3 | 80.86/88.70 | 78.80/82.07
ALBERT-base+BAM-WC | 88.5 | 55.8 | 76.2 | 85.6 | 91.5 | 90.7 | 92.7 | 91.1 | 81.40/88.82 | 78.97/82.23

Results. In Table 5, we compare the results of ALBERT finetuned with
deterministic soft attention with those finetuned with BAM-WC, resuming from
the same checkpoints. We observe consistent improvements from using BAM-WC in
both GLUE and SQuAD datasets even by only using BAM at the finetuning stage.
We leave as future work using BAM at the pretrain stage.

##  5 Conclusion

We have proposed a simple and scalable Bayesian attention module (BAM) that
achieves strong performance on a broad range of tasks but requires
surprisingly few modifications to standard deterministic attention. The
attention weights are obtained by normalizing reparameterizable distributions
parameterized by functions of keys and queries. We learn the distributions in
a Bayesian framework, introducing a key-dependent contextual prior such that
the KL is semi-analytic as regularization. Our experiments on graph node
classification, visual question answering, image captioning, and machine
translation, show that BAM outperforms corresponding baselines consistently
and provides better uncertainty without introducing extra computational and
memory cost. Further, on language understanding benchmarks, we show it is even
possible to finetune a pretrained deterministic attention with BAM and achieve
better performance than finetuning with the deterministic soft attention. With
extensive experiments and ablation studies, we demonstrate the effectiveness
of each component of the proposed architecture, and show that BAM can serve as
an efficient alternative to deterministic attention in the versatile tool box
of attention modules.

## Broader Impact

Attention modules have become critical components for state-of-the-art neural
network models in various applications, including computer vision, natural
language processing, graph analysis, and multi-modal tasks, to name a few.
While we show improvements brought by our work on five representative tasks
from a broad range of domains, our framework is general enough that it could
be used to improve potentially any attention based models. Also, our framework
solves two main issues of previously proposed probabilistic attentions that
restrict their popularity, $i.e.$, optimization difficulty and complicated
model design. We hope that our work will encourage the community to pay more
attention to stochastic attention and study from a probabilistic perspective.

Considering that attention models have been adopted in many machine learning
systems, our work could have an important impact on those systems, such as
self-driving [kim2017interpretable], healthcare [choi2016retain], and
recommender systems [tay2018multi]. However, there are potential risks of
applying such systems in real-life scenario, because the data we encounter in
real-life is biased and long-tailed, and also the discrepancy between training
data and testing data might be large. Therefore, an undue trust in deep
learning models, incautious usage or imprecise interpretation of model output
by inexperienced practitioners might lead to unexpected false reaction in
real-life and unexpected consequences. However, we see opportunities that our
work can help mitigate the risks with uncertainty estimation which is
extremely important as knowing when mistakes happen would enable us to know
when to ask for human-aid if needed for real-life applications
[ovadia2019can].

Bayesian Attention Modules: Appendix

##  Appendix A Proof for Lemma 1

###### Proof.

$\begin{split}\mbox{KL}(q_{\boldsymbol{\phi}}(S)||p_{\boldsymbol{\eta}}(S))=&\mathbb{E}_{q_{\boldsymbol{\phi}}(S)}\left[\sum_{l=1}^{L}(\log
q_{\boldsymbol{\phi}}(S_{l}|S_{1:l-1})-\log
p_{\boldsymbol{\eta}}(S_{l}|S_{1:l-1}))\right]\\\
=&\sum_{l=1}^{L}\mathbb{E}_{q_{\boldsymbol{\phi}}(S)}\left[\log
q_{\boldsymbol{\phi}}(S_{l}|S_{1:l-1})-\log
p_{\boldsymbol{\eta}}(S_{l}|S_{1:l-1})\right]\\\
=&\sum_{l=1}^{L}\mathbb{E}_{q_{\boldsymbol{\phi}}(S_{1:l-1})}\mathbb{E}_{q_{\boldsymbol{\phi}}(S_{l}|S_{1:l-1})}\left[\log
q_{\boldsymbol{\phi}}(S_{l}|S_{1:l-1})-\log
p_{\boldsymbol{\eta}}(S_{l}|S_{1:l-1})\right]\\\
=&\sum_{l=1}^{L}\mathbb{E}_{q_{\boldsymbol{\phi}}(S_{1:l-1})}\underbrace{\mbox{KL}(q_{\boldsymbol{\phi}}(S_{l}|S_{1:l-1})||p_{\boldsymbol{\eta}}(S_{l}|S_{1:l-1}))}_{\text{analytic}}.\\\
\end{split}$ (5)

∎

##  Appendix B Algorithm

$\boldsymbol{\theta},{\boldsymbol{\eta}},\boldsymbol{\phi}\leftarrow$
Initialze parameters, $t\leftarrow 0$, $\rho\leftarrow$ anneal rate

repeat

$\\{\boldsymbol{x}_{i},\boldsymbol{y}_{i}\\}_{i=1}^{M}\leftarrow$ Random
minibatch of M datapoints (drawn from full dataset)

$\\{\boldsymbol{\epsilon}_{i}\\}_{i=1}^{M}\leftarrow$ Random samples

$\lambda=\text{sigmoid}(t*\rho)$

Compute gradients
$\frac{1}{M}\nabla_{\boldsymbol{\theta},{\boldsymbol{\eta}},\boldsymbol{\phi}}\sum_{i}{\mathcal{L}}_{\lambda}(\boldsymbol{x}_{i},\boldsymbol{y}_{i},\boldsymbol{\epsilon}_{i})$
according to Eq. (4)

Update $\boldsymbol{\theta},{\boldsymbol{\eta}},\boldsymbol{\phi}$ with
gradients, $t\leftarrow t+1$

until convergence

return: $\boldsymbol{\theta},{\boldsymbol{\eta}},\boldsymbol{\phi}$

Algorithm 1 Bayesian Attention Modules

##  Appendix C Experiment details

###  C.1 Graph neural networks

####  C.1.1 Model descriptions

As in velivckovic2017graph, we apply a two-layer GAT model. We summarize the
graph attention layer here. Denote the input node features as
${\boldsymbol{h}}=\\{{\boldsymbol{h}}_{1},...,{\boldsymbol{h}}_{N}\\}$, where
$N$ is the number of nodes. Then, the self-attention weights is defined as:

$\alpha_{ij}^{h}=\frac{\exp(\text{LeakyReLU}({\boldsymbol{a}}^{h}[\mathbf{W}^{h}{\boldsymbol{h}}_{i}||\mathbf{W}^{h}{\boldsymbol{h}}_{j}]))}{\sum_{k\in\mathcal{N}_{i}}\exp(\text{LeakyReLU}({\boldsymbol{a}}^{h}[\mathbf{W}^{h}{\boldsymbol{h}}_{i}||\mathbf{W}^{h}{\boldsymbol{h}}_{k}]))},$

where ${\boldsymbol{a}}^{h},\mathbf{W}^{h}$ are neural network weights for
head $h$, and $\mathcal{N}_{i}$ is the set of neighbor nodes for node $i$.
$||$ denotes concatenation.

The output
${\boldsymbol{h}}^{\prime}=\\{{\boldsymbol{h}}_{1}^{\prime},...,{\boldsymbol{h}}_{N}^{\prime}\\}$
is computed as:

${\boldsymbol{h}}_{i}^{\prime}=||_{h=1}^{H}\sigma(\sum_{j\in\mathcal{N}_{i}}\alpha_{ij}^{h}\mathbf{W}^{h}{\boldsymbol{h}}_{j}).$

####  C.1.2 Detailed experimental settings

We follow the same architectural hyperparameters as in velivckovic2017graph.
The first layer consists of $H=8$ attention heads computing $8$ features each,
and the second layer has a single head attention following an exponential
linear unit (ELU) [clevert2015fast] nonlinearity. Then softmax is applied to
obtain probabilities. During training, we apply $L2$ regularization with
$\lambda=0.0005$. Furthermore, dropout [srivastava2014dropout] with $p=0.6$ is
applied to both layers’ inputs, as well as to the normalized attention
coefficients. Pubmed requires slight changes for hyperparameter: the second
layer has $H=8$ attention heads, and the $L2$ regularization weight is
$\lambda=0.001$. Models are initialized using Glorot initialization
[glorot2010understanding] and trained with cross-entropy loss using the Adam
SGD optimizer [kingma2014adam] with an initial learning rate of $0.01$ for
Pubmed, and $0.005$ for all other datasets. In both cases we use an early
stopping strategy on both the cross-entropy loss and accuracy on the
validation nodes, with a patience of $100$ epochs. Here, we summarize the
hyperparameters for BAM, including anneal rate $\rho$ (as in Algorithm 1),
$\sigma_{1}$ and $\sigma_{2}$ for prior Lognormal and posterior Lognormal
respectively, $k$ for Weibull distribution, $\alpha,\beta$ for Gamma
distribution, hidden dimension for contextual prior $d_{\text{mid}}$. On
Pubmed, we use anneal rate $\rho=0.2$ for all methods. For BAM-LF,
$\sigma_{1}=1\mathrm{E}6$,
$\sigma_{2}=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}2}$. For BAM-LC,
$\sigma_{1}=1\mathrm{E}5$,
$\sigma_{2}=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}2}$, and $d_{\text{mid}}=5$.
For BAM-WF, $k=10$, $\beta=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}8}$,
$\alpha=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}4}$. For BAM-WC, $k=10$,
$\beta=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}4}$, and $d_{\text{mid}}=5$. On
Cora, for BAM-LF, $\sigma_{1}=1\mathrm{E}{15}$,
$\sigma_{2}=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}6}$, and $\rho=0.2$. For BAM-
LC, $\sigma_{1}=1\mathrm{E}{15}$,
$\sigma_{2}=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}15}$, $\rho=0.1$, and
$d_{\text{mid}}=1$. For BAM-WF, $k=1$,
$\beta=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}10}$,
$\alpha=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}15}$, and $\rho=0.2$. For BAM-WC,
$k=1$, $\beta=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}10}$, $\rho=0.1$, and
$d_{\text{mid}}=1$. On Citeseer, we use anneal rate $0.1$ for all methods. for
BAM-LF, $\sigma_{1}=1\mathrm{E}{15}$ and
$\sigma_{2}=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}6}$. For BAM-LC,
$\sigma_{1}=1\mathrm{E}{15}$,
$\sigma_{2}=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}5}$, and $d_{\text{mid}}=1$.
For BAM-WF, $k=100$, $\beta=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}15}$, and
$\alpha=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}7}$. For BAM-WC, $k=100$,
$\beta=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}15}$, and $d_{\text{mid}}=1$.

Table 6: Basic statistics on datasets for node classification on graphs.

| Cora | Citeseer | PubMed
---|---|---|---
#Nodes | 2708 | 3327 | 19717
#Edges | 5429 | 4732 | 44338
#Features/Node | 1433 | 3703 | 500
#Classes | 7 | 6 | 3
#Training Nodes | 140 | 120 | 60
#Validation Nodes | 500 | 500 | 500
#Test Nodes | 1000 | 1000 | 1000

###  C.2 Visual question answering

####  C.2.1 Uncertainty evaluation via PAvPU

We adopt hypothesis testing to quantify the uncertainty of a model’s
prediction. Consider $M$ posterior samples of predictive probabilities
$\\{{\boldsymbol{p}}_{m}\\}_{m=1}^{M}$, where ${\boldsymbol{p}}_{m}$ is a
vector with the same dimension as the number of classes. To quantify how
confident our model is about its prediction, we evaluate whether the
difference between the probabilities of the first and second highest classes
(in terms of posterior means) is statistically significant with two-sample
$t$-test.

With the output $p$-values and a given threshold, we can determine whether a
model is certain about its prediction. Then, we evaluate the uncertain using
the Patch Accuracy vs Patch Uncertainty metric [mukhoti2018evaluating] which
is defined as
$\mathrm{PAvPU}={\left(n_{ac}+n_{iu}\right)}/{\left(n_{ac}+n_{au}+n_{ic}+n_{iu}\right)}$,
where $n_{ac},n_{au},n_{ic},n_{iu}$ are the numbers of accurate and certain,
accurate and uncertain, inaccurate and certain, inaccurate and uncertain
samples, respectively. Since for VQA, each sample has multiple annotations,
the accuracy for each answer can be a number between $0$ and $1$ and it is
defined as $\text{Acc}(ans)=\min\\{{(\\#\text{human that said
}ans)}/{3},1\\}.$ Then we generalize the PAvPU for VQA task accordingly:

$\displaystyle\leavevmode\resizebox{211.3883pt}{}{$n_{ac}=\sum_{i}\text{Acc}_{i}\text{Cer}_{i},~{}n_{iu}=\sum_{i}(1-\text{Acc}_{i})(1-\text{Cer}_{i})$\,},\leavevmode\resizebox{211.3883pt}{}{$n_{au}=\sum_{i}\text{Acc}_{i}(1-\text{Cer}_{i}),~{}n_{ic}=\sum_{i}(1-\text{Acc}_{i})(\text{Cer}_{i})$\,},$

where for the $i$th prediction $\text{Acc}_{i}$ is the accuracy and
$\text{Cer}_{i}\in\\{0,1\\}$ is the certainty indicator.

####  C.2.2 Model descriptions

We use the state-of-the-art VQA model, MCAN [yu2019deep], to conduct
experiments. The basic component of MCAN is Modular Co-Attention (MCA) layer.
The MCA layer is a modular composition of two basic attention units: the self-
attention (SA) unit and the guided-attention (GA) unit, where the SA unit
focuses on intra-modal interactions and GA unit focuses on inter-modal
interactions. Both units follow the multi-head structure as in
vaswani2017attention, including the residual and layer normalization
components. The only difference is that in GA, the queries come from a
different modality (images) than the keys and values (questions). By stacking
MCA layers, MCAN enables deep interactions between the question and image
features. We adopt the encoder-decoder structure in MCAN [yu2019deep] with six
co-attention layers.

####  C.2.3 Detailed experimental settings

We conduct experiments on the VQA-v2 dataset, which is split into the training
(80k images and 444k QA pairs), validation (40k images and 214k QA pairs), and
testing (80k images and 448k QA pairs) sets. The evaluation is conducted on
the validation set as the true labels for the test set are not publicly
available [deng2018latent], which we need for uncertainty evaluation. For the
noisy dataset, we add Gaussian noise (mean $0$, variance $5$) to image
features. We follow the hyperparameters and other settings from yu2019deep:
the dimensionality of input image features, input question features, and fused
multi-modal features are set to be $2048$, $512$, and $1024$, respectively.
The latent dimensionality in the multi-head attention is $512$, the number of
heads is set to $8$, and the latent dimensionality for each head is $64$. The
dropout rate is $0.1$. The size of the answer vocabulary is set to $N=3129$
using the strategy in teney2018tips. To train the MCAN model, we use the Adam
optimizer [kingma2014adam] with $\beta_{1}=0.9$ and $\beta_{2}=0.98$. The
learning rate is set to
$\min(2.5t\mathrm{E}{\scalebox{0.5}[1.0]{$-$}5},1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}4})$,
where $t$ is the current epoch number starting from $1$. After $10$ epochs,
the learning rate is decayed by $1/5$ every $2$ epochs. All the models are
trained up to $13$ epochs with the same batch size of $64$. To tune the
hyperparameters in BAM, we randomly hold out $20\%$ of the training set for
validation. After tuning, we train on the whole training set and evaluate on
the validation set. For BAM-LF, $\sigma_{1}=1\mathrm{E}{9}$,
$\sigma_{2}=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}9}$, and $\rho=0.2$. For BAM-
LC, $\sigma_{1}=1\mathrm{E}{9}$,
$\sigma_{2}=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}9}$, $\rho=0.2$, and
$d_{\text{mid}}=20$. For BAM-WF, $k=1000$,
$\beta=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}2}$,
$\alpha=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}3}$, and $\rho=0.2$. For BAM-WC,
$k=1000$, $\beta=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}6}$, $\rho=0.1$, and
$d_{\text{mid}}=20$.

####  C.2.4 More results

Table 7: Performance comparison among different attention modules on visual
question answering.

Accuracy
---
Attention | Original Data |  | Noisy Data
| ALL | Y/N / NUM / OTHER |  | ALL | Y/N / NUM / OTHER
Soft | 66.95 | 84.55 / 48.92 / 58.33 |  | 61.25 | 80.58 / 40.80 / 51.97
BAM-LF | 66.89 | 84.46 / 49.11 / 58.24 |  | 61.43 | 80.95 / 41.51 / 51.85
BAM-LC | 66.93 | 84.58 / 49.05 / 58.24 |  | 61.58 | 80.70 / 41.31 / 52.40
BAM-WF | 66.93 | 84.55 / 48.84 / 58.32 |  | 61.60 | 81.02 / 41.84 / 52.05
BAM-WC | 67.02 | 84.66 / 48.88 / 58.42 |  | 62.89 | 81.94 / 41.90 / 53.96

Uncertainty
---
Attention | Original Data |  | Noisy Data
| ALL | Y/N / NUM / OTHER |  | ALL | Y/N / NUM / OTHER
Soft | 70.04 | 83.02 / 56.81 / 63.66 |  | 65.34 | 78.84 / 49.80 / 59.18
BAM-LF | 69.92 | 82.79 / 56.87 / 63.58 |  | 65.48 | 79.13 / 50.19 / 59.16
BAM-LC | 70.14 | 83.02 / 57.40 / 63.71 |  | 65.60 | 78.85 / 49.87 / 59.70
BAM-WF | 70.09 | 83.00 / 56.85 / 63.78 |  | 65.62 | 79.16 / 50.22 / 59.40
BAM-WC | 71.21 | 83.95 / 58.12 / 63.82 |  | 66.75 | 80.21 / 51.38 / 60.58

###  C.3 Image captioning

####  C.3.1 Model descriptions

We conduct experiments on an attention-based model for image captioning,
Att2in, in rennie2017self. This model uses RNN as its decoder, and at each
step of decoding, image features are aggregated using attention weights
computed by aligning RNN states with the image features. Formally, suppose
$I_{1},...,I_{N}$ are image features, ${\boldsymbol{h}}_{t-1}$ is the hidden
state of RNN at step $t-1$. Then, the attention weights at step $t$ are
computed by:
$\boldsymbol{\alpha}_{t}=\text{softmax}({\boldsymbol{a}}_{t}+b_{\alpha})$, and
$a_{t}^{i}=W\text{tanh}(W_{aI}I_{i}+W_{ah}{\boldsymbol{h}}_{t-1}+b_{a})$,
where $W,W_{aI},W_{ah},b_{\alpha},b_{a}$ are all neural network weights.
Aggregated image feature $I_{t}=\sum_{i=1}^{N}\alpha_{t}^{i}I_{i}$ would then
be injected into the computation of the next hidden state of RNN
${\boldsymbol{h}}_{t}$ (see details in rennie2017self).

####  C.3.2 Detailed experimental settings

We use the code from https://github.com/ruotianluo/self-critical.pytorch and
conduct our experiments on the MS COCO dataset [lin2014microsoft] that
consists of 123,287 images. Each image has at least five captions. We use the
standard data split from karpathy2015deep, with 113,287 training, 5000
validation, and 5000 testing images. The vocabulary size $V$ is 9488 and the
max caption length $T$ is 16. We replace the ResNet-encoded features in
rennie2017self with boxing box features extracted from a pre-trained Faster-
RCNN [ren2015faster] as visual features. Following the original setting in the
code base, we use batch size $10$, Adam optimizer with learning rate
$5\mathrm{E}\scalebox{0.5}[1.0]{$-$}4$, dropout rate of $0.5$ and train $30$
epochs. During training, we use MLE loss only without scheduled sampling or RL
loss, neither of which is compatible with our current framework. For testing,
we use greedy search to generate sequences. For BAM, we use contextual prior
with $d_{\text{mid}}=10$ and $\rho=1$. For BAM-WC, $k=10$,
$\beta=1\mathrm{E}\scalebox{0.5}[1.0]{$-$}6$. For BAM-LC,
$\sigma_{1}=1\mathrm{E}3,\sigma_{2}=0.1$.

###  C.4 Neural Machine Translation

####  C.4.1 Model descriptions

To make comparision with deng2018latent, we adopt the LSTM-based machine
translation model in that paper. The model use a bidirectional LSTM to encode
source sentence to source representations
$\boldsymbol{x}_{1},...,\boldsymbol{x}_{T}$. At the step $j$ of decoding,
current LSTM state $\tilde{\boldsymbol{x}}$ (a function of previous target
words $y_{1:j-1}$) is used as query. The attention weights is computed from an
MLP between the query and encoded source token representations. Then the
aggregated feature is used to produce the distribution over the next target
work $y_{j}$ (see details in deng2018latent or see code in
https://github.com/harvardnlp/var-attn).

####  C.4.2 Detailed experimental settings

We use the same dataset, IWSLT [cettolo2014report], as deng2018latent. We
preprocess the data in the same way: using Byte Pair Encoding over the
combined source/target training set to obtain a vocabulary size of 14,000
tokens [sennrich2015neural]. We train on the sequence length up to 125. We use
a two-layer bi-dreictional LSTM with 512 units and 768 units for encoder and
decoder respectively. In addition, the batch size is $6$, dropout rate is
$0.3$, learning rate is $3\mathrm{E}\scalebox{0.5}[1.0]{$-$}4$ (Adam
optimizer). For testing, we use beam search with beam size 10 and length
penalty $1$ [wu2016google]. For BAM-WC, $k=5$,
$\beta=1\mathrm{E}{\scalebox{0.5}[1.0]{$-$}6}$, $\rho=1$, and
$d_{\text{mid}}=5$.

###  C.5 Pretrained language model

####  C.5.1 Model descriptions

BERT [devlin2018bert] is a state-of-the-art deep bidirectional
transformer[vaswani2017attention] model pretrained on large corpora to extract
contextual word representations. ALBERT [lan2019albert] improves upon BERT in
terms of latency efficiency and performance by using (a) factorized embedding
parameterization, (b) cross-layer parameter sharing, and (c) a sentence-order
prediction (SOP) loss. Our experiment is done on the ALBERT-base model, which
includes $12$ attention layers, each of hidden dimension $768$. The embedding
dimension for factorized embedding is $128$. While BERT-base involves $108M$
parameters, ALBERT-base only has $12M$ parameters.

####  C.5.2 Detailed experimental settings

Our experiment includes both the General Language Understanding Evaluation
(GLUE) and Stanford Question Answering (SQuAD) Datasets. We evaluate on $8$
tasks from GLUE including Corpus of Linguistic Acceptability (CoLA;
[warstadt2019neural]), Stanford Sentiment Treebank (SST;
[socher2013recursive]), Microsoft Research Paraphrase Corpus (MRPC;
[dolan2005automatically]), Semantic Textual Similarity Benchmark
(STS;[cer2017semeval]), Quora Question Pairs (QQP; [iyer2017first]), Multi-
Genre NLI (MNLI; [williams2017broad]), Question NLI (QNLI;
[rajpurkar2016squad]), and Recognizing Textual Entailment (RTE;
[dagan2005pascal]). We evaluate on both SQuAD v1.1 and SQuAD v2.0. Our code is
built on wolf2019transformers, which can be found at
https://github.com/huggingface/transformers. We follow the training settings
as in lan2019albert and summarize them in Table 8. We also include the
hyperparameter setting for BAM-WC. We note, as the model is already pretrained
so we do not anneal KL term. We pick
$\beta=1\mathrm{E}\scalebox{0.5}[1.0]{$-$}2$ and $d_{\text{dim}}=5$ for all
experiments, as we found the performance is not sensitive to them. We include
the $k$ in Table 8.

Table 8: Experiment setting for pretrained language model (LR: learning rate,
BSZ: batch size, DR: dropout rate, TS: training steps, WS: warmping steps,
MSL: maximum sentence length).

| LR | BSZ | ALBERT DR | Classifier DR | TS | WS | MSL | $k$
---|---|---|---|---|---|---|---|---
CoLA |  1.00 $\mathrm{E}$ $-$ 05 | 16 | 0 | 0.1 | 5336 | 320 | 512 | 10
STS |  2.00 $\mathrm{E}$ $-$ 05 | 16 | 0 | 0.1 | 3598 | 214 | 512 | 20
SST  $-$ 2 |  1.00 $\mathrm{E}$ $-$ 05 | 32 | 0 | 0.1 | 20935 | 1256 | 512 | 1000
MNLI |  3.00 $\mathrm{E}$ $-$ 05 | 128 | 0 | 0.1 | 10000 | 1000 | 512 | 5
QNLI |  1.00 $\mathrm{E}$ $-$ 05 | 32 | 0 | 0.1 | 33112 | 1986 | 512 | 500
QQP |  5.00 $\mathrm{E}$ $-$ 05 | 128 | 0.1 | 0.1 | 14000 | 1000 | 512 | 1000
RTE |  3.00 $\mathrm{E}$ $-$ 05 | 32 | 0.1 | 0.1 | 800 | 200 | 512 | 1000
MRPC |  2.00 $\mathrm{E}$ $-$ 05 | 32 | 0 | 0.1 | 800 | 200 | 512 | 100
SQuAD v1.1 |  5.00 $\mathrm{E}$ $-$ 05 | 48 | 0 | 0.1 | 3649 | 365 | 384 | 10
SQuAD  v 2.0 |  3.00 $\mathrm{E}$ $-$ 05 | 48 | 0 | 0.1 | 8144 | 814 | 512 | 2000


# Structural characterization of thin-walled microbubble cavities

Mohammed Zia Jalaludeen  1 Shilong Li  1,* Ke Tian  1,2 Toshio Sasaki  3 and
Síle Nic Chormaic1  1Light-Matter Interactions for Quantum Technologies Unit,
Okinawa Institute of Science and Technology Graduate University, Onna, Okinawa
904-0495, Japan
2College of Physics and Optoelectronic Engineering, Harbin Engineering
University, Harbin 150001, China
3Scientific Imaging Section, Okinawa Institute of Science and Technology
Graduate University, Onna, Okinawa 904-0495, Japan<EMAIL_ADDRESS>
††journal: opticajournal††articletype: Research Article

Whispering gallery mode (WGM) microbubble cavities are a versatile optofluidic
sensing platform owing to their hollow core geometry. To increase the light-
matter interaction and, thereby, achieve a higher sensitivity, thin-walled
microbubbles are desirable. However, a lack of knowledge about the precise
geometry of hollow microbubbles prevents us from having an accurate
theoretical model to describe the WGMs and their response to external stimuli.
In this work, we provide a complete characterization of the wall structure of
a microbubble and propose a theoretical model for the WGMs in this thin-walled
microcavity based on the optical waveguide approach. Structural
characterization of the wavelength-scale wall is enabled by focused ion beam
milling and scanning electron microscopy imaging. The proposed theoretical
model is verified by finite element method simulations. Our approach can
readily be extended to other low-dimensional micro-/nanophotonic structures.

##  1 Introduction

Optical microcavities supporting whispering gallery modes (WGMs) have been
investigated intensively in the past two decades [1, 2] due to their ultrahigh
quality factor (Q-factor), which makes them suitable for various optical
applications ranging from cavity quantum electrodynamics [3, 4, 5, 6, 7] to
label-free optical detection [8, 9, 10, 11, 12]. Compared with the widely used
WGM microcavity geometries such as microspheres [13, 14], microtoroids [15],
and microrings [16], microbubble cavities or microbubbles [17, 18] have the
advantage of a hollow core and can be used as optofluidic devices in an all-
fiber manner [19, 20, 21]. Moreover, the resultant thin-walled structure of
microbubbles provides us with new degrees of freedom, such as the thickness of
the wall and its variation along the cavity axis. These allow us to engineer
properties related to the WGMs, such as the mode field distribution, the mode
dispersion, and the mode spectrum. Such engineered WGMs are particularly
useful for various nonlinear optical processes, for example, four-wave
parametric oscillation and frequency comb generation [22, 23, 24, 25].
Therefore, an accurate determination of the geometry of a microbubble to
precisely characterize its WGMs is an important prerequisite for practical
applications of such cavities.

Several methods to determine the thin-walled structure of microbubble cavities
in a non-destructive way have already been reported. Bright-field microscopy
is probably the simplest approach for measuring the diameter of a microbubble
[26], but the low image contrast at a reasonable field-of-view excludes it as
an effective way for a wall thickness measurement. Confocal microscopy has
been used to measure the wall thickness of microbubble cavities [27]; however,
the image resolution limits the accuracy of the measured thickness to half a
wavelength. Obtaining the structural information of the microbubble by
inferring its response to a certain stimulus seems to be a non-destructive
method for the determination of the wall thickness. For example, the
microbubble wall thickness was measured based on the internal aerostatic
pressure sensing method with a measurement uncertainty on sub-micrometer scale
[28]. Nonetheless, such a method is not ideal as it requires precise knowledge
of the structural information beforehand and it also assumes a constant wall
thickness along the cavity axis. Currently, the only reliable method for
studying the wall structure of a microbubble is a destructive approach, which
involves breaking the microbubble and then measuring its cross-section using,
for example, scanning electron microscopy [22, 29, 30]. However, a constant
wall thickness along the cavity axis is generally assumed. A fully systematic
study on the microbubble wall structure is yet to be carried out.

In this work, we fully characterize the wall structure of a microbubble cavity
using focused ion beam (FIB) milling and scanning electron microscopy (SEM)
imaging. Both the wall thickness and its variation along the cavity axis are
obtained, thus enabling us to precisely model the microbubble geometry.
Considering the wavelength-scale wall thickness, a theoretical model based on
optical waveguide theory is proposed to describe the WGMs in the microbubble
cavity. Finite element method simulations are performed to verify the validity
of the proposed theoretical model. Our results will benefit not only the
development of microbubble cavities but also the exploration of other low-
dimensional micro-/nano-photonic structures.

Figure 1: Structural characterization of a thin-walled microbubble by FIB
milling and SEM imaging. The microbubble’s support stem on the left side was
initially removed through FIB milling, while half of the support stem on the
right side was cut, creating a gap (a). A third FIB milling was conducted from
the center of the left side towards the center of the gap on the right side,
resulting in the removal of half of the microbubble (b). The wall structure of
the microbubble is clearly visible under SEM imaging (c). Due to the high SEM
imaging resolution, the wall thickness variation along the cavity axis can be
determined with accuracy down to the nanometer scale (upper panel, d). To
describe such a wall structure, Gaussian profiles were used to fit the outer
and inner boundaries of the microbubble (lower panel, d). The dependence of
the wall thickness $t$ on the outer diameter $d$ of the thin-walled
microbubble is shown in (e). A reciprocal linear relation is satisfied near
the center of the microbubble at larger outer diameters.

##  2 Experimental results

Silica microbubbles were fabricated as described previously [29] using fused
silica capillaries (360-$\mu$m outer diameter and 250-$\mu$m inner diameter)
and a CO2 laser. The wall structure of a microbubble (with a thin Au layer
coating) was determined by means of FIB milling and SEM imaging (FEI Helios G3
UC), see Fig. 1. Due to the high imaging resolution of both the SEM and the
FIB, the microbubble’s wall structure was clearly visible. The largest
diameter was around 64 $\mu$m at the bubble center, where the wall thickness
was the thinnest ($\sim$0.85 $\mu$m). The bubble diameter gradually decreased,
following a Gaussian, down to around 28 $\mu$m at the support stem, where the
wall was thickest at $\sim$2.60 $\mu$m. With this structural information, the
mode structure of the thin-walled microbubble could be fully determined by
either theoretical models or numerical simulations, as shown below.

The wall thickness is a crucial parameter to measure the performance of
microbubble-based optical sensors [29]. However, due to the lack of an
efficient approach to fully characterize the wall structure, geometrical
approaches [26, 27, 31, 32] have widely been used to estimate the wall
thickness. These models are based on two assumptions, i.e. constant wall
thickness and mass conservation. The first assumption is meaningful when
making order-of-magnitude estimates. The second assumption can be split into
the following two scenarios: area conservation or volume conservation. In the
first case, the microbubble is considered to be the result of cylindrical
expansion of a capillary. The cross-section of the capillary is a ring with an
area of $\pi(d/2)^{2}-\pi(d/2-t)^{2}\approx\pi dt$, where $d$ is the bubble’s
outer diameter and $t$ its wall thickness ($t\ll d$). The conserved area leads
to a reciprocal linear relationship between $d$ and $t$, i.e., $t=c_{1}/d$
with a coefficient $c_{1}$. Similarly, for the second case, the microbubble
can be viewed as the result of spherical expansion of a spherical bubble with
the same diameter as the capillary. Since the volume of the spherical bubble
is $(4/3)\pi(d/2)^{3}-(4/3)\pi(d/2-t)^{3}\approx\pi d^{2}t$, a reciprocal
quadratic relation of $t=c_{2}/d^{2}$ is obtained with $c_{2}$ as a
coefficient. It is generally believed that the cylindrical expansion gives an
over-estimation of the wall thickness, therefore the upper limit for the
measured thickness, while the lower limit is obtained from the spherical
expansion [27].

Figure 1(e) shows the $t$ versus $d$ relationship for the thin-walled
microbubble. The wall thickness of the microbubble varies along the bubble
axis, clearly showing that it does not have a constant wall thickness,
invalidating the first assumption mentioned above. The second assumption is
also invalid because the relationship between $t$ and $d$ is neither
reciprocal linear nor reciprocal quadratic. Nevertheless, around the center of
the thin-walled microbubble ($\pm 40$ $\mu$m from the microbubble center), a
reciprocal linear relationship between $t$ and $d$ is satisfied. We attribute
these seemingly unusual results to the fact that only the center of the
microbubble was melted and fully expanded, while the portions near the support
stems experienced lower temperatures and were unable to fully expand. These
findings demonstrate the importance of developing an efficient approach for
characterizing the wall structure of microbubbles.

##  3 Theoretical model

As a type of optical bottle microresonators (BMRs), light in microbubbles is
trapped in the cross-sectional plane, circulating around the bubble axis,
while confined axially, bouncing back and forth between two turning points
known as caustics; this is similar to the way charged particles are trapped in
magnetic bottles [33]. Such a confinement of light in three dimensions (3D)
results in the quantization of optical fields into a series of optical modes.
With the structural information (Fig. 2(a)) obtained from experimental
measurements, the mode structure of the thin-walled microbubble can be
theoretically modeled by simplifying Maxwell’s equations.

For azimuthally and axially symmetric microbubbles made of isotropic and
homogeneous nonmagnetic dielectric materials, optical spin-orbit coupling is
absent [34, 35]. Furthermore, most of these microbubbles have a relatively
small diameter variation along the axial direction (see Fig. 2(a)). Therefore,
the two sets of polarization modes can be well separated: the transverse
electric (TE) modes with a nonzero axial electric field and the equivalent
transverse magnetic (TM) modes with a nonzero axial magnetic field. Since the
TE modes preferentially exist in thin-walled microbubbles, only they will be
considered in the subsequent analysis. The Helmholtz equation for the nonzero
axial electric field $E_{z}$ of TE modes reads:

$\nabla^{2}E_{z}(\mathbf{r})+k^{2}\mathbf{\varepsilon}_{\mathrm{r}}(\mathbf{r})E_{z}(\mathbf{r})=0.$
(1)

Here, $k=\omega/c$ is the wave vector, $\omega$ is the angular frequency,
$c=1/\sqrt{\varepsilon_{0}\mu_{0}}$ is the speed of light, and
$\varepsilon_{0}$ ($\mu_{0}$) is the permittivity (permeability) in vacuum.
The relative permittivity
$\mathbf{\varepsilon}_{\mathrm{r}}=\mathbf{\varepsilon}/\varepsilon_{0}$,
where $\mathbf{\varepsilon}$ is the permittivity of a material. The relative
permeability $\mathbf{\mu}_{\mathrm{r}}$ of nonmagnetic materials in the
visible and infrared spectral range is close to unity and one may set the
material’s permeability $\mathbf{\mu}=\mu_{0}$.

Figure 2: Theoretical model of a thin-walled microbubble. (a) Reconstructed 3D
geometry of the microbubble from the SEM images shown in Fig. 1. Due to the
geometrical symmetries, the electric field distribution of WGMs in the
microbubble can be calculated by solving the three coupled differential
equations, Eqs. 3–5. Three mode indices, i.e. $n$, $m$, and $l$, are resolved
in the end to identify the WGMs, in addition to their polarization state
(either TM or TE). (b) Radial field distribution. Owing to the wavelength-
scale wall thickness, only the fundamental radial mode with $n=1$ is
considered. The inset illustrates the physical meaning of the effective
permittivity $\varepsilon_{\mathrm{eff}}$. (c) Axial field distribution. The
quasipotential of the microbubble forms a quantum well—due mainly to the
diameter variation—which confines the axial motion such that different axial
modes emerge. (d) Azimuthal field distribution. Only a small portion of the
azimuthal field is shown for better visibility. (e) Dependence of
$\varepsilon_{\mathrm{eff}}$ on the wall thickness. (f) Illusory example
showing a quantum-barrier-like quasipotential (lower panel) formed solely by
the wall thickness variation (upper panel) where a constant outer diameter was
used. (g) Dependence of $k_{\mathrm{circ}}$ on the outer diameter.

The scalar Helmholtz equation for $E_{z}$ is a 3D partial differential
equation. It can be further simplified by the method of separation of
variables in the cylindrical coordinates $(\rho,\varphi,z)$ which are defined
in Fig. 2(a). To this end, $E_{z}$ can be expressed in the separable form:

$E_{z}(\rho,\varphi,z)=P(\rho)\Phi(\varphi)\Psi(z),$ (2)

where $P(\rho)$, $\Phi(\varphi)$, and $\Psi(z)$ are the radial, azimuthal, and
axial field components, respectively. Substituting Eq. 2 into Eq. 1, three
ordinary differential equations for the respective field components are
obtained:

$\displaystyle\left[\rho^{2}\frac{\mathrm{d}^{2}}{\mathrm{d}\rho^{2}}+\rho\frac{\mathrm{d}}{\mathrm{d}\rho}+\varepsilon_{\mathrm{r},z}(\rho)\rho^{2}\right]P(\rho)=$
$\displaystyle\varepsilon_{\mathrm{eff,}z}P(\rho),$ (3)
$\displaystyle\frac{\mathrm{d}^{2}}{\mathrm{d}\varphi^{2}}\Phi(\varphi)=$
$\displaystyle-k^{2}_{\mathrm{circ,}z}\varepsilon_{\mathrm{eff,}z}\Phi(\varphi),$
(4)
$\displaystyle\left[-\frac{1}{\varepsilon_{\mathrm{eff,}z}}\frac{\mathrm{d}^{2}}{\mathrm{d}z^{2}}+k^{2}_{\mathrm{circ,}z}\right]\Psi(z)=$
$\displaystyle k^{2}\Psi(z),$ (5)

which are coupled via two coupling constants $\varepsilon_{\mathrm{eff}}$ and
$k_{\mathrm{circ}}$. Here, $\varepsilon_{\mathrm{eff}}$ couples the dynamics
of the radial and the circular components of the light propagation, while
$k_{\mathrm{circ}}$ couples the dynamics of the lateral and the axial
components of the light propagation. In the end, the discrete spectrum
$\omega=kc$ can be obtained by solving these three ordinary differential
equations, Eqs. 3–5.

The general solution of Eq. 3 consists of a linear combination of Bessel
functions of the first and second kind. Their coefficients are determined by
matching the wall boundary condition (Fig. 2(b)). The Bessel functions and,
therefore, $P(\rho)$ look like oscillating sine or cosine functions that decay
proportionally to $1/\sqrt{\rho}$. Different radial modes can be distinguished
by the number of ‘peaks’ in the $P(\rho)$, with each mode labeled by a unique
radial mode index $n=1,2,...$. Equation 3 quantifies the effect of the wall
thickness on these radial modes by using an effective permittivity
$\varepsilon_{\mathrm{eff}}$, i.e., the first coupling constant, which
measures the degree of light confinement by the wall (see the inset of Fig.
2(b)). Since the diameter of microbubbles is typically in the range of a few
tens of micrometers, the weakly curved condition is satisfied, and the
microbubble wall can be treated as a slab waveguide. Then,
$\varepsilon_{\mathrm{eff}}$ can be easily found via the transcendental
equation based on the well-established optical waveguide theory [36]:

$\mathrm{tan}(\kappa
t)=\frac{\kappa\gamma_{\mathrm{air}}+\kappa\gamma_{\mathrm{core}}}{\kappa^{2}-\gamma_{\mathrm{air}}\gamma_{\mathrm{core}}},$
(6)

where
$\gamma_{\mathrm{air}}^{2}=k^{2}\varepsilon_{\mathrm{eff}}-k^{2}\varepsilon_{\mathrm{air}}$,
$\gamma_{\mathrm{core}}^{2}=k^{2}\varepsilon_{\mathrm{eff}}-k^{2}\varepsilon_{\mathrm{core}}$,
$\kappa^{2}=k^{2}\varepsilon_{\mathrm{r}}-k^{2}\varepsilon_{\mathrm{eff}}$
with $\varepsilon_{\mathrm{air}}$ and $\varepsilon_{\mathrm{core}}$ are the
relative permittivities of the surrounding air and the microbubble’s core
material, respectively. Figure 2(e) shows the calculated
$\varepsilon_{\mathrm{eff}}$ as a function of the wall thickness at a few
wavelengths of interest. It is clear that the microbubble wall plays a crucial
role when its thickness is close to the propagating wavelength, as is the case
for most microbubble cavities.

The general solution of Eq. 4 is $\Phi(\varphi)=A\mathrm{exp}[\pm
i(\beta\varphi+\varphi_{0})]$ with the amplitude $A$ and the initial phase
$\varphi_{0}$, where
$\beta=k_{\mathrm{circ}}\sqrt{\varepsilon_{\mathrm{eff}}}$ can be called the
propagation constant. Note that ‘$\pm$’ corresponds to the clockwise (CW) and
counter-clockwise (CCW) modes [9]. To have a stable optical field
distribution, the solution must satisfy the periodic boundary condition (Fig.
2(d)):

$\sqrt{\varepsilon_{\mathrm{eff},z}}\pi d=m\lambda_{\mathrm{circ,z}},$ (7)

where $\lambda_{\mathrm{circ}}=2\pi/k_{\mathrm{circ}}$ is the wavelength in
the cross-sectional plane. On the one hand, it leads to the WGMs identified by
the azimuthal mode index $m=0,1,2,...$. On the other hand, it determines the
$k_{\mathrm{circ}}$ in this cross-section, i.e., the second coupling constant.
Figure 2(g) shows the dependence of $k_{\mathrm{circ}}$ on the outer diameter.
Generally speaking, the smaller the outer diameter, the stronger the azimuthal
optical confinement.

The last equation, Eq. 5, describes the axial dynamics of the WGMs. It is a
quasi-Schrödinger equation with the quasipotential:

$V_{\mathrm{eff}}(z)=\frac{\hbar c}{e}k_{\mathrm{circ,}z},$ (8)

where the elementary charge $e$ is used to scale the quasipotential in eV.
Although not explicitly mentioned in the literature, the quasipotential in
most microbubble cavities forms a quantum well, which confines the axial
motion of the WGMs (Fig. 2(c)). The quantization of the axial motion in the
quantum well results in different axial modes indicated by the axial mode
index $l=1,2,...$. However, due to the lack of analytical solutions for most
quantum well quasipotentials, the finite difference method is widely employed
as a reliable numerical approximation technique to solve the quasi-Schrödinger
equation [36].

The customizability of their axial modes distinguishes microbottle cavities
from microsphere and microtoroid cavities [37]. As one type of microbottle
cavity, microbubbles provide a new degree-of-freedom, i.e., the wall
structure, in tailoring the axial modes. This becomes clearer with the
theoretical model presented here: both the wall thickness and its variation
contribute to the axial quasipotential in Eq. 8 by determining
$\varepsilon_{\mathrm{eff}}$ through Eq. 6 and subsequently influencing the
value of $k_{\mathrm{circ}}$ as described in Eq. 7. Figure 2(f) shows an
example where a microbubble can provide a quantum barrier for the axial
optical motion if the quasipotential is formed solely by the wall structure.
This is similar to the axial mode engineering in rolled-up microbottle
cavities [38, 36, 39, 40, 41].

##  4 Simulation verification

Figure 3: Verification of the theoretical model of the thin-walled microbubble
by finite element method simulations. (a) Resonant wavelengths for different
azimuthal modes. Simulated field distribution in the cross-section is shown in
the inset. (b) Resonant wavelengths for different axial modes. The inset shows
the axial field distribution. $n=1$ for all cases.

The measured 3D structure of the thin-walled microbubble (Fig. 2(a)) allows us
to characterise the shape with great accuracy. Therefore, a series of
simulations based on the finite element method were carried out and the
results are summarized in Fig. 3. These simulation results were used to verify
the validity of the proposed theoretical model.

Figure 3(a) shows the resonant wavelengths of WGMs with $m$ ranging from 165
to 175 ($l=1$ and $n=1$). The calculated values using the aforementioned
theoretical model agree with the simulation results over a wide spectral
range. Such good agreement confirms the validity of the proposed theoretical
model based on the optical waveguide approximation. This deepens our
understanding of the underlying physics and facilitates the design for device
applications using thin-walled microbubble cavities.

The comparison for the axial modes is made in Fig. 3(b) for WGMs with $m=170$
and $n=1$. Very good agreement between the theoretical model and simulations
is also obtained. This confirms the effectiveness of treating the WGM axial
dynamics in the same way as the dynamics of a particle in a quantum well. By
doing so, thin-walled microbubble cavities become a reliable experimental
platform to test quantum mechanics. On the other hand, the well-developed
quantum theory can be used to engineer the axial modes.

##  5 Conclusion

We have demonstrated an efficient way to fully characterize the wall structure
of a microbubble cavity. The 3D geometry of the microbubble was reconstructed
based on FIB milling and SEM imaging. Owing to the wavelength-scale wall
thickness, a theoretical model based on the optical waveguide approximation
has been proposed to describe the WGMs in the thin-walled microbubble cavity.
Simulations have also been performed using the fabricated microbubble
structure. Very good agreement between the proposed theoretical model and
simulations has been obtained, verifying the validity of the proposed theory.
The demonstrated characterization and modeling approaches are readily
adaptable for other wavelength-scaled photonic devices.

Funding This work was funded by the Okinawa Institute of Science and
Technology Graduate University (OIST), and the Japan Society for the Promotion
of Science (JSPS) KAKENHI through Grant-in-Aid for Scientific Research (C)
with Grant Number 23K04617 and Grant-in-Aid for Early-Career Scientists with
Grant Number 22K14621.

Acknowledgments The authors would like to thank the Engineering Section, the
Scientific Computing & Data Analysis Section, and the Scientific Imaging
Section of Okinawa Institute of Science and Technology Graduate University
(OIST) for technical assistance.

Disclosures The authors declare no conflicts of interest.

Data Availability Data underlying the results presented in this paper are not
publicly available at this time but may be obtained from the authors upon
reasonable request.

## References

  * [1] L. Cai, J. Pan, Y. Zhao, J. Wang, and S. Xiao, “Whispering gallery mode optical microresonators: Structures and sensing applications,” Phys. Status Solidi A 217, 1900825 (2020).
  * [2] D. Yu, M. Humar, K. Meserve, R. C. Bailey, S. Nic Chormaic, and F. Vollmer, “Whispering-gallery-mode sensors for biological and physical sensing,” Nat. Rev. Methods Primers 1, 82 (2021).
  * [3] K. J. Vahala, “Optical microcavities,” Nature (London) 424, 839–846 (2003).
  * [4] S. M. Spillane, T. J. Kippenberg, K. J. Vahala, K. W. Goh, E. Wilcut, and H. J. Kimble, “Ultrahigh-Q toroidal microresonators for cavity quantum electrodynamics,” Phys. Rev. A 71, 013817 (2005).
  * [5] T. Aoki, B. Dayan, E. Wilcut, W. P. Bowen, A. S. Parkins, T. J. Kippenberg, K. J. Vahala, and H. J. Kimble, “Observation of strong coupling between one atom and a monolithic microresonator,” Nature (London) 443, 671–674 (2006).
  * [6] D. J. Alton, N. P. Stern, T. Aoki, H. Lee, E. Ostby, K. J. Vahala, and H. J. Kimble, “Strong interactions of single atoms and photons near a dielectric boundary,” Nat. Phys. 7, 159–165 (2011).
  * [7] E. Will, L. Masters, A. Rauschenbeutel, M. Scheucher, and J. Volz, “Coupling a single trapped atom to a whispering-gallery-mode microresonator,” Phys. Rev. Lett. 126, 233602 (2021).
  * [8] A. M. Armani, R. P. Kulkarni, S. E. Fraser, R. C. Flagan, and K. J. Vahala, “Label-free, single-molecule detection with optical microcavities,” Science 317, 783–787 (2007).
  * [9] J. Zhu, S. K. Ozdemir, Y.-F. Xiao, L. Li, L. He, D.-R. Chen, and L. Yang, “On-chip single nanoparticle detection and sizing by mode splitting in an ultrahigh-Q microresonator,” Nat. Photonics 4, 46–49 (2010).
  * [10] L. Shao, X.-F. Jiang, X.-C. Yu, B.-B. Li, W. R. Clements, F. Vollmer, W. Wang, Y.-F. Xiao, and Q. Gong, “Detection of single nanoparticles and lentiviruses using microcavity resonance broadening,” Adv. Mater. 25, 5616–5620 (2013).
  * [11] M. D. Baaske, M. R. Foreman, and F. Vollmer, “Single-molecule nucleic acid interactions monitored on a label-free microcavity biosensor platform,” Nat. Nanotechnol. 9, 933–939 (2014).
  * [12] Y. Kim and H. Lee, “On-chip label-free biosensing based on active whispering gallery mode resonators pumped by a light-emitting diode,” Opt. Express 27, 34405–34415 (2019).
  * [13] J. M. Ward, D. G. O’Shea, B. J. Shortt, and S. Nic Chormaic, “Optical bistability in Er-Yb codoped phosphate glass microspheres at room temperature,” J. Appl. Phys. 102, 023104 (2007).
  * [14] A. Chiasera, Y. Dumeige, P. Féron, M. Ferrari, Y. Jestin, G. Nunzi Conti, S. Pelli, S. Soria, and G. C. Righini, “Spherical whispering-gallery-mode microresonators,” Laser Photonics Rev. 4, 457–482 (2010).
  * [15] D. K. Armani, T. J. Kippenberg, S. M. Spillane, and K. J. Vahala, “Ultra-high-Q toroid microcavity on a chip,” Nature (London) 421, 925–928 (2003).
  * [16] P. Rabiei, W. H. Steier, C. Zhang, and L. R. Dalton, “Polymer micro-ring filters and modulators,” J. Lightwave Technol. 20, 1968–1975 (2002).
  * [17] M. Sumetsky, Y. Dulashko, and R. S. Windeler, “Optical microbubble resonator,” Opt. Lett. 35, 898–900 (2010).
  * [18] A. Watkins, J. Ward, Y. Wu, and S. Nic Chormaic, “Single-input spherical microbubble resonator,” Opt. Lett. 36, 2113–2115 (2011).
  * [19] G. S. Murugan, M. N. Petrovich, Y. Jung, J. S. Wilkinson, and M. N. Zervas, “Hollow-bottle optical microresonators,” Opt. Express 19, 20773–20784 (2011).
  * [20] J. M. Ward, Y. Yang, F. Lei, X. C. Yu, Y. F. Xiao, and S. Nic Chormaic, “Nanoparticle sensing beyond evanescent field interaction with a quasi-droplet microcavity,” Optica 5, 674–677 (2018).
  * [21] F. Pan, K. Karlsson, A. G. Nixon, L. T. Hogan, J. M. Ward, K. C. Smith, D. J. Masiello, S. Nic Chormaic, and R. H. Goldsmith, “Active control of plasmonic–photonic interactions in a microbubble cavity,” J. Phys. Chem. C 126, 20470–20479 (2022).
  * [22] Y. Yang, X. Jiang, S. Kasumie, G. Zhao, L. Xu, J. M. Ward, L. Yang, and S. Nic Chormaic, “Four-wave mixing parametric oscillation and frequency comb generation at visible wavelengths in a silica microbubble resonator,” Opt. Lett. 41, 5266–5269 (2016).
  * [23] Y. Yang, Y. Ooka, R. M. Thompson, J. M. Ward, and S. Nic Chormaic, “Degenerate four-wave mixing in a silica hollow bottle-like microresonator,” Opt. Lett. 41, 575–578 (2016).
  * [24] S. Kasumie, Y. Yang, J. M. Ward, and S. Nic Chormaic, “Towards visible frequency comb generation using a hollow WGM resonator,” Rev. Laser Engin. 46, 92–96 (2018).
  * [25] Y. Yin, Y. Niu, H. Qin, and M. Ding, “Kerr frequency comb generation in microbottle resonator with tunable zero dispersion wavelength,” J. Lightwave Technol. 37, 5571–5575 (2019).
  * [26] R. Henze, T. Seifert, J. Ward, and O. Benson, “Tuning whispering gallery modes using internal aerostatic pressure,” Opt. Lett. 36, 4536–4538 (2011).
  * [27] A. Cosci, F. Quercioli, D. Farnesi, S. Berneschi, A. Giannetti, F. Cosi, A. Barucci, G. Nunzi Conti, G. Righini, and S. Pelli, “Confocal reflectance microscopy for determination of microbubble resonator thickness,” Opt. Express 23, 16693–16701 (2015).
  * [28] Q. Lu, J. Liao, S. Liu, X. Wu, L. Liu, and L. Xu, “Precise measurement of micro bubble resonator thickness by internal aerostatic pressure sensing,” Opt. Express 24, 20855–20861 (2016).
  * [29] Y. Yang, S. Saurabh, J. M. Ward, and S. Nic Chormaic, “High-Q, ultrathin-walled microbubble resonator for aerostatic pressure sensing,” Opt. Express 24, 294–299 (2016).
  * [30] F. Lei, J. M. Ward, P. Romagnoli, and S. Nic Chormaic, “Polarization-controlled cavity input-output relations,” Phys. Rev. Lett. 124, 103902 (2020).
  * [31] J. Yu, J. Zhang, R. Wang, A. Li, M. Zhang, S. Wang, P. Wang, J. M. Ward, and S. Nic Chormaic, “A tellurite glass optical microbubble resonator,” Opt. Express 28, 32858–32868 (2020).
  * [32] J. Jiang, Y. Liu, K. Liu, S. Wang, Z. Ma, Y. Zhang, P. Niu, L. Shen, and T. Liu, “Wall-thickness-controlled microbubble fabrication for WGM-based application,” Appl. Opt. 59, 5052–5057 (2020).
  * [33] M. Sumetsky, “Whispering-gallery-bottle microcavities: The three-dimensional etalon,” Opt. Lett. 29, 8–10 (2004).
  * [34] L. Ma, S. Li, V. M. Fomin, M. Hentschel, J. B. Götte, Y. Yin, M. R. Jorgensen, and O. G. Schmidt, “Spin-orbit coupling of light in asymmetric microcavities.” Nat. Commun. 7, 10983 (2016).
  * [35] J. Wang, S. Valligatla, Y. Yin, L. Schwarz, M. Medina-Sánchez, S. Baunack, C. H. Lee, R. Thomale, S. Li, V. M. Fomin, L. Ma, and O. G. Schmidt, “Experimental observation of berry phases in optical möbius-strip microcavities,” Nat. Photonics 17, 120–125 (2023).
  * [36] C. Strelow, C. M. Schultz, H. Rehberg, M. Sauer, H. Welsch, A. Stemmann, C. Heyn, D. Heitmann, and T. Kipp, “Light confinement and mode splitting in rolled-up semiconductor microtube bottle resonators,” Phys. Rev. B 85, 155329 (2012).
  * [37] M. Sumetsky, “Optical bottle microresonators,” Prog. Quantum Electron. 64, 1–30 (2019).
  * [38] C. Strelow, H. Rehberg, C. M. Schultz, H. Welsch, C. Heyn, D. Heitmann, and T. Kipp, “Optical microcavities formed by semiconductor microtubes using a bottlelike geometry,” Phys. Rev. Lett. 101, 127403 (2008).
  * [39] S. Li, L. Ma, S. Böttner, Y. Mei, M. R. Jorgensen, S. Kiravittaya, and O. G. Schmidt, “Angular position detection of single nanoparticles on rolled-up optical microcavities with lifted degeneracy,” Phys. Rev. A 88, 033833 (2013).
  * [40] Y. Fang, S. Li, and Y. Mei, “Modulation of high quality factors in rolled-up microcavities,” Phys. Rev. A 94, 033804 (2016).
  * [41] Z. Tian, S. Li, S. Kiravittaya, B. Xu, S. Tang, H. Zhen, W. Lu, and Y. Mei, “Selected and enhanced single whispering-gallery mode emission from a mesostructured nanomembrane microcavity,” Nano Lett. 18, 8035–8040 (2018).