illustration of a non-$i.i.d.$ version, based on draws from an MH candidate
distribution, $q(\boldsymbol{\cdot}|\mathbf{y})$.

The general principle of IS has, of course, two aspects to it: i) the use of
draws simulated from the importance density to compute a weighted mean; and
ii) evaluation of the importance density in the weight. The ‘reciprocal IS’
(RIS) method (Gelfand and Dey, , 1994; Frühwirth-Schnatter, , 2004) uses the
second aspect, whilst taking draws from the posterior itself. Simple
calculations can be used to show that, for some
$q(\boldsymbol{\cdot}|\mathbf{y})$ that is contained in the support of
$p(\boldsymbol{\theta}_{k}\mathbf{|y})$, and defining
$g(\boldsymbol{\theta}_{k}\mathbf{)}=q(\boldsymbol{\theta}_{k}\boldsymbol{|}\mathbf{y})/\left[p(\mathbf{y}|\boldsymbol{\theta}_{k})p(\boldsymbol{\theta}_{k})\right]$,

$\mathbb{E}(g(\boldsymbol{\theta}_{k}\mathbf{)|y)}=\left[p(\mathbf{y}|\mathcal{M}_{k})\right]^{-1}.$
(34)

Again, under stringent regularity conditions, including some on the support of
$q(\boldsymbol{\theta}_{k}\boldsymbol{|}\mathbf{y})$, draws from
$p(\boldsymbol{\theta}_{k}\mathbf{|y})$, can be used to estimate (34), and its
reciprocal used as an estimate of (33) itself. The ‘harmonic mean estimator’
of Newton and Raftery,  (1994) is a special, if infamous, case of (34) in
which
$q(\boldsymbol{\theta}_{k}\boldsymbol{|}\mathbf{y})=p(\boldsymbol{\theta}_{k})$.
The complete untrustworthiness of the output when the prior has fatter tails
than the posterior (Neal, , 1994, 1999) have led to this method being largely
eschewed in the literature. The ‘bridge sampler’, on the other hand, provides
a more robust version of (34) by exploiting draws from both the posterior and
the weight density, $q(\boldsymbol{\theta}_{k}\boldsymbol{|}\mathbf{y})$ (Meng
and Wong, , 1996; Meng and Schilling, , 2002; Frühwirth-Schnatter, , 2004;
Gronau _et al._ , , 2017). More recent versions of the RIS method retain draws
from the posterior, but use VB approximations of
$p(\boldsymbol{\theta}_{k}\mathbf{|y})$ to define
$q(\boldsymbol{\theta}_{k}\boldsymbol{|}\mathbf{y})$ (Fourment _et al._ , ,
2018; Hajargasht and Woźniak, , 2018).

Finally, building on the thermodynamic integration method from theoretical
physics, Gelman and Meng,  (1998) develop the ‘path sampler’. They illustrate
the sense in which the earlier IS schemes for estimating (33), followed by
bridge sampling, and then path sampling, represent a natural methodological
lineage. See also Rischard _et al._ ,  (2018) for a recent amalgam of path
sampling with unbiased MCMC, and Neal,  (2001) for related work using
‘annealed importance sampling’ to estimate (33).232323On a related thread,
note that SMC has often been advocated for estimating the evidence (Doucet _et
al._ , , 2000; Fiel and Wyse, , 2012; Everitt _et al._ , , 2020). Similarly,
techniques incorporating the intractable marginal likelihood as a
supplementary parameter can be traced back to Geyer,  (1993), with more recent
occurrences like noise-contrastive estimation (Gutmann and Hyvärinen, , 2012),
being based on a so-called ‘logistic trick’ that turns the approximation of
the evidence into the estimation of the intercept of a logistic classification
program. See also Barthelmé and Chopin,  (2015) and Lyne _et al._ ,  (2015)
for related work.

####  6.1.2 Multiple runs of MCMC

Given the definition of $p(\boldsymbol{\theta}_{k}\mathbf{|y})$, we can
produce a representation of the marginal likelihood as:

$p(\mathbf{y}|\mathcal{M}_{k})=\frac{p(\mathbf{y}|\boldsymbol{\theta}_{k})p(\boldsymbol{\theta}_{k})}{p(\boldsymbol{\theta}_{k}\mathbf{|y})}.$
(35)

The insight of Chib (1995) was to recognize that (35) holds for all
$\boldsymbol{\theta}$. Hence, an estimate of $p(\mathbf{y}|\mathcal{M}_{k})$
is simply produced as:
$\widehat{p}(\mathbf{y}|\mathcal{M}_{k})=p(\mathbf{y}|\boldsymbol{\theta}_{k}^{\ast})p(\boldsymbol{\theta}_{k}^{\ast})/p(\boldsymbol{\theta}_{k}^{\ast}\mathbf{|y})$,
where the convention is to take $\boldsymbol{\theta}_{k}^{\ast}$ as some high
posterior value. Whilst the (common) availability of the likelihood and prior
in closed form renders the ordinates of the factors in the numerator readily
accessible in most cases, the denominator and, indeed, the value of
$\boldsymbol{\theta}_{k}^{\ast}$ itself are, by the very nature of the
problem, inaccessible without further work. However, defining
$\boldsymbol{\theta}_{k}^{\ast}=(\theta_{k,1}^{\ast},\theta_{k,2}^{\ast},...,\theta_{k,p_{k}}^{\ast})^{\prime}$,
the joint posterior (evaluated at $\boldsymbol{\theta}_{k}^{\ast}$) can be
decomposed as:

$p(\boldsymbol{\theta}_{k}^{\ast}\mathbf{|y})=p(\theta_{k,1}^{\ast}|\theta_{k,2}^{\ast},...,\theta_{k,p_{k}}^{\ast},\mathbf{y})p(\theta_{k,2}^{\ast}|\theta_{k,3}^{\ast},...,\theta_{k,p_{k}}^{\ast},\mathbf{y})...p(\theta_{k,p_{k}}^{\ast}|\mathbf{y}).$
(36)

The last term on the right-hand-side of (36) can simply be estimated in the
usual way using a full run of an MCMC sampler. The remaining conditionals can
be estimated from additional applications of the same simulation scheme, but
with the appropriate sets of parameters held fixed. Modification of the
original (pure Gibbs) approach proposed in Chib,  (1995) to cater for full
conditionals that are not available in closed form, by using MH steps, is
detailed in Chib and Jeliazkov,  (2001). Recent work in Chib _et al._ ,
(2020) highlights the effective use of parallelization to reduce the
computational cost of the additional ‘reduced’ MCMC runs.

####  6.1.3 Nested sampling

Nested sampling (Skilling, , 2007) is yet another method for producing a
simulation-based estimate of $p(\mathbf{y}|\mathcal{M}_{k}).$ Whilst it gained
an immediate foothold in astronomy (Mukherjee _et al._ , , 2006) — possibly
due to the availability of dedicated software like MultiNest and Dynesty — and
remains popular in that field (Feroz _et al._ , , 2019), it has not gained
wide acceptance beyond that field. A cartoon description of the method is as
the simulation version of Lebesgue integration, in that $M$ points are
simulated on slices of the likelihood function (delineated by points $t,$
$t-1$ in the support of $\boldsymbol{\theta}$), $\left\\{\boldsymbol{\theta};\
p(\mathbf{y}|\boldsymbol{\theta}_{t-1})\leq
p(\mathbf{y}|\boldsymbol{\theta})\leq
p(\mathbf{y}|\boldsymbol{\theta}_{t})\right\\},$ with each slice having
approximately a prior probability, $\exp\\{-(t-1)/M\\}$, of occurring. As
first shown in Chopin and Robert,  (2010), nested sampling is a Monte Carlo
method with a $\sqrt{M}$ rate of convergence, whose performance relies on the
ability to efficiently simulate parameters within the above slices, which is
challenging when these are not connected.

###  6.2 Reversible Jump MCMC

All methods described above for computing $p(\mathbf{y}|\mathcal{M}_{k})$ have
one thing in common: the marginal likelihood for each model is tackled as a
separate computational exercise. Once each $p(\mathbf{y}|\mathcal{M}_{k})$ is
computed, the posterior model probabilities follow, and model choice, or model
averaging, can proceed.

Alternatively, uncertainty about the model can be used to augment the
unknowns, and a posterior sampler designed to target this augmented space.
This is the principle adopted in a range of papers, including those that focus
on variable selection in regression models, and the number of components in
finite mixture models; and we refer the reader to George,  (2000), Marin _et
al._ ,  (2005) and Chib,  (2011) for reviews and references. We focus here on
one particular approach only: that of Green,  (1995).

Green,  (1995) characterizes the problem of an unknown model as one in which
the dimension of the (model-specific) unknowns varies, depending on which
model is in play. He thus designs an MCMC sampler that is allowed to jump
between parameter spaces of differing dimensions; coining the term:
‘reversible jump MCMC’ (RJMCMC). At its core though, Green’s approach is one
of augmentation, and can be viewed as a particular application of the original
idea of Tanner and Wong,  (1987), with the extra complexity of dimension
variation as the sampler traverses the augmented space.

In brief, Green,  (1995) assumes a countable collection of candidate models
$\mathcal{M}=\\{\mathcal{M}_{1},\mathcal{M}_{2},...\\}$, indexed by
$k=1,2,.....$ Each model has a $p_{k}$-dimensional set of unknown parameters
$\boldsymbol{\theta}_{k}$. Using obvious notation for the likelihood and prior
for the $kth$ model, and the prior $p(k)$ for the model index itself, the
target of the RJMCMC algorithm is then:
$p(k,\boldsymbol{\theta}_{k}|\mathbf{y})=p(\mathbf{y}|k,\boldsymbol{\theta}_{k})p(\boldsymbol{\theta}_{k}|k)p(k)/p(\mathbf{y})$.
The RJMCMC sampler moves between any two parameters spaces by creating
temporary auxiliary variables that bring the dimensions of the augmented
spaces to be equal, with a reversibility constraint on the proposed moves
between these models. Such draws from the joint space of
$\\{k,\boldsymbol{\theta}_{k}\\}$ can be used to compute any particular
$\mathbb{E}(g(\boldsymbol{\theta}_{k}\mathbf{)|y)}$ of interest. Indeed, the
draws can also be used to compute an expectation of the form:
$\mathbb{E}(g(k)\mathbf{|y)}$, which nests the marginal posterior probability
attached to the $kth$ model: $p(k\mathbf{|y)}$. That is, posterior model
probabilities are an automatic outcome of the simulation scheme. Moreover, the
computation of any expectation of interest incorporates all uncertainty
associated with both the parameters of each model and the model itself; hence
model averaging is automatic. See Green,  (2003), Fan and Sisson,  (2011) and
Geyer, 2011b for reviews of RJMCMC, including: its theoretical and
implementation properties, its links to other ‘multi-model’ samplers, and the
scope of its application.

###  6.3 Computation in Bayesian Prediction

Conditional on the assumed model that underpins the likelihood function being
‘correctly specified’ — i.e. $p(\mathbf{y}|\boldsymbol{\theta})$ coinciding
with the true data generating process (DGP) — the ‘gold standard’ for Bayesian
prediction is the specification of (4) with $g(\boldsymbol{\theta}\mathbf{)}=$
$p(y_{n+1}^{\ast}|\boldsymbol{\theta}\mathbf{,y})$, which yields:

$p(y_{n+1}^{\ast}|\mathbf{y})=\int_{{\Theta}}p(y_{n+1}^{\ast}|\boldsymbol{\theta}\mathbf{,y})p(\boldsymbol{\theta}\mathbf{|y})d\boldsymbol{\theta}.$
(37)

The distribution $p(y_{n+1}^{\ast}|\mathbf{y})$ summarizes all uncertainty
about $y_{n+1}$, conditional on both the assumed model — which underpins the
structure of both the conditional predictive,
$p(y_{n+1}^{\ast}|\boldsymbol{\theta}\mathbf{,y})$, and the posterior itself —
and the prior beliefs that inform $p(\boldsymbol{\theta}\mathbf{|y}).$ Point
and interval predictions of $y_{n+1}$, and indeed any other distributional
summary, can be extracted from (37). In the case where the model itself is
uncertain, and a finite set of models, $\mathcal{M}_{1}$,
$\mathcal{M}_{2}$,…,$\mathcal{M}_{K},$ is assumed to span the model space, the
principle of model averaging can be used to produce a ‘model-averaged’
predictive, $p_{MA}(y_{n+1}^{\ast}|\mathbf{y})$ as

$p_{MA}(y_{n+1}^{\ast}|\mathbf{y})=\sum\limits_{k=1}^{K}p(y_{n+1}^{\ast}|\mathbf{y},\mathcal{M}_{k})p(\mathcal{M}_{k}|\mathbf{y})\text{,}$
(38)

where $p(y_{n+1}^{\ast}|\mathbf{y},\mathcal{M}_{k})$ denotes the density in
(37), but conditioned explicitly on the $kth$ model in the set. In the typical
case where (37) and (38) are unavailable analytically, any of the
computational methods that have been discussed thus far could be used to
compute either $p(y_{n+1}^{\ast}|\mathbf{y})$ or
$p_{MA}(y_{n+1}^{\ast}|\mathbf{y})$, and any summaries thereof. In some ways
then, this could be viewed as completing the Bayesian prediction story.

However, Bayesian prediction is more than just a special case of the general
Bayesian computational problem defined by (4). Predicting outcomes that have
not yet been observed is arguably the most stringent test to which any
statistical methodology can be put. Moreover, out-of-sample accuracy is the
ultimate arbiter in this setting, and any computational method used to produce
predictions needs to be assessed in this light.

Such is the motivation of recent works that pose the following question: In
cases (such as those highlighted in Section 5.3) where the exact posterior is
inaccessible and the exact predictive in (37) is thus also unavailable, what
are the implications for predictive accuracy of adopting an approximation to
$p(\boldsymbol{\theta}\mathbf{|y})$ and, hence,
$p(y_{n+1}^{\ast}|\mathbf{y})$? Frazier _et al._ , 2019a , for instance,
document that an ‘approximate predictive’ produced by replacing
$p(\boldsymbol{\theta}\mathbf{|y})$ with an ABC-based posterior, is
numerically indistinguishable (in the cases investigated) from the exact
predictive and, thus, yields equivalent predictive accuracy. Further, under
certain regularity conditions, the exact and approximate predictives are shown
to be asymptotically (in $n$) equivalent. The tenor of related work exploring
prediction in approximate inference (including VB) settings — Park and Nassar,
(2014), Canale and Ruggiero,  (2016), Koop and Korobilis,  (2018), Quiroz _et
al._ , 2018a and Kon Kam King _et al._ ,  (2019) — is somewhat similar to that
of Frazier _et al._ , 2019a; that is, computing
$p(\boldsymbol{\theta}\mathbf{|y})$ via an approximate method does not
necessarily reduce predictive accuracy.

However, an arguably more challenging question is: How does one even think
about Bayesian prediction — and the use of computation therein — once one
acknowledges that, in reality: i) any given model is misspecified; and ii) any
finite set of models does not span the truth? The ‘probably approximately
correct (PAC)-Bayes’ approach in machine learning (reviewed in Guedj, , 2019)
replaces the likelihood function in the Bayesian update with the exponential
of a general loss function, with a view to producing predictions that are more
targeted to the problem at hand, rather than being tied to a particular model
specification. This approach yields so-called ‘Gibbs posteriors’ (Zhang,
2006a, ; Zhang, 2006b, ; Jiang and Tanner, , 2008), and mimics — in a
prediction setting — the generalized inferential methods proposed in (inter
alia) Bissiri _et al._ ,  (2016), Giummolè _et al._ ,  (2017), Holmes and
Walker,  (2017), Lyddon _et al._ ,  (2019) and Syring and Martin,  (2019). In
addition, PAC-Bayes uses updates based on ‘tempered’, or ‘power’ likelihoods,
in which robustness to model misspecification is sought by raising the
likelihood associated with an assumed model to a particular power.

Other work tackles this issue by estimating weighted combinations of
predictives via either forecast accuracy criteria or the criterion of
predictive calibration (Dawid, , 1982; Gneiting _et al._ , , 2007), without
assuming that the true model lies within the set of constituent predictives
(Billio _et al._ , , 2013; Casarin _et al._ , , 2015; Pettenuzzo and
Ravazzolo, , 2016; Bassetti _et al._ , , 2018; Baştürk _et al._ , , 2019). In
a similar spirit, Loaiza-Maya _et al._ ,  (2020) propose the use of focused
Bayesian prediction, in which the Bayesian update is driven by a criterion
that captures a user-specified measure of predictive accuracy. The authors do
indeed find that focusing on the loss that matters produces superior
predictive performance relative to using a misspecified likelihood update.

In summary, once attention is on the particular version of (4) that yields a
predictive distribution, it is the updating criterion function itself that is
increasingly being viewed as key, with uncertainty about the unknown
parameters — and the computational method used to quantify that uncertainty —
assuming a somewhat secondary role.

##  7 The Future

Our journey with Bayesian computation began in 1763: with a posterior
probability defined in terms of a scalar $\theta$, whose solution challenged
Bayes. We now end our journey in 2020: having referenced posterior
distributions defined over thousands, possibly millions of unknowns, and
computational problems with a degree of complexity — and scale — to match.
Along the way, we have seen the huge variety of imaginative computational
solutions that have been brought to bear on all such problems, over the span
of 250 years. A natural question to ask then is: ‘what is there left to do?’

Judging from the wealth of contributions to ‘Bayes Comp 2020’, in January 2020
(http://users.stat.
ufl.edu/~jhobert/BayesComp2020/Conf_Website/), the answer to that question is:
‘a great deal!’; and most certainly as pertains to matters of scale. Indeed,
of the 129 abstracts recorded on the conference website (and acknowledging
some double counting), 16 make explicit use of the term scalability (or
something comparable); 10 refer to the ability of a proposed computational
method to deal effectively with large data sets, and 22 refer to high-
dimensional problems of one form or another. There are attempts to scale most
categories of computational methods reviewed in this paper, and the scope of
the empirical problems to which these advances are applied — from
probabilistic topic models, health studies on hypertension and sepsis,
problems in neuroimaging, genomics, biology, epidemiology, ecology, psychology
and econometrics, through to probabilistic record linkage and geographic
profiling — is extremely broad, highlighting again the critical importance of
Bayesian computation to diverse fields.

But if scale may be viewed as a key focus of this selection of research, there
is another theme that can also be discerned. A second glance at these
conference proceedings, and at recent journal publications, pinpoints a
growing interest in the impact of model misspecification on computational
methods; specifically: 1) what are the implications for computation if an
assumed parametric model is misspecified?; and, 2) what are the implications
for computation if the conventional likelihood-based paradigm is eschewed
altogether (as is already happening in some prediction settings)? Remembering
that our interest here is on the implications for computation per se of both
misspecified and non-likelihood settings, we note that specific attempts to
address 1) and 2) directly are still quite small in number, if
growing.242424We refer to Kleijn and van der Vaart,  (2012) and Muller,
(2013) for general treatments of Bayesian likelihood-based inference in
misspecified models; to some of the literature cited in Section 6.3, plus
Chernozhukov and Hong,  (2003),Gallant,  (2016), Chib _et al._ ,  (2018), and
Miller and Dunson,  (2019), for various generalizations of the standard
Bayesian paradigm, including posterior up-dates driven by problem-specific
loss (or moment) functions. It can be argued, however, that in none of these
papers are the implications of model misspecification and/or non-likelihood
up-dates for computation per se the primary focus. We complete our review by
briefly summarizing five recent (sets of) papers in this vein:

  1. i)

First, Lyddon _et al._ ,  (2019) and Syring and Martin,  (2019) use bootstrap
principles to compute so-called general Bayesian posteriors, in which the
likelihood function associated with an assumed (and potentially misspecified)
model is replaced by a more general loss function that is not tied to a
particular model specification. Huggins and Miller,  (2019) also use the
bootstrap to construct ‘bagged’ posteriors (BayesBag), and thereby conduct
Bayesian inference that is robust to model misspecification.

  2. ii)

Second, Frazier _et al._ ,  (2020) analyze the theoretical properties of ABC
under model misspecification; outlining when ABC concentrates posterior mass
on an appropriately defined pseudo-true value, and when it does not. The
nonstandard asymptotic behaviour of the ABC posterior, including its failure
to yield credible sets with valid frequentist coverage, is highlighted. The
authors also devise techniques for diagnosing model misspecification in the
context of ABC. Frazier and Drovandi,  (2019) devise a version of BSL that is
robust to model misspecification, and demonstrate that this version can be
much more computationally efficient than standard BSL when the model is
misspecified.

  3. iii)

Third, Wang and Blei, 2019a investigate VB under model misspecification. They
demonstrate that the VB posterior both concentrates on the value that
minimizes the Kullback-Leibler (KL) divergence from the true DGP, and is
asymptotically normal; as is the VB posterior mean. These results generalize
the asymptotic results for VB of Wang and Blei, 2019b , derived under correct
specification, to the misspecification case.

  4. iv)

Fourth, Knoblauch _et al._ ,  (2019) propose what they term generalized
variational inference, by extending the specification of the Bayesian paradigm
to accommodate general loss functions (thereby avoiding the reliance on
potentially misspecified likelihoods) and building an optimization-based
computational tool within that setting.

  5. v)

Fifth, building on earlier work in the context of HMC (and which is cited
therein), Bornn _et al._ ,  (2019) derive an MCMC scheme for sampling on the
lower-dimensional manifold implied by the moment conditions that they adopt
within a Bayesian framework. Whilst this work embeds the moments within a
nonparametric Bayesian set-up — and we have not discussed computation in
nonparametric settings in this review — we make note of this work as an
example of a fundamental shift in computational design that is required when
moving to a particular non-likelihood Bayesian up-date. Whilst not motivated
by this same problem, the work on extending VB to manifolds by Tran _et al._ ,
(2019) is also relevant here.

Bayesian computation in parametric models is thus beginning to confront — and
adapt to — the reality of misspecified DGPs, and the generalizations beyond
the standard likelihood-based up-date that are evolving. Allied with the
growing ability of computational methods to also deal with the scale of modern
problems, the future of the paradigm in the 21st century thus seems assured.
And with this, the 18th century Bayes (and his loyal champion, Price) would no
doubt be duly impressed!

## References

  * Albert and Chib, (1993) Albert, J. and Chib, S. (1993).  Bayesian analysis of binary and polychotomous response data.  J. American Statist. Assoc., 88:669–679.
  * Andrieu _et al._ , (2011) Andrieu, C., Doucet, A., and Holenstein, R. (2011).  Particle Markov chain Monte Carlo.  J. Royal Statist. Society Series B, 72(2):269–342.  With discussion.
  * Andrieu _et al._ , (2004) Andrieu, C., Doucet, A., and Robert, C. (2004).  Computational advances for and from Bayesian analysis.  Statist. Science, 19(1):118–127.
  * Andrieu and Roberts, (2009) Andrieu, C. and Roberts, G. (2009).  The pseudo-marginal approach for efficient Monte Carlo computations.  Ann. Statist., 37(2):697–725.
  * Baker _et al._ , (2019) Baker, J., Fearnhead, P., Fox, E., and Nemeth, C. (2019).  Control variates for stochastic gradient mcmc.  Statist. Comp., 29:599–615.
  * Banterle _et al._ , (2019) Banterle, M., Grazian, C., Lee, A., and Robert, C. P. (2019).  Accelerating Metropolis-Hastings algorithms by delayed acceptance.  Foundations of Data Science, 1(2):103–128.
  * Bardenet _et al._ , (2017) Bardenet, R., Doucet, A., and Holmes, C. (2017).  On Markov chain Monte Carlo methods for tall data.  J. Machine Learning Res., 18(1):1515–1557.
  * Barnard and Bayes, (1958) Barnard, G. and Bayes, T. (1958).  Studies in the history of probability and statistics: IX. Thomas Bayes’s essay towards solving a problem in the doctrine of chances.  Biometrika, 45(3/4):293–315.
  * Barthelmé _et al._ , (2018) Barthelmé, S., Chopin, N., and Cottet, V. (2018).  Divide and conquer in ABC: Expectation-propagation algorithms for likelihood-free inference.  Handbook of Approximate Bayesian Computation, pages 415–34.  Chapman & Hall/CRC. Eds. Sisson, S., Fan, Y., Beaumont, M.
  * Barthelmé and Chopin, (2014) Barthelmé, S. and Chopin, N. (2014).  Expectation propagation for likelihood-free inference.  J. American Statist. Assoc., 109(505):315–333.
  * Barthelmé and Chopin, (2015) Barthelmé, S. and Chopin, N. (2015).  The Poisson transform for unnormalised statistical models.  Stat. Comput., 25:767–780.
  * Bassetti _et al._ , (2018) Bassetti, F., Casarin, R., and Ravazzolo, F. (2018).  Bayesian nonparametric calibration and combination of predictive distributions.  J. American Statist. Assoc., 113(522):675–685.
  * Bauwens and Richard, (1985) Bauwens, L. and Richard, J. (1985).  A 1-1 Poly-$t$ random variable generator with application to Monte Carlo integration.  J. Econometrics, 29(1):19–46.
  * Bayes, (1764) Bayes, T. (1764).  An essay towards solving a problem in the doctrine of chances.  Philosophical Transactions of the Royal Society of London for 1973, 53:370–418.
  * Baştürk _et al._ , (2019) Baştürk, N., Borowska, A., Grassi, S., Hoogerheide, L., and van Dijk, H. (2019).  Forecast density combinations of dynamic models and data driven portfolio strategies.  Journal of Econometrics, 210(1):170–186.
  * Beaumont, (2003) Beaumont, M. (2003).  Estimation of population growth or decline in genetically monitored populations.  Genetics, 164:1139–1160.
  * Beaumont, (2010) Beaumont, M. (2010).  Approximate Bayesian computation in evolution and ecology.  Annual Review of Ecology, Evolution, and Systematics, 41:379–406.
  * Beaumont _et al._ , (2009) Beaumont, M., Cornuet, J.-M., Marin, J.-M., and Robert, C. (2009).  Adaptive approximate Bayesian computation.  Biometrika, 96(4):983–990.
  * Beaumont _et al._ , (2002) Beaumont, M., Zhang, W., and Balding, D. (2002).  Approximate Bayesian computation in population genetics.  Genetics, 162(4):2025–2035.
  * Bédard _et al._ , (2012) Bédard, M., Douc, R., and Moulines, E. (2012).  Scaling analysis of multiple-try MCMC methods.  Stochastic Processes and their Applications, 122(3):758–786.
  * Berger, (1985) Berger, J. (1985).  Statistical Decision Theory and Bayesian Analysis.  Springer-Verlag, New York, second edition.
  * Bernton _et al._ , (2019) Bernton, E., Jacob, P. E., Gerber, M., and Robert, C. P. (2019).  Approximate Bayesian computation with the wasserstein distance.  J. Royal Statist. Society Series B, 81(2):235–269.
  * Besag, (1974) Besag, J. (1974).  Spatial interaction and the statistical analysis of lattice systems.  J. Royal Statist. Society Series B, 36(2):192–326.  With discussion.
  * Besag and Green, (1993) Besag, J. and Green, P. (1993).  Spatial statistics and Bayesian computation.  J. Royal Statist. Society Series B, 55(1):25–37.  With discussion.
  * Betancourt, (2018) Betancourt, M. (2018).  A conceptual introduction to Hamiltonian Monte Carlo.  https://arxiv.org/abs/1701.02434v2.
  * Bierkens _et al._ , (2018) Bierkens, J., Bouchard-Côté, A., Doucet, A., Duncan, A. B., Fearnhead, P., Lienart, T., Roberts, G., and Vollmer, S. J. (2018).  Piecewise deterministic Markov processes for scalable Monte Carlo on restricted domains.  Statist. Prob. Letters, 136:148–154.
  * Bierkens _et al._ , (2019) Bierkens, J., Fearnhead, P., and Roberts, G. (2019).  The zig-zag process and super-efficient sampling for Bayesian analysis of big data.  Ann. Statist., 47(3):1288–1320.
  * Billio _et al._ , (2013) Billio, M., Casarin, R., Ravazzolo, F., and van Dijk, H. (2013).  Time-varying combinations of predictive densities using nonlinear filtering.  Journal of Econometrics, 177(2):213–232.
  * Bishop, (2006) Bishop, C. M. (2006).  Pattern Recognition and Machine Learning.  Springer, New York.
  * Bissiri _et al._ , (2016) Bissiri, P. G., Holmes, C. C., and Walker, S. G. (2016).  A general framework for updating belief distributions.  J. Royal Statist. Society Series B, 78(5):1103–1130.
  * Blei _et al._ , (2017) Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. (2017).  Variational inference: A review for statisticians.  J. American Statist. Assoc., 112(518):859–877.
  * Blum, (2010) Blum, M. (2010).  Approximate Bayesian computation: a non-parametric perspective.  J. American Statist. Assoc., 105(491):1178–1187.
  * Blum _et al._ , (2013) Blum, M. G. B., Nunes, M. A., Prangle, D., and Sisson, S. A. (2013).  A comparative review of dimension reduction methods in approximate Bayesian computation.  Statist. Science, 28(2):189–208.
  * Bornn _et al._ , (2017) Bornn, L., Pillai, N. S., Smith, A., and Woodard, D. (2017).  The use of a single pseudo-sample in approximate Bayesian computation.  Statist. Comp., 27(3):583–590.
  * Bornn _et al._ , (2019) Bornn, L., Shephard, N., and Solgi, R. (2019).  Moment conditions and Bayesian non-parametrics.  J. Royal Statist. Society Series B, 81(1):5–43.
  * Braun and McAuliffe, (2010) Braun, M. and McAuliffe, J. (2010).  Variational inference for large-scale models of discrete choice.  J. American Statist. Assoc., 105(489):324–335.
  * Brooks _et al._ , (2011) Brooks, S., Gelman, A., Jones, G., and Meng, X. (2011).  Handbook of Markov Chain Monte Carlo.  Taylor & Francis.
  * Buchholz and Chopin, (2019) Buchholz, A. and Chopin, N. (2019).  Improving approximate Bayesian computation via quasi-Monte Carlo.  J. Comput. Graph. Statist., 28(1):205–219.
  * Calvet and Czellar, (2015) Calvet, L. E. and Czellar, V. (2015).  Accurate methods for approximate Bayesian computation filtering.  J. Finan. Econometrics, 13(4):798–838.
  * Canale and Ruggiero, (2016) Canale, A. and Ruggiero, M. (2016).  Bayesian nonparametric forecasting of monotonic functional time series.  Electronic Journal of Statistics, 10(2):3265–3286.
  * Cappé _et al._ , (2004) Cappé, O., Guillin, A., Marin, J., and Robert, C. (2004).  Population Monte Carlo.  J. Comput. Graph. Statist., 13(4):907–929.
  * Carpenter _et al._ , (2017) Carpenter, B., Gelman, A., Hoffman, M., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., and Riddell, A. (2017).  Stan: A probabilistic programming language.  Journal of Statistical Software, Articles, 76(1).
  * Carter and Kohn, (1994) Carter, C. K. and Kohn, R. (1994).  On Gibbs sampling for state space models.  Biometrika, 81(3):541–553.
  * Casarin _et al._ , (2015) Casarin, R., Leisen, F., Molina, G., and ter Horst, E. (2015).  A Bayesian beta Markov random field calibration of the term structure of implied risk neutral densities.  Bayesian Analysis, 10(4):791–819.
  * Casella and George, (1992) Casella, G. and George, E. (1992).  An introduction to Gibbs sampling.  American Statist., 46:167–174.
  * Casella _et al._ , (2001) Casella, G., Lavine, M., and Robert, C. P. (2001).  Explaining the perfect sampler.  American Statist., 55(4):299–305.
  * Casella and Robert, (1996) Casella, G. and Robert, C. (1996).  Rao-Blackwellisation of sampling schemes.  Biometrika, 83(1):81–94.
  * Ceruzzi, (2003) Ceruzzi, P. (2003).  A History of Modern Computing.  MIT Press, second edition.
  * Chen _et al._ , (2011) Chen, S., Dick, J., and Owen, A. B. (2011).  Consistency of Markov chain quasi-Monte Carlo on continuous state spaces.  Ann. Statist., 39(2):673–701.
  * Chernozhukov and Hong, (2003) Chernozhukov, V. and Hong, H. (2003).  An MCMC approach to classical estimation.  J. Econometrics, 115(2):293–346.
  * Chib, (1995) Chib, S. (1995).  Marginal likelihood from the Gibbs output.  J. American Statist. Assoc., 90(432):1313–1321.
  * Chib, (2011) Chib, S. (2011).  Introduction to simulation and MCMC methods.  The Oxford Handbook of Bayesian Econometrics, pages 183–217.  OUP. Eds. Geweke, J., Koop, G. and van Dijk, H.
  * Chib and Greenberg, (1995) Chib, S. and Greenberg, E. (1995).  Understanding the Metropolis–Hastings algorithm.  American Statist., 49:327–335.
  * Chib and Greenberg, (1996) Chib, S. and Greenberg, E. (1996).  Markov chain Monte Carlo simulation methods in econometrics.  Econometric Theory, 12(3):409–431.
  * Chib and Jeliazkov, (2001) Chib, S. and Jeliazkov, I. (2001).  Marginal likelihood from the Metropolis–Hastings output.  J. American Statist. Assoc., 96(453):270–281.
  * Chib and Ramamurthy, (2010) Chib, S. and Ramamurthy, S. (2010).  Tailored randomized block MCMC methods with application to DSGE models.  Journal of Econometrics, 155(1):19 – 38.
  * Chib _et al._ , (2018) Chib, S., Shin, M., and Simoni, A. (2018).  Bayesian estimation and comparison of moment condition models.  J. American Statist. Assoc., 113(524):1656–1668.
  * Chib _et al._ , (2020) Chib, S., Sin, M., and Tan, F. (2020).  Estimating high-dimensional DSGE models with the TaRB-MH algorithm.  Technical report, Olin Business School, Washington University.
  * Chopin and Robert, (2010) Chopin, N. and Robert, C. (2010).  Properties of nested sampling.  Biometrika, 97(3):741–755.
  * Christen and Fox, (2005) Christen, J. and Fox, C. (2005).  Markov chain Monte Carlo using an approximation.  J. Comput. Graph. Statist., 14(4):795–810.
  * Clarté _et al._ , (2020) Clarté, G., Robert, C. P., Ryder, R., and Stoehr, J. (2020).  Component-wise approximate Bayesian computation via Gibbs-like steps.  Biometriak.  To appear.
  * Craiu and Meng, (2005) Craiu, R. V. and Meng, X.-L. (2005).  Multiprocess parallel antithetic coupling for backward and forward Markov chain Monte Carlo.  Ann. Statist., 33(2):661–697.
  * Craiu and Meng, (2011) Craiu, R. V. and Meng, X.-L. (2011).  Perfection within reach: exact MCMC sampling.  Handbook of Markov Chain Monte Carlo, pages 199–226.  Chapman & Hall/CRC. Eds. Brooks, S., Gelman, A., Jones, G., Meng, X-L.
  * Damien _et al._ , (1999) Damien, P., Wakefield, J., and Walker, S. (1999).  Gibbs sampling for Bayesian non-conjugate and hierarchical models by using auxiliary variables.  J. Royal Statist. Society Series B, 61(2):331–344.
  * Davis and Rabinowitz, (1975) Davis, P. and Rabinowitz, P. (1975).  Numerical Methods of Integration.  Academic Press, New York.
  * Dawid, (1982) Dawid, A. P. (1982).  The well-calibrated Bayesian.  Journal of the American Statistical Association, 77(379):605–610.
  * De Bruijn, (1961) De Bruijn, N. (1961).  Asymptotic methods in analysis. 2nd Edn., Sect. 4.2.  Amsterdam: North-Holland.
  * Dean _et al._ , (2014) Dean, T. A., Singh, S. S., Jasra, A., and Peters, G. W. (2014).  Parameter estimation for hidden Markov models with intractable likelihoods.  Scandinavian Journal of Statistics, 41(4):970–987.
  * Dehideniya _et al._ , (2019) Dehideniya, M., Overstall, A. M., Drovandi, C. C., and McGree, J. M. (2019).  A synthetic likelihood-based Laplace approximation for efficient design of biological processes.  https://arXiv:1903.04168.
  * Deligiannidis _et al._ , (2018) Deligiannidis, G., Doucet, A., and Pitt, M. K. (2018).  The correlated pseudomarginal method.  J. Royal Statist. Society Series B, 80(5):839–870.
  * Dellaportas and Kontoyiannis, (2012) Dellaportas, P. and Kontoyiannis, I. (2012).  Control variates for estimation based on reversible Markov chain Monte Carlo samplers.  J. Royal Statist. Society Series B, 74(1):133–161.
  * Devroye, (1986) Devroye, L. (1986).  Non-Uniform Random Variate Generation.  Springer-Verlag, New York.
  * Diebolt and Robert, (1994) Diebolt, J. and Robert, C. P. (1994).  Estimation of finite mixture distributions by Bayesian sampling.  J. Royal Statist. Society Series B, 56(2):363–375.
  * Dongarra and Sullivan, (2000) Dongarra, J. and Sullivan, F. (2000).  Guest editors’ introduction: The top 10 algorithms.  Computing in Science & Engineering, 2(1):22–23.
  * Douc and Robert, (2011) Douc, R. and Robert, C. P. (2011).  A vanilla Rao–Blackwellization of Metropolis–Hastings algorithms.  Ann. Statist., 39(1):261–277.
  * Doucet _et al._ , (2001) Doucet, A., de Freitas, N., and Gordon, N. (2001).  Sequential Monte Carlo Methods in Practice.  Springer-Verlag, New York.
  * Doucet _et al._ , (2000) Doucet, A., Godsill, S., and Andrieu, C. (2000).  On sequential Monte-Carlo sampling methods for Bayesian filtering.  Statist. Comp., 10:197–208.
  * Doucet _et al._ , (2015) Doucet, A., Pitt, M. K., Deligiannidis, G., and Kohn, R. (2015).  Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator.  Biometrika, 102(2):295–313.
  * Drovandi _et al._ , (2011) Drovandi, C., Pettitt, A., and Faddy, M. (2011).  Approximate Bayesian computation using indirect inference.  J. Royal Statist. Society Series A, 60(3):503–524.
  * Drovandi _et al._ , (2015) Drovandi, C. C., Pettitt, A. N., and Lee, A. (2015).  Bayesian indirect inference using a parametric auxiliary model.  Statist. Science, 30(1):72–95.
  * Dunson and Johndrow, (2019) Dunson, D. and Johndrow, J. (2019).  The Hastings algorithm at fifty.  Biometrika, 107(1):1–23.
  * Everitt _et al._ , (2020) Everitt, R., Culliford, R., Medina-Aguayo, F., and Wilson, D. (2020).  Sequential Monte Carlo with transformations.  Stat. Comput., 30:663–676.
  * Fan and Sisson, (2011) Fan, Y. and Sisson, S. A. (2011).  Reversible jump MCMC.  Handbook of Markov Chain Monte Carlo, pages 67–92.  Chapman & Hall/CRC. Eds. Brooks, S., Gelman, A., Jones, G., Meng, X-L.
  * Fearnhead, (2011) Fearnhead, P. (2011).  MCMC for state-space models.  Handbook of Markov Chain Monte Carlo, pages 513–529.  Chapman & Hall/CRC. Eds. Brooks, S., Gelman, A., Jones, G., Meng, X-L.
  * Fearnhead, (2018) Fearnhead, P. (2018).  Asymptotics of ABC.  Handbook of Approximate Bayesian Computation, pages 269–288.  Chapman & Hall/CRC. Eds. Sisson, S., Fan, Y., Beaumont, M.
  * Fearnhead _et al._ , (2018) Fearnhead, P., Bierkens, J., Pollock, M., and Roberts, G. O. (2018).  Piecewise deterministic Markov processes for continuous-time Monte Carlo.  Statist. Sci., 33(3):386–412.
  * Fearnhead and Prangle, (2012) Fearnhead, P. and Prangle, D. (2012).  Constructing summary statistics for approximate Bayesian computation: Semi-automatic approximate Bayesian computation.  J. Royal Statist. Society Series B, 74(3):419–474.  With discussion.
  * Feroz _et al._ , (2019) Feroz, F., Hobson, M. P., Cameron, E., and Pettitt, A. N. (2019).  Importance nested sampling and the multinest algorithm.  The Open Journal of Astrophysics, 2(1).
  * Fiel and Wyse, (2012) Fiel, N. and Wyse, J. (2012).  Estimating the evidence: a review.  Statistica Neerlandica, 66(3):288–308.
  * Fienberg, (2006) Fienberg, S. (2006).  When did Bayesian inference become “Bayesian“?  Bayesian Analysis, 1(1):1–40.
  * Fourment _et al._ , (2018) Fourment, M., Magee, A. F., Whidden, C., Bilge, A., Matsen, I., Frederick, A., and Minin, V. N. (2018).  19 dubious ways to compute the marginal likelihood of a phylogenetic tree topology.  https://arXiv:1811.11804.
  * Frazier, (2020) Frazier, D. T. (2020).  Robust and efficient Approximate Bayesian Computation: A minimum distance approach.  arXiv preprint arXiv:2006.14126.
  * Frazier and Drovandi, (2019) Frazier, D. T. and Drovandi, C. (2019).  Robust approximate Bayesian inference with synthetic likelihood.  https://arXiv:1904.04551.
  * (94) Frazier, D. T., Maneesoonthorn, W., Martin, G. M., and McCabe, B. P. (2019a).  Approximate Bayesian forecasting.  Intern. J. Forecasting, 35(2):521–539.
  * Frazier _et al._ , (2018) Frazier, D. T., Martin, G. M., Robert, C. P., and Rousseau, J. (2018).  Asymptotic properties of approximate Bayesian computation.  Biometrika, 105(3):593–607.
  * (96) Frazier, D. T., Nott, D. J., Drovandi, C., and Kohn, R. (2019b).  Bayesian inference using synthetic likelihood: Asymptotics and adjustments.  https://arXiv:1902.04827.
  * Frazier _et al._ , (2020) Frazier, D. T., Robert, C. P., and Rousseau, J. (2020).  Model misspecification in approximate Bayesian computation: consequences and diagnostics.  J. Royal Statist. Society Series B.
  * Frazier, (2018) Frazier, P. (2018).  A tutorial on Bayesian optimization.  https://arxiv.org/abs/1807.02811.
  * Frigessi _et al._ , (2000) Frigessi, A., Gasemyr, J., and Rue, H. (2000).  Antithetic coupling of two Gibbs sampler chains.  The Annals of Statistics, 28(4):1128–1149.
  * Frühwirth-Schnatter, (1994) Frühwirth-Schnatter, S. (1994).  Data augmentation and dynamic linear models.  J. Time Ser. Anal., 15(2):183–202.
  * Frühwirth-Schnatter, (2004) Frühwirth-Schnatter, S. (2004).  Estimating marginal likelihoods for mixture and Markov switching models using bridge sampling techniques.  The Econometrics Journal, 7(1):143–167.
  * Gallant, (2016) Gallant, A. R. (2016).  Reflections on the probability space induced by moment conditions with implications for Bayesian inference.  J. Finan. Econometrics, 14(2):229–247.
  * Gallant and Tauchen, (1996) Gallant, A. R. and Tauchen, G. (1996).  Which moments to match?  Econometric theory, 12(4):657–681.
  * Gelfand and Dey, (1994) Gelfand, A. and Dey, D. (1994).  Bayesian model choice: Asymptotics and exact calculations.  J. Royal Statist. Society Series B, 56(3):501–514.
  * Gelfand and Smith, (1990) Gelfand, A. and Smith, A. (1990).  Sampling based approaches to calculating marginal densities.  J. Amer. Statist. Assoc., 85(410):398–409.
  * Gelman and Meng, (1998) Gelman, A. and Meng, X. (1998).  Simulating normalizing constants: From importance sampling to bridge sampling to path sampling.  Statist. Science, 13:163–185.
  * Gelman and Shirley, (2011) Gelman, A. and Shirley, K. (2011).  Inference from simulations and monitoring convergence.  Handbook of Markov chain Monte Carlo, pages 163–174.  Chapman & Hall/CRC. Eds. Brooks, S., Gelman, A., Jones, G., Meng, X-L.
  * Geman and Geman, (1984) Geman, S. and Geman, D. (1984).  Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images.  IEEE Trans. Pattern Anal. Mach. Intell., 6:721–741.
  * George, (2000) George, E. (2000).  The variable selection problem.  J. American Statist. Assoc., 95(452):1304–1308.
  * Gerber and Chopin, (2015) Gerber, M. and Chopin, N. (2015).  Sequential quasi Monte Carlo.  J. Royal Statist. Society Series B, 77(3):509–579.
  * (111) Geweke, J. (1989a).  Bayesian inference in econometric models using Monte Carlo integration.  Econometrica, 57(6):1317–1340.
  * (112) Geweke, J. (1989b).  Exact predictive densities for linear models with ARCH disturbances.  J. Econometrics, 40(1):63–86.
  * Geweke, (1999) Geweke, J. (1999).  Using simulation methods for Bayesian econometric models: Inference, development, and communication.  Econometric Reviews, 18(1):1–73.  With discussion.
  * Geweke _et al._ , (2011) Geweke, J., Koop, G., and van Dijk, H. (2011).  The Oxford Handbook of Bayesian Econometrics.  OUP.
  * Geyer, (1991) Geyer, C. (1991).  Markov chain Monte Carlo maximum likelihood.  In Keramigas, E., editor, Computing Science and Statistics: Proceedings of the 32rd Symposium on the Interface, pages 156–163, Fairfax. Interface Foundation.
  * Geyer, (1993) Geyer, C. (1993).  Estimating normalizing constants and reweighting mixtures in Markov chain Monte Carlo.  Technical Report 568, School of Statistics, Univ. of Minnesota.
  * (117) Geyer, C. J. (2011a).  Importance sampling, simulated tempering, and umbrella sampling.  Handbook of Markov chain Monte Carlo, pages 295–311.  Chapman & Hall/CRC. Eds. Brooks, S., Gelman, A., Jones, G., Meng, X-L.
  * (118) Geyer, C. J. (2011b).  Introduction to Markov chain Monte Carlo.  Handbook of Markov chain Monte Carlo, pages 3–48.  Chapman & Hall/CRC. Eds. Brooks, S., Gelman, A., Jones, G., Meng, X-L.
  * Ghosal _et al._ , (1995) Ghosal, S., Ghosh, J. K., and Samanta, T. (1995).  On convergence of posterior distributions.  The Annals of Statistics, 23(6):2145–2152.
  * Ghosal and Van der Vaart, (2017) Ghosal, S. and Van der Vaart, A. (2017).  Fundamentals of Nonparametric Bayesian Inference, volume 44.  Cambridge University Press.
  * Giordani _et al._ , (2011) Giordani, P., Pitt, M., and Kohn, R. (2011).  Bayesian inference for time series state space models.  The Oxford Handbook of Bayesian Econometrics, pages 61–124.  OUP. Eds. Geweke, J., Koop, G. and van Dijk, H.
  * Giummolè _et al._ , (2017) Giummolè, F., Mameli, V., Ruli, E., and Ventura, L. (2017).  Objective Bayesian inference with proper scoring rules.  TEST, 28(3):1–28.
  * Glynn, (2016) Glynn, P. W. (2016).  Exact simulation vs. exact estimation.  Proceedings of the 2016 Winter Simulation Conference, WSC16:193–205.
  * Glynn and Rhee, (2014) Glynn, P. W. and Rhee, C.-H. (2014).  Exact estimation for Markov chain equilibrium expectations.  J. Appl. Probab., 51(A):377–389.
  * Gneiting _et al._ , (2007) Gneiting, T., Balabdaoui, F., and Raftery, A. E. (2007).  Probabilistic forecasts, calibration and sharpness.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2):243–268.
  * Golightly _et al._ , (2015) Golightly, A., Henderson, D. A., and Sherlock, C. (2015).  Delayed acceptance particle MCMC for exact inference in stochastic kinetic models.  Statist. Comp., 25(5):1039–1055.
  * Goodfellow _et al._ , (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014).  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680.
  * Gordon _et al._ , (1993) Gordon, N., Salmond, J., and Smith, A. (1993).  A novel approach to non-linear/non-Gaussian Bayesian state estimation.  IEEE Proceedings on Radar and Signal Processing, 140(2):107–113.
  * Gouriéroux _et al._ , (1993) Gouriéroux, C., Monfort, A., and Renault, E. (1993).  Indirect inference.  J. Applied Econometrics, 8:85–118.
  * Gramacy _et al._ , (2010) Gramacy, R., Samworth, R., and King, R. (2010).  Importance tempering.  Statist. Comp., 20(1):1–7.
  * Green, (1995) Green, P. (1995).  Reversible jump MCMC computation and Bayesian model determination.  Biometrika, 82(4):711–732.
  * Green _et al._ , (2015) Green, P., Latuszynski, K., Pereyra, M., and Robert, C. (2015).  Bayesian computation: a summary of the current state, and samples backwards and forwards.  Statist. Comp., 25:835–862.
  * Green, (2003) Green, P. J. (2003).  Trans-dimensional Markov chain Monte Carlo.  Oxford Statistical Science Series, 27:179–198.
  * Gronau _et al._ , (2017) Gronau, Q. F., Sarafoglou, A., Matzke, D., Ly, A., Boehm, U., Marsman, M., Leslie, D. S., Forster, J. J., Wagenmakers, E.-J., and Steingroever, H. (2017).  A tutorial on bridge sampling.  Journal of Mathematical Psychology, 81:80–97.
  * Gubernatis, (2005) Gubernatis, J. E. (2005).  Marshall Rosenbluth and the Metropolis algorithm.  Physics of Plasmas, 12(5):057303.
  * Guedj, (2019) Guedj, B. (2019).  A primer on PAC-Bayesian learning.  arXiv preprint arXiv:1901.05353.
  * Gutmann and Corander, (2016) Gutmann, M. U. and Corander, J. (2016).  Bayesian optimization for likelihood-free inference of simulator-based statistical models.  The Journal of Machine Learning Research, 17(1):4256–4302.
  * Gutmann and Hyvärinen, (2012) Gutmann, M. U. and Hyvärinen, A. (2012).  Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics.  Journal of Machine Learning Research, 13(Feb):307–361.
  * Hajargasht and Woźniak, (2018) Hajargasht, G. and Woźniak, T. (2018).  Accurate computation of marginal data densities using variational Bayes.  https://arXiv:1805.10036.
  * Hammersley and Handscomb, (1964) Hammersley, J. and Handscomb, D. (1964).  Monte Carlo Methods.  John Wiley, New York.
  * Hastings, (1970) Hastings, W. (1970).  Monte Carlo sampling methods using Markov chains and their application.  Biometrika, 57(1):97–109.
  * Higdon, (1998) Higdon, D. M. (1998).  Auxiliary variable methods for Markov chain Monte Carlo with applications.  Journal of the American Statistical Association, 93(442):585–595.
  * Hitchcock, (2003) Hitchcock, D. H. (2003).  A history of the Metropolis–Hastings algorithm.  American Statist., 57(4):254–257.
  * Hoffman and Gelman, (2014) Hoffman, M. D. and Gelman, A. (2014).  The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo.  Journal of Machine Learning Research, 15(1):1593–1623.
  * Holmes and Walker, (2017) Holmes, C. and Walker, S. (2017).  Assigning a value to a power likelihood in a general Bayesian model.  Biometrika, 104(2):497–503.
  * Hoogerheide _et al._ , (2009) Hoogerheide, L. F., van Dijk, H. K., and van Oest, R. D. (2009).  Simulation based Bayesian econometric inference: principles and some recent computational advances.  Handbook of Computational Econometrics, pages 215–280.  John Wiley & Sons. Eds. van Dijk, H. and van Oest, R.
  * Hooper, (2013) Hooper, M. (2013).  Richard Price, Bayes’ theorem, and God.  Significance, 10(1):36–39.
  * Huber, (2016) Huber, M. L. (2016).  Perfect simulation.  Chapman & Hall/CRC.
  * Huggins _et al._ , (2019) Huggins, J. H., Kasprzak, M., Campbell, T., and Broderick, T. (2019).  Validated variational inference via practical posterior error bounds.  https://arXiv:1910.04102.
  * Huggins and Miller, (2019) Huggins, J. H. and Miller, J. W. (2019).  Using bagged posteriors for robust inference and model criticism.  https://arXiv:1912.07104.
  * Iba, (2000) Iba, Y. (2000).  Population-based Monte Carlo algorithms.  Trans. Japanese Soc. Artificial Intell., 16(2):279–286.
  * Jacob _et al._ , (2011) Jacob, P., Robert, C., and Smith, M. (2011).  Using parallel computation to improve independent Metropolis–Hastings based estimation.  J. Comput. Graph. Statist., 20(3):616–635.
  * Jacob _et al._ , (2020) Jacob, P. E., O’Leary, J., and Atchadé, Y. F. (2020).  Unbiased Markov chain Monte Carlo methods with couplings.  J. Royal Statist. Society Series B, 82:1–32.  With discussion.
  * Jacquier _et al._ , (1994) Jacquier, R., Polson, N. G., and Rossi, P. E. (1994).  Bayesian analysis of stochastic volatility models.  J. Business and Economic Statistics, 12(4):371–389.  With discussion.
  * Jahan _et al._ , (2020) Jahan, F., Ullah, I., and Mengersen, K. (2020).  A review of Bayesian statistical approaches for Big Data.  In Mengersen, K., Pudlo, P., and Robert, C., editors, Case Studies in Applied Bayesian Science, pages 17–44. Springer.
  * Jasra, (2015) Jasra, A. (2015).  Approximate Bayesian computation for a class of time series models.  International Statistical Review, 83(3):405–435.
  * Jasra _et al._ , (2012) Jasra, A., Singh, S., Martin, J., and McCoy, E. (2012).  Filtering via approximate Bayesian computation.  Statist. Comp., 22:1223–1237.
  * Jennings and Madigan, (2017) Jennings, E. and Madigan, M. (2017).  AstroABC: an approximate Bayesian computation sequential Monte Carlo sampler for cosmological parameter estimation.  Astronomy and Computing, 19:16–22.
  * Jiang and Tanner, (2008) Jiang, W. and Tanner, M. A. (2008).  Gibbs posterior for variable selection in high-dimensional classification and data-mining.  Annals of Statistics, 36(5):2207–2231.
  * Johndrow _et al._ , (2019) Johndrow, J. E., Smith, A., Pillai, N., and Dunson, D. B. (2019).  MCMC for imbalanced categorical data.  J. American Statist. Assoc., 114(527):1394–1403.
  * Joyce and Marjoram, (2008) Joyce, P. and Marjoram, P. (2008).  Approximately sufficient statistics and Bayesian computation.  Statistical Applications in Genetics and Molecular Biology, 7(1):article 26.
  * Kabisa _et al._ , (2016) Kabisa, S., Dunson, D. B., and Morris, J. S. (2016).  Online variational Bayes inference for high-dimensional correlated data.  J. Comput. Graph. Statist., 25(2):426–444.
  * Kass and Raftery, (1995) Kass, R. and Raftery, A. (1995).  Bayes factors.  J. American Statist. Assoc., 90:773–795.
  * Kim _et al._ , (1998) Kim, S., Shephard, N., and Chib, S. (1998).  Stochastic volatility: likelihood inference and comparison with ARCH models.  The Review of Economic Studies, 65(3):361–393.
  * Kleijn and van der Vaart, (2012) Kleijn, B. and van der Vaart, A. (2012).  The Bernstein-von-Mises theorem under misspecification.  Electron. J. Statist., 6:354–381.
  * Kloek and van Dijk, (1978) Kloek, T. and van Dijk, H. K. (1978).  Bayesian estimates of equation system parameters: an application of integration by Monte Carlo.  Econometrica, 46(1):1–19.
  * Knoblauch _et al._ , (2019) Knoblauch, J., Jewson, J., and Damoulas, T. (2019).  Generalized variational inference.  https://arXiv:1904.02063.
  * Kon Kam King _et al._ , (2019) Kon Kam King, G., Canale, A., and Ruggiero, M. (2019).  Bayesian functional forecasting with locally-autoregressive dependent processes.  Bayesian Anal., 14(4):1121–1141.
  * Koop and Korobilis, (2018) Koop, G. and Korobilis, D. (2018).  Variational Bayes inference in high-dimensional time-varying parameter models.  SSRN 3246472.
  * Koop, (2003) Koop, G. M. (2003).  Bayesian Econometrics.  John Wiley & Sons Inc.
  * Lazar, (2003) Lazar, N. A. (2003).  Bayesian empirical likelihood.  Biometrika, 90:319–326.
  * Lemieux, (2009) Lemieux, C. (2009).  Monte Carlo and quasi-Monte Carlo sampling.  Springer Science & Business Media.
  * (173) Li, W. and Fearnhead, P. (2018a).  Convergence of regression-adjusted approximate Bayesian computation.  Biometrika, 105(2):301–318.
  * (174) Li, W. and Fearnhead, P. (2018b).  On the asymptotic efficiency of approximate Bayesian computation estimators.  Biometrika, 105(2):285–299.
  * Lindley, (1980) Lindley, D. (1980).  Approximate Bayesian methods.  In Bayesian statistics: Proceedings of the first international meeting held in Valencia (Spain), May 28 to June 2, 1979.
  * Lintusaari _et al._ , (2017) Lintusaari, J., Gutmann, M. U., Dutta, R., Kaski, S., and Corander, J. (2017).  Fundamentals and recent developments in approximate Bayesian computation.  Systematic biology, 66(1):e66–e82.
  * Liu, (2001) Liu, J. S. (2001).  Monte Carlo Strategies in Scientific Computing.  Springer Verlag, New-York.
  * Liu _et al._ , (2000) Liu, J. S., Liang, F., and Wong, W. H. (2000).  The multiple-try method and local optimization in Metropolis sampling.  J. American Statist. Assoc., 95(449):121–134.
  * Loaiza-Maya _et al._ , (2020) Loaiza-Maya, R., Martin, G. M., and Frazier, D. T. (2020).  Focused Bayesian prediction.  Forthcoming. Journal of Applied Econometrics.
  * Luo and Tjelmeland, (2019) Luo, X. and Tjelmeland, H. (2019).  A multiple-try Metropolis-Hastings algorithm with tailored proposals.  Computational Statistics, 34(3):1109–1133.
  * Lyddon _et al._ , (2019) Lyddon, S., Holmes, C., and Walker, S. (2019).  General Bayesian updating and the loss-likelihood bootstrap.  Biometrika, 106(2):465–478.
  * Lyne _et al._ , (2015) Lyne, A.-M., Girolami, M., Atchadé, Y., Strathmann, H., and Simpson, D. (2015).  On Russian roulette estimates for Bayesian inference with doubly-intractable likelihoods.  Statist. Sci., 30(4):443–467.
  * Marin _et al._ , (2011) Marin, J., Pudlo, P., Robert, C., and Ryder, R. (2011).  Approximate Bayesian computational methods.  Statist. Comp., 21(2):279–291.
  * Marin and Robert, (2011) Marin, J. and Robert, C. (2011).  Importance sampling methods for Bayesian discrimination between embedded models.  In Chen, M.-H., Dey, D., Müller, P., Sun, D., and Ye, K., editors, Frontiers of Statistical Decision Making and Bayesian Analysis. Springer-Verlag, New York.
  * Marin _et al._ , (2005) Marin, J.-M., Mengersen, K., and Robert, C. (2005).  Bayesian modelling and inference on mixtures of distributions.  Handbook of Statistics, pages 459–507.  Elsevier. Eds. Rao, C. and Dey, D.
  * Marinari and Parisi, (1992) Marinari, E. and Parisi, G. (1992).  Simulated tempering: A new Monte Carlo scheme.  Europhysics Letters, 19(6):451–458.
  * Marjoram _et al._ , (2003) Marjoram, P., Molitor, J., Plagnol, V., and Tavaré, S. (2003).  Markov chain Monte Carlo without likelihoods.  Proc. Natl. Acad. Sci. USA, 100(26):15324–15328.
  * Martin _et al._ , (2019) Martin, G. M., McCabe, B. P., Frazier, D. T., Maneesoonthorn, W., and Robert, C. P. (2019).  Auxiliary likelihood-based approximate Bayesian computation in state space models.  J. Comput. Graph. Statist., 28(3):508–522.
  * Martino, (2018) Martino, L. (2018).  A review of multiple try MCMC algorithms for signal processing.  Digital Signal Processing, 75:134–152.
  * Martino and Riebler, (2019) Martino, S. and Riebler, A. (2019).  Integrated nested Laplace approximations (INLA).  https://arXiv:1907.01248.
  * Meng and Schilling, (2002) Meng, X. and Schilling, S. (2002).  Warp bridge sampling.  J. Comput. Graph. Statist., 11(3):552–586.
  * Meng and Wong, (1996) Meng, X. and Wong, W. (1996).  Simulating ratios of normalizing constants via a simple identity: a theoretical exploration.  Statist. Sinica, 6:831–860.
  * Metropolis _et al._ , (1953) Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and Teller, E. (1953).  Equations of state calculations by fast computing machines.  J. Chem. Phys., 21:1087–1092.
  * Middleton _et al._ , (2018) Middleton, L., Deligiannidis, G., Doucet, A., and Jacob, P. E. (2018).  Unbiased Markov chain Monte Carlo for intractable target distributions.  https://arXiv:1807.08691.
  * Miller and Dunson, (2019) Miller, J. W. and Dunson, D. B. (2019).  Robust Bayesian inference via coarsening.  J. American Statist. Assoc., 114(527):1113–1125.
  * Mukherjee _et al._ , (2006) Mukherjee, P., Parkinson, D., and Liddle, A. (2006).  A nested sampling algorithm for cosmological model selection.  The Astrophysical Journal, 638(2):L51–L54.
  * Muller, (2013) Muller, U. K. (2013).  Risk of Bayesian inference in misspecified models, and the sandwich covariance matrix.  Econometrica, 81(5):1805–1849.
  * Naesseth _et al._ , (2019) Naesseth, C. A., Lindsten, F., Schön, T. B.,_et al._ (2019).  Elements of sequential Monte Carlo.  Foundations and Trends in Machine Learning, 12(3):307–392.
  * Naylor and Smith, (1982) Naylor, J. and Smith, A. (1982).  Application of a method for the efficient computation of posterior distributions.  Applied Statistics, 31(3):214–225.
  * Neal, (1994) Neal, R. (1994).  Contribution to the discussion of “Approximate Bayesian inference with the weighted likelihood bootstrap” by Michael A. Newton and Adrian E. Raftery.  J. Royal Statist. Society Series B, 56(1):41–42.
  * Neal, (1996) Neal, R. (1996).  Sampling from multimodal distributions using tempered transitions.  Statist. Comp., 6:353–366.
  * Neal, (1998) Neal, R. (1998).  Suppressing random walks in Markov chain Monte Carlo using ordered overrelaxation.  Learning in Graphical Models, pages 205–228.  Chapman & Hall/CRC. Ed. Jordan, M. I.
  * Neal, (1999) Neal, R. (1999).  Erroneous results in ‘Marginal likelihood from the Gibbs output’.  Technical report, University of Toronto.
  * Neal, (2001) Neal, R. (2001).  Annealed importance sampling.  Stat. Comput., 11(2):125–139.
  * Neal, (2003) Neal, R. (2003).  Slice sampling.  Ann. Statist., 31(3):705–767.  With discussion.
  * (206) Neal, R. (2011a).  MCMC using ensembles of states for problems with fast and slow variables such as Gaussian process regression.  https://arXiv:1101.0387.
  * (207) Neal, R. (2011b).  MCMC using Hamiltonian dynamics.  Handbook of Markov Chain Monte Carlo, pages 113–162.  Chapman & Hall/CRC. Eds. Brooks, S., Gelman, A., Jones, G., Meng, X-L.
  * Neiswanger _et al._ , (2013) Neiswanger, W., Wang, C., and Xing, E. (2013).  Asymptotically exact, embarrassingly parallel MCMC.  https://arXiv:1311.4780.
  * Nemeth and Fearnhead, (2019) Nemeth, C. and Fearnhead, P. (2019).  Stochastic gradient Markov chain Monte Carlo.  https://arXiv:1907.06986.
  * Newton and Raftery, (1994) Newton, M. and Raftery, A. (1994).  Approximate Bayesian inference by the weighted likelihood bootstrap (with discussion).  J. Royal Statist. Society Series B, 56(1):3–48.
  * Nguyen _et al._ , (2020) Nguyen, H. D., Arbel, J., Lü, H., and Forbes, F. (2020).  Approximate Bayesian computation via the energy statistic.  IEEE Access, 8:131683–131698.
  * Nott _et al._ , (2018) Nott, D., Ong, V. M.-H., Fan, Y., and Sisson, S. (2018).  High-dimensional ABC.  Handbook of Approximate Bayesian Computation, pages 211–242.  Chapman & Hall/CRC. Eds. Sisson, S., Fan, Y., Beaumont, M.
  * Nott and Kohn, (2005) Nott, D. J. and Kohn, R. (2005).  Adaptive sampling for Bayesian variable selection.  Biometrika, 92(4):747–763.
  * (214) Ong, V. M., Nott, D. J., Tran, M.-N., Sisson, S. A., and Drovandi, C. C. (2018a).  Variational Bayes with synthetic likelihood.  Statist. Comp., 28(4):971–988.
  * (215) Ong, V. M.-H., Nott, D. J., Tran, M.-N., Sisson, S. A., and Drovandi, C. C. (2018b).  Likelihood-free inference in high dimensions with synthetic likelihood.  Computational Statistics & Data Analysis, 128:271–291.
  * Ormerod and Wand, (2010) Ormerod, J. T. and Wand, M. P. (2010).  Explaining variational approximations.  American Statist., 64(2):140–153.
  * Owen, (2017) Owen, A. B. (2017).  Statistically efficient thinning of a Markov chain sampler.  J. Comput. Graph. Statist., 26(3):738–744.
  * Park and Nassar, (2014) Park, M. and Nassar, M. (2014).  Variational Bayesian inference for forecasting hierarchical time series.  In Divergence Methods in Probabilistic Inference (DMPI) workshop at International Conference on Machine Learning (ICML). Citeseer.
  * Peters _et al._ , (2012) Peters, G. W., Sisson, S. A., and Fan, Y. (2012).  Likelihood-free Bayesian inference for $\alpha$-stable models.  Comput. Statist. Data Anal., 56(11):3743–3756.
  * Pettenuzzo and Ravazzolo, (2016) Pettenuzzo, D. and Ravazzolo, F. (2016).  Optimal portfolio choice under decision-based model combinations.  Journal of Applied Econometrics, 31(7):1312–1332.
  * Pitt _et al._ , (2012) Pitt, M. K., dos Santos Silva, R., Giordani, P., and Kohn, R. (2012).  On some properties of Markov chain Monte Carlo simulation methods based on the particle filter.  J. Econometrics, 171(2):134–151.
  * Pollock _et al._ , (2020) Pollock, M., Fearnhead, P., Johansen, A. M., and Roberts, G. O. (2020).  Quasi-stationary Monte Carlo and the ScaLE algorithm.  J. Royal Statist. Society Series B, 82(5):1167–1221.  (With discussion.).
  * Polson _et al._ , (1992) Polson, N. G., Carlin, B. P., and Stoffer, D. S. (1992).  A Monte Carlo approach to nonnormal and nonlinear state-space modeling.  J. American Statist. Assoc., 87(418):493–500.
  * Price _et al._ , (2018) Price, L. F., Drovandi, C. C., Lee, A., and Nott, D. J. (2018).  Bayesian synthetic likelihood.  J. Comput. Graph. Statist., 27(1):1–11.
  * Price, (1764) Price, R. (1764).  A demonstration of the second rule in the essay towards the solution of a problem in the doctine of chances.  Philosophical Transactions of the Royal Society of London, 54:296–325.
  * Priddle _et al._ , (2019) Priddle, J. W., Sisson, S. A., Frazier, D. T., and Drovandi, C. (2019).  Efficient Bayesian synthetic likelihood with whitening transformations.  arXiv preprint arXiv:1909.04857.
  * Pritchard _et al._ , (1999) Pritchard, J., Seielstad, M., Perez-Lezaun, A., and Feldman, M. (1999).  Population growth of human Y chromosomes: a study of Y chromosome microsatellites.  Mol. Biol. Evol., 16:1791–1798.
  * Propp and Wilson, (1996) Propp, J. G. and Wilson, D. B. (1996).  Exact sampling with coupled Markov chains and applications to statistical mechanics.  In Proceedings of the Seventh International Conference on Random Structures and Algorithms (Atlanta, GA, 1995), volume 9, pages 223–252.
  * Quiroz _et al._ , (2019) Quiroz, M., Kohn, R., Villani, M., and Tran, M.-N. (2019).  Speeding up MCMC by efficient data subsampling.  J. American Statist. Assoc., 114(526):831–843.
  * (230) Quiroz, M., Nott, D. J., and Kohn, R. (2018a).  Gaussian variational approximation for high-dimensional state space models.  https://arXiv:1801.07873.
  * (231) Quiroz, M., Tran, M.-N., Villani, M., and Kohn, R. (2018b).  Speeding up MCMC by delayed acceptance and data subsampling.  J. Comput. Graph. Statist., 27(1):12–22.
  * Raftery, (1996) Raftery, A. (1996).  Hypothesis testing and model selection.  In Gilks, W., Spiegelhalter, D., and Richardson, S., editors, Markov Chain Monte Carlo in Practice, pages 163–188. Chapman and Hall, New York, London.
  * Rischard _et al._ , (2018) Rischard, M., Jacob, P. E., and Pillai, N. (2018).  Unbiased estimation of log normalizing constants with applications to Bayesian cross-validation.
  * Ritter and Tanner, (1992) Ritter, C. and Tanner, M. (1992).  Facilitating the Gibbs sampler: The Gibbs stopper and the Griddy-Gibbs sampler.  J. American Statist. Assoc., 87(419):861–868.
  * Robert, (2001) Robert, C. (2001).  The Bayesian Choice.  Springer-Verlag, New York, second edition.
  * Robert, (2007) Robert, C. (2007).  The Bayesian Choice.  Springer-Verlag, New York.
  * Robert and Casella, (2004) Robert, C. and Casella, G. (2004).  Monte Carlo Statistical Methods.  Springer-Verlag, New York, second edition.
  * Robert and Casella, (2011) Robert, C. and Casella, G. (2011).  A history of Markov chain Monte Carlo—subjective recollections from incomplete data.  Statist. Science, 26(1):102–115.
  * Robert, (2015) Robert, C. P. (2015).  The Metropolis–Hastings Algorithm, pages 1–15.  Wiley StatsRef: Statistics Reference Online. American Cancer Society.
  * Robert _et al._ , (2018) Robert, C. P., Elvira, V., Tawn, N., and Wu, C. (2018).  Accelerating MCMC algorithms.  Wiley Interdisciplinary Reviews: Computational Statistics, 10(5):e1435.
  * Roberts _et al._ , (1997) Roberts, G., Gelman, A., and Gilks, W. (1997).  Weak convergence and optimal scaling of random walk Metropolis algorithms.  Ann. Applied Prob., 7(1):110–120.
  * Roberts and Rosenthal, (1999) Roberts, G. and Rosenthal, J. (1999).  Convergence of slice sampler Markov chains.  J. Royal Statist. Society Series B, 61(3):643–660.
  * Roberts and Sahu, (1997) Roberts, G. and Sahu, S. (1997).  Updating schemes, covariance structure, blocking and parametrisation for the Gibbs sampler.  J. Royal Statist. Society Series B, 59(2):291–317.
  * Roberts and Rosenthal, (1998) Roberts, G. O. and Rosenthal, J. S. (1998).  Optimal scaling of discrete approximations to Langevin diffusions.  J. Royal Statist. Society Series B, 60(1):255–268.
  * Roberts and Rosenthal, (2009) Roberts, G. O. and Rosenthal, J. S. (2009).  Examples of adaptive MCMC.  J. Comput. Graph. Statist., 18(2):349–367.
  * Roberts _et al._ , (1996) Roberts, G. O., Tweedie, R. L.,_et al._ (1996).  Exponential convergence of Langevin distributions and their discrete approximations.  Bernoulli, 2(4):341–363.
  * Rodrigues _et al._ , (2019) Rodrigues, G., Nott, D. J., and Sisson, S. (2019).  Likelihood-free approximate Gibbs sampling.  https://arXiv:1906.04347.
  * Rosenthal, (2011) Rosenthal, J. S. (2011).  Optimal proposal distributions and adaptive MCMC.  Handbook of Markov Chain Monte Carlo, pages 93–111.  Chapman & Hall/CRC. Eds. Brooks, S., Gelman, A., Jones, G., Meng, X-L.
  * Rubin, (1981) Rubin, D. B. (1981).  The Bayesian bootstrap.  Ann. Statist., 9(1):130–134.
  * Rue and Held, (2005) Rue, H. and Held, L. (2005).  Gaussian Markov Random Fields: Theory and Applications, volume 104 of Monographs on Statistics and Applied Probability.  Chapman & Hall, London.
  * Rue _et al._ , (2009) Rue, H., Martino, S., and Chopin, N. (2009).  Approximate Bayesian inference for latent Gaussian models using integrated nested Laplace approximations.  J. Royal Statist. Society Series B, 71(2):319–392.
  * Scott _et al._ , (2016) Scott, S. L., Blocker, A. W., Bonassi, F. V., Chipman, H. A., George, E. I., and McCulloch, R. E. (2016).  Bayes and big data: The consensus Monte Carlo algorithm.  International Journal of Management Science and Engineering Management, 11(2):78–88.
  * Sisson and Fan, (2011) Sisson, S. and Fan, Y. (2011).  Likelihood-free Markov chain Monte Carlo.  Handbook of Markov Chain Monte Carlo, pages 313–333.  Chapman & Hall/CRC. Eds. Brooks, S., Gelman, A., Jones, G., Meng, X-L.
  * Sisson _et al._ , (2019) Sisson, S. A., Fan, Y., and Beaumont, M. (2019).  Handbook of Approximate Bayesian Computation.  Chapman & Hall/CRC.
  * Sisson _et al._ , (2007) Sisson, S. A., Fan, Y., and Tanaka, M. (2007).  Sequential Monte Carlo without likelihoods.  Proc. Natl. Acad. Sci. USA, 104(6):1760–1765.
  * Skilling, (2007) Skilling, J. (2007).  Nested sampling for Bayesian computations.  Bayesian Analysis, 1(4):833–859.
  * Smith and Roberts, (1993) Smith, A. and Roberts, G. (1993).  Bayesian computation via the Gibbs sampler and related Markov chain Monte Carlo methods.  J. Royal Statist. Society Series B, 55(1):3–24.  With discussion.
  * (258) Stigler, S. (1986a).  The History of Statistics.  Belknap, Cambridge.
  * (259) Stigler, S. (1986b).  Memoir on inverse probability.  Statistical Science, 1(3):359–363.
  * Stigler, (1975) Stigler, S. M. (1975).  Studies in the history of probability and statistics. XXXIV Napoleonic statistics: The work of Laplace.  Biometrika, 62(2):503–517.
  * Stigler, (2018) Stigler, S. M. (2018).  Richard Price, the first Bayesian.  Statist. Sci., 33(1):117–125.
  * Stoehr, (2017) Stoehr, J. (2017).  A review on statistical inference methods for discrete Markov random fields.  https://arXiv:1704.03331.
  * Strachan and van Dijk, (2014) Strachan, R. and van Dijk, H. (2014).  Divergent priors and well behaved Bayes factors.  Central European Journal of Economic Modelling and Econometrics, CEJEME, 6(1):1–31.
  * Syring and Martin, (2019) Syring, N. and Martin, R. (2019).  Calibrating general posterior credible regions.  Biometrika, 106(2):479–486.
  * Tanner and Wong, (1987) Tanner, M. A. and Wong, W. (1987).  The calculation of posterior distributions by data augmentation.  J. American Statist. Assoc., 82(398):528–550.  With discussion.
  * Tavaré _et al._ , (1997) Tavaré, S., Balding, D., Griffith, R., and Donnelly, P. (1997).  Inferring coalescence times from DNA sequence data.  Genetics, 145:505–518.
  * Tawn _et al._ , (2020) Tawn, N. G., Roberts, G. O., and Rosenthal, J. S. (2020).  Weight-preserving simulated tempering.  Statist. Comp., 30:27–41.
  * Tierney, (1994) Tierney, L. (1994).  Markov chains for exploring posterior distributions.  Ann. Statist., 22(4):1701–1762.  With discussion and a rejoinder by the author.
  * Tierney and Kadane, (1986) Tierney, L. and Kadane, J. (1986).  Accurate approximations for posterior moments and marginal densities.  J. American Statist. Assoc., 81(393):82–86.
  * Tierney _et al._ , (1989) Tierney, L., Kass, R., and Kadane, J. (1989).  Fully exponential Laplace approximations to expectations and variances of non-positive functions.  J. American Statist. Assoc., 84(407):710–716.
  * Tierney and Mira, (1998) Tierney, L. and Mira, A. (1998).  Some adaptive Monte Carlo methods for Bayesian inference.  Statistics in Medicine, 18:2507–2515.
  * Tokdar and Kass, (2010) Tokdar, S. and Kass, R. (2010).  Importance sampling: A review.  Wiley Interdisciplinary Reviews: Computational Statistics, 2:54 – 60.
  * Tran, (2018) Tran, K. T. (2018).  A common derivation for Markov chain Monte Carlo algorithms with tractable and intractable targets.  arXiv:, 1607.01985.
  * Tran _et al._ , (2019) Tran, M.-N., Nguyen, D. H., and Nguyen, D. (2019).  Variational Bayes on manifolds.  https://arXiv:1908.03097.
  * Tran _et al._ , (2017) Tran, M.-N., Nott, D. J., and Kohn, R. (2017).  Variational Bayes with intractable likelihood.  J. Comput. Graph. Statist., 26(4):873–882.
  * van der Vaart, (1998) van der Vaart, A. (1998).  Asymptotic Statistics.  Cambridge University Press.
  * Vanslette _et al._ , (2019) Vanslette, K., Alsheikh, A. A., and Youcef-Toumi, K. (2019).  Why simple quadrature is just as good as Monte Carlo.  https://arXiv:1908.00947.
  * Wand, (2017) Wand, M. P. (2017).  Fast approximate inference for arbitrarily large semiparametric regression models via message passing.  J. American Statist. Assoc., 112(517):137–168.
  * Wang and Dunson, (2013) Wang, X. and Dunson, D. (2013).  Parallel MCMC via Weierstrass sampler.  https://arXiv:1312.4605.
  * (280) Wang, Y. and Blei, D. (2019a).  Variational Bayes under model misspecification.  In Advances in Neural Information Processing Systems, pages 13357–13367.
  * (281) Wang, Y. and Blei, D. M. (2019b).  Frequentist consistency of variational Bayes.  J. American Statist. Assoc., 114(527):1147–1161.
  * Wilkinson, (2013) Wilkinson, R. (2013).  Approximate Bayesian computation (ABC) gives exact results under the assumption of model error.  Statistical Applications in Genetics and Molecular Biology, 12(2):129–141.
  * Wiqvist _et al._ , (2018) Wiqvist, S., Picchini, U., Forman, J. L., Lindorff-Larsen, K., and Boomsma, W. (2018).  Accelerating delayed-acceptance Markov chain Monte Carlo algorithms.  https://arXiv:1806.05982.
  * Wood, (2010) Wood, S. (2010).  Statistical inference for noisy nonlinear ecological dynamic systems.  Nature, 466(7310):1102––1104.
  * Wood, (2019) Wood, S. (2019).  Simplified integrated nested Laplace approximation.  Biometrika, 107(1):223–230.
  * Yu _et al._ , (2019) Yu, X., Nott, D. J., Tran, M.-N., and Klein, N. (2019).  Assessment and adjustment of approximate inference algorithms using the law of total variance.  https://arXiv:1911.08725.
  * Zanella and Roberts, (2019) Zanella, G. and Roberts, G. (2019).  Scalable importance tempering and Bayesian variable selection.  J. Royal Statist. Society Series B, 81(3):489–517.
  * Zellner, (1971) Zellner, A. (1971).  An Introduction to Bayesian Econometrics.  John Wiley, New York.
  * Zhang _et al._ , (2018) Zhang, C., Bütepage, J., Kjellström, H., and Mandt, S. (2018).  Advances in variational inference.  IEEE transactions on pattern analysis and machine intelligence, 41(8):2008–2026.
  * Zhang and Gao, (2017) Zhang, F. and Gao, C. (2017).  Convergence rates of variational posterior distributions.  https://arXiv:1712.02519.
  * (291) Zhang, T. (2006a).  From eps-entropy to KL entropy: analysis of minimum information complexity density estimation.  Annals of Statistics, 34:2180–2210.
  * (292) Zhang, T. (2006b).  Information-theoretic upper and lower bounds for statistical estimation.  IEEE Trans. Info. Theory, 52(4):1307–1321.


11institutetext: North Carolina State University, Raleigh, NC 27695, USA

# Mixing Backward- with Forward-Chaining for Metacognitive Skill Acquisition
and Transfer

Mark Abdelshiheed  John Wesley Hostetter  Xi Yang
Tiffany Barnes  Min Chi
{mnabdels  jwhostet  yxi2  tmbarnes<EMAIL_ADDRESS>
###### Abstract

Metacognitive skills have been commonly associated with preparation for future
learning in deductive domains. Many researchers have regarded _strategy-_ and
_time-awareness_ as two metacognitive skills that address _how_ and _when_ to
use a problem-solving strategy, respectively. It was shown that students who
are both strategy- and time-aware $(StrTime)$ outperformed their $nonStrTime$
peers across deductive domains. In this work, students were trained on a logic
tutor that supports a default forward-chaining (FC) and a backward-chaining
(BC) strategy. We investigated the impact of mixing BC with FC on teaching
strategy- and time-awareness for $nonStrTime$ students. During the logic
instruction, the experimental students $(Exp)$ were provided with two BC
worked examples and some problems in BC to practice _how_ and _when_ to use
BC. Meanwhile, their control $(Ctrl)$ and $StrTime$ peers received no such
intervention. Six weeks later, all students went through a probability tutor
that only supports BC to evaluate whether the acquired metacognitive skills
are transferred from logic. Our results show that on both tutors, $Exp$
outperformed $Ctrl$ and caught up with $StrTime$.

###### Keywords:

Strategy Awareness Time Awareness Metacognitive Skill Instruction Preparation
for Future Learning Backward Chaining.

##  1 Introduction

One fundamental goal of education is being prepared for future learning [6] by
transferring acquired skills and problem-solving strategies across different
domains. Despite the difficulty of achieving such transfer [6], prior research
has shown it can be facilitated by obtaining metacognitive skills [1, 2, 3, 4,
8]. It has been believed that metacognitive skills are essential for academic
achievements [9], and teaching such skills impacts learning outcomes [8] and
strategy use [14]. Much prior research has categorized knowing _how_ and
_when_ to use a problem-solving strategy as two metacognitive skills [16],
referred to as strategy- and time-awareness, respectively. Our prior work
found that students who were both strategy- and time-aware —referred to as
$StrTime$— outperformed their $nonStrTime$ peers across deductive domains [3,
4]. In the current work, we provide interventions for the latter students to
catch up with their $StrTime$ peers.

Deductive domains such as logic, physics and probability usually require
multiple problem-solving strategies. Two common strategies in these domains
are forward-chaining (FC) and backward-chaining (BC). Early studies showed
that experts often use a mixture of FC and BC to execute their strategies
[13]. This work investigates the impact of mixing FC and BC on teaching
strategy- and time-awareness for $nonStrTime$ students.

Our study involved two intelligent tutoring systems (ITSs): logic and
probability. Students were first assigned to a logic tutor that supports FC
and BC, with FC being the default, then to a probability tutor six weeks later
that only supports BC. During the logic instruction, $nonStrTime$ students
were split into experimental $(Exp)$ and control $(Ctrl)$ conditions. For
$Exp$, the tutor provided two worked examples solved in BC and presented some
problems in BC to practice _how_ and _when_ to use BC. $Ctrl$ received no such
intervention as each problem was presented in FC by default with the ability
to switch to BC. Our goal is to inspect whether our intervention would make
$Exp$ catch up with the golden standard —$StrTime$ students— who already have
the two metacognitive skills and thus need no intervention. All students went
through the same probability tutor to evaluate whether the acquired
metacognitive skills are transferred from logic. Our results show that $Exp$
outperformed $Ctrl$ and caught up with $StrTime$ on both tutors.

###  1.1 Metacognitive Skill Instruction

Metacognitive skills regulate one’s awareness and control of their cognition
[7]. Many studies have demonstrated the significance of metacognitive skills
instruction on academic performance [9], learning outcomes [1, 3, 8] and
regulating strategy use [14]. Schraw and Gutierrez [14] argued that
metacognitive skill instruction involves feeling what is known and not known
about a task. They stated that such instruction should further compare
strategies according to their feasibility and familiarity from the learner’s
perspective. Chi and VanLehn [8] found that teaching students principle-
emphasis skills closed the gap between high and low learners, not only in the
domain where they were taught (probability) but also in a second domain where
they were not taught (physics).

Strategy- and time-awareness have been considered metacognitive skills as they
respectively address _how_ and _when_ to use a problem-solving strategy [9,
16]. Researchers have emphasized the role of strategy awareness in preparation
for future learning [3, 5] and the impact of time awareness on planning skills
and academic performance [9, 10]. Belenky and Nokes [5] showed that students
who had a higher aim to master presented materials and strategies outperformed
their peers on a transfer task. Fazio et al. [10] revealed that students who
knew when to use each strategy to pick the largest fraction magnitude had
higher mathematical proficiency than their peers. de Boer et al. [9] showed
that students who knew when and why to use a given strategy exhibit long-term
metacognitive knowledge that improves their academic performance. de Boer et
al. emphasized that knowing _when_ and _why_ has the same importance as
knowing _how_ when it comes to strategy choice in multi-strategy domains.

###  1.2 Forward- and Backward-Chaining

FC and BC are two standard problem-solving strategies in deductive domains. In
FC, the reasoning proceeds from the given propositions toward the target goal,
whereas BC is goal-driven in that it works backward from a goal state to a
given state. Substantial work has investigated the impact of FC and BC
strategies in two research categories: empirical studies and post-hoc
observations.

Prior empirical studies have shown the significance of FC over BC in learning
physics [11] and weightlifting movements [12]. Moore and Quintero [12]
compared FC and BC in teaching the clean and snatch movements to novice weight
lifters. The participants showed mastery performance with the FC training but
showed substantially fewer improvements in performance accuracy via the BC
training. All participants mastered the movements when some BC lifts were
changed to FC. Conversely, some studies reported no significant difference
between the two strategies [15]. Slocum and Tiger [15] assessed the children’s
FC and BC strategy preferences on various learning tasks. They found that
children were equally efficient on both strategies and had similar mixed
strategy preferences.

Early research has observed the impact of mixing FC and BC strategies [13].
Priest and Lindsay [13] compared how experts and novices solve physics
problems. Although both groups used a mixture of FC and BC, _only_ the experts
knew how and when to use each strategy and significantly produced more
complete plans and stages than their novice peers. In brief, while no
consensus has been reached on whether FC or BC is most effective in problem-
solving, prior work has observed that the mixture of FC and BC yields the
highest performance accuracy as learners know how and when to use each
strategy.

##  2 Methods

Participants They are Computer Science undergraduates at North Carolina State
University. Students were assigned each tutor as a class assignment and told
that completion is required for full credit. Similar to our prior work, we
utilize the random forest classifier (RFC) that, based on pre-test
performance, predicts the metacognitive label ($StrTime$ or otherwise) before
training on logic and was previously shown to be $96\%$ accurate [3].
Specifically, $StrTime$ students frequently follow the desired behavior of
switching _early_ (within the first $30$ actions) to $BC$, while their peers
either frequently switch late (after the first $30$ actions) or stick to the
default $FC$ [1, 3, 4]. A total of $121$ students finished both tutors and
were classified by the RFC into $26$ $StrTime$ and $95$ otherwise. The latter
students were randomly assigned to $Experimental$ $(Exp:N=49)$ and $Control$
$(Ctrl:N=46)$ conditions. The RFC was $97\%$ accurate in classifying students
who received no intervention —$Ctrl$ and $StrTime$.

(a) FC Strategy

(b) BC Strategy

(c) Modified Training

Figure 1: Logic Tutor

Logic Tutor and Our Intervention The logic tutor teaches propositional logic
proofs by applying inference rules such as Modus Ponens. A student can solve
any problem by either a FC or BC strategy. Students derive a conclusion at the
bottom from givens at the top in _FC_ (Fig. 1a), while they derive a
contradiction from givens and the _negation_ of the conclusion in BC (Fig.
1b). A problem is presented by _default_ in FC with the ability to switch to
BC by clicking a button. The tutor consists of two pre-test, $20$ training and
six post-test problems. The post-test is _much harder_ than the pre-test, and
the first two post-test problems are isomorphic to the two pre-test problems.
The _pre-_ and _post-test_ scores are calculated by averaging the pre- and
post-test problem scores, where a problem score is a function of time,
accuracy, and solution length. The training consists of five ordered levels in
an _incremental degree of difficulty_ , and each level consists of four
problems. We modified the training section to mix BC with FC (Fig. 1c).
Specifically, two worked examples (WE) on BC were implemented, where the tutor
provided a step-by-step solution, and six problems were presented in BC by
default. The two WEs and the six problems are expected to teach students _how_
and _when_ to use BC. Note that the colored problems in Figure 1c were
selected based on the historical strategy switches in our data [4].

Probability Tutor It teaches how to solve probability problems using ten
principles, such as the Complement Theorem. The tutor consists of a textbook,
pre-test, training, and post-test. The textbook introduces the domain
principles, while training consists of $12$ problems, each of which can _only_
be solved by $BC$ as it requires deriving an answer by _writing and solving
equations_ until the target is ultimately reduced to the givens. In pre- and
post-test, students solve $14$ and $20$ open-ended problems graded by
experienced graders in a double-blind manner using a partial-credit rubric.
The _pre-_ and _post-test_ scores are the average grades in their respective
sections, where grades are based _only_ on accuracy. Like the logic tutor, the
post-test is much harder than the pre-test, and each pre-test problem has a
corresponding isomorphic post-test problem.

Procedure Students were assigned to the logic tutor and went through the pre-
test, training and post-test. Before training on logic, the RFC predicted the
metacognitive label for each student, as described in the Participants
section. During training, $Exp$ received the modified tutor shown in Figure
1c, while $Ctrl$ and $StrTime$ received the original tutor, where all problems
are presented in $FC$ by default. Six weeks later, students were trained on
the probability tutor.

##  3 Results

Table 1: Comparing Groups across Tutors |  $Experimental\,(Exp)$ $(N=49)$ |  $Control\,(Ctrl)$ $(N=46)$ |  $StrTime$ $(N=26)$
---|---|---|---
Logic Tutor
$Pre$ |  $61.7\,(18)$ |  $58.7\,(20)$ | $62.1\,(20)$
$Iso$-$Post$ |  $81\,(11)$ |  $70.4\,(14)$ | $81.3\,(10)$
$Iso$-$NLG$ |  $0.27\,(.12)$ |  $0.09\,(.31)$ | $0.29\,(.16)$
$Post$ |  $77.4\,(11)$ |  $66.7\,(14)$ | $79\,(9)$
$NLG$ |  $0.24\,(.15)$ |  $0.06\,(.37)$ | $0.25\,(.18)$
Probability Tutor
$Pre$ |  $74.8\,(14)$ |  $74.2\,(16)$ | $75.8\,(15)$
$Iso$-$Post$ |  $90.4\,(10)$ |  $65.3\,(16)$ | $90.6\,(8)$
$Iso$-$NLG$ |  $0.29\,(.19)$ |  -$0.02\,(.27)$ | $0.26\,(.17)$
$Post$ |  $89.5\,(15)$ |  $62.5\,(18)$ | $88.8\,(7)$
$NLG$ |  $0.26\,(.21)$ |  -$0.08\,(.3)$ | $0.24\,(.15)$
|  |  |

Table 1 compares the groups’ performance across the two tutors showing the
mean and standard deviation of pre- and post-test scores, isomorphic scores,
and the learning outcome in terms of the normalized learning gain $(NLG)$
defined as $(NLG=\frac{Post-Pre}{\sqrt{100-Pre}})$, where $100$ is the maximum
test score. We refer to pre-test, post-test and NLG scores as $Pre$, $Post$
and $NLG$, respectively. On both tutors, a one-way ANOVA found no significant
difference on $Pre$ between the groups.

To measure the improvement on isomorphic problems, repeated measures ANOVA
tests were conducted using {$Pre$, $Iso$-$Post$} as factor. Results showed
that $Exp$ and $StrTime$ learned significantly with $\mathit{p}<0.0001$ on
both tutors, while $Ctrl$ did not perform significantly higher on $Iso$-$Post$
than $Pre$ on both tutors. These findings verify the RFC’s accuracy, as
$StrTime$ learned significantly on both tutors, while $Ctrl$ did not, despite
both receiving no intervention.

A comprehensive comparison between the three groups was essential to evaluate
our intervention. On the logic tutor, A one-way ANCOVA using $Pre$ as
covariate and group as factor found a significant effect on $Post$:
$\mathit{F}(2,117)=14.5,\,\mathit{p}<.0001,\,\mathit{\eta}^{2}=.18$.
Subsequent post-hoc analyses with Bonferroni correction $(\alpha=.05/3)$
revealed that $Exp$ and $StrTime$ significantly outperformed $Ctrl$:
$\mathit{t}(93)=3.8,\,\mathit{p}<.001$ and
$\mathit{t}(70)=3.9,\,\mathit{p}<.001$, respectively. Similar patterns were
observed on $NLG$ using ANOVA and the post-hoc comparisons.

On the probability tutor, a one-way ANCOVA using $Pre$ as covariate and group
as factor showed a significant effect on $Post$:
$\mathit{F}(2,117)=48.1,\,\mathit{p}<.0001,\,\mathit{\eta}^{2}=.35$. Follow-up
pairwise comparisons with Bonferroni adjustment showed that $Exp$ and
$StrTime$ significantly surpassed $Ctrl$:
$\mathit{t}(93)=6.1,\,\mathit{p}<.0001$ and
$\mathit{t}(70)=5.9,\,\mathit{p}<.0001$, respectively. Similar results were
found on $NLG$ using ANOVA and the post-hoc comparisons.

##  4 Conclusion

We showed that mixing BC with FC on the logic tutor improved the experimental
students’ learning outcomes, as $Exp$ significantly outperformed $Ctrl$ on
logic and on a probability tutor that only supports BC. Additionally, $Exp$
caught up with $StrTime$ on both tutors suggesting that $Exp$ students are
prepared for future learning [6] as they acquired BC mastery skills on logic
and transferred them to probability, where they received no intervention.
There is at least one caveat in our study. The probability tutor supported
only one strategy. A more convincing testbed would be having the tutors
support both strategies. The future work involves implementing FC on the
probability tutor.

Acknowledgments: This research was supported by the NSF Grants: 1660878,
1651909, 1726550 and 2013502.

## References

  * [1] Abdelshiheed, M., Hostetter, J.W., Shabrina, P., Barnes, T., Chi, M.: The power of nudging: Exploring three interventions for metacognitive skills instruction across intelligent tutoring systems. In: Proceedings of the 44th annual conference of the cognitive science society. vol. 44 (2022)
  * [2] Abdelshiheed, M., Maniktala, M., Barnes, T., Chi, M.: Assessing competency using metacognition and motivation: The role of time-awareness in preparation for future learning. In: Design Recommendations for Intelligent Tutoring Systems, vol. 9, pp. 121–131 (2022)
  * [3] Abdelshiheed, M., Maniktala, M., Ju, S., Jain, A., Barnes, T., Chi, M.: Preparing unprepared students for future learning. In: Proceedings of the 43rd annual conference of the cognitive science society. vol. 43 (2021)
  * [4] Abdelshiheed, M., Zhou, G., Maniktala, M., Barnes, T., Chi, M.: Metacognition and motivation: The role of time-awareness in preparation for future learning. In: Proceedings of the 42nd annual conference of the cognitive science society. vol. 42 (2020)
  * [5] Belenky, D.M., Nokes, T.J.: Motivation and transfer: The role of mastery-approach goals in preparation for future learning. Journal of the Learning Sciences 21(3), 399–432 (2012)
  * [6] Bransford, J.D., Schwartz, D.L.: Rethinking transfer: A simple proposal with multiple implications. Review of research in education 24(1), 61–100 (1999)
  * [7] Chambres, P.E., Izaute, M.E., Marescaux, P.J.E.: Metacognition: Process, function and use. Kluwer Academic Publishers (2002)
  * [8] Chi, M., VanLehn, K.: Meta-cognitive strategy instruction in intelligent tutoring systems: how, when, and why. Journal of Educational Technology & Society 13(1), 25–39 (2010)
  * [9] De Boer, H., Donker, A.S., Kostons, D.D., Van der Werf, G.P.: Long-term effects of metacognitive strategy instruction on student academic performance: A meta-analysis. Educational Research Review 24, 98–115 (2018)
  * [10] Fazio, L.K., DeWolf, M., Siegler, R.S.: Strategy use and strategy choice in fraction magnitude comparison. Journal of Experimental Psychology: Learning, Memory, and Cognition 42(1), 1 (2016)
  * [11] Larkin, J., McDermott, J., Simon, D.P., Simon, H.A.: Expert and novice performance in solving physics problems. Science 208(4450), 1335–1342 (1980)
  * [12] Moore, J.W., Quintero, L.M.: Comparing forward and backward chaining in teaching olympic weightlifting. Journal of applied behavior analysis 52(1), 50–59 (2019)
  * [13] Priest, A., Lindsay, R.: New light on novice—expert differences in physics problem solving. British journal of Psychology 83(3), 389–405 (1992)
  * [14] Schraw, G., Gutierrez, A.P.: Metacognitive strategy instruction that highlights the role of monitoring and control processes. In: Metacognition: Fundaments, applications, and trends, pp. 3–16. Springer (2015)
  * [15] Slocum, S.K., Tiger, J.H.: An assessment of the efficiency of and child preference for forward and backward chaining. ournal of Applied Behavior Analysis 44(4), 793–805 (2011)
  * [16] Winne, P.H., Azevedo, R.: Metacognition. In: Sawyer, R.K. (ed.) The Cambridge Handbook of the Learning Sciences. Cambridge Handbooks in Psychology, Cambridge University Press, 2 edn. (2014)