# Topological Transformation and Free-Space Transport of Photonic Hopfions

Yijie Shen<EMAIL_ADDRESS>Optoelectronics Research Centre & Centre for
Photonic Metamaterials, University of Southampton, Southampton SO17 1BJ,
United Kingdom  Bingshi Yu  Haijun Wu  Chunyu Li  Zhihan Zhu
<EMAIL_ADDRESS>Wang Da-Heng Center, Heilongjiang Key Laboratory of
Quantum Control, Harbin University of Science and Technology, Harbin 150080,
China  Anatoly V. Zayats<EMAIL_ADDRESS>Department of Physics and London
Centre for Nanotechnology, King’s College London, London WC2R 2LS, United
Kingdom

###### Abstract

Structured light fields embody strong spatial variations of polarisation,
phase and amplitude. Understanding, characterization and exploitation of such
fields can be achieved through their topological properties. Three-dimensional
(3D) topological solitons, such as hopfions, are 3D localized continuous field
configurations with nontrivial particle-like structures, that exhibit a host
of important topologically protected properties. Here, we propose and
demonstrate photonic counterparts of hopfions with exact characteristics of
Hopf fibration, Hopf index, and Hopf mapping from real-space vector beams to
homotopic hyperspheres representing polarisation states. We experimentally
generate photonic hopfions with on-demand high-order Hopf indices and
independently controlled topological textures, including Néel-, Bloch-, and
anti-skyrmionic types. We also demonstrate a robust free-space transport of
photonic hopfions, thus, showing potential of hopfions for developing optical
topological informatics and communications.

Topological solitons with topologically protected spin texture are of
fundamental interest in exploring fascinating physical phenomena and nonlinear
field theories manton2004topological . In particular, numerous low-dimensional
topological solitonic textures have been extensively studied in the recent
years, such as 1D magnetic domain walls and magnetic skyrmions
parkin2008magnetic ; wang2012domain ; ryu2013chiral ; han2019mutual ;
jue2016chiral , as well as 2D electromagnetic skyrmions tsesses2018optical ;
du2019deep ; davis2020ultrafast ; shen2021supertoroidal , merons and bimerons
jani2021antiferromagnetic ; dai2020plasmonic ; shen2021topological . Their
sophisticated topological structures have been considered as promising
information carriers in the next generation of data storage and communication
devices shen2022generation ; gobel2021beyond ; fert2017magnetic ;
bernevig2022progress . 3D topological textures have also been attracting
considerable interest in condensed matter physics and optics, following their
theoretical introduction, extending the frontiers of topological field
manipulation gobel2021beyond ; ackerman2017static ; tang2021magnetic ;
Zheng2021 .

Hopfions, as the most classic 3D topological solitons, were initially proposed
in the Skyrme-Faddeev model faddeev1976some ; skyrme1991non ;
skyrme1962unified . They are also known as Faddeev-Hopf knots
hietarinta1999faddeev ; sutcliffe2007knots  and can be elegantly mapped to a
Hopf fibration and characterized by a Hopf index lyons2003elementary ;
whitehead1947expression . Hopfions have fundamental importance in many
physical systems in high-energy physics faddeev1997stable , chiral magnets
tai2018static ; wang2019current ; liu2020three ; luk2020hopfions ;
gobel2020topological ; kent2021creation , quantum fields hall2016tying ;
lee2018synthetic , condensed matter physics babaev2002dual ;
kawaguchi2008knots ; bisset2015robust ; zou2021formation , cosmology
cruz2007cosmic ; thompson2015classification , fluid dynamics
kleckner2013creation , liquid crystals ackerman2017diversity ;
ackerman2015self , and very recently realized in free-space photonics
sugic2021particle . Due to their rich 3D spin texture, hopfions can
potentially provide many opportunities in investigations and applications of
topological structures, however, only very limited number of hophions were
experimentally realized to date. For instance, magnetic hopfions of Bloch-
(vortex) and Néel- (hedgehog) types can be excited and exist in a stable state
in chiral magnets wang2019current . In photonics, only fundamental-order
hopfions with a unit Hopf index were reported sugic2021particle . Generation
and properties of higher-order hopfions and their topological spin texture
tuning still need to be explored.

In this Letter, we theoretically and experimentally demonstrate a generalized
family of photonic hopfions constructed using superposition of Laguerre-
Gaussian (LG) beams with customized polarization patterns. The generated
complex vector fields have spatially varying 3D polarisation distributions,
which exhibits hopfion topological textures. The observed photonic hopfions
can be freely transformed to various topological states, including Néel-type
(hedgehog), Bloch-type (vortex), and anti-type (saddle) textures and many
intermediate topological states. For each on-demand texture, a topological
charge (Hopf index) of hopfions can also be tuned to arbitrary integers,
realising higher-order hopfions. We also demonstrate a free-space propagation
of hopfions with protected topology, revealing potential to develop technology
of topological information transfer.

Figure 1: (a) The parameter-space visualization of a hypersphere: the
longitude and latitude degrees ($\alpha$ and $\beta$) of a parametric 2-sphere
are represented by hue color and its lightness (dark towards the south pole,
where spin is down, and bright towards the north pole, where spin is up). Each
point on a parametric 2-sphere corresponds to a closed iso-spin line located
in a 3D Euclidean space. (b) The lines projected from the selected points of
the same latitude $\beta$ and different longitude $\alpha$ on the hypersphere
(highlighted by the solid dots with the corresponding hue colors), form torus
knots covering a torus (with different tori corresponding to different
$\beta$). (c) The real-space visualization of a Hopf fibration as a full
stereographic mapping from a hypersphere: toroidally knotted lines (torus
knots) arranged on a set of coaxially nested tori, with each torus
corresponding to different latitude $\beta$ of a parametric 2-sphere. The
black circle corresponds to south pole (spin down) and the axis of the nested
tori corresponds to the north pole (spin up) in (a). (d) The 3D spin
distribution in a hopfion, corresponding to the isospin contours in (c) with
each spin vector colored by its $\alpha$ and $\beta$ parameters of a
parametric sphere in (a) as shown in the insert. (e,f) The cross-sectional
view of the spin distribution in (d): (e) $xy$ ($z=0$) and (f) $yz$ ($x=0$)
cross-sections show skyrmion-like structures with the grey arrows marking the
vorticity of the skyrmions. Colour scale is the same as correspond to the spin
direction in (d).

In topology, a hopfion configuration is usually represented by a 3D real-space
distribution of generalised unit spin vectors,
$\mathbf{S}(x,y,z)=[s_{x}(x,y,z),s_{y}(x,y,z),s_{z}(x,y,z)]$, and achieved by
a stereographic projection from a hypersphere $\mathbb{S}^{3}$
($\bm{\chi}=(\chi_{1},\chi_{2},\chi_{3},\chi_{4})$) to a real space
$\mathbb{R}^{3}$ ($\mathbf{r}=(x,y,z)$), fulfilling the Hopf map
$\mathbf{S}=\langle\bm{\zeta}|\bm{\sigma}|\bm{\zeta}\rangle$, where
$|\bm{\zeta}\rangle=(\chi_{4}+\text{i}\chi_{3},\chi_{1}+\text{i}\chi_{2})^{\text{T}}$
and $\bm{\sigma}=[\bm{\sigma}_{x},\bm{\sigma}_{y},\bm{\sigma}_{z}]$
corresponds to the three Pauli matrices liu2020three . As a geometric
representation of a Hopf map, a point on a $\mathbb{S}^{2}$ sphere (a so-
called 2-sphere) corresponds to a closed line (loop) in a real space rather
than a point in a real space, revealing the additional dimension of a
hypersphere. Each loop corresponds to an isospin contour
($(s_{x},s_{y},s_{z})$= const) in the hopfion texture. Extending this
description to optics, $\bm{\chi}$ components correspond to the real and
imaginary parts of the complex-valued fields projected onto the circular
polarisation basis, thus capturing both polarisation and phase variations in a
3D space. In turn, $\mathbf{S}$ is, therefore, formed by Stokes vectors
$S_{1}$, $S_{2}$ and $S_{3}$, describing spatial variations of polarisation of
an optical field, and a 2-sphere is equivalent to a Poincaré sphere. In this
framework, the isospin contours are iso-polarisation contours of an optical
vector field.

The example of the mapping is shown in Fig. 1, with each color of a point on a
parametric 2-sphere representing a spin vector with specific latitude and
longitude, $\alpha$ and $\beta$, shown in Fig. 1(a) with hue color and
lightness, respectively, and the corresponding stereographic projection of an
iso-polarisation contour in a real space is shown as a loop with the same
color (Fig. 1(b,c)). The loops mapped from the points on the same latitude
$\beta$ of a parametric sphere form a set of the torus knots (a set of knots
that lie on the surface of a special torus), completely covering a torus when
scanning the points with different longitudes $\alpha$ (Fig. 1(b)). Thus, the
full unwrapping of the parametric sphere (for all latitudes and longitudes)
results in the torus knots on the nested tori with each torus corresponding to
different latitude $\beta$, akin to matryoshka (Fig. 1(c)); this construction
is termed the Hopf fibration. In two extreme cases of the mapping, the spin-
down contour (left circular polarisation in optics), corresponding to the
south pole of a 2-sphere, is the ring at the core of the nested tori (black
line), while the spin-up contour (right circular polarisation), corresponding
to the north pole, is the line in $z$-direction along the axis of the nested
tori.

How knotted each torus knot is depends on the topological charge (Hopf index,
$Q_{H}$) (see Supplementary Note I and Fig. S1) hilton1953introduction . The
Hopf index (topological charge) is determined by an around-torus index $p$ and
a through-torus index $q$ via $Q_{H}=p\times q$ describing how many times an
isospin contour goes around the torus ($p$) and through the torus hole ($q$).
A fundamental hopfion corresponds to one around and one through a torus
configuration ($Q_{H}$=1).

The 3D hopfion texture is shown in Fig. 1(d), where the colors of vectors
correspond to the isospin contours in (c). The 3D hopfion is closely related
to 2D skyrmions. Hopfion textures have a skyrmionium (a doughnut-shaped
skyrmion) in $xy$ cross-section at $z=0$ (Fig. 1(e)) and the $yz$ cross-
section ($x=0$) corresponds to two skyrmions with opposite vorticity and
opposite skyrmion topological charges (Fig. 1(f)). These are the signatures of
a hopfion since the hopfion can be represented by an end-to-end twisted
skyrmion tube gobel2020topological . Due to the closed connection to
skyrmions, hopfions can be classified by the type of skyrmionium in the $xy$
cross-section, which in turn is classified by the type of 2D skyrmions forming
it (see Supplementary Note II for details) shen2021topological ;
shen2022generation ; gobel2021beyond .

Hopfion is defined by whitehead1947expression  $Q_{H}=p\times
q=\frac{1}{(4\pi)^{2}}\iiint{\mathbf{F}\cdot\mathbf{A}}\text{d}x\text{d}y\text{d}z$,
where
$F_{i}=\varepsilon_{ijk}\mathbf{S}\cdot(\partial_{j}\mathbf{S}\times\partial_{k}\mathbf{S})/2$,
in which $i,j,k=\\{x,y,z\\}$, $\varepsilon$ is the Levi-Civita tensor, and
$\mathbf{A}$ is the vector potential satisfying
$\nabla\times\mathbf{A}=\mathbf{F}$ and is closely related to the skyrmion
number ($p$ and $q$ correspond to the skyrmion numbers of the skyrmioniums in
the $xz$ and $xy$ cross-sections of a hopfion ackerman2015self ), which can be
defined by the product of topological numbers of polarity $Q_{P}$ and
vorticity $Q_{V}$ of the vector field gobel2021beyond ; shen2022topological .
The polarity $Q_{P}=\frac{1}{2}[\cos\beta(r)]_{r=0}^{r=a}=\pm 1$ is defined by
the vector direction down (up) at the center $r=0$ and up (down) at the
skyrmion boundary $r\to r_{\sigma}$ for $Q_{P}=1$ ($Q_{P}=-1$), and the
vorticity $Q_{V}=\frac{1}{2\pi}[\alpha(\phi)]_{\phi=0}^{\phi=2\pi}$ can be an
arbitrary integer, which controls the azimuthal distribution of the transverse
vector field components. For a fixed vorticity, an initial phase $\theta$
should be added to distinguish the helicity to completely decide the
transverse distribution of the vector field, i.e. $\alpha(\phi)=m\phi+\theta$,
which is termed helicity. For instance, skyrmion with vorticity charge
$Q_{V}=1$ represents a hedgehog-like texture for a helicity $\theta=0$ (named
Néel-type-I), a squeezed hedgehog-like texture when helicity is $\pi$ (named
Néel-type-II), and a right- or left-handed vortex texture when helicity is
$\pi/2$ or $3\pi/2$ (named Bloch-type). While, for the vorticity charge of
$-1$, the skyrmion has a saddle-like texture (named anti-type). The
classification can be applied to hopfions, i.e. Néel-, Bloch-, anti-type
hopfions, given that the skyrmion appears in the $xy$ cross-section of the
hopfion.

Using structured light technologies to sculpture complex nonseparable states
of an optical vector field shen2021rays ; shen2022nonseparable ; lsa , a
photonic hopfion can be constructed by the polarization Stokes vectors of a 3D
structured vector beam with the electric field given by
$\bm{\uppsi}(x,y,z)=\psi_{R}(x,y,z)\hat{\mathbf{e}}_{R}+\psi_{L}(x,y,z)\hat{\mathbf{e}}_{L}$,
where $\hat{\mathbf{e}}_{R}$ ($\hat{\mathbf{e}}_{L}$) is the eigenstate of
right-handed (left-handed) circular polarization and $\psi_{R}$ ($\psi_{L}$)
is the corresponding spatial mode. The Stokes vector is related to the
electric field by
$\mathbf{S}=\langle\bm{\uppsi}|\bm{\sigma}|\bm{\uppsi}\rangle$, using the same
form of a Hopf map mentioned above. The developed experimental set-up (see
Supplementary Note III for the details of the experiment) allowed us to
generate photonic hopfions with tuneable topology, via the interference of
controlled Laguerre-Gaussian (LG) beams, and perform 3D tomography of the
Stokes vectors distribution measuring the spatial polarisation distributions.
The iso-polarization contours exactly follow the Hopf fibration when
$\psi_{R}=\text{LG}_{0,0}+\text{LG}_{1,0}$ and $\psi_{L}=\text{LG}_{0,-1}$,
where $\text{LG}_{p,\ell}$ is the LG beam with radial index $p$ and azimuthal
(orbital angular momentum, OAM) index $\ell$ sugic2021particle . Our set-up
provides a universal approach to achieve arbitrary photonic hopfions by
considering
$\psi_{R}=\text{LG}_{0,0}+\text{e}^{\text{i}\varphi}\text{LG}_{1,0}$ and
$\psi_{L}=\text{e}^{\text{i}\theta}\text{LG}_{0,\ell}$, where $\varphi$ and
$\theta$ are two arbitrary intermodal phases, which do not affect the
configuration of the Hopf fibration. Photonic hopfions of arbitrary
topological numbers $Q_{H}$ and with all kinds of hopfionic textures can be
generated, including Néel-, Bloch-, anti-types, and intermediate states. For
example, choosing $\ell=1$, the texture of a photonic hopfion can be switched
by simply tuning $\theta$ to Néel-type-I ($\theta=0$), Bloch-type
($\theta=\pi/2$), Néel-type-II ($\theta=\pi$) (Fig. 2(a-c)). Choosing
$\ell=-1$, the photonic hopfion of the anti-type is observed in the case of
$\theta=\pi$ (Fig. 2(d)).

The experimentally measured polarisation distributions in the optical hopfions
of Néel-type-I ($\ell=1,\theta=0$), Bloch-type ($\ell=1,\theta=\pi/2$), Néel-
type-II ($\ell=1,\theta=\pi$), and Anti-type ($\ell=-1,\theta=\pi$) are in
excellent agreement to the theoretical results (Fig. 2).

Figure 2: Left: Simulated Stokes vectordistributions in the skyrmionium
textures in the $xy$ ($z=0$) plane of the photonic hopfions of (a) Néel-type-I
($Q_{P}=1,Q_{V}=1,\theta=0$), (b) Bloch-type ($Q_{P}=1,Q_{V}=1,\theta=\pi/2$),
(c) Néel-type-II ($Q_{P}=1,Q_{V}=1,\theta=\pi$), and (d) anti-type
($Q_{P}=1,Q_{V}=-1,\theta=\pi$). Right: theoretical and experimental
polarisation distributions, represented by Poincaré parameters (orientation
and ellipticity of the polarisation ellipse) in $xy$ and $yz$ planes for the
topological hopfions in (a-d). The colour scale is as in Fig. 1 (also the same
as in Fig. S4). Figure 3: Left: the torus-knot configurations of a toroidal
layer in the Hopf fibration for the higher-order hopfions with Hopf indices of
(a) $Q_{H}=2$, (b) $Q_{H}=-2$, and (c) $Q_{H}=-3$. Right: theoretical and
experimental polarisation distributions in $xy$ and $yz$ planes for the the
hopfions in (a-c). The colour scale is as in Fig. 2.

In a fundamental-order hopfion, each fiber as a torus knot in the Hopf
fibration goes one time through and one time around the torus, resulting
topological charge of $Q_{H}=\pm 1$ (“$\pm$” decides chirality). For higher-
order hopfions, if each torus-knot fiber goes $p$ times through and $q$ times
around the torus, the topological charge is $Q_{H}=\pm p\times q$
kobayashi2014torus . The vector fields of various higher-order hopfions can
also be characterized by a closed-form expression related to $p$ and $q$ (see
Supplementary Note I). Three examples of the torus-knot configurations of
three higher-order hopfions with $(p,q)=(-2,1)$, $(p,q)=(2,1)$, $(p,q)=(3,2)$
were generated, using LG beams with $\ell>1$ (Fig. 3). In general, for a given
value of $\ell$, the torus-knot indices are determined by
$(p,q)=(\ell,|\ell|-1)$. For each higher-order hopfion, the $xy$ cross-section
shows corresponding higher-order skyrmionium textures with vorticity equals to
$Q_{H}$, while, the $yz$ cross-section still show two fundamental skyrmions
with opposite vorticity.

In the past, a hopfion was always considered as a static topological
quasiparticle. Our developed approach provides a unique possibility stimulate
and observe free-space transport of hopfions with the preserved topology. This
is achieved by tuning the phase parameter $\varphi$. In the initial
invetsigations in Figs. 2 and 3, $\varphi$=0 and the hopfion center is located
at $z=0$. If the phase parameter is tuned, the hopfion configuration
controllably propagates along $z$-axis, so that the hopfion centre moves in a
$z$ direction. For example, when $\varphi=-\pi/2$, the hopfion is centered at
$z=z_{R}$ (where $z_{R}$ is Rayleigh length). When the value of $\varphi$ is
gradually increased to $\pi/2$, the hopfion center propagates to $z=z_{R}$
(Fig. 4). This propagation effect can be explained and simulated considering
the Gouy phase mediated intramodal phase, resulting in the amplitude profile
of $\psi_{R}$ variation along the radial direction upon propagation
zhong2021gouy , which is crucial for determining a 3D polarisation texture of
hopfions. As the initial intramodal phase $\varphi$ changed, the center of a
hopfion ring, which corresponds to a plane with $\varphi=0$, would move along
the $z$-direction (Supplementary Note IV).

Figure 4: Propagation of photonic hopfions in a free space. Theoretical (top)
and experimental (bottom) polarisation distributions in $xz$ plane for
different $\varphi$ from $-\pi/2$ to $\pi/2$. The hopfion torus center moves
along $z$ from $z=-z_{R}$ to $z=z_{R}$. The grey dashed lines mark the
location of a hopfion torus centers. Grey arrows indicate the vorticity. The
colour scale is as in Fig. 2.

In summary, we presented both theoretical and experimental demonstrations of
reconfigurable photonic hopfions of higher-orders and their free-space
transport. With the development of a high-precision all-digital generation of
hopfions, the topological spin textures including Néel-, Bloch-, and anti-
types can be flexibly manipulated and transformed into each other,
demonstrating excellent agreement with the theoretical simulations. The
proposed and implemented mechanism to propagate a hopfion topological
structure in free space maybe of interest for development of topological
classical and quantum information carriers for optical communications and data
manipulation.

Acknowledgments. This work was supported, in part, by National Natural Science
Foundation of China (grants No. 62075050, 11934013, 61975047), High-Level
Talents Project of Heilongjiang Province (2020GSP12), and the European
Research Council iCOMM project (789340).

## References

  * (1) Manton, N. & Sutcliffe, P.  Topological solitons (Cambridge University Press, 2004).
  * (2) Parkin, S. S., Hayashi, M. & Thomas, L.  Magnetic domain-wall racetrack memory.  Science 320, 190–194 (2008).
  * (3) Wang, X., Yan, P., Shen, Y., Bauer, G. E. & Wang, X.  Domain wall propagation through spin wave emission.  Physical Review Letters 109, 167209 (2012).
  * (4) Ryu, K.-S., Thomas, L., Yang, S.-H. & Parkin, S.  Chiral spin torque at magnetic domain walls.  Nature nanotechnology 8, 527–533 (2013).
  * (5) Han, J., Zhang, P., Hou, J. T., Siddiqui, S. A. & Liu, L.  Mutual control of coherent spin waves and magnetic domain walls in a magnonic device.  Science 366, 1121–1125 (2019).
  * (6) Jué, E. et al. Chiral damping of magnetic domain walls.  Nature materials 15, 272–277 (2016).
  * (7) Tsesses, S. et al. Optical skyrmion lattice in evanescent electromagnetic fields.  Science 361, 993–996 (2018).
  * (8) Du, L., Yang, A., Zayats, A. V. & Yuan, X.  Deep-subwavelength features of photonic skyrmions in a confined electromagnetic field with orbital angular momentum.  Nature Physics 15, 650–654 (2019).
  * (9) Davis, T. J. et al. Ultrafast vector imaging of plasmonic skyrmion dynamics with deep subwavelength resolution.  Science 368 (2020).
  * (10) Shen, Y., Hou, Y., Papasimakis, N. & Zheludev, N. I.  Supertoroidal light pulses as electromagnetic skyrmions propagating in free space.  Nature communications 12, 5891 (2021).
  * (11) Jani, H. et al. Antiferromagnetic half-skyrmions and bimerons at room temperature.  Nature 590, 74–79 (2021).
  * (12) Dai, Y. et al. Plasmonic topological quasiparticle on the nanometre and femtosecond scales.  Nature 588, 616–619 (2020).
  * (13) Shen, Y.  Topological bimeronic beams.  Optics Letters 46, 3737–3740 (2021).
  * (14) Shen, Y., Martínez, E. C. & Rosales-Guzmán, C.  Generation of optical skyrmions with tunable topological textures.  ACS Photonics 9, 296–303 (2022).
  * (15) Göbel, B., Mertig, I. & Tretiakov, O. A.  Beyond skyrmions: Review and perspectives of alternative magnetic quasiparticles.  Physics Reports 895, 1–28 (2021).
  * (16) Fert, A., Reyren, N. & Cros, V.  Magnetic skyrmions: advances in physics and potential applications.  Nature Reviews Materials 2, 1–15 (2017).
  * (17) Bernevig, B. A., Felser, C. & Beidenkopf, H.  Progress and prospects in magnetic topological materials.  Nature 603, 41–51 (2022).
  * (18) Ackerman, P. J. & Smalyukh, I. I.  Static three-dimensional topological solitons in fluid chiral ferromagnets and colloids.  Nature materials 16, 426–432 (2017).
  * (19) Tang, J. et al. Magnetic skyrmion bundles and their current-driven dynamics.  Nature Nanotechnology 1–6 (2021).
  * (20) Zheng, F. et al. Magnetic skyrmion braids.  Nature Communications 12, 5316 (2021).
  * (21) Faddeev, L.  Some comments on the many-dimensional solitons.  Letters in Mathematical Physics 1, 289–293 (1976).
  * (22) Skyrme, T. H. R.  A non-linear field theory.  Proc. R. Soc. A 260, 127–138 (1961).
  * (23) Skyrme, T. H. R.  A unified field theory of mesons and baryons.  Nuclear Physics 31, 556–569 (1962).
  * (24) Hietarinta, J. & Salo, P.  Faddeev-hopf knots: dynamics of linked un-knots.  Physics Letters B 451, 60–67 (1999).
  * (25) Sutcliffe, P.  Knots in the skyrme–faddeev model.  Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 463, 3001–3020 (2007).
  * (26) Lyons, D. W.  An elementary introduction to the hopf fibration.  Mathematics magazine 76, 87–98 (2003).
  * (27) Whitehead, J.  An expression of hopf’s invariant as an integral.  Proceedings of the National Academy of Sciences of the United States of America 33, 117 (1947).
  * (28) Faddeev, L. & Niemi, A. J.  Stable knot-like structures in classical field theory.  Nature 387, 58–61 (1997).
  * (29) Tai, J.-S. B., Smalyukh, I. I. et al. Static hopf solitons and knotted emergent fields in solid-state noncentrosymmetric magnetic nanostructures.  Physical Review Letters 121, 187201 (2018).
  * (30) Wang, X., Qaiumzadeh, A. & Brataas, A.  Current-driven dynamics of magnetic hopfions.  Physical Review Letters 123, 147203 (2019).
  * (31) Liu, Y., Hou, W., Han, X. & Zang, J.  Three-dimensional dynamics of a magnetic hopfion driven by spin transfer torque.  Physical Review Letters 124, 127204 (2020).
  * (32) Luk’Yanchuk, I., Tikhonov, Y., Razumnaya, A. & Vinokur, V.  Hopfions emerge in ferroelectrics.  Nature Communications 11, 1–7 (2020).
  * (33) Göbel, B., Akosa, C. A., Tatara, G. & Mertig, I.  Topological hall signatures of magnetic hopfions.  Physical Review Research 2, 013315 (2020).
  * (34) Kent, N. et al. Creation and observation of hopfions in magnetic multilayer systems.  Nature Communications 12, 1–7 (2021).
  * (35) Hall, D. S. et al. Tying quantum knots.  Nature Physics 12, 478–483 (2016).
  * (36) Lee, W. et al. Synthetic electromagnetic knot in a three-dimensional skyrmion.  Science advances 4, eaao3820 (2018).
  * (37) Babaev, E.  Dual neutral variables and knot solitons in triplet superconductors.  Physical Review Letters 88, 177002 (2002).
  * (38) Kawaguchi, Y., Nitta, M. & Ueda, M.  Knots in a spinor bose-einstein condensate.  Physical Review Letters 100, 180403 (2008).
  * (39) Bisset, R. et al. Robust vortex lines, vortex rings, and hopfions in three-dimensional bose-einstein condensates.  Physical Review A 92, 063611 (2015).
  * (40) Zou, S., Bai, W.-K., Yang, T. & Liu, W.-M.  Formation of vortex rings and hopfions in trapped bose–einstein condensates.  Physics of Fluids 33, 027105 (2021).
  * (41) Cruz, M., Turok, N., Vielva, P., Martinez-Gonzalez, E. & Hobson, M.  A cosmic microwave background feature consistent with a cosmic texture.  Science 318, 1612–1614 (2007).
  * (42) Thompson, A., Wickes, A., Swearngin, J. & Bouwmeester, D.  Classification of electromagnetic and gravitational hopfions by algebraic type.  Journal of Physics A: Mathematical and Theoretical 48, 205202 (2015).
  * (43) Kleckner, D. & Irvine, W. T.  Creation and dynamics of knotted vortices.  Nature physics 9, 253–258 (2013).
  * (44) Ackerman, P. J. & Smalyukh, I. I.  Diversity of knot solitons in liquid crystals manifested by linking of preimages in torons and hopfions.  Physical Review X 7, 011006 (2017).
  * (45) Ackerman, P. J., Van De Lagemaat, J. & Smalyukh, I. I.  Self-assembly and electrostriction of arrays and chains of hopfion particles in chiral liquid crystals.  Nature communications 6, 1–9 (2015).
  * (46) Sugic, D. et al. Particle-like topologies in light.  Nature communications 12, 1–10 (2021).
  * (47) Hilton, P.  An introduction to homotopy theory (Cambridge University Press, 1953).
  * (48) Shen, Y. et al. Topological quasiparticles of light: Optical skyrmions and beyond.  arXiv preprint arXiv:2205.10329 (2022).
  * (49) Shen, Y.  Rays, waves, su (2) symmetry and geometry: toolkits for structured light.  Journal of Optics 23, 124004 (2021).
  * (50) Shen, Y. & Rosales-Guzmán, C.  Nonseparable states of light: From quantum to classical.  Laser & Photonics Reviews 2100533 (2022).
  * (51) He, C., Shen, Y. & Forbes, A.  Towards higher-dimensional structured light.  Light: Science & Applications 11, 205 (2022).
  * (52) Kobayashi, M. & Nitta, M.  Torus knots as hopfions.  Physics Letters B 728, 314–318 (2014).
  * (53) Zhong, R.-Y. et al. Gouy-phase-mediated propagation variations and revivals of transverse structure in vectorially structured light.  Physical Review A 103, 053520 (2021).


# Branching Time Active Inference with Bayesian Filtering

Théophile Champion<EMAIL_ADDRESS>
University of Kent, School of Computing
Canterbury CT2 7NZ, United Kingdom Marek Grześ<EMAIL_ADDRESS>
University of Kent, School of Computing
Canterbury CT2 7NZ, United Kingdom Howard Bowman<EMAIL_ADDRESS>
University of Birmingham, School of Psychology,
Birmingham B15 2TT, United Kingdom
University of Kent, School of Computing
Canterbury CT2 7NZ, United Kingdom

###### Abstract

Branching Time Active Inference (Champion et al., 2021b, a) is a framework
proposing to look at planning as a form of Bayesian model expansion. Its root
can be found in Active Inference (Friston et al., 2016; Da Costa et al., 2020;
Champion et al., 2021c), a neuroscientific framework widely used for brain
modelling, as well as in Monte Carlo Tree Search (Browne et al., 2012), a
method broadly applied in the Reinforcement Learning literature. Up to now,
the inference of the latent variables was carried out by taking advantage of
the flexibility offered by Variational Message Passing (Winn and Bishop,
2005), an iterative process that can be understood as sending messages along
the edges of a factor graph (Forney, 2001). In this paper, we harness the
efficiency of an alternative method for inference called Bayesian Filtering
(Fox et al., 2003), which does not require the iteration of the update
equations until convergence of the Variational Free Energy. Instead, this
scheme alternates between two phases: integration of evidence and prediction
of future states. Both of those phases can be performed efficiently and this
provides a seventy times speed up over the state-of-the-art.

Keywords: Branching Time Active Inference, Bayesian Filtering, Free Energy
Principle

##  1 Introduction

Active inference extends the free energy principle to generative models with
actions (Friston et al., 2016; Da Costa et al., 2020; Champion et al., 2021c)
and can be regarded as a form of planning as inference (Botvinick and
Toussaint, 2012). Over the years, this framework has successfully explained a
wide range of brain phenomena, such as habit formation (Friston et al., 2016),
Bayesian surprise (Itti and Baldi, 2009), curiosity (Schwartenbeck et al.,
2018), and dopaminergic discharge (FitzGerald et al., 2015). It has also been
applied to a variety of tasks such as navigation in the Animal AI environment
(Fountas et al., 2020), robotic control (Pezzato et al., 2020; Sancaktar et
al., 2020), the mountain car problem (Çatal et al., 2020), the game DOOM
(Cullen et al., 2018) and the cart pole problem (Millidge, 2019).

However, because active inference defines the prior over policies as a joint
distribution over the space of all possible policies, the method suffers from
an exponential space and time complexity class. In the reinforcement learning
literature, this problem can be tackled using Monte Carlo tree search (MCTS)
(Browne et al., 2012), whose origins can be found in the multi-armed bandit
problem (Auer et al., 2002). More recently, MCTS has been applied to a large
number of tasks such as the game of Go (Silver et al., 2016), the Animal AI
environment (Fountas et al., 2020), and many others.

More recently, Branching Time Active Inference (BTAI) (Champion et al., 2021b,
a) proposed that planning is a form of Bayesian model expansion guided by the
upper confidence bound for trees (UCT) criterion from the MCTS literature,
i.e. a quantity from the multi-armed bandit problem whose objective is to
minimize the agent’s regret. And because the generative model is dynamically
expanded, variational message passing (VMP) (Winn and Bishop, 2005) was used
to carry out inference over the latent variables. VMP can be understood as a
flexible iterative process that sends messages along the edges of a factor
graph (Forney, 2001), and computes posterior beliefs by summing those messages
together.

Bayesian filtering (BF) (Fox et al., 2003) is an alternative inference method
composed of two phases. In the first phase, Bayes theorem is used to compute
posterior beliefs each time a new observation is obtained from the
environment. In the second phase, posterior beliefs over the present state
($S_{t}$) are used to predict posterior beliefs over the state at the next
time step ($S_{t+1}$). Importantly, this process is not iterative within a
time step, i.e., it only contains a forward pass, and therefore is much more
efficient than VMP.

In Section 2, we present the theory underlying Branching Time Active Inference
when using Bayesian filtering for inference over latent variables. In Section
3, we show that using Bayesian filtering instead of variational message
passing for the inference process provides BTAI with a seventy times speed-up
while maintaining effective planning. Finally, Section 4 concludes this paper
and discusses avenues for future research.

##  2 Branching Time Active Inference with Bayesian Filtering
($\text{BTAI}_{\text{BF}}$)

In this section, we describe the theory underlying our approach. For any
notational uncertainty the reader is referred to Appendix F of Champion et al.
(2021b). We let $\bm{D}$ be a 1-tensor representing the prior over initial
hidden states $P(S_{0})$. Let $\bm{A}$ be a 2-tensor representing the
likelihood mapping $P(O_{\tau}|S_{\tau})$, and $\bm{B}$ be a 3-tensor
representing the transition mapping $P(S_{\tau+1}|S_{\tau},U_{\tau})$.
Additionally, we let $\mathbb{I}$ be the set of multi-indices containing all
the policies (i.e., sequences of actions) that have been explored by the
model. The generative model of BTAI with BF can be formally written as the
following joint distribution:

$\displaystyle P(O_{0},S_{0},O_{\mathbb{I}},S_{\mathbb{I}})=$ $\displaystyle
P(O_{0}|S_{0})P(S_{0})\prod_{I\in\mathbb{I}}P(O_{I}|S_{I})P(S_{I}|S_{I\setminus\text{last}})$

where $S_{I\setminus\text{last}}$ is the parent of $S_{I}$, and:

$\displaystyle P(S_{0})=\text{Cat}(\bm{D})$ $\displaystyle
P(O_{\tau}|S_{\tau})=\text{Cat}(\bm{A})$ $\displaystyle
P(O_{I}|S_{I})=\text{Cat}(\bm{A})$ $\displaystyle
P(S_{I}|S_{I\setminus\text{last}})=\text{Cat}(\bm{B}_{I}).$

where
$\bm{B}_{I}=\bm{B}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}},\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}},I_{\text{last}})$
is the 2-tensor corresponding to $I_{last}$ (i.e., the last action that led to
$S_{I}$), and the likelihood mapping in the past, i.e.,
$P(O_{\tau}|S_{\tau})$, and in the future, i.e., $P(O_{I}|S_{I})$, are both
categorical distributions with parameters $\bm{A}$. This generative model is
depicted in Figure 1, where we assume that the current time step $t$ equals
zero.

Figure 1:  This figure illustrates the expandable generative model used by the
BTAI with BF agent. The future is a tree like generative model whose branches
correspond to the policies considered by the agent. The branches can be
dynamically expanded during planning and the nodes in light gray represent
possible expansions of the current generative model.

Initially, the generative model only contains the initial state $S_{0}$ and
observation $O_{0}$. The prior over the hidden state is known, i.e.
$P(S_{0})=\text{Cat}(\bm{D})$, as well as the likelihood, i.e.,
$P(O_{\tau}|S_{\tau})=\text{Cat}(\bm{A})$, and $P(O_{0})$, the evidence, can
be computed in the usual way by marginalizing over
$P(O_{0},S_{0})=P(O_{0}|S_{0})P(S_{0})$. Thus, we can integrate the evidence
provided to us by the initial observation $O_{0}$ using Bayes Theorem:

$\displaystyle\mathcal{B}(S_{0})=\frac{P(O_{0}|S_{0})P(S_{0})}{P(O_{0})},$ (1)

where $\mathcal{B}(S_{0})$ are the beliefs over the initial hidden state.
Then, we use the UCT criterion to determine which node in the tree should be
expanded. Let the tree’s root $S_{0}$ be called the current node. If the
current node has no children, then it is selected for expansion.
Alternatively, the child with the highest UCT criterion becomes the new
current node and the process is iterated until we reach a leaf node (i.e. a
node from which no action has previously been selected). The UCT criterion
(Browne et al., 2012) for the $j$-th child of the current node is given by:

$\displaystyle UCT_{j}=-\bar{\bm{G}}_{j}+C_{explore}\sqrt{\frac{\ln
n}{n_{j}}},$ (2)

where $\bar{\bm{G}}_{j}$ is the average expected free energy calculated with
respected to the actions selected from the $j$-th child, $C_{explore}$ is the
exploration constant that modulates the amount of exploration at the tree
level, $n$ is the number of times the current node has been visited, and
$n_{j}$ is the number of times the $j$-th child has been visited.

Let $S_{I}$ be the (leaf) node selected by the above selection procedure. We
then expand all the children of $S_{I}$, i.e., all the states of the form
$S_{I::U}$ where $U\in\\{1,...,|U|\\}$ is an arbitrary action, and $I::U$ is
the multi-index obtained by appending the action $U$ at the end of the
sequence defined by $I$. Next, we compute the predicted beliefs over those
expanded hidden states using the transition mapping:

$\displaystyle\mathcal{B}(S_{J})=\mathbb{E}_{\mathcal{B}(S_{I})}\big{[}P(S_{J}|S_{I})\big{]},$
(3)

where we let $J=I::U$ for any action $U$, $\mathcal{B}(S_{I})$ are the
predicted posterior beliefs over $S_{I}$, and according to our generative
model $P(S_{J}|S_{I})=\text{Cat}(\bm{B}_{J})$ with
$\bm{B}_{J}=\bm{B}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}},\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}},J_{\text{last}})$.
The above equation corresponds to the second phase of Bayesian filtering,
i.e., the prediction phase, which involves the calculation of new beliefs,
using the generative model, in the absence of new observations. Then, we need
to estimate the cost of (virtually) taking each possible action. The cost in
this paper is taken to be the expected free energy (Friston et al., 2017):

$\displaystyle\bm{G}_{J}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}D_{\mathrm{KL}}[\mathcal{B}(O_{J})||V(O_{J})]\,\,+\,\,\mathbb{E}_{\mathcal{B}(S_{J})}[\text{H}[P(O_{J}|S_{J})]]$

where the prior preferences over future observations are specified by the
modeller as $V(O_{J})=\text{Cat}(\bm{C})$, according to the generative model
$P(O_{J}|S_{J})=\text{Cat}(\bm{A})$, and the posterior beliefs over future
observations are computed by prediction as follows:

$\displaystyle\mathcal{B}(O_{J})=\mathbb{E}_{\mathcal{B}(S_{J})}[P(O_{J}|S_{J})].$

Next, we assume that the agent will always perform the action with the lowest
cost, and back-propagate the cost of the best (virtual) action toward the root
of the tree. Formally, we write the update as follows:

$\displaystyle\forall
K\in\mathbb{A}_{I}\cup\\{I\\},\quad\bm{G}_{K}\leftarrow\bm{G}_{K}+\min_{U\in\\{1,...,|U|\\}}\bm{G}_{I::U},$
(4)

where $I$ is the multi-index of the node that was selected for (virtual)
expansion, and $\mathbb{A}_{I}$ is the set of all multi-indices corresponding
to ancestors of $S_{I}$. During the back propagation, we also update the
number of visits as follows:

$\displaystyle\forall K\in\mathbb{A}_{I}\cup\\{I\\},\quad n_{K}\leftarrow
n_{K}+1.$ (5)

If we let $\bm{G}^{aggr}_{K}$ be the aggregated cost of an arbitrary node
$S_{K}$ obtained by applying Equation 4 after each expansion, then we are now
able to express $\bar{\bm{G}}_{K}$ formally as:

$\bar{\bm{G}}_{K}=\frac{\bm{G}^{aggr}_{K}}{n_{K}}.$

The planning procedure described above ends when the maximum number of
planning iterations is reached, and the action corresponding to the root’s
child with the lowest average cost is performed in the environment. At this
point, the agent receives a new observation $O_{\tau}$ and needs to update its
beliefs over $S_{\tau}$. First, we predict the posterior beliefs over
$S_{\tau}$ as follows:

$\displaystyle\mathcal{B}(S_{\tau}|U_{\tau-1}=U^{*})=\mathbb{E}_{\mathcal{B}(S_{\tau-1})}\big{[}P(S_{\tau}|S_{\tau-1},U_{\tau-1}=U^{*})\big{]},$
(6)

where $U^{*}$ is the action performed (from the root) in the environment,
$P(S_{\tau}|S_{\tau-1},U_{\tau-1}=U^{*})$ is the 2-tensor
$\bm{B}(\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}},\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}},U^{*})$,
and $\mathcal{B}(S_{\tau-1})$ is the agent’s posterior beliefs over the state
at time $\tau-1$, e.g., after performing the first action in the environment,
$\tau=1$ and $\mathcal{B}(S_{\tau-1})=\mathcal{B}(S_{0})$ as given by Equation
1. Second, we integrate the evidence provided by the new observation
$O_{\tau}$ using Bayes theorem:

$\displaystyle\mathcal{B}(S_{\tau})=\frac{P(O_{\tau}|S_{\tau})\mathcal{B}(S_{\tau}|U_{\tau-1}=U^{*})}{P(O_{\tau})},$
(7)

where $\mathcal{B}(S_{\tau}|U_{\tau-1}=U^{*})$ is used as an empirical prior.
By an empirical prior we mean a posterior distribution of the previous time
step, e.g., $\mathcal{B}(S_{\tau}|U_{\tau-1}=U^{*})$, that is used as a prior
in Bayes theorem. Algorithm 1 concludes this section by summarizing our
approach.

Input : $env$ the environment, $O_{0}$ the initial observation, $\bm{A}$ the
likelihood mapping, $\bm{B}$ the transition mapping, $\bm{C}$ the prior
preferences, $\bm{D}$ the prior over initial states, $N$ the number of
planning iterations, $M$ the number of action-perception cycles.

$\mathcal{B}(S_{0})\leftarrow$ IntegrateEvidence($O_{0}$, $\bm{A}$, $\bm{D}$)

// Using (1)

$root\leftarrow$ CreateTreeNode(beliefs = $\mathcal{B}(S_{0})$, action = -1,
cost = 0, visits = 1)

// Where -1 in the line above is a dummy value

repeat  _$M$_ times

repeat  _$N$_ times

$node\leftarrow$ SelectNode($root$)

// Using (2) recursively

$eNodes\leftarrow$ ExpandChildren($node$, $\bm{B}$)

// Using (3) for each action

Evaluate($eNodes$, $\bm{A}$, $\bm{C}$)

// Compute (2) for each expanded node

Backpropagate($eNodes$)

// Using (4) and (5)

end

$U^{*}\leftarrow$ SelectAction($root$)

// Such that $U^{*}$ minimises the average cost

$O_{\tau}\leftarrow$ $env$.Execute($U^{*}$)

$\mathcal{B}(S_{\tau-1})\leftarrow root.beliefs$

// Get beliefs of the root node

$\mathcal{B}(S_{\tau}|U_{\tau-1}=U^{*})\leftarrow$
ComputeEmpiricalPrior($\bm{B}$, $\mathcal{B}(S_{\tau-1})$, $U^{*}$)

// Using (6)

$\mathcal{B}(S_{\tau})\leftarrow$ IntegrateEvidence($O_{\tau}$, $\bm{A}$,
$\mathcal{B}(S_{\tau}|U_{\tau-1}=U^{*})$)

// Using (7)

$root\leftarrow$ CreateTreeNode(beliefs = $\mathcal{B}(S_{\tau})$, action =
$U^{*}$, cost = 0, visits = 1)

end

Algorithm 1 BTAI with BF: action-perception cycles (with relevant equations
indicated in round brackets).

##  3 Results

In this section, we first present the deep reward environment in which two
versions of BTAI will be compared. Then, we present experimental results
comparing BTAI with VMP and BTAI with BF in terms of running time and
performance.

###  3.1 Deep reward environment

This environment is called the deep reward environment because the agent needs
to navigate a tree-like graph where the graph’s nodes correspond to the states
of the system, and the agent needs to look deep into the future to
diferentiate the good path from the traps. At the beginning of each trial, the
agent is placed at the root of the tree, i.e., the initial state of the
system. From the initial state, the agent can perform $n+m$ actions, where $n$
and $m$ are the number of good and bad paths, respectively. Additionally, at
any point in time, the agent can make two observations: a pleasant one or an
unpleasant one. The states of the good paths produce pleasant observations,
while the states of the bad paths produce unpleasant ones.

If the first action selected was one of the $m$ bad actions, then the agent
will enter a bad path in which $n+m$ actions are available at each time step
but all of them produce unpleasant observations. If the first action selected
was one of the $n$ good actions, then the agent will enter the associated good
path. We let $L_{k}$ be the length of the $k$-th good path. Once the agent is
engaged on the $k$-th path, there are still $n+m$ actions available but only
one of them keeps the agent on the good path. All the other actions will
produce unpleasant observations, i.e., the agent will enter a bad path.

This process will continue until the agent reaches the end of the $k$-th path,
which is determined by the path’s length $L_{k}$. If the $k$-th path was the
longest of the $n$ good paths, then the agent will from now on only receive
pleasant observations independently of the action performed. If the $k$-th
path was not the longest path, then independently of the action performed the
agent will enter a bad path.

To summarize, at the beginning of each trial, the agent is prompted with $n$
good paths and $m$ bad paths. Only the longest good path will be beneficial in
the long term, the others are traps, which will ultimately lead the agent to a
bad state. Figure 2 illustrates this environment.

Figure 2:  This figure illustrates a type of deep reward environment where
$S_{0}$ represents the initial state, $S_{b}$ represents a bad state, $S_{g}$
represents a good state, and $S^{i}_{j}$ is the $j$-th state of the $i$-th
good path. Also, the longest path in the above picture is the first good path
whose length $L_{1}$ is equal to two. Importantly, the longest path
corresponds to the only good path that does not turn out to be a trap.

###  3.2 BTAI with VMP versus BTAI with BF

In this section, we compare BTAI with VMP and BTAI with BF in terms of running
time and performance. The running time reported in Tables 1 and 2 was obtained
by running 100 trials each composed of 20 action-perception cycles. Also, the
trial was stopped whenever the agent reached a bad state or the goal state. As
shown in Tables 1 and 2, both approaches were able to solve the tasks.
However, BTAI with BF ran $69\pm 26$ times faster than BTAI with VMP.

This speed up is possible for two reasons. First, Bayesian filtering does not
require the iteration of the belief updates until convergence of the
variational free energy. Second, when computing the optimal posterior over a
random variable $X$, VMP needs to compute one message for each adjacent
variable of $X$, add them together, and normalise using a softmax function. In
contrast, BF only performs a forward pass, which is essentially implemented as
matrix multiplications.

$n$ | $m$ |  $L_{1}$, $L_{2}$, …, $L_{n}$ | # planning iterations | P(goal) | P(bad) | Running time
---|---|---|---|---|---|---
2 | 5 | 5, 8 | 25 | 1 | 0 | 2.294 sec
2 | 5 | 5, 8 | 50 | 1 | 0 | 4.688 sec
2 | 5 | 5, 8 | 100 | 1 | 0 | 9.045 sec
3 | 5 | 6, 5, 8 | 25 | 1 | 0 | 2.805 sec
3 | 5 | 6, 5, 8 | 50 | 1 | 0 | 5.416 sec
3 | 5 | 6, 5, 8 | 100 | 1 | 0 | 11.288 sec
Table 1: This table presents the results of BTAI with BF on various deep reward environments. Recall, that $n$ and $m$ are the number of good and bad paths, respectively. $L_{i}$ is the length of the $i$-th good path. $P(goal)$ reports the probability of reaching the goal state (i.e., the agent successfully picked the longest path), and $P(bad)$ reports the probability of reaching the bad state (i.e., either by picking a bad action directly of by falling into a trap). $n$ | $m$ |  $L_{1}$, $L_{2}$, …, $L_{n}$ | # planning iterations | P(goal) | P(bad) | Running time
---|---|---|---|---|---|---
2 | 5 | 5, 8 | 25 | 1 | 0 | 4 min 16 sec
2 | 5 | 5, 8 | 50 | 1 | 0 | 4 min 42 sec
2 | 5 | 5, 8 | 100 | 1 | 0 | 6 min 26 sec
3 | 5 | 6, 5, 8 | 25 | 1 | 0 | 4 min 31 sec
3 | 5 | 6, 5, 8 | 50 | 1 | 0 | 5 min 35 sec
3 | 5 | 6, 5, 8 | 100 | 1 | 0 | 7 min 48 sec
Table 2: This table presents the results of BTAI with VMP on various deep
reward environments. Recall, that $n$ and $m$ are the number of good and bad
paths, respectively. $L_{i}$ is the length of the $i$-th good path. $P(goal)$
reports the probability of reaching the goal state (i.e., the agent
successfully picked the longest path), and $P(bad)$ reports the probability of
reaching the bad state (i.e., either by picking a bad action directly of by
falling into a trap).

##  4 Conclusion and future works

In this paper, we proposed a new implementation of Branching Time Active
Inference (Champion et al., 2021b, a), where the inference is carried out
using Bayesian filtering (Fox et al., 2003), instead of using variational
message passing (Champion et al., 2021c; Winn and Bishop, 2005).

This new approach has a few advantages. First, it achieves the same
performance as its predecessor around seventy times faster. Second, the
implementation is simpler and less data structures need to be stored in
memory.

Also, one could argue that there is a trade-off in the nature and extent of
the information inferred by classic active inference, branching-time active
inference with variational message passing ($\text{BTAI}_{\text{VMP}}$) from
Champion et al. (2021b, a), and branching-time active inference with Bayesian
Filtering ($\text{BTAI}_{\text{BF}}$). Specifically, classic active inference
exhaustively represents and updates all possible policies, while
$\text{BTAI}_{\text{VMP}}$ will typically only represent one policly in the
past (i.e., the one undertaken by the agent) and a small subset of the
possible (future) trajectories. These will typically be the more advantageous
paths for the agent to pursue, with the less beneficial paths not represented
at all. Indeed, the tree search is based on the expected free energy that
favors policies that maximize information gain, while realizing the prior
preferences of the agent. $\text{BTAI}_{\text{BF}}$ stores even less data than
$\text{BTAI}_{\text{VMP}}$, because the sequence of past hidden states is
discarded as time passes, and only the beliefs over the current and future
states are stored.

Additionally, full variational inference can update the system’s understanding
of past contingencies on the basis of new observations. As a result, the
system can obtain more refined information about previous decisions, perhaps
re-evaluating the optimality of these past decisions. Because classic active
inference represents a larger space of policies, this re-evaluation could
apply to more policies. When using Bayesian filtering, beliefs about past
hidden states are discarded as time progresses, which makes Bayesian belief
updating (about past hidden states) impossible.

We also know that humans engage in counterfactual reasoning (Rafetseder et
al., 2013), which, in our planning context, could involve the entertainment
and evaluation of alternative (non-selected) sequences of decisions. It may be
that, because of the more exhaustive representation of possible trajectories,
classic active inference can more efficiently engage in counterfactual
reasoning. In contrast, branching-time active inference would require these
alternative pasts to be generated “a fresh” for each counterfactual
deliberation. In this sense, one might argue that there is a trade-off:
branching-time active inference provides considerably more efficient planning
to attain current goals, classic active inference provides a more exhaustive
assessment of paths not taken. In contrast, branching time active inference
implemented with Bayesian filtering does not leave a memory at all, let alone
one upon which conterfactual reasoning could be realized.

The implementation of Branching Time Active Inference with variational message
passing can be found here: $https://github.com/ChampiB/Homing-Pigeon$, and the
implementation of Branching Time Active Inference with Bayesian Filtering is
available on Github:
$https://github.com/ChampiB/Branching\\_Time\\_Active\\_Inference$.

Even with this seventy times speed up, BTAI is still unable to deal with large
scale observations such as images. Adding deep neural networks to approximate
the likelihood mapping is therefore a compelling direction for future
research.

Also, this framework is currently limited to discrete action and state spaces.
Designing a continuous extension of BTAI would enable its application to a
wider range of applications such as robotic control with continuous actions.

Finally, as the depth of the tree increases, the beliefs about future states
tend to become more and more uncertain, which can lead to a drop in
performance. This suggests that there exists an optimal number of planning
iterations, after which the model simply does not have enough information to
keep planning. Future work could thus focus on automatically identifying this
optimal number of planning iterations, in order to improve the robustness of
the approach.

Acknowledgments

TO BE FILLED

## References

  * Auer et al. (2002) Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer.  Finite-time analysis of the multiarmed bandit problem.  _Machine Learning_ , 47(2):235–256, May 2002\.  ISSN 1573-0565.  doi: 10.1023/A:1013689704352.  URL https://doi.org/10.1023/A:1013689704352.
  * Botvinick and Toussaint (2012) Matthew Botvinick and Marc Toussaint.  Planning as inference.  _Trends in Cognitive Sciences_ , 16(10):485 – 488, 2012.  ISSN 1364-6613.  doi: https://doi.org/10.1016/j.tics.2012.08.006.
  * Browne et al. (2012) C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton.  A survey of Monte Carlo tree search methods.  _IEEE Transactions on Computational Intelligence and AI in Games_ , 4(1):1–43, 2012.
  * Champion et al. (2021a) Théophile Champion, Howard Bowman, and Marek Grześ.  Branching time active inference: empirical study.  _arXiv_ , 2021a.
  * Champion et al. (2021b) Théophile Champion, Howard Bowman, and Marek Grześ.  Branching time active inference: the theory and its generality.  _arXiv_ , 2021b.
  * Champion et al. (2021c) Théophile Champion, Marek Grześ, and Howard Bowman.  Realizing Active Inference in Variational Message Passing: The Outcome-Blind Certainty Seeker.  _Neural Computation_ , 33(10):2762–2826, 09 2021c.  ISSN 0899-7667.  doi: 10.1162/neco_a_01422.  URL https://doi.org/10.1162/neco_a_01422.
  * Cullen et al. (2018) Maell Cullen, Ben Davey, Karl J. Friston, and Rosalyn J. Moran.  Active inference in OpenAI Gym: A paradigm for computational investigations into psychiatric illness.  _Biological Psychiatry: Cognitive Neuroscience and Neuroimaging_ , 3(9):809 – 818, 2018.  ISSN 2451-9022.  doi: https://doi.org/10.1016/j.bpsc.2018.06.010.  URL http://www.sciencedirect.com/science/article/pii/S2451902218301617.  Computational Methods and Modeling in Psychiatry.
  * Da Costa et al. (2020) Lancelot Da Costa, Thomas Parr, Noor Sajid, Sebastijan Veselic, Victorita Neacsu, and Karl Friston.  Active inference on discrete state-spaces: A synthesis.  _Journal of Mathematical Psychology_ , 99:102447, 2020.  ISSN 0022-2496.  doi: https://doi.org/10.1016/j.jmp.2020.102447.  URL https://www.sciencedirect.com/science/article/pii/S0022249620300857.
  * FitzGerald et al. (2015) Thomas H. B. FitzGerald, Raymond J. Dolan, and Karl Friston.  Dopamine, reward learning, and active inference.  _Frontiers in Computational Neuroscience_ , 9:136, 2015\.  ISSN 1662-5188.  doi: 10.3389/fncom.2015.00136.  URL https://www.frontiersin.org/article/10.3389/fncom.2015.00136.
  * Forney (2001) G. D. Forney.  Codes on graphs: normal realizations.  _IEEE Transactions on Information Theory_ , 47(2):520–548, 2001.
  * Fountas et al. (2020) Zafeirios Fountas, Noor Sajid, Pedro A. M. Mediano, and Karl Friston.  Deep active inference agents using Monte-Carlo methods.  _arXiv_ , 2020.
  * Fox et al. (2003) V. Fox, J. Hightower, Lin Liao, D. Schulz, and G. Borriello.  Bayesian filtering for location estimation.  _IEEE Pervasive Computing_ , 2(3):24–33, 2003\.  doi: 10.1109/MPRV.2003.1228524.
  * Friston et al. (2016) Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, John O Doherty, and Giovanni Pezzulo.  Active inference and learning.  _Neuroscience & Biobehavioral Reviews_, 68:862 – 879, 2016.  ISSN 0149-7634.  doi: https://doi.org/10.1016/j.neubiorev.2016.06.022.
  * Friston et al. (2017) Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, and Giovanni Pezzulo.  Active Inference: A Process Theory.  _Neural Computation_ , 29(1):1–49, 01 2017.  ISSN 0899-7667.  doi: 10.1162/NECO_a_00912.  URL https://doi.org/10.1162/NECO_a_00912.
  * Itti and Baldi (2009) Laurent Itti and Pierre Baldi.  Bayesian surprise attracts human attention.  _Vision Research_ , 49(10):1295 – 1306, 2009\.  ISSN 0042-6989.  doi: https://doi.org/10.1016/j.visres.2008.09.007.  URL http://www.sciencedirect.com/science/article/pii/S0042698908004380.  Visual Attention: Psychophysics, electrophysiology and neuroimaging.
  * Millidge (2019) Beren Millidge.  Combining active inference and hierarchical predictive coding: A tutorial introduction and case study.  _PsyArXiv_ , 2019.  doi: 10.31234/osf.io/kf6wc.  URL https://doi.org/10.31234/osf.io/kf6wc.
  * Pezzato et al. (2020) Corrado Pezzato, Carlos Hernandez, and Martijn Wisse.  Active inference and behavior trees for reactive action planning and execution in robotics.  _arXiv_ , 2020.
  * Rafetseder et al. (2013) Eva Rafetseder, Maria Schwitalla, and Josef Perner.  Counterfactual reasoning: From childhood to adulthood.  _Journal of experimental child psychology_ , 114(3):389–404, 2013.
  * Sancaktar et al. (2020) Cansu Sancaktar, Marcel van Gerven, and Pablo Lanillos.  End-to-end pixel-based deep active inference for body perception and action.  _arXiv_ , 2020.
  * Schwartenbeck et al. (2018) Philipp Schwartenbeck, Johannes Passecker, Tobias U Hauser, Thomas H B FitzGerald, Martin Kronbichler, and Karl Friston.  Computational mechanisms of curiosity and goal-directed exploration.  _bioRxiv_ , 2018.  doi: 10.1101/411272.  URL https://www.biorxiv.org/content/early/2018/09/07/411272.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.  Mastering the game of go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, 2016.  doi: 10.1038/nature16961.  URL https://doi.org/10.1038/nature16961.
  * Winn and Bishop (2005) John Winn and Christopher Bishop.  Variational message passing.  _Journal of Machine Learning Research_ , 6:661–694, 2005\.
  * Çatal et al. (2020) Ozan Çatal, Tim Verbelen, Johannes Nauta, Cedric De Boom, and Bart Dhoedt.  Learning perception and planning with deep active inference.  _arXiv_ , 2020.