# Photo-induced nonlinear band shift and valence transition in SmS

Yitong Chen  Department of Physics, Graduate School of Science, Osaka
University, Toyonaka, Osaka 560-0043, Japan  Takuto Nakamura
<EMAIL_ADDRESS>Graduate School of Frontier Biosciences,
Osaka University, Suita, Osaka 565-0871, Japan  Department of Physics,
Graduate School of Science, Osaka University, Toyonaka, Osaka 560-0043, Japan
Hiroshi Watanabe<EMAIL_ADDRESS>Graduate School of Frontier
Biosciences, Osaka University, Suita, Osaka 565-0871, Japan  Department of
Physics, Graduate School of Science, Osaka University, Toyonaka, Osaka
560-0043, Japan  Takeshi Suzuki  Institute for Solid State Physics, The
University of Tokyo, Kashiwa, Chiba 277-8581, Japan  Qianhui Ren  Institute
for Solid State Physics, The University of Tokyo, Kashiwa, Chiba 277-8581,
Japan  Kecheng Liu  Institute for Solid State Physics, The University of
Tokyo, Kashiwa, Chiba 277-8581, Japan  Yigui Zhong  Institute for Solid State
Physics, The University of Tokyo, Kashiwa, Chiba 277-8581, Japan  Teruto Kanai
Institute for Solid State Physics, The University of Tokyo, Kashiwa, Chiba
277-8581, Japan  Jiro Itatani  Institute for Solid State Physics, The
University of Tokyo, Kashiwa, Chiba 277-8581, Japan  Shik Shin  Office of
University Professor, The University of Tokyo, Kashiwa, Chiba 277-8581, Japan
Kozo Okazaki  Institute for Solid State Physics, The University of Tokyo,
Kashiwa, Chiba 277-8581, Japan  Keiichiro Imura  Department of Physics, Nagoya
University, Nagoya, Aichi 464-8602, Japan  Hiroyuki S. Suzuki  Institute for
Solid State Physics, The University of Tokyo, Kashiwa, Chiba 277-8581, Japan
Noriaki K. Sato  Center for General Education, Aichi Institute of Technology,
Toyota, Aichi 470-0392, Japan  Shin-ichi Kimura  kimura.shin-
<EMAIL_ADDRESS>Graduate School of Frontier Biosciences, Osaka
University, Suita, Osaka 565-0871, Japan  Department of Physics, Graduate
School of Science, Osaka University, Toyonaka, Osaka 560-0043, Japan
Institute for Molecular Science, Okazaki, Aichi 444-8585, Japan

###### Abstract

The photo-induced band structure variation of a rare-earth-based
semiconductor, samarium monosulfide (SmS), was investigated using high-
harmonic-generation laser-based time-resolved photoelectron spectroscopy. A
nonlinear photo-induced band shift of the Sm 4f multiplets was observed. The
first one is a shift to the high-binding-energy side due to a large surface
photovoltage (SPV) effect of approximately 93 meV, comparable to the size of
the bulk band gap, with a much longer relaxation time than 0.1 ms. The second
one is an ultrafast band shift to the low binding energy side, which is in the
opposite direction to the SPV shift, suggesting an ultrafast valence
transition from divalent to trivalent Sm ions due to photo-excitation. The
latter energy shift was approximately 58 meV, which is consistent with the
energy gap shift from ambient pressure to the boundary between the black
insulator and golden metallic phase with the application of pressure. This
suggests that the photo-induced valence transition can reach the phase
boundary, but other effects are necessary to realize the golden metallic
phase.

Materials in the vicinity of critical points, such as strongly correlated
electron systems, exhibit characteristic pressure- and magnetic-field-induced
phase transitions, such as metal-insulator transitions and magnetically
ordered nonmagnetic transitions.

In addition to traditional external perturbations, photo-excitation has
recently been developed to create nonequilibrium states where charge-melted
metal-insulator transitions [1, 2] and photo-excited superconducting states
[3, 4] appear. The photo-induced creation and manipulation of physical
properties have recently been studied for future applications in ultrafast
electronics[5]. Among the candidate materials, some rare-earth compounds
exhibit quantum phase transitions between magnetically ordered states and
nonmagnetic heavy fermion (HF) states and valence transitions when applying
pressure and magnetic fields [6, 7, 8]. Such materials are also promising
hosts for realizing the photo-excited breakdown of HF states [9, 10] and the
change in mean valences [11, 12].

Samarium monosulfide (SmS) is also located near the valence phase transitions.
At ambient pressure, SmS is a black-colored semiconductor with an indirect
energy gap size of 0.1 eV [13], and Sm ions are almost divalent (4$f^{6}$)
[14]. Above the critical pressure of approximately 0.65 GPa, this material
changes to the golden semimetallic phase together with a valence transition to
the trivalent state (4$f^{5}$) while maintaining the same crystal structure
[15, 16, 17, 18, 19], namely the black-to-golden phase transition (BGT).
Although the phase transition was discovered more than half a century ago, its
origin remains under debate [20]. Recent theoretical and experimental studies
on the phase transition of SmS have proposed an important role for the
generation of electron–hole pairs (excitons) [21, 22]. By the optical
excitation of Sm2+ 4$f$ electrons, an electronic configuration similar to that
in the golden semimetallic phase (4$f^{6}$ $\rightarrow$ 4$f^{5}$) with an
electron–hole pair can be artificially created, expecting a similar phase
transition with the application of pressure. Therefore, SmS is one of the
candidates to investigate the photo-induced nonequilibrium electronic
structure, especially whether BGT can be exhibited by photo-excitation.

Time-resolved photoelectron spectroscopy (TrPES) and time- and angle-resolved
photoelectron spectroscopy (TrARPES) are powerful experimental tools for
investigating the transient electronic structure of the photo-induced state.
The observation of transient band structures by TrARPES has been successful in
the discovery of novel inequivalent phenomena, such as Floquet-Bloch states in
topological insulators [23] and an insulator-to-semimetal transition in
excitonic insulators [24]. However, despite extensive studies on topological
materials and low-dimensional systems, observation of the transient band
structure in rare-earth compounds has not been carried out until now, except
for a few pioneering studies [25]. One of the reasons for the difficulty in
observation is that photoelectrons excited by 6-eV photons in ordinary TrARPES
have poor 4$f$ photoionization cross sections [26].

Figure 1:  (a) Schematic illustration of high-harmonic-generation-laser-based
time-resolved photoelectron spectroscopy (TrPES), as applied to SmS. (b) and
(c) band mappings of SmS with angle-resolved photoelectron spectroscopy
(ARPES) without the photo-irradiation of a pump laser. (b) Wide-range ARPES
image and (c) 4f multiplets in the yellow dashed square area of (b). (d)
Angle-integrated photoelectron spectrum of (c). The multiplet structures of
the Sm2+ 4f5 final state are visible. The peak positions of Sm 4f5 multiplets
are indicated by the dashed lines.

In this study, we investigate the photo-induced change in the electronic
structure of SmS using TrPES combined with extreme ultraviolet photons
($\sim$22 eV) emitted from a high-harmonic-generation laser. The energy shift
of the Sm 4$f$ multiplet structure and modulation of the energy gap in the
photo-induced state were successfully visualized. Two types of characteristic
photo-induced energy shifts of the Sm 4f multiplet structure were observed.
The first is a large surface photovoltage (SPV) effect of approximately 93 meV
on the high-binding-energy side, comparable to the size of the bulk indirect
band gap of SmS. The second is an ultrafast band shift of approximately 58
$\pm$ 4 meV to the low binding energy side, indicating photo-induced band
deformation due to photo-excitation from the 4f to 5d state. Photo-excitation
induces the energy gap narrowing, but the photo-excited phase still has an
energy gap, suggesting that the material is not a metal. This suggests that
the generation of electron–hole pairs by photo-excitation alone cannot cause
BGT, and other effects, such as lattice modulation, are necessary.

Figure 2:  (a) Angle-integrated photoelectron spectra of the Sm 4f multiplet
structure without irradiation of the pump laser (black dashed line) and before
the arrival of the pump pulse ($t<0$) under irradiation of the pump laser with
a fluence. Gray dashed lines indicate the contribution of different parts of
the fitting function applied to the photoelectron spectra, and vertical dot-
dashed lines indicate the energy positions of the central 4f multiplet
structure acquired by fitting. (b) Pump fluence dependency of the energy shift
of the 4f multiplet structure happens at t $<$ 0\. Fitting results obtained
using a horizontal line are shown using a red dashed line.  Figure 3:  Angle-
integrated photoelectron spectra of the multiplet structure of SmS 4f final
state before the arrival of the pump pulse ($t<0$) (black dashed line) and
just after the arrival of the pump pulse (less than 0.1 ps) under irradiation
of the pump laser with a fluence. Similar to the indications in Fig. 2(a),
gray dashed lines represent three Sm $4f$ multiplet peaks, and short vertical
lines indicate the approximate positions of the central peaks. (b) Time
structure of the energy shift of the three 4f multiplet peaks at various pump
fluences. Marks show the energy shift evaluated by the fitting of
photoelectron spectra, while the solid lines show the results of a single-
exponential decay function convoluted with a Gaussian function fitting. (c)
Pump fluence dependence of the maximum energy shift of the 4f multiplet
structure at $t>0$. The red solid line shows the fitting result with a single-
exponential decay function, while the horizontal dot-dashed line shows the
saturation value of the energy shift. (d) Relaxation time of the fitting
function in (b) as a function of pump fluence. (e) Energy shift time after
photo-irradiation as a function of pump fluence.

High-quality single-crystalline SmS was synthesized using the vertical
Bridgman method in a high-frequency induction furnace. A clean SmS surface was
obtained by cleaving in situ in the measurement chamber under a background
pressure of approximately 10-9 Pa. Figure 1(a) illustrates the experimental
system of TrARPES. The pump pulse is 1.55 eV light generated directly from a
commercial Ti:Sapphire regenerative amplifier system (Spectra-Physics,
Solstice Ace), whereas the probe pulse is 21.7 eV light produced by the
seventh harmonic generation of the second harmonic, which is also used from
the identical laser of the pump pulse [27]. The repetition rate and pulse
width of the pump pulse were 10 kHz and 35 fs, respectively, whereas the
temporal and energy resolutions were estimated to be about 70 fs and 250 meV,
respectively. Photoelectrons are detected by using a hemispherical electron
analyzer (Scienta, R4000). All experiments were conducted at room temperature.
Here, we mainly focused on the energy shift of weakly dispersive Sm 4f
multiplet peaks; therefore, we integrated the TrARPES image along the momentum
direction to obtain angle-integrated photoelectron (AIPE) spectra with a high
S/N ratio.

Figure 1(b) shows the wide-range ARPES intensity plot of SmS without photo-
irradiation by the pump laser. Dispersive bands appeared at $\sim$4 eV, and
flat bands appeared at $\sim$1.7 eV, originating from the S 3p bands and Sm 4f
multiplets, respectively. Although the Sm 5$d$ bands are located above the
Fermi level at the X point in the bulk Brillouin zone, no such conduction
bands are observed, even immediately after photo-excitation. The possible
reasons for the absence of such conduction bands might be the ultrafast
relaxation of the excited carrier of less than 70 fs, the time resolution,
and/or the difference in the out-of-plane wave number $k_{z}$. By setting the
inner potential to 14.1 eV [28], the value of $k_{z}$ was evaluated to be 2.9
$\text{\AA}^{-1}$ for a 21.7 eV probe photon, which is located near the center
between the $\Gamma$ point and X point of the bulk Brillouin zone of SmS,
while the bottom of the Sm 5$d$ band is located at the X point. The $k_{z}$
broadening was estimated to be 0.1 Å ($\sim$10 $\%$ of the distance from the
$\Gamma$ point to K point); thus, an average wavenumber over a wide $k_{z}$
area was observed, which could result in the indistinctness of the strongly
dispersive Sm 5$d$ band.

Figures 1(c) and 1(d) show a magnified ARPES image and the corresponding AIPE
spectrum. The three peaks at approximately 1, 1.7, and 2.5 eV indicated by
dashed lines in Fig. 1(d) are the Sm 4f multiplets of bulk-${}^{6}H$
(${}^{6}H$(b)), the overlap of bulk-${}^{6}F$ (${}^{6}F$(b)) and
surface-${}^{6}H$ (${}^{6}H$(s)), and surface-${}^{6}F$ (${}^{6}F$(s)),
respectively. The shape and position of these peaks are consistent with the
previous ARPES data probing by He-I (21.2 eV) photons [29], indicating that
the valence band structure of SmS can be observed by extreme ultraviolet pulse
laser excitation.

Figure 4:  Schematic illustration of the band structure near the surface of
the SmS sample with photo-irradiation. (a) Without pumping. (b) Under photo-
irradiation by pump pulses and before the arrival of the pump pulse. (c)
Within several tens of femtoseconds after the arrival of a pump pulse. (d)
Several picoseconds after the arrival of a pump pulse.

Figure 2(a) shows the AIPE spectra of the multiplet structure of Sm2+ 4f5
final state before the arrival of the pump pulse ($t<0$). Even at $t<0$, the
valence band spectrum is shifted by photo-irradiation. To determine the exact
energy positions of the multiplet peaks, the AIPE spectra were fitted using
the following function:

$I(x)=\sum^{3}_{i=1}I_{i}\text{exp}(-\frac{(x-x_{i})^{2}}{w_{i}^{2}})+I_{b}\text{exp}(-\frac{(x-x_{b})^{2}}{w_{b}^{2}})+BG$

The fitting function consists of four Gaussian functions and a constant
background. The first three Gaussian functions with subscript $i$ are for the
multiplet structure of the Sm2+ 4f5 final state, as shown in Fig. 1(d), and
the other with subscript $b$ is for a structure located at a higher binding
energy. I, x, and w are the fitting parameters of intensity, central binding
energy, and full width at half maximum of the corresponding Gaussian peak,
respectively.

To obtain the overall chemical shift due to photo-irradiation, the widths of
the Gaussian peaks representing the 4f multiplet structure ($w_{i}$), as well
as the energy differences among the three multiplet peaks, were kept unchanged
during the fitting process, while other parameters were allowed to vary. The
fitting result of the spectrum without pump laser irradiation is shown by gray
dashed lines in Fig. 2(a), and the energy positions of the central Gaussian
peak are marked by vertical dot-dashed lines. After fitting to all other
spectra, the size of the peak shift can be evaluated as a constant value of
$95\pm 5$ meV for all photo-irradiation fluences, as shown in Fig. 2(b). The
peak shift energy is saturated even at the lowest fluence of 0.29 J/cm2. This
energy shift could be attributed to the long-living surface photovoltage (SPV)
effect. Further details on this energy shift will be provided later.

Figure 3(a) shows the AIPE spectra of the multiplet structure of the Sm2+ 4f5
final state, obtained immediately before the arrival of the pump pulse (black
dashed line spectrum is for $t<0$), and the spectra obtained immediately after
the arrival (colored solid lines labeled as different fluences) of the pump
pulse. The spectra were fitted to the same function as in Fig. 2(a), and the
energy position of the central Gaussian peak of the 4f multiplet structure is
marked by short vertical lines in each spectrum. After photo-irradiation, the
spectrum shifts toward low binding energies but in the opposite direction to
the shift at $t<0$, that is, a nonlinear shift in time. To investigate the
time development of the energy shift after photoirradiation, the same fitting
procedure as shown in Fig. 2(a) was performed on the obtained data, and the
peak shift was plotted as a function of delay time, as shown in Fig. 3(b). The
AIPE spectrum shifted towards the Fermi level within 0.1 ps and then gradually
shifted back to the position before pumping in the following several
picoseconds. To obtain a quantitative understanding, we fitted the time-
dependent energy shift to a single-exponential decay function convoluted with
a Gaussian function, as shown in Fig. 3(b). The fitting parameters, that is,
the maximum energy shift ($E_{smax}$), recovery time $\tau$, and rise time
$\tau_{r}$ at different pump fluences, are plotted as functions of the pump
fluence in Figs. 3(c)–(e).

Figure 3(c) shows the pump fluence dependency of $E_{smax}$. $E_{smax}$
initially increases as the fluence increases up to a pump fluence of
approximately 1.2 mJ/cm2 and then saturates at a certain value. The saturation
value of $E_{smax}$ was evaluated as $58\pm 4$ meV by assuming a single-
exponential function. The recovery time $\tau$ and rising time $\tau_{r}$ are
shown in Figs. 3(d) and 3(e), respectively. Both parameters remained constant
at each fluence; $\tau$ was as long as 1 ps, and $\tau_{r}$ was several tens
of femtoseconds, which is comparable to the time resolution of the instrument.
This ultrafast band shift and recovery are believed to correspond to the
valence transition of Sm ions, as discussed later.

We now discuss the band shift induced by photo-excitation. SmS is expected to
be an $n$-type semiconductor; thus, a potential point from the bulk to the
surface is created by the electrons trapped by the surface state, and the
bands near the surface bend upwards without photo-irradiation, as shown in
Fig. 4(a). Under photo-irradiation, excited electrons and holes move to the
bulk and surface sides, respectively, driven by the electric field near the
surface, which is known as the SPV effect. Consequently, the surface electric
field is canceled when the irradiation power is sufficient. The observed
energy shift to the high-binding-energy side at $t<0$ can be explained by the
cancelation of surface band bending by the SPV effect. It should be noted that
the energy shift before a pump pulse arrives suggests an extremely long
lifetime of the SPV effect of at least 0.1 ms, according to the repetition
rate of 10 kHz. A similar long-living SPV shift has been reported for SmB6
[25], but to the best of our knowledge, it does not appear in conventional
semiconductors [30]. The lifetime of the SPV effect may originate from the
nondispersive Sm 4f band structure, which is a common feature of SmS and SmB6.

By the pump pulse, localized 4f electrons of Sm ions were excited to the
widely spread 5d conduction band. Then, the ionic valence of Sm changed from a
divalent to nearly trivalent state, and the band structure exhibited a trend
of changing from a divalent to trivalent state, as in Ref. [31], that is, the
5d conduction band is expected to shift downward, while the 4f valence band
shifts upward, thus reducing the band gap, as shown in Fig. 4(c); finally, the
band shift returned to the same value as that at the SPV effect (Fig. 4(d)).
Because the peak shift energy due to the SPV effect at $t<0$ ($93\pm 5$ meV)
is comparable to the bandgap size of SmS ($93\pm 4$ meV) [22], the bottom of
the 5d conduction band is considered to be located just above the Fermi level
at the X point. Furthermore, no evident change in the sample color was
observed during the time-resolved measurement and the 5d occupied state was
not observed in the ARPES mapping after photo-irradiation. Therefore, the
bottom of the 5d band was pinned at the Fermi level even after photo-
irradiation, meaning that SmS continued to be a semiconductor. However, the
photo-induced shift of the 4f valence band toward the Fermi level shown in
Fig. 3 suggests the narrowing of the band gap. Considering the indirect band
gap size of bulk SmS and the saturation value of $E_{smax}$ of $58\pm 4$ meV,
the saturation value of the energy gap size ($\sim$35 meV) under photo-
irradiation is similar to that ($\sim$45 meV) observed just before the BGT
critical pressure of 0.65 GPa is attained [22]. In BGT, the energy gap
suddenly collapsed across the critical pressure. However, the gap size of the
photo-excitation state saturated to the value at the critical pressure,
suggesting that the band shift only occurred in the black phase and did not
cross the phase boundary of the BGT. One of the reasons for this is that the
photo-irradiation time ($\sim$ps) is too short for the lattice to shrink from
the black phase to the golden phase ($>$ ns).

In summary, we performed TrPES on SmS and experimentally specified two
individual energy shifts of the multiplet structure (4f6 $\rightarrow$ 4f5) of
Sm2+ induced by photo-excitation. At $t<0$, we observed an energy shift to the
high-binding-energy side, which was attributed to the SPV effect. At $t>0$,
the 4f multiplet structure shifted to the low binding energy side within
several tens of femtoseconds and recovered completely in a few picoseconds.
This suggests that creating excitons by photo-excitation exhibits the valence
transition of SmS in a short time, but the phase does not change to the golden
phase, which appears at a high pressure. This further suggests that the
creation of excitons is the origin of the narrowing of the energy gap, which
supports the theory that excitons play an important role in BGT. However, the
short period of appearance of excitons did not cause BGT. Other effects,
including lattice shrinking, are considered to be necessary for BGT.

We would like to thank Professor Takahiro Ito for helpful discussions. The
TrARPES experiment in this work was conducted by joint research at the ISSP,
University of Tokyo. This study was supported by JSPS KAKENHI (Grant Nos.
20H04453 and 22K14605) and the Iketani Science and Technology Foundation.

## References

  * Miyano  _et al._ [1997] K. Miyano, T. Tanaka, Y. Tomioka, and Y. Tokura, Phys. Rev. Lett. 78, 4257 (1997).
  * Morrison  _et al._ [2014] V. R. Morrison, R. P. Chatelain, K. L. Tiwari, A. Hendaoui, A. Bruhács, M. Chaker, and B. J. Siwick, Science 346, 445 (2014).
  * Okazaki  _et al._ [2018a] K. Okazaki, H. Suzuki, T. Suzuki, T. Yamamoto, T. Someya, Y. Ogawa, M. Okada, M. Fujisawa, T. Kanai, N. Ishii, J. Itatani, M. Nakajima, H. Eisaki, A. Fujimori, and S. Shin, Phys. Rev. B 97, 121107(R) (2018a).
  * Suzuki  _et al._ [2019] T. Suzuki, T. Someya, T. Hashimoto, S. Michimae, M. Watanabe, M. Fujisawa, T. Kanai, N. Ishii, J. Itatani, S. Kasahara, Y. Matsuda, T. Shibauchi, K. Okazaki, and S. Shin, Communications Physics 2, 115 (2019).
  * de la Torre  _et al._ [2021] A. de la Torre, D. M. Kennes, M. Claassen, S. Gerber, J. W. McIver, and M. A. Sentef, Rev. Mod. Phys. 93, 041002 (2021).
  * Shishido  _et al._ [2005] H. Shishido, R. Settai, H. Harima, and Y. Ōnuki, Journal of the Physical Society of Japan 74, 1103 (2005).
  * Gegenwart  _et al._ [2002] P. Gegenwart, J. Custers, C. Geibel, K. Neumaier, T. Tayama, K. Tenya, O. Trovarelli, and F. Steglich, Phys. Rev. Lett. 89, 056402 (2002).
  * Wada  _et al._ [1997] H. Wada, A. Nakamura, A. Mitsuda, M. Shiga, T. Tanaka, H. Mitamura, and T. Goto, Journal of Physics: Condensed Matter 9, 7913 (1997).
  * Wetli  _et al._ [2018] C. Wetli, S. Pal, J. Kroha, K. Kliemt, C. Krellner, O. Stockert, H. von Loehneysen, and M. Fiebig, Nature Physics 14, 1103 (2018).
  * Zhu  _et al._ [2021] W. Zhu, B. Fauseweh, A. Chacon, and J.-X. Zhu, Phys. Rev. B 103, 224305 (2021).
  * Yokoyama  _et al._ [2019] Y. Yokoyama, K. Kawakami, Y. Hirata, K. Takubo, K. Yamamoto, K. Abe, A. Mitsuda, H. Wada, T. Uozumi, S. Yamamoto, I. Matsuda, S. Kimura, K. Mimura, and H. Wadati, Phys. Rev. B 100, 115123 (2019).
  * Mardegan  _et al._ [2021] J. R. L. Mardegan, S. Zerdane, G. Mancini, V. Esposito, J. R. Rouxel, R. Mankowsky, C. Svetina, N. Gurung, S. Parchenko, M. Porer, B. Burganov, Y. Deng, P. Beaud, G. Ingold, B. Pedrini, C. Arrell, C. Erny, A. Dax, H. Lemke, M. Decker, N. Ortiz, C. Milne, G. Smolentsev, L. Maurel, S. L. Johnson, A. Mitsuda, H. Wada, Y. Yokoyama, H. Wadati, and U. Staub, Phys. Rev. Research 3, 033211 (2021).
  * Kimura  _et al._ [2008] S. Kimura, T. Mizuno, K. Matsubayashi, K. Imura, H. S. Suzuki, and N. K. Sato, Physica B: Condensed Matter 403, 805 (2008).
  * Matsubayashi  _et al._ [2007] K. Matsubayashi, K. Imura, H. S. Suzuki, T. Mizuno, S. Kimura, T. Nishioka, K. Kodama, and N. K. Sato, Journal of the Physical Society of Japan 76, 064601 (2007).
  * Jayaraman  _et al._ [1970] A. Jayaraman, V. Narayanamurti, E. Bucher, and R. G. Maines, Phys. Rev. Lett. 25, 1430 (1970).
  * Bader  _et al._ [1973] S. D. Bader, N. E. Phillips, and D. B. McWhan, Phys. Rev. B 7, 4686 (1973).
  * Keller  _et al._ [1979] R. Keller, G. Güntherodt, W. Holzapfel, M. Dietrich, and F. Holtzberg, Solid State Communications 29, 753 (1979).
  * Raymond  _et al._ [2002] S. Raymond, J. P. Rueff, M. D’Astuto, D. Braithwaite, M. Krisch, and J. Flouquet, Phys. Rev. B 66, 220301(R) (2002).
  * Annese  _et al._ [2006] E. Annese, A. Barla, C. Dallera, G. Lapertot, J.-P. Sanchez, and G. Vankó, Phys. Rev. B 73, 140409(R) (2006).
  * Varma [1976] C. M. Varma, Rev. Mod. Phys. 48, 219 (1976).
  * Watanabe [2021] S. Watanabe, Journal of the Physical Society of Japan 90, 023706 (2021).
  * Mizuno  _et al._ [2008] T. Mizuno, T. Iizuka, S. Kimura, K. Matsubayashi, K. Imura, H. S. Suzuki, and N. K. Sato, Journal of the Physical Society of Japan 77, 113704 (2008).
  * Wang  _et al._ [2013] Y. H. Wang, H. Steinberg, P. Jarillo-Herrero, and N. Gedik, Science 342, 453 (2013).
  * Okazaki  _et al._ [2018b] K. Okazaki, Y. Ogawa, T. Suzuki, T. Yamamoto, T. Someya, S. Michimae, M. Watanabe, Y. Lu, M. Nohara, H. Takagi, N. Katayama, H. Sawa, M. Fujisawa, T. Kanai, N. Ishii, J. Itatani, T. Mizokawa, and S. Shin, Nature Communications 9, 4322 (2018b).
  * Ishida  _et al._ [2015] Y. Ishida, T. Otsu, T. Shimada, M. Okawa, Y. Kobayashi, F. Iga, T. Takabatake, and S. Shin, Scientific Reports 5, 8160 (2015).
  * Yeh and Lindau [1985] J. Yeh and I. Lindau, Atomic Data and Nuclear Data Tables 32, 1 (1985).
  * Suzuki  _et al._ [2021] T. Suzuki, S. Shin, and K. Okazaki, Journal of Electron Spectroscopy and Related Phenomena 251, 147105 (2021).
  * Kaneko  _et al._ [2014] M. Kaneko, M. Saito, T. Ito, K. Imura, T. Hajiri, M. Matsunami, S. Kimura, H. S. Suzuki, and N. K. Sato, Proceedings of the International Conference on Strongly Correlated Electron Systems 3, 011080 (2014).
  * Ito  _et al._ [2002] T. Ito, A. Chainani, H. Kumigashira, T. Takahashi, and N. K. Sato, Phys. Rev. B 65, 155202 (2002).
  * Azuma  _et al._ [2009] J. Azuma, S. Tokudomi, K. Takahashi, and M. Kamada, Physica Status Solidi C 6, 307 (2009).
  * Antonov  _et al._ [2002] V. N. Antonov, B. N. Harmon, and A. N. Yaresko, Phys. Rev. B 66, 165208 (2002).

Supplementary Information:


# Designing explainable artificial intelligence with active inference: A
framework for transparent introspection and decision-making

Mahault Albarracin  VERSES Research Lab, Los Angeles, CA 90016, USA
Département d’informatique, Université du Québec à Montréal, 201, Avenue du
Président-Kennedy, Montréal, H2X 3Y7  Inês Hipólito  VERSES Research Lab, Los
Angeles, CA 90016, USA  Berlin School of Mind & Brain, Humboldt-Universität zu
Berlin, Berlin, Germany  Department of Philosophy, Macquarie University,
Sydney, New South Wales, Australia  Safae Essafi Tremblay  VERSES Research
Lab, Los Angeles, CA 90016, USA  Département de philosophie, Université du
Québec à Montréal, 455, Boulevard René-Lévesque Est, Montréal, H2L 4Y2  Jason
G. Fox  VERSES Research Lab, Los Angeles, CA 90016, USA  Gabriel René  VERSES
Research Lab, Los Angeles, CA 90016, USA  Karl Friston  VERSES Research Lab,
Los Angeles, CA 90016, USA  Wellcome Centre for Human Neuroimaging, University
College London,
London WC1N 3AR, UK  Maxwell J. D. Ramstead  VERSES Research Lab, Los Angeles,
CA 90016, USA  Wellcome Centre for Human Neuroimaging, University College
London,
London WC1N 3AR, UK

###### Abstract

This paper investigates the prospect of developing human-interpretable,
explainable artificial intelligence (AI) systems based on active inference and
the free energy principle. We first provide a brief overview of active
inference, and in particular, of how it applies to the modeling of decision-
making, introspection, as well as the generation of overt and covert actions.
We then discuss how active inference can be leveraged to design explainable AI
systems, namely, by allowing us to model core features of “introspective”
processes and by generating useful, human-interpretable models of the
processes involved in decision-making. We propose an architecture for
explainable AI systems using active inference. This architecture foregrounds
the role of an explicit hierarchical generative model, the operation of which
enables the AI system to track and explain the factors that contribute to its
own decisions, and whose structure is designed to be interpretable and
auditable by human users. We outline how this architecture can integrate
diverse sources of information to make informed decisions in an auditable
manner, mimicking or reproducing aspects of human-like consciousness and
introspection. Finally, we discuss the implications of our findings for future
research in AI, and the potential ethical considerations of developing AI
systems with (the appearance of) introspective capabilities.

###### Contents

  1. 1 Introduction: Explainable AI and active inference
  2. 2 Active inference and introspection
    1. 2.1 A brief introduction to active inference
    2. 2.2 Active inference, introspection, and self-modeling
  3. 3 Using active inference to design self-explaining AI
  4. 4 Discussion
    1. 4.1 Directions for future research
    2. 4.2 Ethical considerations of introspective AI systems
  5. 5 Conclusion

### Acknowledgements

The authors are grateful to VERSES for supporting the open access publication
of this paper. SET is supported in part by funding from the Social Sciences
and Humanities Research Council of Canada (Ref: 767-2020-2276). KF is
supported by funding for the Wellcome Centre for Human Neuroimaging (Ref:
205103/Z/16/Z) and a Canada-UK Artificial Intelligence Initiative (Ref:
ES/T01279X/1). The authors are grateful to Brennan Klein for assistance with
typesetting.

### Conflict of interest statement

The authors disclose that they are contributors to the Institute of Electrical
and Electronics Engineers (IEEE) P2874 Spatial Web Working Group.

##  1 Introduction: Explainable AI and active inference

Artificial intelligence (AI) systems continue to proliferate and, at the time
of writing, have become an integral part of various intellectual and
industrial domains, including healthcare, finance, and transportation [1, 2].
Traditional AI models, such as deep learning neural networks, have been widely
recognized for their ability to achieve high performance and accuracy across
various tasks [3, 4]. However, it is well known that these models almost
invariably function as “black boxes,” with limited transparency and
interpretability of their decision-making processes [5, 6]. This lack of
explainability can lead to skepticism and reluctance to adopt AI systems—and
indeed, to harm, particularly in high-stakes situations, where the
consequences of a wrong decision can be severe and harmful [7, 8, 9, 10].
Indeed, a lack of explainability precludes applications in certain domains,
such as fintech.

The problem of explainable AI (sometimes referred to as the “black box”
problem) is the problem of understanding and interpreting how these models
arrive at their decisions or predictions [11, 12]. While researchers and users
may have knowledge of the inputs provided to the model and the corresponding
outputs that it produces, comprehending the internal workings and decision-
making processes of AI systems can be complex and challenging. This is in no
small part because their intricate architectures and numerous interconnected
layers learn to make predictions by analyzing vast amounts of training data
and adjusting their internal parameters, without explicit instruction from a
programmer [13]. The method by which these systems are trained thus, by
design, limits their explainability. Moreover, the internal computations that
are performed by these models—when they engage in decision-making—can be
highly complex and nonlinear, making it difficult to extract meaningful
explanations of their behavior, or insights into their decision-making process
[14]. This problem is compounded by the fact that most machine learning
implementations of AI fail to represent or quantify their uncertainty;
especially, uncertainty about the parameters and weights that underwrite their
accurate performance. This means that AI, in general, cannot evaluate (or
report) the confidence in its decisions, choices or recommendations.

The lack of interpretability poses several challenges. Firstly, it hampers
transparency and makes audits by third parties next to impossible, as the
designers, users, and stakeholders of these systems may struggle to understand
why a particular decision or prediction was made. This becomes problematic in
critical domains such as healthcare or finance, where the ability to explain
the reasoning behind a decision is essential for trust, accountability, and
compliance with regulations [15, 16]. Secondly, the black box nature of
machine learning models can hinder the identification and mitigation of biases
or discriminatory patterns. Without visibility into the underlying decision-
making process, it becomes challenging to detect and address biases that may
exist within the model’s training data or architecture.

This opacity can lead to unfair or biased outcomes, perpetuating social
inequalities or discriminatory practices [17, 18, 19]. Additionally, the lack
of interpretability of the model limits its ability to provide meaningful
explanations to end-users. Individuals interacting with machine learning
systems often seek explanations for the decisions made by these systems [20,
21]. For instance, in medical diagnosis, patients and healthcare professionals
may want to understand why a particular diagnosis or treatment recommendation
was given [22, 23]; or consider automated suggestions in practical industrial
settings [24]. Without explainability, users may be hesitant to trust the
system’s recommendations or may feel apprehensive (not without good reason)
about relying on the outputs of such models.

Accordingly, the need for explainable AI has become increasingly important
[25]. “Explainable AI” refers to the development of AI systems that can
provide human-understandable explanations for their decisions and actions
[26]. This level of transparency is crucial for fostering trust [27], ensuring
accountability [28], and facilitating inclusive collaboration between humans
and AI systems [29, 30, 31]. Recent efforts to regulate AI may turn
explainability into a requirement for the deployment of any AI system at
scale. For instance, in the United States, the National Institute of Standards
and Technology (NIST) released its Artificial Intelligence Risk Management
Framework (RMF) in 2023, which includes explainability and interpretability as
crucial characteristics of a trustworthy AI system. The RMF is envisioned as a
guide for tech companies to manage the risks of AI and could eventually be
adopted as an industry standard. In a similar vein, US Senator Chuck Schumer
has led a congressional effort to establish US regulations on AI, with one of
the key aspects being the availability of explanations for how AI arrives at
its responses [32].

In the European Union, a proposed Regulation Laying Down Harmonized Rules on
Artificial Intelligence (better known as the “AI Act”) is set to increase the
transparency required for the use of so-called “high-risk” AI systems. For
instance, groups that deploy automated emotion recognition systems may be
obligated to inform those on whom the system is being deployed that they are
being exposed to such a system. The AI Act is expected to be finalized and
adopted in 2023, with its obligations likely to apply within three years’
time. The Council of Europe is also in the process of developing a draft
convention on artificial intelligence, human rights, democracy, and the rule
of law, which will be the first legally binding international instrument on
AI. This convention seeks to ensure that research, development, and deployment
of AI systems are consistent with the values and interests of the EU, and that
they remain compatible with the AI Act and the proposed AI Liability
Directive, which includes a risk-based approach to AI. In addition, the US-EU
Trade and Technology Council published a joint Roadmap for Trustworthy AI and
Risk Management in 2022, which aims to advance collaborative approaches in
international standards bodies related to AI, among other objectives [33].
Therefore, explainability is clearly a major issue in research, development,
and deployment of AI systems, and will remain so for the foreseeable future.

Explainable AI aims to bridge the gap between the complexity and lack of
auditability of contemporary AI systems and the need for human
interpretability and auditability [25, 26, 34]. It seeks to provide insights
into the factors that influence AI decision-making, enabling users to
understand the explicit reasoning and other factors driving the output of AI
systems. Understanding the performance and potential biases of AI systems is
crucial for their ethical and responsible deployment [35, 36]. This
understanding, however, must extend beyond the performance of AI systems on
academic benchmarks and tasks to include a deep understanding of what the
models represent or learn, as well as the algorithms that they instantiate
[37].

Transparency considerations are embedded in the design, development, and
deployment of AI systems, from the societal problems that arise worth
developing a solution, to the data collection stage, and still at the point
where the AI system is deployed in the real world and iteratively improved
[30, 38]. This transparency may enable the implementation of other ethical AI
dimensions like interpretability, accountability, and safety [39].

Researchers have been exploring various approaches to develop more explainable
AI systems [40, 7]. However, these efforts have yet to yield a principled and
widely accepted path method for, or path to, explainability. One promising
direction is to draw inspiration from research into human introspection and
decision-making processes. Furthermore, a two-stage decision-making process,
which includes a reflection stage where the network reflects on its feed-
forward decision, can enhance the robustness and calibration of AI systems
[41]. It has been suggested that explainability in AI systems can be further
enhanced through techniques such as layer-wise relevance propagation [42] and
saliency maps [43], which aid in visualizing the model’s reasoning process. By
translating the internal models of AI systems into human-understandable
explanations, we can foster trust and collaboration between AI systems and
their human users [44]. However, as [37] argue, we must also consider the
metatheoretical calculus that underpins our understanding and use of these
models. This involves not only considering the performance of the model on a
task, but also the implications of the performance of the model for our
understanding of the mind and brain.

In this paper, we investigate the potential of active inference, and the free
energy principle (FEP) upon which is based [45, 46], to enhance explainability
in AI systems, notably by capturing core aspects of introspective processes,
hierarchical decision-making processes, and (cover and overt) forms of action
in human beings [47, 48, 49]. The FEP is a variational principle of
information physics that can be used to model the dynamics of self-organizing
systems like the brain. Active inference is an application of the FEP to model
the perception-action loops of cognitive systems: it provides us with the
basis of a unified theory of the structure and function of the brain (and
indeed, of living and self-organizing systems more generally; [50, 51]. Active
inference allows us to model self-organizing systems like brains as being
driven by the imperative to minimize surprising encounters with the
environment; where this surprise scores how far a thing or system deviates
from its characteristic states (e.g., a fish out of water). By doing so, the
brain continually updates and refines its world model, allowing the agent to
act adaptively and in situationally appropriate ways.

The relevance of using active inference is that the models of cognitive
dynamics—and in particular, introspection—that have been developed using its
tools can be adapted to enable the design of human interpretable and auditable
(and indeed, self-auditable) AI systems. The ethical and epistemological or
epistemic gains that this enables are notable. The proposed active inference
based AI system architecture would enable artificial agents to access and
analyze their own internal states and decision-making processes, leading to a
better understanding of their decision-making processes, and the ability to
report on themselves. Proof of concept for this kind of “self report” is
already at hand [52] and, in principle, is supported in any application of
active inference. At one level, committing to a generative model—implicit in
any active inference scheme—dissolves the explainability problem. This is
because one has direct access to the beliefs and belief-updating of the agent
in question.

Indeed, this is why active inference has been so useful in neuroscience to
model and explain behavioral and neuronal responses in terms of underlying
belief states: e.g., [53, 54, 55, 56, 57]. As demonstrated in [52] it is a
relatively straightforward matter to augment generative models to self-report
their belief states. In this paper, we address a slightly more subtle aspect
of explainability that rests upon “self-access”; namely, when an agent infers
its own “states of mind”—states of mind that underwrite its sense-making and
choices. Crucially, this kind of meta-inference [58, 59, 60, 61] may rest on
exactly the representations of uncertainty (a.k.a., precision) that are absent
in conventional AI.

This paper is organized as follows. We first introduce essential aspects of
active inference. We then discuss how active inference can be used to design
explainable AI systems. In particular, we propose that active inference can be
used as the basis for a novel AI architecture—based on explicit generative
models—that both endows AI systems with a greater degree of explainability and
audibility from the perspective of users and stakeholders, and allows AI
systems to track and explain their own decision-making processes in a manner
understandable to users and stakeholders. Finally, we discuss the implications
of our findings for future research in auditable, human-interpretable AI, as
well as the potential ethical considerations of developing AI systems with the
appearance of introspective capabilities.

##  2 Active inference and introspection

###  2.1 A brief introduction to active inference

Active inference offers a comprehensive framework for naturalizing,
explaining, simulating, and understanding the mechanisms that underwrite
decision-making, perception, and action [62, 63]. The free energy principle
(FEP) is a variational principle of information physics [45]. It has gained
considerable attention and traction since it was first introduced in the
context of computational neuroscience and biology [64, 65]. Active inference
denotes a family of models premised on the FEP, which are used to understand
and predict the behavior of self-organizing systems. The tools of active
inference allow us to model self-organizing systems as driven by the
imperative to minimize surprise, which quantifies the degree to which a given
path or trajectory deviates from its inertial or characteristic path—or its
upper bound, variational free energy, which scores the difference between its
predictions and the actual sensory inputs it receives [50].

Active inference modeling work suggests that decision-making, perception, and
action involve the optimization of a world model that represents the causal
structure of the system generating outcomes of observations [45]. In
particular, active inference models the way that latent states or factors in
the world cause sensory inputs, and how those factors cause each other,
thereby capturing the essential causal structure of the measured or sensed
world [66]. Minimizing surprise or free energy on average and over time allows
the brain to maintain a consistent and coherent internal model of the
world—one that maximizes predictive accuracy while minimizing model
complexity—which, in turn, enables agents to adapt and survive in their
environments [64, 67]. (Strictly speaking, this is the other way around. In
other words, agents who “survive” can always be read as minimizing variational
free energy or maximizing their marginal likelihood (a.k.a., model evidence).
This is often called self-evidencing [68].)

Active inference has instrumental value in allowing us to model, and thereby
hopefully help to understand, core aspects of human consciousness (for a
review, see [64]Friston, 2010). Of particular interest to us here, it enables
us to model the processes involved in introspective self-access (see [48, 49].
Active inference modeling deploys the construct of generative models to make
sense of the dynamics of self-organizing systems. In this context, a
generative model is a joint probability density over the hidden or latent
causes of observable outcomes; see [45] for a discussion of how to interpret
these models philosophically and [61] for a gentle introduction to the
technical implementation of these models.

We depict a simple generative model, apt for perceptual inference, in Figure
1, and a more complex generative model, apt for the selection of actions
(a.k.a. policy selection) in Figure 2. These models specify the way in which
observable outcomes are generated by (typically non-observable) states or
factors in the world.

Figure 1: A basic generative model for precision-weighted perceptual
inference. This figure depicts an elementary generative model that is capable
of performing precision-weighted perceptual inference. States are depicted as
circles and denoted in lowercase: observable states or outcomes are denoted
$o$ and latent states (which need to be inferred) are denoted $s$. Parameters
are depicted as squares and denoted as uppercase. The likelihood mapping A
relates outcomes to the states that cause them, whereas D harnesses our prior
beliefs about states, independent of how they are sampled. The precision term
$\gamma$ controls the precision or weighting assigned to elements of the
likelihood, and implements attention as precision-weighting. Figure from [61].
Figure 2: A generative model for policy selection. This figure depicts a more
sophisticated generative model that is apt for planning and the selection of
actions in the future. The basic model depicted in Figure 1 has now been
expanded to include beliefs about the current course of action or policy
(denoted $\bar{\pi}$), as well as B, C, E, F and G parameters. This kind of
model generates a time series of states ($s_{1}$, $s_{2}$, etc.) and outcomes
($o_{1}$, $o_{2}$, etc.). The state transition (B) parameter encodes the
transition probabilities between states over time, independently of the way
they are sampled. B, C, E, F and G enter into the selection of beliefs about
courses of action, a.k.a. policies. The C vector specifies preferred or
expected outcomes and enters into the calculation of variational (F) and
expected (G) free energies. The E vector specifies a prior preference for
specific courses of action. Figure from [61].

The main advantage of using generative models over current state of the art
black box approaches is interpretability and auditability. Indeed, the factors
that figure in the generative model are explicitly labeled, such that their
contributions to the operations of the model can be read directly off its
structure. This lends the generative model a degree of auditability that other
approaches do not have.

###  2.2 Active inference, introspection, and self-modeling

Active inference modeling has been deployed in the context of the scientific
study of introspection, self-modeling, and self-access, which has led to the
development of several leading theories of consciousness (for a review, see
[69, 49]). Introspection, which is defined as the ability to access and
evaluate one’s own mental states, thoughts, and experiences, plays a pivotal
role in self-awareness, learning, and decision-making and is a pillar of human
consciousness [70]. Self-modeling and self-access can be defined as
interconnected processes that contribute to the development of self-awareness
and to the capacity for introspection. Self-modeling involves the creation of
internal representations of oneself, while self-access refers to the ability
to access and engage with these representations for self-improvement and
learning [71, 72]. These processes, in conjunction with introspection, form a
complex dynamic system that enriches our understanding of consciousness and
the self—and indeed, may arguably form the causal basis of our capacity to
understand ourselves and others.

Introspective self-access has been modeled using active inference by deploying
a hierarchically structured generative model [73]. The basic idea is that for
a system to report or evaluate its own inferences, it must be able to enact
some form of self-access, where some parts of the system can take the output
of other parts as their own input, for further processing. This has been
discussed in computational neuroscience under the rubric of “opacity” and
“transparency” [74, 75, 76, 61]. The idea is that some cognitive processes are
“transparent”: like a (clean, transparent) window, they enable us to access
some other thing (say, a tree outside) while not themselves being perceivable.
Other cognitive processes are “opaque”: they can be assessed per se, as in
introspective self-awareness (i.e., aware that you are looking at a tree as
opposed to seeing a tree). The idea, then, is that introspective processes
make other cognitive processes accessible to the system as such, rendering
them opaque.

In the context of self-access, the transparency and opacity of introspective
processes has been modeled using a three-level generative model [61]. The
model is depicted in Figure 3. This model provides a framework for
understanding how we access and interpret our internal states and experiences.
The first level of the model (in blue), which implements the selection of
overt actions, can be seen as a transparent process. The second,
hierarchically superordinate level (in orange), which implements attention and
covert action [76, 48], represents more opaque processes, which make processes
in the first layer accessible to the system. This layer models mental actions
and shifts in attention that we may not be consciously aware of, or able to
report. The second level takes as its input the inferences (posterior state
estimations) ongoing at the first level, as data for further inference—about
the system’s inferences. Attentional processes are of this sort: they are
about cognitive processes and action, and they modulate the activity of the
first level. The third, final level (in green) implements the awareness of
where one’s attention is deployed. In other words, it both recognizes and
instantiates a particular attentional set via bottom-up and top-down messages
between levels, respectively. On the whole, this three-level architecture
models our self-access and introspective abilities in terms of the processes
regulating transparency and opacity at a phenomenal level of description, or
attentional selection at a psychological level.

Figure 3: A hierarchical generative model capable of self-access. Here, the
generative model depicted in Figure 2 (in blue) has been augmented with two
superordinate hierarchical layers. In this architecture, posterior state
estimates at one level are passed onto the next level as data for further
inference. Note that this induces an architecture where the system is able to
make inferences about its own inferences. Figure from [61].

Ramstead, Albarracin et al. (ramstead2023inner) [48] recently discussed how
active inference enables us to model both overt and covert action (also see
[76, 70, 73, 58, 59]). Overt actions—observable behaviors such as physical
movements or verbal responses—are directly influenced by the brain’s
hierarchical organization and can be modeled using active inference [77, 78,
79]. In contrast, covert actions refer to internal mental processes, such as
attention and imagination, which involve the manipulation and processing of
internal representations in the absence of observable behaviors [80, 81, 82,
83, 84, 85, 86, 87, 88, 89]—of the sort discussed as “mental action” [76, 70,
90, 61]. These actions are essential for higher cognitive functions, which
rely on the brain’s capacity to explore and manipulate abstract concepts and
relationships.

In Smith et al. (smith2019neurocomputational) [91], a hierarchical
architecture of this type was deployed that was augmented with the capacity to
report on its emotional states. Thus, it is possible to use active inference
to design systems that can not only access their own states and perform
inferences on their basis, but also to report on their introspective processes
in a manner that is readily understandable by human users and stakeholders.
With this formulation of how active inference enables agents to model their
overt and covert action, in the following sections, we argue that we can and
ought to research, design, and develop AI systems that mimic these
introspective processes, ultimately leading to more human-like artificial
intelligence.

##  3 Using active inference to design self-explaining AI

We argue that incorporating the design principles of active inference into AI
systems can lead to better explainability. This is for two key reasons. The
first is that, by deploying an explicit generative model, AI systems premised
on active inference are designed explicitly such that their operations can be
interpreted and audited by a user or stakeholder that is fluent in the
operation of such models. We believe that the inherent explainability of
active inference AI might be scaled up, by deploying the kind of explicit,
standardized world modelling techniques that are being developed as open
standards within the Institute of Electrical and Electronics Engineers (IEEE)
P2874 Spatial Web Working Group [92], to formalize contextual relationships
between entities and processes and to create digital twins of environments
that are able to update in real time.

The second is that, by implementing an architecture inspired by active
inference models of introspection, we can build systems that are able to
access—and report on—the reasons for their decisions, and their state of mind
when reaching these decisions.

AI systems designed using active inference can incorporate the kind of
hierarchical self-access described by [61] and by [91], to enhance their
introspection during decision-making. As discussed, in the active inference
tradition, introspection can be understood in the context of the (covert and
overt) actions that AI systems perform. Covert actions, which are internal
computations and decision-making processes that are not directly observable to
users and stakeholders, can be recorded or explained to make the system more
explainable. Overt actions, which are actions that an AI system takes based on
its internal computations, such as making a recommendation or decision, can be
explained to help users understand why the AI system acted as it did. This
kind of deep inference promotes introspection, adaptability, and responses to
environmental changes [93, 94].

The proposed AI architecture includes components that continuously update and
maintain an internal model of its own states, beliefs, and goals. This
capacity for self-access (and implicitly self-report) enables the AI system to
optimize (and report on) its decision-making processes, fostering
introspection (and enhanced explainability). It incorporates metacognitive
processing capabilities, which involve the ability to monitor, control, and
evaluate its own cognitive processes. The AI system can thereby better explain
the factors that contribute to its decisions, as well as identify potential
biases or errors, ultimately leading to improved decision-making and
explainability.

The proposed AI architecture would include introspection and a self-report
interface, which translates the AI system’s internal models and decision-
making processes into human-understandable (natural) language (using, e.g.,
large language models). In effect, the agent would be talking to itself,
describing its current state of mind and beliefs. This interface bridges the
gap between the AI system’s internal workings and human users, promoting
epistemic trust and collaboration. In this way, the system can effectively
mimic human-like consciousness and transparent introspection, leading to a
deeper understanding of its decision-making processes and explainability. This
advancement may be essential in fostering trust and collaboration between AI
systems and their human users, paving the way for more effective and
responsible AI applications.

Augmenting a generative model with black box systems—like large language
models—may be a useful strategy to help AI systems articulate their
“understanding” of the world. Using large language models to furnish an
introspective interface may be relatively straightforward, leveraging their
powerful natural language processing capabilities to create explanations of
belief updating. This architecture—with a hierarchical generative model at its
core—may contribute to the overall performance and explainability of hybrid AI
systems. Attention mechanisms also achieve this purpose by enhancing the
explainability of the AI system’s decision making, emphasizing important
factors in the hierarchical generative model that contribute to its decisions
and actions.

These ideas are not new. Attentional mechanisms, particularly those at the
word-level, have been identified as crucial components in AI architecture,
specifically in the context of hierarchical generative models—and in
generative AI, in the form of transformers. They function by focusing on
relevant aspects during decision-making processes, thereby allowing the system
to effectively process and prioritize information [95]. In fact, the
performance of hierarchical models, which are a type of AI architecture, can
be significantly improved by integrating word-level attention mechanisms.
These mechanisms are powerful because they can leverage context information
more effectively, especially fine-grained information.

The AI architecture that we propose employs a soft attention mechanism, which
uses a weighted combination of hierarchical generative model components to
focus on relevant information. The attention weights are dynamically computed
based on the input data and the AI system’s internal state, allowing the
system to adaptively focus on different aspects of the hierarchical generative
model [96]. This approach is similar to the use of deep learning models for
global coordinate transformations that linearize partial differential
equations, where the model is trained to learn a transformation from the
physical domain to a computational domain where the governing partial
differential equations are simpler, or even linear [97].

The AI architecture that we describe here effectively integrates diverse
information sources for decision-making, mirroring the complex information
processing capabilities observed in the human brain. The hierarchical
structure of the generative model facilitates the exchange of information
between different levels of abstraction. This exchange allows the AI system to
refine and update its internal models based on both high-level abstract
knowledge and low-level detailed information.

In conclusion, the integration of introspective processes in AI systems may
represent a significant step towards achieving more explainable AI. By
leveraging explicit generative models, as well as attention and introspection
mechanisms, we can design AI systems that are not only more efficient and
robust, but also more understandable and trustworthy. This approach allows us
to bridge the gap between the complex internal computations of AI systems and
the human users who interact with them. Ultimately, the goal is to create AI
systems that can effectively communicate the reasons that drive their
decision-making processes, adapt to environmental changes, and collaborate
seamlessly with human users. As we continue to advance in this field, the
importance of introspection in AI will only become more apparent, paving the
way for more sophisticated and ethically sound AI systems.

##  4 Discussion

###  4.1 Directions for future research

The problem of explainable AI is the problem of understanding how AI models
arrive at their decisions or predictions. This problem is especially relevant
to avoid biases and harm in the design, implementation, and use of AI systems.
By incorporating explicit generative models and introspective processing into
the proposed AI architecture, we can create a system that is or seems capable
of introspection and, thereby, that displays greatly enhanced explainability
and auditability. This approach to AI design paves the way for more effective
AI deployment across various real-world applications, by shedding light upon
the problem of explainability, thereby offering opportunities for fostering
trust, fairness, and inclusivity.

The development of the AI architecture based on active inference opens several
potential avenues for future research. One possible direction is to further
investigate the role of attention and introspection mechanisms in both AI
systems and human cognition, as well as the development of more efficient
attentional models to improve the AI system’s ability to focus on salient
information during decision-making. The approach that we propose bridges the
gap between AI and cognitive neuroscience by incorporating biologically-
inspired mechanisms into the design of AI systems. As a result, the proposed
architecture promotes a deeper understanding of the nature of cognition and
its potential applications in artificial intelligence, thus paving the way for
more human-like AI systems capable of introspection and enhanced collaboration
with human users.

Future work could explore more advanced data fusion techniques, such as deep
learning-based fusion or probabilistic fusion, to improve the AI system’s
ability to combine and process multimodal data effectively. Evaluating the
effectiveness of these techniques in diverse application domains will also be
a valuable avenue for research [98, 99]. Furthermore, the explanation
dimension of these AI systems has been a significant topic in recent years,
particularly in decision-making scenarios. These systems provide more
awareness of how AI works and its outcomes, building a relationship with the
system and fostering trust between AI and humans [100].

In addition to the aforementioned avenues for future research, another
promising direction lies in the realm of computational phenomenology (for a
review and discussion, see [101]. Beckmann, Köstner, & Hipólito
(beckmann2023rejecting) [102] have proposed a framework that deploys
phenomenology—the rigorous descriptive study of first-person experience—for
the purposes of machine learning training. This approach conceptualizes the
mechanisms of artificial neural networks in terms of their capacity to capture
the statistical structure of some kinds of lived experience, offering a unique
perspective on deep learning, consciousness, and their relation. By grounding
AI training in socioculturally situated experience, we can create systems that
are more aware of sociocultural biases and capable of mitigating their impact.
Ramstead et al. (ramstead2022generative) [101] propose a similar methodology
based on explicit generative models as they figure in the active inference
tradition. This connection to first-person experience, of course, does not
guarantee unbiased AI. But by moving away from traditional black box AI
systems, we shift towards human-interpretable models that enable the
identification and correction of biases in the AI system. This approach aligns
with our goal of creating AI systems that are not only efficient and
effective, but also ethically sound and socially responsible.

The incorporation of computational phenomenology into our proposed AI
architecture could further enhance its introspective capabilities and its
ability to understand and navigate the complexities of human sociocultural
contexts. This could lead to AI systems that are more adaptable, more
trustworthy, and more capable of meaningful collaboration with human users. As
we continue to explore and integrate such innovative approaches, we move
closer to our goal of creating AI systems that truly mirror the richness and
complexity of human cognition and consciousness.

###  4.2 Ethical considerations of introspective AI systems

Ethical AI starts with the development of AI systems that are ethically
designed; AI systems must be designed in such a way as to be transparent,
auditable, and explainable, and to minimize harm. But as these systems become
increasingly integrated into our daily lives, research on the ethical
implications of introspective AI systems, as well as the development of
regulatory frameworks and guidelines for responsible AI use, become crucial.
The development of introspective AI systems raises several ethical
considerations. Even if these systems provide more human-like decision-making
capabilities and enhanced explainability, it is and will remain crucial to
ensure that their decisions are transparent, fair, and unbiased, and that
their designers and users can be held accountable for harm that their use may
cause.

To address these concerns, future research should focus on developing methods
to audit and evaluate the AI system’s decision-making processes, as well as
identify and mitigate potential biases within the system. Additionally, the
development of ethical guidelines and regulatory frameworks for the use of
introspective AI systems will be essential to ensure that they are deployed
responsibly and transparently. Moreover, as introspective AI systems become
more prevalent, issues related to agency, privacy, and data security may
arise. Ensuring that these systems protect sensitive information by abiding by
data protection regulations, thereby safeguarding agency, will be of paramount
importance.

In conclusion, the development of AI systems based on active inference has
broad implications for both the fields of AI and consciousness studies. As
future research explores the potential of this novel approach, ethical
considerations and responsible use of introspective AI systems must remain at
the forefront of these advancements, ultimately leading to more transparent,
effective, and user-friendly AI applications.

##  5 Conclusion

We have argued that active inference has demonstrated significant potential in
advancing the field of explainable AI. By incorporating design principles from
active inference, the AI system can better tackle complex real-world problems
with improved auditability of decision-making, thereby increasing safety and
user trust.

Throughout our discussions and analysis, we have highlighted the importance of
active inference models as a foundation for designing more human-like AI
systems, seemingly capable of introspection and finessed (epistemic)
collaboration with human users. This novel approach bridges the gap between AI
and cognitive neuroscience by incorporating biologically-inspired mechanisms
into the design of AI systems, thus promoting a deeper understanding of the
nature of consciousness and its potential applications in artificial
intelligence.

As we move forward in the development of AI systems, the importance of
advancing explainable AI becomes increasingly apparent. By designing AI
systems that can not only make accurate and efficient decisions, but also
provide understandable explanations for their decisions, we foster (epistemic)
trust and collaboration between AI systems and human users. This advancement
ultimately leads to more transparent, effective, and user-friendly AI
applications that can be tailored to a wide range of real-world scenarios.

## References

  * [1] Wullianallur Raghupathi and Viju Raghupathi  “Big data analytics in healthcare: Promise and potential”  In _Health Information Science and Systems_ 2, 2014, pp. 1–10  DOI: 10.1186/2047-2501-2-3
  * [2] Miguel Mascarenhas et al.  “The Promise of Artificial Intelligence in Digestive Healthcare and the Bioethics Challenges It Presents”  In _Medicina_ 59.4, 2023, pp. 790  DOI: 10.3390/medicina59040790
  * [3] Ian Goodfellow, Yoshua Bengio and Aaron Courville  “Deep Learning”  MIT press, 2016
  * [4] Yann LeCun, Yoshua Bengio and Geoffrey Hinton  “Deep learning”  In _Nature_ 521.7553, 2015, pp. 436–444  DOI: 10.1038/nature14539
  * [5] Davide Castelvecchi  “Can we open the black box of AI?”  In _Nature News_ 538.7623, 2016, pp. 20  DOI: 10.1038/538020a
  * [6] David Gunning  “Explainable Artificial Intelligence (XAI)”  In _Defense Science Research Projects Agency_ 2.2, 2017, pp. 1  DOI: 10.1609/aimag.v40i2.2850
  * [7] Finale Doshi-Velez and Been Kim  “Towards a rigorous science of interpretable machine learning”  In _arXiv_ , 2017  DOI: 10.48550/arXiv.1702.08608
  * [8] Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin  ““Why should I trust you?” Explaining the predictions of any classifier”  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2016, pp. 1135–1144  DOI: 10.1145/2939672.2939778
  * [9] Abeba Birhane  “The impossibility of automating ambiguity”  In _Artificial Life_ 27.1, 2021, pp. 44–61  DOI: 10.1162/artl_a_00336
  * [10] Abeba Birhane et al.  “Frameworks and Challenges to Participatory AI”  In _Proceeding of the Second Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO ’22)_ , 2022  DOI: 10.48550/arXiv.2209.07572
  * [11] Jean-Christophe Bélisle-Pipon, Erica Monteferrante, Marie-Christine Roy and Vincent Couture  “Artificial intelligence ethics has a black box problem”  In _AI & SOCIETY_, 2022, pp. 1–16  DOI: 10.1007/s00146-021-01380-0
  * [12] Kevin Bauer, Moritz Zahn and Oliver Hinz  “Expl(AI)ned: The impact of explainable artificial intelligence on cognitive processes”  In _Information Systems Research_ , 2021  DOI: 10.1287/isre.2023.1199
  * [13] Sajid Ali et al.  “Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence”  In _Information Fusion_ , 2023, pp. 101805  DOI: 10.1016/j.inffus.2023.101805
  * [14] Jacques A Esterhuizen, Bryan R Goldsmith and Suljo Linic  “Interpretable machine learning for knowledge generation in heterogeneous catalysis”  In _Nature Catalysis_ 5.3, 2022, pp. 175–184  DOI: 10.1038/s41929-022-00744-z
  * [15] Abhishek Mishra  “Transparent AI: Reliabilist and proud”  In _Journal of Medical Ethics_ 47.5, 2021, pp. 341–342  DOI: 10.1136/medethics-2021-107352
  * [16] Warren J Eschenbach  “Transparency and the black box problem: Why we do not trust AI”  In _Philosophy & Technology_ 34.4, 2021, pp. 1607–1622  DOI: 10.1007/s13347-021-00477-0
  * [17] Michael Veale and Reuben Binns  “Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data”  In _Big Data & Society_ 4.2, 2017  DOI: 10.1177/2053951717743530
  * [18] Benjamin Giffen, Dennis Herhausen and Tobias Fahse  “Overcoming the pitfalls and perils of algorithms: A classification of machine learning biases and mitigation methods”  In _Journal of Business Research_ 144, 2022, pp. 93–106  DOI: 10.1016/j.jbusres.2022.01.076
  * [19] Nathalia Nascimento, Paulo Alencar and Donald Cowan  “Comparing Software Developers with ChatGPT: An Empirical Investigation”  In _arXiv_ , 2023  DOI: 10.48550/arXiv.2305.11837
  * [20] Gregor Stiglic et al.  “Interpretability of machine learning-based prediction models in healthcare”  In _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ 10.5, 2020, pp. e1379  DOI: 10.1002/widm.1379
  * [21] Samuli Laato, Miika Tiainen, AKM Najmul Islam and Matti Mäntymäki  “How to explain AI systems to end users: A systematic literature review and research agenda”  In _Internet Research_ 32.7, 2022, pp. 1–31  DOI: 10.1108/INTR-08-2021-0600
  * [22] Emanuele Neri et al.  “Explainable AI in radiology: A white paper of the Italian Society of Medical and Interventional Radiology”  In _La Radiologia Medica_ , 2023, pp. 1–10  DOI: 10.1007/s11547-023-01634-5
  * [23] Luis Oberste et al.  “Designing User-Centric Explanations for Medical Imaging with Informed Machine Learning”  In _Design Science Research for a New Society: Society 5.0: 18th International Conference on Design Science Research in Information Systems and Technology, DESRIST 2023, Pretoria, South Africa, May 31–June 2, 2023, Proceedings_ , 2023, pp. 470–484  DOI: 10.1007/978-3-031-32808-4_29
  * [24] Thi-Thu-Huong Le et al.  “Exploring Local Explanation of Practical Industrial AI Applications: A Systematic Literature Review”  In _Applied Sciences_ 13.9, 2023, pp. 5809  DOI: 10.3390/app13095809
  * [25] Amina Adadi and Mohammed Berrada  “Peeking inside the black-box: A survey on explainable artificial intelligence (XAI)”  In _IEEE Access_ 6, 2018, pp. 52138–52160  DOI: 10.1109/ACCESS.2018.2870052
  * [26] Riccardo Guidotti et al.  “A survey of methods for explaining black box models”  In _ACM Computing Surveys_ 51.5, 2018, pp. 1–42  DOI: 10.1145/3236009
  * [27] Jenna Burrell  “How the machine ‘thinks’: Understanding opacity in machine learning algorithms”  In _Big Data & Society_ 3.1, 2016, pp. 2053951715622512  DOI: 10.1177/2053951715622512
  * [28] Beatriz San Miguel, Aisha Naseer and Hiroya Inakoshi  “Putting accountability of AI systems into practice”  In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , 2021, pp. 5276–5278  DOI: 10.24963/ijcai.2020/768
  * [29] Nadin Kokciyan, Biplav Srivastava, Michael N Huhns and Munindar P Singh  “Sociotechnical perspectives on AI ethics and accountability”  In _IEEE Internet Computing_ 25.6, 2021, pp. 5–6  DOI: 10.1109/MIC.2021.3117611
  * [30] Inês Hipólito, Katie Winkle and Merete Lie  “Enactive Artificial Intelligence: Subverting Gender Norms in Robot-Human Interaction”  In _Frontiers in Neurorobotics_ 17, 2023, pp. 77  DOI: 10.48550/arXiv.2301.08741
  * [31] Abeba Birhane et al.  “The forgotten margins of AI ethics”  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , 2022, pp. 948–958  DOI: 10.1145/3531146.3533157
  * [32] Marianna Drake, Jiayen Ong, Marty Hansen and Lisa Peets  “EU AI Policy and Regulation: What to look out for in 2023”  In _Inside Privacy_ , 2023  URL: https://www.insideprivacy.com/artificial-intelligence/eu-ai-policy-and-regulation-what-to-look-out-for-in-2023/
  * [33] Caleb Skeath, Lindsey Tonsager and Jenna Zhang  “FTC Announces COPPA Settlement Against Ed Tech Provider Including Strict Data Minimization and Data Retention Requirements”  In _Inside Privacy_ , 2023  URL: https://www.insideprivacy.com/childrens-privacy/ftc-announces-coppa-settlement-against-ed-tech-provider-including-strict-data-minimization-and-data-retention-requirements
  * [34] Andrea Brennen  “What Do People Really Want When They Say They Want “Explainable AI?” We Asked 60 Stakeholders”  In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_ , 2020, pp. 1–7  DOI: 10.1145/3334480.3383047
  * [35] Emanuele Ratti and Mark Graves  “Explainable machine learning practices: Opening another black box for reliable medical AI”  In _AI and Ethics_ 2.4, 2022, pp. 801–814  DOI: 10.1007/s43681-022-00141-z
  * [36] Michael Ridley  “Explainable Artificial Intelligence (XAI)”  In _Information Technology and Libraries_ 41.2, 2022  DOI: 10.6017/ital.v41i2.14683
  * [37] Olivia Guest and Andrea E Martin  “On logical inference over brains, behaviour, and artificial neural networks”  In _Computational Brain & Behavior_, 2023, pp. 1–15  DOI: 10.1007/s42113-022-00166-x
  * [38] Inês Hipólito  “The Human Roots of Artificial Intelligence”, 2023  DOI: 10.31234/osf.io/cseqt
  * [39] Muhammad Ali Chaudhry, Mutlu Cukurova and Rose Luckin  “A transparency index framework for AI in education”  In _Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium: 23rd International Conference, AIED 2022, Durham, UK, July 27–31, 2022, Proceedings, Part II_ , 2022, pp. 195–198  DOI: 10.1007/978-3-031-11647-6_33
  * [40] Alejandro Barredo Arrieta et al.  “Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI”  In _Information Fusion_ 58, 2020, pp. 82–115  DOI: 10.1016/j.inffus.2019.12.012
  * [41] Mohit Prabhushankar and Ghassan AlRegib  “Introspective learning: A two-stage approach for inference in neural networks”  In _arXiv_ , 2022  URL: https://openreview.net/forum?id=in1ynkrXyMH
  * [42] Sebastian Bach et al.  “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation”  In _PloS One_ 10.7, 2015, pp. e0130140  DOI: 10.1371/journal.pone.0130140
  * [43] Quanshi Zhang, Ying Nian Wu and Song-Chun Zhu  “Interpretable convolutional neural networks”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8827–8836  DOI: 10.1109/CVPR.2018.00920
  * [44] William Franz Lamberti  “An overview of explainable and interpretable AI”  In _AI Assurance_ , 2023, pp. 55–123  DOI: 10.1016/B978-0-32-391919-7.00015-9
  * [45] Maxwell J D Ramstead et al.  “On Bayesian mechanics: A physics of and by beliefs”  In _Interface Focus_ 13, 2023, pp. 20220029  DOI: 10.1098/rsfs.2022.0029
  * [46] Karl J Friston et al.  “Designing Ecosystems of Intelligence from First Principles”  In _arXiv_ , 2022  DOI: 10.48550/arXiv.2212.01354
  * [47] Jakob Hohwy  “The Predictive Mind”  Oxford University Press, 2013  DOI: 10.1093/acprof:oso/9780199682737.001.0001
  * [48] Maxwell J D Ramstead et al.  “The inner screen model of consciousness: Applying the free energy principle directly to the study of conscious experience”  In _PsyArXiv_ , 2023  DOI: 10.31234/osf.io/6afs3
  * [49] Maxwell J D Ramstead et al.  “Steps towards a minimal unifying model of consciousness: An integration of models of consciousness based on the free energy principle”, 2023
  * [50] Maxwell J D Ramstead, Paul Benjamin Badcock and Karl John Friston  “Answering Schrödinger’s question: A free-energy formulation”  In _Physics of Life Reviews_ 24, 2018, pp. 1–16  DOI: 10.1016/j.plrev.2017.09.001
  * [51] Maxwell J D Ramstead, Axel Constant, Paul B Badcock and Karl J Friston  “Variational ecology and the physics of sentient systems”  In _Physics of Life Reviews_ 31, 2019, pp. 188–205  DOI: 10.1016/j.plrev.2018.12.002
  * [52] Thomas Parr and Giovanni Pezzulo  “Understanding, explanation, and active inference”  In _Frontiers in Systems Neuroscience_ 15, 2021, pp. 772641  DOI: 10.3389/fnsys.2021.772641
  * [53] Ryan Smith, Sahib S Khalsa and Martin P Paulus  “An active inference approach to dissecting reasons for nonadherence to antidepressants”  In _Biological Psychiatry: Cognitive Neuroscience and Neuroimaging_ 6.9, 2021, pp. 919–934  DOI: 10.1016/j.bpsc.2019.11.012
  * [54] Ryan Smith, Samuel Taylor and Edda Bilek  “Computational mechanisms of addiction: Recent evidence and its relevance to addiction medicine”  In _Current Addiction Reports_ , 2021, pp. 1–11  DOI: 10.1007/s40429-021-00399-z
  * [55] Rick A Adams, Stewart Shipp and Karl J Friston  “Predictions not commands: Active inference in the motor system”  In _Brain Structure and Function_ 218.3, 2013, pp. 611–643  DOI: 10.1007/s00429-012-0475-5
  * [56] Rick A Adams et al.  “Everything is connected: Inference and attractors in delusions”  In _Schizophrenia Research_ 245, 2022, pp. 5–22  DOI: 10.1016/j.schres.2021.07.032
  * [57] Philipp Sterzer et al.  “The predictive coding account of psychosis”  In _Biological Psychiatry_ 84.9, 2018, pp. 634–643  DOI: 10.1016/j.biopsych.2018.05.015
  * [58] Stephen M Fleming  “Awareness as inference in a higher-order state space”  In _Neuroscience of Consciousness_ 2020.1, 2020, pp. niz020  DOI: 10.1093/nc/niz020
  * [59] Daniel Yon and Chris D Frith  “Precision and the Bayesian brain”  In _Current Biology_ 31.17, 2021, pp. R1026–R1032  DOI: 10.1016/j.cub.2021.07.044
  * [60] Chris D Frith  “Consciousness, (meta) cognition, and culture”  In _Quarterly Journal of Experimental Psychology_ , 2023, pp. 17470218231164502  DOI: 10.1177/17470218231164502
  * [61] Lars Sandved-Smith et al.  “Towards a computational phenomenology of mental action: Modelling meta-awareness and attentional control with deep parametric active inference”  In _Neuroscience of Consciousness_ 2021.1, 2021  DOI: 10.1093/nc/niab018
  * [62] Axel Constant, Maxwell J D Ramstead, Samuel P L Veissière and Karl J Friston  “Regimes of expectations: An active inference model of social conformity and human decision making”  In _Frontiers in Psychology_ 10, 2019, pp. 679  DOI: 10.3389/fpsyg.2019.00679
  * [63] Lancelot Da Costa, Karl J Friston, Conor Heins and Grigorios A Pavliotis  “Bayesian mechanics for stationary processes”  In _Proceedings of the Royal Society A_ 477.2256, 2021  DOI: 10.1098/rspa.2021.0518
  * [64] Karl J Friston  “Is the free-energy principle neurocentric?”  In _Nature Reviews Neuroscience_ 11.8, 2010, pp. 605–605  DOI: 10.1038/nrn2787-c2
  * [65] Karl J Friston  “A theory of cortical responses”  In _Philosophical Transactions of the Royal Society B: Biological Sciences_ 360.1456, 2005, pp. 815–836  DOI: 10.1098/rstb.2005.1622
  * [66] Yuki Konaka and Honda Naoki  “Decoding reward–curiosity conflict in decision-making from irrational behaviors”  In _Nature Computational Science_ 3.5, 2023, pp. 418–432  DOI: 10.1038/s43588-023-00439-w
  * [67] Karl J Friston  “Life as we know it”  In _Journal of the Royal Society Interface_ 10.86, 2013, pp. 20130475  DOI: 10.1098/rsif.2013.0475
  * [68] Jakob Hohwy  “The self-evidencing brain”  In _Nous_ 50.2, 2016, pp. 259–285  DOI: 10.1111/nous.12062
  * [69] Anil K Seth and Tim Bayne  “Theories of consciousness”  In _Nature Reviews Neuroscience_ 23.7, 2022, pp. 439–452  DOI: 10.1038/s41583-022-00587-4
  * [70] Jakub Limanowski and Karl J Friston  “‘Seeing the dark’: Grounding phenomenal transparency and opacity in precision estimation for active inference”  In _Frontiers in Psychology_ 9, 2018, pp. 643  DOI: 10.3389/fpsyg.2018.00643
  * [71] Garold Murray  “Self-Access Environments as Self-Enriching Complex Dynamic Ecosocial Systems”  In _Studies in Self-Access Learning Journal_ 9.2, 2018  DOI: 10.37237/090204
  * [72] John R Baker  “Going beyond brick and mortar self-access centers: Establishing a satellite activity self-access program”  In _Studies in Self-Access Learning Journal_ 13.1, 2022, pp. 129–141  DOI: 10.37237/130107
  * [73] Jakub Limanowski and Karl J Friston  “Attenuating oneself: An active inference perspective on “selfless” experiences”  In _Philosophy and the Mind Sciences_ 1.I, 2020, pp. 1–16  DOI: 10.33735/phimisci.2020.I.35
  * [74] Thomas Metzinger  “Phenomenal transparency and cognitive self-reference”  In _Phenomenology and the Cognitive Sciences_ 2, 2003, pp. 353–393  DOI: 10.1023/b:phen.0000007366.42918.eb
  * [75] Thomas Metzinger  “Empirical perspectives from the self-model theory of subjectivity: A brief summary with examples”  In _Progress in Brain Research_ 168, 2007, pp. 215–278  DOI: 10.1016/S0079-6123(07)68018-2
  * [76] Thomas Metzinger  “The problem of mental action”  In _Philosophy and Predictive Processing_ Frankfurt am Main: MIND Group, 2017  DOI: 10.15502/9783958573208
  * [77] Karl J Friston, Jérémie Mattout and James Kilner  “Action understanding and active inference”  In _Biological Cybernetics_ 104, 2011, pp. 137–160  DOI: 10.1007/s00422-011-0424-z
  * [78] Karl J Friston et al.  “Deep temporal models and active inference”  In _Neuroscience & Biobehavioral Reviews_ 77, 2017, pp. 388–402  DOI: 10.1016/j.neubiorev.2017.04.009
  * [79] Karl J Friston, Thomas Parr and Bert Vries  “The graphical brain: Belief propagation and active inference”  In _Network Neuroscience_ 1.4, 2017, pp. 381–414  DOI: 10.1162/NETN_a_00018
  * [80] Giovanni Pezzulo  “An Active Inference view of cognitive control”  In _Frontiers in Psychology_ 3, 2012, pp. 478  DOI: 10.3389/fpsyg.2012.00478
  * [81] Harriet Feldman and Karl J Friston  “Attention, uncertainty, and free-energy”  In _Frontiers in Human Neuroscience_ 4, 2010  DOI: 10.3389/fnhum.2010.00215
  * [82] Mark J Edwards et al.  “A Bayesian account of ‘hysteria”’  In _Brain_ 135.11, 2012, pp. 3495–3512  DOI: 10.1093/brain/aws129
  * [83] Jakob Hohwy  “Attention and conscious perception in the hypothesis testing brain”  In _Frontiers in Psychology_ 3, 2012, pp. 96  DOI: 10.3389/fpsyg.2012.00096
  * [84] Harriet Brown et al.  “Active inference, sensory attenuation and illusions”  In _Cognitive Processing_ 14, 2013, pp. 411–427  DOI: 10.1007/s10339-013-0571-3
  * [85] Ryota Kanai, Yutaka Komura, Stewart Shipp and Karl J Friston  “Cerebral hierarchies: predictive processing, precision and the pulvinar”  In _Philosophical Transactions of the Royal Society B: Biological Sciences_ 370.1668, 2015, pp. 20140169  DOI: 10.1098/rstb.2014.0169
  * [86] Simone Vossel, Christoph Mathys, Klaas E Stephan and Karl J Friston  “Cortical coupling reflects Bayesian belief updating in the deployment of spatial attention”  In _Journal of Neuroscience_ 35.33, 2015, pp. 11532–11542  DOI: 10.1523/JNEUROSCI.1382-15.2015
  * [87] Vivien Ainley, Matthew A J Apps, Aikaterini Fotopoulou and Manos Tsakiris  “‘Bodily precision’: a predictive coding account of individual differences in interoceptive accuracy”  In _Philosophical Transactions of the Royal Society B: Biological Sciences_ 371.1708, 2016, pp. 20160003  DOI: 10.1098/rstb.2016.0003
  * [88] Jakub Limanowski  “(Dis-)Attending to the Body — Action and Self-Experience in the Active Inference Framework”  In _Philosophy and Predictive Processing_ Frankfurt am Main: MIND Group, 2017  DOI: 10.15502/9783958573192
  * [89] Thomas Parr and Karl J Friston  “Attention or salience?”  In _Current Opinion in Psychology_ 29, 2019, pp. 1–5  DOI: 10.1016/j.copsyc.2018.10.006
  * [90] Jakub Limanowski  “Precision control for a flexible body representation”  In _Neuroscience and Biobehavioral Reviews_ 134, 2022, pp. 104401  DOI: 10.1016/j.neubiorev.2021.10.023
  * [91] Ryan Smith, Richard D Lane, Thomas Parr and Karl J Friston  “Neurocomputational mechanisms underlying emotional awareness: Insights afforded by deep active inference and their potential clinical relevance”  In _Neuroscience & Biobehavioral Reviews_ 107, 2019, pp. 473–491  DOI: 10.1016/j.neubiorev.2019.09.002
  * [92] “Standard for Spatial Web Protocol, Architecture and Governance”  In _IEEE Standards Association_ , 2020  URL: https://standards.ieee.org/ieee/2874/10375/
  * [93] Somayajulu L N Dhulipala and Ryan C Hruska  “Efficient interdependent systems recovery modeling with DeepONets”  In _arXiv_ , 2022, pp. 1–6  DOI: 10.48550/arXiv.2206.10829
  * [94] Jakob Schoeffer et al.  “On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making”  In _arXiv_ , 2023  DOI: 10.48550/arXiv.2304.08804
  * [95] Tian Lan, Xian-Ling Mao, Wei Wei and Heyan Huang  “Which Kind Is Better in Open-domain Multi-turn Dialog, Hierarchical or Non-hierarchical Models? An Empirical Study”  In _arXiv_ , 2020  DOI: 10.48550/arXiv.2008.02964
  * [96] Mandar Kulkarni and Aria Abubakar  “Soft Attention Convolutional Neural Networks for Rare Event Detection in Sequences”, 2020  DOI: 10.48550/arXiv.2011.02338
  * [97] Craig Gin, Bethany Lusch, Steven L Brunton and J Nathan Kutz  “Deep learning models for global coordinate transformations that linearise PDEs”  In _European Journal of Applied Mathematics_ 32.3, 2021, pp. 515–539  DOI: 10.1017/S0956792520000327
  * [98] Dana Lahat, Tülay Adali and Christian Jutten  “Multimodal data fusion: An overview of methods, challenges, and prospects”  In _Proceedings of the IEEE_ 103.9, 2015, pp. 1449–1477  DOI: 10.1109/JPROC.2015.2460697
  * [99] Microsoft Defender Security Research Team  “Seeing the big picture: Deep learning-based fusion of behavior signals for threat detection”, 2020  URL: https://tinyurl.com/3kpzvk9d
  * [100] Juliana Jansen Ferreira and Mateus Monteiro  “The human-AI relationship in decision-making: AI explanation to support people on justifying their decisions”  In _arXiv_ , 2021  DOI: 10.48550/arXiv.2102.05460
  * [101] Maxwell J D Ramstead et al.  “From generative models to generative passages: A computational approach to (Neuro) Phenomenology”  In _Review of Philosophy and Psychology_ 13.4, 2022  DOI: 10.1007/s13164-021-00604-y
  * [102] Pierre Beckmann, Guillaume Köstner and Inês Hipólito  “Rejecting Cognitivism: Computational Phenomenology for Deep Learning”  In _arXiv_ , 2023  DOI: 10.48550/arXiv.2302.09071