# Plasmonics in the visible domain for a one-dimensional truncated photonic
crystal terminated by graphene: sensing beyond Dirac point’s approximation

A. Alidoust Ghatar, D. Jahani<EMAIL_ADDRESS>W. Fritzche and F.
Garwe

###### Abstract

_Visible surface plasmon resonances (SPRs) could be excited by TE wave
polarization in one-dimensional photonic crystals (PCs) coated by a graphene
layer under the Kretschmann configuration. In this work, the plasmonic Bloch
wave properties beyond Dirac points in a one-dimensional graphene-based
photonic sensing structure have been numerically studied. We demonstrate that
emergent plasmonic dips in the reflectance spectra of the suggested photonic
device in the visible region exhibit tunable characteristics upon modulation
of the chemical potential and the hopping parameter. The sensitivity of the
sensor in the visible domain has been numerically evaluated and also was
compared with those considering the surface plasmon resonances in the
terahertz regime._

_Materials and Energy Research Center, Tehran, Iran_

_Leibniz-Institut for Photonische Technologien, Jena, Germany._

_Keywords_ : _Surface states, Graphene; Plasmonics; Visible domain; Photonic
crystal_

##  1  Introduction

_Plasmonics is a rapidly growing field that has attracted intensive attention
in physics, biology, chemistry, medical diagnostics, organic chemical
detection, etc. due to highly extraordinary sensitive specifications of
surface states [1,2]. Surface plasmons (SPs) are propagating modes which could
be created when a polarized light strikes an electrically conducting surface
at the interface between two media, a conductor (mostly metal) and a
dielectric [3]. The necessary condition for occurring this resonance is
matching the phase of the incoming light and the natural frequency of the
material [4,5]. At this resonance condition, a sharp dip at a particular
wavelength appears in the reflection spectrum and the intensity of the
reflected light significantly shows to be reduced. Then, an unknown analyte
could be detected by observing the variation of this dip in an optical sensing
device [5,6,7]. We note that dip variations are extremely sensitive to changes
in the refractive index of the dielectric medium and this property could be
used as a sensing method especially as an optical biosensor [8,9]. The first
sensing application of surface plasmonic resonance (SPRs) technique was
reported by Liedberg et al. in 1983 [6]. Since then, SPRs have received much
attention due to their extremely sensitive performance in optics. In order to
evaluate the quality and the performance of an optical biosensor relevant
parameters must be considered, for instance, the sensitivity, the depth and
the width of the dip, size, the limit of detection and the cost [10,11,12].
Here, it should be noted that the earliest SPR sensing devices were bulky in
size and mechanically unstable making them not suitable for the remote sensing
applications [4,5,6,7]. To overcome mentioned limitations, manufacturing low-
cost optical structures with high confinement characteristics for surface
states propagation known as Photonic crystals (PCs) is of much interest
[6,7,8,9,10,11,12,13]._

_PCs are artificial optical structure with a periodic change in their
refractive index which could create a forbidden frequency region due to
various reflections and refractions that occur upon lights propagations [14].
Moreover, by considering plasmonic properties of surface states in a PC,
designing optical structures in nanometer size at or less than the input
wavelength will be possible. Therefore, scaling down to nanometer size for the
PCs is possible by considering the sensing properties of the visible light._

_Furthermore, the choice of material for plasmonic waves have highly affected
on sensing performance and better accuracy of SPR sensors. Therefore, a
metallic component that carries large amounts of free electrons are required
in these devices. In fact, gold and silver are mostly used metals in SPR
sensing applications since gold is chemically stable in many environments and
shows a large resonance peak shift. However, this peak resonance band becomes
broader due to the large coefficient absorption which correspondingly reduces
the accuracy of the analyte detection. On the contrary, silver with
intrinsically higher sensitivities than gold shows lower loss and sharper
resonance peaks compared to the other plasmonic materials. Unfortunately,
silver shows rather quickly lost SPR properties with the contact of the
ambient air because it is not chemically stable and could be oxidized easily
[15,16,17,18]. Hence, it could be coated with an utra-thin graphene layer
which could also absorb gas molecules on its surface [19,20]. Regardless of
the use of noble metals for plasmonic materials, these conventional materials
have their limitations. In fact, their optical properties are hardly tunable,
due to large Ohmic losses cause poor propagating distances for plasmon
oscillations. Therefore, we need a structure or materials that compensate for
these limitations. Now, because of the outstanding properties of graphene
which yields a negative imaginary part in optical conductivity in different
frequency ranges from terahertz to visible light, it could be considered as an
excellent alternative to the conventional plasmonic structures and noble
metals [21,22,23,24,25]._

_Graphene is a gapless flat monolayer of graphite with very high electron
mobility whose transport characteristics and optical conductivity
$(\sigma_{g})$ can be tuned by either electrostatic or magneto-static gating
or chemical doping [26,??,27]. In addition, graphene has also been
demonstrated to be a good candidate in optoelectronics with potential
applications in photodetection, optics and plasmonics [28]. Moreover,
advantages of graphene-based plasmonics are the presence of high carrier
concentrations, ultra-relativistic speed for Dirac fermions, high confinement
for surface states with longer propagation distance, flexible feature,
adjustable electro-optical properties, nonlinear optics and large high-
intensity electromagnetic field [29,30,31,32]. We should note that most of the
interesting properties of graphene reflects back from its linear energy
dispersion near two inequivalent points in the momentum space named as Dirac
points. However, beyond Dirac points graphene’s energy relation exhibits
nonlinear properties which must be taken into account for obtaining its
surface conductivity in the visible range [33]. Therefore, in the visible
region, we need to consider optical excitations beyond Dirac points in order
to investigate the surface states in graphene in the nonlinear regime._

_We note that in the previous works infrared light has been used while the
wavelength of this region is relatively large, to be used for designing on-
chip nano-scale optical devices [20,34,35]. In this paper, we investigate the
sensing properties of SPs in the visible region allowing us to down the size
of photonic crystal layers in a nanometer-scale that is comparable to the
thickness of the graphene layer. To best of our knowledge, visible SPs
corresponding to the optical excitations beyond Dirac points in graphene’s
spectrum have been reported up to this stage._

_This paper is organized into the following sections: The model is introduced
in section 2. Following this section, we express the optical conductivity of
the graphene sheet in the visible region, and by the use of the transfer
matrix for the propagation of light among photonic crystals. Then numerical
calculations are addressed in section 3. In the end, the conclusion is
introduced in section 4._

##  2 Structure and Theoretical Modeling

_In order to compute the plasmon dispersion in our photonic structure, we
first employ the transfer matrix method. It is assumed that light is polarized
in the y-direction and propagating in the z-direction. The electromagnetic
fields of the transverse electric (TE) polarized wave can be written_

$\displaystyle H_{1y}=(a_{1}e^{ik_{1z}z}+b_{1}e^{ik_{1z}z})e^{ik_{1x}x}\hskip
28.45274ptz<0,$ (1) $\displaystyle
H_{2y}=(a_{2}e^{ik_{2z}z}+b_{2}e^{ik_{2z}z})e^{ik_{2x}x}\hskip 28.45274ptz>0.$
(2)

_Here, $a_{i}$ and $b_{i}(i=1,2)$ are the field coefficients, $k_{ix}(k_{iz})$
is the $x(z)$ component of the wave-vector
$k_{i}=\sqrt{\varepsilon_{i}}\omega/c$ in which $\omega$ represents the
angular frequency and $c$ is the light speed in the vacuum. In the parenthesis
on the right side, the first (second) term stands for propagation waves along
z-direction. We, then, relate the electric $E_{1}$ and $E_{2}$ and magnetic
fields at the interface by using the following boundary conditions._

$\displaystyle n\times(E_{2}-E_{1})|_{z=0}=0,$ (3) $\displaystyle
n\times(H_{2}-H_{1})|_{z=0}=J,$ (4)

_where $n$ is the unit of the surface normal and $J$ is the surface current
density of the graphene. Subsequently, by applying the above boundary
conditions at $z=0$ [36],_

$\displaystyle\frac{k_{1z}}{\varepsilon_{1}}(a_{1}-b_{1})-\frac{k_{2z}}{\varepsilon_{2}}(a_{2}-b_{2})=0,$
(5) $\displaystyle(a_{1}+b_{1})-(a_{2}+b_{2})=J_{x}.$ (6)

_It should be pointed out that $J$ can be obtained from Ohm’s law, that is,_

$\displaystyle J_{x}=\sigma E_{x}|_{z=0}=\frac{\sigma
k_{2z}}{\varepsilon_{0}\varepsilon_{2}\omega}(a_{2}-b_{{}_{2}}),$ (7)

_where $\varepsilon_{0}$ is the vacuum permittivity and $\sigma$ is the
optical conductivity of graphene in the visible region that follow below. It
this important to mention that in the below equation the well-known Kubo
formula is used, both inter-band and intra-band transitions are considered.
The direction of the external field and optical conductivity is assumed the
same [37]._

$\begin{split}\sigma(\omega)&=\\\ &\frac{e^{2}t^{2}a^{2}\pi}{12\hbar^{2}\omega
A_{c}}\frac{\hbar\omega}{t^{2}\pi^{2}}\frac{1}{\sqrt{(1+\frac{\hbar\omega}{2t})^{2}-[(\frac{\hbar\omega}{2t})^{2}-1]^{2}/4}}\times\\\
&\left[18-\frac{(\hbar\omega)^{2}}{t^{2}}\right]\left[\tanh(\frac{\beta(\hbar\omega+2\mu)}{4})+\tanh(\frac{\beta(\hbar\omega-2\mu)}{4})\right]\\\
&K\left(\frac{\frac{4\hbar\omega}{2t}}{(1+\frac{\hbar\omega}{2t})^{2}-\frac{[(\frac{\hbar\omega}{2t})^{2}-1]^{2}}{4}}\right)+iIm(\sigma(\omega))\end{split}.$
(8)

_with_

$\begin{split}Im(\sigma(\omega))&=\\\
&\frac{e^{2}}{\hbar^{2}\pi\omega}\left(\mu-\frac{2}{9}\frac{\mu^{3}}{t^{2}}\right)-\frac{e^{2}}{4\pi\hbar}\log\frac{|\hbar\omega+2\mu|}{|\hbar\omega-2\mu|}\\\
&-\frac{e^{2}\hbar\omega^{2}}{144\hbar\pi
t^{2}}\log\frac{|\hbar\omega+2\mu|}{|\hbar\omega-2\mu|}\end{split}.$ (9)

_Here $\mu$, $\beta=1/(K_{B}T)$ and $t$ are the chemical potential, the
Boltzmann constant and hopping parameter which is the energy for an electron
to jump into its nearest neighbors, is in the order of $3\ eV$, respectively.
The lattice constant and the area of the unit cell are $a=1.42\ {\AA}$ and
$A_{C}=(3\sqrt{3}a^{2}/2)$ , respectively. The first kind of elliptic integral
$K(r)$ is:_

$K(m)=\int_{0}^{1}dr\left[\left(1-r^{2}\right)\left(1-mr^{2}\right)\right]^{-1}.$
(10)

_with
$m=\frac{\frac{4\hbar\omega}{2t}}{(1+\frac{\hbar\omega}{2t})^{2}-\frac{[(\frac{\hbar\omega}{2t})^{2}-1]^{2}}{4}}$
. The coefficients $a_{1}$ and $b_{1}$ can be related to $a_{2}$ and $b_{2}$
by the $2\times 2$ transmission matrix $D_{1\rightarrow 2}$ combining
equations (1-5)-(1-6) as follows:_

$\binom{a_{1}}{b_{1}}=D_{1\rightarrow 2}\binom{a_{2}}{b_{2}},$ (11)

_where_

$D_{1\rightarrow
2}=\frac{1}{2}\begin{pmatrix}1+\eta_{s}+\xi_{s}&1-\eta_{s}+\xi_{s}\\\
1-\eta_{s}-\xi_{s}&1+\eta_{s}-\xi_{s}\end{pmatrix},$ (12)

_with the parameters $\eta_{s}$ and $\xi_{s}$ provided by:_

$\eta_{s}=\frac{k_{2z}}{k_{1z}},\hskip
28.45274pt\xi_{s}=\frac{\sigma\mu_{0}\omega}{k_{1z}}.$ (13)

_Now, we can examine light propagation in a homogeneous medium. The electric
or magnetic field at $z+\Delta z$ can be related to those at the $z$ position
by a $2\times 2$ propagation matrix_

$P(\Delta z)=\begin{pmatrix}e^{-ik_{z}\Delta z}&0\\\ 0&e^{ik_{z}\Delta
z}\end{pmatrix}.$ (14)

_For a stack of $N$ graphene layers, the transfer matrix can be obtained by
transmission matrices across different interfaces and propagation matrices in
different homogeneous dielectric media. The two sets of field coefficients are
then related by a $2\times 2$ transfer matrix $M$, namely_

$\binom{a_{1}}{b_{1}}=M\binom{a_{N+1}}{b_{N+1}}.$ (15)

_with_

$M=D_{1\rightarrow 2}P(d_{1,2})D_{2\rightarrow
3}P(d_{2,3})....P(d_{N-1,N})D_{N\rightarrow N+1}.$ (16)

_Finally, the transmittance $(T)$ and reflectance $(R)$ can be calculated by
using the matrix elements $M_{11}$ and $M_{12}$ as follows :_

$R=\Big{|}\frac{M_{21}}{M_{11}}\Big{|}^{2},\hskip
28.45274ptT=\Big{|}\frac{1}{M_{11}}\Big{|}^{2}.$ (17)

_Now, using the above expression we are ready to calculate the transmission
spectrum of our 1D photonic crystal. In the next section, therefore, the
numerical results are expressed in more detail._

##  3 Result and discussion

_For our numerical approach, in this section, based on the transmission matrix
method, a trunked photonic crystal coated with single-layer graphene with
Kretschmann configuration [?] is assumed which is demonstrated schematically
in Figure 1. It consists of two dielectric layers with constants
$\varepsilon_{A}$ and $\varepsilon_{B}$ and thicknesses of $d_{A}$ and $d_{B}$
arranged in the periodic pattern, respectively. As it is seen, the graphene
layer is attached to the photonic crystal contact with a sample whose
refractive index changes, and a prism with refractive index $n_{p}=1.77$ at
the entrance side of the proposed sensing device. The surface plasmonic wave
excitations can be observed by a dip appearing in the reflection spectra of
the photonic structure. We choose $Si$ for the $A$, $SiO_{2}$ for the material
$B$ and the period of the crystal is $N=9$. The region of the transmission
spectra which is considered to be in the visible zone $(1-2\ eV)$ requires
choosing proper geometrical parameters for thicknesses of the layers.
Therefore, the thickness of A and B layers are chosen to be $d_{A}=d_{B}=200\
nm$. We note that the temperature in all of our calculations is assumed to be
$T=300\ K$ which is the well-known room temperature._

_Also, the dielectric constants of layers are $12.74$ and $2.16$ for $Si$ and
$SiO2$, respectively. Nore that, the imaginary parts of the refractive indices
of dielectric layers are considered to be zero in this work. Now, for exciting
the surface states we consider visible light to be incident at greater angles
than $\theta_{c}=sin^{-1}(n_{s}/n_{p})$ for which reflection is perfect.
Therefore, one can detect plasmonic surface states have appeared as a dip in
this reflection spectrum near the band-gap-edge in our numerical
calculations._

_As our main aim in this section, we will investigate the dip properties in
the reflection spectra of the introduced truncated photonic crystal by using
(2-16). At first, we present the reflection spectra of the proposed structure
without a graphene layer based on the parameters given above, as shown in Fig.
2. A small dip (blue solid line) appears in the spectrum at $\lambda=637.6\
nm$ wavelength and $\theta=34.4$ degree. Interestingly, by introducing a
graphene layer into the supposed structure with the relatively low chemical
potential of about $\mu=0.1\ eV$ and the hopping parameter $t=3.1\ eV$ no dip
is appeared in the dispersion spectrum (Green solid line). The situation
changes if one goes beyond Dirac points by considering larger values for the
chemical potential for graphene so that a deep dip (dotted red line) in
reflectance spectra of the proposed PC appears in $\lambda=635.6\ nm$ ,
$\mu=1.145\ eV$ and $t=3.1\ eV$ as shown in Fig. 2._

_To proceed, as a matter of more comparison, in Fig. 3 we show both the
reflection and transmission profiles of the proposed sensing structure
terminated by single-layer graphene. As it is evident from this figure the
light is completely absorbed and we have no transmitted light in the
corresponding spectrum which indicates well the emergence of a plasmonic
surface state at the interface between graphene and the sample._

_Moreover, in Fig. 4, the obtained resonant dip is illustrated for two
situations, one for the modulation of the incident angles in constant
wavelength $\lambda=635/nm$ and the other(inset figure) for a specific
interval for the wavelength in constant incident angle $\theta=34.44$ degree
which the plasmonic wave is centered around $635\ nm$. It is clear that one
gets a more narrow dip which indicates proper physical parameters selectivity
and suitable for serving as a gas sensor in the visible region. Then, to
evaluate the angular and wavelength sensitivity (S) of the sensor, we need to
have the slope of these diagrams $S_{\theta}=\Delta\theta/\Delta n$ and
$S_{\lambda}=\Delta\lambda/\Delta n\ (nm)$ which indicate the angular shift
and wavelength peak shift of surface plasmons relative to the refractive index
changes, respectively. In Fig. 5, the reflectance of the optical sensing
layered device is plotted as a function of the wavelength and also the angle
of incidence. Then the angular sensitivity of the sensor is, therefore,
computed at $\lambda=635.6\ nm$ with the refractive indices ranging from 1 to
0.01 in an interval of 0.002 in fig 5(.a) and (c). Also, the wavelength
sensitivity is calculated at $\theta=34.44$ degree with refractive indices
ranging from 1 to 0.001 in an interval of 0.0002 in figure 5(b) and (d).
Moreover, it is evident that the variations of the resonant angle due to
variations of refractive indices show a linear behavior and, therefore, the
sensitivity is equal to $S_{\theta}=35.53\ \circ/RIU$ (Refractive Units
Index). Similarly, the spectra in Fig. 5(b) and (d) demonstrate that the dip
wavelength sensitivity is $S_{\lambda}=1402\ nm/RIU$ and the variation of the
resonance wavelength is almost linear and red-shifted which indicates both
sensitivities are higher than the results reported previously [38,39]._

_In the following, the effect of increasing the chemical potential and the
hopping energy on the refractive spectra and the sensitivity of the device is
investigated. Hence, we investigate the dip modulation by considering the
hopping energies of graphene, $t=2.5\ eV$ and $t=3.1\ eV$ for different
refractive values. As is shown in Fig. 6, by increasing the hopping parameters
which means that the gas molecules in the air are absorbed on graphene’s
surface shifts the refractive spectra without affecting the angular and
wavelength sensitivities meaning that $S_{\theta}=35.42\ \circ/RIU$ and
$S_{\lambda}=1399\ nm/RIU$ remain almost constant. Therefore, our sensor is
sensitive to the different kinds of gas molecules absorption on graphene’s
surface without any change in sensing performance which is of most interest in
sensing applications._

_In the end, since the chemical potential, $\mu$, significantly could affect
the charge carrier’s concentration of graphene, its influence on SPRs can also
be analyzed. In Fig. 7 (a),(b) the refractive spectra is shown to be red-
shifted without affecting much the sensitivities as also reported for the
infrared region in [34]._

##  4 Conclusion

_In this work, we addressed optical sensing properties of SPRs for a 1D
truncated PC coated by graphene in the visible region of the light’s
frequency. In particular, we have numerically calculated the spectra for the
suggested structure in order to analyze the sensing properties of an emergent
dip in reflectance due to the absorption of SPRs in graphene beyond its usual
linear dispersion. As it was discussed these relatively high domains of the
frequency for photons allows the fabricating of nanoscale layered structure
which could be employed in on-chip sensing technology. Then, the refractive
spectra and sensitivity of the proposed sensing device has been presented by
the transfer matrix method._ _The most significant result of this work is the
emergence of highly narrow surface plasmons in the visible regions of the
spectrum for graphene beyond its usual Dirac points approximation. Moreover,
the effect of changing the refractive index of the analyte on the plasmonic
dip in the reflectance spectra of the optical sensor was investigated. Our
results indicated that $S_{\theta}=35.53\ \circ/RIU$ and $S_{\lambda}=1402\
nm/RIU$ for the angular and wavelength sensitivity, respectively meaning the
low-loss and narrow width dip comparable to the previous works [38,39]. The
role of hopping energy and the chemical potential also analyzed which shows
that the plasmonic spectra are shifted SPRs propagations. In fact, with regard
to the prospect of graphene, PCs and plasmonic in the future, the proposed SPR
sensor may be considered an ideal candidate for sensing purposes in the
visible domain._

## References

  * [1] Y. B. Zheng, B. Kiraly, P. S.Weiss, and T. J. Huang, (2012). Molecular plasmonics for biology and nanomedicine. Nanomedicine, 7(5), 751-770.
  * [2] Y. Saylan, S. Akgonullu, and A. Denizli, (2020). Plasmonic Sensors for Monitoring Biological and Chemical Threat Agents. Biosensors, 10(10), 142.
  * [3] I. R. Hooper, and J. R. Sambles, (2002). Dispersion of surface plasmon polaritons on short-pitch metal gratings. Physical Review B, 65(16), 165432.
  * [4] B. D. Gupta, and R. K. Verma, (2009). Surface plasmon resonance-based fiber optic sensors: principle, probe designs, and some applications. Journal of sensors.
  * [5] J. Homola, S. S. Yee, and G. Gauglitz, (1999). Surface plasmon resonance sensors. Sensors and actuators B: Chemical, 54(1-2), 3-15.
  * [6] B. Liedberg, C. Nylander, and I. Lunstrom, (1983). Surface plasmon resonance for gas detection and biosensing. Sensors and actuators, 4, 299-304.
  * [7] X. Yang, Y. Lu, B. Liu, and J. Yao, (2017). Analysis of graphene-based photonic crystal fiber sensor using birefringence and surface plasmon resonance. Plasmonics, 12(2), 489-496.
  * [8] R. S. Zheng, Y. H. Lu, Z. G. Xie, J. Tao, K. Q. Lin, and H. Ming, (2008, November). Surface plasmon resonance sensors based on polymer optical fiber. In 2008 1st Asia-Pacific Optical Fiber Sensors Conference (pp. 1-4). IEEE.
  * [9] S. biosensors for use in the diagnostics of malignant and infectious diseases. Laser Physics Letters, 15(6), 065602.Firdous, S. Anwar, and R. Rafya, (2018). Development of surface plasmon resonance (SPR).
  * [10] K. V. Sreekanth, S. Zeng, K. T. Yong, and T. Yu, (2013). Sensitivity enhanced biosensor using graphene-based one-dimensional photonic crystal. Sensors and Actuators B: Chemical, 182, 424-428.
  * [11] X. Fan, I. M. White, S. I. Shopova, H. Zhu, J. D. Suter, and Y. Sun, (2008). Sensitive optical biosensors for unlabeled targets: A review. analytica chimica acta, 620(1-2), 8-26.
  * [12] A. Panda, P. D. Pukhrambam, and G. Keiser, (2020). Performance analysis of graphene-based surface plasmon resonance biosensor for blood glucose and gas detection. Applied Physics A, 126(3), 1-12.
  * [13] M. Hasan, S. Akter, A. A. Rifat, S. Rana, and S. Ali, (2017, March). A highly sensitive gold-coated photonic crystal fiber biosensor based on surface plasmon resonance. In Photonics (Vol. 4, No. 1, p. 18). Multidisciplinary Digital Publishing Institute.
  * [14] E. Yablonovitch, (1987). Inhibited spontaneous emission in solid-state physics and electronics. Physical review letters, 58(20), 2059.
  * [15] M. Kanso, S. Cuenot, and G. Louarn, (2007). Roughness effect on the SPR measurements for an optical fibre configuration: Experimental and numerical approaches. Journal of Optics A: Pure and Applied Optics, 9(7), 586.
  * [16] J. N. Dash, and R. Jha, (2014). SPR biosensor based on polymer PCF coated with conducting metal oxide. IEEE Photonics Technology Letters, 26(6), 595-598.
  * [17] E. K. Akowuah, T. Gorman, H. Ademgil, S. Haxha, G. K. Robinson, and J. V. Oliver, (2012). Numerical analysis of a photonic crystal fiber for biosensing applications. IEEE Journal of Quantum Electronics, 48(11), 1403-1410.
  * [18] Y. Lu, C. J. Hao, B. Q. Wu, M. Musideke, L. C. Duan, W. Q. Wen, and J. Q. Yao, (2013). Surface plasmon resonance sensor based on polymer photonic crystal fibers with metal nanolayers. Sensors, 13(1), 956-965.
  * [19] W. Qin, S. Li, Y. Yao, X. Xin, and J. Xue, (2014). Analyte-filled core self-calibration microstructured optical fiber based plasmonic sensor for detecting high refractive index aqueous analyte. Optics and Lasers in Engineering, 58, 1-8.
  * [20] L. Wu, H. S. Chu, W. S. Koh, and E. P. Li, (2010). Highly sensitive graphene biosensors based on surface plasmon resonance. Optics express, 18(14), 14395-14400.
  * [21] A. N. Grigorenko, M. Polini, and K. S. Novoselov, (2012). Graphene plasmonics. Nature photonics, 6(11), 749-758.
  * [22] F. H. L. MinovKoppensich, D. E. Chang, S. Thongrattanasiri, and F. G. de Abajo, (2011). Graphene plasmonics: A platform for strong light-matter interactions. Optics and Photonics News, 22(12), 36-36.
  * [23] M. N. Gjerding, M. Pandey, and K. S. Thygesen, (2017). Band structure engineered layered metals for low-loss plasmonics. Nature communications, 8(1), 1-8.
  * [24] Q. Bao, and K. P. Loh, (2012). Graphene photonics, plasmonics, and broadband optoelectronic devices. ACS nano, 6(5), 3677-3694.
  * [25] P. R. West, S. Ishii, G. V. Naik, N. K. Emani, V. M. Shalaev, and , A. Boltasseva (2010). Searching for better plasmonic materials. Laser and Photonics Reviews, 4(6), 795-808.
  * [26] S. Zhang, Z. Li, and F. Xing, (2020). Review of polarization optical devices based on graphene materials. International journal of molecular sciences, 21(5), 1608.
  * [27] O. L. Berman, V. S. Boyko, R. Y. Kezerashvili, A. A. Kolesnikov, and Y. E. Lozovik, (2010). Graphene-based photonic crystal. Physics Letters A, 374(47), 4784-4786.
  * [28] O. L. Berman, and R. Y. Kezerashvili, (2011). Graphene-based one-dimensional photonic crystal. Journal of Physics: Condensed Matter, 24(1), 015305.
  * [29] P. Avouris, and M. Freitag, (2013). Graphene photonics, plasmonics, and optoelectronics. IEEE Journal of selected topics in quantum electronics, 20(1), 72-83.
  * [30] T. Low, and P. Avouris, (2014). Graphene plasmonics for terahertz to mid-infrared applications. ACS nano, 8(2), 1086-1101.
  * [31] Z. Zhen, and H. Zhu, (2018). Structure and properties of graphene. In Graphene (pp. 1-12). Academic Press.
  * [32] Z. Xu, (2018). Fundamental properties of graphene. In Graphene (pp. 73-102). Academic Press.
  * [33] X. Luo, T. Qiu, W. Lu, and Z. Ni, (2013). Plasmons in graphene: recent progress and applications. Materials Science and Engineering: R: Reports, 74(11), 351-376.
  * [34] A. C. Neto, F. Guinea, N. M. Peres, K. S. Novoselov, and A. K. Geim, (2009). The electronic properties of graphene. Reviews of modern physics, 81(1), 109.
  * [35] A. K. Sharma, and B. Kaur, (2018). Analyzing the effect of graphene’s chemical potential on the performance of a plasmonic sensor in infrared. Solid State Communications, 275, 58-62.
  * [36] J. Tang, Y. Ye, J. Xu, Z. Zheng, X. Jin, L. Jiang, … and Y. Xiang, (2020). High-sensitivity terahertz refractive index sensor in a multilayered structure with graphene. Nanomaterials, 10(3), 500.
  * [37] T. Zhan, X. Shi, Y. Dai, X. Liu, and J. Zi, (2013). Transfer matrix method for optics in graphene layers. Journal of Physics: Condensed Matter, 25(21), 215301.
  * [38] T. Stauber, N. M. R. Peres, and A. K. Geim, (2008). Optical conductivity of graphene in the visible region of the spectrum. Physical Review B, 78(8), 085432.
  * [39] E. Kretschmann, and H. Raether, (1968). Radiative decay of non-radiative surface plasmons excited by light. Z. Naturforsch. a, 23(12), 2135-2136.
  * [40] A. V. Baryshev, and A. M. Merzlikin, (2014). Plasmonic photonic-crystal slabs: visualization of the Bloch surface wave resonance for an ultrasensitive, robust and reusable optical biosensor. Crystals, 4(4), 498-508.
  * [41] X. J. Zou, G. G. Zheng, and Y. Y. Chen, (2018). Confinement of Bloch surface waves in a graphene-based one-dimensional photonic crystal and sensing applications. Chinese Physics B, 27(5), 054102.

Figure 1:  Schematic diagram of the designed structure in the presence of a
prism with $n=1.77$ and the period $N=9$ coated by a single-layer graphene.
Figure 2:  Reflection spectrum for the SPRs in the visible domain of the
suggested sensing structure as a function of the incident angle. The blue
solid line represents the graphene-less structure for $\lambda=637.6\ nm$ and
$T=300\ K$. The green line represents the structure with a graphene layer for
$\lambda=637.6\ nm$ , $\mu=0.1\ eV$ and $t=3.1\ eV$. The red dotted line is
for the graphene-based structure for $\lambda=635.6\ nm$ $\mu=1.145\ eV$ and
$t=3.1\ eV$. Figure 3:  Reflection and the transmission spectra for detecting
SPR dip in the case of $\mu=1.145\ eV$ and $t=3.2\ eV$ for $\lambda=635.6\
nm$. As it is clear, no transmitted light is obverted in the transmission
spectra of the SPR photonic sensing structure while a plasmonic dip is emerged
at the wavelength $\lambda=635.6\ nm$. Figure 4:  Results of numerical
calculations for detecting a Plasmonic dip in the reflectance spectrum of the
truncated PC as a function of the incident angles at constant wavelength
around $635\ nm$, and the inset showes the reflectance spectrum as a function
of the wavelength centered around $635\ nm$ in constant incident angle
$\theta=34.44$ degree. Figure 5:  (a) and (c): the angle variations of
reflectance as a function of the wavelength under for different refractive
indices with $\mu=1.145\ eV$ and $t=3.1\ eV$. (b) and (d): Wavelength’s
variation for the reflectance as a function of the incident angle for
different values for refractive indices for $\theta=34.44^{o}$ and $t=3.1\ eV$
keeping all the parameters the same as that in Fig. 2. Figure 6:  (a)
Variation of the plasmonic dip for surface states as a function of the
incident angle and refractive indices for $\lambda=635.6nm$ and, (b) Variation
of the plasmonic dip as a function of the wavelength for the different values
of refractive indices with $\theta=34.44^{o}$, the hopping energy $t=2.5\ eV$
and $3.1\ eV$. Here the chemical potential and wavelength are set to be
$\mu=1.145$ and $\lambda=635.6nm$, respectively. Figure 7:  (a) Variation of
the plasmonic dip for the incident angle as a function of the refractive
indices for $\lambda=635.6\ nm$ and, (b) wavelength’s variation of the SPRs as
a function of the refractive index for $\theta=34.44^{o}$, $\mu=1.125$ and
$1.145\ eV$ and a constant constant hopping parameter $t=3.1\ eV$.


hazard pulse (Fig. 21) seems to vary much more substantially across ISI
conditions than across co-agent types. Re-entry timing is very consistent
relative to exit timing, which especially for the more difficult conditions
can be quite variable. There were a few statistically significant differences
found in median exit and re-entry times across co-agent pairings ($N>140$ for
each group), but the differences in the medians compared to the spread of the
data suggests that these differences are not likely to be practically
significant.

Figure 21: Goal region exit and re-entry timing for each condition. Individual
data points are individual pulses within trials, and all timing information is
synchronised around the falling edge of the hazard pulse. Time flows from
bottom to top on this chart; the participant exits the goal region in advance
of the hazard (below the yellow region), and re-enters the goal region after
the hazard resolves (above the yellow region). Due to the spread of the data
compared to the differences in group medians, statistical significance is not
expected to correspond to practical significance.

Human-Agent Interaction: The length of time between the co-agent’s signal and
the participant’s exit from the goal region is plotted in Fig. 22. A negative
value on this chart indicates that the participant left the goal region before
being cued by the co-agent. Here, we see that the participant exhibits clear
behavioural differences when interacting with each co-agent ($p<<0.05$ in all
comparisons). When paired with a co-agent using a tile-coded trace
representation, the participant nearly always waits for the co-agent signal
before leaving the goal region (data above the dotted line). When paired with
a co-agent using a bit-cascade representation, the participant is much more
likely to exit the goal region before the co-agent gives a signal. Expanding
these box-plots out over trial time in Fig. 23, we can see the participant’s
response time relative to the co-agent signal change over the course of a
trial for each of the conditions. In the fixed ISI condition, when paired with
the tile-coded trace co-agent, the participant seems to move after the co-
agent’s cue as early as the second or third pulse of a trial. Under the same
conditions, when paired with a bit-cascade co-agent, the participant relies
entirely on their own internal timing. For the more difficult conditions, the
participant eventually moves after the cue of either co-agent, but aligns
their movements with the tile-coded trace co-agent’s cue more readily than
with the bit-cascade co-agent’s cue. While it is tempting to interpret this
feature of the data as the participant relying on the tile-coded trace co-
agent’s cue more than the bit-cascade co-agent’s cue, there is insufficient
evidence from these charts alone to conclude how the participant is using
either signal, as we see in Sec. 8.5.

Figure 22: Time interval between signal from co-agent and goal region exit.
Negative intervals indicate the participant leaving the goal region in advance
of the co-agent’s signal. Figure 23: Time interval between signal from co-
agent and goal region exit, shown as a trajectory over the length of trials.
Data are shown as the mean (solid line) and 95% confidence interval (shaded
region) of the signal-to-exit interval for each pulse. Negative data indicate
the participant leaving the goal region in advance of the co-agent’s signal.
Due to the randomization of the starting ISI (refer to Table 2) and fixed
trial duration, some trials with shorter ISIs presented more pulses than
others. This led to the occurrence of one or two trials with high pulse count
(>14), resulting in the large confidence intervals at the ends of these plots.

To gain an intuitive sense of how the participant engages with the co-agent,
we present two specific trials in detail: a typical “simple” trial (Fig. 24,
which corresponds to the fixed ISI condition, paired with the tile-coded trace
co-agent of session 3), and a typical “challenging” trial (Fig. 25, which
corresponds to the random ISI condition, paired with the bit-cascade co-agent
of session 3). Beginning with the simple trial, we see the co-agent providing
a useful (though inconsistent) signal beginning on the second hazard pulse,
and reliably thereafter. The amount of wasted steps leading up to the hazard
pulse diminishes to a narrow margin as the trial progresses. In the
challenging trial we see the co-agent give its first useful signal only by the
time of the fourth pulse. Until this time, the participant has been using
their internal timing to determine when to leave the goal region in advance of
the signal. The co-agent is then unable to give reliable signals for the next
few pulses (pulses 5, 6, 8, 9, and 10), and the participant is hit by the
hazard several times. By the 11th pulse, the participant resumes reliance on
their internal timing to leave the goal region in advance of the co-agent
signal, wasting many steps in order to avoid the hazard.

Figure 24: A typical simple trial (in this case fixed pulse interval condition, with tile-coded trace co-agent, from session 3).“Wasted steps” are counted any time the participant is outside the goal region while the hazard pulse is inactive. “Minimum Useful Signal” corresponds to the signal that gives the participant exactly 0.89s lead-time before the hazard begins, calculated according to the participant’s exit velocity. Figure 25: A typical challenging trial (in this case random pulse interval condition, with bit-cascade co-agent from session 3).“Wasted steps” are counted any time the participant is outside the goal region while the hazard pulse is inactive. “Minimum Useful Signal” corresponds to the signal that gives the participant exactly 0.89s lead-time before the hazard begins, calculated according to the participant’s exit velocity. |  |  Points Cached |  Steps Hit by Hazard |  Heat Gain |  Re-Entry Timing |  Exit Timing |  Signal-to- Exit Time
---|---|---|---|---|---|---|---
A Priori Tests
Fixed | $\chi^{2}(2)$ |  5.2 |  14.0 |  1.4 |  0.9 |  5.5 |  -
| $p$ |  0.0755 |  0.0009 |  0.4966 |  0.6364 |  0.0632 |  -
Drifting | $\chi^{2}(2)$ |  1.4 |  1.4 |  7.2 |  0.5 |  11.5 |  -
| $p$ |  0.4895 |  0.4966 |  0.0273 |  0.7956 |  0.0032 |  -
Random | $\chi^{2}(2)$ |  0.7 |  11.4 |  13.4 |  11.9 |  2.1 |  -
| $p$ |  0.7165 |  0.0033 |  0.0012 |  0.0026 |  0.3540 |  -
Post Hoc Tests (p-values)
Fixed | NvsB |  0.1415 |  0.0432 |  0.9594 |  0.4518 |  0.0398 |  -
| NvsT |  0.1724 |  0.0151 |  0.5550 |  0.4518 |  0.1632 |  -
| BvsT |  0.7344 |  0.1763 |  0.4930 |  0.4633 |  0.1108 |  0.0000
Drifting | NvsB |  0.6831 |  0.4257 |  0.3642 |  0.6026 |  0.1247 |  -
| NvsT |  0.6831 |  0.4881 |  0.3642 |  0.6026 |  0.0018 |  -
| BvsT |  0.5507 |  0.6465 |  0.0206 |  0.6026 |  0.0227 |  0.0000
Random | NvsB |  0.7990 |  0.2026 |  0.0593 |  0.0037 |  0.3397 |  -
| NvsT |  0.7990 |  0.0278 |  0.0151 |  0.0037 |  0.3397 |  -
| BvsT |  0.7990 |  0.0278 |  0.0329 |  0.4455 |  0.2159 |  0.0000
Table 3: Results from statistical analysis of comparisons. Significant results
($\alpha=0.05$) are indicated in bold text. Friedman’s Tests
($\chi^{2}_{critical}(2)=6.20$) followed by Wilcoxon Signed-Rank tests were
conducted for steps hit by hazard, points cached, and heat gain. Kruskal-
Wallis tests ($\chi^{2}_{critical}(2)=5.99$) followed by Mann-Whitney U tests
were conducted for exit and re-entry timing. Mann-Whitney U tests were
conducted for signal-to-exit time.

Co-agent Characteristics: We are able to examine co-agent learning directly
because the co-agent’s learning of the task does not depend in any way on the
participant’s actions. Figure 26 shows the mean interval between the co-agent
signal and the onset of the hazard pulse, for each pulse over the length of
the trials. Recall that this can be interpreted as “how long before the onset
of the pulse did the co-agent’s prediction of the pulse rise above the
threshold for signalling?”. When the co-agent’s cue is less than 0.89s before
the hazard (above the dashed line), the signal doesn’t give the participant
enough time to react given how long it takes to leave the hazard region. In
the simplest prediction task with fixed ISI, the bit cascade co-agent is
unable to reliably give a useful signal (below the dashed line) until after
about the sixth or seventh pulse while the tile-coded-trace co-agent is able
to give a reliably useful signal after only the second pulse. More challenging
conditions introduce more variance in these intervals, but the trend remains
that the tile-coded trace co-agent provides useful signals earlier, and more
reliably. The reason for this can be understood by examination of the
representations and corresponding predictions made by each co-agent. Again for
clarity we present data from two individual trials: an easily predictable case
(Fig. 27, a typical fixed-ISI trial corresponding to session 3 trial 2) and a
more challenging case (Fig. 28, a typical random-ISI trial corresponding to
session 5 trial 3). The regular, small binning of the bit-cascade
representation means several features correspond to the hazard pulse. The
value of each of these features must be updated in turn as they are activated,
meaning that none of them are active for very long, resulting in the slowly
and evenly rising saw-tooth crest in the prediction. By contrast, the tile-
coded trace representation has large representational bins in the time span
near the hazard. The few features here are active for much of the duration of
the pulse, meaning that the predictions corresponding to these features can be
updated rapidly. The more rapidly updated tile-coded trace predictions cross
the signalling threshold sooner than those of the bit-cascade. In the more
challenging condition (Fig. 28), the effects of a changing ISI are clearly
visible in the presentation of the third pulse. In this trial, the third pulse
came along much later than the first two, and during a period of the bit-
cascade representation that had not yet been activated. The tile-coded trace
representation is better able to cope with this discrepancy, as the wide
binning allows a previously activated feature to cover this time period.

The threshold and representation choices are therefore critical to co-agent
signalling behaviour, and in turn the interaction between the co-agent and the
participant. The threshold and representation bin-widths in this experiment
were chosen considering late-trial performance, so that once the predictions
stabilized both representations would give roughly equal notice before a
pulse. A lower threshold or wider feature bins would likely have allowed the
bit-cascade co-agent to provide reliably useful signals earlier in the trials.
The bit-cascade co-agent is able to produce a more accurate prediction of the
hazard than the tile-coded trace co-agent due to the finer feature binning in
the pulse region (e.g., Fig. 7). The trade-off for this accuracy comes in the
form of slower early learning, which was found detrimental to the interaction
between the co-agent and the human participant (a tension with the usual
approach of seeking excellent asymptotic accuracy, perhaps at the expense of
early learning speed).

Figure 26: Mean interval between signal from co-agent and pulse over trial
length. Shaded regions represent the 95% confidence interval. The minimum
useful lead time (shown as dashed line) before the hazard pulse (dotted line)
is 0.89 seconds, corresponding to the time needed to exit the hazard region
given the average speed of the participant during exit events. Due to the
randomization of the starting ISI (refer to Table 2) and fixed trial duration,
some trials with shorter ISIs presented more pulses than others. This led to
the occurrence of one or two trials with high pulse count (>14), resulting in
the large confidence intervals at the ends of these plots. Figure 27: Effect
of representation and threshold choice on prediction and communication (fixed
condition). Examining a single trial (here the fixed pulse interval condition
of session 3, trial 2), we can observe the change in a co-agent’s prediction
over time (solid line). Shaded vertical bars depict each co-agent’s
representation of time; each representation bin is active for a period of time
corresponding to that bin’s width on this chart. Bars below the chart indicate
the time for which the hazard pulse is active and the time for which the co-
agent is signalling. Figure 28: Effect of representation and threshold choice
on prediction and communication (random condition). Examining a single trial
(here the random pulse interval condition of session 5, trial 3), we can
observe the change in a co-agent’s prediction over time (solid line). Shaded
vertical bars depict each co-agent’s representation of time; each
representation bin is active for a period of time corresponding to that bin’s
width on this chart. Bars below the chart indicate the time for which the
hazard pulse is active and the time for which the co-agent is signalling.

###  8.5 Qualitative Questions and Interpretation of Participant Feedback

|  Experimenter-developed questions
---|---
$\circ$ |  Are you trying to figure out how the co-agent (and environment) work?
$\circ$ |  For the whole trial?
$\circ$ |  If not, did you figure it out or just start to trust it?
$\circ$ |  After time, or successes?
$\circ$ |  How much do you notice or think about the other agent at the beginning? The middle? The end?
|  Participant-developed questions
$\circ$ |  Changes in when and how I counted: did I count from the start of the trial? Did I shift to just counting from the agent cue and not counting from the beginning? When did I shift between these and under what conditions or observations on timing?
$\circ$ |  What co-agent behaviours did I like and not like?
$\circ$ |  Adaptation rates: what were my expectations on response or learning times for co-agents?
$\circ$ |  Did I think of co-agents as adaptive systems / predictors or not?
$\circ$ |  What conditions did I lose confidence in the co-agent; when did I gain confidence?
$\circ$ |  When did trust in the co-agent occur quickly?
Table 4: Qualitative questions considered by the participant for compiling
notes during each session. Experimenter-developed questions were posed by the
member of the study team conducting the qualitative analysis at the outset of
the trials. Participant-developed questions were generated independently by
the member of the study team acting as the participant, and evolved as the
study progressed.

In cases where a machine agent is interacting with a human, there are insights
that can be gained from the user’s experience through the use of qualitative
methods. The participant was provided with questions to consider as they
underwent the trials, and also initiated their own questions (Tab. 4).
Participant-generated questions allowed for the participant to contribute to
the data being gathered based on their experience with the system, rather than
being restricted to a priori questions. The participant made notes after each
session. These notes were sent without review by the participant to a
different author who analyzed them. The notes were analyzed in isolation of
the quantitative data initially in order to minimize biases that may have
arisen from seeing the quantitative results. The notes were made after each
session (not trial), so they are broader senses of the interaction and
unfortunately lack specific connections to the type of co-agent. We
acknowledge that the participant is an author on this paper and an expert in
the system used, and that the analysis of the notes taken was done by a person
very familiar with the participant. These factors, along with the
circumstances that prompted the use of only a single participant, ultimately
mean that the findings of this portion cannot be generalized, but provide
insight as to what direction to take in expanded studies; we expect they will
illuminate what outcomes we might find possible or seek to directly test for
in a wider study (Mook,, 1983).

Discourse analysis was the primary methodology used to analyze the notes made
by the participant. The notes were examined for recurring sentiment and
topics, and three themes emerged. These were trust, cognitive load, and use of
the message. Following the initial analysis the participant discussed their
notes and thoughts with the experimenter doing the analysis, raising
additional points of interest.

Trust: Trust was built most rapidly when the system learned quickly. In
response to the prompt “When did trust in the co-agent occur quickly”, the
participant noted: “when it learned fast”. The participant found that when the
system was demonstrably correct earlier in trial, indicated by the participant
receiving the signal from the agent to move followed closely by the hazard
indication, they tended to trust it more. Trust is foundational to the other
themes that emerged from the participant’s notes. Trust changed how the
participant experienced the mental load: “With trust in my co-agent, I can let
[my] mind wander.”, and trust also allowed unexpected uses of the signal: “…
the co-agent helps me feel like I have a lower bound of safety once it is
trained, and then can choose my risk based on its feedback.” It is reasonable
to suggest that if the participant did not trust the agent they would not feel
there was safety, that they could rely on it’s input to make riskier decisions
about the task.

User trust in a machine co-agent may be crucial to any other outcomes we seek
from such a partnership. Future studies may benefit from a direct line of
questioning that specifically focuses on trust. A possible challenge in
investigating the development of trust is that trust is initially in the
participant’s thoughts, but once it is developed the participant no longer
thinks about trust directly. However, the existence of that trust enables the
participant to shift their cognitive load to the agent as well as being a key
factor in the participant’s use of the message in unexpected or unintended
ways.

Cognitive load: The task was designed to make mental timing difficult, to
encourage the use of the agent’s signal. This difficulty is reflected in the
participant’s notes: “Counting in my head is fatiguing, and means I am unable
to allow my mind to wander even for a moment. With a co-agent, I can let my
mind wander and, even if I am still counting myself, can have a ‘heads up’ if
I need to pay attention again when it buzzes.” On the trials with a co-agent
the participant could let their focus lapse to some degree. This reduced the
cognitive load on the participant by offloading some of the responsibility to
the co-agent agent. The participant also reported that their count could be
inaccurate. They reported that when caching the points (which is exciting and
tense) their internal count may have accelerated. This feeling was based on
when the participant received the co-agent cue earlier than expected. “For co-
agent conditions, I think I am too eager to off load the timing to the co-
agent. I think I stop counting and follow the agent feedback cues earlier than
I strategically should.” This comment, coupled with a comment made by the
participant at the top of the second session notes (“Another 8 sessions of
this? Sigh.”), suggests that the participant wanted to offload the dull but
cognitively intensive task on the agent, even in situations where trust hadn’t
yet been established in the system. Further explorations are required in order
to determine whether established trust leads to cognitive off-loading, or
whether the desire to off-load cognitive processes leads to premature trust.
Most likely, the establishment of trust and the gradual reliance on the system
are related processes that are developed simultaneously, as reflected in the
participant’s notes: “…in some runs I found myself gradually stopping counting
and starting to rely on the agent cues even without explicitly choosing to do
so… I’d be part way into a pulse and realize (in a bit of a panic) that I had
forgotten to start counting… but then would get the cue and move and it was
fine.” The transfer of cognitive load to the co-agent in this case was
inadvertent, rather than a conscious choice. The element of panic suggests
that the participant does not yet fully trust the system, but the system’s
competence encouraged further trust. This note also emphasizes that the
participant is not relying on the system to do the task, but is working with
the co-agent to accomplish the task.

Use of the message: The machine agent was designed to signal the participant
with its prediction of the onset of the upcoming wind gust. Despite knowing
this, the participant occasionally used the prompt from the co-agent in ways
that were not anticipated in the original design.

One way the prompt of the co-agent was used was to help the participant
determine what type of trial they were in (fixed, drift, or random).
“…evidence accumulation on my part for what kind of trial I was in was
definitely helped by having a co-agent as off-board memory and a sanity check
for what I perceived so far.” Other statements made by the participant suggest
that over time, they found it less important to determine what sort of trial
they were in. Rather, they began to associate the prompt from the co-agent
with the oncoming hazard in the way that the participant felt they could best
solve the task: “Even if I moved before the co-agent’s cue and did not rely on
the agent in that regard, I might at times be seeing how the pulse and the
agent buzz line up or have changed in their distance from each other, and use
that info to modify how I respond to my own counting strategy.” By making use
of their greater knowledge of the system, the world, and the context of the
task, the participant was able to leverage the agent signal as more than a
simple cue to move. The participant began to associate the prompt with the
hazard in a spatial sense; the participant got a sense for if the hazard was
“approaching” or “receding” based on perceived time between prompt and hazard
signals. In some instances, the participant was confident enough about their
understanding of the prompt-to-hazard timing to engage in risky behaviour: “I
was at times racing the pulse; the co-agent would cue me but I would see the
heat bar almost full and then gamble that it would fill fast enough before the
[hazard] came, given what I knew about relationship between cue and future
[hazard].” A final example of non-standard use of the prompt came in cases
with a particularly short hazard ISI, where the participant used the co-agent
signal as a verification of their mental time-keeping: “…even though I had
agent help, it was not fast enough to be useful in advance of [hazard], so I
mainly used it as a checksum.”

Discussion with Participant: Frequently during discussion, the participant
mentioned using the signal from the co-agent as a reference to the impending
event rather than a direct signal of it, or as a direction to move. Use of the
message as a reference point rather than as a cue to move suggests that the
participant made use of the co-agent’s signal to inform their own tracking and
planning of the task, rather than naively following its advice. Noting again
that our participant is deeply familiar with learning systems and their
limitations, it will be interesting to see in future studies whether more
varied participants have similar interactions.

The participant noted that in early sessions, part of their strategy was to
infer the type of trial they were interacting with (fixed, random, or drift),
in order to inform their timing. However, after repeated interactions with the
system, the participant found it became unimportant to try to identify the
trial condition when choosing how they will act. Instead, they reported that
they built a sense of the relation between the prompt and the hazard and acted
according to that. Despite the participant’s deep initial understanding of the
task, the participant’s approach changed over time, seemingly as a result of
interactions with the co-agent.

###  8.6 Synthesis of Quantitative and
Qualitative Results

In both the quantitative and qualitative analyses, we see human trust of the
co-agent emerging as an important theme. The participant’s notes suggest that
using the sign of the signal-to-exit interval (Fig. 22) as an indicator of
human trust might miss parts of the picture, since the participant makes use
of the co-agent signal in other ways than as simply a cue to move. Other
quantitative measures of trust should be sought, to corroborate this
interpretation. One particular notion of intense trust called out in the
participant notes (when the participant is “racing” the pulse, caching points
after the co-agent signal but before the hazard) is also visible in the
quantitative data. Of the 14 instances where a point caching event is recorded
after a co-agent signal and before a hazard, 13 of these instances occurred
when the participant was paired with the tile-coded trace co-agent. This could
be interpreted as a strong measure of trust between the human and tile-coded
trace co-agent, but it should also be noted that (as shown in Fig. 26) the
tile-coded trace co-agent reliably gives more lead-time than necessary before
the pulse, leaving time for pulse “racing” that the bit-cascade co-agent does
not.

The qualitative analysis also explains some apparently contradictory aspects
of the quantitative data. For example, the fact that the participant does not
appear to use the bit-cascade co-agent in fixed-ISI trials (Fig. 22) yet the
participant is hit by the pulse significantly fewer times as compared to no
co-agent (Fig. 19) can be explained by the participant’s use of the agent
signal as a “sanity check for what [the participant] perceived so far”, or as
a “checksum”, rather than as the intended cue to leave the goal region.

###  8.7 Follow-Up Studies

Specific quantitative measures to assess human trust in the co-agent would be
particularly informative for future studies, especially if such measures could
assess changes in levels of trust over the course of a trial or across
sessions. One such task modification might involve the introduction of a
secondary, voluntary and cognitively demanding task that could be performed
simultaneously while gathering heat. While engaged with the secondary task,
the participant would need to place trust in the co-agent to keep track of the
timing in the primary task. Another point of potential interest would be the
introduction of an adaptive agent that blends features of the two
representations tested in this study; an agent that adapts its feature-binning
to allow for rapid early learning as well as late-trial accuracy may be an
interesting candidate for study.

For future time-based prediction experiments or applications involving human
actors with machine co-agents, we make no particular recommendations about
representation or threshold choices, as we understand these to be task-
specific. We do however stress the importance of these choices, and recommend
that they be made with both early and late learning stages in mind, and
considering the interaction between the human and machine’s actions.

##  9 Discussion

As each independent empirical section (Sec. 6–8) contained relevant discussion
to the results presented in the respective sections, in this final section of
the manuscript we provide discussion on cross-cutting themes. In particular,
we highlight general findings in Pavlovian signalling, comparisons to the
relevant literature, and also cross-cutting observations that connect the
different experiments presented above.

The main contribution of this manuscript is the in-depth exploration of what
we here defined as Pavlovian signalling. At a high level, the results in this
work support a clear value case for using Pavlovian signalling as a lens to
study certain agent-agent relationships. As identified by Pilarski et al.,
(2017), we can frame different dyadic partnerships between agents in terms of
the agency and capacity of the parties engaged in the interaction; while
capacity of a partnership might be limited by having reduced agency by one of
the parties, the simplicity of one partner provides the opportunity for fast
learning of its behaviour by the other party (Pilarski et al.,, 2017). This is
the case with Pavlovian signalling: we see that, with no policy learning on
the part of the co-agent, predictions may be learned rapidly (less than 500
steps), as seen in Figs. 8 and 9 and when distilled down to a low-bandwidth
signal can be readily used for policy learning by a second control learning
agent (also c.f. Pilarski and Sutton, (2012) for a discussion of the way
bandwidth, latency, and explicitness impact agent/co-agent relationships).

In addition to the learnability of the co-agent by the main agent, we see
evidence that there are further benefits to Pavlovian signalling in that it
assumes nothing about the internal structure or mutability of the agent; the
agent can approach the signals from an ungrounded perspective, as in the full
control learning machine agent in Sec. 7 or the human agent in Sec. 8. The
agent can also approach the signals from a grounded perspective, as per the
responsive, hard-wired agent introduced in Sec. 7. Further, there is no reason
to limit the co-agent to only observations of the environment as we have done
in these studies; it is natural to think that information pertaining to the
main agent and its adaptability can be used as well as inputs to the assitant,
further increasing its ability to make relevant predictions and to begin to
build what we have previously termed communicative capital (Pilarski et al.,,
2017). As such, Pavlovian signalling appears to be a viable stepping stone
between between fixed agent-agent relationships and fully, bidirectionally
learned multiagent relationships.

We also note that, across studies in this work, that the interpretation of
representations and how they relate to predictions and finally control
behaviour appears to depend on the agency of the party receiving the signals.
In the case of a Pavlovian signalling co-agent and machine agent (Sec. 7), the
lack of temporal aliasing in the bit cascade representation was observed to
provide a level of consistency in the generated tokens that allowed for agents
partnered with bit cascade co-agents to do better (fixed condition) or equal
to (random and drift conditions) that of co-agent-agent combinations using a
presentation that produced more temporal aliasing (Figs. 13). However, in the
case of a Pavlovian signalling co-agent paired with an agent that was able to
model the co-agent and produce more complex decision making, tile-coded-trace
representations with greater temporal aliasing proved more effective and more
well received as signal inputs to the human agent (e.g., Fig. 23). This is an
interesting finding, and suggests that the interaction between temporal
aliasing/resolution and agent capacity is more nuanced than we expected, and
is an important element to study when formulating agent-agent partnerships.

Returning to our basis in the neuroscience literature, we note that we largely
designed our representations to act as population clocks with different
proprieties. These clocks were then provided to prediction learners that
either served as pattern timing prospective elements (accumulation predictions
forecasting the incidence of future events in time) or interval timing
elements with ramping model properties (countdown predictions forecasting the
remaining interval until a future event). These predictions were used to
inform reflexive action or more complex control, with their utility relating
strongly to the amount and location of temporal aliasing observed in the
prediction at different places within an ISI (c.f., Fig. 7 fixed condition).
From the comparisons of accumulation and countdown predictions, we also see
that our approximation of interval timing GVFs are more likely to get
problematic over longer spans or with more variability in the environment than
would approximations of pattern timing GVFs. Further, even what might be
considered simple ramping cells (bias unit) prove to be important in some
simple cases if the threshold for token generation is tuned well—i.e.,
tracking and not prediction also forms a viable ramping model for interval
timing in our instance of Pavlovian signalling.

In total, we found evidence for our hypothesis that nexting on the part of a
machine agent can be used in token formation to help bridge the gap in time
between past stimulus and later decision points for a human or machine
decision-making agent, improving decision making abilities of the agent in
small or large ways. However, there was no clearly “best” temporal
representation across the different studies and contexts, even if some
representations allowed prediction formation that most closely matched the
true cumulant return calculations of Eq. 1. Or, said differently and to
paraphrase Kearney et al., (2021), from our results we see that the accuracy
of a co-agent’s predictions do not necessarily dictate their usefulness in
Pavlovian signalling for agent-agent interaction.

###  9.1 Preliminary Evidence on more Complex Tokenization Approaches

Figure 29: Schematic example of an agent/co-agent pairing for Pavlovian
signalling with multiple tokens. GVF predictions are here mapped to a vector
of binary tokens according to a pre-determined coding scheme.

As noted in Sec. 6, the Pavlovian signalling we propose in this work as a
process for turning predictions into tokens is in no way limited to the
generation of a single binary token. We may also consider cases where
predictions are mapped in a hard-coded way to the activity of bits in a one-
hot vector of arbitrary size, are hashed into a vector with multiple active
features, or are mapped to a vector of scalar values (with one example shown
schematically in Fig. 29). We also note that the mapping process (e.g., the
threshold or other mapping function) may best be suited to real-time
adaptation during the process of learning either on the part of the co-agent
or by some pre-determined external mechanism. Indeed, the observations from
the random and drift conditions in Figs. 8 and 9, the trends noted for control
learning in different representations for the abstract Frost Hollow
environment, and the qualitative and quantitative human-machine findings all
suggest there might be merit in online adaptation of a Pavlovian signalling
approach. Online adaptation and multiple tokens are outside the scope of the
present work (which the authors note is already quite lengthy), but we believe
may serve as helpful doorways toward understanding more complex emergent
communication learning processes between an agent and a co-agent.

To provide initial insight into a multi-token process, we also performed a
preliminary study on one extension where tokens took the form of a one-hot
vector tokenization scheme (implementing the approach for vectors of length 1,
3, and 5). Our findings showed that control learning agent performance on the
Frost Hollow domain when using these tokens as part of the agent’s state in
place of the single binary token used in our studies above led to a
stratification of agent performance levels, where more complex tokenization
schemes led to potentially slower learning by the control agent, but higher
overall asymptotic performance. We extrapolate from these anecdotal results
that, not surprisingly, increasing the complexity of tokenization will lead to
more complex agent/co-agent interaction and possible performance gains and
losses depending on the implementation of such approaches. Evidence from
Edwards et al., (2016) also suggests that scalar presentation of predictive
tokens with appropriate threshold for minimum and maximum token values may
lead to actionable signals for the receiving agent. Evidence from Pilarski et
al., (2019) further suggests that token presentation based on context (e.g.,
location of stimulus) has possible merits for the decision-making agent.

We also note that it is unclear how tokenization complexity will impact co-
agents learning and adapting their predictions and tokens over extended
periods of time, as partnerships move beyond the limited episodic length
studied in this work and embrace the full continual learning setting for both
co-agent/machine-agent and co-agent/human contexts.

As is well described by Scott-Phillips, (2014), there is a clear route to
progress from grounded signals to what he termed as full, ostensive-
inferential communication; Pezzulo et al., (2013) also describe the power of
signalling in more elaborate acts of coordination and agent-agent alignment.
We are therefore interested to explore how Pavlovian signalling can provide a
functional bridge towards this more natural, expressive communication
interface. Pilarski et al., (2019) provided one view into this pathway,
through a bidirectional learning relationship wherein the agent (a human) made
certain things visible for co-agent learning, and the co-agent subsequently
learned when and how to make its Pavlovian signalling tokens visible to the
agent. Tokens created in Pavlovian signalling with GVFs have the nice property
that they are constructivist in nature and we could imagine them being created
by a co-agent autonomously as opposed to specified by an external designer.
Tokens are for the sender grounded in co-agent-centric (subjectively
specified) GVF question parameters cumulant $c$, time scale $\gamma$, and
policy $\pi$ and also in the mapping approach, in this case is parameterized
by the threshold value $\tau$ used in token generation. For our specific case,
this 4-element tuple

$\\{c,\gamma,\pi,\tau\\}$ (11)

is the grounding of the token, and it can be fully subjective to the sending
agent and not require some reference to an objective frame of reference. This
suggests a solid, understandable basis for more eloquent communication between
co-agents and agents.

Pavlovian signalling as implemented in this work is a process wherein learned,
temporally extended predictions in the form of generalized value functions are
mapped via a fixed threshold to Boolean tokens intended for receipt by a
decision-making agent, and where these signals are grounded for the sender in
the cumulant $c$, time scale $\gamma$, policy $\pi$ and threshold value $\tau$
of their computational elements.

###  9.2 Non-linear Co-agents and
Single-agent Alternatives

A final, natural question relates to the form of the co-agent’s prediction
learning algorithm and architecture. For clarity, learning speed and stability
in an online learning environment, we chose to specifically study co-agents
with linear learning mechanisms. Non-linear learning, if efficient in an
online context, promises even better representation of time and multifaceted
temporal phenomena. For example, we might consider the clockwork RNN of
Koutnik et al., (2014) or an LSTM formulation as per Rafiee et al., (2021).
These are viable areas for future work.

Recently Rafiee et al., (2021) investigated recurrent learning algorithms such
as LSTMs on a suite of problems inspired by experiments in animal learning.
The key question addressed in their work was how effectively a recurrent
learning agent can represent time in trace conditioning. Trace conditioning is
a form of classical conditioning that proceeds in repeated trials. In each
trial the agent is presented with a binary CS that occurs for several
timesteps (with a random unpredictable onset). After several time-steps
(randomized but within some bounded interval), a binary US is presented to the
agent. The time between CS onset and US onset is called the inter-stimulus
interval or ISI. The agent’s job is to predict the US based on the CS, just as
a rabbit predicts the air-puff based on a tone. The challenge is that there
are no sensory signals presented to the agent between the CS and the US; the
agent must construct a representation that somehow encodes the passage of
time. The major finding of Rafiee et al.’s study was that recurrent learning
systems could accurately make trace conditioning predictions for short ISI.
However, as ISI was increased, state-of-the-art algorithms could not learn
accurate predictions of the US even as the amount of Back propagation through
time was increased.

Further, we might consider placing temporal representation elements, such as
an LSTM, inside the main agent and not considering the co-agent as an explicit
system. While there are cases where this could be (and verifiably is) an
excellent choice for learning complex phenomena, we note that the ability to
consider a co-agent separately gives us the capacity to study interactions
with systems not under our control, e.g., a production level system, an
external piece of hardware or software, and importantly as shown in Sec. 8,
advanced biological learning agents of great relevance to daily life: humans
beings.

##  10 Conclusions

In this work, we contributed a concrete definition and exploration of
Pavlovian signalling as implemented via processes of GVF learning. Our
findings, while preliminary in that they are derived from a single human-agent
case series and related fundamental agent-agent experiments, suggest that
Pavlovian signalling by a co-agent can be learned very rapidly in single-shot
real time deployment, improves temporal decision-making of receiving human and
machine control-learning agents, and opens a number of future avenues for
using GVF learning in this way to augment, adapt, and potentially enhance
human and machine perception, action, and cognition—especially in settings
where the connection between the agent and co-agent is tight but perhaps low
in bandwidth. When presented with information from a machine co-agent, we
observed changes in both the timing of decisions of both human and machine
agents (behavioural change, differences in sensorimotor trajectories, and
reaction times) and also the quantitative outcomes of the timing task (score,
number of mistakes). Future study is needed with larger environments or
continual learning settings with distractors, and in tasks that more fully
blend both time and space. In summary, we believe there is great opportunity
for using Pavlovian signalling to understand agent-agent signalling and
communication in complex tasks that unfold in both time and space, and as a
useful way-point between hand-designed communication interfaces and full
machine communication learning.

### Acknowledgements

The authors thank Kevin McKee, Kory Mathewson, Michael Bowling, Ola
Kalinowska, Nathan Wispinski, Andrew Bolt, Alexander Zacherl, Drew Purves,
Edward Hughes, Christopher Summerfield, Richard Sutton, Koray Kavukcuoglu, and
Shanon Phelan, for many helpful conversations, discussions, and technical
insights regarding this work, in addition to a number of other colleagues
within DeepMind and at the University of Alberta. University of Alberta
collaboration on this work by PMP, ASRP, and AW was supported, in part, by the
Canada CIFAR AI Chairs program, the Canada Research Chairs program, the
Natural Sciences and Engineering Research Council (NSERC), and the Alberta
Machine Intelligence Institute (Amii). We also thank the authors and
maintainers of Launchpad, which we used for our experiments (Yang et al.,,
2021). We further note that peer-reviewed versions of components of this paper
have appeared or will appear in Butcher et al., (2022) and Brenneis et al.,
(2021).

### Author Contribution Statement

All authors contributed to the concept or design of the work and to the
acquisition, analysis or interpretation of data; drafted the article or
revised it critically for important intellectual content; approved the version
to be published; and are in agreement to be accountable for all aspects of the
work in ensuring that questions related to the accuracy or integrity of any
part of the work are appropriately investigated and resolved.

## References

  * Amershi et al., (2014) Amershi, S., Cakmak, M., Knox, W. B., and Kulesza, T. (2014).  Power to the people: The role of humans in interactive machine learning.  AI Magazine, 35:105–120.
  * Andrewes, (2006) Andrewes, W. J. (2006).  A chronicle of timekeeping: Our conception of time depends on the way we measure it.  _Scientific American Special Editions_ , 16(1s):46–55.
  * Balleine and O’Doherty, (2010) Balleine, B. W. and O’Doherty, J. P. (2010).  Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action.  Neuropsychopharmacology, 35(1):48–69.
  * Banino et al., (2018) Banino, A., Barry, C., and Uria, B. (2018).  Vector-based navigation using grid-like representations in artificial agents.  Nature, 557:429–433.
  * Brenneis et al., (2021) Brenneis, D. J. A., Parker, A. S. R., Johanson, M. B., Butcher, A., Davoodi, E., Acker, L., Botvinick, M. M., Modayil, J., White, A., and Pilarski, P. M. (2021).  Assessing human interaction in virtual reality with continually learning prediction agents based on reinforcement learning algorithms: A pilot study.  arXiv, 2112.07774 [cs.AI].
  * Buonomano, (2017) Buonomano, D. (2017).  Your brain is a time machine: the neuroscience and physics of time.  W.W. Norton & Company, New York.
  * Butcher et al., (2022) Butcher, A., Johanson, M. B., Davoodi, E., Brenneis, D. J. A., Acker, L., Parker, A. S. R., White, A., Modayil, J., and Pilarski, P. M. (2022).  Pavlovian signalling with general value functions in agent-agent temporal decision making.  arXiv, 2201.03709 [cs.AI].
  * Candidi et al., (2015) Candidi, M., Curioni, A., Donnarumma, F., Sacheli, L. M., and Pezzulo, G. (2015).  Interactional leader–follower sensorimotor communication strategies during repetitive joint actions.  Journal of The Royal Society Interface, 12(110):20150644.
  * Charrier and Sturdy, (2005) Charrier, I. and Sturdy, C. B. (2005).  Call-based species recognition in black-capped chickadees.  Behavioural Processes, 70(3):271–281.
  * Cheney and Seyfarth, (2018) Cheney, D. L. and Seyfarth, R. M. (2018).  How monkeys see the world: Inside the mind of another species.  University of Chicago Press.
  * Clark, (2013) Clark, A. (2013).  Whatever next? Predictive brains, situated agents, and the future of cognitive science.  Behavioral and Brain Sciences, 36(3):181–204.
  * Copernicus, (1543) Copernicus, N. (1543).  De revolutionibus orbium coelestium.
  * Crandall et al., (2018) Crandall, J. W., Oudah, M., Tennom, F. I.-O., Abdallah, S., Bonnefon, J.-F., Cebrian, M., Shariff, A., Goodrich, M. A., and Rahwan, I. (2018).  Cooperating with machines.  Nature Communications, 9:233.
  * Cueva and Wei, (2018) Cueva and Wei (2018).  Emergence of grid-like representations by training recurrent neural networks to perform spatial localization.  International Conference on Learning Representations (ICLR) .
  * Dalrymple et al., (2020) Dalrymple, A. N., Roszko, D. A., Sutton, R. S., and Mushahwar, V. K. (2020).  Pavlovian control of intraspinal microstimulation to produce over-ground walking.  Journal of Neural Engineering, 17(3):036002.
  * Dayan and Berridge, (2014) Dayan, P. and Berridge, K. C. (2014).  Model-based and model-free pavlovian reward learning: revaluation, revision, and revelation.  Cognitive, Affective, & Behavioral Neuroscience, 14(2):473–492.
  * Dretske, (1981) Dretske, F. (1981).  Knowledge and the flow of information.  MIT press, Cambridge.
  * Edwards et al., (2016) Edwards, A. L., Hebert, J. S., and Pilarski, P. M. (2016).  Machine learning and unlearning to autonomously switch between the functions of a myoelectric arm.  Proceedings of the 6th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob2016), June 26-29, pages 514–521.
  * Eichenbaum, (2014) Eichenbaum, H. (2014).  Time cells in the hippocampus: a new dimension for mapping memories.  Nature Reviews Neuroscience, 15:732–744.
  * Eichenbaum, (2017) Eichenbaum, H. (2017).  On the integration of space, time, and memory.  Neuron, 95(ue 5):1007–1018.
  * Gibbon, (1977) Gibbon, J. (1977).  Scalar expectancy theory and Weber’s law in animal timing.  Psychological Review, 84(3):279–325.
  * Gilbert, (2006) Gilbert, D. (2006).  Stumbling on happiness.  A.A. Knopf, New York.
  * Grice, (1975) Grice, H. P. (1975).  Logic and conversation.  In Speech Acts, Syntax and Semantics, vol. 3, pages 41–58.  Brill, Leiden.
  * Grice, (1989) Grice, P. (1989).  Studies in the Way of Words.  Harvard University Press, Cambridge.
  * Grynszpan et al., (2019) Grynszpan, O., Sahaï, A., Hamidi, N., Pacherie, E., Berberian, B., Roche, L., and Saint-Bauzel, L. (2019).  The sense of agency in human-human vs human-robot joint action.  Consciousness and Cognition, 75:102820.
  * Hafting et al., (2005) Hafting, T., Fyhn, M., Molden, S., Moser, M., and Moser, E. (2005).  Microstructure of a spatial map in the entorhinal cortex.  Nature, 436(7052):801–806.
  * Hancock et al., (2011) Hancock, P., Billings, D., Schaefer, K., Chen, J., de Visser, E., and Parasuraman, R. (2011).  A meta-analysis of factors affecting trust in human-robot interaction.  Human Factors, 53:517–527.
  * Harrison et al., (1767) Harrison, J., Maskelyne, N., and Longitude, G. B. C. O. (1767).  The principles of Mr. Harrison’s time-keeper; with plates of the same. London, printed by W. Richardson and S. Clark.  PDF Retrieved from the Library of Congress.
  * Hochreiter and Schmidhuber, (1997) Hochreiter, S. and Schmidhuber, J. (1997).  Long short-term memory.  Neural Computation, 9(8):1735–1780.
  * Jaderberg et al., (2017) Jaderberg, M., Czarnecki, W. M., Osindero, S., Vinyals, O., Graves, A., Silver, D., and Kavukcuoglu, K. (2017).  Decoupled neural interfaces using synthetic gradients.  Proceedings of the 34th International Conference on Machine Learning, pages 1627–1635.
  * Jirenhed et al., (2007) Jirenhed, D.-A., Bengtsson, F., and Hesslow, G. (2007).  Acquisition, extinction, and reacquisition of a cerebellar cortical memory trace.  Journal of Neuroscience, 27(10):2493–2502.
  * Kearney et al., (2021) Kearney, A., Koop, A., and Pilarski, P. M. (2021).  What’s a good prediction? Challenges in evaluating an agent’s knowledge.  arXiv, 2001.08823 [cs.AI].
  * Knoblich et al., (2011) Knoblich, G., Butterfill, S., and Sebanz, N. (2011).  Psychological research on joint action: Theory and data.  In Ross, B., editor, The Psychology of Learning and Motivation Vol. 54, pages 59–101. Academic Press, Burlington.
  * Koutnik et al., (2014) Koutnik, J., Greff, K., Gomez, F., and Schmidhuber, J. (2014).  A clockwork RNN.  Proceedings of the 31st International Conference on Machine Learning, pages 1863–1871.
  * Lazaridou and Baroni, (2020) Lazaridou, A. and Baroni, M. (2020).  Emergent multi-agent communication in the deep learning era.  arXiv, 2006.02419 [cs.CL].
  * Lewis, (2008) Lewis, D. (2008).  Convention: A philosophical study.  John Wiley & Sons, New Jersey.
  * Li et al., (2019) Li, G., Gomez, R., Nakamura, K., and He, B. (2019).  Human-centered reinforcement learning: A survey.  IEEE Transactions on Human-Machine Systems, 49(4):337–349.
  * Ludvig et al., (2012) Ludvig, E. A., Sutton, R. S., and Kehoe, E. J. (2012).  Evaluating the TD model of classical conditioning.  Learning & Behavior, 40(3):305–319.
  * Lusk et al., (2016) Lusk, N. A., Petter, E. A., MacDonald, C. J., and Meck, W. H. (2016).  Cerebellar, hippocampal, and striatal time cells.  Current Opinion in Behavioral Sciences, 8:186–192.
  * MacDonald et al., (2011) MacDonald, C., Lepage, K., Eden, U., and Eichenbaum, H. (2011).  Hippocampal "time cells" bridge the gap in memory for discontiguous events.  Neuron, 71(4):737–749.
  * Maes, (1994) Maes, P. (1994).  Agents that reduce work and information overload.  Communications of the ACM, 37:30–40.
  * Miall and Wolpert, (1996) Miall, R. C. and Wolpert, D. M. (1996).  Forward models for physiological motor control.  Neural Networks, 9(8):1265–1279.
  * Mnih et al., (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., and Ostrovski, G. e. a. (2015).  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533.
  * Modayil and Sutton, (2014) Modayil, J. and Sutton, R. S. (2014).  Prediction driven behavior: Learning predictions that drive fixed responses.  Workshops at the Twenty-Eighth AAAI Conference on Artificial Intelligence.
  * Modayil et al., (2014) Modayil, J., White, A., and Sutton, R. S. (2014).  Multi-timescale nexting in a reinforcement learning robot.  Adaptive Behavior, 22(2):146–160.
  * Mook, (1983) Mook, D. G. (1983).  In defense of external invalidity.  American Psychologist, 38(4):379–387.
  * Mozer, (1993) Mozer, M. (1993).  Neural network architectures for temporal pattern processing.  In A. Weigend & N. Gershenfeld (Eds.), Time Series Prediction: Forecasting the Future and Understanding the Past, pages 243–264.  Addison-Wesley Publishing, Redwood City, CA.
  * Nath et al., (2019) Nath, S., Liu, V., Chan, A., Li, X., White, A., and White, M. (2019).  Training recurrent neural networks online by learning explicit state variables.  International Conference on Learning Representations.
  * Neil et al., (2016) Neil, D., Pfeiffer, M., and Liu, S.-C. (2016).  Phased LSTM: Accelerating recurrent network training for long or event-based sequences.  arXiv, 1610.09513 [cs.LG].
  * O’Keefe and Dostrovsky, (1971) O’Keefe, D. and Dostrovsky, J. (1971).  The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat.  Brain Research, 34(1):171–175.
  * Parisotto et al., (2020) Parisotto, E., Song, F., Rae, J., Pascanu, R., Gulcehre, C., Jayakumar, S., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., et al. (2020).  Stabilizing transformers for reinforcement learning.  International Conference on Machine Learning, pages 7487–7498.
  * Parker et al., (2019) Parker, A. S. R., Edwards, A. L., and Pilarski, P. M. (2019).  Exploring the impact of machine-learned predictions on feedback from an artificial limb.  Proceedings of the IEEE-RAS-EMBS International Conference on Rehabilitation Robotics (ICORR), pages 1239–1246.
  * Paton and Buonomano, (2018) Paton, J. J. and Buonomano, D. V. (2018).  The neural basis of timing: Distributed mechanisms for diverse functions.  Neuron, 98(ue 4):687–705.
  * Pavlov, (1927) Pavlov, I. P. (1927).  Conditioned reflexes: an investigation of the physiological activity of the cerebral cortex.  Translated by G. V. Anrep. Oxford University Press, London.
  * Pesquita et al., (2018) Pesquita, A., Whitwell, R. L., and Enns, J. T. (2018).  Predictive joint-action model: A hierarchical predictive approach to human cooperation.  Psychonomic Bulletin & Review, 25(5):1751–1769.
  * Pezzulo, (2008) Pezzulo, G. (2008).  Coordinating with the future: The anticipatory nature of representation.  Minds and Machines, 18(2):179–225.
  * Pezzulo and Dindo, (2011) Pezzulo, G. and Dindo, H. (2011).  What should I do next? Using shared representations to solve interaction problems.  Experimental Brain Research, 211(3–4):613–630.
  * Pezzulo et al., (2013) Pezzulo, G., Donnarumma, F., and Dindo, H. (2013).  Human sensorimotor communication: A theory of signaling in online social interactions.  PLoS ONE, 8(11):e79876.
  * Pilarski et al., (2019) Pilarski, P. M., Butcher, A., Johanson, M., Botvinick, M. M., Bolt, A., and Parker, A. S. R. (2019).  Learned human-agent decision-making, communication and joint action in a virtual reality environment.  In 4th Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), pages 302–306.
  * Pilarski et al., (2013) Pilarski, P. M., Dawson, M. R., Degris, T., Carey, J. P., Chan, K. M., Hebert, J. S., and Sutton, R. S. (2013).  Adaptive artificial limbs: A real-time approach to prediction and anticipation.  IEEE Robotics Automation Magazine, 20(1):53–64.
  * Pilarski and Sutton, (2012) Pilarski, P. M. and Sutton, R. S. (2012).  Between instruction and reward: Human-prompted switching.  In AAAI 2012 Fall Symposium on Robots Learning Interactively from Human Teachers (RLIHT), pages 46–52.
  * Pilarski et al., (2017) Pilarski, P. M., Sutton, R. S., Mathewson, K. W., Sherstan, C., Parker, A. S. R., and Edwards, A. L. (2017).  Communicative capital for prosthetic agents.  arXiv, 1711.03676 [cs. AI], pages 1–33.
  * Rafiee et al., (2021) Rafiee, B., Abbas, Z., Ghiassian, S., Kumaraswamy, R., Sutton, R., Ludvig, E., and White, A. (2021).  From eye-blinks to state construction: Diagnostic benchmarks for online representation learning.  arXiv 2011.04590v3 [cs.AI], 2021.
  * Rescorla, (1988) Rescorla, R. A. (1988).  Pavlovian conditioning: It’s not what you think it is.  American Psychologist, 43(3):151.
  * Riley et al., (2005) Riley, J. R., Greggers, U., Smith, A. D., Reynolds, D. R., and Menzel, R. (2005).  The flight paths of honeybees recruited by the waggle dance.  Nature, 435(7039):205–207.
  * Risko and Gilbert, (2016) Risko, E. F. and Gilbert, S. J. (2016).  Cognitive offloading.  Trends in Cognitive Sciences, 20:676–688.
  * Schofield et al., (2021) Schofield, J. S., Battraw, M. A., Parker, A. S. R., Pilarski, P. M., Sensinger, J. W., and Marasco, P. D. (2021).  Embodied cooperation to promote forgiving interactions with autonomous machines.  Frontiers in Neurorobotics, 15:38.
  * Scott-Phillips, (2014) Scott-Phillips, T. (2014).  Speaking Our Minds: Why Human Communication is Different, and how Language Evolved to Make it Special.  Palgrave Macmillan, London.
  * Scott-Phillips et al., (2009) Scott-Phillips, T. C., Kirby, S., and Ritchie, G. R. S. (2009).  Signalling signalhood and the emergence of communication.  Cognition, 113(2):226–233.
  * Sebanz et al., (2006) Sebanz, N., Bekkering, H., and Knoblich, G. (2006).  Joint action: bodies and minds moving together.  Trends in Cognitive Sciences, 10(2):70–76.
  * Sebanz and Knoblich, (2009) Sebanz, N. and Knoblich, G. (2009).  Prediction in joint action: What, when, and where.  Topics in Cognitive Science, 1(2):353–367.
  * Seeley et al., (2006) Seeley, T. D., Visscher, P. K., and Passino, K. M. (2006).  Group decision making in honey bee swarms: When 10,000 bees go house hunting, how do they cooperatively choose their new nesting site?  American Scientist, 94(3):220–229.
  * Stachenfeld et al., (2014) Stachenfeld, K. L., Botvinick, M. M., and Gershman, S. J. (2014).  Design principles of the hippocampal cognitive map.  Proceedings of the 27th International Conference on Neural Information Processing Systems, vol. 2, pages 2528–2536.
  * Sutton, (1988) Sutton, R. S. (1988).  Learning to predict by the methods of temporal differences.  Machine Learning, 3(1):9–44.
  * Sutton and Barto, (2018) Sutton, R. S. and Barto, A. G. (2018).  Reinforcement Learning: An Introduction. 2nd Edition.  MIT Press, Cambridge.
  * Sutton et al., (2011) Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., and Precup, D. (2011).  Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction.  Proceedings of 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 761–768.
  * Taga and Bassler, (2003) Taga, M. E. and Bassler, B. L. (2003).  Chemical communication among bacteria.  Proceedings of the National Academy of Sciences, 100(suppl 2):14549–14554.
  * Tallec and Ollivier, (2017) Tallec, C. and Ollivier, Y. (2017).  Unbiased online recurrent optimization.  arXiv 1702.05043 [cs.NE].
  * Tolman, (1948) Tolman, E. C. (1948).  Cognitive maps in rats and men.  Psychological Review, 55(4):189–208.
  * Tsao et al., (2018) Tsao, A., Sugar, J., Lu, L., Wang, C., Knierim, J. J., Moser, M.-B., and Moser, E. I. (2018).  Integrating time from experience in the lateral entorhinal cortex.  Nature, 561:57–62.
  * Umbach et al., (2020) Umbach, G., Kantak, P., Jacobs, J., Kahana, M., Pfeiffer, B. E., Sperling, M., and Lega, B. (2020).  Time cells in the human hippocampus and entorhinal cortex support episodic memory.  Proceedings of the National Academy of Sciences, 117(45):28463-–28474.
  * Vaswani et al., (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017).  Attention is all you need.  Proceedings of the 31st Conference on Neural Information Processing Systems, pages 5998–6008.
  * Vesper et al., (2010) Vesper, C., Butterfill, S., Knoblich, G., and Sebanz, N. (2010).  A minimal architecture for joint action.  Neural Networks, 23(8–9):998–1003.
  * Vogt, (2006) Vogt, P. (2006).  Language evolution and robotics: issues on symbol grounding and language acquisition.  In Angelo Loula, Ricardo Gudwin, João Queiroz (Eds.), Artificial Cognition Systems, pages 176–209. Idea Group, Hershey.
  * Windholz, (1990) Windholz, G. (1990).  The second signal system as conceived by Pavlov and his disciples.  Pavlovian Journal of Biological Science, 25(4), 163–173.
  * Yang et al., (2021) Yang, F., Barth-Maron, G., Stańczyk, P., Hoffman, M., Liu, S., Kroiss, M., Pope, A., and Rrustemi, A. (2021).  Launchpad: A programming model for distributed machine learning research.  arXiv 2106.04516 [cs.DC].