Boyce, Lindsay, Zgonnikov, Rano, & Wong-Lin

# Optimality and limitations of audio-visual integration for cognitive systems

W. Paul Boyce1,+,∗ Tony Lindsay1 Arkady Zgonnikov2,3 Ignacio Rano1,α &
KongFatt Wong-Lin1 1Intelligent Systems Research Centre, Ulster University,
Magee campus, Derry Londonderry, Northern Ireland, UK
2AiTech, Delft University of Technology, Delft, The Netherlands
3Department of Cognitive Robotics, Faculty of Mechanical, Maritime, and
Materials Engineering, Delft University of Technology, Delft, The Netherlands
+WPB is now at the School of Psychology, University of New South Wales,
Sydney, NSW, Australia.
αIR is now at SDU Biorobotics, Mærsk Mc Kinney Møller Institut, University of
Southern Denmark, Denmark.
∗Correspondence: W. Paul Boyce
<EMAIL_ADDRESS>
###### Abstract

Abstract–Multimodal integration is an important process in perceptual
decision-making. In humans, this process has often been shown to be
statistically optimal, or near optimal: sensory information is combined in a
fashion that minimises the average error in perceptual representation of
stimuli. However, sometimes there are costs that come with the optimization,
manifesting as illusory percepts. We review audio-visual facilitations and
illusions that are products of multisensory integration, and the computational
models that account for these phenomena. In particular, the same optimal
computational model can lead to illusory percepts, and we suggest that more
studies should be needed to detect and mitigate these illusions, as artefacts
in artificial cognitive systems. We provide cautionary considerations when
designing artificial cognitive systems with the view of avoiding such
artefacts. Finally, we suggest avenues of research towards solutions to
potential pitfalls in system design. We conclude that detailed understanding
of multisensory integration and the mechanisms behind audio-visual illusions
can benefit the design of artificial cognitive systems.

Keywords—multi-modal processing, multisensory integration, audio-visual
illusions, Bayesian integration, optimality, cognitive systems.

##  1 Introduction

Perception is a coherent conscious representation of stimuli that is arrived
at, via processing signals sent from various modalities, by a perceiver:
either human or non-human animals Goldstein (2008). Humans have evolved
multiple sensory modalities, which include not only the classical five (sight,
hearing, tactile, taste, olfactory) but also more recently defined ones (for
example, time, pain, balance, and temperature Rao . (2001); Fulbright .
(2001); Fitzpatrick McCloskey (1994); Green (2004)). While each modality is
capable of resulting in a modality-specific percept, it is often the case that
stimulus information gathered by two or more modalities is combined in an
attempt to create the most robust representation possible of a given
environment in perception Macaluso . (2000); Ramos-Estebanez . (2007).

Understanding and mapping just how the human brain combines different types of
stimulus information from drastically different modalities is challenging.
Behavioural studies have suggested optimal or near-optimal integration of
multi-modal information Alais Burr (2004); Shams Kim (2010). In the case of
Alais2004ventriloquist they examined the classic spatial ventriloquist effect
through the lens of near optimal binding. The effect in question describes the
apparent ‘capture’ of an auditory stimulus in perceptual space that is then
mapped to the perceptual location of a congruent visual stimulus, the famous
example being the ventriloquist’s voice appearing to emanate from the
synchronously animated mouth of the dummy on their knee.
Alais2004ventriloquist demonstrated that this process of ‘binding’ the
perceived spatial location of an auditory stimulus to the perceived location
of a visual stimulus is an example of near optimal audio-visual integration.
They achieved this by demonstrating that variations of the effect could be
reversed (i.e. a visual stimulus being ‘captured’ and shifted to the same
perceptual space as an auditory stimulus) when the auditory signal was less
noisy relative to the visual stimulus (when extreme blurring noise was added
to the visual stimulus). Additionally, when visual stimuli was blurred, but
not extremely so, neither stimulus source captured the other and a mean
spatial position was perceived. This in turns hints at a weighting process in
audio-visual integration modulated by the level of noise in a given source
signal. These findings are consistent with the notion of inverse
effectiveness: when a characteristic of a given stimulus has low resolution
there tends to be in an increase in ‘strength’ of multisensory integration de
Dieuleveult . (2017); Stevenson James (2009). See holmes2009principle for
potential issues when measuring multisensory integration ‘strength’ from the
perspective of inverse effectiveness.

However, the very existence of audio-visual illusions in these processes
highlights that there can be a cost associated with this optimal approach
Shams, Ma Beierholm (2005); the perceptual illusions here are being considered
as unwanted artefacts (costs) that manifest due to optimal integration of
signals from multiple modalities. One such well established audio-visual
illusion that combines information from both modalities and arrives at an
auditory percept altogether unique is the McGurk-MacDonald effect McGurk
MacDonald (1976). When participants watch footage of someone moving their
lips, while simultaneously listening to an auditory stimulus (a single
syllable repeated in time with the moving lips) that is incongruent to the
moving lips, they have a tendency to ‘hear’ a sound that matches neither the
mouthed syllable or the auditory stimulus. While not gazing at the moving
lips, participants accurately report the auditory stimulus.

The McGurk effect demonstrates that the integration of audio-visual
information is an effective process in most natural settings (even when
modalities provide competing information), but may occasionally result in an
imperfect representation of events. This auditory illusion suggests a “best
guess” can sometimes be arrived at when modalities provide contradictory
information, where different weightings are given to competing modalities.
Crevecoeur2016 highlighted that the nervous system also considers temporal
feedback delays when performing optimal multi-sensory integration (for
example, visual input followed by muscle response is slower than
proprioceptive input followed by a muscle response with a difference of
~50ms). The faster of the two sensory cues is given a dominant weighting in
integration. This shows that temporal characteristics affect optimal
integration of information from different modalities, and should be a factor
in any models of multi-modal integration.

If artefacts such as illusions can occur in an optimal multi-modal system,
these artefacts become a concern when designing artificial cognitive systems.
The optimal approach of integrating information from multiple sources may lead
to inaccurate representation of an environment (an artefact), which in turn
could result in a potentially hazardous outcome. For example, if a autonomous
vehicle was trained in a specific environment and then relocated to a novel
environment, an artefact manifested via optimal integration of stimuli could
compromise the safe navigation through the novel environment and any action
decisions taken therein (this scenario is a combination of the “Safe
Exploration” and “Robustness to Distributional Shift” accident risks as
outlined by Amodei2016concrete).

The remainder of the paper reviews the processes in audio-visual perception
that offers explanations for audio-visual illusions and effects, focusing
mainly on how audition affects visual perception, and what this tells us about
the audio-visual integration system. We continue by building a case for audio-
visual integration as a process of evidence accumulation/discounting, where
differing weights are given to different modalities depending on the stimuli
information (spatial, temporal, featural etc.) being processed, which follows
a hierarchical process (from within-modality discrimination to multi-modal
integration). We highlight how some processes are optimal and others
suboptimal, and how each have their own drawbacks. Following that, we review
cognitive models of multi-modal integration which provide computational
accounts for illusions. We then outline the potential implications of the
mechanisms behind multisensory illusions for artificial intelligent systems,
concluding with our views on future research directions. Additionally, rather
than assuming that all attributions of prior entry (discussed below) are
accurate, this paper expands on the definition of prior entry to encompass
both response bias and undefined non-attentional processes. Doing so
circumvents the granular debates surrounding prior entry in favour of better
discussing the broader processes on the way to audio-visual integration, of
which prior entry is but one. We also consider impletion (discussed below) as
a process distinct from prior entry, but one that complements and/or competes
with prior entry.

##  2 Audio-visual Integration

###  2.1 Visual and Multi-modal Prior Entry

Prior entry, a term coined by E.B. Titchener in 1908, describes a process
whereby a visual stimulus that draws an observer’s attention is processed in
the visual perceptual system before any unattended stimuli in the perceptual
field. This in turn results in the attended stimulus being processed “faster”
relative to subsequently attended stimuli Spence . (2001). This suggests that
when attention is drawn (usually via a cue) to a specific region of space, a
stimulus that is presented to that region is processed at a greater speed than
a stimulus presented to unattended space.

Prior entry as a phenomenon is important in multi-modal integration due to the
fact that the temporal perception of one modality can be significantly altered
by stimuli in another modality (as well as within a modality) Shimojo .
(1997). The mechanisms underlying prior entry have been the subject of
controversy Cairney (1975); Schneider Bavelier (2003); Downing Treisman
(1997), but strong evidence has been provided for its existence via
orthogonally designed crossmodal experiments Spence . (2001); Zampini .
(2005). In the case of the orthogonally designed experiments, related
information between the attended modality and the subsequent temporal order
judgement task was removed, thus ensuring no modality-specific bias.

A classic visual illusion that supports the tenets of prior entry, and
demonstrates just how much temporal perception can be affected by it, is the
line motion illusion, first demonstrated by Hikosaka1993a using visual cues. A
cue to one side of fixation prior to the presentation of a whole line to the
display can result in the illusion of the line being “drawn” from the cued
side (Figure 1). Hikosaka1993 investigated this effect further and
demonstrated illusory temporal order in a similar fashion: namely, cueing one
side of fixation in a temporal order judgement task prior to the simultaneous
onset of both visual targets. Both these effects were replicated using
auditory cues by Shimojo1997.

Figure 1: The paradigm of the line-motion illusion similar to that used by
Hikosaka1993a and Shimojo1997. Participants were presented with a fixation
cross followed by a cue (auditory in this example) to one side of fixation
followed by another fixation display. Finally the target stimulus was
presented. In this example the left side of the target stimulus was cued via
an auditory tone. The resulting perception would be that of the line being
drawn from the same side as the cue as opposed to being presented in its
entirety at the same time.

Shimojo1997 demonstrated that the integration of auditory and visual stimuli
can cause temporal order perception in one modality to be significantly
altered by information in another via audio-visual prior entry. However,
Downing1997 suggested that the original line motion illusion was an example of
what they termed “impletion”: in an ambiguous display multiple stimuli are
combined to reflect a single smooth event in perception. For example, when an
illusion of apparent motion is created using statically flashed stimuli in
different locations (e.g. left space followed by right space), the stimuli can
appear to smoothly change from the first stimulus shape (circle) to the second
stimulus shape (square) Kolers von Grünau (1976); Downing Treisman (1997). It
is suggested that a discriminatory process gathers all available stimuli
information, combines them, and creates a coherent percept; or the most likely
real world outcome where it fills in the gaps on the way to perception.
Downing1997 demonstrated that the line motion illusion could in fact be a
perception of the visual cue itself streaking across the field of display akin
to frames in an animation. Admittedly, when one takes into account the
findings of Shimojo1997 using auditory cues, it may be tempting to dismiss the
account of impletion, but illusory visual motion can be induced via auditory
stimuli Hidaka . (2009), which demonstrates that auditory stimuli can also
induce a perception of motion in visual modality independent of prior-entry.
Despite these alternative explanations for phenomena such as the line motion
illusion, neuroscience has provided strong evidence for the existence of prior
entry: speeded processing when attention was directed to the visual modality
rather than the tactile Vibell . (2007), speeded processing associated with
attending to an auditory stimulus Folyi . (2012), and speeded processing
during a visual task when an auditory tone was presented prior to the onset of
the visual stimuli Seibold Rolke (2014).

Evidence thus suggests that prior entry, and indeed audio-visual prior entry,
is a robust phenomenon. Whether all effects attributed to prior entry are done
so correctly is another matter, but ultimately may be somewhat irrelevant (see
Fuller2009 where evidence for both prior entry and impletion in the line
motion illusion is presented, and suggests prior entry is not requisite). For
instance, even if response bias or some non-attentional processes are
mistakenly attributed to prior entry, these effects are still predictable, and
replicable, and in fact these processes may enhance or exaggerate genuine
prior entry effects.

The prior entry and impletion effects discussed above show that shifts in
attention, or the combination of separate stimuli into the perception of a
single stimulus event, can result in illusory temporal visual perception. It
seems likely that evidence gathered from both the audio and visual modalities
are combined optimally with some sources of information being given greater
weighting in this process. When and how to assign weightings in an artificial
system is an important consideration in design in order to avoid artefacts
such as those described above. While prior entry and/or impletion can result
in an inaccurate representation of temporal events, there exist audio-visual
effects that are facilitatory in nature and thus desirable, which we discuss
next.

###  2.2 Temporal ventriloquism

Illusory visual temporal order, as shown above, can be induced by auditory
stimuli. However, auditory stimuli, when integrated with visual stimuli, can
also facilitate visual temporal perception: Scheier1999 discovered an audio-
visual effect where spatially non-informative auditory stimuli affected the
temporal perception of a visual temporal order judgement task. This effect
became known as temporal ventriloquism, analogous to spatial ventriloquism
where visual stimuli shifts the perception of auditory localisation Radeau
Bertelson (1987); Willey . (1937); Thomas (1941). Temporal ventriloquism was
further investigated by Morein-Zamir2003: when accompanied by auditory tones,
performance in a visual temporal order judgement task was enhanced (Figure 2).
This enhancement was abolished when the tones coincided with the visual
stimuli in time. When the two tones were presented between the visual stimuli
in time (Figure 3), a detriment in performance was observed. In both
conditions the tones appeared to “pull” the perception of the visual stimuli
in time towards the auditory stimuli temporal onsets: further apart in the
enhanced performance and closer together when a detriment in performance was
observed (see however Hairston2006auditory for an argument against the notion
of introducing a temporal gap between the stimuli in visual perception). The
main driver of this effect was believed to be the temporal relationship
between the auditory and visual stimuli, where the higher temporal resolution
of the auditory stimuli carried more weight in integration. This is a potent
example of how assigning weightings in multi-modal integration can have both
positive and negative outcomes in terms of system performance.

Figure 2: Trial events in the temporal ventriloquism paradigm that resulted in
enhanced temporal order judgement performance Morein-Zamir . (2003). The
stimulus onset asynchrony (SOA) between illumination of LEDs varied. The first
auditory tone was presented before the first LED illumination. The second
auditory tone was presented after the second LED illumination. The SOA between
tones also varied.

Figure 3: Trial events in the temporal ventriloquism paradigm that resulted in
a detriment in temporal order judgement performance Morein-Zamir . (2003). The
stimulus onset asynchrony (SOA) between illumination of LEDs varied. The first
auditory tone was presented after the first LED illumination. The second
auditory tone was presented before the second LED illumination. The SOA
between tones also varied.

Morein-Zamir2003 hypothesized that the quantity of auditory stimuli must match
the quantity of visual stimuli in order for the temporal ventriloquism effects
to occur. For example, when a single tone was presented between the
presentation of the visual stimuli in time, there was no reported change in
performance. Morein-Zamir2003 refer to the unity assumption: the more
physically similar stimuli are to each other across modalities, the greater
the likelihood they will be perceived as having originated from the same
source Welch (1999), we discuss this in more detail later.

However, other research questions the assumption that a matching number of
stimuli in both the auditory and visual modality are required to induce
temporal ventriloquism. Getzmann2007 studied an apparent motion paradigm,
where participants perceive two sequentially presented visual stimuli behaving
as one stimulus moving from one position to another. They found that when a
single click was presented between the two visual stimuli, it increased the
perception of apparent motion, essentially “pulling” the visual stimuli closer
together in time in perception. This casts doubt on the idea that the quantity
of stimuli must be equal across modalities in order for, in this instance, an
auditory stimulus to affect the perception of visual events.

Indeed, Boyce2019 demonstrated that a detriment in response bias corresponding
to actual visual presentation order can be achieved with the presentation of a
single tone in neutral space (different space to the visual stimuli) in a
visual ternary response task (temporal order judgement combined with
simultaneity judgements where the participant reports if stimuli were
presented simultaneously). Importantly, this can be achieved consistently when
presenting the single tone prior to the onset of the first visual stimulus
(similar to the trial in Figure 2 but without the second auditory stimulus).
Often participants were as likely to make a simultaneity judgement report as
they were to make a temporal order judgement report corresponding to actual
sequential visual stimuli presentation order. This poses a problem for the
temporal ventriloquism narrative: a single tone before the sequential
presentation of visual stimuli in a ternary task would be expected to “pull”
the perception of the first visual stimulus towards it in time, resulting in
increased reports corresponding to the sequential order of visual stimuli.
Alternatively, it might “pull” the perception of both visual stimuli in time
with no observable effect on report bias should the matching quantity rule be
abandoned. The repeated demonstrations of a decrease in report bias
corresponding to the sequential order of visual stimuli suggest that the
processes underlying temporal ventriloquism may be more flexible than
previously suggested, and may have impletion-esque elements. Specifically,
characteristics of stimuli such as their spatial and temporal relationship,
and the featural similarity of stimuli within a single modality, may be
weighted to arrive at the most likely real world outcome in perception
regardless of whether the number of auditory stimuli equal the number of
visual stimuli or not. Indeed, the number of auditory stimuli relative to
visual in this example appears to modulate the type of temporal ventriloquist
effect that might be expected to be observed.

Clearly not all conditions support the idea that the temporal relationship of
an auditory stimulus to a visual stimulus drives temporal ventriloquism and
similar effects. However, while there are no easy explanations for the audio-
visual integration in temporal ventriloquism, efforts have been made to show
that auditory stimuli do indeed “pull” visual stimuli in temporal perception.
Freeman2008 created an innovative paradigm that tested the idea that temporal
ventriloquism is driven by auditory capture (in a similar fashion to that of
Getzmann2007), that is to say there is a “pulling” of visual stimuli towards
auditory stimuli in temporal perception. They began by determining the
relative timings of visual stimuli that resulted in illusory apparent visual
motion (Figure 4). Once visual stimulus onset asynchronies (SOAs) were
established for the effect, Freeman2008 adjusted the timings to remove bias in
the illusion (Figure 4b). Following that, they introduced auditory stimuli
(Figure 4c) with the same SOAs used to induce the effect in the visual-only
condition (Figure 4a). In the presence of the auditory stimuli, both visual
stimuli were “pulled” towards each other in time perception, and participants
perceived a bias in the illusion.

This demonstrated that auditory stimuli had the ability to “pull” the
respective visual stimuli in perceptual time towards the respective auditory
onsets. In doing so, the visual stimuli now appeared in perception to have the
same SOA as the auditory stimuli. This introduced a perceptual bias consistent
with that observed for the visual stimuli SOA (in the absence of auditory
stimuli) necessary to induce the same bias in illusory apparent visual motion.
This meant predictable manipulation of the effect, and more specifically,
demonstrated auditory capture of visual events in time.

Figure 4:  Illusory apparent visual motion paradigm Freeman Driver (2008); (a)
the visual SOAs (vSOA) between stimuli. ‘L’ denotes the left stimulus, and ‘R’
the right stimulus. When the vSOA was $333$ms, apparent motion in the
direction of the second stimulus was perceived. When there was a vSOA of
$666$ms, no apparent motion was perceived; (b): when vSOAs were $500$ms, there
was no bias in apparent motion. (c): when vSOAs were $500$ms but auditory
stimuli were presented with an SOA (aSOA) of $333$ms, participants perceived
apparent motion in the same direction as would be expected with a vSOA of
$333$ms. When the aSOA was $666$ms, illusory apparent visual motion was
abolished.

Freeman2008 suggest that the timings of the flanker stimuli (the stimuli used
to induce temporal ventriloquism effects) in relation to the visual are the
main drivers of temporal ventriloquism. Roseboom2013a demonstrated that, in
fact, the featural similarity of the flanker stimuli used to induce the
effects described by Freeman2008 have arguably as important a role to play at
these time scales. Specifically, Roseboom2013a replicated the findings of
Freeman2008 using auditory flankers. When flankers were featurally distinct
(e.g. a sine wave and a white-noise burst) or flanker types were mixed via
audio-tactile flankers, a mitigated effect was observed. It was significantly
weaker compared to featurally identical audio-only or tactile-only flankers.
This suggests that temporal capture in-and-of-itself is not sufficient when
describing the underlying mechanisms that account for this effect, or temporal
ventriloquism in general at the reported time scales. Roseboom2013a also
demonstrated that the reported illusory apparent visual motion could be
induced when the flanker stimuli was presented synchronously with the target
visual stimuli. This suggests that temporal ansynchrony is not a requisite to
induce this illusion in a directionally ambiguous display. Keetels2007 further
highlighted the importance of featural characteristics when inducing the
temporal ventriloquism effect. However, at shorter time scales, featural
similarity appears not to play as large a role where timing is reasserted as
the main driver Klimova . (2017); Kafaligonul Stoner (2010, 2012).

The above research is consistent with the unity assumption, where an observer
makes an assumption about two sensory signals, such as auditory and visual (or
indeed, signals from the same modality), originating from a single source or
event Vatakis Spence (2007, 2008); YC. Chen Spence (2017).
vatakis2007crossmodal demonstrated that when auditory and visual stimuli were
mismatched (for example, speech presentation where the voice did not match the
gender of the speaker) participants found it easier to judge which stimulus
was presented first; auditory or visual. The task difficulty increased when
the stimuli were matched suggesting an increased likelihood of perceiving the
auditory and visual stimuli occurring at the same time. This finding provides
support for the unity assumption in audio-visual temporal integration of
speech via the process of temporal ventriloquism. See vatakis2008evaluating
for limitations of the unity assumption’s influence over audio-visual temporal
integration of complex non-speech stimuli. See also chen2017assessing for a
thorough review of the unity assumption and the myriad debates surrounding it,
and how it relates to Bayesian causal inference.

The findings by Roseboom2013a and Keetels2007 show that there is often a much
more complex process of integration than simply auditory stimuli (or other
stimuli of high temporal resolution) capturing visual stimuli in perception.
There would appear to be a process of evidence accumulation and evidence
discounting: when two auditory events are featurally similar, and their
temporal relationship with visual stimuli is close, the auditory and visual
stimuli are integrated. However, when two auditory stimuli meet the temporal
criteria for integration with visual stimuli, but these auditory stimuli are
featurally distinct and therefore deemed to be from unique sources, they are
not wholly integrated with the visual stimuli. In the second example, some of
the accumulated temporal evidence is discounted due to evidence of unique
sources being present.

Taken together, this evidence builds a more complicated picture of temporal
ventriloquism in general, and the modulated illusory apparent visual motion
direction effect Freeman Driver (2008) in particular. Indeed, the unity
assumption potentially plays a role here when one considers the effect
featural similarity has on the apparent grouping of flankers.

###  2.3 Additional audio-visual effects

Most of the research discussed thus far has focused on the effects of audio-
visual integration on space and time perception in the visual modality. As
highlighted by the McGurk effect, audio-visual integration can also have other
surprising outcomes in perception. Shams2002 demonstrated that when a single
flash of a uniform disk was accompanied by two or more tones, participants
tended to perceive multiple flashes of the disk (Figure 5). When multiple
physical flashes were presented and accompanied by a single tone, participants
tended to perceive a single presentation of the disk Andersen . (2004). These
effects were labelled as fission in the case of illusory flashes, and fusion
in the case of illusory single presentation of the disk. Fission and fusion
differ from the likes of temporal ventriloquism and prior entry in that they
increase or decrease the quantity of perceived stimuli. After training, or
when there was a monetary incentive, qualitative differences were detectable
between illusory and physical flashes van Erp . (2013); Rosenthal . (2009).
However, the illusion persisted despite the ability to differentiate.
Similarities may be drawn between the effect reported by Shams2002 and
Shipley1964 where, when the flutter rate of an auditory signal was increased,
participants perceived an increased flicker frequency of a visual signal.
However, there was a relatively small quantitative change in flicker
frequency, whereas fission is a pronounced change in the visual percept (a
single stimulus perceived as multiple stimuli).

Figure 5: Trial events in the multiple flash illusion paradigm Shams . (2002).
A tone was presented before visual stimulus onset, and a tone was presented
after visual stimulus onset. The SOA between tones was 57ms. This paradigm
resulted in the perception of multiple flashes when in fact the visual
stimulus was presented only once for 17ms.

Neuroimaging evidence provided further insights into fission/fusion effects.
Specifically, in the presence of auditory stimuli the BOLD response in the
retinotopic visual cortex increased whether fission was perceived or not
Watkins . (2006). The inverse was true when the fusion illusion was perceived.
This suggests that the auditory and visual perceptual systems are
intrinsically linked, and reflects the additive nature of the fission illusion
and the suppressive nature of the fusion illusion that “removes” information
from visual perception (auditory stimuli has also been shown to have
suppressive effects on visual perception Hidaka Ide (2015)).

Shams2002 proposed the discontinuity hypothesis as an underlying explanation
for the fission effect: discontinuous stimuli

must be present in one modality in order to “dominate” another modality during
integration. However, as alluded to above, Andersen2004 demonstrated that this
was not the case via the fusion illusion. Fission and fusion once again align
with the ideas of impletion and the unity assumption. Consistent with the
influence featural similarity of flankers had on illusory apparent visual
motion, the fission effect was completely abolished when the tones used were
distinct from each other: one a sine wave, the other a white noise burst; or
both featurally distinct sine wave tones (a 300Hz sine wave and a 3500Hz)
Roseboom . (20131); Boyce (2016).

Another effect that seems to be governed by the featural similarity of
auditory stimuli is the stream bounce illusion. In this illusion, two uniform
circles move towards each other from opposite space, and when a tone presented
at the point of overlap of the circles differed featurally to other presented
tones, the circles are perceived to “bounce” off each other. When multiple
tones were featurally identical, the circles often appeared to cross paths and
continue on their original trajectory Sekuler . (1997); Watanabe Shimojo
(2001). Taken with the above and similar research Keetels . (2007); Cook Van
Valkenburg (2009), this suggests that auditory streaming (where a sequence of
auditory stimuli are assigned the same or differing origins) processes play an
integral role in audio-visual illusions and integration in general.

Figure 6:  Trial events in the sound induced illusory visual motion paradigm
Hidaka . (2011). The flashing visual stimulus (white bar above) was presented
with varying eccentricities from fixation (red dot) depending on the trial
condition. The auditory stimulus in the illusory condition was first presented
to one ear and panned to the opposing ear. This paradigm resulted in the
illusion of motion often in the same direction as that of the auditory
stimulus motion.

Auditory motion can also have a profound effect on visual perception where a
static flashing visual target is perceived to move in the same direction as
auditory stimuli (Figure 6, see also Hidaka2011auditory, Fracasso2013).
Perceived location of apparent motion visual stimuli can also be modulated by
auditory stimuli Teramotoa . (2012). The visual motion direction selective
brain region MT/V5 is activated in the presence of moving auditory stimuli,
suggesting processing for auditory motion occurs there Poirier . (2005), which
in turn hints at an intrinsic link between auditory and visual perceptual
systems. These effects taken together again point to evidence accumulation in
audio-visual integration as an optimal process, where different weight is
given to auditory and visual inputs.

Consideration should be lent to how and when multiple stimuli in single
modality should be grouped together as originating from a single source, or
not, before pairing with another modality. As demonstrated above from
neuropsychological evidences, and also from recent systems neuroscience
evidences Ghazanfar Schroeder (2006); Meijer . (2019), the human audio-visual
integration system appears to operate in rather complex processing steps, in
addition to the traditional thinking of hierarchical processing from single
modality. Hence, there is a need for modelling these cognitive processes. When
designing artificial cognitive systems, efforts should be made to isolate
sources of auditory and visual stimuli, and identify characteristics that
would suggest they are related events. As shown above, relying on similar
temporal signatures alone is not a robust approach when integrating signals
across modalities. The illusions discussed above are summarised in Table 1.

Table 1:  Illusions summary.

Illusion | Description
---|---
The line-motion illusion: | When one side of space is cued prior to the presentation of the entire physical line it results in
| the perception of the line being drawn from that cued side of space
Illusory temporal order I: | When a tone is presented to one side of congruent space prior to the simultaneous
| presentation of both targets in a simultaneity judgement task, illusory
sequential order is perceived
Illusory temporal order II: | When a tone is presented in neutral space prior to the simultaneous presentation of both targets in
| a ternary judgement task, and a tone is presented in neutral space after the
onset of both targets, illusory sequential
| order is perceived
Temporal ventriloquism - performance enhancement: | When a tone is presented before the first visual stimulus in a temporal order judgement sequence and a tone
| is presented after the second visual stimulus, performance is improved
Temporal ventriloquism - performance detriment I: | When a tone is presented after the first visual stimulus in a temporal order judgement sequence and a tone
| is presented before the second visual stimulus, performance is worsened
Temporal ventriloquism - performance detriment II: | When a single tone is presented before the first visual stimulus in a temporal order judgement sequence,
| response bias matching the presentation order of visual stimuli is reduced
Temporal ventriloquism - illusory apparent visual motion: | When auditory stimuli are presented in neutral space but with specific SOAs in relation to visual
| stimuli the perception of apparent motion can be modulated
Multiple flash illusion: | When two tones are presented either side of a single presentation of a circle in time, multiple flashes
| of the circle are perceived
Single flash illusion: | When a single tone is presented with multiple flashes of a circle a single presentation of the circle is
| perceived
Sound-induced illusory apparent visual motion: | When auditory stimuli are presented panning from one ear to the other in time with a flashing
| visual target, visual apparent motion is perceived

##  3 Computational Cognitive Models

We have discussed how auditory stimuli can have a pronounced effect on the
perception of visual events, and vice versa, be it temporal or qualitative in
nature. For auditory stimuli affecting the perception of visual signals, some
effects were additive, facilitatory, and others suppressive. Regardless of the
outcome, the influence of auditory stimuli on visual perception provides
evidence of complexity of audio-visual integration processes on the way to
visual perception.

These complex mechanisms of how and when auditory stimuli alter visual
perception have been clarified through computational modelling.
Chandrasekaran2017computational presents a review of computational models of
multisensory integration, categorizing the computational models into
accumulator models, probabilistic models, or neural network models. These
types of models are also typically used in single-modal perceptual decision-
making (e.g. Ratcliff1978theory, Wang2002probabilistic, Wong2006recurrent). In
this review, we will only focus on the accumulator and probabilistic models;
the neural network (connectionist) models provide finer-grained, more
biologically plausible description of neural processes, but on the behavioral
level are mostly similar to the models reviewed here Ma . (2006); Bogacz .
(2006); Wong Wang (2006); Roxin Ledberg (2008); Ma Pouget (2008); Liu .
(2009); Pouget . (2013); Ursino . (2014); Zhang . (2016); Ursino . (2019);
Meijer . (2019).

###  3.1 Accumulator models

The race model is a simple model that accounts for choice distribution and
reaction time phenomena, e.g., faster reaction times of multisensory than
unisensory stimuli Raaja (1962); Gondan Minakata (2016); Miller (2016). More
formally, the multisensory processing time $D_{AV}$ is the winner of two
channel’s processing times $D_{A}$ and $D_{V}$ for audio and visual signals:

$D_{AV}=min(D_{A},D_{V})\newline $ (1)

Another type of accumulator model, the coactivation model Schwarz (1994);
Diederich (1995), is based on the classic accumulator-type drift diffusion
model (DDM) of decision-making Stone (1960); Link (1975); Ratcliff (1978);
Ratcliff Rouder (1998). The DDM is a continuous analogue of a random walk
model Bogacz . (2006), using a drift particle with state $X$ at any moment in
time to represent a decision variable (relating in favour of one over another
choice). This is obtained through integrating noisy sensory evidence over time
in the form of a stochastic differential equation, a biased Brownian motion
equation:

$dX=Adt+cdW,\newline $ (2)

where $A$ is the stimulus signal (i.e. the drift rate), $c$ is the noise
level, and $W$ represents the stochastic Wiener process. Integration of the
sensory evidence begins from an initial point (usually origin point $0$), and
is bounded by the lower and upper decision thresholds, $-z$ and $z$
respectively. Each threshold corresponds to a decision in favour of one of the
two choices. Integration of the sensory information continues until the drift
particle encounters either the upper or the lower threshold, at which stage a
decision is made in favour of the corresponding option. The drift particle is
then reset to the origin point to allow the next decision to be processed. The
DDM response time (RT) is calculated as the time taken for the drift particle
to move from its origin point to the either of the decision thresholds and can
include a brief, fixed non-decision latency. For the simplest DDM, the RT has
a closed form analytical solution Ratcliff (1978); Bogacz . (2006):

$RT=\frac{z}{A}\tanh(\frac{Az}{c^{2}})\newline $ (3)

Similarly, the corresponding analytical solution for the DDM’s error rate (ER)
is:

$ER=\frac{1}{1+\exp(\frac{2Az}{c^{2}})}\newline $ (4)

A simple unweighted coactivation model would combine evidence from two
modalities and integrate it over time using the DDM Schwarz (1994); Diederich
(1995). For example, with unimodal sensory evidence $X_{1}$ and $X_{2}$ (e.g.,
auditory and visual information), the combined evidence is just a simple
summation over time using (2):

$X_{c}=X_{1}+X_{2}\newline $ (5)

###  3.2 Bayesian models

In contrast to these models, the Bayesian modeling framework offers an elegant
approach to modeling multisensory integration Angelaki . (2009), although they
share some similar characteristics with drift-diffusion models Bitzer .
(2014); Fard . (2017). This approach can provide optimal or near optimal
integration of multimodal sensory cues by weighting the incoming evidences
from each modality. For example, if the modalities follow a Gaussian
distribution, the optimal integration estimate $X_{c}$ is Bülthoff Yuille
(1996):

$X_{c}=\frac{k_{1}^{2}}{k_{1}^{2}+k_{2}^{2}}\widehat{X_{1}}+\frac{k_{2}^{2}}{k_{1}^{2}+k_{2}^{2}}\widehat{X_{2}}\newline
$ (6)

where $k_{1}=\frac{1}{\sigma_{1}^{2}}$ and $k_{2}=\frac{1}{\sigma_{2}^{2}}$
and $\widehat{X_{1}}$, $\widehat{X_{2}}$, $\sigma_{1}$ and $\sigma_{2}$ are
the means and standard deviations for modality 1 and 2, respectively.
Interestingly, the model can show that the combined variance (noise) will
always be less than their individual estimates when the latter are
statistically independent Alais Burr (2004). This justifies why combining the
signals help reduce the overall noise. In fact, beck2012 makes a strong case
for suboptimal inference, that the larger variability is due to deterministic,
but suboptimal computation, and that the latter, not internal or external
noise, is the major cause of variability in behaviour.

A more complex model, TWIN (time window of integration), involves a
combination of the race model and the coactivation model Colonius Diederich
(2004). Specifically, whichever modality is first registered (as in winning a
“race”), the size of the window is dynamically adapted to the level of
reliability of the sensory modality. This would ensure, for instance, that if
the less reliable modality wins the race, the window would be increased to
give the more reliable modality a relatively higher contribution in
multisensory integration. This model accounts for the illusory temporal order
induced via a tone after visual stimuli onset, where the more reliable
temporal information (auditory stimulus) dictates the perceptual outcome –
illusory temporal order Boyce . (2020).

The fission and fusion in audio-visual integration were suggested to result
from statistically optimal computational strategy Shams Kim (2010), similar to
Bayesian inference where audio-visual integration implies decisions about
weightings assigned to signals and decisions whether to integrate these
signals. Battaglia2003bayesian applied this Bayesian approach to reconcile two
seemingly separate audio-visual integration theories. The first theory, called
visual capture, is a “winner-take-all” model where the most reliable signal
(least variance) dominates, while the second theory used a maximum-likelihood
estimation to identify the weight average of the sensory input. The visual
signal was shown to be dominant because of the subject perceptual bias, but
the weighting given to auditory signals increased as visual reliability
decreased. Battaglia2003bayesian showed that Equation 6 can naturally account
for both theories by having the weights to vary based on the signals’
variances.

To study how children and adults differ in audio-visual integration,
Adams2016development also used the same Bayesian approach in addition to two
other models of audio-visual integration: a focal switching model, and a
modality-switching model. The focal switching model stochastically sampled
either auditory or visual cues based on subjects’ reports of the observed
stimulus. For the modality-switching model, the stochastically sampled cues
were probabilistically biased towards the likelihood of the stimulus being
observed. Adams2016development found that the sub-optimal switching models
modeled sensory integration in the youngest study groups best. However, the
older participants followed the partial integration of an optimal Bayesian
model.

####  3.2.1 Illusions as a by-product of optimal Bayesian integration

A variety of perceptual illusions have been shown to result from optimal
Bayesian integration of information coming from multiple sensory modalities.
In the context of sound-induced flash illusion (Figure 5), given independent
auditory and visual sensory signals $A,V$, the ideal Bayesian observer
estimates posterior probabilities of the number of source signals as a
normalized product of single-modality likelihoods $P(A|Z_{A})$ and
$P(V|Z_{V})$ and joint priors $P(Z_{A},Z_{V})$

$P(Z_{A},Z_{V}|A,V)=\frac{P(A|Z_{A})P(V|Z_{V})P(Z_{A},Z_{V})}{P(A,V)}.$ (7)

Regardless of the degree of consistency between auditory and visual stimuli,
the optimal observer (7) have been shown to be consistent with the performance
of human observers Shams, Ma Beierholm (2005). Specifically, when the
discrepancy between the auditory and visual source signals is large, human
observers rarely integrate the corresponding percepts. However, when the
source signals overlap to a large degree, the two modalities are partially
combined; in these cases the more reliable auditory modality shifts the visual
percepts, thereby leading to sound-induced flash illusion.

Existence of different causes for signals of different modalities is the key
assumption of the optimal observer model developed in Shams, Ma Beierholm
(2005), which allowed it to capture both full and partial integration of
multisensory stimuli, with the latter resulting in illusions.
Kording2007causal suggested that in addition to integration of sensory
percepts, optimal Bayesian estimation is also used to infer the causal
relationship between the signals; this was consistent with spatial
ventriloquist illusion found in human participants. Alternative Bayesian
accounts developed by Alais2004ventriloquist (using Equation 6) and Sato2007
also suggest that the spatial ventriloquist illusion stems from the near
optimal integration of spatial and auditory signals.

Evidence for optimal Bayesian integration as the primary mechanism behind
perceptual illusions comes from the paradigms involving not only audio-visual,
but also other types of information. wozny2008human applied the model of
Shams2005optimal to trimodal, audio-visuo-tactile perception, through simple
extension of Equation 7. This Bayesian integration model accounted for cross-
modal interactions observed in human participants, including touch-induced
auditory fission, and flash- and sound-induced tactile fission Wozny . (2008).
Further evidence for Bayesian integration of visual, tactile, and
proprioceptive information is provided by the rubber hand illusion Botvinick
Cohen (1998), in which a feeling of ownership of a dummy hand emerges soon
after simultaneous tactile stimulation of both the concealed own hand of the
participant and the visible dummy hand (see lush2020demand for a critique of
control methods used in the ‘rubber hand’ illusion). The optimal causal
inference model Körding . (2007) adapted for this scenario accounted for this
illusion Samad . (2015). Moreover, the model predicted that if the distance
between the real hand and the rubber hand is small, the illusion would not
require any tactile stimulation, which was also confirmed experimentally Samad
. (2015).

Finally, Bayesian integration has recently been shown to account even for
those illusions which were previously striking researchers as “anti-Bayesian”,
for the reason that the empirically observed effects had the direction
opposite to the effects predicted by optimal integration. Such “anti-Bayesian”
effects are the size-weight illusion MAK. Peters . (2016) (of the two objects
with same mass but different size, the larger object is perceived to be
lighter), and the material-weight illusion MA. Peters . (2018) (of the two
objects with the same mass and size, the denser-looking object is perceived to
be lighter). In both cases, the models explaining these two illusions involved
optimal Bayesian estimation of latent variables (e.g., density), which
affected the final estimation of weight.

Altogether, the reviewed evidence from diverse perceptual tasks illustrates
the ubiquity of optimal Bayesian integration and its role in emergence of
perceptual illusions.

####  3.2.2 Temporal dimension in Bayesian integration

Basic Bayesian modelling framework often does not come with a temporal
component, unlike dynamical models such as accumulator. However, a recent
study shows that when optimal Bayesian model is combined with the DDM, it can
provide optimal and dynamic weightings to the individual sensory modalities.
In the case of visual and vestibular integration, using an experimental setup
similar to that of Fetsch2009dynamic, Drugowitsch2014optimal found a Bayes-
optimal DDM to integrate vestibular and visual stimuli in a heading
discrimination task. It allowed the incorporation of time-variant features of
the vestibular motion, i.e. motion acceleration, and visual motion velocity.
The Bayesian framework allowed the calculation of a combined sensitivity
profile $d(t)$ from the individual stimulus sensitivities.

$d(t)=\sqrt{\frac{k_{vis}^{2}(c)}{k_{comb}^{2}(c)}{v^{2}}(t)+\frac{k_{vest}^{2}(c)}{k_{comb}^{2}(c)}{a^{2}}(t)}\newline
$ (8)

where $k_{vis}(c)$, $k_{vest}(c)$, and $k_{comb}(c)$ are the visual,
vestibular and combined stimulus sensitivities, and $v(t)$ and $a(t)$ are the
temporal sensitivities of the visual and vestibular stimuli, respectively.
Drugowitsch2014optimal found that Bayes-optimal DDM led to suboptimal
integration of stimuli when subject response times were ignored. However, when
response times were considered, the decision-making process took longer but
resulted in more accurate responses. That said, a significant limitation of
the study by Drugowitsch2014optimal and related work is that it does not
incorporate delays in information processing. More generally, current Bayesian
models do not consider how temporal delays impact sensory reliability. Delays
are particularly relevant for feedback control in the motor system and
processes like audio-visual speech because different sensory systems are
affected by different temporal delays McGrath Summerfield (1985); Jain .
(2015); Crevecoeur . (2016).

So far, the modeling approaches do not generally take into account the effects
of attention, motivation, emotion, and other “top-down” or cognitive control
factors that could potentially affect multimodal integration. However, there
are experimental studies of top-down influences, mainly attention Talsma .
(2010). More recently, Maiworm2012emotional showed that aversive stimuli could
reduce the ventriloquism effect. Bruns2014reward designed a task paradigm in
which rewards were differentially allocated to different spatial locations
(hemifields), creating a conflict between reward maximization and perceptual
reliance. The auditory stimuli were accompanied by task-irrelevant, spatially
misaligned visual stimuli. They showed that the hemifield with higher reward
had a smaller ventriloquism effect. Hence, reward expectation could modulate
multimodal integration and illusion, possibly through some cognitive control
mechanisms. Future computational studies, e.g., using reward rate analysis
Bogacz . (2006); Niyogi Wong-Lin (2013), should address how reward and
punishment are associated with such effects.

##  4 Audio-visual systems in the artificial

Multimodal integration and sensor fusion in artificial systems have been an
active research field for decades Luo Kay (1989, 2002), since using multiple
sources of information can improve artificial systems in many application
areas, including smart environments, automation systems and robotics,
intelligent transportation systems. Integration of sensory modalities to
generate a percept can occur at different stages, from low (feature) to high
(semantic) level. The integration of several sources of unimodal information
at middle and high level representations Gómez-Eguíluz . (2016); Wu . (1999)
has clear advantages: interpretability, simplicity of system design, and
avoiding the problem of increasing dimensionality of the resulting integrated
feature. Although model dependent, lower dimensionality of the feature space
typically leads to better estimates of parametric models and computationally
faster non-parametric models for a fixed amount of training data, which in
turn can reduce the number of judgement failures. However, percept integration
at the representation level lacks robustness and does not account for the way
humans integrate multisensory information Calvert . (2001); Shams, Iwaki .
(2005); Stein . (2014); watkins2006 to create these percepts Cohen (2001).
Temporal ventriloquism and the McGurk effect are just two examples of the
result of the lower-level integration of sensory modalities in humans to
create percepts, yet the differences with artificial systems go even further.
While human perceptual decision-making is based on a dynamic process of
evidence accumulation of noisy sensory information over time (see above),
artificial systems typically follow a snapshot approach, i.e. percepts are
created on the basis of instantaneous information, and only from data over
time-windows when the perception mainly unfolds over time. Therefore, we can
distinguish between decisions made over accumulated evidence, i.e. decision-
making, and decisions made following the snapshot approach, i.e.
classification, even though sometimes these two approaches are combined.

Audio-visual information integration is one of the multi-sensory mechanisms
that has increasingly attracted research interests in the design of artificial
intelligent systems. This is mainly due to the fact that humans heavily rely
on these sensing modalities, and advances in this area have been facilitated
by the high level of maturity of the individual areas involved, for instance
signal processing, speech recognition, machine learning, and computer vision.
See parisi2017computational and parisi2018neurorobotic for examples of how
human multisensory integration in spatial ventriloquism has been used to model
human-like spatial localisation responses in artificial systems in which –
given a scenario where sensor uncertainty exists in audio-visual information
streams – they propose artificial neural architectures for multisensory
integration. An interesting characteristic of audio-visual processing compared
to other multimodal systems is the fact that the information unfolds over time
for audio signals, but also for visual systems when video is considered
instead of still images. However, most of the research in artificial visual
systems follow the snapshot approach mentioned above to build percepts, while
video processing mainly focuses on integrating and updating of these
instantaneous percepts over time, which can be seen as evidence accumulation.
Like for other multimodal integration modalities, audio-visual integration in
artificial systems can be performed at different levels, although is generally
used for classification purposes, while decision-making, when performed, is
based on the accumulation of classification results. Optimal temporal
integration of visual evidence together with audio information can be prone to
the sort of illusory effects on percepts illustrated above in humans. However,
because artificial systems are designed with very specific objectives, an
emergent deviation of the measurable targets of the system would be considered
as a failure or bug of the system. Therefore, although artificial systems can
display features that could be the emergent results of the multimodal
integration, they will be regarded as failures to be avoided, and most likely
not reported in the literature. A close example related to reinforcement
learning is the reward hacking effect Amodei . (2016), where a learning agent
finds an unexpected (maybe undesired) optimal policy for a given learning
problem.

As stated earlier, multimodal integration is typically performed at high
level, as low-level integration generates higher dimensional data, thereby
increasing the difficulty of processing and analysis. Moreover, the low-level
integration of raw data can have the additional problem of combining data of
very different nature. The dimensionality problem is magnified by the massive
amount of data visual perception produces, therefore most approaches to audio-
visual processing in intelligent systems also address the problem of
integration at a middle and higher levels across diverse applications: object
and person tracking Beal . (2003); Nakadai . (2002), speaker localization and
identification Gatica-Perez . (2007), multimodal biometrics Chibelushi .
(2002), lip reading and speech recognition Guitarte-Pérez . (2005); Sumby
Pollack (1954); Luettin Thacker (1997); T. Chen (2001) and video annotation Y.
Li . (2004); Y. Wang . (2000), and others. The computational models described
above can be identified with these techniques for artificial systems, as they
generate percepts and perform decision-making on the basis of middle-level
fusion of evidence. However, some work in artificial systems deals with the
challenging problem of combining data at the signal level Fisher . (2000);
Fisher Darrell (2004). While artificial visual systems were dominated by
feature definition, extraction, and learning J. Li Allinson (2008), the
success of deep learning and convolutional neural networks in particular has
shifted the focus of computer vision research. Likewise, speech processing is
adopting this new learning paradigm, yet audio-visual speech processing with
deep learning is still based mainly on high-level integration Noda . (2015);
Deng . (2013). Although the human audio-visual processing is not fully
understood, our knowledge of the brain strongly inspires (and biases) the
design of artificial systems. Besides the well-known (yet not widely reported)
reward hacking in reinforcement learning and optimisation Amodei . (2016), to
the best of our knowledge no illusory percepts have been reported in specific-
purpose artificial systems, as they are typically situations to be avoided.

##  5 Discussion

The multi-modal integration processes and related illusions outlined above are
closely related in terms of how audition affects visual perception. Untangling
whether prior entry (whatever form it may take), impletion, temporal
ventriloquism, or featural similarity of auditory stimuli are the drivers of
audio-visual effects can be a challenge, and may be missing the bigger picture
when trying to understand how perception is arrived at in a noisy world. The
most likely explanation of the discussed effects is one of an overarching
unified process of evidence accumulation and evidence discounting. This
perspective would state that evidence is gathered via multiple modalities and
is filtered through multiple sub-processes: prior entry, auditory streaming,
impletion, and temporal ventriloquism. Two or more of these sub-processes will
often interact, with various weightings given to each process. For example,
prior entry using a single auditory cue can induce an illusion of temporal
order, but with the addition of a cue in the unattended side of space after
the presentation of both target visual stimuli, extra information in favour of
temporal order can be accumulated, which would increase the strength of the
illusion Boyce . (2020). Similarly, illusory temporal order can be induced via
spatially neutral tones (an orthogonal design), as demonstrated by Boyce2019,
which appears to combine prior entry and temporal ventriloquism, and
impletion-like processes generally. The illusory temporal order induced via
spatially neutral tones is significantly weaker when compared to the spatially
congruent audio and visual stimuli equivalent, highlighting the relative
weight given to spatial congruency. Additionally, when featurally distinct
tones are used for both of these effects, the prior entry illusory order is
preserved while the illusory order induced by spatially neutral tones is
completely abolished Boyce . (2020). This highlights how spatial information
carries greater weight than the featural information of the tones used when
the auditory and visual stimuli are spatially congruent. Conversely, it also
highlights how featural characteristics carry greater weight in the absence of
audio-visual spatial congruency.

As outlined above, temporal ventriloquism effects can also interact with
auditory streaming where the features of auditory stimuli undergo a process of
grouping, and the outcome can dictate whether the stimuli is paired or not
with visual stimuli. The mere fact that the temporal signature of stimuli is
not in-and-of-itself enough to induce an effect (at the times discussed here)
suggests that sub-processes interact across modalities. Specifically, when
auditory stimuli are not grouped in the streaming process, there is less
evidence that they belong to the same source, and in turn it is less likely
both auditory stimuli belong to the same source as the visual events. These
types of interactions taken with different outcomes in visual perception,
depending on the number of auditory stimuli used, point towards an overarching
process that fits an expanded version of impletion, or a unifying account of
impletion Boyce . (2020) (aligning with Bayesian inference), where the most
likely real world outcome is reflected in perception. The observer weighs
evidence from both modalities in multi-modal perception and also weighs
evidence within a single modality. This suggests an inherent weighted
hierarchy, where spatial, featural, and temporal information are all taken
into account.

The discussed integration processes are often statistically optimal in nature
Shams Kim (2010); Alais Burr (2004). This has implications for designing
artificial cognitive systems. An optimal approach may be an intuitive one:
minimising the average error in perceptual representation of stimuli. However,
as discussed, this approach can come with costs in terms of illusions, or
artefacts, despite a reduction in the average error. Of course, some systems
will not rely wholly on mimicking human integration of modalities, and indeed
will supersede human abilities: for example, a system may be designed to
perform multiple tasks simultaneously, something a human cannot do. However,
future research should aim to identify when an optimal approach is not
suitable in multi-modal integration.

Using illusion research in human perception as a guide, researchers could
identify and model when artefacts occur in multi-modal integration, and apply
these findings to system design. This might take the shape of modulating the
optimality of integration depending on conditions via increasing or decreasing
weightings as deemed appropriate. This approach could contribute to a database
of “prior knowledge” where specific conditions that can result in artefacts
are catalogued and can inform the degree of integration between sources in
order to avoid undesired outcomes. For instance, Roach2006resolving examines
audio-visual integration from just such a perspective using a Bayesian model
of integration, where prior knowledge of events are taken into account and a
balance between benefits and costs (optimal integration and potential
erroneous perception) of integration is reached. They examined interactions
between auditory and visual rate perception (where a judgement is made in a
single modality and the other modality is ‘ignored’) and found that there is a
gradual transition between partial cue integration and complete cue
segregation as inter-modal discrepancy increases. The Bayesian model they
implemented took into account prior knowledge of the correspondence between
audio and visual rate signals, when arriving at an appropriate degree of
integration.

Similarly, a comparison between unimodal information and the final multi-modal
integration might offer a strategy for identifying artefacts. This strategy
might be akin to the study of Sekiyama1994, which demonstrated that Japanese
participants, in contrast to their American counterparts, have a different
audio-visual strategy in the McGurk paradigm: less weight was given to
discrepant visual information, which in turn affected the integration with
auditory stimuli, ultimately resulting in a smaller McGurk effect. The inverse
was shown in the participants with cochlear implants who demonstrated a larger
McGurk effect: more weight was given to visual stimuli in general Rouger .
(2008). Magnotti2017 suggested that a causal inference (determining if audio
and visual stimuli have the same source) “type” calculation is a step in
multisensory speech perception, where some, but not all, incongruent audio-
visual speech stimuli are integrated based on the likelihood of a shared, or
separate, sources. Should that be the case, and this step is part of a near
optimal strategy, a suboptimal process — such as a comparison of unimodal
information and final multi-modal integration, or adjusting relative stimulus
feature weightings when estimating likelihoods of source — could ensure that a
McGurk-like effect is avoided. Additionally, it is worth noting the research
by driver1996enhancement who demonstrated that when there are competing
auditory speech stimuli ostensibly from the same source and a matching visual
speech stimuli from a different spatial location this has the effect of
‘pulling’ the matching auditory stimuli in perceptual space towards the visual
stimuli improving separation of the auditory streams reflected in report
accuracy.

Dynamic adjustment of prior expectations is a vital consideration when
designing local “prior knowledge” databases for artificial cognitive systems.
This is illuminated by the fact that dynamically updated prior expectations
can increase the likelihood of audio-visual integration: When congruent audio-
visual stimuli is interspersed with incongruent McGurk audio-visual stimuli,
the illusory McGurk effect emerges Gau Noppeney (2016). Essentially, when
there is a high instance of audio-visual integration due to congruent stimuli,
incongruent stimuli have a greater chance of being deemed as originating from
the same source and therefore being integrated. These behavioural results were
supported by fMRI recordings that showed the left inferior frontal sulcus
arbitrates between multisensory integration and segregation by combining top-
down prior congruent/incongruent expectations with bottom-up
congruent/incongruent cues Gau Noppeney (2016). This suggests that in
artificial cognitive systems, even though the prior knowledge databeses should
cater for updates, it should not be done so live and “in the wild”. If the
probability of audio and visual stimuli originating from the same source was
calculated near-optimally in the manner described by Gau2016, dynamically it
could result in artefacts in an artificial cognitive system, where, for
example, unrelated audio-visual events could be classified as being
characteristics of the same event. If a dynamic approach is required, an
optimal strategy should be avoided for these reasons.

In addition to the approaches suggested above, it is important that temporal
characteristics, such as processing differences across artificial modalities
are also taken into account. For instance, even though light is many hundreds
of thousands of times faster than sound, the human perceptual system processes
sound stimuli faster than visual stimuli Recanzone (2009). Indeed, it has been
suggested that characteristics such as processing speeds of auditory and
visual stimuli changing as a person ages (for example, visual processing
slowing) may be responsible for increasing audio-visual integration in older
participants where auditory tones had a greater influence on the perceived
number of flashes in the sound-induced flash illusion compared to younger
participants DeLoss . (2013); McGovern . (2014). Similar considerations should
be made for artificial systems. Regardless of how sensitive or fast at
processing a given artificial sensor is, light will always reach a sensor
before sound if the respective stimuli originate from the same
distance/location. Setting aside the physical attribute of the speed of light
versus the speed of sound, there is an additional level of complexity even in
an artificial system where it presumably would require a lot more
computational power, and thus time, to process and separate the stimulus of
interest in a given visual scene (with other factors such as feature
resolution playing a role). Indeed, as mentioned previously, temporal feedback
delay in the nervous system is a factor in optimal multi-sensory integration
Crevecoeur . (2016). Additionally, a unimodal auditory strategy for separating
sources of auditory stimuli in a noisy environment via extracting and
segregating temporally coherent features into separate streams has been
developed by Krishnan . (2014). These considerations taken with the multi-
modal audio-visual strategies deployed in speech (where temporal relationships
of mouth movement and auditory onset play a role, specifically the voice
onsets between 100 and 300ms before the mouth visibly moves Chandrasekaran .
(2009)) highlight the importance of temporal characteristics, correlations,
and strategies when designing artificial cognitive systems. Finally, to handle
noisy sensory information, artificial cognitive systems should perhaps
consider incorporating temporal integration of sensory evidence Yang . (2017);
Rañó . (2017); Mi . (2019) instead of employing snapshot decision processing.

In summary, we highlight a wide range of audio-visual illusory percepts from
the psychological and neuroscience literature, and discussed how computational
cognitive models can account for some of these illusions – through seemingly
optimal multimodal integration. We provide cautions regarding the naïve
adoption of these human multimodal integration computations for artificial
cognitive systems, which may lead to unwanted artefacts. Further
investigations of the mechanisms of multimodal integration in humans and
machines can lead to efficient approaches for mitigating and avoiding unwanted
artefacts in artificial cognitive systems.

## Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any
commercial or financial relationships that could be construed as a potential
conflict of interest.

## Author Contributions

WPB wrote the Introduction, Audio-visual Integration, and Discussion sections.
AL, AZ, and KFW-L wrote the Computational Cognitive Models section. IR wrote
the Audio-visual Systems in the Artificial section. All authors contributed to
structuring, revising, and proofreading of the manuscript.

## Funding

This research was supported by the Northern Ireland Functional Brain Mapping
Project (1303/101154803), funded by Invest NI and the University of Ulster
(KFW-L), The Royal Society (KFW-L, IR), and ASUR (1014-C4-Ph1-071) (KFW-L, IR,
AL).

## Acknowledgements

The authors are also grateful to Shufan Yang for helpful discussions on the
computational modelling.

## References

  * Adams (2016) Adams2016developmentAdams, WJ.  2016\.  The development of audio-visual integration for temporal judgements The development of audio-visual integration for temporal judgements.  PLoS Comput Biol124e1004865.
  * Alais Burr (2004) Alais2004ventriloquistAlais, D. Burr, D.  2004\.  The ventriloquist effect results from near-optimal bimodal integration The ventriloquist effect results from near-optimal bimodal integration.  Current biology143257–262.
  * Amodei . (2016) Amodei2016concreteAmodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J. Mané, D.  2016\.  Concrete problems in AI safety Concrete problems in ai safety.  arXiv preprint arXiv:1606.06565.
  * Andersen . (2004) Andersen2004Andersen, TS., Tiippana, K. Sams, M.  2004\.  Factors influencing audiovisual fission and fusion illusions Factors influencing audiovisual fission and fusion illusions.  Cognitive Brain Research21301–308.
  * Angelaki . (2009) Angelaki2009multisensoryAngelaki, DE., Gu, Y. DeAngelis, GC.  2009\.  Multisensory integration: psychophysics, neurophysiology, and computation Multisensory integration: psychophysics, neurophysiology, and computation.  Current opinion in neurobiology194452–458.
  * Battaglia . (2003) Battaglia2003bayesianBattaglia, PW., Jacobs, RA. Aslin, RN.  2003\.  Bayesian integration of visual and auditory signals for spatial localization Bayesian integration of visual and auditory signals for spatial localization.  JOSA A2071391–1397.
  * Beal . (2003) beal03graphicalBeal, MJ., Jojic, N. Attias, H.  2003\.  A graphical model for audiovisual object tracking A graphical model for audiovisual object tracking.  IEEE Transactions on Pattern Analysis and Machine Intelligence257828-836.
  * Beck . (2012) beck2012Beck, JM., Ma, WJ., Pitkow, X., Latham, PE. Pouget, A.  2012\.  Not noisy, just wrong: the role of suboptimal inference in behavioral variability Not noisy, just wrong: the role of suboptimal inference in behavioral variability.  Neuron74130–39.
  * Bitzer . (2014) bitzer2014perceptualBitzer, S., Park, H., Blankenburg, F. Kiebel, SJ.  2014\.  Perceptual decision making: drift-diffusion model is equivalent to a Bayesian model Perceptual decision making: drift-diffusion model is equivalent to a bayesian model.  Frontiers in human neuroscience8102.
  * Bogacz . (2006) Bogacz2006physicsBogacz, R., Brown, E., Moehlis, J., Holmes, P. Cohen, JD.  2006\.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks.  Psychological review1134700.
  * Botvinick Cohen (1998) botvinick1998rubberBotvinick, M. Cohen, J.  199802\.  Rubber hands ‘feel’ touch that eyes see Rubber hands ‘feel’ touch that eyes see.  Nature3916669756-756.  10.1038/35784
  * Boyce (2016) BoycePhD2016Boyce, WP.  2016\.  Perception of space and time across modalities. Perception of space and time across modalities. Doctoral dissertation.  Swansea University.
  * Boyce . (2020) Boyce2019Boyce, WP., Whiteford, S., Curran, W., Freegard, G. Weidemann, CT.  2020\.  Splitting Time: Sound-Induced Illusory Visual Temporal Fission and Fusion Splitting time: Sound-induced illusory visual temporal fission and fusion.  Journal of Experimental Psychology: Human Perception and Performance..  10.1037/xhp0000703
  * Bruns . (2014) Bruns2014rewardBruns, P., Maiworm, M. Röder, B.  2014\.  Reward expectation influences audiovisual spatial integration Reward expectation influences audiovisual spatial integration.  Attention, Perception, & Psychophysics7661815–1827.
  * Bülthoff Yuille (1996) Bulthoff1996bayesianBülthoff, HH. Yuille, AL.  1996\.  A Bayesian framework for the integration of visual modules A bayesian framework for the integration of visual modules.  Attention and performance XVI: Information integration in perception and communication49–70.
  * Cairney (1975) Cairney1975Cairney, PT.  1975\.  The complication experiment uncomplicated The complication experiment uncomplicated.  Perception4(3)255-265.
  * Calvert . (2001) calvert2001detectionCalvert, GA., Hansen, PC., Iversen, SD. Brammer, MJ.  2001\.  Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the bold effect.  Neuroimage142427–438.
  * Chandrasekaran (2017) Chandrasekaran2017computationalChandrasekaran, C.  2017\.  Computational principles and models of multisensory integration Computational principles and models of multisensory integration.  Current Opinion in Neurobiology4325–34.
  * Chandrasekaran . (2009) Chandrasekaran2009Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A. Ghazanfar, AA.  2009\.  The natural statistics of audiovisual speech The natural statistics of audiovisual speech.  PLoS computational biology57e1000436.
  * T. Chen (2001) chen10audiovisualChen, T.  2001\.  Audiovisual Speech Processing. Lip Reading and Lip Synchronization Audiovisual speech processing. lip reading and lip synchronization.  IEEE Signal Processing Magazine1819-21.
  * YC. Chen Spence (2017) chen2017assessingChen, YC. Spence, C.  2017\.  Assessing the role of the ‘unity assumption’on multisensory integration: A review Assessing the role of the ‘unity assumption’on multisensory integration: A review.  Frontiers in psychology8445.
  * Chibelushi . (2002) chibelushi02reviewChibelushi, CC., Deravi, F. Mason, JS.  2002\.  A review of speech-based bimodal recognition A review of speech-based bimodal recognition.  IEEE transactions on multimedia4123-37.
  * Cohen (2001) cohen01multimodalCohen, M.  2001\.  Multimodal Integration – A Biological View Multimodal integration – a biological view.  Proceedings of the $15^{th}$ International Conference on Arificial Intelligence Proceedings of the $15^{th}$ international conference on arificial intelligence ( 1417-1424).
  * Colonius Diederich (2004) colonius2004multisensoryColonius, H. Diederich, A.  2004\.  Multisensory interaction in saccadic reaction time: a time-window-of-integration model Multisensory interaction in saccadic reaction time: a time-window-of-integration model.  Journal of cognitive neuroscience1661000–1009.
  * Cook Van Valkenburg (2009) Cook2009Cook, LA. Van Valkenburg, DL.  2009\.  Audio-visual organisation and the temporal ventriloquism effect between grouped sequences: evidence that unimodal grouping precedes cross-modal integration. Audio-visual organisation and the temporal ventriloquism effect between grouped sequences: evidence that unimodal grouping precedes cross-modal integration.  Perception38(8)1220-1233.
  * Crevecoeur . (2016) Crevecoeur2016Crevecoeur, F., Munoz, DP. Scott, SH.  2016\.  Dynamic multisensory integration: somatosensory speed trumps visual accuracy during feedback control Dynamic multisensory integration: somatosensory speed trumps visual accuracy during feedback control.  Journal of Neuroscience36338598–8611.
  * de Dieuleveult . (2017) de2017effectsde Dieuleveult, AL., Siemonsma, PC., van Erp, JB. Brouwer, AM.  2017\.  Effects of aging in multisensory integration: a systematic review Effects of aging in multisensory integration: a systematic review.  Frontiers in aging neuroscience980.
  * DeLoss . (2013) deloss2013multisensoryDeLoss, DJ., Pierce, RS. Andersen, GJ.  2013\.  Multisensory integration, aging, and the sound-induced flash illusion. Multisensory integration, aging, and the sound-induced flash illusion.  Psychology and aging283802.
  * Deng . (2013) deng13newDeng, L., Hinton, G. Kingsbury, B.  2013\.  New types of deep neural network learning for speech recognition and related applications: An overview New types of deep neural network learning for speech recognition and related applications: An overview.  IEEE International Conference on Acoustics, Speech and Signal Processing Ieee international conference on acoustics, speech and signal processing ( 8599-8603).
  * Diederich (1995) Diederich1995intersensoryDiederich, A.  1995\.  Intersensory facilitation of reaction time: Evaluation of counter and diffusion coactivation models Intersensory facilitation of reaction time: Evaluation of counter and diffusion coactivation models.  Journal of Mathematical Psychology392197–215.
  * Downing Treisman (1997) Downing1997Downing, PE. Treisman, AM.  1997\.  The line–motion illusion: Attention or impletion?. The line–motion illusion: Attention or impletion?.  Journal of Experimental Psychology: Human Perception and Performance23768–779.
  * Driver (1996) driver1996enhancementDriver, J.  1996\.  Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading.  Nature381657766–68.
  * Drugowitsch . (2014) Drugowitsch2014optimalDrugowitsch, J., DeAngelis, GC., Klier, EM., Angelaki, DE. Pouget, A.  2014\.  Optimal multisensory decision-making in a reaction-time task Optimal multisensory decision-making in a reaction-time task.  Elife3e03005.
  * Fard . (2017) fard2017bayesianFard, PR., Park, H., Warkentin, A., Kiebel, SJ. Bitzer, S.  2017\.  A Bayesian Reformulation of the Extended Drift-Diffusion Model in Perceptual Decision Making A bayesian reformulation of the extended drift-diffusion model in perceptual decision making.  Frontiers in computational neuroscience1129.
  * Fetsch . (2009) Fetsch2009dynamicFetsch, CR., Turner, AH., DeAngelis, GC. Angelaki, DE.  2009\.  Dynamic reweighting of visual and vestibular cues during self-motion perception Dynamic reweighting of visual and vestibular cues during self-motion perception.  The Journal of Neuroscience294915601–15612.
  * Fisher Darrell (2004) fisher04speakerFisher, J. Darrell, T.  2004\.  Speaker association with signal-level audiovisual fusion Speaker association with signal-level audiovisual fusion.  IEEE Transactions on Multimedia63406-413.
  * Fisher . (2000) fisher00learningFisher, J., Darrell, T., Freeman, W. Viola, P.  2000\.  Learning joint statistical models for audio-visual fusion and segregation Learning joint statistical models for audio-visual fusion and segregation.  Neural Information Processing Systems Neural information processing systems ( 772-778).
  * Fitzpatrick McCloskey (1994) Fitzpatrick1994proprioceptiveFitzpatrick, R. McCloskey, D.  1994\.  Proprioceptive, visual and vestibular thresholds for the perception of sway during standing in humans. Proprioceptive, visual and vestibular thresholds for the perception of sway during standing in humans.  The Journal of physiology478Pt 1173.
  * Folyi . (2012) Folyi2012Folyi, T., Fehér, B. Horváth, J.  2012\.  Stimulus-focused attention speeds up auditory processing. Stimulus-focused attention speeds up auditory processing.  International Journal of Psychophysiology84(2)155-163.
  * Fracasso . (2013) Fracasso2013Fracasso, A., Targher, S., Zampini, M. Melcher, D.  2013\.  Fooling the Eyes: The Influence of a Sound-Induced Visual Motion Illusion on Eye Movements. Fooling the eyes: The influence of a sound-induced visual motion illusion on eye movements.  PLOS One8(4): e62131doi:10.1371/journal.pone.0062131.
  * Freeman Driver (2008) Freeman2008Freeman, E. Driver, J.  2008\.  Direction of visual apparent motion driven solely by timing of a static sound. Direction of visual apparent motion driven solely by timing of a static sound.  Current biology181262–1266.
  * Fulbright . (2001) Fulbright2001functionalFulbright, RK., Troche, CJ., Skudlarski, P., Gore, JC. Wexler, BE.  2001\.  Functional MR imaging of regional brain activation associated with the affective experience of pain Functional mr imaging of regional brain activation associated with the affective experience of pain.  American Journal of Roentgenology17751205–1210.
  * Fuller Carrasco (2009) Fuller2009Fuller, S. Carrasco, M.  2009\.  Perceptual consequences of visual performance fields: The case of the line motion illusion Perceptual consequences of visual performance fields: The case of the line motion illusion.  Journal of vision9413–13.
  * Gatica-Perez . (2007) gatica07audiovisualGatica-Perez, D., Lathoud, G., Odobez, J. McCowan, I.  2007\.  Audiovisual probabilistic tracking of multiple speakers in meetings Audiovisual probabilistic tracking of multiple speakers in meetings.  IEEE Transactions on Audio, Speech, and Language Processing152601-616.
  * Gau Noppeney (2016) Gau2016Gau, R. Noppeney, U.  2016\.  How prior expectations shape multisensory perception How prior expectations shape multisensory perception.  NeuroImage124876–886.
  * Getzmann (2007) Getzmann2007Getzmann, S.  2007\.  The effect of brief auditory stimuli on visual apparent motion. The effect of brief auditory stimuli on visual apparent motion.  Perception361089.
  * Ghazanfar Schroeder (2006) ghazanfar2006neocortexGhazanfar, AA. Schroeder, CE.  2006\.  Is neocortex essentially multisensory? Is neocortex essentially multisensory?  Trends in cognitive sciences106278–285.
  * Goldstein (2008) Goldstein2010Goldstein, EB.  2008\.  Sensation and perception. Sensation and perception. (8th ).  Belmont, CalifThomson Wadsworth.
  * Gómez-Eguíluz . (2016) gomez16multimodalGómez-Eguíluz, A., Rañó, I., Coleman, S. T.M.McGinnity.  2016\.  A multi-modal approach to continuous material identification through tactile sensing A multi-modal approach to continuous material identification through tactile sensing.  IEEE/RSJ International Conference on Intelligent Robots and Systems. Ieee/rsj international conference on intelligent robots and systems.
  * Gondan Minakata (2016) Gondan2016tutorialGondan, M. Minakata, K.  2016\.  A tutorial on testing the race model inequality A tutorial on testing the race model inequality.  Attention, Perception, & Psychophysics783723–735.
  * Green (2004) Green2004temperatureGreen, BG.  2004\.  Temperature perception and nociception Temperature perception and nociception.  Journal of neurobiology61113–29.
  * Guitarte-Pérez . (2005) perez05lipGuitarte-Pérez, J., Frangi, A., Lleida-Solano, E. Lukas, K.  2005\.  Lip reading for robust speech recognition on embedded devices Lip reading for robust speech recognition on embedded devices.  Proceedings of the International Conference on Acoustics, Speech, and Signal Processing Proceedings of the international conference on acoustics, speech, and signal processing ( 473-476).
  * Hairston . (2006) Hairston2006auditoryHairston, WD., Hodges, DA., Burdette, JH. Wallace, MT.  2006\.  Auditory enhancement of visual temporal order judgment Auditory enhancement of visual temporal order judgment.  Neuroreport178791–795.
  * Hidaka Ide (2015) Hidaka2015soundHidaka, S. Ide, M.  2015\.  Sound can suppress visual perception Sound can suppress visual perception.  Scientific reports5.
  * Hidaka . (2009) Hidaka2009Hidaka, S., Manaka, Y., Teramoto, W., Sugita, Y., Miyauchi, R., Gyoba, J.Iwaya, Y.  2009\.  The alternation of sound location Induces visual motion perception of a static object The alternation of sound location induces visual motion perception of a static object.  PLoS One4: e8188doi: 10.1371/journal.pone.0008188.
  * Hidaka . (2011) Hidaka2011auditoryHidaka, S., Teramoto, W., Sugita, Y., Manaka, Y., Sakamoto, S. Suzuki, Y.  2011\.  Auditory motion information drives visual motion perception Auditory motion information drives visual motion perception.  PLoS One63e17499.
  * Hikosaka . (19931) Hikosaka1993Hikosaka, O., Miyauchi, S. Shimojo, S.  19931\.  Focal visual attention produces illusory temporal order and motion sensation. Focal visual attention produces illusory temporal order and motion sensation.  Vision research331219–1240.
  * Hikosaka . (19932) Hikosaka1993aHikosaka, O., Miyauchi, S. Shimojo, S.  19932\.  Voluntary and stimulus-induced attention detected as motion sensation. Voluntary and stimulus-induced attention detected as motion sensation.  Perception22517–526.
  * Holmes (2009) holmes2009principleHolmes, NP.  2009\.  The principle of inverse effectiveness in multisensory integration: some statistical considerations The principle of inverse effectiveness in multisensory integration: some statistical considerations.  Brain topography213-4168–176.
  * Jain . (2015) jain2015comparativeJain, A., Bansal, R., Kumar, A. Singh, K.  2015\.  A comparative study of visual and auditory reaction times on the basis of gender and physical activity levels of medical first year students A comparative study of visual and auditory reaction times on the basis of gender and physical activity levels of medical first year students.  International Journal of Applied and Basic Medical Research52124.
  * Kafaligonul Stoner (2010) Kafaligonul2010auditoryKafaligonul, H. Stoner, GR.  2010\.  Auditory modulation of visual apparent motion with short spatial and temporal intervals Auditory modulation of visual apparent motion with short spatial and temporal intervals.  Journal of vision101231–31.
  * Kafaligonul Stoner (2012) Kafaligonul2012staticKafaligonul, H. Stoner, GR.  2012\.  Static sound timing alters sensitivity to low-level visual motion Static sound timing alters sensitivity to low-level visual motion.  Journal of vision12112–2.
  * Keetels . (2007) Keetels2007Keetels, M., Stekelenburg, J. Vroomen, J.  2007\.  Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism. Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism.  Experimental Brain Research180449-456.
  * Klimova . (2017) Klimova2017groupingKlimova, M., Nishida, S. Roseboom, W.  2017\.  Grouping by feature of cross-modal flankers in temporal ventriloquism Grouping by feature of cross-modal flankers in temporal ventriloquism.  Scientific reports717615.
  * Kolers von Grünau (1976) Kolers1976shapeKolers, PA. von Grünau, M.  1976\.  Shape and color in apparent motion Shape and color in apparent motion.  Vision research164329–335.
  * Körding . (2007) Kording2007causalKörding, KP., Beierholm, U., Ma, WJ., Quartz, S., Tenenbaum, JB. Shams, L.  2007\.  Causal inference in multisensory perception Causal inference in multisensory perception.  PLoS one29e943.
  * Krishnan . (2014) Krishnan2014Krishnan, L., Elhilali, M. Shamma, S.  2014\.  Segregating complex sound sources through temporal coherence Segregating complex sound sources through temporal coherence.  PLoS computational biology1012e1003985.
  * J. Li Allinson (2008) li08comprehensiveLi, J. Allinson, N.  2008\.  A comprehensive review of current local features for computer vision A comprehensive review of current local features for computer vision.  Neurocomputing7110-121771-1787.
  * Y. Li . (2004) li04contentLi, Y., Narayanan, S. Kuo, CJ.  2004\.  Content-based movie analysis and indexing based on audiovisual cues Content-based movie analysis and indexing based on audiovisual cues.  IEEE Transactions on Circuits and Systems for Video Technology1481073-1085.
  * Link (1975) Link1975relativeLink, SW.  1975\.  The relative judgment theory of two choice response time The relative judgment theory of two choice response time.  Journal of Mathematical Psychology121114–135.
  * Liu . (2009) liu2009dynamicalLiu, YS., Yu, A. Holmes, P.  2009\.  Dynamical analysis of Bayesian inference models for the Eriksen task Dynamical analysis of bayesian inference models for the eriksen task.  Neural computation2161520–1553.
  * Luettin Thacker (1997) luettin97speechreadingLuettin, J. Thacker, N.  1997\.  Speechreading using probabilistic models Speechreading using probabilistic models.  Computer Vision and Image Understanding652163-178.
  * Luo Kay (1989) luo89multisensorLuo, R. Kay, M.  1989\.  Multisensor Integration and Fusion in Intelligent Systems Multisensor integration and fusion in intelligent systems.  IEEE Transactions on Systems, Man, and Cibernetics195.
  * Luo Kay (2002) luo02multisensorLuo, R. Kay, M.  2002\.  Multisensor Fusion and Integration: Approaches, Applications, and Future Research Directions Multisensor fusion and integration: Approaches, applications, and future research directions.  IEEE Sensors Journal22.
  * Lush (2020) lush2020demandLush, P.  2020\.  Demand characteristics confound the rubber hand illusion Demand characteristics confound the rubber hand illusion.  Collabra: Psychology6(1)22.  http://doi.org/10.1525/collabra.325
  * Ma . (2006) ma2006bayesianMa, WJ., Beck, JM., Latham, PE. Pouget, A.  2006\.  Bayesian inference with probabilistic population codes Bayesian inference with probabilistic population codes.  Nature neuroscience9111432.
  * Ma Pouget (2008) ma2008linkingMa, WJ. Pouget, A.  2008\.  Linking neurons to behavior in multisensory perception: A computational review Linking neurons to behavior in multisensory perception: A computational review.  Brain research12424–12.
  * Macaluso . (2000) Macaluso2000modulationMacaluso, E., Frith, CD. Driver, J.  2000\.  Modulation of human visual cortex by crossmodal spatial attention Modulation of human visual cortex by crossmodal spatial attention.  Science28954821206–1208.
  * Magnotti Beauchamp (2017) Magnotti2017Magnotti, JF. Beauchamp, MS.  2017\.  A Causal Inference Model Explains Perception of the McGurk Effect and Other Incongruent Audiovisual Speech A causal inference model explains perception of the mcgurk effect and other incongruent audiovisual speech.  PLoS computational biology132e1005229.
  * Maiworm . (2012) Maiworm2012emotionalMaiworm, M., Bellantoni, M., Spence, C. Röder, B.  2012\.  When emotional valence modulates audiovisual integration When emotional valence modulates audiovisual integration.  Attention, Perception, & Psychophysics7461302–1311.
  * McGovern . (2014) mcgovern2014soundMcGovern, DP., Roudaia, E., Stapleton, J., McGinnity, TM. Newell, FN.  2014\.  The sound-induced flash illusion reveals dissociable age-related effects in multisensory integration The sound-induced flash illusion reveals dissociable age-related effects in multisensory integration.  Frontiers in aging neuroscience6250.
  * McGrath Summerfield (1985) mcgrath1985intermodalMcGrath, M. Summerfield, Q.  1985\.  Intermodal timing relations and audio-visual speech recognition by normal-hearing adults Intermodal timing relations and audio-visual speech recognition by normal-hearing adults.  The Journal of the Acoustical Society of America772678–685.
  * McGurk MacDonald (1976) McGurkMacDonald1976McGurk, H. MacDonald, J.  1976\.  Hearing lips and seeing voices. Hearing lips and seeing voices.  Nature264746 - 748.
  * Meijer . (2019) meijer2019circuitMeijer, GT., Mertens, PE., Pennartz, CM., Olcese, U. Lansink, CS.  2019\.  The circuit architecture of cortical multisensory processing: Distinct functions jointly operating within a common anatomical network The circuit architecture of cortical multisensory processing: Distinct functions jointly operating within a common anatomical network.  Progress in neurobiology1741–15.
  * Mi . (2019) mi2019spatiotemporalMi, Y., Lin, X., Zou, X., Ji, Z., Huang, T. Wu, S.  2019\.  Spatiotemporal Information Processing with a Reservoir Decision-making Network Spatiotemporal information processing with a reservoir decision-making network.  arXiv preprint arXiv:1907.12071.
  * Miller (2016) Miller2016statisticalMiller, J.  2016\.  Statistical facilitation and the redundant signals effect: What are race and coactivation models? Statistical facilitation and the redundant signals effect: What are race and coactivation models?  Attention, Perception, & Psychophysics782516–519.
  * Morein-Zamir . (2003) Morein-Zamir2003Morein-Zamir, S., Soto-Faraco, S. Kingstone, A.  2003\.  Auditory capture of vision: examining temporal ventriloquism. Auditory capture of vision: examining temporal ventriloquism.  Cognitive Brain Research17154-163.
  * Nakadai . (2002) nakadai02realNakadai, K., Hidai, K. Okuno, H.  2002\.  Real-time speaker localization and speech separation by audio-visual integration Real-time speaker localization and speech separation by audio-visual integration.  Proceedings of the IEEE International Conference on Robotics and Automation Proceedings of the ieee international conference on robotics and automation ( 1).
  * Niyogi Wong-Lin (2013) niyogi2013dynamicNiyogi, RK. Wong-Lin, K.  2013\.  Dynamic excitatory and inhibitory gain modulation can produce flexible, robust and optimal decision-making Dynamic excitatory and inhibitory gain modulation can produce flexible, robust and optimal decision-making.  PLoS computational biology96e1003099.
  * Noda . (2015) noda15audioNoda, K., Yamaguchi, Y., Nakadai, K., Okuno, H. Ogata, T.  2015\.  Audio-visual speech recognition using deep learning Audio-visual speech recognition using deep learning.  Applied Intelligence42722-737.
  * Parisi . (2018) parisi2018neuroroboticParisi, GI., Barros, P., Fu, D., Magg, S., Wu, H., Liu, X. Wermter, S.  2018\.  A neurorobotic experiment for crossmodal conflict resolution in complex environments A neurorobotic experiment for crossmodal conflict resolution in complex environments.  2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2018 ieee/rsj international conference on intelligent robots and systems (iros) ( 2330–2335).
  * Parisi . (2017) parisi2017computationalParisi, GI., Barros, P., Kerzel, M., Wu, H., Yang, G., Li, Z.Wermter, S.  2017\.  A computational model of crossmodal processing for conflict resolution A computational model of crossmodal processing for conflict resolution.  2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob) 2017 joint ieee international conference on development and learning and epigenetic robotics (icdl-epirob) ( 33–38).
  * MA. Peters . (2018) peters2018materialweightPeters, MA., Zhang, LQ. Shams, L.  201810\.  The material-weight illusion is a Bayes-optimal percept under competing density priors The material-weight illusion is a Bayes-optimal percept under competing density priors.  PeerJ6e5760.  10.7717/peerj.5760
  * MAK. Peters . (2016) peters2016sizeweightPeters, MAK., Ma, WJ. Shams, L.  201606\.  The Size-Weight Illusion is not anti-Bayesian after all: a unifying Bayesian account The Size-Weight Illusion is not anti-Bayesian after all: a unifying Bayesian account.  PeerJ4e2124.  10.7717/peerj.2124
  * Poirier . (2005) Poirier2005Poirier, C., Collignon, O., DeVolder, AG., Renier, L., Vanlierde, A., Tranduy, D. Scheiber, C.  2005\.  Specific activation of the V5 brain area by auditory motion processing: An fMRI study Specific activation of the v5 brain area by auditory motion processing: An fmri study.  Cognitive Brain Research25650–658.
  * Pouget . (2013) pouget2013probabilisticPouget, A., Beck, JM., Ma, WJ. Latham, PE.  2013\.  Probabilistic brains: knowns and unknowns Probabilistic brains: knowns and unknowns.  Nature neuroscience1691170.
  * Raaja (1962) Raab1962statisticalRaaja, D.  1962\.  Statistical facilitation of simple reaction time Statistical facilitation of simple reaction time.  Transctions of the New York Academy of Sciences24574–590.
  * Radeau Bertelson (1987) Radeau1987Radeau, M. Bertelson, P.  1987\.  Auditory-visual interaction and the timing of inputs. Auditory-visual interaction and the timing of inputs.  Psychological Research4917–22.
  * Ramos-Estebanez . (2007) Ramos2007visualRamos-Estebanez, C., Merabet, LB., Machii, K., Fregni, F., Thut, G., Wagner, TA.Pascual-Leone, A.  2007\.  Visual phosphene perception modulated by subthreshold crossmodal sensory stimulation Visual phosphene perception modulated by subthreshold crossmodal sensory stimulation.  The Journal of Neuroscience27154178–4181.
  * Rañó . (2017) rano2017driftRañó, I., Khamassi, M. Wong-Lin, K.  2017\.  A drift diffusion model of biological source seeking for mobile robots A drift diffusion model of biological source seeking for mobile robots.  2017 IEEE International Conference on Robotics and Automation (ICRA) 2017 ieee international conference on robotics and automation (icra) ( 3525–3531).
  * Rao . (2001) Rao2001evolutionRao, SM., Mayer, AR. Harrington, DL.  2001\.  The evolution of brain activation during temporal processing The evolution of brain activation during temporal processing.  Nature neuroscience43317–323.
  * Ratcliff (1978) Ratcliff1978theoryRatcliff, R.  1978\.  A theory of memory retrieval. A theory of memory retrieval.  Psychological review85259.
  * Ratcliff Rouder (1998) Ratcliff1998modelingRatcliff, R. Rouder, JN.  1998\.  Modeling response times for two-choice decisions Modeling response times for two-choice decisions.  Psychological Science95347–356.
  * Recanzone (2009) recanzone2009interactionsRecanzone, GH.  2009\.  Interactions of auditory and visual stimuli in space and time Interactions of auditory and visual stimuli in space and time.  Hearing research2581-289–99.
  * Roach . (2006) Roach2006resolvingRoach, NW., Heron, J. McGraw, PV.  2006\.  Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration.  Proceedings of the Royal Society of London B: Biological Sciences27315982159–2168.
  * Roseboom . (20131) Roseboom2013bRoseboom, W., Kawabe, T. Nishida, SY.  20131\.  The cross-modal double flash illusion depends on featural similarity between cross-modal inducers. The cross-modal double flash illusion depends on featural similarity between cross-modal inducers.  Scientific reports31–5.
  * Roseboom . (20132) Roseboom2013aRoseboom, W., Kawabe, T. Nishida, SY.  20132\.  Direction of visual apparent motion driven by perceptual organization of cross-modal signals. Direction of visual apparent motion driven by perceptual organization of cross-modal signals.  Journal of Vision131–13.
  * Rosenthal . (2009) Rosenthal2009Rosenthal, O., Shimojo, S. Shams, L.  2009\.  Sound-Induced Flash Illusion is Resistant to Feedback Training. Sound-induced flash illusion is resistant to feedback training.  Brain topography21185-192.
  * Rouger . (2008) Rouger2008Rouger, J., Fraysse, B., Deguine, O. Barone, P.  2008\.  McGurk effects in cochlear-implanted deaf subjects Mcgurk effects in cochlear-implanted deaf subjects.  Brain research118887–99.
  * Roxin Ledberg (2008) roxin2008neurobiologicalRoxin, A. Ledberg, A.  2008\.  Neurobiological models of two-choice decision making can be reduced to a one-dimensional nonlinear diffusion equation Neurobiological models of two-choice decision making can be reduced to a one-dimensional nonlinear diffusion equation.  PLoS Computational Biology43e1000046.
  * Samad . (2015) samad2015perceptionSamad, M., Chung, AJ. Shams, L.  201502\.  Perception of Body Ownership Is Driven by Bayesian Sensory Inference Perception of Body Ownership Is Driven by Bayesian Sensory Inference.  PLOS ONE102e0117178.  10.1371/journal.pone.0117178
  * Sato . (2007) Sato2007Sato, Y., Toyoizumi, T. Aihara, K.  2007\.  Bayesian inference explains perception of unity and ventriloquism aftereffect: identification of common sources of audiovisual stimuli. Bayesian inference explains perception of unity and ventriloquism aftereffect: identification of common sources of audiovisual stimuli.  Neural computation19(12)3335-3355.
  * Scheier . (1999) Scheier1999Scheier, C., Nijhawan, R. Shimojo, S.  1999\.  Sound alters visual temporal resolution Sound alters visual temporal resolution.  Investigative Ophthalmology & Visual Science Investigative ophthalmology & visual science ( 40, S792–S792).
  * Schneider Bavelier (2003) Schneider2003Schneider, KA. Bavelier, D.  2003\.  Components of visual prior entry. Components of visual prior entry.  Cognitive psychology47(4)333-366.
  * Schwarz (1994) Schwarz1994diffusionSchwarz, W.  1994\.  Diffusion, superposition, and the redundant-targets effect Diffusion, superposition, and the redundant-targets effect.  Journal of Mathematical Psychology384504–520.
  * Seibold Rolke (2014) Seibold2014Seibold, VC. Rolke, B.  2014\.  Does temporal preparation speed up visual processing? Evidence from the N2pc. Does temporal preparation speed up visual processing? evidence from the n2pc.  Psychophysiology51(6)529-538.
  * Sekiyama (1994) Sekiyama1994Sekiyama, K.  1994\.  Differences in auditory-visual speech perception between Japanese and Americans: McGurk effect as a function of incompatibility. Differences in auditory-visual speech perception between japanese and americans: Mcgurk effect as a function of incompatibility.  Journal of the Acoustical Society of Japan15(3)143-158.
  * Sekuler . (1997) Sekuler1997soundSekuler, R., Sekuler, AB. Lau, R.  1997\.  Sound alters visual motion perception. Sound alters visual motion perception.  Nature3856614308.
  * Shams, Iwaki . (2005) shams2005earlyShams, L., Iwaki, S., Chawla, A. Bhattacharya, J.  2005\.  Early modulation of visual cortex by sound: an MEG study Early modulation of visual cortex by sound: an meg study.  Neuroscience letters378276–81.
  * Shams . (2002) Shams2002Shams, L., Kamitani, Y. Shimojo, S.  2002\.  Visual illusion induced by sound. Visual illusion induced by sound.  Cognitive Brain Research14147-152.
  * Shams Kim (2010) Shams2010Shams, L. Kim, R.  2010\.  Crossmodal influences on visual perception. Crossmodal influences on visual perception.  Physics of life reviews7(3)269-284.
  * Shams, Ma Beierholm (2005) Shams2005optimalShams, L., Ma, WJ. Beierholm, U.  2005\.  Sound-induced flash illusion as an optimal percept Sound-induced flash illusion as an optimal percept.  Neuroreport16171923–1927.
  * Shimojo . (1997) Shimojo1997Shimojo, S., Miyauchi, S. Hikosaka, O.  1997\.  Visual motion sensation yielded by non-visually driven attention. Visual motion sensation yielded by non-visually driven attention.  Vision Research371575–1580.
  * Shipley (1964) Shipley1964Shipley, T.  1964\.  Auditory flutter-driving of visual flicker Auditory flutter-driving of visual flicker.  Science1328–1330.
  * Spence . (2001) SpenceKelinShore2001Spence, C., Shore, DI. Klein, RM.  2001\.  Multisensory prior entry. Multisensory prior entry.  Journal of Experimental Psychology130(4)799-831.
  * Stein . (2014) stein2014developmentStein, BE., Stanford, TR. Rowland, BA.  2014\.  Development of multisensory integration from the perspective of the individual neuron Development of multisensory integration from the perspective of the individual neuron.  Nature Reviews Neuroscience158520–535.
  * Stevenson James (2009) stevenson2009audiovisualStevenson, RA. James, TW.  2009\.  Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition.  Neuroimage4431210–1223.
  * Stone (1960) Stone1960modelsStone, M.  1960\.  Models for choice-reaction time Models for choice-reaction time.  Psychometrika253251–260.
  * Sumby Pollack (1954) sumby54visualSumby, W. Pollack, I.  1954\.  Visual contribution to speech intelligibility in noise Visual contribution to speech intelligibility in noise.  Journal of the Acoustical Society of America262212-215.
  * Talsma . (2010) Talsma2010multifacetedTalsma, D., Senkowski, D., Soto-Faraco, S. Woldorff, MG.  2010\.  The multifaceted interplay between attention and multisensory integration The multifaceted interplay between attention and multisensory integration.  Trends in cognitive sciences149400–410.
  * Teramotoa . (2012) Teramoto2012Teramotoa, W., Hidaka, S., Sugita, Y., Sakamoto, S., Gyoba, J., Iwaya, Y. Suzuki, Y.  2012\.  Sounds can alter the perceived direction of a moving visual object Sounds can alter the perceived direction of a moving visual object.  Journal of Vision121–12.
  * Thomas (1941) Thomas1941Thomas, GJ.  1941\.  Experimental study of the influence of vision on sound localization. Experimental study of the influence of vision on sound localization.  Journal of Experimental Psychology282163.
  * Ursino . (2014) ursino2014neurocomputationalUrsino, M., Cuppini, C. Magosso, E.  2014\.  Neurocomputational approaches to modelling multisensory integration in the brain: a review Neurocomputational approaches to modelling multisensory integration in the brain: a review.  Neural Networks60141–165.
  * Ursino . (2019) ursino2019explainingUrsino, M., Cuppini, C., Magosso, E., Beierholm, U. Shams, L.  2019\.  Explaining the Effect of Likelihood Manipulation and Prior Through a Neural Network of the Audiovisual Perception of Space Explaining the effect of likelihood manipulation and prior through a neural network of the audiovisual perception of space.  Multisensory research322111–144.
  * van Erp . (2013) vanErp2013van Erp, JB., Philippi, TG. Werkhoven, P.  2013\.  Observers can reliably identify illusory flashes in the illusory flash paradigm. Observers can reliably identify illusory flashes in the illusory flash paradigm.  Experimental Brain Research22673–79.
  * Vatakis Spence (2007) vatakis2007crossmodalVatakis, A. Spence, C.  2007\.  Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli.  Perception & psychophysics695744–756.
  * Vatakis Spence (2008) vatakis2008evaluatingVatakis, A. Spence, C.  2008\.  Evaluating the influence of the ‘unity assumption’on the temporal perception of realistic audiovisual stimuli Evaluating the influence of the ‘unity assumption’on the temporal perception of realistic audiovisual stimuli.  Acta Psychologica127112–23.
  * Vibell . (2007) Vibell2007Vibell, J., Klinge, C., Zampini, M., Spence, C. Nobre, AC.  2007\.  Temporal order is coded temporally in the brain: early event-related potential latency shifts underlying prior entry in a cross-modal temporal order judgment task. Temporal order is coded temporally in the brain: early event-related potential latency shifts underlying prior entry in a cross-modal temporal order judgment task.  Journal of Cognitive Neuroscience19(1)109-120.
  * XJ. Wang (2002) Wang2002probabilisticWang, XJ.  2002\.  Probabilistic decision making by slow reverberation in cortical circuits Probabilistic decision making by slow reverberation in cortical circuits.  Neuron365955–968.
  * Y. Wang . (2000) wang00multimediaWang, Y., Liu, Z. Huang, JC.  2000\.  Multimedia content analysis-using both audio and visual clues Multimedia content analysis-using both audio and visual clues.  IEEE signal processing magazine17612-36.
  * Watanabe Shimojo (2001) Watanabe2001Watanabe, K. Shimojo, S.  2001\.  When sound affects vision: effects of auditory grouping on visual motion perception. When sound affects vision: effects of auditory grouping on visual motion perception.  Psychological Science12(2)109-116.
  * Watkins . (2006) Watkins2006Watkins, S., Shams, L., Tanaka, S., Haynes, JD. Rees, G.  2006\.  Sound alters activity in human V1 in association with illusory visual perception. Sound alters activity in human v1 in association with illusory visual perception.  NeuroImage311247–1256.
  * Welch (1999) Welch1999Welch, RB.  1999\.  Meaning, attention, and the unity assumption in the intersensory bias of spatial and temporal perceptions. Meaning, attention, and the unity assumption in the intersensory bias of spatial and temporal perceptions.  Advances in psychology129371-387.
  * Willey . (1937) Willey1937Willey, CF., Inglis, E. Pearce, C.  1937\.  Reversal of auditory localization. Reversal of auditory localization.  Journal of Experimental Psychology202114.
  * Wong Wang (2006) Wong2006recurrentWong, KF. Wang, XJ.  2006\.  A recurrent network mechanism of time integration in perceptual decisions A recurrent network mechanism of time integration in perceptual decisions.  The Journal of neuroscience2641314–1328.
  * Wozny . (2008) wozny2008humanWozny, DR., Beierholm, UR. Shams, L.  200803\.  Human trimodal perception follows optimal statistical inference Human trimodal perception follows optimal statistical inference.  Journal of Vision8324-24.  10.1167/8.3.24
  * Wu . (1999) wu99multimodalWu, L., Oviatt, S. Cohen, P.  1999\.  Multimodal Integration – A Statistical View Multimodal integration – a statistical view.  Transactions on Multimedia14.
  * Yang . (2017) yang2017singleYang, S., Wong-Lin, K., Rano, I. Lindsay, A.  2017\.  A single chip system for sensor data fusion based on a Drift-diffusion model A single chip system for sensor data fusion based on a drift-diffusion model.  2017 Intelligent Systems Conference (IntelliSys) 2017 intelligent systems conference (intellisys) ( 198–201).
  * Zampini . (2005) Zampini2005Zampini, M., Shore, DI. Spence, C.  2005\.  Audiovisual prior entry Audiovisual prior entry.  Neuroscience letters381(3)217-222.
  * Zhang . (2016) zhang2016decentralizedZhang, WH., Chen, A., Rasch, MJ. Wu, S.  2016\.  Decentralized multisensory information integration in neural systems Decentralized multisensory information integration in neural systems.  Journal of Neuroscience362532–547.