# Reconfigurable Computation in Spiking Neural Networks

Fabio Schittler Neves  Chair for Network Dynamics, Center for Advancing
Electronics Dresden (cfaed), Dresden, Germany  Institute for Theoretical
Physics, TU Dresden, 01062 Dresden, Germany<EMAIL_ADDRESS>Marc
Timme  Chair for Network Dynamics, Center for Advancing Electronics Dresden
(cfaed), Dresden, Germany  Institute for Theoretical Physics, TU Dresden,
01062 Dresden, Germany  Cluster of Excellence Physics of Life, TU Dresden,
01062 Dresden, Germany  Lakeside Labs, 9020 Klagenfurt am Wörthersee, Austria
<EMAIL_ADDRESS>
###### Abstract

The computation of rank ordering plays a fundamental role in cognitive tasks
and offers a basic building block for computing arbitrary digital functions.
Spiking neural networks have been demonstrated to be capable of identifying
the largest $k$ out of $N$ analog input signals through their collective
nonlinear dynamics. By finding partial rank orderings, they perform
$k$-winners-take-all computations. Yet, for any given study so far, the value
of $k$ is fixed, often to $k$ equal one. Here we present a concept for spiking
neural networks that are capable of (re)configurable computation by choosing
$k$ via one global system parameter. The spiking network acts via pulse-
suppression induced by inhibitory pulse-couplings. Couplings are proportional
to each units’ state variable (neuron voltage), constituting an uncommon but
straightforward type of leaky integrate-and-fire neural network. The result of
a computation is encoded as a stable periodic orbit with $k$ units spiking at
some frequency and others at lower frequency or not at all. Orbit stability
makes the resulting analog-to-digital computation robust to sufficiently small
variations of both, parameters and signals. Moreover, the computation is
completed quickly within a few spike emissions per neuron. These results
indicate how reconfigurable $k$-winners-take-all computations may be
implemented and effectively exploited in simple hardware relying only on basic
dynamical units and spike interactions resembling simple current leakages to a
common ground.

Analog computing, coupled oscillators, network dynamics, nonlinear dynamics,
spiking neural networks, winner-takes-all, heteroclinic dynamics

##  I Introduction

Rank ordering of signals plays a fundamental role in natural and artificial
cognitive computations, in particular in attention-related tasks
Bridewell2016theory ; McKinstry2016Imagery ; Rabinovich2013neural  where a
small set of relevant information must be computed in real time from an array
of sensory inputs. Natural environments present a continuous stream of
concurrent analog signals that often are simultaneously time-dependent, multi-
modal and high-dimensional. Yet such complex signals typically provide a basis
for discrete decisions.

Figure 1:  Reconfigurable $k$-winners-take-all computation in a schematic
network of spiking neurons. Tuning one global parameter, the coupling strength
$\epsilon$, selects $k$. A vector $\xi$, with
$\xi_{1}>\xi_{2}>\xi_{3}>\dots>\xi_{N}$ (gray arrows) serves as an input to a
network of spiking neurons (black circles) that dynamically computes a 2-WTA
function of the input vector $\xi$. Only those neurons receiving the largest
two inputs send spikes whereas spikes from all other neurons are dynamically
suppressed.

Partial rank ordering, determining a subset of strongest signals by comparing
a variety of inputs, offers a fundamental option for providing a discrete
judgment. Finding the largest $k$ out of a set of $N$ analog inputs defines a
$k$-winners-take-all ($k$-WTA) computation, which networks of spiking (formal)
neurons have been shown to perform maass2000computation ; Neves2012computation
via different approaches. While bio-inspired networks of pulse-coupled neurons
can compute one winner-takes-all (1-WTA) functions via a combination of local
excitation and longer range inhibition chen2017mechanisms , e.g. bump states
Laing2001stationary  or dynamical neural fields Sandamirskaya2014dynamics ,
symmetrical networks of pulse-coupled neurons or phase-coupled oscillators may
compute more general $k$-WTA functions exploiting complex periodic orbits akin
to heteroclinic dynamics Krupa1997robust ; rabinovich2001dynamical ;
Timme2002prevalence ; Timme2003unstable ; Ashwin2004encoding ; Ashwin2005 ;
Ashwin2005when ; Neves2012computation ; Wordsworth2008spatiotemporal .
Moreover, a recent study Wang2019wta  also provides an actual simple hardware
implementation of a WTA neural-circuit with promising scalability via careful
network design, also a mix of excitation and inhibition.

In all these studies, however, either the number of winners $k$ is fixed, hard
to design, thus not easily (re)configurable, or computations typically take
many spikes per unit or both rabinovich2001dynamical ; Timme2002prevalence ;
Timme2003unstable ; Ashwin2005 ; rabinovich2006dynamical ;
Wordsworth2008spatiotemporal ; Neves2012computation ; Neves2017Noise . Here,
we propose a class of spiking neural network models that not only computes
fast, in as few as $k$ spikes under ideal conditions (no noise and all
starting from the same voltage), but moreover provides intrinsic
(re)configurability via changes to a single global parameter to perform
$k$-WTA computations for different $k$, see Fig. 1 for a schematic
illustration.

We thus present a system that is directly (re)configurable by setting a single
parameter to different values in order to directly switch between different
well-defined computational tasks. Such reconfiguration is in stark contrast to
learning processes in neural networks during which many parameters (coupling
weights) evolve in a self-organized way to adapt the system to become capable
of solving a given task.

##  II Spiking neural networks with proportional inhibition

We consider a network of $N$ spiking neurons that are oscillatory, i.e. send
spikes periodically in the absence of coupling and external signals. The
neurons’ continuous-time dynamics is a singular type of integrate-and-fire
neurons maass1999Pulse  and is characterized by a one-dimensional phase-like
variable as in the generic model introduced in Mirollo1990synchronization .
Each neural unit $i\in\\{1,\ldots,N\\}$ exhibits a voltage-like state variable
$x_{i}$ satisfying the differential equations

$\frac{dx_{i}}{dt}=I_{i}-\gamma x_{i}+\xi_{i}(t)+\eta_{i}(t)+G(x_{i},t),$ (1)

where

$G(x_{i},t)=\sum_{j=1}^{N}\sum_{t_{j,\ell}\in
P_{j}}g(x_{i})\delta(t-t_{j,\ell}),$ (2)

for $x_{i}\in[0,x^{\theta})$ where $x^{\theta}$ is a spiking threshold. The
parameter $I_{i}$ represents a constant current, $\gamma$ is a dissipation
parameter, $\xi_{i}$ an external signal serving as input and $\eta_{i}(t)$
represents an independent Gaussian white noise signal which strength is
completely determined by its signal variance $\sigma^{2}$. Numerically, by
definition, the noise contribution within any given small time-interval
$\Delta t$ is a random real number with zero mean and variance $(\Delta
t)\sigma^{2}$. Thus, one random number is generated at each integration
interval and summed to the resulting voltage. To avoid numerically driven
synchronization, noise is added at randomly generated times
hansel1998numerical ; Timme2003unstable ; Klinglmayr2012Guaranteeing  (201
points per time unit per neuron) drawn independently from a Poisson
distribution. Whenever a threshold is reached $x_{i}(t^{-})=x^{\theta}$, the
state variable is reset to $x_{i}(t)=0$ and a pulse (spike) is sent to all
other neurons, mathematically reflected in the time $t=t_{j,\ell}$ of the
$\ell$th threshold crossing, $\ell\in\mathbb{Z}$, by the neuron $j$. Without
loss of generality we fixed $x^{\theta}=1$. The sum in (2) is the contribution
of all spikes arriving from the other $N-1$ neurons $j$ to $i$ at time $t$,
where $P_{j}$ is the set of all times $t_{j,\ell}$ of spikes sent by neuron
$j$. We propose to design the coupling function $g(x_{j})$, also known as
sensitivity function in the research field of coupled phase oscillators
strogatz2000kuramoto ; Winfree1967biological , to be state dependent such that
it is inhibitory with amplitude proportional to the voltage $x_{i}$ of the
receiving neuron at times $t_{j,\ell}^{-}$, i.e.

$g(x_{i})=-\epsilon x_{i}$ (3)

where $\epsilon<1$ is the coupling strength. As a consequence, the larger the
voltage $x_{i}$, the larger the inhibitory effect of a received spike signal.
To the best of our knowledge, this model setting has not been analysed or
exploited for solving computational tasks (via spiking neural networks).

Figure 2:  Effect of inhibitory spike reception. The continuous time evolution
of state variables (voltages) $x_{i}(t)$ as a function of time $t$. Each line
represents the voltage of one neuron $i$. When some neuron $j$ (dashed line)
reaches the threshold, $x_{j}(t^{-}_{s})=1$, it is reset, $x_{j}(t_{s})=0$,
and a spike is immediately sent to all other neurons. Because the spike
response strength is proportional to the voltage of receiving neurons
$x_{i}(t_{s})=x_{i}(t^{-}_{s})-\epsilon x_{i}(t^{-}_{s})$, their voltages will
be both decreased, $x_{i}(t_{s})<x_{i}(t^{-}_{s})$ and more strongly
synchronized than before the spike reception, i.e. $d_{2}=(1-\epsilon)d_{1}$
where $d(t)=\max_{i,i^{\prime}\neq j}|x_{i}(t)-x_{i^{\prime}}(t)|$,
$d_{1}=d(t_{s}^{-})$ and $d_{2}=d(t_{s})$. Parameters for this illustration
are $N=5$ and $\epsilon=0.2$.

For illustration, we here focus on networks with fixed parameters $\gamma=1$
and $I_{i}=1.04=:I$. We vary the magnitudes of the input signals $\xi_{i}(t)$
and the coupling strength $\epsilon$ to select different $k$ of the $k$-WTA
computation by achieving different collective dynamics. Furthermore, as
concrete examples, we present networks of sizes $N=5$ and $N=50$ to emphasize
different collective dynamics’ aspects and the pulse-suppression mechanism’s
independence of the network size.

##  III $k$-winners-take-all via Pulse-Suppression

How may spiking neural networks with pulse-suppression perform $k$-WTA
functions? Its overall dynamics is dictated by a simple mechanism: every time
a neuron reaches the firing threshold, it emits a spike and inhibits all other
neurons proportionally to their voltages, thereby bringing their state
variables $x_{i}$ closer together, compare $d_{2}$ to $d_{1}$ in Fig. 2. The
neurons’ voltages $x_{i}$ do not synchronize identically in finite time
(unless they are initialized identically) because any resulting voltage
difference $x_{i}(t)-x_{i^{\prime}}(t)$ is a positive fraction $(1-\epsilon)$
of the original $x_{i}(t^{-})-x_{i^{\prime}}(t^{-})$. Furthermore, in the
absence of external signals or noise, a neuron will also never overtake
another Mirollo1990synchronization ; Kielblock2011Overtaking . This implies
that all neurons repetitively reach the threshold sequentially one after the
other and send a spike, see Fig. 3a. The order in which they reach the
threshold is determined by the ordering in their initial condition, i.e. their
voltages at time zero. Thus, all neurons will exhibit the same spiking
frequency in the absence of external signals and noise.

Figure 3:  Dynamics of 2-winners-take-all computation via pulse-suppression.
The voltage and spike times for all neurons in the network. Each curve
represents the voltage dynamics of one neuron and each dash represents the
time of an elicited spike. (a) shows the dynamics of all five neurons in the
absence of external signals ($\xi(t)=(0,0,0,0,0)^{\intercal}$) such that all
units spike periodically and sequentially. (b) Network dynamics when an
asymmetric external signal $\xi(t)=(\xi_{1},\ldots,\xi_{5})^{\mathsf{T}}$ with
$\xi_{1}>\xi_{2}>\xi_{3}>\xi_{4}>\xi_{5}$ is turned on. In the presence of
external signals only neurons 1 and 2 spike (periodically and sequentially).
Neurons 1 and 2 (blue curves) repeatedly overtake and inhibit neurons 3, 4 and
5 (black curves). Thus promoting 2-wta via pulse suppression. Parameters are
$\epsilon=0.5$ and $\Delta\xi=0.02$.

In the presence of noise (and still without external signals), sporadic
overtaking becomes possible. Some neurons may by chance skip a spike emission
(and a reset) due to inhibition. These events are random and the average
number of spikes emitted by all neurons are still (almost) the same for any
sufficiently large time interval, thus assuming identical average frequencies
in the theoretical limit of infinite observation times.

Figure 4:  (Re)configurable choice of computational task. The computed $k$
depends on the differences $\Delta\xi$ and the coupling strength $\epsilon$.
For a small network of five neurons, we observed five different response
regions in the given parameter range: white, there is not ranking $k=5$;
computing $k\in\\{1,2,3,4\\}$ are represented by different shades of gray or
black. For any fixed input signal (horizontal value), a sorting type ($k$) can
be selected by changing the global parameter $\epsilon$.

The external signals $\xi_{i}(t)$ that we define as inputs may induce much
richer and more interesting collective network dynamics. If input signals with
components of different intensities are concurrently applied to all neurons,
neurons exhibit different intrinsic frequencies for the duration of the signal
and the faster neurons may repeatedly overtake others, see Fig. 3b. As a
result, some neurons may never reach the threshold and deliver a spike, i.e.
they will always skip their turn, (Fig. 3b). We may thus interprete that the
system evaluates some aspects of the external continuous-time analog input
signal to make a decision about which spike emissions to skip and which to
keep. As we will show below, the system performs a $k$-winners-take-all
computation with $k$ depending on a combination of the network parameters and
the magnitude of the difference between input signal’s components.

Figure 5:  Monotonic decrease in computed $k$ with increasing $\epsilon$ for a
network of $N=50$ neurons. Each dash represents the time of one spike. For
these simulations, inputs were fixed and linearly distributed between $0$ and
$0.003$ ($\Delta\xi=6\times 10^{-5}$) and $\epsilon$ was switched to a new
value every 50 time units. The inset shows how $k$ scales with $\epsilon$.

Whether and how many neurons will skip their turn depends on two factors –
driving signals and coupling strength. First, we observe that if neurons are
driven with input signals of different average values, neurons overtake each
other (Fig. 3b). If the input signals are constant in time, the neurons $i$
receiving the larger driving signals $\xi_{i}$ exhibit faster voltage changes
and are thereby candidates for being neurons with larger output spiking
frequencies. We remark that even if a neuron exhibits a fast change in
voltages it may not elicit any spike if inhibited, and the voltage is lowered
by means of leakage rather than spikes delivered to the network. Details of
their dynamics depend on how fast (or if at all) the neurons driven more
strongly can catch up with more weakly driven ones ahead of them and overtake
these. Second, the coupling strength $\epsilon$ determines how close all
neurons’ voltage variables are squeezed together during a spike reception.

Whereas the first factor, different rates of change $dx_{i}/dt$ of the
neurons’ voltages, provides a simple mechanism for spreading the network
voltages proportionally to their total input current into $N$ different
voltage values, the effect of inhibitory spikes is more convoluted. It has two
aspects: first, arriving spikes provide a synchronizing force for the group of
$(N-1)$ neurons concurrently receiving the spike (Fig. 2); second, whether the
voltage difference from a neuron $j$ to the spike-sending neuron $i$ will
increase or decrease depends on the phase $x_{j}(t_{s}^{-})$ at the spike time
and $\epsilon$. For example, for $x_{j}(t_{s}^{-})$ close to one (almost
synchronized) most $\epsilon$ increase the voltage difference
$|x_{i}(t_{s})-x_{j}(t_{s})|$ as $x_{i}(t_{s}^{-})\rightarrow 0$ and thus
decrease the degree of synchrony; on the other hand, for initially large
differences $|x_{i}(t_{s}^{-})-x_{j}(t_{s}^{-})|$, where $x_{j}(t_{s}^{-})$ is
near 0, all $\epsilon$ would increase synchrony. Furthermore, large values of
$\epsilon$ provides a global synchronization force, e.g. if $\epsilon$ close
to one all $x_{j}(t_{s})$ are close to zero independent of $x_{j}(t_{s})$.
Moreover, whereas the first factor (different $dx_{i}/dt$) is acting
continuously in time, the spike interaction is only acting at a discrete set
of spike reception times. These observations imply that an intricate interplay
of overall spiking frequency, differences in input signal strengths and
coupling strength determines which type of output spike pattern results and
how many neurons spike in the presence of a given input signal.

To quantify the contribution of these two factors, we simulate a system of
$N=5$ neurons varying the global coupling strength $\epsilon$ and the degree
of separation of input signal components. Specifically, we generate temporally
constant signals of the form

$\xi_{i}=(N-i)\times\Delta\xi,\qquad i\in\\{1,\ldots,N\\},$ (4)

where $\Delta\xi$ is a constant that characterizes the input signal. The
smallest input difference between any two inputs therefore equals $\Delta\xi$.
For all other system parameters fixed as above, we find that the system is
capable of computing any given $k$, that is $k\in\\{1,2,3,4,5\\}$ neurons
spiking repeatedly whereas $N-k$ neurons remain silent without spikes. Fig. 4
shows that for a single parameter $\epsilon$, the system will perform the same
computation for a broad range of input differences $\Delta\xi$. Furthermore,
for the entire range of $\Delta\xi$ considered the system can compute $k$-WTA
for all $k\in\\{1,\ldots,5\\}$, depending on the coupling strength $\epsilon$.
The computation is thus reconfigurable in this sense. We remark that
$\epsilon$ needs to be sufficiently large, otherwise no neuron is capable of
overtaking another and thus $k=5$, i.e. no ranking is performed. Finally, in
Fig. 5 we present a larger network example with $N=50$. As expected, we found
that the mechanism described above applies in the same manner, and the global
parameter $\epsilon$ controls the computed $k$. Specifically, $k$ decreases
monotonically as $\epsilon$ increases. Furthermore, every $k\leq N$ is in
principle possible for sufficiently well-tuned $\epsilon$.

Figure 6:  Example of noise generating randomness in spike sequences and its
stochastic effects.. The system is set to compute the $k=3$ largest out of 5
inputs, as in (4). (a) For very small noise ($\text{SNR}=100$), the system
seems not affected by noise, as only the three stronger inputs induce spikes
in this sample; (b) as noise increases ($\text{SNR}=1$), some predicted spikes
are missed, replaced by false positives; (c) strong noise ($\text{SNR}=0.01$)
generates a large number of false positives (spikes) and negatives, rendering
the computation more difficult. (d) The fraction of spikes sent from each
neuron in relation to the total network’s number of spikes for different SNRs
(single sample) counted for $t\in[0,2000]$. Even though the number of false
positives increases with noise, the result of a computational task can still
be solved via the average spike rate up to $\text{SNR}=0.01$. For example, for
$\text{SNR}\leq 0.001$ the system computes the wrong ranking and thus fails.
Parameters used: $N=5$, $\epsilon=0.3$ and $\Delta\xi=0.003$

As the voltages of many neurons may approach the threshold almost at the same
time, see Fig. 3, it is important to understand the concurrent effect of noise
and input signals. We here consider a Gaussian white noise signal and inputs
as in (4) for a $N=5$ network. In particular, we compute which fraction of all
spikes comes from which neurons during a 3-WTA computation for different noise
levels. To quantify noise in our system we define a signal-to-noise measure as

$\mbox{SNR}=\frac{\Delta\xi^{2}}{\sigma^{2}},$ (5)

where sigma is the standard deviation of the Gaussian signal and $\Delta\xi$
characterizes our signal strength as in (4). Let us consider the limiting
cases first: in the absence of noise ($\sigma^{2}\rightarrow 0$), we expect
that the two neurons subject to the weaker inputs would not elicit any spike
while the other three neurons exhibit the same fraction ($\approx 33\%$) of
spikes; for strong noise ($\sigma^{2}\gg\Delta\xi$), we expect that most of
the information about the input is lost, thus all neurons contribute roughly
with the same $\approx 20\%$ of elicited spikes. As shown in Fig. 6, our
numerical results corroborate those predictions. Furthermore, as the signal to
noise ratio (SNR) is reduced, the previously silent neurons start to send a
fraction of the spikes inversely proportional to the SNR. This is particularly
interesting, because it shows a form of robustness, that the computation can
still be achieved up to SNRs approximately $0.01$ (Fig. 6d), if the system
computes with the average number of spikes, for example by imposing a
threshold on the firing rates as a readout.

So far, we have shown that the system introduced above is (re)configurable and
how it responds to noise. Adjusting the coupling strength selects some $k$ for
a $k$-WTA computation via changes in the spike rates. A fundamental feature of
spiking systems it that a major part of consumed power is due to spike
generation, i.e. the smaller the number of spikes the smaller the power
consumed. Let us now briefly estimate how many spikes our system requires to
compute at different SNR.

The number of spikes $n^{*}$ required to rank the driving current from a pair
of neurons, say $i$ and $j$ depends on how fast their individual numbers of
generated spikes $n_{i}(t)$ and $n_{j}(t)$ diverge as function of time.
Specifically, the difference $\Delta n_{ij}(t)=n_{i}(t)-n_{j}(t)$ encodes the
rank result in its magnitude and sign, e.g. $\Delta n_{ij}(t)>0$ suggests that
$\xi_{i}>\xi_{j}$ and $\Delta n_{ij}(t)<0$ indicates that $\xi_{i}<\xi_{j}$.
We remark that the sign of $\Delta n_{ij}(t)$ alone is not enough to determine
which neuron received the stronger input, due to variability caused by noise
and initial conditions. We thus numerically estimate an expected failure rate
function $F(\sigma,\Delta\xi,n(t))$ measuring how often $\Delta n_{ij}(t)$
indicates either the wrong signal order, e.g. $\Delta n_{ij}(t)>0$ for
$\xi_{i}<\xi_{j}$, or is inconclusive ($\Delta n_{ij}(t)=0$) after a total of
$n(t)$ network spikes, calculated over 500 simulations per parameter set. The
minimum number $n^{*}$ of spikes from the whole network to compute the partial
rank order with an acceptable failure rate $\delta$, noise standard deviation
$\sigma$ and input amplitude $\Delta\xi$, is then estimated as

$n^{*}(\sigma,\Delta\xi,\delta)=\min\\{n^{\prime}\,|\,F(\sigma,\Delta\xi,n)<\delta\,\text{
for all }\,n>n^{\prime}\\}.$ (6)

The quantity $n^{*}$ thus define the minimum (total) number of spikes after
which the failure rate will never exceed $\delta$ afterwards within a much
larger number of spikes, interval $[0,2000]$.

Figure 7:  Number of spikes required to compute with a coupled and an
uncoupled systems. (a) For small SNR, the coupled system computes with few
spikes, consistent across different input magnitudes and fixed SNR, thus is
robuts against $\Delta\xi$ changes. (b) the uncoupled system requires a large
variety of number of spikes ($n^{*}$) for the same SNR, depending on the input
strength (results for coupled system in green). Because all $N=5$ neurons have
similar spike frequencies, an oscillation with period close to 5 is observed,
as $\Delta n_{ij}$ has a larger probability to be zero for all $i,j$ pairs at
times in which $n(t)$ is a multiple of five (one spike per unit). Deviations
on this period are cause by spike frequency variability (overtakes). (c) The
coupled system consistently exhibits a smaller number of spikes and a smaller
variation at the same SNR when compared to the uncoupled system. At small SNRs
and ideal initial conditions ($V_{i}=0$ for all $i$), in red, only the
absolute minimum number of spikes 3 (n=k) is required. Parameters used are
$\epsilon=0.3$ and $\delta=0.002$.

As an example, we consider a $N=5$ network set to compute a 3-WTA ranking over
five inputs chosen as in (4). In this experiment $k=3$ is known and we are
interested on comparing the three winners to the two losers. Among those
comparisons the smallest difference among input strengths is
$\xi_{3}-\xi_{4}=\Delta\xi$, between neurons 3 and 4. Their difference in
spike frequency determines the overall computing time because it dictates the
longest relevant computing time in the process. Notice that the in-group
differences are irrelevant for the desired computation. Moreover, the in-group
rankings may take longer than determining the three stronger inputs and not
possible to compute for large SNRs. For reference, we compare our results to a
set of independent integrators. For clarity, we consider the exact same model
of neurons described above, but now uncoupled, thus stressing the
computational aspects provided by the network dynamics rather than model
specific features.

Let us first consider the limit case of no noise ($\sigma\rightarrow 0$) and
an ideal initial condition where all neurons start at the same voltage
$V_{i}(0)=0$. In this limit, in the coupled system, neuron 4 does not spike
and the number of spikes to compute is simply $n=3$, one spike from each
winner, and decoding the resulting spike trains reduces to detecting the first
three spikes. On the other hand, the number of spikes to compute with the
uncoupled system is proportional to $\Delta\xi$ and is given by the $n(t)$ in
which $|\Delta n_{34}(t)|>2$, with no actual limit to how large the number of
spikes may become (minimum of 2). In the other extreme limit of very large
noise, $\Delta n_{34}(t)$ is approximately a discrete random walk with step
size one and zero mean, thus no computation is performed in either system.

More interestingly, Fig. 7 illustrate the intermediary cases where noise may
(or may not) promote spike variability on all five neurons. In particular it
shows how the number of spikes needed to compute scales with the SNR in both
coupled and uncoupled systems. We found that not only the coupled system
consistently computes with less spikes across a broad range of SNRs but also
exhibits a small variability when the input magnitude is varied (Fig. 7a).
Furthermore, for ideal initial conditions, $V_{i}(0)=0$ for all $i$, the
coupled system computes with the absolute minimum number of spikes $n^{*}=3$
for $\text{SNR}\geq 25$. In contrast, the uncoupled system (Fig. 7b) does not
exhibit a predictable number of spikes $n^{*}$ to compute as a function of the
SNR, as it is also a function of the input signal differences $\Delta\xi$, and
presents a spike response variability with orders of magnitude of difference
for the same SNR (Fig. 7c). In practice, given an interval of possible SNRs
and input strengths, the largest estimated $n^{*}$ should be chosen as an
effective number of spikes to finish a computation.

The increase in computing speed due to reduced noise observed on the coupled
system occurs because it is itself computing the rank order as it goes, i.e.
the neurons are not only mapping the input strength to spike trains with
different frequencies but actively inhibiting other components in a self-
organized way. In summary, $k$-WTA by pulse-suppression is computed with
reasonably few spikes ($3\leq n^{*}\leq 70$, numerically estimated) if the SNR
is kept relatively large ( $\geq 1$) and for $\text{SNR}\geq 25$ only the
absolute minimum number of spikes $n^{*}=k$ is required, thus the computing
time is only limited by the neurons’ intrinsic spike frequency. As the
mechanism of self-organized pulse suppression is independent of both model
details and network size, these results should hold qualitatively to other
models and network sizes.

##  IV Discussion

Our results show that fast-computing (re)configurable $k$-WTA can be
implemented by simple symmetrical systems of spiking neurons with only
inhibitory interactions, providing a versatile model capable of performing a
variety of functions using the same underlying network structure. We believe
our model bridges and complements a variety of other approaches to $k$-WTA
that trade versatility, e.g. reconfigurability and/or simple topology, for
specific features as output stability and sensitivity to weak input signals.
For example Heteroclinic Computing Neves2009controlled ; Neves2012computation
; Neves2017Noise  offers sensitivity to arbitrarily small signals (up to the
noise level), because it relies only on unstable states, while neural fields
Sandamirskaya2014dynamics ; Strub2017dynamic  provide a soft-WTA with
macroscopic stability (population dynamics) via short-range excitation and
long-range inhibition. Our model makes use of stable orbits and can robustly
compute in a given time over a broad range of input magnitudes. Moreover, we
expect the results on the influence of noise to stay qualitatively the same
across other types of noise as long as they exhibit zero mean and exhibit no
long-term (power law) correlations. Such types would add a bias proportional
to the mean amplitude because noise is simply integrated in-between pulse-
events.

Due to its simplicity, our model may provide the basis for novel hardware
implementations. That is, our neuronal model is one-dimensional and generic,
thus has a variety of well-established implementations, while the amplitude of
interacting spikes is simply proportional to the voltage of the neuron
receiving them, resembling a simple leakage of current to a common ground.
Furthermore, the network topology employed is symmetrical, so it does not rely
on any specifically weighted or complex topological interactions. We remark
that system symmetry is not a necessary condition for such types of
computations even though our theoretical model represents ideal symmetrical
components. Small variations in parameters, e.g. coupling weights and neuron
spiking frequencies, would add a bias to the results with strength
proportional to the given variations because after integration all elements on
the right-hand side of Equation (1) represent simple currents.

As a versatile model capable of, in principle, performing any $k$-WTA
computation for a given network size $N$ and using only few spikes, it may be
a good candidate for implementing functionality on autonomous systems. Varying
$k$ means that a small network is already capable of performing many different
functions while the low spike count may translate in low power approaches to
computation. Thus, our results may potentially also contribute to the growing
field of artificial cognitive computing and related topics
memmesheimer2006designing ; memmesheimer2006designing2 ; schoner_2008 ;
lukovsevivcius2009reservoir ; Steingrube2010pattern ;
Hopfield2015understanding ; abbott2016FunctionalNetworksSpikingNeurons ;
Strub2017dynamic .

## Acknowledgement

This work was supported in part by the German Science Foundation (DFG) under
Germany’s Excellence Strategy – EXC-2068 – 390729961- Cluster of Excellence
Physics of Life of TU Dresden, the DFG and the Free State of Saxony by a grant
towards the Research Cluster Center for Advancing Electronics Dresden (cfaed),
the Free State of Saxony towards the project TransparNET and by the DFG under
grant number TI 629/5-1.

## References

  * [1] W. Bridewell and P. F. Bello.  A theory of attention for cognitive systems.  Annu. Conf. on Adv. Congnit. Sys., 4:1–16, 2016.
  * [2] J. L. McKinstry, J. G. Fleischer, Y. Chen, W. E. Gall, and G. M. Edelman.  Imagery may arise from associations formed through sensory experience: A network of spiking neurons controlling a robot learns visual sequences in order to perform a mental rotation task.  PLoS ONE, 11(9):e0162155, 2016.
  * [3] M. I. Rabinovich and P. Varona.  Neural dynamics of attentional cross-modality control.  PLoS ONE, 8(5):e64406, 2013.
  * [4] W. Maass.  On the computational power of winner-take-all.  Neural Comput., 12:2519–2535, 2000.
  * [5] F. S. Neves and M. Timme.  Computation by switching in complex networks of states.  Phys. Rev. Lett., 109(1):018701, 2012.
  * [6] Y. Chen.  Mechanisms of winner-take-all and group selection in neuronal spiking networks.  Front. in Comput. Neurosci., 11:20, 2017.
  * [7] C. R. Laing and C. C. Chow.  Stationary bumps in networks of spiking neurons.  Neural Comput., 13:1473–1494, 2001.
  * [8] Y. Sandamirskaya.  Dynamic neural fields as a step toward cognitive neuromorphic architectures.  Front. Neurosci., 22, 2014.
  * [9] M. Krupa.  Robust heteroclinic cycles.  J. Nonlinear Sci., 7:129–176, 1997.
  * [10] M. I. Rabinovich, A. Volkovskii, P. Lecanda, R. Huerta, H. D. I. Abarbanel, and G. Laurent.  Dynamical encoding by networks of competing neuron groups: winnerless competition.  Phys. Rev. Lett., 87(6):068102, 2001.
  * [11] M. Timme, F. Wolf, and T. Geisel.  Prevalence of unstable attractors in networks of pulse-coupled oscillators.  Phys. Rev. Lett., 89(15):154105, 2002.
  * [12] M. Timme, F. Wolf, and T. Geisel.  Unstable attractors induce perpetual synchronization and desynchronization.  Chaos, 13(1):377–387, 2003.
  * [13] P. Ashwin and J. Borresen.  Encoding via conjugate symmetries of slow oscillations for globally coupled oscillators.  Phys. Rev. E, 70:026203, 2004.
  * [14] P. Ashwin and J. Borresen.  Discrete computation using a perturbed heteroclinic network.  Phys. Lett. A, 374(4–6):208–214, 2005.
  * [15] P. Ashwin and M. Timme.  When instability makes sense.  Nature, 436:36–37, 2005.
  * [16] J. Wordsworth and P. Ashwin.  Spatiotemporal coding of inputs for a system of globally coupled phase oscillators.  Phys. Rev. E, 78:066203, 2008.
  * [17] J. J. Wang, Q. Yu, S. G. Hu, Y. Liu, R. Guo, T. P. Chen, Y. Yin, and Y. Liu.  Winner-takes-all mechanism realized by memristive neural network.  Appl. Phys. Lett., 115(24):243701, 2019.
  * [18] M. I. Rabinovich, P. Varona, A. I. Selverston, and H. D. I. Abarbanel.  Dynamical principles in neuroscience.  Rev. of Mod. Phys., 78(4):1213, 2006.
  * [19] F. S. Neves, M. Voit, and M. Timme.  Noise-constrained switching times for heteroclinic computing.  Chaos, 27(3):033107, 2017.
  * [20] W. Maass and C. M. Bishop, editors.  Pulsed Neural Networks.  MIT Press, Cambridge, Massachusetts, 1999.
  * [21] R. E. Mirollo and S. H. Strogatz.  Synchronization of pulse-coupled biological oscillators.  SIAM J. Appl. Math., 50(6):1645–1662, 1990.
  * [22] D. Hansel, G. Mato, C. Meunier, and L. Neltner.  On numerical simulations of integrate-and-fire neural networks.  Neural Comput., 10(2):467–483, 1998.
  * [23] J. Klinglmayr, C. Kirst, C. Bettstetter, and M. Timme.  Guaranteeing global synchronization in networks with stochastic interactions.  New J. Phys., 14(7):073031, 2012.
  * [24] S. H. Strogatz.  From Kuramoto to Crawford: exploring the onset of synchronization in populations of coupled oscillators.  Physica D, 143(1-4):1–20, 2000.
  * [25] A. T. Winfree.  Biological rhythms and the behavior of populations of coupled oscillators.  J. Theor. Biol., 16(1):15–42, 1967.
  * [26] H. Kielblock, C. Kirst, and M. Timme.  Breakdown of order preservation in symmetric oscillator networks with pulse-coupling.  Chaos, 21(2):025113, 2011.
  * [27] F. S. Neves and M. Timme.  Controlled perturbation-induced switching in pulse-coupled oscillator networks.  J. Phys. A, 42(34):345103, 2009.
  * [28] C. Strub, G. Schöner, F. Wörgötter, and Y. Sandamirskaya.  Dynamic neural fields with intrinsic plasticity.  Front. Comp. Neurosci., 31, 2017.
  * [29] R. Memmesheimer and M. Timme.  Designing the dynamics of spiking neural networks.  Phys. Rev. Lett., 97(18):188101, 2006.
  * [30] R. Memmesheimer and M. Timme.  Designing complex networks.  Physica D, 224(1-2):182–201, 2006.
  * [31] G. Schöner.  Dynamical Systems Approaches to Cognition, pages 101–126.  Cambridge University Press, 2008.
  * [32] M. Lukoševičius and H. Jaeger.  Reservoir computing approaches to recurrent neural network training.  Comp. Sci. Rev., 3(3):127–149, 2009.
  * [33] S. Steingrube, M. Timme, F. Wörgötter, and P. Manoonpong.  Self-organized adaptation of a simple neural circuit enables complex robot behaviour.  Nature Physics, 6:224–230, 2010.
  * [34] J. J. Hopfield.  Understanding emergent dynamics: Using a collective activity coordinate of a neural network to recognize time-varying patterns.  Neural Comput., 27:2011–2038, 2015.
  * [35] L. F. Abbott, B. DePasquale, and R. Memmesheimer.  Building functional networks of spiking model neurons.  Nature Neurosci., 19(3):350, 2016.


# S2R-ViT for Multi-Agent Cooperative Perception: Bridging the Gap from
Simulation to Reality

Jinlong Li1, Runsheng Xu2, Xinyu Liu1, Baolu Li1, Qin Zou3, Jiaqi Ma2, Hongkai
Yu1∗ 1Cleveland State University, Cleveland Vision $\&$ AI Lab. 2University of
California, Los Angeles, UCLA Mobility Lab. 3Wuhan University, MVR Lab.
*Corresponding Author<EMAIL_ADDRESS>
###### Abstract

Due to the lack of enough real multi-agent data and time-consuming of
labeling, existing multi-agent cooperative perception algorithms usually
select the simulated sensor data for training and validating. However, the
perception performance is degraded when these simulation-trained models are
deployed to the real world, due to the significant domain gap between the
simulated and real data. In this paper, we propose the first Simulation-to-
Reality transfer learning framework for multi-agent cooperative perception
using a novel Vision Transformer, named as S2R-ViT, which considers both the
Deployment Gap and Feature Gap between simulated and real data. We investigate
the effects of these two types of domain gaps and propose a novel uncertainty-
aware vision transformer to effectively relief the Deployment Gap and an
agent-based feature adaptation module with inter-agent and ego-agent
discriminators to reduce the Feature Gap. Our intensive experiments on the
public multi-agent cooperative perception datasets OPV2V and V2V4Real
demonstrate that the proposed S2R-ViT can effectively bridge the gap from
simulation to reality and outperform other methods significantly for point
cloud-based 3D object detection.

##  I Introduction

The recent advancement in multi-agent cooperative perception shows potentials
to overcome the limitation of single-agent perception that suffers from the
challenges of perceiving range and occlusion [1, 2]. By leveraging agent-to-
agent communication technology to share information, multi-agent cooperative
perception systems can significantly enhance perception performance compared
to the single-agent perception [3, 4]. Due to the difficulties of collecting
multi-agent data with communication in the real world, it is expensive and not
easy to collect enough real data in diverse and complex real-world
environments [2]. Furthermore, the ground-truth data labeling and uniform
coordinate projection for multi-agent cooperative perception systems is
particularly time-consuming. Therefore, many existing multi-agent cooperative
perception research works usually select the simulated data for model training
and validating [3, 4].

However, when we apply the models trained with simulated data to the real
world, the perception performance is typically degraded. This phenomenon is
because of the significant domain gap between the simulated and real data. In
this paper, our research is focused on utilizing labeled simulated data and
unlabeled real-world data as transfer learning to reduce the domain gap for
multi-agent cooperative perception. We observe that the domain gap from
simulation to reality for multi-agent cooperative perception includes the
following two perspectives.

  * •

Deployment Gap: As shown in Fig. 1, different with the ideal simulation
setting, the multiple agents might have localization (positional and heading)
errors due to the unavoidable GPS errors and communication latency (time
delay) during the real-world agent-to-agent communication.

  * •

Feature Gap: As illustrated in Fig. 1, the point cloud feature distribution in
real world might differ significantly with that of the simulated data, such as
more complex driving scenarios, different LiDAR channel numbers, mixed traffic
flow, various point cloud variations, and so on.

((a)) Figure 1: Illustration of the domain gap (Deployment Gap, Feature Gap)
for multi-agent cooperative perception from simulation to reality. Here we use
Vehicle-to-Vehicle (V2V) cooperative perception in autonomous driving as
example. CAV indicates the Connected Autonomous Vehicles. Figure 2: Overview
of the proposed S2R-ViT for multi-agent cooperative perception from simulation
to reality, which leverages S2R-UViT module to handle uncertainty in
Deployment Gap and tackles the Feature Gap through S2R-AFA module including
the inter-agent discriminator $D_{i}$ and the ego-agent discriminator $D_{e}$.
Source Domain: labeled simulated data, Target Domain: unlabeled real-world
data. Best viewed in color.

In this paper, we propose the first Simulation-to-Reality (S2R) transfer
learning framework for multi-agent cooperative perception using a novel Vision
Transformer (ViT), named as S2R-ViT, by taking both Deployment Gap and Feature
Gap into consideration. We choose the task of Vehicle-to-Vehicle (V2V)
Cooperative Perception for the point cloud-based 3D object detection as
algorithm development. Specifically, our framework takes the labeled point
cloud data from simulation and the unlabeled data from real world as input, so
as to largely utilize the simulated data. In machine learning research, this
setting is widely called as Unsupervised Domain Adaptation [5] from source
domain (simulation) to target domain (reality).

The proposed S2R-ViT comprises two key components: (1) S2R-UViT: a novel S2R
Uncertainty-aware Vision Transformer to effectively relief the uncertainties
brought by the Deployment Gap. Specifically, S2R-UViT includes a Local-and-
Global Multi-head Self Attention (LG-MSA) module to enhance feature
interactions across all agents’ spatial positions to tolerate the uncertainty
drawbacks and also a Uncertainty-Aware Module (UAM) to enhance the ego-agent
features by considering the shared other-agent features of different
uncertainty levels. (2) S2R-AFA: S2R Agent-based Feature Adaptation to reduce
the Feature Gap. S2R-AFA utilizes the inter-agent and ego-agent discriminators
to extract domain-invariant features to bridge the Feature Gap. Finally, we
conducted extensive experiments on two public datasets, namely simulated OPV2V
[4] and real V2V4Real [2], to justify the effectiveness of our proposed
method. Our contributions are summarized as follows.

  * •

To the best of our knowledge, we propose the first research, named S2R-ViT, on
multi-agent cooperative perception from simulation to reality by investigating
two types of domain gaps, i.e., Deployment Gap and Feature Gap, for point
cloud-based 3D object detection.

  * •

We propose a novel Uncertainty-aware Vision Transformer (S2R-UViT) to
effectively relieve the uncertainties brought by the Deployment Gap.

  * •

We design an Agent-based Feature Adaptation (S2R-AFA) module that includes
inter-agent and ego-agent discriminators to effectively reduce the Feature Gap
between simulation and reality.

  * •

We evaluate our proposed method on the large-scale simulated OPV2V dataset and
the real V2V4Real dataset, whose experiments demonstrate our superior
performance in point cloud-based 3D object detection.

Figure 3: Architecture of the proposed S2R-UViT: Simulation-to-Reality
Uncertainty-aware Vision Transformer.

##  II Related Work

Multi-Agent Perception. Multi-agent perception system can overcome the
challenges of occlusion and short-range perceiving via agent-to-agent
communication technology to achieve the large-range perceiving, which has
attracted the attention of many researchers. Rather than exchanging raw
sensing data or detected outputs, contemporary approaches typically share
intermediate features extracted by neural networks. This strategy offers an
optimal balance between accuracy and bandwidth requirements [6, 7, 8, 1].
V2VNet [3] utilizes a graph neural network to aggregate features from
individual agents. When2com [8] employs a spatial confidence-aware
communication strategy aimed at enhancing performance by reducing
communication overhead. OPV2V [4] proposes an attention fusion framework for
cooperative perception system. SyncNet [9] introduces a latency compensation
module for time-domain synchronization. V2X-ViT [6] and CoBEVT [10] propose
transformer or axial attention based methods to improve the performance.
Although these methods have demonstrated impressive performance, all of them
are implemented in the simulated data, while this paper aims to bridge the gap
from simulation to reality.

Challenges in Multi-Agent Perception. Multi-Agent perception system also
introduces some new challenges, e.g., localization error, communication
latency, adversarial attacks. These challenges might diminish the benefits of
collaborations [7, 11, 6]. To ensure robustness for Multi-Agent perception,
V2X-ViT [6] proposes a ViT based multi-agent perception system, showcasing
robust performance under communication delay. [12] introduces a pose
regression module and consistency module prior to feature aggregation to
rectify pose errors. [8] presents the first latency-aware multi-agent
perception framework, achieving feature-level synchronization. [13] explores
adversarial attacks in multi-agent perception and devises a novel transfer
attack approach within the intermediate collaborative perception. [11]
proposes a lossy communication-aware repair network to ensure the robustness
of multi-agent perception amidst lossy communication data. To promote multi-
agent perception research, several large-scale cooperative perception datasets
are publicized, e.g., simulation based OPV2V [4] and V2XSet [6], real-world
V2V4Real [2]. Because collecting the multi-agent sensing data in the wild and
annotating its ground truth are time and labor consuming, many existing
research works are based on simulated data. In this paper, we focus on
improving point cloud-based cooperative 3D object detection in real world by
largely utilizing labelled simulated data and just unlabelled real-world data.

Domain Adaptation for Perception. Domain adaptation involves adapting the
machine learning model trained on source domain to the target domain. Many
domain adaptation works are mainly focused on the RGB camera data [14, 15, 16,
17], while more domain adaptation works have been proposed to solve this
problem in LiDAR data [18, 19, 20, 7]. Specifically, Semantic Point Generation
(SPG) [19] is presented to enhance the reliability of LiDAR detectors against
domain gaps to generates semantic points at the 3D objects. CoSMix [20] is
designed for 3D LiDAR segmentation, aiming to alleviate the domain gap by
generating two intermediate domains comprising composite point clouds. This is
achieved through the application of a special mixing strategy at the input
level. [7] proposes a domain adaption framework to bridge the domain gap of
shared data in communication for collaborative perception. In this paper, an
agent-based feature adaptation module are proposed to reduce the feature gap
between simulated to real data for multi-agent perception.

##  III Methodology

As illustrated in Fig. 2, the proposed S2R work is an end-to-end unified deep
learning pipeline, including 1) V2V metadata sharing, 2) Feature extraction
and sharing, 3) S2R-ViT, and 4) Detection header.

###  III-A Overview of Architecture

1) V2V metadata sharing. We designate one of the CAVs as the ego vehicle and
construct a spatial graph around it, with each node representing a CAV within
the communication range. After receiving the ego vehicle’s relative pose and
extrinsic information, all other nearby CAVs then project their LiDAR data to
the coordinate frame of the ego vehicle. 2) Feature extraction and sharing. We
utilize the PointPillar backbone [21] to extract intermediate features from
LiDAR data due to its low inference latency and refined memory usage. Each CAV
possesses its own feature extraction module. Following feature extraction by
each CAV, the ego vehicle receives the visual features of neighboring CAVs via
communication. 3) S2R-ViT. The intermediate features aggregated from other
surrounding CAVs are fed into our major component named S2R-ViT, which
consists of S2R-UViT and S2R-AFA modules. Upon receiving the final fused
features, we utilize a prediction header for 3D object classification and
localization.

###  III-B S2R-UViT: Simulation-to-Reality Uncertainty-aware Vision
Transformer

The Deployment Gap from simulation to reality brings different uncertainties
to both ego and neighboring agents, e.g., spatial bias by GPS errors, spatial
misalignment in the coordinate projection because of communication latency.
How to effectively reduce the degradation effects of these uncertainty
drawbacks is an essential problem and open question to the S2R multi-agent
perception research. In this paper, we propose to answer this question from
two perspectives: uncertainties can be relieved by enhancing (1) the feature
interactions across all agents’ spatial positions more comprehensively and (2)
the ego-agent features by considering the shared other-agent features of
different uncertainty levels. These two perspectives motivate us to develop
the novel Local-and-Global Multi-head Self Attention (LG-MSA) Module and
Uncertainty-Aware Module (UAM) respectively. The Fig. 3 presents the two major
modules of the proposed S2R-UViT.

####  III-B1 Local-and-Global Multi-head Self Attention (LG-MSA)

In order to enhance the feature interactions across all agents’ spatial
positions more comprehensively, we propose LG-MSA to promote both local and
global feature interactions across all agents’ spatial positions. In the
proposed LG-MSA, a local feature-based attention is utilized to focus on the
local details of spatial features, while a global feature-based attention is
used to pay attention on the wide range of spatial features. Specifically, the
fused features of the ego agent and other agents are spilt into two branches
(i.e., local branch and global branch) to process local-based and global-based
attentions, respectively, then these two features are concatenated together
and fed into a Self Attention (SA) module to further capture local and global
information.

Inspired by [22, 23], after retrieving the whole features
$F_{e,o}\in\mathbb{R}^{H\times W\times kC}$ with spatial dimension $H\times W$
from all agents ($e$: ego-agent, $o$: other agents, $k$: number of all
agents), we reshape it into $\mathbb{R}^{h\times H\times
W\times\frac{kC}{h}}$. After that, we take $h=8$ as head number and $n=2$ as
window-type number, the multi-head $h$ of standard multi-head Self Attention
Module (MSA) [24] is evenly divided into two groups with different window
sizes, i.e., $4\times 4$ for the local branch and $8\times 8$ for the global
branch. In the local branch, the spilt feature
$F_{e,o}^{l}\in\mathbb{R}^{\frac{h}{n}\times H\times W\times\frac{kC}{h}}$ is
fed into the MSAL with a small window size $4\times 4$ to enhance the local
details of spatial features. In the global branch, another spilt feature
$F_{e,o}^{g}\in\mathbb{R}^{\frac{h}{n}\times H\times W\times\frac{kC}{h}}$ is
fed into the MSAG with a large window size $8\times 8$ to capture global
spatial feature information. Then we concatenate these two-branch output
features and implement the feature interactions by a Self Attention (SA)
module to obtain the promoted feature $F_{e,o}^{p}\in\mathbb{R}^{H\times
W\times kC}$. As show in Fig. 3(a), the proposed LG-MSA computation can be
defined as

$F_{e,o}^{p}=\mathrm{SA}(\mathrm{Concat}(\mathrm{MSA}_{L}(F_{e,o}^{l}),\mathrm{MSA}_{G}(F_{e,o}^{g}))).$
(1)

####  III-B2 Uncertainty-Aware Module (UAM)

To enhance the ego-agent feature based on the shared other-agent features,
their different uncertainty levels should not be neglected. When receiving the
feature $F_{e,o}^{p}$ for all agents, we split them into ego-agent feature
$F_{e}^{p}\in\mathbb{R}^{H\times W\times C}$ and other agent shared feature
$F_{o}^{p}\in\mathbb{R}^{H\times W\times(k-1)C}$. The shared other-agent
feature $F_{o}^{p}$ will be fed into an Uncertainty Prediction Network (UPN)
to predict the uncertainty levels on spatial features, which generates a
uncertainty-level map $M\in\mathbb{R}^{H\times W\times(k-1)C}$ with the same
spatial size of input. The UPN is an encoder-decoder based neural network
simplified from [25], which can be learned during the end-to-end training of
our whole architecture. Inspired by the natural selection in the mechanism of
evolution, the feature values in the predicted uncertainty-level map $M$ with
high uncertainty levels (i.e., low confidences) are reset as 1 using the
median as threshold. It results in a new uncertainty-level map $M_{t}$. Then,
$M_{t}$ is multiplied with $F_{e}^{p}$, by taking the shared other-agent
features of different uncertainty levels into consideration, so as to produce
the enhanced ego feature. In other words, only the low-uncertainty spatial
features of other agents (related to non-ones in $M_{t}$) will contribute to
enhance the ego-agent feature during the matrix multiplication. Finally, the
enhanced ego feature and other agent shared feature are concatenated together
to obtain the combined feature $F_{e,o}^{h}\in\mathbb{R}^{H\times W\times
kC}$. As show in Fig. 3(a), the computation of the proposed UAM can be
formulated as

$F_{e,o}^{h}=\mathrm{Concat}(\Delta[\mathrm{UPN}(F_{o}^{p})]\circledast
F_{e}^{p},F_{o}^{p}),$ (2)

where $\Delta[\cdot]$ represents the threshold process and $\circledast$
denotes the matrix dot product. Combining these local and global attention
branches with standard Transformer designs [22, 23], such as Layer
Normalization (LN), MLPs, and skip-connections, results in our proposed
S2R-UViT block, as depicted in Fig. 3(b). The S2R-UViT block can be expressed
as:

$F_{e,o}^{h}=\mathrm{S2RAttn}(\mathrm{LN}(F_{e,o}))+F_{e,o},$ (3)
$\hat{F_{e,o}^{h}}=\mathrm{MLP}(\mathrm{LN}(F_{e,o}^{h}))+F_{e,o}^{h},$ (4)

where $F_{e,o}^{h}$ and $\hat{F_{e,o}^{h}}$ denote the output features of our
S2RAttn module (i.e., the proposed LG-MSA and UAM) and MLP module.

###  III-C S2R-AFA: Simulation-to-Reality Agent-based Feature Adaptation

To reduce the Feature Gap from simulated feature $F_{s}$ to real feature
$F_{r}$ within both inter-agent and ego-agent features, we design two domain
discriminators/classifiers before and after fusion as shown in Fig. 2, where
all agent feature $F_{s}$ and $F_{r}$ before fusion are fed into the inter-
agent discriminator $D_{i}$ to classify whether it belongs to simulation or
reality, and the fused ego features $F_{s}^{e}$ and $F_{r}^{e}$ after fusion
are also classified into simulation or reality by the ego-agent discriminator
$D_{e}$. The binary cross entropy loss is used for binary domain (Sim/Real)
classification. These two discriminators are adversarially optimized by the
Agent-based Feature Adaptation loss $\mathcal{L}_{AFA}$:

$\min\limits_{G_{m}}\max\limits_{D_{i},D_{e}}\mathcal{L}_{AFA}=\mathbb{E}_{s,r}[D_{i}(F_{s},F_{r})]+\mathbb{E}_{s,r}[D_{e}(F_{s}^{e},F_{r}^{e})],$
(5)

where $\mathbb{E}_{s,r}$ indicates the domain classification error in
simulation and reality respectively, $G_{m}$ is our whole model (backbone,
S2R-UViT, and detection header) which can be thought as the generator of
Generative Adversarial Network [26]. Because of S2R-AFA, our generator model
$G_{m}$ will have the capability of extracting domain-invariant features of
simulation and reality.

We use the focal loss [27] and smooth $L_{1}$ loss as the detection loss for
our model training. Our final loss is the combination of detection loss and
the Agent-based Feature Adaptation loss as follows,

$\mathcal{L}_{total}=w_{1}\mathcal{L}_{det}+w_{2}\mathcal{L}_{AFA},$ (6)

where $w_{1}$ and $w_{2}$ are balance weights with sum as 1.

TABLE I: 3D detection performance on two OPV2V testing sets under Deployment-
Gap Scenario with Perfect Setting and Noisy Setting. All methods are trained
on Perfect Setting.

Models | Setting | V2V CARLA Towns | V2V Culver City
---|---|---|---
<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
AttFuse [4] | Perfect | 0.921 | 0.804 | 0.887 | 0.716
Noisy | 0.851 | 0.472 | 0.865 | 0.565
V2VNet [3] | Perfect | 0.915 | 0.828 | 0.884 | 0.757
Noisy | 0.845 | 0.413 | 0.868 | 0.617
F-cooper [1] | Perfect | 0.907 | 0.810 | 0.893 | 0.746
Noisy | 0.842 | 0.479 | 0.881 | 0.624
V2X-ViT [6] | Perfect | 0.902 | 0.792 | 0.903 | 0.764
Noisy | 0.801 | 0.395 | 0.882 | 0.606
CoBEVT [10] | Perfect | 0.925 | 0.852 | 0.904 | 0.776
Noisy | 0.862 | 0.519 | 0.889 | 0.665
V2VAM [11] | Perfect | 0.916 | 0.849 | 0.903 | 0.794
Noisy | 0.876 | 0.507 | 0.883 | 0.663
S2R-UViT | Perfect | 0.928 | 0.867 | 0.912 | 0.768
Noisy | 0.879 | 0.579 | 0.900 | 0.681

Figure 4: Robustness in Deployment-Gap Scenario including GPS (positional,
heading) errors and communication latency on the CARLA Towns testing set of
OPV2V [4] with the simulated Noisy Setting.

##  IV Experiment

###  IV-A Dataset

We conduct experiments on two public benchmark datasets (OPV2V [4], V2V4Real
[2]) for the V2V cooperative perception task. OPV2V is a large-scale simulated
dataset for V2V cooperative perception, which is collected by CARLA simulator
and OpenCDA [28]. It contains 73 divergent scenes with various numbers of
connected vehicles ($[1,5]$), and its training/validation/testing set is split
into 6,764, 1,981, and 2,719 frames, respectively. V2V4Real is a real-world,
large-scale dataset with diverse driving scenarios, which is collected by two
CAVs driving simultaneously in Columbus, Ohio, USA. It’s split into the
train/validation/test set with 14,210/2,000/3,986 frames, respectively.

###  IV-B Experiments Setup

Evaluation Metrics. We assess the perception performance based on the final 3D
vehicle detection accuracy. Following [4, 6], we define the evaluation range
as $x\in[-140,140]$ meters and $y\in[-40,40]$ meters, encompassing all CAVs
within this spatial range in the experiment. Accuracy is calculated as Average
Precision (AP) at two Intersection-over-Union (IoU) thresholds ($0.5$ and
$0.7$).

Experiment Settings. In this work, we want to address the deployment gap and
feature gap on LiDAR-based object detection and assess our model under two
distinct settings:

  * 1)

Deployment-Gap Scenario: During the training, all models are trained in the
perfect simulated OPV2V training set, then all of them are evaluated on OPV2V
CARLA Towns and Culver City testing sets under two different setting (e.g.,
Perfect and Noisy), respectively. We implement the Noisy Setting following [6]
to simulate the GPS errors, the positional and heading noises of the
transmitter are sampled from a Gaussian distribution with a standard deviation
of $0.2$ m and $0.2^{\circ}$ respectively, and the communication latency is
set as $100$ ms for all the evaluated models. This Deployment-Gap Scenario
only includes the Deployment Gap.

  * 2)

Sim2Real Scenario: We set the labeled training set of simulation dataset OPV2V
[4] as the source domain and the unlabeled training set of the real-world
dataset V2V4Real [2] as the target domain for all models during the training,
by following the same setting in [2]. Then, all trained models are evaluated
on the testing set of V2V4Real to report the performance. This Sim2Real
Scenario includes both the Deployment Gap and Feature Gap from simulation to
reality.

All methods utilize PointPillar [29] as the point cloud encoder/backbone. We
employ the Adam optimizer [30] with an initial learning rate of $10^{-3}$ and
gradually decay it every $10$ epochs based a factor of $0.1$. Following the
hyperparameters outlined in V2X-ViT [6], we train all models on two RTX 3090
GPU cards.

Compared Methods. Six state-of-the-art V2V methods are evaluated, all of which
employ Intermediate Fusion as the primary strategy: AttFuse [4], V2VNet [3],
F-Cooper [1], V2X-ViT [6], CoBEVT [10], and V2VAM [11]. We first train these
methods on the perfect setting of OPV2V training set, and then these methods
are evaluated on the Noisy Setting of OPV2V testing set and V2V4Real testing
set to assess their performance. In addition, to show the effectiveness of
reducing Feature Gap, two domain adaptation methods i.e., gradient reverse
layer (GRL) [31] and adversarial gradient reverse layer (AdvGRL) [15] are
utilized to back-propagate the reversed gradient to adversarially guide the
model for generating domain-invariant features by one feature-level domain
classifier (after fusion) and one object/proposal-level domain classifier (in
detection header).

TABLE II: 3D detection performance on V2V4Real testing set under Sim2Real
Scenario. All methods with domain adaptation are trained following the setting
in Sim2Real Scenario. S2R-UViT w/ S2R-AFA indicates our S2R-ViT.

Method | V2V4Real Testing
---|---
<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
AttFuse [4] | 0.225 | 0.094
AttFuse w/ GRL | 0.356 | 0.139
AttFuse w/ AdvGRL | 0.366 | 0.137
V2VNet [3] | 0.268 | 0.108
V2VNet w/ GRL | 0.376 | 0.122
V2VNet w/ AdvGRL | 0.358 | 0.103
F-Cooper [1] | 0.236 | 0.091
F-Cooper w/ GRL | 0.372 | 0.116
F-Cooper w/ AdvGRL | 0.364 | 0.135
V2X-ViT [6] | 0.274 | 0.103
V2X-ViT w/ GRL | 0.395 | 0.156
V2X-ViT w/ AdvGRL | 0.404 | 0.157
CoBEVT [10] | 0.334 | 0.130
CoBEVT w/ GRL | 0.398 | 0.163
CoBEVT w/ AdvGRL | 0.393 | 0.163
V2VAM [11] | 0.332 | 0.120
V2VAM w/ GRL | 0.390 | 0.146
V2VAM w/ AdvGRL | 0.401 | 0.161
S2R-UViT | 0.367 | 0.138
S2R-UViT w/ GRL | 0.414 | 0.141
S2R-UViT w/ AdvGRL | 0.414 | 0.157
S2R-UViT w/ S2R-AFA | 0.441 | 0.170

((a))

((b))

((c))

Figure 5: Visualization example of point cloud-based 3D object detection on
V2V4Real testing set under the Sim2Real Scenario. Green and red 3D bounding
boxes represent ground truth and prediction respectively. Best viewed in
color.

###  IV-C Quantitative Evaluation

Performance in Deployment-Gap Scenario. Table I shows the performance
comparison on Deployment-Gap Scenario, where all methods are evaluated on
Perfect Setting and Noisy setting, respectively. Under the Perfect Setting,
all cooperative perception methods achieve outstanding performance.
Nevertheless, when these methods deployed on the Noisy Setting scenario, which
have the deployment gap with the Perfect Setting scenario, the V2X-ViT [6],
CoBEVT [10], and V2VAM [11] drop $39.7\%$, $33.3\%$, and $34.2\%$ on for
<EMAIL_ADDRESS>on V2V CARLA Towns testing set. It indicates the highly negative
impacts by V2V deployment gap. While our proposed S2R-UViT achieves the mostly
best performance under both Perfect Setting and Noisy Setting scenarios, which
is highlighted in Table I. To assess the models’ sensitivity on different
Deployment-Gap Scenario, we conduct the experiments on the V2V CARLA Towns
testing set of OPV2V dataset with the difference of Noisy Setting. The Fig. 4
depicts the higher robustness of our S2R-UViT against other fusion methods in
the deployment gap.

Performance in Sim2Real Scenario. The 3D object detection result on the real-
world V2V4Real testing set of Sim2Real Scenario is presented in Table II.
Among all intermediate fusion methods without domain adaptation, our proposed
S2R-UViT achieves the best performance by eliminating the deployment gap in
the real V2VReal testing set. After applying domain adaptation methods e.g.,
GRL [31] and AdvGRL [15], all methods have improved performance. For example,
V2X-ViT is improved by $12.1\%$/$5.3\%$ for<EMAIL_ADDRESS>with GRL, and
$13.0\%$/$5.4\%$ for<EMAIL_ADDRESS>with AdvGRL. Our S2R-ViT with S2R-AFA achieves
the $44.1\%$/$17.0\%$ for<EMAIL_ADDRESS>as the best performance with improvement
of $7.4\%$/$3.2\%$ than S2R-UViT. The examples of 3D object detection results
on V2V4Real testing set under Sim2Real Scenario are visualized in Fig. 5,
where our S2R-ViT generates more accurate 3D detection results.

Ablation Study. As Table II depicts, all of the proposed components within
S2R-ViT have contributed to enhanced detection performance. Specifically, our
S2R-UViT model achieves the highest detection performance among all state-of-
the-art V2V methods, which is $3.3\%$ and $0.8\%$ higher than the second-best
performance model CoBEVT in<EMAIL_ADDRESS>and<EMAIL_ADDRESS>respectively. Adding our
S2R-AFA, our proposed S2R-ViT achieves $44.1\%$ and $17.0\%$ in<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>with the improvement of $7.4\%$ in<EMAIL_ADDRESS>and $3.2\%$ in<EMAIL_ADDRESS>
##  V Conclusions

This paper is the first work that investigates the domain gap on multi-agent
cooperation perception from simulation to reality, specifically focusing on
the deployment gap and feature gap in point cloud-based 3D object detection.
Based on the analysis, we present the first Simulation-to-Reality transfer
learning framework using a novel Vision Transformer, named S2R-ViT, to
mitigate these two types of domain gaps, which mainly contain an Uncertainty-
aware Vision Transformer and an Agent-based Feature Adaptation module. The
experiment shows the effectiveness of S2R-ViT. This research presents a
significant step forward in the multi-agent cooperation perception from
simulation to reality.

## References

  * [1] Q. Chen, X. Ma, S. Tang, J. Guo, Q. Yang, and S. Fu, “F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds,” in _ACM/IEEE Symposium on Edge Computing_ , 2019, pp. 88–100.
  * [2] R. Xu, X. Xia, J. Li, H. Li, S. Zhang, Z. Tu, Z. Meng, H. Xiang, X. Dong, R. Song, _et al._ , “V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 13 712–13 722.
  * [3] T.-H. Wang, S. Manivasagam, M. Liang, B. Yang, W. Zeng, and R. Urtasun, “V2vnet: Vehicle-to-vehicle communication for joint perception and prediction,” in _European Conference on Computer Vision_. Springer, 2020, pp. 605–621.
  * [4] R. Xu, H. Xiang, X. Xia, X. Han, J. Li, and J. Ma, “Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication,” in _International Conference on Robotics and Automation_. IEEE, 2022, pp. 2583–2589.
  * [5] S. Song, H. Yu, Z. Miao, J. Fang, K. Zheng, C. Ma, and S. Wang, “Multi-spectral salient object detection by adversarial domain adaptation,” in _AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 12 023–12 030.
  * [6] R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and J. Ma, “V2x-vit: Vehicle-to-everything cooperative perception with vision transformer,” in _European Conference on Computer Vision_. Springer, 2022, pp. 107–124.
  * [7] R. Xu, J. Li, X. Dong, H. Yu, and J. Ma, “Bridging the domain gap for multi-agent perception,” in _International Conference on Robotics and Automation_ , 2023.
  * [8] Y. Hu, S. Fang, Z. Lei, Y. Zhong, and S. Chen, “Where2comm: Communication-efficient collaborative perception via spatial confidence maps,” in _Advances in Neural Information Processing Systems_ , 2022.
  * [9] Z. Lei, S. Ren, Y. Hu, W. Zhang, and S. Chen, “Latency-aware collaborative perception,” in _European Conference on Computer Vision_. Springer, 2022, pp. 316–332.
  * [10] R. Xu, Z. Tu, H. Xiang, W. Shao, B. Zhou, and J. Ma, “Cobevt: Cooperative bird’s eye view semantic segmentation with sparse transformers,” in _Conference on Robot Learning_ , 2022.
  * [11] J. Li, R. Xu, X. Liu, J. Ma, Z. Chi, J. Ma, and H. Yu, “Learning for vehicle-to-vehicle cooperative perception under lossy communication,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 4, pp. 2650–2660, 2023.
  * [12] N. Vadivelu, M. Ren, J. Tu, J. Wang, and R. Urtasun, “Learning to communicate and correct pose errors,” in _Conference on Robot Learning_. PMLR, 2021, pp. 1195–1210.
  * [13] J. Tu, T. Wang, J. Wang, S. Manivasagam, M. Ren, and R. Urtasun, “Adversarial attacks on multi-agent communication,” in _IEEE International Conference on Computer Vision_ , 2021, pp. 7768–7777.
  * [14] J. Li, Z. Xu, L. Fu, X. Zhou, and H. Yu, “Domain adaptation from daytime to nighttime: A situation-sensitive vehicle detection and traffic flow parameter estimation framework,” _Transportation Research Part C: Emerging Technologies_ , vol. 124, p. 102946, 2021.
  * [15] J. Li, R. Xu, J. Ma, Q. Zou, J. Ma, and H. Yu, “Domain adaptive object detection for autonomous driving under foggy weather,” in _IEEE Winter Conference on Applications of Computer Vision_ , 2023, pp. 612–622.
  * [16] P. Oza, V. A. Sindagi, V. V. Sharmini, and V. M. Patel, “Unsupervised domain adaptation of object detectors: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [17] L. Hoyer, D. Dai, H. Wang, and L. Van Gool, “Mic: Masked image consistency for context-enhanced domain adaptation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 11 721–11 732.
  * [18] L. Yi, B. Gong, and T. Funkhouser, “Complete & label: A domain adaptation approach to semantic segmentation of lidar point clouds,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 363–15 373.
  * [19] Q. Xu, Y. Zhou, W. Wang, C. R. Qi, and D. Anguelov, “Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation,” in _IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15 446–15 456.
  * [20] C. Saltori, F. Galasso, G. Fiameni, N. Sebe, E. Ricci, and F. Poiesi, “Cosmix: Compositional semantic mix for domain adaptation in 3d lidar segmentation,” in _European Conference on Computer Vision_. Springer, 2022, pp. 586–602.
  * [21] A. H. Lang, S. Vora, H. Caesar, L. Zhou, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [22] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.
  * [23] P. Ren, C. Li, G. Wang, Y. Xiao, Q. Du, X. Liang, and X. Chang, “Beyond fixation: Dynamic window visual transformer,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 987–11 997.
  * [24] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021.
  * [25] B. Mildenhall, J. T. Barron, J. Chen, D. Sharlet, R. Ng, and R. Carroll, “Burst denoising with kernel prediction networks,” in _IEEE conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2502–2510.
  * [26] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _arXiv preprint arXiv:1511.06434_ , 2015.
  * [27] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _IEEE International Conference on Computer Vision_ , 2017, pp. 2980–2988.
  * [28] R. Xu, Y. Guo, X. Han, X. Xia, H. Xiang, and J. Ma, “Opencda: An open cooperative driving automation framework integrated with co-simulation,” in _IEEE Intelligent Transportation Systems Conference_ , 2021.
  * [29] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 697–12 705.
  * [30] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _International Conference on Learning Representations._ , 2017.
  * [31] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _International Conference on Machine Learning_. PMLR, 2015, pp. 1180–1189.