# Integrate-and-fire circuit for converting analog signals to spikes using
phase encoding

Javier López-Randulfe , Nico Reeb22footnotemark: 2 , and Alois
Knoll22footnotemark: 2 Corresponding author. E-mail<EMAIL_ADDRESS>TUM
School of Computation, Information, and Technology. Technical University of
Munich, Munich, Germany

###### Abstract

Processing sensor data with spiking neural networks on digital neuromorphic
chips requires converting continuous analog signals into spike pulses. Two
strategies are promising for achieving low energy consumption and fast
processing speeds in end-to-end neuromorphic applications. First, to directly
encode analog signals to spikes to bypass the need for an analog-to-digital
converter (ADC). Second, to use temporal encoding techniques to maximize the
spike sparsity, which is a crucial parameter for fast and efficient
neuromorphic processing. In this work, we propose an adaptive control of the
refractory period of the leaky integrate-and-fire (LIF) neuron model for
encoding continuous analog signals into a train of time-coded spikes. The LIF-
based encoder generates phase-encoded spikes that are compatible with digital
hardware. We implemented the neuron model on a physical circuit and tested it
with different electric signals. A digital neuromorphic chip processed the
generated spike trains and computed the signal’s frequency spectrum using a
spiking version of the Fourier transform. We tested the prototype circuit on
electric signals up to 1 KHz. Thus, we provide an end-to-end neuromorphic
application that generates the frequency spectrum of an electric signal
without the need for an ADC or a digital signal processing algorithm.

_Keywords_ Spiking neural network $\cdot$ Temporal encoding $\cdot$ Sensor
signal processing $\cdot$ Fourier transform

This is the version of the article after acceptance and before editing, as
submitted by an author to Neuromorphic Computing and Engineering. IOP
Publishing Ltd is not responsible for any errors or omissions in this version
of the manuscript or any version derived from it. The Version of Record is
available online at https://doi.org/10.1088/2634-4386/acfe36.

##  1 Introduction

The human brain processes a vast amount of sensory information quickly and
efficiently. This efficient and fast processing is currently unmatched by any
artificial system. Hence, ongoing research aims to understand and mimic the
brain’s sensory processing. An essential part of this research is encoding
incoming information into spikes [1]. Coding schemes are commonly classified
into rate coding and temporal coding, that assume information is stored in the
rate of spike trains for a given time window or in the actual timing of the
spikes, respectively. There is evidence that points toward the existence of
time coding schemes in the brain [2, 3, 4], as well as for rate coding schemes
[5, 6, 7]. These two paradigms can also be generalized for neuron populations,
where redundant information in a group of neurons allows to reduce the
inference time or the uncertainty in the signal [8]. From a conceptual point
of view, rate encoding offers robustness and can be easily generalized for
large populations [7]. In contrast, time coding is faster and more energy
efficient, as it requires few spikes. It also explains the fast reaction
times, especially for sensory pathways, that are crucial for survival [9, 10,
11]. One hypothesis of how neurons implement time coding is by carrying
information in the inter-spike interval, i.e., the time between two
consecutive spikes [12]. Alternatively, time coding schemes may rely on
reference points in time to convey information. These references include the
onset of a stimulus, spikes from other neurons, or periodic waves within the
brain activity [13, 14].

One of the aims of neuromorphic research is to represent data with discrete
spikes, mimicking the efficient operation of the brain. Therefore, spike
encoding is a crucial step in the design of neuromorphic algorithms. When
developing spiking neural networks (SNNs), the choice of the encoding scheme
depends on aspects like the nature of the input data, the network
architecture, or the neuron dynamics model used by the SNN. Neuromorphic
computing research often focuses on high-end tasks that use processed data,
e.g., classification [15], object detection [16], tracking [17], or motor
control [17, 18]. These applications typically use rate encoding, as the main
motivation is to validate neural models in terms of accuracy, rather than the
benefits of sparsity and efficiency provided by temporal encoding schemes
[19]. However, energy and time efficiency becomes more relevant for low-level
neuromorphic applications that directly process sensor data. This is the case
of embedded systems where the pool of energy is limited, such as automotive
applications[20, 21]. There are recent examples of neuromorphic computing
algorithms that deal with low-end tasks and are applied to sensor data, such
as LiDAR [20], event-based cameras [22], FMCW radar [21], electrocardiogram
signals [23], or microphones [24].

The interface between the sensor and the neuromorphic chip is crucial for
obtaining a high energy efficiency and processing speed. When dealing with
analog data, the most efficient approaches directly operate on the sensor
signals by using ad-hoc circuits that encode them to spike trains. The signal-
to-spike encoding mechanisms are typically application-specific, as they need
to be compatible with the SNNs afterward. The idea of encoding analog signals
to spikes for obtaining efficient implementations is not new. In [23] and
[25], authors use temporal contrast to identify positive and negative
gradients, which discriminate high and low frequencies in the analog signal.
[26] introduces a VLSI circuit for implementing the interspike-interval (ISI)
mechanism for encoding analog signals. The authors in [27] and [28] introduce
silicon neurons for generating different spike bursting behaviours, similar to
those observed in biological neurons. We refer the interested reader to [29]
for getting an overview of the different analog circuits used in neuromorphic
systems for real-time applications, and to [30] for a comparison of the
different encoding approaches.

Some neuromorphic applications focus on generating the frequency spectrum of
the incoming signal [24, 31, 32]. In these examples, an analog-to-digital
converter (ADC) samples the sensor data and an SNN computes a higher-level
algorithm on a digital neuromorphic chip afterward. To maximize the efficiency
of these applications, the ADC should be removed by directly encoding the
input signal into a spike train compatible with the digital neuromorphic chip.
For the specific case of an SNN that implements the Fourier transform (FT)
[32], the network needs spikes referenced to a periodic signal, as the FT
algorithm works with data sampled uniformly over time. Thus, the best-suited
temporal approach for these applications is phase encoding, as ISI and
temporal contrast lack a signal that serves as reference over the time
dimension. Additionally, the results in [30] show the benefits of phase
encoding in terms of accuracy and efficiency. The benefits are more clear when
comparing with rate-coded approaches, as phase coding needs down to $6.5$
times less spike operations for processing data [33], resulting in a lower
energy footprint [19].

The work in [32] proposed an SNN that computes a mathematically equivalent
version of the FT by using time encoding. The authors implemented the spiking
FT (S-FT) on a digital neuromorphic chip and validated it on automotive radar
data. Here, we propose adapting the leaky integrate-and-fire (LIF) neuron
model for directly encoding analog signals to phase-encoded spikes, which the
S-FT can use on a digital neuromorphic chip. The proposed approach leads to
the implementation of an end-to-end pipeline that does not require an ADC nor
a digital processing algorithm that encodes digitized signals into spikes. The
proposed analog-to-spike encoder (ASE) employs phase encoding, i.e., each
sample is represented by the time difference between the spike and a reference
pulse. We implemented the ASE on a physical circuit and validated the system
with real data by comparing the reconstructed data from the spikes with the
original data. We modeled the nature of the error of the ASE for different
scenarios. Moreover, we implemented the S-FT on the neuromorphic chip
SpiNNaker 2 [34] and ran it with the ASE output spikes. The results show that
the chip can use the spike train that results from encoding a periodic wave.
To our knowledge, we provide the first end-to-end neuromorphic pipeline for
generating the frequency spectrum of analog signals without employing an ADC.

##  2 Analog-to-spike encoder

The LIF is a century-old single-compartment neuron model that mimics the
behavior of biological neurons [35]. Its simplicity makes it a widely used
artificial neuron in neuromorphic engineering. The model is based on an $R-C$
circuit that charges with the incoming current and discharges in the absence
of input, resembling the flux of ions in a biological neuron through its
dendrites and the constant leak of them over time through its membrane. We can
express the dynamics of the LIF as

$C\frac{du}{dt}+\frac{u(t)-u_{\text{rest}}}{R}=i_{\text{in}}(t)\,,$ (1)

where $u(t)$ is the neuron’s membrane potential, $u_{\text{rest}}$ is the
resting potential, and $i_{\text{in}}(t)$ is the input current to the neuron.
The LIF model does not include the depolarization and hyperpolarization
mechanisms on a biological neuron when it generates a spike. Instead, it is
necessary to manually add the behavior of the spike generation and refractory
period. A spike is triggered at time $t_{s}$ when $u(t)$ crosses the threshold
voltage $u_{\text{th}}$. After the spike generation, $u(t)$ is reset and fixed
to $u_{\text{rest}}$ for the neuron’s refractory period $t_{\text{refr}}$,

$\displaystyle t_{s}:u(t_{s})\geq u_{\text{th}}\quad\text{and}$ (2)
$\displaystyle u(t)=u_{\text{rest}}\quad\text{for}\quad
t_{s}<t<t_{s}+t_{\text{refr}}\,.$ (3)

As artificial sensors often provide information in the form of a voltage, we
modify (1) so the evolution of the membrane voltage depends on an input
voltage $u_{\text{in}}$. We also set $u_{\text{rest}}=0$ for simplicity,

$RC\frac{du}{dt}+u(t)=u_{\text{in}}(t)\,.$ (4)

In the remainder of the paper, we use the time constant $\tau=RC$. By
integrating (4) and setting $u(t_{s})=u_{\text{th}}$, we obtain an encoding
function $f(u_{\text{in}})$ that maps the input voltage $u_{\text{in}}$ to a
spike time $t_{s}$ after onset,

$t_{s}=f(u_{\text{in}})=-\tau
log\left(1-\frac{u_{\text{th}}}{u_{\text{in}}}\right)\,.$ (5)

Accordingly, the ideal decoding function is given by the inverse of
$f(u_{\text{in}})$,

$u_{\text{in}}=f^{-1}(t_{s})=\frac{u_{\text{th}}}{1-e^{-t_{s}/\tau}}\,.$ (6)

We implemented the LIF neuron model (4) for generating a spike train from
input analog signals. We encoded the information into precise spike times
using (5) and controlling the refractory period according to a reference
signal.

Fig. 1 shows a simplified schematic of the electric circuit we used to
implement the encoder. This circuit served as a proof-of-concept for
implementing an end-to-end pipeline. For implementing a high-performance
circuit, we refer the interested reader to literature focused on the VLSI
design of silicon neurons [29].

Figure 1: Simplified schematic of the electric circuit used for implementing
the LIF-based encoder. The $R-C$ formed by $R_{\text{LIF}}$ and
$C_{\text{LIF}}$ circuit generates the $u(t)$ according to (4). $C1$ compares
$u(t)$ with $u_{th}$ and generates spike events. The two D-type flip-flops
synchronize the output of $C1$ with the spike reading hardware, which controls
the reset of the spike s(t) and refractory $S_{\text{refr}}$ signals. The
second flip-flop regulates the working cycle of the LIF circuit.

###  2.1 Encoding analog voltages with temporal spikes

By default, the LIF dynamics described in (4) generates a spike rate
proportional to the input voltage $u_{\text{in}}(t)$. In our approach, we
sample the continuous signal $u_{\text{in}}(t)$ with a constant sampling time
$T_{S}$ by generating a single spike per sample. We repeat the spike
generation process for successive time windows. Thus, the $m^{\text{th}}$
spike represents the signal $u_{\text{in}}$ during the time range
$[t_{m},t_{m}+T_{S}]$. This is inspired by the phase encoding technique
hypothesized for the encoding of information in certain areas of the brain
[14]. A binary variable $S_{\text{refr}}$ forces $u(t)$ to stay in a
refractory state,

$u(t)=u_{\text{rest}},\quad\mbox{iff}\;S_{\text{refr}}=1\,.$ (7)

We achieve an adaptive refractory period, $t_{\text{refr}}=T_{S}-t_{s}$, by
controlling the set and reset time of $S_{\text{refr}}$ for the
$m^{\text{th}}$ sample,

$S_{\text{refr}}=\begin{cases}1&\text{if }t_{m}+t_{s}<t<t_{m}+T_{S}\\\
0&\text{otherwise.}\end{cases}$ (8)

This encoding mechanism requires a periodic reference signal $CLK_{S}$ (see
Fig. 1) that defines the sampling time $T_{S}$ and the end of the refractory
period. We assume the input voltage to be constant during the sampling time of
the ASE, $u_{in}(t)=U_{in}$, as the neuron dynamics is faster than the rate of
change of the input signal.

Figure 2: ASE signals for the generation of one spike. The top plot represents
the membrane voltage for the n-th spike in the time series, the second plot
represents the refractory state of the neuron, the third plot represents the
output spike train, and the bottom plot represents the sampling clock of the
digital neuromorphic chip that uses the spikes generated by the ASE.

The sampling time $T_{N}$ of the hardware component collecting the spikes
determines the spike time resolution $N$,

$N=\frac{T_{S}}{T_{N}}\,.$ (9)

###  2.2 Parameter tuning

The encoder (5) has two parameters that we can tune in order to map a voltage
range $\left[u_{\text{min}},u_{\text{max}}\right]$ onto a time range
$\left[t_{\text{min}},t_{\text{max}}\right]$, namely the time constant $\tau$,
and the threshold voltage $u_{\text{th}}$. Due to the dynamics of the neuron
model, we need to choose $u_{\text{min}}>u_{\text{th}}>0$ to ensure a finite
time range. This leads to $t_{\text{min}}>0$ (see (5)) and hence introduces a
waiting time before the first informative spike can arrive. We can use $\tau$
to reduce $t_{\text{min}}$, but this would also lead to a smaller time range,
as $(t_{\text{max}}-t_{\text{min}})\propto\tau$. To optimize the resolution in
the encoding time window $\Delta t$, we aim for a small waiting time

$\displaystyle
t_{\text{wait}}=t_{\text{min}}=-\tau\ln\left(1-\frac{u_{\text{th}}}{u_{\text{max}}}\right)$
(10)

and a large spike time range

$\displaystyle
t_{\text{spk}}=t_{\text{max}}-t_{\text{min}}=-\tau\ln\left(\frac{u_{\text{max}}(u_{\text{min}}-u_{\text{th}})}{u_{\text{min}}(u_{\text{max}}-u_{\text{th}})}\right).$
(11)

Hence, by maximizing the ratio

$\displaystyle\mu=\frac{t_{\text{max}}-t_{\text{min}}}{t_{\text{min}}}=\frac{t_{\text{spk}}}{t_{\text{wait}}}\,,$
(12)

we can ensure to find an optimal value for $u_{\text{th}}$ (see Fig. 3). The
ratio (12) is independent of $\tau$. Therefore, we can only use $\tau$ as a
scaling factor to fit the time range into the encoding window $\Delta t$. A
voltage threshold $u_{\text{th}}\sim u_{\text{min}}$ maximizes the ratio.

Figure 3:  Voltage dynamics of the ASE for the corner cases. The plot shows
the curves for the maximum and minimum possible inputs, which generate spikes
at the minimum and maximum spike times, respectively. The dashed lines
represent the additive thermal noise (22).

The overall encoding performance also depends on the decoding and the
subsequent algorithms. By tuning the encoding/decoding parameters, we optimize
the process. We establish three pipelines to evaluate the accuracy of the
coding process. First, a coding scheme using the proposed encoding $f(y)$ (5)
and its inverse as decoding $f^{-1}(t)$,

$\displaystyle y\xrightarrow{f}t\xrightarrow{f^{-1}}\hat{y}=y\,.$ (13)

Second, a coding scheme using the proposed encoding $f(y)$ (5) and a linear
decoding $g^{-1}(t)$,

$\displaystyle y\xrightarrow{f}t\xrightarrow{g^{-1}}\hat{y}\,.$ (14)

Finally, we assess the encoder performance using a spiking algorithm $A_{s}$
that takes linearly encoded spike times as input and produces linearly encoded
spike times as output. We apply the algorithm on spike times encoded as
$t=f(x)$ (5). To evaluate the pipeline, we compare the decoded spiking output
$g^{-1}(A_{s}(t))=\hat{y}$ with the output of a non-spiking version of the
algorithm $A(x)=y$,

$\displaystyle x\xrightarrow{f}t\mapsto
A_{s}(t)\xrightarrow{g^{-1}}\hat{y}\,.$ (15)

For (13), the decoding does not need tuning because it is the inverse of the
encoding (5). For (14) and (15), we tune the parameters of the encoding and
decoding to minimize the error

$\displaystyle\varepsilon=\int_{y_{\text{min}}}^{y_{\text{max}}}|\varepsilon(y)|dy=\int_{y_{\text{min}}}^{y_{\text{max}}}|y-\hat{y}|dy\,.$
(16)

We evaluated the encoding on the S-FT algorithm [32]. The S-FT network relies
on linear coding and employs current-based LIF neurons without leak. The model
assumes that the input consists in phase-encoded spikes that follow the
encoding

$\displaystyle t_{s}$ $\displaystyle=g(y)=t_{\text{lin,
min}}+\frac{t_{\text{lin, max}}-t_{\text{lin,
min}}}{y_{\text{max}}-y_{\text{min}}}(y_{\text{max}}-y)$ (17)

and decoding

$\displaystyle y$
$\displaystyle=g^{-1}(t_{s})=y_{\text{max}}-\frac{y_{\text{max}}-y_{\text{min}}}{t_{\text{lin,
max}}-t_{\text{lin, min}}}(t_{s}-t_{\text{lin, min}})\,.$ (18)

By calculating the error

$\displaystyle\varepsilon_{\text{lin}}=\int_{y_{\text{min}}}^{y_{\text{max}}}|y-\hat{y}|dy=\int_{y_{\text{min}}}^{y_{\text{max}}}|y-g^{-1}(f(y))|dy$
(19)

with the encoding $f(y)$ and decoding $g^{-1}(t)$, we see that the
multiplicative time constant $\tau$ cancels out. The error depends on the
decoder’s time range $[t_{\text{lin, min}},t_{\text{lin, max}}]$, the voltage
threshold of the encoder $u_{\text{th}}$ and the input voltage range
$[u_{\text{min}},u_{\text{max}}]$, where the latter is fixed by the given
setup. Due to the complexity of the relationship between the tunable
parameters $u_{\text{th}}$, $t_{\text{lin, min}}$ and $t_{\text{lin, max}}$,
we propose a loss function

$\displaystyle\mathcal{L}=\alpha\varepsilon_{\text{lin}}-\mu\,,$ (20)

that we can use to determine the optimal parameters, where $\alpha$ defines
the weight of the linear error w.r.t. the time ratio, i.e. a small $\alpha$
puts focus on obtaining good time ratios, whereas big $\alpha$ favors small
linear errors. By setting a voltage threshold $u_{\text{th}}$ that optimizes
the time ratio (12), the decoding parameters $t_{\text{lin, min}}$ and
$t_{\text{lin, max}}$ can be determined by directly minimizing the error
$\varepsilon_{\text{lin}}$.

##  3 Validation experiments

This section covers the experiment results for the proposed ASE. We
implemented a prototype on an electric board with standard components. We
tested the three scenarios introduced in section 2.2 with signals obtained
from a function generator. We collected the spikes and generated the reset
signals for the encoder with a microcontroller from the dsPIC33CK family. The
minimum sampling time of the microcontroller is $1.48\,\mu$s, which determines
the resolution $T_{N}$ of the spike times. We finally implemented the S-FT
algorithm on the SpiNNaker 2 chip [34] and ran it with the spikes collected by
the microcontroller.

We powered the circuit with $5$ V, and the input voltage signal was in the
range of $[1,5]$ V. Fig. 4 shows a measurement of the ASE on an oscilloscope
for two consecutive spikes.

Figure 4: Measurement on a oscilloscope of the ASE membrane voltage $u(t)$,
the threshold voltage $u_{\text{th}}$, and the spike event signal $s(t)$ in
red, yellow, and blue, respectively. The measurements show the signals for two
consecutive samples.

###  3.1 Decoding of flat voltages with an ideal decoder

We tested the ASE functionality using constant voltage signals as input. We
collected the generated spikes and applied the ideal decoding function (6) to
the recorded spike times. In this case, the absolute error in the measurement
is mainly due to the quantization error and the circuit’s thermal noise.

The spike quantization introduces a time shift $\delta t_{\text{q}}$ to the
actual spike time $t_{s}$ due to the finite resolution of the digital hardware
reading the spike events. Assuming that the spike time bins are uniformly
distributed with a period of $T_{N}$, we can theoretically model this error
with an average sampling error

$\delta t_{\text{q}}=\frac{T_{N}}{2}\,.$ (21)

On the other hand, voltage dynamics are noisy and voltage fluctuations can
alter the threshold crossing time. The thermal error $\delta
t_{\text{thermal}}$ on the spike time highly depends on the voltage dynamics.
For simplification, we model the fluctuations as a constant added to the
actual membrane voltage, $u(t)+\delta u_{\text{thermal}}$, and we estimate its
impact on the spike time as

$t_{s}-\delta t_{\text{thermal}}=-\tau\ln\left(1-\frac{u_{\text{th}}-\delta
u_{\text{thermal}}}{u_{\text{in}}}\right)\,.$ (22)

From (22), we can conclude that the thermal error on the spike times increases
exponentially for small input voltages $u_{\text{in}}$. Fig. 3 shows that
thermal noise typically anticipates the spike time.

The overall measured spike time is thus obtained by combining the the
quantization and the thermal noise time shifts

$\hat{t}_{\text{s}}=t_{s}+\delta t_{\text{q}}-\delta t_{\text{thermal}}\,,$
(23)

that leads to the decoding error

$\displaystyle\varepsilon(u)$ $\displaystyle=u-f^{-1}(\hat{t}_{\text{s}})$
(24) $\displaystyle=u-\frac{u_{\text{th}}}{1-(1-\frac{u_{\text{th}}\mp\delta
u_{\text{thermal}}}{u_{\text{in}}})e^{-\delta t_{\text{q}}/\tau}}\,.$ (25)

Figure 5:  Measurement errors of the ASE for encoding constant voltages
$u_{\text{in}}\in\left[1,5\right]$ V with varying
$u_{\text{th}}\in\left[0.1,0.5,0.75,0.9\right]$ V. Top: In dark blue, decoding
error $\varepsilon_{u}=|u_{\text{in}}-f^{-1}(\hat{t_{s}})|$ for measured spike
times $\hat{t_{s}}$. In light blue, modelled quantization error
$\varepsilon_{u}=|u_{\text{in}}-f^{-1}(t_{s}+\delta t_{q})|$ with ideal spike
time $t_{s}=f(u_{\text{in}})$. Bottom: normalized error
${\varepsilon_{t}}_{s}=|t_{s}-\hat{t}_{s}|/T_{N}$ between the measured spike
times $\hat{t_{s}}$.

(a)

(b)

Figure 6: Decoding of the spikes obtained from the ASE when feeding constant
input voltages between $1$ and $5$ V, when using (a) an ideal decoding scheme
for $u_{\text{th}}$ values of $0.1,0.5,0.75,$ and $0.9$ V, respectively; and
(b) a linear decoding function for $u_{\text{th}}$ values of $0.1$ and $0.75$
V, respectively. Moreover, the second experiment was also run for a narrow
voltage input range, namely $2$ to $5$ V. The input voltage is shown in red as
reference, as it is the target value for the decoding. Figure 7: On top,
mapping from voltages $U_{in}$ to spike times using the ASE model (5) and a
linear time latency decoder (17), in dark blue and light blue, respectively.
For ease of comparison, the spike times have been normalized for all
experiments, $t_{s}/t_{\text{max}}$. At the bottom, the decoding error
$\varepsilon$ when using (18) for decoding the ASE spike times. The parameters
$t_{\text{min}}$ and $t_{\text{max}}$ for the linear decoded are calculated
with the optimizer described in section 2.2 for minimizing the cumulative
decoding error $E$. The figure depicts the mapping for $u_{\text{min}}=1\,V$
and $2\,V$, on the two leftmost and two rightmost plots, respectively. For
each $u_{\text{min}}$, we tested the ASE for $u_{\text{th}}=0.1\,V$ and
$0.75\,V$.

This experiment converted constant voltages in the working range $[1,5]$ V to
spikes. We ran the experiment for four different setups with different
threshold voltage: $u_{\text{th}}=[0.1,0.5,0.75,0.9]$ V (see Fig. 5).

We measured the absolute decoding error as the difference between the input
voltage $u_{\text{in}}$ and the voltage $\hat{u}_{\text{in}}$ that results
from decoding the spike times,

$\varepsilon_{u}=|u_{in}-\hat{u}_{in}|\,.$ (26)

Moreover, we also measured the error in the spike times by comparing the
obtained spike times $\hat{t}_{s}$ and the ideal spike times
$t_{s}=f(u_{\text{in}})$ calculated with (5),

$\varepsilon_{t_{s}}=|t_{s}-\hat{t}_{s}|/T_{N}\,,$ (27)

where we use $T_{N}$ as a normalization coefficient for making the error
comparable across experiments. Fig. 5 depicts $\varepsilon_{u}$ and
$\varepsilon_{t_{s}}$ for the ASE when using the values of $u_{\text{th}}$
from the previous experiment. We can see that the quantization error is high
for high input voltages, whereas the thermal noise is low. On the other hand,
low input voltages increase the probability of earlier spikes due to thermal
noise, but low quantization errors in this area make spike times more precise.
These errors are inherent to the used hardware and limit the accuracy of the
encoding regardless of the approach followed for decoding the spikes.

We offer in Fig. 6(a) a visual comparison between the decoded voltages
$\hat{u}_{\text{in}}$ and the input voltages $u_{\text{in}}$ for the described
experiment.

###  3.2 Decoding of flat voltages with a linear decoder

We tested the ASE setup from the previous section with a linear decoder (18).
We used the optimization strategy introduced in section 2.2 to find the
parameters from (18) that best fit the curve from the ASE. We used a
differential evolution
method111https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html
that minimizes $\mathcal{L}$ (20) by adjusting $k_{1}$ and $k_{2}$ parameters
that fix the decoding time limits

$t_{\text{lin, min}}=t_{\text{min}}(1+k_{1})\,,$ (28)

and

$t_{\text{lin, max}}=t_{\text{max}}(1+k_{2})\,.$ (29)

As the spike times can only be real-valued, the hard boundaries for $k_{1}$
and $k_{2}$ are $[-1,\infty]$. In our implementation, we chose the boundaries
$k_{1},k_{2}\in[-1,2]$.

For testing the decoding, we reconstructed the original signal
$\hat{u}_{\text{in}}=g^{-1}(\hat{t}_{s})$ from the obtained spike times
$\hat{t}_{s}$ with the inverse of the linear encoding function $g^{-1}$ (18).
We evaluated the reconstruction error in terms of the root mean squared error
between the original signal $u_{\text{in}}(t)$ and the reconstructed signal
$\hat{u}_{\text{in}}(t)$

$E=\sqrt{\frac{\sum_{m}^{M}(u_{\text{in},m}-\hat{u}_{\text{in},m})^{2}}{M}}\,,$
(30)

where $M$ is the number of samples collected during the experiment.

Fig. 7 depicts the decoding results for four scenarios with different
$u_{\text{min}}$ and $u_{\text{th}}$. A higher $u_{\text{min}}$ leads to a
narrower input range, which directly affects the output dynamic range of the
encoder, i.e., narrow voltage ranges increase the waiting time
$t_{\text{wait}}$ until a spike can take place. On the other hand, the mapping
of the ASE has a smaller curvature, which diminishes the error when using a
linear decoder (18). These two effects are counterbalanced with the threshold
voltage, as a higher $u_{\text{th}}$ leads to a higher dynamic range, and
lower values reduce the decoding error.

After validating the optimizer on the theoretical scenarios, we applied the
fitted curves to data from the experiments with real data in section 3.1. Fig.
6(b) shows the obtained decoded voltages to four scenarios, with
$u_{\text{th}}=[0.1,0.75]$ V, and $u_{\text{min}}=[1,2]$ V. Results show that
narrowing down the input voltage range is the best way of improving the
decoding error at the expense of reducing the output dynamic range. The
results for $u_{\text{min}}=1$ V also show that a higher $u_{\text{th}}$ leads
to a larger curvature and thus increases the output dynamic range. This effect
is reduced for the results for $u_{\text{min}}=2$ V because the change in the
ratio $u_{\text{th}}/u_{\text{min}}$ is smaller for the shown experiments.
Increasing $u_{\text{th}}$ to values closer to $u_{\text{min}}$ would bring
more significant improvements to the output dynamic range.

###  3.3 Computing the S-FT with the ASE output

We tested the performance of the ASE for an end-to-end pipeline that computes
the frequency spectrum of the input signal. For the input, we used a function
generator that creates signals consisting of one sinusoidal component. These
signals can be defined as

$h(t)=A\sin{2\pi\nu t}+B\,,$ (31)

where $A$, $\nu$, and $B$ are the amplitude, frequency, and offset of the
sinusoidal wave, respectively. We connected the function generator to the ASE,
generating one spike per sampling period $T_{S}$. Finally, we ran the S-FT
algorithm on a SpiNNaker 2 board with the spikes generated from the ASE.

We tested the implementation for frequencies in the range
$\nu\in[25\,\text{Hz},1\,\text{kHz}]$. Same as for the experiment in Section
3.2, we created experiments with two different setups for the input voltage
range. Namely, a working range of $[1,5]$ V, and a working range of $[2,5]$ V,
i.e., we used the parameters $A=2$ and $B=3$, and $A=1.5$ and $B=3.5$,
respectively. We fixed $u_{th}=0.1$ V and $\tau=3$ ms, and the resolution per
spike to $N=100$ time steps. These parameters lead to maximum spiking times of
$315$ and $155\,\mu$s for the wide and narrow input voltage ranges,
respectively.

In Fig. 8, we compare the result of decoding the spikes of the ASE using an
ideal decoder and a linear decoder for a wide and narrow input voltage range,
respectively. We also show on the right the result of the S-FT for both input
ranges. The results correspond to a setup with a frequency of $\nu=500$ Hz. We
repeated this experiment for frequencies
$\nu\in[25,50,75,100,250,500,750,1000]$ Hz and calculated the root mean
squared error (30) between the obtained frequency spectrum and an FFT computed
with a synthetic sine wave sampled with an ideal ADC. We show the error for
the different frequencies in Fig. 9.

Figure 8: Output of the ASE for a sinusoidal wave of $500\,Hz$. The top plots
correspond to a wave with amplitudes ranging between $1\,\mbox{and}\,5$ V, and
the bottom plots correspond to a wave with amplitudes ranging between $2$ and
$4$ V. From left to right, the plots correspond to the ideal decoding of the
spikes, the decoding using a linear function, and the result of applying the
S-FT on the SpiNNaker 2 board to the output spikes, respectively. The ASE was
tuned with $\tau=3$ ms and $u_{\text{th}}=0.1$ V. The sampling frequency was
$3$ and $5.5$ kHz, respectively, and each sample had a resolution of $100$
time steps. Figure 9: Error of the frequency spectrum when encoding voltages
with the ASE and feeding the spikes to the S-FT in the SpiNNaker 2 board. The
plots show the error for frequencies ranging from $25$ Hz to $1000$ Hz, both
for wide and narrow input voltage ranges, $V=[1,5]$ V and $[2,5]$ V,
respectively.

##  4 Discussion

We modelled the expected output of the proposed ASE from a theoretical
perspective and validated its behaviour with experiments on analog signals. In
Section 3.1 we modelled the error in the spike encoding as a combination of a
quantization error and the thermal error present in analog circuits. We proved
that the quantization error grows exponentially with the value of the input
voltage, and the experiments show evidence that this is the dominating error
in the output spikes. The experiment in Section 3.1 tested this hypothesis by
converting input voltage signals in the range $[1,5]$ V to spikes and decoding
them back into voltages. We show the measured error in the decoding on the top
row of Fig. 5 and compare it with the quantization error equivalent to half of
a time step in the spike times, i.e., the difference between the input voltage
and the voltage that the corresponding spike actually represents. We also show
the errors in the spike times on the bottom row of Fig. 5. The results
indicate that errors in the spike times, mostly due to thermal error, are
larger for low input voltages. However, these errors do not have a big effect
on the decoding error, as it is mostly dominated by the quantization error.
Finally, Fig. 6(a) shows the relationship between input and decoded voltages
for the considered voltage range and for four different values of
$u_{\text{th}}$. We observe that the decoding stays close to the target value,
and that errors are more significant towards high values of $u_{\text{in}}$.

The experiment in in Section 3.2 fit a linear decoder to the mapping curve of
different setups of the ASE. This step is necessary before implementing the
ASE together with the S-FT, as the latter assumes phase-encoded input spike
times. We fit a linear decoder for four different ASE setups, where we vary
$u_{\text{th}}$ and $u_{\text{min}}$, and we evaluated the result in terms of
the decoding error $E$ and the dynamic output range $\mu$. We can observe that
both parameters have a big influence on both indicators. Large parameter
values lead to small errors and narrow dynamic ranges. In general, the actual
choice of parameters will depend on the requirements and limitations of the
application. If the SNN that processes the spikes is not limited to a linear
encoding function the design should aim to maximize $\mu$, as the decoding
error will stay close to zero (see top and bottom plots in Fig. 6).

The experiment described in Section 3.3 tested the ASE together with a digital
neuromorphic chip for computing the frequency spectrum of the input analog
signal. As we aimed to minimize the error in the generated spectrum, we chose
$u_{\text{th}}\approx 0.1$ V. We run experiments for input signal frequencies
up to $1$ kHz and $u_{\text{min}}=1$ and $2$ V, and fit a linear decoding
function for each setup. Fig. 8 depicts the results for $\nu=500$ Hz. The non-
linear behaviour of the ASE (5) can be approximated by a hyperbolic function
assuming small ratios $u_{\text{th}}/u_{\text{in}}\ll 1$, which leads to a
compression of the data for low values of $u_{\text{in}}$, and an expansion of
the encoding range for high values of $u_{\text{in}}$. This translates into
the appearance of harmonics in the resulting S-FT, which is specially
noticeable for larger input dynamic ranges (see top right plot of Fig. 8). We
also plot the total error of the generated frequency spectrum with that of an
FFT applied on a synthetic signal sampled with an ideal ADC, i.e., simulated
on a PC without any added noise. The error stays relatively low, and its
mostly due to the usage of a decoding function different to the encoding one.
We observe bigger errors for higher frequencies and higher input voltage
ranges. For $u_{\text{in}}=[2,5]$, this trend stops at $\nu=500$ Hz , as after
this frequency the second harmonic of the main signal is out of the range of
the generated FT. This source of noise is easy to model, and this error could
be mitigated with a post-processing algorithm.

##  5 Conclusion

We proposed adapting the widely used LIF neuron model for mapping analog
signals into spike trains using phase encoding. By using an adaptive
refractory period, the output spike train keeps a notion of the sampling rate,
which is a fundamental requirement for applying FT-based frequency spectrum
analysis techniques.

We constructed a circuit prototype and ran it on simple periodic waves to
validate the model. Our experiments show that the proposed approach can encode
voltages with high accuracy. Moreover, we used the output of the ASE for
running a spike-based FT on the digital neuromorphic chip SpiNNaker 2. This
implementation is more efficient and simpler than previous approaches, as it
does not require an ADC between the sensor and the digital chip.

Future work shall focus on generating a comprehensive benchmark of the ASE by
creating a VLSI version of the circuit and assessing its performance with
complex data obtained from actual sensors. Those benchmarks shall include
critical parameters like energy efficiency and the robustness to the noise of
the ASE, and compare them with the equivalent parameters of ADCs.

## Acknowledgments

The authors would like to thank Alexander Lenz and Susanne Junghans for their
expertise and input, which was very valuable for implementing and testing the
designed circuit. The authors would also like to thank the Technical
University of Dresden for granting them access to the SpiNNaker 2 chip and its
associated resources.

This research has been funded by the Federal Ministry of Education and
Research of Germany in the framework of the KI-ASIC project, with
identification number 16ES0995.

## References

  * [1] R. Brette, “Philosophy of the spike: rate-based vs. spike-based theories of the brain,” Frontiers in systems neuroscience, p. 151, 2015.
  * [2] W. Bair and C. Koch, “Temporal precision of spike trains in extrastriate cortex of the behaving macaque monkey,” Neural Computation, vol. 8, no. 6, p. 1185–1202, 1996.
  * [3] D. S. Reich, F. Mechler, K. P. Purpura, and J. D. Victor, “Interspike intervals, receptive fields, and information encoding in primary visual cortex,” The Journal of Neuroscience, vol. 20, no. 5, p. 1964–1974, 2000.
  * [4] R. Brasselet, S. Panzeri, N. K. Logothetis, and C. Kayser, “Neurons with stereotyped and rapid responses provide a reference frame for relative temporal coding in primate auditory cortex,” The Journal of Neuroscience, vol. 32, no. 9, p. 2998–3008, 2012.
  * [5] D. H. Hubel and T. N. Wiesel, “Receptive fields and functional architecture of monkey striate cortex,” The Journal of Physiology, vol. 195, 1968.
  * [6] D. J. Heeger, “Normalization of cell responses in cat striate cortex,” Visual Neuroscience, vol. 9, no. 2, p. 181–197, 1992.
  * [7] M. London, A. Roth, L. Beeren, M. Häusser, and P. E. Latham, “Sensitivity to perturbations in vivo implies high noise and suggests rate coding in cortex,” Nature, vol. 466, no. 7302, p. 123–127, 2010.
  * [8] A. Pouget, P. Dayan, and R. S. Zemel, “Inference and computation with population codes,” Annual Review of Neuroscience, vol. 26, no. 1, p. 381–410, 2003.
  * [9] P. Heil, “First-spike latency of auditory neurons revisited,” Current Opinion in Neurobiology, vol. 14, no. 4, pp. 461–467, 2004.
  * [10] T. Gollisch, “Throwing a glance at the neural code: Rapid information transmission in the visual system,” HFSP Journal, vol. 3, no. 1, p. 36–46, 2009.
  * [11] J. G. Martin, C. E. Davis, M. Riesenhuber, and S. J. Thorpe, “Zapping 500 faces in less than 100 seconds: Evidence for extremely fast and sustained continuous visual search,” Scientific Reports, vol. 8, no. 1, 2018.
  * [12] A.-M. M. Oswald, B. Doiron, and L. Maler, “Interval coding. i. burst interspike intervals as indicators of stimulus intensity,” Journal of neurophysiology, vol. 97, no. 4, pp. 2731–2743, 2007.
  * [13] G. Buzsáki and E. I. Moser, “Memory, navigation and theta rhythm in the hippocampal-entorhinal system,” Nature Neuroscience, vol. 16, no. 2, p. 130–138, 2013.
  * [14] N. Hakim and E. K. Vogel, “Phase-coding memories in mind,” PLOS Biology, vol. 16, no. 8, p. e3000012, 2018.
  * [15] B. Rueckauer, I.-A. Lungu, Y. Hu, M. Pfeiffer, and S.-C. Liu, “Conversion of continuous-valued deep networks to efficient event-driven networks for image classification,” Frontiers in neuroscience, vol. 11, p. 682, 2017.
  * [16] J. López-Randulfe, T. Duswald, Z. Bing, and A. Knoll, “Spiking neural network for fourier transform and object detection for automotive radar,” Frontiers in Neurorobotics, vol. 15, p. 688344, 2021.
  * [17] Z. Cao, L. Cheng, C. Zhou, N. Gu, X. Wang, and M. Tan, “Spiking neural network-based target tracking control for autonomous mobile robots,” Neural Computing and Applications, vol. 26, pp. 1839–1847, 2015.
  * [18] R. Stagsted, A. Vitale, J. Binz, L. Bonde Larsen, Y. Sandamirskaya, et al., “Towards neuromorphic control: A spiking neural network based pid controller for uav,” in Robotics: Science and Systems 2020, RSS, 2020.
  * [19] S. Davidson and S. B. Furber, “Comparison of artificial and spiking neural networks on digital hardware,” Frontiers in Neuroscience, vol. 15, p. 651141, 2021.
  * [20] S. Zhou, Y. Chen, X. Li, and A. Sanyal, “Deep scnn-based real-time object detection for self-driving vehicles using lidar temporal data,” IEEE Access, vol. 8, pp. 76903–76912, 2020.
  * [21] B. Vogginger, F. Kreutz, J. López-Randulfe, C. Liu, R. Dietrich, H. A. Gonzalez, D. Scholz, N. Reeb, D. Auge, J. Hille, et al., “Automotive radar processing with spiking neural networks: Concepts and challenges,” Frontiers in Neuroscience, p. 414, 2022.
  * [22] U. Rançon, J. Cuadrado-Anibarro, B. R. Cottereau, and T. Masquelier, “Stereospike: Depth learning with a spiking neural network,” IEEE Access, vol. 10, pp. 127428–127439, 2022.
  * [23] S. Gerber, M. Steiner, G. Indiveri, E. Donati, et al., “Neuromorphic implementation of ecg anomaly detection using delay chains,” in 2022 IEEE Biomedical Circuits and Systems Conference (BioCAS), pp. 369–373, IEEE, 2022.
  * [24] D. Auge, J. Hille, F. Kreutz, E. Mueller, and A. Knoll, “End-to-end spiking neural network for speech recognition using resonating input neurons,” in Artificial Neural Networks and Machine Learning–ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part V 30, pp. 245–256, Springer, 2021.
  * [25] K. Burelo, G. Ramantani, G. Indiveri, and J. Sarnthein, “A neuromorphic spiking neural network detects epileptic high frequency oscillations in the scalp eeg,” Scientific Reports, vol. 12, no. 1, p. 1798, 2022.
  * [26] C. Zhao, Y. Yi, J. Li, X. Fu, and L. Liu, “Interspike-interval-based analog spike-time-dependent encoder for neuromorphic processors,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 25, no. 8, pp. 2193–2205, 2017.
  * [27] P. Livi and G. Indiveri, “A current-mode conductance-based silicon neuron for address-event neuromorphic systems,” in 2009 IEEE international symposium on circuits and systems, pp. 2898–2901, IEEE, 2009.
  * [28] J. H. Wijekoon and P. Dudek, “Compact silicon neuron circuit with spiking and bursting behaviour,” Neural Networks, vol. 21, no. 2-3, pp. 524–534, 2008.
  * [29] E. Chicca, F. Stefanini, C. Bartolozzi, and G. Indiveri, “Neuromorphic electronic circuits for building autonomous cognitive systems,” Proceedings of the IEEE, vol. 102, no. 9, pp. 1367–1388, 2014.
  * [30] E. Forno, V. Fra, R. Pignari, E. Macii, and G. Urgese, “Spike encoding techniques for iot time-varying signals benchmarked on a neuromorphic classification task,” Frontiers in Neuroscience, vol. 16, p. 999029, 2022.
  * [31] G. Orchard, E. P. Frady, D. B. D. Rubin, S. Sanborn, S. B. Shrestha, F. T. Sommer, and M. Davies, “Efficient neuromorphic signal processing with loihi 2,” in 2021 IEEE Workshop on Signal Processing Systems (SiPS), pp. 254–259, IEEE, 2021.
  * [32] J. López-Randulfe, N. Reeb, N. Karimi, C. Liu, H. A. Gonzalez, R. Dietrich, B. Vogginger, C. Mayr, and A. Knoll, “Time-coded spiking fourier transform in neuromorphic hardware,” IEEE Transactions on Computers, vol. 71, no. 11, pp. 2792–2802, 2022.
  * [33] W. Guo, M. E. Fouda, A. M. Eltawil, and K. N. Salama, “Neural coding in spiking neural networks: A comparative study for robust neuromorphic systems,” Frontiers in Neuroscience, vol. 15, p. 638474, 2021.
  * [34] S. Höppner, Y. Yan, A. Dixius, S. Scholze, J. Partzsch, M. Stolba, F. Kelber, B. Vogginger, F. Neumärker, G. Ellguth, et al., “The spinnaker 2 processing element architecture for hybrid digital neuromorphic computing,” arXiv preprint arXiv:2103.08392, 2021.
  * [35] L. F. Abbott, “Lapicque’s introduction of the integrate-and-fire model neuron (1907),” Brain research bulletin, vol. 50, no. 5-6, pp. 303–304, 1999.


# Context-gloss Augmentation for Improving Word Sense Disambiguation

Guan-Ting Lin
National Tsing-Hua University
<EMAIL_ADDRESS>
Manuel Giambi
National Tsing-Hua University
<EMAIL_ADDRESS>

(January 2021)

###### Abstract

The goal of Word Sense Disambiguation (WSD) is to identify the sense of a
polysemous word in a specific context. Deep-learning techniques using BERT
have achieved very promising results in the field and different methods have
been proposed to integrate structured knowledge to enhance performance. At the
same time, an increasing number of data augmentation techniques have been
proven to be useful for NLP tasks. Building upon previous works leveraging
BERT and WordNet knowledge, we explore different data augmentation techniques
on context-gloss pairs to improve the performance of WSD. In our experiment,
we show that both sentence-level and word-level augmentation methods are
effective strategies for WSD. Also, we find out that performance can be
improved by adding hypernyms’ glosses obtained from a lexical knowledge base.
We compare and analyze different context-gloss augmentation techniques, and
the results show that applying back translation on gloss performs the best.

##  1 Introduction

Word Sense Disambiguation (WSD) is a fundamental task and long-standing
challenge in Natural Language Processing (NLP), which aims to find the exact
sense of an ambiguous word in a particular context Navigli (2009). Previous
approaches to WSD can be grouped into two main categories: knowledge-based and
supervised methods.

Knowledge-based WSD methods use lexical databases like WordNet Miller et al.
(1991). WordNet provides a definition for each word sense, called gloss. Gloss
has been used extensively in WSD, first in Lesk algorithm Lesk (1986) and
subsequently in many other approaches Banerjee and Pedersen (2002); Basile et
al. (2014).

As for supervised methods, traditional approaches Zhong and Ng (2010); Shen et
al. (2013); Iacobacci et al. (2016) focus on training many word expert
classifiers which specialise in disambiguating one lemma each. More recent
methods Kågebäck and Salomonsson (2016); Raganato et al. (2017b) using neural
networks aim at improving the performance in the all-word WSD task. Luo et al.
(2018a, b) is the first to use glosses during training. Those attempts show
the importance of including gloss information into the WSD task.

More recently, Huang et al. (2019) proposed a new approach named GlossBERT
that considerably improves the performance on WSD. They generate concatenated
context-gloss pairs using WordNet and then use them to fine-tune a pre-trained
BERT Devlin et al. (2018) model in a binary classification task. In addition,
an increasing number of works Du et al. (2019); Blevins and Zettlemoyer (2020)
adopt two separate BERT models to encode context and gloss respectively. These
works achieved competitive results and Du et al. (2019) also revealed that
adding gloss information can alleviate the poor performance on less frequent
senses (LFS). Guo et al. (2020) jointly encode the context and multi-glosses
of the target word to produce gloss embeddings that are almost orthogonal and
emphasize the differences between glosses, thus achieving a better
performance.

Although there has been a lot of work on WSD recently which has improved
performance and offered new insight on the problem, to the best of our
knowledge there is no research trying to do data augmentation on context-gloss
pairs and we think this is a notable research field to explore.

Data augmentation is a important technique to increase model generality and
prevent overfitting, so an increasing number of NLP data augmentation methods
have been proposed in the recent years. One popular method is back translation
Edunov et al. (2018) which leverages neural a translation model to generate
paraphrases by translating from source to target language and back.
Furthermore, Wei and Zou (2019) showed that simply replacing words with
synonyms is a straight-forward and useful data augmentation technique.
Besides, Wu et al. (2019) and Kumar et al. (2020) leverage pre-trained
language model to augment sentences and consider contextual meaning at the
same time. The above works suggest that textual data augmentation can play an
important role in improving NLP task performance.

This paper brings the following contributions:

  * 1

Consider synset relationships (hypernym and hyponym) in WordNet to augment our
training corpora.

  * 2

Investigate the potential of different word substitution augmentation
techniques applied to context-gloss pairs.

  * 3

Explore the advantages in using back translation as an augmentation method for
both gloss and context, showing that gloss back translation using German
outperforms all other methods.

  * 4

Analyze the insight gained by experimenting with several context-gloss
augmentation techniques.

##  2 Methods

Figure 1: GlossBERT

###  2.1 GlossBERT

We use a pre-trained BERT model and fine-tune it using context-gloss pairs. As
proposed by Huang et al. (2019), we use weak supervision on the context-gloss
pairs. Context here refers to a sentence in the training corpora that contains
the lemma we want to disambiguate. The input to BERT is:

$[CLS]\;context\;[SEP]\;target\;:\;gloss\;[SEP]$ (1)

As shown in Figure 1, we highlight the target word in the context by wrapping
it with the special character ”; we also prepend the target word itself to the
gloss, separated by a : character. For each lemma we want to disambiguate, we
create n records, one for each of the lemma’s senses. The label for each
example is binary. If the gloss is the correct sense, then label it as 1; the
mismatching senses are annotated as 0. We then take the final hidden state of
[CLS] as the representation of the whole sequence and add a binary
classification layer. As a result, the model can learn to find the sense that
is matched with the context among all each word’s senses. GlossBERT is the
baseline for our experiments, and all the following augmentation techniques
aim to increase the number of context-gloss pairs used to train the model.

###  2.2 Context-gloss augmentation

We apply context-gloss augmentation because we believe that there are
different ways to convey the same semantics and small variations in the
sentence, such as paraphrasing and using synonyms, maintain the meaning of the
sentence unchanged. We can use the increased number of records to prevent
overfitting.

####  2.2.1 Back translation

Back translation augmentation is a well-known technique for generating
paraphrases by translating sentences from a source language to a target
language, and then translating back into the source language. Similarly to
Edunov et al. (2018), we increase the number of training examples by adding
back translated context-gloss pairs. We use pre-trained models (see Appendix
Table 4 for details) to perform back translation and experimented with German
(De), Russian (Ru) and French (Fr) as target languages.

method | context
---|---
\- (baseline) | are you utilizing cafeteria space for company ” meetings ” or discussions ?
MLM | are you utilizing cafeteria space holding company ” meetings ” or discussions?
synonym replacement | are you use cafeteria space for company ” meetings ” or discussions?
back translation |  do you use the cafeteria for ” meetings ” or discussions in companies?
method | gloss
\- (baseline) | a formally arranged gathering
back translation (De) | a formally established meeting
hypernym | a group of persons together in one place
hypernym concatenation | a formally arranged gathering [SEP] a group of persons together in one place
Table 1:  Different types of augmentation methods applied to an example
context or gloss.

####  2.2.2 Hypernyms and hyponyms

As suggested by Luo et al. (2018b) we make use of the structured information
contained in WordNet and retrieve the glosses for the senses’ hypernyms and
hyponyms - which is to say the concepts that represent respectively a
generalisation or a specialisation of the given concept. We propose two
approaches that make use of hypernyms’ glosses. One is hypernym concatenation,
which consists in appending a hypernym’s gloss behind the original gloss
separated by the [SEP] token. The second method consists in adding a new
context-gloss pair for each original pair, with the new pair having the
original gloss replaced by the hypernym’s gloss.

####  2.2.3 Synonym replacement

One popular NLP data augmentation technique is to replace the original word
with a synonym. We leverage WordNet as the lexical knowledge base and
substitute words with their synonyms. Each word has the probability $p$ to be
replaced, and we set $p$ to 0.15. We only apply the synonym replacement method
to context because on average the length of glosses is short, and substituting
words in glosses can easily lead the glosses to significantly change meaning.

Augmentation Method | Component | f1 score
---|---|---
\- (baseline) | - | 77.9
hypernyms | gloss | 79.3
hypernyms concatenation | gloss | 79.6
hypernyms and hyponyms | gloss | 78.4
back translation (Ru) | gloss | 78.8
back translation (Fr) | gloss | 79.1
back translation (De) | gloss | 80.0
back translation (Fr+De) | gloss | 79.4
hypernyms concatenation + back translation (De) | gloss | 79.3
hypernyms concatenation + back translation (Fr) | gloss | 79.1
back translation (De) | context | 78.9
synonym replacement | context | 78.5
MLM | context | 78.3
back translation (De) | gloss + context | 79.7
Table 2:  The comparison of different context-gloss augmentation methods. We
calculate testing f1 score based on ALL noun dataset.

####  2.2.4 Contextual word embeddings (MLM)

Language models like BERT have been pre-trained using a masked language model
(MLM). We leverage this method and perform augmentation by masking random
words in context sentence and substituting them with the model’s prediction,
taken from a list of $k$ candidates. Here we set $k$ to 3.

##  3 Experiments

###  3.1 Training dataset and augmentation

Following previous work Luo et al. (2018a, b); Huang et al. (2019) we choose
SemCor3.0 as our basic training corpus, which is the largest corpus manually
annotated with WordNet senses for WSD. We focus on the disambiguation of nouns
only, exploring several data augmentation techniques on context-gloss pairs
from WordNet.

###  3.2 Evaluation datasets

We evaluate our model on the benchmark datasets proposed by Raganato et al.
(2017a) which includes five standard English all-words fine-grained WSD
datasets. They are Senseval-2 (SE2), Senseval-3 (SE3), SemEval-07 (SE7),
SemEval-13 (SE13), and SemEval-15 (SE15). After filtering these datasets so
that they contain noun only senses, we concatenate them to create a new
dataset (ALLn) which we use to evaluate our models.

##  4 Results

Table 2 shows the performance of different context-gloss augmentation methods
on the ALLn evaluation datasets.

All augmentation methods applied to gloss increase the performance. Methods
using hypernyms help performance a lot, as the f1 score increase from 0.779 to
0.793 and 0.796 depending on the method used. Whereas adding a hypernym’s
gloss as a new record might provide confounding signal that leads the model
away from the original sense’s meaning, the hypernym concatenation method
achieves better performance since it provides extra information to the
original sense. However, using both hypernyms and hyponyms can only improve f1
score to 0.784, demonstrating that hyponyms may be too generic and damage
performance.

Remarkably, back translation offers an even more significant improvement. The
f1 score increases to 0.788 when using Russian as a target language, to 0.791
when using French and to 0.800 when using German.

As for context augmentation, back translation using German also takes the f1
score to 0.789, but the performance gain is not as significant when compared
to gloss augmentation. As for the word-level augmentation methods, they both
enhance the performance but not as much as back translation, since word
substitution has a high probability of introducing grammatical errors. MLM
performs worse than synonym replacement using WordNet, with an f1 score of
0.783 and 0.785 respectively.

Even though doing back translation on gloss improves the performance
considerably, the gain is reduced when back translation is used in combination
with other methods, such as hypernym concatenation and back translation on
context.

##  5 Discussion

###  5.1 Comparison among different translation languages

We investigate German, French and Russian as target languages of back
translation. German is one of the most closely related languages to English
and like English belongs to the Germanic language family. Back translated
sentences tend to have a word by word correspondence with the original, with
most words being the same and some being replaced by a synonym. French has a
considerable lexical overlap with English, but French belongs to the Romance
language family and we can see there are some misleading statements in
French’s augmented sentences. Russian belongs to the Baltic-Slavic language
family; the sentence patterns of English and Russian have greater
dissimilarity and the augmentation noise is higher. We observe that the
meaning of some of the augmented sentences changes a lot. A few examples of
back translation by different languages are provided in the Appendix Table 3.

Our results suggest that grammatical similarity has a greater impact than
lexical similarity on the quality of translation for augmentation. Based on
our experiment we think that WSD is sensitive to noise in semantic, so it’s
better to choose target languages that don’t alter the sentence meaning too
much.

###  5.2 Augmentation on context versus gloss

Table 2 shows that back translation (De) performs better when used on gloss
compared to when it’s used on context. There might be three reasons: firstly,
glosses are generally shorter sentences, so the chance of a translation error
distorting the meaning is lower. Secondly, glosses have a more formal and
standardised register when compared to context sentences, as the latter can
originate from a varied array of sources (literary prose, newspaper article,
etc.). Lastly, context sentences often feature compound sentences and more
complex syntactic relationship, whereas glosses usually contain straight-
forward statements.

##  6 Conclusion

In this paper, we explore various types of data augmentation on context-gloss
pairs data to improve WSD performance. This is the first research work that
applies NLP data augmentation methods to WSD. We compare and analyze the
results from different augmentation techniques. In our experiments, we find
out that leveraging back translation on gloss only performs the best and
improves the f1 score to 0.800. In the future we want to verify the generality
of our findings by extending these methods to all-pos WSD and also to apply
augmentation to different baseline models.

## References

  * Banerjee and Pedersen (2002) Satanjeev Banerjee and Ted Pedersen. 2002.  An adapted lesk algorithm for word sense disambiguation using wordnet.  volume 2276, pages 136–145.
  * Basile et al. (2014) Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro. 2014.  Combining distributional semantic models and sense distribution for effective italian word sense disambiguation.
  * Blevins and Zettlemoyer (2020) Terra Blevins and Luke Zettlemoyer. 2020.  Moving down the long tail of word sense disambiguation with gloss informed bi-encoders.  pages 1006–1017.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Du et al. (2019) Jiaju Du, Fanchao Qi, and Maosong Sun. 2019.  Using BERT for word sense disambiguation.  _CoRR_ , abs/1909.08358.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.
  * Guo et al. (2020) Ping Guo, Yue Hu, and Yunpeng Li. 2020.  Mg-bert: A multi-glosses bert model for word sense disambiguation.  In _Knowledge Science, Engineering and Management_ , pages 263–275, Cham. Springer International Publishing.
  * Huang et al. (2019) Luyao Huang, Sun Chi, Xipeng Qiu, and Xuanjing Huang. 2019.  Glossbert: Bert for word sense disambiguation with gloss knowledge.  pages 3500–3505.
  * Iacobacci et al. (2016) Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016.  Embeddings for word sense disambiguation: An evaluation study.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 897–907, Berlin, Germany. Association for Computational Linguistics.
  * Kågebäck and Salomonsson (2016) Mikael Kågebäck and Hans Salomonsson. 2016.  Word sense disambiguation using a bidirectional LSTM.  In _Proceedings of the 5th Workshop on Cognitive Aspects of the Lexicon (CogALex - V)_ , pages 51–56, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020.  Data augmentation using pre-trained transformer models.
  * Lesk (1986) M. Lesk. 1986.  Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone.  In _SIGDOC ’86_.
  * Luo et al. (2018a) Fuli Luo, Tianyu Liu, Zexue He, Qiaolin Xia, Zhifang Sui, and Baobao Chang. 2018a.  Leveraging gloss knowledge in neural word sense disambiguation by hierarchical co-attention.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1402–1411, Brussels, Belgium. Association for Computational Linguistics.
  * Luo et al. (2018b) Fuli Luo, Tianyu Liu, Qiaolin Xia, Baobao Chang, and Zhifang Sui. 2018b.  Incorporating glosses into neural word sense disambiguation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2473–2482, Melbourne, Australia. Association for Computational Linguistics.
  * Miller et al. (1991) George Miller, R. Beckwith, Christiane Fellbaum, Derek Gross, and Katherine Miller. 1991.  Introduction to wordnet: An on-line lexical database*.  3\.
  * Navigli (2009) Roberto Navigli. 2009.  Word sense disambiguation: A survey.  _ACM Comput. Surv._ , 41.
  * Raganato et al. (2017a) Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017a.  Word sense disambiguation: A unified evaluation framework and empirical comparison.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 99–110.
  * Raganato et al. (2017b) Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017b.  Neural sequence learning models for word sense disambiguation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1156–1167, Copenhagen, Denmark. Association for Computational Linguistics.
  * Shen et al. (2013) Hui Shen, Razvan Bunescu, and Rada Mihalcea. 2013.  Coarse to fine grained sense disambiguation in Wikipedia.  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity_ , pages 22–31, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Wei and Zou (2019) Jason W. Wei and Kai Zou. 2019.  EDA: easy data augmentation techniques for boosting performance on text classification tasks.  _CoRR_ , abs/1901.11196.
  * Wu et al. (2019) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019.  Conditional bert contextual augmentation.  In _Computational Science – ICCS 2019_ , pages 84–95, Cham. Springer International Publishing.
  * Zhong and Ng (2010) Zhi Zhong and Hwee Ng. 2010.  It makes sense: A wide-coverage word sense disambiguation system for free text.  pages 78–83.

Language | Lemma | Back translated gloss
---|---|---
\- (original) | face | the front of the human head from the forehead to the chin and ear to ear
De | face | the front of the human head from forehead to chin and from ear to ear
Fr | face | the forehead of the human head from the forehead to the chin and the ear to the ear
Ru | face | front of a person’s head from head to chin and ear to ear
\- (original) | day | time for Earth to make a complete rotation on its axis
De | day | time for the Earth to turn completely around its axis
Fr | day | it’s time for Earth to make a complete rotation on its axis
Ru | day | the time when the Earth will complete a full rotation on its asteroid
\- (original) | account | a short account of the news
De | account | a short report on the news
Fr | account | a brief account of the current situation
Ru | account | news at a glance
Table 3:  Comparison of back translated glosses in German (De), French (Fr) and Russian (Ru).  target language | open-source package | target model | source model
---|---|---|---
german | nlpaug | transformer.wmt19.en-de | transformer.wmt19.de-en
russian | nlpaug | transformer.wmt19.en-ru | transformer.wmt19.ru-en
french | transformers | Helsinki-NLP/opus-mt-en-fr | Helsinki-NLP/opus-mt-fr-en
Table 4:  Open-source packages and models used for back translation

##  Appendix A Experimental Setup

We modify the maximum number of tokens input to BERT from 512 (GlossBERT
usage) to 128 because we have checked that the more than 99 percent of
context-gloss pairs have less than 128 tokens. For the training hyper-
parameters, we set the batch size to 96, learning rate to 2e-5 and the number
of epochs to 4 for all the experiments.

##  Appendix B Error Analysis

A deeper analysis of our results reveal some patterns worth reporting. We
specifically look at our failed predictions and find there are some cases
where the models fails to infer the correct sense and instead predicts a
completely unrelated one. However, in the remaining failed predictions, we
notice that in a sizeable portion of instances the predicted sense is very
similar to the true sense, often with the two senses sharing a very similar
gloss. In some cases the similarity is so pronounced that the uniqueness or
correctness of the true sense can be put into discussion. This suggests that
WordNet might be too fine-grained in how it partitions lemmas into senses for
any practical application of WSD.