# Dynamic Control of Plasmonic Colors by Voltage Actuation MEMS Cantilevers
for Optical Display Applications

Zhengli Han1,†,∗ Christian Frydendahl1,† Noa Mazurski1 and Uriel Levy1,∗

###### Abstract

Abstract: Conventional optical displays using ITO (indium tin oxide) and LC
(liquid crystal) materials present a lot of challenges in terms of long-term
sustainability. We show here how it is possible to generate a cost-effective
and CMOS compatible fast and full range electrically controlled RGB color
display by combining transmission based plasmonic metasurfaces with MEMS
(Microelectromechanical systems) technology, using only two common materials:
Aluminum and silicon oxide. White light is filtered into red, green, and blue
components by plasmonic metasurfaces made of aluminum nanohole arrays, and the
transmission through each color filter is modulated by MEMS miniaturized
cantilevers fabricated with aluminum and silicon oxide on top of the color
filters. We show that the relative transmission of a color subpixel can be
freely modulated from 35% to 100%. Our pixels can also operate well above 800
Hz, enabling future ultrafast displays. Our work provides a road to future
circular economic goals by exploiting advances in structural colors and MEMS
technologies to innovate optical displays.

Department of Applied Physics, The Faculty of Science, The Center for
Nanoscience and Nanotechnology, The Hebrew University of Jerusalem, Jerusalem
91904, Israel.

These authors contributed equally to this work

<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
## Introduction

Optical displays have become an ubiquitous technology in daily life, being an
essential component of smart phones, computers, televisions, etc. These same
devices make up roughly half of the 50 million tonnes of e-waste produced
every year globally1. In estimate, less than 20% of the mass of world’s annual
e-waste is recorded as being recycled1. For example, typically only metals are
extracted from plastic circuit boards. The remaining 40 million tonnes of
waste is mainly unaccounted for, generally ending up in burn pits or land
fills in developing countries where many of the toxic elements contained
within (such as indium, lead, mercury, etc.) pose serious threats to human and
environmental health. Annual e-waste production is predicted to grow to over
120 million tonnes by 2050, with the majority of this increase being from
devices reliant on displays1, 2, 3. One of the main reasons for the low rate
of recycling of displays, is the cost associated with the process due to the
large variety of materials included in a device3, making ’virgin’ materials
favorable over recycled materials in terms of cost - despite the fact that it
is up to 10 times as energy efficient to recycle e.g. metals over extracting
new metals from ore1. There is thus an urgent need for alternative display
technologies that are easier to recycle and are financially feasible to be
incorporated into a circular consumer electronics economy4, 5, 6, 7 \- not
just to preserve finite resources, but also to lower total energy usage of the
manufacturing industry and lower total global carbon emissions1. In
particular, some of the most critical materials to find replacements for are
the liquid crystals (LCs) and the indium tin oxide (ITO) which are used for
the pixel brightness control in all common digital displays such as LCDs
(liquid crystal displays), LEDs (light-emitting diodes displays), and OLEDs
(organic light-emitting diodes displays)8, 2, 3. We show here a new type of
optical display technology, based on structural colors and flexible micro
mechanical shutters, which could be an ideal candidate to enable the future
transition towards a circular electronics economy.

Structural colors generate colors from nanostructures by plasmonic or
dielectric optical resonators or gratings9, 10, 11, 12. While abundant in
nature13, engineered passive structural colors have also been demonstrated to
great effect in the last decade9, 10, 12. Both metallic14, 15, 16, 17, 18 and
dielectric structures19, 11, 20, 21 have been shown, and generic ’structural
color paper’ that can be ’painted’ using a high-power laser has also been
demonstrated22, 23, 24, 19, 25, 26. Apart from passive structural colors,
tunable or ’dynamic’ structural colors for display applications are also
seeing increasing interest10. Recent demonstrations include chemical processes
to change the local refractive index to shift the resonant color27, 28, or
using LC as in a conventional LCD monitor to modulate color pixel
transmission29, 10. Use of electrostatic forces to align plasmonic resonant
metal nanorods to generate colors has also been shown30. The disadvantages of
these techniques are that either their color switching speed is slow, or it is
difficult to individually control pixels, or they do not greatly simplify the
composite materials required to manufacture the display - while the number of
materials in the display unit is closely related to the end-of-life-cycle
management8, 2, 3.

MEMS (Microelectromechanical systems) technology has been used in the past to
generate alternative displays and color filters31, 32. MEMS display
technologies can generally be categorized into several groups, such as digital
micromirror devices33, laser scanning displays34, interferometric modulator
displays35, digital micro-shutters36, grating light valves37, etc. MEMS
generally offers a significant power reduction when compared to most other
display technologies, but each actuation technique has its own unique
advantages and disadvantages31, 32. For example, laser scanning displays offer
almost perfect color gamut displays, as the image is generated by scanning
three different monochromatic red, green, and blue lasers, but generally
suffer from speckle causing imaging artifacts31.

MEMS combined with structural colors was very recently demonstrated38. The
work used SOI (silicon on insulator) wafers to build a silicon grating above a
silicon substrate. By controlling the gap in between the grating and
substrate, the reflection color was changed among green, yellow and red. This
combined approach shows several advantages, such as low power consumption,
CMOS-compatible fabrication, etc. However, from the mechanism of the color
control, it is difficult to cover the full range of RGB colors, and it is
difficult to show black color. Grating based structural colors are also well-
known to have issues with viewing angle due to their diffractive nature, and
as the device works in reflection mode, it is hard to incorporate a dedicated
light source that would allow viewing in low ambient light conditions.

In this paper we bring together plasmonic metasurfaces and MEMS technology to
generate transmission type dynamic color control using only two common and
easily recyclable materials: Aluminum and silicon oxide (glass). We
demonstrate the generation of a full range of RGB color subpixels. The colors
are generated by optical transmission based plasmonic metasurfaces consisting
of aluminum nanohole arrays, and the relative transmission/brightness of each
color subpixel is modulated by a MEMS cantilever made of aluminum and silicon
oxide fabricated on top of the metasurface. By using the nanohole array itself
as the actuation electrode for the cantilever, application of a bias between
the two then causes electrostatic forces to pull the cantilever down and close
off light transmission through the plasmonic array - effectively ’shutting it
off’. By modulating the actuation voltage, we show how it is directly possible
to modulate the overall brightness of each pixel by changing its average
transmission in time via fast modulation.

Our method allows for fast pixel switching speeds well in the excess of 800 Hz
(which is significantly faster than currently available displays), and by
using the metasurfaces themselves as the actuation electrodes, it is possible
to individually address and modulate specific color subpixels - an essential
feature for making a RGB display, and eliminates the need for transparent
electrodes made of e.g. ITO. Furthermore, as the entire fabrication process is
based on lithography and the materials used are generally considered cost-
effective and CMOS (Complementary Metal–Oxide–Semiconductor) compatible, it is
also expected that our approach could enable significant fabrication cost
reductions for mass scale manufacturing of optical displays.

Our work shows that it is possible to make a fast, full color range display,
using just two component materials. This is an important step towards
simplifying implementations of digital displays, and a crucial step towards
display technologies with simplified end-of-life-cycle reprocessing procedures
- essential for achieving future circular economic and e-waste management
goals4, 5, 6, 7.

## Results

Figure 1: Device deign and working principle. a) Schematic of the device. A
plasmonic nanohole array filters white continuum light into one color upon
transmission. The array’s transmission can then be modulated via a mechanical
cantilever which blocks off the array when a voltage is applied between the
cantilever and the nanohole array. b) SEM image of the fabricated cantilevers
and the nanohole array. Scale bar is 50 $\upmu$m. c) Top: SEM images of the
generated nanohole arrays for the red (R), green (G), and blue (B) color pixel
elements. Bottom: White light transmission microscope images for each of the
color pixel elements. Scale bars for SEM images are 250 nm, and for microscope
images 30 $\upmu$m. d) Experimental (solid lines)- and FDTD simulated (dashed
lines) transmission spectra of the nanohole arrays.

###  0.1 Device design and operating principle:

The general working principle of the device is shown in Fig. 1.a. A plasmonic
nanohole array filters white continuum illumination into single colors when it
is transmitted through the array. Then, the transmission through the array is
modulated by a flexible MEMS cantilever, that is actuated by an electrostatic
potential applied between the nanohole array and the cantilever. The
cantilever motion resembles more the unrolling of a rolled up sheet, rather
than a stiff vibrating beam. This allows the metasurfaces to be completely
unobstructed by the cantilevers when in their rolled up default position,
allowing for high optical transmission. An example of a fabricated device can
be seen in the scanning electron microscope (SEM) image in Fig. 1.b, and a
video of the cantilever motion with low frequency actuation voltage can be
seen in the supplementary materials.

To demonstrate the usefulness of our approach towards display applications, we
have fabricated three different types of nanohole arrays, one each to match
respectively to red, green, and blue visible light39. By varying the diameter
and period of the nanoholes, we are able to change the plasmonic resonance
response, and thus control the transmission spectrum at will. The diameters of
the nanohles for red, green, and blue are 208 nm, 154 nm, and 124 nm, where
the corresponding periods between nanohole center for the three colors are 380
nm, 312 nm, and 252 nm. A set of SEM images and transmission microscope images
of the generated arrays can be seen in Fig. 1.c, with finite-difference time-
domain (FDTD) simulation and experimentally measured transmission spectra
available in Fig. 1.d. From the spectral measurements we see that the peak
transmission efficiencies are $\sim$45% for the red color filter, and
$\sim$35% for the green and blue color filters. The measured results show
noticeable broader spectra at longer wavelength for each curve as compared
with the simulation. The details of the spectral measurements and simulations
are available in the methods section below.

The overall fabrication of the device is done in two steps: First, the
nanohole arrays and a solid electrode are defined and fabricated using
electron-beam lithography with an evaporation of a 100 nm aluminum layer and
lift-off. Then, a spacer oxide (to block electrical breakdown when the
cantilever is touching the array) of 210 nm silicon oxide is evaporated on top
of the nanoholes. Besides insulating the array from the cantilever, the spacer
layer also encapsulates and protects the aluminum nanoholes. In the next step,
a sacrificial layer of photoresist is added on top of the array (to later
release the cantilevers). Then, using UV photolithography the cantilevers and
their electrodes are defined. The cantilevers are then fabricated by
sequential sputtering of 100 nm of silicon oxide, followed by 100 nm aluminum.
Reactive-ion etching (RIE) is then used to etch away the sacrificial layer of
photoresist to finally release the cantilevers from the nanohole array and the
device substrate. The total lateral size of a nanohole array and its
accompanying cantilever is $\sim 30\times 60$ $\upmu\text{m}^{2}$. For more
details on the fabrication, see the methods section below.

###  0.2 Modulation of color pixel brightness:

After fabricating the nanohole arrays and cantilevers, we characterize the
modulation of light transmission through them using the setup shown in Fig.
2.a.

Figure 2: Transmission modulation. a) Experimental setup used for the
transmission modulation experiment. A home-built reflection microscope is used
to couple laser light through the nanohole array while the cantilevers are
modulated by a gate voltage. The resulting transmission modulation is recorded
by a silicon photodiode. The imaging part of the microscope (top section) is
used to select which cantilever to measure and to align the laser spot to it.
b - c) Applied $V_{\text{G}}$ and the measured relative transmission of a
laser passing through the nanohole array, as a function of time, for 100 Hz
and 300 Hz gate signal, respectively. d) Frequency response of the normalized
relative transmission modulation through the array (dotted horizontal line
marks -3 dB) and average relative transmission for different frequencies. e)
Average relative transmission of the laser through the array as a function of
$V_{\text{G}}$ peak-to-peak voltages.

By matching the laser spot size to that of one nanohole array/cantilever we
can measure the performance of a single cantilever. The transmission is
measured as the photovoltage of a silicon photodiode connected to an
oscilloscope. For full details of the transmission modulation measurements,
see methods below. It is important to note that the frequency of the light
modulation by the cantilevers is doubled relative to the drive voltage
frequency, see Fig. 2.b. This is because despite the fact that the voltage is
applied in alternating patterns of positive and negative voltage, both
polarities cause the cantilevers to attract towards the nanohole array. Only
when $V_{\text{G}}$ goes towards zero volt do the cantilevers release back to
their natural upward position. This is because the cantilevers are operated by
capacitive charging, so when either polarity is applied, there is a
corresponding accumulation of positive and negative charges on the nanohole
array and the cantilever. This charge accumulation causes the cantilever to
actuate down towards the nanohole array, due to electrostatic attraction,
regardless of the polarity. The gate voltage is applied in alternating
patterns of positive and negative bias to avoid permanent charge accumulation,
which would pin the cantilever down permanently if not discharged40, 41. We
generally apply a sine-wave pattern for $V_{\text{G}}$, as the gradual change
in polarity allows the cantilevers more time to release and recover. If we
apply a square-wave instead, we find that the cantilevers get stuck in the
downward position without modulating the light. This is because there is
effectively no time at which no bias is applied, and there is no time for the
cantilevers to recover to their upward position.

We can write the total force on a cantilever as the sum of a restoring spring
force and an attractive electrostatic force. Assuming a parallel plate
actuator for simplicity, we get40:

$\displaystyle F_{\text{tot}}(x,t)$
$\displaystyle=F_{\text{spring}}(x)-F_{\text{electrostatic}}(x,t),$ (1)
$\displaystyle=kx-\frac{1}{2}\epsilon_{0}\frac{S}{(g-x)^{2}}V_{\text{G}}^{2}(t),$
(2)

here $k$ is the spring constant of the cantilever, which is determined by its
materials and the geometry, $x$ is its displacement, $\epsilon_{0}$ is the
dielectric constant of vacuum, $S$ is the area of the cantilever, $g$ is the
initial gap between the cantilever and the nanohole electrode, and
$V_{\text{G}}$ is the time dependent drive voltage.

We see from Fig. 2.c the positive role of increasing the bias. By increasing
the amplitude of the drive voltage, $V_{\text{G}}$, at high frequencies, the
’closed’ state of the cantilever becomes deepened (i.e. less light goes
through, red versus blue curve in Fig. 2.b). While the cantilever is not able
to become fully closed (as in Fig. 2.b) due to the limited motion speed, a
larger electrostatic force from the increased voltage serves to bring it
closer to a fully closed position, due to the increased downward acceleration.
If we consider the ’well-behaved’ cantilever motion at lower frequencies, Fig.
2.b, we find also that the rise time of the modulation is $\sim$0.23 ms while
the fall time is $\sim$1.2 ms. This tells us that the restoring force is
generally quite a lot stronger than the electrostatic attractive force, when
outside the electrostatic pull-in range of the motion40.

Fig. 2.d’s top panel shows the peak-to-peak modulation of light in dB through
the nanohole array for different drive voltage frequencies. We see from this
that the -3 dB point for the device is at $\sim$400 Hz input, i.e. $\sim$800
Hz actual light modulation. For higher frequencies, the peak-to-peak
modulation of the light is reduced significantly, as the cantilever does not
have time to properly close due to the limited fall-time of the motion.

We can characterize the average transmission of light through the array over
time for different frequencies and voltages. For a given measurement of the
normalized relative transmission through the sample over time,
$T_{\text{rel}}(t)$, we can calculate the average relative transmission as:

$\displaystyle\left<T_{\text{rel}}\right>=\sum_{t=1}^{N}\frac{T_{\text{rel}}(t)}{N},$
(3)

where $N$ is the total amount of equidistant points in time sampled. We can
then see that the average transmission through the sample goes up with the
drive frequency, bottom panel of Fig. 2.d.

Because the speed of the modulation depends on the electrostatic force, we can
also modulate the average transmission by changing the peak-to-peak drive
voltage, as seen in Fig. 2.e.

Figure 3: Color brightness modulation. a) Experimental setup used for the
brightness measurements. The home-built microscope is reconfigured to a
transmission type scheme. b) The resulting brightnesses observed when the
cantilevers are modulated at different peak-to-peak voltages with a 100 Hz
driving sine-wave. 100 images were recorded with a framerate of 32 Hz and
averaged together. An average RGB color is also displayed from within the
color pixel area. c) Relative luminance value from the averaged RGB values
when converted to xyY, normalized to the unmodulated pixel.

If the motion of the cantilevers is generally faster than the flicker fusion
threshold42 of an observer, i.e. the point at which periodic motion/light
modulation can no longer be discretely observed, then the modulated response
will instead look like the averaged transmission signal. For optical displays,
the flicker fusion threshold is generally regarded as being $\sim$60-100 Hz
(different viewing conditions can affect the threshold and it is different for
each individual)42. Motivated by this concept, we decide to test if indeed our
device is able to modulate the overall brightness of the transmitted color, as
would be necessary for optical display applications, by applying a periodic
bias with different gate voltage.

The results can be seen in Fig. 3. Using the transmission imaging setup as
outlined in Fig. 3.a, we can observe the different brightnesses of the color
arrays simply by varying the amplitude of drive voltage, while maintaining a
100 Hz input sine-wave, as seen in Fig. 4.b and c.

Because the frame rate of our camera (32 Hz) is slower than the modulation
speed of the cantilevers, we record a series of images (100) and average them
together to approximate how a human eye would see the blurred motion of the
cantilever. The achieved colors in Fig. 3.b are thus only a rough
approximation of how a human observer would see the dimming of the pixels.

###  0.3 Color mixing:

Figure 4: Chromaticity and color space CIE xyY chromaticity map calculated
from transmission spectra of white light transmitted through the red, green,
and blue color pixels. The range of sRGB colors are shown for comparison.

Finally, we have also characterized the chromaticity of the red, green, and
blue color arrays, the result can be seen in Fig. 4.a. The recorded
transmission spectra were converted into the International Commission on
Illumination (CIE) 1931 color space xyY coordinates (see methods below).

If the colors that make up two points in the CIE xyY chromaticity diagram can
be freely mixed, then any colors on a straight line between those two points
can also be achieved. Thus, inside a triangle of points, if all three colors
corresponding to the points can be mixed freely, then all the colors inside
the triangle can be achieved. For comparison, we have shown here also the
chromaticity points of the sRGB scale, which is the standard chromaticity
scale for most commercial optical displays.

## Discussion

The capabilities of our ’proof of concept’ device shown here could be further
improved in several ways. First, the cantilevers are being actuated in air,
and drag from air resistance is thus limiting the speed of their motion.
Encapsulating the device and operating it in a standard low vacuum of
$\sim$10-3 torr could greatly increase the modulation frequencies that could
be achieved.

Secondly, the mechanical properties of the cantilevers could be further
optimized. It is very possible that by changing the dimensions or the
thicknesses of the SiO2 and Al layers, that even higher frequencies of
modulation could be achieved, and that the average transmission could be
better controlled as well. In the same fashion, adding a high frequency
modulation to the drive voltage could further stabilize the cantilever
movement, or prevent potential high frequent damping41, 43, 40. The magnitudes
of the driving voltages used here (roughly $\pm 70$ V) is also something that
could be greatly reduced through optimization of the cantilever design,
especially by reducing the cantilever size both in width and length, such that
after device fabrication the gap between the cantilever and the bottom
electrode will be reduced, which will dramatically lower the pull-in voltage.
Another method to reduce the driving voltage is to reduce the spring constant
by increasing the cantilever aspect ratio so less total electrostatic force is
needed to pull down the cantilever. It should be noted that the spring
constant should be kept above a certain level to avoid cantilever stiction,
which is a common issue in low spring constant MEMS devices41, 43, 40.

As for the plasmonic nanohole arrays, further optical design with the aim to
narrow the spectral transmission would greatly expand the achievable color
space. For example, our red nanohole array contains a great deal of ’blue’
light in its transmission spectrum (see Fig. 1.d), and in general the arrays
allow for a low amount of ’white light’ transmission. The color filtering
mechanism shown here relies purely on plasmonic resonances. Alternatively,
making a combination of diffractive and plasmonic resonances could potentially
achieve ultra narrow color filters, as has been shown recently in the short
wave infrared regime44. The on/off extinction ratio of our method is also
expected to be very high, as the aluminum cantilever can essentially shut off
optical transmission completely, allowing for high contrast ratios in display
applications.

In terms of scalability, our approach is also expected to enable cost-
effective large scale fabrication of color displays. For one, the entire
device only relies on three lithography steps for its fabrication, and two,
all of the used materials are CMOS-compatible and considered to be low-cost.
While we have utilized e-beam lithography for our particular device, it is
possible in industrial quality extreme UV (EUV) lithography to generate
structures in the range of 5 nm45, which is well below any feature size in our
device.

As our device only consists of aluminum and glass, we believe it would be
relatively easy to recycle via a combination of mechanical and heating
processes, and/or chemical processes. Essentially the color/display element of
a display could be crushed to a fine powder in a mechanical press. Then, as
the melting temperatures of glass and aluminum are significantly different
($T_{\text{melt,Al}}=660$ ∘C vs. $T_{\text{melt,SiO}_{2}}=1723$ ∘C), the
aluminum and glass components could be melted out of the powder at different
temperatures in a furnace. Alternatively, the powder could be suspended in an
acid which would selectively dissolve the aluminum to form an aluminum salt
dissolved in the liquid. The resulting liquid could then be strained of the
remaining glass particles, and the aluminum could be extracted either by
electro-chemical processes or chemical reduction3.

## Conclusion

To conclude, we have shown that it is possible to generate a full range of
red, green, and blue color pixels using plasmonic metasurface in the form of a
nanohole arrays and demonstrate the modulation of the average transmission of
such an array by a micromechanical cantilever which is actuated by applying an
electrostatic potential between the nanohole array and the cantilever.

We find that the -3 dB point for the transmission modulation to be $\sim$800
Hz, which is well above the speeds of conventional display technology (60-124
Hz), and even more importantly, well above the human flicker fusion threshold
(60-100 Hz). Taking advantage of this property, we have shown that the average
transmission through an array can be controlled by changing the magnitude of
the applied voltage or by changing the drive voltage frequency.

Making a pixel that consists of three subpixels each of a red, green, and blue
nanohole array can thus allow for color mixing between the red, green, and
blue subpixels, by individually controlling their average transmission, making
their brightness dim to a human observer.

Finally, from transmission spectra of each of our nanohole arrays, we have
extracted the chromaticity and shown the color space that could be achieved in
such a plasmonic color display.

Our results highlight how plasmonic structural colors and MEMS technology can
be combined in order to generate a color display, based on only two materials
that can be fully recyclable. Given the immense prevalence and importance of
displays in modern society, such reduction of component materials is predicted
to become critically important in the future if circular economic goals of
easily, and economically viably, recyclable consumer goods are to be achieved.

## Methods

###  0.4 FDTD simulation:

FDTD simulations were carried out using the commercial software Lumerical FDTD
(Ansys). The nanohole arrays were simulated using periodic boundary conditions
for the $x$\- and $y$-directions and perfectly matched layers (PMLs) were used
in the $z$-direction. The smallest mesh element size was 2 nm. The dielectric
functions of the aluminum used the values from Palik 46, and the glass was
kept as a lossless dielectric with refractive index 1.45.

###  0.5 Sample fabrication:

The device fabrication includes two lithography techniques, electron beam (EB)
lithography and photo lithography. We use the EB lithography to make the
nanohole arrays for the color filters, and the photo lithography for the MEMS
cantilevers. First, PMMA was coated on a glass substrate, followed by the EB
lithography with a ELS-G100 (Elionix) system. An acceleration voltage of 100
kV and a beam current of 1 nA were used. Then 100 nm thick Aluminum (Al) was
deposited with electron-beam evaporation on the wafer. We used lift-off to
pattern the nanohole arrays and the bottom electrodes for the cantilevers. A
thin layer oxide of 210 nm was deposited with sputtering on the wafer to tune
the color filter of nanohole arrays to the desired color. The oxide layer also
works as an insulation layer for the electrodes when actuating the
cantilevers. Next, a photoresist sacrificial layer was patterned on the wafer
with photo lithography using a MA6 mask aligner (SÜSS MicroTec). A 100 nm
oxide and 100 nm Al were then deposited with sputtering on the wafer. The
cantilever shapes were patterned by photo lithography, followed by Al wet
etching and oxide wet etching. The cantilevers are released by O2 plasma with
a RIE machine.

###  0.6 Transmission spectroscopy:

We characterize the optical transmission of the plasmonic metasurfaces by
illuminating them with a white light source (tungsten-halogen lamp) through a
microscope condenser lens. The transmitted light is collected with a
microscope objective (Nikon, 50×, NA 0.45) and directed into an Ocean Optics
Flame spectrometer. An aperture is placed in the image plane to spatially
select only the metasurface region (around diameter 25 $\upmu$m circular area)
to enter the spectrometer.

###  0.7 Light modulation measurements:

The cantilevers’ modulation of light was measured using a red laser pointer.
Following the setup in Fig. 2.a, the laser light was coupled from above in a
home-built reflection microscope, where it was aligned to a single cantilever
at a time. Tungsten probes were mechanically pressed against the aluminium
electrodes on the top of the device using XYZ 500MIM 3D stages (Quater
Research & Development), and a bias voltage as applied between them using a
SDG 2082X (SIGLENT) function generator. The voltage output of the signal
generator was amplified using a model 2350 high voltage amplifier (TEGAM). The
transmitted laser light through the nanohole array was then measured using a
silicon photodiode (DET100A/M, Thorlabs). The voltage output of the photodiode
and the applied bias were both monitored on an InfiniiVision DSOX2004A
(Keysight) oscilloscope. The measured transmission signals were converted to
$T_{\text{rel}}$ signals of values between 0 and 1, by first subtracting the
off-state voltage from the lowest frequency measurement, and then normalizing
the signals to the highest value of the lowest frequency measurement.

###  0.8 Color mixing and color gamut:

CIE 1931 xyY chromaticity coordinates were calculated from the recorded
transmission spectra. Using the CIE 1931 color matching functions, $\bar{x}$,
$\bar{y}$, and $\bar{z}$47 the tristimulus values, $X$, $Y$, and $Z$ were then
calculated as48:

$\displaystyle
X=\frac{1}{N}\int_{0}^{\infty}I(\lambda)T(\lambda)\bar{x}(\lambda)d\lambda,$
$\displaystyle
Y=\frac{1}{N}\int_{0}^{\infty}I(\lambda)T(\lambda)\bar{y}(\lambda)d\lambda,$
and $\displaystyle
Z=\frac{1}{N}\int_{0}^{\infty}I(\lambda)T(\lambda)\bar{z}(\lambda)d\lambda,$

where $I$ is the spectral power distribution of the illuminating light source,
$T$ is the normalized transmission spectrum of a nanohole array color filter,
$\lambda$ is the wavelength, and $N$ is given as:

$\displaystyle N=\int_{0}^{\infty}I(\lambda)\bar{y}(\lambda)d\lambda.$

The ’brightness’ (luminance) and ’color’ (chromaticity) of the light
transmitted through the nanohole array can then be described by the CIE xyY
coordinates. The tristimulus value $Y$ from above is the luminance per
definition, and the $x$ and $y$ chromaticity coordinates are given by
normalization as:

$\displaystyle x=\frac{X}{X+Y+Z},$ and $\displaystyle y=\frac{Y}{X+Y+Z}.$

The final coordinate $z=Z/(X+Y+Z)=1-x-y$ is not independent and is generally
disregarded.

## References

  * 1 A new circular vision for electronics, time for a global reboot.  tech. rep., World Economic Forum, 2019.
  * 2 Vikrant Bhakar, Aashray Agur, AK Digalwar, and Kuldip Singh Sangwan.  Life cycle assessment of CRT, LCD and LED monitors.  Procedia CIRP, 29:432–437, 2015.
  * 3 Ville Lahtela, Sami Virolainen, Andreas Uwaoma, Mari Kallioinen, Timo Kärki, and Tuomo Sainio.  Novel mechanical pre-treatment methods for effective indium recovery from end-of-life liquid-crystal display panels.  J. Clean. Prod., 230:580–591, 2019.
  * 4 Allen V Kneese.  The economics of natural resources.  Popul. Dev. Rev., 14:281–309, 1988.
  * 5 Walter R Stahel.  The circular economy.  Nature News, 531(7595):435, 2016.
  * 6 Julian Kirchherr, Denise Reike, and Marko Hekkert.  Conceptualizing the circular economy: An analysis of 114 definitions.  Resour. Conserv. Recycl., 127:221–232, 2017.
  * 7 Martin Geissdoerfer, Paulo Savaget, Nancy MP Bocken, and Erik Jan Hultink.  The circular economy–a new sustainability paradigm?  J. Clean. Prod., 143:757–768, 2017.
  * 8 Seung-Jin Lee and Joyce Cooper.  Estimating regional material flows for LCDs.  In Proceedings of the IEEE international symposium on electronics and the environment, volume 19, page 21, 2008.
  * 9 Anders Kristensen, Joel KW Yang, Sergey I Bozhevolnyi, Stephan Link, Peter Nordlander, Naomi J Halas, and N Asger Mortensen.  Plasmonic colour generation.  Nat. Rev. Mater., 2(1):1–14, 2016.
  * 10 Lei Shao, Xiaolu Zhuo, and Jianfang Wang.  Advanced plasmonic materials for dynamic color display.  Adv. Mater., 30(16):1704338, 2018.
  * 11 Taejun Lee, Jaehyuck Jang, Heonyeong Jeong, and Junsuk Rho.  Plasmonic-and dielectric-based structural coloring: from fundamentals to practical applications.  Nano Converg., 5(1):1–21, 2018.
  * 12 Soroosh Daqiqeh Rezaei, Zhaogang Dong, John You En Chan, Jonathan Trisno, Ray Jia Hong Ng, Qifeng Ruan, Cheng-Wei Qiu, N Asger Mortensen, and Joel KW Yang.  Nanophotonic structural colors.  ACS Photonics, 8(1):18–33, 2020.
  * 13 Shuichi Kinoshita and Shinya Yoshioka.  Structural colors in nature: the role of regularity and irregularity in the structure.  ChemPhysChem, 6(8):1442–1459, 2005.
  * 14 Karthik Kumar, Huigao Duan, Ravi S Hegde, Samuel CW Koh, Jennifer N Wei, and Joel KW Yang.  Printing colour at the optical diffraction limit.  Nat. Nanotechnol., 7(9):557–561, 2012.
  * 15 Tal Ellenbogen, Kwanyong Seo, and Kenneth B Crozier.  Chromatic plasmonic polarizers for active visible color filtering and polarimetry.  Nano Lett., 12(2):1026–1031, 2012.
  * 16 Jeppe S Clausen, Emil Højlund-Nielsen, Alexander B Christiansen, Sadegh Yazdi, Meir Grajower, Hesham Taha, Uriel Levy, Anders Kristensen, and N Asger Mortensen.  Plasmonic metasurfaces for coloration of plastic consumer products.  Nano Lett., 14(8):4499–4504, 2014.
  * 17 Xiao Ming Goh, Yihan Zheng, Shawn J Tan, Lei Zhang, Karthik Kumar, Cheng-Wei Qiu, and Joel KW Yang.  Three-dimensional plasmonic stereoscopic prints in full colour.  Nat. Commun., 5(1):1–8, 2014.
  * 18 Zhibo Li, Alasdair W Clark, and Jonathan M Cooper.  Dual color plasmonic pixels create a polarization controlled nano color palette.  ACS Nano, 10(1):492–498, 2016.
  * 19 Xiaolong Zhu, Wei Yan, Uriel Levy, N Asger Mortensen, and Anders Kristensen.  Resonant laser printing of structural colors on high-index dielectric metasurfaces.  Sci. Adv., 3(5):e1602487, 2017.
  * 20 Wenhong Yang, Shumin Xiao, Qinghai Song, Yilin Liu, Yunkai Wu, Shuai Wang, Jie Yu, Jiecai Han, and Din-Ping Tsai.  All-dielectric metasurface for high-performance structural color.  Nat. Commun., 11(1):1–8, 2020.
  * 21 N Roostaei, NS Shnan, and SM Hamidi.  Red and blue color production by flexible all-dielectric structure.  Optik, 230:166345, 2021.
  * 22 Peter Zijlstra, James WM Chon, and Min Gu.  Five-dimensional optical recording mediated by surface plasmons in gold nanorods.  Nature, 459(7245):410–413, 2009.
  * 23 Xi Chen, Yiting Chen, Jin Dai, Min Yan, Ding Zhao, Qiang Li, and Min Qiu.  Ordered Au nanocrystals on a substrate formed by light-induced rapid annealing.  Nanoscale, 6(3):1756–1762, 2014.
  * 24 Xiaolong Zhu, Christoph Vannahme, Emil Højlund-Nielsen, N Asger Mortensen, and Anders Kristensen.  Plasmonic colour laser printing.  Nat. Nanotechnol., 11(4):325–329, 2016.
  * 25 Alexander S Roberts, Sergey M Novikov, Yuanqing Yang, Yiting Chen, Sergejs Boroviks, Jonas Beermann, N Asger Mortensen, and Sergey I Bozhevolnyi.  Laser writing of bright colors on near-percolation plasmonic reflector arrays.  ACS nano, 13(1):71–77, 2018.
  * 26 Vadim P Veiko, Yaroslava Andreeva, Luong Van Cuong, Daria Lutoshina, Dmitry Polyakov, Dmitry Sinev, Vladimir Mikhailovskii, Yury R Kolobov, and Galina Odintsova.  Laser paintbrush as a tool for modern art.  Optica, 8(5):577–585, 2021.
  * 27 Xiaoyang Duan, Simon Kamin, and Na Liu.  Dynamic plasmonic colour display.  Nat. Commun., 8(1):1–9, 2017.
  * 28 Xiaoyang Duan and Na Liu.  Scanning plasmonic color display.  ACS Nano, 12(8):8817–8823, 2018.
  * 29 Daniel Franklin, Russell Frank, Shin-Tson Wu, and Debashis Chanda.  Actively addressed single pixel full-colour plasmonic display.  Nat. Commun., 8(1):1–10, 2017.
  * 30 Nicholas J Greybush, Kristin Charipar, Jeffrey A Geldmeier, Stephen J Bauman, Paul Johns, Jawad Naciri, Nicholas Charipar, Kyoungweon Park, Richard A Vaia, and Jake Fontana.  Dynamic plasmonic pixels.  ACS Nano, 13(4):3875–3883, 2019.
  * 31 Sven TS Holmström, Utku Baran, and Hakan Urey.  MEMS laser scanners: a review.  J. Microelectromech. Syst., 23(2):259–275, 2014.
  * 32 Ji Ma.  Advanced MEMS-based technologies and displays.  Displays, 37:2–10, 2015.
  * 33 Larry J Hornbeck.  Digital light processing for high-brightness high-resolution applications.  In Projection Displays III, volume 3013, pages 27–40. International Society for Optics and Photonics, 1997.
  * 34 Jason Tauscher, Wyatt O Davis, Dean Brown, Matt Ellis, Yunfei Ma, Michael E Sherwood, David Bowman, Mark P Helsel, Sung Lee, and John Wyatt Coy.  Evolution of MEMS scanning mirrors for laser projection in compact consumer electronics.  In MOEMS and Miniaturized Systems IX, volume 7594, page 75940A. International Society for Optics and Photonics, 2010.
  * 35 Jeffrey B Sampsell.  MEMS-based display technology drives next-generation FPDs for mobile applications.  Information Display, 22(6):24, 2006.
  * 36 Nesbitt Hagood, Roger Barton, Tim Brosnihan, John Fijol, Jignesh Gandhi, Mark Halfman, Richard Payne, and J Lodewyk Steyn.  35.5 l: Late-news paper: A direct-view MEMS display for mobile applications.  In SID Symposium Digest of Technical Papers, volume 38, pages 1278–1281. Wiley Online Library, 2007.
  * 37 David M Bloom.  Grating light valve: revolutionizing display technology.  In Projection Displays III, volume 3013, pages 165–171. International Society for Optics and Photonics, 1997.
  * 38 Aaron L Holsteen, Ahmet Fatih Cihan, and Mark L Brongersma.  Temporal color mixing and dynamic beam shaping with silicon metasurfaces.  Science, 365(6450):257–260, 2019.
  * 39 Qin Chen and David RS Cumming.  High transmission and low color cross-talk plasmonic color filters using triangular-lattice hole arrays in aluminum films.  Opt. Express, 18(13):14056–14062, 2010.
  * 40 Hiroshi Toshiyoshi.  Electrostatic actuation.  Comprehensive Microsystems, 2:1–38, 2008.
  * 41 Zhengli Han, Kenta Kohno, Hiroyuki Fujita, Kazuhiko Hirakawa, and Hiroshi Toshiyoshi.  MEMS reconfigurable metamaterial for terahertz switchable filter and modulator.  Opt. Express, 22(18):21326–21339, 2014.
  * 42 Carney Landis.  Determinants of the critical flicker-fusion threshold.  Physiol. Rev., 34(2):259–286, 1954.
  * 43 Zhengli Han, Kenta Kohno, Hiroyuki Fujita, Kazuhiko Hirakawa, and Hiroshi Toshiyoshi.  Tunable terahertz filter and modulator based on electrostatic mems reconfigurable srr array.  IEEE Journal of Selected Topics in Quantum Electronics, 21(4):114–122, 2014.
  * 44 M Saad Bin-Alam, Orad Reshef, Yaryna Mamchur, M Zahirul Alam, Graham Carlow, Jeremy Upham, Brian T Sullivan, Jean-Michel Ménard, Mikko J Huttunen, Robert W Boyd, et al.  Ultra-high-Q resonances in plasmonic metasurfaces.  Nat. Commun., 12(1):1–8, 2021.
  * 45 Geoffrey Yeap, SS Lin, YM Chen, HL Shang, PW Wang, HC Lin, YC Peng, JY Sheu, M Wang, X Chen, et al.  5nm cmos production technology platform featuring full-fledged euv, and high mobility channel finfets with densest 0.021 $\mu$m 2 sram cells for mobile soc and high performance computing applications.  In 2019 IEEE International Electron Devices Meeting (IEDM), pages 36–7. IEEE, 2019.
  * 46 Edward D Palik.  Handbook of optical constants of solids, volume 3.  Academic press, 1998.
  * 47 John Guild.  The colorimetric properties of the spectrum.  Philos. Trans. R. Soc. A, 230(681-693):149–187, 1931.
  * 48 Daniel Malacara.  Color vision and colorimetry: theory and applications/David Malacara.  2002\.

###  0.9 Acknowledgments:

The authors wish to thank Mr. Maurice Saidian and Dr. Shimon Eliav of the The
Hebrew University Center for Nanoscience and Nanotechnology for assistance
with the device fabrication.

Funding: We acknowledge funding from the Israeli Ministry of Science and
Technology. Z.H. is supported by the PBC Fellowship Program.

###  0.10 Author contributions:

The idea for the project was devised by Z.H., C.F. and U.L. Optical design and
FDTD simulations were done by Z.H. Sample fabrication was done by Z.H. and
N.M. SEM images were recorded by N.M. Optical characterization was done by
C.F. and Z.H. Data analysis was done by C.F. The work was supervised by U.L.
All authors contributed to the writing of the manuscript and interpretation of
the results.

###  0.11 Conflicts of interests:

The authors declare no competing financial interests.

###  0.12 Data availability:

Optical spectra and other raw data are available upon request.

###  0.13 Correspondence:

General correspondence should be addressed to<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>


# Modified Zero Forcing Decoder for Ill-conditioned Channels

Ibrahim Al-Nahhal(1), Masoud Alghoniemy(2), Adel B. Abd El-Rahman(3), and Zen
Kawasaki(4)
${}^{(1)},^{(3)}$Department of Electronics and Communications Engineering,
Egypt-Japan University of Science and Technology,
Alexandria, 21934 Egypt (e-mail: {Ibrahim.al-
nahhal},{adel.bedair}@ejust.edu.eg).
(2)Department of Electrical Engineering, University of Alexandria, 21544 Egypt
(e-mail: alghoniemy@alexu.edu.eg).
(4)Department of Information and Communication Technology, Graduate School of
Engineering, Osaka University, Suita,
Osaka, Japan (e-mail<EMAIL_ADDRESS>
###### Abstract

A modified zero forcing decoder (MZF) for Multi-Input Multi-Output (MIMO)
systems in case of ill-conditioned channels is proposed. The proposed decoder
provides significant performance improvement compared to the traditional zero
forcing for ill-conditioned channels. The proposed decoder is the result of
reformulating the $QR$ decomposition of the channel matrix by neglecting the
elements which represent the correlation in the channel. By combining the
traditional ZF with the MZF decoders, a hybrid decoder can be formed that
alternates between the traditional ZF and the proposed MZF based on the
channel condition. We will illustrate through simulations the significant
improvement in performance with little change in complexity over the
traditional implementation of the zero forcing decoders.

###### Index Terms:

MIMO systems, sphere decoder, zero forcing.

##  I Introduction

In multi-input multi-output (MIMO) systems with additive Gaussian noise, the
Maximum-likelihood (ML) decoder is the optimum receiver. However, due to the
high complexity of the ML decoders, lattice-based decoding algorithms such as
the sphere decoding algorithms have been proposed [1]. Although the sphere
decoder (SD) achieves near-ML performance, its complexity is highly dependent
on the received signal to noise ratio (SNR). In order to achieve ML
performance at low SNR, the SD algorithm requires severe pre-processing
operations that involve iterative column vector ordering [2]. On the other
hand, the zero-forcing decoder has a very low complexity at the expense of
poor performance due to its inherent noise enhancement.

In this paper, we propose a modified zero forcing decoder which provides
significant performance improvement by limiting the inherent noise
enhancement. The proposed algorithm performs a simple reformulation of the
$QR$ factorization of the channel by considering the diagonal elements of the
upper-triangular matrix $R$, which represent the well-condition part of the
channel.

Consider the complex-valued baseband MIMO model in Rayleigh fading channels
with $M$ transmit and $N$ received antennas. Let the $N\times 1$ received
signal vector $\dot{y}$ [2],

$\dot{y}=\dot{H}\dot{x}+\dot{w}$ (1)

with the transmitted signal vector $\dot{x}\in Z_{C}^{M}$ whose elements are
drawn from q-QAM constellation and $Z_{C}$ is the set of complex integers, the
$N\times M$ channel matrix $\dot{H}$ whose elements $h_{ij}$ represent the
Rayleigh complex fading gain from transmitter $j$ to receiver $i$ with
$h_{ij}\sim CN(0,1)$ . In this paper, it is assumed that channel realization
is known to the receiver through preamble and/or pilot signals, and $N\geq M$.
The $N\times 1$ complex noise vector $\dot{w}$ has independent complex
Gaussian elements with variance $\sigma^{2}$ per dimension. Throughout the
paper, we will consider the real model of $(1)$

$y=Hx+w$ (2)

where, $m=2M$, $n=2N$, then $y=[R(\dot{y})\,\,I(\dot{y})]^{T}$ $\in R^{n}$,
$x=[R(\dot{x})\,\,I(\dot{x})]^{T}\in Z^{m}$,
$w=[R(\dot{w})\,\,I(\dot{w})]^{T}\in R^{n}$, and
$H=\left(\begin{array}[]{cc}R(\dot{H})&-I(\dot{H})\\\
I(\dot{H})&R(\dot{H})\end{array}\right)\in R^{n\times m}$, where $R(.)$ and
$I(.)$ are the real and imaginary parts, respectively. The ML solution,
$\hat{X}_{ML}$, that minimizes the 2-norm of the residual error is found by
solving the following integer-least squares

$\hat{X}_{ML}=arg\,\underset{x\subset\Lambda}{min}\left\|y-Hx\right\|^{2}$ (3)

where $\Lambda$ is the lattice whose points represent all possible codewords
at the transmitter and $Z^{m}$ is the set of integers of dimension $m$. It
should be noted that (3) is, in general, NP-hard [3].

##  II Traditional Decoders

In this section, we provide a brief overview of the sphere decoder (SD) and
the zero forcing decoders.

###  II-A Sphere Decoder

The sphere decoder (SD) reduces the search complexity by limiting the search
space inside a hyper-sphere of radius $\rho$ centered at the received vector
$y$ [3, 4]. In particular, the solution should satisfy

$\left\|y-Hx\right\|^{2}<\rho^{2}$ (4)

The sphere decoder transforms the closest-point search problem into a tree-
search problem by factorizing the channel matrix $H=QR$, where $Q$ is a
$n\times m$ unitary matrix which represents the orthonormal bases of the
channel $H$, and $R$ is an upper triangular matrix of size $m\times m$ that
represents the correlation in the channel. The energy of the residual error be
written recursively as [3]

$\left\|y-Hx\right\|^{2}=\left\|y-QRx\right\|^{2}=\left\|Q^{*}y-Rx\right\|^{2}<\rho^{2}$
(5)

The SD traverses the tree and computes the path metric for each node in the
tree. Any branch that has a path metric exceeding the pre-defined pruning
constraint $\rho$ will be discarded. Thus, only a subset of the tree is
visited and the complexity is reduced.

###  II-B Zero Forcing

The zero forcing (ZF) decoder provides low complexity through decorrelating
the channel by directly inverting the channel matrix $H$ [3, 5]. In
particular, for non square MIMO systems, multiplying both sides of (2) by the
pseudo-inverse of the channel, the residual error can be written as

$w_{ZF}=H^{+}w.$ (6)

Where $H^{+}=(H^{*}H)^{-1}H^{*}$ is the pseudo-inverse of the channel. It is
clear that if the channel is ill-conditioned, then noise enhancement is
significant. The ZF solution, $\hat{X}_{ZF}$, minimizes the 2-norm of the
residual error. In particular [3],

$\hat{X}_{ZF}=\lfloor H^{+}y\rceil$ (7)

where $\lfloor.\rceil$ is a slicing operation.

##  III The Modified Zero Forcing

In order to reduce the noise enhancement which is a result of ill-conditioned
channels, the MZF only considers the diagonal elements of the upper-triangular
matrix $R$, in the $QR$ factorization of the channel matrix, $H=QR$. It should
be noted that since $Q$ is an orthogonal matrix, then $cond(H)=cond(R)$ [6].

In particular, let $r_{ij}$ be the $ij^{th}$ element of $R$, $j\geq i$; and
let $R=\hat{R}R_{D}$ where $\hat{R}$ is a unit upper triangular matrix with
elements $\hat{r}_{ij}={r_{ij}}/{r_{jj}},\,\,j>i$, $\hat{r}_{ii}=1$ and
$R_{D}$ is a diagonal matrix whose diagonal elements are $r_{ii}$.

It should be noted that the strict upper-diagonal elements $\hat{r}_{ij}$
represent the correlation elements which are responsible for channel bad
conditioning. On the other hand, the diagonal elements $r_{ii}$ represent the
energy of the channel. In order to see this, figure $1$ illustrates the effect
of decomposing the channel matrix $H$ on the condition number. In particular,
the horizontal access represents the condition number of the full channel
matrix $H$, and the horizontal access represents the condition umber of the
factored matrices, $\hat{R},R_{D}$. The blue curves represent the condition
number of $R_{D}$ while the red curves represent the condition number of
$\hat{R}$ for channel matrices of size $2\times 2$ and $4\times 4$. It is
clear that even for highly ill-conditioned channel matrix, the condition
number of $R_{D}$ remains low. It should be noted that figure 1 is generated
by averaging $10,000$ runs. Using the above results, the MZF decoder solves

$\hat{X}_{MZF}=\lfloor R_{D}^{-1}Q^{H}y\rceil$ (8)

Figure 1: The effect of decomposition of the channel matrix H on the condition
number.

##  IV Hybrid Decoder

Based on the previous findings, a hybrid decoder (HD) can be formed by
alternating between the MZF and the ZF decoders based on channel condition
number. In particular, the MZF decoder provides performance improvement in
case of ill-conditioned channels due to considering only the well conditioned
elements of the channel. On the other hand, in case of well-conditioned
channels, the ZF decoder can be used without loss of performance. Hence, the
hybrid decoder alternates between the traditional ZF decoder and the MZF
according to the state of the channel and can be described by the following
pseudo-code.

  1. 1.

Compute the condition number of the channel $cond(H)$.

  2. 2.

Set a threshold $\gamma$ > 1.

  3. 3.

If $cond(H)<\gamma$, then $\hat{X}_{ZF}=\lfloor H^{+}y\rceil$.

  4. 4.

If $cond(H)>\gamma$, then $\hat{X}_{MZF}=\lfloor R_{D}^{-1}Q^{H}y\rceil$.

##  V Complexity Analysis

The complexity of the proposed algorithm can be measured by computing the
number of floating point operations (flops) consumed for execution; which also
can be converted into the execution time. It should be noted that $(7)$ is
solved using the $QR$ factorization algorithm [7]. So $(7)$ can be re-written
as

$R\hat{X}_{ZF}=Q^{H}y$ (9)

Which can be solved using back substitution method with cost [7]:

$flps_{ZF}\approx 2nm^{2}-\frac{2}{3}m^{3}$ (10)

Similarly, for the MZF decoder,

$R_{D}\hat{X}_{MZF}=Q^{H}y$ (11)

The cost of the previous algorithm is the same as the cost of the algorithm of
eq. $(10)$ without the cost of back substitution. Where the cost of back
substitution is $m^{2}$, the number of flops of MZF algorithm is:

$flps_{MZF}\approx flps_{ZF}-m^{2}$ (12)

In HD algorithm; the number of flops is greater than the average between
$flps_{ZF}$ and $flps_{MZF}$ by the number of flops consumed in channel
condition number calculations.

##  VI Simulation Results

For comparison purpose, the performance of the proposed decoders ,(MZF) and
(HD), are compared to the SD and the ZF for uncoded systems. It is assumed
that the transmitted power is independent of the number of transmit antennas,
$M$, and equals to the average symbols energy. We have assumed that the
channel is Rayleigh fading channel. Let $P$ be the percentage of ill-
conditioned channels in the runs. Figures 2, 3, and 4 illustrate the
performance of the traditional and proposed decoders for $N=M=2$ ,$16-QAM$ for
well-conditioned channels $P=0\%$, ill-conditioned channels with
$cond(H)=10^{3}$ for $P=100\%$ and $P=50\%$ versus different values of signal
to noise ratio. Figure 5 illustrates the performance for different values of
channel condition numbers for $SNR=15dB$. As it is clear from the figures, the
SD is superior while the ZF and the MZF decoders are inferior. It should be
noted that in the well-conditioned channel case, as illustrated in figure 2,
the MZF decoder performance has an error floor. This is a natural byproduct of
neglecting of one of the well-conditioned channel component, $\hat{R}$, as
expected. Thus, the HD typically follows the ZF because it acts as a ZF in
well-conditioned channels, $P=0\%$, as indicated before. Similarly, for the
ill-conditioned channels, $P=100$, illustrated in figure 3, the performance of
the ZF decoder produces an error floor. This is expected due to the noise
enhancement inherent in the ZF decoder [6]. Figure 4 shows that as the
percentage of ill-conditioned channels increases, the performance of HD is
very close to SD performance especially in low SNR. Also, as shown in figures
5 and 6 the performance of the HD approaches the SD performance with the
increase of the channel condition number especially in low SNR as shown in
figure 6 .

The complexities of the SD, ZF, MZF, and HD are measured by the execution time
in finding the solution. In particular, figures 7 illustrates the complexity
for the $2\times 2$ MIMO with $16-QAM$ modulation as a function of the SNR. It
is clear that the SD has high complexity while the MZF and the ZF decoders has
low complexity. As it is clear from $(12)$, the complexity of MZF decoder is
less than the complexity of the ZF decoder by a small margin according.

Figure 2: Performance of 2$\times$2 MIMO 16-QAM well-conditioned channels (P =
0%) Figure 3: Performance of 2$\times$2 MIMO 16-QAM ill-conditioned channels
with cond(H) = $10^{3}$ (P = 100%) Figure 4: Performance of 2$\times$2 MIMO
16-QAM (P = 50%) Figure 5: Channel performance varies with a channel condition
number for 2$\times$2 MIMO 16-QAM at SNR = 15dB Figure 6: Channel performance
varies with a channel condition number for 2$\times$2 MIMO 16-QAM at SNR = 5dB
Figure 7: Average Execution Time per Bit for 2$\times$2 MIMO 16-QAM

##  VII Conclusions

A Hybrid MIMO decoder that is based on neglecting the cross correlation
elements of the channel correlation matrix, $R$, in ill-conditioned channels
and acting as ZF in well-conditioned channels has been proposed. The proposed
decoder has better performance in ill-conditioned channels than the
corresponding ZF decoder. The complexity of the proposed decoder is as low as
the ZF decoder.

## References

  * [1] Ramin Shariat-Yazdi and Tad Kwasniewski, “Configurable K-best MIMO Detector Architecture,” ISCCSP 2008, Malta, 12-14 March 2008\.
  * [2] Chung-An Shen and Ahmed M. Eltawil, “An Adaptive Reduced Com- plexity K-Best Decoding Algorithm with Early Termination,” IEEE CCNC 2010 proceedings.
  * [3] Y. Cho et. al, "MIMO-OFDM Wireless Communications with MATLAB", chapter 11, John Wiley & Sons (Asia) Pte Ltd, 2010.
  * [4] Yi Hsuan Wu, Yu Ting Liu, Hsiu-Chi Chang, Yen-Chin Liao, and Hsie- Chia Chang, “Early-Pruned K-Best Sphere Decoding Algorithm Based on Radius Constraints,” IEEE Communications Society, ICC 2008.
  * [5] N. Sathish Kumar and K. R. Shankar Kumar, “Performance analysis and comparison of m x n zero forcing and MMSE equalizer based receiver for mimo wireless channel,” Songklanakarin J. Sci. Univ. 33 (3), 335-340, May - Jun. 2011.
  * [6] Michael T. Heath, “Scientific Computing: An Introductory Survey,” chapter 2, ISBN-10: 0071244891, 2001.
  * [7] Lloyd N. Trefethen, David Bau, “Numerical Linear Algebra,” lecture 11, 1997.


AppAppendix References

# DESED-FL and URBAN-FL: Federated Learning Datasets for Sound Event Detection

David S. Johnson1, Wolfgang Lorenz, Michael Taenzer, Stylianos Mimilakis,
Sascha Grollmisch, Jakob Abeßer and Hanna Lukashevich  Fraunhofer Institute
for Digital Media Technology (IDMT), Ilmenau, Germany
<EMAIL_ADDRESS>
###### Abstract

Research on sound event detection (SED) in environmental settings has seen
increased attention in recent years. The large amounts of (private) domestic
or urban audio data needed raise significant logistical and privacy concerns.
The inherently distributed nature of these tasks, make federated learning (FL)
a promising approach to take advantage of large-scale data while mitigating
privacy issues. While FL has also seen increased attention recently, to the
best of our knowledge there is no research towards FL for SED. To address this
gap and foster further research in this field, we create and publish novel FL
datasets for SED in domestic and urban environments. Furthermore, we conduct
baseline results on the datasets in a FL context for three deep neural network
architectures. The results indicate that FL is a promising approach for SED,
but faces challenges with divergent data distributions inherent to distributed
client edge devices.

###### Index Terms:

federated learning, sound event detection, deep learning, distributed learning

##  I Introduction

The aim of SED is to automatically identify the occurrence of target sound
events, such as glass breaking or dog barking, within an audio signal
capturing an acoustic scene. Identifying these sound events within complex
scenes is a challenging and open research problem that has attracted much
attention in recent years, observed with the increase in the literature and
particularly with the growing research interest in the DCASE
Community111http://dcase.community. Two common SED uses-cases are acoustic
monitoring in domestic [1] and urban [2] environments. In both scenarios, the
use of audio to train detection models in a centralized training context
raises considerable privacy concerns. In these environments, there are speech
and other sounds that are confidential which should not be shared or stored
insecurely.

State-of-the-art approaches to SED are most commonly based on deep learning
[3] which requires large centralized datasets for model training, posing
significant security and logistical challenges. FL [4] offers an attractive
approach to mitigate some of these concerns. Instead of sending private data
to a centralized data store, FL performs model training directly on many
client edge devices (from here on referred to as clients) using locally stored
data. The clients then share only their updated parameters with a coordination
server, which aggregates the shared parameters to update a global model. The
new global model is then transferred back to the clients. This process
continues until convergence, or indefinitely if new data is continuously
acquired [5].

Current research in FL has focused on image or text-based tasks. To our
knowledge, the only known research or practical applications of FL in the
audio domain are related to keyword spotting [6, 7, 8]. Due to the limited
research on FL for SED, there remain questions about the effectiveness of the
approach due to varying acoustic conditions inherent to distributed clients.
For example, data may be captured from clients in multiple locations with
different background noise characteristics or in locations with only a subset
of the sound event classes. This leads to differences in data distributions
amongst the clients involved in the training process. For centralized
training, data from multiple devices is combined into single training dataset
that is typically assumed to be independent and identically distributed (IID),
but with FL distributed data collection leads to models being trained using
data from divergent distributions, i.e., data that is non-IID. Existing SED
datasets do not capture the non-IID characteristics seen with FL. To address
this gap and foster research on FL for SED, we contribute novel SED datasets
specifically designed for FL training. Additionally, we provide baseline
results for three neural network architectures to evaluate the effects of FL
hyperparameters and non-IID data on SED performance.

TABLE I: Sound event and background classes for each of the datasets. Dataset |  Sound Event Classes |  Background Noise Classes
---|---|---
DESED-FL |  $e_{1}$: Dishes; $e_{2}$: Cat; $e_{3}$: Frying; $e_{4}$: Dog; $e_{5}$: Blender; $e_{6}$: Speech; $e_{7}$: Vacuum cleaner; $e_{8}$: Electric shaver/toothbrush; $e_{9}$: Alarm bell; $e_{10}$: Running water |  apartment room, computer interior, computer lab, emergency staircase, and library
URBAN-FL |  $e_{1}$: Children playing; $e_{2}$: Siren; $e_{3}$: Drilling; $e_{4}$: Street music; $e_{5}$: Car horn; $e_{6}$: Gun shot; $e_{7}$: Jackhammer; $e_{8}$: Dog bark; $e_{9}$: Air conditioner; $e_{10}$: Engine idling |  birds, crowd, fountain, rain, and traffic

##  II Related Work

###  II-A Federated Learning

McMahan et al. first proposed the idea of FL as a method to allow data to
remain on distributed devices while training a shared model by aggregating
locally trained updates [4]. For a comprehensive overview of FL, refer to the
technical report by Kairouz et al. [5]. There are two main challenges for FL
methods. First is the need for communicating over unreliable networks to
transmit data. A second issue results from data captured in varying contexts,
leading to datasets that are statistically dissimilar, non-IID, between
clients.

To address these challenges, Sattler et al. [9] proposed a compression
framework sparse ternary compression (STC). Similarly, Lin et al. [10] and
Bernstein et al.[11] proposed methods to remove the redundancies of gradient
information in node-distributed learning frameworks. Hsieh et al. evaluated
the challenges of non-IID data in an FL scenario [12]. They identified
problems with the batch normalization layer, a common layer in many deep
neural network (DNN) architectures, and proposed to use group normalization
[13] instead. Similarly, to address the problems of non-IID data, Sattler et
al. [14] proposed a clustering operation to group clients whose data
distributions have similar characteristics. While the previous research
proposed methods to overcome challenges in FL, there are no known datasets to
evaluate them for SED. We address this gap, by presenting new SED datasets
specifically designed for FL with non-IID data.

###  II-B Sound Event Detection

State-of-the-art SED algorithms build upon deep neural networks, the most
common being convolutional neural network (CNN) and convolutional recurrent
neural network (CRNN) based architectures. Both architectures include
convolutional front-ends, where multiple convolutional layers are trained to
learn sound-specific features. As input to the network, either fixed two-
dimensional signal transformations such as mel spectrograms [15] or raw one-
dimensional audio samples are used (end-to-end learning) [16]. As a back-end,
CNNs use fully-connected layers for classification whereas CRNNs employ
recurrent layers such as gated recurrent unit (GRU) or long short-term memory
(LSTM) layers to model the temporal progression of the extracted features. We
focus our work on CNN architectures for a lightweight approach, to enable
model training on low resource devices required by FL.

Training SED models requires strongly labeled datasets in which onset and
offset times are labeled for each sound event. Because of the laborious effort
required to annotate real-world samples, researchers often use synthetically
generated datasets. This requires mixing events from a curated sound bank with
a background signal to synthesize soundscapes with multiple, possibly
overlapping, events. For example, URBAN-SED [17] is composed of sound events
from the UrbanSound-8K (URBAN-8K) [2] dataset mixed with Brownian noise. A
recent trend is to use a combination of synthetic and real recordings for
training and evaluation as with Domestic Environment Sound Event Detection
(DESED) [1]. For both datasets, however, events are distributed uniformly
during soundscape generation. For FL, events should be distributed in a
structured fashion to simulate real-world distributed learning conditions. To
enable research in FL for SED, our proposed datasets distribute soundscapes to
clients with different background characteristics and class distributions.

##  III Datasets

In this section we present DESED-FL and URBAN-FL, datasets for acoustic
monitoring of domestic and urban environments with FL. Each use case contains
two independent training sets: an IID dataset, in which sound event classes
are distributed evenly to devices, and a non-IID dataset, in which only a
subset of the total classes is assigned to each client. To imitate varying
acoustic conditions, we mix sound events with one of five background noise
classes. Each training dataset is partitioned into 100 clients with 20 clients
per background noise class. It is possible to simulate more than 100 clients
by partitioning further, or less than 100 by combining or removing clients.
For reproducible evaluation, each use case also includes an evaluation dataset
in which sound events are uniformly distributed to each background class.

To generate the soundscapes for DESED-FL, sound events and background noises
are sourced from DESED [1]. URBAN-FL soundscapes are generated using sound
events from URBAN-8K [2], and background noises from Isolated Urban Sound
Database (IUSD) [18]. The sound event and noise classes for each dataset are
listed in Table I.

###  III-A IID and Non-IID Training Datasets

To generate the IID and non-IID dataset variations, two sound event class
distribution schemes are implemented for assigning events to clients. The IID
scheme uniformly distributes event classes to all devices. This is the best
possible case for training FL models since all clients have access to all
classes. To simulate a more realistic scenario, in which clients only have
knowledge of a subset of all classes, the non-IID scheme distributes events to
clients using five class distributions, with each distribution containing a
subset of five classes. To minimize the bias that could result from linking
background sounds to certain subsets of event classes, each of the five
distributions is assigned to four clients per background noise. This results
in each set of 20 clients per background noise comprising five class
distributions. A detailed view of each data distribution is presented in
Appendix A. In the non-IID scheme, each of the five class distributions
contains five event classes for a total of 25 event classes, which is not
divisible by 10, the number of total classes. This means that five event
classes were used three times and five event classes were used only twice.

One goal in designing the distributions is to have each set of classes be as
different as possible from any other set, while ensuring that each class is
used at least once. Hence, the algorithm to select class distributions
minimizes the penalty value $p$ computed by

$p=\sum_{k=1}^{N_{\text{coll}}-1}\sum_{l=k+1}^{N_{\text{coll}}}N_{\text{eq}}^{2}\left(k,l\right),$

where $N_{\text{coll}}$ equals the number of collections and
$N_{\text{eq}}\left(k,l\right)$ represents the number of equal classes in
collections $k$ and $l$. We do not take into account whether classes are
likely to occur together in real-world applications, since there are no
obvious combinations that would never happen together.

The distributions that minimize the penalty $p$ consists of the sets
$\left\\{e_{1},e_{2},e_{4},e_{6},e_{9}\right\\}$,
$\left\\{e_{1},e_{3},e_{4},e_{7},e_{8}\right\\}$,
$\left\\{e_{1},e_{3},e_{5},e_{6},e_{10}\right\\}$,
$\left\\{e_{2},e_{3},e_{5},e_{7},e_{9}\right\\}$, and
$\left\\{e_{2},e_{4},e_{5},e_{8},e_{10}\right\\}$, where each event $e_{i}$ is
assigned one of the available event classes. The mapping of the event classes
to the positions (i.e. $e_{\\#})$ have been randomized, and can be found in
Table I.

###  III-B Data Generation

The datasets consist of ten-second soundscapes synthetically generated using
Scaper [17] by mixing between one and five possibly overlapping source events
with one background noise type. Each event is mixed with an signal-to-noise
ratio (SNR) chosen from $\mathcal{N}(\mu,\,\sigma^{2})$ with
$\mu=$\mathrm{1}\mathrm{0}${dB}$ and $\sigma=$\mathrm{3}${dB}$. The sound
events are selected by sampling from the class distributions as discussed in
Section III-A. Additionally, source events are augmented by pitch shifting the
audio by an amount uniformly sampled from the range $[-2,2]$, and by time
stretching by a value uniformly sampled from the range $[0.8,1.25]$. These
augmentations are only applied to the training data.

Before generating the soundscapes, the source data is split into training and
evaluation sets to ensure that there is no data leakage. For DESED-FL events,
the data is partitioned into training and evaluation data according to split
provided by DESED. For Urban-FL, we take the approach employed for URBAN-SED,
by using the existing stratified folds from the URBAN-8K dataset for the
split: folds 1-6 are used for training and 9-10 for evaluation. We omit folds
7-8, used by URBAN-SED for validation, since the DESED source data does not
contain a validation split. The background noise for each dataset is split
into training and evaluation by splitting each source file into separate
training and evaluation segments.

The final training datasets each contain 100 ten-second soundscapes per edge
device, totaling $10\,000$ soundscapes. The evaluation datasets contain $400$
soundscapes per background class for a total of $2000$ soundscapes. To enable
reproducibility, the dataset creation scripts are available for
download222https://www.idmt.fraunhofer.de/en/publications/idmt-fl.html

##  IV Experimental Setup

###  IV-A Architectures

We propose three baseline architectures to evaluate different model
complexities and their effects on FL. One of the goals driving this research
is developing small models that are able to be trained on low resource
devices, such as neuromorphic hardware. Therefore, we evaluate two standard
CNN architectures of different sizes, and a Residual Network (ResNet)
architecture [19]. The baseline CNN, _CNN-Base_ , is a medium-sized
architecture based on the feature extraction front-end of the 2019 Detection
and Classification of Acoustic Scenes and Events (DCASE) Challenge Task 3
baseline architecture. It is composed of seven convolutional blocks and a
linear classification layer, for a total of $542\,442$ parameters. The second
CNN architecture is a small CNN, called _CNN-Sm_ , and was designed using
neural architecture search with Bayes optimization [20] to limit the model to
nearly $100\,000$ parameters while optimizing the F-score on the URBAN-SED
dataset. The found model has four convolutional blocks, a single feed forward
layer, and a classification layer, resulting in $115\,434$ parameters. Lastly,
we propose a medium-sized ResNet architecture, _ResNet_ , with five
independent component (IC) ResNet blocks [21] and a classification layer for a
total of $422\,090$ parameters. Detailed descriptions for each architecture
may be found in Appendix B.

###  IV-B Preprocessing

####  IV-B1 Input Representation

The input for each model is a perceptually weighted mel spectrogram [19]: The
input signal is first downsampled to $22\,050\text{\,}\mathrm{Hz}$. The short-
time Fourier transform (STFT) is applied with a fast Fourier transform (FFT)
of size $2048$ and a hop size of $512$, and is followed by perceptual
weighting. A mel-filter bank of 256 mel bands is then applied. Finally, 43
windows are stacked together resulting in a feature representation (43x256x1)
of one second.

####  IV-B2 Data Augmentation

Similar to Salamon and Bello [22], we apply pitch shifting to the audio data
before extracting the mel spectrograms. However, instead of applying all
shifts of $\pm 2$ and $\pm 1$ semitones for a total increase factor of 4, we
randomly select one semitone value between $\pm 2$ for each input for a total
augmentation factor of 1 (i.ẽ., doubling the size of the dataset). This helps
significantly reduce the final size of the dataset and has only a minor impact
on model performance.

###  IV-C Experimental Design

In our first experiment, the three proposed network architectures are
evaluated in a centralized training scenario. First, we train the models using
the original URBAN-SED dataset [2] to validate that the architectures work as
expected on a well-known dataset. Then, we train each of the three models
using the FL training sets. We evaluate the models, first, with batch
normalization, and then, replace all batch normalization layers with group
normalization to ensure that this substitution for mitigating non-IID issues
in FL [12] does not significantly affect the baseline results. All models are
trained for 50 epochs with early stopping using a patience of 25 epochs, which
monitors the validation loss. We use the Adam optimizer [23] with a cosine
learning rate schedule in the range $[1e-04,1e-06]$.

The next experiments evaluate the influence of FL hyperparameters on SED
performance, namely the total number of clients, $N$, the percentage of total
clients that participate in each training round (participation rate $r_{p}$),
and number of local epochs performed during each round, $E_{L}$. For $N$ we
evaluate 1, 25, and 100 clients, with $N=1$ providing a FL baseline. For
$r_{p}$, values of $\\{0.25,0.5,1.0\\}$ are used in order to simulate the
uncertainty of client participation during each communication round. At each
round a fraction, $r_{p}$, of the $N$ clients are uniformly sampled (with
replacement at the end of each round) to participate in training. As the
amount of network communication should be limited in FL, we evaluate the
effects of $E_{L}=\\{1,3,5\\}$ to reduce the number of communication rounds by
increasing amount of training steps performed on each client before sending
weight updates. For all FL experiments, the local client models are optimized
using Adam, with a learning rate $lr=1e-04$, proposed by Leroy et al. [6]. The
local weight updates are aggregated by the coordinator using the standard
Federating Averaging algorithm using stochastic gradient descent (SGD) with a
learning rate of $1.0$ [4].

Due to the number of experiments, the size of the datasets, and the training
times required for each experiment, we limit the number of communication
rounds during training to 60 rounds per experiment. Thus, reducing the time
and resources required for evaluation333In real-world FL, this would not be an
issue due to the inherent parallelization of FL, as opposed to simulating FL
on a single server.. While in some cases the models may not have completely
converged, the results provide valuable insights to improve our understanding
of the different effects of FL hyperparameters to better focus future
research.

##  V Results

###  V-A Centralized Training

Table II lists the segment-based F-Scores for each of the proposed datasets
and architectures from centralized training. Additionally, we include the
results of the architectures trained using the URBAN-SED [17] dataset as a
baseline to validate the architectures’ performance on a well-established
dataset.

In the case of URBAN-SED, the models perform comparable or better than the
original baseline of $F=0.57$ from Salamon et al. [17]. In general, the ResNet
architecture is typically the best performing model. Furthermore, replacing
batch normalization with group normalization has only minor effects on the
performance of all architectures, and in many of the cases it improves
performance. The results also indicate that data distribution, either IID or
non-IID, has minimal effect on model performance in a centralized context.
This is expected since all data is used during each training epoch, and the
model does not fit to a particular data distribution.

TABLE II: Centralized Training F-scores.

Dataset | IID | Norm | CNN-Sm | CNN-Base | ResNet
---|---|---|---|---|---
URBAN-SED | - | batch | 0.566 | 0.567 | 0.601
URBAN-SED | - | group | 0.532 | 0.587 | 0.589
URBAN-FL | ✓ | batch | 0.600 | 0.625 | 0.647
URBAN-FL | ✓ | group | 0.574 | 0.638 | 0.647
URBAN-FL | ✗ | batch | 0.593 | 0.609 | 0.634
URBAN-FL | ✗ | group | 0.564 | 0.618 | 0.642
DESED-FL | ✓ | batch | 0.627 | 0.632 | 0.630
DESED-FL | ✓ | group | 0.628 | 0.632 | 0.664
DESED-FL | ✗ | batch | 0.618 | 0.625 | 0.625
DESED-FL | ✗ | group | 0.621 | 0.634 | 0.647

###  V-B Federated Learning

(a) DESED-FL

(b) URBAN-FL

Figure 1: F-scores as a function of communication round for
$PR=$50\text{\,}\mathrm{\char 37\relax}$$, and for $N=\\{1,25,100\\}$. The top
row shows results of the IID datasets and bottom row non-IID. Each column in
the subfigures show the results for a specified $N$.

Figure 7 shows the segment-based F-scores on evaluation data for models
trained with each training datasets. DESED-FL results are shown in Figure
1(a), and URBAN-FL in Figure 1(b). Each subfigure includes the the results
from IID (top row) and non-IID (bottom row) datasets. The results are shown
for hyperparameters $N$, $E_{L}$, and $r_{p}$ as a function of communication
rounds. Here we present the results for $r_{p}=0.5$ with the remaining results
found in Appendix C. The training curves for IID data look similar to what is
expected in a centralized training scenario with the training times (i. e.,
number of communication rounds) being influenced by the architecture type,
participation rate, and the number of local epochs. Typically in the IID
setting, increasing model complexity and $E_{L}$ reduces the amount of
communication needed between client and server by improving the training time;
whereas, increasing $N$ slows down training. This could be attributed to the
fact that each local client has less data resulting in smaller gradient deltas
between the global and newly trained local model at each round. These
observations, however, do not necessarily hold in a non-IID context, where
performance degrades significantly for all models (except for the baseline
$N=1$). In this case, the results indicate that larger models are prone to
overfit to the data seen during an individual training round. This is
especially true with a small number of clients, such as $N=25$, since there is
inherently less variation in the randomly selected client distributions.
However, even in cases when all data is seen during each communication round,
i. e., $r_{p}=1.0$, the ResNet continues to overfit. Reducing $E_{L}$ helps to
alleviate this issue by limiting the gradient values, but slowing down
training. Training with a larger number of overall clients helps to mitigate
this issue as well. In general, the models in a non-IID scenario tend to
overfit to local distributions, and the federating averaging process does not
correct for this on its own. Damping the gradients before performing
aggregation, by lowering the server learning rate or by normalizing gradients
for example, may help to reduce the effects of large local gradients.

##  VI Conclusion

In this work, we introduce DESED-FL and URBAN-FL, two novel datasets to foster
research in FL for SED. To better understand the effects of previously
identified challenges associated with non-IID data in FL, we include both IID
and non-IID training sets for each use case. Additionally, we contribute the
first known research on FL for SED through the evaluation of three baseline
neural network architecture. The results show that while FL is a promising
approach for SED it is prone to challenges with non-IID data similar to
previous FL research [12, 9]. By contributing non-IID datasets, we hope enable
further research to identify potential solutions to mitigate these issues.

Future research directions include addressing the issues of non-IID data
through adaptive techniques such as the automatic adjustment of parameters
[12], or clustering techniques to identify groups of distributions and train
individual models accordingly [14]. Furthermore, an investigation on
strategies for local data management is needed. Since clients may have limited
storage capacities, an evaluation of how much data should be stored and for
how long is needed. Additionally, it is important to be able to identify which
data samples contain the most information and should be stored for later
training. To that end, the selective sampling of data based on entropy [24] is
an interesting research direction.

SED

    sound event detection
CNN

    convolutional neural network
CNN-S

    small CNN
CRNN

    convolutional recurrent neural network
DCASE

    Detection and Classification of Acoustic Scenes and Events
DESED

    Domestic Environment Sound Event Detection
DNN

    deep neural network
FFT

    fast Fourier transform
FL

    federated learning
GRU

    gated recurrent unit
IC

    independent component
IID

    independent and identically distributed
LSTM

    long short-term memory
ReLU

    rectified linear unit
ResNet

    Residual Network
RNN

    recurrent neural network
SGD

    stochastic gradient descent
STC

    sparse ternary compression
STFT

    short-time Fourier transform
SNR

    signal-to-noise ratio
URBAN-8K

    UrbanSound-8K
IUSD

    Isolated Urban Sound Database

## References

  * [1] R. Serizel, N. Turpault, A. Shah, and J. Salamon, “Sound event detection in synthetic domestic environments,” in _Proceedings of ICASSP 2020_ , Barcelona, Spain, May 2020.
  * [2] J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in _22nd ACM International Conference on Multimedia (ACM-MM’14)_ , Orlando, FL, USA, 11 2014, pp. 1041–1044.
  * [3] A. Dang, T. H. Vu, and J. Wang, “A survey of deep learning for polyphonic sound event detection,” in _2017 International Conference on Orange Technologies (ICOT)_ , 2017, pp. 75–78.
  * [4] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_ , 2017.
  * [5] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, and et al., “Advances and Open Problems in Federated Learning,” 2019.
  * [6] D. Leroy, A. Coucke, T. Lavril, T. Gisselbrecht, and J. Dureau, “Federated Learning for Keyword Spotting,” in _Proceedings of ICASSP 2019_ , 2019\.
  * [7] Apple. (2019) Designing for privacy (video and slide deck). [Online]. Available: https://developer.apple.com/videos/play/wwdc2019/708
  * [8] A. Bhowmick, J. Duchi, J. Freudiger, G. Kapoor, and R. Rogers, “Protection against reconstruction and its applications in private federated learning,” _arXiv preprint arXiv:1812.00984_ , 2018.
  * [9] F. Sattler, S. Wiedemann, K.-R. Muller, and W. Samek, “Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2019.
  * [10] Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, “Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training,” 2017.
  * [11] J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar, “signSGD: Compressed optimisation for non-convex problems,” 2018.
  * [12] K. Hsieh, A. Phanishayee, O. Mutlu, and P. B. Gibbons, “The Non-IID Data Quagmire of Decentralized Machine Learning,” 2019. [Online]. Available: http://arxiv.org/abs/1910.00189
  * [13] Y. Wu, “Group Normalization,” _ECCV_ , 2018. [Online]. Available: https://research.fb.com/publications/group-normalization/
  * [14] F. Sattler, K.-R. Müller, and W. Samek, “Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints,” 2019.
  * [15] Y. Hou, Q. Kong, S. Li, and M. D. Plumbley, “Sound Event Detection with Sequentially Labelled Data Based on Connectionist Temporal Classification and Unsupervised Clustering,” in _Proceedings of ICASSP 2019_ , 2019.
  * [16] E. Cakir and T. Virtanen, “End-to-End Polyphonic Sound Event Detection Using Convolutional Recurrent Neural Networks with Learned Time-Frequency Representation Input,” _Proceedings of the International Joint Conference on Neural Networks_ , vol. 2018-July, 2018.
  * [17] J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello, “Scaper: A library for soundscape synthesis and augmentation,” in _Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , 2017.
  * [18] J.-R. Gloaguen, M. Lagrange, A. Can, and J.-F. Petiot, “Isolated urban sound database,” Apr. 2018. [Online]. Available: https://doi.org/10.5281/zenodo.1213793
  * [19] K. Koutini, H. Eghbal-zadeh, M. Dorfer, and G. Widmer, “The Receptive Field as a Regularizer in Deep Convolutional Neural Networks for Acoustic Scene Classification,” in _Proceedings of EUSIPCO 2019_ , 2019.
  * [20] J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian Optimization of Machine Learning Algorithms,” in _NeurIPS_ , Lake Tahoe, Nevada, 2012, pp. 2951–2959.
  * [21] G. Chen, P. Chen, Y. Shi, C.-Y. Hsieh, B. Liao, and S. Zhang, “Rethinking the usage of batch normalization and dropout in the training of deep neural networks,” _arXiv preprint arXiv:1905.05928_ , 2019.
  * [22] J. Salamon and J. P. Bello, “Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification,” _IEEE Signal Processing Letters_ , vol. 24, no. 3, pp. 279–283, 3 2017.
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [24] F. Wiewel and B. Yang, “Entropy-based sample selection for online continual learning,” in _Proceedings of the EUSIPCO 2020_ , 2020.

##  Appendix A Datasets

TABLE III: DESED-FL and URBAN-FL Sound Event Statistics per Client Device Dataset | Version | Max Poly | Avg Poly | Avg Num Events | Avg Dur. (s) | Min Dur. (s) | Max Dur. (s)
---|---|---|---|---|---|---|---
URBAN-FL | IID | 5 | $1.97\pm 0.09$ | $2.98\pm 0.14$ | $2.57\pm 0.10$ | $0.09\pm 0.03$ | $4.98\pm 0.02$
URBAN-FL | Non-IID | 5 | $1.98\pm 0.08$ | $3.00\pm 0.14$ | $2.58\pm 0.15$ | $0.09\pm 0.03$ | $4.98\pm 0.02$
URBAN-FL | Eval | 5 | $1.98\pm 0.05$ | $3.03\pm 0.08$ | $2.52\pm 0.03$ | $0.11\pm 0.00$ | $4.00\pm 0.00$
DESED-FL | IID | 5 | $2.03\pm 0.09$ | $2.98\pm 0.14$ | $3.21\pm 0.20$ | $0.07\pm 0.02$ | $10.0\pm 0.00$
DESED-FL | Non-IID | 5 | $1.99\pm 0.28$ | $2.99\pm 0.15$ | $3.08\pm 1.18$ | $0.08\pm 0.02$ | $9.65\pm 1.16$
DESED-FL | Eval | 5 | $2.02\pm 0.04$ | $3.01\pm 0.07$ | $2.99\pm 0.04$ | $0.09\pm 0.00$ | $10.0\pm 0.00$

In this section, we present a detailed view of the device data and
distributions for the proposed IID and non-IID datasets, DESED-FL and URBAN-
FL. Table III lists detailed statistics regarding polyphony, i.e. the number
of events occurring simultaneously, number and duration of events per sound
soundscape. The statistics indicate that the datasets all exhibit similar
characteristics, with the only significant different in the max durations
between the DESED-FL and URBAN-FL datasets. This is due to the nature of the
source datasets. The max duration of the sound events in the URBAN-8K datasets
is four seconds, whereas in the DESED-FL source dataset, some events can be
longer than four seconds so the max duration is limited to the length of the
soundscapes.

Figures 2 and 4 show the total duration of each of the sound events as
distributed to the background noise classes for each of the training and
evaluation datasets of DESED-FL and URBAN-FL respectively. As expected, the
distributions of data between each background. The data was distributed evenly
amongst background classes to mitigate any potential background biases in the
final trained models. Figures 3 and 5, provide a detailed look at the
individual client distributions. The event durations bar charts for the IID
data, in Figures 3(a) and 5(a), show that the sound events are identically
distributed for all devices. Whereas, in the non-IID data, shown in Figures
3(b) and 5(b), there are five different distributions for each background
class, indicated by the surrounding green boxes. This distribution scheme
reduces the potential for background bias by ensuring all sound events are
distributed among each background type while ensuring non-IID client devices.

(a) DESED-FL IID training data

(b) DESED-FL non-IID training data

(c) DESED-FL evaluation data

Figure 2: DESED-FL: Total sound event durations for each background for the
two versions of training data and the evaluation dataset. The sound event
classes from left to right are: alarm_bell_ringing, blender, cat, dishes, dog,
electric_shaver_toothbrush, frying, running_water, speech, and vacuum_cleaner.

(a) DESED-FL IID training data event duration per client

(b) DESED-FL non-IID training data event duration per client

Figure 3: Distribution of data, by event duration, to client devices for the
DESED-FL dataset. Each row shows the distribution for a specific background
type. The green boxes indicate the five different class distribution groupings
for the non-IID dataset.

(a) URBAN-FL IID training data

(b) URBAN-FL non-IID training data

(c) URBAN-FL evaluation data

Figure 4: URBAN-FL: Total sound event durations for each background for the
two versions of training data and the evaluation dataset. The sound event
classes from left to right are: air_conditioner, car_horn, children_playing,
dog_bark, drilling, engine_idling, gun_shot, jackhammer, siren, street_music

(a) URBAN-FL IID training data event duration per client

(b) URBAN-FL non-IID training data event duration per client

Figure 5: Distribution of data, by event duration, to client devices for the
URBAN-FL dataset. Each row shows the distribution for a specific background
type. The green boxes indicate the five different class distribution groupings
for the non-IID dataset.

##  Appendix B Detailed Architectures

In this section, we provide details on the three architectures used in our
experiments on SED with FL to foster reproducibility of baseline results on
the proposed datasets. As baseline architectures we propose two standard CNN
architectures, a small one with close to $100\,000$ parameters and a medium-
sized one. Additionally, we proprosed a medium-sized ResNet based
architecture.

TABLE IV: CNN-Base Architecture with $542\,442$ parameters Layer | Output | Kernel Size | Droput
---|---|---|---
Conv Block | 16 | (3, 3) | 0.20
Avg Pooling | - | (2, 2) | -
Conv Block | 32 | (3, 3) | 0.20
Avg Pooling | - | (2, 2) | -
Conv Block | 64 | (3, 3) | 0.20
Avg Pooling | - | (1, 2) | -
Conv Block | 128 | (3, 3) | 0.20
Avg Pooling | - | (1, 2) | -
Conv Block | 128 | (3, 3) | 0.20
Avg Pooling | - | (1, 2) | -
Conv Block | 128 | (3, 3) | 0.20
Avg Pooling | - | (1, 2) | -
Conv Block | 128 | (3, 3) | 0.20
Global Avg Pooling | - | - | -
Dense | 10 | - | -
Sigmoid | - | - | -

Both CNN-Base and CNN-Small are standard CNN architectures composed of
convolution blocks each followed by a pooling layer. Each convolutional block,
shown in Figure 6(b), contains a 2D convolution, a normalization layer, an
activation layer, and 20% dropout. Global pooling is performed before the
final output layer. CNN-Base is based on the baseline architecture from the
DCASE 2019 challenge for SED turpault:dcase2019baseline without the recurrent
layers, and is composed of seven convolutional blocks. The recurrent layers
were removed in order to maintain compatibility with neuromorphic hardware.
The architecture and hyperparameters are listed in Table IV. The resulting
architecture contains $542\,442$ trainable parameters.

TABLE V: CNN-S Architecture with $115.434$ parameters Layer | Output | Kernel Size | Dropout
---|---|---|---
Conv Block | 16 | (3, 3) | 0.2
Pooling 2D | - | (2, 2) | -
Conv Block | 32 | (3, 3) | 0.2
Pooling 2D | - | (2, 2) | -
Conv Block | 64 | (3, 3) | 0.2
Pooling 2D | - | (2, 2) | -
Conv Block | 128 | (3, 3) | 0.2
Global Pooling | - | - | -
Dense | 128 | - | -
ReLu | - | - | -
Dense | 10 | - | -
Sigmoid | - | - | -

CNN-Small is a CNN that was designed using Bayesian Optimization neural
architecture search Snoek2012:bayes_opt444Implemented with
http://github.com/fmfn/BayesianOptimization to keep the model close to
$100\,000$ parameters. The final architecture is composed of four
convolutional blocks, seen in 6(b) and a classification network with one
hidden layer and one output layer, for a total of $115\,434$ parameters. The
architecture and hyperparameters are listed in Table IV.

(a) ResNet Block with IC Layers

(b) Convolution Block

Figure 6: Blocks used for the ResNet and CNN architectures TABLE VI: Resnet Architecture with $422\,090$ parameters Layer | Output | Kernel Size 1 | Kernel Size 2 | Dropout
---|---|---|---|---
Conv 2D | 64 | (5, 5) | - | -
Relu | - | - | - | -
ResNet Block | 64 | (3, 3) | (1, 1) | 0.10
Avg Pooling | - | (2, 2) | - | -
ResNet Block | 64 | (3, 3) | (3, 3) | 0.10
Avg Pooling | - | (2, 2) | - | -
ResNet Block | 64 | (3, 3) | (3, 3) | 0.10
Avg Pooling | - | (2, 2) | - | -
ResNet Block | 128 | (3, 3) | (1, 1) | 0.10
Avg Pooling | - | (2, 2) | - | -
ResNet Block | 256 | (1, 1) | (1, 1) | 0.10
Avg Global Pooling | - | - | - | -
Dense | 10 | - | - | -
Sigmoid | - | - | - | -

The ResNet architecture is composed of ResNet blocks with IC layers as
proposed by Chen et al. chen2019.rethinking shown in Fig. 6(a) with
hyperparameters inspired by Koutini et al. Koutini2019 to minimize the
receptive field of the model. The IC blocks contain a normalization layer
followed by dropout. For additional regularization, l2 weight regularization
is performed as well as disabling of centering and scaling during the
normalization process McDonnell2020. The final architecture contains
$422\,090$ trainable parameters.

##  Appendix C Extended Results

In this section, we present extended results from our FL experiments. In
addition, to $r_{p}=0.5$, presented in the main article, results for
$r_{p}=\\{0.25,\>1.0\\}$ are shown in Figures 7 and 8. The percentage of
clients involved in a training round has minimal effects on the training
peformance for IID data, while in the case of non-IID training is more
stabilized as the percentage increases. This is most likely a result of the
central server receiving weight gradients from all possible distributions at
each training round.

(a) N Clients = 1

(b) N Clients = 25

(c) N Client = 100

Figure 7: DESED-FL: Federated Learning F1 (micro average) Score on Validation
Data

(a) N Clients = 1

(b) N Clients = 25

(c) N Client = 100

Figure 8: URBAN-FL: Federated Learning F1 (micro average) Score on Validation
Data

IEEEtran IEEEabrv,literature.bib

  *[SED]: sound event detection
  *[FL]: federated learning
  *[IID]: independent and identically distributed
  *[STC]: sparse ternary compression
  *[DNN]: deep neural network
  *[CNN]: convolutional neural network
  *[CRNN]: convolutional recurrent neural network
  *[CNNs]: convolutional neural network
  *[CRNNs]: convolutional recurrent neural network
  *[GRU]: gated recurrent unit
  *[LSTM]: long short-term memory
  *[URBAN-8K]: UrbanSound-8K
  *[DESED]: Domestic Environment Sound Event Detection
  *[IUSD]: Isolated Urban Sound Database
  *[SNR]: signal-to-noise ratio
  *[ResNet]: Residual Network
  *[DCASE]: Detection and Classification of Acoustic Scenes and Events
  *[IC]: independent component
  *[STFT]: short-time Fourier transform
  *[FFT]: fast Fourier transform
  *[SGD]: stochastic gradient descent
  *[CNN-S]: small