# Single-pixel edge enhancement of object via convolutional filtering with
localized vortex phase

Jigme Zangpo* and Hirokazu Kobayashi  Graduate School of Engineering, Kochi
University of Technology, 185 Miyanokuchi, Tosayamada, Kami City, Kochi
782-8502, Japan<EMAIL_ADDRESS>
###### Abstract

Microscopy is an essential tool in imaging research, and the edge-enhanced
microscope by using the vortex filter is of particular interest as an optical
information processing that highlights amplitude and phase edges of object in
all directions. The application of this technique is not limited to the
visible range, but edge enhancement of object in invisible wavelength is also
crucial for near-infrared fluorescence and electronic circuit inspection
through silicon semiconductors. One disadvantage of near-infrared imaging is
that digital cameras such as CCD and CMOS become much more expensive than
cameras for the visible spectrum. As an cost-effective method to implement
invisible edge enhancement, the Fourier single-pixel imaging has already been
proposed without using a camera, but using a single-pixel detector. However,
this method requires 3 or 4 times more single-pixel measurements due to the
three-phase or four-phase shift to detect optical complex amplitude in Fourier
domain. In response, we propose a method for single-pixel edge enhancement of
object via convolutional filtering with a localized vortex phase, eliminating
the extra single-pixel measurements required by the phase-shifting method. Our
simulation results show that the correlation coefficient between the ideal
edges of an object and the edge enhanced by our proposed method is 0.95,
indicating that our method is effective way to detect the edges. This novel
and effective approach for enhancing and detecting the edges of object can be
valuable in various invisible imaging applications.

††journal: osajournal††articletype: Research Article

##  1 Introduction

Microscopy is an essential tool in imaging research, and edge-enhanced
microscope with a vortex filter has been particularly attracted more attention
to enhance the edges of phase and amplitude objects due to their capability
for all-directional edge enhancement[1, 2, 3, 4, 5, 6], while differential
interference-contrast microscopy enhances edges in only one-direction[7]. The
application of this technique is not limited to the visible range, but edge
enhancement of object in invisible wavelength is also crucial for near-
infrared fluorescence and electronic circuit inspection through silicon
semiconductors. One disadvantage of near-infrared imaging is that digital
cameras, such as silicon-based charge-coupled devices (CCD) and complementary
metal-oxide-semiconductors (CMOS), tend to become expensive than cameras for
the visible spectrum[8].

To address this issue, researchers have opted for a more cost-effective
approach by utilizing single-pixel detectors with reconstruction of two-
dimensional image by correlation calculation[8, 9, 10]. The so-called single-
pixel imaging can detect the weak signal and image with a high speed[11, 12,
13, 14]. In Refs. [15, 16, 17], researchers used single-pixel imaging to
capture complex amplitude distribution of objects using a four-step phase-
shifting technique. Subsequently, researchers attempted to integrate single-
pixel imaging into an edge-enhanced microscope, where Fourier single-pixel
imaging was adopted to capture the edges of the phase object using four or
three-step phase shifting[18, 19, 20, 21]. The advantage of their approach is
the absence of a vortex filter, instead, they utilized a digital micromirror
device (DMD) to generate a reference wave and an optical vortex with the
assistance of a super-pixel technique[22, 23, 24], which is highly robust and
easy to use while offering full spatial control over the phase and amplitude
of a light field[22, 25]. While this method successfully detected and enhanced
the edges of the phase object, the drawback is that due to the four or three-
step phase shifting, resulting in four or three times single-pixel
measurements are required for reconstruction.

To reduce the number of single-pixel measurements, we propose a method,
namely, a single-pixel edge-enhanced microscope via convolutional filtering
with a localized vortex phase, which can eliminate the extra single-pixel
measurements required by the phase-shifting method. We conducted numerical
simulations to validate this approach, resulting in that the correlation
coefficient of edge enhancement of the object through our proposed method and
ideal edge enhancement is 0.95, which indicates that the proposed method is
indeed capable of effectively enhancing the edges of object.

In Section 2 we describe proposed method, the single-pixel edge enhancement of
object, and principle of super-pixel method. In Sections 3, we present the
numerical simulation results. Section 4 concludes the study.

##  2 Single-pixel Edge Enhancement via Convolutional Filtering with Localized
Vortex Phase

###  2.1 Principle of single-pixel edge enhancement via convolutional
filtering

Figure 1 shows $4f$ system with a vortex filter to implement all-directional
edge enhancement placed after the bright-field microscope. The $4f$ system
consists of object plane, Fourier plane and image plane. At the object plane,
the object magnified by the bright-field microscope is a phase-amplitude
object (PAO), whose brightness (from 0 to 1) denotes amplitude and color
denotes phase. In the object, a yellow circle and a horizontal black strip
indicate a phase object and an amplitude object, respectively. The input
object $f_{\text{in}}(\bm{r})$ magnified by the bright-field microscope
undergoes the Fourier transformation by first lens and the Fourier spectrum is
given by $F(\bm{k})=\mathcal{F}\\{f(\bm{r})\\}$, where $\mathcal{F}$
represents the Fourier transform and $\bm{k}=(k_{x},k_{y})$ denotes the two-
dimensional transverse wavenumber vector. Then, $F(\bm{k})$ is modulated by
vortex phase plate, $H(\bm{k})=e^{\\\ i\theta}$ with an azimuthal angle
$\theta$ at the Fourier plane and the image is obtained after inverse Fourier
transform by second lens as follow:

$f_{\text{out}}(\bm{r})=\mathcal{F}^{-1}\\{F(\bm{k})H(\bm{k})\\}=f_{\text{in}}(\bm{r})*h(\bm{r}),$
(1)

where $*$ denotes the convolution and
$h(\bm{r})=\frac{{\mathrm{i}}}{2\pi}\frac{e^{{\mathrm{i}}\theta}}{r^{2}}$ is
the inverse Fourier transform of $H(\bm{k})$ and denotes the point spread
function of the optical system with a vortex phase plate as the Fourier
filter. The PAO, captured by the bright-field microscope, undergoes edge
enhancement by the $4f$ system in Fig. 1, as shown at image plane.

Figure 1: The $4f$ system with a vortex filter for all-directional edge
enhancement..

In Eq. (1),
$h(\bm{r})=\frac{{\mathrm{i}}}{2\pi}\frac{e^{{\mathrm{i}}\theta}}{r^{2}}$
consists of a spiral phase $e^{i\theta}$ and a localized amplitude function
$a(\bm{r})\propto 1/r^{2}$. In general, requirement of the filter function for
the edge enhancement is the localized amplitude function multiplied with the
azimuthal phase, i.e., $h(\bm{r})=a(\bm{r})e^{i\theta}$. The convolution of
$f_{\text{in}}(\bm{r})*h(\bm{r})$ can be rewritten as

$\bm{f_{\text{out}}}(\bm{r})=\iint_{-\infty}^{\infty}f_{\text{in}}(\bm{r^{\prime}})h(\bm{r}-\bm{r^{\prime}})d\bm{r^{\prime}}=\mathcal{F}\\{f_{\text{in}}(\bm{r^{\prime}})h(\bm{r}-\bm{r^{\prime}})\\}(0),$
(2)

where the object $f_{\text{in}}(\bm{r})$ multiplies with shifted patterns of
$h(\bm{r})$ and then Fourier transformed, followed by extracting the center
amplitude. Equation (2) can be implemented as shown in Fig. 2, where the
object light is multiplied with the shifted localized vortex phase
$h(\bm{r}-\bm{r^{\prime}})$ generated by the super-pixel method using the DMD
and then Fourier transformed by Fourier lens in the $2f$ system, followed by
extracting the desired location with an aperture. This aperture has two
functions: the first function is to select the first-order diffraction to
generate the localized vortex phase by super-pixel method as shown later; the
second function is to select the center position according to Eq. (2).
Finally, the single-pixel detector, such as photodiode, measures the
corresponding optical power for different patterns over time to reconstruct
the enhanced edges of the object by reshaping one-dimensional temporal data to
two-dimensional format, as shown in Fig. 2 with the caption ’Edge
Enhancement’.

Figure 2: Single-pixel edge-enhanced microscope via convolutional filtering.
In $m$ different patterns, a white dot denotes the scanning path of patterns,
while a red line denotes the patterns that have completed the scanning. For
each scanning of pattern, the photodiode measures the corresponding optical
power as voltage data in time and the edge-enhanced image can be obtained by
reshaping the temporal voltage data to two-dimensional format.

###  2.2 Generation of localized vortex phase by super-pixel method with DMD

Unlike a spatial light modulator (SLM), which can generate phase and amplitude
holograms[26, 27, 28, 29, 30], the DMD can only generate binary amplitude
hologram represented by binary reflections with 1 or 0 indicating turn on and
off, respectively. In the super-pixel method[25], however, both the amplitude
and phase modulation of light wave can be implemented by extracting the first-
order diffraction generated by the binary amplitude hologram. This method
offers higher beam shaping fidelity than SLMs[31] and much faster refresh
rates over 10 kHz for binary amplitude patterns, whereas most SLMs are limited
to 120 Hz[32]. However, these advantages come at the expense of limited
modulation depth and low diffraction efficiency[31].

Figure 3 (a) shows schematic $4f$ setup of the super-pixel method to obtain
full spatial control over the phase and amplitude of light on target
plane[22]. The DMD placed at the front focal plane of the first lens is
partitioned into super-pixels, which are square sub-groups of $n\times n$
micromirrors. The diffracted light from the DMD is Fourier transformed by the
first lens and an aperture with radius of $R$ acts as a spatial filter on the
Fourier plane to capture the first-order diffraction. The position of the
aperture is selected to ensure that the responses of adjacent pixels within
the super-pixel on the DMD exhibit a phase difference of $2\pi/n^{2}$ in the
$x$-direction and $2\pi/n$ in the $y$-direction, as shown in Fig. 3 (b), where
some pixels are highlighted with green color to indicate that they are turned
on. Figure 3 (c) shows the appropriate aperture position, which is obtained as
$(x,y)=(-\rho,n\rho)$, where $\rho=\frac{-\lambda f}{n^{2}d}$, $\lambda$ is
the wavelength of the light, $f$ is the focal length of the first lens and $d$
is pixel size of the micromirror. Here we notice that optimizing the aperture
radius $R$ is crucial because a small $R$ will reduce the desired signal and a
large $R$ will contain the undesired zero-order diffraction. The first-order
diffraction extracted by the aperture further undergoes inverse Fourier
transform by the second lens and the resultant $E_{\text{out}}$ at the target
plane is proportional to the sum of only valid phases on individual pixel
responses, which are evenly distributed on the complex plane along a circle as
shown in Fig. 3 (d). To determine the resultant of electric field
$E_{\text{out}}$ at the target plane, we used the following equation:

$E_{\text{out}}=E_{\text{in}}\sum_{m=0}^{n^{2}-1}D_{m}e^{{\mathrm{i}}2\pi
m/n^{2}},$ (3)

where $E_{\text{in}}$, $D_{m}$ and n denote the input electric field, the
binary values 0 or 1 of the $m$-th pixel within the super-pixel, and the
super-pixel size, respectively. As shown in Fig. 2, the second $2f$ system in
Fig. 3 (a) is not required in the proposed single-pixel edge enhancement, due
to the Fourier transform in Eq. (2).

Figure 3:  (a) Setup of super-pixel method. (b) Super-pixel with the size of
$4\times 4$ pixels exhibiting a phase difference of $2\pi/n^{2}$ in the
$x$-direction and $2\pi/n$ in the $y$-direction. Two pixels with green color
are turned on with 100% reflectance while the other pixels are turned off with
0% reflectance. (c) Aperture with radius of $R$ placed at $(-\rho,4\rho)$ on
the Fourier plane to extract the first-order diffraction of DMD. (d) The
resultant field $E_{\text{out}}$ (red arrow) on the target plane is
proportional to the sum of the phases on the pixels that are turned on.

In Fig. 4 (a), we plot the complex amplitude for all possible combinations of
$D_{m}$ within the super-pixel with the size of $n=4$. As can be seen from
Fig. 4 (a), the possible complex amplitude is distributed within a disk with
radius of $\text{cot}(\pi/n^{2})\approx 5$ on the complex plane. In the set of
65536 patterns, there are many duplicated complex fields and only 6561
patterns have unique fields. For the purpose of making a hologram using the
super-pixel method, each pixel in the desired complex function is replaced
with a appropriate super-pixel with the size of $n\times n$. The replacement
process involves comparing the resultant field of the super-pixel with each
pixel in the desired complex function and the nearest values from all the
possible fields generated by the super-pixel are then assigned to each pixel
in the desired complex function. Figure 4 (b) illustrates a grid of $16\times
16$ pixels of the desired function with localized vortex phase. By employing
the super-pixel method, each $4\times 4$ super-pixel replaces individual
pixels in the desired complex function in Fig. 4 (b), resulting in a hologram
pattern as shown in Fig. 4 (c) with a dimension of $64\times 64$ pixels.

Figure 4: (a) Complex amplitude generated by all possible combinations of
pixels corresponding to the super-pixel with $n=4$. (b) The desired complex
amplitude distribution with $16\times 16$ pixels. (c) The hologram with
$64\times 64$ pixels generated by replacing super-pixel $4\times 4$ to the
$16\times 16$ desired complex function.

##  3 Numerical Simulation Results of the Proposed Method

###  3.1 Edge enhancement capability of localized amplitude functions

In this section, we consider several types of the localized amplitude
functions $a(\bm{r})$, such as $\frac{1}{r^{2}}$, $\frac{1}{r}$,
$\frac{1}{\sqrt{r}}$ and $e^{-r^{2}/w^{2}}$ with the radius $w$, as plotted in
Fig. 5 (a), where we used the optimized radius $w=35$ µm for the Gaussian
function, explained in Section 3.3. The complex amplitude distribution of
$h(\bm{r})=a(\bm{r})e^{{\mathrm{i}}\theta}$ with the several localized
amplitude functions $a(\bm{r})$ are shown in Figs. 5 (b) - (e). Here we note
that $\frac{1}{r^{2}}$, $\frac{1}{r}$, and $\frac{1}{\sqrt{r}}$ in Fig. 5 are
normalized by their function values at $r_{\text{min}}$, which is the minimum
radial value in the simulation coordinate.

Figure 5: (a) Radial dependence of the amplitude functions, $\frac{1}{r^{2}}$,
$\frac{1}{r}$, $\frac{1}{\sqrt{r}}$ and $e^{-r^{2}/w^{2}}$ with $w=35$ µm.
(b-e) Corresponding complex amplitude distributions of the localized vortex
phase.

Before using the super-pixel method, we used Eq. (1) to numerically verify the
edge enhancement capability of the localized amplitude functions. Here we
consider the PAO in Fig. 6 (a) as the object. We first perform inverse Fourier
transform on $h(\bm{r})$ to obtain $H(\bm{k})$ and then perform a Fourier
transform on the PAO, followed by multiplication with $H(\bm{k})$. Finally, we
apply an inverse Fourier transform to the Fourier spectrum and then take the
absolute square to obtain the edges of the PAO. The simulation parameters are
as follows: a wavelength of 635 nm, a total dimension of $1\text{mm}\times
1\text{mm}$, a step size of $1/199\text{mm}\times 1/199\text{mm}$, which
corresponds to $200\times 200$ pixels.

Figure 6: Edge enhancement capability of several localized amplitude
functions. (a) Complex amplitude distribution of the PAO. (b) Ideal edge
enhancement. (c-f) Edge enhancement using $\frac{1}{r^{2}}$, $\frac{1}{r}$,
$\frac{1}{\sqrt{r}}$, and $e^{-r^{2}/w^{2}}$ with $w=35$ µm in $h(\bm{r})$,
respectively.

Figure 6 shows the numerical simulation results of the edge enhancement using
the localized vortex phase $h(\bm{r})$. Figure 6 (b) shows the ideal edge
enhancement of PAO by using $H(\bm{k})=k_{r}e^{{\mathrm{i}}\theta}$ with
${k}_{{r}}\equiv\sqrt{k_{x}^{2}+k_{y}^{2}}$, while Figs. 6 (c-f) illustrate
edge enhancement using the localized amplitude functions $a(\bm{r})$. The
localized amplitude functions $\frac{1}{r}$ and $\frac{1}{\sqrt{r}}$ failed to
detect the edges of the PAO while the functions $\frac{1}{r^{2}}$ and
$e^{-r^{2}/w^{2}}$ successfully detected the edges of the PAO. The correlation
coefficients between ideal edge enhancement in Fig. 6 (b) and Figs. 6 (c-f)
are 0.80, 0.25, 0.06 and 0.85, respectively. The reason why the amplitude
functions $\frac{1}{r}$ and $\frac{1}{\sqrt{r}}$ of optical vortex failed to
detect the edges of object is that their radial dependence is not tightly
localized compared to $\frac{1}{r^{2}}$ and $e^{-r^{2}/w^{2}}$, i.e., the
tails of the radial dependence for the amplitude functions $\frac{1}{r}$ and
$\frac{1}{\sqrt{r}}$ do not rapidly approach to zero, instead, they exhibit
higher value, as can be seen in Fig. 5 (a).

###  3.2 Fidelity estimation of super-pixel method to generate localized
vortex phase

The modulation accuracy of the super-pixel method between the target field
$E_{\text{out}}$ and the desired field $E_{\text{desire}}$ can be quantified
by calculating their fidelity $F$ as shown below:

$F=\frac{|E_{\text{desire}}^{*}\cdot
E_{\text{out}}|^{2}}{|E_{\text{desire}}|^{2}|E_{\text{out}}|^{2}},$ (4)

where the two-dimensional distributions $E_{\text{out}}$ and
$E_{\text{desire}}$ are reshaped into one column and their inner product is
calculated in the numerator, and $*$ denotes the complex conjugate. $F=1$
indicates that the target fields and the desired fields are the same, whereas
$F=0$ means they are completely different.

We conducted a numerical simulation to estimate the fidelity between the
target field generated by the super-pixel method and the desired localized
vortex phase with the amplitude functions, $(c/r)^{2}$, $c/r$, $\sqrt{c/r}$
and $ce^{-r^{2}/w^{2}}$, where $c$ denotes amplitude coefficient. The
optimizing parameters to obtain the best fidelity in the super-pixel method
are the amplitude coefficient $c$ and the aperture radius $R$. For the
Gaussian amplitude function, an additional parameter $w$ needs to be optimized
and here we used $w=100$ µm, which gives the highest fidelity of 0.89. The
simulation conditions remain the same as those in Section 3.1, with an
additional parameter, a focal length of the lens setting at 300 mm.

Figures 7 (a-d) show the simulation results of the fidelity between the ideal
and the generated localized vortex phases with different $a(\bm{r})$:
$(c/r)^{2}$, $c/r$, $\sqrt{c/r}$ and $ce^{-r^{2}/w^{2}}$, respectively. The
horizontal and vertical axes in Fig. 7 show the aperture radius $R$ normalized
by aperture position $\rho$ and the amplitude coefficient $c$ of $h(\bm{r})$,
respectively. Table 1 shows summary of the maximum fidelities and the
corresponding optimized parameters. The Gaussian amplitude function
$e^{-r^{2}/w^{2}}$ provides fidelity $0.89$, whereas $\frac{1}{r^{2}}$,
$\frac{1}{r}$ and $\frac{1}{\sqrt{r}}$ yield fidelity values below $0.6$
(refer to Table 1), because these power functions increase rapidly as they
approach to the origin, resulting in low fidelity with the finite amplitudes
in the super-pixel method. Even if the fidelity to the desired function is
low, however, edge enhancement might be possible if a certain degree of
localization is achieved by the amplitude function. Thus, in the next section,
we use the amplitude functions $\frac{1}{r^{2}}$ and $e^{-r^{2}/w^{2}}$ which
successfully performed edge enhancement in Fig. 6, to evaluate our single-
pixel method via convolutional filtering.

Figure 7: (a-d) The fidelity of localized vortex phase with amplitude
functions:$\frac{1}{r^{2}}$, $\frac{1}{r}$, $\frac{1}{\sqrt{r}}$ and
$e^{-r^{2}/w^{2}}$ ($w=100$ µm) for different amplitude coefficients $c$ and
aperture radius, $R$ normalized by aperture position $\rho$. Table 1:
Simulation results of maximum fidelity and optimized parameters with different
$a(\bm{r})$

Parameter $a(\bm{r})$ $\frac{1}{r^{2}}$ $\frac{1}{r}$ $\frac{1}{\sqrt{r}}$
$e^{-r^{2}/w^{2}}$ Max. fidelity, $F$ 0.02 0.45 0.57 0.89 Radius, $R/\rho$ 2.5
2.6 2 0.7 Amp. coefficient, $c$ 42 µm 100 µm 100 µm 9

###  3.3 Edge enhancement capability of single-pixel imaging via convolutional
filtering with localized vortex phase

Now, we evaluate edge enhancement capability of our proposed single-pixel
imaging method with the localized amplitude functions, $\frac{1}{r^{2}}$ and
$e^{-r^{2}/w^{2}}$. We first prepared the binary hologram on the DMD to
generate the shifted localized vortex phase $h(\bm{r}^{\prime}-\bm{r})$ by
using the super-pixel method with $n=4$. By following the principles of
Section 2.1, we calculated Fourier transform of the object light multiplied
with the hologram on the DMD and then obtained optical power at the center
point of the aperture, $(x,y)=(-\rho,n\rho)$, to reconstruct the two-
dimensional edge enhanced image. The simulation conditions remain the same as
those in Section 3.1.

Figure 8: (a) Distribution of PAO, (b) and (c) Edge enhancement via proposed
method using optical vortex with the amplitude function $1/r^{2}$ and
$e^{-r^{2}/w^{2}}$ with $w=35$ µm, respectively. Figure 9: Correlation
coefficient between the edge enhanced by the Gaussian amplitude function of
the optical vortex and the ideal edge enhancement for varying values of $w$
and amplitude coefficient $c$.

Figure 8 (a) represent the complex amplitude distribution of the PAO. The
ideal edge to compare with edge enhancement of the object by vortex phases
with the amplitude function $1/r^{2}$ and $e^{-r^{2}/w^{2}}$ is same as Figs.
6 (c) and (d) but with a different pixel size i.e., $50\times 50$, which
represents the total pixel size for the single-pixel edge enhancement
reconstruction. The best edge enhancement using localized vortex phases with
the amplitude function $1/r^{2}$ and $e^{-r^{2}/w^{2}}$ are shown in Figs. 8
(b) and (c), respectively. The amplitude function $e^{-r^{2}/w^{2}}$ could
successfully enhance the edge of the object using the proposed single-pixel
edge enhancement technique while the amplitude function $1/r^{2}$, which could
successfully enhance the edge in Section 3.1, failed to enhance the edge. The
main reason of this failure is considered as low fidelity (0.02), as shown in
Table 1. Figure 9 shows dependence of the correlation coefficient between the
single-pixel edge enhancement by using the Gaussian amplitude function and the
ideal edge enhancement on the optimizing parameters $w$ and $c$. We selected
the starting point of $w$ greater than 15 µm, which exceeds the radius of the
super-pixel size of 10 µm, because $w$ values lower than 15 µm do not detect
the edges of the object. The highest correlation coefficient 0.95 is achieved
when $w=35$ µm and $c=6$. The edge enhancement using these optimized
parameters is illustrated in Fig. 8 (c). The strong and positive relationship
between the ideal edge and the proposed method’s edge demonstrates that the
proposed approach is indeed capable of effectively enhancing the edges of
objects.

##  4 Conclusion

In this paper, we proposed and numerically demonstrated edge enhancement of
objects using single-pixel imaging via convolutional filtering, in which the
localized vortex phase consists of a spiral phase $e^{{\mathrm{i}}\theta}$ and
a localized amplitude function $a(\bm{r})$. We have tested several localized
functions to find the highest fidelity using super-pixel method for use in our
proposed method and a Gaussian amplitude function offers the highest fidelity
(0.89). We numerically demonstrated the proposed method and the correlation
coefficient between ideal edge enhancement and our method was 0.95. The
results show that the proposed method is capable of enhancing the edge of the
PAO while eliminating extra single-pixel measurements required by previous
studies. Our proposed method would be applied to invisible wavelength range,
such as near-infrared fluorescence and electronic circuit inspection through
silicon semiconductor. We plan to conduct experimental demonstration of the
proposed method in the future.

Funding Japan Society for the Promotion of Science (18KK0079, 20K05364);
Research Foundation for Opto-Science and Technology.

Disclosures The authors declare no conflicts of interest.

Data availability Data underlying the results presented in this paper are not
publicly available at this time but may be obtained from the authors upon
reasonable request.

## References

  * [1] S. Fürhapter, A. Jesacher, S. Bernet, and M. Ritsch-Marte, “Spiral phase contrast imaging in microscopy,” Optics Express 13, 689–694 (2005).
  * [2] A. Jesacher, S. Fürhapter, S. Bernet, and M. Ritsch-Marte, “Shadow effects in spiral phase contrast microscopy,” Physical Review Letters 94, 233902 (2005).
  * [3] S. Bernet, A. Jesacher, S. Fürhapter, C. Maurer, and M. Ritsch-Marte, “Quantitative imaging of complex samples by spiral phase contrast microscopy,” Optics Express 14, 3792–3805 (2006).
  * [4] S. Fürhapter, A. Jesacher, C. Maurer, S. Bernet, and M. Ritsch-Marte, “Spiral phase microscopy,” Advances in Imaging and Electron Physics 146, 1–59e (2007).
  * [5] J. Zangpo, T. Kawabe, and H. Kobayashi, “Edge-enhanced microscopy of complex objects using scalar and vectorial vortex filtering,” Optics Express 31, 38388–38399 (2023).
  * [6] J. Zangpo and H. Kobayashi, “Isolation of phase edges using off-axis q-plate filters,” arXiv preprint arXiv:2310.18044 (2023).
  * [7] W. Lang, _Nomarski differential interference-contrast microscopy_ (Carl Zeiss Oberkochen, 1982).
  * [8] M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Processing Magazine 25, 83–91 (2008).
  * [9] G. M. Gibson, S. D. Johnson, and M. J. Padgett, “Single-pixel imaging 12 years on: a review,” Optics Express 28, 28190–28208 (2020).
  * [10] M. P. Edgar, G. M. Gibson, and M. J. Padgett, “Principles and prospects for single-pixel imaging,” Nature Photonics 13, 13–20 (2019).
  * [11] N. Radwell, K. J. Mitchell, G. M. Gibson, M. P. Edgar, R. Bowman, and M. J. Padgett, “Single-pixel infrared and visible microscope,” Optica 1, 285–289 (2014).
  * [12] M. P. Edgar, G. M. Gibson, R. W. Bowman, B. Sun, N. Radwell, K. J. Mitchell, S. S. Welsh, and M. J. Padgett, “Simultaneous real-time visible and infrared video with single-pixel detectors,” Scientific Reports 5, 10669 (2015).
  * [13] M.-J. Sun, M. P. Edgar, G. M. Gibson, B. Sun, N. Radwell, R. Lamb, and M. J. Padgett, “Single-pixel three-dimensional imaging with time-based depth resolution,” Nature Communications 7, 12010 (2016).
  * [14] S. D. Johnson, D. B. Phillips, Z. Ma, S. Ramachandran, and M. J. Padgett, “A light-in-flight single-pixel camera for use in the visible and short-wave infrared,” Optics Express 27, 9829–9837 (2019).
  * [15] Z. Zhang, X. Ma, and J. Zhong, “Single-pixel imaging by means of Fourier spectrum acquisition,” Nature Communications 6, 6225 (2015).
  * [16] Z. Zhang, X. Wang, G. Zheng, and J. Zhong, “Fast Fourier single-pixel imaging via binary illumination,” Scientific Reports 7, 12029 (2017).
  * [17] R. Liu, S. Zhao, P. Zhang, H. Gao, and F. Li, “Complex wavefront reconstruction with single-pixel detector,” Applied Physics Letters 114 (2019).
  * [18] Y. Liu, P. Yu, X. Hu, Z. Wang, Y. Li, and L. Gong, “Single-pixel spiral phase contrast imaging,” Optics Letters 45, 4028–4031 (2020).
  * [19] X. Hu, H. Zhang, Q. Zhao, P. Yu, Y. Li, and L. Gong, “Single-pixel phase imaging by fourier spectrum sampling,” Applied Physics Letters 114 (2019).
  * [20] H. Ren, S. Zhao, and J. Gruska, “Edge detection based on single-pixel imaging,” Optics Express 26, 5501–5511 (2018).
  * [21] Z. Qiu, X. Guo, T. Lu, P. Qi, Z. Zhang, and J. Zhong, “Efficient Fourier single-pixel imaging with Gaussian random sampling,” in _Photonics,_ vol. 8 (MDPI, 2021), p. 319.
  * [22] S. A. Goorden, J. Bertolotti, and A. P. Mosk, “Superpixel-based spatial amplitude and phase modulation using a digital micromirror device,” Optics Express 22, 17999–18009 (2014).
  * [23] Y.-X. Ren, Z.-X. Fang, L. Gong, K. Huang, Y. Chen, and R.-D. Lu, “Dynamic generation of Ince-Gaussian modes with a digital micromirror device,” Journal of Applied Physics 117 (2015).
  * [24] Y.-X. Ren, Z.-X. Fang, L. Gong, K. Huang, Y. Chen, and R.-D. Lu, “Digital generation and control of Hermite–Gaussian modes with an amplitude digital micromirror device,” Journal of Optics 17, 125604 (2015).
  * [25] E. Van Putten, I. M. Vellekoop, and A. Mosk, “Spatial amplitude and phase modulation using commercial twisted nematic LCDs,” Applied Optics 47, 2076–2081 (2008).
  * [26] L. Waller, G. Situ, and J. W. Fleischer, “Phase-space measurement and coherence synthesis of optical beams,” Nature Photonics 6, 474–479 (2012).
  * [27] M. Woerdemann, C. Alpmann, M. Esseling, and C. Denz, “Advanced optical trapping by complex beam shaping,” Laser & Photonics Reviews 7, 839–854 (2013).
  * [28] C. Maurer, A. Jesacher, S. Bernet, and M. Ritsch-Marte, “What spatial light modulators can do for optical microscopy,” Laser & Photonics Reviews 5, 81–101 (2011).
  * [29] B. Bhaduri, C. Edwards, H. Pham, R. Zhou, T. H. Nguyen, L. L. Goddard, and G. Popescu, “Diffraction phase microscopy: principles and applications in materials and life sciences,” Advances in Optics and Photonics 6, 57–119 (2014).
  * [30] A. B. Parthasarathy, K. K. Chu, T. N. Ford, and J. Mertz, “Quantitative phase imaging using a partitioned detection aperture,” Optics Letters 37, 4062–4064 (2012).
  * [31] S. Turtaev, I. T. Leite, K. J. Mitchell, M. J. Padgett, D. B. Phillips, and T. Čižmár, “Comparison of nematic liquid-crystal and DMD based spatial light modulation in complex photonics,” Optics Express 25, 29874–29884 (2017).
  * [32] A. B. Ayoub and D. Psaltis, “High speed, complex wavefront shaping using the digital micro-mirror device,” Scientific Reports 11, 18837 (2021).


# A Survey on Universal Approximation Theorems

Midhun T. Augustine
Automation Lab
Indian Institute of Technology Bombay, India
<EMAIL_ADDRESS>

###### Abstract

This paper discusses various theorems on the approximation capabilities of
neural networks (NNs), which are known as universal approximation theorems
(UATs). The paper gives a systematic overview of UATs starting from the
preliminary results on function approximation, such as Taylor’s theorem,
Fourier’s theorem, Weierstrass approximation theorem, Kolmogorov–Arnold
representation theorem, etc. Theoretical and numerical aspects of UATs are
covered from both arbitrary width and depth.

_K_ eywords Machine learning $\cdot$ Deep learning $\cdot$ Neural Network
$\cdot$ Universal Approximation Theorems.

##  1 Introduction

A neural network (NN) or artificial neural network (ANN) is a network of
artificial neurons arranged in layers [1, 2]. The artificial neurons (also
called perceptrons) are inspired by biological neurons in biological neural
networks (BNNs)[3]. Biological neurons are the signal-processing units of BNN
in the brain, similarly, artificial neurons are data-processing units in ANN.
The rest of the paper discusses only ANN and artificial neurons which will be
referred to simply by NN and neuron. From a mathematical point of view,
neurons are made of compositions of a nonlinear function (also called
activation function) and a linear function [4]. Therefore, NNs which are a
network of neurons can be considered as a nonlinear function. This survey
focuses on the results on feedforward neural networks (FNNs) aka multi-layer
perceptrons (MLPs) which are the simplest and most popular category of NNs.
Nowadays, NN is one of the most trending areas in artificial intelligence (AI)
and machine learning (ML) because of its ability to model complicated
relationships [5, 6, 7]. UATs are theorems associated with the approximation
capabilities of NNs i.e., the ability of an NN to approximate arbitrary
functions [8]. In general, UATs imply that NNs with appropriate parameters can
approximate any continuous functions, i.e. are generalized models that can
represent complicated relationships in the data [9, 10, 11]. Further, UATs
also focus on the expressivity of NNs which describes the class of functions
an NN (with a given structure) can approximate [12, 13]. The structure of an
NN can be characterized by the number of layers in an NN aka depth, and the
maximum number of neurons in a layer aka width. There exist a few surveys on
UATs [8, 14, 15] which focus on specific areas such as soft computing,
differential geometry, etc. The initial surveys on UATs [8, 16] cover the
results from arbitrary width direction only. Even though the approximation
capability of NNs is of both theoretical and practical interest, a
comprehensive survey that covers results from arbitrary width and depth
directions is absent in the literature to the best of the author’s knowledge.
This motivates this paper in which the main objective is to provide a detailed
survey on UATs which is useful to students and researchers interested in this
topic.

The paper is organized as follows. Section 2 gives a brief introduction to NNs
and the terminologies associated with NNs. Section 3 starts with the
preliminary results on function approximation which is followed by UATs in the
direction of arbitrary width and arbitrary depth. Finally, section 4 gives the
conclusions and further reading.

Notations and definitions: $\mathbb{N}$ and $\mathbb{R}$ denote the set of
natural numbers and real numbers, respectively. $\mathbb{R}^{n}$ represents
$n$ \- dimensional Euclidean space and $\mathbb{R}^{m\times n}$ refers to the
space of $m\times n$ real matrices. Matrices and vectors are denoted by
boldface letters ($\textbf{A},\textbf{a}$), scalars by normal font ($A,a$),
and sets by blackboard bold font ($\mathbb{A}$). The $p-$ norm or $l^{p}$ norm
of a vector $\textbf{x}\in\mathbb{X}\subseteq\mathbb{R}^{n}$ is defined as:

$\parallel\textbf{x}\parallel_{p}=\big{(}|x_{1}|^{p}+|x_{2}|^{p}+...+|x_{n}|^{n}\big{)}^{\frac{1}{p}}.$
(1)

The $L^{p}$ norm of a function $\textbf{f}:\mathbb{X}\rightarrow\mathbb{Y}$ is
defined as

$\parallel\textbf{f}\parallel_{p}=\bigg{(}\int_{\mathbb{X}}{\parallel\textbf{f}(\textbf{x})\parallel}_{p}^{p}d\textbf{x}\bigg{)}^{\frac{1}{p}}.$
(2)

Lebesgue spaces or $\mathbb{L}^{p}$ spaces are function spaces containing
measurable functions f satisfying:

$\parallel\textbf{f}\parallel_{p}<\infty.$ (3)

A subset $\mathbb{X}$ of $\mathbb{R}^{n}$ is said to be compact if it is
closed and bounded. A subset $\mathbb{S}$ of a topological space $\mathbb{T}$
is said to be dense in $\mathbb{T},$ if every element of $\mathbb{T}$ either
belongs to $\mathbb{T}$ or arbitrarily close to an element of $\mathbb{T}.$
Examples of Topological spaces are Euclidean spaces, metric spaces, manifolds,
etc.

##  2 Neural Network (NN)

NN is a network of neurons arranged in layers. The layers are connected
sequentially, i.e., the output of each layer goes to the next layer as input.
This results in a network or multi-layer configuration. Consequently, NNs can
represented using composite functions of the form [17]:

$\textbf{y}=\textbf{f}_{\text{NN}}(\textbf{x})=\textbf{f}_{\text{L+1}}(\textbf{f}_{\text{L}}...\textbf{f}_{1}(\textbf{f}_{0}(\textbf{x})))$
(4)

where $\textbf{x}\in\mathbb{R}^{n}$ is the input,
$\textbf{y}\in\mathbb{R}^{m}$ is the predicted output,
$\textbf{f}_{\text{NN}}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ is the NN
function, and $\textbf{f}_{0},\textbf{f}_{1},...,\textbf{f}_{\text{L+1}}$ are
layers of the NN. Here $\textbf{f}_{0}$ represents the input layer,
$\textbf{f}_{\text{L+1}}$ is the output layer, and
$\textbf{f}_{1},...,\textbf{f}_{\text{L}}$ are the hidden layers (see Fig.
1(a)).

Figure 1:  (a) Neural Network (b) Neuron.

The number of hidden layers is denoted by L and the total number of layers in
the NN is $\text{L+2},$ i.e., including the input and output layer. Let the
input to $i^{th}$ layer is denoted as $\textbf{x}_{i}$ and the output of
$i^{th}$ layer as $\textbf{y}_{i}.$ Then each layer of the NN can be
represented as:

$\textbf{y}_{i}=\textbf{f}_{i}(\textbf{x}_{i})=\bm{\sigma}_{i}(\textbf{A}_{i}\textbf{x}_{i}+\textbf{b}_{i})$
(5)

where $\bm{\sigma}_{i}=\left[\begin{matrix}\sigma_{i_{1}}\\\ \vdots\\\
\sigma_{i_{m_{i}}}\end{matrix}\right]$ contains the element-wise activation
functions for the $i^{th}$ layer, $m_{i}$ is the number of neurons in $i^{th}$
layer, $\textbf{A}_{i}\in\mathbb{R}^{m_{i}\times m_{i-1}}$ is the weight
matrix, $\textbf{b}_{i}\in\mathbb{R}^{m_{i}}$ is the bias vector,
$\textbf{x}_{i}\in\mathbb{R}^{m_{i-1}}$ and
$\textbf{y}_{i}\in\mathbb{R}^{m_{i}}$. Note that the layers in Eq. (5) itself
are composite functions and the NN is a composition of layers which also
becomes a composite function. The affine term
$\textbf{A}_{i}\textbf{x}_{i}+\textbf{b}_{i}$ is the linear component of NNs
and the activation function is the nonlinear component. In general, any real-
valued function can be used as an activation function. The following are the
commonly used activation functions [18]:

  1. 1.

ReLU (Rectified Linear Unit) function:

$\sigma(x)=\max(0,x)=\begin{cases}0,\hskip 8.5359pt&if\hskip 2.84544ptx\leq
0\\\ x,\hskip 8.5359pt&if\hskip 2.84544ptx>0.\end{cases}$ (6)

  2. 2.

Step function:

$\sigma(x)=\begin{cases}0,\hskip 8.5359pt&if\hskip 2.84544ptx\leq 0\\\
1,\hskip 8.5359pt&if\hskip 2.84544ptx>0.\end{cases}$ (7)

  3. 3.

Logistic function:

$\sigma(x)=\frac{1}{1+e^{-x}}.$ (8)

  4. 4.

Tanh (Hyperbolic tangent) function:

$\sigma(x)=tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}.$ (9)

The graph of these functions is given in Fig. 2. Note that logistic and tanh
functions come under the category of sigmoid functions. In general, a sigmoid
function is a bounded function (normal range [0,1]) that has a non-negative
derivative at each point in $\mathbb{R}$ and exactly one inflection point,
i.e., sigmoid functions are characterized by an $S-$shaped curve as in Figs.
2(c) and (d).

Figure 2:  Graph of activation functions: (a) ReLU (b) Step (c) Logistic (d)
Tanh.

Terminologies associated with NNs and their brief description are given below:

  * •

Neuron: is the data processing units in NNs. Neurons can be represented as a
scalar-valued function:

$y_{i_{j}}=\sigma_{i_{j}}(\textbf{A}_{i_{j}}\textbf{x}_{i}+b_{i_{j}})$ (10)

where $y_{i_{j}}$ is the output of $j^{th}$ neuron in the $i^{th}$ layer,
$\textbf{A}_{i_{j}}=\left[\begin{matrix}a_{1}&a_{2}&\dots
a_{m_{i-1}}\end{matrix}\right]$ is the $j^{th}$ row of the weight matrix
$\textbf{A}_{i},$ and $b_{i_{j}}$ is the $j^{th}$ element of the bias vector
$\textbf{b}_{i}$ (see Fig. 1(b)).

  * •

Layer: is a collection of neurons that takes the same inputs. In general,
layers are vector-valued functions. Layers can grouped into three categories:
input, output, and hidden layers.

  * •

Input layer: in which each neuron has only one input and it simply passes the
input, i.e.,
$\bm{\sigma}_{0}(\textbf{A}_{0}\textbf{x}+\textbf{b}_{0})=\textbf{x}.$

  * •

Hidden layer: in which each neuron can have more than one input and gives a
scalar output. The activation function is normally nonlinear.

  * •

Output layer: in which each neuron can have more than one input and gives a
scalar output. The activation function is normally linear for regression
problems and nonlinear in the case of classification problems.

  * •

Width: is the maximum number of neurons in a layer, i.e., width
$\text{W}=\underset{i}{\max}\hskip 5.69046ptm_{i}.$

  * •

Depth: is the number of hidden layers plus output layer, i.e., depth D = L+1.

  * •

Shallow NN: is an NN with only one hidden layer, i.e., $\text{D}=2.$

  * •

Deep NN: is an NN with more than one hidden layer, i.e., $\text{D}>2.$

As an example, we can consider an NN with one input, one output, a linear
output layer, and one hidden layer with three neurons which can be represented
as

$y=\textbf{A}_{2}\bm{\sigma}_{1}(\textbf{A}_{1}x+\textbf{b}_{1})+b_{2}.$ (11)

Let
$\textbf{A}_{2}=\left[\begin{matrix}0.1&0.3&0.7\end{matrix}\right],\textbf{A}_{1}=\left[\begin{matrix}3\\\
-1\\\ 2\end{matrix}\right],$ $\textbf{b}_{1}=\left[\begin{matrix}-1\\\ 4\\\
10\end{matrix}\right],$ $b_{2}=2,$ $\bm{\sigma}_{1}=\textbf{tanh}$ which will
expand Eq. 11 as:

$y=0.1\hskip 2.84544pttanh(3x-1)+0.3\hskip 2.84544pttanh(-x+4)+0.7\hskip
2.84544pttanh(2x+10)+2$ (12)

and for $x\in[-10,10]$ we can compute the output by substituting in Eq. (11).
Fig. 3(a) shows the plot of the output $y$.

Figure 3:  Illustrating NN.

Now by changing
$\textbf{A}_{2}=\left[\begin{matrix}-0.4&0.2&-0.3\end{matrix}\right]$ and
keeping $\textbf{A}_{1},b_{1},b_{2}$ as before, we get the output as in Fig.
3(b). Similarly for
$\textbf{A}_{2}=\left[\begin{matrix}-0.3&0.5&0.1\end{matrix}\right]$ and
$\textbf{A}_{2}=\left[\begin{matrix}0.2&-0.7&0.6\end{matrix}\right]$ the
output will be as in Figs. 3(c) and (d), respectively. These examples
illustrate that we can represent complicated nonlinear relationships using
NNs. As the main use of NNs is modeling relationships in the data, the
question of the approximation capabilities of NNs gained interest naturally.
Now the question is can we approximate any continuous function using NNs? This
leads to the theoretical results known as UATs which are discussed next.

##  3 Universal Approximation Theorems (UATs)

Universal approximation theorems (UATs) are theoretical results associated
with the approximation capabilities of NNs. In general, UATs are explored in
the following directions:

  1. 1.

Arbitrary width case: deals with approximation capabilities of NNs with an
arbitrary number of neurons (with a limited number of hidden layers). For eg.
Fig 4(a) shows an NN with one hidden layer that has arbitrary neurons.

  2. 2.

Arbitrary depth case: is associated with NNs with an arbitrary number of
hidden layers (with a limited number of neurons). For eg. Fig 4(b) shows NN
with arbitrary hidden layers and one neuron per layer.

Before moving on to UATs in arbitrary width and depth cases, a brief summary
of the preliminary results on function approximation is given next.

Figure 4:  (a) NN with arbitrary width (b) NN with arbitrary depth.

###  3.1 UATs: Predecessors

The ability of NNs to approximate arbitrary functions with a desired accuracy
has theoretical and practical significance. The theoretical interests
originated from the mathematical areas: linear algebra [4], analysis [19],
topology [20], etc. The problem of representing a continuous function as a
superposition of simpler components (also known as basis functions) has been
studied long back. One of the initial results in this direction is the
Taylor’s theorem[21, 22] which is stated below:

######  Theorem 1 (Taylor, 1715).

Any continuous function $f(x):\mathbb{R}\rightarrow\mathbb{R}$ that is $k-$
times differentiable at $a\in\mathbb{R}$ can be represented as a sum of
polynomials:

$f(x)=f(a)+f^{{}^{\prime}}(a)(x-a)+\frac{f^{{}^{\prime\prime}}(a)}{2}(x-a)^{2}+...+\frac{f^{k}(a)}{k!}(x-a)^{k}+R_{k}(x)=\sum_{i=0}^{k}c_{i}(x-a)^{i}+R_{k}(x)$
(13)

where $c_{i}=\frac{f^{i}(a)}{i!}=\frac{1}{i!}\frac{d^{i}}{dx^{i}}f(x)|_{x=a}$
and $R_{k}(x)=o(|x-a|^{k})$ is the residual term.

A special case of the Eq. 13 for $a=0$ gives:

$f(x)=\sum_{i=0}^{k}c_{i}x^{i}+R_{k}(x)$ (14)

which is a more commonly used version and the summation is called the
Maclaurin series. Taylor/Maclaurin series shows that polynomial functions can
approximate any continuous smooth functions with sufficient accuracy. Taylor’s
theorem can be considered as a mathematical foundation for linearization-based
methods. Similar ideas were presented in the Fourier series[23, 24] in which
the results are initially given for periodic functions:

######  Theorem 2 (Fourier, 1807).

Any continuous and periodic function $f(x)$ can be expressed as a sum of
sinusoids:

$f(x)=A_{0}+\sum_{i=1}^{N}A_{i}cos(\frac{2\pi i}{T}x+\phi_{i}).$ (15)

where $T$ is the period of the function, $A_{i}$ is the amplitude and
$\phi_{i}$ is the phase of $i^{th}$ harmonic component.

Later on, the results are generalized for nonperiodic functions as well, which
resulted in the Fourier transform. The Fourier series and transform resulted
in the origin of frequency domain analysis and design methods. Another
milestone in this area is the Weierstrass approximation theorem [25] which can
be considered as an extension of Taylor’s theorem to arbitrary continuous
functions and is stated below:

######  Theorem 3 (Weierstrass, 1885).

Any continuous real-valued function $f(x):[a,b]\rightarrow\mathbb{R}$ defined
on the interval $[a,b]$ can be approximated with a polynomial function
$p_{N}(x)=\sum_{i=0}^{N}c_{i}x^{i}$ with finite degree $N$ such that:

$|f(x)-p_{N}(x)|<\epsilon$ (16)

for arbitrary $\epsilon>0$.

Weierstrass’s theorem implies that any continuous function on a closed
interval can be uniformly approximated by a polynomial function with arbitrary
accuracy. The Weierstrass approximation theorem can be considered as a
mathematical foundation for polynomial regression and interpolation methods.

Later on in the 1900s, the problem of decomposing continuous multivariable
functions as a finite superposition of continuous univariate and bivariate
functions attracted a lot of research interest. This started with the
continuous variant of Hilbert’s $13^{th}$ problem [26]: Can any continuous
function of more than two variables be expressed as a superposition of
finitely many continuous functions of two variables? This problem was solved
by Arnold and Kolmogorov in 1957 which resulted in the Kolmogorov-Arnold
representation theorem [27] stated below:

######  Theorem 4 (Kolmogorov and Arnold, 1959).

Any continuous multivariate function $f:[0,1]^{n}\rightarrow\mathbb{R}$ can be
written as

$f(\textbf{x})=f(x_{1},x_{2},...,x_{n})=\sum_{j=1}^{2n+1}\beta_{j}\bigg{(}\sum_{i=1}^{n}\alpha_{ij}(x_{i})\bigg{)}$
(17)

where $\alpha_{ij}:[0,1]\rightarrow\mathbb{R}$ and
$\beta_{j}:\mathbb{R}\rightarrow\mathbb{R}.$

Theorems 1 to 4 above gave insights into the approximation capabilities of
sinusoidal functions, polynomial functions, etc. After the introduction of
NNs, approximation capabilities of sigmoid functions gained popularity, since
in the initial versions of NNs sigmoid functions were used as activation
functions. The approximation capabilities of NNs with different activation
functions such as ReLU, step, and tanh were introduced later. This resulted in
several theorems known by the name of UATs which are discussed in the rest of
the paper.

###  3.2 UATs: Arbitrary width case

This section discusses results on approximation capabilities of NNs with
arbitrarily large widths. The depth or number of layers in the NN is
considered to be bounded. As an example, consider the following NNs with one
hidden layer with ReLU activation functions and a linear output layer:

$\displaystyle(a)\hskip 14.22636pty=\text{ReLU}(2x-4)$ (18)
$\displaystyle(b)\hskip 14.22636pty=\text{ReLU}(-x-3)$ $\displaystyle(c)\hskip
14.22636pty=\text{ReLU}(2x-4)+\text{ReLU}(-x-3)$ $\displaystyle(d)\hskip
14.22636pty=0.3\hskip 1.42271pt\text{ReLU}(2x-4)+0.7\hskip
1.42271pt\text{ReLU}(-x-3)-0.5\hskip 1.42271pt\text{ReLU}(4x-20)$

which corresponds to NNs with one hidden layer neuron (a,b), two neurons (c),
and three neurons (d). For these cases, the outputs are shown in Fig. 5. The
plots show that the number of folds in the curve increases with the number of
neurons. For example, the curve in Fig. 5 (c) and (d) has two and three folds
which correspond to NNs with two and three hidden layer neurons, respectively.
This indicates that an arbitrary number of folds can be introduced to the
curve by an NN with arbitrary neurons.

Figure 5:  Output of NNs with one hidden layer and ReLU activation function.

Folded lines are also called a piecewise linear curve. Therefore, with one
hidden layer and ReLU activation function, we can obtain piecewise linear
approximations of continuous functions. A theoretical result on the number of
folds and linear pieces is presented in [28] which is stated below:

######  Theorem 5 (Pascanu et al., 2013).

Consider the graph of $\textbf{f}_{\text{NN}}(\textbf{x})$ which has folds
along $H$ hyperplanes defined linear equations
$\textbf{A}_{1_{j}}\textbf{x}+b_{1_{j}}=0,$ $\textbf{x}\in\mathbb{R}^{n},$
$j=1,2,...,H$. Then the number of linear pieces of $\textbf{f}_{\text{NN}}$
is:

$n_{l}=\sum_{i=0}^{n}{{}^{H}}{C_{i}}.$ (19)

In general, the approximation capabilities (the number of folds or bends in
the outputs increases) as the number of neurons in the hidden layer increases
which can observed from Fig. 5. This makes NNs with a sufficient number of
neurons and hidden layers a universal approximator that can used for modeling
data with arbitrary patterns, i.e., NNs are generalized models. This is the
central idea of UATs.

The initial versions of UATs were introduced during the 1980s which attempted
to extend the Kolmogorov-Arnold representation theorem and other preliminary
results to NNs. In Cun [29] and Lapedes and Farber [30] it has been shown that
continuous functions can be arbitrarily approximated using NNs with two hidden
layers and monotonic activation functions. Irie and Miyake [31] showed that
arbitrary functions can be approximated by a one hidden layer network with an
infinite number of neurons. Later on, Gallant and White [32] showed one hidden
layer with a monotone cosine activation function can give Fourier series
approximation to a given function as its output. However, this approximation
is limited to only square-integrable functions on a compact set (see
Dirichlet’s conditions [33] for Fourier series approximation). Later on
generalizations of these results to arbitrary continuous functions in a
compact set were introduced by Funahashi [9], Hornick et al. [10], and Cybenko
[11] which are the popular versions of UATs in the arbitrary width case. The
UATs are mostly stated using the density of the functions
$\textbf{f}_{\text{NN}}(\textbf{x})$ generated by NNs within a given function
space of interest, i.e. if $\textbf{f}_{\text{NN}}(\textbf{x})$ is dense in a
given space, then, any function in that space can be approximated by
$\textbf{f}_{\text{NN}}$. This leads to the following theorem:

######  Theorem 6 (Funahashi, Hornick et al., and Cybenko, 1989).

Let $\mathbb{X}$ be any compact subset of $\mathbb{R}^{n}$ and $\sigma$ be any
sigmoid activation function, then the finite sum of the form:

$f_{\text{NN}}(\textbf{x})=\textbf{A}_{2}\bm{\sigma}(\textbf{A}_{1}\textbf{x}+\textbf{b}_{1})=\sum_{j=1}^{m_{1}}a_{2_{j}}\sigma(\textbf{A}_{1_{j}}\textbf{x}+b_{1_{j}})$
(20)

is dense in $\mathbb{X}.$ In other words, given any
$f:\mathbb{X}\rightarrow\mathbb{R}$ and $\epsilon>0$, there is a finite sum:
$f_{\text{NN}}$ as in Eq. (20) for which
$|f(\textbf{x})-f_{\text{NN}}(\textbf{x})|<\epsilon$ for all
$\textbf{x}\in\mathbb{X}$.

The above theorem means that NNs with one hidden layer and sigmoid activation
function can approximate any continuous univariate function on a bounded
domain with arbitrary accuracy. Hornik [34] showed that the primary reason
behind the universal approximation capabilities of NNs is the multilayer feed-
forward architecture. In [35] it is shown that MLP with non-polynomial
activation functions are universal approximators which resulted in the
following theorem:

######  Theorem 7 (Leshno et al., 1993).

Let $\mathbb{X}$ be any compact subset of $\mathbb{R}^{n}$ and $\sigma$ be an
activation function, then the finite sum in Eq. 20 is dense in $\mathbb{X}$
iff $\sigma$ is not a polynomial function.

Later on, similar theorems were developed for other activation functions such
as ReLU, step, tanh, etc which are non-polynomial functions. Therefore,
depth-2 NNs (NNs with one hidden layer aka shallow NNs) with a suitable
activation function are universal approximators. The same holds for NNs with
more than one hidden layer aka deep NNs. However, depth-1 NNs or NNs without
any hidden layer have limited approximation capability. Further results of
UATs for shallow NNs can be found in [36, 37].

###  3.3 UATs: Arbitrary depth case

This section focuses on the approximation capabilities of deep NNs, i.e. NNs
with arbitrary depth and bounded width. The arbitrary depth case has attained
a lot of research interest recently, especially after the introduction of deep
learning as a separate domain in ML. To get a better understanding of this
idea, let’s start with NNs with one neuron per layer, i.e. NNs with width
$\text{W}=1$ and arbitrary depth as shown in Fig. 4(b). In that case, each
layer will be a scalar-valued function. Consider the following NNs with ReLU
activation functions in the hidden layer and a linear output layer:

$\displaystyle(a)\hskip 14.22636pty=\text{ReLU}(0x+5)$ (21)
$\displaystyle(b)\hskip 14.22636pty=\text{ReLU}(2x+4)$ $\displaystyle(c)\hskip
14.22636pty=\text{ReLU}\big{(}-0.5[\text{ReLU}(2x+4)]+5\big{)}$
$\displaystyle(d)\hskip
14.22636pty=\text{ReLU}\big{(}-2\big{[}\text{ReLU}\big{(}-0.5[\text{ReLU}(2x+4)]+5\big{)}\big{]}+3\big{)}$

which corresponds to NNs with one hidden layer (a,b), two hidden layers (c),
and three hidden layers (d). For these cases, the outputs are shown in Fig. 6.

Figure 6: Output of NNs with one neuron per layer.

For NNs with one neuron per layer and ReLU activation function, the output can
be either a straight line as in Fig. 6(a), piecewise linear curve with one
fold as in Fig. 6(b), or piecewise linear curve with two folds as in Fig. 6(c)
and (d). The tilted portion in Fig 6(c) can be rotated by multiplying it with
a weight, which changes the slope. However, the horizontal portions in Fig.
6(c) correspond to a constant output. Therefore, multiplying it with a weight
term gives another constant output. This means the horizontal portions in Fig.
6(c) can not be rotated with a linear mapping. Consequently, after two folds,
the subsequent layers cannot add any further folds which can be observed in
6(d). However, by increasing the number of neurons in the hidden layer as in
Eq. (21), the number of folds can be increased beyond 2. This shows that
width-1 NNs with ReLU activation have limited approximation capabilities. This
holds for other activation functions as well. A theoretical result on this
aspect is presented in [13] which is stated below:

######  Theorem 8 (Lu et al., 2017).

Except for a negligible set, all functions
$f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ cannot be approximated by any ReLU
network whose width $W\leq n$.

The above theorem states that width-1 NNs can approximate only a small class
of univariate functions, i.e., the minimum width required for universal
approximation should be greater than 1. Similar results can be found in [38].
This leads to the problem of finding the minimum width for universal
approximation with deep NNs. One of the initial results in this direction
states that $width-n+4$ NNs with ReLU activation function are universal
approximators which is stated below [13]:

######  Theorem 9 (Lu et al., 2017).

For any Lebesgue-integrable function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$
and $\epsilon>0,$ there exists a neural network $f_{\text{NN}}$ of width
$W\leq n+4$ with ReLU activation function which satisfies:

$\int|f(\textbf{x})-f_{\text{NN}}(\textbf{x})|d\textbf{x}<\epsilon.$ (22)

This means that NNs with arbitrary hidden layers and at most $n+4$ number of
neurons per layer can approximate any functions in a Lebesgue integrable space
with sufficient accuracy. Later on, it is shown that the minimum width
required for universal approximation in deep NNs is $n+1$ which leads to the
following theorem [39]:

######  Theorem 10 (Park et al., 2021).

The minimum width required for universal approximation of Lebesgue integrable
functions $\textbf{f}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ is
$max\\{n+1,m\\}.$

Therefore, width-2 NNs with a suitable activation function are also universal
approximators for continuous univariate functions. Similar results of UATs for
width-bounded NNs can be found in [40, 41]. Apart from this, UATs for bounded-
depth and bounded-width NNs are also studied in the literature [42, 43].

##  4 Conclusions and Further reading

This paper discussed the major results and theorems on the approximation
capabilities of NNs. The paper focused on UATs for feedforward NNs or MLPs.
The extension of UATs for other version of NNs such as convolutional NN [44,
45], ResNet [46, 47], recurrent NN [48, 49], transformer [50], etc are
available in the literature. For other topics on NN and ML, the books [4, 7]
can referred to. The Python codes for the examples discussed in this tutorial
are available at GitHub 111https://github.com/MIDHUNTA30/NN-PYTHON.

## References

  * [1] W. McCulloch and W. Pitts, _“A logical calculus of the ideas immanent in nervous activity”,_ The Bulletin of Mathematical Biophysics, vol. 5, pp. 115–133, Dec. 1943.
  * [2] S. Haykin, _“Neural Networks and Learning Machines: Third Edition”,_ Pearson, 2009.
  * [3] E. Kandel, J. Schwartz, T. Jessell, S. Siegelbaum, and A. Hudspeth, _“Principles of Neural Science: Volume 6”,_ McGraw-Hill New York, 2021.
  * [4] G. Strang, _“Linear Algebra and Learning from Data”,_ Wellesley-Cambridge Press, 2019.
  * [5] S. Shwartz and S. David, _“Understanding Machine Learning: From Theory to Algorithms”,_ Cambridge University Press, 2014.
  * [6] I. Goodfellow, Y. Bengio, and A. Courville., _“Deep Learning”,_ MIT Press, 2016.
  * [7] M. Zaki and W. Meira, _“Data Mining and Machine Learning”,_ Cambridge University Press, 2020.
  * [8] D. Tikk, L. Kóczy, and T. Gedeon, _“A survey on universal approximation and its limits in soft computing techniques”,_ International Journal of Approximate Reasoning, vol. 33, pp. 185–203, Jun. 2003.
  * [9] K. Funahashi, _“On the approximate realization of continuous mappings by neural networks”,_ Neural Networks, vol. 2, pp. 183–192, 1989.
  * [10] K. Hornik, M. Stinchcombe, H. White, _“Multilayer Feedforward Networks are Universal Approximators”,_ Neural Networks, vol. 2, pp. 359–366, Mar. 1989.
  * [11] G. Cybenko,,_“Approximation by Superpositions of a Sigmoidal function”,_ Mathematics of Control, Signals, and Systems, vol. 2, pp. 303-314, Dec. 1989.
  * [12] M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, J. Dickstein, _“On the Expressive Power of Deep Neural Networks”,_ $34^{th}$ International Conference on Machine Learning, Sydney, Australia, Aug. 2017.
  * [13] Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang, _“The Expressive Power of Neural Networks: A View from the Width”,_ $31^{st}$ Conference on Neural Information Processing Systems, Long Beach, U.S.A., Dec. 2017.
  * [14] A. Kratsios, _“The Universal Approximation Property Characterization, Construction, Representation, and Existence”,_ Annals of Mathematics and Artificial Intelligence, vol. 89, pp. 435-469, Jan. 2021.
  * [15] A. Kratsios and L. Papon, _“Universal Approximation Theorems for Differentiable Geometric Deep Learning”,_ Journal of Machine Learning Research, vol. 23, pp. 1-73, Jul. 2022.
  * [16] A. Pinkus, _“Approximation theory of the MLP model in neural networks”,_ Acta Numerica, vol. 8, pp. 143 - 195, Jan. 1999.
  * [17] M. Augustine, _“A Tutorial on Neural Networks and Activation Functions”,_ Preprint, http://dx.doi.org/10.13140/RG.2.2.15748.30080, Oct. 2023.
  * [18] A. Apicella, F. Donnarumma, F. Isgro, and R. Prevete, _“A Survey on Modern Trainable Activation functions”,_ Neural Networks, vol. 138, pp. 14-32, Jun. 2021.
  * [19] S. Abbott, _“Understanding Analysis: Second Edition”,_ Springer, 2015.
  * [20] G. Bredon, _“Topology and Geometry”,_ Springer, 1993.
  * [21] B. Taylor, _“Methodus Incrementorum Directa et Inversa (Direct and Reverse Methods of Incrementation)”,_ Typis Pearsonianis prostant apud Gul. Innys ad Insignia Principis in Coemeterio Paulino, pp. p. 21–23, London, 1715.
  * [22] Encyclopedia of Mathematics, _“Taylor formula”,_ EMS Press, 2001.
  * [23] J. Fourier, _“Mémoire sur la propagation de la chaleur dans les corps solides (Treatise on the propagation of heat in solid bodies)”,_ 1807\.
  * [24] A. Pinkus, S. Zafrany, _“Fourier Series and Integral Transforms”,_ Cambridge University Press, 1997.
  * [25] K. Weierstrass, _“Uber die analytische Darstellbarkeit sogenannter willkurlicher Functionen einer reellen Veranderlichen Sitzungsberichteder”,_ Koniglich Preussischen Akademie der Wissenschcaften zu Berlin, 1885.
  * [26] D. Hilbert, _“Über die Gleichung neunten Grades”,_ Mathematische Annalen, vol. 93, pp. 243-250, 1927.
  * [27] A. Kolmogorov, _“On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition”,_ Doklady Akademii Nauk, Russian Academy of Sciences, vol. 114, pp. 953-956, 1959.
  * [28] R. Pascanu, G. Montufar, and Y, Bengio, _“On the number of response regions of deep feedforward networks with piecewise linear activations”,_ ArXiv, Dec. 2013.
  * [29] Y. Cun, _“Modeles connexionistes de l’apprentissage”,_ These de Doctorat, Universite Pierre et Marie Curie, 1987.
  * [30] A. Lapedes and R. Farber, _“How neural networks work”,_ Technical Report. LA-UR-88-418, Los Alamos National Laboratory, 1988.
  * [31] B. Irie and S. Miyake, _“Capabilities of three-layer perceptrons”,_ IEEE Second International Conference on Neural Networks, pp. 641-648, San Diego, U.S.A., 1988.
  * [32] A. Gallant and H. White, _“There exists a neural network that does not make avoidable mistakes”,_ IEEE Second International Conference on Neural Networks, pp. 657-664, San Diego, U.S.A., 1988.
  * [33] P. Dirichlet , _“Sur la convergence des series trigonometriques qui servent à represénter une fonction arbitraire entre des limites donnees”,_ Journal fur die Reine und Angewandte Mathematik, vol. 4, pp. 157-169, 1829.
  * [34] K. Hornik, _“Approximation capabilities of multilayer feedforward networks”,_ Neural Networks, vol. 4, pp. 251–257, 1991.
  * [35] M. Leshno, V. Lin, Y. Vladimir, A. Pinkus, and S. Schocken, _“Multilayer feedforward networks with a nonpolynomial activation function can approximate any function”,_ Neural Networks, vol. 6, pp. 861-867, Jan. 1993.
  * [36] N. Guliyev and V. Ismailov, _“On the approximation by single hidden layer feedforward neural networks with fixed weights”,_ Neural Networks, vol. 98, pp. 296–304, Feb. 2018.
  * [37] J. Park and I. Sandberg, _“Universal Approximation Using Radial-Basis-Function Networks”,_ Neural Computation, vol. 3, pp. 246-257, Jun. 2021.
  * [38] J. Johnson, _“Deep, Skinny Neural Networks are not Universal Approximators”,_ $7^{th}$ International Conference on Learning Representations, New Orleans, U.S.A., May 2019.
  * [39] S. Park, C. Yun, J. Lee, and J. Shin, _“Minimum Width for Universal Approximation”,_ International Conference on Learning Representations, Vienna, Austria, 2021.
  * [40] G. Gripenberg, _“Approximation by neural networks with a bounded number of nodes at each level”,_ Journal of Approximation Theory, vol. 122, pp. 260-266, Jun. 2003.
  * [41] T. Poggio, H. Mhaskar, L. Rosasco, B. Miranda, and Q. Liao, _“Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review”,_ International Journal of Automation and Computing, vol. 14, pp. 503–519, Mar. 2017.
  * [42] V. Maiorov and A. Pinkus, _“Lower bounds for approximation by MLP neural networks”,_ Neurocomputing, vol. 25, pp. 81-91, Apr. 1999.
  * [43] Z. Shen, H. Yang, and S. Zhang, _“Optimal approximation rate of ReLU networks in terms of width and depth”,_ Journal de Mathématiques Pures et Appliquées, vol. 157, pp. 101-135, Jan. 2022.
  * [44] D. Zhou, _“Universality of deep convolutional neural networks”,_ Applied and Computational Harmonic Analysis, vol. 48, pp. 787-794, Mar. 2020.
  * [45] A. Heinecke, J. Ho, W. Hwang, _“Refinement and Universal Approximation via Sparsely Connected ReLU Convolution Nets”,_ IEEE Signal Processing Letters, vol. 27, pp. 1175-1179, Jun. 2020.
  * [46] P. Tabuada and B. Gharesifard, _“Universal Approximation Power of Deep Residual Neural Networks Through the Lens of Control”,_ IEEE Transactions on Automatic Control, vol. 68, pp. 2715–2728, May 2023.
  * [47] H. Lin and S. Jegelka, _“A ResNet with one-neuron hidden layers is a Universal Approximator”,_ $32^{nd}$ Conference on Neural Information Processing Systems, Canada, Dec. 2018.
  * [48] C. Song, G. Hwang, J. Lee, and M. Kang, _“Minimal Width for Universal Property of Deep RNN”,_ Journal of Machine Learning Research, vol. 24, pp. 1 - 41, Feb. 2023.
  * [49] X. Chen, Y. Tao, W. Xu, and S. Yau, _“Recurrent Neural Networks Are Universal Approximators With Stochastic Inputs”,_ IEEE Transactions on Neural Networks and Learning Systems, vol. 34, pp. 7992 - 8006, Oct. 2023.
  * [50] S. Alberti, N. Dern, L. Thesing, and Gitta Kutyniok, _“Sumformer: Universal Approximation for Efficient Transformers”,_ $40^{th}$ International Conference on Machine Learning, Hawaii, U.S.A., Jul. 2023.