# Spatial resolution enhancement in holographic imaging via angular spectrum
expansion

Byung Gyu Chae  Holographic Contents Research Laboratory, Electronics and
Telecommunications Research Institute, 218 Gajeong-ro, Yuseong-gu, Daejeon
34129, Republic of Korea

###### Abstract

Digital holography numerically restores three-dimensional image information
using optically captured diffractive waves. The required bandwidth is larger
than that of hologram pixel at a closer distance in the Fresnel diffraction
regime, which results in the formation of aliased replica patterns in digital
hologram. From the analysis of sampling phenomenon, the replica functions are
revealed to be the components of higher angular spectra of hologram.
Undersampled hologram consists of the moire patterns formed by the modulation
of original function by complex exponential function. There is a one-to-one
correspondence between the replicas in both real and Fourier spaces. The
possibility to acquire high-resolution images over a wide field view is
explored in terms of the expansion process of angular spectrum by using
replicas. Only a low-NA hologram captured over a wide field restores a high-
resolution image when using an optimization algorithm. Numerical simulations
and optical experiments are performed to investigate the proposed scheme.

##  I Introduction

Optical microscopy is a fundamental tool to observe magnified images of a
material through optical lenses. The spatial resolution of an image is limited
by the wave characteristics of light, known as the Abbe diffraction limit,
which depends on the performance of the optical lens that collects the
diffracted light at a higher angular scope [1]. Hence, an upper bound of the
spatial resolution appears when an objective lens with a finite numerical
aperture (NA) is used. To overcome the limited NA of a finite objective,
various approaches have been successfully carried out [2-5], where higher
angular spectra of radiated light from materials are acquired in near-field
and far-field imaging systems. Recently, the necessity of microscopic imaging
technology to observe samples in a wide range together with high precision has
been indicated [6, 7, 8, 9]. This technology would allow the sub-micrometric
structure of materials, particularly those of pathological specimens in
biomedicine, to be inspected effectively.

Digital holography is an interferometric imaging method in which the
diffraction fringe of a diffracted wave is optically captured on an image
sensor, and the image is numerically reconstructed from the captured digital
data based on scalar diffraction theory [10, 11, 12, 13]. The angular spectrum
of a digital hologram can be expanded by measuring multiple holograms using an
oblique incident beam on the specimen, thus resulting in an image resolution
beyond the diffraction limit of existing optical systems. Synthetic-aperture
Fourier holographic microscopy is a representative tool for obtaining high-
resolution images within a wide field-of-view [3, 6, 7, 14, 15, 16]. Another
diffraction imaging technology, i.e., Fourier ptychographic microscopy is used
to extend the Fourier space [17, 18], in which the complex amplitude is
restored from the measured diffraction intensity via phase retrieval process.
Although these imaging approaches effectively capture high-resolution image
information in a wide field using only a low-NA objective, there still remains
a system complexity such as the multiple measurements or post data processing.

Recently, there have been several studies on high-NA digital hologram [19, 20,
21]. High-NA hologram suffers from the aliased replica fringe when captured at
a closer distance. In analysis of sampling theorem, the original image is well
restored despite severe aliased errors. Because the aliasing errors arise from
the required bandwidth larger than that of hologram pixel, it is not certain
whether or not there is a constraint on spatial frequency in the image
restoration.

Holographic imaging is a representative linear system [10]. Thus, digital
hologram of a finite-sized object can be analyzed by a point hologram. Optical
kernel functions directly correspond to elementary functions for point object
in real and Fourier spaces. When the functions are sampled in a digitized
space, they can be represented as the summation of shifted form of original
functions because of complex amplitude with a quadratic phase term [22, 23].

In this study, the relationship between aliased replica functions in both real
and Fourier spaces is mathematically or graphically investigated. Firstly, it
is found out that the replication patterns are moire interfernce fringes,
i.e., the modulation of original function by complex exponential function,
which indicates that they are the components of higher angular spectra of
hologram. A one-to-one correspondence exists between them in both real and
Fourier spaces. A method for extending angular spectrum of hologram is studied
using displacement of replication patterns in the space. This technique
enables high-resolution images to be restored, which reduces the complexity of
imaging system. Numerical simulation is performed to investigate the proposed
scheme, and the results of the optical experiment are presented. Finally, the
possibility of achieving the super-resolution beyond the Abbe diffraction
limit using this strategy is discussed.

##  II Enhancement in spatial resolution of holographic image by angular
spectrum expansion

###  II.1 Abbe theory of spatial resolution in digital holography

Based on scalar diffraction theory [10], the diffractive wave field in the
near-field region is characterized by optical kernel functions, i.e., an
impulse response function in real space,

$h(\rm{\bf{X}})=\it{}h_{0}\exp\left(i\frac{\pi}{\lambda
z}\rm{\bf{XX^{T}}}\right),$ (1)

and a transfer function in the Fourier space,

$H(\rm{\bf{U}})=\it{}H_{0}\exp\left(-i\pi\lambda z\rm{\bf{UU^{T}}}\right),$
(2)

where $h_{0}=\frac{e^{ikz}}{i\lambda z}$ and $H_{0}=e^{ikz}$. $\rm{\bf{X}}$
and $\rm{\bf{X^{{}^{\prime}}}}$ represent two-dimensional (2D) vectors in
$(x,y)$ and $(x^{\prime},y^{\prime})$ coordinates, respectively; and $k$ is
the wavenumber of $\frac{2\pi}{\lambda}$ with wavelength $\lambda$. The
convolution notation, $g(\bf{X})=\it{O}(\bf{X})*\it{h}(\bf{X})$ describes the
diffractive wave field $g(\bf{X})$ that propagates from the object field
$O(\bf{X^{\prime}})$. For a point object $\delta(\bf{X^{{}^{\prime}}})$, the
spherical wave diverges forward in a free space under the constraint of the
angular spectrum, $k\geq 2\pi\sqrt{\bf{UU^{T}}}$, where $\bf{U}$=[$u$ $v$] is
the spatial frequency vector. The optical imaging performance depends on how
much of higher components of angular spectrum in the diverging spherical
wavelets is gathered through an optical lens.

Digital holography optically captures the wavefront of a diffracted wave from
a material in digitized space, which is called the digital hologram. Figure
1(a) shows the geometric structure of a Fresnel holographic imaging system.
The captured hologram data is band-limited in the spatial and frequency
domains due to its finite aperture. Considering the spatial frequency vector
of $\frac{\bf{X}}{\lambda z}$ in the Fresnel diffraction formula, the space
bandwidth $B_{w}$ in a digital hologram with $N$$\times$$N$ pixels of
$\it{\Delta}$ pixel interval is represented as

$B_{w}=\frac{N\it{\Delta}}{\lambda z}.$ (3)

According to the Abbe diffraction theory, the bandwidth value determines the
resolution limit $R_{\rm{lim}}$ of restored image [21, 24, 25]:

$R_{\rm{lim}}=\frac{\lambda z}{N\it{\Delta}}=\frac{\lambda}{2\rm{NA}},$ (4)

where $\rm{NA}=\it{n}\sin\theta=\frac{N\it{\Delta}}{2z}$, and $n$ is the
refractive index of free space. The hologram aperture function is an optical
transfer function that defines the extent to which it collects the spatial
frequency components of the diffracted wave.

###  II.2 Replication properties of undersampled digital hologram in a
digitized space

When the hologram is acquired at a closer distance, the bandwidth is larger
than that of hologram pixel, which resluts in the aliasing errors in digital
hologram. In this case, it is important to establish that the image resolution
would be finer than the hologram pixel value because of the restriction in
spatial frequency, as depicted in Fig. 1.

Holographic imaging is a linear system, and thus digital hologram of a finite
object can be interpretable in terms of only an elementary function of point
object [10]. The response function in real space becomes a hologram of point
object from the relation, $g(\bf{X})=\delta(\bf{X})*\it{h}(\bf{X})$.

Let us investigate the sampling phenomenon of response function in digitized
space. The Fourier transform of the response function sampled by the sampling
matrix, P in real space is expressed as

$\textbf{\emph{FT}}\left\\{\sum_{\bf{n}}h(\bf{nP})\delta(\bf{X}-\bf{nP})\right\\}=\frac{\rm{1}}{|\rm{det}\,\bf{P}|}\sum_{\bf{m}}H(\bf{U}-\bf{mQ}),$
(5)

where $\textbf{\emph{FT}}\\{\\}$ indicates the Fourier transform; Q equals to
$\textbf{P}^{-\rm{T}}$; and n and m are integers in 2D notation. Taking the
inverse Fourier transform into the right-hand side of Eq. (5) yields the
modulated form of the function, which can be rewritten in the shifted form of
a quadratic phase functions [22, 23],

$\sum_{\bf{n}}h(\bf{nP})\delta(\bf{X}-\bf{nP})=\frac{\rm{1}}{|\rm{det}\,\bf{P}|}\sum_{\bf{n}}\it{c_{\bf{n}}h}(\bf{X}-\lambda\it{z}\bf{nQ}).$
(6)

The respective order is multiplied by the constant value,
$c_{n}=e^{\frac{-i\pi\lambda zn^{2}}{\it{\Delta^{\rm{2}}}}}$. The critical
distance $z_{c}$ is defined when the repetition period $\frac{\lambda
z}{\it{\Delta}}$ of function is the same as a field size $N\it{\Delta}$:

$z_{c}=\frac{N\it{\Delta}^{2}}{\lambda}.$ (7)

There appears no aliased fringe at above $z_{c}$.

Similarly, when the transfer function is sampled, the same form as Eq. (6) can
be obtained as follows:

$\sum_{\bf{m}}H(\bf{mP_{0}})\delta(\bf{U}-\bf{mP_{0}})=\frac{\rm{1}}{|\rm{det}\,\bf{P_{0}}|}\sum_{\bf{m}}\it{c_{\bf{m}}H}\left(\bf{U}-\frac{mQ_{0}}{\lambda\it{z}}\right),$
(8)

where $\textbf{P}_{0}$ is the sampling matrix in the Fourier space and
$\textbf{Q}_{0}=\textbf{P}_{0}^{-\rm{T}}$. The transfer function is an angular
spectrum of response function.

The undersampling properties of response function could be analyzed on the
basis of Eq. (6). Principally, the sampling characteristics depend on the
parameters $(z,\it{\Delta})$. As the sampling interval $\it{\Delta}$ increases
or the distance $z$ is smaller compared to a critical distance, the aliased
replica patterns are generated. The shifted quadratic phase function in Eq.
(6) appears as the modulation of original function by complex exponential
function:

$h(\bf{X}-\lambda\it{z}\bf{nQ})=\it{c_{o}h}(\bf{X})\exp\left(-\it{i}\pi\rm{\bf{X}}\bf{nQ}\right).$
(9)

The modulating term, i.e., complex exponential function is constant at
specifications $(z_{c},\it{\Delta})$, where a shifted term becomes original
function itself, and thus no information is extracted.

Figure 2 depicts the graphical analysis for the undersampled digital hologram
of point object. The point hologram in 2D space consisting of 256$\times$256
pixels with a 8-$\rm{\mu}$m pixel pitch was synthesized using a plane wave
with a wavelength of 532 nm. The hologram was made at a distance of a half of
$z_{c}$, which forms four replica Fresnel zones. In this case, the complex
exponential function in replica function,
$e^{\frac{-i\pi\rm{\bf{X}}}{\it{\Delta}}}$ is the sinusoidal wave with a
period of ${2\it{\Delta}}$, in Fig. 2(b). The Fourier-transformed data is the
shifted high-frequency component. All the shifted replica functions become the
higher spatial-frequency components corresponding to those of original
function. The spatial frequency, $\frac{\rm{\bf{X}}}{\lambda z}$ of the
concentric primary pattern linearly increases with a spatial distance. It is
intersesting that although the replica zones seem to be the moire fringes with
a low frequency in appearance, the aliased fringes should be higher spectrum
components according to a spatial distance. Spectrum distributions considering
a finite pixel size are drawn briefly in Fig. 2(c).

Furthermore, a curve shape of replicas by all the sampling periods are
embedded in the form of moire fringe in the primary function, as illustrated
in Fig. 3. The replica functions are formed at a reduced period of
$\frac{\lambda z}{\it{s\Delta}}$ when the function is undersampled by $s$
multiples of $\it{\Delta}$. On the other hand, the number of replicas, $N$ are
larger by a demagnification process, i.e., zoom-out process. This indicates
that the self-similar patterns are formed by means of an extrapolation
operation with a scale parameter of $\varepsilon$. Theoretically, this
phenomenon could be explained as a result of negative fractal dimension [26],
$D=\lim_{\varepsilon\to 0}\frac{\log N(\varepsilon)}{\log(1/\varepsilon)}=-2$.

Figure 4 describes this phenomenon in the Wigner space. The point hologram
made at a distance of one-third of $z_{c}$ forms nine replica zones. The
Fresnel transform is represented as a shearing of the Wigner distribution
function, $W(x,u)=W(x-\lambda zu,u)$. The overlapping of high-order spectra is
inevitable at a distance below $z_{c}$, which causes aliasing fringes. The
Wigner distribution is rotated by $90^{\circ}$ via the Fourier transform [27],
which also represents the replications of angular spectrum. This shows clearly
a one-to-one correspondance with both replica functions in real and Fourier
spaces. The aliased replica components of angular spectrum correspond to the
replication elements of digital hologram.

###  II.3 Method for spatial-resolution enhancement in image retrieval process

The replica functions in the undersampled hologram play a role in higher
angular spectrums of primary function. In the image retrieval process, the
whole area of the digital hologram should be an aperture that satisfies the
resolution relationship of Eq. (4). The hologram aperture extends to the whole
area without a confinement of primary zone.

Obviously, the hologram aperture size could be extended by spatially
replicating the primary function once the primary function is known. In this
strategy, the primary functions both real and Fourier spaces can be
applicable. This expands the angular spectrum of hologram, and thus results in
the retrieval of high-resolution image. Mathematically, this concept would be
possible because the replica function is the same as the original one except
for the multiplication constant in Eqs. (6) and (8). However, in the optically
captured hologram, the addition of external noises due to a measurement
environment is inevitable, where it is hard to extract exact information about
primary hologram function.

Recently, the research to expand the hologram aperture was performed by self-
extraploation technique using a phase retrieval algorithm [13]. This method
could be used for finding replica patterns from the primary function. Here,
the computation performance would be enhanced because the replica patterns
have a self-similarity, and the computational capacity is reduced with
comparison to the non-aliased data.

As another approach, the high-resolution image is retrieved from the
undersampled hologram optically captured while keeping high aperture size. In
this case, the image can be effectively restored by replicating the angular
spectrum in the Fourier space during the iterative optimization algorithm. The
iterative algorithm uses the angular spectrum method:

$O(\textbf{X}^{\prime})=\textbf{\emph{IFT}}\left[\textbf{\emph{FT}}\\{\it{g}(\bf{X})\\}\times\it{H^{-\rm{1}}}(\textbf{U})\right].$
(10)

Unlike the description of Fig. 1, the pixels in both planes have the same
value irrespective of the distance because of the sampling relation,
$\it{\Delta}=\frac{\rm{1}}{N\it{\Delta_{u}}}=\it{\Delta^{\prime}}$, where an
allowable image size is consistent with the hologram size. Undersampling by
$m$-multiple pixels creates $m$-multiple replicas of the primary function. The
$m$-fold expansion of angular spectrum during the reconstruction process helps
to restore the high-resolution image.

Finally, the resolution performance would be enhanced when high-order
diffractions from a material are used. High-order diffractions directly relate
to the higher replica terms in Eq. (6). Optically, these terms can be measured
through a grating device located in front of sample. It was reported that the
captured hologram to collect the highly diffracted beams on centered region
improves the image resolution [28]. Here, there is no need for a beam
collecting process. Figure 5 is the schematic diagram for above proposed
methods.

##  III Numerical analysis of resolution increment of holographic image

Figure 6 illustrates the reconstruction properties of image by using an
extraction process of replica patterns. The initial complex-valued hologram
for the object with 512$\times$512 pixels was synthesized by the Fresnel
diffraction formula, and then cropped by a factor of 2 to obtain the low-NA
hologram with 256$\times$256 pixels, in Fig. 6(a). The real-valued object
consisting of several points with 2-$\mu$m pixel was used, and the initial
hologram made at a half of $z_{c}$ has pixel resolution of 4 $\mu$m according
to a sampling relation between hologram and object planes,
$\it{\Delta^{\prime}}=\frac{\lambda z}{N\it{\Delta}}$.

The replica patterns were extracted through a phase retrieval algorithm based
on the Gerchberg-Saxton algorithm. The absolute value of restored image was
used during the repetition procedure, which is sufficient for a constraint.
Original replication patterns are successfully found at a large repetition,
which results in a retrieval of original image. The one-dimensional profile of
hologram in Fig. 6(e) clearly shows that the extrapolated data well matches
with original one. As not displayed here, it was confirmed that the hologram
made at a fourth of $z_{c}$ retrieves original replication patterns.

Figure 7 is the simulation results by using the undersampled hologram. Initial
high-NA hologram for the USAF resolution target was calculated by the inverse
process of Eq. (10), Both object and hologram spaces consist of 512$\times$512
pixels of a 4-$\mu$m pixel, and undersampled hologram prepared by 2-fold
downsampling has 256$\times$256 pixels of an 8-$\mu$m pixel. Undersampling is
performed by periodically placing sub-pixels of the same size as the original
pixels. This type of hologram can be optically measured using 2D grid mask
attached to the image sensor. The hologram data acquired through the $m$-fold
sub-pixel mask enables $m$-fold expansion of the angular spectrum.

Figure 7(a) is the restored image through the zero-padding technology, where
the retrieval of initial resolution is unavailable. However, as illustrated in
Fig. 7(d), the image with the initial resolution of 4 $\mu$m is reconstructed
from two-multiple expansion of the angular spectrum, which becomes a strong
evidence that the replicated angular spectrum functions as the higher
spectrum. Both magnified images show a clear discrimination. To remove the
interference of high-order images, the extended angular spectrum was
multiplied by a sinc function, $\rm{sinc}(\pi\it{\Delta}\textbf{U})$. There
are still remnents of high-order noises and it shows relatively a poor image
quality.

The iterative optimization algorithm based on the gradient descent method with
a total variational regularization TV was used to optimize the restored image
[29, 30]:

$\arg\min_{O}\frac{1}{2}\parallel\textbf{\emph{Fr} \rm{(} \it{O} \rm{)} -
\it{g}}\parallel^{2}+\gamma\textbf{TV}(O).$ (11)

The objective function consists of the least squares together with the TV
term, and $\gamma$ is a regularized parameter. Fr() denotes the Fresnel
transform of Eq. (10). The reconstructed image $O_{k}$ at respective iteration
is updated by minimizing the loss function, i.e., the squared error between
the original hologram $g_{0}$ and estimated one $g_{k}$. The angular spectrum
of the errors was expanded via its repetition process of $m$ multiples,
resulting in considering the high frequency components. The transfer function
was also extended to $m$-multiple frequency space and the estimated hologram
was downsampled to match original resolution. The convergence rate for
optimization was arbitrarily controlled by changing the external parameters
such as a learning rate $\alpha$ or regularization factor $\gamma$.

The iterative algorithm restores a high-quality image comparable to original
image, when using the 4-fold downsampled hologram for the Shepp-Logan phantom,
in Fig. 8. The PSNR value of the reproduced image is estimated to be
approximately 73 dB. As not shown here, we also confirmed that two-fold or
eight-fold undersampled hologram well restores the image. The sinc function
was used only in reconstruction process using the eight-fold undersampled
hologram, and it was not necessarily required for this algorithm. The spatial
resolution of the reconstructed image is recovered to the original value in
accordance to the expansion extent of the angular spectrum. Figure 8(f) is the
cross-sectional profile of the restored image after optimization. The initial
image with a spatial resolution of 4 $\mu$m is exactly recoverable by using an
iterative optimization algorithm.

##  IV Optical experiments and discussion

Figure 9(a) illustrates an off-axis holographic technique to acquire an
optical hologram for the USAF resolution target. A 532-nm green laser
(LCX-532, Oxxius) was used as a light source. A hologram with 2160$\times$2160
pixels was captured on an image sensor (GS3-U3-89S6M-C, FLIR) at a distance of
86 mm. The captured hologram has a pixel pitch of 3.45 $\mu$m. The first-order
region of the Fourier-transformed data in Fig. 9(c) was cropped to avoid non-
diffraction beams and twin terms, and then moved to the center. The inverse
Fourier transform generated the final complex-amplitude hologram.

The reconstruction of the optical hologram is shown in Fig. 10. The low-
resolution hologram was constructed via a 4-fold undersampling procedure,
which is analogous to a hologram achieved using a 4-fold sub-pixel mask. The
inferior image as compared with the original image was restored from the low-
resolution hologram. However, the expansion of angular spectrum in the
iterative optimization algorithm recovers original resolution, as confirmed in
the magnified pictures in Fig. 10(d).

A digital hologram is a captured diffraction wave. As mentioned previously,
once the primary hologram function is known, a high-resolution image can be
completely recoverable in the numerical reconstruction process. It is
desirable that the primary function is obtained using a sub-pixel mask, where
the enhancement in the spatial resolution is shown to be limited by its
undersampling extent. Principally, as referred to Fig. 3(a), this function
exhibits a perfect form with no aliasing errors at a specified sampling rate.
Therefore, the pixel value of whole area does not differ significantly from
the quantity in the sub-pixel area, thus indicating that the primary function
can be acquired without using a sub-pixel mask. In the numerical study of
Figs. 7 and 8, we observed that even a low-resolution hologram prepared by an
averaging operation of sub-pixel values clearly restores the image. On the
other hand, to relaize this phenomenon in optical hologram, the precise
experiment to capture the hologram well defining the NA should be implemented.

The imaging technology applied in this study overcomes the resolution limit of
finite imaging systems, thus allowing high-resolution image to be restored
over a wide field-of-view. The spatial resolution and field view of the
measured image are determined based on the geometric structure of the imaging
system. The use of an appropriate objective lens increases an imaging
performance.

According to the Abbe resolution criterion, the maximum spatial resolution
reaches a half of wavelength. The optical transfer function used in the
Fresnel diffraction regime can be extended to the form,
$\exp(iz\sqrt{k^{2}-4\pi^{2}\bf{UU^{T}}})$ in the Rayleigh-Sommerfeld region.
Although the extension of angular spectra through their replication is
possible, the replica patterns in the high-NA appear to be distorted.
Furthermore, it is not certain that this scheme is applicable to the
evanescent field region. When these two points are theoretically
comprehensible, a super-resolution imaging using this strategy would be
meaningful.

##  V Conclusions

The replication patterns of the digital hologram and its Fourier-transformed
form in both real and Fourier spaces correlate with higher angular spectra.
This phenomenon is found from the ananlysis of the sampling properties of
hologram in a digitized space. The effective method for obtaining high-
resolution images over a wide field view was developed to recover the
complexity of the optical imaging system. The hologram numerical aperture can
be increased only by expanding the angular spectrum, resulting in the
resolution enhancement of restored image. Several methods of this holographic
imaging were investigated through numerical simulations and optical
experiments. An optical imaging system using a relatively low-NA objective
lens can avoid disadvantages such as a narrow field-of-view and small depth-
of-field.

This work was supported by Institute for Information & Communications
Technology Promotion (IITP) grant funded by the Korea government (MSIP)
(2021-0-00745)

The authors declare no conflicts of interest.

## References

  * [1] M. Born and E. Wolf, Principles of Optics, 7th ed. (Cambridge University, 1999).
  * [2] E. Betzig and J. K. Trautman, “Near-field optics: microscopy, spectroscopy, and surface modification beyond the diffraction limit,” Science 257, 189-195 (1992).
  * [3] R. Heintzmann and C. Cremer, “Laterally modulated excitation microscopy: improvement of resolution by using a diffraction grating,” Proc. SPIE 3568, 185-196 (1998).
  * [4] S. A. Alexandrov, T. R. Hillman, T. Gutzler, and D. D. Sampson, “Synthetic aperture Fourier holographic optical microscopy,” Phys. Rev. Lett. 97, 168102 (2007).
  * [5] H. M. L. Faulkner and J. M. Rodenburg, “Movable aperture lensless transmission microscopy: A novel phase retrieval algorithm,” Phys. Rev. Lett. 93, 023903 (2004).
  * [6] C. J. Schwarz, Y. Kuznetsova, and S. R. J. Brueck, “Imaging interferometric microscopy,” Opt. Lett. 28, 1424-1426 (2003).
  * [7] T. Gutzler, T. R. Hillman, S. A. Alexandrov, and D. D. Sampson, “Coherent aperture-synthesis, wide-field, high-resolution holographic microscopy of biological tissue,” Opt. Lett. 35, 1136-1138 (2010).
  * [8] W. Bishara, T. W. Su, A. F. Coskun, and A. Ozcan, “Lensfree on-chip microscopy over a wide field-of-view using pixel super-resolution,” Opt. Express 18, 11181-11191 (2010).
  * [9] Y. Baek, K. Lee, S. Shin, and Y. Park, “Kramers–Kronig holographic imaging for high-space-bandwidth product,” Optica 6, 2334-2536 (2019).
  * [10] J. W. Goodman, Introdction to Fourier Optics (McGraw-Hill, 1996).
  * [11] U. Schnars and W. P. O. Jüpter, “Direct recording of holograms by a CCD target and numerical reconstruction,” Appl. Opt. 33, 179-181 (1994).
  * [12] E. Cuche, F. Bevilacqua, and C. Depeursinge, “Digital holography for quantitative phase-contrast imaging,” Opt. Lett. 24, 291-293 (1999).
  * [13] T. Latychevskaia and H. W. Fink, “Resolution enhancement in digital holography by self-extrapolation of holograms,” Opt. Express, 21, 7726-7733 (2013).
  * [14] J. H. Massig, “Digital off-axis holography with a synthetic aperture,” Opt. Lett. 27, 2179-2181 (2002).
  * [15] V. Mico, Z. Zalevsky, P. García-Martínez, and J. García, “Synthetic aperture superresolution with multiple off-axis holograms,” J. Opt. Soc. Am A 23, 3162-3170 (2006).
  * [16] T. R. Hillman, T. Gutzler, S. A. Alexandrov, and D. D. Sampson, “High-resolution, wide-field object reconstruction with synthetic aperture Fourier holographic optical microscopy,” Opt. Express 17, 7873-7892 (2009).
  * [17] G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution Fourier ptychographic microscopy,” Nat. Photonics 7, 739-745 (2013).
  * [18] B. Lee, J. Hong, D. Yoo, J. Cho, Y. Jeong, S. Moon, and B. Lee, “Single-shot phase retrieval via Fourier ptychographic microscopy,” Optica 5, 976-983 (2018).
  * [19] L. Onural, “Sampling of the diffraction field,” Appl. Opt. 39, 5929-5935 (2000).
  * [20] A. Stern and B. Javidi, “Analysis of practical sampling and reconstruction from Fresnel fields,” Opt. Eng. 43, 239-250 (2004).
  * [21] B. G. Chae, “Analysis on image recovery for on-axis digital Fresnel hologram with aliased fringe generated from self-similarity of point spread function,” Opt. Commun. 466, 125609 (2020).
  * [22] L. Onural, “Some mathematical properties of the uniformly sampled quadratic phase function and associated issues in Fresnel diffraction simulations,” Opt. Eng. 43, 2557-2563 (2004).
  * [23] B. G. Chae, “Wide viewing-angle holographic display based on enhanced-NA Fresnel hologram,” Opt. Express 29, 38221-38236 (2021).
  * [24] P. Picart and J. Leval, “General theoretical formulation of image formation in digital Fresnel holography,” J. Opt. Soc. Am. A 25, 1744-1761 (2008).
  * [25] D. P. Kelly, B. M. Hennelly, N. Pandey, T. J. Naughton, and W. T. Rhodes, “Resolution limits in practical digital holographic systems,” Opt. Eng. 48, 95801 (2009).
  * [26] B. B. Mandelbrot, The Fractal Geometry of Nature (Freeman, 1982).
  * [27] A. W. Lohmann, R. G. Dorsch, D. Mendlovic, Z. Zalevsky, and C. Ferreira, J. Opt. Soc. Am. A 13, 470 (1996).
  * [28] M. Paturzo, F. Merola, S. Grilli, S. De Nicola, A. Finizio, and P. Ferraro, “Super-resolution in digital holography by a two-dimensional dynamic phase grating,” Opt. Express 16, 17107-17118 (2008).
  * [29] B. G. Chae and S. Lee, “Sparse-view CT image recovery using two-step iterative shrinkage-thresholding algorithm,” ETRI. J. 37, 1251-1258 (2015).
  * [30] E. J. Candes, J. Romberg, and T. Tao, “Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information,” IEEE Trans. Inf. Theory, 52, 489-509(2006).

Figure 1: (a) Schematic illustrations of the Fresnel holographic imaging
system. Point image is restored from digital hologram located at one-half of
$z_{c}$. Green box denotes allowable image space defined by diffraction of
hologram pixel. (b) Change in spatial resolution as a function of
reconstruction distance. Values for digital hologram consisting of
256$\times$256 pixels with a 8-$\mu$m pixel pitch are drawn. Figure 2:
Graphical analysis for the undersampled digital hologram of point object. (a)
Real-valued hologram synthesized at a distance of a half of $z_{c}$. One-
dimensional quadratic phase function along the lateral line at the center is
overlapped on the Fresnel zones. (b) Original function and complex exponential
function corresponding to $+1^{\rm{st}}$-order replica zone. (c) Spectrum
distributions considering a finite pixel size. Figure 3: Sampling phenomena of
(a) response function and (b) transfer function of a point hologram in one-
dimensional space. Black, red, and green markers represent original transfer
function, and undersampled functions by factors of 2 and 4, respectively.
Boxes indicate the corresponding primary functions. $u_{c}$ is the maximum
spatial frequency with respect to bandwidth of original function. Figure 4:
Aliasing effects of undersampled hologram in the Wigner space. (a) Digital
hologram for a point object synthesized at a distance of one-third of $z_{c}$.
(b) Wigner distribution description of digital hologram. Point image is
focused from inclined line designating spherical fringe of hologram. (c)
3-fold expanded angular spectrum corresponding to digital hologram. (d) Wigner
distribution description of angular spectrum. Figure 5: Methods for increasing
image resolution by expanding angular spectrum. (a) Expansion of hologram
aperture size by directly searching replica patterns. (b) Image retrieval
through the expansion process of angular spectrum for acquired low-resolution
hologram. (c) Image retrieval by using captured high-order diffractions.
Figure 6: Reconstruction properties of image by using an extraction process of
replica patterns. (a) Low-NA hologram with 256$\times$256 pixels of an
8-$\mu$m pixel interval for several point objects. (b) Hologram extrapolated
by using a phase retrieval algorithm. Reconstructed images for (c) low-NA
hologram (d) extrapolated hologram. (e) One-dimensional profiles of original
and extrapolated holograms. Figure 7: Simulation results for image
reconstruction of undersampled hologram. (a) Zero-padded angular spectrum into
512$\times$512 pixels in the logarithmic scale and (b) restored USAF target
image through zero-padding technology. Right picture is the magnified image in
red box. (c) 2$\times$2 expanded angular spectra by placing replica in an
array. Red box denotes the zeroth order value. (d) Directly restored image via
two-multiple expansion of angular spectrum in Fourier space. Figure 8: Image
reconstruction of undersampled hologram by using iterative opimization
algorithm. (a) Low-resolution hologram with 128$\times$128 pixels of a
16-$\mu$m pixel interval for the Shepp-Logan phantom. (b) Directly restored
image via four-multiple expansion of angular spectrum. (c) 4$\times$4 expanded
angular spectra in the logarithmic scale after iterative optimization. (d)
Restored image via the iterative optimization algorithm including 4-fold
expansion of angular spectrum. Cross-sectional profiles of reconstructed
images (e) without and (f) with iterative optimization algorithm. Figure 9:
(a) Off-axis holographic technique to acquire an optical hologram. (b)
Captured hologram for the USAF resolution target and (c) its Fourier-
transformed data. Red box indicates the first-order region. $\theta$ is an
off-axis angle. Figure 10: Image reconstruction phenomenon of optical
hologram. (a) Low-resolution optical hologram for USAF resolution target. (b)
Image restored via the iterative optimization algorithm including the
expansion process of angular spectrum. Magnified images and profiles of pixel
values for the restored images (c) without and (d) with the spectrum expansion
process.


# Events Beyond ACE: Curated Training for Events

Ryan Gabbard  Information Sciences Institute, University of Southern
California
gabbard<EMAIL_ADDRESS>Jay DeYoung  Raytheon BBN Technologies
<EMAIL_ADDRESS>Marjorie Freedman  Information Sciences Institute,
University of Southern California
gabbard<EMAIL_ADDRESS>
###### Abstract

We explore a human-driven approach to annotation, _curated training_ (ct), in
which annotation is framed as teaching the system by using interactive search
to identify informative snippets of text to annotate, unlike traditional
approaches which either annotate preselected text or use active learning. A
trained annotator performed 80 hours of ct for the thirty event types of the
nist tac kbp Event Argument Extraction evaluation. Combining this annotation
with ace results in a 6% reduction in error and the learning curve of ct
plateaus more slowly than for full-document annotation. 3 NLP researchers
performed ct for one event type and showed much sharper learning curves with
all three exceeding ace performance in less than ninety minutes, suggesting
that ct can provide further benefits when the annotator deeply understands the
system.

Events Beyond ACE: Curated Training for Events

Ryan Gabbard, Jay DeYoung, Marjorie Freedman††thanks: Work was performed when
Gabbard and Freedman were at BBN
---
USC Information Sciences Insitute, Raytheon BBN Technologies
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

##  1\. Introduction

Identifying events from a given ontology in text and locating their arguments
is an especially challenging task because events vary widely in their textual
realizations and their arguments are often spread across multiple clauses or
sentences. Most event research has been in the context of the 2005 NIST
Automatic Content Extraction (ace) sentence-level event mention task [Walker
et al., 2006], which also provides the standard corpus. Recently, tac kbp has
introduced document-level event argument extraction shared tasks for 2014 and
2015 (kbp ea).

Progress on events since ACE has been limited. Most subsequent work has tried
improve performance through the use of more complex inference [Li et al.,
2013], by transductively drawing on outside sources of information, or both
[Ji and Grishman, 2008, Hong et al., 2011]. Such approaches have produced
modest reductions in error over a pipeline of simple classifiers trained on
ACE.

In our efforts to improve on the kbp ea 2014 systems, we were stymied by a
lack of data, especially for rarer event types. Ten of the 33 event types have
fewer than 25 training examples in ace, and even for more frequent events,
many trigger words and classes of arguments occurred only once. Furthermore,
the 2015 task would include new argument types. These problems motivated the
following question: (a) are we at a plateau in the performance vs. annotation
time curve? (b) is there an viable alternative to full-document annotation,
especially for rarer event types? (c) for novel event types or languages, how
quickly can a useful event model be trained?

In traditional annotation, a static corpus selected to be rich in the target
event types is annotated. Active learning augments existing training data by
having a human oracle annotate system queries (or features [Settles, 2011]).
We explored a novel form of annotation, _curated training_ (ct), in which
_teachers_ (annotators) actively seek out informative training examples.

##  2\. Curated Training

In ct the teacher created a prioritized indicator list of words and phrases
which could indicate a target event’s presence. Given a tool with a search
box, a document list, and a document text pane, teachers searched111over
Gigaword 5 [Parker et al., 2011] using Indri [Strohman et al., 2005] for
indicators in priority order and annotated ten documents each. On loading a
document, they used their browser’s search to locate a single sentence
containing the indicator.

If the sentence mentioned multiple instances of the target event or was
unclear, it was skipped. If it contained no mention of the event, they marked
it negative.222Negated, future, and hypothetical events were all considered
mentions of an event, not negatives Otherwise, they (a) marked the sentence as
event-present; (b) applied the anchor annotation to the tokens333if there were
no anchors, the document was skipped whose presence makes the presence of the
event likely; (c) marked each argument span within the selected sentence; and
(d) marked any other spans they thought might be ‘educational’ as interesting.
(d) was also done for negative sentences.

Teachers were permitted to annotate extra documents if an indicator seemed
ambiguous. They looked very briefly (2-3 seconds) in the context of selected
sentences to see if there were additional informative instances to annotate.
If any non-indicator anchor was marked, it was added to the indicator list
with high priority. The process was repeated for four hours or until the
teacher felt additional ct would not be useful.444See curves in Figure 2. In
many cases annotation appears to terminate early because annotators had no way
of tracking when they hit exactly four hours.

###  2.1. Data Gathered

We recruited three teachers without NLP backgrounds but with annotation
experience. We consider here only the teacher (a) who completed all event
types in time for assessment. Teacher a averaged seven minutes brainstorming
indicators and produced 6,205 event presence, 5137 negative, and 13,459
argument annotations. Every teacher action was time-stamped. For analysis, we
updated the timestamps to remove breaks longer than two minutes.555Annotators
were never to spend more than 2-3 seconds on any decision

Since the ct was stored as character offsets, we aligned it to parses to get
ace-style event mentions. For training our argument attachment models, we omit
any event mentions where any annotation failed to project. Projecting Teacher
a’s data produced 5792 event mentions for trigger training, 5221 for argument
training, and 4,954 negatives.666c.f. roughly 5,300 event mentions in ace

##  3\. Evaluation

Our target evaluation task is kbp-ea [NIST, 2014] which requires mapping a
document to a set of $(t,r,e,m)$ tuples indicating that an entity $e$ plays
the role $r$ in an event of type $t$ with realis $m$. Scoring is F1 over these
tuples.777The details of the scoring are given in [NIST, 2014]. We evalute
over the the 2014 newswire evaluation corpus [Joe Ellis and Strassel, 2015]
using the scorer888https://github.com/isi-nlp/tac-kbp-eal on the evaluation
key augmented with assessments by Teacher a of responses from our system not
found therein.999 The evaluation answer key had assessments of all 2014 system
response and those of an an LDC annotator operating under significant time-
pressure (thirty minutes per document) To focus on event detection and
argument attachment, we enabled the neutralizeRealis and
attemptToNeutralizeCoref scorer options.

###  3.1. Baseline

The highest-performing system in kbp ea 2014, bbn1, ran a pipeline of four
log-linear classifiers (trigger detection, argument attachment, genericity
assignment, and a trigger-less argument model) in a high-recall mode which
output all event mentions and arguments scoring above 10% probability. This
output was fed into a series of inference rules and a score was computed based
on the sub-model scores and the inference rules applied [Chan et al., 2014].

We used this evaluation system for the experiments in this paper with two
changes. First, bbn1 used a multi-class model for trigger detection, while we
use one binary model per event type because with ct each type has a different
set of negative examples. Second, we omitted the ‘trigger-less’ argument
classifier for simplicity. This version, baseline, lags bbn1’s performance by
0.8 F1 but outperforms all other 2014 evaluation systems by a large margin.

To compare against full document annotation, we needed to estimate how long
the event-only portion of ace annotation took.101010Excluding coreference,
etc. The LDC111111personal communication ventured a rough estimate of 1500
words per hour (about twenty minutes per ace document). The LDC human
annotator in kbp-ea 2014 was allocated thirty minutes per document [Freedman
and Gabbard, 2014]. We use the former estimate. To estimate performance with a
fraction of ace, we used the first $n$% documents as needed.

##  4\. Analysis

Figure 1: Performance vs. mean time per event

In aggregate ct’s performance closely tracked ace for small amounts of mean
annotation time per event (Figure 1). However, the performance of ct plateaus
more slowly than ACE, beginning to diverge around ninety minutes per event,
and continuing to increase sharply at the end of our annotation, leaving
unclear what the potential performance of the technique is. When added to ace,
the ct improves performance somewhat, reducing error of P/R/F 1%/5%/6% at
ninety minutes per event before plateauing. ct has a substantial advantage
over ace for event types which are rare in ace, but lags significantly for
event types abundant in ace (Figure 2).121212The anomalously poor performance
on transaction.transfer-money is due to a bug.

Figure 2: Performance vs. average annotation time per event on a per-event
basis. $x\rightarrow y$ indicates there were $x$ event mentions of training
for this type in ace and $y$ argument tuples for it in the evaluation set.
Figure 3: Teacher a vs NLP experts on conflict.demonstrate

The annotation tool designer and two other NLP experts also did ct for
conflict.demonstrate (Figure 3; Table 1). All experts significantly
outperformed Teacher a and ace in terms of F1. In two cases this is because
the experts sacrificed precision for recall. The second expert matched Teacher
a’s precision with much higher recall. Annotators varied widely in the volume
of their annotation and indicator searches, but this did not have a clear
relationship to performance.

| Tch. a | Designer | Exp. 1 | Exp. 2
---|---|---|---|---
Words | 13k | 21k | 13k | 20k
Doc.s | 256 | 466 | 165 | 334
Searches | 28 | 75 | 23 | 37
Prec. | 71 | 42 | 56 | 71
Rec. | 14 | 38 | 43 | 34
Table 1: Teacher a vs NLP experts on conflict.demonstrate

###  4.1. Possible Confounding Factors

Because Teacher a both provided ct and did the output assessment, improvements
may reflect the system learning their biases. We controlled for this somewhat
by having Teacher B dual-assess several hundred responses, resulting in
encouraging agreement rates of 95% for event presence, 98% for role selection,
and 98% for argument assessment.131313aet, aer, and bf in kbp ea terms [Joe
Ellis and Strassel, 2014] For some events, the guidelines changed from ace to
kbp ea 2014 by eliminating ‘trumping’ rules and expanding allowable inference,
which could also account for some improvement. If either of these were
significant factors, it would suggest that ct may be a useful tool for
retargetting systems to new, related tasks.

##  5\. Acknowledgements

Thanks to Elizabeth Boschee and Dan Wholey for doing annotation. This research
was developed with funding from the Defense Advanced Research Projects Agency
(DARPA). The views, opinions, and/or findings expressed are those of the
author(s) and should not be interpreted as representing the official views or
policies of the Department of Defense or the U.S. Government. Distribution
‘A’: Approved For Public Release, Distribution Unlimited.

##  6\. Bibliographical References

## References

  * Chan et al., 2014 Chan, Y. S., Freedman, M., and Gabbard, R.  (2014).  BBN’s KBP EA System.  In Proceedings of NIST TAC 2014.  http://www.nist.gov/tac/protected/2014/TAC2014-workshop-notebook/results.html.
  * Freedman and Gabbard, 2014 Freedman, M. and Gabbard, R.  (2014).  An Overview of the TAC KBP Event Argument Extraction Evaluation.  In Proceedings of NIST TAC 2014.  http://www.nist.gov/tac/protected/2014/TAC2014-workshop-notebook/results.html.
  * Hong et al., 2011 Hong, Y., Zhang, J., Ma, B., Yao, J., Zhou, G., and Zhu, Q.  (2011).  Using Cross-Entity Inference to Improve Event Extraction.  In Proceedings of the ACL 2011.
  * Ji and Grishman, 2008 Ji, H. and Grishman, R.  (2008).  Refining Event Extraction Through Cross-Document Inference.  In Proceedings of the ACL 2008.
  * Joe Ellis and Strassel, 2014 Joe Ellis, J. G. and Strassel, S.  (2014).  TAC KBP 2014 Event Argument Extraction Assessment Guidelines V. 1.4.  http://www.nist.gov/tac/2014/KBP/Event/guidelines/TAC\\_KBP\\_2014\\_Event\\_Argument\\_Extraction\\_Assessment\\_Guidelines\\_V1.4.pdf.
  * Joe Ellis and Strassel, 2015 Joe Ellis, J. G. and Strassel, S.  (2015).  LDC2015E22: TAC KBP English Event Argument Extraction Comprehensive Pilot and Evaluation Data 2014.
  * Li et al., 2013 Li, Q., Ji, H., and Huang, L.  (2013).  Joint Event Extraction via Structured Prediction with Global Features.  In Proceedings of the ACL 2013.
  * NIST, 2014 NIST.  (2014).  TAC KBP 2014 Event Argument Task Description.  http://www.nist.gov/tac/2014/KBP/Event/guidelines/EventArgumentTaskDescription.09042014.pdf.
  * Parker et al., 2011 Robert Parker and David Graff and Junbo Kong and Ke Chen and Kazuaki Maeda.  (2011).  English Gigaword Fifth Edition.  Linguistic Data Consortium, ISLRN 911-942-430-413-0.
  * Settles, 2011 Settles, B.  (2011).  Closing the Loop: Fast, Interactive Semi-Supervised Annotation with Queries on Features and Instances.  In Proceedings of EMNLP 2011.
  * Strohman et al., 2005 Strohman, T., Metzler, D., Turtle, H., and Croft, W. B.  (2005).  Indri: A language model-based search engine for complex queries.  Proceedings of the International Conference on Intelligent Analysis, 2(6):2–6.
  * Walker et al., 2006 Walker, C., Strassel, S., Medero, J., and Maeda, K.  (2006).  ACE 2005 Multilingual Training Corpus.