# Coherent Asymmetric Absorbers

F.S. Cuesta1<EMAIL_ADDRESS>A.D. Kuznetsov1,2 G.A. Ptitcyn1 X.
Wang1 S.A. Tretyakov1 1Department of Electronics and Nanoengineering, Aalto
University, P.O. Box 15500, Aalto FI-00076, Finland
2School of Electronic Engineering, HSE University, Russia

###### Abstract

Most applications of metasurfaces require excitation and control of both
electric and magnetic surface currents. For such purpose, the metasurface must
have a finite thickness to handle magnetic surface currents. For metasurface
sheets of negligible thickness that offer only electric response, coherent
illumination can compensate the need to create discontinuities of the
tangential electric field component using magnetic surface currents. Most of
known coherent metasurfaces are space-uniform and can control only plane-wave
absorption and specular reflection. However, it is also known that periodical
space-modulated (inhomogeneous) metasurfaces can be used to realize anomalous
reflection, refraction, and other useful effects. In this paper, we propose
the concept of a coherently-illuminated space-modulated metasurface that
functions as a coherent asymmetric absorber. We study its behaviour under non-
ideal illuminations and suggest applications related with sensing.

##  I Introduction

The first step in designing metasurface devices is to determine the surface
impedance that corresponds to the desired scattering for a predefined
illumination (e.g., [1, 2]). For different design targets, the required
properties of the metasurface can be different, requiring chirality (for
polarization conversion, for example), time modulation (for frequency
conversion or nonreciprocal response), space modulation (for engineering
reflection and refraction directions), or other properties. In the case of
periodical space modulations, the availability and propagation directions of
the non-specular modes are determined by the period of the surface impedance
variations, as described by the Floquet theory [3]. However, the power
distribution between specular and available non-specular modes is determined
by the surface impedance variation profile [4]. Because of that, space-
modulated metasurfaces can produce such effects as anomalous reflection and
refraction [5, 6, 7, 8, 2], retroreflection [9, 10, 11], or asymmetric
absorption [12].

At the conceptual level, a metasurface behaves as a sheet creating
discontinuities of fields or as an impenetrable boundary. If the objective is
to control reflection only, metasurfaces are often realized as an impedance
sheet above a metallic ground plane, realizing an effective boundary at the
plane of the impedance sheet [13]. On the other hand, metasurfaces designed
for control of transmission do not have a ground plane, behaving as sheets
carrying electric and magnetic surface currents [2]. Many functionalities
cannot be realized using sheets supporting only electric surface currents (for
microwave applications, usually realized as single arrays of thin metal
patches), for example, perfect absorption or retroreflection. In these cases,
metasurfaces must carry also magnetic surface currents, which requires the use
of at least two parallel arrays or some volumetric particles [14]. However,
coherent metasurfaces can produce these desired effects using a secondary
illumination source, even if the metasurface sheet is negligibly thin and no
magnetic current can be excited [15, 16, 17, 18, 19, 20]. Space-modulated
metasurfaces boundaries, realized as finite-size layers carrying both electric
and magnetic surface currents, can behave as asymmetric absorbers [12, 21, 22,
23].

Figure 1: Concept of a coherent asymmetric absorber, which for different
angles of incidence behaves as an absorber or as a reflective surface. The
cutoff angle $\theta_{\rm c}$ is determined by the metasurface period.

In this paper, we introduce the concept of a space-modulated, single-sheet
coherent asymmetric absorber: a device that behaves as an absorber or as a
retroreflector depending on the angle of incidence of illumination by two
coherent waves, as illustrated in Fig. 1. This work focuses on describing how
the angle of incidence and variations in the magnitudes and phases of the
coherent sources affect the response of the metasurface. In addition, this
paper proposes applications where ideal and non-ideal coherent illuminations
can be exploited for sensing and power absorption.

##  II Coherent retroreflection and absorption in space-modulated sheets

Figure 2:  Incident and reflected plane waves from coherent retroreflective
electromagnetic metasurfaces which are designed for (a)–transverse electric
and (b)–transverse magnetic polarizations.

Let us consider a flat negligibly-thin metasurface that divides the space into
two regions. The metasurface is illuminated by two plane waves at the same
angle of incidence $\theta_{\rm I}$, but incident from each region separately.
We start the analysis by designing a metasurface impedance profile that
ensures perfect retroreflection of both illuminating waves at the design angle
$\theta_{\rm I}=\theta_{\rm ms}$, see illustration in Fig. 2. The incident
waves can be polarized either as transverse electric (TE) or transverse
magnetic (TM) waves. This paper will focus on TE-illumination scenarios, as
shown in Fig. 2(a), but the same principles can be also applied to the TM-
case, illustrated in Fig. 2(b). The fields for a TE-polarized illumination
read

$\begin{split}&{\bf E}_{\rm I,1}=E_{\rm I,1}e^{-jk_{0}\left[y{\rm
sin}(\theta_{\rm I})-z{\rm cos}(\theta_{\rm I})\right]}{\bf x},\cr&{\bf
H}_{\rm I,1}=-\dfrac{E_{\rm I,1}}{\eta_{0}}\Big{[}{\rm cos}(\theta_{\rm
I}){\bf y}+{\rm sin}(\theta_{\rm I}){\bf z}\Big{]}e^{-jk_{0}\left[y{\rm
sin}(\theta_{\rm I})-z{\rm cos}(\theta_{\rm I})\right]},\end{split}$ (1)
$\begin{split}&{\bf E}_{\rm R,1}=-E_{\rm R,1}e^{jk_{0}\left[y{\rm
sin}(\theta_{\rm I})-z{\rm cos}(\theta_{\rm I})\right]}{\bf x},\cr&{\bf
H}_{\rm R,1}=-\dfrac{E_{\rm R,1}}{\eta_{0}}\Big{[}{\rm cos}(\theta_{\rm
I}){\bf y}+{\rm sin}(\theta_{\rm I}){\bf z}\Big{]}e^{jk_{0}\left[y{\rm
sin}(\theta_{\rm I})-z{\rm cos}(\theta_{\rm I})\right]},\end{split}$ (2)

in region 1, and

$\begin{split}&{\bf E}_{\rm I,2}=E_{\rm I,2}e^{-jk_{0}\left[y{\rm
sin}(\theta_{\rm I})+z{\rm cos}(\theta_{\rm I})\right]}{\bf x},\cr&{\bf
H}_{\rm I,2}=\dfrac{E_{\rm I,2}}{\eta_{0}}\Big{[}{\rm cos}(\theta_{\rm I}){\bf
y}-{\rm sin}(\theta_{\rm I}){\bf z}\Big{]}e^{-jk_{0}\left[y{\rm
sin}(\theta_{\rm I})+z{\rm cos}(\theta_{\rm I})\right]},\end{split}$ (3)
$\begin{split}&{\bf E}_{\rm R,2}=-E_{\rm R,2}e^{jk_{0}\left[y{\rm
sin}(\theta_{\rm I})+z{\rm cos}(\theta_{\rm I})\right]}{\bf x},\cr&{\bf
H}_{\rm R,2}=\dfrac{E_{\rm R,2}}{\eta_{0}}\Big{[}{\rm cos}(\theta_{\rm I}){\bf
y}-{\rm sin}(\theta_{\rm I}){\bf z}\Big{]}e^{jk_{0}\left[y{\rm
sin}(\theta_{\rm I})+z{\rm cos}(\theta_{\rm I})\right]},\end{split}$ (4)

in region 2, where $E_{\rm I}$ and $E_{\rm R}$ are the amplitudes of the
incident and reflected waves, respectively; $k_{0}$ is the wavenumber of the
incident waves and $\eta_{0}$ is the characteristic impedance of the
surrounding medium. The metasurface must be tailored such that only a single
non-specular mode gets excited when the metasurface is illuminated at the
design angle $\theta_{\rm ms}$. The resulting reflected mode propagates back
in the same direction as the incident wave, with a predetermined relative
reflection amplitude $\Gamma$ and phase $\phi_{\rm R}$ with respect to the
incident waves ($E_{\rm R1}=E_{\rm I1}\Gamma\exp\left[j\phi_{\rm R}\right]$,
$E_{\rm R2}=E_{\rm I2}\Gamma\exp\left[j\phi_{\rm R}\right]$). The excess of
energy supplied by the sources is absorbed by the metasurface.

The metasurface can be modeled as a sheet of electric surface current density
${\bf J}_{\rm s}={\bf E}/Z_{\rm s}$, where ${\bf E}$ is the tangential
component of the macroscopic electric field at the metasurface plane, and
$Z_{\rm s}$ is the surface impedance. Since no magnetic surface currents are
induced, the metasurface does not produce any discontinuity in the tangential
electric field. In this case, symmetric perfect retroreflection can be
realized only if both incident waves have the same amplitude and phase
($E_{\rm I1}=E_{\rm I2}$), so that also the reflected waves have equal complex
amplitudes ($E_{\rm R1}=E_{\rm R2}$). Therefore, for a given incidence
direction, the surface impedance required for perfect coherent retrorefleciton
can be determined by satisfying the boundary conditions for the total
tangential electric and magnetic fields (the electric surface current density
equals to the jump of the tangential magnetic field). For TE or TM
illumination, the surface impedance profile reads, respectively,

$Z_{s}^{\rm TE}\left(y\right)=\frac{\eta_{0}}{2\cos\theta_{\rm
ms}}\frac{\exp\left[-2jk_{0}y\sin\theta_{\rm
ms}\right]-\Gamma\exp\left[j\phi_{R}\right]}{\exp\left[-2jk_{0}y\sin\theta_{\rm
ms}\right]+\Gamma\exp\left[j\phi_{R}\right]},$ (5a) $Z_{s}^{\rm
TM}\left(y\right)=\frac{\eta_{0}\cos\theta_{\rm
ms}}{2}\frac{\exp\left[-2jk_{0}y\sin\theta_{\rm
ms}\right]+\Gamma\exp\left[j\phi_{R}\right]}{\exp\left[-2jk_{0}y\sin\theta_{\rm
ms}\right]-\Gamma\exp\left[j\phi_{R}\right]}.$ (5b)

Through inspection, it can be found that the impedance profile is a periodic
function with the period $D=\lambda_{0}/\left(2\sin\theta_{\rm ms}\right)$,
where $\lambda_{0}$ is the reference design wavelength. A change of the
reflection phase $\phi_{R}$ corresponds to a lateral shift of the impedance
position.

Figure 3: (a) A metasurface designed to create a non-specular mode, acting as
a retroreflector at the design incidence angle $\theta_{\rm ms}$, has an
asymmetric behavior in the form of an absorption region, where non-specular
modes only exist as evanescent waves. This example considers surface
impedances corresponding to $\Gamma=0.99$. (b) In the angular region where the
non-specular mode can propagate, its amplitude can be determined by the
reflection coefficient $\Gamma$ as a design parameter. The structure becomes
lossless and symmetric if the value of $\left|\Gamma\right|=1$ is used,
however, even for low-loss sheets with the reflection coefficients close to
this value there is strong contrast between absorption and reflection regions.

Numerical results of Fig. 3(a) for coherent retroreflectors with different
design angles $\theta_{\rm ms}$ reveal that the angular dependence of
absorption is strongly asymmetric. The absorption is high in the angular
region where there is only one propagating mode available (specular
reflection), and it is weak where the non-specular mode is excited as a
surface wave. By changing the design angle of incidence $\theta_{\rm ms}$, it
is possible to control these angular regions, delimited by the cutoff angle
$\theta_{\rm c}$. The high-absorption regime mostly corresponds to
illuminations with “negative” angles of incidence (tilted to the left from the
normal to the metasurface plane, as defined in Fig. 2).

This asymmetric behaviour allows us to design metasurfaces that exhibit strong
coherent retroreflection for positive angles of incidence but strong coherent
absorption for negative ones, as is illustrated in Fig. 3(a). An additional
design feature is a possibility of controlling the reflection coefficient of
the non-specular mode without disturbing the asymmetric absorption, as
portrayed in Fig. 3(b). The asymmetry is present for almost all values of the
amplitude of the design reflection coefficient $\Gamma$, except for the
extreme values $\left|\Gamma\right|=1$ and $\Gamma=0$, which correspond to
already known cases of the perfect coherent retroreflector [20] and the
perfect coherent absorber [24], respectively. This asymmetric angular
dependence in absorption is similar to the lossy retroreflective boundaries of
Ref. [12], illuminated by only one wave.

##  III Asymmetric Illumination

Figure 4:  As described by the Floquet theory, uniform illumination of a
periodic surface excites a certain range of specular and non-specular waves.
While the mode range is defined by the period of the surface and the incidence
angle of the exciting wave, the amplitude of each excited mode is determined
by the surface impedance profile.

The most attractive feature of coherent metasurfaces is a possibility to tune
the response by varying phase or amplitude differences between the
illuminating waves. To this end, here we analyse the response of coherent
retroreflectors illuminated by an arbitrary set of two plane waves. When a
sheet designed to function as a coherent retroreflector for a given incidence
angle is illuminated by any other source, the reflected field contains
infinitely many plane-wave components, as portrayed in Fig. 4. Let us consider
a periodical metasurface with the surface impedance $Z_{s}$ [in particular,
given by Eq. (5a) or (5b)] with the period $D$, illuminated by a single TE-
polarized wave (similar conclusions are achieved for the TM-polarization) with
the angle of incidence $\theta_{\rm I}$. As described by the Floquet theory,
the field on the metasurface plane can be considered as a combination of
propagating and evanescent plane-wave modes:

$E=E_{\rm
I}e^{-jyk\sin\theta_{i}}+\sum_{n=-\infty}^{\infty}E_{[n]}e^{-jyk_{y[n]}},$ (6)

where $E_{[n]}$ is the amplitude of the $n$-th Floquet harmonic of the field
expansion, defined by the wavevector components $k_{y[n]}=k\sin\theta_{i}+2\pi
n/D$ and $k_{z[n]}=\sqrt{k^{2}-k_{y[n]}^{2}}$. For convenience, we define the
mode angle $\theta_{[n]}=\arcsin{\left(k_{y[n]}/k\right)}$. Note that this
angle is imaginary for evanescent components, and becomes real at the cutoff
angle $\theta_{\rm c}$ where $k_{y[n]}=\left|k\right|$. The tangential
electric field is continuous across the sheet, and the jump of the tangential
magnetic field equals to the surface current density of the induced electric
current:

$\sum_{n=-\infty}^{\infty}2E_{[n]}\cos\theta_{[n]}e^{-jyk_{y[n]}}=-\eta_{0}J_{s}.$
(7)

$\begin{split}\sum_{n=-\infty}^{\infty}\left(2\cos\theta_{[n],1}+\dfrac{\eta_{0}}{Z_{s}}\right)E_{[n],1}e^{-jyk_{y[n],1}}&+\sum_{n=-\infty}^{\infty}\left(2\cos\theta_{[n],2}+\dfrac{\eta_{0}}{Z_{s}}\right)E_{[n],2}e^{-jyk_{y[n],2}}\\\
&=-\dfrac{\eta_{0}}{Z_{s}}\left[E_{\rm I,1}e^{-jyk\sin\theta_{i,1}}+E_{\rm
I,2}e^{-jyk\sin\theta_{i,2}}\right]\end{split}$ (8)

Because the sheet response is linear, the current excited by two coherent
plane waves can be found as the sum of two independent Floquet expansions
corresponding to the two different periods fixed by the two different
incidence angles. The boundary condition $Z_{s}{\bf J}_{e}={\bf E}$ takes the
general form of Eq. (8). In the special case when the two incidence angles are
the same, the periods of the two expansions are equal, and the two series can
be combined into one. For this scenario, Eq. (8) can be rewritten by combining
pairs of scattered harmonics ($E_{[n]}=E_{[n],1}+E_{[n],2}$), and the boundary
condition takes the form

$\begin{split}\sum_{n=-\infty}^{\infty}\left(2\cos\theta_{[n]}+\dfrac{\eta_{0}}{Z_{s}}\right)&E_{[n]}e^{-jyk_{y[n]}}\\\
&=-\dfrac{\eta_{0}}{Z_{s}}e^{-jyk\sin\theta_{i}}\left[E_{\rm I1}+E_{\rm
I2}\right],\end{split}$ (9)

where each scattering mode does not only depend on the interaction between the
fields and the metasurface, but also on the direct interaction between the
incident waves at the boundary.

In order to understand the effect of phase amplitude differences between the
two illuminaitng waves, the amplitude of the source in region 2 can be defined
as proportional to the one in region 1: $E_{\rm I2}=E_{\rm I1}\rho_{\rm
I2}\exp\left[j\phi_{\rm I2}\right]$, where $\rho_{\rm I2}$ is the amplitude
ratio. The total incident field at the metasurface plane is then written as

$E_{\rm tot}=E_{\rm I1}\left[1+\rho_{\rm I2}e^{j\phi_{\rm I2}}\right].$ (10)

The metasurface is functioning as a perfect coherent retroreflector when the
two waves have equal amplitudes ($\rho_{\rm I2}=1$) and phases ($\phi_{\rm
I2}=0$). Otherwise, the metasurface is “mismatched” with respect to the
sources, and its response can be dramatically different. In the limiting case
of the equal amplitude but opposite phase, that is, when $\rho_{\rm I2}=1$ and
$\phi_{\rm I2}=\pi$, there is no interaction between the sheet and the
incident waves.

##  IV Numerical Characterization

The exact values of the scattered waves amplitudes can be determined for
lossless metasurfaces by solving Eq. (8) in combination with the power
conservation law for the propagating modes ($\left|k_{y[n]}\right|\leq k$). On
the other hand, if the metasurface is lossy or it has an active behavior,
power conservation is not usable as the total scattered power is not equal to
the power radiated by the metasurface. For these scenarios, approximate
solutions can be found using numerical means based on the Fourier series
expansion of the surface impedance $Z_{s}$ in combination with the mode-
matching method developed in Refs. [25, 4]. The absorption in the metasurface
is found by subtracting the power reflected into free space from the total
incident power.

Figure 5: (a) A coherently-illuminated metasurface designed with equivalent
electric current densities in mind is matched with the sources when both waves
illuminate the surface at the same phase. Any variation in the phase will
create a mismatch, leading to some specular reflection. (b) Similarly, the
sources are matched with the metasurface as long as both plane waves
illuminate the surface with the same amplitude. Any difference in the
amplitude breaks the system symmetry, exciting undesired specular reflection.
Examples correspond to surface impedances with $\Gamma=0.71$ and $\theta_{\rm
ms}=30^{\circ}$.

Results of Fig. 3 show the behaviour of the proposed structure with a
combination of matching sources, as described in Eq. (10). The strong
absorption regime for “negative angles” ($\theta_{\rm I}<0$) is achieved by
preventing the excitation of the first available non-specular mode ($n=-1$),
leaving only specular reflection ($n=0$). On the other hand, Figs. 5 (a) and
(b) show how a mismatch between the two coherent sources affect absorption in
the metasurface when different phase shifts $\phi_{\rm I2}$ and amplitude
ratios $\rho_{\rm I2}$ are considered, respectively. Obviously, the
metasurface behaves as a lossless, fully transparent structure when the
amplitudes are equal and the phase difference is equal to $\pi$. When one of
the two amplitudes is much larger than the other, we approach the regime of a
single-wave excitation of a sheet of negligible thickness, in which case
absorption cannot be higher than $50\%$ of the incident power.

##  V Potential applications of asymmetric coherent absorbers

Figure 6: (a) By inserting a coherent asymmetric absorber inside a parallel-
plate waveguide, it is expected to transfer its properties into the waveguide
for TE1 mode. (b) and (d) If the metasurface is designed for $\theta_{\rm
ms}=30^{\circ}$, one end of the waveguide becomes highly reflective. (c) and
(e) On the other hand, the waveguide absorbs the TE1 wave when it is fed from
the opposite end of the waveguide. For this example, the metasurface was
designed with $\theta_{\rm ms}=30^{\circ}$, $\Gamma=0.99$, for a reference
frequency of $10$ GHz and with total length of $l=10D$.

Asymmetric absorption and dramatic differences of response of asymmetric
coherent absorbers under coherent and incoherent illuminations can be
exploited in various applications. First, let us consider the setup of Fig.
6(a), where a sample of such metasurface is placed in the symmetry plane of a
parallel-plate waveguide, supporting the TE1 mode. The field of this mode can
be decomposed into two coherent plane waves [26]. By changing the separation
between the plates $h$, the angle of incidence of these partial waves on the
metasurface can be tuned. In the considered example, the metasurface is
designed with $\theta_{\rm ms}=30^{\circ}$ ($\theta_{\rm c}=0^{\circ}$) and
$\Gamma=0.99$, resulting in high reflection for positive angles and strong
absorption for the opposite tilt, as shown in Fig. 3(a). In this
configuration, the waveguide inherits the properties of the metasurface as a
retroreflector or as an absorber, depending on which end of the waveguide is
excited, as portrayed in Figs. 6(b)-(c). This behavior is stable over a wide
frequency range where higher-order modes beyond TE1 cannot propagate, as is
seen in Fig. 6(d)–(e). However, close to the cutoff frequency of the TE1 mode,
the response is affected by the difference between the waveguide
characteristic impedance with and without the metasurface, resulting in high
reflection from both sides. Let us note that this setup does not break
reciprocity, and transmission through the device is symmetric.

We see that this configuration functions as a compact and broadband microwave
absorber in a waveguide that does not contain any terminating boundary and can
freely pass high-frequency electromagnetic radiation or, for example, flow of
air or water. Also, for waves propagation into the opposite direction, the
device works as broadband reflector, although there is no reflecting wall.

Figure 7: Positioning a coherent asymmetric absorber in the field of two
counter-propagating plane waves allows us to obtain a structure capable to
sense small changes in the metasurface angular position. Coherent illumination
at the same angle is realized only for the normal incidence ($\theta_{\rm
I}=0$), in which case the absorption it total. Absorption for any non-normal
illumination is comparable to the single-source scenario, where the absorption
is limited by $50\%$. (a) Geometry of the setup. (b)–(c) The angular
dependence of absorption at oblique illumination can be controlled tuning
$\theta_{\rm ms}$ and $\Gamma$. The example in (b) is for the design angle
$\theta_{\rm ms}=45^{\circ}$ and varying $\Gamma$, while (c) is for the
reflection coefficient of $\Gamma=0.99$ and variable design angle.

The second application exploits the fact that for any mismatched illumination
the absorption is limited by $50\%$ of the incident power, while it can be
perfect at coherent and symmetric illumination. The sources (or one source and
a uniform mirror) are aligned so that the two incident waves propagate in the
opposite directions, forming a standing wave. The metasurface is positioned in
the field of this standing wave with a tilt with respect to the propagation
axis, as illustrated in Fig. 7(a). From the point of view of the metasurface,
one of the sources illuminates it at a positive angle of incidence
$\theta_{\rm I}$, while the other wave has a negative angle of incidence
$-\theta_{\rm I}$. Thus, the condition of coherent illumination that requires
equal incident angle for both waves, is not fulfilled. The only exception,
when the metasurface receives matched illumination, is the case of the normal
incidence $\theta_{\rm I}=0$. At this singular value of the tilt angle, the
absorption is perfect, as shown in Figs. 7(b)–(c). For any other angle of
incidence, the absorption level drops to $50\%$ or more [see Fig. 3]. The
profile, shown in Fig. 7(b), is symmetric, as either both waves illuminate in
the absorption regime, or one of them excites the non-specular mode.
Absorption level for angles compatible with non-specular reflection can be
controlled using the parameter $\Gamma$, down to $25\%$ of the total incident
power. The angular range of these regions of different absorption can be
controlled through the design angle $\theta_{\rm ms}$, as shown in Fig. 7(c).
The case when the metasurface is designed for $\theta_{\rm ms}=30^{\circ}$ is
of special attention, as the non-specular mode can only be excited as a
propagating wave for positive angles of incidence $\theta_{\rm I}$. As a
result, considering a reflection coefficient value of $\Gamma=0.99$, the
difference between the absorption peak at normal illumination and the
absorption at oblique illumination is maximized.

##  VI Conclusions

In this paper we presented the concept of coherent asymmetric absorber that
changes its behavior according to the incidence angle of coherent-wave
illuminations. It was shown that negligibly thin nonuniform sheets
characterized by the determined sheet impedance operate as coherent
counterparts of asymmetric absorbers. Under coherent illumination by two plane
waves, absorption changes from nearly perfect to very small when the incidence
angle varies. The study of arbitrary illuminations of these sheets reveals
possibilities to tune the absorption and reflection changing the amplitude and
phase of one of the two illuminating waves.

Furthermore, this paper proposed two applications of coherent asymmetric
absorbers: as broadband asymmetric waveguide absorbers and sensors of angular
position. By placing a coherent asymmetric absorber sheet inside a parallel-
plate waveguide, equidistant to its walls, it is possible to realize broadband
absorption without a need to block the cross-section opening of the waveguide.

Exploiting the dramatic difference between achievable absorption under perfect
coherent illumination by two waves with the same incident angle and any other
illuminations, we showed that it is possible to detect angular displacements
of objects with extreme accuracy.

## References

  * [1] F. Yang and Y. Rahmat-Samii, Eds., _Surface Electromagnetics, With Applications in Antenna, Microwave, and Optical Engineering_. Cambridge University Press, 2019.
  * [2] C. Simovski and S. Tretyakov, _An Introduction to Metamaterials and Nanophotonics_. Cambridge University Press, 2020.
  * [3] A. Ishimaru, _Periodic Structures and Coupled-Mode Theory_. John Wiley & Sons, Ltd, 2017, ch. 7, pp. 201–231.
  * [4] X. Wang, A. Díaz-Rubio, and S. A. Tretyakov, “Independent control of multiple channels in metasurface devices,” _Phys. Rev. Applied_ , vol. 14, p. 024089, Aug 2020.
  * [5] Z. Li, L. Huang, K. Lu, Y. Sun, and L. Min, “Continuous metasurface for high-performance anomalous reflection,” _Applied Physics Express_ , vol. 7, no. 11, p. 112001, oct 2014.
  * [6] K. Tang, C. Qiu, M. Ke, J. Lu, Y. Ye, and Z. Liu, “Anomalous refraction of airborne sound through ultrathin metasurfaces,” _Scientific Reports_ , vol. 4, no. 1, Oct. 2014.
  * [7] A. Díaz-Rubio, V. S. Asadchy, A. Elsakka, and S. A. Tretyakov, “From the generalized reflection law to the realization of perfect anomalous reflectors,” _Science Advances_ , vol. 3, no. 8, p. e1602714, 2017.
  * [8] D. Sell, J. Yang, E. W. Wang, T. Phan, S. Doshay, and J. A. Fan, “Ultra-high-efficiency anomalous refraction with dielectric metasurfaces,” _ACS Photonics_ , vol. 5, no. 6, pp. 2402–2407, 2018.
  * [9] Y. Jia, J. Wang, Y. Li, Y. Pang, J. Yang, Y. Fan, and S. Qu, “Retro-reflective metasurfaces for backscattering enhancement under oblique incidence,” _AIP Advances_ , vol. 7, no. 10, p. 105315, 2017.
  * [10] V. Asadchy, A. Díaz-Rubio, S. Tcvetkova, D.-H. Kwon, A. Elsakka, M. Albooyeh, and S. Tretyakov, “Flat engineered multichannel reflectors,” _Physical Review X_ , vol. 7, no. 3, Sep. 2017.
  * [11] X. Li, C. Tao, L. Jiang, and T. Itoh, “Blazed metasurface grating with handedness preservation for circularly polarized incident wave,” in _2018 48th European Microwave Conference (EuMC)_ , 2018, pp. 133–136.
  * [12] X. Wang, A. Díaz-Rubio, V. S. Asadchy, G. Ptitcyn, A. A. Generalov, J. Ala-Laurinaho, and S. A. Tretyakov, “Extreme asymmetry in metasurfaces via evanescent fields engineering: Angular-asymmetric absorption,” _Phys. Rev. Lett._ , vol. 121, p. 256802, Dec 2018.
  * [13] S. Tretyakov, _Analytical Modeling in Applied Electromagnetics_. Artech House, 2003.
  * [14] V. S. Asadchy, I. A. Faniayeu, Y. Ra’di, S. A. Khakhomov, I. V. Semchenko, and S. A. Tretyakov, “Broadband reflectionless metasheets: Frequency-selective transmission and perfect absorption,” _Phys. Rev. X_ , vol. 5, p. 031005, Jul 2015.
  * [15] W. Lv, J. Bing, Y. Deng, D. Duan, Z. Zhu, Y. Li, C. Guan, and J. Shi, “Polarization-controlled multifrequency coherent perfect absorption in stereometamaterials,” _Opt. Express_ , vol. 26, no. 13, pp. 17 236–17 244, Jun 2018.
  * [16] X. Feng, J. Zou, W. Xu, Z. Zhu, X. Yuan, J. Zhang, and S. Qin, “Coherent perfect absorption and asymmetric interferometric light-light control in graphene with resonant dielectric nanostructures,” _Opt. Express_ , vol. 26, no. 22, pp. 29 183–29 191, Oct 2018.
  * [17] J. Zou, F. Xiong, J. Zhang, and Z. Zhu, “Coherent perfect absorption in artificially engineered nanometer metal/semiconductor composite films at oblique incidence,” _OSA Continuum_ , vol. 2, no. 11, pp. 3251–3261, Nov 2019.
  * [18] D. G. Baranov, A. Krasnok, T. Shegai, A. Alù, and Y. Chong, “Coherent perfect absorbers: linear control of light with light,” _Nature Reviews Materials_ , vol. 2, no. 12, oct 2017.
  * [19] G. Nie, Q. Shi, Z. Zhu, and J. Shi, “Selective coherent perfect absorption in metamaterials,” _Applied Physics Letters_ , vol. 105, no. 20, p. 201909, 2014\.
  * [20] F. S. Cuesta, G. A. Ptitcyn, M. S. Mirmoosa, and S. A. Tretyakov, “Coherent retroreflective metasurfaces,” _Phys. Rev. Research_ , vol. 3, p. L032025, Jul 2021.
  * [21] A. Song, J. Li, X. Peng, C. Shen, X. Zhu, T. Chen, and S. A. Cummer, “Asymmetric absorption in acoustic metamirror based on surface impedance engineering,” _Phys. Rev. Applied_ , vol. 12, p. 054048, Nov 2019.
  * [22] X. Wang, X. Fang, D. Mao, Y. Jing, and Y. Li, “Extremely asymmetrical acoustic metasurface mirror at the exceptional point,” _Phys. Rev. Lett._ , vol. 123, p. 214302, Nov 2019.
  * [23] S. Dong, G. Hu, Q. Wang, Y. Jia, Q. Zhang, G. Cao, J. Wang, S. Chen, D. Fan, W. Jiang, Y. Li, A. Alù, and C.-W. Qiu, “Loss-assisted metasurface at an exceptional point,” _ACS Photonics_ , vol. 7, no. 12, pp. 3321–3327, 2020\.
  * [24] Y. Fan, F. Zhang, Q. Zhao, Z. Wei, and H. Li, “Tunable terahertz coherent perfect absorption in a monolayer graphene,” _Opt. Lett._ , vol. 39, no. 21, pp. 6269–6272, Nov 2014.
  * [25] X. Wang, A. Díaz-Rubio, H. Li, S. A. Tretyakov, and A. Alù, “Theory and design of multifunctional space-time metasurfaces,” _Phys. Rev. Applied_ , vol. 13, p. 044040, Apr 2020.
  * [26] D. M. Pozar, _Microwave Engineering_ , 3rd ed. Wiley, 2004.


††thanks<EMAIL_ADDRESS><EMAIL_ADDRESS>
#  Identification of the Mott insulating CDW state in 1T-TaS2

Dongbin Shin  Max Planck Institute for the Structure and Dynamics of Matter
and Center for Free-Electron Laser Science, Luruper Chaussee 149, 22761,
Hamburg, Germany  Nicolas Tancogne-Dejean  Max Planck Institute for the
Structure and Dynamics of Matter and Center for Free-Electron Laser Science,
Luruper Chaussee 149, 22761, Hamburg, Germany  Jin Zhang  Max Planck Institute
for the Structure and Dynamics of Matter and Center for Free-Electron Laser
Science, Luruper Chaussee 149, 22761, Hamburg, Germany  Mahmut Sait Okyay
Department of Physics, Ulsan National Institute of Science and Technology
(UNIST), UNIST-gil 50, Ulsan 44919, Korea  Angel Rubio  Max Planck Institute
for the Structure and Dynamics of Matter and Center for Free-Electron Laser
Science, Luruper Chaussee 149, 22761, Hamburg, Germany  Nano-Bio Spectroscopy
Group, Departamento de Fsica de Materiales, Universidad del Pas Vasco, 20018
San Sebastian, Spain  Center for Computational Quantum Physics (CCQ), The
Flatiron Institute, 162 Fifth avenue, New York NY 10010  Noejung Park
Department of Physics, Ulsan National Institute of Science and Technology
(UNIST), UNIST-gil 50, Ulsan 44919, Korea

###### Abstract

We investigate the low-temperature charge-density-wave (CDW) state of bulk
TaS2 with a fully self-consistent DFT+U approach, over which the controversy
has remained unresolved regarding the out-of-plane metallic band. By examining
the innate structure of the Hubbard U potential, we reveal that the
conventional use of atomic-orbital basis could seriously misevaluate the
electron correlation in the CDW state. By adopting a generalized basis,
covering the whole David star, we successfully reproduce the Mott insulating
nature with the layer-by-layer antiferromagnetic order. Similar consideration
should be applied for description of the electron correlation in molecular
solid.

††preprint: APS/DFT+GOU

A first-principle description of interacting many-body states has been a
central issue in many areas of solid state theories. While the density
functional theory (DFT) provides a formal route for finding the ground state
energy of a many-electrons system, the self-consistent density seeking
procedure, as formulated by Kohn and Sham (KS), has been usually practiced
with approximated local or semi-local functionals [1, 2, 3, 4, 5, 6]. Those
practical functionals lack the derivative discontinuity in the energy density
functional [7, 8, 9], and usually underestimates the fundamental band gap and
overestimates charge transfer [10, 11, 12]. Among various complementation
schemes [3, 13], the KS scheme combined with the Hubbard U potential, now
known as DFT+U, has been most widely used particularly in the descriptions of
the correlation-dominated insulating nature– the Mott insulating state [14,
15, 16, 17, 18]. On the other hand, when the on-site potential is not fully
repulsive, as in cases of various bad-metal phases, the statistical samplings
of multiple occupations are essentially required, as treated in the framework
of the dynamical mean field theory (DMFT) [19, 20].

A four-decades-long debate has remained unresolved for the low temperature
phase of bulk TaS2. Whereas many experiments reported the insulating CDW phase
[21, 22, 23, 24, 25, 26, 27, 28], first-principle calculations with DFT and
DFT+U have noticed that the in-plane Mott states constitute the metallic band
dispersion along the out-plane direction [29, 30, 31, 32]. It has
experimentally reported that, on the formation of the low-temperature Mott
insulating phase of the commensurate CDW phase at T$<200$ K, the atomic
structures are reconstructed into the David-star pattern, packed in the
$\sqrt{13}\times\sqrt{13}$ in-plane super-lattice [24, 25, 26, 27, 28]. The
Mott insulating nature of the monolayer has been described with the first-
principle calculations, and also the corresponding the two-dimensional (2D)
Hubbard model Hamiltonian has been evaluated in the frame work of DMFT [33,
34, 35]. However, on stacking of the 2D Mott insulating layers in the bulk
1T-TaS2, the calculation results with DFT, DFT+U and spin-frozen hybrid
functional lose the separation between the lower Hubbard and upper Hubbard
bands, resulting in a spin-unpolarized metallic dispersion along the out-of-
plane direction [29, 30, 31, 32]. These theoretical results are clearly
contrasting with experimentally observed insulating nature [21, 22, 23, 24,
25, 26, 27, 28]. Leaving behind these inconsistencies, variety of interests
have been gathered on complex phases diagram of TaS2 [24, 25]. Also,
controllable phase transitions have been pursued to utilize the phase diagram
as a memory device [36, 37, 38, 26, 39].

To resolve the inconsistency between the previous experiments and theories for
the bulk 1T-TaS2, here we reexamine the innate structure of the DFT+U [29, 30,
31]. In the standard formulation of the DFT+U, the total energy is expressed
as a functional of both the electron density and the density matrix
constructed in the correlated subspace
$E_{DFT+U}={E_{DFT}}[\rho(\mathbf{r})]+{E_{U}}[n]$ [14, 15, 16]. The averaged
two-body energy is separated into the on-site Coulomb ($U$) and exchange
energy ($J$), and by subtracting the doubly counted mean-field energy from the
DFT energy functional, the effective Hubbard U energy can be expressed in
terms of $\bar{U}=U-J$, as follows:

${E_{U}}=\sum\limits_{I}{\frac{{{\bar{U}_{I}}}}{2}\sum\limits_{m,m^{\prime},\sigma}{\left\\{{n_{m,m^{\prime}}^{I,\sigma}{\delta_{m,m^{\prime}}}-n_{m,m^{\prime}}^{I,\sigma}n_{m^{\prime},m}^{I,\sigma}}\right\\}}},$
(1)

where
$n_{m,m^{\prime}}^{I,\sigma}=\sum\limits_{i,\mathbf{k},\sigma}\frac{1}{N_{\mathbf{k}}}{\left\langle{\psi_{i,\mathbf{k}}^{\sigma}\left|{\hat{P}_{m,m^{\prime}}^{I}}\right|\psi_{i,\mathbf{k}}^{\sigma}}\right\rangle}$,
and
$\hat{P}_{m,m^{\prime}}^{I}=\left|{\phi_{m}^{I}}\right\rangle\left\langle{\phi_{m^{\prime}}^{I}}\right|$.
Above, $\psi_{i,\mathbf{k}}^{\sigma}$ indicates the KS orbital for the $i$-th
band with the Bloch vector $\mathbf{k}$ and spin state $\sigma$.
$\phi_{m}^{I}$ represents the atomic-orbital basis, for the $m$-th orbital of
the $I$-th atom. $N_{\mathbf{k}}$ indicates number of sampled k-points in the
Brillouin zone. The functional variation $\delta
E_{DFT+U}/\delta\psi_{i,\mathbf{k}}^{\sigma}$ leads to one-body potential that
depends on the electron density of the band states and the density matrix of
the projected subspace [14, 15, 16].

Figure 1:  (a) Various occupations of the molecular orbitals of the model
diatomic molecule. (b) The Hubbard energy with respect to the electron number
(N) as defined in (a). (c) Density of state for the O2/Cu(100) system. (d) The
charge of O2 with respect to additional electron doping on the O2/Cu(100)
system. In (a) and (b), $|a\rangle$ and $|b\rangle$ indicate the bonding and
anti-bonding molecular orbitals, respectively. Inset of (c) indicates
schematic geometry of O2 on Cu(100) surface. In (c), upward and downward
arrows indicates doubly degenerated spin-up and spin-down states of O2.

Now we examine the characteristics of the functional with a simple molecular
example. For a diatomic molecular orbital consisting of two one-orbital atoms,
the Hubbard energy can be written as
${E_{U}}=\sum\limits_{\sigma}{\frac{\bar{U}}{2}\left[{\left({{n_{b\sigma}}-n_{b\sigma}^{2}}\right)+\left({{n_{a\sigma}}-n_{a\sigma}^{2}}\right)}\right]}$,
where ${n_{b\sigma}}$ and ${n_{a\sigma}}$ represent the occupation numbers of
the bonding and anti-bonding molecular orbitals with the spin state $\sigma$,
respectively [40, 41, 42]. The sequence of electron occupations is
schematically depicted in Fig. 1(a), assuming that the energy splitting
between the bonding and anti-bonding states is substantially larger than the
on-site interaction energy ($\bar{U}$). The sharp kinks in the plot of $E_{U}$
in terms of electron numbers, as shown by the solid line in Fig. 1(b),
illustrate clearly the well known derivative discontinuity of the exact
exchange-correlation functional at the integer numbers of particles [7, 8, 9].

One may consider the same system by counting the occupations of each atomic
orbitals. The Hubbard energy becomes
${E_{U}^{atom}}=\sum\limits_{\sigma}{\frac{\bar{U}}{2}\left[{\left({{n_{1\sigma}}-n_{1\sigma}^{2}}\right)+\left({{n_{2\sigma}}-n_{2\sigma}^{2}}\right)}\right]}$
with ${n_{1\sigma}}$ and ${n_{2\sigma}}$ indicating the occupation of the
first and second atomic orbital with the spin ($\sigma$), respectively. Once
the series of the electron occupation, shown in Fig. 1(a), is subject to the
molecular symmetry, it is natural to claim that the two expressions of the
density matrix are related by
${n_{1\sigma}}={n_{2\sigma}}=\frac{1}{2}\left({{n_{b\sigma}}+{n_{a\sigma}}}\right)$.
As revealed by the dashed red line in Fig. 1(b), the plot of the
$E_{U}^{atom}$ in terms of the electron number largely misses the kink
structure at the point of the integer numbers of electrons [8, 16]. This
result indicates that the application of the Hubbard U potential onto each
constituent atomic orbitals are irrelevant, substantially misevaluating the
electron correlation in molecular-orbital states.

To demonstrate this misbehavior in a more realistic example, we consider the
O2 molecule on the surface of Cu(100) surface: O2 molecule fixed to a distance
of 5Å above the three-layer Cu slab, as depicted in inset of Fig. 1(c). The
density of state of the O2/Cu(100) reveals the spin-triplet O2 molecular state
over the background metallic states of the Cu layer, as indicated by two pairs
of arrows in Fig. 1(c). We added additional electrons to this system, and we
calculated each ground state with the given number of electrons, and the Bader
charge is evaluated to account for the number of electron accommodated in O2
as shown in Fig. 1(d). When we calculated the system by using the Perdew-
Burke-Ehrenof (PBE)-type functional [43], these added electrons are
fractionally distributed between O2 and the metal layer, as shown in Fig.
1(d). We now consider applying the U potential onto generalized basis,
hereafter abbreviated as DFT+GOU, and it is applied on O2 molecular orbital
states for this example. The resulting number of electrons in O2 exhibit a
sharp step-like increase, which proves the discontinuous jump in the Coulomb
energy of the electrons occupying the O2 rather than long-range partial charge
transfer. Notably, the various U parameters in the conventional scheme
resulted in similar fractional distributions, which clearly evidences our
arguments on the misbehavior of the U applied onto the atomic orbitals.

In general, the U parameters are manually selected so as to reproduce the
known electronic structure. However, a few recent approaches has proposed
schemes to determine the on-site Coulomb ($U$) and the exchange ($J$)
parameters in a self-consistent way [16, 44, 17, 45, 46, 18]. One of well-
known methods was suggested by Agapito, Curtarolo, and Nardelli (ACBN0), in
which the $U$ and $J$ parameters are determined through the theory of screened
Hartree-Fock in the correlated subspace [17, 47, 18]. In our study, hereafter,
the Coulomb parameter calculated by this self-consistent ACBN0 scheme is
denoted by $\bar{U}$, while the manually selected one is noted by $U$. For
example, the computed parameter for O2 molecular orbital in the O2/Cu(100)
system is found to be $\bar{U}\approx$ 7.5 eV. Key elements of the
formulations of the DFT+GOU(ACBN0) are summarized and the revival of the
derivative discontinuity are discussed in the Supplementary Material [48].

Figure 2:  (a) An isosurface of the valence band maximum CDW state overlapped
on the atomic geometry of the monolayer 1T-TaS2 (upper panel), and the same
charge density on the line that passes through the central Ta atom of the
star-of-David (lower panel). Band structure of monolayer 1T-TaS2 calculated by
(b) DFT+U (${U}=0.2$eV) and (c) DFT+GOU (${U}=0.2$eV). (d) The energy gap of
monolayer 1T-TaS2 with various Hubbard U values. Two arrows in orange in the
upper panel indicate the lattice vector of the $\sqrt{13}\times\sqrt{13}$
supercell. The lattice length is $a=12.13$ Å. The line for the charge density
plot is depicted by the thick dashed line in both panels. Red and orange ball
indicate Ta and S atoms, respectively. The blue surface is isosurface of
partial charge density of CDW state. Cyan solid lines are the guide to the eye
for David-star pattern of the CDW. In (b) and (c), red and blue solid lines
indicate spin-up and spin-down states, respectively. The inset of (b)
indicates the Bouillon zone of $\sqrt{13}\times\sqrt{13}$ supercell with high
symmetric points.

The aforementioned scheme is well suited for CDW state of the bulk 1T-TaS2.
Figure 2(a) depicts the commensurate CDW state of the David star pattern in
the $\sqrt{13}\times\sqrt{13}$ supercell. The lobe of charge density indicates
that the CDW state mainly consists of $5d_{z}$ orbitals of the Ta atoms at the
center and six surrounding vertex of the David star, which is also confirmed
by the cross-sectional profile displayed in the lower panel of Fig. 2(a). The
band structures of the monolayer calculated by DFT+U and DFT+GOU are displayed
in Figs. 2(b) and 2(c), respectively, in which the flat band state below and
above the Fermi level correspond to the singly occupied and unoccupied CDW
states, respectively. For the generalized orbital to which the U potential is
applied, we set the KS orbitals of the CDW state. Detailed procedures to
derive the generalized basis orbital from the self-consistently converged
Kohn-Sham states are summarized in the Supplementary Material [48]. The energy
gap between the spin-up valence band and spin-down conduction band are
summarized in Fig. 2(d). The splitting between the upper and the lower Hubbard
bands with respect to various U parameters obviously reveals the difference.
The energy gap calculated by the DFT+GOU increases linearly with the U
parameter, while the spin gap size in the DFT+U is mostly inert over the
increase of the U parameter. Note that the difference in the Hubbard potential
energy between occupied ($n_{CDW}=1$) and unoccupied CDW states ($n_{CDW}=0$)
is exactly $U$, which can be read from
$\hat{V}_{U}=\frac{1}{N_{k}}\sum\limits_{\sigma,\mathbf{k}}{U}(\frac{1}{2}-n_{CDW,\sigma})|\phi_{CDW,\sigma,\mathbf{k}}\rangle\langle\phi_{CDW,\sigma,\mathbf{k}}|$.

Now we focus on the band structure of bulk 1T-TaS2, of which the out-of-plane
conductivity has remained controversial for the last decade. In our study, we
chose the A-stacked bulk 1T-TaS2 as a representative example, in which the Ta
center atom of the David star is on the top of the same atom in the adjacent
layer [49, 29, 30, 31]. We doubled the computational supercell to include two
TaS2 layers to investigate the layer-by-layer spin order. The band structure
of the bulk 1T-TaS2 calculated by the DFT exhibit metallic band with a wide
band width ($\sim 0.35$eV) along the out-plane direction ($\Gamma-$A), as
shown in Fig. 3(a). Furthermore, as depicted in the upper panel of Fig. 3(a),
this metallic band doesn’t produce the spin splitting, and the spin-up and
spin-down charge densities exhibit almost the same pattern.

Figure 3:  Band structure of the bulk 1T-TaS2 calculated by (a) DFT and (b)
DFT+GOU (ACBN0) methods. Unfolded band structure of bulk 1T-TaS2 in the
primitive cell calculated by (c) DFT and (d) DFT+GOU (ACBN0) methods. (e) The
gap at zone boundary (A) and the indirect band gap of the bulk 1T-TaS2 with
respect to the various values of the U parameters. (f) Schematic atomic
geometry for A-stacking and L-stacking of the bulk 1T-TaS2. In (a) and (b),
the red and pink arrows indicate the spin-up and spin-down CDW states,
respectively. Upper panel of (a) and (b) indicate partial charge density of
CDW state for spin-up (left) and spin-down (right) at zone boundary (the A
point). The two adjacent layers are denoted by 1 and 2. The inset of (c)
indicates the Bouillon zone of primitive cell with high symmetric points.

On the other hand, the DFT+GOU scheme reveals an Mott insulating phase of the
bulk 1T-TaS2. In this study, the Hubbard U parameter is self-consistently
determined by using the ACBN0 method ($\bar{U}=0.33$ eV), which produces a
sizable gap opening: the energy gap at the zone boundary A point
($\Delta_{\text{A}}$) is $0.16$ eV and the indirect band gap
($\Delta_{\text{indirect}}$) is $0.06$ eV, which is eigen values difference
between valence band maximum state at A point and conduction band minimum
state at the point of 20% along the path G-K. The experimental observed Mott
gap ($\sim 0.12$ eV) is comparable with our computed values [27]. The CDW band
exhibits clear spin splittings, and the spin-up and spin-down band edges
reside in different layer as depicted by the partial charge densities of the
CDW states shown in the upper panel of Fig. 3(b): the spin-down band in the
first layer and the spin-up band in the next layer are denoted by the two
numbered arrows. As a result, we obtained the layer-by-layer anti-
ferromagnetic Mott insulating phase for the ground state of 1T-TaS2.

For the comparison with experimental observations, we calculated the unfolded-
band structure into the primitive cell of 1T-TaS2 using the BandUP code [50].
As shown in Figs. 3(c) and 3(d), the main difference between DFT and DFT+GOU
methods can be found along the $\bar{\Gamma}$-$\bar{\text{A}}$ line. The
unfolded-band structure evaluated by the DFT+GOU(ACBN0) exhibits an apparent
gap, which is in a good agreement with recent angle-resolved photoemission
spectroscopy measurement in $\bar{\Gamma}$-$\bar{\text{A}}$ line [51, 31]. For
comparison, we tested with various U parameters, and the obtained energy gaps
are summarized in Fig. 3(e): the gaps at the zone boundary at A point
($\Delta_{\text{A}}$) and the band gap indirectly formed at different k-points
($\Delta_{\text{indirect}}$) are displayed with solid and empty symbols,
respectively.

Now we consider different stacking orders and show that the obtained Mott
insulating nature, in the result of DFT+GOU, is independent of the stacking
order. The center and vertex Ta atom of the David-star are denoted as C and V
sites, as shown in the left panel of Fig. 4(f), respectively. In the A-staking
geometry, the center Ta atoms (C site) is exactly on top of the same C site in
the adjacent layer, as depicted in the right upper panel of Fig. 3(f). In the
L-staking geometry, the C site of one layer is on top of the V site of the
other layer, as shown in the right bottom panel of Fig. 3(f). In our self-
consistent DFT+GOU calculations with the ACBN0 functional, as summarized in
Fig. 3(e), both the L-stacked and A-stacked bulk phases exhibit a similar size
of the band gap.

Numerous experimental studies have investigated the conductivity changes near
the phase boundary of the bulk TaS2 [21, 22, 23, 24, 25, 26, 27, 28]. Near the
phase transition (T$\sim 200$ K), not only the in-plane conductivity but also
the out-of-plane conductivity changes drastically [28]. Measurements and
discussions on the spin configurations are diverse [33, 34, 52, 28], but the
insulating nature of the bulk TaS2 is experimentally agreed [21, 22, 23, 24,
28]. A recent first principle calculations suggested that a very particular
double-layer stacking order leads to a band gap without spin ordering [53,
31]. On the other hand, a recent scanning tunneling microscopy and a
transmission electron microscopy revealed that the Ta atoms in different
layers are in good order along the out-of-plane direction [49], and such type
of double-layer order is not likely in practice [54]. Our results of the
DFT+GOU, as summarized in Fig. 3(e), indicates that the bulk preserves the
Mott insulating phase irrespective of the stacking order, which supports all
the experimental results mentioned above and suggest that the 3D insulating
nature should obviously be attributed to the Coulomb correlation in the David
star.

In summary, we re-investigated the Mott insulating nature of bulk 1T-TaS2
which electronic structure evaluated by DFT provides inconsistent results from
previous experimental observations. We showed that the conventional Hubbard U
potential can lead to erroneous evaluation of the electron correlation in KS
states rooted in multiple atoms [55]. By applying the U potential on the
generalized basis (DFT+GOU), we successfully reproduced the Mott insulating
phases of the bulk 1T-TaS2 with the layer-by-layer antiferromagnetic order,
which resolves the decade inconsistent occurred by DFT calculation. The
electron correlation in orbitals localized, but extended over atoms, should be
considered with similar perspective.

$\it{Computational\ Details-}$ We performed DFT calculation using the Quantum
Espresso package [56] with PBE-type functional [43]. The projector-augmented-
wave method is used, and the plane wave basis set with 30 Ry and 60 Ry energy
cut-off is used to describe wavefunction for 1T-TaS2 and O2/Cu(100) system,
respectively. The calculated lattice constant for bulk 1T-TaS2 is $a=3.36$Å
and $c=5.9$Å [57]. The Brillouin zone is sampled with $5\times 5\times 10$
$\mathbf{k}$-point mesh for 1T-TaS2 systems, and the $8\times 8\times 1$
$\mathbf{k}$-point sampling is used for O2/Cu(100) system.

###### Acknowledgements.

We further acknowledge financial support from the European Research Council
(ERC-2015-AdG-694097), the Cluster of Excellence ’CUI: Advanced Imaging of
Matter’ of the Deutsche Forschungsgemeinschaft (DFG) - EXC 2056 - project ID
390715994, Grupos Consolidados (IT1249-19), and SFB925 ”Light induced dynamics
and control of correlated quantum systems”. D.S. is supported by Alexander von
Humboldt Foundation. N.P. is supported by National Research Foundation of
Korea (NRF-2019R1A2C2089332). J.Z. acknowledges funding received from the
European Union Horizon 2020 research and innovation program under Marie
Sklodowska-Curie Grant Agreement 886291 (PeSD-NeSL).

## References

  * Perdew and Levy [1983] J. P. Perdew and M. Levy, Phys. Rev. Lett. 51, 1884 (1983).
  * Kohn and Sham [1965] W. Kohn and L. J. Sham, Phys. Rev. 140, A1133 (1965).
  * Perdew and Zunger [1981] J. P. Perdew and A. Zunger, Phys. Rev. B 23, 5048 (1981).
  * Ceperley and Alder [1980] D. M. Ceperley and B. J. Alder, Phys. Rev. Lett. 45, 566 (1980).
  * Perdew  _et al._ [1996a] J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996a).
  * Perdew and Wang [1992] J. P. Perdew and Y. Wang, Phys. Rev. B 45, 13244 (1992).
  * Janak [1978] J. F. Janak, Phys. Rev. B 18, 7165 (1978).
  * Andrade and Aspuru-Guzik [2011] X. Andrade and A. Aspuru-Guzik, Phys. Rev. Lett. 107, 183002 (2011).
  * Chai and Chen [2013] J. D. Chai and P. T. Chen, Phys. Rev. Lett. 110, 033002 (2013).
  * Anisimov  _et al._ [1991] V. I. Anisimov, J. Zaanen, and O. K. Andersen, Phys. Rev. B 44, 943 (1991).
  * Shin  _et al._ [2016] D. Shin, G. Lee, Y. Miyamoto, and N. Park, J. Chem. Theory Comput. 12, 201 (2016).
  * Shin  _et al._ [2013] D. Shin, H. Jung, S. S. Han, C. H. Choi, H. Lee, and N. Park, Chem. Phys. Lett. 570, 85 (2013).
  * Heyd  _et al._ [2003] J. Heyd, G. E. Scuseria, and M. Ernzerhof, J. Chem. Phys. 118, 8207 (2003).
  * Anisimov  _et al._ [1993] V. I. Anisimov, I. V. Solovyev, M. A. Korotin, M. T. Czyzyk, and G. A. Sawatzky, Phys. Rev. B 48, 16929 (1993).
  * Lichtenstein  _et al._ [1995] A. I. Liechtenstein, V.I. Anisimov, and J. Zaanen, Phys. Rev. B 52, R5467 (1995).
  * Cococcioni and de Gironcoli [2005] M. Cococcioni and S. de Gironcoli, Phys. Rev. B 71, 035105 (2005).
  * Agapito  _et al._ [2015] L. A. Agapito, S. Curtarolo, and M. BuongiornoNardelli, Phys. Rev. X 5, 011006 (2015).
  * Tancogne-Dejean  _et al._ [2018] N. Tancogne-Dejean, M. A. Sentef, and A. Rubio, Phys. Rev. Lett. 121, 97402 (2018).
  * Georges  _et al._ [1996] A. Georges, G. Kotliar, W. Krauth, and M. J. Rozenberg, Rev. Mod. Phys. 68, 13 (1996).
  * Kotliar and Vollhardt [2004] G. Kotliar and D. Vollhardt, Phys. Today 57, 53 (2004).
  * Wilson  _et al._ [1975] J. A. Wilson, F. J. Di Salvo, and S. Mahajan, Adv. Phys. 24, 117 (1975).
  * Fazekas and Tosatti [1980] P. Fazekas and E. Tosatti, Physica B+C 99, 183 (1980).
  * Dardel  _et al._ [1992] B. Dardel, M. Grioni, D. Malterre, P. Weibel, Y. Baer, and F. Lévy, Phys. Rev. B 45, 1462 (1992).
  * Sipos  _et al._ [2008] B. Sipos, A. F. Kusmartseva, A. Akrap, H. Berger, L. Forró, and E. Tutǐ, Nat. Mater. 7, 960 (2008).
  * Han  _et al._ [2015] T. R. T. Han, F. Zhou, C. D. Malliakas, P. M. Duxbury, S. D. Mahanti, M. G. Kanatzidis, and C. Y. Ruan, Sci. Adv. 1, e1400173 (2015).
  * Hellmann  _et al._ [2010] S. Hellmann, M. Beye, C. Sohrt, T. Rohwer, F. Sorgenfrei, H. Redlin, M. Kalläne, M. Marczynski-Bühlow, F. Hennies, M. Bauer, A. Föhlisch, L. Kipp, W. Wurth, and K. Rossnagel, Phys. Rev. Lett. 105, 187401 (2010).
  * Hellmann  _et al._ [2012] S. Hellmann, T. Rohwer, M. Kalläne, K. Hanff, C. Sohrt, A. Stange, A. Carr, M. M. Murnane, H. C. Kapteyn, L. Kipp, M. Bauer, and K. Rossnagel, Nat. Commun. 3, 1069 (2012).
  * Martino  _et al._ [2020] E. Martino, A. Pisoni, L. Ćirić, A. Arakcheeva, H. Berger, A. Akrap, C. Putzke, P. J. Moll, I. Batistić, E. Tutiš, L. Forró, and K. Semeniuk, npj 2D Mater. Appl. 4, 7 (2020).
  * Darancet  _et al._ [2014] P. Darancet, A. J. Millis, and C. A. Marianetti, Phys. Rev. B 90, 045134 (2014).
  * Ritschel  _et al._ [2015] T. Ritschel, J. Trinckauf, K. Koepernik, B. Büchner, M. V. Zimmermann, H. Berger, Y. I. Joe, P. Abbamonte, and J. Geck, Nat. Phys. 11, 328 (2015).
  * Lee  _et al._ [2019] S. H. Lee, J. S. Goh, and D. Cho, Phys. Rev. Lett. 122, 106404 (2019).
  * Lazar  _et al._ [2015] P. Lazar, J. Martincova, and M. Otyepka, Phys. Rev. B 92, 224104 (2015).
  * Perfetti  _et al._ [2005] L. Perfetti, T. A. Gloor, F. Mila, H. Berger, and M. Grioni, Phys. Rev. B 71, 153101 (2005).
  * Perfetti  _et al._ [2006] L. Perfetti, P. A. Loukakos, M. Lisowski, U. Bovensiepen, H. Berger, S. Biermann, P. S. Cornaglia, A. Georges, and M. Wolf, Phys. Rev. Lett. 97, 067402 (2006).
  * Yoshioka  _et al._ [2009] T. Yoshioka, A. Koga, and N. Kawakami, Phys. Rev. Lett. 103, 036401 (2009).
  * Vaskivskyi  _et al._ [2015] I. Vaskivskyi, J. Gospodaric, S. Brazovskii, D. Svetin, P. Sutar, E. Goreshnik, I. A. Mihailovic, T. Mertelj, and D. Mihailovic, Sci. Adv. 1, e1500168 (2015).
  * Yoshida  _et al._ [2017] M. Yoshida, T. Gokuden, R. Suzuki, M. Nakano, and Y. Iwasa, Phys. Rev. B 95, 121405(R) (2017).
  * Yu  _et al._ [2015] Y. Yu, F. Yang, X. F. Lu, Y. J. Yan, Y. H. Cho, L. Ma, X. Niu, S. Kim, Y. W. Son, D. Feng, S. Li, S. W. Cheong, X. H. Chen, and Y. Zhang, Nat. Nanotechnol. 10, 270 (2015).
  * Zhang  _et al._ [2019] J. Zhang, C. Lian, M. Guan, W. Ma, H. Fu, H. Guo, and S. Meng, Nano Lett. 19, 6027 (2019).
  * Strebel and Soos [1970] P. J. Strebel and Z. G. Soos, J. Chem. Phys. 53, 4077 (1970).
  * Chittipeddi  _et al._ [1987] S. Chittipeddi, K. R. Cromack, J. S. Miller, and A. J. Epstein, Phys. Rev. Lett. 58, 2695 (1987).
  * Lechermann  _et al._ [2007] F. Lechermann, A. Georges, G. Kotliar, and O. Parcollet, Phys. Rev. B 76, 155102 (2007).
  * Perdew  _et al._ [1996b] J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996b).
  * Kulik and Marzari [2011] H. J. Kulik and N. Marzari, J. Chem. Phys. 134, 094103 (2011).
  * Mosey  _et al._ [2008] N. J. Mosey, P. Liao, and E. A. Carter, J. Chem. Phys. 129, 014103 (2008).
  * Mosey and Carter [2007] N. J. Mosey and E. A. Carter, Phys. Rev. B 76, 155123 (2007).
  * Tancogne-Dejean and Rubio [2019] N. Tancogne-Dejean and A. Rubio, Phys. Rev. B 102, 155117 (2020).
  * [48] See supplemental material at url.
  * Hovden  _et al._ [2016] R. Hovden, A. W. Tsen, P. Liu, B. H. Savitzky, I. El Baggari, Y. Liu, W. Lu, Y. Sun, P. Kim, A. N. Pasupathy, and L. F. Kourkoutis, Proc. Natl. Acad. Sci. 113, 11420 (2016).
  * Medeiros  _et al._ [2015] P. V. C. Medeiros, S. S. Tsirkin, S. Stafström, and J. Björk, Phys. Rev. B 91, 041116(R) (2015).
  * Ngankeu  _et al._ [2017] A. S. Ngankeu, S. K. Mahatha, K. Guilloy, M. Bianchi, C. E. Sanders, K. Hanff, K. Rossnagel, J. A. Miwa, C. Breth Nielsen, M. Bremholm, and P. Hofmann, Phys. Rev. B 96, 195147 (2017).
  * Law and Lee [2017] K. T. Law and P. A. Lee, Proc. Natl. Acad. Sci. U. S. A. 114, 6996 (2017).
  * Ritschel  _et al._ [2018] T. Ritschel, H. Berger, and J. Geck, Phys. Rev. B 98, 195134 (2018).
  * Butler  _et al._ [2020] C. J. Butler, M. Yoshida, T. Hanaguri, and Y. Iwasa, Nat. Commun. 11, 2477 (2020).
  * Xian  _et al._ [2019] L. Xian, D. M. Kennes, N. Tancogne-Dejean, M. Altarelli, and A. Rubio, Nano Lett. 19, 4934 (2019).
  * Giannozzi  _et al._ [2017] P. Giannozzi _et al._ , J. Phys. Condens. Matter 29 (2017).
  * Givens and Fredericks [1977] F. L. Givens and G. E. Fredericks, J. Phys. Chem. Solids 38, 1363 (1977).


# META-LEARNING FOR IMPROVING RARE WORD RECOGNITION IN END-TO-END ASR

###### Abstract

We propose a new method of generating meaningful embeddings for speech,
changes to four commonly used meta learning approaches to enable them to
perform keyword spotting in continuous signals and an approach of combining
their outcomes into an end-to-end automatic speech recognition system to
improve rare word recognition. We verify the functionality of each of our
three contributions in two experiments exploring their performance for
different amounts of classes (N-way) and examples per class (k-shot) in a few-
shot setting. We find that the speech embeddings work well and the changes to
the meta learning approaches also clearly enable them to perform continuous
signal spotting. Despite the interface between keyword spotting and speech
recognition being very simple, we are able to consistently improve word error
rate by up to 5%.

Index Terms—  meta learning, keyword spotting, speech recognition, speech
embedding

##  1 INTRODUCTION

While end-to-end (E2E) [1] deep learning (DL) models brought great
improvements to the field of automatic speech recognition (ASR) in recent
years and reduced word error rates (WER) on benchmark datasets significantly
[2], they also come with a set of problems. Even though they are designed to
work well with unknown words, since they usually concatenate subword-units and
can thus essentially produce any textual transcription, they tend to perform
even worse on rare words than classical DL ASR systems [3]. This is due to
them having an internal latent language model, which is biased and cannot be
tweaked. For example, an ASR trained on the LibriSpeech corpus is very likely
to transcribe the name Hilde as Hilda or the name Josef as Joseph.

Even though those words are rare, they tend to be of great importance for many
tasks. A good example for this is automatic meeting transcription. People
frequently address each other by name in a meeting. And meeting participants
can have very unique names with unique pronunciations. So for the
transcriptions, the prior knowledge which names to expect could be very
helpful. Incorporating it however is a non-trivial task, because E2E
approaches cannot be fine-tuned easily, since they lack a lexicon or a
pronunciation dictionary.

Proper nouns have been identified as a challenging problem in ASR for a while
now [4]. Recently some approaches have arisen to tackle this challenge with
E2E ASR using a specialised architecture and losses [5] or using specific data
and training procedures to better represent contextual information [6]. Our
approach is meant for rare words in general, however in this work we choose
rare proper nouns as exemplary data and use few-shot learning to improve
performance on them.

In meta learning [7] using the metric space approach (MSML) [8], an embedding
function is trained to transform datapoints into a metric space, where
comparisons purely based on positions are possible. Since structural knowledge
about a task is within the embedding function, new information can be
considered by making comparisons to reference samples of data we want to adapt
to. Essentially clustering is performed in a latent space. MSML could provide
solutions for our goal of improving recognition of rare words in highly
specific contexts, since it can operate with as little as one reference
datapoint of the data we want to adapt to, it doesn’t need any retraining or
time consuming tweaking mechanisms for new reference samples, and the
decisions made on the basis of the comparisons in the metric space are simple
and interpretable and can thus be integrated into an E2E workflow with few
complications.

We propose a two part ASR system that integrates metric space representations
of expected difficult keywords into the E2E pipeline. An ASR system is built
using the Transformer architecture [9] which has been shown to outperform any
prior architecture for most speech related tasks[10, 11, 12, 13] by heavily
relying on attention mechanisms [14]. The intermediate representation that the
Transformer-encoder produces is used as prior embedding and then embedded
further into a metric space on a frame-shifted window basis inspired by
previous work on keyword spotting [15]. Prior approaches of embedding audio
include using a skip-gram model for speech [16] and using convolutional neural
nets (CNNs) for general purpose audio detection [17]. By using the Transformer
as embedding function, we hope to get more phonetically rich embeddings. The
metric space embeddings are then compared to reference samples using various
renown MSML approaches such as Siamese Networks [18], Relation Networks [19],
Prototypical Networks [20] and Matching Networks [21]. The results of these
comparisons are taken into account when decoding the orthographic
representation of the utterance.

Our main contributions are as follows: we propose a new method of generating
meaningful embeddings of spoken language, we redesign MSML approaches to spot
keywords in a continuous signal and we showcase an approach to integrate the
results into an E2E ASR system. We verify our proposed methods on two severely
different datasets and observe the embeddings working well, the keyword
spotting achieving high F1 scores and the keyword spotting ASR interface
improving rare-word WER by 1.6% on average.

##  2 PROPOSED APPROACH

An overview of the entire system we built can be seen in figure 1. The ASR
encoder feeds into the ASR decoder, and also into each of the four MSML agents
in a windowed fashion. The ASR decoder produces hypotheses and the MSML agents
(usually only the one you favor) produce a list of recognized keywords. The
selection of the best hypotheses that the decoder’s beamsearch produces takes
the list of recognized keywords into consideration.

Fig. 1: Overview of how the components are combined

###  2.1 Semantics of Audio

We need to abstract away from speaker, microphone, recording environment and
so on, but retain the phonetic information. To bypass this challenge, we use
the output of the encoder of a Transformer ASR as the input to the metric
space embedding function. The Transformer encoder has been trained on
thousands of hours of speech beforehand and thus abstracts away from surface
level features and is robust against data variations already.

To use the ASR system as embedding function, a feature matrix derived from an
arbitrary audio signal is fed to the encoder of the Transformer, yielding an
embedding matrix with 512 dimensions per timestep in our case. The Transformer
applies attention with respect to the entire input sequence itself, so in
order to get stable embeddings we have to limit the attention span to the size
of our window. This is done by windowing over the signal first and encoding
one appropriately sized chunk at a time using the ASR.

###  2.2 Meta Learning for Spotting

MSML approaches are usually used for classification on clean data, mostly
images. We however want a network that can be applied efficiently to a
sequence of windows of a signal and rather than classifying the correct class,
spot whether something is present or not. This requires some major changes to
the architectures of the four meta learning approaches used.

All of the approaches are redesigned to look at one class at a time. Each
network assigns a similarity score to a pair of query and a member of the
support set of a class. The query is a window of ASR-encoded signal. For each
class, the highest similarity score that any of its supports achieves with any
of the windows is saved. In the end, the score of each class is compared
against a manually set threshold and if it peaks above, it is added to the
list of recognized keywords that is passed on to the ASR decoder.

For the Siamese Network, the support and the query are embedded into the
metric space, and the cosine similarity (CosSim, normalized dot product,
measure how well vectors align) of the resulting vectors is computed. The
scoring mechanism is displayed in equation 1. An encoded matrix M is denoted
as enc(M), the query as $q$, the support as $s$

$\text{score}_{\text{class}}^{q}=\max_{s\in\text{class}}\cfrac{\text{enc}(q)\cdot\text{enc}(s)}{||\text{enc}(q)||\cdot||\text{enc}(s)||}$
(1)

In our Relation Network implementation, the query and a support are again
encoded using the metric space encoder. Here however the similarity function
is learnt, not static. The two vectors are concatenated and passed though a
small multi layer perceptron to get the similarity score. The relation scoring
mechanism can be seen in 2, $relat$ denotes the relation function, $concat$
denotes vector-concatenation.

$\text{score}_{\text{class}}^{q}=\max_{s\in\text{class}}\text{relat}(\text{concat}(\text{enc}(q),\text{enc}(s)))$
(2)

Our Prototypical implementation takes in multiple supports at a time, which
are then averaged after they have been encoded to build a prototype of the
corresponding class, which is then compared to the query using CosSim. For the
case of k=1, it is equivalent to the Siamese Network. The scoring mechanism is
displayed in equation 3, the calculation of the prototype is displayed in
equation 4.

$\displaystyle\text{score}_{\text{class}}^{q}$
$\displaystyle=\cfrac{\text{enc}(q)\cdot\text{enc}(p_{\text{class}})}{||\text{enc}(q)||\cdot||\text{enc}(p_{\text{class}})||}$
(3) $\displaystyle p_{class}$
$\displaystyle=\sum_{s\in\text{class}}\cfrac{s}{|\text{class}|}$ (4)

The Matching Network is usually designed to benefit from fully conditional
encoding functions into a metric space, which rely on attention. For this task
however, this setup did not converge at all. Instead we use again a simple
encoder like the other networks, however with the output being sequences. We
apply attention from the query to the support sequence. The conceptual
matching lies within the properties of the resulting attention matrix, namely
how well the two sequences align. And to get a similarity score from the
attention matrix, we pass it through a small long short-term memory layer
[22]. The matching scoring mechanism is displayed in equation 5, $att$ is an
attention mechanism and $reduce$ is any method to reduce over a sequence axis.

$\text{score}_{\text{class}}^{q}=\max_{s\in\text{class}}\text{reduce}(\text{att}(\text{enc}(q),\text{enc}(s)))$
(5)

###  2.3 Integration with Speech Recognition

The decoder of the ASR produces a set of hypotheses using a beamsearch. One of
them is chosen as the final ASR output. This choice usually just depends on
the ranking that the beamsearch itself assigns to the hypotheses. In our
system, this choice is influenced by a list of recognized keywords that the
MSML agents produce. If a keyword is spotted, hypotheses that contain the word
are moved to the front of the ranking. The spotting at this level is binary.

To illustrate how this works, assume we have a sentence that contains the name
Noirtier. Shown below are the beginnings of the hypotheses that our ASR yields
for the sentence.

  * •

$<$eos$>$ nautier was near the bed …

  * •

$<$eos$>$ natier was near the bed …

  * •

$<$eos$>$ nartier was near the bed …

  * •

$<$eos$>$ noirtier was near the bed …

The correct name appears in the hypotheses, however only as the fourth option.
Now if our keyword spotting system detects that the sentence contains the
name, we pick the most probable hypothesis that contains the desired
transcription, which is the fourth here.

##  3 DATASETS

###  3.1 Speech Commands Dataset

We train the meta learning agents on the Speech Commands dataset [23]. It
contains a total of 35 different words with 1000 to 3000 samples each. To
further enrich the variety of the dataset, we add 160 more names as keywords
with 4 to 6 samples each. Those samples come from three different speech
synthesis systems and random data augmentations are applied to them. The
additional keywords are carefully selected in order to avoid having names in
there that are orthographically different, yet pronounced the same.

###  3.2 LibriSpeech Names

The LibriSpeech corpus [24] is the training set for the Transformer ASR and
the development set of the MSML agents. It contains over 1000 hours of read
speech. For tuning the meta learning agents, a subset is created from the dev-
clean portion. A total of 40 proper nouns are selected, which occur between 4
and 10 times in the whole corpus. For each of them, 4 sentences containing
them are selected. From each of the utterances, the name that occurs in them
is manually cut out and labelled to create a collection of high quality
supports for each name. While we choose proper nouns as exemplary data, what
we really want to explore is the performance of rare words, thus the upper
limit for their occurrence counts.

###  3.3 Singapore English National Speech Names

The Singapore English National Speech Corpus [25] contains over 3000 hours of
speech. The speakers have varying native languages and are recorded with a
variety of microphones in different recording environments. We thus believe
that the small sub-corpus of sentences that contain proper nouns which we
derived from it is well suited to reflect the performance of each of the
components in a challenging close-to-real-world scenario. Our derived corpus
contains 120 utterances which contain 30 proper nouns and 4 utterances per
name. Again, the samples we use as supports are selected and cut manually.

##  4 EXPERIMENTS

###  4.1 Setup

For the audio embeddings, we train a Transformer ASR on the LibriSpeech corpus
using the ESPNet toolkit [26, 27]. The recipe uses 80 Mel-frequency buckets as
the input and the training is capped to 40 epochs, similar to the recipe of
the IMS-Speech system [28]. The training took 96 hours on 4 Titan X GPUs.

The two experiments use the exact same setup, but look at different aspects.
The metric space encoder that is used for all of the MSML agents is built as
follows: Two 1D convolutional layers with a kernel size of 3 and 20 filters
using hyperbolic tangent as activation function are stacked on top of each
other, separated by a maximum pooling layer with a window size of 2. A global
max pooling reduces over the sequence axis. The contrastive loss function [29,
30] is found to work best. RMSProp is used as the optimizer with a learning
rate of 0.001 and a discounting factor of 0.9.

To train the networks, we sample pairs of audio and assign the label 1 if they
are instances of the same word or 0 otherwise. We always alternate between
positive and negative examples, even though we expect mostly negative labels
during inference. We do so for 800,000 steps, even though convergence shows
much earlier, since the MSML networks contain only around 40,000 parameters
each.

For the actual experiment we perform a random sampling, since sampling every
possible combination of queries and supports is not feasible. This means for
any N and k, a random query and k random supports for N classes are selected.
Then all four MSML agents are applied to the same randomized setup, as well as
the unmodified system as baseline in the second experiment. We get one result
for every combination of N, k and agent per run. The random sampling is
performed a total of 10 times and the results are averaged.

###  4.2 Continuous Signal Spotting

The first experiment aims to explore the impact that k and N have on the
performance of the various architectures for spotting keywords in full
utterances. The performance is measured in the F1 score, which is the harmonic
mean of precision and recall.

Fig. 2: Spotting F1 scores across different N and agents for k=1

Fig. 3: Spotting F1 scores across different N and agents for k=4

The results on the test set (as seen in figures 2 and 3) show Matching and
Relation Networks achieving the highest scores. Siamese Networks only take the
top spot when given enough supports for each task. The drop in precision and
thus in F1 score is severe for an increase in N, however considering that the
random baseline decreases from 0.5 down to 0.032, the results are still well
above chance. Prototypical Networks do not seem to perform too well overall
for this kind of task on the test set, which we suspect is due to some
supports disturbing the integrity of the prototype.

###  4.3 Speech Recognition Integration

N = | 1 | 5 | 10 | 15 | 20 | 25 | 30
---|---|---|---|---|---|---|---
k = 1 |
Siamese | 12.2 | 9.1 | 10.2 | 12.6 | 9.3 | 20.1 | 13.3
Relation | 12.2 | 9.1 | 10.2 | 12.6 | 9.3 | 20.1 | 13.3
Proto | 12.2 | 9.1 | 10.2 | 12.6 | 9.3 | 20.1 | 13.3
Matching | 12.2 | 9.1 | 10.2 | 12.6 | 9.3 | 21.0 | 14.3
Vanilla | 16.4 | 10.9 | 12.4 | 12.6 | 9.3 | 21.0 | 13.3
k = 4 |
Siamese | 11.7 | 8.8 | 10.1 | 14.2 | 6.9 | 13.2 | 6.1
Relation | 11.7 | 8.8 | 10.1 | 14.2 | 6.9 | 13.2 | 6.1
Proto | 11.7 | 8.8 | 10.1 | 14.2 | 6.9 | 13.2 | 6.1
Matching | 11.7 | 9.7 | 10.1 | 13.4 | 10.0 | 13.2 | 8.6
Vanilla | 15.2 | 12.4 | 15.1 | 14.2 | 6.9 | 15.3 | 6.1
Table 1: Word Error Rates in % across N, k and approaches

The second experiment focuses on the ASR performance and the impact that the
meta learning agents’ results have on it. This is measured using the WER of
the ASR with and without the meta learning agents as baseline (called Vanilla
here). Inconsistencies in the Vanilla WERs originate from the randomized setup
to ensure comparability of the approaches within each N and k. The WER in this
evaluation is only counting mistakes that originate from the keywords. All
other mistakes are ignored, as the meta learning agents have no effect on
them. The reranking in this setup considers all of the hypotheses that the
beam search of the decoder (beamsize = 4) produces

As can be seen in table 1, the meta learning agents help the ASR slightly in
most cases for any N and k. Interestingly there is very little difference
between the meta learning agents. One more remarkable thing is that the
Matching Networks seem to be the only cases where the WER actually increases
over the vanilla system with no augmentations. This is probably due to them
having low precision, but very high recall in the first experiment, so a lot
of false positives.

##  5 DISCUSSION

Impact of Transformer based Embeddings To begin we want to address insights
into the usefulness of the Transformer encoder as embedding function. The
results on the development set and the results on the test set for both
experiments are remarkably close, even though the data is severely different.
This good performance across corpora hints at the Transformer ASR embeddings
working as intended. In comparison, when the same MSML nets are trained on Mel
frequency buckets directly, the performance across corpora shows a massive
drop from development to test set.

The Spotting Process To provide some insight into the spotting process over a
whole signal, figure 4 shows the Siamese similarity score given a support of
the word Prometheus over time. At the point in the sentence where the keyword
actually appears, the score peaks over 0.95. In most longer signals however,
there are random peaks aside from this main peak. Looking at the
transcriptions, unfortunately no specific reason becomes apparent. The margin
between an actual peak and a random peak across signals is however big enough
for a consistent threshold to be set (we choose 0.8).

Fig. 4: Siamese similarity score over frames of an utterance given one support
for the keyword Prometheus

Impact of N, k and Support Quality on MSML Agents Next we want to give some
insights into cases of MSML agent performance which we find interesting. The
MSML agents outperform the random baseline by a huge margin for any N and k,
with the performance decreasing for large N. The Relation Network manages to
perform well even with few supports and surprisingly even in cases where the
only support it is given is pronounced slightly wrong. The Prototypical and
Siamese Networks greatly benefit from more supports, however the prototype is
sometimes disrupted by heavily mispronounced names in the supports. This is
not the case in the development set, where the Siamese and the Prototypical
approach actually yield the best results, together with the Relation approach.
So robustness against imperfect supports appears to be important for close-to-
real-world scenarios.

Impact of imperfect Speech on ASR One interesting insight about the keyword
spotting ASR interface are the cases where the regular ASR tends to make
mistakes. Looking into the data and the hypotheses of the ASR, we find that
usually the most probable ASR hypothesis matches exactly what the speaker of
the utterance articulated. This is however not always the desired
transcription, since speakers tend to be sloppy, especially with names, or
mispronounce them entirely.

Impact of the Keyword Spotting Integration Another interesting observation can
be made with the cases where the hypothesis was not changed, despite the MSML
agents correctly spotting a keyword. This happens quite frequently for names
that follow phonotactics that are very different from English, since they do
not appear in the hypotheses at all and could thus also not be reranked. At
the same time this is also a good thing, since especially the Matching Network
often spotted keywords which were not present. The hypothesis does not change
then, because the word in question simply does not appear in the selection. So
the ASR hypotheses act as a filter to the MSML agents and the MSML agents act
as a filter to the ASR hypotheses.

##  6 Conclusions

In this paper we introduce the idea of using a Transformer ASR encoder as an
embedding function, which enables stable good performance across severely
different corpora. We also discuss a setup that enables four commonly used
MSML approaches to perform keyword recognition on arbitrary reference samples
in continuous signals. Finally we introduce a system that combines ASR with
keyword spotting to enable modular and simple fine-tuning of neural E2E ASR to
expected vocabulary. Experiments show a keyword recognition performance much
higher than chance across different setups and an improvement in WER,
indicating that the fine-tuning works despite the mechanism being a rather
simple proof of concept.

## References

  * [1] Alex Graves and Navdeep Jaitly,  “Towards end-to-end speech recognition with recurrent neural networks,”  in International conference on machine learning, 2014, pp. 1764–1772.
  * [2] Jinyu Li, Yu Wu, Yashesh Gaur, Chengyi Wang, Rui Zhao, and Shujie Liu,  “On the comparison of popular end-to-end models for large scale speech recognition,” 2020.
  * [3] Jinxi Guo, Tara Sainath, and Ron Weiss,  “A spelling correction model for end-to-end speech recognition,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
  * [4] Antoine Laurent, Sylvain Meignier, and Paul Deléglise,  “Improving recognition of proper nouns in ASR through generating and filtering phonetic transcriptions,”  Computer Speech & Language, vol. 28, 07 2014.
  * [5] Cal Peyser, Tara N Sainath, and Golan Pundak,  “Improving proper noun recognition in end-to-end asr by customization of the mwer loss criterion,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7789–7793.
  * [6] Uri Alon, Golan Pundak, and Tara N Sainath,  “Contextual speech recognition with difficult negative training examples,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6440–6444.
  * [7] Jürgen Schmidhuber,  Evolutionary principles in self-referential learning, or on learning how to learn,  Ph.D. thesis, Technische Universität München, 1987.
  * [8] J. Goldberger, S. Roweis, Geoffrey E. Hinton, and R. Salakhutdinov,  “Neighbourhood Components Analysis,”  in NIPS, 2004.
  * [9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Advances in neural information processing systems, 2017, pp. 5998–6008.
  * [10] Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al.,  “A comparative study on transformer vs rnn in speech applications,”  in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 449–456.
  * [11] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu,  “Neural speech synthesis with Transformer network,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2019, vol. 33, pp. 6706–6713.
  * [12] Laura Cross Vila, Carlos Escolano, José AR Fonollosa, and Marta R Costa-jussà,  “End-to-End Speech Translation with the Transformer,”  in IberSPEECH, 2018, pp. 60–63.
  * [13] Linhao Dong, Shuang Xu, and Bo Xu,  “Speech-Transformer: a no-recurrence sequence-to-sequence model for speech recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5884–5888.
  * [14] Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio,  “Neural machine translation by jointly learning to align and translate,”  in 3rd International Conference on Learning Representations, ICLR 2015, 2015.
  * [15] Guoguo Chen, Carolina Parada, and Georg Heigold,  “Small-footprint keyword spotting using deep neural networks,”  in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 4087–4091.
  * [16] Yu-An Chung and James Glass,  “Speech2vec: A sequence-to-sequence framework for learning word embeddings from speech,”  Proc. Interspeech 2018, pp. 811–815, 2018.
  * [17] IBM,  “MAX-Audio-Embedding-Generator,” Sept. 2019.
  * [18] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah,  “Signature verification using a ”siamese” time delay neural network,”  in Advances in neural information processing systems, 1994, pp. 737–744.
  * [19] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales,  “Learning to compare: Relation network for few-shot learning,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1199–1208.
  * [20] Jake Snell, Kevin Swersky, and Richard Zemel,  “Prototypical networks for few-shot learning,”  in Advances in neural information processing systems, 2017, pp. 4077–4087.
  * [21] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.,  “Matching networks for one shot learning,”  in Advances in neural information processing systems, 2016, pp. 3630–3638.
  * [22] Sepp Hochreiter and Jürgen Schmidhuber,  “Long short-term memory,”  Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  * [23] Pete Warden,  “Speech commands: A dataset for limited-vocabulary speech recognition,”  arXiv e-prints, pp. arXiv–1804, 2018.
  * [24] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5206–5210.
  * [25] Jia Xin Koh, Aqilah Mislan, Kevin Khoo, Brian Ang, Wilson Ang, Charmaine Ng, and YY Tan,  “Building the singapore english national speech corpus,”  Malay, vol. 20, no. 25.0, pp. 19–3, 2019.
  * [26] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai,  “ESPnet: End-to-End Speech Processing Toolkit,”  in Interspeech, 2018, pp. 2207–2211.
  * [27] Tomoki Hayashi, Ryuichi Yamamoto, Katsuki Inoue, Takenori Yoshimura, Shinji Watanabe, Tomoki Toda, Kazuya Takeda, Yu Zhang, and Xu Tan,  “Espnet-tts: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7654–7658.
  * [28] Pavel Denisov and Ngoc Thang Vu,  “IMS-speech: A speech to text tool,”  Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2019, pp. 170–177, 2019.
  * [29] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang,  “A tutorial on energy-based learning,”  Predicting structured data, vol. 1, no. 0, 2006.
  * [30] Raia Hadsell, Sumit Chopra, and Yann LeCun,  “Dimensionality reduction by learning an invariant mapping,”  in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). IEEE, 2006, vol. 2, pp. 1735–1742.