# Nonlocal inelastic scattering of light: Enhanced and noiseless signals in
remote-coupled optomechanical systems

Sho Tamaki  Department of Materials Engineering Science, Osaka University,
Toyonaka, Osaka 560-8531, Japan  Current address: Niels Bohr Institute,
University of Copenhagen, Copenhagen, Denmark  Tomohiro Yokoyama
<EMAIL_ADDRESS>Department of Materials Engineering
Science, Osaka University, Toyonaka, Osaka 560-8531, Japan  Hajime Ishihara
Department of Materials Engineering Science, Osaka University, Toyonaka, Osaka
560-8531, Japan  Department of Physics and Electronics, Osaka Prefecture
University, Sakai, Osaka 599-8531, Japan  Center for Quantum Information and
Quantum Biology, Osaka University, Osaka, Japan

###### Abstract

The inelastic scatterings of matter systems, such as Raman scattering, contain
rich information on mechanical vibrations like as resonant frequencies, which
lead to various applications, for example, a sensor for specific molecules.
However, observing output signals requires a sensitive setup because an
inelastic signal is inherently weak and is disturbed by strong input. In this
study, we theoretically investigate a physical scheme to avoid detrimental
impact of the input by distancing it from the emitter and greatly enhancing
the output signals. If two bodies are coupled mechanically and direct optical
communication is forbidden, the nonlocal inelastic scattering signals can be
considerably boosted. We demonstrate this mechanism by considering coupled
optomechanical systems as a typical example that enables control of the two-
body interaction strength. The results present a general scheme to boost
nonlocal inelastic scattering for noiseless and pure signals.

###### pacs:

##  I Introduction

Light-matter interaction causes inelastic scattering of light accompanied by
the generation of mechanical vibration, namely Stokes and anti-Stokes
scatterings. The energy shift of photons owing to such scattering is an
integer multiple of the mechanical vibration frequency. The Stokes shift owing
to the mechanical vibration of molecules provides rich information, e.g.,
chemical bonds, crystallinities, and concentrations of solutions Pettinger
_et al._ (2012); Stiles  _et al._ (2008). Hence, it is an essential tool for
spectroscopy. However, there are bottlenecks for observing inelastic signals,
namely, the signals are inherently weak; therefore, they require a physical
mechanism to enhance them. In surface- and tip-enhanced Raman scattering
microscopes, the electric field is strongly enhanced by the localized surface
plasmon resonances that are sustained near the metallic nanostructures, which
enhances the inelastic scattering at a molecule Nie and Emory (1997); Kneipp
_et al._ (1997). These spectroscopies have successfully observed signals from
single molecules using nanoscale gaps between the apexes of metallic
structures. However, there are still existing issues, namely, a stronger
electromagnetic field causes larger and broader noise, which might disturb
sensitive detection, owing to the masking of the minute signals of inelastic
scattering.

The purpose of this letter is to theoretically propose a nonlocal inelastic-
scattering scheme that generates an enhanced and noiseless signal; thus, it is
free from the above-mentioned bottleneck. This scheme spatially and
energetically separates the signals of inelastic scattering from an intensive
input laser, which is necessary for strong light-matter interaction; however,
it simultaneously generates large noise. For the theoretical demonstration, we
used an artificial quantum system, the optomechanical system (OMS) Law (1995);
Aspelmeyer  _et al._ (2014), as an example of a system that exhibits Stokes
and anti-Stokes scattering, where the light-matter interaction is attributed
to an optical force caused by the enhanced light in a cavity. Several
parameters of the OMS, for example, mechanical frequency, the strength of the
light-matter interaction, and damping constants, can be designed; therefore,
the full merit of the proposed scheme is well demonstrated by this system.
OMSs enable the detection and monitoring of mechanical motion via the
inelastic scattering of light, which can be applied to higher resolution
sensors and efficient information transport Andrews  _et al._ (2014); Forsch
_et al._ (2020); Jiang  _et al._ (2020). However, stronger field incidence
causes larger and broader noise, which might disturb sensitive detection at
the local OMS, owing to the masking of the minute signal of inelastic
scattering.

To separate the output inelastic signals from the input, we consider remotely
and mechanically coupled OMSs, for which an optical coupling between the OMSs
is absent. A mechanical coupling between OMSs has been experimentally examined
to synchronize their mechanical vibrations Colombano  _et al._ (2019) and
achieve mechanical sensing and transduction Letizia Catalini and Schliesser
(2020). In theoretical studies, the mechanical coupling of OMSs has
demonstrated a great potential to achieve many-body interactions, which cause
the phase transition Ludwig and Marquardt (2013) and an emergence of
topological nature Peano  _et al._ (2015). However, the nonlocal response
through mechanical coupling has not been studied thus far. In our system, one
OMS is a controller under a two-tone laser irradiation, and the other is a
target under a monotone laser. (see Fig. 1(a).) We observe the output signals
at the target OMS (t-OMS), where nonlocal inelastic scattering is induced by
an interplay between the monotone input and phonons coming from the controller
OMS (c-OMS). This scheme separates the inelastic signal from noise and heat,
and it might offer long coherence between the photon and phonon because of the
suppression of the heating noise due to two-photon absorption Grutter  _et
al._ (2015); Schneider  _et al._ (2019). This is a critical advantage for
various applications and the measurement of pure inelastic scattering. Here,
we wish to remark that the application of our proposed scheme is not limited
to OMSs, but it includes the general concept of generating nonlocal inelastic
signals, thereby reducing detrimental impact of the strong input.

As depicted in Fig. 1(b), the cavity of each OMS has a single optical mode
described by $\hat{a}_{\alpha},\hat{a}_{\alpha}^{\dagger}$. Here, the label
$\alpha=\mathrm{c,t}$ denotes the c- and t-OMS, respectively. The optical mode
is coupled with the mechanical vibration of the cavity mirror of the OMS or a
specific OMS structure by the optical pressure in the cavity and a shift of
the resonant light frequency owing to the position modulation. The mechanical
vibration is described by $\hat{b}_{\alpha},\hat{b}_{\alpha}^{\dagger}$. Then,
the Hamiltonian of the OMSs is written as follows Law (1995); Aspelmeyer  _et
al._ (2014):

$\displaystyle\hat{H}_{\mathrm{OMS}}$ $\displaystyle=$
$\displaystyle\sum_{\alpha={\rm
c,t}}\left[\left(\hbar\omega_{\alpha}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}+\hbar\Omega_{\alpha}\hat{b}_{\alpha}^{\dagger}\hat{b}_{\alpha}\right)\right.$
(1) $\displaystyle-$ $\displaystyle\left.\hbar
g_{\alpha}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}\left(\hat{b}_{\alpha}^{\dagger}+\hat{b}_{\alpha}\right)\right].$

Here, $\omega_{\alpha}$ and $\Omega_{\alpha}$ are the eigenfrequencies of the
optical and mechanical modes, respectively. $g_{\alpha}$ is the coupling
strength of the optical mode and the mechanical vibration determined by the
OMS structure. We use the unit of $\hbar=1$ for simplicity in the following.
In this study, we consider identical OMSs, $\omega_{\rm c}=\omega_{\rm
t}\equiv\omega_{0}$, $\Omega_{\rm c}=\Omega_{\rm t}\equiv\Omega_{0}$, and
$g_{\rm c}=g_{\rm t}\equiv g_{0}$, to highlight the significant mechanism.

Figure 1:  (a) Image of mechanically remote-coupled OMSs. The c- and t-OMSs
consist of one optical mode ($\hat{a}_{\alpha}$) and one mechanical vibration
mode ($\hat{b}_{\alpha}$) Here, $\alpha={\rm c,t}$. The phonons of the
mechanical vibration modes in the OMSs are coupled to each other via a
substrate, whereas the respective optical cavities are isolated. Two-tone and
monotone incident laser beams were applied to the c-and t-OMSs, respectively.
(b) Schematic summary of the parameters in the proposed system. Colored
parameters indicate the control parameters in this study.

The mechanical coupling between the OMSs via their substrate is given as
$\hat{H}_{\mathrm{coup}}=-K\left(\hat{b}_{\rm t}^{\dagger}\hat{b}_{\rm
c}+\hat{b}^{\dagger}_{\rm c}\hat{b}_{\rm t}\right)$ Ludwig and Marquardt
(2013); Peano  _et al._ (2015), where the coupling strength $K$ relates to the
phonon transport in the substrate. The strength $K$ is designed by the system
fabrication. In some recent fabrications Colombano  _et al._ (2019); Letizia
Catalini and Schliesser (2020), a mechanical coupling strength up to
$K\lesssim\Omega_{0}$ has been achieved. Note again that we assume that the
OMSs do not communicate optically with each other.

In the proposed system, we apply a two-tone laser to the c-OMS with the
resonant frequency of the optical mode, $\omega_{0}$, and the blue- or red-
detuned frequency by the phonon energy, $\omega_{0}\pm\Omega_{0}$, while the
t-OMS is irradiated by a monochromatic laser. We describe the case of
$\omega_{0}$ and $\omega_{0}+\Omega_{0}$ irradiation on the c-OMS and
$\omega_{0}+\Omega_{0}$ irradiation on the t-OMS. The Hamiltonian for the
input lasers is as follows:

$\displaystyle\hat{H}_{\mathrm{drive}}$ $\displaystyle=\left[i\left(E_{{\rm
c},0}e^{-i\omega_{0}t}+E_{{\rm
c},+}e^{-i(\omega_{0}+\Omega_{0})t}\right)\hat{a}_{\rm c}\right.$ (2)
$\displaystyle+\left.iE_{{\rm t},+}e^{-i(\omega_{0}+\Omega_{0})t}\hat{a}_{\rm
t}\right]+\mathrm{h.c.},$

where $E_{\alpha,0}$ and $E_{\alpha,+}$ are the strengths of the resonant and
blue-detuned coherent driving fields, respectively. These two plus one laser
inputs are intended as a typical situation to generate phonons in the c-OMS
and induce the Stokes scattering in the t-OMS by the generated phonons. If we
consider other situations for the laser incidence, the laser amplitude in Eq.
(2) is replaced such as $E_{{\rm t},+}e^{-i(\omega_{0}+\Omega_{0})t}\to
E_{{\rm t},-}e^{-i(\omega_{0}-\Omega_{0})t}$.

The photons of the optical mode and the phonons of the mechanical vibration in
the OMSs relax to the environment. We introduce operators $\hat{c}_{\alpha}$
and $\hat{d}_{\alpha}$ to describe the relaxation of the photons and phonons,
respectively. The Hamiltonian of the environment with the interaction is

$\displaystyle\hat{H}_{\mathrm{env}}$ $\displaystyle=\sum_{\alpha={\rm
c,t}}\left[\int d\omega\ \omega\hat{c}_{\alpha}^{\dagger}\hat{c}_{\alpha}+\int
d\omega\
i\sqrt{\frac{\kappa}{2\pi}}\left(\hat{a}_{\alpha}^{\dagger}\hat{c}_{\alpha}+\hat{c}_{\alpha}^{\dagger}\hat{a}_{\alpha}\right)\right]$
(3) $\displaystyle+$ $\displaystyle\sum_{\alpha={\rm c,t}}\left[\int d\Omega\
\Omega\hat{d}_{\alpha}^{\dagger}\hat{d}_{\alpha}+\int d\Omega\
i\sqrt{\frac{\gamma}{2\pi}}\left(\hat{b}_{\alpha}^{\dagger}\hat{d}_{\alpha}+\hat{d}_{\alpha}^{\dagger}\hat{b}_{\alpha}\right)\right],$

where $\kappa$ and $\gamma$ are the damping constants of the optical and
mechanical vibration modes, respectively.

The entire Hamiltonian is
$\hat{H}=\hat{H}_{\mathrm{sys}}+\hat{H}_{\mathrm{env}}$ with
$\hat{H}_{\mathrm{sys}}=\hat{H}_{\mathrm{OMS}}+\hat{H}_{\mathrm{coup}}+\hat{H}_{\mathrm{drive}}$.
In this model, the photon in the t-OMS is indirectly coupled with the photon
of the control OMS via the mechanical vibrations of the substrate. The
environment photons (phonons) described by $\hat{c}_{\alpha}^{\dagger}$
($\hat{d}_{\alpha}^{\dagger}$) can enter the $\alpha$ OMS. However, we assume
that the $c$-photon ($d$-phonon) is absent in the initial state, and the
relaxed photons (phonons) from the OMSs immediately move away. Moreover, they
do not go into the other OMSs. This assumption corresponds to a low-noise
environment at low temperature. Then, we formulate the system
$\hat{H}_{\mathrm{sys}}$ and treat the influence of the environment
$\hat{H}_{\mathrm{env}}$ only as the relaxation in the Heisenberg equations
(in Sec. III of Supplementary Materials).

We assume conventional cavity-quantum-electrodynamic (QED) systems to build up
the present mechanically coupled OMSs. The parameters for the energies of the
cavity and mechanical vibration modes, coupling strengths, and damping
constants are typically $\omega_{0}\sim 200\,\mathrm{THz}$ ($\lambda\sim
1550\,\mathrm{nm}$), $\Omega_{0}\sim 1\,\mathrm{MHz}-1\,\mathrm{GHz}$,
$g_{0}/\Omega_{0}\sim 10^{-4}$, $\kappa/\Omega_{0}\sim 0.1$,
$\gamma/\Omega_{0}\sim 10^{-4}$, and $K/\Omega_{0}\sim 10^{-3}-1$ Colombano
_et al._ (2019). Hence, the cooperativity $C\equiv 4g_{0}^{2}/(\gamma\kappa)$
and multiphoton cooperativity $C_{\mathrm{m}}\equiv\bar{n}_{\mathrm{cav}}C$,
which are important indices as a figure of merit of the QED systems Clerk  _et
al._ (2020); Aspelmeyer  _et al._ (2014), are much smaller than unity.
$\bar{n}_{\mathrm{cav}}$ is the number of cavity photons. It is worth noting
that the proposed system provides detectable and noiseless Stokes signals,
even for $C_{\mathrm{m}}\ll 1$ at the respective OMSs.

First, we observe the favorable condition for the mechanical components
$\gamma$ and $K$ in a typical parameter regime of
$\omega_{0}\gg\Omega_{0}>\kappa\gg g_{0},\gamma,K$. We present the derivation
and a detailed discussion in Sec. I of the Supplementary Materials; therefore,
we only show the result here. Under a steady-state condition, the present
system can be analytically solved, and we obtain the explicit expression of
the expectation value that explains the enhancement of the Stokes scattering
component (Eq. (1) in Sec. I of the Supplementary Materials). Examining this
expression, we know that the optimal condition is given by $K\sim\gamma/2$
(see Fig. 1 in the Supplementary Materials). This result attributes to that an
increase in $K$ enhances the inelastic signal, while a strong coupling of $K$
does not contribute to the enhancement of $\left|\braket{\hat{a}_{{\rm
t},0}}\right|$ owing to the energy level splitting of the phonons by $K$ (see
Sec. I in the Supplementary Materials).

We discuss the efficiency of the enhancements by numerically calculating the
output power spectrum at the t-OMS under the above-mentioned optimal
condition. For the numerical calculation, the system Hamiltonian
$\hat{H}_{\mathrm{sys}}$ is transferred by a unitary operator,
$\hat{U}_{2}=\exp\left[i(\omega_{0}+\Omega_{0})\left(\sum_{\alpha}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}\right)t\right]$
(for details, see Secs. II and III in the Supplementary Materials).

The output power spectrum of the optical mode in each OMS is evaluated using
the Fourier transformation of the correlation function:

$\mathcal{S}_{\alpha}(\omega)=\int\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}e^{-i\omega
t}dt.$ (4)

$\braket{\hat{\mathcal{O}}(0)}$ denotes the time-averaged steady-state mean
value of an arbitrary system operator $\hat{\mathcal{O}}(t)$. This spectrum
should be evaluated from the steady state (see Sec. V in the Supplementary
Materials).

Figure 2:  (a) Normalized output power spectra $|\tilde{\mathcal{S}}_{\rm
t}(\omega)|$ of the target photons for various conditions of the input laser
beams at the c-OMS. A monotone laser applied to the t-OMS is fixed at
$\omega=\omega_{0}+\Omega_{0}$. The system parameters are as follows if the
parameters are not designated: $E_{\alpha,\xi}/\Omega_{0}=1.0$ ($\alpha={\rm
c,t}$ for the OMSs, $\xi=0,+$), $\kappa/\Omega_{0}=0.23$,
$g_{0}/\Omega_{0}=2.4\times 10^{-4}$, $K/\Omega_{0}=2.35\times 10^{-4}$, and
$\gamma/\Omega_{0}=4.7\times 10^{-4}$. The red, green, orange, and blue lines
indicate the cases of no mechanical coupling ($K=0$), a monotone blue-detuned
$\omega_{0}+\Omega_{0}$ input, a resonant $\omega_{0}$ input, and a two-tone
($\omega_{0}$ and $\omega_{0}+\Omega_{0}$) input at the c-OMS, respectively.
The plotted curves are displayed using vertical shifts by 3 units to assist
clear observation. For the green line, the input laser power at the c-OMS is
stronger ($E_{{\rm c},+}/\Omega_{0}=4.0$) to enhance phonon generation. All
spectra are normalized by their respective maximum value. (b,c) Enlarged views
of the Stokes scattering signals for the respective cases. The line colors
correspond to those in panel (a). The vertical shift is not applied to (b) and
(c). The thin black line in (b) indicates a case of the resonant $\omega_{0}$
input and $K=0$, which has a main peak at $\omega=\omega_{0}$; hence, the plot
is shifted by $+1$ in the horizontal direction.

The evaluated output power spectrum at the t-OMS is shown in Fig. 2 for
several input situations. Here, we consider four cases: no mechanical coupling
($K=0$), a monotone blue-detuned $\omega_{0}+\Omega_{0}$ strong input, a
resonant $\omega_{0}$ input, and a two-tone ($\omega_{0}$ and
$\omega_{0}+\Omega_{0}$) input at the c-OMS. At the t-OMS, a blue-detuned
monotone laser is applied in all cases. In Fig. 2(a), the spectrum exhibits
sharp signals at $\omega\approx\omega_{0}+(n+1)\Omega_{0}$ ($n$ is an
integer). Note that each spectrum is normalized by its maximum value, and the
lines are plotted with vertical shifts. The largest peaks for all the spectra
at $\omega=\omega_{0}+\Omega_{0}$ are attributed to the input laser
frequencies. When the OMSs are disconnected ($K=0$), the mechanical vibration
at the t-OMS is induced only by the monotone input at the t-OMS. Then, the
inelastic-scattering signal induced by locally generated phonons is almost
invisibly weak. Figure 2(b) displays an enlarged view that exhibits the weak
signals at $\omega\approx\omega_{0}$. The peak-and-dip height from a baseline
output due to the $\omega_{0}+\Omega_{0}$ input signal is of the order of
$0.001$ in the figure. Note that the plot in Fig. 2 is in the logarithmic
scale. Hence, a measurement of this Stokes signal requires an extremely high-
quality OMS. At $\omega\approx\omega_{0}+2\Omega_{0}$, an anti-Stokes
component might be present. However, the signal is not identified in the
spectrum. For an incident resonant laser ($\omega=\omega_{0}$) at $K=0$ (see
the black line in Fig. 2(b)), the main peak produced by the incident laser is
especially large, and the Stokes and anti-Stokes signals at
$\omega\approx\omega_{0}\mp\Omega_{0}$, are completely masked by the baseline,
owing to the input (no peak in the black line). Hence, the blue- or red-
detuned laser incidence is appropriate to observe the inelastic signals.

When $K\neq 0$, the phonons generated at the c-OMS contribute to inelastic
scattering at the t-OMS. However, if the input at the c-OMS is monotone, the
inelastic scattering signal at the t-OMS is negligible even if the input is
four times stronger than the blue-detuned input (green) or the resonant
frequency input (orange lines) at the c-OMS. Thus, phonon generation is
ineffective under a monotone laser incidence. In Fig. 2(b), the Stokes
scattering signal for the resonant monotone input at the c-OMS (orange) is
larger than that for the blue-detuned monotone input (green). This indicates
that phonon generation is efficient when the input laser is resonant although
local Stokes and anti-Stokes signals by the locally generated phonon ($K=0$
case) are masked when the input is resonant. Therefore, the present nonlocal
scheme is incredibly useful for detecting phonon generation and inelastic
scattering because it avoids noise caused by laser incidence.

We discuss the drastic enhancement of the phonon generation and inelastic
scatterings beyond those caused by the monotone input mentioned above. The key
ingredient is a two-tone input laser applied only to the c-OMS. We examine the
$\omega_{0}$ and $\omega_{0}+\Omega_{0}$ two-tone input (blue line), as shown
in Fig. 2(a) and (c). The two-tone input “activates” the c-OMS and nonlocally
results in enormously enlarged inelastic scattering signals at the t-OMS. The
Stokes scattering at $\omega=\omega_{0}$ ($n=-1$) is significantly enlarged,
owing to the optical cavity mode. The enhancement from the baseline due to the
input is approximately $10^{3}$ (see Fig. 2(c)). Here, we consider a signal-
to-noise (SN) ratio. The intensity of the noise caused by the input laser is
approximated to be of the same order as that of the output spectrum, owing to
the baseline of the input laser. Then, the ratio between the peak height and
the baseline gives a qualitative estimation of the SN ratio. Let us call this
the $\eta$-factor, $\eta\equiv({\rm peak\ height})/({\rm based\ line})$. The
$\eta$-factor for the local Stokes scattering induced by the blue-detuned
laser with $K=0$ (red) is $\eta\approx 10^{-4.8845}/10^{-4.8860}=10^{0.0015}$
from Fig. 2(b). For the nonlocal Stokes scattering by the resonant monotone
laser (orange), $\eta\approx 10^{0.014}$. This suggests the important insight
that a nonlocal signal would result in a large SN ratio with noise
suppression. However, the enhancement is still weak. In our proposed scheme
under the two-tone laser, the nonlocal Stokes signal indicates $\eta\approx
10^{2.75}$ (blue line), which is considerably larger than that in other
situations. In addition to enhancement (peak), the spectrum indicates
suppression (dip). Such a peak-dip spectrum is attributed to the Fano
resonance between the mechanical vibration mode and the baseline spectrum of
the photons in the cavity.

Figure 3:  Normalized output power spectra $|\tilde{\mathcal{S}}_{\rm
t}(\omega)|$ of the target photons when two-tone and monotone lasers are
applied to the c- and t-OMSs, respectively. For the efficient generation of
phonons, $\omega_{0}$ and $\omega_{0}+\Omega_{0}$ two-tone input is considered
with a monotone input of $\omega=\omega_{0}+2\Omega_{0}$ (orange),
$\omega_{0}+\Omega_{0}$ (purple), $\omega_{0}$ (green), and
$\omega_{0}-\Omega_{0}$ (red lines) at the t-OMS. A case of
$\omega_{0}-\Omega_{0}$ and $\omega_{0}+\Omega_{0}$ input at the c-OMS and
$\omega_{0}+\Omega_{0}$ input at the t-OMS is plotted (blue line). The
respective curves are plotted with lateral offsets of 3 units to assist with
clear observation. The other parameters are the same as those in Fig. 2.
Hence, the purple line is identical to the blue line in Fig. 2. A case of
$\omega_{0}-(1/2)\Omega_{0}$ and $\omega_{0}+(1/2)\Omega_{0}$ input at the
c-OMS and $\omega_{0}+\Omega_{0}$ input at the t-OMS is also shown with a
horizontal offset by $-0.25$ (yellow line). All spectra are normalized by the
respective maximum value. Each maximum point, which value is unity, is marked
by a blank circle.

Next, we examine the dependence on the frequencies of the two-tone and
monotone input lasers. In Fig. 3, $|\tilde{\mathcal{S}}_{\rm t}(\omega)|$
indicates the enhanced signals of the nonlocal inelastic scatterings. The
input frequency to the t-OMS is changed to $\omega_{0}+2\Omega_{0}$ (doubly
blue-detuned, orange), $\omega_{0}+\Omega_{0}$ (blue-detuned, purple),
$\omega_{0}$ (resonant, green), and $\omega_{0}-\Omega_{0}$ (red-detuned, red
lines) with fixed $\omega_{0}$ and $\omega_{0}+\Omega_{0}$ two-tone input at
the c-OMS. The Stokes scattering signal reaches a maximum when the blue-
detuned input is at the t-OMS, whereas the red-detuned input maximizes the
anti-Stokes scattering. The resonant input at the t-OMS (green line) is not an
optimal condition. The maximized condition of the Stokes (anti-Stokes)
scattering is found because the Stokes (anti-Stokes) shift is on the optical
cavity mode of the t-OMS. Then, the peak height of the Stokes (anti-Stokes)
signal is increased to $10^{3}$ from the baseline of the incident frequency
(see purple (red) line at $\omega=\omega_{0}$). For the doubly blue-detuned
input (orange line), the first-order inelastic scatterings are enhanced,
whereas the second-order inelastic scattering caused by two phonons does not
exhibit a large signal in the present consideration, even for the
$\omega=\omega_{0}$ optical cavity mode. These enhancements of the inelastic
scattering signals are supported by the efficient phonon generation at the
c-OMS. Efficient generation is also caused by the $\omega_{0}-\Omega_{0}$ and
$\omega_{0}$ two-tone input. Then, the spectra are overlapped with the shown
results.

Two-phonon generation at the c-OMS is ineffective when the
$\omega_{0}-\Omega_{0}$ and $\omega_{0}+\Omega_{0}$ two-tone laser is applied
(blue line). In this case, the inelastic scattering signal is not found in the
spectrum. We examine the $\omega_{0}-\Omega_{0}/2$ and
$\omega_{0}+\Omega_{0}/2$ two-tone laser incidence (yellow line with a small
horizontal shift). Then, one-phonon generation is effective, and the inelastic
scattering signal is sufficiently large. Its peak height is more than $10^{2}$
from the baseline (see the peak of the yellow line at
$\omega\approx\omega_{0}$). Hence, the two-tone input with an $\Omega_{0}$
frequency difference is one of the required conditions to “activate” the
c-OMS, whereas an optical resonant condition for the cavity mode is not
strictly required. Additionally, coherent phonon transport between the OMSs is
required, and the difference between the mechanical vibration modes of the
OMSs should be small, $|\Omega_{\rm c}-\Omega_{\rm
t}|<\sqrt{\gamma^{2}+K^{2}}$.

In conclusion, nonlocal inelastic scattering by remote and mechanically
coupled OMSs generates noiseless and pure inelastic signals. The two-tone
laser beams “activate” the c-OMS to generate phonons that affords the nonlocal
inelastic signals from the t-OMS. The frequency difference between the two
tones corresponding to the mechanical vibration mode significantly enhances
inelastic signals by several orders of magnitude. If the difference departed
from the mechanical vibration mode, the inelastic signals disappeared
sensitively. In addition, if a molecule or gas is adsorbed into one OMS and,
the mechanical vibration mode is changed, the inelastic scattering signal is
weakened (enlarged), owing to the suppression (enhancement) of phonon
transport, which leads to a sensitive detection of the molecule or gas. Thus,
our scheme can be applied to remote (noiseless), robust, and accurate sensors.

The present conclusions are derived from our specific model of remote-coupled
OMSs. However, the physics discussed in this paper might have a general aspect
toward nonlocal inelastic scattering. The key concept is nonlocal control of
inelastic scattering by two-plus-one color lasers, which has considerable
potential to open optical phononics, for example, in local phonon generation
by a focused laser under a wide-area reference light.

This work is supported in part by JSPS KAKENHI 18K13484 and 18H01151.

##  Appendix A Analytical solution

Figure 4:  (a) Expectation value $|\braket{\hat{a}_{{\rm t},0}}|$, which
exhibits the enhancement of Stokes scattering from the t-OMS in the plane of
the mechanical coupling strength $K$ and mechanical damping constant $\gamma$.
The parameters of the OMSs are $g_{0}=2.4\times 10^{-4}\Omega_{0}$ and
$\kappa=0.23\Omega_{0}$. The incident laser intensities parametrized by
$E_{\alpha,\xi}$ in the Hamiltonian are set to $|E_{\rm c,0}|=|E_{\rm
c,+}|=|E_{\rm t,+}|=\Omega_{0}$. Note that the phase factor of
$E_{\alpha,\xi}$ is irrelevant to $|\braket{\hat{a}_{{\rm t},0}}|$. Thin white
lines denote contour lines with an interval of $0.1$. The thick red line
denotes the maximum position in $K$ when $\gamma$ is fixed, which indicates
the optimal condition for inelastic scattering enhancement. (b) Cross-
sectional plot of $|\braket{\hat{a}_{{\rm t},0}}|$ in (a) at
$\gamma/g_{0}=1.0$ (purple), $1.5$ (green), and $2.0$ (light blue lines).

Here, we examine an analytical solution to discuss the enhancement of Stokes
scattering. We focus on the mechanical components $\gamma$ and $K$ in a
typical parameter regime of $\omega_{0}\gg\Omega_{0}>\kappa\gg
g_{0},\gamma,K$. For the system Hamiltonian $\hat{H}_{\mathrm{sys}}$ described
in the main text with mechanically coupled optomechanical systems (OMSs), we
apply a unitary transformation with respect to the operator,
$\hat{U}_{1}(t)=\mathrm{exp}\left[i\sum_{\alpha={\rm
c,t}}\left(\omega_{0}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}+\Omega_{0}\hat{b}_{\alpha}^{\dagger}\hat{b}_{\alpha}\right)t\right]$,
to formulate the Heisenberg equations of motion in a simple description (see
Sec. C). Here, the indices c and t represent the controller OMS (c-OMS) and
target OMS (t-OMS), respectively. Assuming the condition
$\kappa\ll\Omega_{0}$, where inelastically generated light spectra are
separated from the resonant signal, we take an ansatz for the photons Woolley
and Clerk (2014); Woolley et al. (2008): $\hat{a}_{\rm c}=\hat{a}_{{\rm
c},0}+\hat{a}_{{\rm c},+}e^{-i\Omega_{0}t}+\hat{a}_{{\rm
c},-}e^{i\Omega_{0}t}$, and $\hat{a}_{\rm t}=\hat{a}_{{\rm t},0}+\hat{a}_{{\rm
t},+}e^{-i\Omega_{0}t}$. The labels $0$, $+$, and $-$ represent the resonant,
blue-detuned, and red-detuned components, respectively. Here, the higher-order
inelastic scattered photons are not considered. We assume that for the two-
tone and monotone inputs to the c- and t-OMSs, respectively, the operators
$\hat{a}_{{\rm c},0}$, $\hat{a}_{{\rm c},+}$, and $\hat{a}_{{\rm t},+}$ in the
Heisenberg equations are constant. Then, the equations of motion for
$\hat{a}_{{\rm c},-}$, $\hat{a}_{{\rm t},0}$, $\hat{b}_{\rm c}$, and
$\hat{b}_{\rm t}$ are linearized. This set of inhomogeneous linear equations
of operators is analytically solvable. (See, Eqs.(24)–(27).) When all photons
and phonons are in the steady-states, $d\braket{\hat{a}_{\alpha,\xi}}/dt=0$
and $d\braket{\hat{b}_{\alpha}}/dt=0$, we obtain

$\braket{\hat{a}_{{\rm t},0}}=\frac{i\bar{g}_{\rm t}\bar{g}_{\rm
cc}^{*}K}{\zeta|\bar{g}_{\rm
t}|^{2}-\frac{\kappa}{2}\left(K^{2}+\zeta\gamma/2\right)},$ (5)

with $\zeta=\frac{\gamma}{2}-\frac{|g_{\rm c}|^{2}}{\kappa/2+i\Omega_{0}}$,
$\bar{g}_{\rm cc}=g_{0}\bar{a}_{{\rm c},0}^{*}\bar{a}_{{\rm c},+}$,
$\bar{g}_{\rm c}=g_{0}\bar{a}_{{\rm c},0}$, and $\bar{g}_{\rm
t}=g_{0}\bar{a}_{{\rm t},+}$. (see the detailed derivation in Sec. C).

Figure 4(a) illustrates the absolute value of $\braket{\hat{a}_{{\rm t},0}}$
in the $\gamma$-$K$ plane. For a fixed $K$, $\left|\braket{\hat{a}_{{\rm
t},0}}\right|$ is monotonically reduced by the damping constant $\gamma$ of
the mechanical vibration. When $\gamma$ is fixed, on the other hand, there is
an optimal condition, shown in Fig. 4(b). The condition can be approximated
when the numerator of $d\braket{\hat{a}_{{\rm t},0}}/dK$ is equal to zero,
$K^{2}\sim\frac{\gamma^{2}}{4}$. At $K=0$, an inelastic scattering signal is
not observed because phonon transport is absent. An increase in $K$ enhances
the inelastic signal. However, the stronger coupling of $K$ does not
contribute to the enhancement of $\left|\braket{\hat{a}_{{\rm t},0}}\right|$.
This is attributed to the energy shift of the mechanical vibration modes by
$K$. Hence, the two-tone input laser with $\omega_{0}$ and
$\omega_{0}+\Omega_{0}$ frequencies, which correspond to the resonant and
blue-detuned “eigen” frequencies of the isolated OMS, cannot generate phonons
effectively. The damping constant $\gamma$ indicates the energy-width of the
phonon spectrum. Thus, the optimal condition is given as $K\sim\gamma/2$. If
the “eigen” frequencies of the input laser are appropriately tuned for a large
$K$, the enhancement factor can be larger. However, this condition becomes
system-dependent, which is undesirable for applications.

##  Appendix B Numerical calculation: Fourier spectrum

To discuss the enhancement efficiency, we examine a numerical calculation of
the output power spectrum at the t-OMS. For the numerical calculation, the
system Hamiltonian is transferred by another unitary operator:
$\hat{U}_{2}=\exp\left[i(\omega_{0}+\Omega_{0})\left(\sum_{\alpha}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}\right)t\right]$.
(see Sec. C for details).

The output power spectrum of the optical mode in each OMS is evaluated using
the Fourier transformation of the correlation function:

$\mathcal{S}_{\alpha}(\omega)=\int\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}e^{-i\omega
t}dt.$ (6)

Here, $\braket{\hat{\mathcal{O}}(0)}$ denotes the time-averaged steady-state
mean value of an arbitrary system operator $\hat{\mathcal{O}}(t)$. The time
evolution follows the equations of motion (34)–(36), which consist of multiple
operators. After taking the expectation of the operators, we adopt the cluster
expansion method to solve the equations (see Sec.D and Ref. [Kira and Koch,
2008].) The correlation function is divided into two terms, e.g.,

$\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}=\braket{\hat{a}_{\alpha}^{\dagger}(0)}\braket{\hat{a}_{\alpha}(t)}+\Delta\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}.$
(7)

The second term on the right-hand-side represents the quantum correlation. The
time evolution of $\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}$
also follows Eqs. (34)–(36). The third-order terms, such as
$\Delta\braket{\hat{a}_{\alpha}(0)\hat{a}_{\alpha}^{\dagger}(t)\hat{a}_{\alpha}(t)}$,
and the higher-order terms are deduced in the equations of motion. The order
of the operators increases in the inductive self-consistent treatment of the
equations. There is a cut-off in the weak coupling regime for the respective
OMSs, where $C=4g_{0}^{2}/(\gamma\kappa)\ll 1$ is satisfied, and we can
neglect the third- and higher-order correlations.

##  Appendix C Unitary transformation

In this section, we describe two unitary transformations of the system
Hamiltonian $\hat{H}_{\mathrm{sys}}$ in the main text. These are utilized in
Secs. A and B.

First, we consider a unitary transformation with respect to a rotating frame,
as follows:

$\hat{H}^{\prime}_{\mathrm{sys}}(t)=\hat{U}_{1}(t)\hat{H}_{\mathrm{sys}}\hat{U}_{1}^{\dagger}(t)-i\hat{U}_{1}(t)\frac{\partial}{\partial
t}\hat{U}_{1}^{\dagger}(t),$ (8)

with the unitary operator

$\hat{U}_{1}(t)=\mathrm{exp}\left[i\sum_{\alpha}\left(\omega_{0}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}+\Omega_{0}\hat{b}_{\alpha}^{\dagger}\hat{b}_{\alpha}\right)t\right].$
(9)

Then, we obtain

$\displaystyle\hat{H}^{\prime}_{\mathrm{sys}}$ $\displaystyle=$
$\displaystyle-g_{0}\sum_{\alpha={\rm
c,t}}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}(e^{i\Omega_{0}t}\hat{b}_{\alpha}^{\dagger}+e^{-i\Omega_{0}t}\hat{b}_{\alpha})-K\left(\hat{b}_{\rm
t}^{\dagger}\hat{b}_{\rm c}+\hat{b}_{\rm c}^{\dagger}\hat{b}_{\rm t}\right),$
(10) $\displaystyle\hskip 14.22636pt+\left[i\left(E_{{\rm c},0}+E_{{\rm
c},+}e^{-i\Omega_{0}t}\right)\hat{a}_{\rm c}+iE_{{\rm
t},+}e^{-i\Omega_{0}t}\hat{a}_{\rm t}+\mathrm{h.c.}\right].$

Here, we omit the description “$(t)$” of the operators for simplicity. Note
that this unitary transformation is available for different input laser
frequencies. For the transferred Hamiltonian (10), the Heisenberg equations of
motion for the operators are

$\displaystyle\frac{d}{dt}\hat{a}_{\rm c}$ $\displaystyle=$
$\displaystyle-\frac{\kappa}{2}\hat{a}_{\rm c}+ig_{0}\hat{a}_{\rm
c}(e^{i\Omega_{0}t}\hat{b}_{\rm c}^{\dagger}+e^{-i\Omega_{0}t}\hat{b}_{\rm
c})+E_{{\rm c},0}+e^{-i\Omega_{0}t}E_{{\rm c},+},$ (11)
$\displaystyle\frac{d}{dt}\hat{a}_{\rm t}$ $\displaystyle=$
$\displaystyle-\frac{\kappa}{2}\hat{a}_{\rm t}+ig_{0}\hat{a}_{\rm
t}(e^{i\Omega_{0}t}\hat{b}_{\rm t}^{\dagger}+e^{-i\Omega_{0}t}\hat{b}_{\rm
t})+e^{-i\Omega_{0}t}E_{{\rm t},+},$ (12)
$\displaystyle\frac{d}{dt}\hat{b}_{\alpha}$ $\displaystyle=$
$\displaystyle-\frac{\gamma}{2}\hat{b}_{\alpha}+ig_{0}\hat{a}_{\alpha}\hat{a}_{\alpha}^{\dagger}e^{i\Omega_{0}t}+iK\hat{b}_{\bar{\alpha}}.$
(13)

The first terms on the right-hand side of these equations arise from the
coupling between the system and the environment Gardiner and Collett (1985).
$\bar{\alpha}$ in Eq. (13) represents $\bar{\rm c}={\rm t}$ and $\bar{\rm
t}={\rm c}$.

Let us formulate the analytical solution in Sec. A. In this study, we assume
that the relaxation of the optical cavity mode in the OMSs is much smaller
than the mechanical vibration mode, $\kappa\ll\Omega_{0}$. Then, the side-band
spectra caused by the inelastic scattering of light are well separated from
the main spectrum owing to the input laser. For this situation, we take the
following ansatz Woolley and Clerk (2014); Woolley et al. (2008):

$\displaystyle\hat{a}_{\rm c}$ $\displaystyle=$ $\displaystyle\hat{a}_{{\rm
c},0}+\hat{a}_{{\rm c},+}e^{-i\Omega_{0}t}+\hat{a}_{{\rm
c},-}e^{i\Omega_{0}t},$ (14) $\displaystyle\hat{a}_{\rm t}$ $\displaystyle=$
$\displaystyle\hat{a}_{{\rm t},0}+\hat{a}_{{\rm t},+}e^{-i\Omega_{0}t}$ (15)

by introducing the photon operators $\hat{a}_{\alpha,0}$,
$\hat{a}_{\alpha,+}$, and $\hat{a}_{\alpha,-}$. The labels $0$, $+$, and $-$
represent the resonant, blue-detuned, and red-detuned components,
respectively. Here, the higher-order inelastic scattered photons
$\hat{a}_{\alpha,\pm n}e^{\mp n\Omega_{0}t}$ ($n\geq 2$) are neglected because
their energy is far from the energy of the OMS cavity mode. For the t-OMS, we
do not consider the $\hat{a}_{{\rm t},-}e^{i\Omega_{0}t}$ term because this
non-resonant Fourier component is far from the input frequency
$\omega_{0}+\Omega_{0}$, while the $\hat{a}_{{\rm c},-}e^{i\Omega_{0}t}$ term
is considered because it is a “neighboring” component of the input frequency
$\omega_{0}$ in the c-OMS.

Substituting Eq. (14) into the Heisenberg equation (11), we obtain the
following:

$\displaystyle\frac{d}{dt}\hat{a}_{{\rm c},0}$ $\displaystyle=$
$\displaystyle-\frac{\kappa}{2}\hat{a}_{{\rm c},0}+ig_{0}(\hat{a}_{{\rm
c},+}\hat{b}_{\rm c}^{\dagger}+\hat{a}_{{\rm c},-}\hat{b}_{\rm c})+E_{{\rm
c},0},$ (16) $\displaystyle\frac{d}{dt}\hat{a}_{{\rm c},+}$ $\displaystyle=$
$\displaystyle\left(i\Omega_{0}-\frac{\kappa}{2}\right)\hat{a}_{{\rm
c},+}+ig_{0}\hat{a}_{{\rm c},0}\hat{b}_{\rm c}+E_{{\rm c},+},$ (17)
$\displaystyle\frac{d}{dt}\hat{a}_{{\rm c},-}$ $\displaystyle=$
$\displaystyle\left(-i\Omega_{0}-\frac{\kappa}{2}\right)\hat{a}_{{\rm
c},-}+ig_{0}\hat{a}_{{\rm c},0}\hat{b}_{\rm c}^{\dagger},$ (18)

and the substitution of Eqs. (15) into (12) gives

$\displaystyle\frac{d}{dt}\hat{a}_{{\rm t},0}$ $\displaystyle=$
$\displaystyle-\frac{\kappa}{2}\hat{a}_{{\rm t},0}+ig_{0}\hat{a}_{{\rm
t},+}\hat{b}_{\rm t}^{\dagger},$ (19) $\displaystyle\frac{d}{dt}\hat{a}_{{\rm
t},+}$ $\displaystyle=$
$\displaystyle\left(i\Omega_{0}-\frac{\kappa}{2}\right)\hat{a}_{{\rm
t},+}+ig_{0}\hat{a}_{{\rm t},0}\hat{b}_{\rm t}+E_{{\rm t},+}.$ (20)

Let us consider the expectation values of operators in the steady state when
sufficient strong lasers are applied, where the states are described by the
coherent states. For the two-tone and monotone laser incidences at the c- and
t-OMSs, respectively, the operators in the equations of motion (16), (17), and
(20) are replaced with
$\hat{a}_{\alpha,\xi}\to\braket{\hat{a}_{\alpha,\xi}}=\bar{a}_{\alpha,\xi}$
($\alpha={\rm c,t}$ and $\xi=0,\pm$). Then, we obtain

$\displaystyle\bar{a}_{{\rm c},0}$ $\displaystyle=$
$\displaystyle\frac{E_{{\rm c},0}}{\kappa/2},$ (21)
$\displaystyle\bar{a}_{{\rm c},+}$ $\displaystyle=$
$\displaystyle\frac{E_{{\rm c},+}}{\kappa/2-i\Omega_{0}},$ (22)
$\displaystyle\bar{a}_{{\rm t},+}$ $\displaystyle=$
$\displaystyle\frac{E_{{\rm t},+}}{\kappa/2-i\Omega_{0}},$ (23)

when $g_{0}\ll E_{\alpha,\xi},\kappa,\Omega_{0}$, which is a typical condition
Woolley and Clerk (2014). These input photons become steady states fast if the
incident light intensities are sufficiently strong. On the other hand, the
induced side-band photons and phonons require a longer time-scale to achieve a
steady state compared with the input photons. The side-band photons and
phonons adhere to the following equations of motion:

$\displaystyle\frac{d}{dt}\hat{a}_{{\rm c},-}$ $\displaystyle=$
$\displaystyle\left(-i\Omega_{0}-\frac{\kappa}{2}\right)\hat{a}_{{\rm
c},-}+i\bar{g}_{\rm c}\hat{b}_{\rm c}^{\dagger},$ (24)
$\displaystyle\frac{d}{dt}\hat{a}_{{\rm t},0}$ $\displaystyle=$
$\displaystyle-\frac{\kappa}{2}\hat{a}_{{\rm t},0}+i\bar{g}_{\rm
t}\hat{b}_{\rm t}^{\dagger},$ (25) $\displaystyle\frac{d}{dt}\hat{b}_{\rm c}$
$\displaystyle=$ $\displaystyle-\frac{\gamma}{2}\hat{b}_{\rm c}+iK\hat{b}_{\rm
t}+i\bar{g}_{\rm c}\hat{a}_{{\rm c},-}^{\dagger}+i\bar{g}_{\rm cc},$ (26)
$\displaystyle\frac{d}{dt}\hat{b}_{\rm t}$ $\displaystyle=$
$\displaystyle-\frac{\gamma}{2}\hat{b}_{\rm t}+iK\hat{b}_{\rm c}+i\bar{g}_{\rm
t}\hat{a}_{{\rm t},0}^{\dagger},$ (27)

where we define the effective coupling constants,

$\displaystyle\bar{g}_{\rm cc}$ $\displaystyle=$ $\displaystyle
g_{0}\bar{a}_{{\rm c},0}^{*}\bar{a}_{{\rm c},+},$ (28)
$\displaystyle\bar{g}_{\rm c}$ $\displaystyle=$ $\displaystyle
g_{0}\bar{a}_{{\rm c},0},$ (29) $\displaystyle\bar{g}_{\rm t}$
$\displaystyle=$ $\displaystyle g_{0}\bar{a}_{{\rm t},+}.$ (30)

Eqs. (24)–(27) provide a set of inhomogeneous linear equations of operators.
The solutions are given analytically for the case in which the input laser
intensities are constant. When all photons and phonons become steady states,
$d\braket{\hat{a}_{\alpha,\xi}}/dt=0$ and $d\braket{\hat{b}_{\alpha}}/dt=0$,
we obtain Eq. (5) with

$\zeta=\frac{\gamma}{2}-\frac{|g_{\rm c}|^{2}}{\kappa/2+i\Omega_{0}}.$ (31)

Next, for the numerical calculations described in Sec. B, we consider another
unitary transformation,

$\hat{U}_{2}=\exp\left[i(\omega_{0}+\Omega_{0})\left(\sum_{\alpha}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}\right)t\right],$
(32)

for the system Hamiltonian $\hat{H}_{\mathrm{sys}}$:

$\displaystyle\hat{H}_{\mathrm{sys}}^{{}^{\prime\prime}}$ $\displaystyle=$
$\displaystyle\hat{U}_{2}\hat{H}_{\mathrm{sys}}\hat{U}_{2}^{\dagger}-i\hat{U}_{2}\frac{\partial}{\partial
t}\hat{U}_{2}^{\dagger}$ (33) $\displaystyle=$ $\displaystyle\sum_{\alpha={\rm
c,t}}\left[-\Omega_{0}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}+\Omega_{0}\hat{b}_{\alpha}^{\dagger}\hat{b}_{\alpha}-g_{0}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}\left(\hat{b}_{\alpha}^{\dagger}+\hat{b}_{\alpha}\right)\right]-K\left(\hat{b}_{\rm
c}\hat{b}_{\rm t}^{\dagger}+\hat{b}^{\dagger}_{\rm c}\hat{b}_{\rm t}\right)$
$\displaystyle+\left[i\left(E_{{\rm c},0}e^{i\Omega_{0}t}+E_{{\rm
c},+}\right)\hat{a}_{\rm c}+iE_{{\rm t},+}\hat{a}_{\rm
t}+\mathrm{h.c.}\right].$

The Heisenberg equations for the operators are as follows:

$\displaystyle\frac{d}{dt}\hat{a}_{\rm c}$ $\displaystyle=$
$\displaystyle\left(i\Omega_{0}-\frac{\kappa}{2}\right)\hat{a}_{\rm
c}+ig_{0}\hat{a}_{\rm c}\left(\hat{b}_{\rm c}^{\dagger}+\hat{b}_{\rm
c}\right)+E_{{\rm c},0}e^{i\Omega_{0}t}+E_{{\rm c},+},$ (34)
$\displaystyle\frac{d}{dt}\hat{a}_{\rm t}$ $\displaystyle=$
$\displaystyle\left(i\Omega_{0}-\frac{\kappa}{2}\right)\hat{a}_{\rm
t}+ig_{0}\hat{a}_{\rm t}\left(\hat{b}_{\rm t}^{\dagger}+\hat{b}_{\rm
t}\right)+E_{{\rm t},+},$ (35) $\displaystyle\frac{d}{dt}\hat{b}_{\alpha}$
$\displaystyle=$
$\displaystyle\left(-i\Omega_{0}-\frac{\gamma}{2}\right)\hat{b}_{\alpha}+ig_{0}\hat{a}_{\alpha}^{\dagger}\hat{a}_{\alpha}+iK\hat{b}_{\bar{\alpha}}.$
(36)

Here, $\bar{\alpha}$ in (36) indicates that $\bar{\rm c}={\rm t}$ and
$\bar{\rm t}={\rm c}$. Note that this transformation applies to a rotating
frame with a $\omega_{0}+\Omega_{0}$ laser. Then, the factors in the
Heisenberg equations become time-independent, except for the third term on the
right-hand side in Eq. (34), which describes the $\omega_{0}$ input laser.
These transformations and Heisenberg equations provide an advantage for the
numerical discussion.

##  Appendix D Cluster expansion

In this section, we explain a cluster expansion method Kira and Koch (2008),
which is applied to our numerical calculation in Eqs. (34)–(36). This method
is a conventional treatment in quantum optics. The cluster expansions for
arbitrary operators, $\hat{X},\hat{Y},\hat{Z}$ are given as follows:

$\displaystyle\braket{\hat{X}\hat{Y}}$ $\displaystyle=$
$\displaystyle\braket{\hat{X}}\braket{\hat{Y}}+\Delta\braket{\hat{X}\hat{Y}},$
(37) $\displaystyle\braket{\hat{X}\hat{Y}\hat{Z}}$ $\displaystyle=$
$\displaystyle\braket{\hat{X}}\braket{\hat{Y}}\braket{\hat{Z}}+\braket{\hat{X}}\Delta\braket{\hat{Y}\hat{Z}}+\braket{\hat{Y}}\Delta\braket{\hat{X}\hat{Z}}+\braket{\hat{Z}}\Delta\braket{\hat{X}\hat{Y}}$
(38) $\displaystyle+\Delta\braket{\hat{X}\hat{Y}\hat{Z}}.$

Terms such as $\Delta\braket{\hat{X}\hat{Y}}$ are called doublets, which
originate purely from quantum correlations. Here, we neglect the triplet
correlation terms $\Delta\braket{\hat{X}\hat{Y}\hat{Z}}$ in a weak coupling
regime. In this study, the weak coupling regime corresponds to a small
cooperativity $C=4g_{0}^{2}/(\gamma\kappa)\ll 1$. Note that for a single
operator, $\Delta\braket{\hat{a}_{\alpha}}=\Delta\braket{\hat{b}_{\alpha}}=0$.

Figure 5:  (a) Numerically calculated results of the time evolution of a real
part of $\braket{\hat{a}_{\rm t}(t)}$. The time is normalized by the
mechanical vibration frequency $\Omega_{0}$. Panels (b)–(d) are zoomed-in
views of (a). $\braket{\hat{a}_{\rm t}(t)}$ becomes a steady state in (d). The
time evolution of the imaginary part is qualitatively equivalent to the real
part, hence it is not plotted. We used the steady state for the evaluation of
the power spectrum $\mathcal{S}_{\alpha}(\omega)$.

The cluster expansion provides

$\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}=\braket{\hat{a}_{\alpha}^{\dagger}(0)}\braket{\hat{a}_{\alpha}(t)}+\Delta\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}.$
(39)

To evaluate the output power spectrum as
$\mathcal{S}_{\alpha}(\omega)=\int\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}e^{-i\omega
t}dt$, we numerically solve the equations of motion for
$\braket{\hat{a}_{\alpha}(t)}$ and
$\Delta\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}$. The former
equation of motion follows Eqs. (34) and (35):

$\displaystyle\frac{d}{dt}\braket{\hat{a}_{\rm c}(t)}$ $\displaystyle=$
$\displaystyle\left(i\Omega_{0}-\frac{\kappa}{2}\right)\braket{\hat{a}_{\rm
c}(t)}+ig_{0}\left(\braket{\hat{a}_{\rm c}(t)\hat{b}_{\rm
c}^{\dagger}(t)}+\braket{\hat{a}_{\rm c}(t)\hat{b}_{\rm
c}(t)}\right)+\left(E_{{\rm c},0}e^{i\Omega_{0}t}+E_{{\rm c},+}\right)$ (40)
$\displaystyle=$
$\displaystyle\left(i\Omega_{0}-\frac{\kappa}{2}\right)\braket{\hat{a}_{\rm
c}(t)}+ig_{0}\left\\{\braket{\hat{a}_{\rm c}(t)}\braket{\hat{b}_{\rm
c}^{\dagger}(t)}+\Delta\braket{\hat{a}_{\rm c}(t)\hat{b}_{\rm
c}^{\dagger}(t)}+\braket{\hat{a}_{\rm c}(t)}\braket{\hat{b}_{\rm
c}(t)}+\Delta\braket{\hat{a}_{\rm c}(t)\hat{b}_{\rm c}(t)}\right\\}$
$\displaystyle\hskip 227.62204pt+\left(E_{{\rm c},0}e^{i\Omega_{0}t}+E_{{\rm
c},+}\right),$ $\displaystyle\frac{d}{dt}\braket{\hat{a}_{\rm t}(t)}$
$\displaystyle=$
$\displaystyle\left(i\Omega_{0}-\frac{\kappa}{2}\right)\braket{\hat{a}_{\rm
t}(t)}+ig_{0}\left(\braket{\hat{a}_{\rm t}(t)\hat{b}_{\rm
t}^{\dagger}(t)}+\braket{\hat{a}_{\rm t}(t)\hat{b}_{\rm t}(t)}\right)+E_{{\rm
t},+}$ (41) $\displaystyle=$
$\displaystyle\left(i\Omega_{0}-\frac{\kappa}{2}\right)\braket{\hat{a}_{\rm
c}(t)}+ig_{0}\left\\{\braket{\hat{a}_{\rm t}(t)}\braket{\hat{b}_{\rm
t}^{\dagger}(t)}+\Delta\braket{\hat{a}_{\rm t}(t)\hat{b}_{\rm
t}^{\dagger}(t)}+\braket{\hat{a}_{\rm t}(t)}\braket{\hat{b}_{\rm
t}(t)}+\Delta\braket{\hat{a}_{\rm t}(t)\hat{b}_{\rm t}(t)}\right\\}$
$\displaystyle\hskip 227.62204pt+E_{{\rm t},+}.$

For the latter equation of motion, we obtain, e.g.,

$\displaystyle\frac{d}{dt}\left(\Delta\braket{\hat{a}_{\rm
c}^{\dagger}(0)\hat{a}_{\rm
c}(t)}\right)=\frac{d}{dt}\left(\braket{\hat{a}_{\rm
c}^{\dagger}(0)\hat{a}_{\rm c}(t)}-\braket{\hat{a}_{\rm
c}^{\dagger}(0)}\braket{\hat{a}_{\rm c}(t)}\right)$ $\displaystyle\hskip
14.22636pt=\left(i\Omega_{0}-\frac{\kappa}{2}\right)\left\\{\braket{\hat{a}_{\rm
c}^{\dagger}(0)\hat{a}_{\rm c}(t)}-\braket{\hat{a}_{\rm
c}^{\dagger}(0)}\braket{\hat{a}_{\rm c}(t)}\right\\}$ $\displaystyle\hskip
28.45274pt+ig_{0}\left\\{\braket{\hat{a}_{\rm c}^{\dagger}(0)\hat{a}_{\rm
c}(t)\hat{b}_{\rm c}^{\dagger}(t)}-\braket{\hat{a}_{\rm
c}^{\dagger}(0)}\braket{\hat{a}_{\rm c}(t)\hat{b}_{\rm
c}^{\dagger}(t)}+\braket{\hat{a}_{\rm c}^{\dagger}(0)\hat{a}_{\rm
c}(t)\hat{b}_{\rm c}(t)}-\braket{\hat{a}_{\rm
c}^{\dagger}(0)}\braket{\hat{a}_{\rm c}(t)\hat{b}_{\rm c}(t)}\right\\}$
$\displaystyle\hskip
14.22636pt\approx\left(i\Omega_{0}-\frac{\kappa}{2}\right)\Delta\braket{\hat{a}_{\rm
c}^{\dagger}(0)\hat{a}_{\rm c}(t)}$ $\displaystyle\hskip
28.45274pt+ig_{0}\left\\{\braket{\hat{a}_{\rm c}(t)}\Delta\braket{\hat{a}_{\rm
c}^{\dagger}(0)\hat{b}_{\rm c}^{\dagger}(t)}+\braket{\hat{b}_{\rm
c}^{\dagger}(t)}\Delta\braket{\hat{a}_{\rm c}^{\dagger}(0)\hat{a}_{\rm
c}(t)}+\braket{\hat{a}_{\rm c}(t)}\Delta\braket{\hat{a}_{\rm
c}^{\dagger}(0)\hat{b}_{\rm c}(t)}+\braket{\hat{b}_{\rm
c}(t)}\Delta\braket{\hat{a}_{\rm c}^{\dagger}(0)\hat{a}_{\rm c}(t)}\right\\}.$
(42)

Then, we can calculate all the expectation values for a single operator and
the quantum correlation of the two operators. Note that Eqs. (40) and (41)
include the quantum correlation of the same time operators. Therefore, we must
consider their equations of motion. In the evaluation,
$\braket{\hat{\mathcal{O}}(0)}$ represents the time-averaged steady-state mean
value of the system operator $\hat{\mathcal{O}}(t)$. In this study, we focus
only on the sum of the calculated
$\braket{\hat{a}_{\alpha}^{\dagger}(0)}\braket{\hat{a}_{\alpha}(t)}$ and
$\Delta\braket{\hat{a}_{\alpha}^{\dagger}(0)\hat{a}_{\alpha}(t)}$. These
respective terms should be considered when the cooperativity $C$ is large.

##  Appendix E Time evolution for steady-state

We demonstrate the time evolution using the Heisenberg equations to determine
the steady state. Figure 5(a) shows the oscillated time evolution of ${\rm
Re}[\braket{\hat{a}_{\rm t}(t)}]$ as an example. At the beginning (Fig. 5(b)),
the oscillation amplitude of $\braket{\hat{a}_{\rm t}(t)}$ changes rapidly.
After a short period, the amplitude gradually increases in Fig. 5(b) and (c).
When we examine a sufficient time evolution, $\braket{\hat{a}_{\rm t}(t)}$
becomes a steady state (Fig. 5(d)). In the main text, Sec. B, the output power
spectrum $\mathcal{S}_{\alpha}(\omega)$ is evaluated from such a steady-state.

## References

  * Pettinger  _et al._ (2012) B. Pettinger, P. Schambach, C. J. Villagómez, and N. Scott, Annual Review of Physical Chemistry 63, 379 (2012).
  * Stiles  _et al._ (2008) P. L. Stiles, J. A. Dieringer, N. C. Shah, and R. P. Van Duyne, Annual Review of Analytical Chemistry 1, 601 (2008).
  * Nie and Emory (1997) S. Nie and S. R. Emory, Science 275, 1102 (1997).
  * Kneipp  _et al._ (1997) K. Kneipp, Y. Wang, H. Kneipp, L. T. Perelman, I. Itzkan, R. R. Dasari, and M. S. Feld, Phys. Rev. Lett. 78, 1667 (1997).
  * Law (1995) C. K. Law, Phys. Rev. A 51, 2537 (1995).
  * Aspelmeyer  _et al._ (2014) M. Aspelmeyer, T. J. Kippenberg, and F. Marquardt, Rev. Mod. Phys. 86, 1391 (2014).
  * Andrews  _et al._ (2014) R. W. Andrews, R. W. Peterson, T. P. Purdy, K. Cicak, R. W. Simmonds, C. A. Regal, and K. W. Lehnert, Nature Physics 10, 321 (2014).
  * Forsch  _et al._ (2020) M. Forsch, R. Stockill, A. Wallucks, I. Marinković, C. Gärtner, R. A. Norte, F. van Otten, A. Fiore, K. Srinivasan, and S. Gröblacher, Nature Physics 16, 69 (2020).
  * Jiang  _et al._ (2020) W. Jiang, C. J. Sarabalis, Y. D. Dahmani, R. N. Patel, F. M. Mayor, T. P. McKenna, R. Van Laer, and A. H. Safavi-Naeini, Nature Communications 11, 1166 (2020).
  * Colombano  _et al._ (2019) M. F. Colombano, G. Arregui, N. E. Capuj, A. Pitanti, J. Maire, A. Griol, B. Garrido, A. Martinez, C. M. Sotomayor-Torres, and D. Navarro-Urrios, Phys. Rev. Lett. 123, 017402 (2019).
  * Letizia Catalini and Schliesser (2020) L. Catalini, Y. Tsaturyan, A. Schliesser, Phys. Rev. Appl. 14, 014041 (2020).
  * Ludwig and Marquardt (2013) M. Ludwig and F. Marquardt, Phys. Rev. Lett. 111, 073603 (2013).
  * Peano  _et al._ (2015) V. Peano, C. Brendel, M. Schmidt, and F. Marquardt, Phys. Rev. X 5, 031011 (2015).
  * Grutter  _et al._ (2015) K. E. Grutter, M. Davanço, and K. Srinivasan, IEEE Journal of Selected Topics in Quantum Electronics 21, 61 (2015).
  * Schneider  _et al._ (2019) K. Schneider, Y. Baumgartner, S. Hönl, P. Welter, H. Hahn, D. J. Wilson, L. Czornomaz, and P. Seidler, Optica 6, 577 (2019).
  * Clerk  _et al._ (2020) A. A. Clerk, K. W. Lehnert, P. Bertet, J. R. Petta, and Y. Nakamura, Nature Physics 16, 257 (2020).
  * Woolley and Clerk (2014) M. J. Woolley and A. A. Clerk, Phys. Rev. A 89, 063805 (2014).
  * Woolley et al. (2008) M. J. Woolley, A. C. Doherty, G. J. Milburn, and K. C. Schwab, Phys. Rev. A 78, 062303 (2008).
  * Kira and Koch (2008) M. Kira and S. W. Koch, Phys. Rev. A 78, 022102 (2008).
  * Gardiner and Collett (1985) C. W. Gardiner and M. J. Collett, Phys. Rev. A 31, 3761 (1985).


# Improving Structured Text Recognition with Regular Expression Biasing

Baoguang Shi, Wenfeng Cheng, Yijuan Lu, Cha Zhang, Dinei Florencio
Microsoft
{bashi, wenfeng.cheng, yijlu, chazhang<EMAIL_ADDRESS>
###### Abstract

We study the problem of recognizing structured text, i.e. text that follows
certain formats, and propose to improve the recognition accuracy of structured
text by specifying regular expressions (regexes) for biasing. A biased
recognizer recognizes text that matches the specified regexes with
significantly improved accuracy, at the cost of a generally small degradation
on other text. The biasing is realized by modeling regexes as a Weighted
Finite-State Transducer (WFST) and injecting it into the decoder via dynamic
replacement. A single hyperparameter controls the biasing strength. The method
is useful for recognizing text lines with known formats or containing words
from a domain vocabulary. Examples include driver license numbers, drug names
in prescriptions, etc. We demonstrate the efficacy of regex biasing on
datasets of printed and handwritten structured text and measures its side
effects.

##  I Introduction

A practical OCR system is often applied to recognize _structured text_ , i.e.
text that follows certain formats. Examples include dates, currencies, phone
numbers, addresses, etc. Structured text possesses challenges to OCR
recognizers, as such text contains digits and symbols that are hard to
separate, e.g. “1” (one), “l” (lower-case L), and “/” (slash). Meanwhile,
structured text possesses important information and demands higher recognition
accuracy.

Figure 1: An illustration of regex biasing applied to recognizing a low-
resolution driver license scan. The specified regex consists of 3 parts,
concatenated by the OR (“|”) operator, biasing the recognition of license
number, expiration date, and sex fields. The fields impacted by the biasing
are highlighted by color-coded boxes. This image is a fake sample from
California DMV. Figure 2: A mini WFST decoder representing the language model
comprising words, “foo” and “bar”. Thick-lined circles are start states;
double-lined circles are final states. Transition labels are formatted in
“<input label>:<output label>/<weight>”, or “<input label>:<output label>”
when weight is zero. The auxiliary symbol “#0” is for disambiguation when a
word has more than one spelling (e.g. spelling in lower case and upper case
letters) [20]. In (a), the weight values 6.9 is calculated from
$-\log(0.001)$, meaning unigram probabilities 0.001 for words “foo” and “bar”.
The transition from state 1 to 2 means a 0.01 bigram probability for “foo
bar”.

Improving the recognition of structured text can be seen as a case of domain
adaptation. A common strategy for domain adaption is to finetune recognition
models with domain-specific data. However, finetuning requires collecting a
sufficiently large dataset in the same domain. Therefore, finetuning can be
very expensive, and impractical in many cases due to the sensitivity of the
data in the target domain.

In many cases, the format of the structured text is known in prior and can be
expressed by regular expressions. For example, California car license plate
number follows “\d[A-Z]{3}\d{3}”, meaning “one digit, followed by three
capital letters, then followed by three digits”. The knowledge of the format
may substantially improve the recognition of structured text, as candidate
characters are limited by their positions and contexts.

In this paper, we propose to inject such knowledge into a text recognizer by
biasing it towards user-specified regexes. A biased recognizer will favor text
that matches the specified recognizer over other similar candidates. Figure 1
illustrates the biasing using a regex describing the formats of license
number, expiration date, and sex. A recognizer biased as such will favor “DL
I12345678” over “DL 112345678” because the former matches the specified regex
while the latter does not. Consequently, a biased recognizer recognizes
structured text with significantly improved accuracy at the cost of a
generally small degradation on other text.

We realize regex biasing by expressing the regexes as a Weighted Finite-State
Transducer (WFST) [19] and use it to decode the outputs of the recognition
model. A single hyperparameter $\alpha\in\Re$ controls the weight of WFST and
thus the biasing strength. Specifically, a lower $\alpha$ makes the recognizer
bias the specified regexes with higher strength.

Regex biasing enables convenient domain adaption without training data. Not
only is this method cost-effective, but particularly useful for situations
where training data from the target domain is hard to obtain due to its cost
or sensitivity. We test the efficacy of this method on both printed and
handwritten text data. A side effect of biasing is creating false positives,
especially when biasing strength is high. We measure this side effect in the
experiments.

Since our method only concerns the WFST language model, it can be applied to
any recognizers that are compatible with WFST decoders. We focus on the type
of text recognizer that outputs a sequence of character probabilities, e.g.
[24, 8]. Recognizers of this kind have been shown to deliver state-of-the-art
performance on line-level recognition when working together with an n-gram
language model [11].

The rest of the paper is organized as follows. In Section II, we discuss the
related works. In Section III, we first briefly present the basics of WFST,
then introduce how to model regular expressions as WFSTs and how to use them
for decoding. In Section IV, we present the experiment results, demonstrating
the efficacy and side-effects of regex biasing.

##  II Related works

Text recognition has attracted a lot of research effort in recent years [24,
9, 16, 25, 4, 8, 6, 11, 17, 21]. These works model text recognition as a
sequence prediction problem and employ models such as convolutional network,
LSTM, HMM (Hidden Markov Model), and WFST for sequence modeling. Some works
focus on word-level scene text recognition [24, 9, 16, 25] while others
address line-level handwriting recognition [4, 8, 6, 11]. Our character
recognition model is based on the CNN-LSTM-CTC framework [24] and WFST for the
language modeling part. It has a similar architecture as the one in [6], which
has been demonstrated to outperform attention-based models when trained with a
large-scale dataset and coupled with a language model.

WFST has been highly successful in the research of speech recognition [19] and
handwriting recognition [4]. Because of the flexibility of WFST, it can be
used for modeling n-gram language models, lexicons, etc. Our WFST building
process follows the standard recipe that involves lexicon and grammar modeling
[20].

The idea of biasing WFSTs with domain knowledge has been previously explored
in the speech recognition community [2, 14]. We drew our inspiration from
these works. [2] proposes to improve the recognition of contact names by
dynamic WFST replacement, which we also used for injecting regex patterns.
[14] proposes to improve the recognition of numeric sequence by building
numeric grammar modeled by WFST, which is similar to our idea of improving
structured text with regex-defined grammars.

To the best of our knowledge, the idea of regex biasing has not been
previously proposed in the literature.

##  III Method

Figure 3: Example regex operations and their corresponding WFSTs. x can be a
symbol or a regular expression. In the examples above, x is a symbol.

###  III-A Background: WFST decoder

WFST has a long-standing role in speech recognition [19] for its language
modeling capability. It provides a unified representation of n-gram language
model, lexicon, CTC collapsing rule, etc. We refer the readers to [19] for a
complete tutorial.

A WFST is a finite-state machine whose state transitions are labeled with
input symbols, output symbols, and weights. A state transition consumes the
input symbol, writes the output symbol, and accumulates the weight. A special
symbol $\epsilon$ means consuming no input when used as an input symbol or
outputting nothing when used as an output label. Therefore, a path through the
WFST maps an input string to an output string with a total weight.

A common set of operations are available for WFSTs. _Composition_ ($\circ$)
combines two WFSTs: Denoting the two WFSTs by $T_{1}$ and $T_{2}$, if the
output space (symbol table) of $T_{1}$ matches the input space of $T_{2}$,
they can be combined by the composition algorithm, as in $T=T_{1}\circ T_{2}$.
Applying $T$ on any sequence is equivalent to applying $T_{1}$ first, then
$T_{2}$ on the output of $T_{1}$. _Determinization_ and _minimization_ are two
standard WFST optimization operations. Determinization makes each WFST state
has at most one transition with any given input label and eliminates all input
$\epsilon$-labels. Minimization reduces the number of states and transitions.
The common optimization recipe combines the two operations, as in
$T^{\prime}=\mathrm{optim}(T)=\mathrm{minimize}(\mathrm{determinize}(T))$, and
yields an equivalent WFST that is faster to decode and smaller in size.

In the context of OCR, WFST can be used for decoding the output sequences of a
text recognition model. Below we demonstrate the WFST decoder for a CTC-based
[13] recognizer. The WFST is composed and optimized from three different
WFSTs, denoted by $G$, $L$, and $C$:

$\displaystyle T$ $\displaystyle=\mathrm{optim}(L\circ G)$ (1) $\displaystyle
T_{\mathrm{ctc}}$ $\displaystyle=\mathrm{optim}(C\circ T)$ (2)

Here, $G$ (grammar) models n-gram probabilities. Its input and output symbols
are words (or sub-word units, such as BPE [23]) and its transition weights
represent n-gram probabilities. $L$ models lexicon, i.e. the spelling of every
word in $G$. Its input space is the set of characters supported by the text
recognizer and its output space is the words modeled by $G$. Since a CTC-based
recognizer outputs extra _blank_ symbols, an extra WFST $C$ is left-composed
to perform the “collapsing rule” of CTC. In practice, $C$ is realized by
inserting states and transitions that consume all blanks and repeated
characters to $L\circ G$. In Figure 2, we illustrate $L$, $G$ and $T$ on a
mini language model involving only two words “foo” and “bar”.

Decoding with WFST is to find the most probable word (or sub-word unit)
sequence $\hat{W}=w_{1},...,w_{M}$ given the character observations
$X=x_{1},...,x_{n}$ output by the CTC recognition model:

$\hat{W}=\arg\min_{W}(\lambda\log p(X|W)+\log p(W))$ (3)

Here, $\lambda$ (known as _acoustic weight_ in speech) controls the weight of
the character observations. The most probable path can be approximated by the
beam search algorithm. Open-source toolkits such as Kaldi [20] provide highly
efficient decoding implementations.

###  III-B Modeling regex as WFST

Regular expressions are widely used in computer science for specifying search
patterns. A regex expression can be translated into a deterministic finite
automaton (DFA) by a regex processor, such as the Thompson’s construction
algorithm [1]. Since WFST is also finite automaton, we can convert the DFA of
a regex into a WFST by turning every transition label into a pair of identical
input and output labels and assign a unit weight. The resulting unweighted
WFST is denoted by $R$.

Figure 3 demonstrates some basic regex operations and their corresponding
WFSTs. Using these operators we can build regex to match complex patterns. In
practice, we rely on the open-source grammar compiler _Thrax_ [22] to compile
regexes directly to WFSTs. The syntax of Thrax is similar to the common regex
syntaxes such as that in Python, but it also supports more advanced features
such as functions. The pseudo regex syntax we have used so far can be easily
translated into equivalent Thrax statements. For example, the digit matcher
“\d” in Figure 1 is replaced by a constant “DIGIT” in Thrax’s syntax, which is
defined as “DIGIT = ’’0’’|’’1’’|...|’’9’’”.

We make regex WFSTs have small or even negative transition weight values so
that the paths through regex WFSTs will be favored by the decoder. In
practice, we use a length-linear function to assign the weights in the WFST
transitions. This implemented by left-composing a scoring WFST $S$ with the
unweighted regex WFST $R$:

$T_{r}=S_{\alpha}\circ R$

Here, $S_{\alpha}$ is a scoring WFST that has a single state that is both
start and final state and connects a number of self-loop transitions where the
input and output labels are the supported symbols (characters). The weights of
these transitions are set to a constant $\alpha$. After the composition, the
total weight of a path in $T_{r}$ for a matching text string will be
$n\alpha$, where $n$ is the length of the string. In this way, we can control
the biasing strength by adjusting $\alpha$: lowering $\alpha$ increases the
biasing strength.

###  III-C Decoding with regex biasing

The weighted regex WFST $T_{r}$ cannot be used directly for decoding since it
only accepts text matching the regex. We combine $T_{r}$ with the base
language model $T$ so that the decoder can output any text.

To achieve this, we first modify $T$ to add special transitions that are
labeled with a _nonterminal_ [10] symbol $REGEX. The modified WFST
$T_{\mathrm{root}}$ is known as the _class-based language model_ [5]. One way
to add $REGEX is to add it as a unigram word to the grammar WFST $G$ so that
regex biasing can be applied to part of a sentence, while the rest of the
sentence is scored by $G$. The lexicon WFST $L$ is also modified to add the
spelling of $REGEX, which is one epsilon (“$\epsilon$”). The modified grammar
and lexicon WFSTs are denoted by $G^{\prime}$ and $L^{\prime}$, respectively.
Then we have $T_{\mathrm{root}}=\mathrm{optim}(G^{\prime}\circ L^{\prime})$.

$T_{\mathrm{root}}$ and $T_{r}$ can be combined using the WFST _replacement_
operation:

$T^{\prime}=\mathrm{replace}(T_{\mathrm{root}},T_{r})$

which replaces transitions labeled with $REGEX with its corresponding WFST
$T_{r}$. Figure 4 illustrates this process on the aforementioned mini language
model. The modified base language model $T_{\mathrm{root}}$ has two additional
transitions with $REGEX. After replacement, state 0 and state 7 in
$T^{\prime}$ (corresponding to state 5 in $T_{\mathrm{root}}$) both have a
transition to state 1, effectively acting as the entry and return points of
$T_{r}$. After the replacement, we can make $T^{\prime}$ into a CTC-compatible
decoder $T^{\prime}_{\mathrm{ctc}}$ using the composition in Equation 2.

In practice, we use an operation called _dynamic replacement_ [3] to perform
the combination. We first build CTC-compatible WFSTs from $T_{\mathrm{root}}$
and $T_{r}$ respectively (denoted by $T_{\mathrm{root}}^{c}$ and $T_{r}^{c}$
respectively). Then, during decoding, transitions in $T_{\mathrm{root}}^{c}$
is replaced by $T_{r}^{c}$ on-demand, as in
$T^{\prime}=\mathrm{dyreplace}(T_{\mathrm{root}}^{c},T_{r}^{c})$. Since the
main language model WFST $T_{\mathrm{root}}$ may contain millions of states
and transitions and is costly to update, dynamic replacement allows us to fix
the main language model and only update the regex WFST.

Figure 4: Illustration of WFST replacement. (a) Modified WFST
$T_{\mathrm{root}}$ with nonterminal symbol $REGEX; (b): Regex WFST $T_{r}$
representing regex “ab*” with biasing strength $\alpha$; (c) Unoptimized WFST
after replacement.

##  IV Experiments

In this section, we demonstrate the efficacy of regex biasing through
different combinations of dataset and regex setting. We focus on the
improvements on the structured text of interest as well as the side effects,
i.e. degradation on other text.

###  IV-A Evaluation datasets

Throughout the experiments, recognition is performed on text lines rather than
isolated words. This setting fits real-world scenarios and is necessary for
regex biasing since structured text often comes in multiple words that are
separated by space (e.g. “Date of Birth: Aug 1, 1990”). To the best of our
knowledge, there are no public datasets of such structured text. Therefore, we
collect two datasets of printed text and simulate structured text recognition
on a handwritten text dataset.

Driver Licenses This dataset consists of scans of US driver licenses from
different states. The licenses are fake samples we collected from the
Internet. To simulate the real-world imaging conditions, we printed the
samples on paper and took photos of them with phone cameras. We ran an in-
house text detector to extract text lines and label them manually. There are
737 text lines from 50 distinct licenses. Many lines contain structured text,
such as date of birth and driver’s license number. To simulate poor imaging
conditions, we further augmented each image by degrading image quality and got
4422 text line images for the final dataset.

Passport MRZ This dataset contains 8040 text lines extracted from the machine-
readable zones (MRZ) of a collection of passport scans from different
countries. Each MRZ contains two text lines. Usually, MRZs are scanned by
specialized passport readers and their images are in high resolution. But in
this dataset, images have much lower quality in terms of resolution, lighting
condition, etc.

IAM [18] This dataset has been a standard dataset for handwriting recognition.
The test set of IAM contains 1861 handwritten text lines. Some text lines of
IAM contain out-of-vocabulary words such as people’s names. We use this
dataset to test the efficacy of biasing a word list.

Table I: Regex definition for the driver license dataset in Thrax’s syntax.
“UPPER” and “DIGIT” are predefined constants, meaning upper-case letters and
digits respectively.

###  IV-B Recognition models

Our recognizer consists of a character-level recognition model (character
model) and a language model. The character model follows the ConvNet-LSTM-CTC
design [24]. We use the compact model architecture from [8]. Since our model
needs to be able to perform line-level recognition, we trained our model on an
internal dataset of 1.6 million text lines. Images in this dataset come from
multiple sources, such as documents, receipts, scene images, and are in
printed style. This model is referred to as the _printed model_. Similarly, we
train another model for handwriting recognition on a dataset consisting of
248k handwritten text lines (_handwriting model_).

We use the Kaldi toolkit [20] to build the language model. The language model
is trained on a large text corpus comprising 67 million lines of text,
collected from various sources. The lexicon WFST is built from a vocabulary of
100k words and sub-word units, learned from the training data using the byte-
pair encoding algorithm [23] implemented in the sentencepiece library [15].
The grammar WFST is built from an n-gram model containing unigrams and
bigrams.

We use the OpenGrm toolkit [22] to build regex WFSTs and convert them into
CTC-compatible decoders using a Python binding of Kaldi [7]. Building a regex
WFST typically takes a few seconds.

###  IV-C Regex biasing for driver licenses

The formats of US driver licenses differ from state to state. We set regexes
for a common set of fields, including date of birth, issuing date, expiration
date, etc. The full regex is in Table I. We set different regexes 9 types of
fields. The final regex is the concatenation of the regexes using the OR
operator.

We tested the WFST with different biasing strengths $\alpha$ and measure their
performance in terms of word error rate (WER). The recognition model is the
printed model. To test the efficacy of regex biasing on matching text lines
and its side effects on other text lines, we divide the full set into two
subsets, “fields” and “non-fields”, and calculate their WERs respectively.

Table II: Word error rate (%) on the driver license dataset. “Full” is the full dataset; “Fields” is the subset where text matches specified regex, partially or fully; “Non-fields” is the subset where the text does not match the regex. Relative WER change to “no bias” WER is displayed in percentages. Subset | No bias | $\alpha=0$ | $-1$ | $-3$ | $-5$ | $-10$
---|---|---|---|---|---|---
Full | 5.5 | 5.0 | 4.9 | 4.9 | 4.9 | 5.6
-9% | -11% | -11% | -11% | +2%
Fields | 5.7 | 4.1 | 3.8 | 3.8 | 3.7 | 4.1
-28% | -33% | -33% | -35% | -28%
Non-fields | 5.4 | 5.4 | 5.4 | 5.4 | 5.5 | 6.4
0% | 0% | 0% | +2% | +19%

Table II summarizes the results. Without regex biasing, the WER on the full
set is 5.5, and the WERs on the two subsets are close. When regex biasing is
applied with $\alpha=0$, the WER on fields drops to 4.1%, while the WER on
non-fields does not change. It is worth noting that, since the weights in the
main language model are mostly positive, setting $\alpha=0$ still creates a
bias.

As we further increase the biasing strength, the WER on fields drops to 3.7%
when $\alpha=-5$, reducing 35% of the errors compared with no biasing, while
the WER on non-fields slightly increases to 5.5% due to false positives, i.e.
text that is mistakenly recognized to match the regex. The overall WER is at
its lowest 4.9%.

Beyond $\alpha=-5$, the number of false positives drastically increases, and
the overall WER increases, indicating that the regex biasing is too strong and
creates many false positives. On the other hand, the WER on matching fields
also increases beyond $\alpha=-5$. There are two factors behind this increase:
1) False positives appear in the part where the text does not match the regex;
2) Because beam search sets a maximum weight difference between the allowed
candidates and the best one, a lower $\alpha$ will lead to the pruning of some
correct paths. As we increase the beam size, the WER on fields gets closer to
that of $\alpha=-5$.

The optimal setting of $\alpha$ depends on the application. Lower $\alpha$
improves the accuracy of text matching the regex at the cost of increased
false positives. But as a rule of thumb, $\alpha$ in the range of $[-3,0]$
significantly reduces errors on matching text and brings limited regression.

Figure 5 shows the efficacy and side effects of regex biasing through
examples. With regex biasing, the recognizer correctly recognizes some highly
challenging examples which are hard to recognize without prior knowledge of
their formats. On the other hand, over biasing leads to more false positives.
A failure case of such is displayed in the last row.

Figure 5: Examples of regex biasing on the driver license dataset. “Biased”
results are from regex biasing with $\alpha=-5$. Recognition errors are
highlighted in red color. Table III: Regex definition for passport MRZ.
Definitions for “yy”, “mm”, and “dd” are reused from Table I.

###  IV-D Regex biasing for passport MRZ

Passport MRZ follows the international standard ISO/IEC 7501-1. The standard
specifies the number of characters (44) and the allowed characters at each
position. Using the standard, we wrote the regexes shown in Table III and
compile the final regex mrz_line into a WFST.

We measure the recognition performance on this dataset using character error
rate (CER). We used the printed model for this dataset. Table IV summarizes
the results under different $\alpha$. Again, we observe a significant
improvement in character error rate after biasing. Character error rates are
reduced by 36% at $\alpha=-2$. We also observed that the biased recognizer
avoided many common mistakes in the unbiased recognizer, such as confusing “0”
with “O”. As $\alpha$ further decreases, we see an increase in CER, also
caused by early pruning in the beam search.

Table IV: Character error rate (%) on the MRZ dataset. | no bias | $\alpha=0$ | $-1$ | $-2$ | $-3$ | $-5$
---|---|---|---|---|---|---
Full | 8.5 | 5.4 | 5.4 | 5.4 | 5.5 | 6.8
-36% | -37% | -36% | -35% | -20%

###  IV-E Biasing a domain-specific vocabulary

Regex biasing can be used for biasing a domain-specific vocabulary. A list of
words can be expressed as a regex by joining them with the OR operator, as in
“<word1>|<word2>| ... |<wordN>”. Biasing to such regexes is useful for
recognizing words in a domain-specific vocabulary. For example, when trying to
recognize handwritten prescriptions, we can set a domain vocabulary of common
drug names and bias them by setting $\alpha$.

We use the IAM dataset to simulate such a scenario. Part of the text in IAM
comes from a novel. Some character names have uncommon spellings and therefore
are recognized with lower accuracies. Using the 37 character names, we set the
regex shown in Table V.

Table V: Regex definition for IAM.

We measure the performance of regex biasing using two metrics: 1) the error
rate on the 38 names, calculated by the sum of insertion, deletion, and
substitution error, divided by the number of name appearances; 2) the WER on
the whole dataset. The results are summarized in Table VI. As the biasing
strength increases, we see a significant drop in the errors on the names. The
WER on the full set also goes down until $\alpha=-5$. As $\alpha$ further
decreases, the errors on names keep decreasing but the number of false
positives increases and the overall WER also increases.

Some examples are shown in Figure 6. With regex biasing, the recognizer can
recognize highly ambiguous words from the domain vocabulary, while not
affecting the recognition of other words.

Table VI: Name error rate (%) and full set WER (%) on IAM. Subset | no bias | $\alpha=0$ | $-1$ | $-3$ | $-5$ | $-10$ | $-20$
---|---|---|---|---|---|---|---
Names | 36.0 | 28.0 | 23.7 | 18.0 | 15.7 | 11.7 | 10.3
-22% | -34% | -50% | -56% | -68% | -71%
Full | 14.3 | 14.2 | 14.1 | 14.0 | 14.0 | 14.0 | 15.8
-1% | -1% | -2% | -2% | -3% | +10%
Figure 6: Examples of regex biasing on IAM. “Biased” results are from regex
biasing with $\alpha=-5$. Recognition errors are highlighted in red color.
Images are framed by thin black lines for clarity.

###  IV-F Runtime analysis

Table VII: Runtime analysis of regex biasing on different datasets. Numbers are the average time (in milliseconds) for recognizing one text line. | no bias | $\alpha=0$ | $-1$ | $-5$
---|---|---|---|---
Driver License | 1.6 | 1.6 | 1.6 | 1.5
Passport MRZ | 14.6 | 12.2 | 5.9 | 1.0
IAM | 6.8 | 7.1 | 6.8 | 6.8

Finally, we analyze the impact of regex biasing on inference time in Table
VII. Overall, the amount of added time by regex biasing ranges from less than
1ms to negative. On the driver license dataset, regex biasing has little
impact on runtime, and as $\alpha$ decreases, the inference time decreases.
This is because a lower $\alpha$ reduces the number of search paths and
therefore accelerates the search process. This phenomenon is more pronounced
on the passport dataset, where the inference time drops by 10 times as
$\alpha$ lowers.

##  V Conclusion

We have proposed a novel method for biasing a recognizer using regular
expressions. This method improves the performance of a recognizer on domain-
specific data with efficacy, requires no labeled data and training process,
and has limited impact on runtime speed.

With some modifications, the decoder we have used in this paper may also work
with auto-regressive text recognition models, such as attention-based
recognizers [17, 25, 21] and RNN transducer models [12]. This can be explored
in the future.

Structured text comes in many other forms, where the proposed biasing method
may find its usage. For example, math equations (represented by Latex code)
are strongly structured. A recognizer may benefit from limiting its search
space to the one defined by the Latex syntax. We are also interested in
exploring this direction in the future.

## References

  * [1] Alfred V. Aho, Ravi Sethi, and Jeffrey D. Ullman.  Compilers: Principles, Techniques, and Tools.  Addison-Wesley series in computer science / World student series edition. Addison-Wesley, 1986.
  * [2] Petar S. Aleksic, Cyril Allauzen, David Elson, Aleksandar Kracun, Diego Melendo Casado, and Pedro J. Moreno.  Improved recognition of contact names in voice commands.  In ICASSP 2015, pages 5172–5175. IEEE, 2015.
  * [3] Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mehryar Mohri.  Openfst: A general and efficient weighted finite-state transducer library.  In Jan Holub and Jan Zdárek, editors, CIAA 2007, volume 4783 of Lecture Notes in Computer Science, pages 11–23. Springer, 2007\.
  * [4] Ashish Arora, Paola García, Shinji Watanabe, Vimal Manohar, Yiwen Shao, Sanjeev Khudanpur, Chun-Chieh Chang, Babak Rekabdar, Bagher BabaAli, Daniel Povey, David Etter, Desh Raj, Hossein Hadian, and Jan Trmal.  Using ASR methods for OCR.  In ICDAR 2019, pages 663–668. IEEE, 2019.
  * [5] Peter F. Brown, Vincent J. Della Pietra, Peter V. de Souza, Jennifer C. Lai, and Robert L. Mercer.  Class-based n-gram models of natural language.  Comput. Linguistics, 18(4):467–479, 1992.
  * [6] Meng Cai and Qiang Huo.  Compact and efficient wfst-based decoders for handwriting recognition.  In ICDAR 2017, pages 143–148. IEEE, 2017.
  * [7] Dogan Can, Victor R. Martinez, Pavlos Papadopoulos, and Shrikanth S. Narayanan.  Pykaldi: A python wrapper for kaldi.  In ICASSP 2018. IEEE, 2018.
  * [8] Kai Chen, Li Tian, Haisong Ding, Meng Cai, Lei Sun, Sen Liang, and Qiang Huo.  A compact CNN-DBLSTM based character model for online handwritten chinese text recognition.  In ICDAR 2017, pages 1068–1073. IEEE, 2017.
  * [9] Zhanzhan Cheng, Fan Bai, Yunlu Xu, Gang Zheng, Shiliang Pu, and Shuigeng Zhou.  Focusing attention: Towards accurate text recognition in natural images.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5086–5094. IEEE Computer Society, 2017.
  * [10] Noam Chomsky.  Syntactic structures.  Walter de Gruyter, 2002.
  * [11] Fu’ze Cong, Wenping Hu, Qiang Huo, and Li Guo.  A comparative study of attention-based encoder-decoder approaches to natural scene text recognition.  In ICDAR 2019, pages 916–921. IEEE, 2019.
  * [12] Alex Graves.  Sequence transduction with recurrent neural networks.  CoRR, abs/1211.3711, 2012.
  * [13] Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In William W. Cohen and Andrew W. Moore, editors, ICML 2006, volume 148 of ACM International Conference Proceeding Series, pages 369–376. ACM, 2006.
  * [14] Ben Haynor and Petar S. Aleksic.  Incorporating written domain numeric grammars into end-to-end contextual speech recognition systems for improved recognition of numeric sequences.  In ICASSP 2020, pages 7809–7813. IEEE, 2020.
  * [15] Taku Kudo and John Richardson.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In Eduardo Blanco and Wei Lu, editors, EMNLP 2018, pages 66–71. Association for Computational Linguistics, 2018.
  * [16] Minghui Liao, Jian Zhang, Zhaoyi Wan, Fengming Xie, Jiajun Liang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  Scene text recognition from two-dimensional perspective.  In AAAI 2019, pages 8714–8721. AAAI Press, 2019.
  * [17] Ron Litman, Oron Anschel, Shahar Tsiper, Roee Litman, Shai Mazor, and R. Manmatha.  SCATTER: selective context attentional scene text recognizer.  In CVPR 2020, pages 11959–11969. IEEE, 2020.
  * [18] Urs-Viktor Marti and Horst Bunke.  The iam-database: an english sentence database for offline handwriting recognition.  Int. J. Document Anal. Recognit., 5(1):39–46, 2002.
  * [19] Mehryar Mohri, Fernando Pereira, and Michael Riley.  Weighted finite-state transducers in speech recognition.  Comput. Speech Lang., 16(1):69–88, 2002.
  * [20] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al.  The kaldi speech recognition toolkit.  In IEEE 2011 workshop on automatic speech recognition and understanding, number CONF. IEEE Signal Processing Society, 2011.
  * [21] Zhi Qiao, Yu Zhou, Dongbao Yang, Yucan Zhou, and Weiping Wang.  SEED: semantics enhanced encoder-decoder framework for scene text recognition.  In CVPR 2020, pages 13525–13534. IEEE, 2020.
  * [22] Brian Roark, Richard Sproat, Cyril Allauzen, Michael Riley, Jeffrey Sorensen, and Terry Tai.  The opengrm open-source finite-state grammar software libraries.  In ACL 2012, pages 61–66. The Association for Computer Linguistics, 2012.
  * [23] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In ACL 2016. The Association for Computer Linguistics, 2016.
  * [24] Baoguang Shi, Xiang Bai, and Cong Yao.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  IEEE Trans. Pattern Anal. Mach. Intell., 39(11):2298–2304, 2017\.
  * [25] Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  ASTER: an attentional scene text recognizer with flexible rectification.  IEEE Trans. Pattern Anal. Mach. Intell., 41(9):2035–2048, 2019\.