# Modified sampling method with near field measurements

Xiaodong Liu111Academy of Mathematics and Systems Science, Chinese Academy of
Sciences, Beijing 100190, China<EMAIL_ADDRESS>Shixu Meng222Academy of
Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190,
China<EMAIL_ADDRESS>and Bo Zhang333Academy of Mathematics and Systems
Science, Chinese Academy of Sciences, Beijing 100190, China and School of
Mathematical Sciences, University of Chinese Academy of Sciences, Beijing
100049, China<EMAIL_ADDRESS>
###### Abstract

This paper investigates the inverse scattering problems using sampling methods
with near field measurements. The near field measurements appear in two
classical inverse scattering problems: the inverse scattering for obstacles
and the interior inverse scattering for cavities. We propose modified sampling
methods to treat these two classical problems using near field measurements
without making any asymptotic assumptions on the distance between the
measurement surface and the scatterers. We provide theoretical justifications
based on the factorization of the near field operator in both symmetric
factorization case and non-symmetric factorization case. Furthermore, we
introduce a data completion algorithm which allows us to apply the modified
sampling methods to treat the limited-aperture inverse scattering problems.
Finally numerical examples are provided to illustrate the modified sampling
methods with both full- and limited- aperture near field measurements.

###### keywords:

inverse scattering, sampling method, near field measurements, limited-aperture

##  1 Introduction

Inverse scattering plays an important role in non-destructive testing, medical
imaging, geophysical exploration and numerous problems associated with target
identification. In the last thirty years, sampling methods for shape
reconstruction in inverse scattering problems have attracted a lot of
interest. Classical examples include the linear sampling method [6], the
singular sources method [25] and the factorization method [15]. The basic idea
is to design an indicator which is large inside the underlying scatterer and
relatively small outside. We refer to the monographs [3, 7, 16] for a
comprehensive introduction. Recently, other types of sampling methods are
proposed for the inverse scattering problems, e.g., orthogonality sampling
[25, 10, 11], direct sampling method [12, 19], single-shot method [17],
reverse time migration [5]. These sampling methods inherit many advantages of
the classical ones, e.g., they are independent of any a priori information on
the geometry and physical properties of the unknown objects. The main feature
of these sampling methods is that only inner product of the measurements with
some suitably chosen functions is involved in the imaging function and thus
these sampling methods are robust to noises. In all of the sampling methods,
the measurements may be divided into two types: far-field measurements and
near field measurements. Generally speaking, the analysis is more rich in the
far-field case compared to the near field case. This paper is devoted to
sampling methods in the near field case with both full- and partial- aperture
data.

Compared to orthogonal/direct sampling methods using far-field measurements
[25, 10, 19, 11], the study on near field orthogonal/direct sampling method is
relatively limited. The near field measurements appear in two classical
inverse scattering problems: the inverse scattering for obstacles (see for
instance the monograph [7]) and the interior inverse scattering for cavity
[27, 26, 18, 4, 24]. Is it possible to treat these two classical inverse
scattering problems using one unified framework? The answer is yes and this is
one of our main results in this paper. We survey literatures on both problems
as follows. Concerning the inverse scattering for obstacles, there have been
considerable works on the reverse time migration [5] and direct sampling
methods [12] using near field measurements. In these works, numerical
algorithms illustrated the performance of the imaging method and the analysis
was done when the measurement surface is not close to the scatterer. Our work
differs from these existing literatures because we propose modified sampling
methods to treat these two classical problems using near field measurements
without making any asymptotic assumptions on the distance between the
measurement surface and the scatterers. One difficulty in this treatment is
the non-symmetric factorization [13, 24] of the near field operator and the
other difficulty is the necessity to consider the “distance” related
fundamental solution (as contrary to considering the plane wave in the far-
field case). We show in this paper how to design a sampling method to overcome
these difficulties. Furthermore, the sampling method is completely
theoretically justified in the sense that the proposed imaging function has
both an upper and lower bound which peak when the sampling point is at the
obstacle boundary. On the other aspect, there seems to be no attempts to
design orthogonal/direct imaging methods in the interior inverse scattering
for cavity where the measurements are in the near field naturally. In the
cavity setting, it is not possible to asymptotically consider the near field
measurements whereby the existing methods [5, 12] do not apply. Fortunately,
our proposed sampling method is in a unified framework, which allows us to
treat the interior inverse scattering for cavity by a similar sampling method
using near field measurements. Once again, this modified sampling method
provides both theoretical justifications and efficient numerical algorithms.
This seems a first attempt on orthogonal/direct sampling methods in the
interior inverse scattering for cavity.

In many cases of practical interest, it is difficult or even impossible to
obtain the full-aperture measurements, this motivates us to consider our
modified sampling methods with limited-aperture measurements. Reconstruction
algorithms have been developed using the limited-aperture data directly [1, 8,
14, 16, 21, 22, 31]. Alternatively, [20, 21] first recover the full-aperture
data and then solve the inverse problems. Recently two novel data completion
algorithms were proposed [9] for the inverse scattering problems in the far-
field case, see also [2] for the waveguide case. We aim to apply a similar
data completion algorithm in the near field case to recover the full-aperture
data and then apply our modified sampling methods.

The paper is further organized as follows. In Section 2, we give the
mathematical formulation of the two classical inverse scattering problems with
near field measurements: the inverse scattering for obstacles and the interior
inverse scattering for cavities. We also provide a preliminary result on the
coercivity estimate of the single-layer operator. The modified sampling
methods using near field measurements are investigated in Section 3 for
obstacles and in Section 4 for cavities, respectively. Such modified sampling
methods give both theoretical justifications and numerical algorithms, no
matter the associated near field operators have symmetric factorizations or
not. The key idea is to design properly chosen functions to overcome the above
two difficulties associated with non-symmetric factorization and “distance”
related fundamental solution. Section 5 is devoted to a data completion
algorithm which recovers the full-aperture data and allows us to apply the
modified sampling methods in the previous sections. Finally, numerical
examples are provided in Section 6 to illustrate the modified sampling methods
with both full- and limited- aperture near field measurements.

##  2 Mathematical model and setup

###  2.1 Mathematical models

We consider two classical inverse scattering problems with near field
measurements: the inverse scattering for obstacles and the interior inverse
scattering for cavities. Throughout the paper we focus on the two dimensional
case. The three dimensional case is similar yet to be done. In both problems,
let $k>0$ be the wave number. A point source $\phi(\cdot;y)$ at $y$ is the
fundamental solution with the following explicit expression

$\phi(x;y)=\frac{i}{4}H^{1}_{0}(k|x-y|),\quad x\not=y,$ (1)

where $H^{1}_{0}$ is the Hankel function of the first kind of order zero [7].

Inverse scattering for obstacles: Let $\Omega\subset\mathbb{R}^{2}$ be an open
and bounded domain with Lipschitz boundary $\partial\Omega$ such that
$\mathbb{R}^{2}\backslash\overline{\Omega}$ is connected. The domain $\Omega$
is referred as the obstacles. The scattering for the obstacles due to a point
source $\phi(\cdot;y)$ is to find scattered wave field $u^{s}(\cdot;y)$ such
that

$\displaystyle\Delta_{x}u^{s}(\cdot;y)+k^{2}u^{s}(\cdot;y)=0\quad$ in
$\displaystyle\quad\mathbb{R}^{2}\backslash\overline{\Omega},\quad$ (2)
$\displaystyle u^{s}(\cdot;y)=-\phi(\cdot;y)\quad$ on
$\displaystyle\quad\partial\Omega,$ (3)
$\displaystyle\lim_{r:=|x|\to\infty}\sqrt{r}\left(\frac{\partial
u^{s}(\cdot;y)}{\partial r}-iku^{s}(\cdot;y)\right)=0.$ (4)

This scattering problem is well-posed, see for instance [7]. Here we have
assumed a sound-soft obstacle.

Let $\partial B:=\\{x:|x|=r_{o},\,r_{o}>0\\}$ be the measurement surface and
$B:=\\{x:|x|<r_{o},\,r_{o}>0\\}$ includes $\Omega$ as its interior. The
inverse problem is to determine $\partial\Omega$ from the following near field
measurements:

$\\{u^{s}(x;y):x,y\in\partial B\\}.$ (5)

The inverse problem has a unique solution [7].

Fig. 1:  Example of the inverse scattering for obstacles and the interior
inverse scattering for a cavity.

Interior inverse scattering for cavities: Let $D\subset\mathbb{R}^{2}$ be an
open, connected and bounded domain with Lipschitz boundary $\partial D$. The
domain $D$ is referred as the cavity. The scattering for the cavity due to a
point source $\phi(\cdot;y)$ is to find scattered wave field $u^{s}(\cdot;y)$
such that

$\displaystyle\Delta_{x}u^{s}(\cdot;y)+k^{2}u^{s}(\cdot;y)=0\quad$ in
$\displaystyle\quad D,\quad$ (6) $\displaystyle
u^{s}(\cdot;y)=-\phi(\cdot;y)\quad$ on $\displaystyle\quad\partial D.$ (7)

If $k^{2}$ is not an eigenvalue of $-\Delta$ in the domain $D$, this
scattering problem is well-posed, see for instance [18, 27]. Again here we
have assumed a sound-soft cavity.

Let $\partial C:=\\{x:|x|=r_{i},\,r_{i}>0\\}$ be the measurement surface and
$C:=\\{x:|x|<r_{i},\,r_{i}>0\\}$ is in the interior of $D$. The inverse
problem is to determine $\partial D$ from the following near field
measurements:

$\\{u^{s}(x;y):x,y\in\partial C\\}.$ (8)

In the interior inverse scattering for cavity problem, we always make the
assumptions that $k^{2}$ is not an eigenvalue of $-\Delta$ in the domain $D$
and $C$. Note that the assumption that $k^{2}$ is not an eigenvalue of
$-\Delta$ in $C$ is not a restriction, since one can always choose $C$ such
that this assumption holds. With these assumptions, the inverse problem has a
unique solution [18, 27].

###  2.2 Estimate of Single-layer operator

Let $\Gamma$ be either $\partial D$ or $\partial\Omega$ and assume that
$k^{2}$ is not an eigenvalue of $-\Delta$ in the domain $\Omega$ and $D$,
respectively. We define $\mathrm{T}_{\Gamma}:H^{1/2}(\Gamma)\to
H^{-1/2}(\Gamma)$ by

$\mathrm{T}_{\Gamma}g=h,$ (9)

where, for any $g\in H^{1/2}(\Gamma)$, $h\in H^{-1/2}(\Gamma)$ is the unique
solution to

$g(x)=\int_{\Gamma}\overline{\phi(x;y)}h(y)\,\mbox{d}s_{y},\quad x\in\Gamma.$
(10)

We note that the above integral equation is uniquely solvable due to that
$k^{2}$ is not an eigenvalue of $-\Delta$ in the domain $D$ or $\Omega$, see
for instance [23, 7, 16].

The following coercivity estimate plays an important role in the analysis of
our imaging function.

######  Lemma 1.

Let $\Gamma$ be either $\partial D$ or $\partial\Omega$. For any $g\in
H^{1/2}(\Gamma)$,

$\big{|}\langle\mathrm{T}_{\Gamma}g,g\rangle_{\Gamma}\big{|}\geq
c_{0}\|g\|^{2}_{H^{1/2}(\Gamma)}.$ (11)

Here $\langle\cdot,\cdot\rangle_{\Gamma}$ denotes the duality paring between
$H^{-1/2}(\Gamma)$ and $H^{1/2}(\Gamma)$.

###### Proof.

We prove the theorem for $\Gamma=\partial\Omega$. The proof of the case that
$\Gamma=\partial D$ is exactly the same.

(a) We first show

$\displaystyle-\Im\langle\mathrm{T}_{\Gamma}g,g\rangle_{\Gamma}<0,\quad\forall
g\not=0.$ (12)

Let $\mathrm{T}_{\Gamma}g=h$. Let
$w^{\pm}(x):=\int_{\partial\Omega}\overline{\phi(x;y)}h(y)\,\mbox{d}s_{y}$ for
$x\in\Omega^{\pm}$ where $\Omega^{-}=\Omega$ and
$\Omega^{+}=\mathbb{R}^{2}\backslash\overline{\Omega}$. From equation (10), we
have from the jump relations of the single layer potential that
$\overline{h}=\frac{\partial\overline{w}^{-}}{\partial\nu}|_{\partial\Omega}-\frac{\partial\overline{w}^{+}}{\partial\nu}|_{\partial\Omega}$
(here $\nu$ is the outward normal to $\partial\Omega$). Then we have that

$\displaystyle-\Im\langle\mathrm{T}_{\Gamma}g,g\rangle_{\Gamma}=\Im\overline{\langle
h,w^{-}\rangle_{\Gamma}}=\Im\langle\frac{\partial\overline{w}^{-}}{\partial\nu}-\frac{\partial\overline{w}^{+}}{\partial\nu},\overline{w}^{-}\rangle_{\Gamma}$
$\displaystyle=$
$\displaystyle\Im\Big{(}\|\nabla\overline{w}\|^{2}_{L^{2}(B_{R})}+\|\overline{w}\|^{2}_{L^{2}(B_{R})}-\int_{\partial
B_{R}}\frac{\partial\overline{w}^{+}}{\partial\nu}{w}\,\mbox{d}s\Big{)}.\mbox{
(Integration by parts)}$ $\displaystyle=$ $\displaystyle-\Im\int_{\partial
B_{R}}\frac{\partial\overline{w}^{+}}{\partial\nu}{w}\,\mbox{d}s=\Im\int_{\partial
B_{R}}\frac{\partial{w}^{+}}{\partial\nu}\overline{w}\,\mbox{d}s.$

The proof is completed if

$\displaystyle-\Im\langle\mathrm{T}_{\Gamma}g,g\rangle_{\Gamma}=\Im\int_{\partial
B_{R}}\frac{\partial{w}^{+}}{\partial\nu}\overline{w}\,\mbox{d}s<0,\quad\forall
g\not=0.$ (13)

This is valid due to the following. Assume on the contrary that

$\displaystyle-\Im\langle\mathrm{T}_{\Gamma}g,g\rangle_{\Gamma}=\Im\int_{\partial
B_{R}}\frac{\partial{w}^{+}}{\partial\nu}\overline{w}\,\mbox{d}s\geq 0,$

Since $\overline{w}$ is a radiating solution to the Helmholtz equation in
$\mathbb{R}^{2}\backslash\overline{B_{R}}$, by [7, Theorem 2.13]
$\overline{w}$ vanishes in $\mathbb{R}^{2}\backslash\overline{B_{R}}$, and by
unique continuation $\overline{w}$ vanishes in
$\mathbb{R}^{2}\backslash\overline{\Omega}$ and hence $w|_{\partial\Omega}=0$.
Since ${w}$ satisfies the Helmholtz equation in $\Omega$, and the assumption
that $-k^{2}$ is not an eigenvalue of $-\Delta$ in $\Omega$, we conclude that
$w$ vanishes in $\Omega$. Recall that $\overline{w}$ vanishes in
$\mathbb{R}^{2}\backslash\overline{\Omega}$, then by the jump relation of the
single layer potential, we conclude that $h$ vanishes and hence $g$ vanishes.
This proves (12).

(b) Note that $\mathrm{T}_{\Gamma}$ is the sum of a strictly coercive self-
adjoint operator and a compact operator [7, Lemma 5.38]. This together with
[3, Lemma 7.28] or [7, Lemma 5.37], we conclude that

$\big{|}\langle\mathrm{T}_{\Gamma}g,g\rangle_{\Gamma}\big{|}\geq
c_{0}\|g\|^{2}_{H^{1/2}(\Gamma)}.$

and this completes the proof. ∎

##  3 Inverse scattering for extended obstacles

###  3.1 Near field operator and its factorization

The near field operator $\mathrm{N}:L^{2}(\partial B)\to L^{2}(\partial B)$ is
denoted by

$\big{(}\mathrm{N}g\big{)}(x):=\int_{\partial
B}u^{s}(x;y)g(y)\,\mbox{d}s_{y},\quad x\in\partial B.$ (14)

To facilitate the analysis, we define $\mathrm{H}:L^{2}(\partial B)\to
H^{1/2}(\partial\Omega)$ by

$\big{(}\mathrm{H}g\big{)}(x):=\int_{\partial
B}\phi(x;y)g(y)\,\mbox{d}s_{y},\quad x\in\partial\Omega,$ (15)

and thereby $\mathrm{H}^{*}:H^{-1/2}(\partial\Omega)\to L^{2}(\partial B)$ is
given by (which can be directly verified again by the definition that
$\langle\mathrm{H}^{*}h,g\rangle_{\partial B}=\langle
h,\mathrm{H}g\rangle_{\partial\Omega}$)

$\big{(}\mathrm{H}^{*}h\big{)}(x)=\int_{\partial\Omega}\overline{\phi(x;y)}h(y)\,\mbox{d}s_{y},\quad\forall
h\in H^{-1/2}(\partial\Omega).$ (16)

For any function $g$ in a complex Banach space, we define the “conjugate
operator” $\mathrm{R}$ by $\mathrm{R}g=\overline{g}$.

The near field operator has the following factorization.

######  Theorem 2.

It holds that

$\mathrm{N}=-\mathrm{R}\mathrm{H}^{*}\mathrm{T}_{\partial\Omega}\mathrm{R}\mathrm{H},$
(17)

where $\mathrm{T}_{\partial\Omega}:H^{1/2}(\partial\Omega)\to
H^{-1/2}(\partial\Omega)$ is defined via (9)–(10).

###### Proof.

Let $w(x):=\int_{\partial B}u^{s}(x;y)g(y)\,\mbox{d}s_{y}$ for
$x\in\mathbb{R}^{2}\backslash\overline{\Omega}$. From the definition of
$\mathrm{H}$ (15) and superposition principle, we have that

$\displaystyle\Delta w+k^{2}w=0\quad$ in
$\displaystyle\mathbb{R}^{2}\backslash\overline{\Omega},\quad$ $\displaystyle
w=-\mathrm{H}g\quad$ on $\displaystyle\quad\partial\Omega,$

and $w$ satisfies the Sommerfeld radiation condition. Thereby we look for

$w(x)=\int_{\partial\Omega}{\phi(x;y)}\psi(y)\,\mbox{d}s_{y}$

with unknown density $\psi\in H^{-1/2}(\partial\Omega)$. From the boundary
condition of $w$ on $\partial\Omega$, we have that

$\int_{\partial\Omega}{\phi(x;y)}\psi(y)\,\mbox{d}s_{y}=-\mathrm{H}g\mbox{ on
}\partial\Omega.$

Taking the conjugate of the above equation yields

$\int_{\partial\Omega}\overline{\phi(x;y)}\overline{\psi(y)}\,\mbox{d}s_{y}=-\overline{\mathrm{H}g}\mbox{
on }\partial\Omega.$

This together with the definition of $\mathrm{T}_{\partial\Omega}$ (9)–(10)
yields that the density is solved by
$\overline{\psi}=\mathrm{T}_{\partial\Omega}(-\overline{\mathrm{H}g})=\mathrm{T}_{\partial\Omega}\mathrm{R}(-\mathrm{H}g)$.
Therefore we have that, for any $x\in\partial B$,

$\displaystyle\big{(}\mathrm{N}g\big{)}(x)$ $\displaystyle=$ $\displaystyle
w(x)=\mathrm{R}\overline{w}=\mathrm{R}\int_{\partial\Omega}\overline{\phi(x;y)}\overline{\psi(y)}\,\mbox{d}s_{y}=\mathrm{R}\int_{\partial\Omega}\overline{\phi(x;y)}\big{(}\mathrm{T}_{\partial\Omega}\mathrm{R}(-\mathrm{H}g)\big{)}\,\mbox{d}s_{y}$
$\displaystyle=$
$\displaystyle-\mathrm{R}\mathrm{H}^{*}\mathrm{T}_{\partial\Omega}\mathrm{R}\mathrm{H}g.$

This completes the proof. ∎

There are other ways to factorize the near field operator [13] using near
field measurements. The key point is that these factorizations are non-
symmetric factorizations which is in contrast to the symmetric factorization
using far-field measurements. We show in the following section how to design a
sampling method even though the factorization is non-symmetric. Furthermore,
the sampling method is completely theoretically justified in the sense that
the proposed imaging function has both an upper and lower bound which peak
when the sampling point is at the obstacle boundary.

###  3.2 Imaging function

We introduce the imaging function

$I_{obstacle}(z):=\big{|}\langle\mathrm{N}\varphi_{z},\overline{\varphi_{z}}\rangle_{\partial
B}\big{|}$ (18)

where $\langle\cdot,\cdot\rangle_{\partial B}$ denotes the $L^{2}(\partial
B)$-inner product, and $\varphi_{z}$ is given by

$\varphi_{z}(y):=\sum_{n=-M}^{M}\frac{4}{i|y|\pi(1+\delta_{0n})}\frac{J_{n}(k|z|)}{H_{n}^{(1)}(k|y|)}\cos(n\theta_{yz}),\quad
y\in\partial B$ (19)

with $M$ chosen to be a positive integer and $\theta_{xy}$ denoting the angle
between $x$ and $y$ for any $x,y\in\mathbb{R}^{2}$. Moreover we also consider

$\varphi_{z,\infty}(y):=\sum_{n=-\infty}^{\infty}\frac{4}{i|y|\pi(1+\delta_{0n})}\frac{J_{n}(k|z|)}{H_{n}^{(1)}(k|y|)}\cos(n\theta_{yz}),\quad
y\in\partial B,$ (20)

which is the infinite series version of (19).

######  Remark 1.

The function $\varphi_{z,\infty}$ (20) is well-defined. Indeed, note that
$H_{n}^{(1)}(k|y|)=H_{n}^{(1)}(kr_{o})$ does not vanish, so first
$\varphi_{z}(y)$ (19) is well-defined. Letting $M\to+\infty$, we have from the
asymptotic of Hankel and Bessel functions [7, Section 3.5] that

$\displaystyle H_{n}^{(1)}(kr_{0})\sim\frac{2^{n}(n-1)!}{\pi
i(kr_{o})^{n}},\quad J_{n}(k|z|)\sim\frac{(k|z|)^{n}}{2^{n}n!},\quad n\gg 1,$

which yields that

$\frac{J_{n}(k|z|)}{H_{n}^{(1)}(k|y|)}\sim\frac{(k|z|)^{n}\pi
i(kr_{o})^{n}}{2^{n}n!2^{n}(n-1)!}$

and hence by ratio test $\varphi_{z,\infty}$ (20) is well-defined for sampling
points $z$ in a bounded sampling region.

The next theorem gives the explicit expressions of $\mathrm{H}\varphi_{z}(x)$
and $\mathrm{H}\varphi_{z,\infty}(x)$.

######  Theorem 3.

It holds that

$\Big{(}\mathrm{H}\varphi_{z}\Big{)}(x)=\sum_{n=-M}^{M}J_{n}(k|x|)J_{n}(k|z|)\cos(n\theta_{xz}),\quad
x\in\partial\Omega,$ (21)

and

$\Big{(}\mathrm{H}\varphi_{z,\infty}\Big{)}(x)=\sum_{n=-\infty}^{\infty}J_{n}(k|x|)J_{n}(k|z|)\cos(n\theta_{xz})=J_{0}(k|x-z|),\quad
x\in\partial\Omega.$ (22)

Here the convergence is uniformly for $x\in\partial\Omega$ and $z$ in a
bounded sampling region.

###### Proof.

We first prove (21). From [3, 7],

$\phi(x;y)=\frac{i}{4}\sum_{n=-\infty}^{\infty}H_{n}^{(1)}(k|y|)J_{n}(k|x|)\cos(n\theta_{xy}),\quad
x\in\partial\Omega,y\in\partial B.$

This together with (19) yields that

$\displaystyle\Big{(}\mathrm{H}\varphi_{z}\Big{)}(x)$ $\displaystyle=$
$\displaystyle\int_{\partial B}\phi(x;y)\varphi_{z}(y)\,\mbox{d}s_{y}$
$\displaystyle=$ $\displaystyle
r_{o}\int_{-\pi}^{\pi}\Big{(}\frac{i}{4}\sum_{n=-\infty}^{\infty}H_{n}^{(1)}(kr_{o})J_{n}(k|x|)\cos(n\theta_{xy})\Big{)}$
$\displaystyle\qquad\times\Big{(}\sum_{n=-M}^{M}\frac{4}{ir_{o}\pi(1+\delta_{0m})}\frac{J_{n}(k|z|)}{H_{n}^{(1)}(kr_{o})}\cos(m\theta_{yz})\Big{)}\,\mbox{d}\theta_{y}$
$\displaystyle=$
$\displaystyle\sum_{n=-M}^{M}J_{n}(k|x|)J_{n}(k|z|)\cos(n\theta_{xz}),$

where in the last step we have applied the orthogonality property that

$\displaystyle\int_{-\pi}^{\pi}\cos(n\theta_{xy})\cos(m\theta_{yz})\,\mbox{d}\theta_{y}$
(23) $\displaystyle=$
$\displaystyle\int_{-\pi}^{\pi}\big{(}\cos(n\theta_{x})\cos(n\theta_{y})+\sin(n\theta_{x})\sin(n\theta_{y})\big{)}\big{(}\cos(m\theta_{z})\cos(m\theta_{y})+\sin(m\theta_{z})\sin(m\theta_{y})\big{)}\,\mbox{d}\theta_{y}$
$\displaystyle=$
$\displaystyle\cos(n\theta_{x})\cos(n\theta_{z})\pi(1+\delta_{0n})+\sin(n\theta_{x})\sin(n\theta_{z})\pi=\pi(1+\delta_{0n})\cos(n\theta_{xz}),$

where $\delta$ is the Kronecker delta. This completes the proof of (21).
Letting $M\to\infty$ and using the asymptotic of the Bessel function [7,
Section 3.5], we prove

$\Big{(}\mathrm{H}\varphi_{z,\infty}\Big{)}(x)=\sum_{n=-\infty}^{\infty}J_{n}(k|x|)J_{n}(k|z|)\cos(n\theta_{xz}),\quad
x\in\partial\Omega,$

where the convergence is uniformly for $x\in\partial\Omega$ and $z$ in a
bounded sampling region. Note that [3, 7] when $|x|>|z|>0$,

$H_{0}^{(1)}(k|x-z|)=\sum_{n=-\infty}^{\infty}H_{n}^{(1)}(k|x|)J_{n}(k|z|)\cos(n\theta_{xz}),\quad|x|>|z|>0$

and when $|z|>|x|>0$

$H_{0}^{(1)}(k|x-z|)=\sum_{n=-\infty}^{\infty}H_{n}^{(1)}(k|z|)J_{n}(k|x|)\cos(n\theta_{zx}),\quad|z|>|x|>0,$

we then take the real parts of the above equations to get

$\sum_{n=-\infty}^{\infty}J_{n}(k|x|)J_{n}(k|z|)\cos(n\theta_{xz})=J_{0}(k|x-z|),\quad
x\in\partial\Omega.$

This proves (22). This completes the proof. ∎

The following theorem plays an important role in the analysis of the imaging
function. Recall the assumption that $k^{2}$ is not an eigenvalue of $-\Delta$
in the domain $\Omega$.

######  Theorem 4.

There exists positive constants $c^{\prime}_{1},c^{\prime\prime}_{1}$ and
$c^{\prime}_{2},c^{\prime\prime}_{2}$ independent of $z$ such that

$c^{\prime}_{1}\|\mathrm{H}\varphi_{z}\|^{2}_{H^{1/2}(\partial\Omega)}\leq\big{|}I_{obstacle}(z)\big{|}\leq
c^{\prime}_{2}\|\mathrm{H}\varphi_{z}\|^{2}_{H^{1/2}(\partial\Omega)}.$ (24)

and

$c^{\prime\prime}_{1}\|\mathrm{H}\varphi_{z,\infty}\|^{2}_{H^{1/2}(\partial\Omega)}\leq\big{|}I_{obstacle}(z)\big{|}\leq
c^{\prime\prime}_{2}\|\mathrm{H}\varphi_{z,\infty}\|^{2}_{H^{1/2}(\partial\Omega)}.$
(25)

###### Proof.

From the factorization of $\mathrm{N}$, we have that

$-\langle\mathrm{N}\varphi_{z},\overline{\varphi_{z}}\rangle_{\partial
B}=\langle\mathrm{R}\mathrm{H}^{*}\mathrm{T}_{\partial\Omega}\mathrm{R}\mathrm{H}\varphi_{z},\overline{\varphi_{z}}\rangle_{\partial
B}=\overline{\langle\mathrm{H}^{*}\mathrm{T}_{\partial\Omega}\mathrm{R}\mathrm{H}\varphi_{z},\varphi_{z}\rangle}_{\partial
B}=\overline{\langle\mathrm{T}_{\partial\Omega}\mathrm{R}\mathrm{H}\varphi_{z},\mathrm{H}\varphi_{z}\rangle}_{\partial\Omega}.$
(26)

By Theorem 3, we have that $\mathrm{H}\varphi_{z}$ is indeed a real-valued
function in $H^{1/2}(\partial\Omega)$, therefore

$\displaystyle\big{|}I_{obstacle}(z)\big{|}=\big{|}\langle\mathrm{T}_{\partial\Omega}\mathrm{H}\varphi_{z},\mathrm{H}\varphi_{z}\rangle\big{|}.$

Therefore we can apply the coercivity estimate in Lemma 1 to derive that

$\displaystyle\big{|}I_{obstacle}(z)\big{|}=\big{|}\langle\mathrm{T}_{\partial\Omega}\mathrm{H}\varphi_{z},\mathrm{H}\varphi_{z}\rangle\big{|}\geq
c^{\prime}_{1}\|\mathrm{H}\varphi_{z}\|^{2}_{H^{1/2}(\partial\Omega)}.$

The upper bound follows from the fact that $\mathrm{T}_{\partial\Omega}$ is
bounded. Note that both the positive constants $c^{\prime}_{1}$ and
$c^{\prime}_{2}$ are independent of $z$. Using exactly the same argument, the
same estimate (with constants $c^{\prime\prime}_{1}$ and
$c^{\prime\prime}_{2}$) also holds when replacing $\varphi_{z}$ by
$\varphi_{z,\infty}$. This completes the proof. ∎

######  Remark 2.

Though the factorization (17) of the near field operator is non-symmetric, we
are still able to design our modified sampling method. The key point is that
$\mathrm{H}\varphi_{z}$ and $\mathrm{H}\varphi_{z,\infty}$ are indeed real-
valued functions in $H^{1/2}(\partial\Omega)$ by Theorem 3. Furthermore, the
sampling method is completely theoretically justified in the sense that the
proposed imaging function has both an upper and lower bound which peak when
the sampling point is at the obstacle boundary.

It is seen from Theorem 3 that

$\Big{(}\mathrm{H}\varphi_{z,\infty}\Big{)}(x)=J_{0}(k|x-z|),\quad
x\in\partial\Omega,$ (27)

and when $M$ is large,

$\Big{(}\mathrm{H}\varphi_{z}\Big{)}(x)\approx J_{0}(k|x-z|),\quad
x\in\partial\Omega.$ (28)

In this sense, we find that when the sampling point $z$ approaches
$x\in\partial\Omega$, $\big{(}\mathrm{H}\varphi_{z}\big{)}(x)$ peaks when $z$
coincides with $x$. Unfortunately, it is not possible to plot
$\mathrm{H}\varphi_{z}\|^{2}_{H^{1/2}(\partial\Omega)}$ since $\Omega$ is the
unknown obstacle. What is significant about Theorem 4 is that it shows that
the imaging function $I_{obstacle}(z)$ is qualitatively the same as
$\|\mathrm{H}\varphi_{z}\|^{2}_{H^{1/2}(\partial\Omega)}$ without the
knowledge of $\Omega$. Therefore from Theorem 4 and Theorem 3, we conclude
that the imaging function $I_{obstacle}(z)$ peaks when the sampling point $z$
is at the obstacle boundary $\partial\Omega$.

######  Remark 3.

The main feature of sampling methods in the literature [5, 12] is that only
inner product of the measurements with some suitably chosen functions is
involved in the imaging function. In these works, numerical algorithms
illustrated the performance of the imaging method and the analysis was done
when the measurement surface is not close to the scatterer, which might be due
to that those suitably chosen functions are “distance” related function (as
contrary to considering the plane wave in the far-field case), e.g. depending
on the distance between the measurement surface and the scatterers. The chosen
functions $\varphi_{z,\infty}$ (20) and $\varphi_{z}$ (19) are different from
the existing chosen functions in the literature [5, 12] and they play a very
important role in the analysis of our modified sampling method. Though our
chosen $\varphi_{z,\infty}$ and $\varphi_{z}$ are also “distance” related,
they “cancel” such “distance” together with $\mathrm{H}\varphi_{z}(x)$ and
$\mathrm{H}\varphi_{z,\infty}(x)$ without making any asymptotic assumptions on
the distance between the measurement surface and the scatterers (see Theorem
3).

######  Remark 4.

In the sampling method using far-field measurements [19], it is seen from the
Jacob-Anger expansion of $e^{ikx\cdot d}$ that the imaging function $I(z)$ in
[19] in fact has lower and upper bound given by

$c_{1}\Big{\|}\int_{|d|=1}e^{ikx\cdot d}e^{-ikz\cdot
d}\,\mbox{d}s_{d}\Big{\|}^{2}_{H^{1/2}(\partial\Omega)}\leq I(z)\leq
c_{2}\Big{\|}\int_{|d|=1}e^{ikx\cdot d}e^{-ikz\cdot
d}\,\mbox{d}s_{d}\Big{\|}^{2}_{H^{1/2}(\partial\Omega)},$

and

$\Big{\|}\int_{|d|=1}e^{ikx\cdot d}e^{-ikz\cdot
d}\,\mbox{d}s_{d}\Big{\|}_{H^{1/2}(\partial\Omega)}\sim
c\Big{\|}J_{0}(k|x-z|)\Big{\|}_{H^{1/2}(\partial\Omega)}\mbox{ for some
constant }c.$

Together with (22) in Theorem 3, we find that this resolution with far-field
measurements is in fact the same as the resolution using our modified sampling
method with near field measurements. This theoretical result is further
confirmed via numerical examples in Section 6.

Finally we summarize the imaging algorithm.

Imaging Algorithm for obstacles.

  * •

Collect the near field measurements $u^{s}(x;y),\,x,y\in\partial B$.

  * •

Select a sampling region in $\mathbb{R}^{2}$ with a fine mesh containing
$\Omega$.

  * •

Compute the imaging function $I_{obstacle}(z)$ in (18) with $\varphi_{z}$
given by (19) for all sampling points.

  * •

Plot the imaging function $I_{obstacle}(z)$ over the sampling region to
determine $\Omega$.

##  4 Interior inverse scattering for cavities

###  4.1 Near field operator and its factorization

The near field operator $\mathrm{N}_{\partial C}:L^{2}(\partial C)\to
L^{2}(\partial C)$ is denoted by

$\big{(}\mathrm{N}_{\partial C}g\big{)}(x):=\int_{\partial
C}u^{s}(x;y)g(y)\,\mbox{d}s_{y},\quad x\in\partial C.$ (29)

To facilitate the analysis, we define $\mathrm{S}:L^{2}(\partial C)\to
H^{1/2}(\partial D)$ by

$\big{(}\mathrm{S}g\big{)}(x):=\int_{\partial
C}\phi(x;y)g(y)\,\mbox{d}s_{y},\quad x\in\partial D,$ (30)

and thereby $\mathrm{S}^{*}:H^{-1/2}(\partial D)\to L^{2}(\partial C)$ is
given by (which can be directly verified by the definition that
$\langle\mathrm{S}^{*}h,g\rangle_{\partial C}=\langle
h,\mathrm{S}g\rangle_{\partial D}$)

$\big{(}\mathrm{S}^{*}h\big{)}(x)=\int_{\partial
D}\overline{\phi(x;y)}h(y)\,\mbox{d}s_{y},\quad\forall h\in H^{-1/2}(\partial
D).$ (31)

The near field operator has the following factorization.

######  Theorem 5.

It holds that

$\mathrm{N}_{\partial C}=-\mathrm{S}^{*}\mathrm{T}_{\partial D}\mathrm{S},$
(32)

where $\mathrm{T}_{\partial D}:H^{1/2}(\partial D)\to H^{-1/2}(\partial D)$ is
defined via (9)–(10).

###### Proof.

The proof is almost the same as the proof of Theorem 2. For completeness we
outline the proof. Let $w(x):=\int_{\partial C}u^{s}(x;y)g(y)\,\mbox{d}s_{y}$
for $x\in D$. From the definition of $\mathrm{S}$ (30) and superposition
principle, we have that

$\displaystyle\Delta w+k^{2}w=0\quad$ in $\displaystyle\quad D,\quad$
$\displaystyle w=-\mathrm{S}g\quad$ on $\displaystyle\quad\partial D.$

This is a classical boundary value problem which can be solved via integral
equation method. In particular, we look for

$w(x)=\int_{\partial D}\overline{\phi(x;y)}\psi(y)\,\mbox{d}s_{y}$

with unknown density $\psi\in H^{-1/2}(\partial D)$. From the boundary
condition of $w$ on $\partial D$, we have that

$\int_{\partial
D}\overline{\phi(x;y)}\psi(y)\,\mbox{d}s_{y}=-\mathrm{S}g\mbox{ on }\partial
D.$

This together with the definition of $\mathrm{T}_{\partial D}$ (9)–(10) yields
that the density is solved by $\psi=\mathrm{T}_{\partial D}(-\mathrm{S}g)$.
Therefore we have that, for any $x\in\partial C$,

$\big{(}\mathrm{N}_{\partial C}g\big{)}(x)=w(x)=\int_{\partial
D}\overline{\phi(x;y)}\psi(y)\,\mbox{d}s_{y}=\int_{\partial
D}\overline{\phi(x;y)}\big{(}\mathrm{T}_{\partial
D}(-\mathrm{S}g)\big{)}\,\mbox{d}s_{y}=-\mathrm{S}^{*}\mathrm{T}_{\partial
D}\mathrm{S}g.$

This completes the proof. ∎

###  4.2 Imaging function

The imaging function is given by

$I_{cavity}(z):=\big{|}\langle\mathrm{N}_{\partial
C}\psi_{z},\psi_{z}\rangle_{\partial C}\big{|}$ (33)

where $\langle\cdot,\cdot\rangle_{\partial C}$ denotes the $L^{2}(\partial
C)$-inner product, and $\psi_{z}$ is given by

$\psi_{z}(y):=\sum_{n=-\mathfrak{M}}^{\mathfrak{M}}\frac{4}{i|y|\pi(1+\delta_{0n})}\frac{J_{n}(k|z|)}{J_{n}(k|y|)}\cos(n\theta_{yz}),\quad
y\in\partial C,$ (34)

where $\mathfrak{M}$ is chosen to be a positive integer. Note that $k^{2}$ is
not an eigenvalue of $-\Delta$ in $C$, therefore $J_{n}(k|y|)=J_{n}(kr_{i})$
never vanishes for any $n$, i.e. $\psi_{z}$ (34) is well-defined. We now prove
the resolution analysis result for the cavity case.

######  Theorem 6.

There exists positive constants $c_{1}$ and $c_{2}$ independent of $z$ such
that

$c_{1}\|\mathrm{S}\psi_{z}\|^{2}_{H^{1/2}(\partial D)}\leq I_{cavity}(z)\leq
c_{2}\|\mathrm{S}\psi_{z}\|^{2}_{H^{1/2}(\partial D)}.$ (35)

###### Proof.

From the factorization of $\mathrm{N}_{\partial C}$ (32), we have that

$-\langle\mathrm{N}_{\partial C}\psi_{z},\psi_{z}\rangle_{\partial
C}=\langle\mathrm{S}^{*}\mathrm{T}_{\partial
D}\mathrm{S}\psi_{z},\psi_{z}\rangle_{\partial C}=\langle\mathrm{T}_{\partial
D}\mathrm{S}\psi_{z},\mathrm{S}\psi_{z}\rangle_{\partial D}.$ (36)

Therefore we can apply the coercivity estimate in Lemma 1 to derive that

$\displaystyle I_{cavity}(z)=\big{|}\langle\mathrm{T}_{\partial
D}\mathrm{S}\psi_{z},\mathrm{S}\psi_{z}\rangle_{\partial D}\big{|}\geq
c_{1}\|\mathrm{S}\psi_{z}\|^{2}_{H^{1/2}(\partial D)}.$

The upper bound follows from the fact that $\mathrm{T}_{\partial D}$ is
bounded. Note that both the positive constants $c_{1}$ and $c_{2}$ are
independent of $z$. This completes the proof. ∎

Theorem 6 states that the imaging function $I(z)$ is qualitatively the same as
$\|\mathrm{S}\psi_{z}\|_{H^{1/2}(\partial D)}$. The next theorem gives the
explicit expression of $\mathrm{S}\psi_{z}(x)$.

######  Theorem 7.

$\Big{(}\mathrm{S}\psi_{z}\Big{)}(x)=\sum_{n=-\mathfrak{M}}^{\mathfrak{M}}H_{n}^{(1)}(k|x|)J_{n}(k|z|)\cos(n\theta_{xz}),\quad
x\in\partial D.$ (37)

###### Proof.

From [3, 7],

$\phi(x;y)=\frac{i}{4}\sum_{n=-\infty}^{\infty}H_{n}^{(1)}(k|x|)J_{n}(k|y|)\cos(n\theta_{xy}),\quad
x\in\partial D,y\in\partial C.$

This together with (34) yields that

$\displaystyle\Big{(}\mathrm{S}\psi_{z}\Big{)}(x)$ $\displaystyle=$
$\displaystyle\int_{\partial C}\phi(x;y)\psi_{z}(y)\,\mbox{d}s_{y}$
$\displaystyle=$ $\displaystyle
r_{i}\int_{-\pi}^{\pi}\Big{(}\sum_{n=-\infty}^{\infty}H_{n}^{(1)}(k|x|)J_{n}(kr_{i})\cos(n\theta_{xy})\Big{)}$
$\displaystyle\qquad\times\Big{(}\sum_{n=-\mathfrak{M}}^{\mathfrak{M}}\frac{1}{r_{i}\pi(1+\delta_{0m})}\frac{J_{n}(k|z|)}{J_{n}(kr_{i})}\cos(m\theta_{yz})\Big{)}\,\mbox{d}\theta_{y}$
$\displaystyle=$
$\displaystyle\sum_{n=-\mathfrak{M}}^{\mathfrak{M}}H_{n}^{(1)}(k|x|)J_{n}(k|z|)\cos(n\theta_{xz}),$

where we have applied the orthogonality property (23). This completes the
proof. ∎

When $\mathfrak{M}$ is large, it can be seen from the series expansion of
Hankel and Bessel function that

$\Big{(}\mathrm{S}\psi_{z}\Big{)}(x)\approx
H_{0}^{(1)}(k|x-z|),\quad|x|>|z|,\quad x\in\partial D.$ (38)

We find that, as the sampling point $z$ approaches $x\in\partial D$ from the
interior of the cavity $D$, $\big{(}\mathrm{S}\psi_{z}\big{)}(x)$ peaks when
$z$ coincides with $x$. Therefore from Theorem 6 and Theorem 7, we may
conclude that the imaging function $I_{cavity}(z)$ peaks when the sampling
point $z$ approaches the boundary $\partial D$ from the interior of the cavity
$D$.

######  Remark 5.

We have seen a difference between the obstacle case and the cavity case. In
the obstacle case, we have considered the limit case (20) by letting
$M\to\infty$ in (19). This is not feasible in the cavity case since we have
the following asymptotic for $|y|=r_{i}$

$\frac{J_{n}(k|z|)}{J_{n}(k|y|)}\sim\left(\frac{|z|}{|r_{i}|}\right)^{n},\quad
n\gg 1$ (39)

which does not allow us to take the limit $\mathfrak{M}$ $\to\infty$ in the
definition of $\psi_{z}$ (34) for all sampling points $z$. It is the same
reason that would limit us to consider a $\mathfrak{M}$ that is not too large
in the numerical examples, since otherwise $\psi_{z}$ (34) would become too
large which may lead to numerical instability. Since the radius $r_{i}$ of $C$
appears in the denominator of of the asymptotic (39), smaller $r_{i}$ may lead
to numerical instability for noisy measurements. We further illustrate these
in the numerical examples section.

We end this section by summarizing the following imaging algorithm.

Imaging Algorithm for the cavity.

  * •

Collect the near field measurements $u^{s}(x;y),\,x,y\in\partial C$.

  * •

Select a sampling region in $\mathbb{R}^{2}$ with a fine mesh containing $D$.

  * •

Compute the imaging function $I_{cavity}(z)$ in (33) with $\psi_{z}$ given by
(34) for all sampling points.

  * •

Plot the imaging function $I_{cavity}(z)$ over the sampling region to
determine $D$.

##  5 Imaging with limited-aperture data

In this section we consider imaging with the following limited-aperture
“backscattering” data

$\\{u^{s}(x;y):\,x,y\in\Gamma_{0}\\}$ (40)

where $\Gamma_{0}:=\\{r(\cos\alpha,\sin\alpha):\theta\in[-\alpha,\alpha]\\}$
is a subset of the measurement circle $\Gamma$ (which is either $\partial C$
or $\partial B$), here $0<\alpha\leq\pi$ ($\alpha=\pi$ gives full-aperture
data) and we denote the radius of $\Gamma$ by $r$ in this section without the
danger of confusion. In the following, we first apply a data completion
algorithm to recover the full-aperture data in appropriate sense and then
apply the modified sampling methods in the previous sections. The idea of the
data completion algorithm is to represent the full-aperture data in the form
of Fourier series, and to relate the corresponding Fourier coefficients to the
limited-aperture data via a prolate matrix. This is in a similar fashion to
the far-field case [9] which considers two data completion algorithms: one
based on Fourier series and the other one based on boundary integral equation.
For completeness and to shed light on the near field case, we give the details
on the data completion algorithm based on Fourier series in a modified way. We
refer to [9] for more related data completion algorithms.

######  Notation 8.

In this section we are going to work with matrices and vectors with negative
indexes for notational convenience. For $-N_{1}\leq n\leq N_{2}$, we denote by

$b:=\Big{(}b_{n}\Big{)},\quad-N_{1}\leq n\leq N_{2}$

as a $N_{1}+N_{2}+1$ dimensional vector.

For $-M_{1}\leq m\leq M_{2}$ and $-N_{1}\leq n\leq N_{2}$, we denote by

$A:=\Big{(}A_{mn}\Big{)},\quad-M_{1}\leq m\leq M_{2},-N_{1}\leq n\leq N_{2}$

as a $(M_{1}+M_{2}+1)\times(N_{1}+N_{2}+1)$ dimensional matrix.

Limited-aperture data using Fourier basis: To begin with, we introduce polar
coordinates $x=r\cos\theta_{x}$ with $\theta_{x}\in(-\pi,\pi)$. At a fixed
$y\in\Gamma_{0}$, for the limited-aperture backscattering far field
measurements $u^{s}(x;y)$ with $\theta_{x}\in[-\alpha,\alpha]$, we introduce
the the infinite dimensional vector $C^{\alpha}_{\infty}$ with $p$-th entry
given by $c^{\alpha}_{p}$

$c_{p}^{\alpha}:=\int_{-\alpha}^{\alpha}u^{s}(x;y)\overline{\phi_{p}(\theta_{x})}\,\mbox{d}s_{\theta_{x}},\qquad
p=0,\pm 1,\cdots,$ (41)

here the Fourier basis is given by
$\phi_{p}(\theta)=\frac{1}{\sqrt{2\pi}}e^{ip\theta},\quad p=0,\pm 1,\pm
2,\cdots$.

Full-aperture data using Fourier basis: At a fixed $y\in\Gamma_{0}$, for the
full-aperture backscattering far field measurements $u^{s}(x,y)$ where
$x\in\Gamma$, we introduce the infinite dimensional vector $C_{\infty}$ with
$p$-th entry given by $c_{p}$

$c_{p}:=\int_{-\pi}^{\pi}u^{s}(x;y)\overline{\phi_{p}(\theta_{x})}\,\mbox{d}s_{\theta_{x}},\qquad
p=0,\pm 1,\cdots.$ (42)

The full-aperture backscattering far field measurements in the Fourier basis
correspond to the infinite dimensional matrix $C_{\infty}$.

Furthermore, given the knowledge of $C_{\infty}$, we can write down $u^{s}$ at
a fixed $y\in\Gamma_{0}$ in Fourier series as

$u^{s}(x;y)=\sum_{m=-\infty}^{\infty}c_{m}\phi_{m}(\theta_{x}),\quad\theta_{x}\in(-\pi,\pi),$
(43)

and approximate $u^{s}$ using a truncated Fourier series as

$u^{s}(x;y)\approx\sum_{m=-J}^{J}c_{m}\phi_{m}(\theta_{x}),\quad\theta_{x}\in(-\pi,\pi),$
(44)

for $J$ large enough so that the approximation error is sufficiently small in
the $L^{2}$ sense.

Relation between limited-aperture data and full-aperture data: We first derive
a relation between the limited-aperture data and full-aperture data as
follows.

######  Lemma 9.

Let $C^{\alpha}_{\infty}$ and $C_{\infty}$ be given by (41) and (42)
respectively. It holds that

$C^{\alpha}_{\infty}=\mathbb{P}C_{\infty},$ (45)

where the infinite dimensional matrix $\mathbb{P}$ is the prolate matrix (with
dimension infinity) given by (46) whose $mn$-th entry is given by

$\mathbbm{p}_{mn}:=\int_{-\alpha}^{\alpha}\phi_{m}(\theta)\overline{\phi_{n}(\theta)}\,\mbox{d}\theta=\frac{1}{2\pi}\int_{-\alpha}^{\alpha}e^{i(m-n)\theta}\,\mbox{d}\theta=\bigg{\\{}\begin{array}[]{cc}\frac{\alpha}{\pi},&m=n\\\
\frac{\sin((m-n)\alpha)}{\pi(m-n)},&m\not=n\end{array}.$ (46)

###### Proof.

Assume that the full-aperture measurements are given, then there is the
Fourier series expansion (43). Plugging this expression into the definition
$\eqref{def C alpha}$ yields

$c^{\alpha}_{p}=\mathbbm{p}_{mp}c_{m}.$

Note that $\mathbb{P}$ is symmetric, this proves (45) and completes the proof.
∎

Finite dimensional case: In practice, the measurements are discrete data. This
motivates us to consider a finite dimensional space consisting of
$\phi_{m}(\theta)=\frac{1}{\sqrt{2\pi}}e^{im\theta},\,m=0,\pm 1,\cdots,\pm J$
for a sufficiently large $J$.

The following theorem in the finite dimensional case follows immediately from
Lemma 9.

######  Theorem 10.

Let $C^{\alpha}:=\Big{(}c^{\alpha}_{p}\Big{)}_{-J\leq p\leq J}$ and
$C:=\Big{(}c_{p}\Big{)}_{-J\leq p\leq J}$ with $c^{\alpha}_{p}$ and $c_{p}$
given by (41) and (42) respectively. It holds that

$C^{\alpha}=\mathbb{P}C,$ (47)

where $\mathbb{P}$ is the $(2J+1)\times(2J+1)$ prolate matrix given by (46).

From limited-aperture data to full-aperture data: Now it is clear that the
limited-aperture data is related to the full-aperture data via (47). Our goal
is then to find $C$ or its approximation from $C^{\alpha}$ via (47) at each
fixed $y\in\Gamma_{0}$. From the properties of the prolate matrix in [9, Lemma
3.2], we have that the eigenvalues of $\mathbb{P}$ are all positive, but they
are clustered near $1$ and $0$, and hence the matrix $\mathbb{P}$ is severely
ill-conditioned. In fact the eigenvalues decay exponentially to $0$ when $J$
becomes large. We refer to [28, 29] for more details on the prolate matrix.
Therefore we can only hope to invert $\mathbb{P}$ using regularization
techniques in order to find $C$ from $C^{\alpha}$. We shall apply the
following Regularization (48) to find approximate inverse of $\mathbb{P}$.

Regularization: Similar to [9], our choice is to consider the regularization
such that

$\mathbb{P}^{\dagger}=\mathbb{U}\left(\frac{1}{\sigma_{j}+\epsilon}\right)\mathbb{U}^{*},$
(48)

where $\epsilon>0$ is a regularization parameter, and

$\mathbb{P}=\mathbb{U}\Sigma\mathbb{U}^{*},\qquad\Sigma=\mbox{diag}(\sigma_{-J},\sigma_{-J+1},\cdots,\sigma_{J}).$

In this case, we take $\mathbb{P}^{\dagger}$ to approximate $\mathbb{P}^{-1}$.
From the point of view of Slepian’s spheroidal wave functions [28, 29], this
method attempts to use some information of the spheroidal wave functions with
“small energy” on the interval $[-\alpha,\alpha]$. Now we are ready to
summarize the imaging algorithm with limited-aperture data.

Imaging Algorithm.

  * •

Data completion for $\\{u^{s}(x;y):\,x\in\Gamma,\,y\in\Gamma_{0}\\}$: At each
fixed $y\in\Gamma_{0}$, recover the full-aperture data
$\\{u^{s}(x;y):\,x\in\Gamma,\,y\in\Gamma_{0}\\}$ approximately using the
limited-aperture data $\\{u^{s}(x;y):\,x\in\Gamma_{0},\,y\in\Gamma_{0}\\}$.

    * –

Step I: Compute $C^{\alpha}=\Big{(}c_{p}^{\alpha}\Big{)}$ from the
measurements $\\{u^{s}(x;y):\,x\in\Gamma_{0}\\}$ by (41).

    * –

Step II: Approximate $C$ by $\mathbb{P}^{\dagger}C^{\alpha}$, where
$\mathbb{P}^{\dagger}$ is the approximate inverse of $\mathbb{P}$ using
Regularization (48).

    * –

Step III: Recover the full-aperture data by (44).

  * •

At each fixed $x\in\Gamma$, repeat the above Data Completion to recover the
full-aperture data $\\{u^{s}(x;y):\,y\in\Gamma,\,x\in\Gamma\\}$ approximately
using the recoved data $\\{u^{s}(x;y):\,y\in\Gamma_{0},\,x\in\Gamma\\}$ in the
above step (note the reciprocity relation $u^{s}(x;y)=u^{s}(y;x)$ [7]).

  * •

Modified sampling method: Reconstruct the object by the imaging function (18)
for the obstacle case and (33) for the cavity case.

##  6 Numerical Examples

In this section, we present some numerical examples to illustrate the
performance of the modified sampling methods proposed in the previous
sections. The numerical examples are divided into three groups.

  * •

Reconstructions of obstacles using the imaging function (18);

  * •

Reconstructions of cavities using the imaging function (33);

  * •

Reconstructions of obstacles with limited-aperture data (40).

The boundaries of the scatterers used in our numerical experiments are
parameterized as follows:

Circle: $\displaystyle\quad x(t)\ =(a,b)+\,r(\cos t,\sin t),\quad 0\leq t\leq
2\pi,$ (49) Ellipse: $\displaystyle\quad x(t)\ =(a,b)+\,(2\cos t,3\sin
t),\quad 0\leq t\leq 2\pi,$ (50) Round Square: $\displaystyle\quad x(t)\
=(a,b)+(1.5\cos^{3}t+1.5\cos t,1.5\sin^{3}t+1.5\sin t),\quad 0\leq t\leq
2\pi,$ (51) Peanut: $\displaystyle\quad x(t)\
=(a,b)+\,1.5\sqrt{3\cos^{2}t+1}(\cos t,\sin t),\quad 0\leq t\leq 2\pi,$ (52)
Kite: $\displaystyle\quad x(t)\ =(a,b)+\,(1.1\cos t+0.625\cos 2t-0.625,1.5\sin
t),\quad 0\leq t\leq 2\pi,$ (53)

with $(a,b)$ be the location of the scatterer which will be specified in
different examples.

In our simulations, the boundary integral equation method is used to compute
the scattered fields $u^{s}(x;y)$ for $L$ equidistantly distributed
measurement points and $L$ equidistantly distributed source points over the
measurement surface. These data are then stored in the matrices
$\mathbb{N}\in{\mathbb{C}}^{L\times L}$. We further perturb $\mathbb{N}$ by
random noise using

$\displaystyle\mathbb{N}^{\delta}\ =\
\mathbb{N}+\delta\|\mathbb{N}\|\frac{R_{1}+R_{2}i}{\|R_{1}+R_{2}i\|},$

where $R_{1}$ and $R_{2}$ are two $L\times L$ matrices containing pseudo-
random values drawn from a normal distribution with mean zero and standard
deviation one. The value of $\delta$ used in our code is
$\delta:=\|\mathbb{N}^{\delta}-\mathbb{N}\|/\|\mathbb{N}\|$ which represents
the relative error.

###  6.1 Obstacle

We first illustrate the performance of the imaging function $I_{obstacles}$
for obstacles. In the simulations, we use a grid $\mathcal{G}$ of $301\times
301$ equally spaced sampling points on the rectangle $[-5,5]\times[-5,5]$. We
use $L=128$ equidistant sensors $y_{i},i=1,2,\cdots,128$ on the circle
$\partial B_{5}:=\\{y\in{\mathbb{R}}^{2}:\,|y|=5\\}$. We take $k=10$.
Therefore the wavelength $\lambda=2\pi/k\approx 0.618$ and consequently the
measurement surface $\partial B_{5}$ is just two or three wavelengths away
from the obstacles. As suggested in the arguments after Theorem 4, we take
$M=32$ in the definition $\varphi_{z}$ of (19). For each sampling point
$z\in\mathcal{G}$, we define the indicator function

$\displaystyle W_{obstacle}(z)\ :=|\Phi_{z}^{T}\mathbb{N}^{\delta}\Phi_{z}|,$
(54)

where
$\Phi_{z}=(\varphi_{z}(y_{1}),\varphi_{z}(y_{2}),\cdots,\varphi_{z}(y_{128}))^{\top}\in{\mathbb{C}}^{128}$.
Clearly, the indicator function is independent of any a priori information of
the unknown obstacles.

We report five examples. In the first three examples, the underlying obstacles
are round square, peanut and kite located at $(0,0)$, respectively. In the
fourth example the obstacle $\Omega$ is given by the union of two disjoint
components $\Omega:=\Omega_{1}\cup\Omega_{2}$, where $\Omega_{1}$ is a disk
with radius $1$ and center $(-2.5,0)$ while $\Omega_{2}$ is a kite located at
$(2,0)$. For the fifth example the obstacle $\Omega$ is also given by the
union of two disjoint components $\Omega:=\Omega_{1}\cup\Omega_{2}$, where
$\Omega_{1}$ is a disk with radius $1$ and center $(0,2.5)$ while $\Omega_{2}$
is a peanut located at $(-1.2,0)$. These five exact obstacles are shown in the
first column of Figure 2.

Reconstructions are shown in the second and third column of Figure 2.
Obviously, the indicator function $W_{obstacle}$ peaks on the boundaries of
the obstacles and is capable to reconstruct the locations and shapes. As shown
in the last two examples, the different components are also well
reconstructed. As shown in the third column of Figure 2, the reconstructions
with $10\%$ noise are almost the same as the those without noises. This
further implies that our modified sampling methods are quite stable with
respect to noises. These reconstructions are comparable to those using similar
imaging function with far-field measurements [19].

Fig. 2: Reconstructions of obstacles. The first column is the exact obstacles.
The second column shows the reconstructions without noises, while the third
column shows the reconstructions with $10\%$ noise.

###  6.2 Cavity

This subsection is denoted to illustrating the performance of the imaging
function $I_{cavity}$ for cavities. In the simulations, we used a grid
$\mathcal{G}$ of $81\times 81$ equally spaced sampling points on the rectangle
$[-4,4]\times[-4,4]$. As suggested in Remark 5, we take $\mathfrak{M}=3$ in
the definition $\psi_{z}$ of (34). For each sampling point $z\in\mathcal{G}$,
we define the indicator function

$\displaystyle W_{cavity}(z)\ :=|\Psi_{z}^{\ast}\mathbb{N}^{\delta}\Psi_{z}|,$

where
$\Psi_{z}=(\psi_{z}(y_{1}),\psi_{z}(y_{2}),\cdots,\psi_{z}(y_{128}))^{\top}\in{\mathbb{C}}^{L}$
and the superscript $*$ denotes the conjugate transpose.

The cavities we considered are round square, disk, ellipse, peanut and kite
with center $(0,0)$. The radius of the disk is $2$. We refer to the first
column of Figure 3 for the exact cavities. In the cavity case, general
reconstructions using low frequency waves seem to perform better than those
using high frequency waves [30], we therefore take the wave number $k=0.2$.

In Figure 3, we show the reconstructions of various cavities in the second
column (without noises) and third column (with $1\%$ noise). We find that the
indicator function $W_{cavity}$ starts to peak as the sampling point
approaches the boundary from the interior of the cavities and this allows us
to reconstruct the shapes. As shown in the third column of Figure 3, the
reconstructions are not that robust to noises compared to the obstacle case.

As discussed in Remark 5, measurement circle with smaller radius may lead to
numerical instability for noisy measurements, we illustrate this in Figures
3-5 by considering three measurement circles with radius $r=1$, $r=0.5$ and
$r=0.005$, respectively. Without noise, the proposed imaging functional
$W_{cavity}$ gives quite good reconstructions. Surprisingly, as shown in
Figure 5, even the measurement surface is very small and there are only $8$
sensors, the cavities can be well reconstructed. However, the reconstructions
are more sensitive to noises for measurement circle with smaller radius.

Fig. 3: Reconstructions of cavities. There are $L=32$ sensors on the
measurement circle with radius $1$. The first column is the exact cavities.
The second column shows the reconstructions without noises, while the third
column shows the reconstructions with $1\%$ noise.

Fig. 4: Reconstructions of cavities. There are $L=32$ sensors on the
measurement circle with radius $0.5$. The second column shows the
reconstructions without noises, while the third column shows the
reconstructions with $1\%$ noise.

Fig. 5: Reconstructions of cavities. There are $L=8$ sensors on the
measurement circle with radius $0.005$. The second column shows the
reconstructions without noises, while the third column shows the
reconstructions with $1\%$ noise.

###  6.3 Limited-aperture

In this subsection, we illustrate the performance of our modified sampling
methods using limited-aperture data. We consider $64$ sensors (of the total
$128$ sensors) equidistantly located on the upper half circle $\partial
B_{+}:=\\{(x_{1},x_{2})\in\partial B_{5}:\,x_{2}>0\\}$. These data are then
stored in the matrices $\mathbb{N}_{partial}\in{\mathbb{C}}^{64\times 64}$. We
consider the following two strategies for the limited-aperture problems.

  * •

Reconstruct the obstacles using the limited-aperture data directly with the
indicator function

$\displaystyle W_{limit}(z)\
:=|\widetilde{\Phi}_{z}^{T}\mathbb{N}_{partial}\widetilde{\Phi}_{z}|,$

where
$\widetilde{\Phi}_{z}=(\varphi_{z}(y_{1}),\varphi_{z}(y_{2}),\cdots,\varphi_{z}(y_{64}))^{\top}\in{\mathbb{C}}^{64}$.

  * •

Recover the full aperture data using the data completion algorithm in Section
5, where the Regularization (48) is used with parameter $\epsilon=10^{-3}$.
With the recovered full aperture data, we then apply the modified sampling
methods to reconstruct the obstacles with the indicator function (54).

The first two columns of Figure 6 show the reconstructions using these two
strategies, respectively. For comparison, we show in the third column of
Figure 6 the results using full aperture data. $10\%$ noise is added in all of
these reconstructions. It is observed that the upper half parts of the
obstacles can be well reconstructed and the resolution indeed can be improved
with the help of the data completion algorithm.

Fig. 6:  Reconstructions using limited-aperture measurements. The first two
columns show the reconstructions with measurements on $\partial B_{+}$. The
first column is the reconstructions using limited-aperture data directly,
while the second column shows the reconstructions combined with the data
completion algorithm. The third column shows the reconstructions with full
aperture data.

## Acknowledgments

The research of X. Liu is supported by the NNSF of China grant 11971471 and
the Youth Innovation Promotion Association, CAS. The research of B. Zhang is
partially supported by the NNSF of China grant 91630309.

## References

  * [1] G. Bao and J. Liu, Numerical solution of inverse problems with multi-experimental limited aperture data, SIAM J.Sci.Comput. 25, (2003), 1102-1117.
  * [2] L. Borcea, F. Cakoni, and S. Meng, A direct approach to imaging in a waveguide with perturbed geometry, J. Comput. Phys., 392 (2019), 556–577.
  * [3] F. Cakoni and D. Colton.  A qualitative approach to inverse scattering theory, volume 767\.  Springer, 2014.
  * [4] F. Cakoni, D. Colton, and S. Meng.  The inverse scattering problem for a penetrable cavity with internal measurements.  Inverse problems and applications, 615:71–88, 2014.
  * [5] J. Chen Z. Chen, and G. Huang.  Reverse time migration for extended obstacles: acoustic waves.  Inverse Problems, 29(8):085005, 2013.
  * [6] D. Colton and A. Kirsch, A simple method for solving inverse scattering problems in the resonance region. Inverse Problems 12 (1996), 383-393.
  * [7] D. Colton and R. Kress.  Inverse acoustic and electromagnetic scattering theory, volume 93.  Springer Nature, 2019.
  * [8] D. Colton and P. Monk, Target identification of coated objects IEEE Trans. Antennas Propagat., 54, (2006), 1232-1242.
  * [9] F. Dou, X. Liu, S. Meng and B. Zhang  Data completion algorithms and their applications in inverse acoustic scattering with limited-aperture backscattering data.  arXiv:2106.11101, 2021.
  * [10] R Griesmaier.  Multi-frequency orthogonality sampling for inverse obstacle scattering problems.  Inverse Problems, 27(8):085005, 2011.
  * [11] I Harris, D-L Nguyen.  Orthogonality sampling method for the electromagnetic inverse scattering problem.  SIAM Journal on Scientific Computing, 42(3):722–737, 2020.
  * [12] K. Ito, B. Jin, and J. Zou, A direct sampling method to an inverse medium scattering problem, Inverse Problems, 28, (2012), 025003.
  * [13] G. Hu, J. Yang, B. Zhang, and H. Zhang, Near-field imaging of scattering obstacles with the factorization method, Inverse Problems , 30(9):095005, 2014.
  * [14] M. Ikehata, E. Niemi, and S. Siltanen, Inverse obstacle scattering with limited-aperture data, Inverse Probl. Imaging 1, (2012), 77-94.
  * [15] A. Kirsch, Characterization of the shape of a scattering obstacle using the spectral data of the far field operator, Inverse Problems 14, (1998), 1489-1512.
  * [16] A. Kirsch and N. Grinberg.  The factorization method for inverse problems.  Number 36. Oxford University Press, 2008.
  * [17] J. Li, H. Liu and J. Zou, Locating multiple multiscale acoustic scatterers, SIAM Multiscale Model. Simul., 12, (2014), 927–952.
  * [18] X. Liu.  The factorization method for cavities.  Inverse problems, 30(1):015006, 2013.
  * [19] X. Liu, A novel sampling method for multiple multiscale targets from scattering amplitudes at a fixed frequency, Inverse Problems 33, (2017), 085011.
  * [20] X. Liu and J. Sun, Data recovery in inverse scattering problems: from limited-aperture to full-aperture, J. Comput. Phys. 386(1), (2019), 350-364.
  * [21] S. Lu, B. Xu, and X. Xu, Unique continuation on a line for the Helmholtz equation. Appl. Anal. 91(9), (2012), 1761-1771.
  * [22] R.D. Mager and N. Bleistein, An examination of the limited aperture problem of physical optics inverse scattering, IEEE Trans. Antennas Propag. 26, (1978), 695-699.
  * [23] W. McLean  Strongly elliptic systems and boundary integral equations. Cambridge university press, 2000.
  * [24] S. Meng, H. Haddar, and F. Cakoni.  The factorization method for a cavity in an inhomogeneous medium.  Inverse Problems, 30(4):045008, 2014.
  * [25] R. Potthast.  A study on orthogonality sampling.  Inverse Problems, 26(7):074015, 2010.
  * [26] H. Qin and F. Cakoni.  Nonlinear integral equations for shape reconstruction in the inverse interior scattering problem.  Inverse Problems, 27(3):035005, 2011.
  * [27] H. Qin and D. Colton.  The inverse scattering problem for cavities.  Applied Numerical Mathematics, 62(6):699–708, 2012.
  * [28] D. Slepian, Prolate spheroidal wave functions, Fourier analysis, and uncertainty V: The discrete case, Bell System Tech. J. 57, (1978), 1371-1430.
  * [29] J. M. Varah, The prolate matrix, Linear Algebra Appl. 187, (1993), 269-278.
  * [30] F. Zeng and S. Meng.  The interior inverse electromagnetic scattering for an inhomogeneous cavity.  Inverse Problems, 37(2):025007, 2021.
  * [31] A. Zinn, On an optimisation method for the full- and limited-aperture problem in inverse acoustic scattering for a sound-soft obstacle, Inverse Problems 5, (1989), 239-253.


# Contrastive Regularization for Multimodal Emotion Recognition Using Audio
and Text

###### Abstract

Speech emotion recognition is a challenge and an important step towards more
natural human-computer interaction (HCI). The popular approach is multimodal
emotion recognition based on model-level fusion, which means that the
multimodal signals can be encoded to acquire embeddings, and then the
embeddings are concatenated together for the final classification. However,
due to the influence of noise or other factors, each modality does not always
tend to the same emotional category, which affects the generalization of a
model. In this paper, we propose a novel regularization method via contrastive
learning for multimodal emotion recognition using audio and text. By
introducing a discriminator to distinguish the difference between the same and
different emotional pairs, we explicitly restrict the latent code of each
modality to contain the same emotional information, so as to reduce the noise
interference and get more discriminative representation. Experiments are
performed on the standard IEMOCAP dataset for 4-class emotion recognition. The
results show a significant improvement of 1.44% and 1.53% in terms of weighted
accuracy (WA) and unweighted accuracy (UA) compared to the baseline system.

Index Terms—  multimodal emotion recognition, contrastive learning,
regularization

##  1 INTRODUCTION

In the human-centered era, natural human-computer interaction (HCI) becomes
more and more important. An ideal HCI system is required to detect the users
emotion for enriching their experience, so as to provide better feedback. Many
studies [1, 2, 3] show that a prominent automatic emotion recognition system
should be multimodal since it is closer to the human sensor system.

The core challenge of multimodal emotion recognition is how to integrate the
emotional knowledge from various modalities. With the popularity of deep
learning, model-level fusion has become the current mainstream method of
multimodal emotion recognition, which is implemented by concatenating
embeddings obtained from different modalities into a combined representation
as the input of a classification model for end-to-end training [4, 5, 6]. It
can capture complex non-linear multimodal feature correlations and make better
advantages of deep neural networks.

However, inevitably, there are several samples that contain noise, which
affects the complementarity between the modalities. Specifically, the noise
may cause the various modalities of a sample to tend to different emotion
categories. For example, when a model is well-trained on the 4-class emotional
recognition task using audio and text, the audio score of a noisy sample
indicates that the speech is neutral, but its text score indicates that the
speech is happy. Although the model prediction based on scores summation of
two modalities may be correct, there is a conflict between them. This
phenomenon shows that, due to noise or other factors, the complementarity
between various modalities of a sample is lost, resulting in the failure of
learning good representation, which greatly affects the generalization of a
model.

To alleviate the above problem, we propose a novel contrastive regularization
method. Our motivation is that the learned embeddings should be restricted to
contain the same emotional information as much as possible, so as to reduce
the influence of noise and improve the system robustness. Contrastive learning
[7] is a paradigm that leverages abundant negative example pairs to constrain
information contained in the learned representation, which satisfies our
demand well. More specifically, a discriminator is introduced to compare
positive and negative example pairs as the explicit constraint with the same
emotion, where the positive example pair is the embeddings obtained by
encoding the audio and text from the same sample, and the negative example
pair is two modal embeddings that one is the audio embedding in the positive
pair, and the other is the text embedding from different samples as well as
different emotions. Therefore, the complementarity between the two modalities
is greatly enhanced.

##  2 RELATED WORK

There are currently three main fusion methods in multimodal emotion
recognition: _feature-level_ fusion, _decision-level_ fusion and _model-level_
fusion. Feature-level fusion [8, 9, 10], a.k.a. early fusion, is implemented
by simply concatenating features from multiple modalities into a combined
vector as the input of a predictive model. However, this type of fusion method
might suffer from the curse of dimensionality on small datasets, meanwhile
larger dimensional features might stress computational resources during model
training. Decision-level fusion [10, 11, 12], a.k.a. late fusion, on the
contrary, combines predictions obtained from different modalities for a final
prediction via a voting strategy. Nevertheless, it does not take into account
the interaction between various modalities. As a compromise of early fusion
and late fusion, model-level fusion which fuses the intermediate
representations of different modalities may be the best choice for the fusion
strategy [4].

However, most previous works based on model-level fusion for emotion
recognition seldom take into account the effect of noise on modal
complementarity. In fact, there are always samples whose multiple modalities
tend to be different emotional categories due to the effect of noise, no
matter how well the model is trained. Our intuition is that, by adding the
regularization constraint of the same emotion explicitly, the performance of
model-level fusion method can be improved. To this end, we draw inspirations
from the idea of contrastive learning [7], where the difference between
positive and negative examples is captured, and propose a novel regularization
method. In contrast to random negative sampling commonly used in self-
supervised learning [13], we draw negative examples from different emotions to
restrict the learned representation to contain the same emotional information
and experimentally demonstrate the effectiveness of our proposed method.

##  3 METHODOLOGY

###  3.1 Model-level fusion

We start by introducing the model-level fusion for emotion recognition using
audio and text. The audio features $\bm{x}_{a}$ and text features $\bm{x}_{t}$
pass through an individual encoder to obtain the embedding vectors
$\bm{e}_{a}$ and $\bm{e}_{t}$, respectively. Then the embedding vectors are
concatenated together to classify emotions. We argue that, when embeddings are
learned well enough, only a linear classifier is required,

$\bm{s}=\bm{W}\cdot[\bm{e}_{a};\bm{e}_{t}]$ (1)

where $\bm{s}$ is the score vector and $\bm{W}$ is the weight matrix. Eq. (1)
can also be written as:

$\bm{s}_{a}=\bm{W}_{1}\cdot\bm{e}_{a}$ (2)
$\bm{s}_{t}=\bm{W}_{2}\cdot\bm{e}_{t}$ (3) $\bm{s}=\bm{s}_{a}+\bm{s}_{t}$ (4)

The weight matrix $\bm{W}$ is the concatenation of $\bm{W}_{1}$ and
$\bm{W}_{2}$. In other words, the score of audio and text connected vector is
the summation of their respective scores. For $i$-th sample, the
classification loss is cross-entropy as follows,

$\mathcal{L}_{1}=-log\frac{exp(s_{ij})}{\sum_{c=1}^{C}exp(s_{ic})}$ (5)

where $C$ is the number of categories and $s_{ij}$ is the score of $i$-th
sample $j$-th class.

###  3.2 Contrastive regularization

Fig. 1: Overview of our model framework

Due to the influence of noise or other factors, the categories corresponding
to the largest scores of $\bm{s}_{a}$ and $\bm{s}_{t}$ are inconsistent. As a
result, the complementarity between the two modalities is destroyed, which
greatly affect the generalization of a model.

To overcome the above problem, we propose a contrastive regularization method
and its model framework is shown in Fig. 1. The upper block diagram represents
a Convolution Neural Network (CNN) with the input of spectrograms as well as
the output of audio embeddings. And the lower block diagram represents a Long-
Short Time Memory (LSTM) with the input of word sequences as well as the
output of text embeddings. The two output on the right side indicate the
emotional classification loss and contrastive loss used, respectively.

More specifically, a four-layers CNN is leveraged as the audio encoder for
audio spectrogram features to capture the emotional information. Meanwhile, a
two-layers LSTM with attention mechanism is used as the text encoder. The
input of LSTM is GloVe [14] word embeddings which embed a sequence of
individual words from video segment transcripts into a sequence of word
vectors that represent spoken text. The ensemble of CNN and LSTM is a backbone
framework of both the baseline and our method. In addition, a discriminator is
introduced and fed by either a positive example pair or negative example pairs
and trained to separate them. Specifically, for each sample, we select its
corresponding audio and text features as a positive pair, and at the same time
select multiple text features with other different emotions as negative pairs.
Through the comparison of positive and negative pairs, the learned
representations are constrained to contain the same emotional information.
Therefore, the noise is reduced and the loss of complementarity can be well
mitigated.

For $i$-th sample, the contrastive loss is similar to the form of cross-
entropy, which is called _infoNCE_ by [15],

$\mathcal{L}_{2}=-log\frac{exp(d(\bm{e}_{a}^{i},\bm{e}_{t}^{i}))}{exp(d(\bm{e}_{a}^{i},\bm{e}_{t}^{i}))+\sum_{n=1}^{N}exp(d(\bm{e}_{a}^{i},\bm{e}_{t}^{n^{-}}))}$
(6)

where $d(,)$ is the discriminator with the input of either positive example
pair $(\bm{e}_{a}^{i},\bm{e}_{t}^{i})$ or negative example pair
$(\bm{e}_{a}^{i},\bm{e}_{t}^{n^{-}})$, and $N$ is the number of negative
pairs. In this work, a single hidden layer fully connected neural network is
selected as the discriminator where the output is a real value, and $N$ is
equal to batch size.

The final loss function is as follows,

$\displaystyle\mathcal{L}$
$\displaystyle=(1-\alpha)\mathcal{L}_{1}+\alpha\mathcal{L}_{2}$ (7)

where $\alpha$ is a hyperparameter that controls the strength of the
regularization term. In this work, we use a grid search with a step size of
0.1 to find the optimal $\alpha$.

##  4 EXPERIMENTS AND RESULTS

Fig. 2: Confusion matrices of baseline and our method. Left and right figures
represent baseline and our method respectively.

###  4.1 Database

We use the IEMOCAP database [16] to evaluate the performance of different
methods, which contains approximate 12 hours of audio-video recordings of
dyadic interactions with 10 different speakers split in pair over 5 sessions
in English. There are about 10000 utterances in the database that has been
given emotion labels by at least 3 annotators. 12 unique raters annotate the
database by choosing labels out of the 9 possible emotional labels per
utterance. Emotional labels are happiness, anger, sadness, neutral, fear,
surprise, frustration, excitement, and others.

In order to compare with majority of previous works [17, 18], we select
utterances annotated with four basic emotions: neutral, happiness, sadness and
anger. Samples with excitement are merged with happiness resulting in a total
of 5531 utterances (neutral: 1708, happiness: 1636, sadness: 1084, anger:
1103).

###  4.2 Preprocessing

For audio features, we extract spectrograms using the python toolkit, pylab
[19]. We calculate a short-time Fourier transform with 256 FFT points and the
hop length is 50% of the number of FFT points. In addition, all audio
utterances are divided into a uniform length of approximate one second. Thus,
the input audio features are 128$\times$128 spectrograms.

For text features, we extract GloVe [14] word embeddings utilizing gensim
toolkit [20]. The _word2vec-google-news-300_ [21], a mass of pre-trained
vectors trained on a part of the Google News dataset (about 100 billion
words), is loaded. We split or pad each sentence into 30 words and acquire a
300-dimensional word embedding sequence of length 30.

###  4.3 Experimental setup

For the CNN block used to process the spectrograms, a 3×3 convolution kernel
is utilized with padding 1, and the max-pooling kernel is 2×2 with a 2×2
stride. Moreover, the activation function is ReLU and the output channels are
all 8. The LSTM which is used to process word embedding sequences has 200
hidden nodes. It is a single-directional framework with two layers. An Adam
optimizer is used with the learning rate of 0.001, and batch size is set to
64. The hyperparameter $\alpha$ is selected as 0.1 through a grid search in
the range of [0.0, 1.0] with a step size of 0.1.

Following [17, 18], we utilize 10-fold cross validation with a 8:1:1 split for
training, validation and testing, respectively. The performance of the system
is measured in terms of weighted accuracy (WA) and unweighted accuracy (UA)
[22].

###  4.4 Results and Analysis

Table 1 presents the performance of our approach for emotion recognition
compared with other methods. The baseline model is based on model-level fusion
that concatenate embeddings obtained from audio and text modalities into a
combined representation as the input of a linear classification model. DS-LSTM
[23] is a dual-level model that predicts emotions based on both MFCC features
and mel-spectrograms, and it is the state-of-the-art unimodal emotion
recognition only using audio. MDRE [17] encodes the information from audio and
text sequences using dual recurrent neural network (RNN) and then combines the
information from these sources to predict the emotion class. MHA-2 [18] uses a
multi-hop attention mechanism to exploit both textual and acoustic information
in tandem.

Table 1: Methods performance comparison on IEMOCAP database (%) Models | WA | UA
---|---|---
Model-level fusion (Baseline) | 74.62 | 75.92
DS-LSTM [23] | 72.7 | 73.3
MDRE [17] | 71.8 | —
MHA-2 [18] | 76.5 | 77.6
Contrastive regularization (Proposed) | 76.06 | 77.45

As illustrated in Table 1, our proposed method outperforms DS-LSTM by 3.36%
and 4.15% in terms of WA and UA, which further demonstrates the previous
conclusion that the multimodal method is better than the unimodal method. In
addition, it is worth noting that MHA-2, the state-of-the-art multimodal
emotion recognition using audio and text, achieves 76.5% in WA and 77.6% in
UA. Although our results are not directly comparable due to distinct model
framework used and domain specialized optimizations, the idea behind our
contrastive regularization method, which leverages abundant negative example
pairs to acquire stronger representations, can be characterized as an
additional contrastive loss and serve as a regularization term. Therefore, our
proposed method is more applicable than MHA-2 due to the limitation of multi-
hop attention mechanism which is only used for audio and text sequential data.

More notably, the 1.44% and 1.53% improvement in terms of WA and UA compared
to the baseline are realized by our proposed method. Meanwhile, as confusion
matrices shown in Fig. 2, our proposed method acquires the same accuracy
68.77% compared to the baseline method in classifying neutral emotion and
outperforms the baseline method in other three types of emotions. Moreover,
the baseline model incorrectly classifies more examples of one emotion as
another emotion than our proposed method, e.g., 16.08% to 14.15% for happy,
11.01% to 9.63% for angry, except for misclassification of neutral as happy
(14.52% to 16.44%). For this type of misclassification, we guess that randomly
sampling other different emotions against neutral class may sample more happy
emotions due to the slight imbalance of the samples.

##  5 CONCLUSIONS

In this paper, we proposed a novel regularization method via contrastive
learning for multimodal emotion recognition task. The proposed method is
designed to constrain latent representations of audio and text to contain the
same emotional information, so that it enhances the complementarity between
the two modalities and improves the generalization of a model. Experiments
show that the proposed method outperforms the baseline system in classifying
the four emotion categories on IEMOCAP database by 1.44% and 1.53% in terms of
WA and UA respectively and is competitive with the state-of-the-art system.

##  6 ACKNOWLEDGEMENTS

This research is supported by the National Natural Science Foundation of China
under grant No.U1736210.

## References

  * [1] M. Soleymani, M. Pantic, and T. Pun,  “Multimodal emotion recognition in response to videos,”  IEEE Transactions on Affective Computing, vol. 3, pp. 211–223, 2012\.
  * [2] Hillel Aviezer, Y. Trope, and A. Todorov,  “Body cues, not facial expressions, discriminate between intense positive and negative emotions,”  Science, vol. 338, pp. 1225 – 1229, 2012.
  * [3] M. Pantic, N. Sebe, J. Cohn, and T. Huang,  “Affective multimodal human-computer interaction,”  in MULTIMEDIA ’05, 2005.
  * [4] Mihalis A. Nicolaou, H. Gunes, and M. Pantic,  “Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space,”  IEEE Transactions on Affective Computing, vol. 2, pp. 92–105, 2011\.
  * [5] Lang He, Dongmei Jiang, Le Yang, Ercheng Pei, P. Wu, and H. Sahli,  “Multimodal affective dimension prediction using deep bidirectional long short-term memory recurrent neural networks,”  in AVEC ’15, 2015.
  * [6] Zhaocheng Huang, Ting Dang, Nicholas Cummins, Brian Stasak, P. Le, V. Sethu, and J. Epps,  “An investigation of annotation delay compensation and output-associative fusion for multimodal continuous emotion prediction,”  in AVEC ’15, 2015.
  * [7] Ankesh Anand,  “Contrastive self-supervised learning,” https://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html, 2020\.
  * [8] C. Busso, Zhigang Deng, S. Yildirim, M. Bulut, C. Lee, A. Kazemzadeh, S. Lee, U. Neumann, and Shrikanth S. Narayanan,  “Analysis of emotion recognition using facial expressions, speech and multimodal information,”  in ICMI ’04, 2004.
  * [9] Jinming Zhao, Ruichen Li, Shizhe Chen, and Q. Jin,  “Multi-modal multi-cultural dimensional continues emotion recognition in dyadic interactions,”  in AVEC’18, 2018.
  * [10] Jing Han, Zixing Zhang, Z. Ren, and B. Schuller,  “Implicit fusion by joint audiovisual training for emotion recognition in mono modality,”  ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5861–5865, 2019.
  * [11] M. Valstar, J. Gratch, B. Schuller, Fabien Ringeval, D. Lalanne, Mercedes Torres, S. Scherer, Giota Stratou, R. Cowie, and M. Pantic,  “Avec 2016: Depression, mood, and emotion recognition workshop and challenge,”  Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, 2016.
  * [12] Jing Han, Zixing Zhang, M. Schmitt, M. Pantic, and B. Schuller,  “From hard to soft: Towards more human-like emotion recognition by modelling the perception uncertainty,”  Proceedings of the 25th ACM international conference on Multimedia, 2017.
  * [13] Suzanna Becker and Geoffrey E. Hinton,  “Self-organizing neural network that discovers surfaces in random-dot stereograms,”  Nature, vol. 355, pp. 161–163, 1992.
  * [14] Jeffrey Pennington, R. Socher, and Christopher D. Manning,  “Glove: Global vectors for word representation,”  in EMNLP, 2014.
  * [15] A. Oord, Y. Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,”  ArXiv, vol. abs/1807.03748, 2018.
  * [16] C. Busso, M. Bulut, Chi-Chun Lee, A. Kazemzadeh, E. Provost, S. Kim, J. N. Chang, S. Lee, and Shrikanth S. Narayanan,  “Iemocap: interactive emotional dyadic motion capture database,”  Language Resources and Evaluation, vol. 42, pp. 335–359, 2008.
  * [17] Seunghyun Yoon, Seokhyun Byun, and K. Jung,  “Multimodal speech emotion recognition using audio and text,”  2018 IEEE Spoken Language Technology Workshop (SLT), pp. 112–118, 2018.
  * [18] Seunghyun Yoon, Seokhyun Byun, Subhadeep Dey, and K. Jung,  “Speech emotion recognition using multi-hop attention mechanism,”  ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2822–2826, 2019.
  * [19] J. D. Hunter,  “Matplotlib: A 2d graphics environment,”  Computing in Science & Engineering, vol. 9, 2007.
  * [20] Radim Rehurek and P. Sojka,  “Gensim – statistical semantics in python,”  2011\.
  * [21] Tomas Mikolov, W. Yih, and G. Zweig,  “Linguistic regularities in continuous space word representations,”  in HLT-NAACL, 2013.
  * [22] Kun Han, Dong Yu, and I. Tashev,  “Speech emotion recognition using deep neural network and extreme learning machine,”  in INTERSPEECH, 2014.
  * [23] Jianyou Wang, Michael Xue, Ryan Culhane, Enmao Diao, J. Ding, and V. Tarokh,  “Speech emotion recognition with dual-sequence lstm architecture,”  ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6474–6478, 2020.