# Sharp uniqueness and stability of solution for an inverse source problem for
the Schrödinger equation

1 O.Y. Imanuvilov and 2 M. Yamamoto

###### Abstract.

The manuscript is concerned with uniqueness and stability for inverse source
problem of determining spatially varying factor $f(x)$ of a source term given
by $R(t)f(x)$ with suitable given $R(t)$ in the right hand side of the
Schrödinger equation with time independent coefficients. In order to establish
these results we provide a simple proof of a logarithmic conditional stability
of the Cauchy problem for the Schrödinger equation with time-independent
coefficients and the zero Dirichlet boundary conditions on the whole boundary
and a proof of uniqueness of solution to the Cauchy problem for the
Schrödinger equation with data on an arbitrary small part of a lateral
boundary. We do not assume any geometrical constraints on subboundary and a
time interval is arbitrary. The key is an integral transform, with a kernel
solving a null controllability problem for the 1-D Schrödinger, which changes
a solution of the Schrödinger equation to a solution of an elliptic equation.

1 Department of Mathematics, Colorado State University, 101 Weber Building,
Fort Collins CO 80523-1874, USA<EMAIL_ADDRESS>
2 Graduate School of Mathematical Sciences, The University of Tokyo, Komaba,
Meguro, Tokyo 153-8914, Japan e-mail<EMAIL_ADDRESS>
##  1\. Introduction and main results

Let $\Omega$ be a bounded domain in $\mathbb{R}^{n}$ with $C^{2}$-boundary. We
set $x=(x_{1},...,x_{n})\in\mathbb{R}^{n}$, $\partial_{t}u=\frac{\partial
u}{\partial t}$, $\partial_{x_{j}}u=\frac{\partial u}{\partial x_{j}}$ for
$1\leq j\leq n$, and $\nabla=(\partial_{x_{1}},...,\partial_{x_{n}})$.
Moreover, $\nu=\nu(x)$ denotes the unit outward normal vector to
$\partial\Omega$ at $x$ and $\partial_{\nu}u=\nabla u\cdot\nu$ on
$\partial\Omega$. We set $Q:=(0,T)\times\Omega$ and $i:=\sqrt{-1}$,
$D=(D_{0},D^{\prime}),D^{\prime}=(D_{1},\dots,D_{n}),D_{0}=\frac{1}{i}\partial_{t},D_{j}=\frac{1}{i}\partial_{x_{j}},j\in\\{1,\dots,n\\}.$

We consider the Schrödinger equation:

(1.1)
$P(x,D)u:=i\partial_{t}u-\sum_{j,k=1}^{n}\partial_{x_{j}}(a_{jk}(x)\partial_{x_{k}}u(t,x))+\sum_{j=1}^{n}b_{j}(x)\partial_{x_{j}}u(t,x)+c(x)u(t,x),\quad(t,x)\in
Q,$

Throughout this article, we make the following assumptions: the coefficients
$a_{jk}$ are real-valued functions for all $1\leq j,k\leq n$ and

(1.2) $a_{jk}=a_{kj}\in C^{1}(\overline{\Omega}),\quad b_{j}\in
W^{1}_{\infty}(\Omega),\quad c\in L^{\infty}(\Omega)\quad\mbox{for all
$j,k\in\\{1,\dots,n\\}$}$

and there exists a constant $\beta>0$ such that

(1.3)
$\sum_{j,k=1}^{n}a_{jk}(x)\xi_{j}\xi_{k}\geq\beta\sum_{j=1}^{n}\xi_{j}^{2}\quad\mbox{for
all $x\in\Omega$, $\xi_{1},...,\xi_{n}\in\mathbb{R}$}.$

We set

$A(x,D^{\prime})v:=-\sum_{j,k=1}^{n}\partial_{x_{j}}(a_{jk}(x)\partial_{x_{k}}v(x))+\sum_{j=1}^{n}b_{j}(x)\partial_{x_{j}}v(x)+c(x)v(x)$

for $x\in\Omega$.

Let $\Gamma\subset\partial\Omega$ be a suboundary and let $T>0$.

We define

$H^{1,2}(Q):=\\{u\in L^{2}(Q);\,\partial_{t}u,\,\partial_{x}^{\alpha}u\in
L^{2}(Q)\,\,\mbox{if $|\alpha|\leq 2$}\\},$

where $\alpha:=(\alpha_{1},...,\alpha_{n})\in(\mathbb{N}\cup\\{0\\})^{n}$,
$|\alpha|:=\alpha_{1}+\cdots+\alpha_{n}$ and
$\partial_{x}^{\alpha}:=\partial_{x_{1}}^{\alpha_{1}}\cdots\partial_{x_{n}}^{\alpha_{n}}$.

In this manuscript we concern with the following inverse problem:

Let $u\in C([0,T];L^{2}(\Omega))$ and $f\in L^{2}(\Omega)$ satisfy

(1.4) $P(x,D)u=i\partial_{t}u+A(x,D^{\prime})u=R(t)f(x)\quad\mbox{in}\quad
Q,\quad u(0,\cdot)=0.$

Here $R$ is a given real-valued function. Determine the source term $f$ based
on the following data : $\partial_{\nu}u|_{\Gamma}.$

We have

######  Theorem 1.

Let pair $(u,f)$ satisfies (1.4) and $R\in C^{1}[0,T].$
(i) We assume that $u,\partial_{t}u\in H^{1,2}(Q)$ and

(1.5) $R(0)\neq 0.$

If $u=\partial_{\nu}u=0$ on $(0,T)\times\Gamma$ with arbitrary fixed subdomain
$\Gamma$ of $\partial\Omega$, then $f=0$ in $\Omega$ and $u=0$ in $Q$.
(ii) We assume that $u\in C([0,T];L^{2}(\Omega)),f\in L^{2}(\Omega)$ and

(1.6) $R$ is not identically zero on $[0,T]$.

Let $u=0$ on $(0,T)\times\partial\Omega$ and $\partial_{\nu}u=0$ on
$(0,T)\times\Gamma$ with arbitrary fixed subdomain $\Gamma$ of
$\partial\Omega$, then $f=0$ in $\Omega$ and $u=0$ in $Q$.

Remark. A function $u\in C([0,T];L^{2}(\Omega))$ is a solution (1.4) such that
$u|_{(0,T)\times\partial\Omega}=\partial_{\nu}u|_{(0,T)\times\Gamma}=0$ if

$(u,P^{*}(x,D)w)_{L^{2}(Q)}=(Rf,w)_{L^{2}(Q)}\quad\forall w\in
H^{1,2}(Q),w=0\quad\mbox{on}\,\,(0,T)\times\partial\Omega\setminus\Gamma,\,w(T,\cdot)=0.$

We do not know whether $f=0$ in $\Omega$ and $u=0$ in $Q$ follow from
$u=\partial_{\nu}u=0$ on $(0,T)\times\Gamma$ under the weaker assumption
(1.6). Moreover, in terms of Theorem 1, we can prove

######  Theorem 2.

Let pair $(u,f)\in H^{1,2}(Q)\times L^{2}(\Omega)$ satisfies (1.4), $R\in
C^{2}[0,T]$. Moreover $\partial^{2}_{t}u,\partial_{t}u\in H^{1,2}(Q)$ and
$u|_{(0,T)\times\partial\Omega}=0.$ We assume (1.5) and

$\|\partial_{t}u\|_{H^{1,2}(Q)}+\|u\|_{H^{1,2}(Q)}\leq M,$

where $M>0$ is an arbitrarily chosen constant. Then there exists a constant
$C>0$ such that

(1.7)
$\|f\|_{L^{2}(\Omega)}\leq\frac{C}{|\log\|\partial_{\nu}u\|_{H^{2}(0,T;L^{2}(\Gamma))}|}.$

The inverse problem of recovery of a source term in the right hand side of the
Schrödinger equation was studied in Baudoin and Mercado [1], Baudouin and Puel
[2], [3], Mercado, Osses and Rosier [10], Yuan and Yamamoto [17].

The authors, in the above mentioned papers, studied right hand side of more
general form, namely $R(t,x)f(x)$ and proved uniqueness of determination of
the right hand side and Lipschitz stability of the determination. The
machinery of these papers is based on the Carleman estimates with boundary for
the Schrödinger equation. Such an approach is based on pioneering paper of A.
Bukhgeim and M. Klibanov [6]. Unfortunately in the case of Schrödinger
equation a Carleman estimate with boundary require an existence of an
appropriate pseudoconvex function. This condition is restrictive. For example,
in case of the principal part of $A$ be the Laplace operator, it require for
observed part of the boundary $\Gamma$ to be sufficiently large. In the case
of the general elliptic operator $A$ existence of pseudoconvex function
implies the non-trapping condition for the Hamiltonian flow with the
Hamiltonian $H(x,\xi)=\sum_{j,k=1}^{n}a_{jk}(x)\xi_{j}\xi_{k}.$

Proof of the theorem 1 and theorem 2 are based on the following uniqueness and
conditional stability results for the Schrödinger equation:

Assuming the zero Dirichlet boundary condition on the whole lateral boundary
$(0,T)\times\partial\Omega$, we can prove a conditional stability estimate of
logarithmic rate.

######  Theorem 3.

Let a suboundary $\Gamma\subset\partial\Omega$ and a constant $M>0$ be
arbitrarily chosen. Let $u,\partial_{t}u\in H^{1,2}(Q)$ satisfy

(1.8) $P(x,D)u:=i\partial_{t}u+A(x,D^{\prime})u=0\quad\mbox{in}\quad Q$

and

(1.9) $u=0\quad\mbox{on $(0,T)\times\partial\Omega$}.$

If

(1.10)
$\|u\|_{H^{1,2}((0,T)\times\Omega)}+\|\partial_{t}u\|_{H^{1,2}((0,T)\times\Omega)}\leq
M,$

then there exists a constant $C>0$ independent of $u$ such that

(1.11)
$\|u\|_{C([0,T];L^{2}(\Omega))}\leq\frac{C}{|\log\|\partial_{\nu}u\|_{H^{1}((0,T)\times\Gamma)}|}.$

We emphasize that in addition to (1.2) and (1.3), we do not assume any extra
conditions for the principal coefficients $a_{jk}$, no geometrical constraints
on $\Gamma$ and no limitation of length of time interval of observation $T.$

We remark that if $a_{jk}$ satisfies some conditions called the pseudo-
convexity (e.g., [2], [3], Yuan and Yamamoto [17]) and $\Gamma$ is some large
portion of $\partial\Omega$, then one can prove the Lipschitz stability rather
than the logarithmic stability. In a particular case where $a_{jj}=1$ and
$a_{jk}=0$ for $j\neq k$ and $j,k=1,...,n$, if
$\Gamma\supset\\{x\in\partial\Omega;\,(x-x_{0})\cdot\nu(x)\geq 0\\}$ with some
$x_{0}\in\mathbb{R}^{n}$ (e.g., [2], [3]), then

$\|u(0,\cdot)\|_{L^{2}(\Omega)}\leq
C\|\partial_{\nu}u\|_{L^{2}((0,T)\times\Gamma)}.$

Here we emphasize that this is unconditional stability without any a priori
condition such as (1.10).

The conclusion holds under the a priori boundedness condition (1.10) and so is
conditional stability.

Finally we formulate the uniqueness result for the Cauchy problem for the
Schrödinger equation with solution from $C([0,T];L^{2}(\Omega)).$

######  Theorem 4.

Let $\Gamma\subset\partial\Omega$ be an arbitrarily chosen subboundary and
$T>0$ be arbitrary. We assume that $u\in C([0,T];L^{2}(\Omega))$ satisfies
(1.1). Then $u=\partial_{\nu}u=0$ on $(0,T)\times\Gamma$ yields $u=0$ in
$(0,T)\times\Omega$.

Remark. Function $u\in C([0,T];L^{2}(\Omega))$ is solution (1.1) such that
$u=\partial_{\nu}u=0$ on $(0,T)\times\Gamma$ if

$(u,P^{*}(x,D)w)_{L^{2}(Q)}=0\quad\forall w\in H^{1,2}(Q),$
$w=\partial_{\nu}w=0\quad\mbox{on}\,\,(0,T)\times\partial\Omega\setminus\Gamma,\,w(T,\cdot)=w(0,\cdot)=0.$

The unique continuation is a fundamental property of partial differential
equations. In order to conclude that $u=0$ in $Q$, the conditions on
$\Gamma\subset\partial\Omega$ and $T>0$ depend on the types of partial
differential equations under consideration. For the parabolic equation
$\partial_{t}u+Au=0$, for any subboundary $\Gamma$ and any $T>0$, without any
assumptions on $a_{jk}$ except for (1.2), (1.3) we can prove the unique
continuation (e.g., Mizohata [11], Saut and Scheurer [14]). Here we do not
intend to create any comprehensive references. Moreover, the same sharp unique
continuation holds also in the case where the coefficients depend on $t$ and
satisfy (1.2) and (1.3). Here and henceforth we do not exploit the minimum
regularity of the coefficients.

For the hyperbolic equation $\partial_{t}^{2}u+Au=0$, in the case where
$a_{jk}$ do not depend on $t$ or analytic in $t$, a sharp unique continuation
of global type similar to the parabolic equation holds provided that $T>0$ is
sufficiently large (e.g., Robbiano [13], Tataru [15]).

Compared with the parabolic and hyperbolic equations, it seems that the
researches on the unique continuation for Schrödinger equations, are not
completed. Although we may be suggested to be able to prove the sharp unique
continuation similar to the parabolic equation by comprehensive researches on
Carleman estimates, we can not find a short proof in the existing
publications, and one of main purposes of this article is to provide such a
self-contained and simple proof.

The logarithmic conditional stability results for the unique continuation of
the hyperbolic equation were established in [4] and [5].

The article is composed of six sections and appendix. In Sections 2 and 3, we
prove Theorem 3 statement I and Theorem 4 respectively. Section 4 is devoted
to the proof Theorem 3 statement II. In Section 5, prove Theorem 1 and in
section 6 we give a proof of Theorem 2. In appendix we solve some
controllability problem for 1-D Schrödinger equation in order to construct a
kernel of integral transform.

##  2\. Proof of Theorem 1, statement I

We define the operator by

(2.1) $(Kv)(t):=R(0)v(t)+\int^{t}_{0}R^{\prime}(t-s)v(s)ds,\quad 0<t<T.$

Let $u$ satisfy (1.4) and $u(0,\cdot)=0$ in $\Omega$ and $u=|\nabla u|=0$ on
$(0,T)\times\Gamma$. Consider the equation with respect to $z(t,x)$:

(2.2) $\partial_{t}u(t,x)=(Kz)(t,x),\quad 0<t<T,\,x\in\Omega.$

Since $R(0)\neq 0$, the operator $K$ is the Volterra operator of the second
kind, and we see that $K^{-1}:H^{1}(0,T)\longrightarrow H^{1}(0,T)$ exists and
is bounded. Therefore, $z(\cdot,x)\in H^{1}(0,T)$ is well defined for each
$x\in\Omega$, and

(2.3) $\partial_{t}u(t,x)=R(0)z(t,x)+\int^{t}_{0}R^{\prime}(t-s)z(s,x)ds,\quad
0<t<T,\,x\in\Omega$

by $\partial_{t}u\in H^{1,2}(Q)$.

Since $u(0,\cdot)=0$ in $\Omega$ and

$R(0)z(t,x)+\int^{t}_{0}R^{\prime}(t-s)z(s,x)ds=\partial_{t}\left(\int^{t}_{0}R(t-s)z(s,x)ds\right),$

we obtain

$u(t,x)=\int^{t}_{0}R(t-s)z(s,x)ds,\quad 0<t<T,\,x\in\Omega,$

that is,

(2.4) $u(t,x)=\int^{t}_{0}R(s)z(t-s,x)ds,\quad 0<t<T,\,x\in\Omega.$

We will prove that $z\in H^{1,2}(Q)$ satisfies

(2.5) $i\partial_{t}z(t,x)+A(x,D^{\prime})z(t,x)=0\quad\mbox{in
$(0,T)\times\Omega$},$

where $t_{*}\in(0,T)$ is some constant, and

(2.6) $z=|\nabla z|=0\quad\mbox{on $(0,T)\times\Gamma$}.$

We can readily verify (2.6), because $\partial_{t}u=|\nabla\partial_{t}u|=0$
on $(0,T)\times\Gamma$ implies $Kz=|K(\nabla z)|=0$ on $(0,T)\times\Gamma$, so
that the injectivity of $K$ directly yields (2.6).

Using $\partial_{t}u\in H^{1,2}(Q)\subset C([0,T];L^{2}(\Omega))$ by (2.3), we
have

$\partial_{t}u(0,x)=R(0)z(0,x),\quad x\in\Omega.$

On the other hand, substituting $t=0$ in (1.4) we obtain

$\partial_{t}u(0,x)=-iR(0)f(x),\quad x\in\Omega.$

Hence $R(0)z(0,x)=-iR(0)f(x)$ for $x\in\Omega$. By $R(0)\neq 0$, we reach

(2.7) $z(0,x)=-if(x),\quad x\in\Omega.$

Now we will prove (2.5). In terms of (2.4) and (2.7), we have

$\displaystyle\partial_{t}u(t,x)=R(t)z(0,x)+\int^{t}_{0}R(s)\partial_{t}z(t-s,x)ds$
$\displaystyle=$ $\displaystyle-
iR(t)f(x)+\int^{t}_{0}R(s)\partial_{t}z(t-s,x)ds$

and

$A(x,D^{\prime})u(t,x)=\int^{t}_{0}R(s)A(x,D^{\prime})z(t-s,x)ds,\quad
0<t<T,\,x\in\Omega.$

Consequently (1.4) implies

$\displaystyle-iR(t)f(x)=\partial_{t}u-iA(x,D^{\prime})u(t,x)$
$\displaystyle=$ $\displaystyle-
iR(t)f(x)+\int^{t}_{0}R(s)(\partial_{t}z-iA(x,D^{\prime})z)(t-s,x)ds,$

that is,

$\int^{t}_{0}R(s)(\partial_{t}z-iA(x,D^{\prime})z)(t-s,x)ds=0,\quad
0<t<T,\,x\in\Omega.$

Hence, setting
$Z(s):=\|(\partial_{s}z-iA(x,D^{\prime})z)(s,\cdot)\|_{L^{2}(\Omega)}$ for
$0<t<T$, we reach

$\int^{t}_{0}R(s)Z(t-s)ds=0,\quad 0<t<T.$

By the Titchmarsh convolution theorem (e.g., Titchmarsh [16]), there exists
$t_{*}\in[0,T]$ such that

$R(s)=0\quad\mbox{for $0<s<T-t_{*}$},\quad Z(s)=0\quad\mbox{for $0<s<t_{*}$}.$

Since $R(0)\neq 0$ in $[0,T]$, we see that $t^{*}=T.$ Thus the verification of
(2.5) is complete.

From (2.5) and (2.6) applying the uniqueness theorem 4 we obtain
$z(0,\cdot)=0.$ Then equality (2.7) implies $f=0.$ $\blacksquare$

##  3\. Proof of Theorem 4.

In order to prove the second part of theorem 1 we need the uniqueness result
for the Cauchy problem for the Schrödinger equation with data on an arbitrary
small part of the boundary.

Proof. Let $\widetilde{\Omega}$ be a bounded domain in $\mathbb{R}^{n}$ with
smooth boundary such that $\Omega\subset\widetilde{\Omega}$,
$\partial\Omega\setminus\Gamma\subset\partial\widetilde{\Omega}$ and
$\omega=\widetilde{\Omega}\setminus\overline{\Omega}$ is an open set. We
extend a function $u$ by zero on $(0,T)\times\tilde{\Omega}\setminus\Omega$.
Function $u$ satisfies

(3.1) $P(x,D)u=0\quad\mbox{in}\,\,(0,T)\times\tilde{\Omega},\quad
u|_{(0,T)\times\omega}=0.$

Let $\mathcal{K}(t,\tau)\in C^{\infty}([-1,1]\times[0,T])$ be solution to the
boundary controllability problem for the Shrödinger equation:

$\displaystyle
i\partial_{\tau}\mathcal{K}-\partial^{2}_{t}\mathcal{K}=0\quad\mbox{in}\quad
t\in(-1,1)\,\,\tau\in(0,T),$ (3.2)
$\displaystyle\quad\mathcal{K}(-1,\tau)=\psi(\tau)\,\,\mbox{on}\,\,(0,T),\,\quad\mathcal{K}(\cdot,0)=\mathcal{K}(\cdot,T)=0.$

Here $\psi\in C^{\infty}_{0}(0,T)$ is fixed complex valued function. Control
is located on the part of the boundary $t=1.$ For existence of such a solution
see e.g. [9]. Consider the function
$w(x)=\int_{0}^{T}\mathcal{K}(t,\tau)u(\tau,x)d\tau.$ This function satisfies
the equation

(3.3)
$L(x,D)w=-\partial_{t}^{2}w+A(x,D^{\prime})w=0\quad\mbox{in}\quad(-1,1)\times\tilde{\Omega},\quad
w=0\quad\mbox{on}\quad(-1,1)\times\omega.$

The second inequality in (3.3) follows from (3.1) immediately. In order to
prove the first equality we observe that for any $p\in
H^{2}_{0}((-1,1)\times\tilde{\Omega})$

$(w,L^{*}(x,D)p)_{L^{2}((-1,1)\times\tilde{\Omega})}=(w,-\partial_{t}^{2}p+A^{*}(x,D^{\prime})p)_{L^{2}((-1,1)\times\tilde{\Omega})}=$
$(\int_{0}^{T}\mathcal{K}(t,\tau)u(\tau,x)d\tau,(-\partial_{t}^{2}p+A^{*}(x,D^{\prime})p)_{L^{2}((-1,1)\times\tilde{\Omega})}=$
$(u,-\int_{-1}^{1}\mathcal{K}(t,\tau)\partial_{t}^{2}pdt+A^{*}(x,D^{\prime})\int_{-1}^{1}\mathcal{K}(t,\tau)pdt)_{L^{2}((0,T)\times\tilde{\Omega})}=$
$(u,-i\partial_{\tau}\int_{-1}^{1}\mathcal{K}(t,\tau)pdt+A^{*}(x,D^{\prime})\int_{-1}^{1}\mathcal{K}(t,\tau)pdt)_{L^{2}((0,T)\times\tilde{\Omega})}=(u,P^{*}(x,D)\tilde{p})_{L^{2}(Q)}=0$

where $\tilde{p}(\tau,x)=\int_{-1}^{1}\mathcal{K}(t,\tau)p(t,x)dt.$

The equations (3.3) are established. By assumptions (1.2), (1.3) the operator
$L$ is the second order elliptic operator with $C^{1}$ coefficients in the
principal part and $L^{\infty}$ coefficients in first and zero order terms.
Then theorem 8.9.1 from [7] which establish uniqueness of the Cauchy problem
for this operator can be applied. This implies that function $w$ is
identically equal to zero on the cylinder $(-1,1)\times\Omega.$ In particular

$w(-1,x)=\int_{0}^{T}\mathcal{K}(-1,\tau)u(\tau,x)d\tau=\int_{0}^{T}\psi(\tau)u(\tau,x)d\tau=0\quad\mbox{on}\quad\Omega.$

Since the space $C^{\infty}_{0}(0,T)$ is dense in $L^{2}(0,T)$ we have
$u\equiv 0.$ Proof of theorem is complete. $\blacksquare$

##  4\. Proof of Theorem 1, statement II

Proof. Without loss of generality we may assume that

(4.1) $0\in\mbox{supp}\,R.$

Indeed, since $\mbox{supp}\,R$ is not empty there exist a point
$\tilde{t}\in\mbox{supp}\,R.$ Denote by $t_{0}$ infimum over such points.
Function $u$ is zero on $(0,t_{0})\times\Omega.$ We replace function $u$ and
$R$ in (1.4) by $y(t+t_{0},\cdot)$ and $R(t+t_{0})$ respectively to obtain
(4.1).

Next we reduce the inverse problem to the case when $R(0)=0.$ Function
$w(x)=\int_{0}^{t}u(\tau,x)d\tau$ satisfies :

$P(x,D)w=\int_{0}^{t}R(\tau)d\tau f(x)\quad\mbox{in}\quad Q,\quad
w(0,\cdot)=0,\quad
w|_{(0,T)\times\partial\Omega}=\partial_{\nu}w|_{(0,T)\times\Gamma}=0.$

Obviously function $\tilde{R}(t)=\int_{0}^{t}R(\tau)d\tau$ satisfies

$\tilde{R}(0)=0.$

Moreover, since $\frac{d\tilde{R}}{dt}=R$ by (1.6), function $\tilde{R}$ is
not identically equal zero on $(0,T).$ Hence, besides (1.6) we may
additionally assume

(4.2) $R(0)=0.$

Let $\widetilde{\Omega}$ be a bounded domain in $\mathbb{R}^{n}$ with smooth
boundary such that $\Omega\subset\widetilde{\Omega}$,
$\partial\Omega\setminus\Gamma\subset\partial\widetilde{\Omega}$ and
$\omega=\widetilde{\Omega}\setminus\overline{\Omega}$ is an open set. We
extend functions $u$ and $f$ by zero on
$(0,T)\times\tilde{\Omega}\setminus\Omega$. By our assumptions we have

$P(x,D)u=Rf\quad\mbox{in}\quad(0,T)\times\tilde{\Omega},\quad
u|_{(0,T)\times\partial\tilde{\Omega}}=0,\,\,u=0\quad\mbox{on}\quad(0,T)\times\omega.$

Next we consider function

(4.3) $y(t,x)=\int_{0}^{t}R(t-\tau)v(\tau,x)d\tau,\quad x\in\tilde{\Omega},$

where function $v\in C([0,T];L^{2}(\tilde{\Omega}))$ solves the initial value
problem

$P(x,D)v=0\quad\mbox{in}\quad(0,T)\times\tilde{\Omega},\quad
v(0,\cdot)=-if,\quad v|_{(0,T)\times\partial\tilde{\Omega}}=0.$

By (4.3) function $y(t,x)$ satisfies the initial and boundary conditions

(4.4) $\quad y|_{(0,T)\times\partial\tilde{\Omega}}=0,\quad y(0,\cdot)=0.$

The short computations imply

(4.5) $P(x,D)y=Rf\quad\mbox{in}\quad(0,T)\times\tilde{\Omega}.$

Indeed, by (4.2) for any $p\in
H^{1,2}((0,T)\times\tilde{\Omega}),p(T,\cdot)=0,p|_{(0,T)\times\partial\tilde{\Omega}}=0$
we have

$(y,P^{*}p)_{L^{2}((0,T)\times\tilde{\Omega})}=(y,i\partial_{t}p+A^{*}p)_{L^{2}((0,T)\times\tilde{\Omega})}=(\int_{0}^{t}R(t-\tau)v(\tau,x)d\tau,i\partial_{t}p+A^{*}p)_{L^{2}((0,T)\times\tilde{\Omega})}=$
$(\int_{0}^{t}R(t-\tau)v(\tau,x)d\tau,A^{*}p)_{L^{2}((0,T)\times\tilde{\Omega})}-(\int_{0}^{t}\partial_{t}R(t-\tau)v(\tau,x)d\tau,ip)_{L^{2}((0,T)\times\tilde{\Omega})}=$
$(\int_{0}^{t}R(t-\tau)v(\tau,x)d\tau,A^{*}p)_{L^{2}((0,T)\times\tilde{\Omega})}+(\int_{0}^{t}\partial_{\tau}R(t-\tau)v(\tau,x)d\tau,ip)_{L^{2}((0,T)\times\tilde{\Omega})}=$
$\int_{0}^{T}(v,P^{*}R(t-\cdot)p(t,\cdot))_{L^{2}((0,T)\times\tilde{\Omega})}dt=(f,p)_{L^{2}((0,T)\times\tilde{\Omega})}.$

Hence function $y$ solves the initial value problem (4.5), (4.4). By the
uniqueness of solution to this initial value problem $u=y$ and

$y|_{(0,T)\times\omega}=0.$

This equality implies

$\int_{0}^{t}R(t-\tau)v(\tau,x)d\tau=0\quad\mbox{on}\quad(0,T)\times\omega.$

Hence

$\int_{0}^{t}R(t-\tau)\|v(\tau,\cdot)\|_{L^{2}(\omega)}d\tau=0.$

Applying the Titchmarsh convolution theorem we have there exist $\lambda\geq
0$ and $r\geq 0$ such that

(4.6) $\lambda+r\geq T$

and function $R$ is equal to zero on $(0,\lambda)$ and function
$\|v(\tau,\cdot)\|_{L^{2}(\omega)}$ is equal to zero on $(0,r).$ By (1.6) and
(4.1) parameter $\lambda=0.$ Hence by (4.6) we have $r=T$ and

$v=0\quad\mbox{on}\quad(0,T)\times\omega.$

Then theorem 4 implies

$v(t,x)=0\quad\mbox{on}\quad(0,T)\times\Omega.$

Hence, by (4.3)

$y(t,x)=0\quad\mbox{on}\quad(0,T)\times\Omega.$

This implies

$Rf=0\quad\mbox{on}\quad(0,T)\times\Omega.$

Since function $R$ is not identically equal zero on $(0,T)$ we have $f\equiv
0.$ $\blacksquare$

##  5\. Proof of Theorem 3.

Proof. Without loss of generality we may assume that

$\|\partial_{\nu}u\|_{L^{2}((0,T)\times\Gamma)}\leq
1\quad\mbox{and}\quad\|\partial_{\nu}\partial_{t}u\|_{L^{2}((0,T)\times\Gamma)}\leq
1.$

Let $y\in\Omega$ and $\mathcal{K}(t,\tau)$ be solution to the boundary
controllability problem for the one dimensional Shrödinger equation:

$\displaystyle
i\partial_{\tau}\mathcal{K}-\partial^{2}_{t}\mathcal{K}=0\quad\mbox{in}\quad
t\in(0,2),\,\tau\in(0,T),$ (5.1)
$\displaystyle\quad\mathcal{K}(0,\tau)=\psi(\tau)\,\,\mbox{on}\,\,(0,T),\quad\mathcal{K}(\cdot,0)=\mathcal{K}(\cdot,T)=0.$

Control is located on the part of the boundary $t=2:$

$\mathcal{K}(2,\tau)=v(\tau).$

A given function $\psi$ is smooth function but in general does not satisfy to
the compatibility conditions at time moment $\tau=0$ or $\tau=T.$ Solution to
the controllability problem (5) is not unique and in general is not smooth,
since function $\psi$ in general does not satisfy the compatibility
conditions. On the other hand one can chose solution to the controllability
problem in such a way that

(5.2) $\|\mathcal{K}\|_{L^{2}((0,2)\times(0,T))}\leq C,$

provided that function $\psi$ belongs to some bounded set in $C^{2}[0,T].$ We
give the proof of this fact in appendix.

Consider function $w(t,x)=\int_{0}^{T}\mathcal{K}(t,\tau)u(\tau,x)d\tau$ and
$r(t,x)=\int_{0}^{T}\mathcal{K}(t,\tau)\partial_{\nu}u(\tau,x)d\tau.$ Observe
that the function $w$ belongs to the space $L^{2}((0,2)\times\Omega).$ Indeed

$\|w\|^{2}_{L^{2}((0,2)\times\Omega)}=\int_{0}^{2}\|\int_{0}^{T}\mathcal{K}(t,\tau)u(\tau,x)d\tau\|^{2}_{L^{2}(\Omega)}dt\leq\int_{0}^{2}\int_{0}^{T}\|\mathcal{K}(t,\tau)u(\tau,x)\|^{2}_{L^{2}(\Omega)}d\tau
dt$ $\leq\int_{0}^{2}\int_{0}^{T}|\mathcal{K}(t,\tau)|^{2}d\tau
dt\int_{0}^{T}\|u(\tau,\cdot)\|^{2}_{L^{2}(\Omega)}d\tau.$

Similarly, by (5.2) we have

(5.3) $\|r\|_{L^{2}((0,2)\times\Gamma)}\leq
C\|\partial_{\nu}u\|_{L^{2}((0,T)\times\Gamma)}.$

Function $w$ satisfies the equation

(5.4)
$L(x,D)w=-\partial_{t}^{2}w+A(x,D^{\prime})w=0\quad\mbox{in}\quad(0,2)\times\Omega,$
(5.5) $\quad
w=0\quad\mbox{on}\quad(0,2)\times\Omega,\quad\partial_{\nu}w=r\quad\mbox{on}\quad(0,2)\times\Gamma.$

So, if

(5.6) $\|u\|_{H^{1,2}(Q)}\leq M,$

since

$\|w(0,\cdot)\|_{H^{2}(\Omega)}+\|w(2,\cdot)\|_{H^{2}(\Omega)}\leq
C\|u\|_{L^{2}(0,T;H^{2}(\Omega))}$

we have

(5.7) $\|w\|_{H^{2}((0,2)\times\Omega)}\leq CM.$

Next we claim that there exists $\hat{\theta}>0$ and constant $C$ , both are
independent of $w$, such that

(5.8)
$\|w(1,\cdot)\|_{L^{2}(\Omega)}+\|\partial_{t}w(1,\cdot)\|_{L^{2}(\Omega)}\leq
CM\|r\|^{\hat{\theta}}_{L^{2}((0,2)\times\Gamma)}.$

Indeed, let $\psi\in C^{2}(\bar{\Omega})$ be a function such that
$\nabla\psi\neq 0$ on $\Omega,$ and $\partial_{\nu}\psi<0$ on
$\partial\Omega\setminus\Gamma,\psi>0$ on $\Omega.$ (For existence of such a
function see e.g. [8].) Let $\phi(t,x)=\psi(x)-K(t-1)^{2}$ with sufficiently
large $K$ such that

(5.9)
$\inf_{x\in\Omega}\phi(1,x)>\mbox{max}\\{\|\phi(0,\cdot)\|_{C^{0}(\bar{\Omega})},\|\phi(2,\cdot)\|_{C^{0}(\bar{\Omega})}\\}.$

We set $\varphi(t,x)=e^{\lambda\phi(t,x)}$. By direct computations one can
check that there exists $\lambda_{0}$ such that for all
$\lambda\geq\lambda_{0}$ the function $\varphi(t,x)$ is pseudo-convex with
respect to the symbol $-\xi_{0}^{2}-\sum_{k,j=1}^{n}a_{kj}(x)\xi_{k}\xi_{j}.$

By (5.9), for all $\lambda$ sufficiently large, one can chose numbers
$\varphi_{I}$ and $\varphi_{II}$ such that

(5.10)
$\varphi_{I}<\mbox{inf}_{x\in\Omega}\varphi(1,x),\quad\mbox{and}\,\,\varphi_{I}>\varphi_{II}>\mbox{max}\\{\|\varphi(0,\cdot)\|_{C^{0}(\bar{\Omega})},\|\varphi(2,\cdot)\|_{C^{0}(\bar{\Omega})}\\}.$

We fix some $\lambda>\lambda_{0}$ such that (5.10) holds true.

Since function $\varphi$ is pseudo-convex with respect to the symbol
$-\xi_{0}^{2}-\sum_{k,j=1}^{n}a_{kj}(x)\xi_{k}\xi_{j}$ the following Carleman
estimate for the second order elliptic equation holds true (see e.g. [8]):

$\displaystyle s^{3}\|we^{s\varphi}\|^{2}_{L^{2}((0,2)\times\Omega)}+s\|\nabla
we^{s\varphi}\|^{2}_{L^{2}((0,2)\times\Omega)}+\|L_{+}(x,D,s)(we^{s\varphi})\|^{2}_{L^{2}((0,2)\times\Omega)}$
$\displaystyle\leq
C(s^{3}\|we^{s\varphi}(0,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(0,\cdot)\|^{2}_{L^{2}(\Omega)}$ (5.11)
$\displaystyle+s^{3}\|we^{s\varphi}(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|\partial_{\nu}we^{s\varphi}\|^{2}_{L^{2}((0,2)\times\Gamma)})\quad\forall
s\geq s_{0},$

where

$L_{+}(x,D,s)=\partial_{t}^{2}+\sum_{k,j=1}^{n}a_{kj}\partial^{2}_{x_{k}x_{j}}+s^{2}((\partial_{t}\varphi)^{2}+\sum_{k,j=1}^{n}a_{kj}\partial_{x_{k}}\varphi\partial_{x_{j}}\varphi).$

The inequality (5) implies

$\displaystyle\frac{1}{s}\|\partial_{t}^{2}w+\sum_{k,j=1}^{n}a_{kj}\partial^{2}_{x_{k}x_{j}}w\|^{2}_{L^{2}((0,2)\times\Omega)}\leq
C(s^{3}\|we^{s\varphi}(0,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(0,\cdot)\|^{2}_{L^{2}(\Omega)}$ (5.12)
$\displaystyle+s^{3}\|we^{s\varphi}(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|\partial_{\nu}we^{s\varphi}\|^{2}_{L^{2}((0,2)\times\Gamma)})\quad\forall
s\geq s_{0}.$

Let $\rho(t)\in C^{\infty}_{0}[\frac{1}{8},\frac{15}{16}]$ be such that
$\rho=1$ on $(\frac{1}{2},\frac{3}{2}).$ By (5) and (5) we have

$\displaystyle\frac{1}{s}\|\partial_{t}^{2}(\rho
w)+\sum_{k,j=1}^{n}a_{kj}\partial^{2}_{x_{k}x_{j}}(\rho
w)\|^{2}_{L^{2}((0,2)\times\Omega)}\leq
C(s^{3}\|we^{s\varphi}(0,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(0,\cdot)\|^{2}_{L^{2}(\Omega)}$ (5.13)
$\displaystyle+s^{3}\|we^{s\varphi}(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|\partial_{\nu}we^{s\varphi}\|^{2}_{L^{2}((0,2)\times\Gamma)})\quad\forall
s\geq s_{0}.$

Function $\rho w$ satisfies the zero Dirichlet boundary conditions on
$\partial((\frac{1}{8},\frac{15}{16})\times\Omega).$ Applying the $L^{2}$ a
priori estimates for elliptic operators we have

$\displaystyle\frac{1}{s}\|\sum_{k,j=0}\partial^{2}_{x_{k}x_{j}}(\rho
w)\|^{2}_{L^{2}((0,2)\times\Omega)}\leq
C(s^{3}\|we^{s\varphi}(0,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(0,\cdot)\|^{2}_{L^{2}(\Omega)}$ (5.14)
$\displaystyle+s^{3}\|we^{s\varphi}(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|\partial_{\nu}we^{s\varphi}\|^{2}_{L^{2}((0,2)\times\Gamma)})\quad\forall
s\geq s_{0}.$

Finally, combining estimates (5) and (5) we obtain

$\displaystyle s^{3}\|we^{s\varphi}\|^{2}_{L^{2}((0,2)\times\Omega)}+s\|\nabla
we^{s\varphi}\|^{2}_{L^{2}((0,2)\times\Omega)}+\frac{1}{s}\|\sum_{j,k=0}^{n}\partial_{x_{j}}\partial_{x_{k}}(we^{s\varphi})\|^{2}_{L^{2}((\frac{1}{2},\frac{3}{2})\times\Omega)}$
$\displaystyle\leq
C(s^{3}\|we^{s\varphi}(0,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(0,\cdot)\|^{2}_{L^{2}(\Omega)}$ (5.15)
$\displaystyle+s^{3}\|we^{s\varphi}(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(2,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|\partial_{\nu}we^{s\varphi}\|^{2}_{L^{2}((0,2)\times\Gamma)})\quad\forall
s\geq s_{0}.$

Setting $\varphi_{M}>\|\varphi\|_{C^{0}([0,2]\times\bar{\Omega})}$ from (5)
and the trace theorem we have

$\displaystyle\frac{1}{s}(\|w(1,\cdot)\|^{2}_{H^{1}(\Omega)}+\|\partial_{t}w(1,\cdot)\|^{2}_{L^{2}(\Omega)})e^{s\varphi_{I}}$
$\displaystyle\leq
Ce^{s\varphi_{II}}(\|w(0,\cdot)\|^{2}_{L^{2}(\Omega)}+\|(\nabla
w)(0,\cdot)\|^{2}_{L^{2}(\Omega)}+\|w(2,\cdot)\|^{2}_{L^{2}(\Omega)}$ (5.16)
$\displaystyle+\|(\nabla
w)(2,\cdot)\|^{2}_{L^{2}(\Omega)})+Ce^{s\varphi_{M}}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}\quad\forall
s\geq s_{0}.$

Dividing both sides of inequality (5) by $e^{s\varphi_{I}}/s$ we obtain

$\displaystyle\|w(1,\cdot)\|^{2}_{H^{1}(\Omega)}+\|\partial_{t}w(1,\cdot)\|^{2}_{L^{2}(\Omega)}$
$\displaystyle\leq Ce^{-s\alpha}(\|w(0,\cdot)\|^{2}_{L^{2}(\Omega)}+\|(\nabla
w)(0,\cdot)\|^{2}_{L^{2}(\Omega)}$ (5.17)
$\displaystyle+\|w(2,\cdot)\|^{2}_{L^{2}(\Omega)}+\|(\nabla
w)(2,\cdot)\|^{2}_{L^{2}(\Omega)})+Ce^{s\beta}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}\quad\forall
s\geq s_{0}.$

Here $\alpha=\varphi_{I}-\varphi_{II},\beta=\varphi_{M}-\varphi_{I}.$ By
(5.10) these numbers are strictly positive. From (5), using (5.7) we have

(5.18)
$\|w(1,\cdot)\|^{2}_{H^{1}(\Omega)}+\|\partial_{t}w(1,\cdot)\|^{2}_{L^{2}(\Omega)}\leq
Ce^{-s\alpha}M^{2}+Ce^{s\beta}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}.$

Let

(5.19)
$M^{2}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}<\mbox{min}\\{e^{-s_{0}(\alpha+\beta)},1\\}.$

Setting in (5.18)
$s=-a\log(M^{2}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}),a=\frac{1}{\alpha+\beta}$
we obtain

$\displaystyle\|w(1,\cdot)\|^{2}_{H^{1}(\Omega)}+\|\partial_{t}w(1,\cdot)\|^{2}_{L^{2}(\Omega)}$
$\displaystyle\leq
C(M^{2}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)})^{a\alpha}M^{2}+C(M^{2}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)})^{-a\beta}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}$
(5.20) $\displaystyle\leq
CM^{1(1-\theta)}\|\partial_{\nu}w\|^{2\theta}_{L^{2}((0,2)\times\Gamma)},$

where $\theta=\alpha/(\alpha+\beta).$ Hence the proof of estimate (5.8) is
complete.

Next we need the analog of the estimate (5) for $t$ from $(0,1].$ We claim
that there exists a positive $\theta_{0}$ such that

(5.21)
$\|w(t,\cdot)\|_{L^{2}(\Omega)}+\|\partial_{t}w(t,\cdot)\|_{L^{2}(\Omega)}\leq
CM\|r\|^{t\theta_{0}}_{L^{2}((0,2)\times\Gamma)}\quad\forall
t\in(0,\frac{1}{2}].$

In order to prove the estimate (5.21) we again use technique bases on the
Carleman estimates. Observe that the function $\tilde{\varphi}(t)=e^{\lambda
t}-1$ for all $\lambda$ sufficiently large is strictly pseudoconvex with
respect to the symbol $-\xi_{0}^{2}-\sum_{k,j=1}^{n}a_{kj}(x)\xi_{k}\xi_{j}.$
Then the following Carleman estimate holds true (see e.g. [8]):

$\displaystyle
s^{3}\|we^{s\tilde{\varphi}}\|^{2}_{L^{2}((0,1)\times\Omega)}+s\|\nabla
we^{s\tilde{\varphi}}\|^{2}_{L^{2}((0,1)\times\Omega)}+\|(L(x,D)+(s\partial_{t}\tilde{\varphi})^{2})(we^{s\tilde{\varphi}})\|^{2}_{L^{2}((0,1)\times\Omega)}$
$\displaystyle\leq
C(s^{3}\|we^{s\tilde{\varphi}}(0,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\varphi})(0,\cdot)\|^{2}_{L^{2}(\Omega)}$ (5.22)
$\displaystyle+s^{3}\|we^{s\tilde{\varphi}}(1,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\tilde{\varphi}})(1,\cdot)\|^{2}_{L^{2}(\Omega)})\quad\forall s\geq
s_{1}.$

Repeating the arguments (5)-(5) we obtain from (5):

$\displaystyle
s^{3}\|we^{s\tilde{\varphi}}\|^{2}_{L^{2}((0,1)\times\Omega)}+s\|\nabla
we^{s\tilde{\varphi}}\|^{2}_{L^{2}((0,1)\times\Omega)}+\frac{1}{s}\|\sum_{k,j=1}^{n}\partial^{2}_{x_{k}x_{j}}(we^{s\tilde{\varphi}})\|^{2}_{L^{2}((0,\frac{1}{2})\times\Omega)}$
$\displaystyle\leq
C(s^{3}\|we^{s\tilde{\varphi}}(0,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\tilde{\varphi}})(0,\cdot)\|^{2}_{L^{2}(\Omega)}$ (5.23)
$\displaystyle+s^{3}\|we^{s\tilde{\varphi}}(1,\cdot)\|^{2}_{L^{2}(\Omega)}+s\|(\nabla
we^{s\tilde{\varphi}})(1,\cdot)\|^{2}_{L^{2}(\Omega)})\quad\forall s\geq
s_{2}.$

From (5) and the trace theorem for all $t$ in $(0,\frac{1}{2}]$ we have

(5.24)
$\displaystyle\frac{1}{s}(\|w(t,\cdot)\|^{2}_{H^{1}(\Omega)}+\|\partial_{t}w(t,\cdot)\|^{2}_{L^{2}(\Omega)})e^{s\tilde{\varphi}(t)}\leq
C(\|w(0,\cdot)\|^{2}_{L^{2}(\Omega)}$ $\displaystyle+\|(\nabla
w)(0,\cdot)\|^{2}_{L^{2}(\Omega)}+e^{s\tilde{\varphi}(1)}(\|w(1,\cdot)\|^{2}_{L^{2}(\Omega)}+\|(\nabla
w)(1,\cdot)\|^{2}_{L^{2}(\Omega)})\forall s\geq s_{2}.$

Dividing both sides of inequality (5) by $e^{s\tilde{\varphi}(t)}/s$ for some
positive $\tilde{\alpha}$ and $\tilde{\beta}$ we obtain

$\displaystyle\|w(t,\cdot)\|^{2}_{H^{1}(\Omega)}+\|\partial_{t}w(t,\cdot)\|^{2}_{L^{2}(\Omega)}$
$\displaystyle\leq
Ce^{-s\tilde{\alpha}t}(\|w(0,\cdot)\|^{2}_{L^{2}(\Omega)}+\|(\nabla
w)(0,\cdot)\|^{2}_{L^{2}(\Omega)})$ (5.25)
$\displaystyle+e^{s\tilde{\beta}t}\|w(1,\cdot)\|^{2}_{L^{2}(\Omega)}+\|(\nabla
w)(1,\cdot)\|^{2}_{L^{2}(\Omega)})\quad\forall s\geq
s_{2}\,\,\mbox{and}\,\,\forall t\in(0,\frac{1}{2}].$

From (5), using (5.7) we have

(5.26)
$\|w(t,\cdot)\|^{2}_{H^{1}(\Omega)}+\|\partial_{t}w(t,\cdot)\|^{2}_{L^{2}(\Omega)}\leq
Ce^{-s\tilde{\alpha}t}M^{2}+Ce^{s\tilde{\beta}t}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}\quad\forall
s\geq s_{2}\,\,\,\,\forall t\in(0,\frac{1}{2}].$

Let

(5.27)
$M^{2}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}<\mbox{min}\\{e^{-s_{0}(\alpha+\beta)},e^{-s_{2}(\tilde{\alpha}+\tilde{\beta})},1\\}.$

Setting in (5.26)
$s=-\tilde{a}\ln(M^{2}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}),\tilde{a}=\frac{1}{\tilde{\alpha}+\tilde{\beta}}$
we obtain

$\displaystyle\|w(t,\cdot)\|^{2}_{H^{1}(\Omega)}+\|\partial_{t}w(t,\cdot)\|^{2}_{L^{2}(\Omega)}$
$\displaystyle\leq
C(M^{2}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)})^{\tilde{a}\tilde{\alpha}t}M^{2}+C(M^{2}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)})^{-\tilde{a}\tilde{\beta}t}\|\partial_{\nu}w\|^{2}_{L^{2}((0,2)\times\Gamma)}$
(5.28) $\displaystyle\leq
CM^{2}\|\partial_{\nu}w\|^{2t\theta_{0}}_{L^{2}((0,2)\times\Gamma)}\quad\forall
t\in(0,\frac{1}{2}],$

where $\theta_{0}=\alpha/(\alpha+\beta).$ Hence the proof of estimate (5.21)
is complete.

Finally, using (5.3) and (5.21), we estimate the trace of the function $w$ at
time moment $t=0:$

$\displaystyle\|w(0,\cdot)\|_{L^{2}(\Omega)}\leq\|\int_{0}^{\frac{1}{2}}\partial_{t}wdt\|_{L^{2}(\Omega)}+\|w(\frac{1}{2},\cdot)\|_{L^{2}(\Omega)}$
$\displaystyle\leq\int_{0}^{\frac{1}{2}}CM\|r\|^{t\theta_{0}}_{L^{2}((0,2)\times\Gamma)}dt+CM\|r\|^{\theta_{0}}_{L^{2}((0,2)\times\Gamma)}=$
$\displaystyle
CM(\int_{0}^{\frac{1}{2}}e^{t\log(\|r\|^{\theta_{0}}_{L^{2}((0,2)\times\Gamma)})}dt+\|r\|^{\theta_{0}}_{L^{2}((0,2)\times\Gamma)})$
$\displaystyle=CM\left(\frac{1}{\log\|r\|^{\theta_{0}}_{L^{2}((0,2)\times\Gamma)}}(e^{\frac{1}{2}\log(\|r\|^{\theta_{0}}_{L^{2}((0,2)\times\Gamma)})}-1)+\|r\|^{\theta_{0}}_{L^{2}((0,2)\times\Gamma)}\right)$
(5.29) $\displaystyle\leq
C/|\log\|\partial_{\nu}u\|_{L^{2}((0,T)\times\Gamma)}.|.$

Here in order to get the last inequality we used (5.3). If inequality (5.27)
fails the estimate (5) is trivial.

Next we set
$\tilde{w}(t,x)=\int_{0}^{T}\mathcal{K}(t,\tau)\partial_{\tau}u(\tau,x)d\tau$
and
$\tilde{r}(t,x)=\int_{0}^{T}\mathcal{K}(t,\tau)\partial_{\nu}\partial_{\tau}u(\tau,x)d\tau.$
Repeating the previous arguments we obtain the estimate similar to (5):

(5.30) $\displaystyle\|\tilde{w}(0,\cdot)\|_{L^{2}(\Omega)}\leq
C/|\log\|\partial_{\nu}\partial_{t}u\|_{L^{2}((0,T)\times\Gamma)}|.$

Now we estimate $L^{2}$ norm of the function $u$. The a priori for the
Schrödinger equation imply

(5.31) $\|u\|_{C([0,T];L^{2}(\Omega))}\leq C\|u(0,\cdot)\|_{L^{2}(\Omega)}.$

Next we estimate $u(0,\cdot).$ We use the following notations

$w_{\psi}(t,x)=\int_{0}^{T}\mathcal{K}(t,\tau)u(\tau,x)d\tau,\quad\tilde{w}_{\psi}(t,x)=\int_{0}^{T}\mathcal{K}(t,\tau)\partial_{\tau}u(\tau,x)d\tau,$

where $\mathcal{K}$ is solution to the controllability problem (5). Short
computations imply

$w_{e^{i\kappa
t}}(0,x)=\int_{0}^{T}e^{i\kappa\tau}u(\tau,x)d\tau=-\frac{1}{i\kappa}\int_{0}^{T}(e^{i\kappa\tau}-e^{i\kappa
T})\partial_{\tau}u(\tau,x)d\tau-\frac{1}{i\kappa}(e^{-i\kappa T}-e^{i\kappa
T})u(0,x).$

Then, taking $\kappa$ such that $(e^{-i\kappa T}-e^{i\kappa T})\neq 0$ we
obtain

(5.32) $u(0,x)=\frac{i\kappa}{(e^{-i\kappa T}-e^{i\kappa
T})}\left(w_{e^{i\kappa t}}(0,x)+\frac{1}{i\kappa}(\tilde{w}_{e^{i\kappa
t}}(0,x)-\tilde{w}_{e^{i\kappa T}}(0,x))\right).$

Estimating the right hand side of (5.32) using (5) and (5.30) we obtain

$\|u(0,\cdot)\|_{L^{2}(\Omega)}\leq
C/|\log\|\partial_{\nu}u\|_{L^{2}((0,T)\times\Gamma)}|+C/|\log\|\partial_{\nu}\partial_{t}u\|_{L^{2}((0,T)\times\Gamma)}|$

This estimate and (5.31) implies (1.11). $\blacksquare$

##  6\. Proof of Theorem 2.

Proof. The Volterra operator $K$ constructed in the proof of the theorem 1 is
invertible. Therefore

$\displaystyle\|\partial_{\nu}z\|_{H^{1}(0,T;L^{2}(\Gamma))}\leq
C\|\partial_{\nu}u\|_{H^{2}(0,T;L^{2}(\Gamma))};$
$\displaystyle\quad\|z\|_{H^{1,2}((0,T)\times\Omega)}+\|\partial_{t}z\|_{H^{1,2}((0,T)\times\Omega)}\leq
C(\|u\|_{H^{1,2}((0,T)\times\Omega)}+\|\partial_{t}u\|_{H^{1,2}((0,T)\times\Omega)}+\|\partial^{2}_{t}u\|_{H^{1,2}((0,T)\times\Omega)})$

and function $z$ satisfies the equation

$P(x,D)z=0\quad\mbox{in}\,(0,T)\times\Omega,\quad
z|_{(0,T)\times\partial\Omega}=0,\quad z(0,\cdot)=-if.$

By theorem 3 we have

(6.1)
$\|z\|_{L^{2}((0,T)\times\Omega)}\leq\frac{C}{|\log\|\partial_{\nu}z\|_{H^{1}(0,T;L^{2}(\Gamma))}|}\leq\frac{\tilde{C}}{|\log\|\partial_{\nu}u\|_{H^{2}(0,T;L^{2}(\Gamma))}|}.$

Then the standard a priori estimate for the Schrödinger equation imply (1.7).
$\blacksquare$

##  7\. Appendix

Let $\Psi(t,\tau)\in C^{2}([0,3]\times[0,T])$ be a function such that
$\Psi=\psi.$ Consider the controllability problem for the Schrödinger equation
with locally distributed control $v$:

$\displaystyle
MK=i\partial_{\tau}K-\partial^{2}_{t}K=M\Psi+v\,\,\mbox{in}\quad
t\in(0,3),\,\tau\in(0,T),\,\,\mbox{supp}\,v\subset(2,3)\times(0,T),$ (7.1)
$\displaystyle\quad K(0,\tau)=0\,\,\mbox{on}\,\,(0,T),\quad
K(\cdot,0)=\Psi(\cdot,0),\quad K(\cdot,T)=\Psi(\cdot,T).$

Then function $\Psi-K$ would solve the controllability problem (5).

In order to solve problem (7) it suffice to prove an observability estimate

(7.2) $\|p\|_{L^{2}(G)}\leq C(\|p\|_{L^{2}(G_{1})}+\|Mp\|_{L^{2}(G)}),$

where $G=(0,3)\times(0,T),G_{1}=(2,3)\times(0,T).$ The estimate (7.2) imply
(see e.g. [18]) the existence of solution to problem (7) such that

$\|K\|_{L^{2}(G)}\leq
C(\|\Psi(\cdot,T)\|_{L^{2}(0,3)}+\|\Psi(\cdot,0)\|_{L^{2}(0,3)}+\|M\Psi\|_{L^{2}(G)}).$

In order to prove (7.2) we use the following observability estimate obtained
in [2]: let $b_{1},b_{0}\in L^{\infty}(G)$ there exist a constant $C$ such
that

(7.3) $\|p\|_{L^{2}(0,T;H^{1}(0,3))}\leq
C(\|Mp+b_{1}\partial_{t}p+b_{0}p\|_{L^{2}(G)}+\|p\|_{L^{2}(0,T;H^{1}(2,3))}).$

Consider the function
$\tilde{p}(\tau,t)=\rho(t)(\int_{t}^{3}\rho(\tilde{t})p(\tau,\tilde{t})d\tilde{t}-\int_{0}^{3}\rho(\tilde{t})p(\tau,\tilde{t})d\tilde{t}),$
where $\rho(t)\in
C^{\infty}[0,3],$$\rho|_{[0,2]}=1,$$\rho|_{[\frac{5}{2},3]}=0.$ We derive
equation for the function $\tilde{p}$. Observe that

$M(\rho p)=\rho Mp+2\partial_{t}\rho\partial_{t}p+\partial^{2}_{t}\rho p.$

Integrating this equation on the interval $(t,3)$ we have

$i\partial_{\tau}\int_{t}^{3}\rho pd\tilde{t}+\partial_{t}(\rho
p)+f=M\int_{t}^{3}\rho pd\tilde{t}=M(\int_{t}^{3}\rho
pd\tilde{t}-\int_{0}^{3}\rho pd\tilde{t}),$ $f=\int_{t}^{3}\rho
Mpd\tilde{t}+2\int_{t}^{3}\partial_{t}\rho\partial_{t}pd\tilde{t}+\int_{t}^{3}\partial^{2}_{t}\rho
pd\tilde{t}.$

For the function $f$ we have the following estimate

(7.4) $\|f\|_{L^{2}(G)}\leq C(\|Mp\|_{L^{2}(G)}+\|p\|_{L^{2}(G_{1})}).$

Finally

$M\tilde{p}=-\rho f-\partial^{2}_{t}\rho(\int_{t}^{3}\rho
pd\tilde{t}-\int_{0}^{3}\rho pd\tilde{t})+2\partial_{t}\rho p.$

Applying the estimate (7.3) to the above equation and using (7.4) we obtain

$\|\rho p\|_{L^{2}(G)}\leq C(\|\int_{0}^{3}\rho
pd\tilde{t}\|_{L^{2}(0,T)}+\|Mp\|_{L^{2}(G)}+\|K\|_{L^{2}(G_{1})}).$

This estimate implies

(7.5) $\|p(0,\cdot)\|_{L^{2}(0,3)}\leq C\|p\|_{L^{2}(G)}\leq
C(\|\int_{0}^{3}\rho
pd\tilde{t}\|_{L^{2}(0,T)}+\|Mp\|_{L^{2}(G)}+\|p\|_{L^{2}(G_{1})}).$

This estimate and uniqueness compactness argument imply (7.2). Indeed suppose
that (7.5) fails. Then there exist a sequence $\\{p_{n}\\}$ such that
$\|p_{n}(0,\cdot)\|_{L^{2}(0,3)}=1$ and

(7.6) $\|p_{n}(0,\cdot)\|_{L^{2}(0,3)}\geq
n(\|Mp_{n}\|_{L^{2}(G)}+\|p_{n}\|_{L^{2}(G_{1})}).$

By (7.5) and Holgrem’s uniqueness theorem one can take a converging
subsequence, which we denote again $p_{n}$ such that
$p_{n}(0,\cdot)=\sum_{n=0}^{\infty}b_{n,k}e_{k}(t)\rightarrow 0$ weakly in
$L^{2}(0,3)$ and $e_{k}(t)=\sin(\frac{k\pi t}{T}).$ In particularly

(7.7) $b_{n,k}\rightarrow 0\quad\mbox{as}\quad n\rightarrow+\infty.$

By (7.5) we have

(7.8) $\|p_{n}(0,\cdot)\|_{L^{2}(0,3)}\leq C\|p_{n}\|_{L^{2}(G)}\leq
C\|\int_{0}^{3}\rho p_{n}dt\|_{L^{2}(0,T)}+o(1)\quad\mbox{as}\quad
n\rightarrow\infty.$

Since the function $p_{n}$ is given by formula
$p_{n}(\tau,t)=\sum_{k=1}^{\infty}e^{ik^{2}\pi^{2}\tau/T^{2}}b_{n,k}e_{k}(t).$
Using this formula we obtain

$\int_{0}^{3}p_{n}(t,\tau)\rho(t)dt=\sum_{k=1}^{\infty}e^{ik^{2}\pi^{2}\tau/T^{2}}b_{n,k}\int_{0}^{3}\rho(t)e_{k}(t)dt.$

On the other hand

$\|\int_{0}^{3}\rho(t)p_{n}(\tau,t)dt\|^{2}_{L^{2}(0,T)}\leq
C\sum_{k=1}^{\infty}(b_{n,k}\int_{0}^{3}\rho(t)e_{k}(t)dt)^{2}.$

Taking into account that $\int_{0}^{3}\rho(t)e_{k}(t)dt=O(\frac{1}{k})$ and
using (7.7) we obtain that

$\|\int_{0}^{3}\rho(t)p_{n}(\tau,t)dt\|^{2}_{L^{2}(0,T)}\leq
C\sum_{k=1}^{\infty}(b_{n,k}\int_{0}^{3}\rho(t)e_{k}(t)dt)^{2}\rightarrow
0\quad\mbox{as}\quad n\rightarrow\infty$

we obtain the contradiction to (7.8). $\blacksquare$

## References

  * [1] L. Baudouin, and A. Mercado, An inverse problem for Schrodinger equations with discontinuous main coefficient, Appl. Anal., 87, (2008), 1145-1165.
  * [2] L. Baudouin and J-P Puel, Corrigendum Uniqueness and stability in an inverse problem for the Schrodinger equation, Inverse Problems, 23, (2008), no. 3, 1327-1328.
  * [3] L. Baudouin and J-P Puel, Uniqueness and stability in an inverse problem for the Schrodinger equation, Inverse Problems, 18 (2002), no. 6, 1537-1554.
  * [4] R. Bosi, Y. Kurylev and M. Lassas, Stability of the unique continuation for the wave operator via Tataru inequality and applications, Journal of Differential Equations, 260, (2016) 6451-6492.
  * [5] R. Bosi, Y. Kurylev and M. Lassas, Stability of the unique continuation for the wave operator via Tataru inequality and applications: the local case, Journal d’Analyse Mathématique, 260, (2018) 157-199.
  * [6] A.L. Bukhgeim and M.V.Klibanov, Global uniqueness of class of multidimentional inverse problems, Soviet Math. Dokl. 24 (1981) 244-247.
  * [7] L. Hörmander, Linear Partial Differential Operators, Springer-Verlag, Berlin, 1963.
  * [8] O.Yu. Imanuvilov, Controllability of parabolic equations, Math. Sb., 186, (1995), 879-900.
  * [9] P. Martina, L. Rosier and P. Rouchona, Controllability of the 1D Schrödinger equation using flatness, Automatica, 91, (2018), 208-216.
  * [10] A. Mercado, A. Osses and L. Rosier, Inverse problems for the Schrödinger equation via Carleman inequalities with degenerate weights, Inverse Problems, 24 (2008), no. 1, 015017.
  * [11] S. Mizohata Unicité du prolongement des solutions pour quelques opérateurs différentiels paraboliques, Mem. College Sci. Univ. Kyoto A 31 (1958), 219-239.
  * [12] X. Huang, Y. Kian, E. Soccorsi and M. Yamamoto Carleman estimate for the Schrödinger equation and application to magnetic inverse problems, Journal of Mathematical Analysis and Applications, 474, Issue 1, (2019), 116-142.
  * [13] L. Robbiano, Théorème d’unicité adapté au contrôle des solutions des problèmes hyperboliques, Comm. Partial Differential Equations, 16 (1991), no. 4-5, 789-800.
  * [14] J-C Saut and B. Scheurer, Unique continuation for some evolution equations, J. Diff. Eqns 66 (1987), 118-139.
  * [15] D. Tataru, Unique continuation for solutions to PDE’s: between Hörmander’s theorem and Holmgrem theorem, Comm. Partial Differential Equations, 20, (1995), no 5-6, 855-884.
  * [16] E.C. Titchmarsh, The zeros of certain integral functions, Proc. London Math. Soc. 25 (1926) 283-302.
  * [17] G. Yuan and M. Yamamoto, Carleman estimates for the Schrödinger equation and applications to an inverse problem and an observability inequality, Chin. Ann. Math., 31B(4), (2010), 555-578.
  * [18] E. Zuazua Remarks on the controllability of the Schrödinger equation, Centre de Recherches Mathématiques CRM Proceedings and Lecture Notes.


# Dialogue with Robots: Proposals for Broadening Participation
and Research in the SLIVAR Community

Casey Kennington
Boise State
University Malihe Alikhani∗
Northeastern
University Heather Pon-Barry∗
Mount Holyoke
College Katherine Atwell
University of
Pittsburgh Yonatan Bisk
Carnegie Mellon
University Daniel Fried
Carnegie Mellon
University Felix Gervits
US Army Zhao Han
University of
South Florida Mert Inan
Northeastern
University Michael Johnston
Amazon Raj Korpan
Hunter College,
City University of
New York Diane Litman
University of
Pittsburgh Matthew Marge
DARPA Cynthia Matuszek
University of Maryland,
Baltimore County Ross Mead
Semio Shiwali Mohan
Xerox PARC Raymond Mooney
University of Texas,
Austin Natalie Parde
University of Illinois,
Chicago Jivko Sinapov
Tufts
University Angela Stewart
University of
Pittsburgh Matthew Stone
Rutgers
UniversityStefanie Tellex
Brown
University Tom Williams
Colorado
School of Mines  Authors Kennington, Alikhani, and Pon-Barry proposed and
organized the workshop. The remaining authors are listed alphabetically.
Corresponding author:
<EMAIL_ADDRESS>
###### Abstract

The ability to interact with machines using natural human language is becoming
not just commonplace, but expected. The next step is not just text interfaces,
but speech interfaces and not just with computers, but with all machines
including robots. In this paper, we chronicle the recent history of this
growing field of spoken dialogue with robots and offer the community three
proposals, the first focused on education, the second on benchmarks, and the
third on the modeling of language when it comes to spoken interaction with
robots. The three proposals should act as white papers for any researcher to
take and build upon.

##  1 Introduction

Daily use of technology that allows humans to interact with machines using
natural language by the general population is accelerating. The field of
Natural Language Processing (nlp) was transformed by transformers Vaswani et
al. (2017) since BERT Devlin et al. (2018), and more recent scaling of
transformer language models like ChatGPT has brought what used to be models
only known in the nlp research community to consumer use where generating
language is a valuable outcome, for example in generating drafts of emails or
company policies. However, language use is not just confined to text; much of
human interaction is still face-to-face where spoken language is used to refer
to objects and events in the world. For example, cooking, yard work, travel,
construction and many other domains of every day life require humans to
interact with physical objects that large language models like ChatGPT can
talk about generally, but not refer to specifically. Robots, in contrast,
exist in a physical embodiment and are being used in cooking, yard work,
travel, and construction to assist humans, automate mundane tasks such as
vacuuming, perform tasks that are dangerous to humans, and act as socially
aware and interactive agents for people who are lonely.

Kennington (2021b) explained that a newly forming _Special Interest Group for
Spoken Language Interaction with Virtual Agents and Robots_ (SLIVAR) aims to
bring the broad areas of spoken dialogue systems, robotics, and human-robot
interaction together with the goal of empowering people to communicate with
robots the way that humans largely communicate with each other: natural human
language, particularly spoken dialogue. A _robot_ is an actuated mechanism
programmable in two or more axes with a degree of autonomy, moving within its
environment, to perform intended tasks that serve the needs of a user Eskenazi
and Zhao (2020).111This, as well as other arguments and definitions that
follow are unceremoniously lifted from Roger Moore’s excellent keynote at RO-
MAN 2020 (https://tinyurl.com/rmoreroman),222See also this Twitter thread:
https://twitter.com/BLeichtmann/status/1314080122169970688 A _spoken dialogue
system_ (sds) is a an automated system that is able to converse with a human
with voice.333Here, the focus is on robots and speech, though in many cases
the arguments made here also extend to virtual agents and communication using
a text chat medium.

Recent workshops and events have facilitated the convergence of dialogue
interaction with robots. Two U.S. National Science Foundation (NSF) workshops
were convened in October, 2019. First, the _Future Directions Workshop, Toward
User-Oriented Agents: Research directions and Challenges_ Eskenazi and Zhao
(2020) which focused on the role of intelligent agents and how to make them
more user-oriented. The participants of the workshop identified broad areas
and themes for future directions including applications, infrastructure,
dynamic views of user-agent interaction, and made several recommendations in
building low-cost dialogue systems, multimodal, grounded, and situated
interaction, robust and flexible dialogue management, and intelligent agents
as good actors. The second workshop, _Spoken Language Interaction with Robots_
Marge et al. (2022), focused on speech and the complexities thereof when
robots are involved.

A Dagstuhl (Germany) Seminar also convened in January 2020 on the topic of
SLIVAR.444Dagstuhl ID 20021
https://drops.dagstuhl.de/opus/volltexte/2020/12400/ This resulted in
organization of other events such as a special session on robots and dialogue
(RoboDial 2.0) at the SIGDIAL 2020 conference,555https://robodial.github.io a
workshop on natural language generation at the HRI 2020
conference,666https://hbuschme.github.io/nlg-hri-workshop-2020/ and a workshop
ROBOTDIAL at the IJCAI 2020
conference.777http://sap.ist.i.kyoto-u.ac.jp/ijcai2020/robotdial/ The primary
goal of the Dagstuhl Seminar was to provide discussion and establish a
community. Discussions revolved around ethics, usability, scenarios for human-
agent / human-robot groups, evaluation, architectures, and situated language
understanding. Other recent, related events include a AAAI Symposium on
Natural Communication for Human-Robot Collaboration
(2017),888https://www.ttic.edu/nchrc/ and RoboNLP workshops including spatial
language understanding.999See links to prior workshops https://splu-
robonlp.github.io/

More recently, on 7-8 April 2023, a workshop funded by NSF convened to discuss
ten important topics that follow directly from the prior workshops and ongoing
research trends:

  * •

Research platforms for robots+spoken dialogue systems

  * •

Collection of situated and multimodal corpora

  * •

Multimodality: which modalities (beyond vision) are necessary, and how to
capture and leverage them

  * •

Natural language understanding in co-located, situated, multimodal systems

  * •

Natural language generation in co-located, situated, multimodal systems

  * •

Representation of the state of the world, robot, and human interlocutor, and
building of common ground

  * •

Dialogue with robots: clarification, turn-taking, handling ambiguity

  * •

Identifying ethical issues surrounding human-robot dialogue

  * •

What should a robotics researcher know when they want to add the ability to
talk to their robots, and visa-versa?

  * •

Identify and plan tasks and benchmarks

Furthermore, the workshop included keynote addresses by Yonatan Bisk (CMU) and
Stefanie Tellex (Brown University) on the broad topics of robots, dialogue,
and vision about where the field is headed given the current state of
technology advancement.

The purpose of this paper is to report on that workshop in the form of 3
overarching proposals with the goal to build the SLIVAR community by (1)
providing educational resources, (2) establishing benchmarks and challenges,
and (3) integrating large language models effectively with robots while
meeting important requirements for natural interaction. In the subsequent
sections below, each proposal is explained. We then conclude.

##  2 Proposal 1: Educational Resources

Robotics, natural language processing (nlp), spoken dialog systems (sds), and
human-robot interaction (hri) are separate fields, each requiring a lot of
educational preparation before a student is ready to perform research ranging
from mechanical and electrical engineering to computer and data science as
well as machine learning. There are many technical skills for each, ranging
from understanding hardware and software to social interactions. The breadth
of topics is daunting, yet ample depth in each area is necessary for effective
research.

The depth that each student needs in each area is partially dependent on the
kinds of research that the student will engage in. If the student is more
interested, for example, in language/symbol grounding, then more coursework
that enables a deeper understanding of semantic theories, as well as in-depth
understanding of nlp and computer vision is crucial. Whereas, a student more
interested in the social aspects of interaction between humans and robots may
not need to take courses where they learn how to build robots or worry about
navigation, though they will need to be able to build systems and work with
various sensor information.

###  2.1 Proposal

We propose the creation of a central resource that allows members of the
community to share syllabi, course content, lecture slides, example code,
assignments, assessment methods, platforms, and research tools used in
coursework. Much of the coursework in certain areas like Math and Computer
Science is likely already covered at many institutions of higher education,
but courses in more specific areas like nlp, sds, hri, and robotics might need
supplementation for students where those courses are not available.

####  2.1.1 Existing Work

Courses that have public content exist that could be adopted by others who
seek to incorporate relevant coursework to their curricula, including Math,
Computer Science, and Machine Learning courses. More specific to the area of
interest here are existing courses (linked in the repository above) like
Grounding Natural Language, Talking to Robots, Multimodal Machine Learning,
Grounded NLP, NLP with a bent on computational semantics including symbol
grounding Kennington (2021a), Probabilistic Robots for Human-Robot
Interaction, Language and Vision, and Dialogue Systems. Some robot platforms
such as GoPiGo101010https://gopigo.io and
DuckieTown111111https://www.duckietown.org offer educational resources with
their robot platforms.

Below we outline a proposed sequence of courses for a student to take, and
offer some ideas as to how this effort could be organized.

####  2.1.2 Coursework

Below is a sample of known courses that could be offered in an undergraduate
curriculum that would prepare a student to do research at the intersection of
robotics, sds, and hri. Each bullet point can either be an entire course, or
crucial topics to be taught within a course.

  * •

Math

    * –

Calculus

    * –

Linear Algebra

    * –

Statistics

    * –

Probability and Information Theory

    * –

Differential Equations

  * •

Computer Science

    * –

Data Structures

    * –

Algorithms

    * –

Distributed Systems

  * •

Robotics

    * –

Morphology & Action Planning

    * –

Localization & Navigation

  * •

Data Science and Machine Learning

    * –

Data Science (data analysis, munging, visualization)

    * –

Machine Learning

    * –

Deep Learning

  * •

Artificial Intelligence

    * –

Natural Language Processing (large language models)

    * –

Computer Vision and Grounding (multimodal machine learning)

    * –

Spoken Dialogue Systems (incremental, multimodal)

  * •

Human-Robot Interaction

    * –

Human-Robot Interaction

    * –

Scientific Methods (human subjects, evaluation)

  * •

Ethics and Society

    * –

Measuring Harm in Human-Robot Dialogue

    * –

Bias

    * –

Efficiency and Environmental Cost

    * –

Accessibility

###  2.2 Sharing Resources

We propose the creation of a repository of resources for education of students
at this crucial, yet complex intersection between hri, sds, and robotics. To
make it easy to add and share resources, we propose using a GitHub repository
where the main landing page has links to the resources. Contributors can use
git functionality like pull requests to add information. We have started the
https://github.com/bsu-slim/slivar-resources/ repository to serve this
purpose. We also propose some kind of forum for discussion about resources
such as Discord that anyone can join.121212Contact the corresponding author if
you want to join the Discord server.

##  3 Proposal 2: Benchmarks & Challenges

Benchmarks and challenges help research move forward by giving a community a
common way to compare their work. For example, the GLUE benchmark is a well-
known suite of tasks in nlp that has been widely used; the 2018 paper has over
4,000 citations Wang et al. (2018). However, benchmarks have their drawbacks.
Bowman and Dahl (2021) explained how GLUE and other language understanding
benchmarks are not guaranteed to properly test models (indeed, even models
that perform well on benchmarks fail at simple language phenomena). According
to the authors, benchmarks should:

  1. 1.

Offer a valid test of the relevant language phenomena

  2. 2.

Be built around consistently labeled data

  3. 3.

Offer adequate statistical power

  4. 4.

Disincentivize the use of systems with potentially harmful biases

These criteria act as useful guides for working towards a benchmark for
dialogue on robots. We add here several requirements for interactive dialogue
with robots that a benchmark should fulfill:

##### Multimodal

the data needs to have speech (or text), but also information about the
physical situation including sensor information from sensors such as one or
more cameras, as well as robot state information.

##### Co-located

the data needs to contain dialogue that talks about objects in a shared
environment that the robot can manipulate (e.g., identify, grasp, or move).

##### High-stakes Dialogue

the task needs to require that the user and robot work together to accomplish
something in a scenario where the stakes are sufficiently high so as to
require important dialogue artifacts such as clarification requests,
interruptions, building common ground, and providing explanations of
decisions. For example, a fast-paced or gamified scenario increases stakes
above simple, goalless verbal interaction.

##### User-centered

the user needs to feel like they are not just giving the robot commands;
rather, they see the benefit for both the user and the robot’s involvement.

##### Community-agnostic

the benchmark should use different robot platforms, be affordable, work in
both a virtual (including the possibility of VR headset integrations) and real
environments.

Other properties of the benchmark could include the ability to evaluate
scenarios where multiple robots and/or people are participating, learning
during interaction, and focus is put on underrepresented communities.

###  3.1 Existing Work

Figure 1:  Example of the Alexa Arena benchmark: user utterances and simulated
robot responses, from Gao et al. (2023b)

Benchmarks are not new in sds research. The Dialogue State Tracking Challenge,
for example, ran for several years Williams and Park (2016); Kim et al.
(2017). There were multiple tasks over the years including finding bus
timetable information and restaurant search. However, the goal of dialogue
state tracking is only part of what makes a dialogue system work, and the only
modality was transcribed speech (i.e., text).

Closer to meeting the above requirements is the ALFRED (Action Learning from
Realistic Environments and Directives) benchmark for learning a mapping from
natural language instructions and egocentric vision to sequences of actions
for household tasks such as navigating and moving objects Shridhar et al.
(2019). The world is represented virtually, but the agent can freely move
around in a complex environment. Similar in some ways to ALFRED, the TEACh
benchmark Padmakumar et al. (2021) uses a _commander_ and _follower_ paradigm
where human users take the commander role and instruct the robot follower in a
household setting performing tasks of varying complexity. In both challenges,
the primary interaction is through a text-input chat interface, clarification
requests are part of the TEACh dataset.

Similar but closer to the above requirements is Alexa Arena Gao et al.
(2023b), see Figure 1. The platform is a new user-centric “Embodied" AI
platform which focuses on building generalizable agents through reasoning and
procedural learning, a benchmark of 3k unique tasks (e.g., move, lift) and 46k
human-annotated instructions and dialogues. The platform includes a web-based
user interface for easy integration and real-time user interaction (see Figure
1). Simple “verbal joystick” actions are possible such as _turn left_ or _move
forward_ , but more complex, relational referring expressions and interactive
dialogue are also required to complete tasks. This dataset is interesting to
researchers because there are many breakdowns in communication that require
sds, the setting is complex enough to require complex language, yet simple
enough to not be completely open domain. Robots are not required as the
setting is virtual.

These benchmarks offer a valuable starting point for a dialogue-with-robots
benchmark, but they do not fulfill all of the requirements listed above. Below
we sketch a proposal towards the goal of a benchmark that does.

###  3.2 Proposal

The work in developing the benchmark will have three phases: (1) requirements
gathering (what do researchers need in a benchmark?), (2) developing technical
infrastructure including virtual platforms, and (3) establishing the first
cohort of participants in a challenge that uses the benchmark.

##### Requirements Gathering

Research is driven by research questions. The kinds of robots and tasks that
are interesting to particular researchers dictates the kinds of robots and
platforms that they adopt. Some are more focused on robot navigation, others
on language/symbol grounding, still others put more focus on the social
aspects of interaction. In any case, the benchmark should include aspects of
safety and bias.

The proposed work will involve requirements gathering to determine what
platforms and tasks researchers are using, and what kind of research they are
focused on. Kennington (2021b) noted that the fields of robotics, sds, and hri
have different, yet complementary goals (see Figure 2) to enable robots and
humans to work together more naturally and effectively, which offers a general
starting point. Though requirements for each researcher will be different, the
design of the infrastructure can have some degree of navigation, dialogue
interaction including social aspects, and interaction with objects in the
environment for language/symbol grounding (i.e., connecting language to the
physical world).

Figure 2: The fields of robotics, hri, and sds have different, yet
complementary goals, from Kennington (2021b).

##### Building Infrastructure

The challenge should include a virtual (e.g., similar to Alexa Arena) and
real-world version of the the same task. The virtual version will enable more
researchers access to the benchmark without expensive hardware. The real-world
version will serve as proving grounds that the benchmark will work in a real
robot.

Similar in some ways to DuckieTown which offers a robot platform and
environment (though DuckieTown is focused on navigation), the platform should
make the environment available to the
community.131313https://www.duckietown.org Our goal is to allow as many
objects to be easily obtained (e.g., through 3D printing) at low cost and to
allow any robot platform within certain size dimensions, and we will maintain
a public listing of robots being used for the challenge.

##### Challenge

We suggest that there be an initial challenge that helps guide a cohort of
teams through the benchmark. This will serve as a test for the benchmark as
the teams will receive materials without cost and they will be able to help
work through inevitable technical difficulties. This approach follows recent
NSF-backed research infrastructure to build a modular social
robot.141414https://www.nsf.gov/awardsearch/showAward?AWD_ID=2235042&HistoricalAwards=false

The first challenge should begin with a cohort of 5 teams to beta-test the
challenge with real researchers. The teams will be chosen based on their
experience in the field of sds and hri and their proposal of the method they
will use to fulfill the challenge. Each team will be issued the virtual
environment, a robot (yet to be determined; we look to GoPiGo as a possible
starting point because it is cheap and simple, yet can sense the environment
and move around in it), and the environment. There will be a setup, training,
and evaluation phase of the first challenge. Teams will be able to report the
results of their work in a culminating workshop that will serve as a report
for the first cohort and an open invitation for others interested in using the
benchmark.

##  4 Proposal 3: Large Language Models and Robots

_Large Language Models_ (llm) have been a mainstay of nlp since attention
neural mechanisms used in a transformer Vaswani et al. (2017) and the BERT
transformer were introduced and systematically used for language tasks Devlin
et al. (2018). llms like BERT use a pre-train, fine-tune paradigm: during pre-
training, the training regime tasks the model with guessing _masked_ words
within a sequence of words (i.e., they can be in a lexical context or a next
word/sentence to be predicted). This training regime enables the model to
learn a distributional approximation of meanings of words and sentences from
the text they are trained on. During the fine-tuning phase, the models can be
arranged in such a way that they can take in text as input and be tuned to a
specific task like machine translation, sentiment classification, or topic
modeling. llms effectively allow one to train using a straightforward training
regime where only text is required, then tune to specific tasks where only
little data might be available.

Can llms be used for dialogue with robots? A social robot could straight-
forwardly be hooked up to a speech recognizer which transcribes speech, then
that transcription is given to a language model which produces a response.
This is fine for social tasks where the goal is chatting, but the limits of
training only on text, and the ability for llms to only take text as input are
quickly apparent when one wants the robot to refer to objects in a physical
environment—llms trained only on text have no notion of the physical world
unless the physical world is somehow represented in text symbols as input to
the llm. Being trained on text only also hasn’t prevented llms from potential
ethical issues such as generating abusive, stereotyped or biased language, or
fabricating information that is nonfactual Bender et al. (2021). Moreover, the
SLIVAR workshop on the Ethics of Language-Capable Robots identified a range of
key ethical challenges facing roboticists developing language-capable robots,
especially those hoping to use LLMs as part of their robotics solutions
Williams et al. (2023). These include key challenges surrounding trust and
influence in and by language capable-robots; identity performance by language-
capable robots; and privacy concerns surrounding what is heard and/or
understood by language-capable robots. Mitigating these shortcomings in llms
is an ongoing research trend by the broader community Williams et al. (2024a),
but it is something that will need to be taken into account and addressed in
the setting of human-robot dialogue.

A more recent trend in llm advancement has been to enrich language models with
visual information, known as _multi-modal language models_ (mm-llm). For
example, the VilBERT model uses BERT coupled with a visual representation of
images Lu et al. (2019). Others have proposed language and vision tasks to
evaluate such models, such as Visual Dialogue Das et al. (2019) and Visual
Question Answering Antol et al. (2015). More recent work has explored how
different training regimes and architectures work better for vision and
language tasks Dou et al. (2022). More recent “embodied” models allow for
incorporating modalities beyond text (and vision) into the language model, but
are yet to be tested within human-robot interaction studies Driess et al.
(2023). More directly for robots, Gao et al. (2023a) showed how vision-
language llms can be used for robotic manipulation of objects. Finally, the
One-Peace model Wang et al. (2023) can fold any input modality into the self-
attention steps of a llm, tested on vision and audio, but theoretically could
be used with other modalities important to robots such as internal state
representations. The research cited here represents steps in the right
direction because they open the visual world up to llms which is crucial for
robots.

Beyond spoken interaction or visual language tasks, llms are being used for
any sequential task or mapping from language to some kind of logical form
(which robots are often represented by). For example, Schick et al. (2023)
introduced the _Toolformer_ model that can map from language to ‘tool‘ access
(i.e., API calls) for example, if numbers are detected in a sentence, the
Toolformer would call a calculator tool, or if a word in a foreign language is
detected then the Toolformer calls a translation tool. This is useful for
robotic platforms where if someone utters an instruction (e.g., _pick up the
book_), a llm like the Toolformer could map that to the tool (e.g., arm
movement) that is necessary to complete the task.

Zhang et al. (2023) used language models to reason about a situation in order
to build cooperative embodied (in their case, virtual) agents. For example,
one agent can inform another agent about objects in a room and the other agent
can take actions on those objects. The inform step is natural language
generated by a language model. Min et al. (2021) introduced _FILM_ that maps
language sequences to action sequence subtasks (e.g., _(Pan, PickUp)_) that is
then combined with visual input, then the agent can actually take actions.
Another model, ProgPrompt, can map from a natural language prompt to python-
like programming language instructions that can then be directly executed to
control a robot Singh et al. (2023). Language models have even been used to
generation emotional behaviors on a robot platform Mishra et al. (2023).

Taken together, there are two ways that researchers make use of language
models:

  * •

Convert the physical world into tokens, then fine-tune a language model on the
input/output at the level of symbolic tokens (e.g., FILM and ProgPrompt)

  * •

Represent the physical world a fine-grained level (usually in vector form) and
fuse within the model itself (e.g., One-Peace)

llms are clearly a model of choice for a growing number of tasks for robots
ranging from observing the visual world, interacting with human users, and
generating emotional displays. llms are well suited for sequential tasks.
However, llms do need a full input sequence to map to a desired output label
or sequence. llms do not work _incrementally_ in that they cannot simply
accept a single token at a time and produce output tokens monotonically. As a
result, they are not inherently temporal in nature, which is important in
real-time interaction. However, see recent work in making transformer language
models work incrementally (i.e., word-by-word) Kahardipraja et al. (2021).

###  4.1 Gaps

##### Closed LLMs

llms like ChatGPT and GPT-4 are impressive and useful, but it is not obvious
how to hook them up to live robots without encoding everything about an
ongoing interaction between a human and robot symbolically using text and
coming up with a textual response that is not only what the robot should say,
but also what the robot should do. Additionally, they have security and bias
issues. Moreover, the latency for getting a response from a public llm is
usually too long for co-located interaction to be effective. Finally, as
Rogers (2023) points out, closed llms (_closed_ meaning we don’t know what the
architectures are or what data they were trained on) make bad baselines to
compare against for ongoing research because much about them are not
sufficiently explained as to be reproducible.

##### Model Size

llms are called _large_ for a reason. BERT has 110 million parameters Devlin
et al. (2018) and RoBERTa has 125 million parameters Liu et al. (2019). These
are more manageable than, say, ChatGPT (estimated around 175 billion
parameters), but the compute resources required for pre-training can be out of
reach to many researchers. One promising model for research involving robots
and language is Palm-E, a model that incorporates multiple modalities beyond
just text, and which is based on the PALM model Chowdhery et al. (2022)
required over 6,000 TPUv3 chips (a cost of roughly $20,000 per hour) to train.
These kinds of models that require excessive compute resources limit the kinds
of fundamental research that is needed for human-robot dialogue. While fine-
tuning such models on specific tasks is a common practice, there are many open
research questions in the domain of human-robot dialogue that will require
custom pre-training, but work needs to be done to make the models small, yet
effective.

##### Data Size

The amount of data needed to train mm-llms is, as the number of model
parameters, a lot. Text data, and even vision data which are currently used to
train mm-llms are much easier to come by than data recording robot behaviors
and states, as well as interactions between robots and humans. The kinds of
systems required are complex, making data collection challenging, resulting in
very little effective data. llms tend to be general purpose models, but what
is needed in dialogue with robots is specific purpose: robots need to interact
with humans using speech, but also attend to objects using other robot
modalities such as pointing or grasping. Without effective learning on small
amounts of data, it will be difficult to advance important research in this
area.

Relatedly, many llms are trained end-to-end with clear text input, text output
tasks. For example, for question-answering there is an input question and an
output answer. For dialogue, there is an input utterance and an output
response. Can end-to-end modeling be used for human-robot interaction where
dialogue is present? It is possible, but data is required—a lot of data. As
noted above, it is also not clear how to “hook up" the modalities of the robot
(cameras, bump sensors, depth sensors, internal states) and the ongoing
interaction to the end-to-end mm-llm. At the moment, it is more prudent to
construct systems that are modular where llms are present, perhaps to
understand utterances and generate responses (explored below).

###  4.2 Open Questions

To address the above gaps, we propose research on small language models that
require fewer compute resources to train and run in real-time, but also
require less data to be effective. This goal is of growing concern in the
broader community given new smaller language models like BabyBERTa Huebner et
al. (2021), distillation Kim et al. (2022); Hsieh et al. (2023), and a recent
challenge for BabyLMs that require less training data Warstadt et al. (2023).
We echo the need for these efforts and encourage research in this direction.

More work needs to be done to see how llms, particularly mm-llms can work
together with robots in a dialogue setting. llms are already being used for
language understanding tasks in dialogue interaction with robots, and they are
already being used to generate responses.

Figure 3: Example of the ISABEL bot designed as part of the Amazon Alexa
Taskbot Challenge 2023. ISABEL is the first multimodal bot that can interact
with signers around the world add the role of LLM Malihe Sicilia et al.
(2023).

One open question is how the world and unfolding interaction can (or should)
be encoded for the robot to effectively interact. llms require text as input.
Should models of the world be encoded in symbolic text, and if so, how? What
is the “vocabulary" of representing the world that would be effective for
llms? Such an approach could be effective because it would not require costly
remodelling of the llm to include new modalities, though researchers would
need to find new ways to textually represent newly added modalities (e.g., a
bump sensor). Williams et al. (2024b) called llms that are used as stand-ins
for modules ’scarecrows’ in that they are used as black-box modules to enable
full-pipeline solutions. For example, using a llm for modeling what a human
might say to the robot is a scarecrow. These kinds of uses of llms have some
theoretical implications, for example that using a text-only llm to make
decisions about all aspects of an embodied interaction makes a tacit
assumption that all knowledge and decisions are language-based (e.g.,
following the Sapir-Whorf hypothesis). Moreover, and more importantly, the
issues with bias that exist in pre-trained llms would negatively affect the
qualities of the interaction, particularly when the humans involved are part
of underrepresented groups, which are often the focus of human-robot
interaction research (e.g., autistic children or teenagers with mental
illnesses).

Alternatively, similar to the Palm-E and OnePeace models, information from
each modality can be embedded in a vector and incorporated into the model.
This has some theoretical implications in that it encodes knowledge and
decisions not only as text. It also makes modeling more difficult as new
modality need to somehow be represented vectorally and incorporated into the
model, and the model needs enough data to learn how to make use of the
information from the new modality, negating the effectiveness of some pre-
traind models.

One promising line of work that will affect human-robot dialogue is recent
work on _reinforcement learning with human feedback_ (rlhf). Reinforcement
learning in sds research is not new, spanning back to early efforts in
dialogue state tracking Schlangen (2003), and effective learning from humans
while interacting has long been a goal of sds and hri research, such as
learning from explicit examples. The explicit learning approaches used in rlhf
have direct implications for human-robot dialogue as robots are often tasked
with learning as they verbally interact with humans.

###  4.3 Proposal

We encourage work on language model research that (1) focuses on smaller, yet
effective LLMs (particularly MLLMs) that can work on a small piece of hardware
on a robot Sicilia and Alikhani (2022), (2) focuses on effective learning with
small amounts of data, (3) focuses on mitigating the biases found in LLMs
(e.g., Dana et al. (2023); Sicilia and Alikhani (2023)), (4) support
populations with diverse needs and capabilities (e.g., Yin et al. (2021);
Viegas et al. (2022); Inan et al. (2022); Tong et al. (2024)), and (5) support
users in achieving their goals and common ground with them.

Concerning bias, safety, and inclusivity, it is paramount to address these
aspects thoroughly within the context of LLM development. Bias mitigation
strategies are essential to counteract and reduce the negative impacts of
inherent biases present in LLMs, necessitating recent research that identifies
and corrects these biases effectively Dana et al. (2023); Weidinger et al.
(2023) this includes grounding our models in theoretical foundations of
machine learning that can help us learn from the long tail Sicilia and
Alikhani (2023), active learning approaches for data augmentation and training
Hassan and Alikhani (2023a) and employing cognitively motivated interventions
such as generating paraphrases and counter narratives Atwell et al. (2022);
Hassan and Alikhani (2023b). Safety measures must be integrated to ensure that
LLMs do not produce or propagate harmful, misleading, or inapproprmoreiate
content, thereby protecting users from potential risks associated with
automated language generation. Inclusivity requires the design and development
of LLMs that accommodate and support the varied needs and abilities of diverse
user populations Yin et al. (2021); Viegas et al. (2022); Inan et al. (2022);
Tong et al. (2024), ensuring accessibility and utility across different socio-
cultural and linguistic contexts. Figure3 shows one of the few multimodal
dialogue that can interact with signers. This is just one example of ways we
can be thinking about accommodating needs of our diverse community of users.
This holistic approach towards bias, safety, and inclusivity underscores the
importance of creating LLMs that are not only technologically sophisticated
but also ethically sound and socially responsible.

##  5 Conclusion

This paper synthesizes the discussions from the NSF-funded workshop in 2023 on
the topic of robots and dialogue. The three proposals above represent the
ongoing and future work that we urge the community to address in the near
future.

As coursework evolves to train new graduates that are ready for sds-hri
research, more fundamental research and practical applications will enable
people to be more productive and ready to interact with robots in the
workforce. As benchmarks for spoken dialogue with robots improve, models and
datasets will follow that build on those improvements. As large language
models become more multi-modal, process in real time, and learn from more than
just text as they interact, their usefulness as a model of language
understanding and generation with robots will increase.

We are optimistic that the field of spoken interaction with robots will
continue to grow and impact society in positive ways.

##### Acknoledgements

This workshop was funded by the National Science Foundation Award #2306113. We
want to thank those who participated in other ways, including Erin Walker
(University of Pittsburgh), Dan Bohus (Microsoft Research), and Joyce Chai
(University of Michigan). We want to thank the University of Pittsburgh for
hosting the event, and thanks to the administrative staff who helped organized
the workshop.

## References

  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  VQA: Visual question answering.  In _Proceedings of the IEEE International Conference on Computer Vision_.
  * Atwell et al. (2022) Katherine Atwell, Sabit Hassan, and Malihe Alikhani. 2022.  Appdia: A discourse-aware transformer-based style transfer model for offensive social media conversations.  _arXiv preprint arXiv:2209.08207_.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, pages 610–623, New York, NY, USA. Association for Computing Machinery.
  * Bowman and Dahl (2021) Samuel R Bowman and George Dahl. 2021.  What will it take to fix benchmarking in natural language understanding?  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4843–4855, Online. Association for Computational Linguistics.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  PaLM: Scaling language modeling with pathways.
  * Dana et al. (2023) Kristin J Dana, Clinton Andrews, Kostas Bekris, Jacob Feldman, Matthew Stone, Pernille Hemmer, Aaron Mazzeo, Hal Salzman, and Jingang Yi. 2023.  Socially cognizant robotics for a technology enhanced society.  _arXiv preprint arXiv:2310.18303_.
  * Das et al. (2019) Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Stefan Lee, Jose M F Moura, Devi Parikh, and Dhruv Batra. 2019.  Visual dialog.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 41(5).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of deep bidirectional transformers for language understanding.
  * Dou et al. (2022) Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. 2022.  An empirical study of training end-to-end vision-and-language transformers.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18166–18176. IEEE.
  * Driess et al. (2023) Danny Driess, Fei Xia, Mehdi S M Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. 2023.  PaLM-E: An embodied multimodal language model.
  * Eskenazi and Zhao (2020) Maxine Eskenazi and Tiancheng Zhao. 2020.  Report from the NSF future directions workshop, toward User-Oriented agents: Research directions and challenges.  _arXiv_.
  * Gao et al. (2023a) Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. 2023a.  Physically grounded Vision-Language models for robotic manipulation.
  * Gao et al. (2023b) Qiaozi Gao, Govind Thattai, Xiaofeng Gao, Suhaila Shakiah, Shreyas Pansare, Vasu Sharma, Gaurav Sukhatme, Hangjie Shi, Bofei Yang, Desheng Zheng, Lucy Hu, Karthika Arumugam, Shui Hu, Matthew Wen, Dinakar Guthy, Cadence Chung, Rohan Khanna, Osman Ipek, Leslie Ball, Kate Bland, Heather Rocker, Yadunandana Rao, Michael Johnston, Reza Ghanadan, Arindam Mandal, Dilek Hakkani Tur, and Prem Natarajan. 2023b.  Alexa arena: A User-Centric interactive platform for embodied AI.
  * Hassan and Alikhani (2023a) Sabit Hassan and Malihe Alikhani. 2023a.  D-calm: A dynamic clustering-based active learning approach for mitigating bias.  _arXiv e-prints_ , pages arXiv–2305.
  * Hassan and Alikhani (2023b) Sabit Hassan and Malihe Alikhani. 2023b.  Discgen: A framework for discourse-informed counterspeech generation.
  * Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023.  Distilling Step-by-Step! outperforming larger language models with less training data and smaller model sizes.
  * Huebner et al. (2021) Philip A Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021.  BabyBERTa: Learning more grammar with Small-Scale Child-Directed language.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 624–646, Online. Association for Computational Linguistics.
  * Inan et al. (2022) Mert Inan, Yang Zhong, Sabit Hassan, Lorna Quandt, and Malihe Alikhani. 2022.  Modeling intensification for sign language generation: A computational approach.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2897–2911.
  * Kahardipraja et al. (2021) Patrick Kahardipraja, Brielen Madureira, and David Schlangen. 2021.  Towards incremental transformers: An empirical analysis of transformer models for incremental NLU.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1178–1189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kennington (2021a) Casey Kennington. 2021a.  Natural language processing for computer scientists and data scientists at a large state university.  In _Proceedings of the Fifth Workshop on Teaching NLP_ , pages 115–124, Online. Association for Computational Linguistics.
  * Kennington (2021b) Casey Kennington. 2021b.  The state of SLIVAR: What’s next for robots, human-robot interaction, and (spoken) dialogue systems?
  * Kim et al. (2022) Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, et al. 2022.  Soda: Million-scale dialogue distillation with social commonsense contextualization.  _arXiv preprint arXiv:2212.10465_.
  * Kim et al. (2017) Seokhwan Kim, Luis Fernando D’Haro, Rafael E Banchs, Jason D Williams, and Matthew Henderson. 2017.  The fourth dialog state tracking challenge.  In _Dialogues with Social Robots_ , pages 435–449. Springer.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  ViLBERT: Pretraining Task-Agnostic visiolinguistic representations for Vision-and-Language tasks.
  * Marge et al. (2022) Matthew Marge, Carol Espy-Wilson, Nigel G Ward, Abeer Alwan, Yoav Artzi, Mohit Bansal, Gil Blankenship, Joyce Chai, Hal Daumé, Debadeepta Dey, Mary Harper, Thomas Howard, Casey Kennington, Ivana Kruijff-Korbayová, Dinesh Manocha, Cynthia Matuszek, Ross Mead, Raymond Mooney, Roger K Moore, Mari Ostendorf, Heather Pon-Barry, Alexander I Rudnicky, Matthias Scheutz, Robert St Amant, Tong Sun, Stefanie Tellex, David Traum, and Zhou Yu. 2022.  Spoken language interaction with robots: Recommendations for future research.  _Comput. Speech Lang._ , 71:101255.
  * Min et al. (2021) So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. 2021.  FILM: Following instructions in language with modular methods.
  * Mishra et al. (2023) Chinmaya Mishra, Rinus Verdonschot, Peter Hagoort, and Gabriel Skantze. 2023.  Real-time emotion generation in human-robot dialogue using large language models.  _Front Robot AI_ , 10:1271610.
  * Padmakumar et al. (2021) Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramithu, Gokhan Tur, and Dilek Hakkani-Tur. 2021.  TEACh: Task-driven embodied agents that chat.
  * Rogers (2023) Anna Rogers. 2023.  Closed AI models make bad baselines.  https://hackingsemantics.xyz/2023/closed-baselines/.  Accessed: 2023-4-27.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.
  * Schlangen (2003) David Schlangen. 2003.  A coherence-based approach to the interpretation of non-sentential utterances in dialogue.
  * Shridhar et al. (2019) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2019.  ALFRED: A benchmark for interpreting grounded instructions for everyday tasks.
  * Sicilia and Alikhani (2022) Anthony Sicilia and Malihe Alikhani. 2022.  Leather: A framework for learning to generate human-like text in dialogue.  _arXiv preprint arXiv:2210.07777_.
  * Sicilia and Alikhani (2023) Anthony Sicilia and Malihe Alikhani. 2023.  Learning to generate equitable text in dialogue from biased training data.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2898–2917.
  * Sicilia et al. (2023) Anthony Sicilia, Yuya Asano, Katherine Atwell, Qi Cheng, Dipunj Gupta, Sabit Hassan, Mert Inan, Jennifer Nwogu, Paras Sharma, and Malihe Alikhani. 2023.  Isabel: An inclusive and collaborative task-oriented dialogue system.  In _Alexa Prize TaskBot Challenge 2 Proceedings_.
  * Singh et al. (2023) Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023.  ProgPrompt: Generating situated robot task plans using large language models.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 11523–11530.
  * Tong et al. (2024) Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024\.  Eyes wide shut? exploring the visual shortcomings of multimodal llms.  _arXiv preprint arXiv:2401.06209_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Adv. Neural Inf. Process. Syst._ , 30.
  * Viegas et al. (2022) Carla Viegas, Mert Inan, Lorna Quandt, and Malihe Alikhani. 2022.  Including facial expressions in contextual embeddings for sign language generation.  _arXiv preprint arXiv:2202.05383_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355.
  * Wang et al. (2023) Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. 2023.  ONE-PEACE: Exploring one general representation model toward unlimited modalities.  _arXiv preprint arXiv:2305. 11172_.
  * Warstadt et al. (2023) Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Gotlieb Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Adina Williams, Bhargavi Paranjabe, Tal Linzen, and Ryan Cotterell. 2023.  Findings of the 2023 BabyLM Challenge: Sample-efficient pretraining on developmentally plausible corpora.  In _Proceedings of the 2023 BabyLM Challenge_. Association for Computational Linguistics (ACL).
  * Weidinger et al. (2023) Laura Weidinger, Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Anne Hendricks, Juan Mateos-Garcia, Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, et al. 2023.  Sociotechnical safety evaluation of generative ai systems.  _arXiv preprint arXiv:2310.11986_.
  * Williams and Park (2016) Jason D Williams and Menlo Park. 2016.  The dialog state tracking challenge series : A review.  _Dialogue & Discourse_, 7(3):4–33.
  * Williams et al. (2023) Tom Williams, Cynthia Matuszek, Kristiina Jokinen, Raj Korpan, James Pustejovsky, and Brian Scassellati. 2023.  Voice in the machine: Ethical considerations for language-capable robots.  _Communications of the ACM_ , 66(8):20–23.
  * Williams et al. (2024a) Tom Williams, Cynthia Matuszek, Ross Mead, and Nick DePalma. 2024a.  Scarecrows in Oz: The use of large language models in HRI.  _ACM Transactions on Human-Robot Interaction_ , 13(1).
  * Williams et al. (2024b) Tom Williams, Cynthia Matuszek, Ross Mead, and Nick Depalma. 2024b.  Scarecrows in oz: The use of large language models in HRI.  _J. Hum.-Robot Interact._ , 13(1):1–11.
  * Yin et al. (2021) Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. 2021\.  Including signed languages in natural language processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7347–7360.
  * Zhang et al. (2023) Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. 2023.  Building cooperative embodied agents modularly with large language models.