∎

11institutetext: Xiazhi Hao 22institutetext: College of Science, Zhejiang
University of Technology, Hangzhou, China
33institutetext: S. Y. Lou∗ 44institutetext: School of Physical Science and
Technology, Ningbo University, Ningbo, 315211, China
∗Corresponding author 44email<EMAIL_ADDRESS>
# Higher dimensional integrable deformations of the modified KdV equation

Xiazhi Hao  S. Y. Lou∗

(Received: date / Accepted: date)

###### Abstract

The derivation of nonlinear integrable evolution partial differential
equations in higher dimension has always been the holy grail in the field of
integrability. The well-known modified KdV equation is a prototypical example
of integrable evolution equation in one spatial dimension. Do there exist
integrable analogs of modified KdV equation in higher spatial dimensions? In
what follows, we present a positive answer to this question. In particular,
rewriting the (1+1)-dimensional integrable modified KdV equation in
conservation forms and adding deformation mappings during the process allow
one to construct higher dimensional integrable equations. Further, we
illustrate this idea with examples from the modified KdV hierarchy, also
present the Lax pairs of these higher dimensional integrable evolution
equations.

###### Keywords:

Higher dimensional integrable equation, Conservation form, Deformation
mapping, Lax integrability, Symmetry integrability

###### MSC:

15A04 76M60 83C15

##  1 Introduction

Integrable systems have a significant influence on both theory and
phenomenology. Important physical applications range from fluid mechanics and
nonlinear optics to plasma physics and quantum gravity sbib11 ; abib25 ;
mbib11 . The modern history of integrable systems begins with the solution of
the initial value problem of the Korteweg de Vries (KdV) equation by the
inverse scattering method cbib11  and with the formulation of integrable
equations as the compatibility condition of two linear eigenvalue equations
called a Lax pair pbib4 . As the inverse scattering method develops, the
approaches to the problem for constructing nonlinear partial differential
equations to which this method applies get themselves improved bbib2 ; abib14
. Such nonlinear partial differential equations may be referred to, somewhat
conventionally, as integrable abib26 ; sbib44 ; jbib13 ; cbib7 ; fbib11 . Of
particular interest is the higher dimensional integrable equations, which
contain derivatives with respect to more than two independent variables vbib5
; bbib10 ; cbib10 .

Of course, integrable equations are not easy to find. Recently, in the paper
lou , we proposed a beautiful idea to construct Lax and symmetry integrable
equations by using deformation algorithm. If the original equation has
conservation laws, then the deformation algorithm induces a higher dimensional
integrable equation for the original equation. This is a highly efficient way
of generating higher dimensional integrable equations. In this paper, we put
this construction into a concrete example in order to better understand which
is the essential idea that makes the construction possible. In addition to
this, we construct a full family of higher dimensional integrable hierarchy.

Our main objective is to present a detailed account of results obtained by
means of the deformation for (1+1)-dimensional integrable systems. Our purpose
is not merely to give a formulation of a problem and final result, but also to
explain the method in sufficient detail if it is nontrivial. In the following
section, we shall review briefly a necessary deformation algorithm and give a
complete answer to a natural question: what higher dimensional integrable
equations are the deformations of the (1+1)-dimensional ones? An concrete
example is investigated in detail in section 3. In sections 4 and 5, we
discuss two topics concerning integrability associated with higher dimensional
modified KdV-HD (Harry-Dym) equation, in particular, the Lax integrability and
symmetry integrability. Section 6 is devoted to a particular kink-like
solution formulated in terms of implicitly expression. In last section, we
make some concluding remarks and mention some difficult problems.

##  2 Preliminaries

In this section, we briefly review the deformation algorithm, which closely
follows lou  and provides a method to deform lower dimensional integrable
equations to higher dimensional integrable ones on the situation in which the
lower dimensional integrable equations possess the conservation laws. All
higher dimensional integrable equations considered in the following
deformation algorithm are local. This method is based on the use of
conservation laws.

Deformation algorithm. If a general (1+1)-dimen
sional integrable local evolution equation

$\begin{split}&u_{t}=F(u,\ u_{x},\ \ldots,\ u_{xn}),\\\
&u_{xn}=\partial_{x}^{n}u,u=(u_{1},\ u_{2},\ \ldots,\ u_{m}),\end{split}$ (1)

has some conservation laws

$\begin{split}&\rho_{it}=J_{ix},\ i=1,\ 2,\ \ldots,D-1,\
\rho_{i}=\rho_{i}(u),\\\ &J_{i}=J_{i}(u,\ u_{x},\ \ldots,\
u_{xN}),\end{split}$ (2)

where the conserved densities $\rho_{i}$ are dependent only on the field $u$,
then the deformed $(D+1)$-dimensional equation

$\hat{T}u=F(u,\ \hat{L}u,\ \ldots,\ \hat{L}^{n}u)\ $ (3)

may be integrable suppose that the deformed operators depend on $D$ variables
in such a way that

$\hat{L}\equiv\partial_{x}+\sum_{i=1}^{D-1}a_{i}\rho_{i}\partial_{x_{i}},\
\hat{T}\equiv\partial_{t}+\sum_{i=1}^{D-1}a_{i}\bar{J}_{i}\partial_{x_{i}}$
(4)

and the deformed flows are

$\bar{J_{i}}=J_{i}|_{u_{xj}\rightarrow\hat{L}^{j}u,\ j=1,\ 2,\ \ldots,\ N}.$
(5)

This algorithm leads directly to the possibility of transforming lower
dimensional integrable equations to higher dimensional ones by means of
conservation laws. Furthermore, the same deformation map is enjoyed by their
Lax pairs. Indeed by extending the deformation map to the Lax pairs, we give
more credence to this possibility. Suppose that Lax pair of the
(1+1)-dimensional integrable local evolution equation is

$\displaystyle M\psi=0,N\psi=0.$ (6)

The compatibility condition $[M,N]=MN-NM=0$ holds provided that $u$ satisfies
the original equation. Finding the Lax pair of the $(D+1)$-dimensional
equation is quite straightforward which can be achieved by using the
deformation relations

$\displaystyle\partial_{x}\rightarrow\hat{L},\
\partial_{t}\rightarrow\hat{T}.$ (7)

If we apply (7) to (6), we readily deduce the Lax pair

$\displaystyle\hat{M}\psi=0,\ \hat{T}\psi=0.$ (8)

Strictly speaking, equation (3) can be derived as compatibility condition of a
Lax representation in terms of the (8) which can be obtained from (6) with the
use of two deformation relations (7).

##  3 Higher dimensional modified KdV-HD equations

Our main point is to construct higher dimensional integrable equations whose
Lax pairs and symmetries can be found using deformation algorithm as well. We
first examine the procedure for constructing such equation on a classical
example, the modified KdV equation

$\displaystyle u_{t}=u_{xxx}-6u^{2}u_{x},$ (9)

which occurs, for instance, in the analysis of quasi-one-dimensional solids
nbib1 , liquid-crystal hydrodynamics vbib6 , acoustic wave propagation in
anharmonic lattices mbib19 , and in the theory of nonlinear Alfvén waves in a
collisionless plasma tbib2 . The modified KdV equation is characterized by the
existence of an infinite number of conservation laws rbib23 ; mbib15  and the
first three conservation laws are

$\displaystyle(u)_{t}=(u_{xx}-2u^{3})_{x},$
$\displaystyle(u^{2})_{t}=(2uu_{xx}-u_{x}^{2}-3u^{4})_{x},$
$\displaystyle(u^{4}+u_{x}^{2})_{t}=(2u_{x}u_{xxx}-u_{xx}^{2}+4u^{3}u_{xx}-12u^{2}u_{x}^{2}-4u^{6})_{x}.$

Only the first two which correspond to conservations of momentum and energy,
respectively, will be of concern to us. By a (3+1)-dimensional modified KdV-HD
equation formed from deformation algorithm with two conservation laws, we mean
an equation of the form

$\displaystyle\hat{T}u=\hat{L}(\hat{L^{2}}u-2u^{3})$ (10)

with deformed operators

$\displaystyle\hat{L}=\partial_{x}+a_{1}u\partial_{y}+a_{2}u^{2}\partial_{z},$
$\displaystyle\hat{T}=\partial_{t}+a_{1}\bar{J_{1}}\partial_{y}+a_{2}\bar{J_{2}}\partial_{z}$

and deformed flows

$\displaystyle\bar{J_{1}}=\hat{L^{2}}u-2u^{3},\bar{J_{2}}=2u\hat{L^{2}}u-(\hat{L}u)^{2}-3u^{4},$

or, in more detail

$\displaystyle\begin{split}u_{t}&=3bu_{z}(\hat{L}u)^{2}+(u_{xx}-2u^{3}+\frac{3}{2}a^{2}u^{2}u_{yy}+3auu_{xy}\\\
&+3bu^{2}u_{xz}+\frac{3}{2}b^{2}u^{4}u_{zz}+3abu^{3}u_{zy})_{x}+a(a^{2}u^{3}u_{yy}\\\
&-u^{4}+\frac{3}{2}au^{2}u_{xy}+\frac{9}{4}abu^{4}u_{zy}+\frac{6}{5}b^{2}u^{5}u_{zz}\\\
&+2bu^{3}u_{xz})_{y}+b(b^{2}u^{6}u_{zz}-\frac{3}{5}u^{5}+\frac{3}{2}bu^{4}u_{xz}\\\
&+\frac{3}{4}a^{2}u^{4}u_{yy}+\frac{9}{5}abu^{5}u_{zy}+au^{3}u_{xy})_{z}.\end{split}$
(11)

An interesting remark is that the equation (11) should be integrable, due to
its direct map via a deformation relation to integrable modified KdV equation
(9). Nevertheless, integrability tests such as those based on the Painlevé
property do not work for equation (11) because the reciprocal transformations
are included in higher dimensional equation and a leading term cannot be even
identified.

The usual (1+1)-dimensional modified KdV equation (9) is a simple reduction of
equation (11) with $u_{y}=u_{z}=0$. In analogy, we can take $u$ under
consideration to depend only on $y$, then equation (11) is transferred into

$\displaystyle u_{t}=a(a^{2}u^{3}u_{yy}-u^{4})_{y}.$ (12)

Equation (12) can be written in conservation forms as

$\displaystyle\begin{split}&\rho_{t}^{\pm}=J_{y}^{\pm},~{}\rho^{\pm}=u^{\pm
1},~{}J^{+}=a(a^{2}u^{3}u_{yy}-u^{4}),\\\
&J^{-}=a[2u^{2}-\displaystyle\frac{a^{2}}{2}(u^{2})_{yy}].\end{split}$ (13)

If $u$ merely contains one spatial variable $z$, then equation (11) becomes a
new reciprocal link of the modified KdV equation (9)

$\displaystyle u_{t}$ $\displaystyle=$ $\displaystyle
3b^{3}u^{4}u_{z}^{3}+b(b^{2}u^{6}u_{zz}-\frac{3}{5}u^{5})_{z}.$ (14)

The first two conservation laws associated with equation (14) are expressed in
the forms

$\displaystyle\begin{split}&(u^{-1})_{t}=(bu^{3}-\frac{b^{3}u^{3}(u^{2})_{zz}}{2})_{z},\\\
&(u^{-2})_{t}=-(2b^{3}u^{3}u_{zz}+3bu^{2}(b^{2}u_{z}^{2}-1))_{z}.\end{split}$
(15)

Applying the deformation algorithm to equations (12) and (14) with their
conservation laws (LABEL:utyc) and (LABEL:utzc), respectively, then,
(3+1)-dimensional modified KdV-HD equation (11) is again obtained which we
mention without going into detail.

The consideration of $u$ being independent of $x$ leads equation (11) to be a
(2+1)-dimensional modified KdV-HD equation

$\displaystyle\begin{split}u_{t}&=3buu_{z}(au_{y}+buu_{z})^{2}+a(a^{2}u^{3}u_{yy}-u^{4}+\frac{9}{4}abu^{4}u_{zy}\\\
&+\frac{6}{5}b^{2}u^{5}u_{zz})_{y}+b(b^{2}u^{6}u_{zz}-\frac{3}{5}u^{5}+\frac{3}{4}a^{2}u^{4}u_{yy}\\\
&+\frac{9}{5}abu^{5}u_{zy})_{z}.\end{split}$ (16)

Performing a further reduction $u_{y}=0$, another (2+1)-dimensional modified
KdV-HD equation is obtained

$\displaystyle\begin{split}u_{t}&=3bu_{z}(u_{x}+bu^{2}u_{z})^{2}+(u_{xx}-2u^{3}+3bu^{2}u_{xz}\\\
&+\frac{3}{2}b^{2}u^{4}u_{zz})_{x}+b(b^{2}u^{6}u_{zz}-\frac{3}{5}u^{5}+\frac{3}{2}bu^{4}u_{xz})_{z}.\end{split}$
(17)

The last (2+1)-dimensional reduction

$\displaystyle\begin{split}u_{t}&=(u_{xx}-2u^{3}+\frac{3}{2}a^{2}u^{2}u_{yy}+3auu_{xy})_{x}\\\
&+a(a^{2}u^{3}u_{yy}-u^{4}+\frac{3}{2}au^{2}u_{xy})_{y}\end{split}$ (18)

is obtained by imposing the condition $u_{z}=0$.

##  4 The Lax pair of the (3+1)-dimensional modified KdV-HD equation

The (3+1)-dimensional modified KdV-HD equation (11) is a nonlinear higher
dimensional integrable local evolution equation in four independent variables.
Here integrability means the existence of a Lax pair and infinitely many
higher order symmetries. The investigation of the Lax pair for equation (11)
can be carried out under an analogous deformation map.

The modified KdV equation (9) has the significant property that it results
from the compatibility condition of a certain auxiliary linear problem

$\displaystyle(\partial_{x}^{2}-u^{2}+u_{x}+\lambda)\psi\equiv M\psi=0,$
$\displaystyle(\partial_{t}-4\partial_{x}^{3}+6u^{2}\partial_{x}-6u_{x}\partial_{x}+6uu_{x}-3u_{xx})\psi\equiv
N\psi=0.$

This Lax pair takes after the same deformation map

$\displaystyle\partial_{x}\rightarrow\hat{L},\ \partial_{t}\rightarrow\hat{T}$

the form

$\displaystyle(\hat{L}^{2}-u^{2}+(\hat{L}u)+\lambda)\psi\equiv\hat{M}\psi=0,$
$\displaystyle(\hat{T}-4\hat{L}^{3}+6u^{2}\hat{L}-6(\hat{L}u)\hat{L}+6u(\hat{L}u)-3(\hat{L}^{2}u))\psi\equiv\hat{N}\psi=0,$

which is exactly the Lax representation of the (3+1)-dimensional modified KdV-
HD equation (11).

##  5 The (3+1)-dimensional integrable modified KdV-HD hierarchy and its Lax
pair

The existence of an infinite hierarchy of commuting flows is one of the most
important properties of integrability. All classical integrable equations such
as KdV pbib7 , Kadomtsev-Petviashvili sbib15 , Ablowitz–Kaup–Newell–Segur
sbib45 , Davey-Stewartson sbib20 , etc., were discovered as equations with
infinitely many symmetries. For the modified KdV equation, there exists a
hierarchy of higher commuting flows with related symmetries and recursion
operator as well. The modified KdV hierarchy is defined by the infinite
sequence of flows with respect to the times $u_{t_{2n+1}}$, with the $n^{th}$
member given by the equation bbib12 ; mbib18 ; zbib20 ; sbib43 ; wbib27 ;
pbib12

$\begin{split}&u_{t_{2n+1}}=K_{2n+1}=\Phi^{n}u_{x},\
\Phi=\partial_{x}^{2}-4\partial_{x}u\partial_{x}^{-1}u,\\\ &\ n=0,\ 1,\
\ldots,\ \infty.\end{split}$ (19)

For $n=1$, equation (LABEL:mKdVH) is exactly the modified KdV equation while
the fifth order modified KdV equation

$\displaystyle u_{t_{5}}=(u_{xxxx}-10uu_{x}^{2}-10u^{2}u_{xx}+6u^{5})_{x}$
(20)

is related to (LABEL:mKdVH) with $n=2$. When $n>2$, (LABEL:mKdVH) gives the
higher order versions of the modified KdV equations. For the purpose of
deforming the $n^{th}$ order modified KdV-HD equation, we first introduce the
conserved densities $u$ and $u^{2}$ with the differential polynomial flows in
the forms

$\displaystyle u_{t_{2n+1}}$ $\displaystyle=$ $\displaystyle(J_{2n+1})_{x},\
J_{2n+1}=\partial_{x}^{-1}\Phi^{n}u_{x},\ $ $\displaystyle(u^{2})_{t_{2n+1}}$
$\displaystyle=$ $\displaystyle(G_{2n+1})_{x},\
G_{2n+1}=2\partial_{x}^{-1}u\Phi^{n}u_{x},\ $ (21)

where $J_{2n+1}$ and $G_{2n+1}$ are all differential polynomials of $u$ with
respect to $x$. For $n=2$, (21) yields

$\displaystyle\begin{split}J_{5}&=u_{xxxx}-10uu_{x}^{2}-10u^{2}u_{xx}+6u^{5},\\\
G_{5}&=2uu_{xxxx}-2u_{x}u_{xxx}+u_{xx}^{2}-20u^{3}u_{xx}\\\
&-10u^{2}u_{x}^{2}+10u^{6}.\end{split}$ (22)

On the basis of the conservation laws (21) and the deformation algorithm, the
whole (3+1)-dimensional modified KdV-HD hierarchy follows (LABEL:mKdVH)
directly as
$\partial_{t}\rightarrow\hat{T}_{2n+1},u_{xm}\rightarrow\hat{L}^{m}u$

$\displaystyle\hat{T}_{2n+1}u=(\Phi^{n}u_{x})|_{u_{xm}\rightarrow\hat{L}^{m}u},m=1,2,\ldots,2n+1,$
(23)

where
$\hat{T}_{2n+1}=\partial_{t_{2n+1}}+a_{1}\bar{J}_{2n+1}\partial_{y}+a_{2}\bar{G}_{2n+1}\partial_{z},~{}~{}\hat{L}=\partial_{x}+a_{1}u\partial_{y}+a_{2}u^{2}\partial_{z},~{}~{}\bar{J}_{2n+1}=J_{2n+1}|_{u_{xm}\rightarrow\hat{L}^{m}u}$
and $\bar{G}_{2n+1}=G_{2n+1}|_{u_{xm}\rightarrow\hat{L}^{m}u},m=1,\ 2,\
\ldots,\ 2n+1$. The (3+1)-dimensional modified KdV-HD hierarchy (23) is
equivalent to

$\displaystyle\begin{split}u_{t_{2n+1}}&=(1-a_{1}u_{y}\partial_{x}^{-1}-2a_{2}u_{z}\partial_{x}^{-1}u)\\\
&\Phi^{n}u_{x}|_{u_{xm}\rightarrow\hat{L}^{m}u,\ m=1,\ 2,\ \ldots,\
2n+1}\equiv\bar{K}_{2n+1}\end{split}$ (24)

with the first three $\bar{K}_{2n+1}$ being

$\displaystyle\bar{K}_{1}$ $\displaystyle=$ $\displaystyle u_{x},$
$\displaystyle\bar{K}_{3}$ $\displaystyle=$
$\displaystyle(\hat{L}-a_{1}u_{y})(\hat{L}^{2}u-2u^{3})-a_{2}u_{z}[2u\hat{L}^{2}u-(\hat{L}u)^{2}-3u^{4}],$
$\displaystyle\bar{K}_{5}$ $\displaystyle=$
$\displaystyle(\hat{L}-a_{1}u_{y})[\hat{L}^{4}u-10u(\hat{L}u)^{2}-10u^{2}\hat{L}^{2}u+6u^{5}]$
$\displaystyle-$ $\displaystyle
a_{2}u_{z}[2u\hat{L}^{4}u-2(\hat{L}u)(\hat{L}^{3}u)+(\hat{L}^{2}u)^{2}-20u^{3}\hat{L}^{2}u$
$\displaystyle-$ $\displaystyle 10u^{2}(\hat{L}u)^{2}+10u^{6}].$

It can be confirmed by a calculation that the compatibility condition,
$u_{t_{2n+1}t_{2m+1}}=u_{t_{2m+1}t_{2n+1}}$, of the (3+1)-dimensional modified
KdV-HD hierarchy (24) is clearly satisfied, i.e.,

$\displaystyle[\bar{K}_{2n+1},\
\bar{K}_{2m+1}]=\lim_{\epsilon=0}\frac{\mbox{d}}{\mbox{d}\epsilon}[\bar{K}_{2n+1}(u+\epsilon\bar{K}_{2m+1})$
$\displaystyle-\bar{K}_{2m+1}(u+\epsilon\bar{K}_{2n+1})]=0.$

The family of the KdV hierarchy is generated by making use of the recursion
operator

$\displaystyle\Psi=\partial_{x}^{2}+4v+2v_{x}\partial_{x}^{-1}$ (25)

in the differential relation

$\displaystyle v_{t_{2n+1}}=\Psi^{n}v_{x},\quad n=0,1,2,\dots.$ (26)

The KdV hierarchy is also recognized as the solvability condition for the Lax
representation pbib12 ; xbib27

$\displaystyle\psi_{xx}=(\lambda-v)\psi,$ (27)
$\displaystyle\psi_{t_{2n+1}}=\frac{\Gamma_{x}(v,\lambda)}{2}\psi-\Gamma(v,\lambda)\psi_{x}$
(28)

with

$\displaystyle\Gamma(v,\lambda)=-2\sum_{k=0}^{n}((4\lambda)^{n-k}L_{k}[v]),$
$\displaystyle\partial_{x}L_{k+1}[v]=(\partial_{xxx}+4v\partial_{x}+2v_{x})L_{k}[v],~{}L_{0}[v]=\frac{1}{2},$

which holds good for all equations in the KdV hierarchy. The nonlinear
transformation of Miura or the so-called Miura transformation mbib18 ; abib24

$\displaystyle v=u_{x}-u^{2}$ (29)

converts equations (27)-(28) into the Lax representation

$\displaystyle\psi_{xx}=(\lambda-(u_{x}-u^{2}))\psi,$ (30)
$\displaystyle\psi_{t_{2n+1}}=\frac{\Gamma_{x}((u_{x}-u^{2}),\lambda)}{2}\psi-\Gamma((u_{x}-u^{2}),\lambda)\psi_{x}$
(31)

with

$\displaystyle\Gamma((u_{x}-u^{2}),\lambda)=-2\sum_{k=0}^{n}((4\lambda)^{n-k}L_{k}[u_{x}-u^{2}]),$
$\displaystyle\partial_{x}L_{k+1}[u_{x}-u^{2}]=(\partial_{xxx}+4(u_{x}-u^{2})\partial_{x}$
$\displaystyle+2(u_{x}-u^{2})_{x})L_{k}[u_{x}-u^{2}],~{}L_{0}[u_{x}-u^{2}]=\frac{1}{2}$

of the whole modified KdV hierarchy (LABEL:mKdVH). It is of interest to note
that the Lax representation of all equations in the (3+1)-dimensional modified
KdV-HD hierarchy (23) can be identified from (30)-(31) through the deformation
map $\partial_{x}\rightarrow\hat{L},\ \partial_{t}\rightarrow\hat{T}_{2n+1}.$

##  6 Anomalous kink wave

Despite integrability manifested in the existence of a Lax pair and higher
order symmetries, the analytical explicit solutions sbib10 ; sbib3 ; rbib21 ;
ybib17 ; rbib17 ; rbib18 , however, of the (3+1)-dimensional modified KdV-HD
equation (11) are not easily accessible because the direct application of the
standard analytic techniques, such as Bäcklund transformation, Hirota’s
method, etc. abib3 ; abib27 ; abib28 , for deriving explicit solutions of the
equation (11) fails. Therefore, taking the simplest traveling wave solution
into account, a solitary wave solution of the (3+1)-dimensional mKdV-HD
equation (11) can be sought in the form

$\displaystyle u=U(\xi),~{}\xi=kx+py+qz+\omega t+\xi_{0}$ (32)

with $k,p,q,\omega,\xi_{0}$ constants, where $U$ satisfies a third order
ordinary differential equation

$\displaystyle\omega U_{\xi}$ $\displaystyle=$
$\displaystyle[(k+apU+bqU^{2})^{3}U_{\xi\xi}-U^{3}(2k+apU$ $\displaystyle+$
$\displaystyle\frac{3}{5}bqU^{2})]_{\xi}+3bq(k+apU+bqU^{2})^{2}U_{\xi}^{3}.$

The implicit expression

$\displaystyle U(\xi)=-c\tanh[\frac{c(ap\ln(U-c)+bqU-\xi)}{acp-bc^{2}q-k}]$
(33)

with $\omega=bqc^{4}-2kc^{2}$ gives the single kink-like solution of equation
(11). It can be seen from this expression that when $p$ and $q$ are equal to
zero, leads to the standard kink solution of the usual modified KdV equation.
As $|p|$ and $|q|$ increasing, the kink shape will deform to an anomalous kink
wave. In Fig. 1, the $u$ is plotted as a function of $\xi$ for various values
of $p$ and $q$. We are only able to derive an anomalous kink wave of the
(3+1)-dimensional modified KdV-HD equation (11), but no further multi-soliton
solution.

Figure 1: The images of different single kink shapes of (33) under the same
parameters $a=1,b=1,c=-1,k=1$ and different deformation parameters $p$ and
$q$. (a). A quasi-symmetric kink with small deformation parameter selections
$p=-1/100,q=1/100$. (b). An asymmetric kink with parameter selections
$p=-2,q=2$. (c). An anomalous kink under the larger parameter selections
$p=-5,q=5$.

##  7 Conclusions and discussions

In this study, we have presented a method for generating higher dimensional
integrable modified KdV-HD equations by deformation algorithm which have not
previously been derived. Moreover, these higher dimensional modified KdV-HD
equations turn out to be integrable in the sense that associated with them
there exist compatible pairs of linear systems and infinitely many symmetries.
The strength of the deformation algorithm is that it constructs
(D+1)-dimensional integrable systems, as well as integrable hierarchies, in a
straightforward manner. The whole (3+1)-dimensional modified KdV-HD hierarchy
and its Lax pair have also been obtained via the deformation algorithm from
the (1+1)-dimensional modified KdV hierarchy.

It is plain that the method employed in this paper provides the possibility to
deform a large number of new higher dimensional integrable systems. The
modified KdV equation specifically discussed above is merely an example of the
vistas opened by this methodology to deform new higher dimensional integrable
systems.

Finally, we give several remarks.

First, conservation laws provide an efficient method for the construction of
higher dimensional integrable systems in a regular way. Although we have been
concerned with the modified KdV equation, this method can be applied to a wide
class of nonlinear evolution equations. An analogous construction of higher
dimensional integrable equations can be carried out whenever the conservation
laws exist in the original integrable equations.

Second, from our viewpoint, the most important consequence of these results is
that they establish a connection between lower dimensional integrable systems
and higher dimensional ones. For any higher dimensional deformed integrable
systems, there exist some new (1+1)-dimensional integrable reductions. Again,
applying the deformation algorithm to these reductions, one can re-obtain the
same higher dimensional deformed integrable systems.

Third, this method has the advantage that it constructs integrable higher
dimensional local nonlinear systems, as well as their Lax pairs and higher
order symmetries, in a straightforward manner, but it has the disadvantage
that it may destroy other exact integrability. For instance, integrability
tests such as those based on the Painlevé property do not hold because the
reciprocal transformations which account for the anomalous nature of the
higher dimensional equations are included.

Last but not least, finding the exact solutions of the higher dimensional
integrable equations are quite difficult even for the solitary solutions. Of
course, there are some standard analytic techniques for obtaining solutions of
nonlinear integrable evolution equations, but, for the higher dimensional
integrable equations, some have proved unsuccessful while others have not been
attempted. Research on this issue is in progress.

## Acknowledgement

The work was sponsored by the National Natural Science Foundations of China
(Nos. 12235007, 11975131, 11435005, 12275144, 11975204), K. C. Wong Magna Fund
in Ningbo University, Natural Science Foundation of Zhejiang Province No.
LQ20A010009.

## References

  * (1) Ablowitz, M.J., Baldwin, D.E.: Nonlinear shallow ocean-wave soliton interactions on flat beaches.  Phys. Rev. E 86, 036305 (2012)
  * (2) Antonowicz, M., Fordy, A.P., Wojciechowski, S.: Integrable stationary flows: Miura maps and bi-hamiltonian structures.  Phys. Lett. A 124, 143–150 (1987)
  * (3) Calogero, F.: New C-integrable and S-integrable systems of nonlinear partial differential equations.  J. Nonlinear Math. Phys. 24, 142–148 (2017)
  * (4) Cao, C.W., Geng, X.G.: Classical integrable systems generated through nonlinearization of eigenvalue problems.  Nonlinear Phys. pp. 68–78 (1990)
  * (5) Choudhuri, A., Talukdar, B., Das, U.: The modified Korteweg-de Vries hierarchy: Lax pair representation and bi-Hamiltonian structure.  Z. Naturforsch. A 64, 171–179 (2009)
  * (6) Clarkson, P.A., Joshi, N., Mazzocco, M.: The Lax pair for the MKdV hierarchy.  Séminaires et Congrès 14, 53–64 (2006)
  * (7) Flytzanis, N., Pnevmatikos, S., Remoissenet, M.: Kink, breather and asymmetric envelope or dark solitons in nonlinear chains: I. Monatomic chain.  J. Phys. C: Solid State Phys. 18, 4603–4629 (1985)
  * (8) Fokas, A.S.: Integrable nonlinear evolution partial differential equations in 4+2 and 3+1 dimensions.  Phys. Rev. Lett. 96, 190201 (2006)
  * (9) Fokas, A.S., Ablowitz, M.J.: On the inverse scattering transform of multidimensional nonlinear equations related to first order systems in the plane.  J. Math. Phys. 25, 2494–2505 (1984)
  * (10) Fordy, A.P., Gibbons, J.: Integrable Nonlinear Klein-Gordon Equations and Toda Lattices.  Commun. Math. Phys. 77, 21–30 (1980)
  * (11) Gardner, C.S., Greene, J.M., Kruskal, M.D., Miura, R.M.: Method for solving the Korteweg-de Vries equation.  Phys. Rev. Lett. 19, 1095–1097 (1967)
  * (12) Gu, Z.Q.: Two finite-dimensional completely integrable Hamiltonian systems associated with the solutions of the MKdV hierarchy.  J. Math. Phys. 32, 1531–1536 (1991)
  * (13) Hao, X.: Nonlocal symmetries and molecule structures of the KdV hierarchy.  Nonlinear Dyn. 104, 4277–4291 (2021)
  * (14) Kakutani, T., Ono, H.: Weak nonlinear hydromagnetic waves in a cold collisionless plasma.  J. Phys. Soc. Japan 26, 1305–1318 (1969)
  * (15) Kamenskii, V.G., Rozhkov, S.S.: Higher-order invariants in the nonlinear dynamics of nematics.  Zh. Eksp. Teor. Fiz. 89, 106–115 (1985)
  * (16) Konopelchenko, B.G.: The two-dimensional second-order differential spectral problem: compatibility conditions, general BTs and integrable equations.  Inverse Probl. 4, 151–163 (1988)
  * (17) Konopelchenko, B.G., Dubrovsky, V.G.: Some new integrable nonlinear evolution equations in 2+1 dimensions.  Phys. Lett. A 102, 15–17 (1984)
  * (18) Kupershmidt, B.A., Wilson, G.: Modifying Lax equations and the second Hamiltonian structure.  Invent. Math. 62, 403–436 (1981)
  * (19) Lax, P.D.: Integrals of nonlinear equations of evolution and solitary waves.  Commun. Pur. Appl. Math. 21, 467–490 (1968)
  * (20) Li, Y., Yao, R.X., Xia, Y.R., Lou, S.Y.: Plenty of novel interaction structures of soliton molecules and asymmetric solitons to (2+1)-dimensional Sawada-Kotera equation.  Commun. Nonlinear Sci. Numer. Simulat. 100, 105843 (2021)
  * (21) Lou, S.Y.: Symmetries of the KdV equation and four hierarchies of the integrodifferential KdV equations.  J. Math. Phys. 35, 2390–2396 (1994)
  * (22) Lou, S.Y.: Consistent Riccati expansion for integrable systems.  Stud. Appl. Math. 134, 372–402 (2015)
  * (23) Lou, S.Y., Chen, C.L., Tang, X.Y.: (2+1)-dimensional (M+N)-component AKNS system: Painlevé integrability, infinitely many symmetries, similarity reductions and exact solutions.  J. Math. Phys. 43, 4078–4109 (2002)
  * (24) Lou, S.Y., Cheng, X.P., Tang, X.Y.: Dressed dark solitons of the defocusing nonlinear Schrödinger equation.  Chin. Phys. Lett. 31, 070201 (2014)
  * (25) Lou, S.Y., Hao, X.Z., Jia, M.: Deformation Conjecture: Deforming Lower Dimensional Integrable Systems to Higher Dimensional Ones by Using Conservation Laws.  arXiv:2211.06844v1 (2022)
  * (26) Lou, S.Y., Hu, X.B.: Infinitely many symmetries of the Davey-Stewartson equation.  J. Phys. A: Math. Gen. 27, L207–L212 (1994)
  * (27) Lou, S.Y., Hu, X.B.: Infinitely many Lax pairs and symmetry constraints of the KP equation.  J. Math. Phys. 38, 6401–6427 (1997)
  * (28) Lou, S.Y., Tong, B., Hu, H.C., Tang, X.Y.: Coupled KdV equations derived from two-layer fluids.  J. Phys. A: Math. Gen. 39, 513–527 (2006)
  * (29) Ma, W.X.: Symmetry constraint of MKdV equations by binary nonlinearization.  Phys. A 219, 467–481 (1995)
  * (30) Miura, R.M., Gardner, C.S., Kruskal, M.D.: Korteweg-de Vries equation and generalizations. II. Existence of conservation laws and constants of motion.  J. Math. Phys. 9, 1204–1209 (1968)
  * (31) Mohamad, M.N.B.: Exact solutions to the combined KdV and mKdV equation.  Math. Meth. Appl. Sci. 15, 73–78 (1992)
  * (32) Olver, P.J.: Evolution equations possessing infinitely many symmetries.  J. Math. Phys. 18, 1212–1215 (1977)
  * (33) Qu, C.Z., Song, J.F., Yao, R.X.: Multi-component integrable systems and invariant curve flows in certain geometries.  SIGMA 9, 1–19 (2013)
  * (34) Wadati, M., Sanuki, H., Konno, K.: Relationships among inverse method, Bäcklund transformation and an infinite number of conservation laws.  Prog. Theor. Phys. 53, 419–436 (1975)
  * (35) Wazwaz, A.M.: Integrable (3+1)-dimensional Ito equation: variety of lump solutions and multiple-soliton solutions.  Nonlinear Dyn. 109, 1929–1934 (2022)
  * (36) Wazwaz, A.M.: Multi-soliton solutions for integrable (3+1)-dimensional modified seventh-order Ito and seventh-order Ito equations.  Nonlinear Dyn. 110, 3713–3720 (2022)
  * (37) Wazwaz, A.M., El-Tantawy, S.A.: New (3+1)-dimensional equations of Burgers type and Sharma-Tasso-Olver type: multiple-soliton solutions.  Nonlinear Dyn. 87, 2457–2461 (2017)
  * (38) Weiss, J.: On class of integrable systems and the Painlevé property.  J. Math. Phys. 25, 13–24 (1984)
  * (39) Wojciechowski, S.: Construction of integrable systems by dressing a free motion with a potential.  Phys. Lett. A 96, 389–392 (1983)
  * (40) Yao, R.X., Li, Y., Lou, S.Y.: A new set and new relations of multiple soliton solutions of (2+1)-dimensional Sawada-Kotera equation.  Commun. Nonlinear Sci. Numer. Simulat. 99, 105820 (2021)
  * (41) Zakharov, V.E., Manakov, S.V.: Construction of higher-dimensional nonlinear integrable systems and of their solutions.  Funct. Anal. Appl. 19, 89–101 (1985)
  * (42) Zhang, R.F., Bilige, S.D., Liu, J.G., Li, M.C.: Bright-dark solitons and interaction phenomenon for p-gBKP equation by using bilinear neural network method.  Phys. Scr. 96, 025224 (2020)
  * (43) Zhang, R.F., Bilige, S.D., Temuer, C.: Fractal Solitons, Arbitrary Function Solutions, Exact Periodic Wave and Breathers for a Nonlinear Partial Differential Equation by Using Bilinear Neural Network Method.  J. Syst. Sci. Complex. 34, 122–139 (2021)


# The Scenario Refin _er_ : Grounding subjects in images at the morphological
level

Tagliaferri Claudia
Utrecht University<EMAIL_ADDRESS>
Axioti Sofia
Leiden University<EMAIL_ADDRESS>
Gatt Albert
Utrecht University
<EMAIL_ADDRESS>
Paperno Denis
Utrecht University
<EMAIL_ADDRESS>
###### Abstract

Derivationally related words, such as “runner” and “running”, exhibit semantic
differences which also elicit different visual scenarios. In this paper, we
ask whether Vision and Language (V&L) models capture such distinctions at the
morphological level, using a a new methodology and dataset. We compare the
results from V&L models to human judgements and find that models’ predictions
differ from those of human participants, in particular displaying a
grammatical bias. We further investigate whether the human-model misalignment
is related to model architecture. Our methodology, developed on one specific
morphological contrast, can be further extended for testing models on
capturing other nuanced language features.

##  1 Introduction

Vision and language (V&L) models are trained to ground linguistic descriptions
in visual data. These models differ in pre-training and architecture. In
particular, there are differences in the cross-modal information exchange
between the textual and visual streams of the models Frank et al. (2021);
Parcalabescu and Frank (2022), even though sometimes, as shown for V&L models
based on the BERT architecture Devlin et al. (2019), architectural differences
have little impact on downstream performance for many benchmarks (Bugliarello
et al., 2021).

Pre-trained V&L models achieve high performance on diverse benchmarks, such as
question answering, image retrieval and word masking Tan and Bansal (2019).
However, they have limitations in tasks requiring fine-grained understanding
Bugliarello et al. (2023), including the ability to reason compositionally in
visually grounded settings Thrush et al. (2022), distinguish spatial
relationships and quantities Parcalabescu et al. (2020, 2022), and identify
dependencies between verbs and arguments Hendricks and Nematzadeh (2021). Most
of these fine-grained linguistic phenomena are at the interface between syntax
and semantics.

Far less attention has been paid to grounding fine-grained linguistic features
at the morphological level. We aim to address this gap by investigating
multimodal alignment at the morphological level. We focus on derived nouns
with the agentive suffix -er (e.g. baker) and the corresponding verbal form
(baking). Such derivationally related pairs involve both category-level and
semantic contrasts, with corresponding differences in the typical visual
scenarios they evoke. For instance, human judges would accept the description
x is baking for a variety of visual scenes depicting a person (hereafter
referred to as ‘the subject’) performing a particular action. Only a subset of
such images would, however, also be judged as corresponding to x is a baker,
since the agentive noun introduces additional expectations, for example about
the way the subject is dressed or the physical environment they are in. By
analysing the same stem (e.g. bake) in different parts of speech, we explore
the ability of V&L models to capture the subtle differences in meaning and
visual representation. To do this, we rely on a zero-shot setting in which we
test the probability with which pretrained V&L models match an image to a
corresponding text containing an agentive noun or a verb, comparing this to
human judgments about the same image-text pairs.

Our contributions are: (i) a methodology for testing V&L models on
morphological contrasts; (ii) a dataset of images that highlights the contrast
between verbs and derived nouns, annotated with human judgements; (iii) an
analysis of the V&L models’ predictions on the contrast between derivationally
related verbs and nouns, in comparison to human judgements.

##  2 Related work

###  2.1 Models

Various V&L model architectures have been proposed, differing a.o. in the way
visual vs. textual features are processed. One important distinction, common
among models based on the BERT architecture, is between single- and dual-
stream models. The former concatenate inputs in the two modalities and process
them through a common transformer stack; the latter first process each
modality through its own transformer stack, before performing cross-modal
attention at a later stage Bugliarello et al. (2021). Another influential
architecture is the dual encoder Radford et al. (2021), which is trained to
project visual and textual embeddings into a common multimodal space. Among
their pretraining objectives, BERT-based V&L models typically include image-
text matching, whereby the model returns a probability that an image
corresponds with a caption. Thus, such models can be tested zero-shot on
image-text pairs. For dual encoders, similar insights can be obtained by
comparing the distance in multimodal space between a text and an image
embedding.

We aim to understand the impact of these architectures on the morphological
contrast between word categories and whether the classification depends on
specific visual information. Three models with different architectures and
pre-training phases are tested: CLIP Radford et al. (2021), ViLT Kim et al.
(2021), and LXMERT Tan and Bansal (2019).

#### CLIP

employs a _dual encoder_ architecture and projects image and text embeddings
in a common space, such that corresponding image-text pairs are closer than
non-corresponding ones. CLIP is pre-trained using cross-modal contrastive
learning on internet-sourced image-text pairs, resulting in strong multimodal
representations Radford et al. (2021). Two different visual backbones are used
for the image encoder: ResNet50 He et al. (2016), which uses attention
pooling; and the Vision Transformer Dosovitskiy et al. (2020) which is
modified by the addition of an additional layer normalisation to the combined
patch and position embedding. The text encoder is a Transformer which operates
on a lower-cased byte pair encoding (BPE) representation of the text. CLIP
computes the cosine similarity between an image and a text.

#### LXMERT

follows a _dual-stream_ approach, utilising three encoders: an object
relationship encoder which acts upon the output of a faster-RCNN visual
backbone Ren et al. (2015), a language encoder, and a cross-modality
transformer stack which applies attention across the two modalities. The pre-
training involves five tasks, including masked cross-modality language
modelling and image question answering, enabling the model to establish intra-
modality and cross-modality relationships Tan and Bansal (2019). LXMERT is
also pretrained with an image-text alignment head, which computes the
probability that a text and an image correspond.

#### ViLT

Kim et al. (2021) is the simplest V&L architecture used in this study. It is a
single-stream model in which a single transformer stack processes the
concatenation of visual and textual features. In contrast to other models, no
pre-trained visual backbone is used; rather, the model works directly on
pixel-level inputs, in the spirit of Dosovitskiy et al. (2020). It has been
shown that the usage of word masking and image augmentations improves its
performance Kim et al. (2021). In ViLT, the embedding layers of raw pixels and
text tokens are shallow and computationally light. This architecture thereby
concentrates most of the computation on modelling modality interactions. Like
LXMERT, ViLT is also pre-trained with an image-text alignment head, in
addition to the multimodal masked modelling objective.

###  2.2 Related studies

Our work is related to studies focusing on the _typicality_ of the word-image
relationship and the interplay with category labels for images depicting
people. For example, people can be described using generic expressions
referring to gender or more specific expressions highlighting individual
properties or aspects. Visual properties that align with our conceptual
knowledge of the noun may lead us to prefer agentive expressions over generic
nouns such as “man” or “woman” Corbetta (2021). Gualdoni et al. (2022a, b)
proposed ManyNames, a small dataset that explores the factors that affect
naming variation for visual objects, for instance, the different
conceptualisations of the same object (e.g., “woman” vs. “tennis player”) or
the disambiguation of the nature of the object (e.g., “horse” vs. “pony”).
Understanding the effects of context and naming preferences is crucial for V&L
models to gain comprehensive understanding of visual scenes. The typicality of
the context determines the occurrence of specific names based on the global
scene where the subject is situated.

The current study explores the impact of typicality of the context at the
morphological level. Derivational relations, relating two words or whole
paradigms of words Bonami and Strnadová (2019), involve contrasts at different
levels, including form, syntax – where the words are related but belong to
different word categories – and semantics, where the meaning of one member
contrasts with the meanings of the other members. For instance, runner and run
belong to the same paradigm, but the suffix -er changes the word category and
alters the referential meaning of the verb. For example, “the man is a runner”
evokes a fit person who frequently trains, while “the man is running” could
equally well portray a man casually running to catch a train. Thus, derived
noun subjects should embody characteristics of the verb and/or common
knowledge. Therefore, syntactic and relational knowledge has to be integrated
with semantic knowledge, common imaginary and visual information, as has been
argued from the language acquisition perspective Tyler and Nagy (1989).

##  3 Methodology

###  3.1 Dataset

Noun | Verb | Noun | Verb
---|---|---|---
supporter | supporting | lover | loving
baker | baking | surfer | surfing
runner | running | swimmer | swimming
hunter | hunting | driver | driving
painter | painting | skier | skiing
walker | walking | dancer | dancing
singer | singing | gamer | gaming
teacher | teaching | reader | reading
cleaner | cleaning | smoker | smoking
Table 1: Noun-verb pairs in the Scenario Refiner dataset

We create the Scenario Refin _er_ dataset highlighting the cognitive and
semantic differences between the verb and its derived noun by contrasting one
image with two annotations. The dataset is based on 18 word pairs, each
consisting of a verb in the -ing form and a derived agentive (-er) noun. The
pairs are summarised in Table 1. The lexical pairs are classified into four
conceptual domains: the professional domain (like _baker_ or _teacher_), the
sports domain (like _runner_ or _skier_), the artistic domain (like _dancer_
or _painter_), and general (_lover_ or _smoker_).

Six images were selected for each of the 18 word pairs. These were manually
selected from various sources: Visual Genome Krishna et al. (2017), Wikipedia
Commons, MSCOCO Lin et al. (2014) and Geograph (https://www.geograph.org.uk/).

For the 18 word pairs, we want to compare images which correspond to the
stereotypical representation of the agent role described by the derived noun,
versus the more general scenario described by the verb. In order to depict the
subject denoted by a derived noun, the images need to include additional
information compared to the verb, for example, specific objects like tools or
outfits for _painter_ or _surfer_ ; or a specific environment like a stage for
_dancer_ or _singer_. The verbs correspond to a more general scenario, which
creates a linguistic and visual contrast with the scenario evoked by the
derived noun. This allows us to examine the contrast in parts of speech and
their typicality within the defined global scene Gualdoni et al. (2022b).

For each word pair, 6 images were selected. Each image is accompanied by two
captions, as shown in Figure 1. Each caption received a judgement on a Likert
scale.

Annotation 1: The man and the woman are supporters
Annotation 2: The man and the woman are supporting Annotation 1: The woman
with pink gloves is a driver
Annotation 2: The woman with pink gloves is driving Figure 1: Sample of
stimuli for supporter-supporting and driver-driving

###  3.2 Data collection

We implemented a survey on Qualtrics and distributed it on Prolific. The
survey included 162 images, consisting of 54 fillers and (18 $\times$ 6 =) 108
target images representing the 18 selected lexical pairs.

Our survey also included fillers of several types. In one type, images were
accompanied by a verb-based description and a derived noun in _-er_ , enhanced
by an adjective based on the mood or facial expression of the depicted
subjects. For instance, a smiling subject wearing appropriate outfit on a ski
slope was paired with the captions “The man is a happy skier” and “The man is
skiing”. This type of filler aimed to investigate if participants would alter
their evaluation when the mental representation of the derived noun is
reinforced by additional linguistic information. Another type of filler
contrasted the verb and its derived adjective in _-ive_ , offering insights
into the classification of other members in the morphological paradigm. For
example, four men intensely engaged in a video game were paired with the
sentences “The men are competitive” and “The men are competing”. A third type
of filler contrasted verbs to bare adjectives, descriptive or emotional, to
determine participants’ preference between verbal and adjectival descriptions.
For instance, a couple swimming happily in a lake was matched with “The man
and woman are happy” and “The man and woman are swimming”; an image of a man
speaking in a classroom was paired with “The man is upright” and “The man is
teaching”. The fourth type of filler included images with true and false
descriptions of the visual content, used to maintain participants’ attention
and allowing to control the quality of their responses.

For each image, participants were asked to what extent both captions describe
the visual scenario, using a seven-point Likert scale ranging from totally
disagree to totally agree. By asking to evaluate both captions for each
picture, it is possible to extract a reliable measure of contrast between the
derived noun and the verb.

In order not to risk rough human evaluations and minimise participant dropout
rates due to the length of the survey, the target images were divided equally
between two surveys (each with a total of 81 images where 54 were target
images and 27 fillers).

Twenty native British English speakers completed the online questionnaire and
were randomly assigned to one of the two surveys. Thus, each image is
evaluated by 10 participants for both captions. For the instructions see
Appendix A

##  4 Results

Our analysis proceeds in two stages. We first consider the category
preference: for an image with two captions (one with a derived noun and one
with a verb), we ask whether human judges (resp. V&L models) exhibit a
preference for the noun or the verb with respect to a given image. We then
compute correlations between the preferences exhibited by human judges and by
models for the two categories.

###  4.1 The word category preference

To analyse which of the two captions is preferred for each image by human
judges, we compare the average ratings of the annotations. For V&L models, we
consider the difference in probability estimated by a model’s image-text
matching head (in the case of ViLT and LXMERT) for the caption containing the
noun or verb, or the difference in cosine distance between image and caption
embeddings (in the case of CLIP). Note that we include results for three
versions of CLIP, with different visual backbones. We use a Fisher test to
determine whether there is a significant difference in category preference
between human judges and V&L models.

Table 2 displays the proportion of times the derived noun or the verb was
preferred by humans and by each of the models.

#### Human judgments

Overall, human judges exhibit a preference for captions containing the verb,
with only a small percentage of preferences for captions containing agent
nominals. These types of classifications are distributed across different
domains. This could be due to variation in the images in the extent to which
they gave clear visual cues as to the role of the person depicted. There were
some exceptions to this trend. In the sports domain, these included images of
a skier wearing skiing gear with a cape, and a couple of surfers in surfing
attire with surfboards. In the profession domain, they included two images
depicting individuals engaged in driving and one image of teachers with pupils
posing for a class photo. Four agent nominals belonged to the artistic and
general domains, such as images of women dancing on a stage, two subjects
getting cigarettes, and a woman in a bookshop. On the other hand, the
difference in preference some noun-verb pairs was lower than for others (with
differences in the 0–0.5 range). An example is shown in Figure 2, where
participants interpreted both captions as appropriate. Interestingly, the
versions of CLIP and LXMERT seem to agree with the human ratings in this
example, showing low contrast between the verb and the noun, with LXMERT
assigning higher probability to verb caption for (c) and CLIP estimating lower
distance between image and verb caption for (a). On the other hand, ViLT
assigned a higher probability to the verbal caption for all the images in
Figure 2.

(a) M = 5.50 (noun, verb), SD = 1.20 (noun, verb)

(b) M = 6.30 (noun, verb), SD = 0.90 (noun, verb)

(c)  M = 5.30 (noun, verb), SD = 0.90 (noun), 1.00 (verb)

Figure 2: Mean (M) human judgments and standard deviations (SD) for an example
image set corresponding to lover-loving.

#### V&L models

Unlike participants, V&L models exhibit a tendency to prefer deverbal nouns to
verbs. The exceptions are CLIP with the ViT-B/32 backbone, and ViLT, both of
which have a slightly higher preference for captions with verbs. The
performance of CLIP seems to depend on the visual backbone. Of the three
versions, ViT-L/14 displays the greatest similarity to human judgments. We
observed a tendency for ViT-B/32 to prefer captions with derived nouns where
there are clear visual cues suggesting a role or activity, such as the
microphone and the stage in Figure 3(b). In contrast, while CLIP-RN50 prefers
the noun caption in Figure 3(b)(a), it shows the opposite trend, in favour of
the verb-based caption, in (b), perhaps because the stage is less clearly
visible.

(a) noun: M = 6.20, SD = 1.17; verb: M = 6.50, SD = 1.02

(b) noun: M = 6.20, SD = 1.17; verb: M = 6.50, SD = 1.02

Figure 3: Mean (M) human judgments and standard deviations (SD) for an example
image set corresponding to singer-singing

.

The difference between the judgements of humans vs. V&L models is
statistically significant (Fisher’s exact test, p $<$ 0.001 for all contrasts
between models and human judgments).

| Derived noun | Verb
---|---|---
Humans | 8.3% | 91.7%
CLIP ViT-L/14@336px | 51.9% | 48.1%
CLIP RN50x64 | 52.8% | 47.2%
CLIP ViT-B/32 | 49.1% | 50.9%
ViLT | 47.2% | 52.8%
LXMERT | 51.9% | 48.1%
Table 2:  Preference for derived noun vs. verb, in human judgments and V&L
model image-text alignment.

###  4.2 Correlations between judgements

We also estimate the correlation between human and automatic judgements as a
more fine-grained measure than binary preference. Overall, the correlation
between the human and the automatic judgements varies depending on
architecture and on the conceptual domain.

We assess correlations between three kinds of values: the (human- or model-
produced) scores for a) noun and b) verb-based captions, as well as c) the
difference between the noun and verb scores. We refer to the latter as the
morphological contrast.

#### Participant consistency

To assess the consistency of collected human judgements, we split participants
randomly into two equal-sized samples and calculate Pearson correlation
coefficients between the average scores of the two samples. The resulting
correlation coefficients for all conceptual domains are reported in Table 3.
Correlation coefficients for noun, verb and contrast are generally consistent,
with the exception of the artistic domain, for which correlations between
judgments for verb-based captions, and as a consequence, also for the
contrast, exhibit more variation.

Domain | Derived noun | Verb | Morphological contrast
---|---|---|---
Professional domain | 0.76 | 0.84 | 0.75
Sport domain | 0.69 | 0.70 | 0.60
Artistic domain | 0.79 | 0.31 | 0.51
General | 0.92 | 0.88 | 0.94
All domains | 0.80 | 0.81 | 0.78
Table 3:  Human judgements: Pearson correlations of judgments for captions containing derived nouns and verbs, and for the difference (contrast). Model | Derived noun | Verb | Morphological contrast
---|---|---|---
CLIP ViT-L/14@336px | 0.13 | 0.08 | 0.15
CLIP RN50x64 | 0.09 | 0.08 | -0.01
CLIP ViT-B/32 | 0.09 | 0.18 | 0.08
ViLT | 0.07 | 0.26 | 0.32
LXMERT | 0.16 | 0.03 | 0.21
Table 4:  Human judgments and V&L models overall: Pearson correlations between human judgements and model image-text alignment for captions containing derived nouns, verbs, and the contrast between them. | Sport domain | Professional domain
---|---|---
| Deriv. noun | Verb | Morph. contrast | Deriv. noun | Verb | Morph. contrast
CLIP ViT-L/14@336p | 0.02 | -0.25 | -0.06 | -0.04 | 0.19 | 0.33
CLIP RN50x64 | 0.02 | -0.31 | -0.11 | -0.22 | 0.33 | 0.23
CLIP ViT-B/32 | -0.04 | -0.26 | -0.11 | -0.16 | 0.23 | 0.40
ViLT | -0.01 | -0.04 | 0.45 | 0.30 | 0.45 | 0.68
LXMERT | 0.08 | -0.32 | 0.17 | -0.10 | 0.18 | 0.40
| Artistic domain | General
| Derived noun | Verb | Contrast | Derived noun | Verb | Contrast
CLIP ViT-L/14@336p | -0.03 | 0.01 | 0.22 | 0.28 | 0.18 | -0.06
CLIP RN50x64 | 0.06 | 0.21 | 0.27 | 0.08 | 0.23 | 0.10
CLIP ViT-B/32 | -0.09 | -0.04 | -0.007 | 0.23 | 0.25 | -0.06
ViLT | 0.44 | 0.15 | 0.39 | -0.06 | 0.05 | 0.25
LXMERT | 0.29 | 0.26 | 0.42 | -0.01 | -0.25 | 0.12
Table 5:  Human judgement and V&L models by domain: Pearson correlation
between human judgments and model image-text matching estimates for captions
containing derived nouns, verbs, and the morphological contrast between the
derived noun and the verb.

#### Models vs human judgments

Table 4 displays the overall correlations between human judgments and model
image-text alignment for verbs, nouns and the morphological contrast. The
correlations are moderate-to-weak, suggesting a lack of alignment between
human intuitions and V&L models. This is consistent with our earlier
observation that models tend to exhibit different preferences for nouns versus
verbs, compared to humans. Interestingly, ViLT emerges as the most correlated
model with human judgement in the verbal evaluation, but it exhibits the least
correlation in the evaluation of the derived noun. Additionally, ViLT displays
a moderate positive relationship with the contrast between verb and derived
noun, whereas the other models demonstrate weaker positive correlations or
very weak negative correlations with this particular contrast.

Table 5 breaks down correlations by conceptual domain. In the professional
domain, correlations are generally stronger, especially for ViLT, LXMERT and
CLIP ViT-B/32. Overall, it appears that models correlate with human judges in
some domains more than others. Nevertheless, correlations are often negative,
and these results suggest a qualitative difference between the image-text
alignment performed by models, and the types of knowledge and inferences that
humans bring to bear to support the grounding of nominal agentive versus
verbal forms in visual stimuli.

##  5 Discussion

The findings revealed a discrepancy between models and human judgments. Humans
displayed a preference for captions containing verbs, whereas V&L models
exhibited a preference for nominal descriptions. Participants prefer the
derived noun only for a few instances that had additional characteristics
elicited by visual elements, or by the kind of action performed by the human
subjects in the images. For instance, they prefer the derived noun for two
images showing a person getting or purchasing cigarettes (smoker-smoking),
meaning that participants interpreted the intention as a characteristic that
corresponds to the derived noun. In contrast, the tested models appeared to
prioritise more the action itself rather than the individual who performs the
action.

However, examining certain lexical pairs, we observed a greater variance in
the pattern of interpretation, highlighting the difficulty in defining the
human evaluation of the derived noun. For example, in the sport domain,
participants rarely seem to rely on the outfit worn by the subject to base
their interpretation, with the exception of _skier_ , which happened to be
paired only with an image of a subject also exhibiting their competition
number. As a surprising contrast, two pictures for _runner-running_ similarly
depicted subjects with their competition numbers are not evaluated as such by
participants. Specifically, one image depicts a man running in a race track,
while the other image depicts three men wearing specific outfits running in
the countryside. The contrast between the means of the human evaluation is
less than or equal to 0.50, indicating the preference for the verbal
description.

The models, too, exhibit variety in the subject classification for these
images. For example, while CLIP-ViT-L/14@336p, CLIP-ViT-B/32 and ViLT display
a similar preference for the nominal form, as humans do, for _skier-skiing_ ,
CLIP-RN50x64 and LXMERT prefer the verb-based caption. Similarly, while
participants slightly prefer the verb for the subjects wearing a competition
number for _runner-running_ , models prefer the nominal description. The three
versions of CLIP strongly prefer the derived noun for these subjects, ViLT
prefers the verbal description only for the single subject running in a race
track and LXMERT prefers the verbal description only for the three subjects
running in the countryside. While CLIP exhibited a preference for the derived
noun in presence of additional visual elements, ViLT and LXMERT do not seem to
base their preference on such a visual cue since they assign a high
probability to the verbal description too.

##  6 Conclusion

We studied the morphological difference between derived nouns in _-er_ and
verbs for visual grounding, comparing human judgements with pre-trained Vision
and Language models. The dataset we presented allows us to assess vision and
language models on their understanding of verbs, deverbal agent nouns, and
most importantly the contrast between the two. Our results show that while
some models, especially ViLT, show strong results for some of the conceptual
domains, they do not support the conclusion that models ground the
morphological differences between derived nouns and verbs in a humanlike way.

Highlighting and investigating such a morphological and cognitive difference
can refine and improve the alignment of textual and visual input of V&L
models. By exploring the visual classification at the morphological level, the
aim was to investigate not only the linguistic and morphological influence in
the automatic recognition of subjects carrying certain visual information, but
also to individuate which architecture of the model better executes the task.
In our study, the single-stream ViLT model tends to correlate better with
human judgments. Nevertheless, these results are based on a relatively small
test set and focus on a restricted set of models, with much scope for further
experimentation. In an effort to encourage the community to undertake further
investigation of these phenomena, we have shared our code and our dataset
text.111https://github.com/ClaudiaTagliaferri/Scenario_Refiner.git.

## References

  * Bonami and Strnadová (2019) Olivier Bonami and Jana Strnadová. 2019.  Paradigm structure and predictability in derivational morphology.  _Morphology_ , 29(2):167–197.
  * Bugliarello et al. (2021) Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. 2021\.  Multimodal pretraining unmasked: A meta-analysis and a unified framework of vision-and-language berts.  _Transactions of the Association for Computational Linguistics_ , 9:978–994.
  * Bugliarello et al. (2023) Emanuele Bugliarello, Laurent Sartran, Aishwarya Agrawal, Lisa Anne Hendricks, and Aida Nematzadeh. 2023.  Measuring progress in fine-grained vision-and-language understanding.  _arXiv preprint arXiv:2305.07558_.
  * Corbetta (2021) Daniela Corbetta. 2021.  Effects of typicality and category label on referring expressions in context: Empirical analysis in the domain of “people”.  _Departament de Traducció i Ciències del Llenguatge_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beye, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv_ , 2010.11929.
  * Frank et al. (2021) Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021.  Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers.  _arXiv preprint arXiv:2109.04448_.
  * Gualdoni et al. (2022a) Eleonora Gualdoni, Thomas Brochhagen, Andreas Mädebach, and Gemma Boleda. 2022a.  Horse or pony? visual typicality and lexical frequency affect variability in object naming.  _Proceedings of the Society for Computation in Linguistics_ , 5(1):241–243.
  * Gualdoni et al. (2022b) Eleonora Gualdoni, Thomas Brochhagen, Andreas Mädebach, and Gemma Boleda. 2022b.  Woman or tennis player? visual typicality and lexical frequency affect variation in object naming.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 44.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep Residual Learning for Image Recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, Las Vegas, NV, USA. IEEE.
  * Hendricks and Nematzadeh (2021) Lisa Anne Hendricks and Aida Nematzadeh. 2021.  Probing image-language transformers for verb understanding.  _arXiv preprint arXiv:2106.09141_.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , pages 5583–5594. PMLR.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017\.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123:32–73.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’a r, and C. Lawrence Zitnick. 2014.  Microsoft COCO: common objects in context.  _CoRR_ , abs/1405.0312.
  * Parcalabescu et al. (2022) Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. 2022.  VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8253–8280, Dublin, Ireland. Association for Computational Linguistics.
  * Parcalabescu and Frank (2022) Letitia Parcalabescu and Anette Frank. 2022.  MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks.  ArXiv:2212.08158 [cs].
  * Parcalabescu et al. (2020) Letitia Parcalabescu, Albert Gatt, Anette Frank, and Iacer Calixto. 2020.  Seeing past words: Testing the cross-modal capabilities of pretrained v&l models on counting tasks.  _arXiv preprint arXiv:2012.12352_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  In _Advances in Neural Information Processing Systems 28 (NeurIPS 2015)_ , Montreal, Canada.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _arXiv preprint arXiv:1908.07490_.
  * Thrush et al. (2022) Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022.  Winoground: Probing vision and language models for visio-linguistic compositionality.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5238–5248.
  * Tyler and Nagy (1989) Andrea Tyler and William Nagy. 1989.  The acquisition of english derivational morphology.  _Journal of Memory and Language_ , 28(6):649–667.

##  Appendix A Appendix

Instructions for participants:

Welcome to our survey! Our project focuses on improving existing annotation
accompanying pictures. You will be presented with pictures and asked to
indicate to which degree you agree with some statements. The study should take
you around 15-20 minutes to complete. Your participation in this research will
be paid only if you complete the survey. Please make sure to be redirected to
Prolific at the end of the survey. In such a way, we can check if you
completed the study and pay your participation.The ProlificID and all the
sensitive data will be deleted once the payment is done. In the next page, you
will be able to read more about the study and how we are doing with the data.
If you would like to contact us to receive more information about the
annotation project, please<EMAIL_ADDRESS>or<EMAIL_ADDRESS>