Exactly solvable piecewise analytic double well potential
$V_{\text{D}}(x)=\text{min}[(x+d)^{2},(x-d)^{2}]$ and its dual
single well potential $V_{\text{S}}(x)=\text{max}[(x+d)^{2},(x-d)^{2}]$

Ryu Sasaki

Department of Physics, Tokyo University of Science, Noda 278-8510, Japan

###### Abstract

By putting two harmonic oscillator potential $x^{2}$ side by side with a
separation $2d$, two exactly solvable piecewise analytic quantum systems with
a free parameter $d>0$ are obtained. Due to the mirror symmetry, their
eigenvalues $\\{E\\}$ for the even and odd parity sectors are determined
exactly as the zeros of certain combinations of the confluent hypergeometric
function ${}_{1}F_{1}$ of $d$ and $E$, which are common to $V_{\text{D}}$ and
$V_{\text{S}}$ but in two different branches. The eigenfunctions are the
piecewise square integrable combinations of ${}_{1}F_{1}$, the so called $U$
functions. By comparing the eigenvalues and eigenfunctions for various values
of the separation $d$, vivid pictures unfold showing the tunneling effects
between the two wells.

Published

Journal of Mathematical Physics 64 (2023) 022102

##  1 Introduction

Double well potentials in quantum mechanics are discussed in various contexts,
e.g. tunneling or spontaneous symmetry breaking, etc. Most commonly studied
are the quartic potentials $V_{\text{Q}}(x)=x^{4}-ax^{2}+bx$, ($a>0$), whose
eigenfunctions are quite complicated and the system is far from exactly
solvable. Some solvable examples are the square double well and the double
Dirac delta potential. Recently Miloslav Znojil introduced an interesting
double and single well potentials (2.3), (2.6) [1]. They are just mirror
symmetrically separated harmonic oscillator potentials, which are not analytic
at the origin but exactly solvable due to the harmonic oscillator nature. One
motivation of this paper is to supplement the pioneering work of Znojil.

Another profound motivation is to disseminate the possibility to enlarge the
list of exactly solvable quantum mechanical systems by combining the technique
of constructing mirror symmetric non-analytic solvable potentials [1, 2, 3, 4]
with other known methods of solvability such as the factorisation [5, 6],
shape-invariance [7], the exceptional and multi-indexed polynomials [8, 9, 10,
11], non-polynoial extensions [12], etc, in particular, the Krein-Adler
deformations [13, 14, 15]. By incorporating the present solvable model
construction method to the simplest Krein-Adler deformation of the harmonic
oscillator potential [16], a new double well potential is proposed in §5
(5.3). It is expected to be the breakthrough point for constructing a
multitude of similarly exactly solvable potentials.

The present paper is prepared in a plain style so that non-experts can easily
understand. This paper is organised as follows. In section two, after a brief
introduction of the double and single well potentials, the simplification of
the connection conditions in mirror symmetric potentials due to the separation
into the even and odd sectors is recapitulated. The symmetric relationships
between the connection conditions of the double and single well potentials are
stated as Theorem 2.1. A simple lower bound of the eigenvalues of the single
well potential is mentioned as Remark 2.2. In section three, elementary
polynomial type solutions are briefly surveyed. The connection conditions
determining the eigenvalues of the even and odd sectors are expressed as the
zeros of the Hermite polynomials and the derivatives in Theorem 3.1. The
polynomial type eigenfunctions are displayed in Theorem 3.2 together with the
explicit expressions of the connection conditions in Tables 1 and 2. The
relationship with the results in [1] is remarked. The upper bound of the
greatest zeros of the Hermite polynomials is mentioned in connection with
Remark 2.2. In §4, starting with the Kummer differential equation, the
piecewise square integrable combination of the confluent hypergeometric
functions are introduced. Theorem 4.1 states that the eigenvalues are obtained
as the zeros of the connection conditions for the even and odd sectors. The
‘duality’ of the connection conditions for the double and single well
potentials is alluded in Remark 4.2. Seven lowest eigenvalues for the double
and single well potentials for a few small values of $d$’s are shown in Tables
3 and 4. Several graphs of some lower eigenfunctions are shown in Figures 2–4.
A simple interpretation of the tunneling effects on the even and odd sector
eigenvalues are presented. In §4.2 it is shown that various quantities and
expressions in §4.1 are simplified for odd integer eigenvalues. In section
five, after a very brief summary, two new piecewise analytic double and single
well potentials are proposed (5.1), (5.4) together with the graphs of the
potentials in Figure 6 and 6.

##  2 Mirror symmetric and piecewise analytic potential

Here we explore a new exactly solvable double well potential $V_{\text{D}}(x)$
and its dual single well potential $V_{\text{S}}(x)$ with $d>0$,

$\displaystyle
V_{\text{D}}(x)=\text{min}[(x+d)^{2},(x-d)^{2}]=\left\\{\begin{array}[]{cc}(x-d)^{2}&x\geq
0\\\\[2.0pt] (x+d)^{2}&x\leq 0\end{array}\right.,\qquad
V_{\text{D}}(x)=V_{\text{D}}(-x),$ (2.3) $\displaystyle
V_{\text{S}}(x)=\text{max}[(x+d)^{2},(x-d)^{2}]=\left\\{\begin{array}[]{cc}(x+d)^{2}&x\geq
0\\\\[2.0pt] (x-d)^{2}&x\leq 0\end{array}\right.,\qquad
V_{\text{S}}(x)=V_{\text{S}}(-x),$ (2.6)

which were recently introduced by Miloslav Znojil in a pioneering work [1].
This paper will be cited as I hereafter. For $d=0$, $V_{\text{D}}$ and
$V_{\text{S}}$ reduce to the well known harmonic oscillator
$V_{\text{D}}(x)=V_{\text{S}}(x)=x^{2}$. Since the min, max definitions of
$V_{\text{D}}(x)$ and $V_{\text{S}}(x)$ are obviously symmetric with
$d\leftrightarrow-d$, we have restricted to $d>0$ and the $d$-dependence of
the potentials, the wavefunctions, eigenvalues etc is usually suppressed for
the simplicity of presentation.

These potentials are obviously mirror symmetric $V(x)=V(-x)$ and analytic on
either half line $x>0$ and $x<0$ and the non-analyticity occurs only at the
origin $x=0$. That is, the wavefunctions on either half line are analytic
functions. That is, their wavefunctions $\psi_{\text{D}}(x)$ and
$\psi_{\text{S}}(x)$ of the Schrödinger equations

$\displaystyle-\frac{d^{2}\psi_{\text{D}}(x,E)}{dx^{2}}+V_{\text{D}}(x)\psi_{\text{D}}(x,E)=E\psi_{\text{D}}(x,E),\quad-\frac{d^{2}\psi_{\text{S}}(x,E)}{dx^{2}}+V_{\text{S}}(x)\psi_{\text{S}}(x,E)=E\psi_{\text{S}}(x,E),$
(2.7)

are piecewise analytic

$\displaystyle\psi_{\text{D}}(x,E)=\left\\{\begin{array}[]{cc}\psi_{\text{D}}^{(+)}(x,E)&x>0\\\\[2.0pt]
\psi_{\text{D}}^{(-)}(x,E)&x<0\end{array}\right.,\qquad\psi_{\text{S}}(x,E)=\left\\{\begin{array}[]{cc}\psi_{\text{S}}^{(+)}(x,E)&x>0\\\\[2.0pt]
\psi_{\text{S}}^{(-)}(x,E)&x<0\end{array}\right..$

Let us assume that the above wavefunctions are piecewise square integrable for
both D and S,

$\int_{0}^{\infty}\left(\psi^{(+)}(x,E)\right)^{2}dx<\infty,\quad\int_{-\infty}^{0}\left(\psi^{(-)}(x,E)\right)^{2}dx<\infty.$

This selects one solution in the two-dimensional solution space of the above
Schrödinger equations (2.7) for generic $E$. Like other one-dimensional
quantum mechanical systems with piecewise analytic potentials, we require the
continuity of the wavefunctions and their first derivatives

$\psi^{(+)}(0,E)=\psi^{(-)}(0,E),\quad\frac{d\psi^{(+)}}{dx}(0,E)=\frac{d\psi^{(-)}}{dx}(0,E).$
(2.8)

These select the eigenvalues $\\{E_{n}\\}$, $n=0,1,\ldots$, since the
continuous wavefunctions are square integrable eigenfunctions
$\\{\psi(x,E_{n})\\}$,

$\int_{-\infty}^{\infty}\psi(x,E_{n})^{2}dx<\infty,\qquad n=0,1,\ldots.$

Thanks to the mirror symmetry, this solution process is simplified extensively
as demonstrated in other similar examples, $V(x)=-g^{2}{\rm exp}(-|x|)$ [2],
$V(x)=g^{2}{\rm exp}(2|x|)$ [3] and symmetric Morse potential [4], etc. Due to
the mirror symmetry of the potentials the wavefunctions are split into the
even and odd parity sectors

$\displaystyle{\rm even:}$
$\displaystyle\quad\psi^{(+)}(x,E)=\psi^{(-)}(-x,E),\qquad\ \ {\rm odd:}$
$\displaystyle\quad\psi^{(+)}(x,E)=-\psi^{(-)}(-x,E).$ (2.9)

The continuity of the wavefunctions and their first derivatives provides the
equations determining the eigenvalue $E$, which have the same forms for D and
S,

$\displaystyle{\rm even:}$
$\displaystyle\quad\psi^{(+)}(0,E)=\psi^{(-)}(0,E),\qquad\ \
\frac{d\psi^{(+)}(0,E)}{dx}=0,$ (2.10) $\displaystyle{\rm odd:}$
$\displaystyle\quad\frac{d\psi^{(+)}(0,E)}{dx}=\frac{d\psi^{(-)}(0,E)}{dx},\qquad\psi^{(+)}(0,E)=0.$
(2.11)

Thanks to the mirror symmetry, the first condition is trivially satisfied for
both sectors by fixing the relative scales of the $\psi^{(+)}(x)$ and
$\psi^{(-)}(x)$, The second condition determines the eigenvalues $\\{E_{n}\\}$
$n=0,1,\ldots$ as functions of the system parameters. In the present case they
are $d$. The second conditions can be replaced by the equivalent one
$\frac{d\psi^{(-)}(0,E)}{dx}=0$ for the even sector and $\psi^{(-)}(0,E)=0$
for the odd sector. Obviously the even parity condition (2.10) is the Neumann
boundary condition and the odd parity one (2.11) is the Dirichlet boundary
condition. This is the rare occasion that the Neumann b.c. appears in quantum
mechanics. The Dirichlet b.c. appears wherever an impenetrable barrier stands.

Another simplification is built in due to the forms of the double and single
well potentials $V_{\text{D}}(x)$ (2.3) and $V_{\text{S}}(x)$ (2.6). On the
positive half line $x>0$, $V_{\text{D}}(x)=(x-d)^{2}$,
$V_{\text{S}}(x)=(x+d)^{2}$ and they interchange by $d\leftrightarrow-d$. The
same situation happens on the negative half line, $x<0$, too. Therefore, when
the Neumann b.c. equation (2.10) is written down for the $V_{\text{D}}(x)$
wavefunctions, the equation for the $V_{\text{S}}(x)$ wavefunctions is simply
obtained by changing $d$ into $-d$, and vice versa. The situation is the same
for the Dirichlet b.c. equation (2.11). Let us write down the equations
determining the eigenvalues $\\{E\\}$ due to the Neumann b.c. (2.10) (for the
even sector) and due to the Dirichlet b.c. (2.11) (for the odd sector) of the
$V_{\text{D}}$ and $V_{\text{S}}$ systems as

$\displaystyle{\rm Double\ well:}\quad$ $\displaystyle
C_{\text{D}}^{(e)}(d,E)=0,\quad C_{\text{D}}^{(o)}(d,E)=0,$ (2.12)
$\displaystyle{\rm Single\ well:}\quad$ $\displaystyle
C_{\text{S}}^{(e)}(d,E)=0,\quad C_{\text{S}}^{(o)}(d,E)=0,$ (2.13)

The following theorem states their close relationship.

###### Theorem 2.1

For the even and odd sectors, the functions for the double well and single
well are simply related by

$\displaystyle{\rm even:}\
C_{\text{D}}^{(e)}(x,E)=C_{\text{S}}^{(e)}(-x,E),\qquad{\rm odd:}\
C_{\text{D}}^{(o)}(x,E)=C_{\text{S}}^{(o)}(-x,E),\quad x\in\mathbb{R},$ (2.14)

up to some irrelevant constant factors.

This is why we call $V_{\text{S}}(x)$ is the dual potential of
$V_{\text{D}}(x)$, and vice versa. Without determining these functions, we can
safely make the following statement concerning the lower bounds of the
eigenvalues of the $V_{\text{S}}$ system.

###### Remark 2.2

For both even and odd sectors, the eigenvalues of the single well system are
greater than $d^{2}$,

$V_{\text{S}}(x)\geq d^{2}\ \Longrightarrow E>d^{2}.$ (2.15)

##  3 Polynomial type solutions

The most basic result of one-dimensional quantum mechanics is that the Hermite
polynomials $\\{H_{n}(x)\\}$ provide the complete set of eigenfunctions of the
quadratic potential $x^{2}$. This means,

$\displaystyle-\frac{d^{2}\Psi_{p}(x,2n+1)}{dx^{2}}+(x+d)^{2}\Psi_{p}(x,2n+1)=(2n+1)\Psi_{p}(x,2n+1),$
$\displaystyle\hskip
142.26378pt\Psi_{p}(x,2n+1)=e^{-(x+d)^{2}/2}H_{n}(x+d),\qquad\
n\in\mathbb{Z}_{\geq 0},$ (3.1)
$\displaystyle-\frac{d^{2}\Psi_{m}(x,2n+1)}{dx^{2}}+(x-d)^{2}\Psi_{m}(x,2n+1)=(2n+1)\Psi_{m}(x,2n+1),$
$\displaystyle\hskip
142.26378pt\Psi_{m}(x,2n+1)=\alpha_{n}e^{-(x-d)^{2}/2}H_{n}(x-d),\quad
n\in\mathbb{Z}_{\geq 0},$ (3.2)

in which $\alpha_{n}$ is a constant. Here the subscript $p$ means ‘plus’ $d$,
i.e. $(x+d)^{2}$ potential and $m$ means ‘minus’ $d$, $(x-d)^{2}$ potential.
The degree $n$ Hermite polynomial $H_{n}(x)$ has the parity
$H_{n}(-x)=(-1)^{n}H_{n}(x)$. This means that for $E=2n+1$, for example,
$\psi_{\text{S}}^{(+)}(x,2n+1)=e^{-(x+d)^{2}/2}H_{n}(x+d)$ is a piecewise
square integrable wavefunction of the single well system on the right half
line $x>0$. Thus we arrive at a theorem.

###### Theorem 3.1

The Neumann (2.10) and Dirichlet (2.11) b.c. provide the equations

$\displaystyle{\rm even:}\ H_{n}^{\prime}(d)-dH_{n}(d)=0,\qquad{\rm odd:}\
H_{n}(d)=0,\qquad\bigl{(}H_{n}^{\prime}(d)=2nH_{n-1}(d)\bigr{)},$ (3.3)

determining a finite number of $\\{d\\}$’s with which the continuous
connection with the left half line wavefunction
$\psi_{\text{S}}^{(-)}(x,2n+1)$ is realised.

Due to the parity of the Hermite polynomial, the contents of these equations
are the same when $d$ is changed to $-d$, meaning that the above equations
apply to the double well system, too. According to Theorem 2.1 we arrive at
the following theorem.

###### Theorem 3.2

To each positive odd integer $2n+1$ ($n\in\mathbb{N}$) correspond two sets of
distinct positive parameters $\\{d_{j}^{e}\\}$, $j=1,\ldots,[\langle\\!\langle
n\rangle\\!\rangle/2]$, and $\\{d_{j}^{o}\\}$, $j=1,\ldots,[\langle
n\rangle/2]$, satisfying
$H_{n}^{\prime}(d_{j}^{e})-d_{j}^{e}H_{n}(d_{j}^{e})=0$ and
$H_{n}(d_{j}^{o})=0$ (3.3), respectively. For the even type $d_{j}^{e}$, the
Schrödinger equations with $V_{\text{D}}(x)$ and $V_{\text{S}}(x)$ potential
have an even parity eigenstate with the eigenvalue $2n+1$,

$\displaystyle\psi_{\text{D},j}^{(e)}(x,2n+1)$
$\displaystyle=\left\\{\begin{array}[]{rc}e^{-(x+d_{j}^{e})^{2}/2}H_{n}(x+d_{j}^{e})&-\infty<x\leq
0\\\\[2.0pt] (-1)^{n}e^{-(x-d_{j}^{e})^{2}/2}H_{n}(x-d_{j}^{e})&0\leq
x<\infty\end{array}\right.\ ,$ (3.6)
$\displaystyle\psi_{\text{S},j}^{(e)}(x,2n+1)$
$\displaystyle=\left\\{\begin{array}[]{rc}(-1)^{n}e^{-(x-d_{j}^{e})^{2}/2}H_{n}(x-d_{j}^{e})&-\infty<x\leq
0\\\\[2.0pt] e^{-(x+d_{j}^{e})^{2}/2}H_{n}(x+d_{j}^{e})&0\leq
x<\infty\end{array}\right..$ (3.9)

For the odd type $d_{j}^{o}$, the Schrödinger equations with $V_{\text{D}}(x)$
and $V_{\text{S}}(x)$ potential have an odd parity eigenstate with the
eigenvalue $2n+1$,

$\displaystyle\psi_{\text{D},j}^{(o)}(x,2n+1)$
$\displaystyle=\left\\{\begin{array}[]{rc}-e^{-(x+d_{j}^{o})^{2}/2}H_{n}(x+d_{j}^{o})&-\infty<x\leq
0\\\\[2.0pt] (-1)^{n}e^{-(x-d_{j}^{o})^{2}/2}H_{n}(x-d_{j}^{o})&0\leq
x<\infty\end{array}\right.\ ,$ (3.12)
$\displaystyle\psi_{\text{S},j}^{(o)}(x,2n+1)$
$\displaystyle=\left\\{\begin{array}[]{rc}(-1)^{n}e^{-(x-d_{j}^{o})^{2}/2}H_{n}(x-d_{j}^{o})&-\infty<x\leq
0\\\\[2.0pt] -e^{-(x+d_{j}^{o})^{2}/2}H_{n}(x+d_{j}^{o})&0\leq
x<\infty\end{array}\right..$ (3.15)

Here $[a]$ denotes the greatest integer not exceeding $a$ and
$\langle\\!\langle n\rangle\\!\rangle=n+1$ for odd $n$ and $\langle\\!\langle
n\rangle\\!\rangle=n$ for even $n$. Likewise $\langle n\rangle=n+1$ for even
$n$ and $\langle n\rangle=n$ for odd $n$. Obviously these numbers are distinct
$\\{d_{j}^{e}\\}\cap\\{d_{j}^{o}\\}=\phi$. Here we list the explicit
expressions of the connection conditions (3.3) and the corresponding values of
$\\{d_{j}^{e}\\}$ and $\\{d_{j}^{o}\\}$ for $n$ upto 6.

Table 1: Even parameters $n$ | $-\bigl{(}H_{n}^{\prime}(d)-dH_{n}(d)\bigr{)}$ |  $d_{j}^{e}$: six digits
---|---|---
1 | $2(-1+d^{2})$ | 1
2 | $2d(-5+2d^{2})$ | 1.58114
3 | $4(3-9d^{2}+2d^{4})$ | 0.602114, 2.03407
4 | $4d(27-28d^{2}+4d^{4})$ | 1.07461, 2.41769
5 | $8(-15+75d^{2}-40d^{4}+4d^{6})$ | 0.476251, 1.47524, 2.75624
6 | $8d(-195+330d^{2}-108d^{4}+8d^{6})$ | 0.881604, 1.82861, 3.06251
Table 2: Odd parameters $n$ | $H_{n}(d)$ |  $d_{j}^{o}$: six digits
---|---|---
2 | $4(-1+2d^{2})$ | 0.707107
3 | $24d(-3+2d^{2})$ | 1.22474
4 | $96(3-12d^{2}+4d^{4})$ | 0.524648, 1.65068
5 | $960d(15-20d^{2}+4d^{4})$ | 0.958572, 2.02018
6 | $5760(-15+90d^{2}-60d^{4}+8d^{6})$ | 0.436077, 1.33585, 2.3506

It is straightforward to verify that the even connection condition (3.3) is
the same as (I.21) of Znojil’s paper [1] and the odd condition (3.3) agrees
with (I.24).

The upper bound of the zeros of the Hermite polynomial $H_{n}(x)$ is known
[17] (6.32.6),

$H_{n}(x_{j}^{(n)})=0,\qquad
x_{j}^{(n)}<\sqrt{2n+1}-\frac{c}{(2n+1)^{1/6}},\quad c=1.85575\ldots.$ (3.16)

This means that the eigenvalues of these explicitly known odd states are
greater than the corresponding $(d_{j}^{o})^{2}$,

$(d_{j}^{o})^{2}<2n+1,$ (3.17)

which is consistent with Remark 2.2. We do not know a corresponding bound for
the even sector, that is the zeros of $H_{n}^{\prime}(d)-dH_{n}(d)=0$.

We would not call these exactly solvable states QES (quasi-exactly solvable
states) [18]. A quantum mechanical system with a quasi-exactly solvable
potential has a finitely many exactly solvable states. In most cases these
states are related by $sl(2,R)$ algebra [19]. In the present case, a double
(2.3) or single well (2.6) potential with the parameter $d$ being the zeros of
(3.3) has only one exactly solvable state. This is a totally different
situation from QES.

We will come back to the topic of the integer eigenvalues in the second half
of the subsequent section. Before closing this section, let us emphasise the
fact that the above connection conditions (3.3), the Neumann and Dirichlet
b.c. including the fact that they are identical for the double and single well
systems, are intuitively quite easy to understand.

##  4 Non-polynomial exact eigenfunctions

###  4.1 Confluent hypergeometric functions

It is well known that the one dimensional Schrödinger equation with the
quadratic potential $x^{2}$ can be rewritten as an equation of the confluent
hypergeometric function $\varphi(z,E)$,

$\displaystyle-\frac{d^{2}\psi(x,E)}{dx^{2}}+x^{2}\psi(x,E)=E\psi(x,E),\quad\psi(x,E)=e^{-x^{2}/2}\varphi(z,E),\quad
z\stackrel{{\scriptstyle\text{def}}}{{=}}x^{2},$ $\displaystyle\hskip
85.35826pt\Longrightarrow
z\frac{d^{2}\varphi(z,E)}{dz^{2}}+(b-z)\frac{d\varphi(z,E)}{dz}-a\varphi(z,E)=0,$
(4.1) $\displaystyle\hskip 85.35826pta=\frac{(1-E)}{4},\quad b=\frac{1}{2}.$
(4.2)

The two fundamental solutions of the above Kummer’s differential equation
(4.1) are

$\displaystyle\varphi_{1}(z,E)$
$\displaystyle={}_{1}F_{1}(a,b\,;z)=\sum_{k=0}^{\infty}\frac{(a)_{k}}{(b)_{k}}\frac{z^{k}}{k!},$
(4.3) $\displaystyle\varphi_{2}(z,E)$
$\displaystyle=z^{1-b}{}_{1}F_{1}(a+1-b,2-b\,;z)=z^{1-b}\sum_{k=0}^{\infty}\frac{(a+1-b)_{k}}{(2-b)_{k}}\frac{z^{k}}{k!},$
(4.4)

in which $(a)_{n}$ is the shifted factorial,

$(a)_{n}\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{\Gamma(a+n)}{\Gamma(a)}=\prod_{k=0}^{n-1}(a+k)=a(a+1)\cdots(a+n-1).$
(4.5)

The well-known piecewise square integrable combination of the fundamental
solutions is

$U(a,b\,;z)\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{\Gamma(1-b)}{\Gamma(a-b+1)}{}_{1}F_{1}(a,b\,;z)+\frac{\Gamma(b-1)}{\Gamma(a)}z^{1-b}{}_{1}F_{1}(a-b+1,2-b\,;z).$
(4.6)

Since $b=\tfrac{1}{2}$,
$\Gamma(-\tfrac{1}{2})=-2\Gamma(\tfrac{1}{2})=-2\sqrt{\pi}$, we introduce

$\bar{U}(a,\tfrac{1}{2}\,;z)\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{1}{\Gamma(a+\tfrac{1}{2})}{}_{1}F_{1}(a,\tfrac{1}{2}\,;z)-\frac{2}{\Gamma(a)}z^{1/2}{}_{1}F_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;z),$
(4.7)

for simplicity of presentation. For the present case, corresponding to the two
types of $\sqrt{z}$ as a function of $x$, $\sqrt{z_{p}}=\pm(x+d)$,
$\sqrt{z_{m}}=\pm(x-d)$, we choose the branch of $\Psi_{p}(x,E)$ and
$\Psi_{m}(x,E)$ in such a way $\sqrt{z_{p}}>0$ and $\sqrt{z_{m}}>0$ at
infinity, so that the wavefunctions are damped at plus and minus infinity,

$\displaystyle-\frac{d^{2}\Psi_{p}(x,E)}{dx^{2}}$
$\displaystyle+(x+d)^{2}\Psi_{p}(x,E)=E\Psi_{p}(x,E),\quad
z_{p}\stackrel{{\scriptstyle\text{def}}}{{=}}(x+d)^{2},$
$\displaystyle\Psi_{p}(x,E)$
$\displaystyle=\left\\{\begin{array}[]{cc}\Psi_{p}^{(+)}(x,E)=e^{-z_{p}/2}\bar{U}_{p}^{(+)}(a,\tfrac{1}{2}\,;z_{p})&x\geq
0,\ \sqrt{z_{p}}=x+d,\\\\[3.0pt] \Psi_{p}^{(-)}(x,E)=\alpha
e^{-z_{p}/2}\bar{U}_{p}^{(-)}(a,\tfrac{1}{2}\,;z_{p})&x\leq 0,\ \
\sqrt{z_{p}}=-(x+d),\end{array}\right.,$ (4.10)
$\displaystyle-\frac{d^{2}\Psi_{m}(x,E)}{dx^{2}}$
$\displaystyle+(x-d)^{2}\Psi_{m}(x,E)=E\Psi_{m}(x,E),\quad
z_{m}\stackrel{{\scriptstyle\text{def}}}{{=}}(x-d)^{2},$
$\displaystyle\Psi_{m}(x,E)$
$\displaystyle=\left\\{\begin{array}[]{cc}\Psi_{m}^{(+)}(x,E)=e^{-z_{m}/2}\bar{U}_{m}^{(+)}(a,\tfrac{1}{2}\,;z_{m})&x\geq
0,\ \sqrt{z_{m}}=x-d,\\\\[3.0pt] \Psi_{m}^{(-)}(x,E)=\beta
e^{-z_{m}/2}\bar{U}_{m}^{(-)}(a,\tfrac{1}{2}\,;z_{m})&x\leq 0,\
\sqrt{z_{m}}=d-x,\end{array}\right..$ (4.13)

in which $\alpha$ and $\beta$ are constants to be determined later. Within the
interval $-d<x<d$, which include the connection point $x=0$, the wavefunctioms
$\psi_{\text{D}}^{(\pm)}(x,E)$ and $\psi_{\text{S}}^{(\pm)}(x,E)$ are
expressed by

$\displaystyle\psi_{\text{D}}^{(+)}(x,E)$
$\displaystyle=\Psi_{m}^{(+)}(x,E),\quad\psi_{\text{D}}^{(-)}(x,E)=\Psi_{p}^{(-)}(x,E),$
(4.14) $\displaystyle\psi_{\text{S}}^{(+)}(x,E)$
$\displaystyle=\Psi_{p}^{(+)}(x,E),\quad\psi_{\text{S}}^{(-)}(x,E)=\Psi_{m}^{(-)}(x,E).$
(4.15)

The boundary values of the double well wavefunctions are

$\displaystyle e^{d^{2}/2}\Psi_{\text{D}}^{(+)}(0,E)$
$\displaystyle=\frac{1}{\Gamma(a+\tfrac{1}{2})}{}_{1}F_{1}(a,\tfrac{1}{2}\,;d^{2})+\frac{2d}{\Gamma(a)}{}_{1}F_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;d^{2}),$
$\displaystyle e^{d^{2}/2}\Psi_{\text{D}}^{(+)^{\prime}}(0,E)$
$\displaystyle=\frac{d}{\Gamma(a+\tfrac{1}{2})}\left\\{{}_{1}F_{1}(a,\tfrac{1}{2}\,;d^{2})-2{}_{1}\dot{F}_{1}(a,\tfrac{1}{2}\,;d^{2})\right\\}$
$\displaystyle-\frac{2}{\Gamma(a)}\left\\{(1-d^{2})\cdot{}_{1}F_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;d^{2})+2d^{2}{}_{1}\dot{F}_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;d^{2})\right\\},$

in which

${}_{1}\dot{F}_{1}(a,b\,;z)\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{d\,{}_{1}F_{1}(a,b\,;z)}{dz}=\frac{a}{b}\cdot{}_{1}F_{1}(a+1,b+1\,;z).$
(4.16)

According to Theoren 2.1 the corresponding quantities for the single well
wavefunctions are obtained by changing $d$ to $-d$, These lead to the
following theorem.

###### Theorem 4.1

The connection conditions for the double and single well wavefunctions are

$\displaystyle{\rm D,even:}$
$\displaystyle\quad\alpha=1,\quad\frac{d}{\Gamma(a+\tfrac{1}{2})}\left\\{{}_{1}F_{1}(a,\tfrac{1}{2}\,;d^{2})-2\,{}_{1}\dot{F}_{1}(a,\tfrac{1}{2}\,;d^{2})\right\\}$
$\displaystyle\qquad\qquad-\frac{2}{\Gamma(a)}\left\\{(1-d^{2})\cdot{}_{1}F_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;d^{2})+2d^{2}{}_{1}\dot{F}_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;d^{2})\right\\}=0,$
(4.17) $\displaystyle{\rm D,odd:}$
$\displaystyle\quad\alpha=-1,\quad\frac{1}{\Gamma(a+\tfrac{1}{2})}{}_{1}F_{1}(a,\tfrac{1}{2}\,;d^{2})+\frac{2d}{\Gamma(a)}{}_{1}F_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;d^{2})=0.$
(4.18) $\displaystyle{\rm S,even:}$
$\displaystyle\quad\beta=1,\quad\frac{d}{\Gamma(a+\tfrac{1}{2})}\left\\{{}_{1}F_{1}(a,\tfrac{1}{2}\,;d^{2})-2\,{}_{1}\dot{F}_{1}(a,\tfrac{1}{2}\,;d^{2})\right\\}$
$\displaystyle\qquad\qquad+\frac{2}{\Gamma(a)}\left\\{(1-d^{2})\cdot{}_{1}F_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;d^{2})+2d^{2}{}_{1}\dot{F}_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;d^{2})\right\\}=0,$
(4.19) $\displaystyle{\rm S,odd:}$
$\displaystyle\quad\beta=-1,\quad\frac{1}{\Gamma(a+\tfrac{1}{2})}{}_{1}F_{1}(a,\tfrac{1}{2}\,;d^{2})-\frac{2d}{\Gamma(a)}{}_{1}F_{1}(a+\tfrac{1}{2},\tfrac{3}{2}\,;d^{2})=0.$
(4.20)

The zeros of these equations provide the eigenvalues $\\{E_{n}\\}$ of the
$V_{\rm D}$ and $V_{\rm S}$ systems in the even and odd sectors. The
corresponding eigenfunctions are those listed in (4.14) and (4.15).

###### Remark 4.2

Roughly speaking, the functions in the $V_{\rm S}$ system (4.19),(4.20) are
the other branches of the corresponding $\dot{U}$ and $U$ functions in the
$V_{\rm D}$ system (4.17),(4.18) and vice versa.

We show the seven lowest eigenvalues, 4 from the even sector and 3 from the
odd sector, of the double (Table 3) and single (Table 4) well potentials for a
selected small values of $d$. For each value of $d$, the eigenvalues of the
odd sectors are greater than the corresponding ones in the even sectors, as
dictated by the oscillation theorem. Each specific eigenvalue of the
$V_{\text{S}}$ system increases monotonically with the parameter $d$. The
$d$-dependence of the states of the $V_{\text{D}}$ system is quite
interesting. As $d$ increases above 3, the split between the even and odd
sectors diminishes appreciably. One could say for almost safely that when $d$
increases the tunneling effects of the lowest $n$ ($n<d^{2}/2$) states in each
sector disappear. This would mean that for large $d$, the eigenvalues of
$V_{\text{D}}(x)$ approach to $E_{n}^{e}=1+2n-\epsilon_{n}$,
$E_{n}^{o}=1+2n+\epsilon_{n}^{\prime}$ with very small
$\epsilon_{n},\epsilon_{n}^{\prime}>0$. It is a good challenge to find out the
asymptotic behaviours of $E_{n}^{e}(d)$ and $E_{n}^{o}(d)$ of the
$V_{\text{S}}(x)$ system.

Table 3: 7 lowest eigenvalues of $V_{\text{D}}$ d | $E_{0}^{e}$ | $E_{0}^{o}$ | $E_{1}^{e}$ | $E_{1}^{o}$ | $E_{2}^{e}$ | $E_{2}^{0}$ | $E_{3}^{e}$
---|---|---|---|---|---|---|---
0 | 1 | 1 | 3 | 3 | 5 | 5 | 7
1/10 | 0.895426 | 2.78209 | 4.72612 | 6.66950 | 8.62731 | 10.5849 | 12.5497
1/4 | 0.768973 | 2.48392 | 4.34603 | 6.20358 | 8.09868 | 9.99237 | 11.9046
1/2 | 0.635529 | 2.06077 | 3.79417 | 5.50548 | 7.29817 | 9.08421 | 10.9098
3/4 | 0.590301 | 1.72471 | 3.34471 | 4.90343 | 6.59770 | 8.27404 | 10.0146
1 | 0.618919 | 1.46847 | 3 | 4.39493 | 5.99720 | 7.56038 | 9.21846
3/2 | 0.801494 | 1.15748 | 2.64868 | 3.64627 | 5.10400 | 6.41679 | 7.92382
2 | 0.951419 | 1.03576 | 2.73504 | 3.22301 | 4.67082 | 5.64089 | 7.04349
3 | 0.999551 | 1.00039 | 2.99252 | 3.00604 | 4.94552 | 5.03982 | 6.79866
4 | 0.999999 | 1.000000 | 2.99998 | 3.00001 | 4.99977 | 5.00020 | 6.99802
Table 4: 7 lowest eigenvalues of $V_{\text{S}}$ d | $E_{0}^{e}$ | $E_{0}^{o}$ | $E_{1}^{e}$ | $E_{1}^{o}$ | $E_{2}^{e}$ | $E_{2}^{0}$ | $E_{3}^{e}$
---|---|---|---|---|---|---|---
1/10 | 1.12121 | 3.23353 | 5.29034 | 7.34657 | 9.38899 | 11.4312 | 13.4665
1/4 | 1.33487 | 3.61368 | 5.75688 | 7.89681 | 10.0032 | 12.1086 | 14.1970
1/2 | 1.77790 | 4.32871 | 6.61797 | 8.89589 | 11.1096 | 13.3194 | 15.4967
3/4 | 2.33218 | 5.14812 | 7.58472 | 9.99898 | 12.3203 | 14.6339 | 16.9002
1 | 3 | 6.07439 | 8.65856 | 11.2076 | 13.6366 | 16.0533 | 18.4086
3/2 | 4.68276 | 8.25537 | 11.1329 | 13.9472 | 16.5907 | 19.2113 | 21.7441
2 | 6.83597 | 10.8843 | 14.0506 | 17.1244 | 19.9803 | 22.8017 | 25.5108
5/2 | 9.46595 | 13.9704 | 17.4196 | 20.7471 | 23.8127 | 26.8318 | 29.7154

A few remarks on the numerical calculations of the eigenvalues. For
$z\to+\infty$, $U(a,b\,;z)$ (4.6) behaves asymptotically $\sim z^{-a}$. The
expressions in Theorem 4.1 (4.17)–(4.20) increase drastically $\sim d^{E/2}$
as $d$ and $E$ increase. With certain reduction factors the zeros (the
eigenvalues) of these expressions can be determined as precisely as wanted for
a specified parameter $d$. This preciseness propagates to the preciseness of
the eigenfunctions. This is why the systems with $V_{\text{D}}(x)$ and
$V_{\text{S}}(x)$ belong to the category of potentials of non-polynomial exact
solvability [1].

In order to share the vidid images of polynomial and non-polynomial type
eigenfunctions of the low lying eigenstates we present four figures. The two
lowest eigenfunctions of the $V_{\text{S}}(x)$ potential with $d=1$ are shown
in Fig.2, the ground state $E_{0}^{e}=3$ and the first excited state
$E_{0}^{o}=6.07439$. For comparison, we show the three lowest energy states of
the $V_{\text{D}}$ system with $d=1$, $E_{0}^{e}=0.618919$ $E_{0}^{o}=1.46846$
and $E_{1}^{e}=3$ in Fig.2. The two lowest ones are not of the polynomial
type. Two $E=5$ eigenstates in the odd sector with $d=1/\sqrt{2}$ are shown in
Fig.4 and those in the even sector with $d=\sqrt{5/2}$ are displayed in Fig.4.
Those of the $V_{\text{S}}$ system have red lines and those in $V_{\text{D}}$
blue.

Figure 1: $d=1$, $E=3,\ 6.07$, S |  Figure 2: $d=1$, $E=0.619,1.468,\ 3$, D
---|---
Figure 3: $E=5,d=1/\sqrt{2}$ odd |  Figure 4: $E=5,d=\sqrt{5/2}$ even
---|---

Fig.2 shows the effect of splitting due to tunneling. Let us introduce a pair
of potentials restricted to the right and left half line by an infinite
impenetrable wall at the origin,

$V_{\rm DR}(x)=\left\\{\begin{array}[]{cc}(x-d)^{2}&x>0\\\
+\infty&x=0\end{array}\right.,\qquad V_{\rm
DL}(x)=\left\\{\begin{array}[]{cc}+\infty&x=0\\\ (x+d)^{2}&x<0\\\
\end{array}\right.,\qquad d>0.$ (4.21)

They have the same set of eigenvalues. The eigenfunctions are restricted to
the right and left line satisfying the Dirichlet b.c. at the origin. The
function vanishing on the left half line and take the odd line of Fig.2 at
$x>0$ is the ground state eigenfunction of $V_{\rm DR}(x)$ with $E=1.46846>1$,
$d=1$, since $V_{\rm DR}(x)|_{d=1}>(x-1)^{2}$ on the left half line. Likewise
the vanishing on the right half line and take the odd line of Fig.2 at $x<0$
is the ground state eigenfunction of $V_{\rm DL}(x)$ with $E=1.46846$, $d=1$.
When the infinite barrier is removed and $V_{\rm DR}(x)$ and $V_{\rm DL}(x)$
merge to become $V_{\text{D}}(x)$, the odd combination of these states becomes
the first excited states with the same eigenvalue $E=1.46846$. The removal of
the infinite barrier at the origin could be rephrased as the addition of an
infinitely deep and narrow well at the origin. This has no effect on the odd
combination of the original ground state eigenfunctions as they vanish at the
origin. However, this has the effect of increasing the even combination of the
wavefunctions at the origin to the point of satisfying the Neumann b.c. and
thus decreasing the eigenvalue. This mechanism applies to all the
eigenfunctions of the $V_{\rm DR}(x)$ and $V_{\rm DL}(x)$ systems. Therefore,
the splitting of the eigenlevels in the $V_{\text{D}}(x)$ potential means
pushing down the even states whereas the odd states stay at the original
eigenvalues of $V_{\rm DL}$ and $V_{\rm DR}$. The situation is essentially the
same for the $V_{\text{S}}(x)$ potential. Instead of (4.21)

$V_{\rm SR}(x)=\left\\{\begin{array}[]{cc}(x+d)^{2}&x>0\\\
+\infty&x=0\end{array}\right.,\qquad V_{\rm
SL}(x)=\left\\{\begin{array}[]{cc}+\infty&x=0\\\ (x-d)^{2}&x<0\\\
\end{array}\right.,\qquad d>0.$ (4.22)

are the potentials to be considered.

###  4.2 Revisiting polynomial type solutions

Since the parameter $a$ in the Kummer’s differential equation (4.1) is
$a=(1-E)/4$ (4.2), various quantities and expressions in the previous
subsection simplify a lot for the odd integer eigenvalues $E$,

$E=4n+1,\ 4n+3,\ \Longleftrightarrow a=-n,\ a+\tfrac{1}{2}=-n,\,\qquad
n\in\mathbb{Z}_{\geq 0}.$ (4.23)

For these values the confluent hypergeometric functions ${}_{1}F_{1}(a,b\,;z)$
and ${}_{1}F_{1}(a+\tfrac{1}{2},b\,;z)$ terminate and become polynomials in
$z$. Many expressions in §4.1 reduce to those in §3. Here we list them for
comparison.

#### (i) $E=4n+1$

This means

$a=-n\ \Rightarrow\frac{1}{\Gamma(a)}=\frac{1}{\Gamma(-n)}=0,\quad
U(-n,\tfrac{1}{2}\,;z)=\frac{\Gamma(\tfrac{1}{2})}{\Gamma(-n+\tfrac{1}{2})}{}_{1}F_{1}(-n,\tfrac{1}{2}\,;z).$

Here ${}_{1}F_{1}(-n,\tfrac{1}{2}\,;x^{2})$ is a degree $n$ polynomial in
$x^{2}$,

${}_{1}F_{1}(-n,\tfrac{1}{2}\,;x^{2})=\frac{n!}{(\tfrac{1}{2})_{n}}L_{n}^{(-\tfrac{1}{2})}(x^{2}),\qquad
H_{2n}(x)=(-1)^{n}n!2^{2n}L_{n}^{(-\tfrac{1}{2})}(x^{2}),$ (4.24)

in which $L_{n}^{(-\tfrac{1}{2})}(x)$ is the Laguerre polynomial.

#### (ii) $E=4n+3$

This means

$a+\tfrac{1}{2}=-n\
\Rightarrow\frac{1}{\Gamma(a+\tfrac{1}{2})}=\frac{1}{\Gamma(-n)}=0,\quad
U(-n-\tfrac{1}{2},\tfrac{1}{2}\,;z)=\frac{\Gamma(-\tfrac{1}{2})}{\Gamma(-n-\tfrac{1}{2})}\sqrt{z}\,{}_{1}F_{1}(-n,\tfrac{3}{2}\,;z).$

Here ${}_{1}F_{1}(-n,\tfrac{3}{2}\,;x^{2})$ is a degree $n$ polynomial in
$x^{2}$,

${}_{1}F_{1}(-n,\tfrac{3}{2}\,;x^{2})=\frac{n!}{(\tfrac{3}{2})_{n}}L_{n}^{(\tfrac{1}{2})}(x^{2}),\qquad
H_{2n+1}(x)=(-1)^{n}n!2^{2n+1}xL_{n}^{(\tfrac{1}{2})}(x^{2}),$ (4.25)

By using these relations, one can easily verify that (4.17)–(4.20) reduce to
(3.3).

##  5 Summary and Comments

Some basic facts, expressions and numbers related with the eigenvalues and
eigenfunctions of the piecewise analytic and exactly solvable potentials
$V_{\text{D}}(x)$ and $V_{\text{S}}(x)$ are explored. For the applications,
the norms of some of the lower lying eigenstates would be needed. This would
require a substantial work.

It is well known that a piecewise linear potential $V_{\text{L}}(x)=g^{3}|x|$
is exactly solvable. It is expected that the potentials

$V_{\rm LD}(x)=\text{min}[V_{\text{L}}(x+d),V_{\text{L}}(x-d)],\qquad V_{\rm
LS}(x)=\text{max}[V_{\text{L}}(x+d),V_{\text{L}}(x-d)],$ (5.1)

would be exactly solvable by the same procedures as those used in this paper.
More interesting would be the double and single well versions of a Krein-Adler
deformation [13, 14, 15] of the harmonic oscillator potential,

$\displaystyle\hskip 113.81102ptV_{\rm
KA}(x)=x^{2}+3+\frac{32x^{2}}{(2x^{2}+1)^{2}}-\frac{8}{2x^{2}+1},$ (5.2)
$\displaystyle V_{\rm KAD}(x)=\text{min}[V_{\rm KA}(x+d),V_{\rm
KA}(x-d)],\quad V_{\rm KAS}(x)=\text{max}[V_{\rm KA}(x+d),V_{\rm
KA}(x-d)],\quad d>0.$ (5.3)

Figure 5: $V_{\rm KA}(x)$ |  Figure 6: $V_{\rm KAD}(x)$ blue, $V_{\rm KADS}(x)$ red, $d=1$
---|---

Dubov et al [16] introduced the exactly solvable $V_{\rm KA}(x)$, which has
the regular singular points at $x=\pm\frac{i}{\sqrt{2}}$ with the
characteristic exponents $(-1,2)$. The complete set of the eigenvalues and
eigenfunctions are

${\psi}_{\rm{KA},n}(x)=\frac{e^{-x^{2}/2}\text{W}[H_{1},H_{2},H_{n}](x)}{4(2x^{2}+1)},\quad\mathcal{E}(n)=2n,\quad
n\in\mathbb{Z}_{\geq 0}\backslash\\{1,2\\},$ (5.4)

in which

$\displaystyle\text{W}\,[f_{1},\ldots,f_{m}](x)\stackrel{{\scriptstyle\text{def}}}{{=}}\det\Bigl{(}\frac{d^{j-1}f_{k}(x)}{dx^{j-1}}\Bigr{)}_{1\leq
j,k\leq m},$ (5.5)

is the Wronskian of functions $\\{f_{1},\ldots,f_{m}\\}$. At least we can find
the polynomial type eigenfunctions of $V_{\rm KAD}(x)$ and $V_{\rm KAS}(x)$
quite easily. If these $V_{\rm KAD}(x)$ and $V_{\rm KAS}(x)$ turn out to be
exactly solvable, we would have an infinitely many similar potentials by the
Krein-Adler prescriptions [15].

## Acknowledgements

R. S. thanks Milosh Znojil for sending [1] just after publication.

Informed Consent Statement: Not applicable.
Data Availability Statement: No new data created
Conflicts of Interest: There is no conflict of interests.

## References

  * [1] M. Znojil, “Displaced harmonic oscillator $V\sim\text{min}[(x+d)^{2},(x-d)^{2}]$ as a benchmark double well potential,” Quantum Rep. 4 (2022) 309-323.
  * [2] R. Sasaki and M. Znojil, “One-dimensional Schrödinger equation with non-analytic potential $V(x)=-g^{2}\exp(-|x|)$ and its exact Bessel-function solvability,” J. Phys. A49 (2016) 445303 (12pp), arXiv:1605.07310[math-ph].
  * [3] M. Znojil, “Symmetrized exponential oscillator,” Mod. Phys. Lett. A31 (2016) 1650195;
R. Sasaki, “Confining non-analytic exponential potential $V(x)=g^{2}{\rm
exp}(2|x|)$ and its exact Bessel-function solvability. arXiv:1611.02467.

  * [4] M. Znojil, “Morse potential, symmetric Morse potential and bracketed bound-state energies,” Mod. Phys. Lett. A31 (2016) 1650088;
R. Sasaki, “Symmetric Morse potential is exactly solvable,” arXiv:1611.05952.

  * [5] L. Infeld and T. E. Hull, “The factorization method,” Rev. Mod. Phys. 23 (1951) 21-68.
  * [6] F. Cooper, A. Khare and U. Sukhatme, “Supersymmetry and quantum mechanics,” Phys. Rep. 251 (1995) 267-385.
  * [7] L. E. Gendenshtein, “Derivation of exact spectra of the Schroedinger equation by means of supersymmetry,” JETP Lett. 38 (1983) 356-359.
  * [8] D. Gomez-Ullate, N. Kamran and R. Milson, “An extended class of orthogonal polynomials defined by a Sturm-Liouville problem,” J. Math. Anal. Appl. 359 (2009) 352-367, arXiv:0807.3939[math-ph]; “An extension of Bochner’s problem: exceptional invariant sub-spaces,” J. Approx. Theory 162 (2010) 987-1006, arXiv:0805.3376[math-ph].
  * [9] C. Quesne, “Exceptional orthogonal polynomials, exactly solvable potentials and supersymmetry,” J. Phys. A: Math. Theor. 41 (2008) 392001 (6 pp), arXiv:0807.4087[quant-ph].
  * [10] S. Odake and R. Sasaki, “Infinitely many shape invariant potentials and new orthogonal polynomials,” Phys. Lett. B679 (2009) 414-417, arXiv:0906.0142[math-ph].
  * [11] S. Odake and R. Sasaki, “Exactly solvable quantum mechanics and infinite families of multi-indexed orthogonal polynomials,” Phys. Lett. B702 (2011) 164-170, arXiv:1105.0508[math-ph].
  * [12] S. Odake and R. Sasaki, “Non-polynomial extensions of solvable potentials á la Abraham-Moses,” J. Math. Phys. 54 (2013) 102106 (19pp), arXiv:1307.0910[math-ph].
  * [13] M. G. Krein, “On continuous analogue of a formula of Christoffel from the theory of orthogonal polynomials,” (Russian) Doklady Acad. Nauk. CCCP, 113 (1957) 970-973.
  * [14] V. É. Adler, “A modification of Crum’s method,” Theor. Math. Phys. 101 (1994) 1381-1386.
  * [15] S. Odake and R. Sasaki, “Krein-Adler transformations for shape-invariant potentials and pseudo virtual states,” J. Phys. A 46 (2013) 245201 (24pp) arXiv:1212.6595[math-ph]
  * [16] S. Yu. Dubov, V. M. Eleonskiĭ and N. E. Kulagin, “Equidistant spectra of anharmonic oscillators,” Soviet Phys. JETP 75 (1992) 446-451; “Equidistant spectra of anharmonic oscillators,” Chaos 4 (1994) 47-53.
  * [17] G. Szegö, Orthogonal Polynomials, Amer. Math. Soc. Colloquium Publications Vol. 23 (Amer. Math. Soc., New York, 1939).
  * [18] A. G. Ushveridze, Quasi-Exactly Solvable Models in Quantum Mechanics, IOPP: Bristol, UK, (1994).
  * [19] A. V. Turbiner, “Quasi-exactly solvable problems and sl(2) algebra,” Commun. Math. Phys. 118 (1988) 467-474.


# Approximated Prompt Tuning for Vision-Language Pre-trained Models

Qiong Wu12, Shubin Huang1, Yiyi Zhou12, Pingyang Dai1, Annan Shu3, Guannan
Jiang3, Rongrong Ji12
1 Key Laboratory of Multimedia Trusted Perception and Efficient Computing,
Ministry of Education of China, Xiamen University, 361005, P.R. China.
2 Institute of Artificial Intelligence, Xiamen University, 361005, P.R. China.
3 Intelligent Manufacturing Department, Contemporary Amperex Technology Co.
Limited
{qiong, shubinhuang}stu.xmu.edu.cn, {zhouyiyi, pydai}xmu.edu.cn,
{shuan01, jianggn}catl.com, rrjixmu.edu.cn

###### Abstract

Prompt tuning is a parameter-efficient way to deploy large-scale pre-trained
models to downstream tasks by adding task-specific tokens. In terms of vision-
language pre-trained (VLP) models, prompt tuning often requires a large number
of learnable tokens to bridge the gap between the pre-training and downstream
tasks, which greatly exacerbates the already high computational overhead. In
this paper, we revisit the principle of prompt tuning for Transformer-based
VLP models and reveal that the impact of soft prompt tokens can be actually
approximated via independent information diffusion steps, thereby avoiding the
expensive global attention modeling and reducing the computational complexity
to a large extent. Based on this finding, we propose a novel _Approximated
Prompt Tuning_ (APT) approach towards efficient VL transfer learning. To
validate APT, we apply it to two representative VLP models, namely ViLT and
METER, and conduct extensive experiments on a bunch of downstream tasks.
Meanwhile, the generalization of APT is also validated on CLIP for image
classification. The experimental results not only show the superior
performance gains and computation efficiency of APT against the conventional
prompt tuning methods, e.g., $+6.6\%$ accuracy and $-64.62\%$ additional
computation overhead on METER, but also confirm its merits over other
parameter-efficient transfer learning approaches111Our code is given in
supplementary materials and will be publicly released after acceptance..

##  1 Introduction

Prompt tuning [4, 42, 32, 8, 44, 23, 60, 59, 33] is a parameter-efficient way
to adapt large-scale pre-trained models to downstream tasks. It inserts
multiple prompt tokens into the input sequence to unify the pre-trained and
downstream data distributions [42, 44], thereby avoiding the expensive full
fine-tune of pre-trained models. Recent advances [32, 23, 60, 59] resort to
trainable tokens to replace the hand-craft ones to improve the adaption on
downstream tasks, which is named _soft prompt tuning_ [32, 33]. In terms of
the way of insertion, soft prompt tuning can be further categorized into _deep
prompt tuning_ [23] and _shallow prompt tuning_ [32], respectively. Currently,
prompt tuning has achieved great success in _natural language processiong_
(NLP) [42, 32, 28] and also been recently applied to shallow-fusion based
vision-language pre-trained (VLP) models like CLIP [44] for image
classification [60, 59].

Figure 1:  (a) The relative performance of soft prompt tuning to full fine-
tune _w.r.t._ the prompt number on VL, NLP and CV tasks 333These results are
from VPT [23] on VTAB datasets (CV), Prefix [32] on DART (NLP) and deep
prompting on VQA (V-L) . VL tasks often require more prompt tokens than
single-modal ones. (b) The comparison between our APT and soft prompt tuning
in terms of information diffusion efficiency, _i.e._ , the attention weights
to the input sequence. The adaption efficiency of soft prompting is much less
efficient on VL tasks.

However, it is still more challenging to apply prompt tuning to common VLP
models, _e.g._ , ViLT [24] and METER [11], than the language ones [32]. Above
all, most VL tasks are greatly different from the VL pre-training objectives,
such as _visual question answering_ (VQA) [14] and _image captioning_ [5].
Thus, hand-craft prompts often fail to adapt to these tasks. Soft prompt
tuning is applicable to VLP models [23]. But due to the large task gap and the
increase of modalities, more learnable tokens are often required to adapt VLP
models to downstream tasks, as shown in Fig. 3-a. In addition, we also notice
that even with a bunch of tokens, soft prompt tuning still has limited impacts
on the input sequence, _i.e._ , attention weights, leading to a sub-optimal
adaption, as shown in Fig. 3-b. Considering that these tokens are often
involved in self-attention, of which computation is quadratic to the input
length [10], this inefficient adaption will significantly increase the already
high computational overhead of VLP models.

By revisiting the principle of prompt tuning, we find that there exists a
potential solution for efficient VL adaption. Particularly, prompt tuning aims
to use additional tokens to influence the input sequence, so as to minimize
the gap between pre-training and downstream tasks [8, 42]. In terms of soft
prompt tuning, the tokens are usually inserted into the self-attention layers
of VLP models [59, 32, 23]. Via analyzing self-attention, we observe that the
obtained attention weight matrix can be actually divided into four sub-parts
as shown in Fig. 2-a. Here, we call them _input-only_ , _input2prompt_ ,
_prompt2input_ and _prompt-only_ attention matrices, respectively. Under the
setting of layer-wise prompt tuning [23], the computations of _prompt2input_
and _prompt-only_ can be indeed skipped and will not affect the prompt tuning
of next layer. And the _input-only_ is the default operation of pre-trained
models that cannot be changed. In this case, the key to improving prompt
tuning lies on the _input2prompt_ , which is essentially an information
diffusion step from the prompt tokens to the input sequence under the
perspective of graph theory [61]. However, we find that its functionality can
be actually approximated via a more effective process independent to global
attention, thereby improving the efficiency of VL adaption.

Figure 2: The illustrations of global self-attention matrices with and without
APT. (a) is the attention matrices of common prompt tuning. In (b), the
prompt-input and prompt-only parts are removed, and the input-prompt attention
is approximated by APT.

Motivated by this observation, we propose a novel _approximated prompt tuning_
(APT) approach for VLP models in this paper. Similar to existing prompt tuning
methods [23, 32], APT inserts a set of learnable tokens into each self-
attention layer of the VLP model. As shown in Fig. 2-b, a key difference is
that we separate these tokens from the expensive global self-attention and
approximate their effect independently by aggregating the prompt tokens with
low-rank transformations. In this way, the proposed APT method can effectively
diffuse more information from prompt tokens to the input sequence while
avoiding the expensive global self-attention, as shown in Fig. 3-b.

To validate APT, we apply it to two deep-fusion based VLP models, namely ViLT
[24] and METER [11], on three VL benchmarks including VQA [2], NLVR2 [50] and
Flickr30K [43]. In addition, we also validate APT on CLIP [44] and conduct
experiments on 11 benchmark datasets of the _base-to-new_ classification task
[60]. The experimental results not only confirm the merits of APT over the
conventional prompt tunning methods [32, 23, 60, 59], but also yield better
performance than most parameter-efficient transfer learning (PETL) approaches
[23, 19, 52, 16], _e.g._ $70.94\%$ on VQA [52] for ViLT and $80.97\%$ on NLVR2
[16] for METER. Its generalization is also confirmed on CLIP, _e.g._ ,
$77.68\%$ on image classification.

Overall, our contributions are three-fold:

  * •

We identify the key challenges of prompt tuning on common VLP models, _e.g._
ViLT [24] and METER [11], which are excessive computation overhead and low
prompt tuning efficiency.

  * •

We propose a novel _approximated prompt tuning_ (APT) method for both
parameter- and computation-efficient prompt tuning, which approximates the
influence of prompt tokens to the input sequence via independent prompt
aggregation steps.

  * •

The proposed APT not only outperforms existing prompt tuning methods but also
achieves better performance than other PETL approaches on 2 VLP models and 4
VL tasks. Its generalization is also validated on CLIP for image
classification tasks.

##  2 Related Work

###  2.1 Vision-Language Pre-training

Following the great success in natural language processing (NLP) [32, 28],
large-scale pre-training has also become a general solution for vision-
language study. Similar to NLP pre-training paradigms [10, 4, 45, 29, 34, 26],
VL pre-training also apply generative prediction tasks, _e.g._ _masked
language modeling_ (MLM) and _maksed image modeling_ (MIM), to achieve self-
supervised learning on massive cross-modal data. A key difference is that VLP
models usually require two encoders to process image and language information,
_e.g._ , BERT [10] and Faster-RCNN [47], and their Transformer-based backbone
networks are not only used for high-level representation learning, but also
for cross-modal deep fusion and interaction [35, 54, 55, 31, 49, 6, 21, 34,
24, 11, 20]. Specifically, ViLBERT [35] and LXMERT [54] introduce two
independent Transformer-based branches [55] for region and text features,
based on which another Transformer block is used for cross-modal interaction.
With a simpler framework, VisualBERT [31], VL-BERT [49] and UNITER [6]
directly feed both region and text features into a single Transformer to
jointly learn two modalities. To break the limitation of object detection
backbones, Pixel-BERT [21], CLIP-ViL [34], ViLT [24] and METER [11] directly
use convolutional neural networks (CNNs) as the image encoder to extract the
grid visual features for multi-modal pre-training. Further, SOHO [20] replaces
the visual feature extraction by applying a learned vision dictionary to
discretize grid features. More recently, ViLT [24] simplifies the VLP models
to an end-to-end structure, where the word embeddings and image patches are
directly learned by the deep Transformer network. In addition to the above
common VLP models, another important trend in VL pre-training is applying
cross-modal contrastive learning for vision-language alignment, _e.g._ CLIP
[44]. This type of methods [44, 22, 27] often have a shallow fusion layer upon
the encoders of two modalities, which are mainly designed for cross-modal
retrieval or prompt-based image classification [60, 59]. Although there are
existing works [60, 59] to apply prompt tuning on CLIP, there is still a lack
of in-depth explorations for VL pre-training in a larger scope. This paper
presents the first attempt at prompt tuning on common deep-fusion based VLP
models in addition to CLIP.

Figure 3:  Illustration of the proposed _approximated prompt tuning_ (APT).
(a) APT approximates the attention weights from prompt tokens to input
sequence in a low-rank subspace ($\mathbf{W}_{t}$). (b) APT is independent to
the original self-attention, for better efficiency.

###  2.2 Prompt Tuning

Prompt tuning [4, 42, 32, 8, 44, 23, 60, 59, 33] is a parameter-efficient way
to adapt pre-trained models to downstream tasks, which is originally designed
for large pre-trained language models such as GPT-3 [4]. Concretely, for hand-
crafted prompts [42, 44], also coined as _hard prompt tuning_ , it often
inserts a pre-defined prompt phrase into the input sequences, thus reminding
the model of pre-trained knowledge, _e.g._ , “ _This is a picture of_ [X]”.
However, hard prompt tuning heavily relies on manual design, and a slight
different template may lead to a drastic change in performance [33]. To
overcome this issue, soft prompt tuning [32, 23, 60] is proposed to
automatically learn prompt tokens, which are often learnable and optimized via
downstream task adaption. In terms of the prompt placement, soft prompt tuning
can be further divided into two patterns, _i.e._ , the shallow [32] and the
deep [23] ones. Shallow prompt tuning methods [28, 32] only expand the input
sequence with trainable vectors at the first layer. In contrast, deep prompt
tuning methods [23] expand the input sequence between any two layers with
trainable tokens. In the field of vision-language learning, some recent works
[60, 59] start applying prompt tuning to shallow-fusion based VLP models like
CLIP for image classification. However, these methods mainly focus on
inserting the prompt tokens to make the language expression match the image
features. This paper presents the first attempt of bridging the gap between
common VL tasks and the pre-training objectives via soft prompt tuning.

###  2.3 Parameter Efficient Transfer Learning

Parameter Efficient Transfer Learning (PETL) [18, 37, 36, 13, 58, 52, 19, 15,
53, 16, 39, 51, 57] aims to update a small number of parameters to approach
the fully-tuned performance on downstream tasks. In addition to prompt tuning,
a common paradigm is the adapter-based methods [18, 37, 36, 13, 58, 52],
called _Adapter_ for short. Compared with prompt tuning, which inserts
additional tokens to bridge the gap between pre-training and downstream data
distributions, Adapter often inserts lightweight feed-forward networks into
the pre-trained model, thereby projecting hidden features onto downstream data
space [18, 52]. To avoid the additional computation overhead during inference,
Hu _et al._ propose a low-rank adaption (LoRA) [19] method, where the low-rank
weight matrices can be re-parameterized into the model after training.
Furthermore, some recent trials [16, 39] combine prompt tuning, Adapter and
LoRA into one adaption scheme, thereby improving performance. Besides, Side
tuning [57], which runs in parallel with the pre-trained module to adapt
downstream tasks, can be used with non-network sources of information. In VL
tasks, LST [51] stacks the outputs of the pre-trained module in a parallel way
and directly predict the results without feedback to save the memory cost in
training. In the field of vision-language learning, VL-adapter [52] insert
low-dimensional networks into a pre-trained language model to adapt to common
VL tasks [5, 14, 50]. Its key difference to this paper is that the language
model is not pre-trained on massive VL data, making its results lack
sufficient generalization for common VLP models.

##  3 Preliminary

Before introducing our approach, we first recap the principle of prompt tuning
for VLP models. Concretely, given a pre-trained vision-language (VLP) model,
denoted as $G(\cdot)$, and the image-text example of the downstream task,
denoted as $(I,T)$, the target of prompt tuning is to minimize the adaption
loss with a set of prompt tokens $\mathbf{P}\in\mathbb{R}^{p\times d}$:

$\displaystyle\operatorname*{argmin}_{\mathbf{P}}\mathcal{L}\big{(}G(I,T,P|\theta^{+})\big{)},$
(1)

where $\theta^{+}$ is the pre-trained weights of $G$ and will be fixed during
prompt tuning444In most case, the classifier will be trained for a specific
task. $\mathcal{L}$ is the objective function of the downstream task.

Considering that the parameters are fixed during adaption, the features of the
input sequence are hard to update for the downstream task. In this case,
prompt tokens $\mathbf{P}$ are often used in the self-attention of VLP models
for diffusing task-related information to the input sequence
$\mathbf{X}\in\mathbb{R}^{n\times d}$:

$\displaystyle=SA(\mathbf{X}||\mathbf{P}),$ (2)

where $SA(\cdot)$ represents the self-attention module. $\mathbf{X}^{\prime}$
and $\mathbf{P}^{\prime}$ are the corresponding outputs of $\mathbf{X}$ and
$\mathbf{P}$, respectively.

In particular, X’ and P’ are obtained by

$\displaystyle\mathbf{X}^{\prime}$
$\displaystyle=\mathbf{A}_{I}\mathbf{X}\mathbf{W}_{v}+\mathbf{A}_{IP}\mathbf{P}\mathbf{W}_{v},$
(3) $\displaystyle\mathbf{P}^{\prime}$
$\displaystyle=\mathbf{A}_{PI}\mathbf{X}\mathbf{W}_{v}+\mathbf{A}_{P}\mathbf{P}\mathbf{W}_{v},$

where $\mathbf{A}_{I}$, $\mathbf{A}_{IP}$, $\mathbf{A}_{PI}$ and
$\mathbf{A}_{P}$ are the sub-attention matrices, corresponding to the _input-
only_ , _input2prompt_ , _prompt2input_ and _prompt-only_ parts described in
introduction and shown in Fig. 2. $\mathbf{W}_{q}$, $\mathbf{W}_{k}$ and
$\mathbf{W}_{v}$ are the weight matrices of $Q$, $K$, $V$ projections in $SA$.
Under the layer-wise setting [23], the prompt tokens are initialized for each
layer and will not be used in the next $SA$. In this case, the computation of
$\mathbf{P}^{\prime}$ can be indeed removed, which can reduce the complexity
by $O(2pd^{2}+4npd+2p^{2}d)$, where $p$ is often a large value on VL tasks.

Eventually, the feature update of VLP models with prompt tokens can be relaxed
to

$\displaystyle\mathbf{X}^{\prime}$
$\displaystyle=\mathbf{A}_{I}\mathbf{X}\mathbf{W}_{v}+\mathbf{A}_{IP}\mathbf{P}\mathbf{W}_{v}$
(4)
$\displaystyle=\frac{\mathbf{\gamma}_{I}}{\mathbf{\gamma}_{I}+\mathbf{\gamma}_{IP}}\sigma(\frac{\mathbf{X}\mathbf{W}_{q}(\mathbf{X}\mathbf{W}_{k})^{T}}{\sqrt{d}})\mathbf{X}\mathbf{W}_{v}$
$\displaystyle+\frac{\mathbf{\gamma}_{IP}}{\mathbf{\gamma}_{I}+\mathbf{\gamma}_{IP}}\sigma(\frac{\mathbf{X}\mathbf{W}_{q}(\mathbf{P}\mathbf{W}_{k})^{T}}{\sqrt{d}})\mathbf{P}\mathbf{W}_{v},$
where $\displaystyle\mathbf{\gamma}_{I}=\sum
e^{\mathbf{Q}\mathbf{K}^{T}_{i}},\mathbf{\gamma}_{IP}=\sum
e^{\mathbf{Q}{\mathbf{P}_{k}}^{T}_{j}}$

Here, $\sigma(\cdot)$ is the Softmax function, and $\mathbf{\gamma}_{I}$ and
$\mathbf{\gamma}_{IP}$ are the attention proportion for the input sequence and
prompt tokens, respectively. In Eq. 4, the first term is the self-attention
update of input features, which is the compulsory operation of VLP models. To
this end, the effectiveness of prompt tuning lies in the second term, which is
essentially a weighted information diffusion step from $\mathbf{P}$ to
$\mathbf{X}$. Since the scale-dot product is still required, this diffusion
step is also expensive.

##  4 Approximated Prompt Tuning

Based on the above observation, we propose _approximated prompt tuning_ (APT)
to model the attention impacts of prompt tokens. In particular, we can get the
prompt tuning process of APT with the following formula:

$\displaystyle\mathbf{X}^{\prime}=SA(\mathbf{X})+APT(\mathbf{X},\mathbf{P}).$
(5)

For simplicity, we regard the information diffusion from P to X as $\Delta X$:

$\displaystyle\Delta\mathbf{X}$ $\displaystyle=APT(\mathbf{X},\mathbf{P})$ (6)
$\displaystyle=\frac{\mathbf{\gamma}_{IP}}{\mathbf{\gamma}_{I}+\mathbf{\gamma}_{IP}}\sigma(\mathbf{X}\mathbf{W}_{q}(\mathbf{P}\mathbf{W}_{k})^{T})\mathbf{P}\mathbf{W}_{v}.$

To approximate $\Delta X$, we first focus on the information aggregation of
prompt tokens, denoted as $\Delta X^{\prime}$:

$\displaystyle\Delta\mathbf{X}^{\prime}=\sigma(\mathbf{X}\mathbf{W}_{q}\mathbf{W}_{k}^{T}\mathbf{P}^{T})\mathbf{P}\mathbf{W}_{v}.$
(7)

Note that, $\mathbf{W}_{v}$ is fixed in SA, and $\mathbf{P}$ is a trainable
matrix. To this end, we can directly update the projection of prompt tokens
onto the $V$ subspace, _i.e._ , put $\mathbf{P}\mathbf{W}_{v}$ as
$\mathbf{P}^{\prime}$. Similarly, we can simplify
$\mathbf{P}\mathbf{W}_{k}\mathbf{W}_{q}^{T}$ as $K$. Then, the X can be
directly taken as the $Q$ without projection, and the computation from
transforming $\mathbf{X}$ and $\mathbf{P}$ into $Q$, $K$ and $V$ of $SA$ can
be saved.

Next, we show that $V$ can be linearly transformed to $K$:

$\displaystyle\Delta\mathbf{P}$
$\displaystyle=\mathbf{P}\mathbf{W}_{k}\mathbf{W}_{q}^{T}-\mathbf{P}\mathbf{W}_{v},$
(8)
$\displaystyle=\mathbf{P}(\mathbf{W}_{k}\mathbf{W}_{q}^{T}-\mathbf{W}_{v}).$

Here, $\Delta\mathbf{P}$ denotes the difference between $V$ and $K$.Because
$V$ can be transformed to $K$ by a linear transformation, we approximate Eq. 7
as

$\displaystyle\Delta\mathbf{X}^{\prime}=\sigma\big{(}\mathbf{X}(\mathbf{P}^{\prime}\mathbf{W}_{p}+\mathbf{P}^{\prime})^{T}\big{)}\mathbf{P}^{\prime},$
(9)

where $\mathbf{W}_{p}\in\mathbb{R}^{d\times d}$ aims at transforming prompt
tokens from $V$ to $K$. However, calculating $\mathbf{P}\mathbf{W}_{p}$ is
still not cheap due to the high feature dimension.

As the low intrinsic dimension component [30, 1] plays a dominant role in
model optimization, the rank for $\mathbf{W}_{p}$ is finite according to the
theorem of the rank of matrices:

$\displaystyle rank(\mathbf{W}_{p})\leq
rank(\mathbf{W}_{k}\mathbf{W}_{q}^{T})+rank(\mathbf{W}_{v}),$ (10)

where $rank(\cdot)$ is the rank of the matrix. We can approximate the
aggregation of prompt tokens in a low-rank way:

$\displaystyle\Delta\mathbf{X}^{\prime}=\sigma\big{(}\mathbf{X}(\mathbf{P}^{\prime}\mathbf{W}_{1}\mathbf{W}_{2}+\mathbf{P}^{\prime})^{T}\big{)}\mathbf{P}^{\prime}.$
(11)

Here, $\mathbf{W}_{1}\in\mathbb{R}^{d\times r}$ and
$\mathbf{W}_{2}\in\mathbf{R}^{r\times d}$ are two low-dimensional matrix,
where $r\ll d$. The rank of projection matrix $\mathbf{W}_{1}\mathbf{W}_{2}$
is limited by $r$. As shown in Fig. 3(a), the way we obtain $Q$, $K$ and $V$
matrices for attention modeling is cheaper than the original global attention.

Afterwards, we consider the way to merge the original output of the self-
attention module $SA(\mathbf{X})$ and the information of prompt tokens
$\Delta\mathbf{X}$. Because the calculation of attention is still related to
the input sequence, it is difficult to reduce the complexity of the
approximation via independent computation. In this case, to adaptively adopt
the impact of each prompt token, a simple solution is activating the attention
matrix with $ReLU$ instead of $Softmax$ and omitting the weight item. Then,
Eq. 6 can be represented as

$\displaystyle\Delta\mathbf{X}=\psi\big{(}\mathbf{X}(\mathbf{P}^{\prime}\mathbf{W}_{1}\mathbf{W}_{2}+\mathbf{P}^{\prime})^{T}\big{)}\mathbf{P}^{\prime},$
(12)

where $\psi(\cdot)$ represent ReLU activation. In this manner, the weights for
prompts depend on the norm of prompt tokens and their relation to input
sequence.

Furthermore, from the weight calculation in Eq. 6, we observe that the effect
of prompt tokens is not only influenced by their dependency to the input
sequence, but also by the sum of attentions to the input sequence. With the
intrinsic of the Softmax function that the maximum value has the most impact,
we re-define Eq.6 by

$\displaystyle\Delta\mathbf{X}$
$\displaystyle=\alpha\cdot\psi(\mathbf{X}(\mathbf{P}^{\prime}\mathbf{W}_{1}\mathbf{W}_{2}+\mathbf{P}^{\prime})^{T})\mathbf{P}^{\prime},$
(13) $\displaystyle\text{where}\alpha$
$\displaystyle=max\\{\mathbf{P}^{\prime}\mathbf{W}_{1}\mathbf{W}_{2}+\mathbf{P}^{\prime}\\},$

where $max\\{\cdot\\}$ is the maximum function for the weight of each token.
Thus, APT can globally adjust the information diffusion from the prompt
tokens. Since the activation function of Eq. 13 no longer relies on the
original attention matrix, the APT is easier to deploy for VLP models.

Up to now, we have fully considered the effect of prompt tokens in diffusing
task-related information to the input sequence. Then, we also take into
account the effect of prompt tokens on the original attention matrix. As shown
in Eq. 4, the information diffusion also influences the original attention
matrix by increasing the denominator of the weight for the item from VLP
module. To this end, we add a learnable scale $s$ for the entire output, and
the proposed APT can be summarised as follow:

$\displaystyle\mathbf{X}^{\prime}$
$\displaystyle=\mathbf{A}_{I}\mathbf{X}\mathbf{W}_{v}+\mathbf{A}_{IP}\mathbf{P}\mathbf{W}_{v}$
(14) $\displaystyle\approx
e^{s}\cdot\big{(}SA(\mathbf{X})+\alpha\cdot\psi(\mathbf{X}(\mathbf{P}^{\prime}\mathbf{W}_{1}\mathbf{W}_{2}+\mathbf{P}^{\prime})^{T})\mathbf{P}^{\prime}\big{)},$
$\displaystyle\text{where}\
\alpha=max\\{\mathbf{P}^{\prime}\mathbf{W}_{1}\mathbf{W}_{2}+\mathbf{P}^{\prime}\\}.$

Here, the learnable value $s$ control the total amount of information diffused
by APT and also make the output of attention modules fit the following layers.

Eventually, the proposed APT method separates the effect of prompt tokens from
the original attention module, as shown in Fig. 3(b). The independence of APT
brings two main benefits: (1) Information diffusion can break the limitation
of patterns from VLP model, _i.e._ not constrained by Softmax-based
normalization. (2) The computational overhead is significantly reduced by
about $O(2pd^{2})$. In practice, it can save about $82.30\%$ and $62.62\%$
computations for ViLT [24] and METER [11] compare to conventional prompt
tuning methods, respectively.

##  5 Experiments

Table 1:  Comparisons of APT and conventional prompt tuning methods for ViLT
and METER on VQA, NLVR2 and Flickr30K. The best performance is bold while the
second one is underlined.

Backbone | Method |  Updated Parameter |  Additional FLOPs | VQA | NLVR2 | Flickr30K | Avg.
---|---|---|---|---|---|---|---
test-dev | test-P | IR R@1 | TR R@1
ViLT | Full Tuning | 115.43M | 0.0 | 71.26 | 76.13 | 64.40 | 83.50 | 73.82
Shallow Prompt | 0.15M | 19.53G | 66.47 | 66.47 | 55.92 | 74.80 | 65.92
Deep Prompt | 1.84M | 5.14G | 69.30 | 73.34 | 58.64 | 79.50 | 70.20
APT | 1.92M | 0.91G | 70.94 | 75.92 | 63.26 | 81.60 | 72.93
METER | Full Tuning | 323.31M | 0.0 | 77.43 | 83.05 | 82.22 | 94.30 | 84.25
Deep Prompt (cross) | 1.84M | 6.53G | 66.27 | 67.32 | 69.96 | 86.90 | 72.61
Deep Prompt (both) | 3.68M | 13.05G | 67.57 | 65.79 | 70.90 | 87.70 | 72.99
Shallow Prompt | 0.30M | 28.71G | 68.51 | 65.69 | 74.20 | 88.60 | 74.25
Deep Prompt (self) | 1.84M | 6.35G | 70.78 | 72.64 | 78.84 | 89.40 | 77.92
APT | 3.83M | 2.31G | 75.45 | 80.97 | 80.88 | 92.90 | 82.55

Table 2:  Comparisons of APT and the state-of-the-art PETL methods for ViLT
and METER on VQA, NLVR2 and Flickr30K. The best performance is bold and the
second best is underlined.

Backbone | Method |  Updated Parameter |  Additional FLOPs | VQA | NLVR2 | Flickr30K | Avg.
---|---|---|---|---|---|---|---
test-dev | test-P | IR R@1 | TR R@1
ViLT | Full Tuning | 115.43M | 0.0 | 71.26 | 76.13 | 64.40 | 83.50 | 73.82
Classifier Only | - | 0.0 | 65.75 | 66.08 | 57.42 | 78.00 | 66.81
Deep Prompt | 1.84M | 5.14G | 69.30 | 73.34 | 58.64 | 79.50 | 70.20
LoRA | 0.15M | 0.0 | 68.44 | 72.77 | 57.44 | 77.70 | 69.09
Scaled PA | 1.80M | 0.44G | 70.40 | 75.13 | 61.88 | 79.00 | 71.60
Adapter | 3.56M | 0.86G | 70.85 | 75.51 | 62.68 | 81.40 | 72.61
APT | 1.92M | 0.91G | 70.94 | 75.92 | 63.26 | 81.60 | 72.93
METER | Full Tuning | 323.31M | 0.0 | 77.43 | 83.05 | 82.22 | 94.30 | 84.25
Classifier Only | - | 0.0 | 69.93 | 73.23 | 78.80 | 89.00 | 77.74
Deep Prompt (self) | 1.84M | 6.53G | 70.78 | 72.64 | 78.84 | 89.40 | 77.92
LoRA | 0.29M | 0.0 | 74.00 | 78.82 | 79.86 | 92.60 | 81.32
Adapter | 5.34M | 1.64G | 74.70 | 79.93 | 80.38 | 91.90 | 81.73
Scaled PA | 3.59M | 1.12G | 75.11 | 80.38 | 80.40 | 93.20 | 82.27
APT | 3.83M | 2.31G | 75.45 | 80.97 | 80.88 | 92.90 | 82.55

###  5.1 Datasets and Experimental Setup

VQA2.0 [14] is one of the most popular datasets for visual question answering
(VQA) task. It uses images from MS-COCO [46] and has about $443,757$,
$214,254$ and $447,793$ VQA examples for training, validation and testing,
respectively. NLVR2 [50] is built for visual reasoning. It contains $107,292$
examples of human-written English sentences for pairs of photographs.
Flickr30k [43] is a widely-used benchmark dataset in this image-text matching
task. The dataset consists of $31,783$ images, and each has five corresponding
captions. For CLIP, we validate APT on $11$ popular image classification
datasets, including ImageNet [9], Caltech101 [12], OxfordPets [41],
StandfordCars [25], Flowers102 [40], Food101 [3], FGVCAircraft [38], SUN397
[56], DTD [7], EuroSAT [17], UCF101 [48] 555The details of these datasets are
given in Appendix.. This comprehensive benchmark comprises datasets that cover
a diverse set of vision tasks, including classification on generic objects,
scenes, actions, and fine-grained categories. It also includes specialized
tasks like recognizing textures and satellite imagery.

Implementation details. We validate APT on two deep-fusion based VLP models,
namely ViLT [24] and METER [11], and one shallow-fusion based VLP network
called CLIP [44]. In terms of ViLT, we add APT to its each SA layer. We set
the rank value $r$ in Eq. 11 to $4$ and the number of prompt tokens $p=200$ as
the default setting. The prompt tokens are initialized by a normal
distribution with a mean of $0.0$ and a variance of $0.02$. And we only apply
a single attention rather than the multi-head one [10] for the proposed APT
method. During the training, we update the classifier, class tokens and modal-
type embeddings, while the rest parameters of ViLT are kept fixed. For each
task, we follow its default settings and increase the learning rate by five
times. In terms of METER, APT is inserted into its self-attention and cross-
attention layers. The rest settings are the same as ViLT. For CLIP [44], we
insert APT into the self-attention layers of its text encoder, and we set the
rank $r=2$ and the number of prompts $p=4$. APT is optimized by SGD with a
learning rate of $0.0002$ and weight decay of $0.3$ for $10$ epochs. Following
[44], we also uses a hard prompt phrase of “ _a photo of_ [X]”, of which word
embeddings are feed to the text encoder of CLIP.

###  5.2 Experimental results

####  5.2.1 Quantitative analysis

Comparison with prompt tuning methods. We first compare APT with two common
soft prompt tuning methods, _i.e._ , deep prompt [23] and shallow prompt [32],
in Tab. 1. For all methods, the number of prompts is set to $200$ for a fair
comparison. Here, the suffixes after deep prompt, _i.e._ , _self_ , _cross_
and _both_ denotes the insertions of prompts to self-attention, cross-
attention and both attention modules of METER, respectively. From Tab. 1, the
performance of existing prompt tuning methods is far behind the full tuning
one, _i.e._ $-7.90\%$ to $-3.62\%$ on ViLT and $-11.64\%$ to $-6.33\%$ on
METER. These results are also worse than their performance on NLP [32] and CV
tasks [23], showing the challenge of prompt tuning on VLP models. Among these
compared methods, Deep Prompt shows better results than shallow prompts at
most cases, while its parameter size is larger. The additional parameter size
of our APT is close to that of Deep Prompt. However, the average improvements
of APT to these prompt methods are $+7.01\%$ to $+2.73\%$ on ViLT and
$+9.94\%$ to $+4.63\%$ on METER, respectively, while the saved additional
computations can be up to $82.30\%$ on ViLT and $63.62\%$ on METER. Meanwhile,
the performance of APT almost approaches full tuning, _e.g._ , $-0.89\%$ and
$-1.70\%$ on average for ViLT and METER, respectively. Considering the small
number of parameters updated, these results are indeed significant. Tab. 1
shows the great merits of APT over existing prompt tuning methods on VLP
models in both performance and computation.

Comparison with existing PETL methods. Next, we compare APT with a bunch of
recently proposed PETL methods, including LoRA [19], VL-Adapter (Adapter) [52]
and Scaled Parallel Adapter (Scaled PA) [16], of which results are given in
Tab. 2666These results are reproduced by us because there are no ready-made
literature to refer to.. Their detailed implementations are given in Appendix.
From this table, we can first see that LoRA is most efficient in both
parameters and computation due to its low-rank re-parameterization scheme.
Compared to pre-trained language models [34, 4], its performance on VLP models
is much inferior, especially on the tasks that are greatly different from pre-
training, _e.g._ , VQA and NLVR2, suggesting the challenge of VL adaption. We
can also find that although the adapter-based methods show better
adaptabilities than LoRA, they still perform worse than our APT. Compared with
VL-Adapter, APT can achieve obvious gains on ViLT and METER, while saving
about $46.07\%$ and $28.28\%$ parameters, respectively. In terms of the most
advanced Scaled PA, APT is slightly inferior in parameter and computation
costs, but its adaption performance is consistently better than Scaled PA on
two VLP models. Overall, these results suggest that our APT is a competitive
method in PETL with great potential.

Ablation Study.

Table 3:  Ablation study on different constructions and the number of prompt
tokens. Results for ViLT on VQA and NLVR2. *the default setting.

Prompt | Rank value |  Additional Parameter | VQA | NLVR2
---|---|---|---|---
test-dev | test-P
200 | Identity | 1.84M | 70.42 | 75.27
200 | dense | 8.93M | 71.07 | 75.67
100 | 4 | 0.99M | 70.11 | 74.81
150 | 4 | 1.46M | 70.49 | 74.87
400 | 4 | 3.76M | 70.64 | 75.68
400 | 16 | 3.98M | 71.01 | 75.95
200 | 4 | 1.92M | 70.94* | 75.92*

Table 4:  Ablation study of the scaling designs of APT, _i.e._ , $\alpha$ and
$e^{s}$. *the results of fine-tuning classifier used as reference.

APT w/o scaling | with $\alpha$ | with $e^{s}$ | VQA | NLVR2
---|---|---|---|---
test-dev | test-P
- | - | - | 65.75* | 66.08*
$\checkmark$ | - | - | 70.41 | 74.93
$\checkmark$ | $\checkmark$ | - | 70.58 | 75.35
$\checkmark$ | - | $\checkmark$ | 70.48 | 74.87
$\checkmark$ | $\checkmark$ | $\checkmark$ | 70.94 | 75.92

We first examine the impact of prompt number and the rank value in Eq. 11, of
which results are given in Tab. 3. Here, “ _identify_ ” denotes that directly
using prompt tokens as $K$ and $V$ in Eq. 9, while “ _dense_ ” means that low-
rank transformation is not used in Eq. 11. The first observation from Tab. 3
is that the increase of prompt tokens is beneficial to VLP models, which can
obtain improvements on both tasks, _e.g._ , 100 to 200. However, when
exceeding 200, its gains are marginal in contrast to other prompt methods as
shown in Fig. 4, which also suggests the effectiveness of APT for VLP models.
In terms of the rank value, the performance of “identity” suggests that
directly using prompt tokens for attention is suboptimal. And the low-rank
approximation can better trade off performance and parameter cost, _e.g._ rank
value $r=4$, which is even superior than the dense transformation on NLVR2.

Table 5:  Comparison of zero-shot CLIP (_CLIP_), CoOp, CoCoOp and APT on the
_base to new_ classification task [59]. We report their transferring
performance on both base and new tasks.

Dataset | Method
---|---
Base | New
CLIP | CoOp | CoCoOp | APT | CLIP | CoOp | CoCoOp | APT
ImgNet [9] | 72.43 | 76.47 | 75.98 | 75.97 | 68.14 | 67.88 | 70.43 | 71.23
Cal101 [12] | 96.84 | 98.00 | 97.96 | 97.93 | 94.00 | 89.81 | 93.81 | 94.13
Pets [41] | 91.17 | 93.67 | 95.20 | 94.97 | 97.26 | 95.29 | 97.69 | 97.60
Cars [25] | 63.37 | 78.12 | 70.49 | 76.10 | 94.89 | 60.40 | 73.59 | 75.13
Flowers [40] | 72.08 | 97.60 | 94.87 | 94.47 | 77.80 | 59.67 | 71.75 | 70.13
Food [3] | 90.10 | 88.33 | 90.70 | 90.17 | 91.22 | 82.26 | 91.29 | 90.70
Aircraft [38] | 27.19 | 40.44 | 33.41 | 38.63 | 36.29 | 22.30 | 23.71 | 33.97
SUN [56] | 69.36 | 80.60 | 79.74 | 81.50 | 75.35 | 65.89 | 76.86 | 78.20
DTD [7] | 53.24 | 79.44 | 77.01 | 81.00 | 59.90 | 41.18 | 56.00 | 48.53
SAT [17] | 56.48 | 92.19 | 87.49 | 91.10 | 64.05 | 54.74 | 60.04 | 62.13
UCF [48] | 70.53 | 84.69 | 82.33 | 85.13 | 77.50 | 56.05 | 77.64 | 76.93
Average | 69.34 | 82.69 | 80.47 | 82.72 | 74.22 | 63.22 | 71.69 | 72.64

Figure 4:  The comparison between APT and other PETL methods in terms of
performance and parameter size. APT has a better trade-off between performance
and parameter costs.  Figure 5:  The visualizations of the attention results
of APT, Shallow Prompt and Deep Prompt with ViLT on VQA2.0 dataset, which
suggests the degree of information exchange between prompts and the input
sequence. We select the 15 most active tokens of the visual and text inputs,
and the top 30 prompt tokens for visualization, 60 tokens in total. The color
denotes the degree of attention, the redder the higher and vice versa.
Compared with Shallow Prompt and Deep Prompt, APT can more effectively diffuse
prompt information to the input sequence from the low layers of ViLT.

Tab. 4 shows the ablation of the scaling designs we used for APT, _i.e._ ,
$\alpha$ in Eq. 13 and $s$ in Eq. 14. The first row is the results of only
fine-tuning classifier used as reference. From this table, we can see that the
basic assumption of APT is supported, _i.e._ , “APT _w/o_ scaling”, which
achieves distinct gains than tuning classifier. Via considering the max value
impact discussed in Eq. 13, _i.e._ , _with_ $\alpha$, and the denominator
factor discussed in Eq. 4, _i.e._ , _with_ $e^{s}$, the performance of APT can
be further improved, _i.e._ , $70.94\%$ and $75.92\%$ in the last row.
However, these two scaling factors need to be used together to maximize the
benefit of APT, as shown in the second and third rows of Tab. 4.

Overall, the results of Tab. 3 and Tab. 4 well confirm the motivation and
effectiveness of our designs towards prompt attention approximation.

Generalization on CLIP. We further examine the generalization ability of APT
on the shallow-fusion based VLP model, _i.e._ , CLIP [44], under the base-to-
new classification task [59], of which results are given in Tab. 5. In this
task, the model needs to adapt to the base dataset, and will be further
evaluated on unseen data (new dataset). The detailed setting is given in
Appendix. The compared methods include zero-shot CLIP, CoOp [60] and CoCoOp
[59]. From this table, we first observe that zero-shot CLIP has a strong
transferring learning ability. Due to its large-scale pre-training, it can
obtain superior performance under the new task evaluations. However, without
tuning on base datasets, its performance is much inferior than PETL methods.
In terms of CoOp, it can achieve satisfactory performance for base task
adaption. However, its generalization is very limited to new tasks, only
$63.22\%$ on average, suggesting an over-fitting problem. In a stark contrast,
APT can obtain a good performance in adapting base tasks while generalizing
well to the new ones. Compared to the latest PETL methods for CLIP, _i.e._
CoCoOp, its performance is also consistently better under two settings. These
results confirm the generalization ability of APT.

Parameter Efficiency. In Fig. 4, we also present the performance comparison of
APT to other PETL methods with different parameter costs. As can be seen, Deep
Prompt are much inferior than other methods in terms of parameter efficiency
and performance, suggesting its difficulty on VL adaption. Adapter [52] and
Scaled PA [16], as the advanced PETL methods, are all parameter efficient, and
their adaptions are also plausible on VQA. However, the overall performance of
these two methods is close, which is beyond expectation. Compared to these
adapter-based methods, the performance of APT can achieve obvious gains at a
scale of about 2M parameters, which becomes stable as the parameter size
grows. Overall, these results well confirm the effectiveness of APT towards
efficient VLadaption.

####  5.2.2 Qualitative analysis

To obtain more intuitive comparisons to existing prompt tuning methods, we
further visualize the attention results of APT, shallow prompt tuning and deep
prompt tuning in Fig. 5. The global attention matrices can be divided into six
sub-parts, _i,e._ _Text-Text_ , _Text-Image_ , _Text-Prompt_ , _Image-Text_ ,
_Image-Image_ , _Image-Prompt_. From these examples, we can first observe that
in the lower layers of the VLP model, the information exchanges mainly happens
among the tokens of the same modality under the common prompt tuning methods.
Thus, the prompts barely affect the statues of the input sequence. As the
inference progress, their impacts of common prompts become slightly more
pronouns. In terms of shallow prompts, the impact of its tokens is still
marginal, while deep prompt will be better at the last few layers of the
model. The above results are also consistent with their performance on VL
tasks. In a stark contrast to these methods, the proposed APT can effectively
diffuse prompt information to the input sequence from the first layer of the
VLP models. And its attention weights become more intensive in the higher
layers of the VLP models, suggesting its effectiveness towards task adaption.

##  6 Conclusion

In this paper, we focus on the issues of high computation overhead and
inefficient adaption of prompt tuning on vision-language pre-trained (VLP)
models. By revisiting the principle of prompt tuning, we can figure out that
the key to improve prompt tuning lies in its information diffusion to the
input sequence, which can be indeed independent to the expensive global self-
attention via effective approximations. Motivated by this observation, we
propose a novel _approximation prompt tuning_ (APT) approach towards effective
VL adaption. APT approximates the impacts of prompt tokens to the input
sequence via a low-rank token aggregation design, reducing the computation
cost to a large extent. We validate APT on 2 VLP models and 3 VL benchmarks,
and also test its generation on CLIP for image classification. The
quantitative and qualitative results not only show the obvious merits of APT
over existing prompt tuning methods in both computation efficiency and
performance, but also outperform the compared PETL methods.

##  7 Acknowledgement

This work was supported by National Key R&D Program of China
(No.2022ZD0118202), the National Science Fund for Distinguished Young Scholars
(No.62025603), the National Natural Science Foundation of China (No. U21B2037,
No. U22B2051, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No.
62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural
Science Foundation of Fujian Province of China (No.2021J01002, No.2022J06001).

## References

  * [1] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, ACL, pages 7319–7328. Association for Computational Linguistics, 2021.
  * [2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh.  VQA: visual question answering.  In ICCV, pages 2425–2433. IEEE Computer Society, 2015.
  * [3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101 - mining discriminative components with random forests.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, ECCV, volume 8694 of Lecture Notes in Computer Science, pages 446–461. Springer, 2014.
  * [4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, NeurIPS, 2020.
  * [5] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO captions: Data collection and evaluation server.  CoRR, abs/1504.00325, 2015.
  * [6] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  UNITER: universal image-text representation learning.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, volume 12375 of Lecture Notes in Computer Science, pages 104–120. Springer, 2020.
  * [7] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In CVPR, pages 3606–3613. IEEE Computer Society, 2014.
  * [8] Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang.  Template-based named entity recognition using BART.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, ACL Findings, volume ACL/IJCNLP 2021 of Findings of ACL, pages 1835–1845, 2021.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, pages 248–255. IEEE Computer Society, 2009.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, NAACL-HLT, pages 4171–4186. Association for Computational Linguistics, 2019\.
  * [11] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng.  An empirical study of training end-to-end vision-and-language transformers.  In CVPR, pages 18145–18155. IEEE, 2022.
  * [12] Li Fei-Fei, Robert Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  Comput. Vis. Image Underst., 106(1):59–70, 2007.
  * [13] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  CoRR, abs/2110.04544, 2021.
  * [14] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the V in VQA matter: Elevating the role of image understanding in visual question answering.  In CVPR, pages 6325–6334. IEEE Computer Society, 2017.
  * [15] Demi Guo, Alexander M. Rush, and Yoon Kim.  Parameter-efficient transfer learning with diff pruning.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, ACL, pages 4884–4896. Association for Computational Linguistics, 2021.
  * [16] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  In ICLR. OpenReview.net, 2022.
  * [17] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 12(7):2217–2226, 2019.
  * [18] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, volume 97 of Proceedings of Machine Learning Research, pages 2790–2799. PMLR, 2019.
  * [19] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In ICLR. OpenReview.net, 2022.
  * [20] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu.  Seeing out of the box: End-to-end pre-training for vision-language representation learning.  In CVPR, pages 12976–12985. Computer Vision Foundation / IEEE, 2021.
  * [21] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu.  Pixel-bert: Aligning image pixels with text by deep multi-modal transformers.  CoRR, abs/2004.00849, 2020.
  * [22] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In Marina Meila and Tong Zhang, editors, ICML, volume 139 of Proceedings of Machine Learning Research, pages 4904–4916. PMLR, 2021\.
  * [23] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In ECCV, volume 13693, pages 709–727, 2022.
  * [24] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In Marina Meila and Tong Zhang, editors, ICML, volume 139 of Proceedings of Machine Learning Research, pages 5583–5594. PMLR, 2021\.
  * [25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In ICCV, pages 554–561. IEEE Computer Society, 2013.
  * [26] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  ALBERT: A lite BERT for self-supervised learning of language representations.  In ICLR. OpenReview.net, 2020.
  * [27] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He.  Stacked cross attention for image-text matching.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, ECCV, volume 11208 of Lecture Notes in Computer Science, pages 212–228. Springer, 2018.
  * [28] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, EMNLP, pages 3045–3059, 2021.
  * [29] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, ACL, pages 7871–7880. ACL, 2020.
  * [30] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski.  Measuring the intrinsic dimension of objective landscapes.  In ICLR. OpenReview.net, 2018.
  * [31] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  CoRR, abs/1908.03557, 2019.
  * [32] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, ACL, pages 4582–4597. Association for Computational Linguistics, 2021.
  * [33] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.  GPT understands, too.  CoRR, abs/2103.10385, 2021.
  * [34] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692, 2019.
  * [35] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, NeurIPS, pages 13–23, 2019.
  * [36] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.  Compacter: Efficient low-rank hypercomplex adapter layers.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, NeurIPS, pages 1022–1035, 2021.
  * [37] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson.  Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, ACL, pages 565–576. Association for Computational Linguistics, 2021.
  * [38] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  CoRR, abs/1306.5151, 2013.
  * [39] Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa.  Unipelt: A unified framework for parameter-efficient language model tuning.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, ACL, pages 6253–6264. Association for Computational Linguistics, 2022.
  * [40] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In ICVGIP, pages 722–729. IEEE Computer Society, 2008.
  * [41] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In CVPR, pages 3498–3505. IEEE Computer Society, 2012.
  * [42] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller.  Language models as knowledge bases?  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, ACL, pages 2463–2473. Association for Computational Linguistics, 2019.
  * [43] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  IJCV, 123(1):74–93, 2017.
  * [44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, ICML, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021\.
  * [45] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  JMLR, 21:140:1–140:67, 2020.
  * [46] Mengye Ren, Ryan Kiros, and Richard Zemel.  Exploring models and data for image question answering.  NeurIPS, 28, 2015.
  * [47] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, NeurIPS, pages 91–99, 2015.
  * [48] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  UCF101: A dataset of 101 human actions classes from videos in the wild.  CoRR, abs/1212.0402, 2012.
  * [49] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  VL-BERT: pre-training of generic visual-linguistic representations.  In ICLR. OpenReview.net, 2020.
  * [50] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi.  A corpus for reasoning about natural language grounded in photographs.  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, ACL, pages 6418–6428. Association for Computational Linguistics, 2019.
  * [51] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  LST: ladder side-tuning for parameter and memory efficient transfer learning.  CoRR, abs/2206.06522, 2022.
  * [52] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  VL-ADAPTER: parameter-efficient transfer learning for vision-and-language tasks.  In CVPR, pages 5217–5227. IEEE, 2022.
  * [53] Yi-Lin Sung, Varun Nair, and Colin Raffel.  Training neural networks with fixed sparse masks.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, NeurIPS, pages 24193–24205, 2021.
  * [54] Hao Tan and Mohit Bansal.  LXMERT: learning cross-modality encoder representations from transformers.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, EMNLP, pages 5099–5110. ACL, 2019.
  * [55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, NeurIPS, pages 5998–6008, 2017.
  * [56] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba.  SUN database: Large-scale scene recognition from abbey to zoo.  In CVPR, pages 3485–3492. IEEE Computer Society, 2010.
  * [57] Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, and Jitendra Malik.  Side-tuning: A baseline for network adaptation via additive side networks.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, volume 12348 of Lecture Notes in Computer Science, pages 698–714. Springer, 2020.
  * [58] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Tip-adapter: Training-free clip-adapter for better vision-language modeling.  CoRR, abs/2111.03930, 2021.
  * [59] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In CVPR, pages 16795–16804. IEEE, 2022.
  * [60] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  IJCV, 130(9):2337–2348, 2022.
  * [61] Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Gen Luo, Xiaopeng Hong, Jinsong Su, Xinghao Ding, and Ling Shao.  K-armed bandit based multi-modal network architecture search for visual question answering.  In Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann, editors, ACM MM, pages 1245–1254. ACM, 2020.