it is just the pullback along ${\rm pr}_{1}$ of the first order Lagrangian for
Einstein–Hilbert gravity.

In the usual Routh reduction, the reduced Routhian is a $m$-form on
$J^{1}\big{(}\tau_{\Sigma}\circ\overline{p}\big{)}$; in this case, Lemma 10.2
allows us to consider the form ${\cal L}_{\rm EH}^{(1)}$ on
$J^{1}\tau_{\Sigma}$ as the Routhian. Therefore, we can forget about the
degrees of freedom associated to the factor $\mathop{{\rm
Lin}}{\big{(}\tau_{\Sigma}^{*}TM,\widetilde{\mathfrak{k}}\big{)}}$ in the
quotient bundle, and take as the quotient bundle for Palatini gravity the jet
bundle $J^{1}\tau_{\Sigma}$; this is the way in which we will proceed from
this point.

##  11 Einstein–Hilbert gravity as Routh reduction
of Palatini gravity

We will devote the present section to establish the two main results of the
article, namely, Theorem 11.4 regarding reduction of Palatini gravity and
Theorem 11.7 dealing with reconstruction of metrics verifying Einstein
equations of gravity. The strategy, as we mention in the introduction, is to
compare the equations of motion (lifted to the corresponding spaces of forms
$W_{\rm EH}$ and $W_{\rm PG}$) in a bundle containing every relevant degree of
freedom; this role is played below by a pullback bundle of the bundle $W_{\rm
EH}$ along a suitable map. So, let us define

$F_{\omega}:={\rm pr}_{1}\circ g_{\omega}\circ p_{K}^{J^{1}\tau}\colon\
\mathcal{T}_{0}\longrightarrow J^{1}\tau_{\Sigma},$

namely

$F_{\omega}\left(j_{x}^{1}s\right)=j_{x}^{1}\left(p_{K}^{LM}\circ s\right)$

for every $j_{x}^{1}s\in\mathcal{T}_{0}$. Then we have the diagram

${W_{\rm
PG}}$${\wedge_{2}^{m}\left(T^{*}\mathcal{T}_{0}\right)}$${F_{\omega}^{*}\left(W_{\rm
EH}\right)}$${W_{\rm
EH}}$${\mathcal{T}_{0}}$${J^{1}\tau_{\Sigma}}$$\scriptstyle{\pi_{\rm
PG}}$$\scriptstyle{\overline{\tau}^{m}_{\mathcal{T}_{0}}}$$\scriptstyle{{\rm
pr}_{1}^{\omega}}$$\scriptstyle{\widetilde{F_{\omega}}}$$\scriptstyle{{\rm
pr}_{2}^{\omega}}$$\scriptstyle{\pi_{\rm EH}}$$\scriptstyle{F_{\omega}}$
(11.1)

where

$\widetilde{F_{\omega}}\colon\ F_{\omega}^{*}(W_{\rm
EH})\longrightarrow\wedge_{2}^{m}(T^{*}\mathcal{T}_{0})\colon\
\big{(}j_{x}^{1}s,\rho\big{)}\mapsto\rho\circ T_{j_{x}^{1}s}F_{\omega}$

and

$\displaystyle{\rm pr}_{1}^{\omega}\colon\ F_{\omega}^{*}(W_{\rm
EH})\longrightarrow\mathcal{T}_{0},\qquad{\rm pr}_{2}^{\omega}\colon\
F_{\omega}^{*}(W_{\rm EH})\longrightarrow W_{\rm EH}$

are the canonical projections of the pullback bundle.

######  Remark 11.1.

Let us give a local version of the map $F_{\omega}$; recall that locally
$\mathcal{T}_{0}$ is described by the set of equations

$e^{k}_{\sigma}e^{\nu}_{k\rho}=e^{k}_{\rho}e^{\nu}_{k\sigma},$

where $\big{(}x^{\mu},e_{k}^{\sigma},e^{\rho}_{k\mu}\big{)}$ is a set of
adapted coordinates induced by a coordinate chart $(\phi=(x^{\mu}),U)$ on $M$.
In these coordinates, the canonical quotient map $p_{K}^{LM}\colon
LM\to\Sigma$ is given by

$p_{K}^{LM}\big{(}x^{\mu},e_{k}^{\sigma}\big{)}=\big{(}x^{\mu},\eta^{kl}e_{k}^{\rho}e_{l}^{\sigma}\big{)};$

accordingly, map $F_{\omega}$ will become

$F_{\omega}\big{(}x^{\mu},e_{k}^{\sigma},e^{\rho}_{k\mu}\big{)}=\big{(}x^{\mu},\eta^{kl}e_{k}^{\sigma}e_{l}^{\rho},\eta^{kl}\big{(}e^{\sigma}_{k\mu}e_{l}^{\rho}+e^{\sigma}_{k}e_{l\mu}^{\rho}\big{)}\big{)}.$

######  Lemma 11.2.

The bundle map $\widetilde{F_{\omega}}$ is an affine bundle isomorphism on
$\mathcal{T}_{0}$ between $W_{\rm PG}$ and $F_{\omega}^{*}(W_{\rm EH})$.

###### Proof.

It is consequence of equation (10.1) and Proposition 7.4. ∎

We will use diagram (11.1) as a means to compare the equations of motion of
Palatini gravity and Einstein–Hilbert gravity; the idea is to use Propositions
4.5 and 8.5 in order to represent these equations in terms of the spaces of
forms $W_{\rm PG}$ and $W_{\rm EH}$ respectively, and to pull them back to the
common space $F_{\omega}^{*}(W_{\rm EH})$. Crucial to this strategy is the
following result.

######  Proposition 11.3.

The following relation holds

$\widetilde{F_{\omega}}^{*}\lambda_{\rm PG}=\big{(}{\rm
pr}_{2}^{\omega}\big{)}^{*}\lambda_{\rm EH}.$

###### Proof.

Let $\big{(}j_{x}^{1}s,\rho\big{)}\in F^{*}_{\omega}(W_{\rm EH})$ be an
arbitrary element in this pullback bundle; then using diagram (11.1) we will
have that

$\displaystyle\lambda_{\rm PG}\big{|}_{\rho\circ
T_{j_{x}^{1}s}F_{\omega}}\circ T_{(j_{x}^{1}s,\rho)}\widetilde{F_{\omega}}$
$\displaystyle=\big{(}\rho\circ T_{j_{x}^{1}s}F_{\omega}\big{)}\circ
T_{\rho\circ
T_{j_{x}^{1}s}F_{\omega}}\overline{\tau}^{m}_{\mathcal{T}_{0}}\circ
T_{(j_{x}^{1}s,\rho)}\widetilde{F_{\omega}}$ $\displaystyle=\big{(}\rho\circ
T_{j_{x}^{1}s}F_{\omega}\big{)}\circ T_{(j_{x}^{1}s,\rho)}{\rm
pr}_{1}^{\omega}=\rho\circ T_{\rho}\pi_{\rm EH}\circ T_{(j_{x}^{1}s,\rho)}{\rm
pr}_{2}^{\omega}$ $\displaystyle=\lambda_{\rm EH}|_{\rho}\circ
T_{(j_{x}^{1}s,\rho)}{\rm pr}_{2}^{\omega},$

using the fact that $\pi_{\rm EH}\colon W_{\rm EH}\to J^{1}\tau_{\Sigma}$ is
the restriction of the canonical projection

$\overline{\tau}^{m}_{J^{1}\tau_{\Sigma}}\colon\
\wedge^{m}_{2}\big{(}T^{*}J^{1}\tau_{\Sigma}\big{)}\longrightarrow
J^{1}\tau_{\Sigma}$

to $W_{\rm EH}$. This identity proves the proposition. ∎

###  11.1 Routh reduction of Palatini gravity

We are now ready to prove the first result on Routh reduction of Palatini
gravity; essentially, we will prove that any section obeying the equations of
motion for Palatini gravity projects along the map $F_{\omega}$ to a holonomic
section obeying Einstein–Hilbert equations of motion.

######  Theorem 11.4.

Let $\widehat{Z}\colon U\subset M\to\mathcal{T}_{0}$ be a section that obeys
the Palatini gravity equations of motion. Then the section

$F_{\omega}\circ\widehat{Z}\colon\ U\to J^{1}\tau_{\Sigma}$

is holonomic and obeys the Einstein–Hilbert gravity equations of motion.

###### Proof.

The idea of the proof is encoded in the following diagram

${W_{\rm PG}}$${F_{\omega}^{*}(W_{\rm EH})}$${W_{\rm
EH}}$${\mathcal{T}_{0}}$${J^{1}\tau_{\Sigma}}$${M}$$\scriptstyle{\pi_{\rm
PG}}$$\scriptstyle{\widetilde{F_{\omega}}}$$\scriptstyle{{\rm
pr}_{1}^{\omega}}$$\scriptstyle{{\rm pr}_{2}^{\omega}}$$\scriptstyle{\pi_{\rm
EH}}$$\scriptstyle{F_{\omega}}$$\scriptstyle{\tau_{1}}$$\scriptstyle{(\tau_{\Sigma})_{1}}$$\scriptstyle{\widehat{Z}}$$\scriptstyle{\Gamma}$$\scriptstyle{\Gamma^{\prime}}$$\scriptstyle{\widetilde{\Gamma}}$
(11.2)

Using Proposition 4.5, we construct $\Gamma\colon U\to W_{\rm PG}$ out of
$\widehat{Z}$; the Palatini gravity equations of motion will become

$\Gamma^{*}(Z\lrcorner{\rm d}\lambda_{\rm PG})=0$

for any $Z\in\mathfrak{X}^{V(\tau_{1}\circ\pi_{\rm PG})}(W_{\rm PG})$. Using
Lemma 11.2 we can define

$\Gamma^{\prime}:=\big{(}\widetilde{F}_{\omega}\big{)}^{-1}\circ\Gamma\colon\
U\longrightarrow F_{\omega}^{*}(W_{\rm EH});$

then the Palatini equations of motion translate into

$({\Gamma}^{\prime})^{*}\big{(}{Z}^{\prime}\lrcorner{\rm
d}\widetilde{F_{\omega}}^{*}\lambda_{\rm PG}\big{)}=0$

for any ${Z}^{\prime}\in\mathfrak{X}^{V(\tau_{1}\circ{\rm
pr}_{1}^{\omega})}(F^{*}(W_{\rm EH}))$. Then using Proposition 11.3 and the
fact that ${\rm pr}_{2}^{\omega}\colon F^{*}_{\omega}(W_{\rm
EH})\allowbreak\to W_{\rm EH}$ is a submersion, we can conclude that the
section

$\widetilde{\Gamma}:={\rm pr}_{2}^{\omega}\circ\Gamma^{\prime}\colon\ U\to
W_{\rm EH}$

obeys the equations of motion

$\widetilde{\Gamma}^{*}\big{(}\widetilde{Z}\lrcorner{\rm d}\lambda_{\rm
EH}\big{)}=0,$

where $\widetilde{Z}\in\mathfrak{X}^{V((\tau_{\Sigma})_{1}\circ\pi_{\rm
EH})}(W_{\rm EH})$ is an arbitrary vertical vector field on $W_{\rm EH}$.
Also, using diagram (11.2), we have that

$\pi_{\rm EH}\circ\widetilde{\Gamma}=\pi_{\rm EH}\circ{\rm
pr}_{2}^{\omega}\circ\Gamma^{\prime}=F_{\omega}\circ{\rm
pr}_{1}^{\omega}\circ\Gamma^{\prime}=F_{\omega}\circ\widehat{Z}.$

The theorem then follows from Proposition 8.5. ∎

######  Remark 11.5 (reduction theorem in local coordinates).

Let us look at the reduction theorem in local terms. First, we have that the
equations of motion on $J^{1}\tau$ can be written as

$\displaystyle\frac{\partial\Gamma^{\beta}_{\mu\nu}}{\partial
x^{\beta}}-\frac{\partial\Gamma^{\beta}_{\mu\beta}}{\partial
x^{\nu}}+\Gamma^{\beta}_{\sigma\beta}\Gamma^{\sigma}_{\mu\nu}-\Gamma^{\beta}_{\sigma\nu}\Gamma^{\sigma}_{\mu\beta}=0,$
$\displaystyle\Gamma^{\sigma}_{\mu\nu}=-e_{\mu}^{k}e^{\sigma}_{k\nu},$
$\displaystyle e_{\mu}^{k}e_{k\nu}^{\sigma}-e_{\nu}^{k}e_{k\mu}=0.$ (11.3)

On the other hand, from Remark 11.1 we have that

$F_{\omega}\big{(}x^{\mu},e_{k}^{\sigma},e^{\rho}_{k\mu}\big{)}=\big{(}x^{\mu},\eta^{kl}e_{k}^{\sigma}e_{l}^{\rho},\eta^{kl}\big{(}e^{\sigma}_{k\mu}e_{l}^{\rho}+e^{\sigma}_{k}e_{l\mu}^{\rho}\big{)}\big{)}.$

Thus, Theorem 11.4 tells us that any local section

$\widehat{Z}\big{(}x^{\mu}\big{)}=\big{(}x^{\mu},e^{\mu}_{k}(x),e^{\mu}_{k\nu}(x)\big{)}$

obeying equations (11.3) will gives rise to a solution of Einstein equations
of motion when composed with $F_{\omega}$.

###  11.2 … and reconstruction

We will give now a (somewhat partial) converse to Theorem 11.4. That is, given
a section $\zeta\colon U\subset M\to\Sigma$ such that $j^{1}\zeta\colon U\to
J^{1}\tau_{\Sigma}$ is extremal for the Einstein–Hilbert variational problem,
find a section

$\widehat{Z}\colon\ U\to\mathcal{T}_{0}$

such that $F_{\omega}\circ\widehat{Z}=j^{1}\zeta$ and $\widehat{Z}$ is an
extremal for the Palatini variational problem. From Fig. 1 it is clear that we
need to lift the section $j^{1}\zeta$ through the quotient map
$p_{K}^{J^{1}\tau}\big{|}_{\mathcal{T}_{0}}\colon\mathcal{T}_{0}\to\mathcal{T}_{0}/K$,
which has the structure of a principal bundle on $\mathcal{T}_{0}/K$. It is
clear that any principal bundle can be trivialized by a convenient restriction
of the base space. As discussed in [6], it is not the way in which this kind
of reconstruction problems are solved. Rather, the problem of lifting sections
along the projection in a principal bundle is reduced to the problem of
deciding if certain connection is flat; moreover, it is expected that this
connection is related to the connection used to define the Routhian. We will
present in this section a theory of reconstruction along these lines. With
this goal in mind, we will recall here some of the details developed in [6];
for proofs we refer to the original article. We begin with a pair of diagrams
(11.4):

${P}$${P/G}$${M}$$\scriptstyle{p^{P}_{G}}$$\scriptstyle{\pi}$$\scriptstyle{\overline{\pi}}$$\scriptstyle{\zeta}$$\scriptstyle{s}$
${\zeta^{*}P}$${P}$${M}$${P/G}$$\scriptstyle{{\rm pr}_{2}}$$\scriptstyle{{\rm
pr}_{1}}$$\scriptstyle{p^{P}_{G}}$$\scriptstyle{\zeta}$ (11.4)

Then we have the following result.

######  Lemma 11.6.

There exists a section $s\colon M\rightarrow P$ covering the section
$\zeta\colon M\to P/G$ if and only if $\zeta^{*}P$ is a trivial bundle.

Using that $\zeta^{*}P$ is a principal bundle, being trivial can be
characterized in terms of a flat connection [29]:

######  Theorem 11.7.

Let $\pi\colon P\rightarrow M$ be a $G$-principal bundle with $M$ simply
connected. Then $P$ is trivial if and only if there exists a flat connection
on $P$.

If $M$ is not simply connected, then one can ask for a flat connection with
trivial holonomy and obtain a similar result. For the sake of simplicity, we
will assume that $M$ is simply connected to apply Theorem 11.7 when needed.
For later use, we also observe that the section constructed in the proof of
Theorem 11.7 has horizontal image with respect to the given connection.

We wish now to apply the previous discussion to the case of the bundle
$p_{K}^{\mathcal{T}_{0}}:=p_{K}^{J^{1}\tau}\big{|}_{\mathcal{T}_{0}}\colon\mathcal{T}_{0}\to\mathcal{T}_{0}/K$.
We have the situation depicted in diagram (11.5) (left): $Z\colon
M\to\mathcal{T}_{0}/K$ is a given section and $\zeta\colon M\to\Sigma$ is the
induced section. The basic question we want to address is whether there exists
a section $\widehat{Z}\colon M\rightarrow\mathcal{T}_{0}$ such that
$p_{K}^{\mathcal{T}_{0}}\circ\widehat{Z}=Z$:

${\mathcal{T}_{0}}$${\mathcal{T}_{0}/K}$${LM}$${\Sigma}$${M}$$\scriptstyle{p^{\mathcal{T}_{0}}_{K}}$$\scriptstyle{\tau_{10}^{\prime}}$$\scriptstyle{\overline{\tau}^{\prime}_{10}}$$\scriptstyle{p^{LM}_{K}}$$\scriptstyle{\tau^{\prime}}$$\scriptstyle{\tau_{\Sigma}}$$\scriptstyle{\zeta}$$\scriptstyle{\widehat{Z}}$$\scriptstyle{Z}$
${\zeta^{*}(LM)}$${LM}$${M}$${\Sigma}$$\scriptstyle{{\rm
pr}_{2}}$$\scriptstyle{{\rm
pr}_{1}}$$\scriptstyle{p_{K}^{LM}}$$\scriptstyle{\zeta}$ (11.5)

Now, using the fact that

$\mathcal{T}_{0}/K\simeq\Sigma\times C_{0}(LM),\qquad\mathcal{T}_{0}\simeq
LM\times C_{0}(LM)$

we have that $Z=(\zeta,\Gamma)$ is composed by the metric plus the Levi-Civita
connection $\Gamma$; therefore, we will have that

$\widehat{Z}=\big{(}\widehat{\zeta},\Gamma\big{)}$

where $\widehat{\zeta}\colon M\to LM$ is some lift of the section $\zeta\colon
M\to\Sigma$. Then, we can then construct the pullback bundle $\zeta^{*}(LM)$
(diagram (11.5), right) and particularize Lemma 11.6 to conclude the
following:

######  Lemma 11.8.

Assume that $M$ is simply connected. If $\zeta^{*}(LM)$ admits a flat
connection then there exists a section $\widehat{Z}\colon
M\rightarrow\mathcal{T}_{0}$ such that

$\big{(}p_{K}^{LM}\circ\tau_{10}^{\prime}\big{)}\circ\widehat{Z}=\zeta$

and $\widehat{Z}^{*}\omega_{\mathfrak{p}}=0$. Conversely, every such section
gives rise to a flat connection on $\zeta^{*}(TM)$.

###### Proof.

Because $\zeta^{*}(LM)$ is a $K$-principal bundle, Theorem 11.7 and Lemma 11.6
allow us to find a section $\widehat{\zeta}\colon M\to LM$ iff there exists a
flat connection on it. Thus if $\omega^{\zeta}$ is flat, we can construct a
lift $\widehat{\zeta}\colon M\to LM$ for $\zeta$ and so

$\widehat{Z}=\big{(}\widehat{\zeta},\Gamma\big{)}$

is the desired lift to $\mathcal{T}_{0}$, where $\Gamma\colon M\to C(LM)$ is
the Levi-Civita connection for $\zeta$.

Conversely, let us suppose that we have a lift

$\widehat{\zeta}:=\tau_{10}^{\prime}\circ\widehat{Z}\colon\ M\to LM$

for the metric $\zeta\colon M\to\Sigma$. Recall that, for every
$(x,u)\in\zeta^{*}(LM)$,

$T_{(x,u)}\zeta^{*}(LM)=\big{\\{}(v_{x},V_{u})\colon
T_{x}\zeta(v_{x})=T_{u}p_{K}^{LM}(V_{u})\big{\\}}\subset T_{x}M\times
T_{u}(LM).$

Then we construct the following $K$-invariant distribution $H$ on
$\zeta^{*}(LM)$: If $k\in K$ fullfils the condition $u=\widehat{\zeta}(x)\cdot
k$, then

$H_{(x,u)}:=\big{\\{}\big{(}v_{x},T_{\widehat{\zeta}(x)}R_{k}\big{(}T_{x}\widehat{\zeta}(v_{x})\big{)}\big{)}\colon
v_{x}\in T_{x}M\big{\\}}.$

It can be shown that it defines a flat connection on $\zeta^{*}(LM)$. ∎

So, in order to find a lift for the section $\zeta$, it is sufficient to
construct a flat connection on the $K$-principal bundle $\zeta^{*}(LM)$.

To this end, we will define

$\omega^{\zeta}:=\pi_{\mathfrak{k}}\circ({\rm
pr}_{2})^{*}\omega_{0}\in\Omega^{1}(\zeta^{*}(LM),\mathfrak{k})$

where $\omega_{0}\in\Omega^{1}(LM,\mathfrak{gl}(m))$ is a principal connection
on $LM$ and $\pi_{\mathfrak{k}}\colon\mathfrak{gl}(m)\to\mathfrak{k}$ is the
canonical projection onto $\mathfrak{k}$. Lemma 11.8 allows us to establish
the following definition, inspired in the analogous concept from regular Routh
reduction.

######  Definition 11.9 (flat condition for Palatini gravity).

We will say that a metric $\zeta\colon M\to\Sigma$ satisfies the _flat
condition regarding the principal connection
$\omega_{0}\in\Omega^{1}(LM,\mathfrak{gl}(m))$_ if and only if the associated
connection $\omega^{\zeta}$ is flat.

######  Remark 11.10 (flat condition and parallelizability).

This condition yields to a relationship between the metric $\zeta\colon
M\to\Sigma$ and the principal connection $\omega_{0}$; the physical relevance
of this relationship remains unclear for the author. In order to get this
condition mathematically, let us discuss the meaning of the bundle
$\zeta^{*}(LM)$. By definition, $(x,u=(X_{1},\dots,X_{m}))\in M\times LM$
belongs to $\zeta^{*}(LM)$ if and only $X_{i}\in T_{x}M$ for all $i=1,\dots,m$
and also

$\displaystyle\zeta(x)^{\sharp}=\eta^{ij}X_{i}\otimes X_{j}.$ (11.6)

Here $g^{\sharp}$ indicates the contravariant $2$-tensor associated to the
metric $g$; an equivalent way to express this is given by the formula

$\zeta(x)\circ(T_{u}\tau\otimes
T_{u}\tau)=\eta_{ij}\theta^{i}\big{|}_{u}\otimes\theta^{j}\big{|}_{u}.$

It means in particular that we can identify

$\displaystyle\iota\colon\
\zeta^{*}(LM)\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}O_{\zeta}(M),$

where $O_{\zeta}(M)\subset LM$ is the orthogonal subbundle associated to the
metric $\zeta$. This identification goes as follows: Given
$(x,u)\in\zeta^{*}(LM)$, equation (11.6) tells us that $u\in O_{\zeta}(M)$;
conversely, given $u\in O_{\zeta}(M)$, the pair $(\tau(u),u)$ belongs to
$\zeta^{*}(LM)$. Therefore, we have the commutative diagram

${\zeta^{*}(LM)}$${LM}$${O_{\zeta}(M)}$$\scriptstyle{\iota}$$\scriptstyle{{\rm
pr}_{2}}$inc

Thus, under the previous identification, the $\mathfrak{gl}(m)$-valued
$1$-form ${\rm pr}_{2}^{*}\omega_{0}$ is the pullback of the form $\omega_{0}$
to the subbundle $O_{\zeta}(M)$; because the decomposition

$\mathfrak{gl}(m)=\mathfrak{k}\oplus\mathfrak{p}$

is $K$-invariant, the $\mathfrak{k}$-valued $1$-form
$\pi_{\mathfrak{k}}\circ({\rm pr}_{2})^{*}\omega_{0}$ is a connection form on
$O_{\zeta}(M)$. Thus, flatness of this connection is equivalent to
$O_{\zeta}(M)$ being trivial. Now, it means that $M$ is parallelizable;
therefore, we have proved that the flat condition for Palatini gravity is
equivalent to parallelizability of the spacetime manifold $M$. In particular,
the reconstruction scheme can be carried out only locally.

Also, it is necessary to establish the following result regarding the map
$F_{\omega}$.

######  Lemma 11.11.

The following diagram commutes

${\mathcal{T}_{0}}$${J^{1}\tau_{\Sigma}}$${\mathcal{T}_{0}/K}$${\Sigma}$$\scriptstyle{p_{K}^{\mathcal{T}_{0}}}$$\scriptstyle{F_{\omega}}$$\scriptstyle{(\tau_{\Sigma})_{10}}$$\scriptstyle{\overline{\tau}^{\prime}_{10}}$
(11.7)

###### Proof.

In fact, for $j_{x}^{1}s\in\mathcal{T}_{0}$ we have

$\displaystyle(\tau_{\Sigma})_{10}\big{(}F_{\omega}\big{(}j_{x}^{1}s\big{)}\big{)}=(\tau_{\Sigma})_{10}\big{(}j_{x}^{1}\big{(}p_{K}^{LM}\circ
s\big{)}\big{)}=[s(x)]_{K}$

and also

$\overline{\tau}^{\prime}_{10}\big{(}p_{K}^{\mathcal{T}_{0}}\big{(}j_{x}^{1}s\big{)}\big{)}=\overline{\tau}^{\prime}_{10}\big{(}\big{[}j_{x}^{1}s\big{]}_{K}\big{)}=[s(x)]_{K},$

and the lemma follows. ∎

With this in mind, we are ready to formulate the reconstruction side of this
version of Routh reduction for Palatini gravity.

######  Theorem 11.12 (reconstruction in Palatini gravity).

Let $\zeta\colon M\to\Sigma$ be a metric satisfying the flat condition and the
Einstein–Hilbert equations of motion. Then there exists a section

$\widehat{Z}\colon\ M\to\mathcal{T}_{0}$

that is extremal of the Griffiths variational problem for Palatini gravity.

###### Proof.

The holonomic lift

$j^{1}\zeta\colon\ M\to J^{1}\tau_{\Sigma}$

is extremal for the variational problem $\big{(}J^{1}\tau_{\Sigma},{\cal
L}_{\rm EH}^{(1)},cI^{\Sigma}_{{\rm con}}\big{)}$; then, by Proposition 8.5,
there exists a section

$\widetilde{\Gamma}\colon\ M\to W_{\rm EH}$

such that $\tau_{\rm EH}\circ\widetilde{\Gamma}=j^{1}\zeta$ and

$\widetilde{\Gamma}^{*}(X\lrcorner{\rm d}\lambda_{\rm EH})=0$ (11.8)

for all $Z\in\mathfrak{X}^{V((\tau_{\Sigma})_{1}\circ\tau_{\rm EH})}(W_{\rm
EH})$.

On the other hand, by Lemma 11.8 we have a lift

$\widehat{Z}\colon\ M\to\mathcal{T}_{0}$

such that

$\overline{\tau}_{10}^{\prime}\circ
p_{K}^{\mathcal{T}_{0}}\circ\widehat{Z}=\zeta;$

by diagram (11.7) we have that

$\displaystyle\zeta=\overline{\tau}_{10}^{\prime}\circ
p_{K}^{\mathcal{T}_{0}}\circ\widehat{Z}=(\tau_{\Sigma})_{10}\circ
F_{\omega}\circ\widehat{Z}.$ (11.9)

We will define the map

$\Gamma^{\prime}:=\big{(}\widehat{Z},\widetilde{\Gamma}\big{)}\colon\
M\to\mathcal{T}_{0}\times W_{\rm EH}$

and show that it is a section of ${\rm pr}_{1}^{\omega}\colon
F^{*}_{\omega}(W_{\rm EH})\to\mathcal{T}_{0}$; namely, we have to show that

$F_{\omega}\circ\widehat{Z}=\pi_{\rm EH}\circ\widetilde{\Gamma}.$

It is important to this end to note that the conclusion of Proposition 7.4 can
be translated to this context into

$Tp_{K}^{LM}\circ\omega_{\mathfrak{p}}=F_{\omega}^{*}\overline{\omega};$

moreover, by Lemma 11.8 we know that $\widehat{Z}^{*}\omega_{\mathfrak{p}}=0$,
so

$\big{(}F_{\omega}\circ\widehat{Z}\big{)}^{*}\overline{\omega}=\widehat{Z}^{*}\big{(}F_{\omega}^{*}\overline{\omega}\big{)}=Tp_{K}^{LM}\circ\big{(}\widehat{Z}^{*}\omega_{\mathfrak{p}}\big{)}=0.$

Then the section

$F_{\omega}\circ\widehat{Z}\colon\ M\to J^{1}\tau_{\Sigma}$

is holonomic; finally, from equation (11.9) we must conclude that

$F_{\omega}\circ\widehat{Z}=j^{1}\zeta.$

But $j^{1}\zeta=\pi_{\rm EH}\circ\widetilde{\Gamma}$ by construction of the
section $\widetilde{\Gamma}$; then

$F_{\omega}\circ\widehat{Z}=\pi_{\rm EH}\circ\widetilde{\Gamma}$

and $\Gamma^{\prime}$ is a section of $F_{\omega}^{*}\left(W_{\rm EH}\right)$,
as required.

Now define the section

${\Gamma}:=\widetilde{F}_{\omega}\big{(}\widehat{Z},\widetilde{\Gamma}\big{)}\colon\
M\to W_{\rm PG};$

Therefore, for any $Z\in\mathfrak{X}^{V(\tau_{1}\circ\pi_{\rm PG})}(W_{\rm
PG})$ that is $\big{(}{\rm pr}_{2}^{\omega}\circ
F_{\omega}^{-1}\big{)}$-projectable, we have that

$\displaystyle\Gamma^{*}(Z\lrcorner{\rm d}\lambda_{\rm PG})$
$\displaystyle=(\Gamma^{\prime})^{*}\big{(}\big{(}TF_{\omega}^{-1}\circ
Z\big{)}\lrcorner{\rm d}F_{\omega}^{*}\lambda_{\rm
PG}\big{)}=(\Gamma^{\prime})^{*}\big{(}\big{(}TF_{\omega}^{-1}\circ
Z\big{)}\lrcorner{\rm d}\big{(}{\rm pr}_{2}^{\omega}\big{)}^{*}\lambda_{\rm
EH}\big{)}$ $\displaystyle=\big{(}{\rm
pr}_{2}^{\omega}\circ\Gamma^{\prime}\big{)}^{*}\big{(}\big{(}T{\rm
pr}_{2}^{\omega}\circ TF_{\omega}^{-1}\circ Z\big{)}\lrcorner{\rm
d}\lambda_{\rm EH}\big{)}$
$\displaystyle=\widetilde{\Gamma}^{*}\big{(}\big{(}T{\rm pr}_{2}^{\omega}\circ
TF_{\omega}^{-1}\circ Z\big{)}\lrcorner{\rm d}\lambda_{\rm EH}\big{)}=0$

because $\widetilde{\Gamma}$ obeys equation (11.8). ∎

##  12 Conclusions and outlook

We adapt the Routh reduction scheme developed in [6] to the case of affine
gravity with vielbeins. It suggests that this formalism could be fit to deal
with Griffiths variational problems more general than the classical, at least
with cases when the differential restrictions are a subset of those imposed by
the contact structure. Extensions of this scheme to gravity interacting with
matter fields will be studied elsewhere.

##  Appendix A An important algebraic result

First, we want to state the following algebraic proposition.

######  Proposition A.1.

Let $\left\\{c_{ijk}\right\\}$ be a set of real numbers such that

$\begin{cases}c_{ijk}\mp c_{jik}=b_{ijk},\\\ c_{ijk}\pm
c_{ikj}=a_{ijk}\end{cases}$

for some given set of real numbers $\\{a_{ijk}\\}$ and $\\{b_{ijk}\\}$ such
that $b_{ijk}\mp b_{jik}=0$ and $a_{ijk}\pm a_{ikj}=0$. Then,

$c_{ijk}=\frac{1}{2}(a_{ijk}+a_{jki}-a_{kij}+b_{ijk}+b_{kij}-b_{jki})$

is the unique solution for this linear system.

###### Proof.

From first equation we see that

$\pm c_{jik}=c_{ijk}-b_{ijk}.$

The trick now is to form the following combination

$\displaystyle a_{ijk}+a_{jki}-a_{kij}=c_{ijk}\pm c_{ikj}+c_{jki}\pm
c_{jik}-\left(c_{kij}\pm c_{kji}\right)=2c_{ijk}-b_{ijk}-b_{kij}+b_{jki},$

where in the permutation of indices was used the remaining condition. ∎

##  Appendix B Proof of Proposition 4.5

In order to do this proof, it will be necessary to bring some facts from [5].
First, we have the bundle isomorphism on $\mathcal{T}_{0}$

$W_{\rm PG}\simeq E_{2}$

where $p_{2}^{\prime}\colon E_{2}\to\mathcal{T}_{0}$ is the vector bundle

$E_{2}:=\wedge^{m-1}_{1}(\mathcal{T}_{0})\otimes S^{*}(m),$

with
$S^{*}(m):=\big{(}\mathbb{R}^{m}\big{)}^{*}\odot\big{(}\mathbb{R}^{m}\big{)}^{*}$
the set of symmetric forms on $\mathbb{R}^{m}$, and

$\wedge^{m-1}_{1}(\mathcal{T}_{0}):=\big{\\{}\gamma\in\wedge^{m-1}(\mathcal{T}_{0})\colon\gamma\text{
is horizontal respect to the projection
}\tau_{1}^{\prime}\colon\mathcal{T}_{0}\to M\big{\\}}.$

The bundle $E_{2}$ is a bundle of forms with values in a vector space;
therefore, it has a canonical $(m-1)$-form

$\Theta:=\Theta_{ij}e^{i}\odot e^{j}.$

Using the structure equations for the canonical connection on $J^{1}\tau$
(pulled back to $\mathcal{T}_{0}$), we have that the differential of the
Lagrangian form $\lambda_{\rm PG}$ is given by

$\displaystyle{\rm d}\lambda_{\rm
PG}|_{\rho}=\big{[}2\eta^{kp}(\omega_{\mathfrak{p}})_{k}^{i}\wedge\theta_{il}-(\omega_{\mathfrak{p}})^{s}_{s}\wedge\eta^{kp}\theta_{kl}+\eta^{ip}\Theta_{il}|_{\beta}\big{]}\wedge\Omega^{l}_{p}$
$\displaystyle\hphantom{{\rm d}\lambda_{\rm
PG}|_{\rho}=}{}+\eta^{ik}\big{[}{\rm
d}\Theta_{ij}|_{\beta}+\eta^{rq}\eta_{li}\Theta_{rj}|_{\beta}\wedge(\omega_{\mathfrak{k}})^{l}_{q}-\Theta_{ip}|_{\beta}\wedge(\omega_{\mathfrak{k}})^{p}_{j}\big{]}\wedge(\omega_{\mathfrak{p}})^{j}_{k}.$

The equations of motion

$\displaystyle\Gamma^{*}(X\lrcorner d\lambda_{\rm PG})=0,\qquad
X\in\mathfrak{X}^{V(\tau_{1}^{\prime}\circ\tau_{\rm PG})}(W_{\rm PG})$ (B.1)

are obtained by choosing a convenient set of vertical vector fields; because
of the identification given above, it is sufficient to give a set of vertical
vector fields on $\mathcal{T}_{0}$ and on $E_{2}$. It results that a global
basis of vertical vector fields on $\mathcal{T}_{0}$ is

$B^{\prime}:=\big{\\{}A_{J^{1}\tau},M_{rs}\big{(}\theta^{r},\big{(}E^{s}_{j}\big{)}_{LM}\big{)}^{V}\colon
A\in\mathfrak{gl}(m),M_{pq}-M_{qp}=0\big{\\}};$

in fact, the equation defining $\mathcal{T}_{0}$

$e_{\rho}^{k}e_{k\sigma}^{\mu}=e_{\sigma}^{k}e_{k\rho}^{\mu}$

is invariant by the ${\rm GL}(m)$-action, and also

$\big{(}\theta^{r},\big{(}E^{s}_{j}\big{)}_{LM}^{V}\big{)}\cdot\big{(}e_{\rho}^{k}e_{k\sigma}^{\mu}-e_{\sigma}^{k}e_{k\rho}^{\mu}\big{)}=e^{\mu}_{j}\big{(}e^{s}_{\sigma}e^{r}_{\rho}-e^{r}_{\sigma}e^{s}_{\rho}\big{)}.$

Given that $E_{2}$ is a vector bundle on $\mathcal{T}_{0}$, any section
$\beta\colon\mathcal{T}_{0}\to E_{2}$ gives rise to a vertical vector field;
the equations of motion associated to these kind of vector fields are the
metricity conditions

$\omega_{\mathfrak{p}}=0.$

Therefore, fixing an Ehresmann connection on the bundle $p_{2}^{\prime}\colon
E_{2}\to\mathcal{T}_{0}$, we can produce the set of vertical vector fields on
$E_{2}$

$(A_{\mathcal{T}_{0}})^{H},\qquad
M:=M_{rs}\big{(}\theta^{r},\big{(}E^{s}_{j}\big{)}_{LM}\big{)}^{H};$

the equations of motion associated to $M$ are

$\eta^{ks}\Gamma^{*}\big{(}M_{rs}\Theta_{kt}\wedge\theta^{r}\big{)}=0.$

The unique solution of these equations is $\Theta_{kt}=0$. In fact, by writing

$\Gamma^{*}\Theta_{kt}=\eta^{lp}N_{ktp}\theta_{l}$

and taking into account the symmetry properties of $M_{pq}$, we have that the
set of quantities $N_{pqr}$ must satisfy

$N_{pqr}-N_{qpr}=0,\qquad N_{pqr}+N_{prq}=0;$

by Proposition A.1, it results that $N_{pqr}=0$, as desired. The rest of the
equations of motion can be calculated is the same fashion that in the
$J^{1}\tau$ case; therefore, the equations (B.1) are equivalent to the
equations for the extremals of the Palatini variational problem.

##  Appendix C Proof of Proposition 6.1

First, let us write down

$\left(\frac{\partial}{\partial
x^{\mu}}\right)^{H}=M^{\nu}_{\mu}\frac{\partial}{\partial x^{\nu}}+N_{\mu
k}^{\nu}\frac{\partial}{\partial e^{\nu}_{k}}.$

Then from

$Tp_{K}^{LM}\left(\left(\frac{\partial}{\partial
x^{\mu}}\right)^{H}\right)=\frac{\partial}{\partial x^{\mu}}$ (C.1)

it results

$M^{\nu}_{\mu}=\delta^{\nu}_{\mu};$

the condition

$\omega_{K}\left(\left(\frac{\partial}{\partial x^{\mu}}\right)^{H}\right)=0$

implies

$N^{\nu}_{\mu
k}\big{(}\eta^{pk}e_{\nu}^{l}-\eta^{lk}e_{\nu}^{p}\big{)}+\eta^{lq}e^{p}_{\sigma}f^{\sigma}_{q\mu}-\eta^{pq}e^{l}_{\sigma}f^{\sigma}_{q\mu}=0.$

In order to understand this equation for the unknowns $N^{\nu}_{\mu k}$, let
us change of variables through the formula

$N^{\nu}_{\mu k}=g_{\sigma\rho}e^{\rho}_{k}N^{\nu\sigma}_{\mu};$

in terms of these new variables, and the Christoffel symbols of the connection
$\omega_{0}$

$\overline{\Gamma}^{\sigma}_{\alpha\mu}=-e_{\alpha}^{k}f_{k\mu}^{\sigma},$

the previous equations can be expressed as

$\displaystyle 0$
$\displaystyle=g_{\alpha\rho}e^{\rho}_{k}N^{\nu\alpha}_{\mu}\big{(}\eta^{pk}e_{\nu}^{l}-\eta^{lk}e_{\nu}^{p}\big{)}-\eta^{lq}e^{p}_{\sigma}e_{q}^{\alpha}\overline{\Gamma}^{\sigma}_{\alpha\mu}+\eta^{pq}e^{l}_{\sigma}e^{\alpha}_{q}\overline{\Gamma}^{\sigma}_{\alpha\mu}$
$\displaystyle=N^{\nu\alpha}_{\mu}\big{(}g_{\alpha\rho}e^{\rho}_{k}\eta^{pk}e_{\nu}^{l}-g_{\alpha\rho}e^{\rho}_{k}\eta^{lk}e_{\nu}^{p}\big{)}+\big{(}\eta^{pq}e^{\alpha}_{q}e^{l}_{\sigma}-\eta^{lq}e^{\alpha}_{q}e^{p}_{\sigma}\big{)}\overline{\Gamma}^{\sigma}_{\alpha\mu}$
$\displaystyle=N^{\nu\alpha}_{\mu}\big{(}g_{\alpha\rho}g^{\rho\beta}e_{\beta}^{p}e_{\nu}^{l}-g_{\alpha\rho}g^{\rho\beta}e_{\beta}^{l}e_{\nu}^{p}\big{)}+\big{(}e^{p}_{\beta}e^{l}_{\sigma}-e^{l}_{\beta}e^{p}_{\sigma}\big{)}g^{\alpha\beta}\overline{\Gamma}^{\sigma}_{\alpha\mu}$
$\displaystyle=N^{\sigma\beta}_{\mu}\big{(}e_{\beta}^{p}e_{\sigma}^{l}-e_{\beta}^{l}e_{\sigma}^{p}\big{)}+\big{(}e^{p}_{\beta}e^{l}_{\sigma}-e^{l}_{\beta}e^{p}_{\sigma}\big{)}g^{\alpha\beta}\overline{\Gamma}^{\sigma}_{\alpha\mu}$
$\displaystyle=\big{(}e_{\beta}^{p}e_{\sigma}^{l}-e_{\beta}^{l}e_{\sigma}^{p}\big{)}\big{(}N^{\sigma\beta}_{\mu}+g^{\alpha\beta}\overline{\Gamma}^{\sigma}_{\alpha\mu}\big{)}.$

The operator in the left is essentially an antisymmetrizator because of the
formula

$e^{\mu}_{p}e^{\nu}_{l}\big{(}e_{\beta}^{p}e_{\sigma}^{l}-e_{\beta}^{l}e_{\sigma}^{p}\big{)}=\delta_{\beta}^{\mu}\delta_{\sigma}^{\nu}-\delta_{\beta}^{\nu}\delta_{\sigma}^{\mu};$

therefore

$\displaystyle
N^{\sigma\beta}_{\mu}+g^{\alpha\beta}\overline{\Gamma}^{\sigma}_{\alpha\mu}=S^{\sigma\beta}_{\mu},$
(C.2)

where

$S^{\sigma\beta}_{\mu}-S^{\beta\sigma}_{\mu}=0.$

Finally, from the condition (C.1) we obtain

$N_{\mu
k}^{\sigma}\big{(}\eta^{kq}e_{q}^{\rho}\delta_{\sigma}^{\alpha}+\eta^{kq}e_{q}^{\alpha}\delta_{\sigma}^{\rho}\big{)}=0$

or, in terms of the variables $N^{\nu\sigma}_{\mu}$

$\displaystyle 0$
$\displaystyle=g_{\nu\beta}e^{\beta}_{k}N_{\mu}^{\sigma\nu}\big{(}\eta^{kq}e_{q}^{\rho}\delta_{\sigma}^{\alpha}+\eta^{kq}e_{q}^{\alpha}\delta_{\sigma}^{\rho}\big{)}=N_{\mu}^{\sigma\nu}\big{(}g_{\nu\beta}e^{\beta}_{k}\eta^{kq}e_{q}^{\rho}\delta_{\sigma}^{\alpha}+g_{\nu\beta}e^{\beta}_{k}\eta^{kq}e_{q}^{\alpha}\delta_{\sigma}^{\rho}\big{)}$
$\displaystyle=N_{\mu}^{\sigma\nu}\big{(}g_{\nu\beta}g^{\beta\rho}\delta_{\sigma}^{\alpha}+g_{\nu\beta}g^{\beta\alpha}\delta_{\sigma}^{\rho}\big{)}=N_{\mu}^{\sigma\nu}\big{(}\delta_{\nu}^{\rho}\delta_{\sigma}^{\alpha}+\delta_{\nu}^{\alpha}\delta_{\sigma}^{\rho}\big{)}.$

From equation (C.2) it results that

$g^{\alpha\beta}\overline{\Gamma}^{\sigma}_{\alpha\mu}+g^{\alpha\sigma}\overline{\Gamma}^{\beta}_{\alpha\mu}-2S^{\sigma\beta}_{\mu}=0,$

or equivalently

$N^{\sigma\beta}_{\mu}=\frac{1}{2}\big{(}g^{\alpha\sigma}\overline{\Gamma}^{\beta}_{\alpha\mu}-g^{\alpha\beta}\overline{\Gamma}^{\sigma}_{\alpha\mu}\big{)}.$

Therefore, we have

$\left(\frac{\partial}{\partial x^{\mu}}\right)^{H}=\frac{\partial}{\partial
x^{\mu}}+\frac{1}{2}g_{\beta\rho}e^{\rho}_{k}\big{(}g^{\alpha\sigma}\overline{\Gamma}^{\beta}_{\alpha\mu}-g^{\alpha\beta}\overline{\Gamma}^{\sigma}_{\alpha\mu}\big{)}\frac{\partial}{\partial
e^{\sigma}_{k}}.$

Additionally, we need to construct the horizontal lifts

$\left(\frac{\partial}{\partial
g^{\mu\nu}}\right)^{H}=P_{\mu\nu}^{\sigma}\frac{\partial}{\partial
x^{\sigma}}+Q_{\mu\nu k}^{\sigma}\frac{\partial}{\partial e^{\sigma}_{k}}$

with $P_{\mu\nu}^{\sigma}-P_{\nu\mu}^{\sigma}=0$, $Q_{\mu\nu
k}^{\sigma}-Q_{\nu\mu k}^{\sigma}=0$. The equation

$\displaystyle Tp_{K}^{LM}\left(\left(\frac{\partial}{\partial
g^{\mu\nu}}\right)^{H}\right)=\frac{\partial}{\partial g^{\mu\nu}}$

and the identity (6.1) imply

$P_{\mu\nu}^{\sigma}\frac{\partial}{\partial x^{\sigma}}+Q_{\mu\nu
k}^{\sigma}\left(\eta^{kq}e_{q}^{\rho}\delta_{\sigma}^{\alpha}+\eta^{kq}e_{q}^{\alpha}\delta_{\sigma}^{\rho}\right)\frac{\partial}{\partial
g^{\alpha\rho}}=\frac{\partial}{\partial g^{\mu\nu}},$

namely

$P_{\mu\nu}^{\sigma}=0$

and (given the symmetry properties of $g^{\mu\nu}$)

$\displaystyle\frac{1}{2}\big{(}\delta_{\mu}^{\alpha}\delta^{\rho}_{\nu}+\delta_{\nu}^{\alpha}\delta^{\rho}_{\mu}\big{)}=Q_{\mu\nu
k}^{\sigma}\big{(}\eta^{kq}e_{q}^{\rho}\delta_{\sigma}^{\alpha}+\eta^{kq}e_{q}^{\alpha}\delta_{\sigma}^{\rho}\big{)}=\eta^{kq}e_{q}^{\rho}Q_{\mu\nu
k}^{\alpha}+\eta^{kq}e_{q}^{\alpha}Q_{\mu\nu k}^{\rho}.$ (C.3)

The horizontality condition

$\omega_{K}\left(\left(\frac{\partial}{\partial
g^{\mu\nu}}\right)^{H}\right)=0$

will be equivalent to

$\displaystyle\big{(}\eta^{pk}e^{l}_{\sigma}-\eta^{lk}e^{p}_{\sigma}\big{)}Q_{\mu\nu
k}^{\sigma}=0.$ (C.4)

These conditions can be understood by introducing the variables

$Q_{\mu\nu}^{\alpha\rho}:=\eta^{kq}e_{q}^{\rho}Q_{\mu\nu k}^{\alpha};$

then, equation (C.3) becomes

$Q_{\mu\nu}^{\alpha\rho}+Q_{\mu\nu}^{\rho\alpha}=\frac{1}{2}\left(\delta_{\mu}^{\alpha}\delta^{\rho}_{\nu}+\delta_{\nu}^{\alpha}\delta^{\rho}_{\mu}\right)$

and equation (C.4) is equivalent to

$Q_{\mu\nu}^{\alpha\rho}-Q_{\mu\nu}^{\rho\alpha}=0.$

Therefore

$\displaystyle
Q_{\mu\nu}^{\alpha\rho}+Q_{\mu\nu}^{\rho\alpha}=\frac{1}{2}\big{(}Q_{\mu\nu}^{\alpha\rho}+Q_{\mu\nu}^{\rho\alpha}\big{)}+\frac{1}{2}\big{(}Q_{\mu\nu}^{\alpha\rho}-Q_{\mu\nu}^{\rho\alpha}\big{)}=\frac{1}{4}\big{(}\delta_{\mu}^{\alpha}\delta^{\rho}_{\nu}+\delta_{\nu}^{\alpha}\delta^{\rho}_{\mu}\big{)}$

and so

$\displaystyle\left(\frac{\partial}{\partial g^{\mu\nu}}\right)^{H}=Q_{\mu\nu
k}^{\alpha}\frac{\partial}{\partial
e^{\alpha}_{k}}=\frac{1}{4}\eta_{kl}e^{l}_{\rho}\big{(}\delta_{\mu}^{\alpha}\delta^{\rho}_{\nu}+\delta_{\nu}^{\alpha}\delta^{\rho}_{\mu}\big{)}\frac{\partial}{\partial
e^{\alpha}_{k}}=\frac{1}{4}g_{\rho\beta}e^{\beta}_{k}\big{(}\delta_{\mu}^{\alpha}\delta^{\rho}_{\nu}+\delta_{\nu}^{\alpha}\delta^{\rho}_{\mu}\big{)}\frac{\partial}{\partial
e^{\alpha}_{k}}.$

### Acknowledgements

The author thanks the CONICET and UNS for financial support, and Eduardo
García-Toraño for valuable discussion regarding aspects of Routh reduction
contained in this article, as well as for pointing me out reference [28].
Also, the author would like to warmly thank the referees for the care they put
in reviewing this work. The article has been greatly improved by their
suggestions.

## References

  * [1] Arnowitt R., Deser S., Misner C.W., The dynamics of general relativity, Gen. Relativity Gravitation 40 (2004), 1997–2027, arXiv:gr-qc/0405109.
  * [2] Barbero-Liñán M., Echeverría-Enríquez A., Martín de Diego D., Muñoz Lecanda M.C., Román-Roy N., Skinner–Rusk unified formalism for optimal control systems and applications, J. Phys. A: Math. Gen. 40 (2007), 12071–12093, arXiv:0705.2178.
  * [3] Capriotti S., Differential geometry, Palatini gravity and reduction, J. Math. Phys. 55 (2014), 012902, 29 pages, arXiv:1209.3596.
  * [4] Capriotti S., Routh reduction and Cartan mechanics, J. Geom. Phys. 114 (2017), 23–64, arXiv:1606.02630.
  * [5] Capriotti S., Unified formalism for Palatini gravity, Int. J. Geom. Methods Mod. Phys. 15 (2018), 1850044, 33 pages, arXiv:1707.06057.
  * [6] Capriotti S., García-Toraño Andrés E., Routh reduction for first-order Lagrangian field theories, Lett. Math. Phys. 109 (2019), 1343–1376, arXiv:1909.10088.
  * [7] Capriotti S., Gaset J., Román-Roy N., Salomone L., Griffiths variational multisymplectic formulation for Lovelock gravity, arXiv:1911.07278.
  * [8] Castrillón López M., García P.L., Rodrigo C., Euler–Poincaré reduction in principal bundles by a subgroup of the structure group, J. Geom. Phys. 74 (2013), 352–369.
  * [9] Castrillón López M., Muñoz Masqué J., The geometry of the bundle of connections, Math. Z. 236 (2001), 797–811.
  * [10] Castrillón López M., Muñoz Masqué J., Rosado María E., First-order equivalent to Einstein–Hilbert Lagrangian, J. Math. Phys. 55 (2014), 082501, 9 pages, arXiv:1306.1123.
  * [11] Castrillón López M., Ratiu T.S., Reduction in principal bundles: covariant Lagrange–Poincaré equations, Comm. Math. Phys. 236 (2003), 223–250.
  * [12] Castrillón López M., Ratiu T.S., Shkoller S., Reduction in principal fiber bundles: covariant Euler–Poincaré equations, Proc. Amer. Math. Soc. 128 (2000), 2155–2164, arXiv:math.DG/9908102.
  * [13] Cattaneo A.S., Schiavina M., The reduced phase space of Palatini–Cartan–Holst theory, Ann. Henri Poincaré 20 (2019), 445–480, arXiv:1707.05351.
  * [14] Crampin M., Mestdag T., Routh’s procedure for non-abelian symmetry groups, J. Math. Phys. 49 (2008), 032901, 28 pages, arXiv:0802.0528.
  * [15] Dadhich N., Pons J.M., On the equivalence of the Einstein–Hilbert and the Einstein–Palatini formulations of general relativity for an arbitrary connection, Gen. Relativity Gravitation 44 (2012), 2337–2352, arXiv:1010.0869.
  * [16] Echeverría-Enríquez A., López C., Marín-Solano J., Muñoz Lecanda M.C., Román-Roy N., Lagrangian–Hamiltonian unified formalism for field theory, J. Math. Phys. 45 (2004), 360–380, arXiv:math-ph/0212002.
  * [17] Ellis D.C.P., Gay-Balmaz F., Holm D.D., Ratiu T.S., Lagrange–Poincaré field equations, J. Geom. Phys. 61 (2011), 2120–2146, arXiv:0910.0874.
  * [18] García-Toraño Andrés E., Mestdag T., Yoshimura H., Implicit Lagrange–Routh equations and Dirac reduction, J. Geom. Phys. 104 (2016), 291–304, arXiv:1509.01946.
  * [19] Gaset J., Román-Roy N., Multisymplectic unified formalism for Einstein–Hilbert gravity, J. Math. Phys. 59 (2018), 032502, 39 pages, arXiv:1705.00569.
  * [20] Gotay M.J., An exterior differential systems approach to the Cartan form, in Symplectic Geometry and Mathematical Physics (Aix-en-Provence, 1990), Progr. Math., Vol. 99, Birkhäuser Boston, Boston, MA, 1991, 160–188.
  * [21] Gotay M.J., Isenberg J., Marsden J.E., Montgomery R., Momentum Maps And Classical Relativistic Fields. Part I: Covariant field theory, arXiv:physics/9801019.
  * [22] Griffiths P.A., Exterior differential systems and the calculus of variations, Progress in Mathematics, Vol. 25, Birkhäuser, Boston, Mass., 1983\.
  * [23] Hehl F.W., McCrea J.D., Mielke E.W., Ne’eman Y., Metric-affine gauge theory of gravity: field equations, Noether identities, world spinors, and breaking of dilation invariance, Phys. Rep. 258 (1995), 1–171, arXiv:gr-qc/9402012.
  * [24] Hehl F.W., von der Heyde P., Kerlick G.D., Nester J.M., General relativity with spin and torsion: foundations and prospect, Rev. Modern Phys. 48 (1976), 393–416.
  * [25] Higham N.J., $J$-orthogonal matrices: properties and generation, SIAM Rev. 45 (2003), 504–519.
  * [26] Hsu L., Calculus of variations via the Griffiths formalism, J. Differential Geom. 36 (1992), 551–589.
  * [27] Ibort A., Spivak A., On a covariant Hamiltonian description of Palatini’s gravity on manifolds with boundary, arXiv:1605.03492.
  * [28] Kharlamov M.P., Characteristic class of a bundle and the existence of a global Routh function, Funct. Anal. Appl. 11 (1977), 80–81.
  * [29] Kobayashi S., Nomizu K., Foundations of differential geometry. Vol. I, Interscience Publishers, New York – London, 1963.
  * [30] Krupka D., Introduction to global variational geometry, Atlantis Studies in Variational Geometry, Vol. 1, Atlantis Press, Paris, 2015.
  * [31] Langerock B., López M.C., Routh reduction for singular Lagrangians, Int. J. Geom. Methods Mod. Phys. 7 (2010), 1451–1489, arXiv:1007.0325.
  * [32] Marsden J.E., Ratiu T.S., Scheurle J., Reduction theory and the Lagrange–Routh equations, J. Math. Phys. 41 (2000), 3379–3429.
  * [33] Nawarajan D., Visser M., Global properties of physically interesting Lorentzian spacetimes, Internat. J. Modern Phys. D 25 (2016), 1650106, 15 pages, arXiv:1601.03355.
  * [34] Pars L.A., A treatise on analytical dynamics, John Wiley & Sons, Inc., New York, 1965.
  * [35] Peldán P., Actions for gravity, with generalizations: a review, Classical Quantum Gravity 11 (1994), 1087–1132, arXiv:gr-qc/9305011.
  * [36] Prieto-Martínez P.D., Román-Roy N., A new multisymplectic unified formalism for second order classical field theories, J. Geom. Mech. 7 (2015), 203–253, arXiv:1402.4087.
  * [37] Romano J.D., Geometrodynamics vs. connection dynamics, Gen. Relativity Gravitation 25 (1993), 759–854, arXiv:gr-qc/9303032.
  * [38] Sardanashvily G., Classical gauge gravitation theory, Int. J. Geom. Methods Mod. Phys. 8 (2011), 1869–1895, arXiv:1110.1176.
  * [39] Saunders D.J., The geometry of jet bundles, London Mathematical Society Lecture Note Series, Vol. 142, Cambridge University Press, Cambridge, 1989.
  * [40] Tsamparlis M., On the Palatini method of variation, J. Math. Phys. 19 (1978), 555–557.
  * [41] Vey D., Multisymplectic formulation of vielbein gravity: I. De Donder–Weyl formulation, Hamiltonian $(n-1)$-forms, Classical Quantum Gravity 32 (2015), 095005, 50 pages, arXiv:1404.3546.


# MaPLe: Multi-modal Prompt Learning

Muhammad Uzair Khattak1, , Hanoona Rasheed1, Muhammad Maaz1
Salman Khan1, 2, Fahad Shahbaz Khan1,3
1Mohamed bin Zayed University of AI, UAE
2Australian National University, Australia
3Linköping University, Sweden
<EMAIL_ADDRESS>
###### Abstract

Pre-trained vision-language (V-L) models such as CLIP have shown excellent
generalization ability to downstream tasks. However, they are sensitive to the
choice of input text prompts and require careful selection of prompt templates
to perform well. Inspired by the Natural Language Processing (NLP) literature,
recent CLIP adaptation approaches learn prompts as the textual inputs to fine-
tune CLIP for downstream tasks. We note that using prompting to adapt
representations in a single branch of CLIP (language or vision) is sub-optimal
since it does not allow the flexibility to dynamically adjust both
representation spaces on a downstream task. In this work, we propose Multi-
modal Prompt Learning (MaPLe) for _both_ vision and language branches to
improve alignment between the vision and language representations. Our design
promotes strong coupling between the vision-language prompts to ensure mutual
synergy and discourages learning independent uni-modal solutions. Further, we
learn separate prompts across different early stages to progressively model
the stage-wise feature relationships to allow rich context learning. We
evaluate the effectiveness of our approach on _three_ representative tasks of
generalization to novel classes, new target datasets and unseen domain shifts.
Compared with the state-of-the-art method Co-CoOp, MaPLe exhibits favorable
performance and achieves an absolute gain of 3.45% on novel classes and 2.72%
on overall harmonic-mean, averaged over 11 diverse image recognition datasets.
Code: https://tinyurl.com/2dzs8f3w.

##  1 Introduction

Foundational vision-language models such as CLIP (Contrastive Language-Image
Pretraining) (Radford et al., 2021) have shown excellent generalization
ability to downstream tasks. Such models are trained to align language and
vision modalities on web-scale data _e.g._ , 400 million text-image pairs in
the case of CLIP. The resulting model can reason about the open-vocabulary
visual concepts, thanks to the rich supervision provided by natural language.
During inference, hand-engineered text prompts are used _e.g._ , ‘a photo of a
$<$category$>$’ as a query for text encoder. The output text embeddings are
matched with the visual embeddings from an image encoder to predict the output
class. Designing high quality contextual prompts have been proven to enhance
the performance of CLIP and other V-L models (Jin et al., 2021; Yao et al.,
2021b).

Despite the effectiveness of CLIP towards generalization to new concepts, its
massive scale and the scarcity of training data (_e.g._ , few-shot setting)
makes it infeasible to fine-tune the full model for downstream tasks. Such
fine-tuning can also forget the useful knowledge acquired in the large-scale
pretraining phase and can pose a risk of overfitting to the downstream task.
To address the above challenges, existing works propose language prompt
learning to avoid manually adjusting the prompt templates and providing a
mechanism to adapt the model while keeping the original weights frozen (Zhou
et al., 2022a; b; Lu et al., 2022; Huang et al., 2022; Manli et al., 2022).
Owing to the inspiration from Natural Language Processing (NLP) models, these
approaches only explore prompt learning for the text encoder in CLIP (Fig.
1:a) while adaptation choices together with an equally important image encoder
of CLIP remains an unexplored topic in the literature.

Our motivation derives from the multi-modal nature of CLIP, where a text and
image encoder co-exist and _both_ contribute towards properly aligning the
vision-language modalities. We argue that any prompting technique should adapt
the model completely and therefore, learning prompts only for the text encoder
in CLIP is not sufficient to model the adaptations needed for the image
encoder. To this end, we set out to achieve completeness in the prompting
approach and propose Multi-modal Prompt Learning (MaPLe) to adequately fine-
tune the text and image encoder representations such that their optimal
alignment can be achieved on the downstream tasks (Fig. 1:b). Our extensive
experiments on three key representative settings including base-to-novel class
generalization, cross-dataset evaluation, and domain generalization
demonstrate the strength of MaPLe. On base-to-novel class generalization, our
proposed MaPLe outperforms existing prompt learning approaches across 11
diverse image recognition datasets (Fig. 1:c) and achieves absolute average
gain of 3.45% on novel classes and 2.72% on harmonic-mean over the state-of-
the-art method Co-CoOp (Zhou et al., 2022a). Further, MaPLe demonstrates
favorable generalization ability and robustness in cross-dataset transfer and
domain generalization settings, leading to consistent improvements compared to
existing approaches. Owing to its streamlined architectural design, MaPLe
exhibits improved efficiency during both training and inference without much
overhead, as compared to Co-CoOp which lacks efficiency due to its image
instance conditioned design.

In summary, the main contributions of this work include:

  * •

We propose _multi-modal_ prompt learning to favourably align the vision-
language representations in CLIP. To the best of our knowledge, this is the
first multi-modal prompting approach developed for fine-tuning CLIP.

  * •

In order to link prompts learned in text and image encoders, we propose a
_coupling function_ to explicitly condition vision prompts on their language
counterparts. It acts as a bridge between the two modalities and allows mutual
propagation of gradients to promote synergy.

  * •

Our multi-modal prompts are learned across multiple transformer blocks in both
vision and language branches to _progressively_ learn the synergistic
behaviour of both modalities. This deep prompting strategy allows modeling the
contextual relationships independently, thus providing more flexibility to
align the vision-language representations.

Figure 1: Comparison of MaPLe with standard prompt learning methods. (a)
Existing methods adopt uni-modal prompting techniques to fine-tune CLIP
representations as prompts are learned only in a single branch of CLIP
(language or vision). (b) MaPLe introduces branch-aware hierarchical prompts
that adapt both language and vision branches simultaneously for improved
generalization. (c) MaPLe surpasses state-of-the-art methods on 11 diverse
image recognition datasets for novel class generalization task.

##  2 Related Work

Vision Language Models: The combined use of language supervision with natural
images is found to be of great interest in the computer vision community. In
contrast to models learned with only image supervision, these vision-language
(V-L) models encode rich multimodal representations. Recently, V-L models like
CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), LiT (Zhai et al.,
2022), FILIP (Yao et al., 2021a) and Florence Yuan et al. (2021) have
demonstrated exceptional performance on a wide spectrum of tasks including
few-shot and zero-shot visual recognition. These models learn joint image-
language representations in a self-supervised manner using abundantly
available data from the web. For example, CLIP and ALIGN respectively use
$\sim$400M and $\sim$1B image-text pairs to train a multi-modal network.
Although these pre-trained V-L models learn generalized representations,
efficiently adapting them to downstream tasks is still a challenging problem.
Many works have demonstrated better performance on downstream tasks by using
tailored methods to adapt V-L models for few-shot image-recognition (Gao et
al., 2021; Zhang et al., 2022a; Kim et al., 2022), object detection (Rasheed
et al., 2022; Maaz et al., 2022; Zhou et al., 2022c; Gu et al., 2021; Zang et
al., 2022; Feng et al., 2022), and segmentation (Li et al., 2022; Rao et al.,
2022; Ding et al., 2022; Lüddecke & Ecker, 2022). In this work, we propose a
novel multi-modal prompt learning technique to effectively adapt CLIP for few-
shot and zero-shot visual recognition tasks.

Prompt Learning: The instructions in the form of a sentence, known as text
prompt, are usually given to the language branch of a V-L model, allowing it
to better understand the task. The prompt can be handcrafted for a downstream
task or learned automatically during fine-tuning stage. The latter is referred
to as ‘Prompt Learning’ which was first used in NLP (Li & Liang, 2021; Lester
et al., 2021; Liu et al., 2021) followed by the adaptation in V-L (Zhou et
al., 2022b; a; Zhu et al., 2022) and vision-only (Jia et al., 2022; Zhang et
al., 2022b; Wang et al., 2022a; b) models.

Prompt Learning in Vision Language models: Full fine-tuning and linear probing
(Gao et al., 2021) are two typical approaches to adapt a V-L model (_i.e._
CLIP) to the downstream tasks. The complete fine-tuning results in degrading
the previously learned joint V-L representation while linear probing limits
the zero-shot capability of CLIP. To this end, inspired from prompt learning
in NLP, many works have proposed to adapt V-L models by learning the prompt
tokens in an end-to-end training. CoOp (Zhou et al., 2022b) fine-tunes CLIP
for few-shot image classification by optimizing continuous set of prompt
vectors at its language branch. Co-CoOp (Zhou et al., 2022a) highlights the
inferior performance of CoOp on novel classes and solves the generalization
issue by explicitly conditioning prompts on image instances. Lu et al. (2022)
proposes to optimize multiple set of prompts by learning the distribution of
prompts. Ju et al. (2021) adapt CLIP by learning prompts for video
understanding tasks. Bahng et al. (2022) perform visual prompt tuning on CLIP
by prompting on the vision branch. We note that the existing methods follow
independent uni-modal solutions and learn prompts either in the language or in
the vision branch of CLIP, thus adapting CLIP partially. In this paper, we
explore an important question: given the multimodal nature of CLIP, is
complete prompting (_i.e._ , in both language and vision branches) better
suited to adapt CLIP? Our work is the first to answer this question by
investigating the effectiveness of multi-modal prompt learning in order to
improve alignment between vision and language representations.

##  3 Method

Our approach concerns with fine-tuning a pre-trained multi-modal CLIP for
better generalization to downstream tasks through context optimization via
prompting. Fig. 2 shows the overall architecture of our proposed MaPLe (Multi-
modal Prompt Learning) framework. Unlike previous approaches (Zhou et al.,
2022b; a) which learn context prompts only at the language branch, MaPLe
proposes a joint prompting approach where the context prompts are learned in
both vision and language branches. Specifically, we append learnable context
tokens in the language branch and explicitly condition the vision prompts on
the language prompts via a coupling function to establish interaction between
them. In order to learn hierarchical contextual representations, we introduce
deep prompting in both branches through separate learnable context prompts
across different transformer blocks. During the fine-tuning stage, only the
context prompts along with their coupling function are learned while the rest
of the model is frozen. Below, we first outline the pre-trained CLIP
architecture and then present our proposed fine-tuning approach.

Figure 2: Overview of our proposed MaPLe (Multi-modal Prompt Learning)
framework for prompt learning in V-L models. MaPLe tunes both vision and
language branches where only the context prompts are learned, while the rest
of the model is frozen. MaPLe conditions the vision prompts on language
prompts via a V-L coupling function $\mathcal{F}$ to induce mutual synergy
between the two modalities. Our framework uses deep contextual prompting where
separate context prompts are learned across multiple transformer blocks.

###  3.1 Revisiting CLIP

We build our approach on a pre-trained vision-language (V-L) model, CLIP,
which consists of a text and vision encoder. Consistent with existing
prompting methods (Zhou et al., 2022b; a), we use a vision transformer (ViT)
(Dosovitskiy et al., 2021) based CLIP model. CLIP encodes an image
$I\in\mathbb{R}^{H\times W\times 3}$ and a corresponding text description as
explained below.

Encoding Image: Image encoder $\mathcal{V}$ with $K$ transformer layers
$\\{\mathcal{V}_{i}\\}_{i=1}^{K}$, splits the image $I$ into $M$ fixed-size
patches which are projected into patch embeddings $E_{0}\in\mathbb{R}^{M\times
d_{v}}$. Patch embeddings $E_{i}$ are then input to the $(i+1)^{\text{th}}$
transformer block $(\mathcal{V}_{i+1})$ along with an appended learnable class
(CLS) token $\text{c}_{i}$ and sequentially processed through $K$ transformer
blocks,

$\displaystyle[{\text{c}}_{i},E_{i}]$
$\displaystyle=\mathcal{V}_{i}([\text{c}_{i-1},E_{i-1}])\qquad\qquad
i=1,2,\cdots,K.$

To obtain the final image representation $x$, the class token $\text{c}_{K}$
of last transformer layer $(\mathcal{V}_{K})$ is projected to a common V-L
latent embedding space via ImageProj,

$\displaystyle x$ $\displaystyle=\texttt{ImageProj}({\text{c}}_{K})\ \ \ \ \
\qquad x\in\mathbb{R}^{d_{vl}}.$

Encoding Text: CLIP text encoder generates feature representations for text
description by tokenizing the words and projecting them to word embeddings
${W}_{0}=[w_{0}^{1},w_{0}^{2},\cdots,w_{0}^{N}]\in\mathbb{R}^{N\times d_{l}}$.
At each stage, ${W}_{i}$ is input to the $(i+1)^{\text{th}}$ transformer layer
of text encoding branch ($\mathcal{L}_{i+1})$,

$\displaystyle\hskip 71.13188pt[{W}_{i}]=\mathcal{L}_{i}({W}_{i-1})\ \ \
\qquad\qquad\qquad i=1,2,\cdots,K.$

The final text representation $z$ is obtained by projecting the text
embeddings corresponding to the last token of the last transformer block
$\mathcal{L}_{K}$ to a common V-L latent embedding space via TextProj,

$\displaystyle z=\texttt{TextProj}({w_{K}^{N}})\ \ \ \ \ \qquad
z\in\mathbb{R}^{d_{vl}}.$

Zero-shot Classification: For zero-shot classification, text prompts are hand-
crafted with class labels $y\in\\{1,2,\ldots C\\}$ (_e.g._ , ‘a photo of a
$<$category$>$’) having $C$ classes. Prediction $\hat{y}$ corresponding to the
image $I$ having the highest cosine similarity score ($sim(\cdot)$) is
formulated as,

$\displaystyle
p(\hat{y}|x)=\frac{\text{exp}(sim(x,z_{\hat{y}})/\tau)}{\sum_{i=1}^{C}\text{exp}(sim(x,z_{i}))}.$

Here, $\tau$ is a temperature parameter.

###  3.2 MaPLe: Multi-modal Prompt Learning

To efficiently fine-tune CLIP for downstream image recognition tasks, we
explore the potential of multi-modal prompt tuning. We reason that prior works
that have predominantly explored uni-modal approaches are less suitable as
they do not offer the flexibility to dynamically adapt both language and
vision representation spaces. Thus to achieve completeness in prompting, we
underline the importance of multi-modal prompting approach. In Fig. 3, we
visualize and compare the image embeddings of MaPLe with recent state-of-the-
art work, Co-CoOp. Note that the image embeddings of CLIP, CoOp and Co-CoOp
will be identical as they do not learn prompts in the vision branch. The
visualization shows that image embeddings of MaPLe are more separable
indicating that learning vision prompts in addition to language prompts leads
to better adaptation of CLIP.

Figure 3: t-SNE plots of image embeddings in uni-modal prompting method Co-
CoOp, and MaPLe on 3 diverse image recognition datasets. MaPLe shows better
separability in both base and novel classes.

In addition to multi-modal prompting, we find that it is essential to learn
prompts in the deeper transformer layers to progressively model stage-wise
feature representations. To this end, we propose to introduce learnable tokens
in the first $J$ (where $J<K$) layers of both vision and language branches.
These multi-modal hierarchical prompts utilize the knowledge embedded in CLIP
model to effectively learn task relevant contextual representations (see Fig.
4).

####  3.2.1 Deep Language Prompting

To learn the language context prompts, we introduce $b$ learnable tokens
$\\{P^{i}\in\mathbb{R}^{d_{l}}\\}_{i=1}^{b}$, in the language branch of CLIP.
The input embeddings now follow the form $[P^{1},P^{2},\cdots,P^{b},W]$, where
$W=[w^{1},w^{2},\cdots,w^{N}]$ corresponds to fixed input tokens. New
learnable tokens are further introduced in each transformer block of the
language encoder ($\mathcal{L}_{i}$) up to a specific depth $J$,

$\displaystyle[\ \underline{\hskip 8.5359pt},\
W_{i}]=\mathcal{L}_{i}([P_{i-1},W_{i-1}])\qquad\qquad i=1,2,\cdots,J.$ (1)

Here $[\cdot,~{}\cdot]$ refers to the concatenation operation. After
$J^{\text{th}}$ transformer layer, the subsequent layers process previous
layer prompts and final text representation $z$ is computed,

$\displaystyle[P_{j},\ W_{i}]$
$\displaystyle=\mathcal{L}_{j}([P_{j-1},W_{j-1}])\qquad\qquad j=J+1,\cdots,K,$
(2) $\displaystyle z$ $\displaystyle=\texttt{TextProj}(w_{K}^{N}).$ (3)

When $J=1$, the learnable tokens $P$ are only applied at the input of first
transformer layer, and this deep language prompting technique degenerates to
CoOp (Zhou et al., 2022b).

####  3.2.2 Deep Vision Prompting

Similar to deep language prompting, we introduce $b$ learnable tokens
$\\{\tilde{P}^{i}\in\mathbb{R}^{d_{v}}\\}_{i=1}^{b}$, in the vision branch of
CLIP alongside the input image tokens. New learnable tokens are further
introduced in deeper transformer layers of the image encoder ($\mathcal{V}$)
up to depth $J$.

$\displaystyle[c_{i},E_{i},\ \underline{\hskip 8.5359pt}\ ]$
$\displaystyle=\mathcal{V}_{i}([c_{i-1},E_{i-1},\tilde{P}_{i-1}])\
\qquad\qquad i=1,2,\cdots,J,$ $\displaystyle[c_{j},E_{j},\tilde{P}_{j}]$
$\displaystyle=\mathcal{V}_{j}([c_{j-1},E_{j-1},\tilde{P}_{j-1}])\qquad\qquad
j=J+1,\cdots,K,$ $\displaystyle x$
$\displaystyle=\texttt{ImageProj}({c}_{K}).$

Our deep prompting provides the flexibility to learn prompts across different
feature hierarchies within the ViT architecture. We find that sharing prompts
across stages is better compared to independent prompts as features are more
correlated due to successive transformer block processing. Thus, the later
stages do not provide independently-learned complimentary prompts as compared
to the early stages.

####  3.2.3 Vision Language Prompt Coupling

We reason that in prompt tuning it is essential to take a multi-modal approach
and simultaneously adapt both the vision and language branch of CLIP in order
to achieve completeness in context optimization. A simple approach would be to
naively combine deep vision and language prompting, where both the language
prompts $P$, and the vision prompts $\tilde{P}$, will be learned during the
same training schedule. We name this design as ‘Independent V-L Prompting’.
Although this approach satisfies the requirement of completeness in prompting,
this design lacks synergy between vision and language branch as both branches
do not interact while learning the task relevant context prompts.

To this end, we propose a branch-aware multi-modal prompting which tunes
vision and language branch of CLIP together by sharing prompts across both
modalities. Language prompt tokens are introduced in the language branch up to
$J^{\text{th}}$ transformer block similar to deep language prompting as
illustrated in Eqs. 1-3. To ensure mutual synergy between V-L prompts, vision
prompts $\tilde{P}$, are obtained by projecting language prompts $P$ via
vision-to-language projection which we refer to as V-L coupling function
$\mathcal{F(\cdot)}$, such that $\tilde{P}_{k}={\mathcal{F}_{k}(P_{k})}$. The
coupling function is implemented as a linear layer which maps $d_{l}$
dimensional inputs to $d_{v}$. This acts as a bridge between the two
modalities, thus encouraging mutual propagation of gradients.

$\displaystyle[c_{i},E_{i},\ \underline{\hskip 8.5359pt}\ ]$
$\displaystyle=\mathcal{V}_{i}([c_{i-1},E_{i-1},\bm{\mathcal{F}_{i-1}(P_{i-1})}])\
\qquad\qquad i=1,2,\cdots,J$ $\displaystyle[c_{j},E_{j},\tilde{P}_{j}]$
$\displaystyle=\mathcal{V}_{j}([c_{j-1},E_{j-1},\tilde{P}_{j-1}])\ \
\qquad\qquad\qquad j=J+1,\cdots,K$ $\displaystyle x$
$\displaystyle=\texttt{ImageProj}({c}_{K})$

Unlike independent V-L prompting, the explicit conditioning of $\tilde{P}$ on
$P$ helps to learn prompts in a shared embedding space between language and
vision branch, thus improving the mutual synergy.

##  4 Experiments

###  4.1 Benchmark setting

We evaluate and conduct experiments in three benchmark settings for image
recognition.

Generalization from Base-to-Novel Classes: To evaluate the generalizability of
our approach, we follow a zero-shot setting where the datasets are split into
base and novel classes. The model is trained only on the base classes in a
few-shot setting and evaluated on both base and novel categories.

Cross-dataset Evaluation: To validate the potential of our approach in cross-
dataset transfer, we evaluate our ImageNet trained model directly on other
datasets. Consistent with Co-CoOp, our model is trained on all 1000 ImageNet
classes in a few-shot manner.

Domain Generalization: We evaluate the robustness of our method on out-of-
distribution datasets. Similar to cross-dataset evaluation, we test our
ImageNet trained model directly on four other ImageNet datasets that contain
various types of domain shifts.

Datasets: For generalization from base-to-novel classes and cross-dataset
evaluation, we follow Zhou et al. (2022b; a) and evaluate the performance of
our method on 11 image classification datasets which covers a wide range of
recognition tasks. This includes two generic-objects datasets, ImageNet (Deng
et al., 2009) and Caltech101 (Fei-Fei et al., 2004); five fine-grained
classifcation datasets, OxfordPets (Parkhi et al., 2012), StanfordCars (Krause
et al., 2013), Flowers102 (Nilsback & Zisserman, 2008), Food101 (Bossard et
al., 2014), and FGVCAircraft (Maji et al., 2013); a scene recognition dataset
SUN397 (Xiao et al., 2010); an action recognition dataset UCF101 (Soomro et
al., 2012); a texture dataset DTD (Cimpoi et al., 2014) and a satellite-image
dataset EuroSAT (Helber et al., 2019). For domain generalization experiments,
we use ImageNet as source dataset and its four variants as target datasets
including ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al.,
2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNet-R (Hendrycks et al.,
2021a).

###  4.2 Implementation Details

We use a few-shot training strategy in all experiments where the number of
shots are set to 16 which are randomly sampled for each class. We apply prompt
tuning on a pre-trained ViT-B/16 CLIP model where $d_{l}=512$, $d_{v}=768$ and
$d_{vl}=512$. For MaPLe, we set prompt depth $J$ to 9 and the language and
vision prompt lengths to 2. All models are trained for 5 epochs with a batch-
size of 4 and a learning rate of 0.0035 via SGD optimizer on a single NVIDIA
A100 GPU. We report base and novel class accuracies and their harmonic mean
(HM) averaged over 3 runs. We initialize the language prompts of the first
layer $P_{0}$ with the pre-trained CLIP word embeddings of the template ‘a
photo of a $<$category$>$’, while for the subsequent layers they are randomly
initialized from a normal distribution. For training MaPLe on all 1000 classes
of ImageNet as a source model, prompt depth $J$ is set to 3 and the model
trained for 2 epochs with learning rate of 0.0026. Hyper-parameters for deep
language prompting, deep vision prompting, and independent V-L prompting are
detailed in Appendix A. The corresponding hyper-parameters are fixed across
all datasets.

###  4.3 Prompting CLIP via Vision-Language Prompts

Prompting Variants: We first evaluate the performance of different possible
prompting design choices as an ablation for our proposed branch-aware multi-
modal prompting, MaPLe. These variants include deep language prompting, deep
vision prompting and independent V-L prompting. In Table 1, we present the
results averaged over the 11 image recognition datasets. Deep language
prompting (row-2) shows improvements over deep vision prompting (row-1),
indicating that prompts learned at the language branch provide better
adaptation of CLIP. Although separately combining the above two approaches
(row-3) further improves the performance, it struggles to achieve
comprehensive benefits from the language and vision branches. We hypothesize
that this is due to the lack of synergy between the learned vision and
language prompts as they do not interact with each other during training.
Meanwhile, MaPLe (row-4) combines the benefits of prompting in both branches
by enforcing interactions through explicit conditioning of vision prompts on
the language prompts. It provides improvements on novel and base class
accuracies which leads to the best HM of 78.55%. We explore other possible
design choices and present the ablations in Appendix C.

Table 1:  Comparison of MaPLe with different deep prompting designs in base-
to-novel generalization. Results are averaged over 11 datasets. Co-CoOp ( Zhou
et al. (2022a)) is the previous state-of-the-art. HM refers to harmonic mean.

Method | Base Acc. | Novel Acc. | HM
---|---|---|---
Co-CoOp (CVPR’22) | 80.47 | 71.69 | 75.83
1: Deep vision prompting | 80.24 | 73.43 | 76.68
2: Deep language prompting | 81.72 | 73.81 | 77.56
3: Independent V-L prompting | 82.15 | 74.07 | 77.90
4: MaPLe (Ours) | 82.28 | 75.14 | 78.55

###  4.4 Base-to-Novel Generalization

Generalization to Unseen Classes: Table 3 presents the performance of MaPLe in
base-to-novel generalization setting on 11 recognition datasets. We compare
its performance with CLIP zero-shot, and recent prompt learning works
including CoOp (Zhou et al., 2022b) and Co-CoOp (Zhou et al., 2022a). In case
of CLIP, we use hand-crafted prompts that are specifically designed for each
dataset.

In comparison with the state-of-the-art work Co-CoOp, our method shows
improved performance on both base and novel categories on all 11 datasets with
an exception of marginal reduction on only the base class performance of
Caltech101. With mutual synergy from the branch-aware multi-modal prompting,
MaPLe better generalizes to novel categories on all 11 datasets in comparison
with Co-CoOp, where we obtain an overall gain from 71.69% to 75.14%. When
taking into account both the base and novel classes, MaPLe shows an absolute
average gain of 2.72% over Co-CoOp.

In comparison with CLIP on novel classes, Co-CoOp improves only on 4/11
datasets dropping the average novel accuracy from 74.22% to 71.69%. MaPLe is a
strong competitor which improves accuracy over CLIP on novel classes on 6/11
datasets, with an average gain from 74.22% to 75.14%.

Generalization and Performance on Base Classes: Co-CoOp solves the poor
generalization problem in CoOp by explicitly conditioning prompts on image
instances and shows significant gains in novel categories. However on base
classes, it improves over CoOp only on 3/11 datasets with an average drop in
performance from 82.69% to 80.47%. Meanwhile, the completeness in prompting
helps MaPLe improve over CoOp on base classes in 6/11 datasets maintaining the
average base accuracy to around 82.28%, in addition to its improvement in
generalization to novel classes.

Table 2: Generalization comparison of MaPLe with CoOp†.

| Base | Novel | HM
---|---|---|---
CoOp | 82.69 | 63.22 | 71.66
Co-CoOp | 80.47 | 71.69 | 75.83
CoOp† | 80.85 | 70.02 | 75.04
MaPLe | 82.28 | 75.14 | 78.55

We find that the training strategies of Co-CoOp can be used to substantially
boost the generalization performance of vanilla CoOp (6.8% gain in novel
classes). We therefore compare our method with CoOp†, which trains CoOp in Co-
CoOp setting. In comparison with CoOp†, the vanilla CoOp model seems to
overfit on base classes. When compared to CoOp† which attains an average base
accuracy of 80.85%, MaPLe shows an improvement of 1.43% with the average base
accuracy of 82.28% (Table 2).

Table 3: Comparison with state-of-the-art methods on base-to-novel
generalization. MaPLe learns multi-modal prompts and demonstrates strong
generalization performance over existing methods on 11 different recognition
datasets. Absolute gains over Co-CoOp are indicated in blue.

(a) Average over 11 datasets | Base | Novel | HM
---|---|---|---
CLIP | 69.34 | 74.22 | 71.70
CoOp | 82.69 | 63.22 | 71.66
Co-CoOp | 80.47 | 71.69 | 75.83
MaPLe | 82.28 | 75.14 | 78.55
| +1.81 | +3.45 | +2.72

(b) ImageNet. | Base | Novel | HM
---|---|---|---
CLIP | 72.43 | 68.14 | 70.22
CoOp | 76.47 | 67.88 | 71.92
Co-CoOp | 75.98 | 70.43 | 73.10
MaPLe | 76.66 | 70.54 | 73.47
| +0.68 | +0.11 | +0.37

(c) Caltech101 | Base | Novel | HM
---|---|---|---
CLIP | 96.84 | 94.00 | 95.40
CoOp | 98.00 | 89.81 | 93.73
Co-CoOp | 97.96 | 93.81 | 95.84
MaPLe | 97.74 | 94.36 | 96.02
| -0.22 | +0.55 | +0.18

(d) OxfordPets | Base | Novel | HM
---|---|---|---
CLIP | 91.17 | 97.26 | 94.12
CoOp | 93.67 | 95.29 | 94.47
Co-CoOp | 95.20 | 97.69 | 96.43
MaPLe | 95.43 | 97.76 | 96.58
| +0.23 | +0.07 | +0.15

(e) StanfordCars | Base | Novel | HM
---|---|---|---
CLIP | 63.37 | 74.89 | 68.65
CoOp | 78.12 | 60.40 | 68.13
Co-CoOp | 70.49 | 73.59 | 72.01
MaPLe | 72.94 | 74.00 | 73.47
| +2.45 | +0.41 | +1.46

(f) Flowers102 | Base | Novel | HM
---|---|---|---
CLIP | 72.08 | 77.80 | 74.83
CoOp | 97.60 | 59.67 | 74.06
Co-CoOp | 94.87 | 71.75 | 81.71
MaPLe | 95.92 | 72.46 | 82.56
| +1.05 | +0.71 | +0.85

(g) Food101 | Base | Novel | HM
---|---|---|---
CLIP | 90.10 | 91.22 | 90.66
CoOp | 88.33 | 82.26 | 85.19
Co-CoOp | 90.70 | 91.29 | 90.99
MaPLe | 90.71 | 92.05 | 91.38
| +0.01 | +0.76 | +0.39

(h) FGVCAircraft | Base | Novel | HM
---|---|---|---
CLIP | 27.19 | 36.29 | 31.09
CoOp | 40.44 | 22.30 | 28.75
Co-CoOp | 33.41 | 23.71 | 27.74
MaPLe | 37.44 | 35.61 | 36.50
| +4.03 | +11.90 | +8.76

(i) SUN397 | Base | Novel | HM
---|---|---|---
CLIP | 69.36 | 75.35 | 72.23
CoOp | 80.60 | 65.89 | 72.51
Co-CoOp | 79.74 | 76.86 | 78.27
MaPLe | 80.82 | 78.70 | 79.75
| +1.08 | +1.84 | +1.48

(j) DTD | Base | Novel | HM
---|---|---|---
CLIP | 53.24 | 59.90 | 56.37
CoOp | 79.44 | 41.18 | 54.24
Co-CoOp | 77.01 | 56.00 | 64.85
MaPLe | 80.36 | 59.18 | 68.16
| +3.35 | +3.18 | +3.31

(k) EuroSAT | Base | Novel | HM
---|---|---|---
CLIP | 56.48 | 64.05 | 60.03
CoOp | 92.19 | 54.74 | 68.69
Co-CoOp | 87.49 | 60.04 | 71.21
MaPLe | 94.07 | 73.23 | 82.35
| +6.58 | +13.19 | +11.14

(l) UCF101 | Base | Novel | HM
---|---|---|---
CLIP | 70.53 | 77.50 | 73.85
CoOp | 84.69 | 56.05 | 67.46
Co-CoOp | 82.33 | 73.45 | 77.64
MaPLe | 83.00 | 78.66 | 80.77
| +0.67 | +5.21 | +3.13

###  4.5 Cross-Dataset Evaluation

We test the cross-dataset generalization ability of MaPLe by learning multi-
modal prompts on all the 1000 ImageNet classes and then transferring it
directly on the remaining 10 datasets. Table 4 shows the performance
comparison between MaPLe, CoOp and Co-CoOp. On the ImageNet source dataset,
MaPLe achieves performance comparable to competing approaches but demonstrates
a much stronger generalization performance by surpassing CoOp in 9/10 and Co-
CoOp in 8/10 datasets. Overall, MaPLe shows competitive performance leading to
the highest accuracy of 66.30% averaged over all datasets. This suggests that
the use of branch-aware V-L prompting in MaPLe facilitates better
generalization for cross-dataset transfer.

Table 4:  Comparison of MaPLe with existing approaches on cross-dataset
evaluation. Overall, MaPLe achieves competitive performance providing highest
average accuracy, indicating better generalization.

| Source | Target
---|---|---
|  ImageNet |  Caltech101 |  OxfordPets |  StanfordCars |  Flowers102 |  Food101 |  Aircraft |  SUN397 |  DTD |  EuroSAT |  UCF101 |  _Average_
CoOp | 71.51 | 93.70 | 89.14 | 64.51 | 68.71 | 85.30 | 18.47 | 64.15 | 41.92 | 46.39 | 66.55 | 63.88
Co-CoOp | 71.02 | 94.43 | 90.14 | 65.32 | 71.88 | 86.06 | 22.94 | 67.36 | 45.73 | 45.37 | 68.21 | 65.74
MaPLe | 70.72 | 93.53 | 90.49 | 65.57 | 72.23 | 86.20 | 24.74 | 67.01 | 46.49 | 48.06 | 68.69 | 66.30

###  4.6 Domain Generalization

We show that MaPLe generalizes favourably on out-of-distribution datasets as
compared to CoOp and Co-CoOp. We evaluate the direct transferability of
ImageNet trained model to various out-of-domain datasets, and observe that it
consistently improves against all the existing approaches as indicated in
Table 5. This indicates that utilizing multi-modal branch-aware prompting
helps MaPLe in enhancing the generalization and robustness of V-L models like
CLIP.

Table 5:  Comparison of MaPLe with existing approaches in domain
generalization setting. MaPLe shows highest performance on all target
datasets.

| Source | Target
---|---|---
| ImageNet | ImageNetV2 | ImageNet-Sketch | ImageNet-A | ImageNet-R
CLIP | 66.73 | 60.83 | 46.15 | 47.77 | 73.96
CoOp | 71.51 | 64.20 | 47.99 | 49.71 | 75.21
Co-CoOp | 71.02 | 64.07 | 48.75 | 50.63 | 76.18
MaPLe | 70.72 | 64.07 | 49.15 | 50.90 | 76.98

###  4.7 Ablation Experiments

Prompt Depth: In Fig. 4 (left), we illustrate the effect of prompt depth $J$
for MaPLe and ablate on the depth of language and vision branch
_individually_. In general, the performance improves as prompt depth
increases. As earlier methods (CoOp and Co-CoOp) utilize shallow language
prompting ($J=1$), we find it interesting to compare our method with deep
language prompting. Overall, MaPLe achieves better performance than deep
language prompting. We observe that MaPLe achieves maximum performance on
validation set at a depth of 9 .

Figure 4: Ablation on prompt depth _(left)_ and prompt length _(right)_ in
MaPLe. We report average results on the held-out validation sets of all
datasets.

Prompt Length: Fig. 4 (right) shows the effect of prompt length for MaPLe. As
the prompt length increases, the performance on base classes is generally
maintained, while the novel class accuracy decreases. This indicates over-
fitting which inherently hurts the generalization to novel classes.

Prompt Initialization: Table 6 shows the effect of prompt initialization on
MaPLe. Best performance is achieved when the learnable prompts in the first
layer are initialized with the prompt ‘a photo of a $<$category$>$’ and rest
of the layers are initialized randomly (row-3). Initializing prompts with a
similar template in all layers leads to lower performance suggesting that this
is redundant as these prompts learn hierarchically different contextual
concepts in different layers (row-1). However, complete random initialization
of prompts provides competitive performance (row-2).

Table 6:  Ablation on prompt initialization. In general, the performance of MaPLe is affected by the choice of prompt initialization. Best results are achieved when only first layer prompts are initialized with the prompt ‘a photo of a $<$category$>$’.  Method | Base Acc. | Novel Acc. | Harmonic Mean (HM)
---|---|---|---
1: MaPLe: All layers: ‘a photo of a’ | 81.90 | 74.22 | 77.88
2: MaPLe: Random initialization | 82.27 | 75.10 | 78.52
3: MaPLe: Only first layer: ‘a photo of a’ | 82.28 | 75.14 | 78.55
Figure 5: Percentage of classes where MaPLe $>$ Co-CoOp increases as diversity
of dataset increases (left to right).

Effectiveness of Multi-modal Prompting: Fig. 5 shows the analysis of per class
accuracy for selected datasets in the order of increasing diversity. It
indicates that the performance gains of MaPLe in comparison to Co-CoOp varies
across different datasets. MaPLe provides significant gains over Co-CoOp for
datasets that have large distribution shifts from the pretraining dataset of
CLIP, and vision concepts that are usually rare and less generic. Further
analysis is provided in Appendix B.

Comparing MaPLe with Heavier Co-CoOp The multi-modal deep prompting
architecture design of MaPLe along with its V-L coupling function
$\mathcal{F}$ constitutes more learnable parameters as compared to CoOP and
Co-CoOp. For a fair comparison, we retrain a heavier Co-CoOp that matches the
parameter count of MaPLe by stacking multiple additional layers in its Meta-
Net block. Table 7 indicates the effectiveness of multi-modal prompting in
MaPLe over the heavier Co-CoOp. This shows that the difference in the number
of parameters is not the cause of gain in our case and the proposed design
choices make a difference.

Table 7:  Comparison of MaPLe with a heavier Co-CoOp model. We retrain a heavier version of Co-CoOp which is comparable with MaPLe in terms of total parameter count.  Method | Base Acc. | Novel Acc. | Harmonic Mean (HM)
---|---|---|---
Co-CoOp | 80.47 | 71.69 | 75.83
Heavier Co-CoOp | 80.14 | 72.02 | 75.86
MaPLe | 82.28 | 75.14 | 78.55

Prompting Complexity Table 8 shows the computational complexity of MaPLe in
comparison with other approaches. Although MaPLe utilizes multi-modal prompts,
its overall FLOPS (Floating Point Operations) exceeds only by 0.1% over CoOp
and Co-CoOp. The independent V-L prompting also provides comparable FLOP
count. In terms of inference speed, Co-CoOp is significantly slower and the
FPS (Frames Per Second) remains constant as the batch size increases. In
contrast, MaPLe has no such overhead and provides much better inference and
training speeds. Further, MaPLe provides better convergence as it requires
only half training epochs as compared to Co-CoOp (5 vs 10 epochs).

Table 8: Comparison of computational complexity among different prompting methods.  Method | GFLOPS | % GFLOPS | FPS | BS Overhead
---|---|---|---|---
w.r.t Co-CoOp | BS=1 | BS=4 | BS=100
CoOp | 166.8 | 0.0 | 13.8 | 55.3 | 1353.0 | No
Co-CoOp | 166.8 | 0.0 | 13.9 | 14.9 | 15.1 | Yes
Independent V-L prompting | 167.1 | 0.2 | 14.1 | 56.7 | 1350.0 | No
MaPLe | 167.0 | 0.1 | 14.1 | 56.3 | 1365.0 | No

##  5 Conclusion

Adaptation of large-scale V-L models, _e.g._ , CLIP (Radford et al., 2021) to
downstream tasks is a challenging problem due to the large number of tunable
parameters and limited size of downstream datasets. Prompt learning is an
efficient and scalable technique to tailor V-L models to novel downstream
tasks. To this end, the current prompt learning approaches either consider
only the vision or language side prompting. Our work shows that it is critical
to perform prompting for both vision and language branches to appropriately
adapt V-L models to downstream tasks. Further, we propose a strategy to ensure
synergy between vision-language modalities by explicitly conditioning the
vision prompts on textual prompt across different transformer stages. Our
approach improves the generalization towards novel categories, cross-dataset
transfer and datasets with domain shifts.

## References

  * Bahng et al. (2022) Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola.  Visual prompting: Modifying pixel space to adapt pre-trained models.  _arXiv preprint arXiv:2203.17274_ , 2022.
  * Bossard et al. (2014) Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In _The European Conference on Computer Vision_ , pp. 446–461. Springer, 2014.
  * Cimpoi et al. (2014) Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3606–3613, 2014.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 248–255. Ieee, 2009.
  * Ding et al. (2022) Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai.  Decoupling zero-shot semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11583–11592, 2022.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Fei-Fei et al. (2004) Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In _2004 conference on computer vision and pattern recognition workshop_ , pp. 178–178. IEEE, 2004.
  * Feng et al. (2022) Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma.  Promptdet: Towards open-vocabulary detection using uncurated images.  In _The European Conference on Computer Vision_ , 2022.
  * Gao et al. (2021) Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  _arXiv preprint arXiv:2110.04544_ , 2021.
  * Gu et al. (2021) Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  _arXiv preprint arXiv:2104.13921_ , 2021.
  * Helber et al. (2019) Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 12(7):2217–2226, 2019.
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8340–8349, 2021a.
  * Hendrycks et al. (2021b) Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15262–15271, 2021b.
  * Huang et al. (2022) Tony Huang, Jack Chu, and Fangyun Wei.  Unsupervised prompt learning for vision-language models.  _arXiv preprint arXiv:2204.03649_ , 2022.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pp. 4904–4916. PMLR, 2021.
  * Jia et al. (2022) Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _The European Conference on Computer Vision_ , 2022.
  * Jin et al. (2021) Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren.  A good prompt is worth millions of parameters? low-resource prompt-based learning for vision-language models.  _arXiv preprint arXiv:2110.08484_ , 2021.
  * Ju et al. (2021) Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie.  Prompting visual-language models for efficient video understanding.  In _The European Conference on Computer Vision_ , 2021.
  * Kim et al. (2022) Konwoo Kim, Michael Laskin, Igor Mordatch, and Deepak Pathak.  How to adapt your large-scale vision-and-language model, 2022.  URL https://openreview.net/forum?id=EhwEUb2ynIa.
  * Krause et al. (2013) Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 554–561, 2013.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In _Conference on Empirical Methods in Natural Language Processing_ , 2021.
  * Li et al. (2022) Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl.  Language-driven semantic segmentation.  In _International Conference on Learning Representations_ , 2022.
  * Li & Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , 2021.
  * Liu et al. (2021) Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , 2021.
  * Lu et al. (2022) Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian.  Prompt distribution learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5206–5215, 2022.
  * Lüddecke & Ecker (2022) Timo Lüddecke and Alexander Ecker.  Image segmentation using text and image prompts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7086–7096, 2022.
  * Maaz et al. (2022) Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang.  Class-agnostic object detection with multi-modal transformer.  In _The European Conference on Computer Vision_. Springer, 2022.
  * Maji et al. (2013) Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Manli et al. (2022) Shu Manli, Nie Weili, Huang De-An, Yu Zhiding, Goldstein Tom, Anandkumar Anima, and Xiao Chaowei.  Test-time prompt tuning for zero-shot generalization in vision-language models.  2022\.
  * Nilsback & Zisserman (2008) Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In _2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing_, pp. 722–729. IEEE, 2008.
  * Parkhi et al. (2012) Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In _2012 IEEE conference on computer vision and pattern recognition_ , pp. 3498–3505. IEEE, 2012.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Rao et al. (2022) Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.  Denseclip: Language-guided dense prediction with context-aware prompting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18082–18091, 2022.
  * Rasheed et al. (2022) Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shahbaz Khan.  Bridging the gap between object and image-level representations for open-vocabulary detection.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International Conference on Machine Learning_ , pp. 5389–5400. PMLR, 2019.
  * Soomro et al. (2012) Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ , 2012.
  * Wang et al. (2019) Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  In _Advances in Neural Information Processing Systems_ , volume 32, 2019.
  * Wang et al. (2022a) Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al.  Dualprompt: Complementary prompting for rehearsal-free continual learning.  In _The European Conference on Computer Vision_ , 2022a.
  * Wang et al. (2022b) Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.  Learning to prompt for continual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 139–149, 2022b.
  * Xiao et al. (2010) Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In _2010 IEEE computer society conference on computer vision and pattern recognition_ , pp. 3485–3492. IEEE, 2010.
  * Yao et al. (2021a) Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  _arXiv preprint arXiv:2111.07783_ , 2021a.
  * Yao et al. (2021b) Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun.  Cpt: Colorful prompt tuning for pre-trained vision-language models.  _arXiv preprint arXiv:2109.11797_ , 2021b.
  * Yuan et al. (2021) Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  _arXiv preprint arXiv:2111.11432_ , 2021.
  * Zang et al. (2022) Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Open-vocabulary detr with conditional matching.  In _The European Conference on Computer Vision_ , 2022.
  * Zhai et al. (2022) Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18123–18133, 2022.
  * Zhang et al. (2022a) Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Tip-adapter: Training-free clip-adapter for better vision-language modeling.  In _The European Conference on Computer Vision_ , 2022a.
  * Zhang et al. (2022b) Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu.  Neural prompt search.  _arXiv preprint arXiv:2206.04673_ , 2022b.
  * Zhou et al. (2022a) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16816–16825, 2022a.
  * Zhou et al. (2022b) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , pp. 1–12, 2022b.
  * Zhou et al. (2022c) Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In _The European Conference on Computer Vision_ , 2022c.
  * Zhu et al. (2022) Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang.  Prompt-aligned gradient for prompt tuning.  _arXiv preprint arXiv:2205.14865_ , 2022.

Supplemental Material

This section contains supplementary material that provides additional details
for the main paper and further experimental analysis. This section follows the
contents in the following order.

  * •

Additional implementation details (Appendix A)

  * •

Analysis for understanding multi-modal prompts (Appendix B)

  * •

Analysis for alternate prompting design choices (Appendix C)

##  Appendix A Additional Implementation details

In this section, we provide further hyper-parameter details of the proposed
approaches presented in the main paper. Table 9 shows the hyper-parameters
chosen for vision, language and independent V-L prompting techniques. We use a
learning rate of 0.0025 for language and vision prompting, and 0.0035 for
independent V-L prompting.

Table 9:  Hyper-parameter settings for deep prompting variants.  Method | Prompt Depth ($K$) | Visual tokens $\tilde{P}$ | Textual tokens $P$
---|---|---|---
Language prompting | 12 | 0 | 4
Vision prompting | 12 | 4 | 0
Independent V-L prompting | 9 | 2 | 2

CoOp in CoCoOp setting: The CoOp approach trained in CoCoOp setting (denoted
by CoOp†) uses training configurations of CoCoOp and trains the standard CoOp
for 10 epochs instead of default 200 epochs. We use a batch size of 4 with a
learning rate of 0.0035.

##  Appendix B Analysis for Understanding Multi-modal Prompts

Our experimental results in Section 4.4 indicates that the performance gains
of MaPLe in comparison to Co-CoOp varies significantly across different
datasets. For some datasets, like ImageNet and Caltech101, the gains are less
than 1%, while on other datasets like EuroSAT, FGVCAircrafts and DTD, MaPLe
shows significant improvements over Co-CoOp. To better understand when MaPLe
is most effective, we dissect the individual dataset performances and perform
an exhaustive per-class analysis. Fig. 5 shows per class analysis for selected
datasets in the order of increasing diversity (distribution gap w.r.t CLIP
pretraining dataset, _i.e._ generic objects). The overall trend indicates that
MaPLe is more effective than Co-CoOp as the diversity of the dataset
increases. We conjecture that this is because fine-tuning or prompting bridges
the gap between the distribution of the downstream and the pretraining dataset
and thus improves the performance. However, the effectiveness would therefore
be less substantial for datasets with little distribution shifts. This
intriguing property is also validated for visual prompting in literature
(Bahng et al., 2022). MaPLe provides completeness in prompting by learning
both vision and language prompts to effectively steer CLIP, this makes it more
adaptive than Co-CoOp to improve on datasets with larger distribution shifts.

Additionally, we note that MaPLe benefits on categories which would have been
rarely seen by CLIP during its pretraining (400 million image caption dataset,
obtained from internet images). We observe that MaPLe provides significant
gains over Co-CoOp for vision concepts that tend to be rare and less generic,
_e.g._ , satellite images. In contrast, MaPLe performs competitively to Co-
CoOp on frequent and more generic categories _e.g._ , forest, river, dog,
_etc_. Multi-modal prompts allow MaPLe to better adapt CLIP for visual
concepts that are rarely occurring as compared to existing uni-modal prompting
techniques. In Table 10, we highlight category-wise comparison between MaPLe
and Co-CoOp for some selected datasets.

Table 10:  Analyzing the nature of categories where MaPLe performs better than Co-CoOp. Co-CoOp performs favourably well on generic categories, while MaPLe provides benefits on classes that are typically rare.  Dataset | MaPLe is better than Co-CoOp | Co-CoOp is better than MaPLe
---|---|---
Caltech | Crontosaurus, | Elephant,
(Generic Objects) | Gerenuk, Sea Horse | Ceiling Fan, Cellphone
Eurosat | Annual Crop Land, |
(Satellite Image) | Permanent Crop Land | -
UCF | Handstand Walking, | Walking With Dog,
(Action recognition) | Playing Daf | Horse Riding

##  Appendix C Analysis for Alternate Design Choices

Exploring other prompting designs: We provide analysis on other possible
multi-modal prompting design choices in comparison to MaPLe. As learnable
prompts in different transformer layers do not interact with each other, we
explore a progressive prompting approach where the prompts at each block are
conditioned on the prompts from the previous block. We apply this approach to
shallow versions (prompt depth $J=1$) of independent V-L prompting (row-1) and
MaPLe (row-2). To analyze whether independent V-L prompting and MaPLe provide
complementary gains, we explore a design choice combining them together
(row-3) in the same model. The results in Table 11 indicate that MaPLe
provides best performance as compared to other design choices.

Table 11:  Analysis on alternative design choices for V-L prompting. Overall, MaPLe proves to be the best variant among alternate prompting-related design choices.  Method | Base Acc. | Novel Acc. | Harmonic Mean (HM)
---|---|---|---
1: Independent V-L prompting + Progressive prompting | 81.20 | 74.92 | 77.93
2: MaPLe + Progressive prompting | 81.45 | 75.04 | 78.11
3: MaPLe + Independent V-L prompting | 82.27 | 74.05 | 77.94
4: MaPLe | 82.28 | 75.14 | 78.55
Table 12: Projecting from $P$ to $\tilde{P}$ provides the best results.

Prompt Proj. | Base | Novel | HM
---|---|---|---
$\tilde{P}$ $\rightarrow$ $P$ | 81.37 | 73.25 | 77.10
$P$ $\rightarrow$ $\tilde{P}$ | 82.28 | 75.14 | 78.55

Direction of prompt projection: As discussed in Section 3.2.3, MaPLe
explicitly conditions the vision prompts $\tilde{P}$ on the language prompts
$P$ ($P$ $\rightarrow$ $\tilde{P}$) using a V-L coupling function
$\mathcal{F}$. Here, we provide analysis for an alternative design choice
where $P$ is conditioned on $\tilde{P}$ ($\tilde{P}$ $\rightarrow$ $P$). Table
12 shows that our approach ($P$ $\rightarrow$ $\tilde{P}$) is a better choice
which can be reasoned by the lower information loss in such a design since
$d_{v}>d_{l}$.