$$0 \to \bbs^{\nu_{I \backslash j} (\alpha)}_{M,A,N} \to \bbs^{\nu_{I} (\alpha)}_{M,A,N} \to \bbs^{\nu_{I \backslash j} ( p_{< j} (\alpha) \odot p_{\geq j} (\alpha))}_{M , A ,N} \to 0.$$

Let us consider the filtrations of the outer two terms: by the inductive hypothesis, $\bbs^{\nu_{I \backslash j} (\alpha)}_{M,A,N}$ has a filtration with graded pieces of the form

$$\bbs^{\sigma_{K} (\alpha)}_A,$$

where $K \subset [\ell -1 ]$ ranges over all subsets with $K \subset I \backslash j$. Likewise, the term $\bbs^{\nu_{I \backslash j} ( p_{< j} (\alpha) \odot p_{\geq j} (\alpha))}_{M , A ,N}$ has a filtration with graded pieces of the form

$$\bbs^{\sigma_{L} ( p_{< j} (\alpha) \odot p_{\geq j} (\alpha))}_{M , A ,N},$$

where $L \subset [\ell - 2]$ ranges over all subsets $L \subset s_j (I \backslash j)$. Notice that $p_{< j} (\alpha) \odot p_{\geq j} (\alpha))$ is the same as $\sigma_j (\alpha)$, in which case ranging over all $K$ and $L$ as above is the same as just ranging over all subsets $J \subset [\ell -1]$ with $J \subset I$. The result hence follows from Corollary <ref>.

Let $A = S(V)$, the symmetric algebra on some free $R$-module $V$. Then $A$ is a $\gl (V)$-representation and Lemma <ref> implies that there is a $\gl (V)$-equivariant filtration of the tensor power $V^{\otimes d}$ with graded pieces of the form $\bbs^{\alpha}_A$, where $\alpha$ ranges over all compositions of $d$ (since $V^{\otimes d} = \bbs^{\nu_{[d-1]} (1^d)}_{S(V)}$).

Assuming $R$ is a field of characteristic $0$, the ring $R [ \gl (V)]$ is semisimple and hence this yields a $\gl(V)$-equivariant direct sum decompositions

$$V^{\otimes d} = \bigoplus_{|\alpha|=d} \bbs^{\alpha}_A.$$

Finally, we arrive at the main result of this section; this result furnishes the multi-Schur functors with a canonical filtration whose associated graded pieces are easily described as tensor products of the ribbon Schur functors of Section <ref>.

Let $\alpha^1 , \dots , \alpha^n$ be a sequence of $\ell$-partitioned compositions and $A^1 , \dots , A^n$ be a sequence of Koszul $R$-algebras and $N^i$ (resp. $M^i$) be Koszul left (resp. right) $A^i$-modules for $1 \leq i \leq n$. Then the multi-Schur module $\bbs^{\underline{\alpha}}_{\underline{M} , \underline{A} , \underline{N}}$ admits a canonical filtration with graded pieces of the form

$$\bbs^{\sigma_{I_1} (\alpha^1)}_{(M^1,A^1,N^1)} \otimes_R \bbs^{\sigma_{I_2} (\alpha^2)}_{(M^2, A^2, N^2)} \otimes_R \cdots \otimes_R \bbs^{\sigma_{I_n} (\alpha^n)}_{(M^n,A^n, N^n)},$$

where the subsets $I_1, \dots , I_n \subset [\ell -1 ]$ range over all choices such that $I_1 \cap I_2 \cap \cdots \cap I_n = \varnothing$.

Notice that the parameterizing set for the associated graded pieces is inherently symmetric under permuting the tensor factors, which is expected by the invariance of the multi-Schur module on the ordering of the inputs.

In general, the number of ways to choose $n$ subsets of $[\ell-1]$ with no common intersection is $(2^n-1)^{\ell-1}$, so the number of associated graded pieces grows exponentially with respect to both $n$ and $\ell$.

The adjective “canonical"  used in Theorem <ref> means that this filtration is really true at the level of functors; in other words, the functor $\bbs^{\underline{\alpha}}$ which takes as inputs $n$-tuples of Koszul $R$-algebras and outputs the associated multi-Schur module admits a canonical filtration whose associated graded pieces are given by the functors of the form

$$\bbs^{\sigma_{I_1} (\alpha^1)} \otimes_R \bbs^{\sigma_{I_2} (\alpha^2)} \otimes_R \cdots \otimes_R \bbs^{\sigma_{I_n} (\alpha^n)}.$$

Proceed by induction on $n$ (the numbers of Koszul algebras), with base case $n = 2$. To prove the base case, we induct on $\ell$ (where the case $\ell = 1$ is vacuous). Let $\ell > 1$ and write $\alpha = \alpha' \cdot p_\ell (\alpha)$ and $\beta = \beta' \cdot p_\ell (\beta)$. By Lemma <ref> there is a short exact sequence

$$0 \to \bbs^{\alpha , \beta'}_{A,B} \otimes_R \bbs^{p_\ell (\beta)}_B \to \bbs^{\alpha,\beta}_{A,B} \to \bbs^{\alpha,\beta}_{A,B} \to \bbs^{\alpha' \odot p_\ell (\alpha) , \beta}_{A,B} \to 0.$$

By the inductive hypothesis, the outer $2$ terms admit filtrations with graded pieces of the correct form. This establishes the base case.

Assume now that $n > 2$. By Lemma <ref> the multi-Schur module $\bbs^{\alpha^1 , \dots , \alpha^n}_{A^1 , \dots , A^n}$ admits a canonical filtration with associated graded pieces of the form

$$\bbs^{\sigma_I (\alpha^1) , \dots , \sigma_I (\alpha^{n-1})}_{A^1 , \dots , A^{n-1}} \otimes_R \bbs^{\nu_{[\ell] \backslash I} (\alpha^n)}_{A^n},$$

where $I$ ranges over all subsets of $[\ell-1]$. By the inductive hypothesis, each of the modules $\bbs^{\sigma_I (\alpha^1) , \dots , \sigma_I (\alpha^{n-1})}_{A^1 , \dots , A^{n-1}}$ admits a canonical filtration with associated graded pieces of the form

$$\bbs^{\sigma_{I_1}}_{A^1} \otimes_R \bbs^{\sigma_{I_2} (\alpha^2)}_{A^2} \otimes_R \cdots \otimes_R \bbs^{\sigma_{I_{n-1}} (\alpha^{n-1})}_{A^{n-1}},$$

where the above ranges over all choices of $I_1 , \dots , I_{n-1}$ with $I_1 \cap \cdots \cap I_{n-1} = I$. On the other hand, the module $\bbs^{\nu_{[\ell-1] \backslash I} (\alpha^n)}_{A^n}$ admits a canonical filtration with associated graded pieces of the form $\bbs^{\sigma_{I_n} (\alpha^n)}_{A^n}$, where $I_n$ ranges over all subsets $I_n \subset [\ell-1]$ such that $I_n \cap I = \varnothing$. This is overall the same thing as ranging over all choices $I_1 , \dots , I_n \subset [\ell -1]$ such that $I_1 \cap \cdots \cap I_n = \varnothing$.

Let $A$ and $B$ be Koszul algebras. Let us use the argument of Theorem <ref> to filter the multi-Schur module $\bbs^{(1^3),(1^3)}_{A,B}$, helping to illustrate the idea of the proof in a concrete setting. Recall by Example <ref> that we have the short exact sequence

$$\bbs^{(1^3),(1^2)}_{A , B} \otimes_R B_1 \to \bbs^{(1^3),(1^3)}_{A , B} \to \bbs^{(1,2),(1^3)}_{A, B} \to 0.$$

Iteratively applying this sequence to the outer two terms yields

$$0 \to A_3 \otimes_R (B_1)^{\otimes 3} \to \bbs^{(1^3),(1^2)}_{A , B} \to \bbs^{(2,1)}_A \otimes_R \bbs^{(1^2)}_B \otimes_R B_1 \to 0,$$

$$0 \to \bbs^{(1,2)}_A \otimes_R B_1 \otimes_R \bbs^{(1^2)}_A \to \bbs^{(1,2),(1^3)}_{A, B} \to A_3 \otimes_R \bbs^{(1^3)}_B \to 0.$$

The terms involving $B$ appearing on the ends of these short exact sequences are precisely the filtration factors of Lemma <ref>, and are filtered further by Lemma <ref>, yielding the filtration factors of Theorem <ref>.

Let $2^{[\ell -1]}$ denote the Boolean poset on $n$-elements. Then the product $(2^{[\ell-1]})^{\times n}$ is naturally a ranked poset (isomorphic in fact to the Boolean poset $2^{[\ell-1] \times [n]}$), and the set $\cat{S}$ of all tuples $(I_1 , \dots , I_n)$ with $I_1 \cap \cdots \cap I_n = \varnothing$ is a ranked subposet. Choosing any total order $<$ refining the partial order on this subposet, the filtration of the multi-Schur module $\bbs^{\alpha^1 , \dots , \alpha^n}_{A^1 , \dots , A^n}$ is parameterized by $<$. In other words, the filtration is of the form

$$\{ F_{(I_1 , \dots , I_n)} \}_{(I_1 , \dots , I_n) \in \cat{S}},$$

$$F_{(I_1 , \dots , I_n)}/F_{\text{pred} (I_1 , \dots , I_n)} \cong \bbs^{\sigma_{I_1} (\alpha^1)}_{A^1} \otimes_R \cdots \otimes_R \bbs^{\sigma_{I_n} (\alpha^n)}_{A^n}.$$

In the above, $\text{pred}$ denotes the predecessor function (ie, the largest element strictly smaller with respect to $<$).

Let $\alpha = (1,2) \cdot (2) \cdot (1)$ and $\beta = (3) \cdot (4,2) \cdot (2)$ be two $3$-partitioned compositions and let us compute the composition factors of the multi-Schur module $\bbs^{\alpha,\beta}_{A,B}$ for any two Koszul algebras $A$ and $B$. The subposet (in fact, meet semilattice) of $(2^{[2]})^{\times 2}$ that parametrizes the filtration factors has Hasse diagram

\[\begin{tikzcd}
{(\{1,2\} , \varnothing)} & {(\{1 \} , \{ 2 \})} & {(\{2 \} , \{ 1 \})} & {(\varnothing , \{1,2 \})} \\
{(\{ 1 \} , \varnothing)} & {( \{ 2 \} , \varnothing)} & {(\varnothing , \{ 1 \} )} & {(\varnothing , \{ 2 \} )} \\
& {(\varnothing,\varnothing)}
\arrow[no head, from=3-2, to=2-1]
\arrow[no head, from=3-2, to=2-2]
\arrow[no head, from=3-2, to=2-3]
\arrow[no head, from=3-2, to=2-4]
\arrow[no head, from=2-1, to=1-1]
\arrow[no head, from=2-1, to=1-2]
\arrow[no head, from=1-1, to=2-2]
\arrow[no head, from=2-2, to=1-3]
\arrow[no head, from=2-3, to=1-4]
\arrow[no head, from=2-4, to=1-4]
\arrow[no head, from=2-4, to=1-2]
\arrow[no head, from=2-3, to=1-3]
\end{tikzcd}\]

This translates to filtration factors of the following form:

\[\begin{tikzcd}
{\bbs^{(1,5)}_A \otimes_R \bbs^{(3,4,2,2)}_B} & {\bbs^{(1,4,1)}_A \otimes_R \bbs^{(3,4,4)}_B} & {\bbs^{(1,2,3)}_A \otimes_R \bbs^{(7,2,2)}_B} & {\bbs^{(1,2,2,1)}_A\otimes_R \bbs^{(7,4)}_B} \\
{\bbs^{(1,4,1)}_A \otimes_R \bbs^{(3,4,2,2)}_B} & {\bbs^{(1,2,3)}_A \otimes_R \bbs^{(3,4,2,2)}_B} & {\bbs^{(1,2,2,1)}_A \otimes_R \bbs^{(7,2,2)}_B} & {\bbs^{(1,2,2,1)}_A \otimes_R \bbs^{(3,4,4)}_B} \\
& {\bbs^{(1,2,2,1)}_A \otimes_R \bbs^{(3,4,2,2)}_B}
\arrow[no head, from=3-2, to=2-1]
\arrow[no head, from=3-2, to=2-2]
\arrow[no head, from=3-2, to=2-3]
\arrow[no head, from=3-2, to=2-4]
\arrow[no head, from=2-1, to=1-1]
\arrow[no head, from=2-1, to=1-2]
\arrow[no head, from=1-1, to=2-2]
\arrow[no head, from=2-2, to=1-3]
\arrow[no head, from=2-3, to=1-4]
\arrow[no head, from=2-4, to=1-4]
\arrow[no head, from=2-4, to=1-2]
\arrow[no head, from=2-3, to=1-3]
\end{tikzcd}\]

If for instance $A = B = S(V)$, the symmetric algebra, then the above filtration is also $\gl (V) \times \gl (V)$ equivariant.

§ APPLICATIONS TO EQUIVARIANT STRUCTURE OF DERIVED INVARIANTS

The following section is the reward for enduring the technical details of Sections <ref> and <ref>; we are able to arrive at the other end with a rather robust theory that allows us to give elegant and simple closed form descriptions of higher derived invariants associated to (Veronese/Segre subalgebras of) Koszul algebras.

§.§ Tor and Ext

Let $A$ be a Koszul algebra and $M$ (resp. $N$) a Koszul right (resp. left) $A$-module of initial degree $t$ (resp. $s$). Then there is a canonical isomorphism of $A$-modules

$$\tor_i^A (M,N) \cong \bbs^{(1^{i})}_{M,A,N} \quad \text{for all} \ i > 0$$

If $M$ is instead a Koszul left $A$-module, then there is a canonical isomorphism

$$\ext^i_A (M,N) \cong \bbs^{(i)}_{M^!,A^!,(N^*)^!} \quad \text{for all} \ i >0.$$

In particular, both of the above modules are flat $R$-modules annihilated by $A_+$.

By the definition of Tor combined with Theorem <ref>, the module $\tor_i (M,N)$ may be computed by looking at the homology of the complex

$$\cdots \to M \otimes_R \bbs^{(1^{i+1})}_{A,N} \to M \otimes_R \bbs^{(1^i)}_{A,N} \to M \otimes_R \bbs^{(1^{i-1})}_{A,N} \to \cdots.$$

Splitting this complex into graded pieces, there is a commutative diagram:

\[\begin{tikzcd}
{M_j \otimes_R \bbs^{(1^i)}_{A,N}} && {M_{j+1} \otimes_R \bbs^{(1^{i-1})}_{A,N}} \\
& {M_j \otimes_R A_1 \otimes_R \bbs^{(1^{i-1})}_{A.N}} \\
{\bbs^{(1)}_{M_{\geq j},A} \otimes_R \bbs^{(1^{i-1})}_{A,N}}
\arrow["{(d_i)_{j+i+s}}", from=1-1, to=1-3]
\arrow[hook, from=1-1, to=2-2]
\arrow[two heads, from=2-2, to=1-3]
\arrow[hook, from=3-1, to=2-2]
\end{tikzcd}\]

This implies that there is an equality

$$\ker (d_i)_{i+j+s} = \left( M_j \otimes_R \bbs^{(1^i)}_{A,N} \right) \cap \left( \bbs^{(1)}_{M_{\geq j} , A} \otimes_R \bbs^{(1^{i-1})}_{A,N} \right).$$

By Observation <ref>, this latter intersection is precisely $\bbs^{(1^i)}_{(M_{\geq j} , A , N)}$. On the other hand, by definition of the Priddy differential there is also a commutative diagram for all $j > s$:

\[\begin{tikzcd}
{M_{j-1} \otimes_R \bbs^{(1^{i+1})}_{A,N}} & {M_j \otimes_R \bbs^{(1^i)}_{A,N}} \\
{\bbs^{(1^i)}_{M_{\geq j},A,N}} & {M_j \otimes_R A_1 \otimes_R \bbs^{(1^{i-1})}_{A,N}}
\arrow["{(d_{i+1})_{j+i+s}}", from=1-1, to=1-2]
\arrow[hook, from=1-2, to=2-2]
\arrow[two heads, from=1-1, to=2-1]
\arrow[hook, from=2-1, to=2-2]
\end{tikzcd}\]

This implies that there is also an equality

$$\im (d_{i+1})_{j+i+s} = \bbs^{(1^i)}_{M_{\geq j} , A , N}.$$

Putting both of the above equalities together, it follows that

$$\tor_i^A (M , N)_{i+j+s} = \begin{cases}
\bbs^{(1^i)}_{M , A , N} & \text{if} \ j = s, \\
0 & \text{otherwise}.
\end{cases}$$

To prove the isomorphism for Ext, recall first that there is a canonical isomorphism

$$\ext^i_A (M , N) \cong \left( \tor_i^A (N^* , M) \right)^*.$$

By Observation <ref>, the graded dual $N^*$ is a Koszul right $A$-module, and by the isomorphism just proved for Tor there is an isomorphism

$$\tor_i^A (N^* , M) \cong \bbs^{(1^i)}_{N^*,A,M}.$$

Dualizing and using the isomorphism of Corollary <ref>(3), the result follows immediately.

Assume that $A$ is any Koszul algebra and recall that

$$\bbs_{A,A_+^d}^{(1^i)} = \bbs^{(1^i,d)}_A.$$

By Theorem <ref>, the minimal free resolution of $A_+^d$ thus has the form

$$\cdots \to A \otimes_R \bbs^{(1^i,d)}_A \to A \otimes_R \bbs^{(1^{i-1},d)}_A \to \cdots \to A \otimes_R A_d \to A_+^d.$$

The ribbon Schur module $\bbs^{(1^i,d)}_{A}$ may be presented as the cokernel of the composition

$$(A^!)^*_{i+2} \otimes_R A_{d-2} \to (A^!)^*_{i+1} \otimes_R A_1 \otimes_R A_{d-2} \to (A^!)^*_{i+1} \otimes_R A_{d-1},$$

in which case we see that there is an isomorphism with the ribbon Schur functor:

$$\bbs^{(1^i,d)}_{A} \cong L^A_{i-1,d-1},$$

where the module $L^A_{i+1,d-1}$ is defined as in [6].

Let $A$ be any Koszul algebra such that $\bbs^{(1^i)}_{A} = 0$ for all $i > 1$. Then every Koszul module over $A$ is a flat $R$-module. If $A$ is $R$-projective, then every Koszul module over $A$ is $R$-projective.

§.§ Veronese Subalgebras

Recall the definition of the Veronese subalgebra as in Definition <ref>. The following Observation that the operation $(-)^{(d)}$ on compositions as defined in Definition <ref> interacts in a functorial way with formation of the ribbon Schur functor:

Let $A$ be any Koszul algebra and $M$ (resp. $N$) any Koszul right (resp. left) $A$-module. For any integer $d > 0$ and integers $i,j \in \bbz$ there is are isomorphisms

$$\bbs^{\alpha}_{\veralg{d}} = \bbs^{\alpha^{(d)}}_{A}, \quad \text{and} \quad \bbs^{ \alpha }_{\veralgg{M}{d} , \veralg{d}, \veralgg{N}{d}} \cong \bbs^{ \alpha^{(d)}}_{M,A,N}.$$

Combining Theorem <ref> with Observation <ref> immediately yields:

Let $A$ be any Koszul algebra and $M$ (resp. $N$) any Koszul right (resp. left) $A$-module. For any integer $d > 0$ there is an isomorphism

$$\tor_i^{\veralg{d}} ( \veralgg{M}{d} , \veralgg{N}{d}) = \bbs^{(d^i)}_{M,A,N},$$

where $t$ (resp. $s$) is the initial degree of $M$ (resp. $N$). In particular, there are canonical isomorphisms

$$\tor_i^{\veralg{d}} (\vermod{d}{r} , \vermod{d}{r'} ) \cong \bbs^{(r,d^i,r')}_{A}.$$

The isomorphism

$$\tor_i^{\veralg{d}} (\vermod{d}{r} , \vermod{d}{r'} ) \cong \bbs^{(r,d^i,r')}_{A}.$$

was originally proved in the case $A = S(V)$ (the symmetric algebra) in the work [2]. However, one notices that the original proof of this isomorphism does not invoke anything more than Koszulness properties of the symmetric algebra (and its truncations), which leads to the generalization presented in Corollary <ref>.

§.§ Segre Subalgebras

In this subsection we apply the construction of multi-Schur modules and their filtrations to study Segre products of Koszul algebras. The following observation is a straightforward translation of the quadratic dual for a Koszul $R$-algebra:

Let $A^1, \dots , A^n$ be a collection of Koszul algebras and $M^i$ a Koszul left $A^i$-module for $1 \leq i \leq n$. Then there are isomorphisms of $R$-modules

$$\left( (A^1 \circ \cdots \circ A^n)^! \right)^*_i = \bbs^{(1^i)}_{A^1 , \dots , A^n}, \quad \text{and}$$

$$\left( (M^1 \circ \cdots \circ M^n)^! \right)^*_i = \bbs^{(1^i)}_{(A^1,M^1) , \dots , (A^n , M^n)}.$$

Let $A$ be any Koszul algebra and consider the filtration of Theorem <ref> applied to computing $(A^{[2] \ !})^*_3 = \bbs^{(1^3)}_{A,A}$. The poset parametrizing the filtration terms are the same as those of Example <ref>, which yields the filtration factors

\[\begin{tikzcd}
{\bbs^{(3)}_A \otimes_R \bbs^{(1^3)}_A} & {\bbs^{(2,1)}_A \otimes_R \bbs^{(1,2)}_A} & {\bbs^{(1,2)}_A \otimes_R \bbs^{(2,1)}_A} & {\bbs^{(1^3)}_A \otimes_R \bbs^{(3)}_A} \\
{\bbs^{(2,1)}_A \otimes_R \bbs^{(1^3)}_A} & {\bbs^{(1,2)}_A \otimes_R \bbs^{(1^3)}_A} & {\bbs^{(1^3)}_A \otimes_R \bbs^{(2,1)}_A} & {\bbs^{(1^3)}_A \otimes_R \bbs^{(1,2)}_A} \\
& {\bbs^{(1^3)}_A \otimes_R \bbs^{(1^3)}_A}
\arrow[no head, from=3-2, to=2-1]
\arrow[no head, from=3-2, to=2-2]
\arrow[no head, from=3-2, to=2-3]
\arrow[no head, from=3-2, to=2-4]
\arrow[no head, from=2-1, to=1-1]
\arrow[no head, from=2-1, to=1-2]
\arrow[no head, from=1-1, to=2-2]
\arrow[no head, from=2-2, to=1-3]
\arrow[no head, from=2-3, to=1-4]
\arrow[no head, from=2-4, to=1-4]
\arrow[no head, from=2-4, to=1-2]
\arrow[no head, from=2-3, to=1-3]
\end{tikzcd}\]

Suppose now that $A = S(V)$, the symmetric algebra on a vector space $V$. If $R$ is a field of characteristic $0$, this induces a $\gl (V) \times \gl (V)$-equivariant direct sum decomposition of $(A^{[2] \ !})^*_3$ into irreducibles:

$$(A^{[2] \ !})^*_3 = \left( \bigwedge^3 V \otimes \bigwedge^3 V \right) \oplus \left( \bigwedge^3 V \otimes_R \bbs^{(2,1)} (V) \right)^{\oplus 4}$$

$$\oplus \left( \bigwedge^3 V \otimes S_3 (V) \right)^{\oplus 2} \oplus \left( \bbs^{(2,1)} (V) \otimes \bbs^{(2,1)} (V) \right)^{\oplus 2}.$$

Let $\alpha \in C(d)$ be any composition of some integer $d>0$ and $A^1 , \dots , A^n$ a sequence of Koszul $R$-algebras admitting a compatible $G^i$-action for each $1 \leq i \leq n$. Assume that $M^i$ (resp. $N^i$) is a Koszul right (resp. left) $A^i$-module admitting a compatible $G^i$-action. Then the multi-Schur module

$$\bbs^{\alpha}_{(M^1 , A^1 , N^1), \dots , (M^n,A^n,N^n)}$$

admits a $G^1 \times \cdots \times G^n$-equivariant filtration with associated graded pieces of the form

$$\bbs^{\alpha^1}_{(M^1,A^1,N^1)} \otimes_R \cdots \otimes_R \bbs^{\alpha^n}_{(M^n,A^n,N^n)},$$

where $\alpha^1 , \dots , \alpha^n$ range over all compositions of $|\alpha|$ satisfying $\alpha^1 \w \cdots \w \alpha^n = \alpha$.

In particular, if each of the group rings $R[G^i]$ is semisimple, then there is a $G^1 \times \cdots \times G^n$-equivariant decomposition

$$\bbs^{\alpha}_{(M^1 , A^1 , N^1), \dots , (M^n,A^n,N^n)} \cong \bigoplus_{\substack{(\alpha^1 , \dots , \alpha^n) \in C(d)^{\times n} \\ \alpha^1 \w \cdots \w \alpha^n = \alpha}} \bbs^{\alpha^1}_{(M^1,A^1,N^1)} \otimes_R \cdots \otimes_R \bbs^{\alpha^n}_{(M^n,A^n,N^n)}.$$

This is just a retranslation of Theorem <ref> combined with the fact that the refinement poset on $\alpha$ is isomorphic to the Boolean poset on $[\ell(\alpha)-1]$, and the meet operation corresponds to intersection under this isomorphism.

The appearance of compositions ranging over the refinement poset as filtration factors is likely related to the connection between Segre products and internal cohomomorphisms as discovered by Manin (see [9] and [10]). Indeed, another perspective on the filtration <ref> is as a canonical filtration of the graded pieces of the cohomomorphism algebra.

Next, we use Theorem <ref> to prove a symmetric function identity; we first need to recall some notation related to Schur polynomials and establish some multi-index conventions:

Let $n \geq 1$ be any integer and consider sets of indeterminates $\xx^1, \dots , \xx^n$. Recall that the Schur polynomial associated to a skew shape $\lambda / \mu$ is the polynomial

$$s_{\lambda / \mu} (\xx) := \sum_{T \in \operatorname{SST} (\lambda / \mu)} \xx_T,$$

where $\xx_T$ denotes the multigraded character of the semistandard tableau $T$. If $\alpha$ is a composition, the notation $s_\alpha (\xx)$ denotes the Schur polynomial corresponding to the ribbon shape determined by $\alpha$. The complete symmetric polynomial $h_d (\xx)$ is defined to be $s_{(d)} (\xx)$.

Given a tuple of compositions $\underline{\alpha} = (\alpha^1 , \dots , \alpha^n)$, use the notation

$$s_{\underline{\alpha}} (\underline{\xx}) := s_{\alpha^1} (\xx^1) \cdot s_{\alpha^2} (\xx^2) \cdots s_{\alpha^n} (\xx^n).$$

In particular, for a single composition $\alpha = (\alpha_1 , \dots , \alpha_n)$ there is the equality

$$h_\alpha (\underline{\xx}) = h_{\alpha_1} (\xx^1) \cdot h_{\alpha_2} (\xx^2) \cdots h_{\alpha_n} (\xx^n).$$

With notation as in Notation <ref>, there is an equality of symmetric polynomials

$$h_{d^n + \alpha} (\underline{\xx}) = \sum_{i=1}^d  \sum_{\substack{\beta^1 , \dots , \beta^n \\ |\beta^1| = \cdots = |\beta^n| = i \\ \beta^1 \w \cdots \w \beta^n = 1^i}} (-1)^{i+1} h_{(d-i)^n} (\underline{\xx} ) \cdot s_{\underline{\beta} \cdot \alpha} ( \underline{\xx}).$$

In the statement of Corollary <ref>, if $\underline{\beta} = (\beta^1 , \dots , \beta^n)$ is a tuple of compositions and $\alpha = (\alpha_1 , \dots , \alpha_n)$ is a composition, then we are using the convention

$$\underline{\beta} \cdot \alpha := \left( \beta^1 \cdot (\alpha_1) , \beta^2 \cdot (\alpha_2) , \dots , \beta^n \cdot (\alpha_n)\right).$$

Assume $R = k$ is a field of characteristic $0$ and let $A = S(V)$ for some vector space $V$. Note that $A$ and $A_{\geq d} = A_+^d$ are polynomial functors for all $d \geq 1$. Taking Segre products, this means that $A^{[n]}$ and $A_+^\alpha := A_+^{\alpha_1} \circ A_+^{\alpha_2} \circ \cdots \circ A_+^{\alpha_n}$ are also polynomial functors and there is hence an equality

\begin{equation}\label{eqn:charIdentity}
\ch (A_+^\alpha) = \ch (A^{[n]}) \cdot \sum_{i \geq 0} (-1)^i \ch \left( \tor_i^{A^{[n]}} (A_+^\alpha , k) \right),\end{equation}

where $\ch (-)$ denotes the multigraded character. By definition there are equalities

$$\ch (A_+^\alpha) = \sum_{d \geq 1} h_{d^n + \alpha} (\underline{\xx}),$$

$$\ch (A^{[n]} ) = \sum_{d \geq 0} h_{d^n} (\underline{\xx}).$$

On the other hand, by Theorem <ref> there is an equality

$$\ch \left( \tor_i^{A^{[n]}} (A_+^\alpha , k) \right) = \sum_{\substack{\beta^1 , \dots , \beta^n \\ |\beta^1| = \cdots = |\beta^n| = i \\ \beta^1 \w \cdots \w \beta^n = 1^i}} s_{\underline{\beta} \cdot \alpha} (\underline{\xx}).$$

Combining all of these expressions and comparing degrees on each side of the equality (<ref>) yields the result.

When $n = 1$, this reduces to the well-known classical character identity

$$h_{d+a} (\xx) = \sum_{i=1}^d (-1)^{i+1} h_{d-i} (\xx) s_{(1^i,a)} (\xx).$$

Moreover, taking $s$th Veronese powers of the polynomial ring and performing an identical argument yields the same identity, but every composition is replaced with its $s$th rescaling (ie, apply the operation $(-)^{(s)}$ to all compositions).

Let $n=2$, $d=2$, and $\alpha = (\alpha_1 ,
\alpha_2)$ be any composition. Then the identity of Corollary <ref> reads

$$h_{2+\alpha_1} (\xx^1) h_{2+\alpha_2} (\xx^2) = h_{d-1} (\xx^1) h_{d-1} (\xx^2) s_{(1 , \alpha_1)} (\xx^1) s_{(1,\alpha_2)} (\xx^2)$$

$$- s_{(1^2,\alpha_1)} (\xx^1) s_{(1^2 , \alpha_2)} - s_{(1^2,\alpha_1)} (\xx^1) s_{(2,\alpha_2)} (\xx^2) - s_{(2,\alpha_1)} (\xx^1) s_{(1^2,\alpha_2)} (\xx^2).$$

Our next corollary is an evident consequence of Theorem <ref> combined with Observation <ref> (by setting $\alpha = (1^i)$).

Let $A^1 , \dots , A^n$ a sequence of Koszul $R$-algebras admitting a compatible $G^i$-action for each $1 \leq i \leq n$. Assume that $M^i$ (resp. $N^i$) is a Koszul right (resp. left) $A^i$-module admitting a compatible $G^i$-action. Then the module

$$\tor_i^{A^1 \circ \cdots \circ A^n} (M^1 \circ \cdots \circ M^n , N^1 \circ \cdot \circ N^n)$$

admits a $G^1 \times \cdots \times G^n$-equivariant filtration with associated graded pieces of the form

$$\bbs^{\alpha^1}_{M^1,A^1,N^1} \otimes_R \cdots \otimes_R \bbs^{\alpha^n}_{M^n,A^n,N^n},$$

where the compositions range over all tuples $(\alpha^1 , \dots , \alpha^n) \in C(i)^{\times n}$ with $\alpha^1 \w \cdots \w \alpha^n = (1^{i})$.

Likewise, the module

$$\tor_i^{(A^1 \circ \cdots \circ A^n)^{(d)}} ((M^1 \circ \cdots \circ M^n)^{(d)}, (N^1 \circ \cdot \circ N^n)^{(d)})$$

admits a $G^1 \times \cdots \times G^n$-equivariant filtration with associated graded pieces of the form

$$\bbs^{\alpha^1}_{M^1,A^1,N^1} \otimes_R \cdots \otimes_R \bbs^{\alpha^n}_{M^n,A^n,N^n},$$

where the compositions range over all tuples $(\alpha^1 , \dots , \alpha^n) \in C(di)^{\times n}$ with $\alpha^1 \w \cdots \w \alpha^n = (d^{i})$.

Let $A$ be any Koszul algebra and let us compute the filtration factors of $(A^{[3]!})^*_2 = \bbs^{(1^2)}_{A^{[3]}}$ (unfortunately, trying this for $\bbs^{(1^3)}_{A^{[3]}}$ yields $49$ filtration factors, which is too big of an example). The poset parametrizing the filtration factors is:

\[\begin{tikzcd}
{(\{1\} , \{ 1 \} , \varnothing)} & {(\{1 \} , \varnothing , \{ 1 \})} & {(\varnothing, \{1 \}  , \{ 1 \} )} \\
{(\{1\} , \varnothing , \varnothing)} & {(\varnothing, \{ 1 \} , \varnothing)} & {(\varnothing , \varnothing , \{ 1 \} )} \\
& {(\varnothing,\varnothing,\varnothing)}
\arrow[no head, from=3-2, to=2-1]
\arrow[no head, from=3-2, to=2-2]
\arrow[no head, from=3-2, to=2-3]
\arrow[no head, from=2-3, to=1-3]
\arrow[no head, from=2-3, to=1-2]
\arrow[no head, from=2-2, to=1-3]
\arrow[no head, from=2-2, to=1-1]
\arrow[no head, from=2-1, to=1-2]
\arrow[no head, from=2-1, to=1-1]
\end{tikzcd}\]

This yields filtration factors:

\[\begin{tikzcd}
{\bbs^{(2)}_A \otimes_R \bbs^{(2)}_A \otimes_R \bbs^{(1^2)}_A} & {\bbs^{(2)}_A \otimes_R \bbs^{(1^2)}_A \otimes_R \bbs^{(2)}_A} & {\bbs^{(1^2)}_A \otimes_R \bbs^{(2)}_A \otimes_R \bbs^{(2)}_A} \\
{\bbs^{(2)}_A \otimes_R \bbs^{(1^2)}_A \otimes_R \bbs^{(1^2)}_A} & {\bbs^{(1^2)}_A \otimes_R \bbs^{(2)}_A \otimes_R \bbs^{(1^2)}_A} & {\bbs^{(1^2)}_A \otimes_R \bbs^{(1^2)}_A \otimes_R \bbs^{(2)}_A} \\
& {\bbs^{(1^2)}_A \otimes_R \bbs^{(1^2)}_A \otimes_R \bbs^{(1^2)}_A}
\arrow[no head, from=3-2, to=2-1]
\arrow[no head, from=3-2, to=2-2]
\arrow[no head, from=3-2, to=2-3]
\arrow[no head, from=2-3, to=1-3]
\arrow[no head, from=2-3, to=1-2]
\arrow[no head, from=2-2, to=1-3]
\arrow[no head, from=2-2, to=1-1]
\arrow[no head, from=2-1, to=1-2]
\arrow[no head, from=2-1, to=1-1]
\end{tikzcd}\]

Let $A$ be any Koszul algebra and let $T(V)$ denote the tensor algebra on some projective $R$-module $V$. Then by definition the $R$-module $\bbs^{(1^i)}_{A,T(V)}$ admits a filtration with graded pieces of the form

$$\bbs^{\alpha^1}_{A} \otimes_R \bbs^{\alpha^2}_{T(V)},$$

with $\alpha^1 \w \alpha^2 = (1^i)$. Notice that the only choice of $\alpha^2$ for which $\bbs^{\alpha^2}_{T(V)}$ is nonzero is for $\alpha^2 = (i)$. Thus $\alpha^1 = (1^i)$ and we find

$$\bbs^{(1^i)}_{A,T(V)} = \bbs^{(1^i)}_A \otimes_R \bbs^{(i)}_{T(V)} = (A^!)^*_i \otimes_R V^{\otimes i}.$$

In particular, after dualizing and collecting graded pieces, there is an isomorphism of $R$-algebras

$$(A \circ T(V))^! = A^! \circ T(V^*).$$

Of course, this could be verified using more direct methods, but the point is to demonstrate the utility of Theorem <ref>.

Let $A = S(V)$, $B = S(W)$ be symmetric algebras on free $R$-modules $V$ and $W$ both of rank $2$. Let $M = A_+^r \circ B$ and $N = A \circ B_+^{r'}$, viewed as modules over the Segre product $A \circ B$. By Corollary <ref> there is an equality

$$\tor^{A \circ B}_i (M , N) = \bbs^{(1^i)}_{(A,A_+^r) \circ (B,B_+^{r'})}.$$

Of course, the module $\bbs^{(1^i)}_{(A,A_+^r) \circ (B,B_+^{r'})}$ has a $\gl (V) \times \gl (W)$-equivariant filtration with associated graded pieces of the form

$$\bbs^{\alpha}_{A,A_+^r} \otimes_R \bbs^{\beta}_{B,B_+^{r'}},$$

where $\alpha$ and $\beta$ range over all partitions with $\alpha \w \beta = (1^i)$. Notice that since $V$ and $W$ have rank $2$, we are really only ranging over all partitions $\alpha$ and $\beta$ such that all parts of $\alpha^t$ and $\beta^t$ are at most $2$, and such that $\alpha^t \vee \beta^t = (i)$. Retranslating this in terms of subsets of the Boolean poset, this is asking for all ways to partition the set $[i-1]$ into the union of two totally disconnected sets $I_\alpha \cup I_\beta$, where $1 \in I_{\beta}$ and $i-1 \in I_\alpha$. One quickly sees that there is no such decomposition if $i-1$ is odd and only $1$ such decomposition when $i-1$ is even:

$$[i-1] = \{2, 4 , \dots , i-1 \} \cup \{ 1 , \dots , i-2 \}.$$

Retranslating this in terms of compositions, we find there there is a $\gl (V) \times \gl (W)$-equivariant isomorphism:

$$\tor^{A \circ B}_i (M , N) = \begin{cases}
S_{r-1} (V) \otimes_R \det(V)^{\frac{i+1}{2}} \otimes_R S_{r'-1} (W) \otimes_R \det(W)^{\frac{i+1}{2}} & \text{if} \ i > 0 \ \text{is odd}, \\
0 & \text{if} \ i > 0 \ \text{is even.}
\end{cases}$$

[1]
Kaan Akin, David A Buchsbaum, and Jerzy Weyman, Schur functors and schur
complexes, Advances in Mathematics 44 (1982), no. 3, 207–278.

[2]
Ayah Almousa, Michael Perlman, Alexandra Pevzner, Victor Reiner, and Keller
VandeBogert, Equivariant resolutions over veronese rings, arXiv
preprint arXiv:2210.16342 (2022).

[3]
Jörgen Backelin, A distributiveness property of augmented algebras and
some related homological results, Ph.D. thesis, Ph.d Thesis, Stockholm,

[4]
David A Buchsbaum and David Eisenbud, Generic free resolutions and a
family of generically perfect ideals, Advances in Mathematics 18
(1975), no. 3, 245–301.

[5]
Alexander Beilinson, Victor Ginzburg, and Wolfgang Soergel, Koszul
duality patterns in representation theory, Journal of the American
Mathematical Society 9 (1996), no. 2, 473–527.

[6]
Eleonore Faber, Martina Juhnke-Kubitzke, Haydee Lindo, Claudia Miller, Rebecca
R.G, and Alexandra Seceleanu, Canonical resolutions over koszul
algebras, arXiv e-prints (2020), arXiv–2011.

[7]
Mark Goresky, Robert Kottwitz, and Robert MacPherson, Equivariant
cohomology, koszul duality, and the localization theorem, Inventiones
mathematicae 131 (1997), no. 1, 25–83.

[8]
George R Kempf, On the collapsing of homogeneous bundles., Inventiones
mathematicae 37 (1976), 229–240.

[9]
Yu I Manin, Some remarks on koszul algebras and quantum groups, Annales
de l'institut Fourier, vol. 37, 1987, pp. 191–205.

[10]
Yuri Manin, Topics in non-commutative geometry, vol. 9, Princeton
University Press, 2014.

[11]
Matthew Mastroeni and Jason McCullough, Chow rings of matroids are
koszul, Mathematische Annalen (2022), 1–33.

[12]
Leonid Positselski, Galois cohomology of a number field is koszul,
Journal of Number Theory 145 (2014), 126–152.

[13]
Alexander Polishchuk and Leonid Positselski, Quadratic algebras,
vol. 37, American Mathematical Soc., 2005.

[14]
Stewart B Priddy, Koszul resolutions, Transactions of the American
Mathematical Society 152 (1970), no. 1, 39–60.

[15]
SA Yuzvinsky, Orlik-solomon algebras in algebra and topology, Russian
Mathematical Surveys 56 (2001), no. 2, 293.


# PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-
training via Prompting

Zixin Guo  Aalto UniversityEspooFinland<EMAIL_ADDRESS>,  Tzu-Jui Julius
Wang  Aalto UniversityEspooFinland<EMAIL_ADDRESS>,  Selen Pehlivan
Aalto UniversityEspooFinland<EMAIL_ADDRESS>,  Abduljalil Radman
Aalto UniversityEspooFinland<EMAIL_ADDRESS>and  Jorma Laaksonen
Aalto UniversityEspooFinland<EMAIL_ADDRESS>
(2023)

###### Abstract.

Vision-language (VL) Pre-training (VLP) has shown to well generalize VL models
over a wide range of VL downstream tasks, especially for cross-modal
retrieval. However, it hinges on a huge amount of image-text pairs, which
requires tedious and costly curation. On the contrary, weakly-supervised VLP
(W-VLP) (Wang et al., 2023) explores means with object tags generated by a
pre-trained object detector (OD) from images. Yet, they still require paired
information, i.e. images and object-level annotations, as supervision to train
an OD.

To further reduce the amount of supervision, we propose Prompts-in-The-Loop
(PiTL) that prompts knowledge from large language models (LLMs) to describe
images. Concretely, given a category label of an image, e.g. refinery, the
knowledge, e.g. a refinery could be seen with large storage tanks, pipework,
and …, extracted by LLMs is used as the language counterpart. The knowledge
supplements, e.g. the common relations among entities most likely appearing in
a scene. We create IN14K, a new VL dataset of 9M images and 1M descriptions of
14K categories from ImageNet21K (Deng et al., 2009) with PiTL. Empirically,
the VL models pre-trained with PiTL-generated pairs are strongly favored over
other W-VLP works on image-to-text (I2T) and text-to-image (T2I) retrieval
tasks, with less supervision. The results reveal the effectiveness of PiTL-
generated pairs for VLP.

Vision-language Retrieval; Pre-training; Knowledge Prompting

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
46th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 23–27, 2023; Taipei, Taiwan††booktitle:
Proceedings of the 46th International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR ’23), July 23–27, 2023, Taipei,
Taiwan††doi: XXXXXXX.XXXXXXX††ccs: Information systems Retrieval models and
ranking

(a) VLP: Vision-language Pre-training. W-VLP: Weakly-supervised VLP. (b) Our
proposed Prompts-in-The-Loop (PiTL).

Figure 1. (a) Different VLP problem settings. The de facto VLP requires the
aligned images and the corresponding descriptions. W-VLP learns on multiple
object-level annotations, such as the object bounding boxes and the labels.
Our proposed W-VLP, Prompts-in-The-Loop (PiTL), comes with the least
supervision from the image category labels. (b) PiTL leverages the shared
prompts (P1-3) on the category label and the responses (D1-D3) from a large
language model to generate image-description pairs. Please refer to Table 1
for the nine prompts used in this work.

##  1\. Introduction

Vision-language (VL) models have been advancing rapidly with the introduction
of various Vision-language Pre-training (VLP) methods. The models for VLP can
adapt to various downstream tasks involving VL modalities, such as cross-modal
retrieval (Radford et al., 2021; Luo et al., 2022; Chen et al., 2020; Kim et
al., 2021; Jia et al., 2021), visual question answering (Zhang et al., 2021;
Byun et al., 2022; Li et al., 2020a; Kim et al., 2021; Gan et al., 2020),
visual captioning (Li et al., 2021a, 2022a, 2022b, 2023; Guo et al., 2022),
etc. The secret recipe of a VLP model comes with (1) a decent amount of webly-
supervised111Some of the gathered image-text pairs may not be highly relevant
as they are not validated by a human. image-text pairs, (2) effective pre-
training objectives which learn cross-modal interaction, and (3) sufficient
resources, e.g. GPUs/TPUs, which enable large-scale training. Those models can
also scale well with more image-text data, i.e. the models excel better in
downstream tasks with more pre-training data (Radford et al., 2021; Jia et
al., 2021; Li et al., 2021b; Singh et al., 2022; Zellers et al., 2021; Alayrac
et al., 2022; Li et al., 2022b, 2023, c; Wang et al., 2022a; Li et al., 2022a;
Yang et al., 2022a).

###  1.1. Weakly-supervised VLP

While the success of VLP methods relying on huge amounts of image-text
annotations has been proven, a less visited research path is emerging to
pursue more data-efficient VLP. The data-efficiency of a VL model can be
viewed from the amount of supervision. That is, would a VL model pre-trained
with less image-text data remain as performant in downstream tasks? This
question leads to works (Li et al., 2020b; Wang et al., 2022b; Zhou et al.,
2022; Chen et al., 2022; Wang et al., 2023) in weakly-supervised VLP (W-VLP)
that aims at not relying on image-text pairs from, e.g. SBU Captions (Ordonez
et al., 2011) and Conceptual Captions (CC) (Sharma et al., 2018; Changpinyo et
al., 2021).

Without the aligned images and texts, these works instead resort to a pre-
trained object detector (OD) that generates object tags, i.e. the visual
entities detected in the given image. The paired images and the object tags
offer weaker supervision than those from the image-sentence pairs, but are
still effective as the cross-domain bridge. However, training an OD relies on
object-level annotations, which is still a form of supervision (Wang et al.,
2023). This seems to deviate these VLP works from the fully unsupervised path,
which aims to remove any kind of cross-modal supervision.

###  1.2. Towards Unsupervised VLP

The unsupervised VLP (U-VLP), which aims at learning a VL model without any
supervision across modalities, remains a daunting challenge. As a step towards
U-VLP, we introduce Prompts-in-The-Loop (PiTL) that generates highly effective
image-text pairs for W-VLP without an OD. PiTL capitalizes on image-level,
i.e. a category label per image, instead of object-level supervision from the
object bounding boxes and the corresponding object categories. This leads to a
much harder W-VLP setting since much underlying information about an entity
could no longer be inferred, such as the common co-occurrence of the visual
entities, e.g. a chair and a desk, in a scene, and the entity relations. e.g.
a person usually sits on a bench.

Specifically, given images with category labels, e.g. a duck as shown in Fig.
1, we prompt large language models (LLMs), e.g. GPT-3 (Brown et al., 2020), to
generate descriptions as the external knowledge of the category labels of the
images. In fact, different prompts provide different focuses on each target
category, e.g. one that emphasizes colors: ”Describe the colors seen from a/an
¡category¿?”, and another that emphasizes relations with other entities:
”Describe what could a/an ¡category¿ be seen with?”. This encourages a VL
model to associate all plausible visual traits, entities, actions, and scenes
pertaining to the target categories.

The prompting paradigm is becoming trendy. Recent works (Pratt et al., 2022;
Menon and Vondrick, 2022; Yang et al., 2022b) reveal that textual prompts
generated by LLMs lead to significant improvement in zero-shot image
classification with VLP models like CLIP (Radford et al., 2021). For instance,
prompting a VL model with the LLM-generated description, ”Goldfish are small,
orange fish with shiny scales.”, is more likely to find matches in the visual
domain than the generic ”A photo of a goldfish.”. Likewise, our work explores
if the LLM-generated descriptions could be proved useful as well in the W-VLP
setting.

Our contributions are summarized as follows. Firstly, we propose PiTL that
generates an image-text dataset IN14K containing 9M images with 1M
descriptions of 14K categories from the ”Winter21” release of ImageNet-21K
(Deng et al., 2009). Second, trained with half of the samples in IN14K, our
models are shown competitive to the state of the arts in image-to-text (I2T)
and text-to-image (T2I) retrieval tasks. With full IN14K, our models
significantly outperform them, e.g. on MSCOCO-5K (Lin et al., 2014) by 11% and
10%, respectively, on I2T and T2I. Moreover, our models are comparable with
VLP models trained with 4M aligned image-text pairs from, e.g. CC3M and SBU
Captions, etc. Lastly, PiTL does not only come with the least cross-modal
supervision among W-VLP works, but also leads to a small gap between the W-VLP
and VLP performances.

##  2\. W-VLP with Image-level Supervision

VLP aims at learning VL alignments given a large number of image-sentence
pairs. The methodology is concluded as (1) learning shared semantics around VL
modalities, (2) learning cross-modal context, e.g. masked modeling (Devlin et
al., 2018), and (3) learning to explicitly match images and texts. With the
same aim, the existing W-VLP methods leverage OD-generated tags to form VL
pairs. What is usually neglected is the cost of pre-training such an OD, which
usually requires 10+ object-level annotations to be effective. The proposed
PiTL aims at relaxing the requirement of having an OD via prompting LLMs
(Brown et al., 2020) to generate descriptions for the object categories.

###  2.1. Forming Image-text Pairs via Prompting

PiTL elicits knowledge about an object category from nine prompts of different
perspectives with an LLM. Five descriptions are collected for each prompt.
Table 1 summarizes the nine prompts and their focuses. Some of them are more
visually-relevant (P1-6), some focus more on knowledge around the target
category (P7-8), and some are more open-ended (P9). We study the effectiveness
of the descriptions generated by each prompt later in Sec. 3.3.

Among PiTL-generated pairs, an image can be paired with different descriptions
as long as they are of the same category. In pre-training, the positive pairs
for the Image-Text Contrastive and Image-Text Matching losses (introduced
later in Sec. 2.3) are drawn from the images and descriptions of the same
categories, and the negative pairs from those of the different categories. In
this setup, pre-training with PiTL-generated pairs encourages the VL models to
learn cross-modal alignment at the category level, i.e. images of a target
category aligned to a group of descriptions, instead of instance level, i.e.
an image aligned with a description as in other VLP works. As such, a given
image would be associated with the plausible categorical visual traits,
entities, actions, and scenes through the VL models.

Table 1. LLM prompts and focuses for ¡category¿, e.g. duck.

Types |  Prompts |  Focuses
---|---|---
P1 |  Describe colors of a ¡category¿ |  colors
P2 |  Describe shapes of a ¡category¿ |  shapes
P3 |  Describe textures of a ¡category¿ |  textures
P4 |  Describe visual appearances of a ¡category¿ |  summarized visual appearances
P5 |  Describe a ¡category¿ in a scene |  scenes
P6 |  Describe what a ¡category¿ could be seen with |  relations with other entities
P7 |  Describe the places a ¡category¿ has been seen |  places
P8 |  Describe the main activities of a ¡category¿ |  activities
P9 |  Describe what is it like to be a ¡category¿ |  first-person view

###  2.2. VL Model Architecture

Our model architecture follows a state-of-the-art VL model, ALBEF (Li et al.,
2021b), which has a multi-modal encoder fusing the representations generated
by visual and textual encoders. Indeed, any VL model with image-text inputs
could also be used instead, as proposing a new VL architecture is not the
focus of this work.

Specifically, given an image $V$ and its paired text description $T$, the
vision encoder $g_{v}(\cdot)$ follows ViT (Dosovitskiy et al., 2020)
consisting of a 12-layer Transformer that generates the image embedding as
$g_{v}(V)=(x_{v}^{CLS},x_{v}^{1},\dots,x_{v}^{n_{v}})$. The text encoder
$g_{t}(\cdot)$ is a 6-layer Transformer encoder that embeds the input text as
$g_{t}(T)=(x_{t}^{CLS},x_{t}^{1},\dots,x_{t}^{n_{t}})$, where $x^{CLS}_{v}$
and $x^{CLS}_{t}$ are the representations of the $[CLS]$ tokens summarizing
the image and the text, respectively. $n_{v}$ and $n_{t}$ are the numbers of
image patches and textual tokens, respectively. A fusion encoder
$g_{f}(\cdot)$ consisting of a 6-layer Transformer learns the interaction
across the VL modalities encoded as $g_{v}(V)$ and $g_{t}(T)$, and generates
$g_{f}(g_{v}(V),g_{t}(T))=(x_{f}^{CLS},x_{f}^{1},\dots,x_{f}^{n_{t}})$.

###  2.3. Pre-training Losses

Our PiTL VL-models are pre-trained with four losses (Li et al., 2021b; Yang et
al., 2022a) that all contribute equally to the total loss $\mathcal{L}$:

(1)
$\displaystyle\mathcal{L}=\mathcal{L}_{ITC}+\mathcal{L}_{ITM}+\mathcal{L}_{MLM}+\mathcal{L}_{IMC},$

where each objective is described as follows.
Image-Text Contrastive (ITC) aims to retain high and low similarities between
the positive and negative image-text pairs, respectively. To obtain the ITC
loss, one first calculates

(2) $\displaystyle
p_{m}^{IT}(V)=\frac{{e^{s(V,T_{m})/\tau}}}{\sum_{i=1}^{M}e^{s(V,T_{i})/\tau}},\;p_{m}^{TI}(T)=\frac{{e^{s(T,V_{m})/\tau}}}{\sum_{i=1}^{M}e^{s(T,V_{i})/\tau}},$

where $s(V,T)=(x_{v}^{CLS})^{T}x_{t}^{CLS}$ measures the dot-product
similarity of an image-text pair $(V,T)$. $p_{m}^{IT}(V)$ and $p_{m}^{TI}(T)$
are image-to-text and text-to-image similarities, respectively. $\tau$ is a
learnable temperature parameter and $M$ is the size of the queues storing the
image and textual class embeddings. The ITC loss is then defined as

(3) $\displaystyle\mathcal{L}_{ITC}=\frac{1}{2}\mathbb{E}_{V,T\sim
D}[H(\textbf{y}^{IT}_{V},\textbf{p}^{IT}(V))+H(\textbf{y}^{TI}_{T},\textbf{p}^{TI}(T))],$

where $D$ denotes the pool of image-text pairs, $\textbf{y}^{IT}_{V}$ and
$\textbf{y}^{TI}_{T}$ are $M$-dimensional binary vectors encoding ground-truth
similarity, and $H(\cdot,\cdot)$ refers to the cross-entropy function.

Image-Text Matching (ITM) aims to predict whether an image-text pair is
matched. The token embedding $x_{f}^{CLS}$ of the fusion encoder predicts the
binary classification probability $\textbf{p}^{ITM}$. The ITM loss is defined
as

(4)
$\displaystyle\mathcal{L}_{ITM}=\mathbb{E}_{V,T}[H(\textbf{y}^{ITM},\textbf{p}^{ITM}(V,T))],$

where $\textbf{y}^{ITM}$ is a binary vector indicating the matching pairs.

Masked Language Modeling (MLM) predicts the masked tokens in a sentence given
an image and the unmasked textual tokens in the same sentence. 15% masking
probability is set. The MLM loss (Devlin et al., 2018) is denoted as
$\mathcal{L}_{MLM}$.

Intra-Modal Contrastive (IMC) aims to differentiate the semantics between the
positive and negative pairs within the same modality (Yang et al., 2022a),
i.e. image-image and text-text pairs with similarities:

(5) $\displaystyle
p_{m}^{II}(V)=\frac{{e^{s(V,V_{m})/\tau}}}{\sum_{i=1}^{M}e^{s(V,V_{i})/\tau}},\;p_{m}^{TT}(T)=\frac{{e^{s(T,T_{m})/\tau}}}{\sum_{i=1}^{M}e^{s(T,T_{i})/\tau}}.$

The IMC objective is defined as

(6) $\displaystyle\mathcal{L}_{IMC}=\frac{1}{2}\mathbb{E}_{V,T\sim
D}[H(\textbf{y}^{II}_{V},\textbf{p}^{II}(V))+H(\textbf{y}^{TT}_{T},\textbf{p}^{TT}(T))],$

where $\textbf{y}^{II}_{V}$ and $\textbf{y}^{TT}_{T}$ indicate whether the
pair is matched or not. It is worth noting that $\mathcal{L}_{IMC}$ encourages
the model to retain the uni-modal semantics provided by the pre-trained
weights of the vision and textual encoders, complementing $\mathcal{L}_{ITC}$,
$\mathcal{L}_{ITM}$, and $\mathcal{L}_{MLM}$, all of which promote multi-modal
alignments.

##  3\. Experiments

###  3.1. Settings

Our vision encoder is instantiated by ViT (Dosovitskiy et al., 2020) and
initialized with DEiT (Touvron et al., 2021) or BEiT-B/16 (Bao et al., 2021)
pre-trained weights. The textual encoder is initialized with BERT-Base (Devlin
et al., 2018). The proposed PiTL W-VLP model is pre-trained on three subsets
IN1K, IN6K, and IN14K created from ImageNet21K. Note that IN6K and IN14K are
created in this work. Specifically, IN14K contains IN6K which also contains
IN1K samples. Each prompt, out of the nine shown in Table 1, generates five
responses for a category. Individual prompts are created for multiple
synonyms, e.g. snorkeling and snorkel_diving under the same category.
Statistics of IN1K, IN6K, IN14K along with other datasets, e.g. CC3M,
BookCorpus (BC) (Zhu et al., 2015), and VL-Full (Wang et al., 2022b) used by
other VLP methods, are shown in Table 2 under the Pre-training Corpus column.
We assess the pre-training quality on I2T and T2I with MSCOCO-5K and Flickr30K
(Ordonez et al., 2011). The retrieval models are evaluated on recall at rank K
(R@K).

Table 2. Results of image-text retrieval tasks on MSCOCO-5K and Flickr30K test
splits. ∗ denotes the results reported by (Wang et al., 2023). $\dagger$
denotes BEiT (Bao et al., 2021) self-supervised pre-trained on the images of
ImageNet21K. $\ddagger$ denotes BEiT$\dagger$ intermediate fine-tuned on the
images and labels of ImageNet1K. $\star$ denotes BEiT$\dagger$ intermediate
fine-tuned on the images and labels of ImageNet21K.

VLP |  Supervision |  Visual Init | Pre-training Corpus | MSCOCO-5K | Flickr30K
---|---|---|---|---|---
|  |  # of Images |  # of Texts | I2T | T2I | I2T | T2I
|  |  |  |  R@1 |  R@5 |  R@10 |  R@1 |  R@5 |  R@10 |  R@1 |  R@5 |  R@10 |  R@1 |  R@5 |  R@10
ViLT (Kim et al., 2021) |  image-level annotations \+ aligned VL pairs |  ViT-B/32 |  4M |  4M |  61.5 |  86.3 |  92.7 |  42.7 |  72.9 |  83.1 |  83.5 |  96.7 |  98.6 |  64.4 |  88.7 |  93.8
ALBEF (Li et al., 2021b) |  |  ViT-B/16 (DeiT) |  4M |  4M |  73.1 |  91.4 |  96.0 |  56.8 |  81.5 |  89.2 |  94.3 |  99.4 |  99.8 |  82.8 |  96.7 |  98.4
TCL (Yang et al., 2022a) |  |  ViT-B/16 (DeiT) |  4M |  4M |  75.6 |  92.8 |  96.7 |  59.0 |  83.2 |  89.9 |  94.9 |  99.5 |  99.8 |  84.0 |  96.7 |  98.5
W-VLP |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
U-VisualBERT (Li et al., 2020b) |  image-level + object-level annotations |  BERT-Base |  CC: 3M |  CC+BC: 5.5M |  – |  – |  – |  – |  – |  – |  67.8∗ |  90.7∗ |  94.9∗ |  55.4 |  82.9 |  89.8
WFH (Wang et al., 2023) |  |  BERT-Base |  CC: 3M |  CC: 3M |  – |  – |  – |  – |  – |  – |  72.0 |  91.3 |  95.6 |  56.4 |  83.2 |  89.9
E2E-UVLP (Chen et al., 2022) |  |  Swin-B/32 |  CC: 3M |  CC: 3M |  – |  – |  – |  – |  – |  – |  – |  – |  – |  66.4 |  89.7 |  94.1
VLMixer (Wang et al., 2022b) |  – |  BERT-Base |  – |  – |  57.4 |  84.0 |  91.6 |  44.0 |  74.1 |  84.1 |  – |  – |  – |  – |  – |  –
VLMixer (Wang et al., 2022b) |  image-level + object-level annotations |  BERT-Base |  CC: 3M |  CC: 3M |  62.2 |  86.3 |  92.8 |  47.4 |  76.2 |  85.4 |  – |  – |  – |  – |  – |  –
VLMixer (Wang et al., 2022b) |  |  BERT-Base |  VL-Full: 5.9M |  VL-Full: 22.4M |  64.8 |  88.6 |  94.2 |  50.1 |  78.4 |  86.9 |  – |  – |  – |  – |  – |  –
PiTL |  – |  BEiT-B/16$\dagger$ |  – |  – |  58.0 |  84.2 |  91.6 |  42.3 |  71.2 |  80.9 |  64.9 |  88.8 |  93.9 |  48.2 |  75.1 |  82.6
PiTL |  image-level annotations |  BEiT-B/16$\dagger$ |  IN1K: 1.3M |  IN1K: 45K |  63.0 |  86.9 |  93.3 |  46.9 |  75.6 |  84.8 |  78.1 |  95.9 |  98.2 |  62.8 |  87.1 |  91.7
PiTL |  |  BEiT-B/16$\dagger$ |  IN6K: 4.5M |  IN6K: 0.4M |  63.7 |  88.7 |  94.1 |  48.6 |  76.8 |  85.7 |  81.8 |  96.4 |  98.8 |  65.6 |  87.9 |  92.8
PiTL |  |  BEiT-B/16$\dagger$ |  IN14K: 9M |  IN14K: 1M |  67.4 |  90.2 |  95.2 |  51.3 |  78.6 |  86.9 |  83.9 |  97.5 |  98.6 |  68.9 |  89.8 |  93.8
PiTL |  – |  ViT-B/16 (DeiT) |  – |  – |  58.9 |  85.0 |  91.8 |  44.1 |  73.1 |  81.9 |  68.6 |  89.7 |  94.6 |  53.7 |  79.4 |  85.4
PiTL |  image-level annotations |  ViT-B/16 (DeiT) |  IN1K: 1.3M |  IN1K: 45K |  62.1 |  86.6 |  93.1 |  46.7 |  75.8 |  84.7 |  74.1 |  93.4 |  96.4 |  61.0 |  85.3 |  90.6
PiTL |  |  ViT-B/16 (DeiT) |  IN6K: 4.5M |  IN6K: 0.4M |  65.6 |  88.5 |  94.5 |  49.5 |  77.4 |  86.1 |  80.1 |  95.2 |  97.8 |  66.6 |  88.7 |  92.7
PiTL |  |  ViT-B/16 (DeiT) |  IN14K: 9M |  IN14K: 1M |  68.2 |  90.4 |  95.1 |  51.9 |  78.7 |  86.7 |  84.3 |  97.3 |  99.0 |  70.3 |  91.0 |  94.3
PiTL |  – |  BEiT-B/16$\ddagger$ |  – |  – |  62.1 |  86.7 |  93.3 |  46.2 |  75.2 |  84.3 |  75.7 |  95.2 |  97.8 |  59.8 |  84.9 |  89.9
PiTL |  image-level annotations |  BEiT-B/16$\ddagger$ |  IN1K: 1.3M |  IN1K: 45K |  63.5 |  88.3 |  93.9 |  48.3 |  76.8 |  85.4 |  79.8 |  95.7 |  98.0 |  64.7 |  88.4 |  92.5
PiTL |  |  BEiT-B/16$\ddagger$ |  IN6K: 4.5M |  IN6K: 0.4M |  66.0 |  88.9 |  94.6 |  49.8 |  78.2 |  86.4 |  82.9 |  96.6 |  98.9 |  67.9 |  89.6 |  93.9
PiTL |  |  BEiT-B/16$\ddagger$ |  IN14K: 9M |  IN14K: 1M |  69.8 |  91.0 |  95.7 |  53.9 |  79.8 |  87.8 |  88.0 |  98.7 |  99.5 |  74.6 |  92.4 |  95.6
PiTL |  – |  BEiT-B/16$\star$ |  – |  – |  67.5 |  90.0 |  94.7 |  51.0 |  78.9 |  87.3 |  81.6 |  96.4 |  98.7 |  67.0 |  88.9 |  92.7
PiTL |  image-level annotations |  BEiT-B/16$\star$ |  IN1K: 1.3M |  IN1K: 45K |  69.1 |  90.4 |  95.2 |  52.8 |  79.4 |  87.4 |  86.8 |  97.6 |  99.3 |  72.3 |  91.3 |  95.1
PiTL |  |  BEiT-B/16$\star$ |  IN6K: 4.5M |  IN6K: 0.4M |  70.4 |  91.0 |  95.6 |  53.5 |  80.0 |  87.6 |  87.2 |  98.4 |  99.3 |  73.4 |  92.4 |  95.6
PiTL |  |  BEiT-B/16$\star$ |  IN14K: 9M |  IN14K: 1M |  71.9 |  92.2 |  96.4 |  55.0 |  80.8 |  88.2 |  90.7 |  98.7 |  99.5 |  76.2 |  93.5 |  95.9

###  3.2. Quantitative Results

Table 2 shows the main results of PiTL and the comparisons against the state-
of-the-art VLP and W-VLP on the I2T and T2I tasks.

Effects of Initialization and Dataset Sizes. We initialize the image encoder
with weights pre-trained with no supervision (i.e. the self-supervised
BEiT$\dagger$) and with image-level supervision (i.e. ViT, BEiT$\ddagger$, and
BEiT$\star$). The best performances are obtained with BEiT$\star$, whose
weights contain the strongest visual semantics, on IN1K to IN14K. PiTL’s
results steadily improve with more images and descriptions. Compared to models
without pre-training, BEiT$\dagger$ pre-trained on IN1K has larger
improvements in R@1 than the other initializations, i.e. 8.6% for I2T and
10.8% for T2I R@1 on MSCOCO-5K.

PiTL with BEiT-B/16$\dagger$ vs. W-VLP Works. To purely assess the generated
image-text pairs, models initialized with self-supervised BEiT-B/16$\dagger$
weights are mainly benchmarked. VLMixer starts out as a better model than PiTL
when both are not pre-trained on any image-text pairs. However, once pre-
trained, PiTL appears to be strongly competitive. For instance, PiTL pre-
trained on IN1K outperforms VLMixer pre-trained on CC3M, across all the I2T
metrics, with fewer images and texts. Pre-trained on IN6K and IN14K, PiTL is
strongly competitive and better than VLMixer pre-trained on VL-Full. On
Flickr30K, PiTL is also comparable to E2E-UVLP with more images and fewer
texts.

PiTL Models Pre-trained on IN14K vs. W-VLP Works. While empowered by the pre-
trained weights, i.e. ViT, BEiT-B/16$\ddagger$, and BEiT-B/16$\star$, PiTL is
strongly favored over other W-VLP works across all the metrics. We stress that
obtaining pre-trained weights requires far less amount of supervision, i.e.
14M labels for 14M images versus 10+ object annotations per image, compared to
an effectively pre-trained OD.

###  3.3. Ablation Studies

We study the retrieval performances on Flickr30K on the types of prompts
(P1-P9 in Table 1) and the effect of the paired images and descriptions
generated by PiTL.

On Effects of Prompts P1-P9. We dissect the effect contributed by prompts P1-9
to the retrieval performance. We pre-train models initialized with
BEiT-B/16$\star$ on descriptions from each prompt. Table 3 shows that, with
all the descriptions available, the models achieve the best R@1 scores for
both retrieval tasks. On I2T, P9 results in the best average recall over
R@1,5,10. We reckon that P9 is a more open-ended question than the others,
which focus specifically on appearances, relations, etc. The open-endedness
could result in more diverse descriptions and hence better I2T result. On T2I,
P1-4 usually yield better T2I results than the other prompts. This seems to be
expected since P1-4 are more visually relevant.

Table 3. Effects of each type of prompt of Table 1 on retrieval tasks. VA is
short for visual appearances.

| I2T | T2I |  Overall
---|---|---|---
Types |  R@1 |  R@5 |  R@10 |  AvgR |  R@1 |  R@5 |  R@10 |  AvgR |  AvgR
P1: colors |  85.4 |  97.7 |  98.9 |  94.0 |  71.7 |  91.6 |  94.9 |  86.0 |  90.0
P2: shapes |  86.1 |  97.7 |  98.9 |  94.2 |  70.7 |  92.0 |  95.2 |  86.0 |  90.1
P3: textures |  84.9 |  97.3 |  99.4 |  93.9 |  72.0 |  92.0 |  95.4 |  86.5 |  90.2
P4: summarized VA |  86.1 |  97.3 |  98.9 |  94.1 |  70.5 |  92.1 |  95.4 |  86.0 |  90.0
P5: scenes |  85.7 |  98.2 |  99.3 |  94.4 |  71.7 |  91.5 |  95.6 |  86.3 |  90.3
P6: relations |  84.8 |  98.3 |  99.0 |  94.0 |  70.3 |  91.1 |  95.1 |  85.5 |  89.8
P7: places |  83.7 |  97.5 |  99.3 |  93.5 |  70.0 |  91.4 |  95.1 |  85.5 |  89.5
P8: activities |  85.2 |  98.1 |  99.0 |  94.1 |  71.4 |  91.4 |  95.1 |  86.0 |  90.0
P9: first-person |  85.6 |  98.2 |  99.5 |  94.4 |  70.0 |  91.5 |  95.2 |  85.5 |  90.0
All |  86.8 |  97.6 |  99.3 |  94.6 |  72.3 |  91.3 |  95.1 |  86.2 |  90.4

On the Effect of Increased Uni-modal Shuffled Sources. To study whether the
improvements actually come from the proposed prompt-based weak supervision,
rather than from the pure increase in the number of images and texts, we pre-
train the models (initialized with BEiT-B/16$\star$) on IN1K and IN6K with
shuffled images and descriptions. Indeed, on Flickr30K, the models degrade in
R@1 from 60.7 to 56.1 for I2T, and from 48.2 to 42.7 for T2I, when pre-trained
on shuffled IN1K and IN6K, respectively. This concludes the effectiveness of
PiTL-generated pairs for VLP.

##  4\. Conclusion

In this work, we proposed Prompts-in-The-Loop (PiTL), a weakly-supervised
method to pre-train VL-models for cross-modal retrieval tasks. Without object-
level supervision based on OD, our PiTL comes with image-level supervision
from images and their categories described by large language models. Retrieval
results on the cross-modal datasets demonstrated the effectiveness of PiTL.

###### Acknowledgements.

This work is supported by the Academy of Finland in project 345791. We
acknowledge the LUMI supercomputer, owned by the EuroHPC Joint Undertaking,
hosted by CSC and the LUMI consortium.

## References

  * (1)
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. 2022\.  Flamingo: a visual language model for few-shot learning.  _arXiv preprint arXiv:2204.14198_ (2022).
  * Bao et al. (2021) Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021\.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ (2021).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Byun et al. (2022) Jaeseok Byun, Taebaek Hwang, Jianlong Fu, and Taesup Moon. 2022\.  GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-training. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIX_. Springer, 395–412.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021\.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3558–3568.
  * Chen et al. (2022) Chi Chen, Peng Li, Maosong Sun, and Yang Liu. 2022\.  End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. 10799–10810.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020\.  Uniter: Universal image-text representation learning. In _European conference on computer vision_. Springer, 104–120.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Gan et al. (2020) Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020\.  Large-scale adversarial training for vision-and-language representation learning.  _Advances in Neural Information Processing Systems_ 33 (2020), 6616–6628.
  * Guo et al. (2022) Zixin Guo, Tzu-Jui Wang, and Jorma Laaksonen. 2022.  CLIP4IDC: CLIP for Image Difference Captioning. In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing_. 33–42.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021\.  Scaling up visual and vision-language representation learning with noisy text supervision. In _International Conference on Machine Learning_. PMLR, 4904–4916.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision. In _International Conference on Machine Learning_. PMLR, 5583–5594.
  * Li et al. (2022c) Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. 2022c.  mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections.  _arXiv preprint arXiv:2205.12005_ (2022).
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023\.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  _arXiv preprint arXiv:2301.12597_ (2023).
  * Li et al. (2022b) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _International Conference on Machine Learning_. PMLR, 12888–12900.
  * Li et al. (2021b) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021b.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ 34 (2021), 9694–9705.
  * Li et al. (2020b) Liunian Harold Li, Haoxuan You, Zhecan Wang, Alireza Zareian, Shih-Fu Chang, and Kai-Wei Chang. 2020b.  Unsupervised vision-and-language pre-training without parallel images and captions.  _arXiv preprint arXiv:2010.12831_ (2020).
  * Li et al. (2021a) Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021a.  UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 2592–2607.
  * Li et al. (2022a) Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2022a.  UNIMO-2: end-to-end unified vision-language grounded learning.  _arXiv preprint arXiv:2203.09067_ (2022).
  * Li et al. (2020a) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020a.  Oscar: Object-semantics aligned pre-training for vision-language tasks. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16_. Springer, 121–137.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 740–755.
  * Luo et al. (2022) Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022.  CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning.  _Neurocomputing_ 508 (2022), 293–304.
  * Menon and Vondrick (2022) Sachit Menon and Carl Vondrick. 2022.  Visual Classification via Description from Large Language Models.  _arXiv preprint arXiv:2210.07183_ (2022).
  * Ordonez et al. (2011) Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011\.  Im2text: Describing images using 1 million captioned photographs.  _Advances in neural information processing systems_ 24 (2011).
  * Pratt et al. (2022) Sarah Pratt, Rosanne Liu, and Ali Farhadi. 2022.  What does a platypus look like? Generating customized prompts for zero-shot image classification.  _arXiv preprint arXiv:2209.03320_ (2022).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018\.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2556–2565.
  * Singh et al. (2022) Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022.  Flava: A foundational language and vision alignment model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15638–15650.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021.  Training data-efficient image transformers & distillation through attention. In _International conference on machine learning_. PMLR, 10347–10357.
  * Wang et al. (2022b) Teng Wang, Wenhao Jiang, Zhichao Lu, Feng Zheng, Ran Cheng, Chengguo Yin, and Ping Luo. 2022b.  Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix. In _International Conference on Machine Learning_. PMLR, 22680–22690.
  * Wang et al. (2023) Tzu-Jui Julius Wang, Jorma Laaksonen, Tomas Langer, Heikki Arponen, and Tom E Bishop. 2023.  Learning by Hallucinating: Vision-Language Pre-training with Weak Supervision. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 1073–1083.
  * Wang et al. (2022a) Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022a.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  _arXiv preprint arXiv:2208.10442_ (2022).
  * Yang et al. (2022a) Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. 2022a.  Vision-language pre-training with triple contrastive learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15671–15680.
  * Yang et al. (2022b) Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. 2022b.  Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification.  _arXiv preprint arXiv:2211.11158_ (2022).
  * Zellers et al. (2021) Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021\.  Merlot: Multimodal neural script knowledge models.  _Advances in Neural Information Processing Systems_ 34 (2021), 23634–23651.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021\.  Vinvl: Revisiting visual representations in vision-language models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5579–5588.
  * Zhou et al. (2022) Mingyang Zhou, Licheng Yu, Amanpreet Singh, Mengjiao Wang, Zhou Yu, and Ning Zhang. 2022\.  Unsupervised vision-and-language pre-training via retrieval-based multi-granular alignment. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16485–16494.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _Proceedings of the IEEE international conference on computer vision_. 19–27.