# Submanifolds of some Hartogs domain and the complex Euclidean space
111Keywords: Hartogs domain, isometric embedding, Bergman metric, Nash
algebraic function, rational function; 2010 Subject classes: 32H02, 32Q40,
53B35.

Xu Zhang, Donghai Ji

###### Abstract

Two Kähler manifolds are called relatives if they admit a common Kähler
submanifold with the same induced metrics. In this paper, we show that a
Hartogs domain over an irreducible bounded symmetric domain equipped with the
Bergman metric is not a relative to the complex Euclidean space. This
generalizes the results in [5, 4] and the novelty here is that the Bergman
kernel of the Hartogs domain is not necessarily Nash algebraic.

##  1 Introduction

Holomorphic isometric embedding is an important topic in complex geometry.
Calabi [2] obtained the celebrated results on the global extendability and
rigidity of a local holomorphic isometry into a complex space form. In
particular, Calabi proved that any complex space form cannot be locally
isometrically embedded into another complex space form with a different
curvature sign with respect to the canonical Kähler metrics, respectively.
Along this line, Di Scala and Loi generalized Calabi’s non-embeddability
result to the case of Hermitian symmetric spaces of different types [7]. For
related problems on the existence of the Kähler-Einstein submanifold or the
Kähler-Ricci soliton of complex space forms, the interested readers may refer
to [16, 12]. For the study of holomorphic isometric embeddings between bounded
symmetric domains, the interested readers may refer to [13, 14, 20] and
references therein.

On the other hand, Umehara [17] studied a more general question whether two
complex space forms can share a common submanifold with the induced metrics
and proved that two complex space forms with different curvature signs cannot
share a common Kähler submanifold by using Calabi’s idea of diatasis. When two
complex manifolds share a common Kähler submanifolds with induced metrics,
they are called relatives by Di Scala and Loi [8]. Furthermore, Di Scala and
Loi proved that a bounded domain equipped with its canonical Bergman metric
can not be a relative to a Hermitian symmetric space of compact type equipped
with the canonical metric. Since any irreducible Hermitian symmetric space of
compact type can be holomorphically isometrically embedded into a complex
project space by the classical Nakagawa-Takagi embedding, in order to show
that a Kähler manifold is not a relative of a projective manifold with the
induced metric, it suffices to show that it is not a relative to the complex
projective space with the Fubini-Study metric. In fact, it follows from the
result of Umehara [17], the complex Euclidean space and the irreducible
Hermitian symmetric space of compact type cannot be relatives. Later, Huang
and Yuan showed that a complex Euclidean space and a Hermitian symmetric space
of noncompact type cannot be relatives [10]. Cheng, Di Scala and Yuan also
considered the relativity problem for indefinite complex space forms [3].

More recently, Su, Tang and Tu showed that a complex Euclidean space and the
symmetrized polydisc are not relatives [15]. Cheng and Hao provided a
sufficient condition to determine whether a real analytic Kähler manifold is
not a relative to a complex space form equipped with its canonical metric. As
consequences, minimal domains, bounded homogeneous domains and some Hartogs
domains equipped with their Bergman metrics are not relatives to the complex
Euclidean space [4]. In these results, one key property for the canonical
Bergman metrics is that the Bergman kernel functions are Nash algebraic (even
rational) functions.

In [5], Cheng and Niu considered the relativity problem for the complex
Euclidean space and the Cartan-Hartogs domain. The fundamental difference in
this problem is that the Bergman kernel function of the Cartan-Hartogs domain
is no longer Nash algebraic. Cheng and Niu were able to show a special case
that any submanifold passing the origin of the Cartan-Hartogs domain cannot be
the submanifold of the complex Euclidean space. In this paper, we solve the
general case, even for the Hartogs domain over a bounded, complete circular,
homogeneous, Lu Qi-Keng domain equipped with the Bergman metric.

Let $D\subset\mathbb{C}^{d}$ be a domain and $\varphi$ be a continuous
positive function on $D$. The domain

$\Omega=\left\\{(\xi,z)\in\mathbb{C}^{d_{0}}\times
D:||\xi||^{2}<\varphi(z)\right\\}$ (1)

is called a Hartogs domain over $D$ with $d_{0}$-dimensional fibers. When $D$
is a bounded homogeneous domain and $\varphi(z)=K_{D}(z,\overline{z})^{-s}$,
Ishi, Park and Yamamori [11] proved that the Bergman kernel is

$K((z,\xi),\overline{(w,\eta)})=\frac{K_{D}(z,\overline{w})^{d_{0}s+1}}{\pi^{d_{0}}}\sum_{j=0}^{d}\frac{c(s,j)(j+d_{0})!}{(1-t)^{j+d_{0}+1}},$
(2)

with $t=K_{D}(z,\overline{w})^{s}<\xi,\eta>$, where the constants $c(s,j)$
satisfy that $F(ks)=\sum_{j=0}^{d}c(s,j)(k+1)_{j}$ and $F$ is the polynomial
given by (18) in [11]. If $D$ is a bounded homogeneous domain whose Bergman
kernel function $K_{D}(z,\overline{w})$ is a rational function on
$D\times\hbox{conj}(D)$, then $K((z,\xi),\overline{(w,\eta)})$ is also a
rational function on $\Omega\times\text{conj}(\Omega)$ for $s\in\mathbb{Z}$.
In this case, $\Omega$ equipped with the Bergman metric is not a relative to
$\mathbb{C}^{n}$ equipped with the Euclidean metric (cf. [4]). However, in
general, for $s\not\in\mathbb{Z}$, $K((z,\xi),\overline{(w,\eta)})$ is a
product of a rational function to a power and a rational function in $t$ and
the problem becomes totally different.

Denote the Euclidean metric on ${\mathbb{C}}^{n}$ by
$\omega_{{\mathbb{C}}^{n}}$ and the Bergman metric on $\Omega$ by
$\omega_{\Omega}$. We will show that there do not simultaneously exist
holomorphic isometric immersions
$F=(f_{1},\cdots,f_{n}):(U,\omega_{U})\rightarrow({\mathbb{C}}^{n},\omega_{{\mathbb{C}}^{n}})$
and
$G=(g_{1},\cdots,g_{d_{0}},h_{1},\cdots,h_{d}):(U,\omega_{U})\rightarrow(\Omega,\omega_{\Omega})$.
As a consequence, a complex Euclidean space and a Hartogs domain over an
irreducible bounded symmetric domain equipped with the Bergman metric cannot
be relatives. Indeed, we prove the following slightly stronger result:

######  Theorem 1.1.

Let $\Omega$ be a Hartogs domain over an irreducible bounded symmetric domain
$D$ defined by (1). Let $U\subset{\mathbb{C}}$ be a connected open subset.
Suppose that $F:U\rightarrow{\mathbb{C}}^{n}$ and $G:U\rightarrow\Omega$ are
holomorphic mappings such that

$F^{*}\omega_{{\mathbb{C}}^{n}}=\mu
G^{*}\omega_{\Omega}~{}~{}\text{on}~{}~{}U$ (3)

for a real constants $\mu$. Then $F$ must be a constant map.

By the same argument, we are able to show a slightly general result. Assume
each $\Omega$ to be a Hartogs domain over an irreducible bounded symmetric
domain defined by (1). Suppose $F:U\rightarrow{\mathbb{C}}^{n}$ and
$G_{j}:U\rightarrow\Omega_{j}$ are holomorphic mappings such that

$F^{*}\omega_{{\mathbb{C}}^{n}}=\sum_{j=1}^{J}\mu_{j}G_{j}^{*}\omega_{\Omega_{j}}~{}~{}\text{on}~{}~{}U$
(4)

for certain real constants $\mu_{1},\cdots,\mu_{J}$. Then $F$ must be a
constant map. Furthermore, if all $\mu_{j}^{\prime}s$ are positive, then $G$
is also a constant map.

The next result follows directly from Theorem 1.1.

######  Corollary 1.2.

There does not exist a Kähler manifold $(X,\omega_{X})$ that can be
holomorphic isometrically embedded into the complex Euclidean space
$({\mathbb{C}}^{n},\omega_{{\mathbb{C}}^{n}})$ and also into a Hartogs domain
$(\Omega,\omega_{\Omega})$ over an irreducible bounded symmetric domain
defined by (1).

The corollary generalizes the results in [4, 5] for arbitrary
$s\in\mathbb{R}$. Again, the key difference in this work from previous works
is the fact that the Bergman kernel for a Hartogs domain over an irreducible
bounded symmetric domain is no longer Nash algebraic, which makes the general
argument in [10] inapplicable (cf. [4]).

We conclude the paper with a remark. One notes that all results stated above
hold for the Hartogs domain $\Omega$ over a bounded, complete circular,
homogeneous, Lu Qi-Keng domain equipped with the Bergman metric. All the
argument works the same provided that Lemma 2.2 holds for such Hartogs domain
and we explain there why this is the case.

##  2 Proof of Theorem 1.1

Our proof fundamentally uses ideas developed in [9, 10]. First, we prove a
generalization of Lemma 2.2 in [10].

######  Lemma 2.1.

Let $V\subset{\mathbb{C}}^{k}$ be a connected open set. Let
$H_{1}(\xi),\cdots,H_{K}(\xi)$ and $H(\xi)$ be holomorphic Nash algebraic
functions on $V$ and $R(t)$ be a holomorphic rational function. Assume that

$\exp^{H(\xi)}=R(H_{1}^{\mu_{1}}(\xi)H_{2}^{\mu_{2}}(\xi))\prod_{j=3}^{K}H_{j}^{\mu_{j}}(\xi),\
\xi\in V,$ (5)

for certain real numbers $\mu_{1},\cdots,\mu_{K}$. Then $H(\xi)$ is constant.

###### Proof.

We include a brief argument here and refer the readers to the proof of Lemma
2.2 in [10] for details.

Suppose that $H$ is not constant. After a linear transformation in $\xi$ and
the dimension reduction, we can assume, without loss of generality, that,
$H(\xi)$ is not constant for a certain fixed $\xi_{2},\cdots,\xi_{k}$. Then
$H$ is a non-constant Nash-algebraic holomorphic function in $\xi_{1}$ for
such fixed $\xi_{2},\cdots,\xi_{k}$. Hence, we can assume that $j=1$ to
achieve a contradiction. Write $H=H(\xi)$ and $H_{j}=H_{j}(\xi)$ for each
$1\leq j\leq K$. Use $S\subset{\mathbb{C}}$ to denote the union of branch
points, poles and zeros of $H(\xi)$ and $H_{j}(\xi)$ for each $j$. Given a
$p\in{\mathbb{C}}\setminus S$ and a real curve in ${\mathbb{C}}\setminus S$
connecting $p$ and $V$, by holomorphic continuation, the equation (5) holds on
an open neighborhood of the curve. Suppose that some branch $H^{(*)}$ of $H$
blows up at $\xi_{0}\in\mathbb{C}\cup\\{\infty\\}$. Then by Puiseux expansion
of $H$ near $\xi_{0}$, $|e^{H^{(*)}(x)}|$ goes to infinity exponentially fast
as $\xi$ tends to $\xi_{0}$. However, the right hand side of (5) grows at most
polynomially. This is a contradiction. ∎

The following key result is a consequence of Proposition 2 in [18]. Although
in [18] a more general result was merely stated for the Hartogs domain over an
irreducible bounded symmetric domain, if the reader applies the argument in
[19, 1] carefully, it is not difficulty to find out Lemma 2.2 also holds for
the Hartogs domain $\Omega$ over a bounded, complete circular, homogeneous, Lu
Qi-Keng domain.

######  Lemma 2.2.

For any point $(\xi_{0},z_{0})\in\Omega$, there exists $\Phi\in$
Aut$(\Omega)$, such that $\Phi(\xi_{0},z_{0})=(\xi^{\prime}_{0},0)$.

Now, we apply the argument in [12] to show the algebraicity.

######  Lemma 2.3.

Writing
$S=\\{\phi_{1},\cdots,\phi_{l}\\}=\\{f_{1},\cdots,f_{n},g_{1},\cdots,g_{d_{0}},h_{1},\cdots,h_{d}\\}$,
then there exists a maximal algebraic independent subset
$\\{\phi_{1},\cdots,\phi_{k}\\}\subset S$ over the field $\cal{R}$ of rational
functions on $U$, and Nash algebraic functions
$\hat{\phi}_{j}(t,X_{1},\cdots,X_{k})$, such that
$\phi_{j}(t)=\hat{\phi}_{j}(t,\phi_{1}(t),\cdots,\phi_{k}(t))$ for all $1\leq
j\leq l$ after shrinking $U$ toward the origin if needed.

###### Proof.

The proof is identical to the argument in [12] and we include the proof here
for completeness. Let $k$ be the transcendence degree of the field extension
$\cal{R}(S)/\cal{R}$. If $k=0$, then every element in $S$ is holomorphic Nash
algebraic. Otherwise, we have a maximal algebraic independent subset over
$\cal R$, denoted by $\phi_{1},\cdots,\phi_{k}$. By the algebraic version of
the existence and uniqueness of the implicit function theorem, there exist a
connected open set, still denoted by $U$ and Nash algebraic functions
$\hat{\phi}_{j}(t,X_{1},\cdots,X_{k})$, defined in a neighborhood $\hat{U}$ of
$\\{(s,\phi_{1}(s),\cdots,\phi_{k}(s))|s\in U\\}$, such that
$\phi_{j}(t)=\hat{\phi}_{j}(t,\phi_{1}(t),\cdots,\phi_{k}(t))$ for all $s\in
U$ and $1\leq j\leq l$. ∎

We are now in a position to prove the main result.

Proof of Theorem 1.1:

Without loss of generality, we assume $0\in U$. By composing $F$ with
Aut$(\mathbb{C}^{n})$ and $G$ with $\Phi\in$ Aut$(\Omega)$, we may assume
$F(0)=0,G(0)=(\xi_{0},0)$. Since the metrics are invariant under
automorphisms, the equation (4) still holds. We still denote the maps by $F$
and $G$. Let $F=(f_{1},\cdots,f_{n}):D\rightarrow{\mathbb{C}}^{n},\ \
G=(g_{1},\cdots,g_{d_{0}},h_{1},\cdots,h_{d}):U\rightarrow\Omega$ be
holomorphic maps. We argue by contradiction. Suppose that $F$ is not constant.
By equation (4), we have

$\partial\bar{\partial}\left(\sum_{i=1}^{n}|f_{i}(s)|^{2}\right)=\mu\partial\bar{\partial}\log
K(G(s),\overline{G(s)})~{}~{}\text{for~{}~{}}s\in U,$

where $K((\xi,z),\overline{(\eta,w)})$ is the Bergman kernel on $\Omega$.
Since $\Omega$ is a complete circular domain, by the standard argument (cf.
[13, 10]), $K((\xi,z),\overline{(\eta,w)})$ does not contain any nonconstant
pure holomorphic terms in $(\xi,z)$. By comparing the pure holomorphic and
anti-holomorphic terms in $(\xi,z)$ (cf. [6]), one can get rid of
$\partial\bar{\partial}$ to obtain the following functional identity:

$\sum_{i=1}^{n}|f_{i}(s)|^{2}=\mu\log
K(G(s),\overline{G(s)})~{}~{}\text{~{}for~{}any~{}}s\in U.$ (6)

After polarization, (6) is equivalent to

$\sum_{i=1}^{n}f_{i}(s)\bar{f_{i}}(t)=\mu\log
K(G(s),\bar{G}(t))~{}~{}\text{~{}for~{}~{}}(s,t)\in U\times\hbox{conj}({U}),$
(7)

where $\bar{f}_{i}(s)=\overline{f_{i}(\overline{s})}$ and
$\hbox{conj}({U})=\\{z\in{\mathbb{C}}|\bar{s}\in U\\}$.

Define

$\Psi(s,X,t)=\sum_{i=1}^{n}\hat{f}_{i}(s,X)\bar{f}_{i}(t)-\mu\log
K((\hat{g}_{1}(s,X),\cdots,\hat{h}_{d}(s,X)),(\bar{g}_{1}(t),\cdots,\bar{h}_{d}(t)))$

and

$\Psi_{i}(s,X,t)=\frac{\partial^{i}\Psi}{\partial t^{i}}(s,X,t)$

for $(s,X,t)\in\hat{U}\times\hbox{conj}(U)$, where $X=(X_{1},\cdots,X_{k})$.

######  Lemma 2.4.

For any $t$ near 0 and any $(s,X)\in\hat{U}$, $\Psi(s,X,t)\equiv\Psi(s,X,0)$.

###### Proof.

It suffices to show that $\Psi_{i}(s,X,0)\equiv 0$ for all $i>0$. Since
$K_{D}(z,\bar{w})$ is a rational function and also
$K_{D}((\hat{h}_{1}(s,X),\cdots,\hat{h}_{d}(s,X)),\overline{(h_{1}(0),\cdots
h_{d}(0))})=K_{D}((\hat{h}_{1}(s,X),\cdots,\hat{h}_{d}(s,X)),(0,\cdots,0))\equiv
1$, then $\Psi_{i}(s,X,0)$ is a Nash algebraic function in $(s,X)$. Assume it
is not constant. Then there exists a holomorphic polynomial
$P(s,X,y)=A_{d}(s,X)y^{d}+\cdots+A_{0}(s,X)$ of degree $d$ in $y$, with
$A_{0}(s,X)\not\equiv 0$ such that $P(s,X,\Psi_{i}(s,X,0))\equiv 0$. As
$\Psi(s,\phi_{1}(s),\cdots,\phi_{k}(s),t)\equiv 0$ for $s\in{U}$, it follows
that $\Psi_{i}(s,\phi_{1}(s),\cdots,\phi_{k}(s),0)\equiv 0$ and therefore
$A_{0}(s,\phi_{1}(s),\cdots,\phi_{k}(s))\equiv 0$. This means that
$\\{\phi_{1}(s),\cdots,\phi_{k}(s)$} are algebraic dependent over $\cal{R}$.
This is a contradiction and it follows that $\Psi_{i}(s,X,0)$ is a constant.
Therefore, $\Psi_{i}(s,X,0)=\Psi_{i}(s,\phi_{1}(s),\cdots,\phi_{k}(s),0)\equiv
0.$

∎

######  Lemma 2.5.

There exists $(s_{0},t_{0})\in U\times\hbox{conj}(U)$ with $t_{0}\not=0$ such
that

$\sum_{i=1}^{n}\hat{f}_{i}(s_{0},X)\bar{f}_{i}(t_{0})\not\equiv\text{constant}.$

###### Proof.

Assume the conclusion is false. Letting $t=\overline{s}$ and
$X=(\phi_{1}(s),\cdots,\phi_{k}(s))$, it follows that

$\sum_{i=1}^{n}|f_{i}(s)|^{2}=\sum_{i=1}^{n}\hat{f}_{i}(s,\phi_{1}(s),\cdots,\phi_{k}(s))\bar{f}_{i}(\overline{s})\equiv\text{constant},\
\hbox{over}\ U.$

This implies that $f_{i}(s)\equiv 0$ for all $1\leq i\leq n$ and therefore
contradicts to the assumption that $F=(f_{1},\cdots,f_{n})$ is a non-constant
map. ∎

Choosing $s_{0},t_{0}$ as in Lemma 2.5,
$\sum_{i=1}^{n}\hat{f}_{i}(s_{0},X)\bar{f}_{i}(t_{0})$ is a nonconstant
holomorphic Nash algebraic function in $X$. It follows from Lemma 2.4 that

$e^{\frac{1}{\mu}\sum_{i=1}^{n}\hat{f}_{i}(s_{0},X)\bar{f}_{i}(t_{0})}=\frac{K((\hat{g}_{1}(s_{0},X),\cdots,\hat{h}_{d}(s_{0},X)),(\bar{g}_{1}(t_{0}),\cdots,\bar{h}_{d}(t_{0})))}{K((\hat{g}_{1}(s_{0},X),\cdots,\hat{h}_{d}(s_{0},X)),(0,\cdots,0,\bar{h}_{1}(0),\cdots,\bar{h}_{d}(0)))}.$

Note that this is precisely in the form of the equation (5). Hence we arrive
at a contradiction by Lemma 2.1. Thus $F$ must be a constant map. This is
again a contradiction and it completes the proof of Theorem 1.1.

## References

  * [1] Ahn, H.; Byun, J.; Park, J.-D. Automorphisms of the Hartogs type domains over classical symmetric domains, Internat. J. Math. 23 (2012), no. 9, 1250098, 11 pp.
  * [2] Calabi, E. Isometric imbedding of complex manifolds, Ann. of Math. (2) 58, (1953). 1-23.
  * [3] Cheng, X.; Di Scala, A. J.; Yuan, Y. Kähler submanifolds and the Umehara algebra, Internat. J. Math. 28 (2017), no. 4, 1750027, 13 pp.
  * [4] Cheng, X.; Hao, Y. On the non-existence of common submanifolds of Kähler manifolds and complex space forms, Ann. Global Anal. Geom. 60 (2021), no. 1, 167-180.
  * [5] Cheng, X.; Niu, Y. Submanifolds of Cartan-Hartogs domains and complex Euclidean spaces, J. Math. Anal. Appl. 452 (2017), no. 2, 1262-1268.
  * [6] Clozel, L. and Ullmo, E. Correspondances modulaires et mesures invariantes, J. Reine Angew. Math. 558 (2003), 47–83.
  * [7] Di Scala, A. and Loi, A. Kähler maps of Hermitian symmetric spaces into complex space forms, Geom. Dedicata 125 (2007), 103-113.
  * [8] Di Scala, A. and Loi, A. Kähler manifolds and their relatives, Ann. Sc. Norm. Super. Pisa Cl. Sci. (5) 9 (2010), no. 3, 495-501.
  * [9] Huang, X. and Yuan, Y. Holomorphic isometry from a Kähler manifold into a product of complex projective manifolds, Geom. Funct. Anal. 24 (2014), no. 3, 854-886.
  * [10] Huang, X., Yuan, Y. Submanifolds of Hermitian symmetric spaces, in: A. Baklut, A.E. Kacim, S. Kallel, N. Mir (Eds.), Analysis and Geometry, in: Springer Proc. Math. Stat., Vol 127, Springer, Cham, 2015, pp. 197-206.
  * [11] Ishi, H.; Park, J.-D.; Yamamori, A. Bergman kernel function for Hartogs domains over bounded homogeneous domains, J. Geom. Anal. 27 (2017), no. 2, 1703-1736.
  * [12] Loi, A., Mossa, R. Kähler immersions of Kähler-Ricci solitons into definite or indefinite complex space forms, Proc. Amer. Math. Soc. 149 (2021), no. 11, 4931-4941.
  * [13] Mok, N. Geometry of holomorphic isometries and related maps between bounded domains, Geometry and analysis. No. 2, 225-270, Adv. Lect. Math. (ALM) 18, Int. Press, Somerville, MA, 2011.
  * [14] Mok, N. Some recent results on holomorphic isometries of the complex unit ball into bounded symmetric domains and related problems, Geometric complex analysis, 269-290, Springer Proc. Math. Stat., 246, Springer, Singapore, 2018.
  * [15] Su, G.; Tang, Y.; Tu, Z. Kähler submanifolds of the symmetrized polydisk, C. R. Math. Acad. Sci. Paris 356 (2018), no. 4, 387-394.
  * [16] Umehara, M. Kaehler submanifolds of complex space forms, Tokyo J. Math. 10 (1987), no. 1, 203-214.
  * [17] Umehara, M. Diastasis and real analytic functions on complex manifolds, J. Math. Soc. Japan. 40 (1988), no. 3, 519-539.
  * [18] Wang, A.; Yin, W.; Zhang, L.; Roos, G. The Kähler-Einstein metric for some Hartogs domains over symmetric domains, Sci. China Ser. A 49 (2006), no. 9, 1175-1210.
  * [19] Yin, W.; Lu, K.; Roos, G. New classes of domains with explicit Bergman kernel, Sci. China Ser. A 47 (2004), no. 3, 352-371.
  * [20] Yuan, Y. Local holomorphic isometries, old and new results, Proceedings of the Seventh International Congress of Chinese Mathematicians. Vol. II, 409-419, Adv. Lect. Math. (ALM), 44, Int. Press, Somerville, MA, 2019. Xu Zhang (xzhang.2018math@yahoo.com); Donghai Ji (jidonghai@126.com);
Department of Mathematics, Harbin University of Science and Technology,
Harbin, Heilongjiang 150080, China.


#  Genixer: Empowering Multimodal Large Language Models as a Powerful Data
Generator

Henry Hengyuan Zhao1∗ Pan Zhou2† Mike Zheng Shou1†
1Show Lab, National University of Singapore, 2Sea AI Lab

###### Abstract

Large Language Models (LLMs) excel in understanding human instructions,
driving the development of Multimodal LLMs (MLLMs) with instruction tuning.
However, acquiring high-quality multimodal instruction tuning data poses a
significant challenge. Previous approaches relying on GPT-4 for data
generation prove expensive and exhibit unsatisfactory performance for certain
tasks. To solve this, we present Genixer, an innovative data generation
pipeline producing high-quality multimodal instruction tuning data for various
tasks. Genixer collects datasets for ten prevalent multimodal tasks, and
designs instruction templates to transform these datasets into instruction
tuning data. It then trains pretrained MLLMs to generate task-specific
instruction data and proposes an effective data filtering strategy to ensure
high quality. To evaluate Genixer, a base MLLM model Kakapo is built and
achieves SoTA performance in image captioning and visual question answering
(VQA) tasks across multiple datasets. Experimental results show that filtered
data from Genixer continually improves Kakapo for image captioning and VQA
tasks. For the SoTA Shikra MLLM model on the image-region-related tasks, e.g.,
region caption and detection, Genixer also successfully generates
corresponding data and improves its performance. Genixer opens avenues for
generating high-quality multimodal instruction data for diverse tasks,
enabling innovative applications across domains. The code and models will be
released soon.

Figure 1: Compared with GPT-4V [1], our proposed data generator Genixer is
capable of generating question-answer pairs with high accuracy and relevance
for REC and REG tasks, whereas GPT-4V fails to accurately identify and provide
the correct cropped regions. ††footnotetext: †Corresponding author, ∗ Work
done during a partial internship at Sea AI Lab. Figure 2:  Comparison of
prompting the current MLLMs to generate a conversation for the given image.
LLaVA [2], mPLUG-owl [3], and Shikra [4] treat the conversation generation
task as a dense captioning task while InstructBLIP [5] directly fails. Our
GenixerI finetunes on InstructBLIP is capable of generating the question and
answer pairs that are highly relevant to the content of the image,
demonstrating the effectiveness of our proposed pipeline.  Table 1: Image
sources comparison between common Vision-Language (VL) Datasets and the
datasets used in this work. These common VL datasets often display a lack of
variety in their image content.

Dataset | COCO Caption [6] | VQAv2 [7] | OKVQA [8] | A-OKVQA [9] | LLaVA-Ins-150K [10] | RefCOCO [11] |  $1/4$ Visual Genome [12] | Ours
---|---|---|---|---|---|---|---|---
Image Source | COCO [13] | COCO [13] | COCO [13] | COCO [13] | COCO [13] | COCO [13] | COCO [13] |  LAION [14], CC [15], SBU [16]

##  1 Introduction

Large Language Models (LLMs) [17, 18, 19, 20] have achieved remarkable success
in understanding human instructions and demonstrated their high proficiency in
tackling complex natural language tasks. This progress has recently extended
to Multimodal Large Language Models (MLLMs) [21, 22, 5, 10, 23, 20, 24, 4, 3,
25, 26, 27], where visual information is encoded by a dedicated visual encoder
and seamlessly integrated into LLMs. This integration empowers MLLMs with the
ability to analyze and reason about multimodal tasks effectively.

The training of a satisfactory MLLM relies heavily on the availability of
sufficient high-quality visual instruction tuning data. This data is, in many
ways, the lifeblood of contemporary general-purpose intelligence systems, much
like oil powers industrial processes. To attain superior performance, one
pioneer model like InstructBLIP [5] requires training on multiple held-in
datasets, such as VQAv2 [7] and OKVQA [8], from traditional Vision-Language
(VL) tasks. Unfortunately, these held-in datasets suffer from a limitation in
image diversity yet as illustrated in Tab. 1, as they all originate from the
COCO dataset [13], potentially restricting the model generalization ability on
the unseen visual data. This leads us to the fundamental question: how can we
obtain a sufficient quantity of high-quality multimodal instruction data for
training MLLMs?

As of now, there is no effective solution to this challenge yet. The
traditional approach of manual labeling proves prohibitively expensive and
impractical, particularly since MLLM training often necessitates billion-scale
multimodal instruction tuning data to ensure robust generalization. Another
potential solution is to leverage powerful models to generate high-quality
data. Unfortunately, almost all open-sourced pretrained MLLMs fall short in
this regard. As exemplified in Fig. 2, models like LLaVA-1.5 [2], mPLUG-owl
[3], and Shikra [4] tend to treat conversation generation as a dense
captioning task and struggle to produce natural and coherent dialog data; and
IntructBLIP [5] directly fails. The API of GPT-4V [1] provides one viable
option for generating multimodal instruction tuning data, but it is
accompanied by high generation costs. For instance, producing a million of
samples, each with thousands tokens, with GPT-4V [1] would cost millions of
dollars. Moreover, as illustrated in Fig. 1, GPT-4V cannot generate satisfied
multimodal instruction tuning data of certain types (e.g., referring
expression comprehension), further limiting its utility for specific tasks.

In this work, we address the challenge of generating highly qualified
multimodal data. To this end, we introduce Genixer, a novel pipeline that
leverages the capabilities of MLLMs to generate multimodal instruction data.
We begin by categorizing current Vision-Language (VL) tasks into ten distinct
types, including Common Visual Question Answering (VQA), Adversarial-Based
(Adv) VQA, Reading Comprehension (RC) VQA, Chain-of-Thought (CoT) VQA, Multi-
turn Dialogue (MD), and five Image-Region based tasks. See details in Sec.
3.1. Then, we compile datasets that correspond to each of these ten tasks and
consistently reformat the input data to align with instruction-following
requirements for subsequent training. Interestingly, through well-designed
training on specifically crafted instructional data, we discover that existing
MLLMs like Shikra [4] have significant potential to serve as data generators.
For instance, as demonstrated in Fig. 2, GenixerI finetunes on InstructBLIP
performs well in generating diverse question and answer pairs based on the
visual content.

Figure 3:  Overview of Genixer. Before empowering the current MLLMs, we first
collect the instruction data and then design the instruction template.
Finally, we conduct data generation and filtering.

Moreover, recognizing the inaccessibility of most advanced MLLMs for
reproducing the training phase and the need to thrive in multimodal domains,
we introduce a new MLLM foundational model called Kakapo, which servers as a
base model to test our generated data by Genixer. This model achieves state-
of-the-art (SOTA) performance on several representative tasks, including GQA
[28], OKVQA [8], and Flickr30K Caption [29]. We propose a modularized training
paradigm to separately train different components, reducing computation costs
and enhancing instruction understanding. Moreover, compared with previous SoTA
models like InstructBLIP [5], our Kakapo achieves strong performance on zero-
shot evaluation using fewer training data (see detail comparison in Sec.
5.4.1).

By employing Genixer to generate additional VQA-like tuning data and train our
Kakapo, we observe a significant performance enhancement compared to the
vanilla Kakapo across a diverse set of tasks, such as GQA [28] and Flickr30K
Caption [29]. Furthermore, our data generator is versatile and extends its
utility to generate instruction-tuning data for visual grounding tasks, akin
to those encountered in the SoTA grounding model, Shikra. These results
underscore the remarkable efficacy of our proposed data generation pipeline,
Genixer. In summary, our contributions are:

  * •

We introduce an innovative multimodal instruction data generation pipeline,
Genixer, that enables existing MLLM models to function as potent data
generators. Genixer shows the capabilities in generating diverse multimodal
instruction data, e.g., CoT VQA data, multi-turn dialogues, and adversarial
and region-related data.

  * •

We propose a novel foundational model Kakapo, with empirical results
showcasing its effectiveness, surpassing other SoTA MLLMs across several
multimodal tasks.

Table 2: Data mixture of training Genixer. We categorize the VL tasks into two
types: 5 nonregion-based tasks and 5 image-region based tasks. Counting110K†
is manually collected from PointQA [30] subset. POPE‡ is generated via the
pipeline from POPE [31].

| Tasks | Datasets | Size | Description
---|---|---|---|---
Nonregion tasks | Common VQA |  VQAv2, GQA, Counting110K† | 420K |  It asks the model to generate question-answer pairs related to object existence, counting, color, relative positioning, scene comprehension, and activity descriptions.
Adv VQA |  POPE‡ | 30K |  POPE [31], have shown the issue of object hallucination that models tend to produce descriptions of objects that do not align with.
RC VQA |  OCRVQA, TextVQA | 80K |  Reading Comprehension VQA refers to understanding the text information within an image and answering.
CoT VQA | GQA | 90K | It asks the model to generate VQA pairs that involve reasoning steps.
MD | LLaVA-150K | 150K | It asks the model to produce multi-turn dialogues for building a multimodal chatbot.
Image-region tasks | REC | VG, RefCOCO | 1M |  The task aims to localize a target object in an image described by a referring expression.
REG |  VG, RefCOCO | 1M | The purpose of REG is to generate a unique description for a specified location.
PointQA |  PointQA Local, Visual7W | 218K |  Ask models to answer questions about the region specified by the user, either by center point or box.
Q$\rightarrow$CBoxA |  Shikra (GPT-4 Generated) | 4K | It asks models to generate grounded explanations for the answer.
RD |  Shikra (GPT-4 Generated) | 1.8K | Models need to answer the question from the user by referring to specific object locations.

##  2 Related Work

###  2.1 Multimodal Large Language Models

Large Language Models (LLMs) showcased the remarkable complicated reasoning
abilities. The well-known open-resouced LLMs include FlanT5 [32], OPT [33],
LLaMA [34], Vicuna [35] and LLaMA-2 [36]. By leveraging these LLMs, a surge of
multimodal modes [37, 10, 5, 27, 4, 38, 26, 39, 40, 41, 42, 21, 22, 43, 3, 20]
are proposed to integrate the visual information for diverse multimodal
reasoning tasks such as image captioning [6, 44, 29] and visual question
answering [7, 8, 28, 45]. LLaVA [10] is a pioneering approach that adopts a
single linear layer to project the CLIP [46] encoder extracted visual features
to the input of LLM. Different from LLaVA, InstructBLIP [5] employs an
instruction-aware feature extractor and obtains advanced performance on
various tasks building upon the pretrained BLIP2 [37]. Besides focusing on
traditional multimodal tasks, Shikra [4] and PVIT [40] pay attention to the
image-region based multimodal tasks by carefully designing the visual
instruction data or employing the region-based vision encoder.

In this work, we introduce a new MLLM foundational model called Kakapo to
thriving the development of MLLMs and provide a backbone model to evaluate the
effectiveness of data generation pipeline Genixer.

###  2.2 Multimodal Instruction Data

High-quality multimodal instruction data is crucial for training a robust
MLLM. Two main setups are as follows: 1) Transforming the current vision-
language datasets into the instruction-tuning format, e.g., InstructBLIP. Such
choice is limited by the diversity of image sources, as in Tab. 1. 2) Some
approaches LLaVA, VisionLLM[42], and Shikra, resort to prompting the GPT-4
[18] language model to generate corresponding instruction data. This way
requires the image datasets to have enough captions or region-based
annotations (e.g., bounding boxes), which heavily restricts the data scale.
Additionally, prompting commercial LLMs incurs high costs, and even GPT-4V [1]
may not address the data generation effectively on some specific domains, as
illustrated in Fig. 1. To this end, we introduce Genixer, an innovative
pipeline that explores the capabilities of MLLMs to generate high-quality
multimodal instruction data.

##  3 Genixer

We propose Genixer, which is a novel pipeline to automatically generate high-
quality multimodal instruction data for advancing the MLLM domain. The model
trained in the pipeline is referred to as GenixerI and GenixerS, indicating
that the model is fine-tuned from InstructBLIP and Shikra, respectively. As
illustrated in Fig. 3, Genixer contains four key steps, including 1)
instruction data collection, 2) instruction template design, 3) empowering
current MLLMs, and 4) data generation and filtering. In the following, we will
elaborate on these four key steps.

###  3.1 Instruction Data Collection

In accordance with the prevalence and practical relevance of real-world multi-
modal tasks, we have carefully selected ten representative multimodal tasks as
listed in Tab. 2 for corresponding data generation.

###  3.2 Instruction Template Design

After assembling datasets for various multimodal tasks, we begin to transform
these datasets into an instruction-tuning format for subsequent training of
Genixer.

Specifically, we design the instruction templates as follows:

<s> SYSTEM MESSAGE. USER: <image> General Instruction. Specific Instruction.
ASSISTANT: Question: <question> Answer: <answer> </s>

The tags <image>, <question>, and <answer> serve as placeholders for inserting
the embeddings of the visual image, the text question, and the text answer,
respectively. Regarding General Instruction, it is designed to have over 150
choices and is randomly selected from an instruction pool Tab. 13. Such
instruction aims to let the generator be able to generate QA pairs.

Then, for Specific Instruction, it denotes designating the specific question
type, such as “This is a Common VQA task.”, allowing us to control the output
type of the generated data. Refer to the example generated by GenixerI
illustrated in Fig. 4. It demonstrates that when provided with various
specific instructions for the same image, the resulting QA pairs generated are
distinct. Throughout the training phase, we will include the specific
instruction with a probability constant denoted as $\tau$. This constant
provides the model with the flexibility to generate appropriate QA types based
on its own judgment while enabling it to follow the user’s specific
instructions simultaneously.

During training, given the system and user information, the model is trained
to predict “Question: <question> Answer: <answer>” in an autoregressive manner
[10, 5].

Figure 4:  Three examples from GenixerI show that different specific
instructions for the same image lead to distinct generated results. Text,
text, and text denote the general instruction, specific instruction, and
generated question-answer pairs, respectively.

###  3.3 Empowering Current MLLMs

Here we introduce how to use the above instruction data to train our Genixer
for multimodal data generation. As summarized in Tab. 2, real-world multi-
modal tasks can be roughly divided into two categories due to their quite
different requirements for MLLMs. So we train two data generators, GenixerI
and GenixerS, to respectively generate multimodal data for the two task
categories, namely, nonregion-based tasks and image-region based tasks.

Training of GenixerI. GenixerI first trains the pretrained InstructBLIP [5], a
SoTA model for nonregion-based tasks, on the VQA and Adv VQA datasets to
ensure the ability for generating standard QA pairs for an image. Then
GenixerI is further trained on RC VQA, CoT VQA, and MD datasets to generate
more complex QA pairs. In all of the datasets we only sample a subset for
training to achieve training efficiency. Detailed data size is in Tab. 2.

Training of GenixerS. Training a data generator to produce region-based QA
pairs is intricate because of the high precision requirements of referred
coordinates. To achieve high-quality generation, we utilize the entirety of
the RefCOCO [11] and VG [12] datasets. Nonetheless, as indicated in Tab. 2,
the dataset size for RD is relatively small. To counteract potential biases in
the generation, GenixerS that finetunes a SoTA model Shikra [4] on region-
based tasks adopts two-phased training. The first phase focuses on the REC and
REG data generation. In the second phase, GenixerS adds PointQA,
Q$\rightarrow$CBoxA, and RD data while deliberately reducing REC and REG data
for data trade-off.

Our pipeline Genixer is flexible and indeed can use any MLLM for data
generation without modifying their network architectures. Here we select
InstructBLIP and Shikra because of their SoTA performance in the corresponding
area.

###  3.4 Automatic Data Generation and Filtering

To generate new multimodal instruction data, we choose the images from
Flickr30K [29], LLaVA-LCS [10], and SBU [16] as the image corpus for both
nonregion-based and image-region based data generation.

To ensure the high quality of region-based generated data, data filtering is
necessary. Specifically, we compute the CLIP-similarity score [46] between the
text expression and the corresponding image region. Given an example of REC
task, the generated QA pair is {Question: “I need the coordinates of the
person at the bottom left of the image. Can you assist?” Answer:
[0.005,0.332,0.249,0.984]}. Then we use CLIP to compute the similarity of the
expression “person at the bottom left of the image” and the image region whose
coordinate is [0.005,0.332,0.249,0.984]. For all generated data, we filter out
low-quality samples whose CLIP scores are lower than 0.5.

##  4 Kakapo

To assess the quality of the multimodal instructional data generated by the
above two Genixer models, we employ these data to train a multimodal large
language model (MLLM). However, reproducing the training phase of most
advanced MLLMs is challenging due to unavailability111For some advanced MLLMs,
e.g., InstructBLIP, they only release their models but do not release their
training codes. and some in-house training data. To address this, we introduce
a novel foundational MLLM model, termed Kakapo, designed for data efficiency
and possessing robust instructional comprehension through a modularized
training paradigm. This foundational MLLM serves not only as the base model
for evaluating the efficacy of our data generation pipeline Genixer but also
contributes to the advancement of the multimodal domain by offering a new and
publicly released potent MLLM222We will make our Kakapo available online to
facilitate further research and progress in MLLM.. In the subsequent sections,
we delve into the model architecture of Kakapo in Sec. 4.1 and its training
methodology in Sec. 4.2.

###  4.1 Model Architecture Overview

As illustrated in Fig. 5, our proposed Kakapo comprises three key components:
a vision encoder, a text-guided adapter, and a large language model. The
vision encoder processes visual input $X_{I}$ to align the learned visual
features with the text input $X_{T}$. Next, the text-guided adapter fuses the
visual feature $F_{I}$ and text input $X_{T}$ based on the Learnable Query
$Q$. Finally, the language model utilizes multimodal query tokens $Z_{Q}$ and
the current text embedding $Z_{T}$ to perform specific tasks.

For the vision encoder, we follow previous methods [5] and implement it with
the pretrained EVA-CLIP vision encoder [47]. For the large language model, we
adopt the widely used instruction fine-tuned model, Vicuna-7B [35]. The text-
guided adapter utilizes the 12-layered pretrained BERT-Base model [48] with
the cross-attention module.

Figure 5:  Kakapo architecture. Kakapo comprises three primary components: a
vision encoder, a text-guided adapter, and a large language model.

###  4.2 Modularized Training of Kakapo

To train Kakapo, we propose a novel modularized training paradigm. The modules
in Kakapo are divided into two groups to respectively achieve modality-
alignment and visual instruction tuning. The first group includes the text-
guided adapter, text input embedding module, and output embedding module
presented in Fig. 5. These modules aim to align visual input $X_{I}$ and text
input $X_{T}$ for modality alignment when the vision encoder and LLM are
frozen. The second group consists of LLM, the text input and output embedding
module, focusing on achieving high-capacity visual instruction tuning. Then we
train these two groups of modules to achieve their target as introduced below.

Modality-Alignment Training. During this phase, we freeze the parameters of
the pretrained EVA-CLIP vision encoder and the Vicuna LLM. We only update the
parameters of the learnable query $Q$, text-guided adapter, and the output
embedding after the Vicuna LLM. The goal is to align visual and text inputs,
encouraging the text-guided adapter to understand the instruction in the input
text and fuse related visual and text embeddings. This process prepares the
subsequent LLM for analysis and reasoning based on the fused output feature.

Visual Instruction Tuning. In this phase, we freeze the visual encoder and
text-guided adapter, and train the Vicuna LLM, along with its corresponding
input and output embedding modules. This unleashes the instruction
understanding ability of LLM when facing complex instructions.

(a) Question length comparison.

(b) Distribution of Nouns.

(c) Distribution of Verbs.

Figure 6:  The statistics of 210K generated data from SBU [16] image corpus.
We compare the question length between our generated data and real data, i.e.,
VQAv2 [7]. The distribution of nouns and verbs demonstrates the diverse
vocabulary of our generator GenixerI.  Figure 7: Comparison between real QA
and GenixerI generated QA for the same image. This comparison reveals that
human annotators sometimes provide irrelevant questions, whereas GenixerI
consistently generates questions and answers that are strictly relevant to the
image content.

##  5 Experiments

Here, we evaluate the quality of the generated multimodal data by Genixer from
several aspects, including statistic evaluation, human evaluation, and
evaluation via training MLLMs. All experiments, unless specifically stated
otherwise, were conducted on an 8 Nvidia A100 (40G) GPU setup.

Training Details. The training of GenixerI, including five nonregion-based
tasks. We employ general instructions on the Common QA task and both general
and specific instructions on the other four tasks with $\tau$ set to 0.5,
spanning two 5-epoch phases at a learning rate of $3.5\times 10^{-5}$ and
$2\times 10^{-5}$. The batch size is 256. The training takes 47 hours. To
train the GenixerS, we set the $\tau$ equal to 1 for each task, which occurs
in two 5-epoch phases at learning rates of $3\times 10^{-5}$ and $1\times
10^{-5}$ respectively, with a batch size of 128.

###  5.1 Statistics Evaluation

We sample 210K images from the SBU dataset [16] for the QA pairs generation
with the general instruction to evaluate the statistics. As shown in Fig. 6
(a), we compare the question length between our GenixerI data and real data
from the VQAv2 [7] training set. The question length generated from our
GenixerI has a larger proportion in the longer sentences. The nouns and verbs
distribution illustrated in Fig. 6 (b) and (c) illustrate the diverse
vocabulary of generated questions. More Statistics can be found in Fig. 11,
Fig. 14, and Fig. 16.

The statistics of the data generated by GenixerS can be found in Fig. 12, Fig.
15 and Fig. 17. We analyze the region width and height of generated REC data
in Fig. 13. In Tab. 7, we compare the number of images, objects, and the
average length of expression with other grounding-based datasets.

###  5.2 Human Evaluation

We conduct the human evaluation to analyze the generated question type and
corresponding correctness manually. We employ GenixerI to generate QA samples.
Since the image content will influence the generation types, we randomly chose
200 images from the COCO validation set as the held-in set and chose 200
images from Flickr30K as the held-out set. As shown in Tab. 3, we divide the
questions into seven types, such as “Action” and “Color”. “Action” and “Object
Type” are two most popular question types of both held-in and held-out
datasets. Among held-in samples, “Yes/No” question also has a notable
proportion with high correctness. Different from the held-in set, there are 46
out of 200 samples belong to “Relative Position” in the held-out set. Since
the quality of the generated data of GenixerI is adequate, we just slightly
control the question types when generating synthetic data for subsequent model
training.

Table 3: The human evaluation of our Genixer on a random sample of 200
examples of held-in (COCO Val) and held-out (Flickr30K) datasets.

Question Type | Held-in (COCO val) | Held-out (Flickr30K)
---|---|---
#Samples |  Correct ($\sim$%) | #Samples |  Correct($\sim$%)
Action | 31 | 89 | 60 | 93
Color | 20 | 70 | 2 | 50
Counting | 20 | 90 | 16 | 81
Object Type | 54 | 79 | 46 | 61
Relative Position | 18 | 83 | 46 | 57
Yes/No | 44 | 82 | 17 | 88
Others | 19 | 84 | 13 | 77

###  5.3 Qualitative Evaluation

Besides the statistic analysis and human evaluation, we further qualitatively
evaluate GenixerI in generating diverse QA pairs with various images and
instruction guidance as in Fig. 8. It is noted that GenixerI is capable of
generating various QA pairs for diverse multimodal tasks. Fig. 7 indicates
that that human annotators sometimes provide irrelevant questions, whereas
GenixerI consistently generates questions and answers that are strictly
relevant to the image content. As shown in Fig. 1, 9, and 10 by enhancing
Shikra, our GenixerS is adept at creating complex question-and-answer (QA)
pairs that require precise object localization and referring expression
capabilities.

Table 4: Zero-shot evaluations on Image Captioning and VQA. $\ddagger$
represents the model trained with our generated data.

Model | Language Model | Image Caption | General VQA
---|---|---|---
| Nocaps
---
val
| Flickr30K
---
karpathy-test
| OKVQA
---
val
| GQA
---
test
| VizWiz
---
test-dev
Flamingo-9B [49] | LLaMA-7B | - | 61.5 | 44.7 | - | 28.8
Kosmos-1 [21] | - | - | 67.1 | - | - | 29.2
Kosmos-2 [22] | - | - | 80.5 | - | - | -
AnyMAL [43] | LLaMA2-13B | - | - | 33.1 | - | 24.4
Shikra [4] | Vicuna-13B | - | 73.9 | 47.2 | - | -
BLIP-2 [37] | Vicuna-7B | 107.5 | 74.9 | - | 41.3 | 25.3
BLIP-2 [37] | Vicuna-13B | 103.9 | 71.6 | 45.9 | 32.3 | 19.6
InstructBLIP [5] | Vicuna-7B | 123.1 | 82.4 | - | 49.2 | 34.5
InstructBLIP [5] | Vicuna-13B | 121.9 | 82.8 | - | 49.5 | 33.4
Kakapo (ours) | Vicuna-7B | 117.3 | 83.1 | 52.0 | 50.6 | 34.2
Kakapo‡ (ours) | Vicuna-7B | 120.5 | 84.2 | 52.6 | 52.3 | 35.2

Table 5: Object hallucination evaluation. Acc. denotes accuracy.

Model | Popular | Adversarial | Random
---|---|---|---
mPLUG-Owl [3] | 50.90 | 50.67 | 53.97
LLaVA [10] | 49.87 | 49.70 | 50.37
MiniGPT-4 [26] | 69.73 | 65.17 | 79.67
InstructBLIP [5] | 82.77 | 72.10 | 88.57
Shikra [4] | 83.97 | 83.10 | 86.90
Kakapo (ours) | 84.03 | 82.13 | 85.26
Kakapo‡ (ours) | 84.70$+0.67$ | 83.36$+1.23$ | 86.22$+0.96$

Table 6: Results on Referring Expression Comprehension task. ${\ddagger}$
represents the model implemented by ourselves trained with adding synthetic
data.

Model | RefCOCO | RefCOCO+ | RefCOCOg | Avg
---|---|---|---|---
val | test-A | test-B | val | test-A | test-B | val | test
GPV-2 [50] | - | - | - | - | - | - | - | - | -
OFA-L* [51] | 79.96 | 83.67 | 76.39 | 68.29 | 76.00 | 61.75 | 67.57 | 67.58 | 72.65
Unified-IO [52] | - | - | - | - | - | - | - | - | -
VisionLLM-H [42] |  | 86.70 | - | - | - | - | - | - | -
Shikra-7B [4] | 87.01 | 90.61 | 80.24 | 81.60 | 87.36 | 72.12 | 82.27 | 82.19 | 82.92
Shikra-7B‡ | 87.48 | 91.09 | 81.51 | 81.73 | 87.02 | 73.47 | 81.78 | 82.89 | 83.37

###  5.4 Evaluation via Training MLLMs

####  5.4.1 Evaluation on Nonregion-based Tasks

Here we describe the experimental setting of training vanilla Kakapo,
including datasets, training settings, and additional synthetic data
generation.

Datasets. Following earlier studies [10, 4, 5, 2], we begin by adopting LLaVA-
CC3M [10], LLaVA-LCS [10] and PTP-4M [53] datasets for image captioning tasks,
and for VQA tasks, we adopt only VQAv2 [7]. It should be noted that our Kakapo
employs 1.9M samples from these captioning datasets in total, in contrast to
InstructBLIP’s use of 14M samples, which underscores the efficiency of our
approach. Additionally, we integrate region-based datasets RefCOCO [11] and VG
[12]. Finally, we adopt the LLaVA-Instruct-150K [10] for dialogue ability. We
use these datasets in both training phases introduced in Sec. 4.2.

Training settings. We utilize AdamW [54] for optimization and a cosine
annealing learning rate scheduler [55]. In the initial phase of the
modularized training, we set the learning rate to $3.5\times 10^{-5}$ with a
global batch size of 256. The second phase continues with a learning rate of
$2\times 10^{-5}$. The training of both phases is approximately two days.

Synthetic Data. To enhance the multimodal reasoning capabilities of our base
model, Kakapo, in nonregion-based VQA tasks, we employed the trained GenixerI
to create approximately 440K VQA-like datasets. This was achieved by selecting
440K images from the Flickr30K [29], LAION-400M [56], SBU [16], and CC3M [57]
datasets.

Zero-shot Evaluation. Tab. 4 presents the zero-shot performance of our model
on two captioning datasets, Nocaps [44], and Flickr30K [29], as well as four
general VQA datasets: OKVQA [8], GQA [28], VizWiz [45], and ScienceQA [58].
The results of comparing multimodal reasoning works are from the literature
that reports zero-shot results on these benchmarks. We emphasize zero-shot
performance to best evaluate the model’s capabilities in handling open-ended
queries during inference.

In conclusion, our vanilla Kakapo model, powered by the Vicuna-7B LLM [35],
outperforms the previous SOTAs on the Flickr30K, OKVQA, and GQA datasets by
margins of 0.3%, 4.8%, and 1.1%, respectively. It has also yielded 35.2%
results on the Vizwiz dataset. Furthermore, the integration of 440K VQA-like
synthetic data has allowed our enhanced model, Kakapo‡, to further improve
performance across Nocaps, Flickr30K, OKVQA, and GQA by 3.2%, 1.1%, 0.6%, and
1.7%, respectively, when compared to the vanilla Kakapo model. This
improvement, particularly in the GQA dataset, can be attributed to the
GenixerI, which was trained on a subset of the VQAv2 and GQA training data and
generated data mirroring the question types found in these datasets. Notably,
this has led to an 11% and 2.8% improvement over the previous SOTAs, BLIP-2
(Vicuna-7B) and InstructBLIP (Vicuna-7B), on the GQA dataset.

Figure 8:  GenixerI generated examples of Common VQA, Adversarial-based VQA,
Chain-of-thought VQA, Reading Comprehension VQA and Multi-turn Dialogue.
Figure 9:  GenixerS generated examples of Referring Expression Comprehension
(REC), Referring Expression Generation (REG) and Q$\rightarrow$CboxA. REC
focuses on localizing the expressions provided in questions. REG entails
providing expressions for a specified region. Q$\rightarrow$CboxA is dedicated
to generating grounded explanations for answers.  Figure 10:  GenixerS
generated examples of Referring Dialogue (RD) and PointQA. RD is about
referring to the object regions when questioning and answering. PointQA
involves using either a bounding box or a point to respond to questions, which
may be presented in a multiple-choice format, as illustrated in the first
example.

Object Hallucination. Recent work, as highlighted by POPE [31], has brought
attention to the problem of object hallucination in current MLLMs. To assess
our model Kakapo${\ddagger}$ resistance to hallucination, we conducted
evaluations on the three hallucination subsets introduced by POPE. Our
Kakapo${\ddagger}$ outperforms other MLLMs, achieving the highest accuracy on
Popular and Adversarial sets. When compared to the baseline model, the
enhanced version Kakapo${\ddagger}$ also shows a consistent improvement in
performance. These results suggest that our model effectively comprehends
misleading instructions and does not exhibit a bias towards responding with
“Yes”, a tendency noted in other models such as LLaVA.

####  5.4.2 Evaluation on Image-Region Based Tasks

Furthermore, to assess the quality of data produced by GenixerS, we select
Shikra [4] as the backbone model, and process a corpus of 1.4M images (i.e.,
558K from LLaVA-LCS [10], 400K from SBU [16], and 441K from OpenImages [59]).
We filter out 845K high-quality REC-like data using the CLIP-similarity Score
metric detailed in Sec. 3.4. Adhering Shikra’s published code, we incorporated
our synthetic data into the training phases, maintaining consistent training
iterations to ensure fair comparison.

Table 7: Comparison GenixerS generated 845K REC-like data with existing visual
grounding datasets.

Dataset | Images | Objects | Avg Length
---|---|---|---
Flickr Entities [60] | 31,783 | 275,775 | -
RefCOCOg [61] | 26,711 | 54,822 | 8.43
RefCOCO [11] | 19,994 | 50,000 | 3.61
RefCOCO+ [11] | 19,992 | 49,856 | 3.53
Visual Genome [12] | 108,077 | 4,102,818 | -
GenixerS(845K) | 845,110 | 1,007,910 | 6.30

Referring Expression Comprehension. We incorporated this synthetic data into
the training phases of Shikra [4]. The results, presented in Tab. 6, indicate
that with 830K synthetic data, our improved model, Shikra-7B‡, achiing the
best performance on 6 out of 8 datasets, obtaining an average high performance
of 83.37%. These findings imply that our pipeline can be an alternative
approach to generate image-region based multimodal instruction data, which is
typically challenging for manually labeling and expensive and not satisfied
for prompting GPT-4V as shown in Fig. 1.

##  6 Conclusion

This paper introduces a novel approach, named Genixer, designed to generate
high-quality instruction-tuning data, thereby enhancing the capabilities of
current MLLMs. We have systematically gathered 10 prominent vision-language
tasks, encompassing both non-region-based and image-region-based aspects, to
train the existing MLLM in generating these multimodal instructional data.
Relying on our bespoke general and specific instructional designs, we can
control the diversity of the generated data. The experimental results
demonstrate that the data generated using this approach significantly improves
the performance of our base model Kakapo and SOTA Shikra.

## References

  * [1] OpenAI.  Gpt-4v(ision) system card, 2023.
  * [2] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning, 2023.
  * [3] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.  mplug-owl: Modularization empowers large language models with multimodality.  arXiv:2304.14178, 2023.
  * [4] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.  Shikra: Unleashing multimodal llm’s referential dialogue magic.  arXiv:2306.15195, 2023.
  * [5] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  2023\.
  * [6] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv:1504.00325, 2015.
  * [7] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In CVPR, 2017.
  * [8] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In CVPR, 2019.
  * [9] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  arXiv, 2022.
  * [10] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In NeurIPS, 2023.
  * [11] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.  Referitgame: Referring to objects in photographs of natural scenes.  In EMNLP, 2014.
  * [12] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  IJCV, 2017.
  * [13] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [14] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  arXiv:2210.08402, 2022.
  * [15] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In CVPR, 2021.
  * [16] Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2text: Describing images using 1 million captioned photographs.  NeurIPS, 2011.
  * [17] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  NeurIPS, 2020.
  * [18] OpenAI.  Gpt-4 technical report, 2023.
  * [19] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  arXiv:2305.10403, 2023.
  * [20] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al.  Llama-adapter v2: Parameter-efficient visual instruction model.  arXiv:2304.15010, 2023.
  * [21] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al.  Language is not all you need: Aligning perception with language models.  arXiv:2302.14045, 2023.
  * [22] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.  Kosmos-2: Grounding multimodal large language models to the world.  arXiv:2306.14824, 2023.
  * [23] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  arXiv:2303.16199, 2023.
  * [24] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  arXiv:2305.03726, 2023.
  * [25] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.  Pandagpt: One model to instruction-follow them all.  arXiv:2305.16355, 2023.
  * [26] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv:2304.10592, 2023.
  * [27] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  arXiv preprint arXiv:2308.12966, 2023.
  * [28] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In CVPR, 2019.
  * [29] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  ACL, 2014.
  * [30] Arjun Mani, Nobline Yoo, Will Hinthorn, and Olga Russakovsky.  Point and ask: Incorporating pointing into visual question answering.  arXiv preprint arXiv:2011.13681, 2020.
  * [31] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  arXiv:2305.10355, 2023.
  * [32] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  arXiv preprint arXiv:2210.11416, 2022.
  * [33] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv:2205.01068, 2022.
  * [34] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv:2302.13971, 2023.
  * [35] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [37] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv:2301.12597, 2023.
  * [38] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang.  Cogvlm: Visual expert for pretrained language models.  2023\.
  * [39] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny.  Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.  arXiv preprint arXiv:2310.09478, 2023.
  * [40] Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu.  Position-enhanced visual instruction tuning for multimodal large language models.  arXiv preprint arXiv:2308.13437, 2023.
  * [41] Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji.  Cheap and quick: Efficient vision-language instruction tuning for large language models.  2023\.
  * [42] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al.  Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.  arXiv preprint arXiv:2305.11175, 2023.
  * [43] Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, et al.  Anymal: An efficient and scalable any-modality augmented language model.  arXiv preprint arXiv:2309.16058, 2023.
  * [44] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson.  nocaps: novel object captioning at scale.  In ICCV, 2019.
  * [45] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham.  Vizwiz grand challenge: Answering visual questions from blind people.  In CVPR, 2018.
  * [46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [47] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  arXiv preprint arXiv:2303.15389, 2023.
  * [48] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv:1810.04805, 2018.
  * [49] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  NeurIPS, 2022.
  * [50] Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi.  Webly supervised concept expansion for general purpose vision models.  In European Conference on Computer Vision, pages 662–681. Springer, 2022.
  * [51] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In ICML, 2022.
  * [52] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  arXiv:2206.08916, 2022.
  * [53] Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan.  Position-guided text prompt for vision-language pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23242–23251, 2023.
  * [54] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [55] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [56] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [57] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [58] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  NeurIPS, 2022.
  * [59] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  IJCV, 2020.
  * [60] Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, J. Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  International Journal of Computer Vision, 123:74–93, 2015.
  * [61] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.  Generation and comprehension of unambiguous object descriptions.  In CVPR, 2016.
  * [62] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu.  Mimic-it: Multi-modal in-context instruction tuning.  2023\.
  * [63] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In ICDAR, 2019.

Supplementary Material

##  7 Comaparison with GPT-4(V)

We illustrate two type examples to compare the generation ability between our
Genixer and GPT-4(V).

GPT-4. Previous approaches [10, 4, 62] utilized the GPT-4 language model to
create multimodal instructional data supplemented by text annotations. The
main limitations of using GPT-4 for this purpose are twofold: firstly, it
requires additional image captions to provide context, and secondly, it is
incapable of generating pertinent question-answer pairs based on the spatial
positioning of objects within images. In contrast, our model, referred to as
GenixerI, demonstrates a comprehensive understanding of the instructions,
enabling it to produce accurate and concise question-answer pairs. Unlike
GPT-4, which relies on text annotations and often merely reformulates
contextual information into a single lengthy sentence, our model exhibits an
acute awareness of spatial details. This contrast in performance is documented
in Tab. 10.

GPT-4V. We present the specifics of the prompts used to engage GPT-4V in
producing instances of Referring Expression Comprehension (REC) and Referring
Expression Generation (REG) question-answer pairs, as seen in Tab. 11 for REC
and Tab. 12 for REG. The outcomes indicate that GPT-4V falls short in
accurately generating REC and REG data. Conversely, our GenixerS has the
proficiency to create succinct and relevant question-answer pairs.

##  8 Data

The detailed dataset mixture are displayed in Tab. 8, we adopt three
captioning datasets, one common VQA dataset, two image-region base VQA
datasets and multi-turn dialogue dataset to train our Kakapo.

LLaVA-CC3M [10] and LLaVA-LCS [10] are two image captioning datasets proposed
by LLaVA [10]. We use the full size of these datasets.

PTP-4M [53] is an image captioning dataset used in vision-language pertaining
tasks. We randomly sample 720K samples for additional supplements for image
captioning ability.

Compared with InstructBLIP [5] that adopt 14M web captioning data, we only
adopt 1.9M image captioning samples and achieve competitive results as shown
Tab. 3 in main paper. Such experiments demonstrate that our Kakapo is data-
efficient compared with InstructBLIP. Further, different previous methods [27]
adopt billion-level image captioning data, and our image captioning datasets
mixture is small and beneficial for reproducible and fast training time.

VQAv2 [7] is a widely used VQA dataset for multimodal model training. We only
adopt VQAv2 as the training data to keep the lower level of training data.
Different from InstructBLIP, which uses additional OKVQA [8] and A-OKVQA [9],
OCRVQA [63] as the training set, such a single dataset demonstrates the
effectiveness of our Kakapo on multiple VQA datasets such as GQA [28], OKVQA,
VizWiz [45] in Tab. 3 in main paper and further benefits us to adding
synthetic VQA-like data to evaluate the improvements.

Visual Genome and RefCOCO are two widely used grounding datasets for phrase
grounding. We randomly sample 90K and 280K, respectively for ensure the fine-
grained image-region related VQA and it will improve the models performance on
avoiding object hallucination problem [4].

LLaVA-Instruct-150K [10] is a GPT-4 generated multi-turn dialogue dataset base
on COCO images and annotations. We adopt it for multi-turn dialogue ability.

Table 8: The datasets used in training Kakapo.

Datasets | Size | Type
---|---|---
LLaVA-CC3M | 595K | Image Captioning
LLaVA-LCS | 585K | Image Captioning
PTP-4M | 720K | Image Captioning
VQAv2 | 443K | VQA
RefCOCO | 280K | Image-region based VQA
Visual Genome | 90K | Image-region based VQA
LLaVA-Instruct-150K | 150K | Multi-turn Dialogue

##  9 Instruction Templates

We also illustrate some examples of general instructions for training GenixerI
and GenixerS in Tab. 13.

##  10 Additional Experiments

Ablation Study of Modularized training

Table 9: Results of ablation studies of training stages. $\ddagger$ represents
the model trained with our generated data.

Model | Language Model |  | Nocaps
---
val
| Flickr30K
---
karpathy-test
| OKVQA
---
val
| GQA
---
test
| VizWiz
---
test-dev
Kakapo Stage1 | Vicuna-7B | 116.8 | 80.9 | 48.7 | 49.5 | 31.8
Kakapo Stage1+Stage2 | Vicuna-7B | 117.3 | 83.1 | 52.0 | 50.6 | 34.2
Kakapo Stage1+Stage2‡ | Vicuna-7B | 120.5 | 84.2 | 52.6 | 52.3 | 35.2

To evaluate the impact of the two-stage training proposed in Sec. 4.2 in the
main paper. As illustrated in Tab. 9, training base model Kakapo without the
second stage leads to a notable decrease in performance across all test
datasets. However, incorporating our generated VQA-like data into the second
stage of training enhances performance on all datasets, particularly on the
GQA dataset, which shares a similar VQA style with the training data used for
our generator GenixerI.

##  11 Statistic of Generated Data

In this section, we will illustrate the data statistics of two types of
generated data used in training Kakapo and Shikra in the main paper Tab. 3 and
Tab. 5, respectively.

440K VQA-like Data. The distribution of question lengths in our generated
dataset of 440K VQA-like data shows that the majority of questions contain six
words. Additionally, there is a significant presence of sentences exceeding
ten words.

Figure 11 presents the word distribution in generated questions. Additionally,
we depict the 30 most frequent nouns and verbs from a 440K VQA-like dataset in
Fig. 14, and the 15 most frequent nouns and verbs in Fig. 16.

Figure 11:  The distribution of words in a question of 440K VQA-like data.
Figure 12:  The distribution of the word length in a region expression.

Figure 13:  The distribution of the normalized region width and height of 845K
generated REC-like training data.

Figure 14:  The distribution of top 30 nouns and verbs in 440K VQA-like
training data.

845K REC-like Data As mentioned in Sec. 10, we expand the generated REC-like
dataset to 845K entries by integrating OpenImages into our image corpus. We
will now provide a statistical analysis focusing on the words of regions,
width, and height of described regions.

Figure 12 presents the word distribution in expressions. Figure 15 illustrates
the distribution of normalized widths and heights in the generated data,
revealing similar patterns for both dimensions, with the most common widths
and heights occurring around 0.2. Additionally, we depict the 30 most frequent
nouns and verbs from an 845K REC-like dataset in Fig. 15, and 15 most frequent
nouns and verbs in Fig. 17.

Figure 15:  The distribution of top 30 nouns and verbs in 845K REC-like
training data.

Figure 16:  The pie chart distribution of top 15 nouns and verbs in 440K VQA-
like training data.

Figure 17:  The pie chart distribution of top 15 nouns and verbs in 845K REC-
like training data.  Table 10: An example of asking GPT-4 to generate specific
question type question and answer.

Generating a question-answer pair of the given image:
---
Context |  A baseball batter up to plate about to hit the ball. A baseball player swinging a bat at a ball A baseball player swinging a bat at a ball A baseball player swinging a baseball bat at a baseball. A baseball game with a left handed batter swinging at a pitch and the catcher and umpire behind him.
Instruction |  You are an AI visual assistant, and you are seeing a single image. What you see are provided with five sentences, describing the same image you are looking at. The answers should be in a tone that a visual AI assistant is seeing the image and answering the question. Can you give me a straightforward question and answer? Include the question asking about the visual content of the image, including object locations, and relative positions between objects.
GPT-4 |  Question: Can you describe the position of the baseball player in relation to the ball and the other figures present in the image? Answer: The baseball player is centered in the foreground, with his body positioned to the left side of the image, indicating he is a left-handed batter. He is in mid-swing with the bat angled towards the ball, suggesting he is about to make contact. The ball is directly in front of him, slightly to the right, appearing close to the bat’s path. Behind the batter, the catcher is crouched in anticipation, and the umpire stands slightly offset to the right, both observing the ball’s trajectory.
Image |
Instruction |  Can you provide a clear and direct question and answer by analyzing the image?
GenixerI (ours) |  Question: What is the man’s position on the baseball field? Answer: The man is in the batter’s box, which is a designated area on the baseball field where the batter stands to hit the ball.

Table 11: Generating a Referring Expression Comprehension (REC) question-
answer pair by asking GPT-4V and GenixerS.

Generating a REC question-answer pair of the given image:
---
Prompt |  Can you design a clear and direct question and answer by analyzing the image? Here is the format of the question and answer. The question should contain an expression of a specific region of the image and the answer should contain the coordinates. The format for the coordinates is [x1, y1, x2, y2], where [x1, y1] represents the top-left corner and [x2, y2] represents the bottom-right corner of the bounding box. The values of x1, y1, x2, and y2 range from 0 to 1.
GPT-4V |  Question: Can you provide the normalized coordinates for the bounding box around the man sitting in the image? Answer: The bounding box around the man sitting in the image can be approximated with normalized coordinates at [0.65, 0.75, 0.95, 1.00].
Visualization |
Prompt |  I’d appreciate a precise question and your answer. Keep image in mind. This is a REC task
GenixerS (ours) |  Question: I’d like to request the coordinates of A chalkboard menu within the photo. Answer: [0.320,0.283,0.702,0.635].
Visualization |

Table 12: Generating a Referring Expression Generation (REG) question-answer
pair by asking GPT-4V and GenixerS.

Generating a REG question-answer pair of the given image:
---
Prompt |  Can you design a clear and direct question and answer by analyzing the image? Here is the format of the question and answer. The question should contain the coordinates and the answer should contain an expression of a specific region of the image. The format for the coordinates is [x1, y1, x2, y2], where [x1, y1] represents the top-left corner and [x2, y2] represents the bottom-right corner of the bounding box. The values of x1, y1, x2, and y2 range from 0 to 1.
GPT-4V |  Question: What is the activity captured within the coordinates [0.2, 0.05, 0.8, 0.95] in the image? Answer: The region within the coordinates [0.2, 0.05, 0.8, 0.95] shows a person engaged in the process of stone sculpting.
Visualization |
Prompt |  I’d appreciate a precise question and your answer. Keep image in mind. This is a REG task
GenixerS (ours) |  Question: What are the unique aspects of the selected rectangular area [0.064,0.277,0.406,0.586] in image? Answer: A large rock on a workstation.
Visualization |

Table 13: There are some examples for randomly choosing general instructions.

General Instruction Pool “Can you give me a straightforward question and
answer? Include questions asking about the visual content of the image” “I
require a brief and clear question and answer regarding the image” “Please
take a look at the image and promptly provide a question and answer” “Examine
the image. I’m looking for a direct question and answer” “Can you provide a
concise question and answer based on the image¿‘ “I want to know the question
and answer. Refer to the image and give a clear question and answer”
“Regarding the image, please quickly give a question and answer” “I’d
appreciate a precise question and answer. keeping the image in mind” “Can you
provide a clear and direct question and answer by analyzing the image¿‘
“Looking at the image, can you quickly give a question and answer¿‘ “Please
give me an exact question and answer by referring to the image” “Could you
please check the image and provide a straightforward question and answer¿‘
“Here is an image. Please give a brief and clear question and answer” “For
this image, I want a simple and direct question and answer” “Please examine
the image and provide a straightforward question and answer” “Can you take a
look at the image and give a concise question and answer” “Can you provide a
brief and direct question and answer based on the image¿‘ “I need a clear
question and answer in regards to the image” “Please provide a quick and
direct question and answer after examining the image” “Give me a concise
question and answer while keeping the image in mind” “Considering the image,
please provide a straightforward question and answer” “Please give a question
and answer with reference to the attached image, in a clear and simple way” …