# Downslope granular flow through a forest of obstacles

Baptiste Darbois Texier, Yann Bertho, Philippe Gondret  Université Paris-
Saclay, CNRS, Laboratoire FAST, 91405 Orsay, France.

###### Abstract

We investigate the effect of a forest of pillars on a granular layer steadily
flowing over a rough inclined plane. We quantify experimentally how the steady
flow rate of grains is affected by the inter-pillars distance for different
layer thicknesses and slope angles. We then propose a model based on a depth-
average approximation associated with $\mu(I)$ rheology that considers the
additional force exerted by the pillars on the granular layer. This model
succeeds in accounting for most of the observed results when taking into
account some inertia due to the nonvanishing Froude number of the flow.

##  I Introduction

The inclined plane is a classical configuration to investigate the rheological
properties of a flowing material. In steady conditions, it gives access to the
unknown bottom friction that balances the known gravity force. This
configuration has been proved to be suitable for studying the rheology of
yield stress fluids [1], dense suspensions [2] and dry granular materials [3,
4]. For dry granular matter, experiments on an incline show that steady flows
only develop between two critical inclination angles $\theta_{1}$ and
$\theta_{2}$ [5, 4]. For a slope less than $\theta_{1}$ there is no flow, and
for a slope greater than $\theta_{2}$ the flow is accelerated. Between these
two limits, steady flows are well described by the $\mu(I)$ rheology [6] which
relates the local friction coefficient $\mu$ to the local velocity gradient
taking into account the local pressure through the inertial number $I$ [7, 8].
Even if this rheological model suffers from some weaknesses [9], this local
rheology succeeds in explaining most of the observed results in different
geometries and complex flow configurations such as the collapse of a granular
column [10, 11] or the flow around a cylinder [12]. The knowledge of the
rheological behavior of granular matter is crucial for both industrial [13,
14] and natural [15] situations. In particular, the understanding of granular
flows is a first step towards a better prediction of snow avalanches as well
as other hazardous geophysical granular flows such as debris flows, lahars and
pyroclastic flows. The presence of natural or artificial obstacles is known to
reduce the destructive power of these events [16, 17, 18, 19, 20]. Thus, the
physical description of the interaction between granular avalanches and
obstacles is a key input for mitigation of various geophysical phenomena. In
order to develop a precise understanding of the avalanche-obstacles
interaction, several experimental and numerical works have investigated
configurations where obstacles are placed in the path of a noncohesive
granular flow on an inclined plane. Faug et al. have inspected numerically the
case of grains flowing down an incline and impacting a wall normal to the
incoming flow [21]. They showed that the wall generates a triangular stagnant
zone upstream that sharply increases the mean force experienced by the wall at
low inertial numbers. Cui and Gray investigated the gravity-driven granular
flow around a circular cylinder in the supercritical regime [22]. In this
situation, they observed that a very sharp bow shock wave is generated in
front of the cylinder and a grain-free region forms on the lee side. They
studied the position of the bow shock and the granular vacuum as a function of
the flow properties and they proposed a simple two-parameters depth-averaged
avalanche model to capture their observations.

Other studies have moved beyond the case of a single obstacle and considered
the effect of multiple obstacles. Benito et al. have investigated the role of
a forest of cylindrical pillars on the stability of a granular layer on an
incline [23]. They revealed that the presence of pillars increases the
stability of the granular layer towards larger slope angles. They succeeded in
rationalizing this effect by a model that takes into account the additional
friction force exerted by the pillar forest onto the granular layer. More
recently, Luong et al. have studied the spreading of a mass of grains on a
slope through a regular array of pillars [24]. They found that the presence of
pillars slows down the spreading of the granular mass and enhances its lateral
dispersion. Luong et al. proposed an empirical model to capture the slow-down
of the granular mass using an effective friction coefficient that depends on
inter-pillars distance. However, the stationary flow of grains through a
forest of obstacles has not been considered yet while it is of interest to
better understand the coupling between granular flows and multiple obstacles,
and to test the ability of a continuum rheological model to account for the
observed results.

This paper presents experimental results and modeling of the stationary
granular flow that establishes down an inclined plane where a forest of
obstacles is present. Section II details the setup used to study this steady
gravity-driven granular flow and presents the experimental results for the
flow rate as a function of pillar density, layer thickness, and slope angle.
In Sec. III, a modeling based on a depth-average approach associated with a
$\mu(I)$ rheology that accounts for the additional resistive force exerted by
the pillar forest is shown to succeed in explaining the experimental results.
A final discussion is then proposed in Sec. IV.

##  II Experiments

###  II.1 Setup

The experimental setup sketched in Fig. 1(a) consists of a plane of length
$L=50$ cm and width $W=37$ cm that can be inclined by an angle $\theta$ from
the horizontal up to $45^{\circ}$. The peculiarity of this setup is to include
a forest of cylindrical pillars positioned regularly (according to a centered
square lattice) perpendicular to the plane surface with a separating distance
denoted $\Delta$ [Fig. 1(b)]. The diameter of the pillars is $D=2$ mm and
their height (35 mm) is always larger than the thickness of the flowing
granular layer. Three different forests have been built with $\Delta=20$,
$14$, and $10$ mm corresponding to three pillar densities $\chi=1/\Delta^{2}$
equal to $0.25$, $0.51$, and $1.00$ cm-2, respectively. The null pillar
density ($\chi=0$) corresponds to a plane without any pillars. The surface
fraction occupied by the pillars, $\phi_{s}=(\pi/4)(D/\Delta)^{2}$, is equal
to about $0.8$, $1.6$, and $3.1\%$, respectively, for the three different
forests, far from the maximal value $\phi_{s}=\pi/4\simeq 78.5\%$
corresponding to the limit case $\Delta=D$. The particles used in the
experiments are sieved glass beads (density $\rho_{g}=2520$ kg m-3) of
diameter $d=450$ $\mu$m with a relative dispersion in diameter of order 10%.
The same glass beads have been used to cover the bottom of the inclined plane
to ensure a no-slip condition. The cylinder-grain size ratio $D/d$ equals 4.4
and the number of grains between two pillars $(\Delta-D)/d$ is always larger
than 17, which prevents clogging [25].

Figure 1: (a) Sketch of the experimental setup and notations used. (b) Top
view of the inclined plane with the organization of the forest of pillars
along a centered square lattice. (c) Picture of the granular medium flowing
through a pillar array of separating distance $\Delta=20$ mm.

An experiment consists in filling with grains the reservoir placed at the top
of the incline and opening the gate with an aperture $H$ at the bottom of the
reservoir. Using the scale placed at the end of the incline, we measure the
mass of grains $M$ falling from the plane as a function of time $t$ from which
we derive the instantaneous mass flow rate $Q=dM/dt$. A low-incident laser
sheet and a camera are used to estimate the mean thickness $h$ of the flowing
layer with an accuracy of $0.1$ mm. The thickness of the layer is measured at
the center of the slope, away from the proximity of the pillars, and averaged
over the longitudinal extent of the laser sheet, which is about 5 cm. It is
observed that the thickness of the layer remains constant for the duration of
the steady state. We first realize a set of 60 experiments where the slope is
constant $\theta=29.5^{\circ}$ and where the mean layer thickness is varied
between 5 and 22 times the grain diameter. Then, we realize 45 experiments
where the inclination is varied between $24.5^{\circ}$ and $32.5^{\circ}$ by
steps of $0.5^{\circ}$ and the layer thickness is maintained approximately
constant around ten grain diameters. In order to maintain a constant
thickness, we manually adjust the gate opening for each inclination angle to
compensate for the fact that the average thickness of the layer decreases as
the slope increases. For all the experiments, no significant variation of the
flowing layer thickness along the transverse and lateral directions of the
plane has been observed during the steady regime (within experimental
uncertainties) as long as the measurements were not made in the vicinity of
the gate or the exit of the inclined plane where entry and exit effects occur.
We also measure the velocity of the grains at the surface of the flowing layer
and we do not observe any significant variation along the transverse
direction, except in a thin layer near the sidewalls ($\approx 3$ mm wide).
Neglecting the presence of this layer on the estimate of the flow velocity
averaged over the layer thickness, $U=Q/\rho\,W\,h$, leads to a relative error
less than 1%, where $\rho=\phi\,\rho_{g}=1510$ kg m-3 is the bulk density of
the flowing material and $\phi$ the grain packing fraction. The volume
fraction $\phi$ is estimated by weighting the grains contained on a given
surface averaged on the thickness of the flowing layer $h$. We find that the
averaged volume fraction is constant, $\phi=0.60\pm 0.01$, and does not show a
clear dependency on the slope inclination nor on the thickness of the flowing
layer in agreement with the observations of Benito et al. [23].

Note that our experimental device may be reminiscent of the “Galton board”
that has been used to study transverse dispersion of grains along their
falling path through a forest of obstacles [26, 27]. However, our setup
presents several differences with the Galton board: a continuous injection of
grains over the whole width of the plane, a large inter-obstacles distance,
and a significant friction of the grains with the walls.

###  II.2 Results

In order to characterize our system, we first measure the thickness
$h_{\rm{stop}}$ at which the granular flow stops as a function of $\theta$ for
different pillar densities, following the procedure described by Pouliquen
[4]. This critical thickness is the thickness below which no steady flow can
be observed. Figure 2 shows the stopping curves $h_{\rm{stop}}(\theta)$ for
the different pillar densities $\chi$ considered in this paper.

Figure 2: Normalized thickness $h_{\rm{stop}}/d$ of the remaining layer after
an avalanche as a function of the slope angle $\theta$ for different pillar
densities ($\circ$) $\chi=0$ cm-2, ($\vartriangle$) $\chi=0.25$ cm-2,
($\diamond$) $\chi=0.51$ cm-2, and ($\square$) $\chi=1.00$ cm-2. The solid
lines are the best fits of the data by Eq. (1) with $\lambda=1.55$,
$\theta_{2}=36.6^{\circ}$, and
$\theta_{1}=22.2^{\circ},\,22.4^{\circ},\,25.0^{\circ}$, and $26.2^{\circ}$.
Inset: Variation of $\tan\theta_{1}$ with the pillar density $\chi$. The solid
line is the best linear fit through the data of equation
$\tan\theta_{1}=0.404+10.2\,\chi Dd$.

We observe that the presence of pillars shifts the curve towards larger slope
angles for larger $\chi$, in agreement with the observations of Benito et al.
[23]. According to [23], these results can be fitted by the equation

$\frac{h_{\rm{stop}}(\theta,\chi)}{d}=\lambda\,\frac{\tan\theta_{2}-\tan\theta}{\tan\theta-\tan\theta_{1}(\chi)},$
(1)

where $\theta_{1}$ and $\theta_{2}$ correspond to the characteristic stopping
angles of the granular layer of infinite height ($h\rightarrow\infty$) and
vanishing layer thickness ($h\rightarrow 0$), respectively, and $\lambda$ is a
numerical prefactor characterizing the influence of the rough bottom surface
on the stopping height. Note that in Eq. (1), the dependence of
$\tan\theta_{1}$ with $\chi$ results from the additional friction of the
granular layer on pillars and does not correspond to a change in the bulk
properties of the medium. For the case without pillars ($\chi=0$), the best
fit of the data is found for $\theta_{1}=22.2^{\circ}$,
$\theta_{2}=36.6^{\circ}$, and $\lambda=1.55$. For the cases with pillars
($\chi\neq 0$), we keep $\theta_{2}$ and $\lambda$ unchanged following the
findings of Benito et al. who show that only $\theta_{1}$ depends
significantly on $\chi$ [23]. The best fits of the curves for the three pillar
densities $\chi=0.25$, $0.51$, and $1.00$ cm-2 are found for
$\theta_{1}=22.4^{\circ}$, $25.0^{\circ}$, and $26.2^{\circ}$, respectively.
These fits appear as solid lines in Fig. 2 and pass quite well through the
data. As expected, increasing $\chi$ increases $\theta_{1}$ as shown in the
inset of Fig. 2. A linear fit of the form
$\tan\theta_{1}(\chi)=\tan\theta_{1}(0)+\alpha\chi Dd$, with $\alpha\simeq
10$, passes rather well through the data and is in relatively good agreement
with the previous findings of Benito et al. [23].

Thereafter, we focus on the granular flow that establishes during a discharge
of the reservoir. Figure 3(a) presents typical evolution of the mass of grains
$M$ as a function of time $t$ after opening of the reservoir at a given
aperture of the gate ($H=5.5$ mm), a given pillar density ($\chi=0.51$ cm-2),
and different slope angles ($28^{\circ}\leq\theta\leq 32^{\circ}$). The inset
of Fig. 3(a) shows the instantaneous mass flow rate $Q(t)=dM/dt$ derived from
these measurements. After a transient regime, for $t\gtrsim 5$ s, we observe
that the mass flow rate quickly increases up to a constant plateau value that
is maintained for most of the discharge duration before vanishing when the
reservoir is empty. The flow rate from the reservoir is kept approximately
constant since the pressure at the outlet does not depend on the height of
grains in the reservoir due to Janssen effect [28].

Figure 3: (a) Mass of grains $M$ as a function of time $t$ after opening the
reservoir at a given gate aperture ($H=5.5$ mm), a pillar density ($\chi=0.51$
cm-2), and nine different slope angles from $28.0^{\circ}$ to $32.0^{\circ}$
by steps of $0.5^{\circ}$. Inset: derived mass flow rate $Q=dM/dt$ as a
function of time. (b) Normalized steady flow rate $\tilde{Q}=\langle
Q\rangle/\rho Wg^{1/2}d^{3/2}$ as a function of the normalized layer thickness
$h/d$ at slope angle $\theta=29.5^{\circ}$, for different pillar densities
($\circ$) $\chi=0$ cm-2, ($\vartriangle$) $\chi=0.25$ cm-2, ($\diamond$)
$\chi=0.51$ cm-2, and ($\square$) $\chi=1.00$ cm-2.

In the following, we focus on the steady flow rate $\langle Q\rangle$
developing in between these two starting and stopping transients and its
dependency with the granular layer thickness and the slope angle. Each
experiment has been reproduced at least three times in order to determine the
average value of the steady flow rate and to estimate the dispersion of the
results. Figure 3(b) presents the mean normalized flow rate $\tilde{Q}=\langle
Q\rangle/\rho Wg^{1/2}d^{3/2}$ as a function of the normalized thickness of
the flowing layer $h/d$ for a given slope angle $\theta=29.5^{\circ}$. In the
absence of pillars ($\chi=0$), a supra-linear increase of $\tilde{Q}$ with
$h/d$ is observed. The presence of the pillar forest ($\chi\neq 0$) reduces
the granular flow rate and the trend is no longer supra-linear but rather sub-
linear. The reduction of the flow rate is more significant as the thickness of
the granular layer or the density of pillars increases. For the smallest
pillar density ($\chi=0.25$ cm-2), the flow rate is only significantly reduced
for large enough layer thicknesses ($h/d\gtrsim 7$), which is consistent with
the fact that the stopping layer $h_{\rm{stop}}$ at a slope angle of
$\theta=29.5^{\circ}$ is not significantly increased by a pillar forest of
such a low density. Interestingly, the addition of pillars not only reduces
the mass flow rate but also changes the dependence of $\tilde{Q}$ with the
layer thickness in a nontrivial way. In particular, for $\chi=1.00$ cm-2, the
flow rate is almost constant and no longer depends on the thickness of the
granular layer [Fig. 3(b)].

Then we measure the steady flow rate $\langle Q\rangle$ as a function of the
slope angle while keeping the flowing layer thickness constant in the best
possible way. Indeed, the layer thickness can hardly be maintained strictly
constant because the relation between the gate opening and the granular layer
thickness is nontrivial as pointed out by Cui and Gray [22]. Figures 4(a) and
4(b) show the simultaneous measurements of $\tilde{Q}$ and $h/d$ as a function
of the slope angle $\theta$. The normalized flow rate is observed to increase
with $\theta$ above a threshold angle which is about $24^{\circ}$ in the
absence of pillars ($\chi=0$). This threshold value corresponds to the value
$\theta_{\rm{stop}}(h/d)$ for the forced thickness $h/d\simeq 10$ that is kept
approximately constant for different $\theta$ [Fig. 4(b)]. This observation is
consistent with Fig. 2 where the flow stops for $h/d\simeq 10$ when
$\theta=24^{\circ}$ and $\chi=0$. When the pillar density is increased, it
induces a shift of the threshold angle towards larger values up to about
$30^{\circ}$ for the highest density ($\chi=1.00$ cm-2). Note that this
increase of the flow rate is mainly due to an increase in velocity as the
thickness of the flowing layer is approximately constant in the whole range of
slope angle $\theta$ for each value of $\chi$ [Fig. 4(b)].

Figure 4: (a) Normalized steady flow rate $\tilde{Q}=\langle Q\rangle/\rho
Wg^{1/2}d^{3/2}$ and (b) normalized layer thickness $h/d$, as a function of
the slope angle $\theta$ for different pillar densities ($\circ$) $\chi=0$
cm-2, ($\vartriangle$) $\chi=0.25$ cm-2, ($\diamond$) $\chi=0.51$ cm-2, and
($\square$) $\chi=1.00$ cm-2.

For granular flows down a rough incline without any pillar forest ($\chi=0$),
the mean velocity $U$ normalized by $(gh)^{1/2}$ has been observed to scale
linearly with the thickness $h$ normalized by the stopping thickness
$h_{\rm{stop}}$ [4, 29, 8]. This implies that the dimensionless flow rate
$\tilde{Q}$ should scale as $h^{5/2}/(h_{\rm{stop}}~{}d^{3/2})$. By testing
this scaling with our data (Fig. 5), we notice that the different sets of data
obtained for $\chi=0$ follow well such a scaling with the numerical
coefficient 0.14 obtained by Pouliquen [4] for flowing glass beads on a bottom
of glued glass beads. However, the data sets corresponding to a pillar forest
($\chi\neq 0$) do not follow such a scaling, or with a slope that decreases
significantly as $\chi$ increases. In the next section, we present a model
that rationalizes these results.

Figure 5: Normalized steady flow rate $\tilde{Q}=\langle Q\rangle/\rho
Wg^{1/2}d^{3/2}$ as a function of the normalized thickness
$(h/d)^{3/2}\,h/h_{\rm{stop}}(\chi)$ for different pillar densities ($\circ$)
$\chi=0$ cm-2, ($\vartriangle$) $\chi=0.25$ cm-2, ($\diamond$) $\chi=0.51$
cm-2, and ($\square$) $\chi=1.00$ cm-2. The solid line corresponds to the
linear trend with the slope of 0.14 proposed by Pouliquen [4] for glass beads
flowing down a slope without pillars.

##  III Model

Let us now model the granular flow down the incline driven by the gravity $g$
with a depth-average approach by considering the $\mu(I)$ rheology [7]. This
rheology relates the friction coefficient of the granular medium $\mu$,
defined as the ratio of the shear stress over the normal stress $P$, with the
inertial number defined as $I=\dot{\gamma}\,d/\sqrt{P/\rho_{g}}$, where
$\dot{\gamma}$ is the shear rate of the flow. Under the depth-average
approximation, a slice of the granular layer experiences its weight and a
basal friction dictated by the $\mu(I)$ rheology, two forces that balance in a
stationary regime [29]. In steady state, the force balance on a slice of the
granular layer of thickness $h$ and of unit length and unit width reads

$\rho gh\sin\theta-\mu_{b}(I_{b})\rho gh\cos\theta=0,$ (2)

where the basal friction coefficient $\mu_{b}$ is a sole function of the
inertial number $I_{b}$ calculated at the base of the granular layer. For a
Bagnold velocity profile in the granular layer, the basal shear rate
$\dot{\gamma_{b}}$ is related to the mean flow speed $U=\langle Q\rangle/\rho
Wh$ through the relation $\dot{\gamma_{b}}=5U/2h$. Considering that the
pressure at the bottom of the granular layer is $P=\rho gh\cos\theta$, we get
that the inertial number at this location is $I_{b}(h,U)=5\,U\,d/2\,\sqrt{\phi
gh^{3}\cos\theta}$. Experimental results obtained in different flow
configurations have permitted the establishment of an expression for the
dependency of the basal friction coefficient with flow parameters

$\mu_{b}(I_{b})=\tan\theta_{1}+\frac{\tan\theta_{2}-\tan\theta_{1}}{1+I_{0}/I_{b}},$
(3)

where $I_{0}$ is the characteristic inertial number above which the friction
coefficient increases from $\tan\theta_{1}$ to $\tan\theta_{2}$ [8]. Note that
$I_{0}$ may be viewed as the inverse of a characteristic length scaled by the
grain diameter, and depends on the granular material and bed roughness [30].
Inserting the friction law Eq. (3) in Eq. (2), we get a prediction for the
normalized flow rate in the absence of pillars:

$\tilde{Q}(h,\theta)=\frac{2I_{0}}{5}\frac{\tan\theta-\tan\theta_{1}}{\tan\theta_{2}-\tan\theta}\sqrt{\phi\cos\theta}\left(\frac{h}{d}\right)^{5/2}.$
(4)

In order to compare this classical prediction with our observations made in
the absence of pillars ($\chi=0$), we present the normalized flow rate
$\tilde{Q}=\langle Q\rangle/\rho Wg^{1/2}d^{3/2}$ as a function of $h/d$ for
$\theta=29.5^{\circ}$ in Fig. 6(a) and the normalized flow rate $\tilde{Q}$
divided by $(h/d)^{5/2}$ as a function of $\theta$ in Fig. 6(b), respectively.
The blue lines in these plots indicate the best fits of the data with Eq. (4)
considering the values determined previously for the angles $\theta_{1}$ and
$\theta_{2}$ (Sec. II.2) and considering $I_{0}$ as a free parameter. The best
fits are found for $I_{0}=0.35$ in reasonable agreement with previous
estimates made for glass beads ($I_{0}\simeq 0.28-0.30$) [6, 7]. The model
fits quite well the experimental data in Fig. 6(a) which means that the
dimensionless flow rate $\tilde{Q}$ follows the expected power scaling law
$(h/d)^{5/2}$. The model also fits well the data in Fig. 6(b) for moderate
slope angle $\theta$ but seems to overestimate $\tilde{Q}$ for $\theta\gtrsim
30^{\circ}$. Note that for a given slope angle in between $\theta_{1}$ and
$\theta_{2}$, this model predicts a vanishing steady flow for a vanishing
thickness $h$ and fails to predict the stopping of the flow at
$h_{\rm{stop}}$. This is a known weakness of the $\mu(I)$ rheology as the
complex liquid-solid transition of granular matter is not captured by this
local approach [8].

Figure 6: (a) Normalized steady flow rate $\tilde{Q}=\langle Q\rangle/\rho
Wg^{1/2}d^{3/2}$ as a function of the normalized layer thickness $h/d$ for a
given slope angle $\theta=29.5^{\circ}$. (b) Normalized steady flow rate
$\tilde{Q}$ divided by $(h/d)^{5/2}$ as a function of the slope angle
$\theta$. Same data as in Figs. 3(b) and 4. Dashed lines show the noninertial
predictions ($C_{d}=0$) from Eq. (7) for each pillar density with $I_{0}=0.35$
and $\beta=1.5$. Solid lines present the inertial numerical solutions of Eq.
(6) with $\beta=1.5$ and $C_{d}=3$.

Subsequently, we develop a model to rationalize the effect of pillars on
granular flow. Note that a satisfactory model cannot be achieved by
considering different parameters of the $\mu(I)$ rheology. Indeed, such an
approach would not capture the change in the dependence of the flow rate on
layer thickness and slope angle. For this reason, we consider a new ingredient
in the model which is the force applied by the pillars on the granular layer.
The force experienced by a single pillar immersed in a granular flow depends
on the flow regime which is determined by the Froude number
$\mathrm{Fr}=U/\sqrt{gh\cos\theta}$ based on the mean flow speed $U$ and the
layer thickness $h$ [31]. In the limit of small Froude numbers
($\mathrm{Fr}\ll 1$), the drag force results from the hydrostatic-like
pressure, $\rho gz\cos\theta$, applying on the exposed surface of the pillar,
$h\,D$, and expresses as $f_{h}=\beta\rho gh^{2}D\cos\theta$, where $\beta$ is
a numerical prefactor [32]. In the opposite limit of large Froude numbers
($\mathrm{Fr}\gg 1$), the drag force results from inertia and expresses
$f_{i}=C_{d}\,\rho\,hD\,U^{2}$, where $C_{d}$ is a dimensionless drag
coefficient.

Figure 7: Froude number $\mathrm{Fr}$ as a function of (a) the normalized
layer thickness $h/d$ and (b) the slope angle $\theta$ for four different
pillar densities ($\circ$) $\chi=0$ cm-2, ($\vartriangle$) $\chi=0.25$ cm-2,
($\diamond$) $\chi=0.51$ cm-2, and ($\square$) $\chi=1.00$ cm-2.

The variations of the Froude number $\mathrm{Fr}$ as a function of $h/d$ and
$\theta$ are shown in Figs. 7(a) and 7(b), respectively. Only a few
$\mathrm{Fr}$ values are a little larger than 1 and all the $\mathrm{Fr}$
values corresponding to the cases with a pillar forest ($\chi\neq 0$) are
smaller than 0.7. In this range, the drag force is expected to correspond
mainly to a quasi-static force term with a little inertia. This implies that
both contributions to the drag force have to be considered in our model. For
simplicity, we will consider that the resulting force in this transitional
regime corresponds to the addition of the two limit terms $f_{h}+f_{i}$. This
crude modeling has been already used to reproduce quite well the variation of
the force with $\mathrm{Fr}$ [31], and correctly predicts the dynamics of an
object impacting a granular material [33, 34]. In order to account for the
presence of multiple pillars, we assume that there is no interaction between
pillars and that the sum of the individual forces exerted by the pillars on
the granular layer results in a global force per unit surface acting on the
whole layer of grains which thus simply reads $(f_{h}+f_{i})\chi$. Indeed, the
granular flow that establishes in the vicinity of a cylindrical obstacle has
been studied experimentally, revealing that the velocity perturbation vanishes
on a distance that is roughly one cylinder diameter [35]. In our experiments,
the minimal distance between pillars ($\Delta=10$ mm) is much larger than the
pillar diameter ($D=2$ mm) so that no interaction between pillars is expected.
Under these approximations, the force balance for a slice of the granular
layer in the presence of pillars becomes

$\rho gh\sin\theta-\mu_{b}(U,h)\rho gh\cos\theta-\beta\rho g\chi
h^{2}D\cos\theta-C_{d}\rho\chi hD\,U^{2}=0,$ (5)

which can be written in a dimensionless form as

$\tan\theta-\mu_{b}(U,h)-\beta\chi hD-C_{d}\frac{\chi
D\,U^{2}}{g\cos\theta}=0.$ (6)

Solving Eq. (6) provides the dependency of the mean flow speed $U$ (or
equivalently the normalized flow rate $\tilde{Q}$) with the relevant
parameters of the problem ($h$, $\theta$, $\chi$, …). Note that in the limit
where $\chi\rightarrow 0$, Eq. (5) leads to Eq. (2) and we recover the
prediction of the standard $\mu(I)$ rheology. In the particular case where
$C_{d}=0$, a simple analytical solution of Eq. (6) can be found for the
normalized flow rate $\tilde{Q}$:

$\tilde{Q}=\frac{2I_{0}}{5}\,\frac{\tan\theta-\tan\theta_{1}-\beta\chi
hD}{\tan\theta_{2}+\beta\chi
hD-\tan\theta}\,\sqrt{\phi\cos\theta}\,\left(\frac{h}{d}\right)^{5/2}.$ (7)

In this limit of vanishing inertial drag ($C_{d}=0$) corresponding to low
enough $\mathrm{Fr}$, Eq. (7) predicts that the presence of pillars changes
the discharge flow curve $\tilde{Q}(h,\theta)$ through a shift of both
friction coefficients $\mu_{1}=\tan\theta_{1}$ and $\mu_{2}=\tan\theta_{2}$
(associated to the inclined plane without pillars) by a quantity $\beta\chi
hD$ that is directly proportional to the density $\chi$ of pillars and their
surface $hD$ exposed to the flow. At a given angle, this correction mitigates
the increase of the flow rate with $h$ in qualitative agreement with the
observations presented in Fig. 3. The shift of $\tan\theta_{2}$ predicted by
Eq. (7) in the case of a flowing layer contrasts with the fact that no
variation of $\tan\theta_{2}$ with $\chi$ is observed in the case of a
stopping layer (as discussed in Sec. II.2). This difference can be understood
from the competition between the friction of the granular layer with the
bottom wall and with pillars: indeed, when $\theta$ approaches $\theta_{2}$,
the stopping granular layer necessarily has a negligible thickness and its
behavior is dominated by the friction with the bottom wall ($\beta\chi h_{\rm
stop}D\ll\mu_{b}$), whereas for a flowing layer with a larger thickness the
resistance of pillars can no longer be neglected ($\beta\chi hD\sim\mu_{b}$).
In order to compare quantitatively the predictions of Eq. (7) with our data,
we used the values determined in the absence of pillars for the parameters
$\tan\theta_{1}$, $\tan\theta_{2}$, and $I_{0}$, and we consider the force
coefficient $\beta$ as a free parameter. The best fits of our experimental
data with the predictions of Eq. (7) are found for $\beta=1.5\pm 0.1$, and are
shown with dashed lines in Figs. 6(a) and 6(b) for all the pillar densities
investigated in this paper. The predictions of the noninertial model capture
the main trend in the data, i.e., the lowering of the granular flow with
increasing pillar density. Nonetheless, the assumption of null inertial drag
($C_{d}=0$) is too strong to rationalize finely our experimental observations,
in particular the variation of the dimensionless flow rate $\tilde{Q}$ with
the slope angle $\theta$ as observed in Fig. 6(b).

Let us now consider the general case where the inertial contribution is taken
into account ($C_{d}\neq 0$). For the sake of simplicity, we do not make
explicit the analytical solution of Eq. (6) but we present its numerical
solutions. In order to compare these predictions with our observations, we
imposed the values determined in the absence of pillars for the parameters
$I_{0}$, $\tan\theta_{1}$, and $\tan\theta_{2}$ and we consider the two force
coefficients $\beta$ and $C_{d}$ as free parameters and adjust them to get the
best fit between experiments and theory. The best fits are found for
$\beta=1.5\pm 0.1$ and $C_{d}=3\pm 1$, and are presented with solid lines in
Figs. 6(a) and 6(b). Accounting for the inertial contribution of the pillar
drag permits us to have a much better agreement between theory and
experiments, especially for the dependency of the normalized flow rate with
the slope angle. Our value of the quasi-static coefficient, $\beta\simeq 1.5$,
is rather close to the value $\beta=2.4$ found by Albert et al. [32] in the
limit where the object size is large compared to grain diameter ($D/d>3$). The
difference between these two values remains in the usual range of incertitude
for force coefficients associated to quasi-static granular flows [31].
Concerning the drag coefficient, our value $C_{d}\simeq 3$ is in very good
agreement with the range $C_{d}=3\pm 2$ reported by Faug [31]. The ratio of
the inertial and the quasi-static drag forces on pillars can be written as
$f_{i}/f_{h}=(C_{d}/\beta)\,\mathrm{Fr}^{2}$. As $C_{d}/\beta\simeq 2$, the
transition between these two regimes is expected to be around the critical
Froude number $\mathrm{Fr}_{c}=(\beta/C_{d})^{1/2}\simeq 0.7$. This confirms
that our data obtained with a pillar forest ($\chi\neq 0$) correspond to a
quasi-static regime with a little inertia. Nonetheless, the model developed
hereby has a major limitation as it predicts that the granular flow of thick
layers should be stopped by the largest pillar density for $h/d\gtrsim 12.5$
at $\theta=29.5\,^{\circ}$ (see the orange solid line in Fig. 6(a)
corresponding to $\chi=1.00$ cm-2), a fact that is not observed in our
experiments. This discrepancy highlights the limitation of the spatial
averaging assumption. In practice, the pillars have a restricted zone of
influence and cannot stop the flow away from their field of action, a behavior
that is not accounted for the averaging assumption.

##  IV Discussion

The model developed in the limit of vanishing inertia ($C_{d}\simeq 0$ when
$\mathrm{Fr}\ll 1$) shows that the presence of pillars is accounted through
the parameter $\beta\chi hD$ for a steady flow of thickness $h$. This
parameter, which is proportional to $h$, induces a reduction of flow rate that
is larger for larger $h$ at a given slope angle $\theta$ and for a pillar
forest of given density $\chi$. Note that the stopping thickness
$h_{\rm{stop}}$ was found to depend on $\chi$ through the parameter
$\alpha\chi Dd$ [inset of Fig. 2(b)]. Combining the linear variation of
$\tan(\theta_{1}(\chi))$ with Eq. (6) for vanishing $U$ leads to an estimate
for the stopping thickness $h_{\rm{stop}}/d=\alpha/\beta\simeq 6$ considering
$\alpha\simeq 10$ and $\beta\simeq 1.5$. This estimate is in rather good
agreement with our data as we do not observe any steady flow for $h/d\lesssim
5$ [Figs. 3(b) and 6(a)]. According to Eq. (7), the influence of the array of
pillars on the granular flow rate is significant when the parameter $\beta\chi
hD$ is of the same order as $\tan\theta-\tan\theta_{1}$ or
$\tan\theta_{2}-\tan\theta$. In the intermediate case where
$\theta=(\theta_{1}+\theta_{2})/2$, we have
$\tan\theta-\tan\theta_{1}=\tan\theta_{2}-\tan\theta=0.17$ and the term
resulting from the presence of the pillar is equal to one fifth of this value
when the pillar density reaches a critical value $\chi_{c}\simeq 0.02/hD$.
Under these conditions, the presence of pillars is expected to reduce the mass
flow rate by a factor of two thirds according to Eq. (7). In the case of the
setup studied previously, where the pillar diameter is $D=2$ mm and the layer
thickness is about $5$ mm, the critical pillar density above which the
presence of pillars impacts significantly the flow rate is $\chi_{c}=0.2$
cm-2. Transposing the same considerations to an avalanche of granular material
occurring through a tree forest with typical values $h\sim 1$ m and $D\sim
0.2$ m, the critical obstacle density should be $0.1$ m-2 corresponding to one
tree every 3 m.

In this paper, we have explored the effect of a forest of cylindrical pillars
on the steady granular flow running down an incline. This experiment
constitutes a paradigm for geophysical flow mitigation by a forest of
obstacles. In addition to reduce the granular flow rate, the presence of
pillars changes drastically the dependency of the flow rate with the thickness
of the granular layer, i.e., the discharge flow curve. We accounted for the
nontrivial impact of the forest of pillars on the granular flow with a model
based on a depth-averaged approach and the $\mu(I)$ rheology. This model
includes the effect of pillars through an averaged force applying on the
granular layer and neglecting interactions between pillars. This approach has
been proved to predict quantitatively the reduction of the granular flow rate
induced by the presence of pillars. It can be used to determine the minimal
pillar density necessary to reduce significantly the energy carried by the
granular flow. These results prove that the $\mu(I)$ rheology can be used to
describe nonunidirectional granular flows as the one studied here.
Nonetheless, some discrepancies between model predictions and experiments are
remaining and should motivate future work to relax the averaging approximation
and to study the granular flow occurring in the inter-space between pillars.
Indeed, the velocity field of the grain flow around a cylinder has been
already characterized together with the granular temperature field and both
have been shown to be localized [35, 36] but it would be interesting to
investigate the possible modification of these fields in the case of an
incline plane with multiple obstacles. Finally, this paper opens several
questions such as the influence of the grain size in relation to the diameter
of pillars or the effect of pillar arrangement on the flow. Indeed, if it has
already been confirmed that the proposed scaling laws are valid for different
grain sizes in the case of a pillar-free plane, it could be interesting to
confirm these results in the case of pillar forests. Moreover, it could be
useful to consider a random distribution of pillars on the inclined plane,
perhaps closer to situations encountered in nature, that should generate local
clogging and preferred paths for the granular flow.

###### Acknowledgements.

We are grateful to P. Jenffer and G. Marteau for the conception and
realization of the experimental setup, and to J. Amarni, A. Aubertin, L.
Auffray, C. Manquest, and R. Pidoux for their technical support. This work was
supported by ANR PIA Grant No. ANR-20-IDEES-0002.

## References

  * Coussot and Boyer [1995] P. Coussot and S. Boyer, Rheol. Acta 34, 534 (1995).
  * Bonnoit  _et al._ [2010] C. Bonnoit, T. Darnige, E. Clement, and A. Lindner, J. Rheol. 54, 65 (2010).
  * GDR MiDi [2004] GDR MiDi, Eur. Phys. J. E 14, 341 (2004).
  * Pouliquen [1999] O. Pouliquen, Phys. Fluids 11, 542 (1999).
  * Daerr and Douady [1999] A. Daerr and S. Douady, Nature 399, 241 (1999).
  * Jop  _et al._ [2006] P. Jop, Y. Forterre, and O. Pouliquen, Nature 441, 727 (2006).
  * Forterre and Pouliquen [2008] Y. Forterre and O. Pouliquen, Annu. Rev. Fluid Mech. 40, 1 (2008).
  * Andreotti  _et al._ [2013] B. Andreotti, Y. Forterre, and O. Pouliquen,  _Granular media: between fluid and solid_ (Cambridge University Press, 2013).
  * Barker  _et al._ [2015] T. Barker, D. G. Schaeffer, P. Bohórquez, and J. Gray, J. Fluid Mech. 779, 794 (2015).
  * Lacaze and Kerswell [2009] L. Lacaze and R. R. Kerswell, Phys. Rev. Lett. 102, 108305 (2009).
  * Lagrée  _et al._ [2011] P.-Y. Lagrée, L. Staron, and S. Popinet, J. Fluid Mech. 686, 378 (2011).
  * Seguin  _et al._ [2016] A. Seguin, A. Lefebvre-Lepot, S. Faure, and P. Gondret, Eur. Phys. J. E 39, 1 (2016).
  * Gaudel  _et al._ [2016] N. Gaudel, S. K. de Richter, N. Louvet, M. Jenny, and S. Skali-Lami, Phys. Rev. E 94, 032904 (2016).
  * Gaudel and Kiesgen de Richter [2019] N. Gaudel and S. Kiesgen de Richter, Powder Technology 346, 256 (2019).
  * Delannay  _et al._ [2017] R. Delannay, A. Valance, A. Mangeney, O. Roche, and P. Richard, J. Phys. D: Appl. Phys. 50, 053001 (2017).
  * Feistl  _et al._ [2014] T. Feistl, P. Bebi, M. Teich, Y. Bühler, M. Christen, K. Thuro, and P. Bartelt, J. Glaciology 60, 124 (2014).
  * Teich  _et al._ [2014] M. Teich, J.-T. Fischer, T. Feistl, P. Bebi, M. Christen, and A. Grêt-Regamey, Nat. Hazards Earth Syst. Sci. 14, 2233 (2014).
  * Yan  _et al._ [2021] K. Yan, J. He, Q. Cheng, J. Zhang, and R. Fuentes, Landslides 18, 1761 (2021).
  * Védrine  _et al._ [2022] L. Védrine, X. Li, and J. Gaume, Nat. Hazards Earth Syst. Sci. 22, 1015 (2022).
  * Liang  _et al._ [2023] Z. Liang, C. E. Choi, Y. Zhao, Y. Jiang, and J. Choo, Comput. Geotech. 155, 105194 (2023).
  * Faug  _et al._ [2009] T. Faug, R. Beguin, and B. Chanut, Phys. Rev. E 80, 021305 (2009).
  * Cui and Gray [2013] X. Cui and J. M. N. T. Gray, J. Fluid Mech. 720, 314 (2013).
  * Benito  _et al._ [2012] J. Benito, Y. Bertho, I. Ippolito, and P. Gondret, EPL 100, 34004 (2012).
  * Luong  _et al._ [2020] T. H. Luong, J. L. Baker, and I. Einav, Granular Matter 22, 1 (2020).
  * Zuriguel  _et al._ [2005] I. Zuriguel, A. Garcimartín, D. Maza, L. A. Pugnaloni, and J. Pastor, Phys. Rev. E 71, 051303 (2005).
  * Bruno  _et al._ [2003] L. Bruno, A. Calvo, and I. Ippolito, Eur. Phys. J. E 11, 131 (2003).
  * Benito  _et al._ [2007] J. G. Benito, G. Meglio, I. Ippolito, M. Re, and A. M. Vidales, Granular Matter 9, 159 (2007).
  * Bertho  _et al._ [2003] Y. Bertho, F. Giorgiutti-Dauphiné, and J.-P. Hulin, Phys. Rev. Lett. 90, 144301 (2003).
  * Forterre and Pouliquen [2003] Y. Forterre and O. Pouliquen, J. Fluid Mech. 486, 21 (2003).
  * Gray and Edwards [2014] J. Gray and A. Edwards, J. Fluid Mech. 755, 503 (2014).
  * Faug [2015] T. Faug, Eur. Phys. J. E 38, 1 (2015).
  * Albert  _et al._ [1999] R. Albert, M. A. Pfeifer, A.-L. Barabási, and P. Schiffer, Phys. Rev. Lett. 82, 205 (1999).
  * Seguin  _et al._ [2009] A. Seguin, Y. Bertho, P. Gondret, and J. Crassous, EPL 88, 44002 (2009).
  * Katsuragi and Durian [2013] H. Katsuragi and D. J. Durian, Phys. Rev. E 87, 052208 (2013).
  * Seguin  _et al._ [2011] A. Seguin, Y. Bertho, P. Gondret, and J. Crassous, Phys. Rev. Lett. 107, 048001 (2011).
  * Seguin  _et al._ [2013] A. Seguin, Y. Bertho, F. Martinez, J. Crassous, and P. Gondret, Phys. Rev. E 87, 012201 (2013).


# Exploring Perceptual Limitation of Multimodal Large Language Models

Jiarui Zhang  Jinyi Hu  Mahyar Khayatkhoei  Filip Ilievski  Maosong Sun

###### Abstract

Multimodal Large Language Models (MLLMs) have recently shown remarkable
perceptual capability in answering visual questions, however, little is known
about the limits of their perception. In particular, while prior works have
provided anecdotal evidence of MLLMs’ sensitivity to object size, this
phenomenon and its underlying causes have not been explored comprehensively.
In this work, we quantitatively study the perception of small visual objects
in several state-of-the-art MLLMs and reveal a pervasive limitation in
answering questions about small objects in images. Next, we identify four
independent factors that can contribute to this limitation—object quality,
size, distractors, and location—and conduct controlled intervention studies to
measure the effect of each factor on MLLMs’ perception. In particular, we find
that lower object quality and smaller object size can both independently
reduce MLLMs’ ability to answer visual questions. More surprisingly, we find
that the location of the object in the image and the presence of visual
distractors can also significantly reduce MLLMs’ question answering accuracy.
Our study provides a better understanding of the perceptual limitation of
MLLMs and contributes new evaluation protocols for analyzing the perception of
future MLLMs. To facilitate further investigations, we release our code and
data here.

Machine Learning, ICML

##  1 Introduction

Figure 1:  Failure cases of GPT-4V (OpenAI, 2023) in perceiving small objects
when serving as web agents. Our research studies this perceptual limitation in
several recent MLLMs.

The development of Multimodal Large Language Models (MLLMs) (OpenAI, 2023;
Team et al., 2023; Liu et al., 2023b; Dai et al., 2023) has significantly
broadened the capabilities of Large Language Models (LLMs) (OpenAI, 2023;
Touvron et al., 2023), enabling them to navigate and interpret the visual
domain. Leveraging pre-trained visual encoders like CLIP-ViT (Dosovitskiy et
al., 2020; Radford et al., 2021), MLLMs have extended the powerful textual
understanding of LLMs to multimodal scenarios, such as visual question
answering (Li et al., 2023a), visual conversations (Liu et al., 2023b), non-
verbal reasoning (Ahrabian et al., 2024), and multimodal in-context learning
(Alayrac et al., 2022; Zhao et al., 2023). To serve as multimodal agents (Yang
et al., 2023; Hong et al., 2023) and accomplish complex embodied tasks (Driess
et al., 2023; Mu et al., 2023), MLLMs need to recognize and interpret visual
information with different quality, size, and location, including large
central objects and small peripheral pieces of text.

Despite the remarkable advancements of current MLLMs, accurately identifying
small objects within images seems to remain a challenge. As Figure 1 shows,
the state-of-the-art GPT-4V (OpenAI, 2023) struggles to discern specific
details like small textual descriptions. Prior research suggests that
increasing the resolution of input images can generally enhance the response
accuracy towards the question (Bai et al., 2023; Yu et al., 2023a).
Furthermore, ViCrop (Zhang et al., 2023) and V*(Wu & Xie, 2023) have
respectively introduced methods for image cropping and visual searching to aid
MLLMs in recognizing finer details. However, the extent of this limitation and
the underlying factors that lead to this challenge have not been
systematically examined yet.

To bridge this gap, we quantitatively study MLLMs’ perceptual sensitivity to
relative object sizes and identify various visual factors that contribute to
this sensitivity. We conduct a comprehensive experiment with seven state-of-
the-art MLLMs on two common visual question-answering datasets, GQA (Hudson &
Manning, 2019) and TextVQA (Singh et al., 2019). By grouping the answers based
on the relative size of target objects, we observe a significant performance
drop with a decrease in object sizes, a trend that persists in all MLLMs.
Next, to identify the individual contribution of various visual factors on the
MLLMs’ ability to perceive small objects in images, we study four factors:
object quality, object size, object distractors, and object location. Our
controlled experiments yield the following findings regarding MLLMs’ visual
question answering performance:

  * •

Object quality (sampling rate) higher than a certain threshold does not affect
MLLMs’ performance, and this threshold seems to align well with human
perception.

  * •

Smaller object size, while controlling for object quality, results in a
significant decline in MLLMs’ performance. This trend is less apparent in
models enhanced by training on data containing annotation of smaller objects.

  * •

The presence of visual distractors can reduce MLLMs’ performance.

  * •

The performance of MLLMs fluctuates significantly with the location of the
object (visual target of the question) in the image.

The significance of these findings is threefold. First, our results suggest
that MLLMs should be used with caution, especially when the task relies on
accurately identifying visual details. Second, our findings provide novel
insights for developing more reliable MLLMs, especially when dealing with data
of lower quality, objects of smaller size, various distractors, and specific
object positions. Third, we provide a new evaluation protocol for studying
future MLLMs. This protocol can be applied, for example, to measure the
robustness of an MLLM in response to different positions by showing the
difference between maximum and minimum performance across different object
locations.

##  2 Related work

Multimodal Large Language Model. MLLMs like GPT-4V (OpenAI, 2023) and Gemini-
pro-vision (Team et al., 2023) demonstrate a strong capability for visual
understanding. MLLMs typically have three primary components: a vision
encoder, a bridge module, and an LLM backbone (Yu et al., 2023a). (1) Vision
Encoder: Commonly, MLLMs utilize CLIP-ViT (Radford et al., 2021) as the vision
encoder, which divides the input image into patches and feeds them into
Transformer blocks sequentially in a raster-scan order. (2) Bridge Module: The
resulting visual features from the vision encoder are then either linearly
projected (Liu et al., 2023b) or condensed into a fixed-sized representation
(Li et al., 2023a) to align with the textual representation space. (3) LLM
Backbone: The transformed visual features are then prepended to the text
embedding within the LLM. We consider seven state-of-the-art MLLMs in this
work, whose architectures are summarized in Table 1. Both BLIP-2 (Li et al.,
2023a) and InstructBLIP (Dai et al., 2023) utilize the Q-Former as a bridge
module, while InstructBLIP integrates instructions into the Q-Former for an
instruction-awarding visual feature. LLaVA-1.5 (Lin et al., 2023a) projects
the visual feature from ViT into the LLM space with an MLP layer. Qwen-VL-Chat
(Bai et al., 2023) chooses a larger vision encoder ViT-bigG and a one-layer
cross-attention module to perceive visual features. Fuyu-8B (Bavishi et al.,
2023) uniquely removes the external vision encoder, directly incorporating
pixel information into the language decoder. The training of MLLMs typically
undergoes an initial pre-training on extensive image-text datasets such as
LAION (Schuhmann et al., 2022), followed by specialized multimodal instruction
tuning (Liu et al., 2023b). Enhancements in MLLMs have been pursued through
various means, including increasing image resolution (Yu et al., 2023a),
scaling data and model size (Wang et al., 2023), extending to multilingual
context (Hu et al., 2023), and introducing interleaved data formats (Lin et
al., 2023b).

Table 1: Architectural overview of the MLLMs used in this study.

Model |  Vision Encoder |  Bridge Module |  LLM Backbone
---|---|---|---
BLIP-2 | ViT-g | Q-Former | Flan-T5XXL
InstructBLIP | ViT-g | Q-Former | Vicuna-13B
LLaVA-1.5 | ViT-l | Linear | Vicuna-13B
Qwen-VL-Chat | ViT-bigG | Resampler | Qwen-7B
Fuyu-8B | - | - | Persimmon-8B
GPT-4V | Not released
Gemini-Pro-Vision | Not released

Robustness Analysis to MLLMs. The capabilities of MLLMs have been evaluated
using general benchmarks like the traditional VQA benchmark VQAv2 (Antol et
al., 2015) and GQA (Hudson & Manning, 2019), alongside newer benchmarks such
as MM-Bench (Liu et al., 2023c) and MMMU (Yue et al., 2023). Some works have
shown that MLLMs suffer from object hallucination (Li et al., 2023b; Yu et
al., 2023b) and a lack of robustness in processing visual details (Zhang et
al., 2023). The MMVP benchmark (Tong et al., 2024) further highlights these
visual shortcomings, particularly emphasizing the discrepancy between the
embedding spaces of CLIP and the vision-only self-supervised space of DINOv2.
The V* algorithm (Wu & Xie, 2023) offers an innovative approach with its LLM-
guided visual search method, specifically targeting the focus on visual
details. Our paper builds upon these insights, quantitatively exploring MLLMs’
performance in handling visual details, considering four distinct factors.

##  3 Can MLLMs Perceive Small Objects?

Figure 2:  The performances of multiple popular MLLMs on GQA and TextVQA show
a clear positive correlation with relative size of target objects. The
accuracy is computed with inclusion match. *A small part of questions is
skipped due to safety policy of API models. †The model has been reported to be
trained on the dataset.  Table 2: Intervals of the number of pixel values (#
Pixels) within the bounding box after the input images are unified to size
224$\times$224 on 5 data quantiles of GQA and TextVQA tested in Section 3, and
the average number of object/textual distractors (# D) in two datasets.
Data | GQA | TextVQA
---|---|---
Quantile | # Pixels | # D | # Pixels | # D
1 | [100, 1409] | 19.2 | [26, 80] | 14.9
2 | [1409, 5043] | 18.1 | [80, 143] | 13.8
3 | [5045, 11967] | 17.4 | [143, 296] | 13.1
4 | [11971, 24571] | 16.7 | [296, 697] | 11.7
5 | [24588, 50176] | 14.6 | [698, 50176] | 7.6

Recent anecdotal evidence (Zhang et al., 2023) suggests that MLLMs face
challenges in perceiving small visual details compared to larger ones.
Inspired by this research, we conduct an extensive quantitative experiment to
study the sensitivity to size of recent SOTA MLLMs on two standard VQA
benchmarks. We evaluate the seven representative models shown in Table 1 on
two prominent visual question-answering datasets, GQA (Hudson & Manning, 2019)
for compositional reasoning on real-world objects and TextVQA (Singh et al.,
2019) for reading and comprehending texts presented in the real-world image.
Both datasets offer the advantage of bounding box annotations, pinpointing
areas of interest within images. For GQA, we aggregate bounding boxes
encompassing all related objects. For TextVQA, we focus on the bounding box
with the highest textual similarity to the ground-truth answer. To facilitate
a nuanced assessment, we categorize these datasets into quintiles based on the
relative size of the target area. The accuracy is measured via inclusion match
(Liu et al., 2023d) (see Appendix B for exact match results).

Our findings, depicted in Figure 2, demonstrate a consistent issue across all
models: a marked decline in processing accuracy for smaller visual elements.
Such a trend is most notable in BLIP-2, whose performance gap across different
quantiles is 16.71% and 21.83% on GQA and TextVQA, respectively. In addition,
the two leading closed-source API models, GPT-4V and Gemini-provision, have a
7.32% and 6.39% performance gap on GQA and 9.05% and 3.31% on TextVQA,
respectively, also exhibiting performance gaps.

To better understand the underlying reasons affecting the performance in
Figure 2, we compute the range of the number of pixels within each quantile
after unifying the input images to 224$\times$244 in Table 2. For models with
higher resolutions, the number of pixels increases proportionally. In TextVQA,
the numbers of pixels of the target texts on the first three quantiles are
notably limited, indicating that the information of the target object only
accounts for a small portion of the whole input. Meanwhile, the limited number
of pixels also causes a low image quality presenting the target object.
Furthermore, we also compute the average number of distracting OCR tokens (OCR
tokens that are not related to the answer) in each quantile of TextVQA and the
average number of distractor objects in GQA, the decreased number of
distractors could also potentially contribute to Figure 2.

Based on the analysis, we summarize the underlying reasons for such
limitations as four potential factors. The following section first
systematically formulates these factors and then explores their impact on the
MLLMs’ capacity to recognize and interpret small visual objects.

Figure 3:  An illustration of the Downsample-Upsample (upper) and Crop-
Upsample (lower) procedure described in Section 4.2 and 4.3. The upper process
reduces object quality 6 times while keeping the same size and position. The
lower increases object size three times while keeping the object quality.

##  4 What Factors Affect MLLMs’ Perception of Small Objects?

We focus on the following four factors: object quality, object size, object
distractors, and object location. While the identified factors are by no means
exhaustive, they aim to illuminate some of the fundamental perceptual
limitations of current MLLMs, thereby informing both practical applications
and future enhancements of these models.

Figure 4:  The effect of changing text sampling rate (quality) on model’s
performance of reading texts while keeping the size of the text. It is
noticeable that from the sampling rate of 8 (marked as red), the image starts
to become fully recognizable as ‘5934549’.

Object Quality. We define quality as the original sampling rate of an object
(in pixels per inch, or pixels per vector graphic range), that is, the
original resolution of an object in a given image. To vary object quality, we
adopt a downsample-upsample strategy on an original high-resolution image of
the object, which is illustrated in the upper part of Figure 3. Starting from
an original 300-pixel by 300-pixel raster image of a vector graphic digit
($D_{orig}$), we reduce its quality six times by down-sampling that raster
image to 50 pixels by 50 pixels ($D_{down}$). Then we upsample the $D_{down}$
six times, and the resulting $D_{down\\_up}$ reaches the same image size with
$D_{orig}$, but a six times lower sampling rate. Note that image upsampling
does not inherently change the sampling rate of the object despite the
increase in pixel values. In this paper, we use the terms ‘sampling rate’ and
‘quality’ interchangeably.

Object Size. The object size is defined as the number of pixels that belong to
an object in the input image to MLLMs. Note that we can modify the object size
while keeping its quality constant by upsampling the object to the desired
size. To this end, we adopt a crop-upsample strategy, as is illustrated in
Figure 3 (lower). Given a 300-pixel by 300-pixel raster image of a digit (of a
particular quality due to the original sampling rate), we crop the $D_{orig}$
at the center to 100 pixels by 100 pixels ($D_{crop}$). Then we upsample the
$D_{crop}$ three times, resulting $D_{crop\\_up}$ with the same sampling rate
and image pixel size with $D_{orig}$, while having a three times larger object
size.

Object Distractors. Object distractors are objects that belong to the same
distribution as a target object of interest (e.g.,, other numbers when the
object of interest is a particular number in the image).

Object Location. Current MLLMs share the same manner for image processing,
where a complete image is divided into numerous patches, which are
subsequently transformed into individual image tokens. Formally, the input
image $\mathrm{x}\in\mathbb{R}^{H\times W\times C}$ with spatial dimensions
$(H,W)$ and $C$ color channels is first reshaped into $2D$ patches
$\mathrm{x_{p}}\in\mathbb{R}^{N\times P^{2}\times C}$, and the resulting $N$
image patches are mapped to $N$ token embedding as the input of transformer
architectures. Given the architecture, an input object could be cut by image
patch boundaries and divided into different image patches. In light of this,
we investigate two complementary location-related factors: the global location
on the image and the local patch boundary cut on the target object.

###  4.1 Experimental Setup

Text-Reading Objective. In our experiments, we focus exclusively on the text-
reading ability of MLLMs. This decision is driven by the idea that text
reading involves recognizing diverse shapes and their spatial relationships,
providing a clear and definitive framework for assessment. Compared to other
visual tasks like identifying object colors or types, text recognition offers
reduced ambiguity in evaluation. To facilitate controlled comparisons, we use
synthetic digital texts, rendered in the widely used Arial sans-serif font,
and overlaid on plain white backgrounds. Here, the ‘sampling rate’ is defined
in terms of the font size used during text creation, which correlates with the
vertical pixel count of the text characters. During the evaluation, the
accuracy of the MLLMs’ responses is assessed against the actual text in the
images using Gestalt Pattern Matching (GPM) (Ratcliff et al., 1988). This
metric is a widely used smooth metric for OCR task assessments.

Evaluated Models. Due to the prohibitive cost of running granular experiments
on commercial MLLMs, we will consider the five open-source models as
representative examples of current MLLMs: BLIP2 (Li et al., 2023a),
InstructBLIP (Dai et al., 2023), LLaVA-1.5 (Liu et al., 2023a), Qwen-VL-Chat
(Bai et al., 2023) and Fuyu-8B (Bavishi et al., 2023). The architectures of
five models are introduced in Section 4. Notably, BLIP-2 has not been
explicitly trained on OCR-oriented tasks, relying instead on image-text pairs
with text annotations within the images. InstructBLIP and LLaVA-1.5 have
undergone training on several OCR-oriented tasks, including OCR-VQA (Mishra et
al., 2019) and TextCaps (Sidorov et al., 2020). Qwen-VL-Chat, having been
trained on a substantial 25M OCR-oriented dataset, demonstrates enhanced OCR
capabilities, and is thus referred to as an OCR-enhanced-MLLM in our analysis.
The training specifics for Fuyu-8B are not publicly disclosed, but based on
its performance, we presume its OCR training to be similar to that of Qwen-VL.

###  4.2 Quality Sensitivity Study

Our goal in this section is to study the ability of MLLMs in reading small
text of varying quality (sampling rates). We adopt the Downsample-Upsample
strategy which is described in Figure 3 and construct a dataset with a
sampling rate from 2 to 20 in increments of 2, examples are shown at the
bottom of Figure 4. Our experimental tasks involve reading 3, 5, and 7 digits,
signifying three tiers of task complexity, placed at the center of an image.
Each tier includes 500 random numbers to read. We prompt MLLMs with the
question “What is the number on the image?”.

MLLMs’ response to object quality is threshold-dependent. As shown in Figure
4, we observed a significant improvement in the MLLMs’ performance as the
sampling rate increased from 4 to 8. However, after this point, the
performance stabilized with increasing sampling rate, indicating a threshold-
dependent trend in the MLLMs’ ability to read text of varying qualities.

The threshold is universal and aligns well with human perception. Remarkably,
the threshold of a sampling rate of 8 is consistently observed across all MLLM
models, irrespective of their text recognition capabilities and the varying
levels of task complexity. This threshold seems to be consistent with human
perceptual ability, as it becomes hard to read text below this threshold for
our own eyes. These findings suggest that the MLLMs’ response to image quality
is more influenced by the intrinsic properties of the images rather than the
internal differences among the MLLMs. Considering this threshold-dependent
performance improvement, the continuous improvement in performance within
image size observed in Figure 2 cannot be solely attributed to image quality
improvements. In the following sections, we conduct further experiments to
investigate other factors that can affect the perception of small objects by
MLLMs.

###  4.3 Size Sensitivity Study

Figure 5:  The effect of changing text size on model’s performance of reading
texts while keeping the sampling rate of the text.  Figure 6:  The effect of
changing the number of distractors on MLLMs’ performance of reading texts.

In the preceding section, we observed that the sampling rate of text does not
significantly challenge MLLMs after a certain threshold. This leads us to
inquire about the impact of object size on MLLMs’ performance with a fixed
sampling rate (quality). To explore this, we follow the Crop-Upsample strategy
described in Figure 3. Specifically, for $D_{orig}$, we place an 8-font size
text in the center of the image, then in $D_{crop\\_up}$ the original text is
enlarged 1 to 5.5 times, with a step of 0.5, illustrated at the bottom of
Figure 5. The tasks include recognizing 500 random numbers with 3, 5, and 7
digits following Section 4.2. We prompt MLLMs with the question “What is the
number on the image?”.

At a fixed object quality, most MLLMs perform better at recognizing larger
objects. As shown in Figure 5, except for the OCR-enhanced model Qwen-VL-Chat,
the performance of MLLMs improves with the increase of object size while
maintaining a constant quality (sampling rate). Notably, the performance
trajectory of Fuyu-8B exhibits a significant enhancement in the early stages
of size increase. In contrast, BLIP-2 and InstructBLIP show a more gradual
improvement in performance with increasing object size. LLaVA-1.5, however,
demonstrates a relatively stable performance across varying sizes, indicating
a lesser sensitivity to changes in object size. Furthermore, we observe that
for tasks with greater complexity (recognizing more digits), the increase in
object size has a larger impact on the models’ accuracy. This phenomenon may
be attributed to two reasons. First, larger object sizes occupy more image
patches. These patches translate into transformer tokens, which, during the
self-attention mechanisms of the transformer architecture, allow for a more
extensive fusion of information. Second, the majority of MLLM image-text
matching data for pre-training, only present textual descriptions for the main
visual components in the image which are often larger, diminishing their
capability of perceiving smaller objects. The second reason is supported by
the fact that the OCR-enhanced model Qwen-VL-Chat, which is trained on large-
scale synthetic data with 41 English fonts and 11 Chinese fonts, maintains its
accuracy when processing smaller objects.

###  4.4 Distractor Sensitivity Study

Table 3: The input image patch number and patch size of the MLLMs considered in our experiment. *Fuyu-8B has a fixed patch size of 30$\times$30 but does not have a fixed patch number. We set it to 10$\times$10 in our experiment. Model |  Patch Number |  Patch Size |  Resolution
---|---|---|---
BLIP-2 |  16$\times$16 |  14$\times$14 |  224$\times$224
InstructBLIP |  16$\times$16 |  14$\times$14 |  224$\times$224
LLaVA-1.5 |  24$\times$24 |  14$\times$14 |  336$\times$336
Qwen-VL-Chat |  32$\times$32 |  14$\times$14 |  448$\times$448
Fuyu-8B |  10$\times$10* |  30$\times$30 |  300$\times$300*
Figure 7:  The effect text (number) location in the image on MLLMs’ ability to
read the text correctly, with and without distractors (bottom and top,
respectively). Higher values are presented in lighter colors.  Figure 8:  The
performance of MLLMs in text recognition tasks demonstrates notable
variability when textual content is vertically (left) and horizontally (right)
cut by image patch boundaries. Gray area indicates that the target texts are
cut by a patch boundary. We provide two local illustrations below showing that
a text is shifted between two adjacent image patches. Due to space
constraints, we only present the middle part of the entire shifting (range
ratio from 0.25 to 0.75), the complete plots are presented in the Figure 9.

Small objects in an image, in addition to the inherent effect of their size we
observed in the previous section, can also affect MLLMs’ perception by
allowing for the presence of more distractors in the image. Our goal in this
section is to study the effect of distractors on MLLMs’ perception of small
objects. To that end, we place the answer text (number) at the center of the
image, then we introduce $k$ distractor numbers, positioning them at random
locations throughout the image. The answer digit text is assigned to the
variable ‘a’, while the distractor numbers are assigned to ‘b’ and subsequent
letters (‘c’, ‘d’……). We vary the number of distractors from 0 to 9, and
prompt MLLMs with ”What is the number assigned to variable ‘a’ in the image?”.
We experiment with text font sizes 8 and 12 without resampling to gain 2 tiers
of task difficulty, each tier including 100 random numbers (3 digits) to read,
and the random position of distractors for each number is varied 5 times.

Increasing the number of distractors makes perception harder for MLLMs. As
shown in Figure 6, the increase in the number of distractors consistently
decreases MLLMs’ performance regardless of their overall performance.
Specifically, the OCR-enhanced MLLM Qwen-VL-Chat reaches a perfect score
across varying distractor numbers on font size 12, while facing a 10-point
performance drop during the increase of distractor numbers on font size 8.
Among the other models, Fuyu-8B, InstructBLIP, and BLIP-2 present heightened
sensitivity to the additional distractors while LLaVA keeps a relatively minor
performance drop. It is worth noting that although Fuyu-8B has superior
performance over LLaVA-1.5 in Figure 5, it appears to lack robustness when
facing more complex visual questions.

###  4.5 Location Sensitivity Study

Another factor that can significantly vary for small objects is their location
in the image, which can in turn affect MLLMs’ perception. We study two
complementary location-related factors in this section: the global location on
the image and the local patch boundary cut on the target object (described in
detail at the start of Section 4).

####  4.5.1 Global location

Table 3 outlines the patch sizes and counts of the MLLMs evaluated in our
study. To augment patch capacities, we amalgamate every four adjacent
$14\times 14$ image patches from models like BLIP-2, InstructBLIP, LLaVA-1.5,
and Qwen-VL-Chat into a single $28\times 28$ patch. Texts are centrally placed
within each merged patch, maintaining a consistent sampling rate of 8. In this
experiment, following the setting of Section 4.4, we examine MLLMs’ text
recognition and localization performance under variations in distractor
presence and global text positioning. For assessing MLLMs’ capabilities, we
introduce scenarios with zero and $k$ distractors—zero distractors that
evaluate pure text recognition ability across different image locations and
$k$ distractors that require localizing the target text. Specifically, the
OCR-enhanced Qwen-VL-Chat model is tested with nine distractors, while all
other models with one distractor. We include 100 random numbers (3 digits)
placed all through the image patches. We prompt MLLMs with the question ”What
is the number assigned to variable ‘a’ in the image?” during evaluation.

MLLMs exhibit inconsistent text recognition and localization performance
across different global locations. It is observed that the majority of models,
except LLava-1.5, encounter challenges in recognizing or localizing text on
the right side of an image. Moreover, BLIP2 and InstructBLIP also experience
difficulties with text on the left side. Notably, the OCR-enhanced model Qwen-
VL-Chat, despite obtaining a near-perfect score in most locations,
demonstrates a significant performance disparity of 58 points across different
locations. Also, Fuyu-8B experiences a sharp decrease in its performance in
the first row. This observation suggests that MLLMs are susceptible to
positional bias when processing images. While including more training datasets
can lead to much better overall performance, performance drops on certain
image regions still exist.

####  4.5.2 Local Patch Boundary Cut

We construct a dataset where the generated digital text gradually crosses an
image boundary. For vertical patch boundary cut, the digit text is anchored at
a predetermined vertical location, while being horizontally moved across the
full span of the image. For horizontal cuts, the digit text is fixed at a
specific horizontal position and moved vertically. An illustrative example of
vertical cut is shown at the bottom of Figure 8. We determine the number of
reading digits depending on the maximum digit capacity for a single image
patch, specifically setting at six digits for Fuyu-8B and three digits for the
remaining models. We include 100 random numbers for each experiment. We prompt
MLLMs with ”What is the number on the image?” during evaluation.

Model’s performance is lower when target objects remain undivided by patch
boundaries. For image patch boundary vertical cutting, as observed in Figure 8
(left), a common trend among all models is the performance decline at the
center of the patch, where texts remain undivided by patch boundaries(white
parts). Notably, although presenting a near-perfect score, Qwen-VL-Chat still
presents an around 10 percent gap between different patch boundary cuts. The
only model that does not show this trend is Fuyu-8B - we assume this is due to
its enlarged patch size, making the performance inside an image patch more
robust. This phenomenon indicates that contrary to intuitions, texts divided
across multiple patches may be more effectively recognized by MLLMs.
Therefore, even with the same size and quality, small objects seem to be more
recognizable by MLLMs when they are divided into different image patches.

Horizontal cuts hurt the performance more than vertical cuts. Figure 8 (right)
demonstrates the performance of the five models when the target text is
horizontally cut by a patch boundary. Consistent with vertical cuts, in
LLaVA-1.5, we observe a notable performance peak at the boundary cuts.
However, the remaining models do not show such a trend. We hypothesize two
factors contributing to this observation. First, at the horizontal cut, all
characters presented are divided into two separate parts, while the vertical
cut divides at most only one character into different patches. This effect
potentially diminishes the completeness of shape information. Second, for the
horizontal cut, the two resulting image tokens are positioned further apart
after the image is translated into sequence input of transformers; for the
vertical cut, the two corresponding image patches remain continuous in the
resulting sequence.

##  5 Conclusion

In this paper, we expose notable limitations of current MLLMs on perceiving
small visual details. To gain a further understanding of the limitation, we
identify four independent relevant factors: object quality, size, distractors,
and location. We extensively explore the effect of each of the factors by
conducting carefully controlled intervention studies. Based on our study, we
suggest that: 1) Object quality does not pose an additional obstacle for MLLMs
after a certain threshold, however, object quality should be carefully
considered when images are resampled before feeding to MLLMs; 2) most MLLMs
fall short in perceiving small objects, even with enough object quality,
explicit training could potentially overcome this gap; 3) MLLMs’ performance
is significantly affected by the target objects’ global location in the image;
4) MLLMs are good at recognizing small objects that are divided into more
image tokens, while horizontal cutting could hurt performance due to the
distance between vertically adjacent image patches. In addition to the
findings, our study also provides a new evaluation protocol for the future
enhancement of MLLMs’ perception.

## Impact Statement

This research identifies and analyzes critical limitations in Multimodal Large
Language Models (MLLMs) regarding the recognition of small visual details,
emphasizing the roles of object quality, size, distractors, and location. Our
findings will potentially offer insights to improve visual processing
capabilities. The introduction of a new evaluation protocol provides a
foundation for future advancements, aiming to enhance MLLMs’ applicability in
diverse real-world scenarios. This work contributes to the development of more
robust and reliable MLLMs.

## References

  * Ahrabian et al. (2024) Ahrabian, K., Sourati, Z., Sun, K., Zhang, J., Jiang, Y., Morstatter, F., and Pujara, J.  The curious case of nonverbal abstract reasoning with multi-modal large language models.  _arXiv preprint arXiv:2401.12117_ , 2024.
  * Alayrac et al. (2022) Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Antol et al. (2015) Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D.  VQA: Visual Question Answering.  In _International Conference on Computer Vision (ICCV)_ , 2015.
  * Bai et al. (2023) Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  _arXiv preprint arXiv:2308.12966_ , 2023.
  * Bavishi et al. (2023) Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., and Taşırlar, S.  Introducing our multimodal models, 2023.  URL https://www.adept.ai/blog/fuyu-8b.
  * Dai et al. (2023) Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S.  InstructBLIP: Towards general-purpose vision-language models with instruction tuning, 2023.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Proceedings of ICLR_ , 2020.
  * Driess et al. (2023) Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Hong et al. (2023) Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al.  Cogagent: A visual language model for gui agents.  _arXiv preprint arXiv:2312.08914_ , 2023.
  * Hu et al. (2023) Hu, J., Yao, Y., Wang, C., Wang, S., Pan, Y., Chen, Q., Yu, T., Wu, H., Zhao, Y., Zhang, H., Han, X., Lin, Y., Xue, J., Li, D., Liu, Z., and Sun, M.  Large multilingual models pivot zero-shot multimodal learning across languages.  2023\.
  * Hudson & Manning (2019) Hudson, D. A. and Manning, C. D.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 6700–6709, 2019.
  * Li et al. (2023a) Li, J., Li, D., Savarese, S., and Hoi, S.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), _Proceedings of ICML_ , volume 202, pp. 19730–19742. PMLR, 23–29 Jul 2023a.
  * Li et al. (2023b) Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R.  Evaluating object hallucination in large vision-language models.  In Bouamor, H., Pino, J., and Bali, K. (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 292–305, Singapore, December 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.20.  URL https://aclanthology.org/2023.emnlp-main.20.
  * Lin et al. (2023a) Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., and Yuan, L.  Video-llava: Learning united visual representation by alignment before projection.  _arXiv preprint arXiv:2311.10122_ , 2023a.
  * Lin et al. (2023b) Lin, J., Yin, H., Ping, W., Lu, Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi, M., and Han, S.  Vila: On pre-training for visual language models.  _arXiv preprint arXiv:2312.07533_ , 2023b.
  * Liu et al. (2023a) Liu, H., Li, C., Li, Y., and Lee, Y. J.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_ , 2023a.
  * Liu et al. (2023b) Liu, H., Li, C., Wu, Q., and Lee, Y. J.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023b.
  * Liu et al. (2023c) Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.  Mmbench: Is your multi-modal model an all-around player?  _arXiv preprint arXiv:2307.06281_ , 2023c.
  * Liu et al. (2023d) Liu, Y., Li, Z., Li, H., Yu, W., Huang, M., Peng, D., Liu, M., Chen, M., Li, C., Jin, L., and Bai, X.  On the hidden mystery of ocr in large multimodal models.  _ArXiv_ , abs/2305.07895, 2023d.  URL https://api.semanticscholar.org/CorpusID:258685422.
  * Mishra et al. (2019) Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A.  Ocr-vqa: Visual question answering by reading text in images.  In _2019 international conference on document analysis and recognition (ICDAR)_ , pp. 947–952. IEEE, 2019.
  * Mu et al. (2023) Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., and Luo, P.  EmbodiedGPT: Vision-language pre-training via embodied chain of thought.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * OpenAI (2023) OpenAI.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , pp. 8748–8763. PMLR, 2021.
  * Ratcliff et al. (1988) Ratcliff, J. W., Metzener, D., et al.  Pattern matching: The gestalt approach.  _Dr. Dobb’s Journal_ , 13(7):46, 1988.
  * Schuhmann et al. (2022) Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.  LAION-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Sidorov et al. (2020) Sidorov, O., Hu, R., Rohrbach, M., and Singh, A.  Textcaps: a dataset for image captioning with reading comprehension.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pp. 742–758. Springer, 2020.
  * Singh et al. (2019) Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M.  Towards vqa models that can read.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8317–8326, 2019.
  * Team et al. (2023) Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Tong et al. (2024) Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S.  Eyes wide shut? exploring the visual shortcomings of multimodal llms.  _arXiv preprint arXiv:2401.06209_ , 2024.
  * Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.  LLaMA: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. (2023) Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., Xu, J., Xu, B., Li, J., Dong, Y., Ding, M., and Tang, J.  Cogvlm: Visual expert for pretrained language models, 2023.
  * Wu & Xie (2023) Wu, P. and Xie, S.  V*: Guided visual search as a core mechanism in multimodal llms.  _arXiv preprint arXiv:2312.14135_ , 2023.
  * Yang et al. (2023) Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., Fu, B., and Yu, G.  Appagent: Multimodal agents as smartphone users.  _arXiv preprint arXiv:2312.13771_ , 2023.
  * Yu et al. (2023a) Yu, T., Hu, J., Yao, Y., Zhang, H., Zhao, Y., Wang, C., Wang, S., Pan, Y., Xue, J., Li, D., Liu, Z., Zheng, H.-T., and Sun, M.  Reformulating vision-language foundation models and datasets towards universal multimodal assistants.  2023a.
  * Yu et al. (2023b) Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., Sun, M., et al.  Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.  _arXiv preprint arXiv:2312.00849_ , 2023b.
  * Yue et al. (2023) Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  _arXiv preprint arXiv:2311.16502_ , 2023.
  * Zhang et al. (2023) Zhang, J., Khayatkhoei, M., Chhikara, P., and Ilievski, F.  Visual cropping improves zero-shot question answering of multimodal large language models.  _Advances in Neural Information Processing Systems Workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models_ , 2023.
  * Zhao et al. (2023) Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., Liu, Z., Wang, S., Han, W., and Chang, B.  Mmicl: Empowering vision-language model with multi-modal in-context learning.  _arXiv preprint arXiv:2309.07915_ , 2023.

##  Appendix A Complete result of patch boundary cut.

Figure 9 shows the complete result of horizontal (upper) and vertical (lower)
cuts, the overall trend stays the same.

Figure 9:  Complete result of vertical and horizontal cut.

##  Appendix B Result from GQA and TextVQA on different matching strategies.

In addition to inclusion matching, in Figure 10, we use exact string matching
to compute the accuracy. The most notable difference is that some models’
performance diminishes, as their output do not follow the dataset format
strictly. Despite the above, the overall trend that most of the models have
difficulty perceiving smaller details stays the same.

Figure 10:  The performances of 7 MLLMs on GQA and TextVQA quantiles by object
relative size. The models’ predictions are computed with exact matching. The
overall trend stays the same despite some variances in performance. *A small
part of the dataset is skipped due to safety policy of API models. †The model
has been reported to be trained on the dataset.

##  Appendix C Further analysis on the role of object distractors.

Table 2 presents the number of distractors in each group of TextVQA and GQA,
from which it’s also clear that the number of distractors decreases when the
object’s relative size gets larger. Hence, it is unclear which factor plays
the most important role in Figure 2. To this end, we divide the GQA and
TextVQA into five quantiles by the number of object/OCR token distractors, the
result of both metrics is presented in Figure 12. From the plot, we observe
that object distractors in GQA seem to affect the MLLM’s performance, while in
TextVQA, we do not observe a clear correlation between number of distractors
and performance.

##  Appendix D What is causing the variance in positional bias?

The different positional biases observed from Section 4.5 may stem from the
bias from textual training data. Typically, textual content in training
datasets is oriented from left to right and concentrated towards the center of
images. This common formatting convention may inadvertently lead to the under-
representation of text located on the right side of images and along their
margins.

Figure 11: MLLMs’ performance on TextVQA and GQA quantiles divided by number
of distractors, accuracy is computed using exact matching.

Figure 12: MLLMs’ performance on TextVQA and GQA quantiles divided by number
of distractors, accuracy is computed using inclusion matching.

##  Appendix E Why does Fuyu-8B have a noticeable low performance in its first
row?

In Figure 7, we notice a sharp decrease in Fuyu-8B’s performance score within
the first row. We assume this unexpected phenomenon is related to its unique
pure transformer decoder architecture. To this end, we choose several images
and present the attention map of Fuyu-8B, providing observations for further
investigation.

Figure 13: The attention map of Fuyu-8B on six different input images.
Detailed descriptions in Appendix E.

In Figure 13, we provide the attention map for each of the 36 layers of
Fuyu-8B. The input image is the synthetic image we construct in the location
study in Section 4.5, where a single ‘a=665’ is placed in an image patch’s
center. The position of the patch is: 0, 4, 9, 19, 49, 99, in the raster scan
order of the original image (with $10\times 10$ image tokens), the input
position can also be seen from the yellow attention outlier in Layer 1. The
attention map is computed for the next token after promoting Fuyu-8B with
‘Question: What is the number in the image? Short answer:’ and we track the
attention of the next token with respect to each image patch. From the
attention map, we can tell that ranging from approximately 13-27 layers, for
the image whose text is placed in $i_{th}$ position, there are consistently
high attention values in the first $k$ tokens, where $k=i$ if $i\leq 9$
otherwise $k=9$. Such a result could be linked to the low performance observed
in the first row since the high attention among those layers stays consistent
within the tokens in the first row. For the deeper reason behind the
phenomenon, we leave them as open future works.