# Prediction of flow dynamics using point processes

Yoshito Hirata  Institute of Industrial Science, The University of Tokyo,
4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan  Thomas Stemler  School of
Mathematics and Statistics, The University of Western Australia, Crawley, WA
6009, Australia  Potsdam Institute of Climate Impact Research, P.O. Box 60 12
03, 14412 Potsdam, Germany  Deniz Eroglu  Potsdam Institute for Climate Impact
Research, P.O. Box 60 12 03, 14412 Potsdam, Germany  Department of Physics,
Humboldt University of Berlin, Newtonstrasse 15, 12489 Berlin, Germany
Norbert Marwan  Potsdam Institute of Climate Impact Research, P.O. Box 60 12
03, 14412 Potsdam, Germany

###### Abstract

Describing a time series parsimoniously is the first step to study the
underlying dynamics. For a time-discrete system, a generating partition
provides a compact description such that a time series and a symbolic sequence
are one-to-one. But, for a time-continuous system, such a compact description
does not have a solid basis. Here, we propose to describe a time-continuous
time series using a local cross section and the times when the orbit crosses
the local cross section. We show that if such a series of crossing times and
some past observations are given, we can predict the system’s dynamics with
fine accuracy. This reconstructability does not depend strongly on the size
nor the placement of the local cross section if we have a sufficiently long
database. We demonstrate the proposed method using the Lorenz model as well as
the actual measurement of wind speed.

Current developments of measurement techniques and hardware enable us to
record time-continuous data with very high sampling rates for long times. To
understand such data intuitively, we need to describe the data parsimoniously
so that such description can reproduce the original time series. Thus, here we
propose to represent such time-continuous data only by the series of times
when the orbit passes a local cross section. We show that a time series
together with some measurements of the past dynamics of the system is
sufficient information to predict the future dynamics of the system. There are
two applications: an immediate application is to save, or send, time-
continuous data using small memory or low channel capacities and therefore
make big data more manageable; a more abstract application is to approximate
details in a time series when only few observations are possible, based on
detailed measurements taken during a different time.

Recent advances in measurement technology enable us to record data of time-
continuous systems with very high sampling rates. While these advances are
welcome, there are several problems in accessing, analysing and even storing
such data sets. These problems are nowadays summarised under the umbrella of
_big data_ and arise in diverse areas ranging from bioinformatics to systems’
health engineering Bio ; health . Here we are presenting a method based on the
recurrence of dynamical systems to categorise and rank such data sets. Ranking
the recurrence times of our current state against the recurrence times of the
big data set, allows us to predict the future dynamics of the system.

Our research is motivated by recent achievements in the time series analysis
of non-uniformly sampled data Irregular1 ; Irregular2 ; hirata2009 ;
hirata2012 ; hirata2014 ; ozken2015 ; eroglu2016 . We assume, we have a time-
continuous dynamical system, that we are able to measure for a certain amount
of time. This observational series of the high dimensional state space is our
database. After recording the full dynamics in the phase space for our
database, we no longer measure the full dynamics but record the crossing
times. These crossing times are the times when the trajectory of the system
passes through a local cross section on the attractor of the dynamical system.
This data set is our non-uniform sampled data set, which we use to estimate
the current state of the system. This estimation is done by finding
similarities between the current crossing time sequence and the crossing times
calculated from the database of the full dynamics.

To detect the similarity we facilitate the Victor and Purpura distance
victor1997 . The information contained in such crossing times has been studied
since Poincaré him  and is the foundation of recurrence–based time series
analysis NM . The Victor and Purpura distance victor1997  offers a natural
metric to rank and detect similarities between consecutive time windows of the
crossing times victor1997 ; hirata2009 ; hirata2012 ; hirata2014 ; ozken2015 ;
hirata2016 ; eroglu2016 . Using cross prediction levanquyen1999 , we exploit
these similarities to estimate the current state and predict the future
system’s dynamics.

Our paper is organised as follows. After giving a short summary of our method,
we first give details on the information contained in the crossing time series
and introduce the Victor and Purpora distance. Then we introduce the cross
prediction method and illustrate our method using the low-dimensional Lorenz
system lorenz1963 . In addition we apply our method to predict actual wind
speed data.

Predicting the dynamics of the system will be done by facilitating the
information in the database. This record of the past dynamics is used in two
ways. First to compare the crossing time series of the database with our
current crossing time series. Second we predict the future dynamics by
approximating the future state of the system from the past dynamics in the
database. The length of the database is critical and determines the quality of
our prediction. The recording time has to be chosen long enough to allow first
a good state estimation and then a realistic approximation of the future
dynamics. Since our state estimation as well as the cross prediction rely on
similarity between past and future dynamics the longer the record the more
reliable the prediction becomes. One of the main results of this paper is that
even for realistic short recording times our method results in quite reliable
predictions.

Formally we assume a dynamical system $f_{t}:M\rightarrow M$, where $M$ is an
$m$-dimensional manifold and $f_{t}$ is the diffeomorphism representing how a
point $y$ in $M$ moves to a point $f_{t}(y)$ in $M$ after some time $t$. Our
database contains the full dynamics of $y(t)$ for $t=[0,\Delta t,2\Delta
t,\ldots{},L]$, where $L$ denotes the recording time and $\Delta t$ is the
sampling time. If the full dynamics cannot be directly measured, one has to
reconstruct the dynamics using delay embedding takens1981 ; sauer1991 . We
will discuss this further, when we introduce our environmental example of
predicting wind speeds.

Figure 1:  A schematic overview of our prediction method. (a) Local section
$\Sigma$ with radius $r$, oriented perpendicular to the flow; (b) spike train
having a non-zero amplitude every time the trajectory passes through $\Sigma$;
(c) $x$-component of the Lorenz system (blue) and $\tilde{x}$ our prediction
(green). The blue part of the trajectory represents the database used for the
cross prediction.

From $t>L$ onwards we do not have access to the full dynamics anymore, but
measure the times $\\{t_{i}\\}$ the trajectory $y(t)$ crosses a local section
$\Sigma\subset M$, e.g. $y(t_{i})\in\Sigma$. We choose $\Sigma$ to be
perpendicular to the flow at a point of the trajectory and the size of
$\Sigma$ is given by its radius $r$ (c.f. Fig. 1(a)). (In practice, we choose
a point in the time series and calculate the direction the point is moving.
This direction gives the the normal vector for our local cross section
$\Sigma$.) A lot is known about such crossing time series. For example it can
be shown that a $d$-dimensional vector
$I_{i}=(t_{i+1}-t_{i},t_{i+2}-t_{i+1},\ldots,t_{i+d}-t_{i+d-1})$ provides a
one–to–one embedding of the dynamics under mild conditions relating to
periodic points in the phase space sauer1994 ; huke2007 ; smale2011  as long
as $d>2m$. We exploit this result but instead of measuring $d+1$ consecutive
crossings, our measuring time $w$ is chosen so that our crossing time series
always satisfies the inequality $d>2m$:

$\max_{i}\\{t_{i+2m+1}-t_{i}\\}<w.$ (1)

We therefore record the crossing time series of our system for $w$ time units.
To determine $w$ we use the time series $y(t)$ in the database and measure the
crossing time series $\\{t_{i}\\}_{DB}$. This set of times $\\{t_{i}\\}_{DB}$
is going to be used to determine $w$. Therefore $w$ is the maximum time span
needed in the recorded history (the database) to satisfy the condition $d>2m$.
Note that, especially if the recording time $L$ is short, choices of $w$, as
the upper bound of eq. (1), should be taken conservatively. As we will see, in
practice one can even use a small value for $w$, e.g. an upper bound for only
a section $i$ of the time series, and still get a good prediction. We discuss
techniques used to improve the prediction for such choices of $w$ later.

Given the current crossing time series $\\{t_{i}\\}$, we use the Victor and
Purpura distance metric victor1997  to find similar sequences of length $w$ in
$\\{t_{i}\\}_{DB}$. The closest sequences in $\\{t_{i}\\}_{DB}$ provide the
base for our state estimation and the prediction of the dynamics for $t>L$. To
apply the Victor and Pupura distance we represent the crossing time series
$\\{t_{i}\\}$ and $\\{t_{i}\\}_{DB}$ as their corresponding spike trains. The
value of such spike trains are everywhere 0 and only equal to 1 when the
trajectory $y(t)\in\Sigma$ (c.f. Fig. 1 (b)). The Victor and Purpura metric
determines the minimum cost to convert one spike train into another one. This
metric has been used extensively in the context of neuroscience, e.g.
hirata2009 , where the spike trains naturally arise as the output of idealised
neurons.

Let the current spike train be $U=\\{u_{\xi}|\xi=1,2,\ldots,\Xi\\}$ and let
$V=\\{v_{\theta}|\theta=1,2,\ldots,\Theta\\}$ represent one of the possible
spike trains of the database. $u_{\xi}$ denotes the time for the $\xi$th event
within $U$ and $v_{\theta}$ is the time for the $\theta$th event within $V$.
Thus, given two initial conditions $x_{u},x_{v}\in X\subseteq M$ at the
beginnings of the two windows, we have
$f_{u_{\xi}}(x_{u}),f_{v_{\theta}}(x_{v})\in\Sigma$ for each $\xi$ and
$\theta$. When we apply the shift of events, we pair up one event $u_{\xi}$
from $U$ with another $v_{\theta}$ from $V$. Therefore each $u_{\xi}$ and
$v_{\theta}$ may not belong to more than one pair. In addition, we define $C$
as the set of such pairs taken from $U$ and $V$. With this we can define the
Victor Purpura distance as:

$\delta(U,V)=\min_{C}\\{\sum_{(u_{\xi},v_{\theta})\in
C}\lambda|u_{\xi}-v_{\theta}|+\Xi+\Theta-2|C|\\}.$ (2)

This edit distance $\delta$ satisfies the necessary three conditions for a
metric; (i) non-negative or ,zero iff two time windows are identical, (ii)
symmetric, and (iii) the triangle inequality.

Intuitively the metric can be understood by taking into account the possible
edits that can transform $U$ into $V$: either we can align event $u_{\xi}$ and
$v_{\theta}$ or we have to delete / create events in the spike trains. Both
these edits appear in the RHS of (2). The cost of aligning two events is
proportional to the time difference between the two events, e.g.
$\lambda|u_{\xi}-v_{\theta}|$ controls the cost. Since the cost of creating
and deleting events is chosen to be equal to 1, $\delta$ depends on the
difference between the total number of events – given by $\Xi+\Theta$ – and
twice the number of possible pairs – $2|C|$. This Victor Purpura distance has
been used to evaluate synchonization among neurons macleod1998 ; nomura2015 .
Further details and a thorough mathematical description of the Victor Purpura
distance can be found in the literature suzuki2010 ; ozken2015 ; hirata2015 .

Comparing the current crossing time sequence in this way with all the crossing
time sequences in the database, we identify the 10 sequences that are closest
in terms of their distance $\delta$. The average of their end points is going
to be the state estimate at the current time. For extremely long database
recording times and very high data precision one might use just the end point
of the sequence with minimal $\delta$ as the current state estimate. Under
realistic conditions, however, using the average and standard deviation of
about 10 sequences with similar recurrence properties – in the Victor Purpura
distance sense – improves the stability of the prediction and helps us to
evaluate its reliability. One can understand the Victor-Purpura distance for a
spike train as being similar to the Euclidean distance for usual vector
spaces.

Using cross prediction has the advantage that we do not require access to the
equations governing the dynamics of $y(t)$. Instead the prediction of the
future dynamics is provided as the average of the recorded evolution of the 10
states defining our state estimate. We define $t_{0}$ as the current point in
time and use $h=1,2,\ldots{},10$ as the index of the 10 states. Consequently
$t_{0,h}$ denotes the end points of the 10 most similar sequences in the
database. Then the estimation of the current state is:

${\color[rgb]{0,0,0}\tilde{y}(t_{0}|t_{0})}=\frac{1}{10}\sum_{h=1}^{10}y_{h}(t_{0,h}).$
(3)

Similarly the prediction of $\tilde{y}$ for $n\Delta t$ time steps in the
future is:

${\color[rgb]{0,0,0}\tilde{y}(t_{0}+n\Delta
t|t_{0})}=\frac{1}{10}\sum_{h=1}^{10}y_{h}(t_{0,h}+n\Delta t),$ (4)

and its corresponding prediction uncertainty can be measured by the variations
of the 10 sequences:

${\color[rgb]{0,0,0}\sigma_{\tilde{y}}(t_{0}+n\Delta
t|t_{0})}=\sqrt{\frac{1}{10}\sum_{h=1}^{10}(y_{h}(t_{0,h}+n\Delta
t)-\tilde{y}(t_{0}+n\Delta t))^{2}}.$

We use the uncertainty of our prediction to accept or reject the prediction of
$t_{0}+n\Delta t$. The main issue we will have is that the ensemble of the 10
sequences with the lowest $\delta$ might lose coherence, because they might be
close in the Victor-Purpura sense, but not all of them have to be close to the
system’s state on $\Sigma$. When this happens the uncertainty of the
prediction grows: $\sigma_{\tilde{y}}(t_{0}+n\Delta
t|t_{0})>\sigma_{\tilde{y}}(t_{0}+n\Delta t|t_{0}+n\Delta t)$. In such a case
we would reject the prediction and instead let $\tilde{y}(t_{0}+n\Delta
t|t_{0}+n\Delta t)$ be our prediction and replace $t_{0}$ by $t_{0}+n\Delta t$
to reinitialise for further prediction.

Especially when $L$ is short, it can be necessary to work with a small value
of $w$. For such $w$ the inequality (1) will not be satisfied for all times,
but only for some part of the time series in the database. To still achieve a
good state estimation, we consider that just before or after the current time
window, there is or is not a spike. For our current spike train, we therefore
have the option that a spike does (not) appear just before (after) the
beginning (end) of the window. We combine these 4 options with the
corresponding 4 options of our database $\\{t_{i}\\}_{DB}$ and determine the
$\delta$ of the 16 possibilities. As we see below, including these 16
possibilities in our $\delta$ minimisation greatly improves the accuracy of
the prediction.

Figure 2: Comparison between the predicted $\tilde{x}$ (dashed line) and the
original $x$ (solid line) time series of the Lorenz system: (a) recording time
of the database $L=150$, radius of local section $r=10$, upper bound of eq.
(1) $w=2$ and $\Delta t=0.2$; (b) $L=4950$, $r=3$, $w=10$ and $\Delta t=0.2$;
at the bottom of both panels the corresponding spike trains are shown and
$\tilde{x}(t_{0})$ is given by eqs. (3,4).

Such a prediction can be seen as the orange line in Fig. 1(c). Clearly visible
are the time periods during which we rejected the prediction (see the
instances of constant amplitude). The coincide with periods of low or no
recurrence, that are large gaps in the train sequence in panel (b). For
practical applications we would use a larger value of $w$ to eliminate these
areas of prediction rejection. To qualitatively assess the prediction we
evaluate the cross-correlation coefficient $R(y,\tilde{y})$ between the
prediction and the true dynamics of the system. For each $t_{0}$ we are going
to predict the dynamics using eq. (4) and evaluate $R(y,\tilde{y})$.

As a first application of this method, we apply our algorithm to the Lorenz
1963 system lorenz1963 , using the standard parameters $\sigma=28$, $\rho=10$
and $b=8/3$. We are going to systematically vary the radius $r$ of the local
section $\Sigma$ and the database recording time $L$. These parameter changes
directly impact on the upper bound $w$ of eq. (1). We calculated statistics
$Cv$ and $Lv$ from these spike trains based on Ref. shinomoto2003 , which
evaluate the global and local variabilities for spike trains for judging
whether the spike trains follow a Poisson process or not. If a spike train
follows a Possion process, its $Cv$ and $Lv$ fluctuate around 1 shinomoto2003
. We found that $Cv$ and $Lv$ were $0.8603\pm 0.1904$ and $0.6276\pm 0.1790$,
respectively, over 10 samples. While the estimate for $Cv$ was not
significantly different from 1, that for $Lv$ was significantly different from
1 ($P\approx 0.019$), meaning that the spike trains are likely to be different
from a Poisson process, or a standard point process shinomoto2003 .

Figure 3: The cross-correlation coefficient $R(x,\tilde{x})$ between the
prediction and the true dynamics in dependence of the size of the partition
and recording length. Cross-correlation was calculated for a prediction of 50
time units. Shown is the average correlation for 10 randomly placed $\Sigma$s
as well as the 50% confidence interval. (a) fixed upper bound $w=2$ and
$L=150$ while varying the size of the portion and (b) $w=10$ and a fixed
radius $r=3$ of the partition while varying the recording length $L$. N.B.
that we tend to have positive correlation coefficients because we predicted
the values generated from the true dynamics based on spike trains.

In Fig. 2 we see two outputs of our algorithm in comparison with the true
$x$-component of the Lorenz system. Since it is hard to visualise the state
estimates together with their predictions, we instead only show the state
estimates together with the prediction up to the next estimate. As a
reinitialisation time we used $\Delta t=0.2$ in both experiments. Since this
value is very small, the prediction is never rejected and the data shown
highlights the accuracy of our state estimate more than the quality of our
prediction. The main difference between the two data sets shown in panel (a)
and (b) is the recording time $L$ and the radius $r$ of the local section
$\Sigma$. We use $r=10$ in panel (a) and $r=3$ in (b). Consequently the
recurrence rate with which the trajectory returns to $\Sigma$ is much higher
in (a) than in (b) as we can see from the spike trains below the trajectory.
Given the high recurrence rate we need a short recording time $L=150$ for
$r=10$, while for $r=3$ we need longer recording times ($L=4950$). Similarly
the recurrence rate helps us to chose the upper bound $w$ of eq. (1): (a)
$w=2$ and (b) $w=10$. The comparison of the true and predicted dynamics
clearly shows that our algorithm can be optimised to work for both choices of
$r$ and approximates the dynamics of the system well.

The large deviations of $\tilde{x}$ in Fig. 2 (a) around $t=12$ and $22$ are
caused by our choice of $w=2$ and a very short recording time $L=150$. Given
$w=2$, it is not always possible to satisfy the condition (1). For that reason
our prediction diverges from the true dynamics. In addition the short
recording of the past dynamics in combination with the cross-prediction
technique does not allow us to get the amplitude of some of the oscillations
right, e.g. $25<t<30$. As we can see for $L=4950$ and $w=10$ these problems
disappear and we get better estimates and predictions (Fig. 2 (b)). While the
recurrence rate is lower, we have more knowledge about the past dynamics and
the higher value of $w$ makes it easier to satisfy the condition (1).

We want to understand the influence of $L$ and $r$ on our prediction algorithm
in more detail. For this we determine the correlation between the original
$x(t)$ time series of Lorenz with predictions of the next 50 time units, while
varying $L$ or $r$. Moreover we use these experiments to demonstrate that the
location of the local section $\Sigma$ is of minor importance for the
predictions. For each of our experiments we therefore use 10 random positions
for $\Sigma$ and report in Fig. 3 the mean value as well as the 50% confidence
interval. For Fig. 3 (a) we fixed $w=2$, $L=150$ and vary the size of the
partition, while in (b) we chose $w=10$, $r=3$ and vary $L$.

A sufficiently large size of $\Sigma$ enables us to cross-predict the original
time series almost perfectly (Fig. 3(a)), because for large $r$ we have a high
recurrence rate and the condition of eq. (1) is likely to be met. When we
reduce the size of the local cross section, the correlation coefficient
between the original time series and the cross-predicted time series
decreases. The lower recurrence rate makes it impossible to satisfy the
condition eq. (1) , namely $d>2m$, for all times, leading to the worse
prediction. But, even in such a case, a sufficiently large recording time $L$
as well as a larger $w$ leads to a high correlation coefficient (Fig. 3(b)).
Thus, we expect for a database of a sufficiently long recording time that we
can still reconstruct the original time series faithfully. Moreover the
medians and 50% confidence intervals in Fig. 3(a) and (b) show that the
positions of the local cross sections are of less importance for our
prediction algorithm. For example, reconstructions from local sections at
$x=0$ with $\dot{x}<0$ or $\dot{x}>0$ are not so different from each other as
shown in Fig. 4.

Figure 4:  A spike train generated by the local cross section at $x=0$ and
$dx/dt>0$, and the reconstruction of the original signal. The original signal
is shown in the blue solid line and its reconstruction, the red dashed line.
Here we used $L=19950$, $\Delta t=0.2$ and $w=30$. At the bottom we show the
spike train and reconstruction for the local cross section at $x=0$ but
$dx/dt<0$. The correlation coefficient between the original and reconstructed
signals is 0.3926. (Because we have a fewer events, we needed the larger $L$
and $w$ to have the better results.)

To further understand these results in Figs. 2 and 3 and gain some insight
about the importance of the different measurements (recorded history in the
database and crossing times), we analysed the predictability of flows from
crossing times as an information-theoretic problem. Our analysis shows that
when a local cross section becomes small and therefore generated less frequent
events, the time resolution of our data becomes the dominating factor. For our
analysis we partition our spike train time series into $\alpha$ bins of equal
size. Let $\beta$ be the number of events within the spike train. Moreover
each bin can only contain one spike. Then, the number of signals $N$ we can
send by a spike train is

$N=\left(\begin{array}[]{c}\alpha\\\
\beta\end{array}\right)=\frac{\alpha!}{\beta!(\alpha-\beta)!}.$ (5)

Assuming that all signals are equally likely, we can calculate the amount of
information in the series as cover2006

$\log_{2}N=\log_{2}\frac{\alpha!}{\beta!(\alpha-\beta)!}.$ (6)

Applying Stirling’s approximation, we get

$\begin{array}[]{cl}&\log_{2}N\\\
\approx&\alpha\\{-\frac{\beta}{\alpha}\log_{2}\frac{\beta}{\alpha}-\frac{\alpha-\beta}{\alpha}\log_{2}\frac{\alpha-\beta}{\alpha}\\}\\\
=&\beta\log_{2}\frac{\alpha}{\beta}+(\alpha-\beta)\log_{2}\frac{\alpha}{\alpha-\beta}\\\
\geq&\beta\log_{2}\frac{\alpha}{\beta}.\end{array}$ (7)

Given that $\alpha\gg\beta$, we can assume that $\alpha$ and $\beta$ are
related by $\alpha=\beta 2^{\gamma}$. Therefore the RHS of the equation can be
written as

$\beta\log_{2}\frac{\alpha}{\beta}=\beta\log_{2}2^{\gamma}=\beta\gamma.$ (8)

Since $\beta$ gives the total number of spikes in the sequence and we only
allow one spike per partition, the maximum $\beta$ is directly related to the
time resolution of the sequence. We conclude that therefore the time
resolution given by $\beta$ is directly proportional to the total amount of
information contained in the time series. This has two consequences. For real
world data the time resolution of the data is of upmost importance. Using our
algorithm in real world applications would require instruments with high
precision, but at least it is only the time precision and not also amplitude
precision on top. The second consequence of eq. (8) relates to the efficiency
of our method. Once the database is recorded, the data required to predict the
future dynamics has a very small footprint. For example the database needed to
predict the dynamics shown in Fig. 2(b) has $40,000$ bytes and its crossing
time series $\\{t_{i}\\}_{DB}$ has $152\pm 40$ bytes without compression in
disk space (We used here the MATLAB command “whos” to evaluate the disk space
we need for storing these variables.). The spike train recorded to estimate
the state and consequently predict the dynamics was $0.38\pm 0.10$ % of their
combined size. Having such a small memory demand makes this method attractive
for real world applications.

As one example for a real world application, we are use our algorithm to
predict wind speed hirata2008 . The original measurements were observed for 24
hours from around 2pm on 1 September 2005 at about 1m above from the ground
level at the Institute of Industrial Science, The University of Tokyo, Tokyo,
Japan using an ultrasonic anemometer. The original measurements were made with
50Hz, but we first sub-sample the time series using every tenth point. This
sub-sampling allows us to use a longer $L$ and therefore the database contains
more of the long term changes in the dynamics. Since we do not have access to
the full state space of this environmental system, we have to reconstruct the
attractor of the dynamics by using delay embedding takens1981 ; sauer1991 .
Given that the system seems to have features of high-dimensional dynamics
hirata2015srep , we have to use a high dimensional embedding. But the total
recording time of the wind speeds is 10,000 seconds, and consequently we
cannot use too high an embedding dimension, without making the recurrence rate
too low for our prediction algorithm. In addition we only use the first 9,000
seconds as our database and use the remainder for comparison with our
prediction. As a compromise and after several tests, we decided to use a 10
dimensional embedding space. We find that our algorithm succeeded in capturing
the large scale features of the dynamics, i.e., prolonged changes of the wind
velocity are similar in the experimental data and the prediction (Fig. 5). On
the other hand our predictions fail to reproduce the finer details of the
system’s dynamics. We conjecture that a higher embedding dimension together
with a longer time series would be able to overcome these limitations. Another
way to improve the performance of the method could be to use marked spike
trains, by assigning some additional value to each event. Several distances
have been proposed for such marked point processes schoenberg2008 ; suzuki2010
; hino2015 ; iwayama2017 , which might increase the prediction performance for
such high dimensional dynamics.

Figure 5: Example of the wind speed time series (solid line) together with our
prediction (dash-dotted line). The parameters of our algorithms are the size
of the time window $w=200\text{s}$, recording time $L=9,000\text{s}$, $\Delta
t=4\text{s}$ and the size of the local section $\Sigma=5\text{m}/\text{s}$.
Our algorithm predicts the following 1,000 seconds.

Applying the method of Ref. kennel1997  to the above wind data shows the value
greater than 99% point of $2.36$, meaning that the wind data should be
regarded as non-stationary. The applicability of our method for non-stationary
data should be evaluated more closely in future research, although our results
in Fig. 5 seem to show some promise in this line of research.

Our numerical results and to some extent the results on wind speeds show that
a local cross section $\Sigma$ has the generic property to reconstruct the
underlying dynamics of a flow almost perfectly. This is similar to generating
partitions which have been extensively studied in maps grassberger1985 ;
davidchack2000 ; plumecoq2000a ; plumecoq2000b ; kennel2003 ; hirata2004 . In
maps it is a non-trivial task to find a generating partition. Our results
show, that in flows this is much easier and the position of $\Sigma$ does not
matter much for the prediction quality (c.f. Fig. 3).

While there are many studies that try to predict the next crossing time based
on previous crossings (for example sauer1994 ), the method presented here is
different. Instead of just focussing on the next crossing, we are able, using
cross-prediction, to estimate the complete future dynamics. If we regard the
observations at a local cross section as a coincidence detector konig1996 ;
azouz2000  our results could explain how we can share the same experience even
if we perceive a phenomenon in different ways, or in our case by different
local cross sections. Hence, our results might also have some implications in
the field of theoretical neuroscience.

We presented a method for state estimation and prediction of flows, given a
database of the past dynamics and a crossing time series. Identifying the
current state is done by ranking the crossing time sequences of the database
according to their distance from the current crossing time series. We showed
that our method requires little memory to store and send time series
information via a spike train if the database is shared between sender and
receiver. Our method does not require us to store and send an entire time
series under this assumption. Instead we simply record or send the times when
a trajectory passes the local cross section. This advantage can be important
for sensor networks akyildiz2002 , where each device has to store and
communicate lots of environmental information under severe energy constraints.
Similarly, our method may be used to reconstruct missing data within
observations, such as gaps in, e.g., satellites images partly covered by
clouds liu2017 , historical series of sunspot numbers clette2014  or
historical phenological data such as the start of the cherry blossom in Japan
aono2008 .

Y. H. would like to appreciate Prof. Kazuyuki Aihara for the discussions. The
research of Y. H. is supported by Kozo Keikaku Engineering Inc. The research
of D. E. has been supported by German-Israeli Foundation for Scientific
Research and Development (GIF), GIF Grant No. I-1298-415.13/2015.

## References

  * (1) C.S. Greene, J. Tan, M. Ung, J.H. Moore, and C. Cheng, Journal of Cellular Physiology 229, 1896–1900 (2014).
  * (2) W.Q. Meeker and Y. Hong, Quality Engineering 26, 102–116 (2014).
  * (3) M. McCullough, K. Sakellariou, T. Stemler, and M. Small, Chaos 26, 123103 (2016).
  * (4) K. Sakellariou, M. McCullough, T. Stemler, and M. Small, Chaos 26, 123104 (2016).
  * (5) Y. Hirata and K. Aihara, J. Neurosci. Methods 183, 277-286 (2009).
  * (6) Y. Hirata and K. Aihara, Physica A 391, 760-766 (2012).
  * (7) Y. Hirata, E. J. Lang, and K. Aihara, Recurrence plots and the analysis of multiple spike trains, In. Springer Handbook of Bio-/Neuroinformatics, Ed. Nikola Kasabov, Springer Berlin Heidelberg, pp.735-744, 2014.
  * (8) I. Ozken, D. Eroglu, T. Stemler, N. Marwan, G. B. Bagci and J. Kurths, Phys. Rev. E 91, 062911 (2015).
  * (9) D. Eroglu, F. H. McRobie, I. Ozken, T. Stemler, K.-H. Wyrwoll, S. F. M. Breitenbach, N. Marwan and J. Kurths, Nat. Comm. 7, 12929 (2016).
  * (10) J. D. Victor and K. P. Purpura, Network: Comput. Neural Syst. 8, 127-164 (1997).
  * (11) H. Poincaré, “Sur le problème des trois corps et les équations de la dynamique”. Acta Math., 13, 1-270 (1890).
  * (12) N. Marwan, M. C. Romano, M. Thiel, J. Kurths, Recurrence Plots for the Analysis of Complex Systems, Physics Reports, 438(5-6), 237-329, (2007).
  * (13) Y. Hirata, K. Iwayama and K. Aihara, Phys. Rev. E 94, 042217 (2016).
  * (14) M. Le Van Quyen, J. Martinerie, C. Adam, and F. J. Varela, Physica D 127, 250-266 (1999).
  * (15) E. N. Lorenz, J. Atmos. Sci. 20, 130-141 (1963).
  * (16) F. Takens, Lect. Notes Math. 898, 366 (1981).
  * (17) T. Sauer, J. A. Yorke and M. Casdagli, J. Stat. Phys. 65, 579 (1991).
  * (18) T. Sauer, Phys. Rev. Lett. 72, 3811-3814 (1994).
  * (19) J. P. Huke and D. S. Broomhead, Nonlinearity 20, 2205-2244 (2007).
  * (20) S. Smale, Stable manifolds for differential equations and diffeomorphisms, in Topologia differenziale pp. 93-126, 2011.
  * (21) K. MacLeod, A. Bäcker, and G. Laurent, Nature 395, 693-698 (1998).
  * (22) R. Nomura, Y. Liang, and T. Okada, PLoS One 10, e0140774 (2015).
  * (23) S. Suzuki, Y. Hirata and K. Aihara, Int. J. Bifurcat. Chaos 20, 3699-3708 (2010).
  * (24) Y. Hirata and K. Aihara, Chaos 25, 123117 (2015).
  * (25) S. Shinomoto, K. Shima, and J. Tanji, Neural Comp. 15, 2823-2842 (2003).
  * (26) T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., John Wiley & Sons, Hoboken, 2006.
  * (27) Y. Hirata, D. P. Mandic, H. Suzuki and K. Aihara, Phil. Trans. R. Soc. A 366, 591-607 (2008).
  * (28) Y. Hirata, T. Takeuchi, S. Horai, H. Suzuki, and K. Aihara, Sci. Rep. 5, 15736 (2015).
  * (29) F. P. Schoenberg and K. E. Tranbarger, Environnmentrics 19, 271-286 (2008).
  * (30) H. Hino, K. Takano and N. Murata, R Journal 7, 237-248 (2015).
  * (31) K. Iwamaya, Y. Hirata and K. Aihara, Phys. Lett. A 381, 257-262 (2017).
  * (32) M. B. Kennel, Phys. Rev. E 56, 316-321 (1997).
  * (33) P. Grassberger and H. Kantz, Phys. Lett. 113A, 235 (1985).
  * (34) R. L. Davidchack, Y.-C. Lai, E. M. Bollt, and M. Dhamala, Phys. Rev. E 61, 1353 (2000).
  * (35) J. Plumecoq and M. Lefranc, Phsyica D 144, 231 (2000).
  * (36) J. Plumecoq and M. Lefranc, Phsyica D 144, 259 (2000).
  * (37) M. B. Kennel and M. Buhl, Phys. Rev. Lett. 91, 084102 (2003).
  * (38) Y. Hirata, K. Judd and D. Kilminster, Phys. Rev. E 70 016215 (2004).
  * (39) P. König, A. K. Engel and W. Singer, Trends Neurosci. 19, 130-137 (1996).
  * (40) R. Azouz and C. M. Gray, Proc. Natl. Acad. Sci. USA 97, 8110-8115 (2000).
  * (41) I. F. Akyildiz, W. Su, Y. Sankarasubramaniam, and E. Cayirci, IEEE Communications Magazine 40, 102 (2002).
  * (42) R. Liu, R. Shang, Y. Liu, X. Lu, Remote Sensing of Environment 189, 164-179 (2017).
  * (43) F. Clette, L. Svalgaard, J.M. Vaquero, E.W. Cliver, Space Science Reviews 186, 35–103 (2014).
  * (44) Y. Aono, K. Kazui, Int. J. of Climatology 28, 905–914 (2008).


#  Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens

Fan Ma1, Xiaojie Jin2∗, Heng Wang2, Yuchen Xian1, Jiashi Feng2, Yi Yang1∗
1Zhejiang University 2ByteDance Inc.

###### Abstract

Recent advances in large video-language models have displayed promising
outcomes in video comprehension. Current approaches straightforwardly convert
video into language tokens and employ large language models for multi-modal
tasks. However, this method often leads to the generation of irrelevant
content, commonly known as “hallucination”, as the length of the text
increases and the impact of the video diminishes. To address this problem, we
propose Vista-LLaMA, a novel framework that maintains the consistent distance
between all visual tokens and any language tokens, irrespective of the
generated text length. Vista-LLaMA omits relative position encoding when
determining attention weights between visual and text tokens, retaining the
position encoding for text and text tokens. This amplifies the effect of
visual tokens on text generation, especially when the relative distance is
longer between visual and text tokens. The proposed attention mechanism
significantly reduces the chance of producing irrelevant text related to the
video content. Furthermore, we present a sequential visual projector that
projects the current video frame into tokens of language space with the
assistance of the previous frame. This approach not only captures the temporal
relationship within the video, but also allows less visual tokens to encompass
the entire video. Our approach significantly outperforms various previous
methods (_e.g.,_ Video-ChatGPT, MovieChat) on four challenging open-ended
video question answering benchmarks. We reach an accuracy of 60.7 on the zero-
shot NExT-QA and 60.5 on the zero-shot MSRVTT-QA, setting a new state-of-the-
art performance. This project is available at
https://jinxxian.github.io/Vista-LLaMA.

††∗ Corresponding author

##  1 Introduction

Figure 1: Video language processing with LLaMA [24] and our Vista-LLaMA. The
vanilla LLaMA treats visual tokens () the same as other language tokens (),
weakening the impact for tokens in long distance. Our model retains the same
mechanism for language tokens and strengthens the impact of the visual tokens.
The intensity of the impact of each token is conveyed through the depth of the
line color (). Our model provides the accurate response for the presented
scenario.

The surge in multi-modal vision-and-language models [1, 12, 14], capable of
comprehending both visual (_e.g._ , image/video) and language data, can be
attributed to the recent achievements of large language models (LLMs) such as
GPT [8], GLM [35], and LLaMA [37]. Video-language models (video-LMs) pose
greater challenges in scaling due to increased computational and annotation
costs compared to image-language models (image-LMs). Recent research has
therefore focused on effectively training video-LMs by utilizing pre-trained
image-LMs [19, 37].

Video-LMs benefit from this warm-start approach to enhance visual
representation learning by projecting video frames into language space and
treating videos as several prompt language tokens [32, 15]. However, this
diminishes the visual impact of videos on text generation and lacks explicit
temporal modeling in videos. The generated text is often not related to the
video content, as depicted in Figure 1, a phenomenon known as hallucination in
language processing [6]. The distance between the generated text token and the
visual tokens in large language models may be a contributing factor,
especially when the visual tokens are distant from the generated text.
Additionally, handling longer videos poses a challenge because of the
constraints on context length in large language models, where numerous visual
tokens take up a substantial portion of the context.

We present Vista-LLaMA, an innovative video language framework to tackle above
issues. The primary concept is to preserve equal distance between all visual
tokens and any language tokens, while also retaining the relative distance
between any two language tokens, as depicted in Figure 1. The rotary position
embedding is only applied to language tokens to capture relative distance when
calculating similarity between language tokens. When computing similarity
between visual and text tokens, the rotary position embedding is removed to
reduce the impact of relative distance. With the equal distance to visual
tokens (EDVT) attention, the impact of visual cues on text production is
amplified without compromising the text production capability. The experiments
also show that our design produces more precise text description for input
videos and the phenomenon of visual hallucinations occurring in language
models has been greatly reduced.

To further improve the temporal modeling of video, we introduce a sequential
visual projector. Instead of mapping each frame of the video independently
into fixed-length visual tokens, which ignores the temporal relationship
between frames, we generate visual tokens for each frame using the previous
projected visual tokens. It not only incorporates the temporal relationship
between frames into the language model to improve video comprehension without
any extra parameters, but also enables the language model to encode longer
videos with fewer visual tokens by sampling fewer projected frames.

We demonstrate the effectiveness of Vista-LLaMA on four challenging video
question answering (QA) benchmarks, where our method outperforms several
previous works, and achieves the state-of-the-art in zero-shot NExT-QA [28],
and MSRVTT [30]. We also show that our attention design and the temporal
modeling mechanism improves capacity of large language model on video-question
answering by a large margin. Comprehensive experiments are conducted to
demonstrate the effectiveness of our designs. We summarize our contributions
as follows:

  * •

We introduce a novel video-language model, dubbed as Vista-LLaMA, to enhance
the video understanding and facilitate temporal modeling within the language
model.

  * •

A novel multi-modal attention is proposed to enable reliable video text
generation by maintaining equal distance to visual tokens. Additionally,
temporal modeling is facilitated by employing a sequential visual projector.

  * •

Our method exhibits superior empirical performance, establishing new
benchmarks for zero-shot open-ended video question answering tasks.

  * •

A detailed analysis further explains the design choices inherent in our
proposed framework, contributing significantly to a better comprehension of
multi-modal dynamics in large language models.

##  2 Related Work

###  2.1 Large Language Models

Large language models (LLMs) [26, 39, 16, 2, 25] have demonstrated exceptional
proficiency in understanding language and reasoning, resulting in the
generation of high-quality natural language text in diverse domains. LLMs have
already initiated a technological revolution and have found extensive
application in various domains. Additionally, a series of open source large
models, including LLaMA [24], and OPT [38], have significantly contributed to
technological advancements and made remarkable contributions to the NLP
community. Leveraging the foundation established by these impressive LLMs,
researchers have further expanded their capabilities and developed exceptional
models for diverse NLP tasks (_e.g._ Vicuna [4]). Our work is also built upon
these remarkable LLMs, and we equip the language models with a novel attention
mechanism and a sequential visual projector that enhances their abilities to
comprehend visual content in videos.

###  2.2 Multi-modal Large Language Models

Researchers have been diligently exploring the application of LLMs for
processing multi-modal problems [13, 9]. The existing approaches can be
classified into two primary categories. In the first category, LLMs are
treated as controllers while the multi-modal models as tools [10]. When
presented with the specific task, the LLMs first interact with user
instructions and decide which tools to employ. Subsequently, it generates
comprehensive responses by amalgamating the outcomes derived from these
readily available multi-modal models. These approaches, such as Visual ChatGPT
[27], HuggingGPT [18], and ViperGPT [22], have shown impressive results on
various multi-modal tasks without training models. For the second category,
large-scale multi-modal models are trained on the multi-modal data. The
principle behind this line of researches is to align other modal pre-trained
models with textual LLMs. Flamingo [1] incorporates a perceiver resampler and
a gated cross-attention layer to connect a frozen image encoder with the LLM.
BLIP2 [12] introduces a Q-Former to map each image into fixed-length tokens in
the language embedding space via the learned queries. LLaVA [14], mPLUG-owl
[33], and MiniGPT4 [39] develop the image-LLMs utilizing image-instruction
training pairs. Video Chat [13] extends image encoders to enable large models
to comprehend visual content in videos. Video-ChatGPT [15] is trained on video
instructional data to give appropriate answers for multi-modal inputs. In this
work, we improve the understanding of videos in a video-language model by
incorporating temporal modeling and a novel multi-modal attention mechanism.

Figure 2: The framework of Vista-LLaMA. The visual encoder and large language
model are both frozen () during training, while the projector is trainable ()
to map video into the language’s space. The attention operation in each layer
is present on the right part. Only the text tokens are applied with rotary
position embedding to include relative distance information. The attention
weights between visual and language tokens are calculated without the rotary
position embedding. The casual mask is applied to the bottom-right attention
weights.

###  2.3 Video Question Answering

Video Question Answering (VideoQA) is a task that involves answering language
questions that are based on a given video [30]. It requires the ability to
understand and reason across different semantic levels, which in turn demands
a capacity for multi-modal understanding. Previous VideoQA benchmarks
primarily concentrated on short videos, asking questions according to the
specified visual facts (_e.g._ , location and objects) [31, 7]. Recently,
several new benchmarks have been proposed to focus on resolving temporal and
causal questions in longer video clips [23]. NExT-QA [28] is an example of
such a benchmark, aiming to uncover the causalities or intentions of
particular events, and infer subsequent actions within the video. In this
work, we investigate the video-language model on zero-shot open-ended VideoQA
where no training question-answer pairs are provided in the training stage.

##  3 Method

###  3.1 Overview

Vista-LLaMA comprises three fundamental components: a visual encoder, a visual
projector, and a pre-trained large language model. Figure 2 shows an overview
of Vista-LLaMA. The components’ design and implementation details are provided
below:

  * •

Visual Encoder. We utilize pre-trained EVA-CLIP-g [21] and ViT-L [17] as
visual encoder. The last layer of ViT encoder is removed because it
specializes in aggregating the features of the first token for contrastive
learning.

  * •

Visual Projector. The visual projector maps the output of the visual encoder
into tokens that occupy the same space as the text features from word
embedding. Various visual projectors are considered in Sec. 3.3. The visual
projector is to be trained in our work.

  * •

Pre-trained Large Language Model. In this study, we utilize LLaVa [14], which
is fine-tuned on Vicuna-7B [4] using instructional image-text pairs, for
processing videos and texts. The causal mask is employed in all attention
processes, incorporating the attention interplay between visual and text
tokens. The pre-trained language model is frozen in the present study, and a
new attention mechanism is developed to maintain a consistent distance to
visual tokens for all textual tokens.

In our framework, the video is first encoded into visual tokens using the
frozen visual encoder and the trainable visual projector. These visual tokens
are then combined with text tokens, which are projected with word embedding
from the prompt and question sentences. The combined visual and text tokens
are then input into the language model to generate answers.

###  3.2 Equal Distance to Visual Tokens

Preliminary. For the concatenate visual-text input, three linear projection
layers are applied in every attention layer to produce the query $\mathbf{Q}$,
key $\mathbf{K}$, and value $\mathbf{V}$. Let $\mathbf{q}_{j}$ be $j^{th}$
query in $\mathbf{Q}$. The conventional attention mechanism updates the input
by initially determining the similarity between the query and key, then
applying the attention weights to the value state. Specifically, for the
$j^{th}$ position, the update procedure can be articulated as follows:

$\scriptsize\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})_{j}=\frac{\sum_{i=1}^{j}sim(\mathbf{q}_{j},\mathbf{k}_{i})\mathbf{v}_{i}}{\sum_{i=1}^{j}sim(\mathbf{q}_{j},\mathbf{k}_{i})},$
(1)

where
$sim(\mathbf{q}_{j},\mathbf{k}_{i})=exp(\mathbf{q}_{j}^{T}\mathbf{k}_{i}/\sqrt{d})$.
Here, a causal mask is utilized so that the $j^{th}$ query can only attend to
the key positioned less than $j$.

Algorithm 1 Pseudocode of EDVT attention in a PyTorch-like style.

⬇

# x: hidden input in each attention laye

# v_mask: indcitate which input are from visual tokens

def edvt_attention_layer(x, v_mask):

# query, answer, value projection

q, k, v = qkv_proj(x)

# apply RoPE for query and key inputs

r_q, r_k = rope(q, k)

# attention weights without RoPE

attention = bmm(q.T, k)

# attention weights with RoPE

r_attention = bmm(r_q, r_k)

# Merge attention weights based on visual token

attention = v_mask * attention + (1 - v_mask) * r_attention

attention = Softmax(attention, dim=-1)

# Update representation based on attention weights

v = bmm(attention, v)

out = linear_proj(v)

return out

qkv_proj: linear projection layer; bmm: batch matrix multiplication; rope:
apply rotary position embedding.

EDVT-Attention. The vanilla attention model lacks positional awareness, with
no encoded relative distance for natural language processing. In contrast,
Rotary Positional Embeddings (RoPE) [20] encodes the position data of tokens
using a rotation matrix, which inherently includes an explicit relative
position dependency. Within each attention layer, RoPE is implemented across
all projected query and key inputs in order to compute the attention weights
via leveraging relative distance between tokens. The query situated at the
$j^{th}$ position incorporates rotary position embedding through
$\tilde{\mathbf{q}}_{j}\\!=\\!\mathbf{R}_{j}\mathbf{q}_{j}$, wherein
$\mathbf{R}_{j}\\!\in\\!\mathbb{R}^{d\times d}$ represents the rotary matrix
for the $j^{th}$ position. Consequently, the attention involving relative
position embedding is expressed as follows:

$\scriptsize\text{Attention}_{rope}(\mathbf{Q},\mathbf{K},\mathbf{V})_{j}=\frac{\sum_{i=1}^{j}sim(\mathbf{R}_{j}\mathbf{q}_{j},\mathbf{R}_{i}\mathbf{k}_{i})\mathbf{v}_{i}}{\sum_{i=1}^{j}sim(\mathbf{R}_{j}\mathbf{q}_{j},\mathbf{R}_{i}\mathbf{k}_{i})}.$
(2)

RoPE inherently integrates relative position data via the multiplication of
rotation matrices rather than appending it to the input as a positional
embedding. In natural language processing, this relative proximity between two
words is vital, given that remote words should have less influence than
adjacent words when generating the current word. However, using the same
attention mechanism for visual and text tokens may result in unintentional
text generation, a phenomenon often referred to as hallucination in LLMs. With
multi-modal input, the generated text should depend on the visual content,
disregarding the influence of relative distance.

To alleviate this issue and enhance the video understanding with LLMs, we
introduce the EDVT-Attention where the equal distance to visual tokens is
maintained while the relative distance between text tokens is attained. As
shown in Figure 2, the rotary position embedding is only applied on text
tokens. Let $\mathcal{V}$ and $\mathcal{T}$ be the set of visual and text
tokens. The EDVT-Attention is formulated as:

$\scriptsize\text{Attention}_{edvt}(\mathbf{Q},\mathbf{K},\mathbf{V})_{j}=\frac{\sum\limits_{k_{i}\in\mathcal{T}}sim(\tilde{\mathbf{q}}_{j},\tilde{\mathbf{k}}_{i})\mathbf{v}_{i}+\sum\limits_{k_{i}\in\mathcal{V}}sim(\mathbf{q}_{j},\mathbf{k}_{i})\mathbf{v}_{i}}{\sum\limits_{k_{i}\in\mathcal{T}}sim(\tilde{\mathbf{q}}_{j},\tilde{\mathbf{k}}_{i})+\sum\limits_{k_{i}\in\mathcal{V}}sim(\mathbf{q}_{j},\mathbf{k}_{i})},$
(3)

where $\tilde{\mathbf{q}}_{j}$ and $\tilde{\mathbf{k}}_{j}$ denote the query
and key applied with rotary embedding, separately. The distance between
language tokens is determined using the rotary matrix, while the correlation
between visual and textual inputs remains unaffected by the rotary embedding.
This improves the influence of visual information on long-term text generation
and decreases the incidence of hallucinations wherein the fabricated content
is absent in the videos.

For ease comprehension of how EDVT-Attention works, Algorithm 1 exhibits the
pseudo-code in the decoder layer of LLMs. It involves the combinations of
attention weights prior to and subsequent to the implementation of rotary
embedding with the visual mask.

Figure 3: Comparison of three visual projectors. (a): Each frame feature is
projected into the visual tokens independently with the linear projection.
(b): Q-Former uses shared learnable query tokens to separately map each frame
into fixed-length tokens. (c): The sequential Q-Former with linear projection
layer to enable temporal modeling.

###  3.3 Sequential Visual Projector

The visual projector aims to map video features into the language embedding
space, allowing for the fusion and processing of visual and textual inputs by
the substantial language model. As shown in Figure 3, earlier visual
projectors either employ the linear layer or the query transformer (Q-Former)
[12] to directly convert frame features into language tokens. However, the
lack of temporal relationship in these methods impedes thorough video
understanding in LLMs. We introduce the sequential visual projector in Figure
3 to encode the temporal context into the visual tokens.

Let $\mathbf{o}\in\mathbb{R}^{t\times l\times d}$ be the extracted video
feature of length $t$ where $l$ denotes the length of visual tokens. We use a
Q-Former $f_{Q}$ to map each frame into the fixed length of $k$
representations
$\mathbf{x}_{i}=f_{Q}(\mathbf{o}_{i},\mathbf{p})\in\mathbb{R}^{k\times d}$
where $\mathbf{p}\in\mathbb{R}^{k\times d}$ is the learnable query embedding.
In the prior approach, all projected frame features are merely combined with
word tokens to serve as mixed input for LLMs. For encoding temporal context,
we utilize the previously projected frame feature as the query to attend to
the current frame feature. The current projected frame feature is then updated
accordingly:

$\mathbf{x}_{t}=f_{Q}(\mathbf{o}_{i},\mathbf{x}_{t-1}),$ (4)

where $\mathbf{x}_{t-1}\in\mathbb{R}^{k\times d}$ represents the previous
projected frame feature. This allows the visual tokens to encode the temporal
relationship, as the current frame’s visual token is generated using the
previous feature.

Moreover, we can represent the entire video with fewer tokens by merely
sampling a small number of visual tokens. This method tackles the challenge of
encoding lengthy videos by employing sequential encoding. By integrating prior
visual context into subsequent visual tokens, we can accomplish an adequate
representation of the entire video through sparse sampling of projected frame
features. Besides, this technique permits larger language models to manage
much longer videos without constraints of context length.

###  3.4 Implementation Details

We fine-tune the model based on LLaVA [14] and use 100$K$ video instruction
pairs provided by [15]. We add Q-Former initialized from BLIP2 [12] to project
frame features into fixed-length tokens. We test our model with both ViT-L-14
[17] and EVA-CLIP-g [21] visual encoders in experiments. We only update the
visual projector, which contains the Q-Former and linear projection layer to
project the video features to the LLMs’ input space. The visual backbone and
the language model are frozen during training. The Vista-LLaMA is fine-tuned
for 3 epochs on 8 A100 80GB GPUs with a learning rate of 2$e^{-5}$ and an
overall batch size of 32. We run all the inference experiments with FP16 to
save memory and faster testing.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. Our method is evaluated on four datasets:

  * •

NExT-QA [28] is designed to advance video understanding from descriing to
explaining the temporal actions. It comprises 5,440 videos and approximately
52$K$ manually annotated QA pairs, which are categorized into temporal (Tem.),
causal (Cau.), and descriptive (Des.) questions.

  * •

MSVD-QA [30] is a dataset built upon Microsoft Research Video Description
Corpus [3], commonly used in video caption tasks. The MSVD-QA dataset
comprises a total of 1,970 video clips with 50,505 QA pairs.

  * •

MSRVTT-QA [30] is based on MSR-VTT dataset [31], which includes 10$K$ videos
and 243$K$ QA pairs with larger and has more complex scenes.

  * •

ActiviytNet-QA [34] is a fully annotated and large-scale videoQA dataset. It
contains 58$K$ QA pairs derived from 5,800 complex web videos derived from the
popular ActivityNet dataset [11].

Evaluation. We adopt two metrics to evaluate the performance of video-language
models.

  * •

Open-Ended Zero-Shot Question-Answer Evaluation. As video-language models
generate responses of varying lengths to open-ended questions, it is challenge
to evaluate models with traditional word matching strategy. We employ LLM-
Assisted evaluation, in line with [15], for assessment purposes. Given the
question, correct answer, and predicted answer by model, GPT-3.5 model is used
to return True or False judgement and relative score ($0$ to $5$).

  * •

Video-Based Text Generation Benchmarking. We evaluate the text generation
performance following [15] from five aspects: Correctness of Information,
Consistency, Detail Orientation, Contextual Understanding, and Temporal
Understanding. The test set for this evaluation is based on the
ActivityNet-200 [7], featuring videos with rich, dense descriptive captions
and associated question-answer pairs from human annotations. The evaluation
pipeline is also built with the GPT-3.5 model and relative score ($0$ to $5$)
is generated.

Method | NExT-QA [28] | MSVD-QA [30] | MSRVTT-QA [30] | ActivityNet-QA [34]
---|---|---|---|---
| Accuracy | Score | Accuracy | Score | Accuracy | Score | Accuracy | Score
FrozenBiLM [32] | - | - | 32.2 | - | 16.8 | - | 24.7 | -
Video Chat [13] | 56.2 | 3.2 | 56.3 | 2.8 | 45.0 | 2.5 | 26.5 | 2.2
LLaMA Adapter [37] | - | - | 54.9 | 3.1 | 43.8 | 2.7 | 34.2 | 2.7
Video LLaMA [36] | - | - | 51.6 | 2.5 | 29.6 | 1.8 | 12.4 | 1.1
MovieChat [19] | 49.9 | 2.7 | 61.0 | 2.9 | 49.7 | 2.8 | 51.5 | 3.1
Video-ChatGPT [15] | 54.6 | 3.2 | 64.9 | 3.3 | 49.3 | 2.8 | 35.2 | 2.7
Vista-LLaMA (Ours) | 60.7 | 3.4 | 65.3 | 3.6 | 60.5 | 3.3 | 48.3 | 3.3
|  |  |  |  |  |  |  |

Table 1: Comparison with SoTA methods on zero-shot VideoQA. See §4.2 for more
details.

###  4.2 Comparison to State-of-the-Arts

For the zero-shot open-ended video question answering tasks, we compare our
model with FrozenBiLM [32], Video Chat [13], LLaMA Adapter [37], VideoLLaMA
[36], Video-ChatGPT [15], and MovieChat [19] in Tab. 1. FrozenBiLM adapts
frozen the bidirectional language model pre-trained on web-scale multi-modal
data, showing promising results in zero-shot VideoQA settings. Other compared
models are built on recent large auto-regressive language models. Despite pre-
existing models have produced substantial results, Vista-LLaMA consistently
outperforms them, achieving state-of-the-art (SoTA) performance across three
datasets: NExT-QA [28], MSVD-QA [30], and MSVTT-QA [30]. Our method obtains
the highest results, with 60.5% accuracy on MSRVTT, elevating the performance
of the second-best model by nearly 10%. Additionally, our method attains 60.7%
accuracy on NExT-QA, markedly superior to Video-ChatGPT [15], which employs
similar architectures as ours and is trained using the same datasets. These
results demonstrate Vista-LLaMA’s capability to comprehend video content and
produce precise answers.

Method | Cr. | Cs. | De. | Ct. | Te.
---|---|---|---|---|---
Video Chat [13] | 2.23 | 2.24 | 2.50 | 2.53 | 1.94
LLaMA Adapter [37] | 2.03 | 2.15 | 2.32 | 2.30 | 1.98
Video LLaMA [36] | 1.96 | 1.79 | 2.18 | 2.16 | 1.82
Video-ChatGPT [15] | 2.40 | 2.37 | 2.52 | 2.62 | 1.98
Vista-LLaMA (Ours) | 2.44 | 2.31 | 2.64 | 3.18 | 2.26
|  |  |  |  |

Table 2: Quantitative results on video-based text generation with different
video-language methods (§4.2). For clarity, five scores are reported (“Cr.”:
Correctness of Information, “Cs.”: Consistency, “De.”: Detail Orientation,
“Ct”: Contextual Understanding, “Te.”: Temporal Understanding).

We present the results of the evaluation of video-based text generation in
Tab. 2. The results reveal its competent performance across all key aspects
when compared with the recently introduced video-language models, Video
Chat[13], VideoLLaMA [36], and Video-ChatGPT [15]. Despite being trained with
identical datasets, our model outperforms Video-ChatGPT in four aspects. Our
mothod offers a more comprehensive interpretation, and its responses are more
in tune with the overarching context of the video content than comparable
approaches. By utilizing the EDVT-Attention and sequential temporal modeling
techniques, our model demonstrates a strong ability to generate text that is
contextually appropriate, detailed, and includes precise timing for video
inputs.

###  4.3 Abalation Study

Effect of Design Choices. We investigate the effect of our two designs,
including the equal-distance to visual-tokens (EDVT) attention and the
sequential visual projector. As shown in Figure 4, our model is tested on
NExT-QA [28], which comprises three different types of questions. The baseline
model employs LLaVA [14] as the language model, and Q-Former in BLIP-2 [12] as
the visual projector. We first incorporate temporal modeling into the
baseline, utilizing the projected tokens from the previous frame as the query
tokens to generate visual tokens for the current frame, a setup denoted as “w
Seq” in Figure 4. Further enhancing this version with EDVT-Attention, we
introduce “w Seq&EDVT”, ultimately forming our final Vista-LLaMA. The results
affirm the efficacy of our design across all question types.

Figure 4: Comparison of different design choices on NExT-QA [28](§4.3). For
clarity, accuracy of base model and two variants are given (“Baseline”: the
frozen LLaVA [14] with trainable Q-Former [12], “w Seq”: base model with
sequential visual projector, “w Seq&EDVT”: base model with both sequential
visual projector and EDVT-Attention).

EDVT-Attention. We here validate the effectiveness of our core EDVT-Attention.
Table 3 reports the comparison results of different models in combination with
our EDVT-Attention on NExT-QA [28]. Initially, we integrate the EDVT-Attention
into Video-ChatGPT [15], where only the attention component is substituted
with our EDVT-Attention during training. Further deployment of the EDVT-
Attention into Video-ChatGPT [15] yields significant performance gains
(_e.g.,_ 65.1% escalates to 72.8% for causal questions). The accuracy is
enlarged across all three setting types, indicating that our EDVT-Attention
considerably enhances the multi-modal understanding in large language models.

We further illustrate the attention weights of both EDVT-Attention and
conventional attention in Figure 5. For this experiment, we employ 128 visual
tokens and integrate the attention weights into the first four tokens to
enhance visualization. The attention weights assigned to visual tokens are
markedly greater in our EDVT-Attention compared to those in conventional
attention. This indicates that the impact of visual tokens on language tokens
is considerably more substantial under the proposed EDVT-Attention, shedding
light on why the performance notably improves with our design.

| NExT-QA [28]
---|---
Method | Tem. | Cau. | Des. | Avg.
Video-ChatGPT [15] | 37.6 | 65.1 | 54.9 | 54.6
\+ EDVT-Attention |  39.5 (+1.9) |  72.8 (+7.7) | 54.8 |  59.3 (+4.7)
Vista-LLaMA (ViT-L-14) | 34.0 | 69.1 | 42.2 | 53.6
\+ EDVT-Attention |  36.8 (+2.8) |  72.2 (+3.1) |  47.7 (+5.5) |  56.5 (+2.9)
Vista-LLaMA (EVA-CLIP-g) | 34.3 | 65.8 | 55.9 | 54.1
\+ EDVT-Attention |  40.7 (+6.4) |  72.3 (+6.5) |  57.0 (+1.1) |  59.7 (+5.6)
|  |  |  |

Table 3: Comparison of EDVT-Attention design with diffent visual encoders on
NExT-QA [28] (§4.3). For clarity, accuracy of all questions and three types of
questions are reported (“Tem.”: temporal, “Cau.”: causal, “Des.”: descriptive,
“Avg.”: average).

(a) Vanilla attention

(b) EDVT-Attention (Ours)

Figure 5: Comparison of attention weights for varing context lengths. Lighter
colors represent higher weights. To improve clarity, we have combined visual
token weights into the first four tokens. We recommend zooming in for optimal
viewing.

|  NExT-QA [28]
---|---
Visual projector | Tem. | Cau. | Des. | Avg.
Linear Projector | 37.6 | 65.1 | 54.9 | 54.6
Q-Former (BERT init.) | 35.2 | 62.7 | 49.2 | 51.8
Q-Former (BLIP-2 init.) | 34.3 | 65.8 | 55.9 | 54.1
SeqQ-Former (BLIP-2 init.) | 36.2 | 68.5 | 51.1 | 55.4

Table 4: Comparison of different visual projectors on NExT-QA [28]. The linear
projector is initialized with pre-trained weights in LLaVa [14]. BERT init.
and BLIP-2 init. indicate that the visual projector is initialized with
weights from BERT [5] and BLIP-2 [12]. SeqQ-Former is the proposed sequential
visual projector. See §4.3 for more details.

Visual Projector. The visual projector is trainable within the model.
Therefore, we have examined the impacts of different visual projector designs,
as shown in Table 4. We evaluated three visual projector variants. The
Q-Former, initialized using BLIP-2 [12], considerably outperforms the Q-Former
that was initialized with BERT [5]. This indicates that visual projectors,
when pre-trained on image-text pairs, can significantly enhance video
comprehension. The model’s performance peaks when integrating the sequential
design, referred to as “SeqQ-Former” in the figure, outperforming all other
visual projectors in terms of accuracy.

Figure 6: The effect of training visual tokens on NExT-QA [28]. Accuracy of
all questions and three types of questions, including temporal (Tem.),
descriptive (Des.), and causal (Cau.), are presented with different colors.
Figure 7: Visualization results on different video questions. The questions
and annotated answers are located on the left side. The generated text from
Video-ChatGPT [29] and our model is presented in the green and orange boxes,
respectively. See §4.3 for more details.

Number of Training Visual Tokens. We assess the impact of visual tokens on
NExT-QA as illustrated in Figure 6. In this study, a frame is converted into
32 visual tokens using Q-Former. Frames were sampled at various timestamps
throughout the training process. The overall accuracy improves as more visual
tokens are incorporated into the language model. However, for temporal
queries, precision declined with an increased number of visual tokens. The
accuracy of temporal reasoning questions is notably lower than that of other
question types. Temporal reasoning presents more difficulties, and the
language model may not excel at the temporal modeling of visual tokens. In
terms of descriptive question types, the accuracy steadily increases as the
model gains more visual information.

Quantitative Results. We visualize the generated responses of Video-ChatGPT
[15] and Vista-LLaMA for different videos in Figure 7. The frames sampled at
different timestamps is presented, with questions listed below the images. In
the first video, the mobile phone is turned, sequentially exposing each side
over time. The question asks for the side of the mobile phone at the end of
the video. Video-ChatGPT gives the wrong response, as the camera is never
present at the bottom left side in the video. The hallucination occurs in
Video-ChatGPT, whereas our model predicts the answer correctly. The responses
in three cases demonstrate that Video-ChatGPT provides unrelated answers that
do not correspond to the video content. In the second case, the person in pink
is always standing while the man is sitting on the ground, but Video-ChatGPT
incorrectly states that the person in pink is also sitting on the ground. In
the third video, the man is consistently walking up the mountain and never
walking down. However, Video-ChatGPT falsely claims that the man continues
walking down the mountain. It is quite common for the video language model to
produce unrelated responses. In contrast, our model provides the correct and
reliable responses, aligning with the video content. This demonstrates that
our model significantly reduces hallucination in video understanding and
delivers more accurate responses.

##  5 Conclusion

In this work, we present Vista-LLaMA to improve the video understanding in
large language model. A new vision-aware attention is introduced to maintain
same relative distance between all visual tokens and language tokens. In
addition, we propose a sequential visual projector to map video into the
language space to enable temporal modeling. Experiments on several video
question answering task demonstrate that our designs significant improves the
current SoTA. Current work is built on the pre-trained image-text
architecture, the power of our designs could be enlarged when applied in pre-
training stage. We would consider evaluating the model in more general multi-
modal tasks and evaluating the capacity for vision language pre-training.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In _NeurIPS_ , pages 23716–23736, 2022.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In _NeurIPS_ , pages 1877–1901, 2020.
  * Chen and Dolan [2011] David L. Chen and William B. Dolan.  Collecting highly parallel data for paraphrase evaluation.  In _Annual Meeting of the Association for Computational Linguistics_ , 2011.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2023.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dhuliawala et al. [2023] Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston.  Chain-of-verification reduces hallucination in large language models.  _arXiv preprint arXiv:2309.11495_ , 2023.
  * Fabian Caba Heilbron and Niebles [2015] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 961–970, 2015.
  * Floridi and Chiriatti [2020] Luciano Floridi and Massimo Chiriatti.  Gpt-3: Its nature, scope, limits, and consequences.  _Minds and Machines_ , 30:681–694, 2020.
  * Gao et al. [2022] Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yezhou Yang, and Mike Zheng Shou.  Mist : Multi-modal iterative spatial-temporal transformer for long-form video question answering.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 14773–14783, 2022.
  * Gupta and Kembhavi [2023] Tanmay Gupta and Aniruddha Kembhavi.  Visual programming: Compositional visual reasoning without training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14953–14962, 2023.
  * Heilbron et al. [2015] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 961–970, 2015.
  * Li et al. [2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _ArXiv_ , abs/2301.12597, 2023a.
  * Li et al. [2023b] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  _arXiv preprint arXiv:2305.06355_ , 2023b.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _ArXiv_ , abs/2304.08485, 2023.
  * Maaz et al. [2023] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  _arXiv preprint arXiv:2306.05424_ , 2023.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Shen et al. [2023] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  _arXiv preprint arXiv:2303.17580_ , 2023.
  * Song et al. [2023] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al.  Moviechat: From dense token to sparse memory for long video understanding.  _arXiv preprint arXiv:2307.16449_ , 2023.
  * Su et al. [2021] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding.  _arXiv preprint arXiv:2104.09864_ , 2021.
  * Sun et al. [2023] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  _arXiv preprint arXiv:2303.15389_ , 2023.
  * Surís et al. [2023] Dídac Surís, Sachit Menon, and Carl Vondrick.  Vipergpt: Visual inference via python execution for reasoning.  _arXiv preprint arXiv:2303.08128_ , 2023.
  * Tapaswi et al. [2015] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler.  Movieqa: Understanding stories in movies through question-answering.  In _CVPR_ , pages 4631–4640, 2015.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _ArXiv_ , abs/2302.13971, 2023.
  * Tsimpoukelli et al. [2021] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill, and Zacharias Janssen.  Multimodal few-shot learning with frozen language models.  In _NeurIPS_ , 2021.
  * Workshop et al. [2022] BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Luccioni, et al.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_ , 2022.
  * Wu et al. [2023] Chenfei Wu, Sheng-Kai Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  _ArXiv_ , abs/2303.04671, 2023.
  * Xiao et al. [2021] Junbin Xiao, Xindi Shang, Angela Yao, and Tat seng Chua.  Next-qa: Next phase of question-answering to explaining temporal actions.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9772–9781, 2021.
  * Xu et al. [2023] Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  _ArXiv_ , abs/2304.01196, 2023.
  * Xu et al. [2017] D. Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  _Proceedings of the 25th ACM international conference on Multimedia_ , 2017.
  * Xu et al. [2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5288–5296, 2016.
  * Yang et al. [2022] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Zero-shot video question answering via frozen bidirectional language models.  _Advances in Neural Information Processing Systems_ , 35:124–141, 2022.
  * Ye et al. [2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yi Zhou, Junyan Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qiang Qi, Ji Zhang, and Feiyan Huang.  mplug-owl: Modularization empowers large language models with multimodality.  _ArXiv_ , abs/2304.14178, 2023.
  * Yu et al. [2019] Zhou Yu, D. Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao.  Activitynet-qa: A dataset for understanding complex web videos via question answering.  In _AAAI_ , 2019.
  * Zeng et al. [2022] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_ , 2022.
  * Zhang et al. [2023a] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_ , 2023a.
  * Zhang et al. [2023b] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_ , 2023b.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models.  _ArXiv_ , abs/2205.01068, 2022.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _ArXiv_ , abs/2304.10592, 2023.

Supplementary Material

##  6 Additional Results

Visualization Results. We present additional visualization results for
different video questions in Figures 8 and 9. In comparison to Video-ChatGPT
[15], our Vista-LLaMA provides more reasonable answers and descriptions that
align better with the video content. Video-ChatGPT often responds with
irrelevant information, resulting in hallucinations. For example, in the first
video where the baby appears tired, Video-ChatGPT incorrectly states that the
baby was eating a snack, even though there is no eating action shown in the
video. We have more examples that demonstrate the improvement of our method on
NExT-QA [28]. We only showcase a small portion of these cases to highlight the
differences. Our methods achieve much better performance and offer more
reliable replies due to the proposed EDVT-Attention, which maintains an equal
distance to all visual tokens.

Figure 8: Visualization results on different video questions. Figure 9:
Visualization results on different video questions.

(a) Vanilla attention

(b) EDVT-Attention (Ours)

Figure 10: Comparison of attention weights for varing context lengths in
different layers. Lighter colors represent higher weights. To improve clarity,
we have combined visual token weights into the first four tokens. We recommend
zooming in for optimal viewing.

Attention Weights in Different Layers. In Figure 10, we present the attention
weights in different layers. Different from Figure 5, here we sum instead of
average the attention weights of 32 heads to present clear comparison. From
the figure, we show that the attention weights between text tokens in the
EDVT-Attention are larger than attention weights in Vanilla attention. It
indicates that the EDVT-Attention strengthen the impact of visual tokens on
generating text.

In Figure 10, we visualize the attention weights in different layers. In
contrast to Figure 5, where we averaged the attention weights of 32 heads,
here we present the sum for a clearer comparison. The figure reveals that the
attention weights between text tokens in the EDVT-Attention are greater than
those in Vanilla attention. This suggests that the EDVT-Attention enhances the
influence of visual tokens on text generation.

|  |  NExT-QA [28]
---|---|---
Query | Key | Tem. | Cau. | Des. | Avg.
RoPE | RoPE | 34.3 | 65.8 | 55.9 | 54.1
FixVPE | FixVPE | 37.0 | 70.5 | 56.7 | 57.6
RoPE | EDVT | 32.2 | 48.1 | 41.8 | 42.0
EDVT | EDVT | 40.7 | 72.3 | 57.0 | 59.7

Table 5: Comparison of positional embedding strategies on NExT-QA [28]. We
provide a list of various positional embedding strategies used for query and
key vectors in the attention layer. The “RoPE” indicates the use of rotary
positional embedding for all visual and text tokens. The “FixVPE” refers to
the fixed position rotary positional embedding used for all visual tokens.
Lastly, “EDVT” indicates that the rotary positional embedding is exclusively
applied to text tokens.

Positional Embedding Study. We explored various strategies for positional
embedding in the attention layer, focusing on the query and key vectors.
According to Tab. 5, the model achieves the highest accuracy when only text
tokens have rotary positional embedding applied to both the query and key
vectors. When only the query vectors have RoPE applied and the key vectors do
not, the performance decreases significantly. This is because the relative
distance is compromised when only the query has RoPE. We also attempted to use
fixed positional embedding on all visual tokens. Unlike in DEVT, all visual
tokens have RoPE applied with the same position index of 0. Compared to the
baseline, this modification also improves performance on different question
types. However, it is still inferior to our design. This demonstrates that the
proposed EDVT design truly enhances video understanding in LLMs.

##  7 Movie Evaluation

Method | Overall | Description | Temporality | Spaciality | Intention | Perception
---|---|---|---|---|---|---
| Score | Accuracy | Score | Accuracy | Score | Accuracy | Score | Accuracy | Score | Accuracy | Score | Accuracy
MovieChat | 2.11 | 20.86 | 2.41 | 23.67 | 1.97 | 16.32 | 1.98 | 16.40 | 2.41 | 30.19 | 1.97 | 21.80
Video-LLAMA | 2.27 | 23.17 | 2.31 | 19.30 | 2.12 | 16.35 | 2.19 | 21.95 | 2.47 | 31.94 | 2.35 | 27.70
Video-ChatGPT | 2.60 | 34.11 | 2.55 | 26.24 | 2.60 | 34.11 | 2.50 | 30.62 | 2.94 | 46.36 | 2.43 | 31.77
Vista-LLaMA (Ours) | 2.98 | 44.90 | 2.79 | 31.46 | 2.92 | 46.22 | 2.73 | 35.63 | 3.38 | 61.89 | 3.12 | 47.49
|  |  |  |  |  |  |  |  |  |  |  |

Table 6: Performance Comparison on CineClipQA of different methods on various
classifications.  Figure 11: CineClipQA, a novel dataset meticulously crafted
to probe the capabilities of visual language models in comprehending and
interpreting plot-driven video content. Figure 12: CineClipQA, the detailed
description for the dataset. Figure 13: CineClipQA, the detailed description
for the dataset. Figure 14: CineClipQA, the detailed description for the
dataset. Figure 15: CineClipQA, the detailed description for the dataset.
Figure 16: CineClipQA, the detailed description for the dataset.

Dataset Collection. In this paper, we introduce a new dataset named
CineClipQA. The CineClipQA dataset encompasses a collection of 153 curated
video clips, derived from five movies that span diverse genres and
storytelling styles. Each clip, representing one or more distinct segment of
the movie plot, is accompanied by a set of 16 tailored questions, thereby
totaling 2,448 questions in various dimensions, as is presented in Figure 11.
The question consists of two parts: System Prompt and Questions. The System
Prompt contains basic information about key characters in the current video
clip and provides prompts for the initial actions of characters when
necessary. The Questions are primarily divided into five categories:
Recognizing, Temporality (Prediction), Spaciality (Interaction), Intention,
and Perception. Specifically, Recognizing includes questions about Location
and Action; Temporality encompasses questions about the next action, the
previous action, simultaneous actions, and predicted actions; Spaciality
involves questions about spatial information between Object and Human;
Intention involves three similar types of questions about the purpose of
actions; finally, Perception examines the recognition of emotions and
inquiries about the ”how” (approaches, manners…). At last, we provide a
detailed explanation and corresponding case for all 16 types, shown in Figure
12, Figure 13, Figure 14, Figure 15, Figure 16.

Method Comparison. In our study, we evaluated the performance of several SOTA
methods, including MovieChat, VideoLLAMA, Video-ChatGPT, and our own Vista-
LLaMA, on the newly proposed CineClipQA dataset. This dataset, designed to
assess comprehension of complex movie content, encompasses challenges in high
content complexity, extensive scene variety, and prolonged temporal
dimensions. The empirical results from our evaluations demonstrate that all
methods, including our Vista-LLaMA, achieved state-of-the-art (SOTA)
performance on the CineClipQA dataset. This outcome unequivocally confirms the
superior ability of our approach in understanding videos with intricate
content, wide-ranging scenes, and extended time frames.Notably, across all
tested models, the highest accuracy was observed in the Intention category of
the CineClipQA dataset. This suggests a particularly effective grasp of human
behavioral reasoning, likely attributed to the rich prior knowledge embedded
within these large language models (LLMs). The Intention category, by its
nature, demands an in-depth analysis of purpose and motive behind actions
depicted in the video clips, a task which seems to align well with the
inherent strengths of current LLMs.Furthermore, this finding underscores the
potential of LLMs in bridging the gap between mere visual recognition and
deeper narrative understanding. The ability of these models to not only
identify characters and actions but also infer underlying intentions is
indicative of their advancing sophistication. It highlights a significant
stride in the evolution of AI, where models are increasingly capable of
nuanced interpretation akin to human-like understanding.

##  8 Discussion

Advantages. In this study, we present two innovations, namely the EDVT-
Attention and the sequential visual projector, aimed at enhancing video
comprehension in LLMs. Our evaluation primarily focuses on zero-shot question
answering benchmarks. The model used is built upon LLaVA [14], which is pre-
trained and then fine-tuned with video instruction data. Vista-LLaMAachieves a
notable enhancement in the proposed innovations when tested on NExT-QA [28]
and MSRVTT-QA [31]. Additionally, we conduct several ablations to illustrate
the effectiveness of our innovations. The outcomes demonstrate the significant
potential of our approach to enhance video comprehension with LLMs.

Limitations. There are also limitations in our work. For the VideoQA task, the
evaluation process is assisted with GPT-3.5, which may result in some false
judgments. GPT-4 might provide more accurate evaluations, but it comes at a
higher cost since it is 20 times more expensive than GPT-3.5. Additionally,
evaluating with GPT-4 requires the use of huge tokens, further increasing the
expense. Furthermore, the evaluation speed is limited by query restrictions,
and GPT-4 takes more time compared to training. We have evaluated a few cases
using GPT-3.5, and the response has been reasonable and the accuracy has
remained stable. When the same results are evaluated on NExT-QA three times,
the variance is lower than 0.5 in the experiments.

Since this work only focuses on fine-tuning rather than pre-training, the full
potential of EDVT-Attention may not be fully explored. EDVT-Attention can also
be utilized for image-text related tasks. However, the impact of EDVT-
Attention on pre-training, image-text related tasks, or other multi-modal
tasks is not investigated in this manuscript. Additionally, the use of rotary
positional embedding in some LLMs restricts the applicability of the current
design. In this work, the rotary positional embedding is removed to ensure the
same distance to visual tokens in decoder layers of LLMs. There may be
alternative dynamic designs that can achieve this objective without
eliminating the positional embedding. All these aspects are worth considering.
Although the number of hallucination cases is reduced with our method, there
are still instances where the model provides irrelevant replies. Further
studies are necessary to address this issue. To enhance the current
manuscript, our future work will focus on developing more general designs for
practical cases.