# Realistic continuous-variable quantum teleportation using a displaced Fock
state channel

Deepak<EMAIL_ADDRESS>Arpita Chatterjee<EMAIL_ADDRESS>Department of Mathematics, J. C. Bose University of Science and Technology,
YMCA, Faridabad 121006, India

###### Abstract

We investigate ideal and non-ideal continuous-variable quantum teleportation
protocols realized by using an entangled displaced Fock state resource. The
characteristic function formulation is applied to measure the relative
performance of displaced Fock state for teleporting squeezed and coherent
states. It is found that for such single-mode input fields, the average
fidelity remains at the classical threshold, suggesting that the displaced
Fock states are not advantageous for teleportation. We also discuss the major
decoherence effects, caused by the inaccuracy in Bell measurements and photon
losses for the propagation of optical fields via fibre channels. The changes
in the teleportation fidelity are described by adjusting the gain factor
($g$), reflectivity ($R$), mode damping ($\tau$), and the number of thermal
photons ($n_{\mathrm{th}}$). The possibility of successful teleportation can
be optimized by fixing these realistic parameters.

###### keywords:

displaced Fock state, characteristic function, ideal and realistic quantum
teleportation, fidelity

##  1 Introduction

Displaced Fock state (DFS) [1, 2, 3], a highly nonclassical quantized
electromagnetic field state, owns a significant role in quantum optics,
especially for representing various quasiprobability distributions. It acts as
the eigenstate of the Jaynes-Cummings model consisting of coherently driven
atoms in the cavity QED system [4]. Cahill and Glauber introduced the
displaced Fock state to express different quasiprobabilities in a simplified
way [5, 6]. Later Boiteux and Levelut [7] suggested a specific notation for
these types of states, named as semi-coherent states and investigated their
basic properties. They derived a generalized completeness relation for these
states. Roy and Singh [8] and Satyanarayana [9] considered these states in the
framework of generalized coherent states and analyzed different properties. de
Oliveira et. al. [10] proposed these states with major attention on their
physical properties as amplified occupation number states and labeled them as
displaced number states. C. F. Lo discussed the possibility of producing
displaced and squeezed Fock states from Fock state using a general driven
time-dependent oscillator [11]. Marchiolli et. al. studied in detail the
quantum interference effects induced by the superposition of DFSs in the
trapped-ion system [12]. A number of optical schemes have been presented by
Podoshvedov to understand the irreversible analog of the Hadamard gate while
extracting displaced number states from squeezed coherent state [13]. An
experimental realization of the displaced number state is reported in [14],
where Ziesel et. al. first prepared Fock states ($\ket{0}$, $\ket{1}$ and
$\ket{2}$) by using a laser-driven technique to lift the Jaynes-Cummings
ladder, followed by a displacement (of amplitudes up to $\alpha\approx 2.8$)
fabricated by means of a sudden shift in the electrostatic trapping potential.
Recently, Priya et. al. [15] investigated the collective effect of adding and
subtracting multiple photons on the nonclassicality and phase properties of a
DFS. It seems that different characters of DFS are studied extensively.
However, it may be noted that a few initiatives have been taken to expose the
applicability of displaced Fock state as a quantum channel in continuous-
variable (CV) teleportation. In the present work, using the characteristic
function formalism, we investigate how useful the displaced Fock states are
for CV teleportation in ideal and realistic conditions.

Theoretically, a displaced Fock state is developed by applying the unitary
displacement operator to a Fock state as $\ket{\alpha,n}\equiv
D(\alpha,\alpha^{*})\ket{n}$, $n=0,1,2,\ldots$ and $\alpha$ is the amount of
displacement [16]. This procedure is equivalent of adding a non–zero complex
quantity $\alpha$ to the Fock state $\ket{n}$, having $n$ number of photons
and thus developing a new type of quantum state $\ket{\alpha,n}$, which
comprises the coherent state $\ket{\alpha}\equiv\ket{\alpha,0}\equiv
D(\alpha,\alpha^{*})\ket{0}$ as well as the Fock state
$\ket{n}\equiv\ket{0,n}\equiv\frac{1}{\sqrt{n!}}a^{\dagger n}\ket{0}$ as
particular cases. These states configure two types of fundamental sets: one
because of their coherent state character, and next as they can act as number
states in a displaced structure [1]. In the first formulation, DFSs describe
an overcomplete basis (just similar to the coherent states) and in the next,
the states form a complete basis (because they are orthonormal
$\langle{\alpha,m}|{\alpha,n}\rangle=\delta_{m,n}$). Using these, many quantum
states can be reported as an infinite superposition of displaced number
states. Interest in studying the characteristics of displaced Fock state has
been boosted because of their robust nonclassical features, such as quantum
entanglement and negative values of the quasiprobability phase-space
distributions, which are urgently needed for implementing quantum information
and communication protocols [17, 18, 19, 20, 21] in an efficient way and for
quantum estimation tasks [22]. In this paper, we attempt to figure out the
performance of the displaced Fock state as a quantum channel for continuous-
variable quantum teleportation. The input state is assumed to be either
coherent or squeezed state in our approach, and the success under the
Braunstein and Kimble protocol is measured in terms of the average fidelity
calculated for transporting that single-mode optical state. At the same time,
we try to understand the functioning of different realistic parameters in the
successful teleportation of a single-mode input state through a two-mode
entangled channel.

Bennett et. al. introduced the concept of quantum teleportation in the context
of discrete variables [23], and later it was realized experimentally by
Bouwmeester [24] and Popescu [25]. Vaidman extended the idea of quantum
teleportation in the continuous variable regime [26]. Braunstein and Kimble
provided the actual quantum-optical procedure (Braunstein and Kimble (BK)
protocol) for transmitting quadrature amplitudes of a light field from sender
to receiver[27], which was experimentally executed by Furusawa et. al. soon
after [28, 29, 30]. In a standard teleportation protocol, an entangled
resource comprising of two modes $A$ and $B$, is shared between two users,
Alice and Bob, and a single-mode input state $\ket{\mathrm{in}}$ in the
sender’s place is the state to be teleported. The input state
$\ket{\mathrm{in}}$ coupled with mode $A$ of the entangled resource by a 50:50
beam splitter, results the states $\ket{\mathrm{out}^{\prime}}$ and
$A^{\prime}$ at the output modes of the beam-splitter. A destructive homodyne
measurement is carried out by Alice on the output modes. The results thus
obtained are transmitted to Bob in a classical way; Bob implements a unitary
displacement on mode $B$, which leads to the desired teleported state. This
unique BK protocol has been presented in the literature in different ways [31,
32, 33]. Advancement in the theoretical description as well as the
experimental implementation of CV quantum teleportation using BK protocol lies
at the heart of interest of the quantum information community because of the
accelerating uses of CV entangled resources in quantum information processing
and quantum computing [34]. In this work, our main objective is to inspect how
the performance of CV teleportation protocol with DFS resource is affected by
decoherence emerging from photon losses in fibre and imperfections in Bell
measurements.

The paper is arranged as follows: in Sec. 2, the relative performance of
displaced Fock state for sending different single-mode input states in ideal
condition is exhibited. In the next section, we concentrate on the
effectiveness of the quantum information protocol in the presence of certain
fixed values of various realistic parameters. In Sec. 4, we draw a conclusion
based on the results obtained and discuss about some possible future
extensions of the work.

##  2 Ideal quantum teleportation with displaced Fock state channel

Teleportation, an indispensable strategy for processing of quantum
information, dispatches an unknown qubit from the sender to the receiver,
travelling across an entangled channel shared between those two parties [23].
The measurement of teleportation can be performed in terms of the
characteristic functions of the arbitrary input and resource states involved
in the process [35]. The characteristic function formalism is especially
helpful while managing non-Gaussian resources, because it inevitably works on
the computational challenges. It is appropriate to use the fidelity of
teleportation ${F}$ to evaluate the success probability of a protocol. The
fidelity is a state-dependent quantity, which computes the overlap between the
input state $\rho_{\mathrm{in}}$ and the teleported state
$\rho_{\mathrm{out}}$, that is,
${F}=\mathrm{Tr}[\rho_{\mathrm{in}}\,\rho_{\mathrm{out}}]$. In the case of
${F}=1$, these two states are identical, which means perfect teleportation is
achieved [36]. In the characteristic function description, the fidelity is
defined as [37, 38]

$\displaystyle{F}=\frac{1}{\pi}\int{d^{2}\gamma}\,{\chi}_{\mathrm{in}}(\gamma)\,{\chi}_{\mathrm{out}}(-\gamma),$
(1)

where

$\displaystyle{\chi}_{\mathrm{out}}(\gamma)={\chi}_{\mathrm{in}}(\gamma){\chi}_{\mathrm{ch}}(\gamma^{*},\gamma),$
(2)

${\chi}_{\mathrm{in}}$ and ${\chi}_{\mathrm{ch}}$ are the characteristic
functions of the input quantum state to be transported and the two-mode
quantum channel, respectively. To calculate the average fidelity of CV
teleportation, it is necessary to find out the characteristic functions of
various associated input states and the two-mode entangled channel. For a
single-mode optical field, the symmetric characteristic function
${\chi}(\gamma)$ can easily be derived by using
${\chi}(\gamma)=\mathrm{Tr}[D(\gamma)\rho]$, where $D(\gamma)$ is the usual
displacement operator and $\rho$ is the density matrix of the related state.

Using the mathematical formula [37]

$\displaystyle\langle
m|D(\alpha)|n\rangle={\left(\frac{n!}{m!}\right)}^{1/2}{\alpha}^{m-n}e^{-\frac{1}{2}|\alpha|^{2}}L_{n}^{(m-n)}(|\alpha|^{2}),$
(3)

where $L_{n}^{(m-n)}(|\alpha|^{2})$ is the associated Laguerre polynomial, the
characteristic function for the coherent state $\ket{\alpha}$ is obtained as

$\displaystyle{\chi}_{\mathrm{coh}}(\gamma)=e^{-\frac{1}{2}|\gamma|^{2}}e^{(\alpha^{*}\gamma-\alpha\gamma^{*})}$
(4)

and for squeezed state $\ket{\varepsilon}=S(\varepsilon)\ket{0}$
($\varepsilon=re^{i\varphi}$), it is

$\displaystyle{\chi}_{\mathrm{squ}}(\gamma)=e^{-\frac{1}{2}|\xi|^{2}},$ (5)

with $\xi=\gamma\cosh{r}+\gamma^{*}e^{i\varphi}\sinh{r}$ [37]. A two-mode
displaced Fock state is defined by [39]

$\displaystyle\ket{\alpha_{1},\alpha_{2},n_{1},n_{2}}=D(\alpha_{1})D(\alpha_{2})\ket{n_{1},n_{2}},$
(6)

where $\ket{n_{1},n_{2}}\equiv\ket{n_{1}}\otimes\ket{n_{2}}$ is a bimodal Fock
state. The characteristic function for the two-mode DFS is obtained as

$\displaystyle{\chi}_{\mathrm{DFS}}(\gamma_{1},\gamma_{2})$ $\displaystyle=$
$\displaystyle\langle D(\gamma_{1})D(\gamma_{2})\rangle$ (7) $\displaystyle=$
$\displaystyle
e^{-\frac{1}{2}|\gamma_{1}|^{2}-\frac{1}{2}|\gamma_{2}|^{2}}e^{(\alpha_{1}^{*}\gamma_{1}-\alpha_{1}\gamma_{1}^{*})}e^{(\alpha_{2}^{*}\gamma_{2}-\alpha_{2}\gamma_{2}^{*})}$
$\displaystyle L_{n_{1}}(|\gamma_{1}|^{2})L_{n_{2}}(|\gamma_{2}|^{2})$

An apparent way to calculate the fidelity function in (1) for a number of
different input states and an entangled DFS resource is by applying the
integration over Gaussian averages as

$\displaystyle\int\frac{d^{2}z}{\pi}\exp(\xi|z|^{2}+\nu z+\eta
z^{*})=\frac{1}{\sqrt{\xi^{2}}}\exp(-\frac{\xi\nu\eta}{\xi^{2}}),$

[see A for detailed calculation]

###  2.1 For input coherent state $\ket{\alpha}$

The fidelity for input coherent state is obtained as

${F}=\frac{1}{\pi}\int{d^{2}\gamma}\,e^{-|\gamma|^{2}}\,e^{-(\alpha_{1}^{*}-\alpha_{2})\gamma^{*}-(\alpha_{2}^{*}-\alpha_{1})\gamma-|\gamma|^{2}}L_{n_{1}}(|\gamma|^{2})L_{n_{2}}(|\gamma|^{2})$
(8)

Figure 1: Fidelity ${F}$ as a function of the DFS parameters $\alpha_{1}$ and
$\alpha_{2}$ and for different photon numbers in mode 1 and mode 2 such as
$n_{1}=n_{2}=0$ (red dash-dotted line), $n_{1}=n_{2}=1$ (green dotted line),
$n_{1}=n_{2}=5$ (blue dashed line) and $n_{1}=3$, $n_{2}=7$ (magenta solid
line), respectively.

In Fig. 1, the dependence of fidelity on the DFS parameters $\alpha_{1}$ and
$\alpha_{2}$ is shown for different $n_{1}$, $n_{2}$ values. It is important
to note that the complex amplitude $\alpha$ of the coherent state to be
teleported is not arising directly in the fidelity equation. For larger values
of $n_{1}$ and $n_{2}$, the fidelity decreases steadily. If $n_{1}\neq n_{2}$,
the fidelity is less as compared to the fidelity attained when $n_{1}=n_{2}$.
Unfortunately, the maximum fidelity obtained here is 0.499 which is below the
classical benchmark. The maximum fidelity is realized at $n_{1}=n_{2}=0$, that
means when the channel becomes a purely bipartite coherent state.

###  2.2 For input squeezed state $\ket{\varepsilon}$

Figure 2: Fidelity in ideal condition with respect to the squeezing parameter
$r$ and for DFS parametric amplitudes as (a) $\alpha_{1}=\alpha_{2}=0$, (b)
$\alpha_{1}=0$ and $\alpha_{2}=0.5$, (c) $\alpha_{1}=1$ and $\alpha_{2}=0$. In
(a)-(c), the squeezing parameter $\varphi$ is 0 whereas $\varphi$ corresponds
to $\pi$ in (d)-(f). Figure 3: Average fidelity (${F}$) for teleportation of a
single-mode squeezed state (of different amplitudes) by using a two-mode
displaced Fock state. ${F}$ is shown as a function of DFS parameters
$\alpha_{1}$ and $\alpha_{2}$ and for different squeezing parameter values (a)
$r=0$ and $\varphi=0$, (b) $r=1$ and $\varphi=0$, (c) $r=2$ and $\varphi=0$.
In (d)-(f), ${F}$ corresponds to $\varphi=\pi$ and same $r$ as in (a)-(c),
respectively.

The Fidelity for input squeezed state is plotted in Fig. 2 with respect to the
squeezing parameter $r$ and for different coherent amplitudes of the two-mode
DFS channel. In this case, the maximum fidelity (0.497 in Fig. 2(a) and 0.499
in Fig. 2(d)) values are attained when both the DFS parameters $\alpha_{1}$
and $\alpha_{2}$ are zero. For any of the $\alpha_{1}$ and $\alpha_{2}$
values, the fidelity is maximum when $n_{1}=n_{2}=0$, and it decreases
steadily as $n_{1}$ and $n_{2}$ are increasing. The DFS channel seems to be
more suitable for teleporting a squeezed state while $\varphi$ changes from 0
to $\pi$.

Fig. 3 gives the variation of ${F}$ as a function of the complex amplitudes
$\alpha_{1}$ and $\alpha_{2}$ corresponding to the squeezing parameters
$r=0,1,2$. We note that as $r$ increases from 0 to 2, ${F}_{\mathrm{max}}$ is
dropped from 0.499 to 0.132. Even for highly squeezed states ($r=2$), no
enhancement in the maximum value of the average fidelity above classical limit
is seen in Fig. 3. Thus the displaced Fock state is not resourceful for
quantum teleportation of a squeezed state.

##  3 Non-ideal quantum teleportation with displaced Fock state channel

In this section, we study the performance of the DFS resources in realistic
Braunstein-Kimble CV teleportation protocol [40]. The amount of success in a
teleportation arrangement is strongly affected by the inefficiency of the
homodyne detectors performing the Bell-measurement and the decoherence effect
caused by the photon losses in noisy channel. A diagrammatic proposal for the
non-ideal teleportation protocol is given in [38, 41] where at the beginning,
the input and entangled resource states are considered to be pure.

For any arbitrary single-mode input state and an entangled channel, the
collective effect of the passage of the field mode in a damping channel and
non-ideal Bell-measurement decides the characteristic function
${\chi}_{\mathrm{out}}(x_{2},p_{2})$ of the final output state as follows
[38]:

$\displaystyle{\chi}_{\mathrm{out}}(x_{2},p_{2})$ $\displaystyle=$
$\displaystyle{\chi}_{\mathrm{in}}(gTx_{2},gTp_{2})\,\,{\chi}_{\mathrm{ch}}(gTx_{2},-gTp_{2};e^{-\frac{\tau}{2}}x_{2},e^{-\frac{\tau}{2}}p_{2})$
(9) $\displaystyle\exp[-\frac{1}{2}{\Gamma_{\tau,R}}(x_{2}^{2}+p_{2}^{2})],$

where $g$ is the gain factor, $\Upsilon$ is the mode damping rate,
$\tau=\Upsilon t$, $\Gamma_{\tau,R}$ is the thermal phase-space covariance
given by

$\displaystyle\Gamma_{\tau,R}=(1-e^{-\tau})\left(\frac{1}{2}+n_{\mathrm{th}}\right)+g^{2}R^{2}$
(10)

and $T^{2}$ and $R^{2}$ are the transmissivity and reflectivity of the beam
splitter that characterizes the losses, respectively, with $T^{2}+R^{2}=1$.
This equation (9) highlights how the teleportation protocol is affected by the
two sources of noise, associated with the damping rate $\Upsilon$ and the
reflectivity parameter $R^{2}$. A scaling factor $T$ is included in the
arguments of the input characteristic function ${\chi}_{\mathrm{in}}$ and
resource characteristic function ${\chi}_{\mathrm{ch}}$ to provide the effect
of realistic Bell measurements. The streamlined output characteristic function
in the ideal situation

$\displaystyle{\chi}_{\mathrm{out}}(x_{2},p_{2})$ $\displaystyle=$
$\displaystyle{\chi}_{\mathrm{in}}(x_{2},p_{2})\,\,{\chi}_{\mathrm{ch}}(x_{2},-p_{2};x_{2},p_{2}),$
(11)

can be recovered from (9) while substituting $R=0$ that means $T=1$,
$\Upsilon=0$ i.e. $\tau=0$, and $g=1$. In the mechanism of characteristic
functions, fidelity refers

$\displaystyle{F}$ $\displaystyle=$ $\displaystyle\frac{1}{\pi}\int
d^{2}\gamma\,{\chi}_{\mathrm{in}}(\gamma){\chi}_{\mathrm{out}}(-\gamma)$ (12)
$\displaystyle=$ $\displaystyle\frac{1}{2\pi}\int
dx_{2}dp_{2}\,{\chi}_{\mathrm{in}}(x_{2},p_{2}){\chi}_{\mathrm{out}}(-x_{2},-p_{2}),$

where $\gamma=\frac{1}{\sqrt{2}}(x_{2}+ip_{2})$,
$d^{2}\gamma=\frac{1}{2}dx_{2}\,dp_{2}$, and
${\chi}_{\mathrm{out}}(\gamma)={\chi}_{\mathrm{out}}(x_{2},p_{2})$ is given by
(9). This fidelity equation measures the performance of a displaced Fock state
channel for transferring quantum information across long distance in a
realistic environment.

###  3.1 For input coherent state $\ket{\alpha}$

After simple calculations, the fidelity for input coherent state in the
realistic situation is as follows:

$\displaystyle{F}$ $\displaystyle=$
$\displaystyle\frac{1}{2\pi}\int{d^{2}\gamma}\,e^{-(1+g^{2}T^{2})\frac{|\gamma|^{2}}{4}}\,e^{(1-gT)(\alpha^{*}\gamma-\alpha\gamma^{*})/\sqrt{2}-\Gamma_{\tau,R}|\gamma|^{2}}$
(13)
$\displaystyle\exp\left\\{(gT\alpha_{1}-e^{-\tau/2}\alpha_{2}^{*})\gamma/\sqrt{2}-(gT\alpha_{1}^{*}-e^{-\tau/2}\alpha_{2})\gamma^{*}/\sqrt{2}-(g^{2}T^{2}+e^{-\tau})/4|\,\gamma|^{2}\right\\}$
$\displaystyle
L_{n_{1}}\bigg{(}e^{-\tau}\frac{|\gamma|^{2}}{2}\bigg{)}L_{n_{2}}\bigg{(}g^{2}T^{2}\frac{|\gamma|^{2}}{2}\bigg{)}$

Figure 4 describes the variation of average fidelity for sending a coherent
state $\ket{\alpha}$ across a long channel. In order to understand the
simultaneous effect of different sources of decoherence in losing fidelity,
the realistic parameters are fixed at $R=0.8$, $n_{\mathrm{th}}=0$ and $g=1$.
The losses mainly due to the imperfections in Bell measurements (when $\tau$
is very low, $\tau=$0.02) are shown by broad lines. The corresponding thin
lines exhibit the fidelity for sufficiently noisy resources ($\tau=0.8$). It
is observed that the fidelity decreases monotonically with the increase of
photon numbers. This behaviour is similar to the case of the ideal protocol.
Interestingly, when the channel decay rate increases from 0.02 to 0.8, the DFS
channel displays larger fidelity values in a certain range of $\alpha$. When
$\alpha\approx 4$, the fidelity curve corresponding to $\tau=0.8$ lowers than
the curve associated with $\tau=0.02$. This strange behaviour may be explained
as follows: for not too large values of $\alpha$, the destructive quality of
$\tau$ is reduced, while the presence of a large number of optical photons
($\geq 4$) stimulates the decoherence, and leads to a suppression of fidelity.
In Fig. 5, choosing the same values of $R$, $\tau$, $n_{\mathrm{th}}$ as in
Fig. 4, we plot ${F}$ as a function of DFS parameters $\alpha_{1}$ and
$\alpha_{2}$, for fixed $\alpha$. It is clear that the disruptive effect of
the noisy channel is dominating over fidelity for a large number of optical
photons only, which supports our findings from Fig. 4. The fidelity $F$ is
plotted with respect to different realistic parameters in Fig. 6. It is noted
that increasing the inefficiency of the photodetectors has a mere effect on
fidelity. The maximum fidelity is obtained when $R\approx 0.8$.

Figure 4: Fidelities of realistic teleportation in terms of the input coherent
state amplitude $\alpha$ with $R=0.8$, $\tau=0.02$, $n_{\mathrm{th}}=0$,
$g=1$, and in (a) $\alpha_{1}=\alpha_{2}=0$, (b) $\alpha_{1}=0$,
$\alpha_{2}=0.5$ and (c) $\alpha_{1}=1$, $\alpha_{2}=0$. Here different types
of lines represent the fidelities of teleporting an input coherent state
$\ket{\alpha}$ corresponding to the following cases: $n_{1}=n_{2}=0$ (red
dash-dotted line), $n_{1}=n_{2}=1$ (green dotted line), $n_{1}=n_{2}=5$ (blue
dashed line), and $n_{1}=3$, $n_{2}=7$ (magenta solid line), respectively. The
corresponding thin lines in (a) to (c) are for the fidelities with $\tau=0.8$.
Figure 5: Fidelity in realistic condition with parameters $R$, $\tau$,
$n_{\mathrm{th}}$, $g$ as in Fig. 4 and as a function of DFS amplitudes
$\alpha_{1}$ and $\alpha_{2}$, with (a) $\alpha=0$, (b) $\alpha=2$, and (c)
$\alpha=4$, respectively. Figure 6: Variation of fidelity for teleportation of
a coherent state $\ket{\alpha}$ with respect to different realistic
parameters, assuming $\alpha_{1}=\alpha_{2}=1$, $R=0.8$, $\tau=0.8$,
$n_{\mathrm{th}}=0$ and $g=1$ whenever needed.

###  3.2 For input squeezed state $\ket{\varepsilon}$

The Fidelity for a squeezed state input after doing some simple calculations
is derived as follows:

$\displaystyle{F}$ $\displaystyle=$
$\displaystyle\frac{1}{2\pi}\int{d^{2}\gamma}\,e^{-(1+g^{2}T^{2})\frac{|\xi|^{2}}{4}-\Gamma_{\tau,R}|\gamma|^{2}}$
(14)
$\displaystyle\exp\left\\{(gT\alpha_{1}-e^{-\tau/2}\alpha_{2}^{*})\gamma/\sqrt{2}-(gT\alpha_{1}^{*}-e^{-\tau/2}\alpha_{2})\gamma^{*}/\sqrt{2}-(g^{2}T^{2}+e^{-\tau})/4|\,\gamma|^{2}\right\\}$
$\displaystyle
L_{n_{1}}\bigg{(}e^{-\tau}\frac{|\gamma|^{2}}{2}\bigg{)}L_{n_{2}}\bigg{(}g^{2}T^{2}\frac{|\gamma|^{2}}{2}\bigg{)}$

Figure 7: Behaviour of fidelity in non-ideal case with respect to squeezing
parameter $r$ and (a) $\varphi=0$, $\alpha_{1}=\alpha_{2}=0$, (b) $\varphi=0$,
$\alpha_{1}=0$, $\alpha_{2}=0.5$, (c) $\varphi=0$, $\alpha_{1}=1$,
$\alpha_{2}=0$. The second row corresponds to $\varphi=\pi$ and $\alpha_{1}$,
$\alpha_{2}$ are same as in (a)-(c), respectively. The plots are for different
values of the realistic parameters such as $R=0.8$, $n_{\mathrm{th}}=0$, $g=1$
and $\tau=0.02$ ($0.8$) corresponding to the thick (thin) lines. Figure 8:
Average fidelity as a function of DFS parameters $\alpha_{1}$ and
$\alpha_{2}$, with squeezing amplitude $\varphi=0$ and (a) $r=0$, (b) $r=1$,
(c) $r=2$. In (d)-(f), ${F}$ corresponds to $\varphi=\pi$ and same $r$ values
as in (a)-(c), respectively. Here $R$, $\tau$, $n_{\mathrm{th}}$ and $g$ have
same values as in Fig. 7. Figure 9: Fidelity for input squeezed state with
respect to different realistic parameters, assuming $r=0$,
$\alpha_{1}=\alpha_{2}=1$, $\varphi=0$, $R=0.8$, $\tau=0.8$,
$n_{\mathrm{th}}=0$ and $g=1$ in different cases.

In Fig. 7, we continue with the same fixed parametric values as in Fig. 4,
i.e., $R=0.8$, $n_{\mathrm{th}}=0$, $g=1$, and $\tau=0.02$ as well as 0.8, and
plot the non-ideal teleportation fidelity with respect to the squeezing
parameter $r$ and $\varphi=0$ (see Figs. 7(a)–(c)); $r$ and $\varphi=\pi$ (see
Figs. 7(d)–(f)). It can be noticed that the maximum fidelity value 0.324 is
obtained when $\varphi=\pi$ and squeezing is very small. Unlike the input
coherent state, the decay in fidelity for increasing $r$ dominates over the
degradation due to the damping $\tau$ in the entire range of $r$. Figure 9
gives the variation of fidelity with respect to different realistic
parameters. Changing of only $\varphi$ has a negligible effect on fidelity. It
is also seen that as the thermal photon number increases from 0 to 10,
$F_{\mathrm{max}}$ decreases in a regular way.

##  4 Conclusion

In this article, the efficiency of the displaced Fock states as a quantum
pathway for teleporting coherent and squeezed states are investigated. The
characteristic function description is used for studying the competence of DFS
in ideal and realistic scenarios. We have observed that the maximum fidelity
for both the protocols is obtained when $n_{1}=n_{2}=0$. How the fidelity
decoheres for transmitting via a noisy channel and imperfect homodyne
detectors are analyzed. We have checked that the fidelity remains below the
classical boundary which recommends that the highly-nonclassical displaced
Fock states are not suitable for quantum teleportation.

For both the input states, the optimization of fidelity is performed at fixed
values of $R$, $\tau$, $n_{\mathrm{th}}$, and $g$. In the context of an
operational approach, manipulating these parameters is equivalent to control
different features of the experimental appliances, including the shortfall of
the photodetectors and the decay rate of the noisy channel. Thus we believe
that the present results will be of further use in displaced Fock state based
continuous-variable quantum information processing tasks. The role of photon
addition and/or subtraction for enhancing the performance of displaced Fock
states in realistic teleportation will further be explored.

##  Appendix A Details of Gaussian Integration

Here we outline the description of the Gaussian integration in detail.

$\displaystyle I$ $\displaystyle=$
$\displaystyle\int{dx\,dy\,\exp(a_{1}x^{2}+a_{2}y^{2}+a_{3}xy+a_{4}x+a_{5}y)L_{m}[a(x^{2}+y^{2})]L_{n}[b(x^{2}+y^{2})]}$
(15) $\displaystyle=$
$\displaystyle\sum_{p,q=0}^{m,n}\frac{(-a)^{p}(-b)^{q}m!n!}{p!^{2}q!^{2}(m-p)!(n-q)!}$
$\displaystyle\times\int{dx\,dy\,\exp(a_{1}x^{2}+a_{2}y^{2}+a_{3}xy+a_{4}x+a_{5}y)(x^{2}+y^{2})^{p+q}}$
$\displaystyle=$
$\displaystyle\sum_{p,q,r=0}^{m,n,p+q}\frac{(-a)^{p}(-b)^{q}m!n!(p+q)!}{p!^{2}q!^{2}(m-p)!(n-q)!r!(p+q-r)!}$
$\displaystyle\times\int{dx\,dy\,\exp(a_{1}x^{2}+a_{2}y^{2}+a_{3}xy+a_{4}x+a_{5}y)x^{2r}y^{2p+2q-2r}}$

Now, rearranging the parameters as
$\mu_{x}=\frac{2a_{2}a_{4}-a_{3}a_{5}}{4a_{1}a_{2}-a_{3}^{2}},~{}~{}\mu_{y}=\frac{2a_{1}a_{5}-a_{3}a_{4}}{4a_{1}a_{2}-a_{3}^{2}},~{}~{}\rho=-\frac{a_{3}}{2\sqrt{a_{1}a_{2}}},~{}~{}\sigma_{x}=\sqrt{-\frac{2a_{2}}{4a_{1}a_{2}-a_{3}^{2}}},~{}~{}\sigma_{y}=\sqrt{-\frac{2a_{1}}{4a_{1}a_{2}-a_{3}^{2}}},\,\,a_{6}=\frac{a_{2}a_{4}^{2}+a_{1}a_{5}^{2}-a_{3}a_{4}a_{5}}{4a_{1}a_{2}-a_{3}^{2}}$
and substituting $x=\sigma_{x}z_{1}+\mu_{x}$ and $y=\sigma_{y}(\rho
z_{1}+\sqrt{1-\rho^{2}}z_{2})+\mu_{y}$, ($z_{1}$ and $z_{2}$ are standard
normal variate) in (15), we obtain [42]

(21)

by using [43]

$\displaystyle\int{dx\,e^{-ax^{b}}x^{n}}=\left\\{\begin{array}[]{lll}\frac{2\,\Gamma[(n+1)/b]}{b\,a^{\frac{n+1}{b}}},\,\,\,\text{if
$n$ is even}\\\ \\\
0,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\text{otherwise}\end{array}\right.$
(25)

ACKNOWLEDGEMENT

Deepak acknowledges support from the Council of Scientific and Industrial
Research (CSIR), Govt. of India (Award no. 09/1256(0006)/2019-EMR-1).

DATA AVAILABILITY STATEMENT

Data will be made available on reasonable request.

## References

  * [1] A. Wunsche, Displaced Fock states and their connection to quasiprobabilities, Quant. Optics 3, 359 (1991).
  * [2] H. Moya-Cessa, and P. L. Knight, Series representation of quantum-field quasiprobabilities, Phys. Rev. A 48, 2479 (1993).
  * [3] S. M. Dutra, P. L. Knight, and H. Moya-Cessa, Large-scale fluctuations in the driven Jaynes-Cummings model, Phys. Rev. A 49, 1993 (1994).
  * [4] P. Alsing, D.-S. Guo and H. Carmichael, Dynamic Stark effect for the Jaynes-Cummings system, Phys. Rev. A 45(7), 5135 (1992).
  * [5] K. E. Cahill and R. J. Glauber, Ordered expansions in boson amplitude operators, Phys. Rev. 177, 1857 (1969).
  * [6] K. E. Cahill and R. J. Glauber, Density operators and quasiprobability distributions, Phys. Rev. 177, 1882 (1969).
  * [7] M. Boiteux and A. Levelut, Semicoherent states, J. Phys. A: Math. Nucl. Gen. 6, 589 (1973).
  * [8] S. M. Roy and V. Singh, Generalized coherent states and the uncertainty principle, Phy J. Rev. D 25, 3413 (1982).
  * [9] M. V. Satyanarayana, Generalized coherent states and generalized squeezed coherent states, Phys. Rev. D 32, 400 (1985).
  * [10] F. A. M. de. Oliveira, M. S. Kim, P. L. Knight and V. Buiek, Properties of displaced number states, Phys. Rev. A 41, 2645 (1990).
  * [11] C. Lo, Generating displaced and squeezed number states by a general driven time-dependent oscillator, Phys. Rev. A 43(1), 404 (1991).
  * [12] M. A. Marchiolli, W. D. José, Engineering superpositions of displaced number states of a trapped ion, Physica A 337(1-2), 89-108 (2004).
  * [13] S. A. Podoshvedov, Extraction of displaced number states, J. Opt. Soc. Am. B 31, 2491 (2014).
  * [14] F. Ziesel et. al., Experimental creation and analysis of displaced number states, J. Phys. B: At. Mol. Opt. Phys. 46, 104008 (2013).
  * [15] P. Malpani, K. Thapliyal, N. Alam, A. Pathak, V. Narayanan and S. Banerjee, Impact of photon addition and subtraction on nonclassical and phase properties of a displaced Fock state, Ann. Phys. (Berlin) 11, 1900141 (2019).
  * [16] W. Quintero, and C. L. Ladera, Teleportation of displaced Fock states: Fidelity and their teleported photon number distributions, J. of Phys.: Conference Series 274, 012146 (2011).
  * [17] F. Dell’Anno, S. De Siena, and F. Illuminati, Multiphoton quantum optics and quantum state engineering, Phys. Rep. 428, 53 (2006).
  * [18] M. S. Kim, W. Son, V. Bušek, and P. L. Knight, Entanglement by a beam splitter: Nonclassicality as a prerequisite for entanglement, Phys. Rev. A 65, 032323 (2002).
  * [19] A. Kitagawa, M. Takeoka, M. Sasaki, and A. Chefles, Entanglement evaluation of non-Gaussian states generated by photon subtraction from squeezed states, Phys. Rev. A 73, 042310 (2006).
  * [20] V. V. Dodonov, and L. A. de Souza, Decoherence of superpositions of displaced number states, J. Opt. B: Quantum Semiclass. Opt. 7, S490 (2005).
  * [21] N. J. Cerf, O. Krüger, P. Navez, R. F. Werner, and M. M. Wolf, Non-Gaussian cloning of quantum coherent states is optimal, Phys. Rev. Lett. 95, 070501 (2005).
  * [22] G. Adesso, F. Dell’Anno, S. De Siena, F. Illuminati, and L. A. M. Souza, Optimal estimation of losses at the ultimate quantum limit with non-Gaussian states, Phys. Rev. A 79, 040305(R) (2009).
  * [23] C. H. Bennett, G. Brassard, C. Crepeau, R. Jozsa, A. Peres, and W. K. Wootters, Teleporting an unknown quantum state via dual classical and Einstein-Podolsky-Rosen channels, Phys. Rev. Lett. 70, 1895 (1993).
  * [24] D. Bouwmeester, J. W. Pan, K. Mattle, M. Eibl, H. Weinfurter, and A. Zeilinger, Experimental quantum teleportation, Nature London 390, 575 (1997).
  * [25] D. Boschi, S. Branca, F. De Martini, L. Hardy, and S. Popescu, Experimental realization of teleporting an unknown pure quantum state via dual classical and Einstein-Podolsky-Rosen Channels, Phys. Rev. Lett.80, 1121 (1998).
  * [26] L. Vaidman, Teleportation of quantum states, Phys. Rev. A 49, 1473 (1994).
  * [27] S. L. Braunstein and H. J. Kimble, Teleportation of continuous quantum variables, Phys. Rev. Lett. 80, 869 (1998).
  * [28] A. Furusawa, J. L. Sørensen, S. L. Braunstein, C. A. Fuchs, H. J. Kimble, and E. S. Polzik, Unconditional quantum teleportation, Science 282, 706 (1998).
  * [29] W. P. Bowen, N. Treps, B. C. Buchler, R. Schnabel, T. C. Ralph, Hans-A. Bachor, T. Symul, and P. K. Lam, Experimental investigation of continuous-variable quantum teleportation, Phys. Rev. A 67, 032302 (2003).
  * [30] N. Takei, H. Yonezawa, T. Aoki, and A. Furusawa, High-fidelity teleportation beyond the no-cloning limit and entanglement swapping for continuous variables, Phys. Rev. Lett. 94, 220502 (2005).
  * [31] S. J. van Enk, Discrete formulation of teleportation of continuous variables, Phys. Rev. A 60, 5095 (1999).
  * [32] A. Vukics, J. Janszky, and T. Kobayashi, Nonideal teleportation in coherent-state basis, Phys. Rev. A 66, 023809 (2000).
  * [33] H. F. Hofmann, T. Ide, T. Kobayashi, and A. Furusawa, Fidelity and information in the quantum teleportation of continuous variables, Phys. Rev. A 62, 062304 (2000).
  * [34] S. L. Braunstein, and P. van Loock, Quantum information with continuous variables, Rev. Mod. Phys. 77, 513 2005.
  * [35] P. Marian, and T. A. Marian, Continuous-variable teleportation in the characteristic-function description, Phys. Rev. A 74, 042306 (2006).
  * [36] W. Quintero, and C. L. Ladera, Teleportation of displaced Fock states: Fidelity and their teleported photon number distributions, J. of Phys.: Conference Series 274, 012146 (2011).
  * [37] F. Dell’Anno, S. De Siena, L. Albano, and F. Illuminati, Continuous-variable quantum teleportation with non-Gaussian resources, Phys. Rev. A 76,022301 (2007).
  * [38] F. Dell’Anno, S. De Siena, and F. Illuminati, Realistic continuous-variable quantum teleportation with non-Gaussian resources, Phys. Rev. A 81, 012333 (2010).
  * [39] Y. -X. Ping, B. Zhang, Z. Cheng, and Q. Xu, Two-mode entanglement via superpositions of two-mode coherent states, Mod. Phys. Lett. B 21, 1253 (2007).
  * [40] H. S. Dhar, A. chatterjee, and R. Ghosh, Generating continuous variable entangled states for quantum teleportation using a superposition of number-conserving operations, J. Phys. B: At. Mol. Opt. Phys. 48, 185502 (2015).
  * [41] U. Leonhardt, and H. Paul, Realistic optical homodyne measurements and quasiprobability distributions, Phys. Rev. A 48, 4598 (1993).
  * [42] S. C. Gupta, and V. K. Kapoor, Fundamentals of Mathematical Statistics, SC & Sons New Delhi, (2018).
  * [43] V. H. Moll, Special integrals of Gradshteyn and Ryzhik the proofs, Volume II, Chapman and Hall/CRC, (2015).


# Question-Instructed Visual Descriptions
for Zero-Shot Video Question Answering

David Romero
MBZUAI
<EMAIL_ADDRESS>
&Thamar Solorio
MBZUAI
<EMAIL_ADDRESS>

###### Abstract

We present Q-ViD, a simple approach for video question answering (video QA),
that unlike prior methods, which are based on complex architectures,
computationally expensive pipelines or use closed models like GPTs, Q-ViD
relies on a single instruction-aware open vision-language model (InstructBLIP)
to tackle videoQA using frame descriptions. Specifically, we create captioning
instruction prompts that rely on the target questions about the videos and
leverage InstructBLIP to obtain video frame captions that are useful to the
task at hand. Subsequently, we form descriptions of the whole video using the
question-dependent frame captions, and feed that information, along with a
question-answering prompt, to a large language model (LLM). The LLM is our
reasoning module, and performs the final step of multiple-choice QA. Our
simple Q-ViD framework achieves competitive or even higher performances than
current state of the art models on a diverse range of videoQA benchmarks,
including NExT-QA, STAR, How2QA, TVQA and IntentQA.

##  1 Introduction

Recently, Vision-Language models have shown remarkable performances in image
question-answering tasks (Goyal et al., 2017; Marino et al., 2019; Schwenk et
al., 2022), with models such as Flamingo (Alayrac et al., 2022), BLIP-2 (Li et
al., 2023b), InstructBlip (Dai et al., 2023) and mPLUG-Owl (Ye et al., 2023)
showing strong reasoning capabilities in the vision-language space. Image
captioning (Vinyals et al., 2015; Ghandi et al., 2023) is one of the
capabilities in which these models truly excel, as they can generate detailed
linguistic descriptions from images. Different works have leveraged this
capability in many ways for zero-shot image-question answering, such as giving
linguistic context to images (Hu et al., 2022; Ghosal et al., 2023),
addressing underspecification problems in questions (Prasad et al., 2023),
coordination of multiple image captions to complement information (Chen et
al., 2023b), or by combining captions with other type of linguistic
information from the image (Berrios et al., 2023). In this manner, the
reasoning capabilities of LLMs can be directly used to reason about the
linguistic image descriptions and generate an answer for the given visual
question.

This approach has been successful for images, but in the case of video-
question answering tasks (Lei et al., 2018; Li et al., 2020b; Xiao et al.,
2021; Wu et al., 2021; Li et al., 2023a) this is more challenging. Video
possesses multiple image frames that have relationships between each other and
involve the recognition of objects, actions, as well as the inference about
semantic, temporal, causal reasoning and much more (Zhong et al., 2022). Thus,
some works (Chen et al., 2023a; Wang et al., 2023) have focused on using
powerful LLMs like ChatGPT to either ask visual questions to image-language
models like BLIP-2 or to respond and retrieve useful information from large
datasets with detailed information from the video. Similarly, Zhang et al.,
(2023a) have leveraged the reasoning capabilities of GPT-3.5 to create textual
summaries from the video, and later perform video QA using only textual
information. While others (Wang et al., 2022b; Zeng et al., 2022) combine
linguistic information from multiple sources such as captions, visual
tokenization or even subtitles of input speech. In summary, current methods
for video QA rely on any combination of closed LLMs, expensive training
regimes, and complex architectures with multiple modules (Yang et al., 2022;
Ko et al., 2023; Yu et al., 2023; Momeni et al., 2023; Li et al., 2023c; Zhang
et al., 2023a). In contrast, we introduce Q-ViD a simple Question-Instructed
Visual Descriptions for video QA approach that relies on an instruction-aware
vision-language model, InstructBLIP (Dai et al., 2023), to automatically
generate rich captions from the frames. In this manner, we effectively turn
the vQA task into a text QA task. More specifically, given an input video $V$
we sample $n$ number of frames, then, we generate question-specific
instructions to prompt the multimodal instruction tuned model to generate
captions for each frame. Afterwards, we form a video description by
concatenating all the generated question-dependent captions from InstructBLIP,
and use it along with the question, options and an instruction prompt as input
to the LLM-based reasoning module that generates an answer to the multiple-
choice question about the video. We demonstrate the effectiveness of Q-ViD on
five challenging multiple choice video question answering tasks (NExT-QA,
STAR, How2QA, TVQA, IntentQA), showing that this simple framework can achieve
strong performances comparable with more complex pipelines. Our contributions
are summarized as follows:

  * •

We propose Q-ViD, a simple gradient-free approach for zero-shot video QA that
relies on an open instruction-tuned multimodal model to extract question-
specific descriptions of frames to transform the video QA task into a text QA
one.

  * •

Our approach achieves strong zero-shot performance that is competitive or even
superior to more complex architectures such as SeViLa, Internvideo, and
Flamingo. It even compares favorably with recent solutions that include GPT
APIs, like LloVi and ViperGPT.

Figure 1: Overview of Q-ViD. We propose relying on a instructed-tuned
multimodal model to generate question-dependent frame captions to perform
video QA using text. This simple approach achieves competitive results with
more complex architectures or GPT-based methods

##  2 Related Work

###  2.1 Multimodal Pretraining for Video QA

The strong reasoning capabilities of LLMs (Chung et al., 2022; Touvron et al.,
2023; Brown et al., 2020; Hoffmann et al., 2022) in natural language
processing tasks has motivated to apply these models for visual understanding.
Currently, LLMs have been successfully adapted to understand images (Li et
al., 2023b; Ye et al., 2023; Chen et al., 2023c), but applying the same
principles for video is more challenging. Approaches for VideoQA rely on
image-language models, and adapt those to process video by using fixed amounts
of video frames as input (Alayrac et al., 2022; Yu et al., 2023; Yang et al.,
2022), or by selecting key-frames from the initial sequence (Yu et al., 2023;
Li et al., 2023c).

Commonly, these works use frozen visual and language models and focus only on
modality alignment. Models like Flamingo (Alayrac et al., 2022) uses a fixed
amount of video frames as input and bridges modalities by training a perceiver
resampler and gated attention layers between Chinchilla LLMs (Hoffmann et al.,
2022). While others, like SeViLa (Yu et al., 2023) relies on BLIP-2 (Li et
al., 2023b) for modality alignment, using an intermediate pretrained module
called Q-former. SeViLa, first perform key-frame localization and then video
QA with Flan-T5 LLMs (Chung et al., 2022). On the other hand, other works
apart from using frozen vision models, adapt the LLM to visual inputs using
adapter tokens (Zhang et al., 2023b) or intermediate trainable modules
(Houlsby et al., 2019). Models like Flipped-VQA (Ko et al., 2023) focuses on
adapting LLaMa (Touvron et al., 2023) to video QA by using adapter tokens
along with different training objectives to leverage the temporal and causal
reasoning abilities of LLMs. Similarly, FrozenBilm (Yang et al., 2022) exploit
the strong zero-shot performance of BILM, a frozen bidirectional language
model that is adapted to video QA by using lightweight trainable modules.
Despite the success of all these models, they require complex architectures
and training regimes, unlike these works we build a simple, gradient-free,
approach for zero-shot video QA.

###  2.2 Image Captions for Video Understanding

One of the core strengths of image-language models (Alayrac et al., 2022; Li
et al., 2023b; Dai et al., 2023) is the generation of image captions, thus due
to the current strong zero-shot capabilities of LLMs, captions can be directly
use to reason about visual content. This has been successfully leveraged in
the image-language space for image question-answering with approaches such as
Lens (Berrios et al., 2023), Img2LLM (Guo et al., 2023) and PromptCat (Hu et
al., 2022) that gather image captions and other type of linguistic information
to answer a visual question. While similar approaches have been taken for
videos, the use of large models like GPTs is very common, with models such as
ChatCaptioner (Chen et al., 2023a), ViperGPT (Surís et al., 2023) , ChatVideo
(Wang et al., 2023), VidIL (Wang et al., 2022b), Socratic Models (Zeng et al.,
2022), and LLoVi (Zhang et al., 2023a) have been applied for video-language
tasks, common methods use GPTs to either interact with image-language models
to get visual descriptions, or to make summaries from captions and other type
of information such as visual tokenization, subtitles of speech and more.
Unlike these approaches, we do not use GPTs or multiple computationally
expensive modules in any part of our pipeline to achieve strong zero-shot
performance on video QA.

##  3 Method

Recently, vision-language models trained with instruction tuning (Dai et al.,
2023; Zhu et al., 2023; Liu et al., 2023) have shown impressive capabilities
to faithfully follow instructions and extract visual representations adapted
to the task at hand. Thus, with Q-ViD (Fig. 2), we propose to leverage these
capabilities for multiple-choice video QA, and turn this task into textual QA
using InstructBLIP (Dai et al., 2023). We use a question-dependent captioning
prompt as the input instruction, to guide InstructBLIP to generate video frame
descriptions that are more relevant for the given question. Afterwards, we
reuse the LLM from InstructBLIP and use it as our reasoning module. This LLM
(Flan-T5) takes a question-answering prompt as input, that consists of a video
description formed by the concatenation of all the question-dependent frame
captions, the question, options and a task instruction. Considering that
Flan-T5 is also originally trained with instructions, we aim to leverage its
reasoning capabilities to correctly answer the question given only the text we
just described as input. Our simple approach does not rely on complex
pipelines or closed GPT models, which makes it easy, cheaper and straight
forward to use for zero-shot video QA. On the other hand, Q-ViD is flexible
and model agnostic, which means we can use any multimodal models available.
This section presents our approach in detail. First, we introduce some
preliminary information on InstructBLIP, which serves as the foundation of our
work, and then we provide a detailed overview for all components from our
Q-ViD framework.

Figure 2: Our pipeline for Zero-shot Video QA. Q-ViD prompts InstructBLIP, to
obtain video frame descriptions that are tailored to the question needing
answer.

###  3.1 Preliminaries: InstructBLIP

We rely on InstructBLIP (Dai et al., 2023) as the foundational architecture of
Q-ViD. InstructBLIP is a vision-language instruction tuning framework based on
a Query Transformer (Q-former) and frozen vision and language models. Unlike
BLIP-2 (Li et al., 2023b), which is based on an instruction-agnostic approach,
InstructBLIP can obtain visual features depending on specific instructions of
the task at hand using an instruction-aware Q-former, which in addition to
query embeddings, uses instruction tokens to guide the Q-former in extracting
specific image features. Subsequently, a LLM (Flan-T5) uses these features to
generate visual descriptions depending on the input instructions. In our
approach we adopt this model to obtain frame captions that are dependant on
the video questions, thus, we aim to gather the most important information
from each part of the video and use it as input for our reasoning module to
answer the given question. Because of our Q-ViD framework is a zero-shot
approach, we do not train any part of InstructBLIP, and keep all of its parts
frozen.

###  3.2 Q-ViD: Generating Frame Descriptions for Video QA

We focus on automatically generating meaningful captions that can provide
enough information about what is happening in the video to the LLM. We assume
that if captions for the frames contain relevant information related to the
question needing answer, then an LLM should be able to answer the question
correctly without additional need for frame/video input. As shown in Fig. 2,
given an input video $V$, we use a uniform sampling strategy and extract a set
of $n$ video frames $\\{f_{1},f_{2},...,f_{n}\\}$. We then use InstructBLIP,
refered as $I_{b}$, to obtain instruction-aware visual captions $c_{i}$ for
each frame $f_{i}$, as follows $c_{i}=I_{b}(f_{i},E)$, where $E$ represents
the question-dependent captioning instruction. Q-ViD automatically generates
$E$ by concatenating a request for a caption, referred as $B$ (e.g "Provide a
detailed description of the image related to the question:") and a question,
referred as $Q$ (e.g "Why did the man in white held tightly to the boy in
white?"), represented as follows $E=concat(B,Q)$. Specifically, $E$ is used as
input to Q-former and to the LLM of InstructBLIP to obtain specific visual
representations and frame descriptions respectively. Thus, we represent the
input video $V$ as a set of question-dependent frame captions
$c=(c_{1},c_{2},...,c_{n})$, where each caption is conformed by a sequence of
$w_{m}$ words $c_{i}=(w_{1},w_{2},...,w_{m})$. In this way, we extract textual
information from $V$, the captions, that is going to be useful for the
question answering task. Next, we describe the reasoning module of Q-ViD and
how these question-dependent captions are used to perform video QA.

###  3.3 Q-ViD: Reasoning Module

We reuse the frozen LLM (Flan-T5) from InstructBLIP and implement it as the
reasoning module of Q-ViD. In order to perform video QA using language, we
first concatenate the question-dependent frame captions
$C=[c_{1},c_{2},...,c_{n}]$ in the same order they appear in the video. Then,
we create a question-answering instruction $L$ as follows:
$L=concat(C,Q,A,T)$. In other words, we concatenate in $L$ the list of
captions $C$, question $Q$, possible answers $A$ and a task description $T$
(e.g "Considering the information presented in the captions, select the
correct answer in one letter (A,B,C) from the options."). Our goal is to
leverage the LLM reasoning linguistic capabilities by providing a set of
captions that were tailored to be relevant for the specific question $Q$. Our
experiments in Section 4, show that this simple approach works surprisingly
well, showing to be competitive, and even superior in some cases, in
comparison with more complex pipelines. Next, we describe in more detail the
prompts used for question-dependent captioning and video QA.

###  3.4 Q-ViD: Prompt Design

First, to get question-dependent captions for each frame, given the question
$Q$ we prompt InstructBLIP with a question-dependent captioning instruction:
"Provide a detailed description of the image related to the question: {Q}".
This instruction is used along with queries as input to the frozen Q-Former
and LLM modules of InstructBLIP to extract specific visual features and
generate question-dependent descriptions. Afterwards, to perform QA with the
reasoning module, given the list of captions $C$ and the list of possible
answers $A=[a_{1},...,a_{m}]$ with $m$ being the number of options provided in
each dataset, we prompt the language model as follows: "Captions: {C}
Question: {Q}. Option A: a1 Option B: a2 Option C: a3. Considering the
information presented in the captions, select the correct answer in one letter
from the options (A,B,C)". In this prompt, in addition to $C$ , $Q$ and $A$,
we added a small instruction at the end to specify in detail that a single
letter is needed as output.

##  4 Experiments

In this section, we present our experiments for zero-shot video QA. First, we
describe the datasets we used and the implementation details. Then, we
evaluate our approach, compare Q-ViD with other state of the art models for
video QA and provide a comprehensive analysis of the model’s performance.
Lastly, we conduct some ablation studies of Q-ViD regarding the instructions
prompt design.

Models | NExT-QA | STAR | How2QA | TVQA
---|---|---|---|---
| Tem. | Cau. | Des. | Avg. |  | Int. | Seq. | Pre. | Fea. | Avg. |  |
GPT-Based Models |  |  |  |  |  |  |  |  |  |  |  |
ViperGPT (Surís et al., 2023) | - | - | - | 60.0 |  | - | - | - | - | - | - | -
LLoVi (Zhang et al., 2023a) | 61.0 | 69.5 | 75.6 | 67.7 |  | - | - | - | - | - |  |
Flamingo-9B (Alayrac et al., 2022) | - | - | - | - |  | - | - | - | - | 41.8 | - | -
Flamingo-80B (Alayrac et al., 2022) | - | - | - | - |  | - | - | - | - | 39.7 | - | -
FrozenBILM (Yang et al., 2022) | - | - | - | - |  | - | - | - | - | - | 41.9 | 29.7
VFC (Momeni et al., 2023) | 51.6 | 45.4 | 64.1 | 51.6 |  | - | - | - | - | - | - | -
InternVideo (Wang et al., 2022a) | 48.0 | 43.4 | 65.1 | 49.1 |  | 43.8 | 43.2 | 42.3 | 37.4 | 41.6 | 62.2 | 35.9
BLIP-2voting (Yu et al., 2023) | 59.1 | 61.3 | 74.9 | 62.7 |  | 41.8 | 39.7 | 40.2 | 39.5 | 40.3 | 69.8 | 35.7
BLIP-2concat (Yu et al., 2023) | 59.7 | 60.8 | 73.8 | 62.4 |  | 45.4 | 41.8 | 41.8 | 40.0 | 42.2 | 70.8 | 36.6
SeViLa (Yu et al., 2023) | 61.3 | 61.5 | 75.6 | 63.6 |  | 48.3 | 45.0 | 44.4 | 40.8 | 44.6 | 72.3 | 38.2
VideoChat2 (Li et al., 2024) | 57.4 | 61.9 | 69.9 | 61.7 |  | 58.4 | 60.9 | 55.3 | 53.1 | 59.0 | - | 40.6
Q-ViD (Ours) | 61.6 | 67.6 | 72.2 | 66.3 |  | 48.2 | 47.2 | 43.9 | 43.4 | 45.7 | 71.4 | 41.0
Table 1: Zero-shot results on video question answering. For fair comparison we
gray out methods that rely on closed GPTs. We bold the best results, and
underline the second-best results. QIViD shows to be competitive and even
outperform some more complex framworks for zero-shot video QA.

###  4.1 Datasets

To test our approach, we conduct experiments on the following multiple-choice
video QA benchmarks. To make comparisons with prior work we use the validation
set in NExT-QA, STAR, How2QA and TVQA, meanwhile in IntentQA we use the test
set. More details are shown below:

  * •

NExT-QA (Xiao et al., 2021): A benchmark focused on Temporal, Causal and
Descriptive reasoning type of questions. Contains 5,440 videos and 48K
multiple-choice questions in total. We perform our experiments using the
validation set that is conformed by 570 videos and 5K multi-choice questions.

  * •

STAR (Wu et al., 2021): A benchmark that evaluates situated reasoning in real-
world videos, is focused on interaction, sequence, prediction and feasibility
type of questions. It contains 22K situation video clips and 60K questions. We
perform evaluations on the validation set with 7K multiple-choice questions.

  * •

HOW2QA (Li et al., 2020a): A dataset that consists on 44K question-answering
pairs for 22 thousand 60-second clips selected from 9035 videos. We perform
experiments on the validation set with 2.8K questions.

  * •

TVQA (Lei et al., 2018): A large scale video QA dataset based on six popular
TV shows. It has 152K multiple-choice questions and 21K video clips. For our
zero-shot evaluations we use the validation set with 15K video-question pairs.

  * •

IntentQA (Li et al., 2023a): A dataset focused on video intent reasoning. It
contains 4K videos and 16K multiple-choice question-answer samples. In this
case, we use the test set for our zero-shot evaluations which contains 2K
video-question answering samples.

###  4.2 Implementation Details

For Q-ViD we adopt InstructBLIP-Flan-T5XXL with 12.1B parameters, as a default
vision encoder it uses VIT-g/14 (Fang et al., 2023), and as language model
FlanT5XXL (Chung et al., 2022). We extract 64 frames per video, as in
preliminary experiments this number worked well. For frame captioning, we use
a maximum number of 30 tokens per description and top-p sampling with
$top_{p}=0.7$ to get varied captions. Regarding our reasoning module, we reuse
and adopt the corresponding Flan-T5 language model from InstructBLIP. In this
case we do not use top-p sampling. Our experiments were conducted using 4
NVIDIA A100 (40GB) GPUs using the Language-Vision Intelligence library LAVIS
(Li et al., 2022) and the released code from SeViLa (Yu et al., 2023) .

###  4.3 Overall Performance

Table 1 provides a detailed overview on the performance of Q-ViD on the
validation set of NExT-QA, STAR, HOW2QA and TVQA. We compare our approach with
current state of the art methods such as SeViLa (Yu et al., 2023), FrozenBILM
(Yang et al., 2022) and VideoChat2 (Li et al., 2024), as well as, with GPT-
based models like ViperGPT (Surís et al., 2023) and LloVi (Zhang et al.,
2023a). The results obtained from our experiments demonstrate the surprisingly
competitive nature of Q-ViD, outperforming or being competitive with previous
methods with more complex architectural pipelines such as SeViLa, VideoChat2
and LLoVi. For fair comparisons, we gray out methods that use GPTs.

Specifically, on NExT-QA, Q-ViD outperforms SeViLa by 2.7% of average
accuracy, and achieves almost the same state of the art results of Llovi, a
framework based of GPT-3.5. Notably, Q-ViD is the best-performing model on
causal questions, temporal questions, and overall average performance among
methods that are not based on GPTs, showing the ability of this approach to
perform action reasoning, which is the target of NExT-QA. With STAR, Q-ViD
achieves the second best average accuracy behind VideoChat2, outperforming all
other methods like SeViLa by 1.1%, BLIP-2concat by 3.5%, InternVideo by 4.1%
and Flamingo-80B by 6%. Also note that Q-ViD achieves the second best
performances on sequence and feasibility type of question of STAR. Lastly, on
How2QA we achieve the second best performance behind SeViLa, and achieves the
best overall performance for TVQA with an improvement of 0.4% to the previous
best-performing method VideoChat2.

On the other hand, in Table 2 we evaluate our approach on IntentQA, we use the
test set of this benchmark in order to compare with prior works. We take the
same comparison made from (Zhang et al., 2023a), and divide the table in two
categories, Supervised and Zero-shot approaches. Q-ViD continues showing
strong results, greatly outperforming all supervised methods and the SeViLa
zero-shot performance by 2.7%. Interestingly, Q-ViD almost achieves the best
overall performance from the GPT-based method Llovi. These results demonstrate
that our approach can be used among different video QA tasks and be able to
achieve strong zero-shot performances.

Models | Acc.(%)
---|---
Supervised |
HQGA (Surís et al., 2023) | 47.7
VGT (Alayrac et al., 2022) | 51.3
BlindGPT (Alayrac et al., 2022) | 51.6
CaVIR (Alayrac et al., 2022) | 57.6
Zero-shot |
SeViLA (Yu et al., 2023) | 60.9
LLoVi (Zhang et al., 2023a) | 64.0
Q-ViD (Ours) | 63.6
Table 2: Performance on IntentQA. Q-ViD shows to outperform supervised
approaches, strong zero-shot baselines like SeViLa and obtain almost the same
performance from the GPT-based model LLoVi.

###  4.4 Ablation Studies

In this section, we perform some ablation studies related to the instruction
prompt selection for Q-ViD. For these experiments, we chose NExT-QA and STAR
as our benchmarks, and report results on the validation sets on each dataset.
Specifically, we test two model variations, using InstructBLIP-FlanT5XL
(Q-ViDXL) and the one used to report our main results, using InstructBLIP-
FlanT5XXL (Q-ViDXXL), we test different prompts to analyze and compare the use
of question-dependent and general descriptive captions. Additionally, we also
make some ablation experiments for the question-answering instruction prompt
that is used by the reasoning module to perform multi-choice QA. We discuss
our findings in detail below.

####  4.4.1 Prompt Analysis

We focus on analyzing the impact on performance of the Captioning and QA
templates of Q-ViD. First, for captioning templates (Fig.3) we compare two
variants: (1) General prompts and (2) Question-dependent prompts. With general
prompts we focus on obtaining general descriptions of the images (frames) and
with question-dependent prompts on information related to the question of the
task at hand. In order to leverage as much as possible the instruction-based
capabilities of InstructBLIP we create these prompts based on similar
templates to the ones used by InstructBLIP in its original training setup. For
these experiments we use the Base Question-Answering prompt that is used as
input of the reasoning module to perform multiple-choice question-answering.

Figure 3: Variation of captioning templates. We focus on comparing general and question-dependent captioning prompts (Top). For both cases we use the same Base QA instruction prompt (Bottom). Method | NExT-QA | STAR
---|---|---
| Tem. | Cau. | Des. | Avg. |  | Int. | Seq. | Pre. | Fea. | Avg.
Q-ViDXL |  |  |  |  |  |  |  |  |  |
(1) General | 57.3 | 60.3 | 62.0 | 60.5 |  | 47.0 | 45.2 | 42.7 | 42.2 | 44.3
(2) General | 57.8 | 60.1 | 60.8 | 60.1 |  | 47.4 | 44.8 | 44.7 | 42.8 | 44.9
(1) Dependent | 55.9 | 59.8 | 57.5 | 59.1 |  | 45.0 | 41.7 | 40.5 | 40.2 | 41.8
(2) Dependent | 56.6 | 58.8 | 61.1 | 59.0 |  | 45.8 | 40.6 | 40.2 | 39.5 | 41.5
Q-ViDXXL |  |  |  |  |  |  |  |  |  |
(1) General | 57.5 | 64.6 | 67.4 | 62.7 |  | 44.7 | 39.5 | 42.6 | 36.3 | 40.8
(2) General | 57.1 | 64.8 | 68.0 | 62.8 |  | 44.6 | 39.5 | 43.1 | 38.7 | 41.5
(1) Dependent | 62.0 | 66.5 | 71.2 | 65.8 |  | 47.8 | 44.2 | 42.1 | 41.8 | 44.0
(2) Dependent | 61.6 | 67.6 | 72.2 | 66.3 |  | 48.2 | 47.2 | 43.9 | 43.4 | 45.7
Table 3: Comparing the impact on performance using different captioning
templates. Base: Refer to the base captioning template (1) General: More
detailed templates for general frame descriptions and (2) Dependent: templates
for frame descriptions depending on a question. All experiments use the base
QA instruction prompt.

Table 3 compares the performance of Q-ViDXL and Q-ViDXXL using the general,
and question-dependent captioning prompts. It can be seen that performance
varies between both models, Q-ViDXL achieves better performances by using
general descriptive prompts. When comparing its best variants using the
(2)General and (1)Dependent templates, the former further increases the
average accuracy by +1.4% on NExT-QA and +3.1% on STAR. On the other hand, the
same behaviour is not shown using a bigger model, Q-ViDXXL achieves
significant improvements in average performance by using question-dependent
prompts, when comparing the best variants using the (2)General and
(2)Dependent templates, the latter obtains improvements of +3.5% on NExT-QA
and +4.2% on STAR. Unsurprisingly, Q-ViDXXL provides significant performance
boosts when compared to its smaller version Q-ViDXL achieving better
performances on all type of questions in both datasets, showing a better
capability to follow instructions, however, this also demonstrates that using
question-dependent frame captions to obtain specific information for the task
at hand, performs better than general visual descriptions for zero-shot Video
QA.

Figure 4: Variation of QA prompt templates. We focus on exploring two more
complex and detailed variations for the Question Answering instruction prompt
(Bottom). We use the best captioning templates (Top) for Q-ViDXL (General) and
Q-ViDXXL (Dependent).

Next, in Table 4 we investigate the impact on performance of the Question-
Answering Instruction Prompt. We propose two variations that are shown in Fig.
4 in addition to the Base QA prompt (Fig. 3). We these new templates we aim to
give more details to our reasoning module based on Flan-T5, because of this
LLM is also a model trained with instructions, we explore if using more
complex and detailed QA prompts we can achieve better performances. For this
comparison we take the best variants (Table 3) of Q-ViDXL and Q-ViDXXL using
the (2)General and (2)Dependent captioning prompts respectively for each
model, and explore their performances with different QA instruction templates.
As shown in Table 4 using more complex variants of the initial Base Question-
Answering Instruction prompt does not have a big impact on performance of any
of the models, it even slightly affects results in some cases, showing that
the simplest base prompt was enough for the LLM to understand the task. With
this ablation study we can highlight the fact that the input instructions used
to obtain dedicated frame descriptions are far more important than elaborated
question-answering instruction prompts for zero-shot video QA.

Model | Templates |  | NExT-QA |  | STAR
---|---|---|---|---|---
| Captioning | QA |  | Tem. | Cau. | Des. | Avg. |  | Int. | Seq. | Pre. | Fea. | Avg.
|  | Base |  | 57.8 | 60.1 | 60.8 | 60.1 |  | 47.4 | 44.8 | 44.7 | 42.8 | 44.9
Q-ViDXL | (2)General | (1)QA |  | 56.4 | 60.6 | 58.4 | 60.2 |  | 47.7 | 44.9 | 43.5 | 41.0 | 44.3
|  | (2)QA |  | 56.8 | 60.9 | 57.5 | 60.1 |  | 47.0 | 44.1 | 43.1 | 40.6 | 43.7
|  | Base |  | 61.6 | 67.6 | 72.2 | 66.3 |  | 48.2 | 47.2 | 43.9 | 43.4 | 45.7
Q-ViDXXL | (2)Dependent | (1)QA |  | 61.7 | 65.8 | 73.7 | 65.5 |  | 48.9 | 46.8 | 43.5 | 43.8 | 45.8
|  | (2)QA |  | 61.5 | 65.6 | 73.9 | 65.5 |  | 49.1 | 45.9 | 42.9 | 42.6 | 45.1
Table 4: Performance using different variants for the QA template. Base: Refer
to the base QA instruction prompt. For the captioning prompts all models use
their best variants, Q-ViDXL with (2)General and Q-ViDXXL with (2)Dependent.
These results suggest that there is no improvements using more complex and
detailed QA instruction prompts for the reasoning module, achieving more
consistent performances with the simpler base template.

##  5 Conclusion

In this paper, we introduce Q-ViD, a simple, gradient-free approach for zero-
shot video QA. Q-ViD turns video QA into textual QA using frame captions, for
this it relies on an instruction-aware visual language model and uses
question-dependent captioning instructions to obtain specific frame
descriptions useful for the task at hand. This information is later used by a
reasoning module with a question-answering instruction prompt to perform
multiple-choice video QA. Our simple approach achieves competitive or even
higher performances than more complex architectures and methods that rely on
closed models like GPT’s. In our ablation studies we show that using dedicated
instructions to get question-dependent captions work better than common
prompts to get general descriptions from frames to perform video QA using
captions.

## Limitations

Even though, Q-ViD has shown to achieve strong performances for zero-shot
video question answering, our approach suffers from some limitations. While
the adopted instruction-aware multimodal model, InstructBlip, shows to
successfully follow instructions from the question and extract meaningful
information that can help the reasoning module to come up with the right
answer, we have seen that in some cases the model tends to show hallucinations
in the captions, or generate direct short one-word answers instead of a
detailed and question-specific description of the image. On the other hand,
even though experiments with really long videos are not within the scope of
this paper, our approach would no be recommended in those cases, due to the
high memory usage that comes with saving detailed frame captions to create an
entire video description, which would also affect the LLM-based reasoning
module because of the limited amount of tokens allowed as input or due to
memory constrains to process the entire video description.

## References

  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj Bińkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. 2022.  Flamingo: a visual language model for few-shot learning.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 23716–23736. Curran Associates, Inc.
  * Berrios et al. (2023) William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, and Amanpreet Singh. 2023.  Towards language models that can see: Computer vision through the lens of natural language.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chen et al. (2023a) Jun Chen, Deyao Zhu, Kilichbek Haydarov, Xiang Li, and Mohamed Elhoseiny. 2023a.  Video chatcaptioner: Towards the enriched spatiotemporal descriptions.  _arXiv preprint arXiv:2304.04227_.
  * Chen et al. (2023b) Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. 2023b.  Language models are visual reasoning coordinators.  In _ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models_.
  * Chen et al. (2023c) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. 2023c.  Pali: A jointly-scaled multilingual language-image model.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023.  Instructblip: Towards general-purpose vision-language models with instruction tuning.
  * Fang et al. (2023) Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023.  Eva: Exploring the limits of masked visual representation learning at scale.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 19358–19369.
  * Ghandi et al. (2023) Taraneh Ghandi, Hamidreza Pourreza, and Hamidreza Mahyar. 2023.  Deep learning approaches on image captioning: A review.  _ACM Computing Surveys_ , 56(3):1–39.
  * Ghosal et al. (2023) Deepanway Ghosal, Navonil Majumder, Roy Lee, Rada Mihalcea, and Soujanya Poria. 2023.  Language guided visual question answering: Elevate your multimodal language model using knowledge-enriched prompts.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 12096–12102, Singapore. Association for Computational Linguistics.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Guo et al. (2023) Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. 2023.  From images to textual prompts: Zero-shot visual question answering with frozen large language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10867–10877.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022.  Training compute-optimal large language models.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.
  * Hu et al. (2022) Yushi* Hu, Hang* Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022.  Promptcap: Prompt-guided task-aware image captioning.  _arXiv preprint arXiv:2211.09699_.
  * Ko et al. (2023) Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo J Kim. 2023.  Large language models are temporal and causal reasoners for video question answering.  In _EMNLP_.
  * Lei et al. (2018) Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018.  Tvqa: Localized, compositional video question answering.  In _EMNLP_.
  * Li et al. (2022) Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. 2022.  Lavis: A library for language-vision intelligence.
  * Li et al. (2023a) Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. 2023a.  Intentqa: Context-aware video intent reasoning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 11963–11974.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
  * Li et al. (2024) Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. 2024.  Mvbench: A comprehensive multi-modal video understanding benchmark.
  * Li et al. (2020a) Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020a.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  In _EMNLP_.
  * Li et al. (2020b) Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020b.  HERO: Hierarchical encoder for Video+Language omni-representation pre-training.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2046–2065, Online. Association for Computational Linguistics.
  * Li et al. (2023c) Yicong Li, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-Seng Chua. 2023c.  Discovering spatio-temporal rationales for video question answering.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023.  Improved baselines with visual instruction tuning.
  * Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Momeni et al. (2023) Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. 2023.  Verbs in action: Improving verb understanding in video-language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 15579–15591.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.
  * Prasad et al. (2023) Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. 2023.  Rephrase, augment, reason: Visual grounding of questions for vision-language models.
  * Schwenk et al. (2022) Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022.  A-okvqa: A benchmark for visual question answering using world knowledge.
  * Surís et al. (2023) Dídac Surís, Sachit Menon, and Carl Vondrick. 2023.  Vipergpt: Visual inference via python execution for reasoning.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wang et al. (2023) Junke Wang, Dongdong Chen, Chong Luo, Xiyang Dai, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. 2023.  Chatvideo: A tracklet-centric multimodal and versatile video understanding system.
  * Wang et al. (2022a) Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. 2022a.  Internvideo: General video foundation models via generative and discriminative learning.
  * Wang et al. (2022b) Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng Ji. 2022b.  Language models with image descriptors are strong few-shot video-language learners.
  * Wu et al. (2021) Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan. 2021.  STAR: A benchmark for situated reasoning in real-world videos.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Xiao et al. (2021) Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021.  Next-qa: Next phase of question-answering to explaining temporal actions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9777–9786.
  * Yang et al. (2022) Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2022.  Zero-shot video question answering via frozen bidirectional language models.  In _NeurIPS_.
  * Ye et al. (2023) Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023.  mplug-owl: Modularization empowers large language models with multimodality.
  * Yu et al. (2023) Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. 2023.  Self-chained image-language model for video localization and question answering.  In _NeurIPS_.
  * Zeng et al. (2022) Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. 2022.  Socratic models: Composing zero-shot multimodal reasoning with language.
  * Zhang et al. (2023a) Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. 2023a.  A simple llm framework for long-range video question-answering.
  * Zhang et al. (2023b) Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2023b.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.
  * Zhong et al. (2022) Yaoyao Zhong, Wei Ji, Junbin Xiao, Yicong Li, Weihong Deng, and Tat-Seng Chua. 2022.  Video question answering: Datasets, algorithms and challenges.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6439–6455, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_.

##  Appendix A Licences

We use standard licenses from the community for the datasets, codes, and
models that we used in this paper:

  * •

NExT-QA (Xiao et al., 2021): MIT

  * •

STAR (Wu et al., 2021): Apache

  * •

How2QA (Li et al., 2020a): MIT

  * •

TVQA (Lei et al., 2018): MIT

  * •

IntentQA (Li et al., 2023a): N/A

  * •

SeViLa (Yu et al., 2023): BSD 3 - Clause

  * •

LAVIS (Li et al., 2022): BSD 3-Clause

  * •

Pytorch (Paszke et al., 2019): BSD Style

  * •

Q-ViD (Ours): Available soon under the BSD Style license.

##  Appendix B Use of Artifacts

In this work we adopt a open multimodal model, InstructBLIP (Dai et al.,
2023), its application in our approach is consistent with its original
intended use. For Q-ViD we will release our code and we hope it will be useful
for future works.