# X-ray measurement model incorporating energy-correlated material variability
and its application in information-theoretic system analysis

Yijun Ding  1,* and Amit Ashok  1,2  1Wyant College of Optical Sciences,
University of Arizona, Tucson AZ 85721
2Department of Electrical and Computer Engineering, University of Arizona,
Tucson AZ 85721<EMAIL_ADDRESS>
###### Abstract

Extending our prior work, we propose a multi-energy X-ray measurement model
incorporating material variability with energy correlations to enable the
analysis and exploration of the performance of X-ray imaging and sensing
systems. Based on this measurement model, we provide analytical expressions
for bounds on the probability of error, $\mathrm{P_{e}}$, to quantify the
performance limits of an X-ray measurement system for binary classification
task. We analyze the performance of a prototypical X-ray measurement system to
demonstrate the utility of our proposed material variability measurement
model.

††journal: osajournal††articletype: Research Article

##  1 Introduction

Imaging and sensing based on X-ray attenuation is commonly used to non-
destructively discriminate materials in security screening, medical imaging
and industrial inspection [1, 2, 3]. An X-ray measurement model with accurate
statistics of the measurement data is desirable for many purposes such as,
evaluating system performance, developing detection classification algorithms,
and optimization of object-reconstruction algorithms.

The performance of attenuation-based X-ray imaging and sensing systems is
limited by at least two fundamental factors in the measurement data: shot
noise and inherent material variability [4]. The shot noise stems from the
randomness in the generation, attenuation and detection of X-ray photons. The
material variability, arising from the inherent fluctuations in material
composition and density, limits the discrimination of materials. Therefore, a
rigorous evaluation of an X-ray imaging or sensing system must take into
account both shot noise and material variability.

An X-ray measurement model that considers only the shot noise has been
analyzed in the context of material-discrimination applications by Huang et.
at and Lin et. al. [5, 6]. Recently, Masoudi et. at. [7] improved the model by
incorporating material variability under the assumption of energy
(statistical) independence. However, the energy-correlations are intrinsic in
X-ray attenuation and cannot be ultimately ignored. In this paper, we propose
an X-ray measurement model that considers energy-correlated material
variability as well as shot noise.

Many imaging systems are used to perform binary-classification tasks. For
example, a luggage scanner at an airport checkpoint is used to determine
whether a luggage bag contains threat material or not; mammography is often
used to classify tumor-present or tumor-absent; in industrial non-destructive
evaluation, radiographs of parts are used to examine the existence of defects.
An objective assessment of an system must take into account the task of the
system [8, 4].

Task-specific information (TSI) [8], which is the information content relevant
to the task, is commonly used as an objective assessment metric. For
classification tasks, Shannon mutual information ($\mathrm{I_{S}}$) [9] is a
natural choice as an information-theoretic metric, because it can be used to
bound probability of classification error ($\mathrm{P_{e}}$) [10, 9, 11, 12,
13]. Although $\mathrm{I_{S}}$ is expensive to compute for non-trivial
distributions, we are able to derive closed-form expressions for bounds on
$\mathrm{I_{S}}$ and bounds on $\mathrm{P_{e}}$ for mixture distributions with
the help of a recent work [14].

This paper is organized as follows. We derive the measurement model in Sec 2.
Sec 3 reviews $\mathrm{I_{S}}$, $\mathrm{P_{e}}$ for binary-classification
tasks and derives closed-form expressions for bounds on $\mathrm{I_{S}}$ and
bounds on $\mathrm{P_{e}}$. In Sec 4, as an example application, we apply the
model to a simplified luggage-scanner and present the simulation results. Sec
5 discusses the advantages and drawbacks of the model. Sec 6 provides a
succinct conclusion.

##  2 Measurement model

The measurement model relates the data to the object and describes the
statistical properties of the data. A general form of the measurement model
can be written as

${\bm{g}}=\mathcal{H}{\bm{f}}+{\bm{n}}=\mathcal{H}(\bar{\bm{f}}+\Delta{\bm{f}})+{\bm{n}},$
(1)

where ${\bm{g}}$ is the data, $\mathcal{H}$ describes the system, $\bm{f}$ is
the object, and $\bm{n}$ is the system noise, $\bar{\bm{f}}$ is the ensemble
mean of the object, and $\Delta{\bm{f}}$ describes the material variability.

###  2.1 X-ray attenuation coefficient $\mu$

In the energy range commonly used for X-ray transmission imaging, the
interaction between X-ray photons and the medium can be categorized into the
following three processes: photoelectric absorption, Compton scattering and
coherent (Rayleigh) scattering. In photoelectric absorption, a photon
disappears and the energy of the photon transfers to an electron in the
material; in Compton scattering, an X-ray photon is deflected and transfers a
portion of its energy to an electron; and in coherent scattering, an X-ray
photon is deflected, but retains its energy. The strength of each interaction
process can be characterized by the energy of the photon $E$, the atomic
number of the medium $Z$ and the density of the medium $\rho$. For a material
with fixed $Z$ and $\rho$, the X-ray attenuation coefficient $\mu(E)$ is a
function of the X-ray energy $E$. Variations in $\mu(E)$ due to variability in
$Z$ and $\rho$ demonstrate intrinsic energy correlation.

For multi-element compounds and mixtures, the attenuation coefficient is

$\mu(E)=\rho\sum_{c}\frac{w_{c}}{\rho_{c}}\mu_{c}(E)$ (2)

where $\rho$ represents the density of the medium, $w_{c}$ is the weight
fraction of the $c^{th}$ element in the compound or mixture, $\mu_{c}(E)$ and
$\rho_{c}$ are the attenuation and the density of the $c^{th}$ element. When
there is variability inherent in the description of a material, which may stem
from density fluctuations, composition variations and packaging differences,
the attenuation coefficient $\mu(E)$ becomes a random process.

To consider material variation, we assume $\mu(E)$ is a Gaussian random
process with covariance function $\Sigma_{\mu}(E,E^{\prime})$. For any set of
energy $E_{1}$, $E_{2}$, ..,$E_{R}$, the random variable $\mu(E_{1})$,
$\mu(E_{2})$, …, $\mu(E_{R})$ are joint Gaussian random variables. If we
denote the set of random variables $\mu(E_{i})$ by the vector $\bm{\mu}$,
whose mean is $\bm{\mu}_{0}$ and the covariance matrix is $\Sigma_{\mu}$, then
the joint probability density is

$\mathrm{pr}(\bm{\mu})=\mathcal{N}(\bm{\mu}|\bm{\mu}_{0},\Sigma_{\mu})=\frac{\exp[-\frac{1}{2}(\bm{\mu}-\bm{\mu}_{0})^{T}\Sigma_{\mu}^{-1}(\bm{\mu}-\bm{\mu}_{0})]}{\sqrt{2\pi^{R}|\Sigma_{\mu}|}},$
(3)

where $|\Sigma_{\mu}|$ is the determinant of $\Sigma_{\mu}$, $(\cdot)^{T}$ is
the transpose of matrix $(\cdot)$, $R$ is the length of the vector $\bm{\mu}$.
When $\Sigma_{\mu}$ is singular, the Gaussian distribution turns into a Dirac
delta function in the corresponding dimension.

###  2.2 X-ray attenuation $\tau$

Figure 1: Illustration of X-ray penetrating multiple objects.

Now consider a beam of X-ray penetrating multiple items, as illustrated in
Figure 1, where each item contains one material. If there are a total of
$N_{it}$ items between the X-ray source and the detector, the total
attenuation along the path is

$\tau(E)=\int\mu(E,l)~{}\mathrm{dl}=\sum_{t=1}^{N_{it}}\mu_{t}(E)\,l_{t},$ (4)

where $t$ is the index for items along the X-ray path, $l_{t}$ is the length
of the material along the path, and $\mu_{t}(E)$ is the attenuation profile of
the material in the ${t}^{th}$ item. In the following discussion, we refer to
$\tau(E)$ as the total attenuation.

Under the assumption that the fluctuations of $\mu(E)$ in two different items
are independent, $\tau(E)$ is a Gaussian random processes, since it is the sum
of independent Gaussian random processes, $\mu_{t}(E)$. For a set of energies
$E_{1}$, $E_{2}$, ..,$E_{K}$, the set of total attenuation $\bm{\tau}$ follows
a multivariate normal distribution

$\mathrm{pr}(\bm{\tau})=\mathcal{N}(\bm{\tau}|\bm{\tau}_{0},\Sigma_{\tau})=\frac{\exp[-\frac{1}{2}(\bm{\tau}-\bm{\tau}_{0})^{T}\Sigma_{\tau}^{-1}(\bm{\tau}-\bm{\tau}_{0})]}{\sqrt{2\pi^{R}|\Sigma_{\tau}|}},$
(5)

where $R$ is the length of the vector $\bm{\tau}$ and

$\bm{\tau}_{0}=\sum_{t=1}^{N_{it}}\bm{\mu}_{0,t}\,l_{t},$ (6)

$\Sigma_{\tau}=\sum_{t=1}^{N_{it}}\Sigma_{\mu,t}\,l_{t}^{2}.$ (7)

###  2.3 Beer’s law

An illustration of the propagation of X-ray photons starting from the X-ray
source, through the object, and ending with photon detection is shown in
Figure 1. The X-ray attenuation follows beer’s law,

$J(E)=\frac{N_{0}S(E)}{t}e^{-\tau(E)},$ (8)

where $J(E)$ is the mean spectral flux incident on a detector element at
energy $E$, $N_{0}S(E)/t$ is the source spectral flux, $\tau(E)$ is the total
attenuation as a function of $E$, and $t$ is the exposure time. More
specifically, $N_{0}$ is the number of photons emitted from the X-ray tube in
the solid angle extended by a detector element over the exposure time $t$, and
$S(E)$ is the normalized x-ray source spectrum. The units of $J(E)$ and $S(E)$
are (s$\cdot$keV)-1 and keV-1, respectively. We denote
$\tau(E)=\tau_{0}(E)+\Delta\tau(E)$, where $\Delta\tau(E)$ is fluctuation or
perturbation in attenuation around $\tau_{0}(E)$, and
$J_{0}(E)=N_{0}S(E)e^{-\tau_{0}(E)}/t$. When the material variance
(perturbation) is small, or more specifically, when $\Delta\tau(E)\ll 1$, we
can approximate $e^{-{\Delta\tau(E)}}$ with the first two terms in the Taylor
expansion, resulting in:

$J(E)\approx J_{0}(E)-J_{0}(E){\Delta\tau(E)}.$ (9)

The remainder of the approximation can be bounded by

$|R_{2}(E)|<\frac{J_{0}(E)|\Delta\tau(E)|^{3}}{6}.$ (10)

If a requirement on the remainder is to be less than $1\%$ of $J_{0}(E)$,
$|\Delta\tau(E)|$ should be less than 0.39. One can always guarantee such a
requirement by splitting a material with large variations into multiple
materials with smaller variations. For example, 40$\%$ sugar water with sugar
content varying from 20$\%$ to 60$\%$ can be split into two materials with
sugar content varying from 20$\%$ to 40$\%$ and 40$\%$ to 60$\%$,
respectively.

In the limit of small variability, the two-term approximation of $J(E)$, as
defined by Equation (9), follows a normal distribution when $\tau(E)$ follows
a normal distribution. Therefore, $\bm{J}$, which is $J(E)$ at a set of
energies, approximately follows a normal distribution:

$\mathrm{pr}(\bm{J})=\mathcal{N}(\bm{J}|{\bm{J}_{0}},\Sigma_{J}).$ (11)

The mean and covariance matrix are

${\bm{J}}_{0}=\frac{N_{0}\bm{S}}{t}\odot e^{-\bm{\tau}_{0}},$ (12)

and

$\Sigma_{J}=(\bm{J}_{0}\bm{J}_{0}^{T})\odot\Sigma_{\tau},$ (13)

where $\odot$ is element-wise multiplication.

###  2.4 Detector response and energy binning

If the energy response of the detector is linear, one can express the mean
photon count collected in the $m^{th}$ energy bin as

$J_{m}=t\int_{0}^{\infty}J(E)D_{m}(E)\mathrm{dE},$ (14)

where $D_{m}(E)$ is the detector response of the $m^{th}$ energy bin to a
photon with energy $E$ and $t$ is the exposure time.

With a total of $M$ energy bins, the mean photon count
$\\{J_{1},J_{2},...,J_{M}\\}$ after energy binning can be represented by a
vector $\mathbf{J}_{d}$, and

$\mathbf{J}_{d}=t\mathcal{D}\,J(E),$ (15)

where $\mathcal{D}$ is the detector response operator and the subscript $d$
denotes detector.

When $\mathcal{D}$ is a linear operator and $J(E)$ is a Gaussian random
process, $\bm{J}_{d}$ follows a normal distribution. The mean and variance are

$\mathbf{J}_{d0}=tD\mathbf{J}_{0}$ (16)

and

$\Sigma_{J_{d}}=t^{2}D\Sigma_{J}D^{T},$ (17)

where $D$ is the matrix form of the operator $\mathcal{D}$ for a set of $R$
energies.

###  2.5 Shot noise

In attenuation-based X-ray imaging, the data collected is often the number of
detected X-ray photons. Photon counting intrinsically introduces shot noise,
hence the number of X-ray photons detected in one energy bin, $g$, follows a
Poisson distribution:

$\mathrm{Pr}(g|J_{d})={\mathcal{P}oiss}(g|J_{d})=\frac{(J_{d})^{g}e^{-J_{d}}}{g!},$
(18)

where $\mathcal{P}oiss$ indicates a Poisson distribution and $J_{d}$ is the
mean photon count in the energy bin.

When the mean photon count is relatively large (i. e. more than say, 10
photons), the Poisson distribution can be approximated by a Gaussian/normal
distribution [barrett1996radiological]. Denoting the continuous variable
$x=g$,

$\mathrm{pr}(x|J_{d})\approx\mathcal{N}(x;J_{d},J_{d})=\frac{\exp[-(x-J_{d})^{2}/(2J_{d})]}{\sqrt{2\pi
J_{d}}}.$ (19)

When the mean photon count $J_{d}\geq 100$, the error introduced by the
Gaussian approximation is less than $4\%$ for $x=\bar{x}\pm\sigma(x)$, where
$\bar{x}$ and $\sigma(x)$ are the mean and standard deviation of $x$,
respectively. A derivation of the percentage error is given in Appendix B.

###  2.6 Combined model

In our measurement model, we combine statistics of the shot noise and the
statistics of the energy-correlated material variability. Thus, the covariance
of the measurement data is a summation of the covariance matrices
corresponding to the shot noise and the material variability. More
specifically, if we denote the continuous data vector as $\bm{x}$, the
probability density function (PDF) of $\bm{x}$ with mean photon counts,
$\bm{J}_{d0}$, and a covariance matrix induced by material-variation,
$\Sigma_{J_{d}}$, is

$\mathrm{pr}(\bm{x}|\bm{J}_{d0},\Sigma_{J_{d}})\approx\mathcal{N}(\bm{x};\,\bm{J}_{d0},\,\Sigma_{J_{d}}+\text{diag}(\bm{J}_{d0})).$
(20)

A detailed derivation is provided in Appendix C.

We define the combined data over all detector pixels as $\bm{g}$. The
probability distribution function of $\bm{g}$ is the joint distribution of all
$\bm{x}_{n}$,

$\mathrm{pr}(\bm{g})=\mathrm{pr}_{\bm{x}_{1},\bm{x}_{2},...\bm{x}_{N}}(\bm{x}_{1},\bm{x}_{2},...\bm{x}_{N}),$
(21)

where $n$ is the detector-pixel index and $N$ is the number of detector
pixels. The joint distribution is determined by the geometry of the object and
setup of the system.

Now, consider an ensemble of K objects that are examined by an X-ray system.
Then the data representing the ensemble of objects can be described by the
following mixture distribution:

$\mathrm{pr}(\bm{g}|\bm{a})=\sum_{i=1}^{K}a_{i}\mathrm{pr}_{i}(\bm{X}),$ (22)

where $a_{i}$ is the probability of the $i_{th}$ object occurring in the
ensemble, $K$ is the number of objects in the ensemble, $\sum a_{i}=1$, and
$\mathrm{pr}_{i}(\bm{X})$ is the probability distribution function of
measurement data $X$ when the $i^{th}$ object is imaged. Note that we have
used the discretized data $\bm{g}$ and the continuous variables
$\\{\bm{x}_{1},\bm{x}_{2},...\bm{x}_{N}\\}$ interchangeably.

##  3 TSI for binary classification tasks

Shannon mutual information, $\mathrm{I_{S}}$, has long been used as a metric
to quantify the task-specific fidelity of a measurement with respect to
classification tasks [8]. This is because $\mathrm{I_{S}}$ is related to the
error probability, $\mathrm{P_{e}}$ through Fano’s inequality [10] and
Kovalevskij’s inequality [11, 12, 13]. In this section, we provides a brief
summary of the relation between $\mathrm{I_{S}}$ and $\mathrm{P_{e}}$ and
provides closed-form expressions for bounds on $\mathrm{I_{S}}$ and
$\mathrm{P_{e}}$ defined on our X-ray measurement model.

The system performance is object dependent. Properties of the object, such as
the size, the material and the geometry, affect the distribution of the data
and hence the difficulty of the classification task. To reduce the dependence
on test objects, a general assessment of a system should consider a large
ensemble of objects. In the following discussion, we consider an ensemble of
$K$ objects that consists of $K_{1}$ objects in the first class and $K_{2}$
objects in the second class. The probability of the $i^{th}$ object in the
ensemble is $a_{i}$. The probabilities of the two class labels are $P_{1}$ and
$P_{2}$, where $P_{1}+P_{2}=1$ for binary classification.

###  3.1 Bounds on $\mathrm{I_{S}}$

The $\mathrm{I_{S}}$ is defined as

$\mathrm{I_{S}}(\mathbf{g};C)=\mathrm{H}(\bm{g})-\sum_{c=1}^{2}P_{c}\mathrm{H}(\bm{g}|C=c),$
(23)

where $\mathrm{H}(\bm{g})$ is the Shannon entropy of the distribution of the
measured data, and $\mathrm{H}(\bm{g}|C=c)$ is the entropy of the conditional
distribution of the measured data given the class $C$.

When the data is a mixture distribution, the $\mathrm{I_{S}}$ between data and
class label has no closed-form expressions. However, $\mathrm{I_{S}}$ can be
bounded by bounds of the entropy [14],

$\begin{split}\mathrm{I_{S}}(\mathbf{g};C)&\geq\hat{\mathrm{H}}_{BD}({\bf
g})-\sum_{c=1}^{2}P_{c}\hat{\mathrm{H}}_{KL}({\bf g}|C=c),\\\
\mathrm{I_{S}}(\mathbf{g};C)&\leq\hat{\mathrm{H}}_{KL}({\bf
g})-\sum_{c=1}^{2}P_{c}\hat{\mathrm{H}}_{BD}({\bf g}|C=c),\end{split}$ (24)

where $\hat{\mathrm{H}}_{BD}({\bf g})$ and $\hat{\mathrm{H}}_{KL}({\bf g})$
are lower and upper bounds on entropy based on pair-wise Bhattacharyaa
distance (BD) and pair-wise Kullback-Leibler (KL) divergence, respectively.
More specifically, the bound on entropy based on either divergence is given by

$\hat{\mathrm{H}}_{D}(\mathbf{g})=\sum_{i=1}^{K}a_{i}\mathrm{H}({\mathrm{pr}}_{i})-\sum_{i=1}^{K}a_{i}\ln\sum_{j=1}^{K}a_{j}\exp(-\mathrm{D}({{\mathrm{pr}}_{i},{\mathrm{pr}}_{j}})),$
(25)

where ${\mathrm{pr}}_{i}$ is the PDF of the data $\bm{g}$ if the $i^{th}$ bag
is measured and $\mathrm{D}(p,q)$ can be either Bhattacharyaa distance or KL
divergence, which are defined by

$\begin{split}\mathrm{BD}(p,q)&=-\ln\int\mathrm{dx}~{}\sqrt{p(x)\,q(x)},\\\
{\text{a}nd}\quad\mathrm{KL}(p,q)&=\int\mathrm{dx}~{}p(x)\ln\frac{p(x)}{q(x)},\end{split}$
(26)

respectively.

An upper bound on $\mathrm{I_{S}}$ based on pair-wise KL divergence and a
lower bound based on pair-wise BD are provided in Appendix A. The minimum of
the two upper bounds and the maximum of the two lower bounds can be used as
the tighter version of the bounds on $\mathrm{I_{S}}$.

###  3.2 Bounds on $\mathrm{P_{e}}$

Starting from $\mathrm{I_{S}}$, the Fano’s inequality [10] provides a lower
bound on $\mathrm{P_{e}}$ for binary classification, as following

$P_{e}\geq h_{b}^{-1}[\mathrm{H}(C)-\mathrm{I_{S}}(\mathbf{g};C)],$ (27)

where $h_{b}(x)=-x\log_{2}(x)-(1-x)\log_{2}(1-x)$ is the binary entropy
function, $h_{b}^{-1}(\cdot)$ is the inverse function of $h_{b}(\cdot)$, and
$\mathrm{H}(C)=h_{b}(P_{1})$ is the Shannon entropy of the class label $C$.
More specifically, one can calculate $P_{e}$ by placing the value
$\mathrm{H}(C)-\mathrm{I_{S}}(\mathbf{g};C)$ on the left side of the binary
entropy function and solving for $x$. When $P_{e}\ll 1$,
$\mathrm{H}(C)-\mathrm{I_{S}}(\mathbf{g};C)\approx-\mathrm{P_{e}}\log{\mathrm{P_{e}}}$
and hence is on the same order of magnitude as $P_{e}$.

An upper bound on binary classification errors $\mathrm{P_{e}}$, which is
tighter than Kovalevskij’s inequality, has been reported recently [13],

$P_{e}\leq{\text{min}}\left\\{P_{min},\,f_{ub}^{-1}[\mathrm{H}(C)-\mathrm{I_{S}}(\mathbf{g};C)]\right\\},$
(28)

where $P_{min}$ is min$\\{P_{1},P_{2}\\}$, and $f_{ub}(x)$ is an upper bound
function defined by

$f_{ub}(x)=-P_{min}\log_{2}\frac{P_{min}}{x+P_{min}}-x\log_{2}\frac{x}{x+P_{min}},$
(29)

and $f_{ub}^{-1}(\cdot)$ is the inverse function of $f_{ub}(\cdot)$.

###  3.3 Closed-from expressions for pair-wise BD and KL divergence

If we assume that the measurement data at different pixels are statistically
independent with each other, the PDF of the measurement data becomes a product
of the PDFs of the data measured at all pixels:

$\mathrm{pr}(\bm{g})=\prod_{n=1}^{N}\mathrm{pr}(\bm{x}_{n}\,|\,\bm{J}_{d0,n},\Sigma_{J_{d},n})\approx\prod_{n=1}^{N}\mathcal{N}(\bm{x}_{n}\,;\bm{J}_{d0,n},\Sigma_{J_{d},n}),$
(30)

where $n$ is the detector-pixel index, and $N$ is the total number of detector
pixels.

Calculation of bounds on $\mathrm{I_{S}}$ and $\mathrm{P_{e}}$ require pair-
wise Bhattacharyaa distance and pair-wise KL divergence. To simplify notation,
we define

$\Delta\mathbf{J}_{n}=\mathbf{J}_{d0,n,i}-\mathbf{J}_{d0,n,j},$ (31)
$\Sigma_{n,i}=\Sigma_{J_{d},n,i}+\text{diag}(\bm{J}_{d0,n,i}),$ (32)

and

$\Sigma_{n}=\Sigma_{n,i}+\Sigma_{n,j}.$ (33)

The analytical form of Bhattacharyya distance can be expressed as

$\mathrm{BD}(\mathrm{\mathrm{pr}}_{i},\mathrm{\mathrm{pr}}_{j})=\sum_{n=1}^{N}\left[\frac{\Delta\mathbf{J}^{T}_{n}\Sigma_{n}^{-1}\Delta\mathbf{J}_{n}}{4}-\frac{\ln{|\Sigma_{n,i}\Sigma_{n,j}|}}{4}+\frac{\ln{|\Sigma_{n}|}}{2}-\frac{M\ln{2}}{2}\right];$
(34)

and the analytical form of KL divergence can be expressed as

$\mathrm{KL}(\mathrm{\mathrm{pr}}_{i},\mathrm{\mathrm{pr}}_{j})=\frac{1}{2}\sum_{n=1}^{N}\Delta\mathbf{J}^{T}_{n}(\Sigma_{n,j})^{-1}\Delta\mathbf{J}_{n}-\ln{|\Sigma_{n,i}|}+\ln{|\Sigma_{n,j}|}+\mathrm{tr}(\Sigma_{n,j}^{-1}\Sigma_{n,i})-M,$
(35)

where $\mathrm{tr}(\cdot)$ is the trace of the matrix.

##  4 Illustrative System Study and Results

In this section, we apply our measurement model to study a simple X-ray
measurement system for the task of material-based threat detection (i. e. a
binary classification problem). The X-ray system, as illustrated in Figure 2,
has 10 X-ray sources that produce parallel pencil-beams and 10 corresponding
photon-counting energy-sensitive detector elements. A source with a tungsten
target operating at 160 kVp is assumed, and the corresponding source spectrum
was generated with SpekCalc [poludniowski2009spekcalc]. The energy-sensitive
detector can have 1, 2 and 3 energy bins. The bin edges are determined by
balancing the photon count after attenuation. More specifically, the bin edges
are [30, 160] keV for one bin, [30, 70, 160] keV for two bins, and [30, 60,
85, 160] keV for three bins. The objects under inspection contain 10 vials of
materials, and the location of each vial is along one parallel-beam X-ray
path. Each vial contains 4 materials, which is randomly sampled from a library
of materials. The lengths of the vials are randomly chosen between 0.5 cm to
20 cm.

A material library and a variability model of the material composition has
been previously developed [7]. The material library contains 25 threat
materials and 33 non-threat materials. Examples of threat materials are
ammonium nitrite, hydrogen peroxide and gun powder; examples of non-threat
materials are milk, toothpaste and polyethylene. The composition and variance
of weight fractions of each material were determined based on industrial
standards. The density variation was folded in either by varying the density
$\rho$ or by adding air as a component. For each material, 1000 composition
realizations were randomly generated; and for each material realization, an
X-ray attenuation profile was computed based on the NIST XCOM database
[berger1998photon]. For each material, the mean and covariance of the 1000
attenuation profiles were used as the mean and the covariance of $\mu(E)$ in
our model. The number of energy samples in $\bm{\mu}$ and the source spectrum
is $R=180$.

We simulate objects, where each object has 10 vials, in pairs and each pair
consists of one object containing one threat material and one object
containing no threat material. A threat object and a non-threat object in a
pair share the same geometry (aka. vial lengths and materials) and are thus
different by only one material. An illustration of an object pair is shown in
Figure 2. We simulate objects in pairs because of the following two reasons:
(1) the system performance is dominated by objects that are located close to
the class boundary in the data space; and (2) in general, the distance in the
data space between two objects in a pair is closer than that between two
random objects.

An ensemble of 160 bag-pairs was simulated. Equal prevalence of each class and
equal probability of occurrence of each bag-pair were assumed for the
calculation of the TSI measure.

Figure 2: Illustration of the simulated X-ray luggage scanner.

There is no spatial correlation between data measured from different detector
elements in this prototypical example. The closed-form expressions derived
under the assumption of spatial independence were used to calculate the bounds
on $\mathrm{I_{S}}$ and $\mathrm{P_{e}}$. First, we study the behavior of
bounds on $\mathrm{I_{S}}$ and bounds on $\mathrm{P_{e}}$ for one, two and
three energy bins. Second, we compare the following three measurement models
in terms of bounds on $\mathrm{P_{e}}$: (1) shot noise only, (2) material
variation only, and (3) combined model. Lastly, we compare the energy-
correlated measurement model and material-variation model with the energy-
uncorrelated models in terms of lower bound on $\mathrm{P_{e}}$.

###  4.1 Behavior of TSI metrics

In this section, we present results calculated with the energy-correlated
measurement model that incorporates both shot noise and material variation and
illustrate the behavior of the TSI metrics.

Figure 3: Upper bound (blue) and lower bound (red) on $\mathrm{I_{S}}$,
$\mathrm{H}(C)-\mathrm{I_{S}}$ and detection error probability
$\mathrm{P_{e}}$ for one (top), two (middle) and three (bottom) energy bins.
The maximum possible value of $\mathrm{P_{e}}$, which is 0.5 for equal prior,
is plotted in black dashed line.

Figure 3 presents bounds on $\mathrm{I_{S}}$ and $\mathrm{P_{e}}$ as functions
of source photon budget $N_{0}$ for one, two and three energy bins,
respectively. The upper bound (blue solid line) and lower bound (red dashed
line) of $\mathrm{I_{S}}$, $\mathrm{H}(C)-\mathrm{I_{S}}$, and
$\mathrm{P_{e}}$ are shown from left to right in each figure. Note that for
binary classification tasks with equal prevalence, $\mathrm{H}(C)=1$. The
$\mathrm{H}(C)-\mathrm{I_{S}}$ is plotted, because it is in the same order of
magnitude with $\mathrm{P_{e}}$ when $\mathrm{P_{e}}\ll 1$.

For a binary classification task with equal prevalence of the two classes,
$\mathrm{I_{S}}\in[0,1]$ and $\mathrm{P_{e}}\in[0,0.5]$. The task-specific
performance improves when $\mathrm{I_{S}}$ increases and $\mathrm{P_{e}}$
decreases. When the source photon budget $N_{0}$ increases, both upper bound
and lower bound of $\mathrm{I_{S}}$ increases, which means that, as source
count increases, the imaging system captures more information for the binary
classification task. However, both $\mathrm{I_{S}}$ and $\mathrm{P_{e}}$
saturates at high photon regime, which is due to material variability.

Figure 4: Bounds on detection error rate $P_{e}$ for 1, 2, and 3 energy bins.

To show $\mathrm{P_{e}}$ for all three energy binning occasions, Figure 4
presents the bounds on $\mathrm{P_{e}}$ for 1, 2, and 3 bins. At a fixed
source count $N_{0}$, bounds on $\mathrm{P_{e}}$ decrease as the number of
energy bins increases. In fact, for systems with only one energy bin, the
upper bound on $\mathrm{P_{e}}$ is 0.5 and the lower bound saturates at around
0.25, which means that binary classification with such a system is close to
random guess. The specific number of bounds on $\mathrm{P_{e}}$ changes with
the material library and other simulation setups, but the general trend that
more energy bins provides higher task-specific information is valid.

###  4.2 Three measurement models

Figure 5: Lower bound (left) and upper bound (right) on $\mathrm{P_{e}}$ as
functions of source photon count for measurement model that considers only the
shot noise (dotted lines), that considers only the energy-correlated material
variation (dashed line) and that considers both the shot noise and the energy-
correlated material variation. Results for detectors that have one, two and
three energy bins are indicated in blue, red and green.

Figure 5 shows the lower bound and the upper bound of $\mathrm{P_{e}}$ for
three measurement models as a function of source photon number ($N_{0}$). The
three measurement modes consist of a model that considers only the shot noise
(approximated as Gaussian, dot-dashed lines), a model that considers only the
energy-correlated material variation (dashed lines), and a model that
considers both the shot noise and the energy-correlated material variation
(solid lines). Results for detectors with 1, 2 and 3 energy bins are presented
in different colors. At low count region, the system performance is limited by
the quantum noise as the bounds on $\mathrm{P_{e}}$ of the combined model is
similar to that of the model considering only quantum noise; at high count
region, the system performance is limited by the material variation as the
bounds on $\mathrm{P_{e}}$ of the combined model approaches that of the model
considering only material variation.

###  4.3 Energy-correlated model vs. energy-uncorrelated model

Figure 6: Comparison of the energy-correlated model with energy-uncorrelated
model in terms of lower bound on $\mathrm{P_{e}}$. The material-variation-only
model (blue lines) and the full measurement model (red dashed lines) are
considered.

We compare the energy-correlated models with the energy-uncorrelated models in
Figure 6. From top to bottom, results of detectors with 1, 2 and 3 energy bins
are presented. In each figure, the results calculated from the material-
variation model are also presented for reference (red dashed line). When there
is one energy bin, the bound on $\mathrm{P_{e}}$ of the energy-uncorrelated
model is lower than that of the correlated model. When the number of energy
bins equals to 2, the bound on $\mathrm{P_{e}}$ of the uncorrelated model is
close to that of the correlated model. When the number of energy bins equals
to 3, the bound on $\mathrm{P_{e}}$ of the uncorrelated model is much higher
than that of the correlated model.

The difference in bound on $\mathrm{P_{e}}$ calculated from the energy-
uncorrelated model and the energy-correlated model is due to the difference in
the material variation considered in the two models, as indicated by the
difference in the red-dashed lines. When energy correlation in the data is
considered, $\Sigma_{J}$, which is the covariance matrix of $J(E)$, is
calculated based on Equation (13); in contrary, when energy correlation is not
considered, the off-diagonal elements of $\Sigma_{J}$ are ignored. Since the
energy-binning process is an integration over energy, each element of
$\Sigma_{J_{d}}$ is the sum of a block matrix in $\Sigma_{J}$. For example,
when there is one energy bin, $\Sigma_{J_{d}}$ is the sum of all elements in
$\Sigma_{J}$ when energy correlation is considered; while in the uncorrelated
model, $\Sigma_{J_{d}}$ is the sum of only the diagonal elements of
$\Sigma_{J}$. Therefore, $\Sigma_{J_{d}}$ of the energy-correlated model is
larger than that of the uncorrelated model for the one energy bin scenario.

When there are more than one energy bin, the comparison is more complex and we
introduce a method to quantitatively compare the material variability model
with energy correlation and that without energy correlation. The size of the
material variability can be quantified by the volume of the noise bubble
induced by the material variability. If we define the volume of the noise
bubble as the volume of a $M$-dimensional ellipsoid determined by the
covariance matrix $\Sigma_{J_{d}}$, where $M$ is the number of energy bins.
The eigenvector of the covariance matrix from singular value decomposition
provides the principle axes of the ellipsoid and the square root of the
eigenvalues are half the length of the principle axes. For example, when
energy correlation is not considered, $\Sigma_{J_{d}}$ is diagonal with each
element equals to the variance of detected photon count in the corresponding
energy bin; the square root of an eigenvalue is the standard deviation of the
detected photon count; and the volume of the ellipsoid determined by the
covariance matrix is the product of the standard deviation in each energy bin.

To compare the material variability estimated by the energy-correlated model
and uncorrelated model, we can calculate the ratio of the two volumes
determined by the two ellipsoids. Denote this ratio as
$r=V_{corr}/V_{uncorr}$. In the dataset used in Figure 6, $\log_{10}(r)$ is
$1.00\pm 0.07$ (mean $\pm$ standard deviation) for one energy bin, $0.98\pm
0.45$ for two energy bins and $-1.08\pm 1.05$ for three energy bins. For the
set of bags we studied, the uncorrelated model always underestimates the
material variability for one energy bin; and with three energy bins, the
uncorrelated model often severely overestimates the material variability.

##  5 Discussion

To incorporate material variation, a linear perturbation of the beer’s law is
used in the derivation from Equation 8 to Equation 9. This perturbation is
equivalent to approximating the sum of correlated log-normal random variables
with a normal distribution, which has been justified in Section 2 for small
material variations. One direction of future work is to examine other
approximations of the sum of log-normal random variables and developing a
measurement model that accommodates large material variations. Such a model
could further reduce the number of materials in the material library and hence
reduce the number of objects needed in a study.

At low count region, the measurement model should be applied with caution, due
to the break down of using Gaussian distribution to approximate the statistics
of the shot noise. In comparison, Masoudi et. at. [7] incorporated the Poisson
distribution in an energy-uncorrelated model. Their model performances better
than our model, when shot noise is dominating and material variation is not
important. However, many X-ray systems are not photon starving and the effect
of material variation should be considered. To address the concern about the
accuracy of the Poisson-Gaussian approximation, we provide an error analysis
in Appendix B.

We assumed the attenuation profiles, $\mu(E)$, follow normal distributions and
derived that the measurement data $\bm{g}$ following normal distributions
under the following two assumptions: (1) The material variation $\Delta\tau$
is small, and (2) the distribution of the shot noise can be approximated by a
Gaussian. Although our derivation relied heavily on the normality of data, the
mean and covariance matrix of the data $\bm{g}$ are independent to the shape
of the distributions, as long as the above assumption (1) is still valid.

The main goals of this work include: presenting the framework, justifying the
approximations used in the measurement model, and studying the effect of
energy correlation. In future work, we will incorporate spatial correlation
and other sources of variations, such as source fluctuation and detector
variation.

##  6 Conclusion

In this work, we have presented an energy-correlated X-ray measurement model
that incorporates both material variation and shot noise. Energy correlations
are inherent in a material’s attenuation profile and affect the system
performance. Therefore, it is important to consider the energy correlations.
We successfully modeled the shot noise and energy-correlated material
variation by a multivariate Gaussian model. Furthermore, under the assumption
of no spatial correlation, we provided analytical forms for TSI metrics,
including bounds on $\mathrm{I_{S}}$ and bounds on $\mathrm{P_{e}}$, for
binary classification tasks. Spatial correlation of the X-ray measurement data
is not explored in this work and will be a focus of future study.

## Funding

The authors gratefully acknowledge the support of the US Department of
Homeland Security. The research for this project was conducted under contract
with the U.S Department of Homeland Security (DHS) Science and Technology
Directorate (S&T), contract HSHQDC-16-C-B0014. The opinions contained herein
are those of the contractors and do not necessarily reflect those of DHS S&T.

## Disclosures

The authors declare that there are no conflicts of interest related to this
article.

## References

  * [1] K. Wells and D. Bradley, “A review of x-ray explosives detection techniques for checked baggage,” Applied Radiation and Isotopes 70, 1729–1746 (2012).
  * [2] J. Hsieh, _Computed Tomography Principles, Design, Artifacts, and Recent Advances_ (John Wiley & Sons, 2009).
  * [3] R. Hanke, T. Fuchs, and N. Uhlmann, “X-ray based methods for non-destructive testing and material characterization,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 591, 14–18 (2008).
  * [4] H. H. Barrett, “Objective assessment of image quality: effects of quantum noise and object variability,” JOSA A 7, 1266–1278 (1990).
  * [5] J. Huang and A. Ashok, “Information optimal compressive x-ray threat detection,” in _Computational Optical Sensing and Imaging,_ (Optical Society of America, 2015), pp. CTh2E–4.
  * [6] Y. Lin, G. G. Allouche, J. Huang, A. Ashok, Q. Gong, D. Coccarelli, R.-I. Stoian, and M. E. Gehm, “Information-theoretic analysis of x-ray photoabsorption based threat detection system for check-point,” in _Anomaly Detection and Imaging with X-Rays (ADIX),_ vol. 9847 (International Society for Optics and Photonics, 2016), p. 98470F.
  * [7] A. Masoudi, J. Voris, D. Coccarelli, J. Greenberg, M. Gehm, and A. Ashok, “X-ray measurement model and information-theoretic system metric incorporating material variability (conference presentation),” in _Anomaly Detection and Imaging with X-Rays (ADIX) III,_ vol. 10632 (International Society for Optics and Photonics, 2018), p. 106320H.
  * [8] M. A. Neifeld, A. Ashok, and P. K. Baheti, “Task-specific information for imaging system analysis,” JOSA A 24, B25–B41 (2007).
  * [9] T. M. Cover and J. A. Thomas, _Elements of information theory_ (John Wiley & Sons, 2012).
  * [10] R. M. Fano and D. Hawkins, “Transmission of information: A statistical theory of communications,” American Journal of Physics 29, 793–794 (1961).
  * [11] V. Kovalevskij, “The problem of character recognition from the point of view of mathematical statistics,” Character Readers and Pattern Recognition (1967).
  * [12] D. Tebbe and S. Dwyer, “Uncertainty and the probability of error (corresp.),” IEEE Transactions on Information Theory 14, 516–518 (1968).
  * [13] B.-G. Hu and H.-J. Xing, “An optimization approach of deriving bounds between entropy and error from joint distribution: Case study for binary classifications,” Entropy 18, 59 (2016).
  * [14] A. Kolchinsky and B. D. Tracey, “Estimating mixture entropy with pairwise distances,” Entropy 19, 361 (2017).


# PIVOT: Prompting for Video Continual Learning

Andrés Villa1,2, Juan León Alcázar2, Motasem Alfarra2, Kumail Alhamoud2, Julio
Hurtado3,
Fabian Caba Heilbron4, Alvaro Soto1, Bernard Ghanem2
1Pontificia Universidad Católica de Chile, 2King Abdullah University of
Science and Technology (KAUST),
3University of Pisa, 4Adobe Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Modern machine learning pipelines are limited due to data availability,
storage quotas, privacy regulations, and expensive annotation processes. These
constraints make it difficult or impossible to train and update large-scale
models on such dynamic annotated sets. Continual learning directly approaches
this problem, with the ultimate goal of devising methods where a deep neural
network effectively learns relevant patterns for new (unseen) classes, without
significantly altering its performance on previously learned ones. In this
paper, we address the problem of continual learning for video data. We
introduce PIVOT, a novel method that leverages extensive knowledge in pre-
trained models from the image domain, thereby reducing the number of trainable
parameters and the associated forgetting. Unlike previous methods, ours is the
first approach that effectively uses prompting mechanisms for continual
learning without any in-domain pre-training. Our experiments show that PIVOT
improves state-of-the-art methods by a significant 27% on the 20-task
ActivityNet setup.

##  1 Introduction

Modern Deep Neural Networks (DNNs) are at the core of many state-of-the-art
methods in machine vision [21, 44, 38, 20, 46, 10] tasks. To achieve their
remarkable performance, most DNNs rely on large-scale pre-training [26, 46,
11, 36], thereby enabling feature reuse in related downstream tasks [4].
However, adapting and fine-tuning a pre-trained DNN on a novel dataset
commonly leads to catastrophic forgetting [16]. This phenomenon explains how
the effectiveness of the fine-tuned DNNs drastically reduces in the original
training distribution, in favor of increased performance on the downstream
task.

This undesirable property has driven multiple research efforts in the area of
Continual Learning (CL) [29, 7, 8, 25, 18], yielding techniques that enable
fine-tuning a DNN on a sequence of tasks while mitigating the performance drop
along the intermediate steps. One of the most challenging scenarios for the
study of CL is Class Incremental Learning (CIL), where the labels and data are
mutually exclusive between tasks, training data is available only for the
current task, and there are no task identifiers on the validation step (i.e
task boundaries are not available at test time). Such a setup requires
learning one model that, despite the continuous adaptation to novel tasks,
performs well on all the seen classes.

Figure 1: Performance improvement by each PIVOT component. We report the
average accuracy on all tasks under the 10-task CIL on UCF101 and ActivityNet.
We report the performance of basic CLIP, and then gradually equip it with
other components: Spatial Prompting, Memory Buffer, Multi-modal Contrastive
Learning, Temporal Encoder, to finally reach our proposed PIVOT method. The
addition of each proposed component generally boosts the performance on both
datasets. Stars represent the upper bound performance on each benchmark.

Recent advances in mitigating catastrophic forgetting rely on deploying
episodic memory or regularization techniques [37, 8, 25, 2]. Nevertheless,
most of this progress has been directed toward analyzing the catastrophic
forgetting of DNNs in the image domain. The works of [35, 34, 47] introduced
the CIL setup to the video domain, in particular the action recognition task.
Unlike its image counterpart, video CIL requires careful modeling of the
temporal information, making it an even more challenging setup. Despite the
success in small-scale datasets (like UCF101), state-of-the-art methods have
shown limited effectiveness on more challenging video test-beds built upon
larger action taxonomies, such as Kinetics and ActivityNet [47].

Currently, state-of-the-art methods for video CIL rely on temporal masking and
feature distillation to mitigate catastrophic forgetting[34, 47]. In this
paper, we take inspiration from recent advances in large-scale DNNs for zero-
shot image classification[36] and learnable prompts for continual learning
[50], and we propose a novel strategy for CIL in the video domain. We show
that a zero-shot baseline pre-trained in the image domain already outperforms
the best CIL methods in the action recognition task***A similar result for
image domain CIL is outlined in the concurrent work Thengane et al. [45].
Moreover, we show that this baseline can be significantly improved by enabling
temporal reasoning and augmenting the modality encoders with a novel prompting
strategy. As a consequence, our proposed method, PIVOT, outperforms every
other baseline, setting a new state-of-the-art in all the 3 datasets included
in the challenging vCLIMB benchmark for video CIL [47]. Figure 1 summarizes
the performance improvements of our approach.

Notably and following the core ideas of the vCLIMB [47] benchmark, PIVOT does
not rely on any in-distribution pre-training (a common feature of prompting
methods for CL [50, 45]). Rather, it leverages the vast and general visual
knowledge contained in the CLIP visual encoder (trained on massive amounts of
paired static images and text) and maps that knowledge into a feature space
suitable for video understanding in a continual learning setup.

Contributions. This paper proposes PIVOT (PromptIng for Video cOnTinual
learning), a novel strategy for continual learning in the video domains that
leverages large-scale pre-trained networks in the image domain. Our work
brings the following contributions: (i) We show that a multimodal classifier
(Video-Text) mitigates catastrophic forgetting while greatly increasing the
final average CIL accuracy. (ii) We design the first prompt-based strategy for
video CIL. Our approach leverages image pre-training to significantly mitigate
forgetting when learning a sequence of video action recognition tasks. (iii)
We conduct extensive experimental analysis to demonstrate the effectiveness of
PIVOT. Our results show that PIVOT outperforms state-of-the-art methods in the
challenging vCLIMB benchmark by 31%, 27%, and 17.2% in the 20-task setups of
Kinetics, ActivityNet, and UCF101, respectively.

##  2 Related Work

#### Image Continual Learning.

Different approaches to tackle Continual Learning have been proposed over the
last few years. These methods can broadly be classified into three categories:
regularization, memory, and parameter isolation methods [9]. Regularization-
based approaches penalize changes to weights associated with previous tasks.
Among these methods, there is a first sub-group that attempted to minimize the
differences in weights that are relevant to previous tasks. There are several
techniques to determine the relevance of each weight, including the Fisher
matrix [25], the gradient [55, 2], the uncertainty [12], among others [3, 40].
An alternative to previous approaches is to use distillation over previous
task representations while training on the new one [28, 42, 17]. While
regularization methods can mitigate forgetting, they usually under-perform in
the video domain, as their extension to temporal data remains an open research
topic [47].

Memory-based methods store a subset of training samples from past tasks to be
used in the future as replay [8, 14] or regularization [29, 7]. Alternatively,
some methods train generative models to approximate the image distribution of
past tasks. Synthetic images drawn from the generative model are then used as
replay [41, 27, 19, 1]. While showing high effectiveness in the image domain,
these methods are not directly applicable to the video domain for two main
reasons. First, due to the higher memory usage when storing video samples and
high complexity of generating videos. Second, due to the associated challenge
of selecting the key video segments to store in memory [47].

On the other hand, parameter-isolation methods aim to overcome the
distribution shift by learning isolated parameters for each task [22]. While
some of these methods clone the model when a new task arrives [39, 15], others
prefer to use masks [30, 52] or conditioning a trainable knowledge base [13,
23] to select relevant information for the current input. A critical problem
with these approaches is identifying the corresponding weights for each input.
Given that only a sub-group of weights is specialized for a task, the task-id
is needed during training and validation to identify the corresponding group.

Adapting General Visual Knowledge. Instead of continually learning reusable
representations [23, 32], some approaches use fixed large-scale pre-trained
models as knowledge bases [31, 50, 45]. The large amount of data used to train
these models helps them generalize to multiple concepts, making it possible to
apply zero-shot classification for continual learning [45] without fine-
tuning. Other works have augmented these pre-trained models with learnable
prompts [50, 49], extracting better representations, significantly improving
performance, and reducing forgetting in image CIL. However, these methods do
not consider the temporal nature of videos, so they are not directly
applicable. Although some works [33, 54] have explored transferring image
domain knowledge of [45] to video domain, [6] has demonstrated that CLIP pre-
training on image-text pairs is more transferable than a pre-training on noisy
video-text pairs. In this work, we take a first step in leveraging large-scale
pre-trained networks, _i.e_. CLIP, and equip them with temporal reasoning and
learnable prompts to mitigate forgetting in video CIL.

Video Continual Learning. Only a few works have addressed the unique
challenges of video continual learning by proposing video-specific strategies
to mitigate forgetting. In the case of [34], the authors proposed to encode
the temporal information through temporal-based knowledge distillation. A
second work [47] resolves the complexity of storing videos in memory by
employing a temporal consistency loss, which reduces the number of frames
stored. Despite the progress, such methods do not scale well on real-world
action recognition benchmarks such as Kinetics [24] and ActivityNet [5]. In
this paper, we extend the ideas presented in previous works and propose PIVOT,
which exploits an image-language pre-trained model (CLIP) and extends it
temporally. PIVOT enables significant performance gains on both small and
large-scale video CIL benchmarks.

##  3 Prompting for Video Continual Learning

Figure 2:  Outline of the PIVOT Model at inference time. Our model follows 3
main stages to make a prediction in video CIL setup: (1) PIVOT selects the
specific task prompts for a video instance based on its similarity with the
keys $K$ of each task prompt (Blue box). (2) Then, PIVOT adds the selected
spatial and temporal task prompts to the corresponding encoder spatial (CLIP)
or Temporal Encoder (green box). (3) Finally, PIVOT classifies the video
following an MCL, considering the similarity with text embeddings from the
labels of all the available classes computed with the CLIP text encoder (black
box).

Our main objective is to leverage the knowledge available in large-scale
image-language models for video CIL. That is, we want to learn how to adapt
large visual-language DNNs available in the image domain into a single end-to-
end network that is effective in video CIL. Furthermore, we want to make sure
that such a transfer of knowledge involves no fine-tuning of the image
encoder, and requires minimal extensions over the original network. Such
constraints will ultimately favor less forgetting in the CIL setup, as the key
parameters will suffer less drift during fine-tuning on different tasks.

Our approach, PIVOT, is divided into three major components. (i) We adapt a
large-scale image-language pre-trained model, CLIP [36], to extract visual
information from video data. (ii) We aggregate the temporal information
extracted from several frames within a video via a transformer encoder. (iii)
We reinforce the learned knowledge from each task by deploying learnable
tokens, _i.e_. prompts, to prevent the forgetting of previously learned tasks.
Despite its multiple components, we train PIVOT in an end-to-end fashion. We
highlight that this training process is lightweight since the image and text
encoders are frozen. Before we delve into the details of each of our model
components, we give a brief preliminary on the Video Class Incremental
Learning (VCIL) problem and the multi-modal contrastive classifier proposed in
CLIP, as it plays a key role in our approach.

###  3.1 Problem Definition

We follow the recent vCLIMB benchmark [47] training a single model
$f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}$, _i.e_. a DNN parameterized
with $\theta$, that learns to predict an action label $y\in\mathcal{Y}$ given
an input video $x\in\mathcal{X}$. The model learns from a sequence of tasks
$\\{(X_{1},Y_{1}),(X_{2},Y_{2}),\dots,(X_{n},Y_{n})\\}$ where
$X_{i}\subseteq\mathcal{X}$ and $Y_{i}\subseteq\mathcal{Y}\,\,\forall i$. In
the class-incremental setup, we assume that different tasks present the model
with a different set of labels, _i.e_. $Y_{i}\cap Y_{j}=\emptyset\,\,\forall
i\neq j$. We define $Acc_{i}$ as the average classification accuracy of all
the tasks up to and including task $i$. Our objective is to train a single
model $f_{\theta}$ that maximizes the average accuracy $Acc$, which is the
average over all observed tasks.

###  3.2 Multi-modal Contrastive Classifier for Video Continual Learning

Our method builds on top of the image-language pre-trained CLIP encoder. CLIP
[36] demonstrated that textual information promotes the learning of
transferable visual features. PIVOT re-purposes the CLIP encoder for video CIL
by using its multi-modal contrastive classifier as a video feature generator.
We use CLIP’s language encoder to extract text embeddings from the labels of
all the available classes, and its image encoder to extract visual embeddings
for arbitrary frame data. These paired embeddings allow us to perform a
pseudo-classification by looking for tuples that minimize the distance between
visual and textual embeddings. By keeping the CLIP’s encoder frozen and
forwarding video related data (in the visual and language streams), we allow
the transfer of its extensive knowledge into the video domain in a zero-shot
fashion.

We observe three advantages in Multi-modal Contrastive Learning (MCL) for the
continual learning setup. First, this approach is parameter-free. Hence, our
classifier would not suffer from _any_ forgetting when learning different
tasks. This mitigates the need to deploy a large replay buffer to maintain the
performance of the classifier on previously learned tasks. Second, this
approach directly leverages the frozen CLIP encoder in the video CIL setup. We
empirically find that, despite the simplicity of this approach and the domain
gap between image and video data, a zero-shot CLIP significantly outperforms
all continual learning methods in the literature on both Kinetics and
ActivityNet VCIL setups. Third, we find the core idea of the zero-shot
approach (multi-modal contrastive loss) could be used to augment the zero-shot
strategy, thus enabling prompting mechanisms, temporal modeling, and replay
memories that still rely on a frozen CLIP encoder.

###  3.3 PIVOT Model

Prompting methods [50, 49] have emerged as an alternative to traditional
rehearsal methods. While prompting mechanisms have been shown effective in
drastically reducing memory sizes, they rely on in-distribution pre-training
to make suitable prompt selections [50]. We approach a more general case,
where we learn to prompt for video data while relying exclusively on image-
language knowledge bases. To this end, we propose PIVOT, a Prompting-based
Model for Video Continual Learning. PIVOT leverages CLIP $(f_{sp})$ to encode
spatial information and a transformer as temporal encoder $(f_{tp})$ to model
the temporal relations essential to understanding actions. Additionally, we
define a novel spatial and temporal prompting scheme that captures specific
properties of each task to increase the model performance while mitigating the
forgetting of previously learned tasks.

We outline a 3-stage process to effectively train and select prompts for the
spatial and temporal dimensions of videos by leveraging extensive image domain
pre-training. Despite having no video domain pre-training, our prompting
strategy is effective on the action recognition task. To the best of our
knowledge, this is the first prompting method that can work in the presence of
such a domain shift.

Out-of-distribution Adaptation. The first training stage adapts an image pre-
trained model to extract video features. Given an input video
$v\in\mathbb{R}^{T\times H\times W\times C}$, where $T$ is the number of video
frames, we utilize CLIP to encode the video frames into meaningful spatial
features $v_{sp}=f_{sp}(v)\in\mathbb{R}^{T\times D_{m}}$. These spatial
features are then passed through the temporal encoder where we append a
$[class]$ token that aggregates the temporal information of the feature
sequence, resulting in the temporal feature of the video
$v_{tp}=f_{tp}(v_{sp})[class]\in\mathbb{R}^{D_{m}}$. During this process, the
CLIP encoder is completely frozen to avoid any distribution shift. In
contrast, the temporal encoder is trained with the current task data
$(X_{i},Y_{i})$. Specifically, we train the temporal encoder following the MCL
approach outlined in Section 3.2. By using an approach that is informed by the
CLIP frozen features, we ensure that the learned features of the temporal
model stay consistent with the knowledge of the original image-language model.

#### Task-specific Prompt Generation.

After tuning the temporal component of our model, the second training stage
aims at learning task-specific prompts. Current prompting methods [50] would
rely on continuously updating a prompt pool along the full task set. Since we
can not rely on in-distribution pre-training to update and select prompts, we
adopt task-specific prompts that preserve individual task knowledge along the
CL process. Essentially, we train a new set of prompts as the data for a new
task becomes available, then we append these new prompts into the pre-existent
pool.

In addition to maintaining task-specific prompts, we propose to train the
prompts and the prompt selector separately. At training time, we optimize the
task-specific prompts $P_{n}$ for task $n$ following the MCL approach, while
the prompts associated to previous tasks ($P_{n-1}\cup P_{n-2}\cup...P_{0}$)
remain fixed. To address the temporal nature of video data, each task will
estimate independent spatial and temporal prompts
$P_{n}=(P_{n}^{sp},P_{n}^{tp})$, where $P_{n}^{sp}\in\mathbb{R}^{N_{p}\times
L_{sp}\times D_{in}}$ and $P_{n}^{tp}\in\mathbb{R}^{N_{p}\times L_{tp}\times
D_{m}}$, $N_{p}$ is the number of prompts per task, $L_{sp}$ and $L_{tp}$ are
the lengths of spatial and temporal prompts, respectively, and $D_{in}$ is the
output dimension of the CLIP input layer $f^{e}_{sp}$.

Prompt Selection Unlike L2P [49], we can not use extensive in-domain knowledge
to select the best prompts for each input. To alleviate CLIP’s lack of domain
comprehension, and thus improve its selection performance, we train the
temporal encoder to select the prompts. We first associate each task-specific
prompt with the feature set estimated from the frozen CLIP representations. To
this end, we use the feature estimated on the language stream of CLIP, using
the labels as textual input to the stream, this feature set will define the
key $K_{n}$ to index a prompt:
$\\{(K_{1},P_{1}),(K_{2},P_{2}),\dots,(K_{n},P_{n})\\}$, where
$K_{n}\in\mathbb{R}^{M_{n}\times D_{m}}$, $M_{n}$ is the number classes on
task $n$, and $D_{m}$ is the output dimension of CLIP Text encoder.

We train the temporal encoder to select the corresponding task prompts for
each input following a pseudo-classification based on the distance estimated
by the MCL. We select the task prompts with the most similar key to the final
video representation $v_{tp}$. Likewise, once the model selects the prompts,
we train the temporal encoder to support them with an MCL. Figure 2
illustrates the PIVOT pipeline.

###  3.4 PIVOT Forward Pass

In Section 3.3, we outlined the construction and selection of both spatial and
temporal prompts. Next, we discuss how do we use these prompts during
inference.

First, we note that CLIP $f_{sp}$ is based on the ViT architecture, so, it is
composed of input layer $f^{e}_{sp}$ and self-attention layers $f^{s}_{sp}$.
Given an input video $v$, we pass it through $f^{e}_{sp}$, which divides each
frame into $L$ patches and encodes them to
$v_{sp}=f^{e}_{sp}(v)\in\mathbb{R}^{T\times L\times D_{in}}$. We leverage the
spatial prompts at this stage, and concatenate $P_{n}^{sp}$ with $v_{sp}$
resulting in
$v^{e}_{sp}=[P_{n}^{sp};v_{sp}]\in\mathbb{R}^{T\times(N_{p}L_{sp}+L)\times
D_{in}}$ and pass the extended sequence to the self-attention layers
$v^{s}_{sp}=f^{s}_{sp}(v^{e}_{sp})\in\mathbb{R}^{T\times(N_{p}L_{sp}+L)\times
D_{m}}$. Following [50], for the final spatial representation we take an
average pooling (AvgPool) through the first $N_{p}L_{sp}$ tokens of
$v^{s}_{sp}$ resulting in $v^{avg}_{sp}\in\mathbb{R}^{T\times D_{m}}$. It is
important to note that, we use the same spatial prompt for all frames in a
given video.

The previous process results in an embedding vector for each frame that takes
into account both the extracted knowledge from CLIP and the learnt spatial
prompts. We then aim at aggregating the information across all frames by
leveraging a temporal encoder $f_{tp}$. Before we do so, we employ the learnt
temporal prompts in a similar fashion to the spatial ones. In particular, we
first concatenate $x=[P_{n}^{tp};v^{avg}_{sp}]$ and then pass it through the
temporal encoder followed by an average pooling layer. Formally
$v_{tp}=AvgPool(f_{tp}(x)[:N_{p}L_{tp}])\in\mathbb{R}^{D_{m}}$. At last, we
use $v_{tp}$ as a global feature representation for the input video to
classify it with the MCL classifier outlined in Section 3.2. See Section A in
supplementary material for more details.

###  3.5 Training PIVOT

Our training pipeline is divided into three stages for each task in the VCIL
setup. We first train our temporal encoder to correctly classify input data
within the current task. Then, we leverage the knowledge of both CLIP and the
trained temporal encoder to learn per task learnable tokens, _i.e_. prompts.
Finally, we fine tune the temporal encoder to select and leverage the learned
prompts to enhance the performance and reduce forgetting. It is worth
mentioning that we freeze CLIP throughout the entire training process.

First Stage: Out-of-distribution Adaptation. In this stage, we want to align
the temporal encoder with the extracted representation of a given video from
CLIP. To do so, we train our temporal encoder to classify a video based on the
features extracted from CLIP with an MCL loss. Let $\theta_{tp}$ be the
learnable parameters for $f_{tp}$, we seek to minimize the loss for the
$n^{th}$ task:

$\min_{\theta_{tp}}\mathbb{E}_{(v,y)\sim(X_{n},Y_{n})}\left[\mathcal{L}\left(f_{tp}(f_{sp}(v)),y\right)\right],$
(1)

where y is the representation of the class label of v obtained with the CLIP
text encoder, $\mathcal{L}$ is the MCL loss, and $(X_{n},Y_{n})$ is the data
distribution for the $n^{th}$ task. We note here that relying exclusively on
Eq. (1) for training might result in forgetting previously learnt tasks.

Second Stage: Task-specific Prompt Generation. In this stage, we leverage the
knowledge encoded on the temporal encoder to learn meaningful prompts which
are specific for task $n$. To that end, we freeze the temporal encoder during
this stage. Then we randomly initialize a set of prompts for task $n$ denoted
as $P_{n}=(P_{n}^{sp},P_{n}^{tp})$. We include $P_{n}$ in the forward
propagation of our pipeline as discussed in Section 3.4. We then learn the
prompts $P_{n}$ by minimizing the following loss:

$\min_{P_{sp},P_{tp}}\mathbb{E}_{(v,y)\sim(X_{n},Y_{n})}\left[\mathcal{L}\left(f_{tp}(f_{sp}(v,P^{sp}_{n}),P^{tp}_{n}),y\right)\right].$
(2)

Note that Eq. (2) aims at aligning the learnable prompts with the knowledge of
both CLIP and the learnt temporal encoder for the same task. This is
accomplished by the prompts $P_{n}$. We note here that we freeze the prompts
for a given task while learning prompts for other tasks.

Third Stage: Prompt Selection. In the last stage, we fine-tune the temporal
encoder over the replay memory data $M$, $M$ includes data samples for the
current task. This is accomplished through minimizing the combined MCL loss
from Equations 1 and 2:

$\begin{split}\min_{\theta_{tp}}\mathbb{E}_{(v,y)\sim
M}\big{[}\mathcal{L}(f_{tp}(f_{sp}(v,P_{sp}),P_{tp}),y)\\\
+\mathcal{L}(f_{tp}(f_{sp}(v)),y)\big{]}.\end{split}$ (3)

It is worth mentioning that the last training step in Eq. (3) aligns the
parameters of the temporal encoder to solidify the knowledge of a given task.
This is achieved by training over the selected prompts and the network’s
input. This will reduce forgetting previously learnt tasks as their
corresponding prompts are frozen.

##  4 Experiments

Table 1:  Results on the two most challenging datasets of vCLIMB. We follow
the vCLIMB Benchmark [47] and report the average accuracy (Acc) and the
backward forgetting (BWF) at 10 and 20 tasks. We highlight that CLIP zero-shot
[36] outperforms regularized-based methods such as EWC [25] and MAS [2] and
Memory-based methods such as iCaRL [37] and BiC [53]. Our approach PIVOT,
which builds on top of CLIP, outperforms all the previous methods, by up to
$31\%$ in the Kinetics and ActivityNet 20-task sequence.

Model | Kinetics | ActivityNet-Trim
---|---|---
Mem. Video Instances | 10 Tasks | 20 Tasks |  Mem. Video Instances | 10 Tasks | 20 Tasks
Acc $\uparrow$ | BWF $\downarrow$ | Acc $\uparrow$ | BWF $\downarrow$ | Acc $\uparrow$ | BWF $\downarrow$ | Acc $\uparrow$ | BWF $\downarrow$
EWC | None | 5.81% | 16.05% | 2.95% | 32.70% | None | 4.02% | 5.32% | 1.28% | 3.77%
MAS | None | 7.81% | 10.12% | 4.25% | 5.54% | None | 8.11% | 0.18% | 4.61% | 0.1%
BiC | 8000 | 27.90% | 51.96% | 23.06% | 58.97% | 4000 | 51.96% | 24.27% | 46.53% | 15.95%
iCaRL | 8000 | 32.04% | 38.74% | 26.73% | 42.25% | 4000 | 48.53% | 19.72% | 43.33% | 21.57%
CLIP | None | 46.50% | 9.80% | 46.52% | 11.32% | None | 64.1% | 10.47% | 64.13% | 11.71%
PIVOT | 4000 | 55.13% | 26.50% | 55.04% | 26.41% | 2000 | 73.8% | 11.41% | 73.84% | 11.94%
PIVOT w/o prompts | 4000 | 58.61% | 20.78% | 57.51% | 21.98% | 2000 | 72.22% | 13.71% | 73.43% | 13.45%
Upper bound | – | 73.9% | – | 73.9% | – | 2000 | 88% | – | 88% | –

We perform extensive experiments to benchmark existing methods against PIVOT
in the VCIL action recognition task. In addition to comparing PIVOT directly
to the existing methods reported in vCLIMB, we benchmark the performance of
our frozen CLIP baseline. Then, we conduct thorough ablation studies to
understand the role that each component of PIVOT contributes to its final
performance. See our supplementary material for additional experiments.

###  4.1 vCLIMB Benchmark

We follow the vCLIMB benchmark [47] and conduct our experiments on Kinetics
[24], ActivityNet [5], and UCF101 [43] datasets where classes of each dataset
are presented sequentially over 10 or 20 tasks. For example, in the 10-task
schedule of ActivityNet, the 200 classes are broken into 10 sets, each
containing 20 non-overlapping categories. We report the average accuracy (Acc)
of $f_{\theta}$ on all observed tasks ($n$), where higher values indicate
better classification performance. Additionally, we follow the standard
practice in VCIL [47], and report the backward forgetting (BWF), where smaller
values indicate less forgetting and hence are better. Equation 4 shows the
formulas for the metrics, where $A_{i,j}$ is the accuracy of task $j$ after
training task $i$.

$\begin{split}Acc&=\frac{1}{n}\sum_{i=1}^{n}A_{n,i}\\\
BWF&=\frac{1}{n-1}\sum_{i=1}^{n-1}A_{i,i}-A_{n,i}\end{split}$ (4)

#### Implementation details.

PIVOT is augmented with task-specific prompts (see Algorithm 2 in the
supplementary material). We add $N_{p}=1$ set of spatial and temporal prompts
per task with lengths $L_{sp}=3$ and $L_{tp}=3$, respectively. Therefore, at
the end of the CL training process, PIVOT has 10 and 20 sets of spatial and
temporal prompts in the 10-tasks and 20-tasks scenarios, respectively. Hence,
PIVOT incorporates all three stages outlined in 3.3, each stage taking 40
epochs using SGD optimizer with a constant learning rate of 0.01 and a batch
size of 50. The PIVOT temporal encoder is a transformer encoder with 2 heads
and 3 layers. We use the CLIP model for all experiments, employing a ViT-B/32
architecture. $T=8$ frames per video are sampled based on the strategy
proposed in [48]. Moreover, we define "PIVOT without prompts" (see Algorithm 1
in the supplementary material). It uses the very same architecture as PIVOT
but lacks any prompting mechanisms. It is trained for 40 epochs using the same
optimizer, learning rate, and batch size as PIVOT, following the MCL approach
on every CIL task. It’s worth noting that all methods with a temporal encoder
in our ablation study (see Section 4.3) follow the same optimization setup as
PIVOT w/o prompts.

On the contrary, the augmented CLIP model with spatial prompts follows the
prompting approach of [50]. Therefore, it follows a similar setup as [50]: an
Adam optimizer with a learning rate of 0.03, $\beta_{1}=0.9$,
$\beta_{2}=0.999$, and a batch size of 50. For this spatial prompting, we
consider 10 prompts in total, each with a length $L_{sp}=5$, and select 5
prompts per video instance using the same selection approach as [50].

###  4.2 Comparison with Baselines.

We compare the performance of our proposed methods PIVOT and PIVOT w/o
prompts, and our CLIP baseline, against the approaches reported in the recent
video class incremental learning benchmark vCLIMB [47], namely EWC, MAS, BiC,
and iCaRL. PIVOT differs from PIVOT w/o prompts by employing task-specific
prompts while sequentially learning the video tasks. We further report the
upper bound performance that is acquired by training the TSN backbone [48]
offline, i.e., on all the dataset classes simultaneously. The results for
Kinetics and ActivityNet are shown in Table 1, while the results for UCF101
are reported in Table 2. Note that some methods store past-task exemplars to
mitigate forgetting. We set the memory budget for these replay-based methods
proportionally to the amount of classes of the dataset. The number of video
instances used in each setup is stated in the respective table.

SOTA Comparisons to PIVOT. Our results show that PIVOT and PIVOT w/o prompts
outperform all the methods in vCLIMB by large margins. We first highlight that
the highest performing method in vCLIMB was iCaRL, which had an average
accuracy of $26.73\%$ and $43.33\%$ on 20-task Kinetics and 20-task
ActivityNet respectively. The average accuracy of $57.51\%$ obtained by PIVOT
w/o prompts is $31\%$ higher than iCaRL in the Kinetics 20 task set. Moreover,
using PIVOT results in a remarkable improvement of $30\%$ over the best result
reported in vCLIMB for the 20 task set on the ActivityNet dataset. Regardless
of the dataset used, our approach leverages image-pretraining, and a multi-
modal contrastive learning approach to consistently outperform the prior VCIL
art.

Image CLIP is a strong VCIL baseline. Following [45], we also propose to use a
frozen CLIP [36] as a baseline for VCIL. Note that this baseline is trained on
images and evaluated in a zero-shot manner, so it requires no fine-tuning on
the sequence of video tasks. We denote this as CLIP in Table 1. Despite not
being trained on video data, the CLIP baseline significantly outperforms the
memory-based methods iCaRL and BiC reported in the vCLIMB benchmark. The
performance gap is especially large in the 20-task setups, where the accuracy
of CLIP outperforms iCaRL by $20\%$ on both Kinetics and ActivityNet. This
confirms the importance of having models trained on a large volume of data,
even when the domains are different, and the relevance of the MCL that allows
using it in a zero-shot manner.

Table 2:  Results on UCF101. We report the average accuracy (Acc) and the
backward forgetting (BWF) at 10 and 20 tasks. It is important to note that in
this benchmark, the CLIP zero-shot does not outperform the baselines proposed
in [47]. However, our PIVOT model outperforms the previous baselines with
great consistency through the number of tasks.

Model |  Mem. Video Instances | UCF101
---|---|---
10 Tasks | 20 Tasks
Acc $\uparrow$ | BWF $\downarrow$ | Acc $\uparrow$ | BWF $\downarrow$
EWC | None | 9.51% | 98.94% | 4.71% | 92.12%
MAS | None | 10.89% | 11.11% | 5.90% | 5.31%
BiC | 2020 | 78.16% | 18.49% | 70.69% | 24.90%
iCaRL | 2020 | 80.97% | 18.11% | 76.59% | 21.83%
CLIP | None | 65.73% | 11.14% | 65.38% | 12.35%
PIVOT | 1010 | 93.36% | 4.47% | 93.07% | 3.90%
PIVOT w/o prompts | 1010 | 94.80% | 3.89% | 93.70% | 4.77%
Upper bound | – | 94.9% | – | 94.9% | –

The Effect of Long CIL Sequences. Our results in Table 1 show that a frozen
image-pretrained model does not experience extra forgetting when the length of
the CIL sequence increases. Moving from 10 to 20 tasks in Kinetics does not
affect CLIP’s performance as it only changes from $46.5\%$ to $46.52\%$.
Similarly, for ActivityNet, the accuracy values of CLIP barely differ, going
from $64.1\%$ to $64.13\%$. This makes sense as the full clip encoder is
frozen, so presenting the action categories in a shorter or longer sequence
does not change the model’s behavior along the task set.

Likewise, the last two rows of Table 1, show that PIVOT and PIVOT w/o prompts
significantly improve the CLIP baseline performance without resulting in a
performance gap between the 10-task and 20-task sequences. In both Kinetics
and ActivityNet, our models PIVOT and PIVOT w/o prompts are consistent
regardless of the task sequence length. They present at most 1.21% between the
accuracy in the two scenarios, which is much less than the 5.43% presented on
[47].

Results on UCF101. We verify the effectiveness of PIVOT by comparing its
performance to SOTA methods on the UCF101 dataset. While iCaRL and BiC perform
reasonably well in the 10-task CIL scenario on this less challenging dataset,
the final accuracy of frozen CLIP on UCF is around $15,24\%$ behind them,
suggesting that this baseline may not always beat traditional CIL approaches.
Yet, by effectively leveraging the extensive image pre-training through
temporally aware MCL and prompting, PIVOT results in a significant improvement
of $28\%$ in accuracy over the CLIP baseline. Consistent with our results on
Kinetics and ActivityNet, both PIVOT and PIVOT w/o prompts outperform the
methods in vCLIMB with up to $14\%$ improvement over iCaRL, and exhibit a
minimal drop in performance between the 10 task and 20 task scenario.

Comparison with the Fully-supervised Upper Bound. We would like to highlight
from Table 1 and Table 2 that the significant improvements, acquired by using
PIVOT or PIVOT w/o prompts, result in a final CIL performance that is
comparable to the upper bound of training on the whole data offline. While
there is a significant gap to the upper bound performance on Kinetics, the gap
to the upper bound in the ActivityNet 20-task setting is decreased from around
$45\%$ to $15\%$. Moreover, the accuracy of $94.8\%$ obtained by PIVOT w/o
prompts in the 10-task UCF101 benchmark already matches the upper bound
performance of $94.9\%$.

Table 3:  Ablation study results. We analyze the impact of each component of
PIVOT: the memory buffer, the temporal encoder, the Multimodal Contrastive
Loss (MCL), and the prompting approach. For the prompting approach, we compare
PIVOT with an extension of L2P for the video domain using CLIP as the backbone
(Spatial Prompting). As can be observed, every component of our model
contributes significantly to its final performance, which outperforms all the
state-of-the-art approaches.

Method |  Uses Memory | UCF101 | ActivityNet-Trim
---|---|---|---
10 Task | 20 Task | 10 Task | 20 Task
Acc $\uparrow$ | BWF $\downarrow$ | Acc $\uparrow$ | BWF $\downarrow$ | Acc $\uparrow$ | BWF $\downarrow$ | Acc $\uparrow$ | BWF $\downarrow$
CLIP Baseline | ✗ | 65.73% | 11.14% | 65.38% | 12.35% | 64.1% | 10.47% | 64.13% | 11.71%
\+ Spatial Prompting | ✗ | 77.66% | 9.79% | 68.66% | 15.19% | 64.21% | 12.57% | 57.89% | 15.14%
\+ Memory Buffer | ✓ | 88.42% | 11.91% | 87.52% | 11.94% | 63.33% | 26.17% | 62.10% | 28.29%
\+ Classification with MCL | ✓ | 86.17% | 10.65% | 87.59% | 9.05% | 69.53% | 12.44% | 69.32% | 13.51%
\+ Temporal encoder | ✓ | 91.09% | 5.95% | 90.04% | 5.19% | 70.37% | 10.83% | 72.15% | 10.02%
\+ improved prompting (PIVOT) | ✓ | 93.36% | 4.47% | 93.07% | 3.90% | 73.8% | 11.41% | 73.84% | 11.94%
PIVOT w/o prompts | ✓ | 94.80% | 3.89% | 93.70% | 4.77% | 72.22% | 13.71% | 73.43% | 13.45%

###  4.3 Relevance of each component of PIVOT

To demostrate the effectiveness and contribution of each component of PIVOT,
we provide a series of experiments on UCF101 and ActivityNet. We start from
the frozen CLIP baseline and we iteratively add one element of PIVOT to test
its effect. The results are summarized in Table 3.

Effect of spatial prompts. We first add spatial learnable prompts to modulate
the frozen CLIP, and we replace the MCL strategy with a linear classifier. The
goal of these prompts is to guide the prediction of the classifier based on
task-specific spatial information. We see from Table 3 these spatial prompts
are useful in UCF101, resulting in a $12\%$ increase in accuracy in the
10-task scenario. However, spatial prompting alone is not useful in
ActivityNet. This can be explained by the complicated temporal dependencies
that are found in ActivityNet but not in UCF101. We show later that augmenting
these spatial prompts with temporal prompts can increase the performance in
ActivityNet.

Effect of memory replay. We extend the previous method by allowing the model
to store a few video samples for future replay. As an incremental analysis, we
keep using spatial prompts from the previous. This addition results in a
further performance increase of $11\%$ in the 10-task setting of UCF101 and
$19\%$ in the 20-task setting. This model is already ahead of the CLIP
baseline by around $23\%$.

Figure 3: MCL robustness to the memory size. All the experiments were done in
the 10-task scenario of UCF101 considering different memory sizes (1010, 505,
and 202) and two types of classifiers (Linear and MCL) for our PIVOT w/o
prompts model.

Effect of the Multimodal Contrastive Classifier. To address the temporal
variation, we employ the MCL strategy explained in 3.2. Table 3 shows that
this addition results in a $5\%$ increase in final accuracy over the CLIP
baseline in the challenging ActivityNet dataset. Moreover, as observed in
Figure 3, the MCL is more robust to the number of instances in the memory than
the traditional linear classifier.

Effect of the Temporal Modeling. As one of the limitation for using CLIP in
videos context is the lack of temporal modeling in it architecture. To improve
this point in VCIL, we utilize a transformer encoder to model the temporal
information of the input videos. This encoder further mitigates forgetting,
decreasing the BWF from $10.65\%$ to $5.95\%$ in the 10-task UCF101 benchmark.

Effect of the improved Prompts. Given the transformer encoder, we can learn
the temporal prompts. The resulting model is PIVOT, which further improves the
accuracy on ActivityNet by around $3.5\%$. Finally, we show that PIVOT w/o
prompts, which removes the prompting strategy from this model but keeps all
the other components, results in the best performance on UCF101.

##  5 Conclusion and Limitations

We proposed PIVOT, a novel method that leverages large-scale pretraining from
the image domain to mitigate catastrophic forgetting in video class
incremental learning. PIVOT is the first prompting-based approach for video
continual learning. While requiring much less memory than existing baselines,
PIVOT outperforms these baselines by $31\%$, $27\%$, and $17.2\%$ in the
20-task setups of the vCLIMB benchmark. Despite PIVOT’s advantages, we found
it difficult to fine-tune the CLIP encoder for VCIL. PIVOT overcomes this
problem by leveraging learnable prompts and a parameter-free multimodal
contrastive classifier. However, we believe that recent advances in robustly
fine-tuning large pre-trained models under distribution shifts [51] could lead
to even greater improvements for PIVOT.

Acknowledgments. This work was supported by the King Abdullah University of
Science and Technology (KAUST) Office of Sponsored Research (OSR) under Award
No. OSR-CRG2021-4648, as well as the SDAIA-KAUST Center of Excellence in Data
Science and Artificial Intelligence (SDAIA-KAUST AI). Likewise, it was also
partially funded by FONDECYT grant 1221425 and the National Center for
Artificial Intelligence CENIA FB210017, Basal ANID.

## References

  * [1] Motasem Alfarra, Zhipeng Cai, Adel Bibi, Bernard Ghanem, and Matthias Müller.  Simcs: Simulation for online domain-incremental continual segmentation.  arXiv preprint arXiv:2211.16234, 2022.
  * [2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In Eur. Conf. Comput. Vis., pages 139–154, 2018.
  * [3] Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars.  Selfless sequential learning.  In Int. Conf. Learn. Represent., 2019.
  * [4] Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil.  Multi-task feature learning.  Adv. Neural Inform. Process. Syst., 19, 2006.
  * [5] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 961–970, 2015\.
  * [6] Santiago Castro and Fabian Caba.  Fitclip: Refining large-scale pretrained image-text models for zero-shot video understanding tasks.  In Brit. Mach. Vis. Conf., 2022.
  * [7] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-gem.  arXiv preprint arXiv:1812.00420, 2018.
  * [8] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning.  arXiv preprint arXiv:1902.10486, 2019.
  * [9] M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE Trans. Pattern Anal. Mach. Intell., pages 1–1, 2021.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [12] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach.  Uncertainty-guided continual learning in bayesian neural networks.  In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., 2019.
  * [13] Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach.  Adversarial continual learning.  Eur. Conf. Comput. Vis., 2020.
  * [14] Sayna Ebrahimi, Suzanne Petryk, Akash Gokul, William Gan, Joseph E. Gonzalez, Marcus Rohrbach, and trevor darrell.  Remembering for the right reasons: Explanations reduce catastrophic forgetting.  In International Conference on Learning Representations, 2021.
  * [15] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra.  Pathnet: Evolution channels gradient descent in super neural networks.  arXiv preprint arXiv:1701.08734, 2017.
  * [16] Robert M French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135, 1999.
  * [17] Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang.  R-dfcil: Relation-guided representation learning for data-free class incremental learning.  In Eur. Conf. Comput. Vis., 2022.
  * [18] Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem.  Real-time evaluation in online continual learning: A new paradigm.  arXiv preprint arXiv:2302.01047, 2023.
  * [19] Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan.  Remind your neural network to prevent catastrophic forgetting.  In Eur. Conf. Comput. Vis. Springer, 2020.
  * [20] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Int. Conf. Comput. Vis., pages 2961–2969, 2017.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 770–778, 2016\.
  * [22] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao Tao, Dongyan Zhao, Jinwen Ma, and Rui Yan.  Overcoming catastrophic forgetting for continual learning via model adaptation.  In Int. Conf. Learn. Represent., 2019.
  * [23] Julio Hurtado, Alain Raymond, and Alvaro Soto.  Optimizing reusable knowledge for continual learning via metalearning.  Adv. Neural Inform. Process. Syst., 34:14150–14162, 2021.
  * [24] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [25] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [26] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [27] Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat.  Generative models from the perspective of continual learning.  In Int. Joint Conf. on Neural Networks. IEEE, 2019.
  * [28] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2935–2947, 2017\.
  * [29] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  Adv. Neural Inform. Process. Syst., 30, 2017.
  * [30] Arun Mallya, Dillon Davis, and Svetlana Lazebnik.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  Eur. Conf. Comput. Vis., 2018.
  * [31] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell.  An empirical investigation of the role of pre-training in lifelong learning.  arXiv preprint arXiv:2112.09153, 2021.
  * [32] Jorge A Mendez and Eric Eaton.  How to reuse and compose knowledge for a lifetime of tasks: A survey on continual learning and functional composition.  arXiv preprint arXiv:2207.07730, 2022.
  * [33] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li.  St-adapter: Parameter-efficient image-to-video transfer learning.  In Adv. Neural Inform. Process. Syst., 2022.
  * [34] Jaeyoo Park, Minsoo Kang, and Bohyung Han.  Class-incremental learning for action recognition in videos.  In Int. Conf. Comput. Vis., pages 13698–13707, October 2021.
  * [35] Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, and Davide Maltoni.  Latent replay for real-time continual learning.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10203–10209. IEEE, 2020.
  * [36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, Int. Conf. on Mach. Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021.
  * [37] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 2001–2010, 2017\.
  * [38] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Adv. Neural Inform. Process. Syst., 28, 2015.
  * [39] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [40] Gobinda Saha and Kaushik Roy.  Gradient projection memory for continual learning.  In Int. Conf. Learn. Represent., 2021.
  * [41] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  Adv. Neural Inform. Process. Syst., 30, 2017.
  * [42] James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira.  Always be dreaming: A new approach for data-free class-incremental learning.  Int. Conf. Comput. Vis., pages 9374–9384, October 2021.
  * [43] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [44] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In Int. Conf. on Mach. Learning, pages 6105–6114. PMLR, 2019.
  * [45] Vishal Thengane, Salman Khan, Munawar Hayat, and Fahad Khan.  Clip model is an efficient continual learner.  arXiv preprint arXiv:2210.03114, 2022.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Adv. Neural Inform. Process. Syst., 30, 2017.
  * [47] Andrés Villa, Kumail Alhamoud, Victor Escorcia, Fabian Caba, Juan León Alcázar, and Bernard Ghanem.  vclimb: A novel video class incremental learning benchmark.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 19035–19044, June 2022.
  * [48] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks for action recognition in videos.  IEEE Trans. Pattern Anal. Mach. Intell., 41(11):2740–2755, 2019\.
  * [49] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al.  Dualprompt: Complementary prompting for rehearsal-free continual learning.  Eur. Conf. Comput. Vis., 2022.
  * [50] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.  Learning to prompt for continual learning.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 139–149, June 2022\.
  * [51] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [52] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi.  Supermasks in superposition.  Adv. Neural Inform. Process. Syst., 33, 2020.
  * [53] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In IEEE Conf. Comput. Vis. Pattern Recog., June 2019.
  * [54] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  Conf. on Empirical Methods in Natural Language Processing, abs/2109.14084, 2021.
  * [55] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In Int. Conf. on Mach. Learning, 2017.

##  Appendix A Prompt Selector

Figure 4: Outline of Prompt Selector. Our prompt selector leverages the CLIP
$f_{sp}$ and Temporal Encoder $f_{tp}$ as the query function $q=f_{sp}\circ
f_{tp}$ to select the suitable task prompts based on the similarity between
the encoded video $q(\textit{{v}})$ and the keys $K$ of all task prompts.

In Section 3.3, we outlined our key components in PIVOT. The Prompt Selector
module in PIVOT selects the most suitable prompts for classifying a given
input. Here we delve into the details of our Prompt Selector module.

Given a class incremental sequence of $n$ tasks, PIVOT learns a set of $n$
task-specific prompts $P_{i}=\\{(P_{i}^{sp},P_{i}^{tp})\\}_{i=1}^{n}$, where
each prompt $P_{i}$ is a key-value pair $(K_{i},P_{i})$. Note that $K_{i}$
consists of the encoded representations of the $M_{i}$ labels found in task
$i$. These representations are obtained using the CLIP Text encoder. As can be
seen in Figure 4, our prompt selector passes the video v through the CLIP
visual encoder $f_{sp}$ and our temporal encoder $f_{tp}$ to compute a query
$f_{tp}(f_{sp}(\textit{{v}}))$. Following the tokenization procedure of CLIP,
each input representation includes a [class] token. The computation of our
query takes into account the [class] tokens encoded in the representations of
both the visual and temporal encoders. PIVOT selects the corresponding task
prompts for v based on the similarity between the query and all keys.

We present Algorithm 2, which summarizes the prompt selection process of PIVOT
and how it uses the selected prompts. Likewise, Algorithm 1 clarifies the
forward pass of our base PIVOT w/o prompts, which does not require the task-
specific prompts.

Data:

$\textit{{Y}}=(\textit{{y}}_{1},...,\textit{{y}}_{M})$ ;

/* The representations of all learned classes computed with the CLIP Text
Encoder. */

Components:

$f_{sp},f_{tp}$ ;

/* CLIP Model, Temporal Encoder */

$\gamma(.,.),f_{cls}$ ;

/* Cosine distance function, Select the class label whose representation is
most similar to the video */

Forward Pass:

$\textit{{v}}_{tp}=f_{tp}(f_{sp}(\textbf{{v}}))$ where $\textbf{{v}}\in D_{t}$
;

/* Compute the video embedding */

$y=f_{cls}(\textit{{v}}_{tp},\textit{{Y}})$ ;

/* Classify the video */

Algorithm 1 PIVOT w/o prompts Forward Pass

##  Appendix B Number of Trainable Parameters in PIVOT

Considering the implementation details presented in section 4, we analyze and
compare the trainable parameters of our PIVOT model against the baseline
models. Note that the vCLIMB baselines utilize TSN as a backbone and train it
at every task. On the other hand in PIVOT w/o prompts, we leverage the
extensive knowledge of CLIP by freezing its visual and text encoders, so we
only train our temporal encoder. In PIVOT, we further learn task-specific
prompts, which does not result in a significance increase in the number of
parameters. As a result of leveraging the knowledge in CLIP without fine-
tuning it, we substantially reduce the number of total trainable parameters.
Table 4 shows that PIVOT w/o prompts and PIVOT train at most 40.56% of the
parameters that the vCLIMB baselines train. It is worth highlighting that the
task-specific prompts correspond to an increase of 0.40% and 0.80% of PIVOT
w/o prompts parameters in the 10-task and 20-task scenarios, respectively.
Thus, the resulting PIVOT is comparable in the number of parameters to PIVOT
w/o prompts.

Table 4:  The number of trainable parameters. All the vCLIMB Baselines use the TSN with ResNet50 as a backbone, in addition to a linear layer to perform the classification. These models have approximately the same number of parameters. We consider ActivityNet, which have 200 classes in total, to compute the number of parameters of the linear layers. We note that PIVOT w/o prompts and PIVOT results in an order of magnitude reduction in the number of parameters to train in video class incremental learning.  Model | Num. Trainable Parameters
---|---
vCLIMB Baselines | $23.610632\times 10^{6}$
PIVOT (10-task) | $9.496064\times 10^{6}$
PIVOT (20-task) | $9.534464\times 10^{6}$
PIVOT w/o prompts | $9.457664\times 10^{6}$

##  Appendix C Prompt Hyper-parameter Analysis

As observed in Table 5, we explored different configurations for the prompt
length $L$ and number of prompts $N_{p}$ per task on the validation set of the
most challenging dataset we evaluated, ActivityNet. We considered the same $L$
for both spatial and temporal prompts $(L=L_{sp}=L_{tp})$. PIVOT is more
sensitive to the number of prompts per task than their length. It is important
to note that $N_{p}=1$ and $L=3$ for both spatial and temporal prompts work
better for ActivityNet. For simplicity, we use the same setup for the other
datasets we evaluated.

Table 5: Prompt Hyper-parameters. We vary the prompt length $(L)$ and number of prompts per task $(N_{p})$ and report PIVOT performance. To assess $L$, we fix $N_{p}=1$ and use the same $L$ for both spatial and temporal prompts $(L=L_{sp}=L_{tp})$. Likewise, for $N_{p}$, we set $L$ to its optimal value $(L=3)$.  Length of prompts | Acc | Num. Prompts | Acc
---|---|---|---
PIVOT $(L=1)$ | 73.2% |  PIVOT $(N_{p}=1)$ | 73.8%
PIVOT $(L=3)$ | 73.8% |  PIVOT $(N_{p}=2)$ | 72.60%
PIVOT $(L=5)$ | 73.1% |  PIVOT $(N_{p}=4)$ | 70.59%
PIVOT $(L=7)$ | 72.7% |  PIVOT $(N_{p}=6)$ | 69.11%

Data:

$\textit{{Y}}=(\textit{{y}}_{1},...,\textit{{y}}_{M})$ ;

/* The representations of all learned classes computed with the CLIP Text
Encoder */

Components:

$f_{sp},f_{tp}$ ;

/* CLIP Model, Temporal Encoder */

$f_{sp}=f_{sp}^{e}\circ f_{sp}^{s}$’ ;

/* Where $f_{sp}^{e}$ is the input layer and $f_{sp}^{s}$ the self-attention
layers */

$f^{avg}_{sp},f^{avg}_{tp}$ ;

/* Compute an average pooling through the added spatial and temporal prompts
*/

$(K_{i},P_{i})_{i=1}^{n}$ ;

/* Set of prompts for the n tasks */

$K_{i}\in\mathbb{R}^{M_{i}\times D_{m}}$ ;

/* Keys of task $i$, where $M_{i}$ is the number classes of task $i$ */

$P_{i}=(P^{sp}_{i},P^{tp}_{i})$ ;

/* Spatial and Temporal Prompts of task $i$ */

$\gamma(.,.),f_{cls}$ ;

/* Cosine distance function, Select the class label whose representation is
most similar to the video */

Forward Pass:

$\textit{{v}}_{tp}=f_{tp}(f_{sp}(\textbf{{v}}))$ where $\textbf{{v}}\in D_{t}$
;

/* Compute the Query */

$P_{k}=f_{min}(\gamma(\textit{{v}}_{tp},K))$ ;

/* Select the task prompt that is closet the Query */

$\textit{{v}}^{e}_{sp}=[P_{k}^{sp};f^{e}_{sp}(\textbf{{v}})]$ ;

/* Add the spatial prompt */

$\textit{{v}}^{avg}_{sp}=f^{avg}_{sp}(f^{s}_{sp}(\textit{{v}}^{e}_{sp}))$ ;

/* Compute the representation of the frames per video */

$\textit{{v}}_{tp}=f_{tp}^{avg}(f_{tp}([P^{tp}_{k},\textit{{v}}^{avg}_{sp}]))$
;

/* Add the temporal prompt and Compute the final video embedding */

$y=f_{cls}(\textit{{v}}_{tp},\textit{{Y}})$ ;

/* Classify the video */

Algorithm 2 PIVOT Forward Pass