# Neural Network Layers for Prediction of Positive Definite Elastic Stiffness
Tensors

Charles F. Jekel111This document was prepared as an account of work sponsored
by an agency of the United States government. Neither the United States
government nor Lawrence Livermore National Security, LLC, nor any of their
employees makes any warranty, expressed or implied, or assumes any legal
liability or responsibility for the accuracy, completeness, or usefulness of
any information, apparatus, product, or process disclosed, or represents that
its use would not infringe privately owned rights. Reference herein to any
specific commercial product, process, or service by trade name, trademark,
manufacturer, or otherwise does not necessarily constitute or imply its
endorsement, recommendation, or favoring by the United States government or
Lawrence Livermore National Security, LLC. The views and opinions of authors
expressed herein do not necessarily state or reflect those of the United
States government or Lawrence Livermore National Security, LLC, and shall not
be used for advertising or product endorsement purposes. LLNL-JRNL-832991
<EMAIL_ADDRESS>Kenneth E. Swartz  Daniel A. White  Daniel A. Tortorelli  Seth
E. Watts

###### Abstract

Machine learning models can be used to predict physical quantities like
homogenized elasticity stiffness tensors, which must always be symmetric
positive definite (SPD) based on conservation arguments. Two datasets of
homogenized elasticity tensors of lattice materials are presented as examples,
where it is desired to obtain models that map unit cell geometric and material
parameters to their homogenized stiffness. Fitting a model to SPD data does
not guarantee the model’s predictions will remain SPD. Existing Cholsesky
factorization and Eigendecomposition schemes are abstracted in this work as
transformation layers which enforce the SPD condition. These layers can be
included in many popular machine learning models to enforce SPD behavior. This
work investigates the effects that different positivity functions have on the
layers and how their inclusion affects model accuracy. Commonly used models
are considered, including polynomials, radial basis functions, and neural
networks. Ultimately it is shown that a single SPD layer improves the model’s
average prediction accuracy.

###### keywords:

Neural networks , Symmetric positive definite , Cholesky decomposition ,
Eigendecomposition

††journal: CMAME

[1]organization=Lawrence Livermore National Laboratory, addressline=L-227, PO
Box 808, postcode=94551, city=Livermore, CA, country=USA

[2]organization=Department of Mechanical Sciences and Engineering,
addressline=University of Illinois, city=Urbana, IL, postcode=61801,
country=USA

##  1 Introduction

The use of multiscale structures has become a popular choice to generate
superior designs which can now be manufactured using 3D printing techniques
[1, 2, 3, 4]. These structures are typically designed from unit cells arrays,
i.e. lattices. To optimize these structures, various surrogate models are
often used to predict the unit cell’s homogenized elastic response from their
geometric and material properties. Most of these surrogate models do not
explicitly consider whether the predicted homogenized elastic tensors are
physically admissible, i.e. symmetric positive definite (SPD). This work
addresses these concerns and investigates what is needed to ensure the
surrogate produces SPD elasticity tensors.

Topology optimization distributes a given amount of material over a design
domain to maximize a structure’s performance subject to constraints (e.g. on
maximum stress, compliance). The topology optimization problem is inherently
ill-posed, but restrictions or relaxation methods can be used to formulate
well-posed problems.

Density based topology optimization [5, 6, 7] use restriction whereby the
material volume fraction is driven to either material or near zero in regions
with a compliant ersatz material that models voids. Fine-scale oscillations in
the volume fraction are prevented by imposing perimeter constraints, slope
constraints, or filtration. Homogenization based methods use relaxation
whereby the fine scale oscillations are used to define microstructures that
vary throughout the structure [8]. To accommodate this the volume fraction
field is replaced by a material field that allows for all possible composite
materials that can be generated from mixtures of solid and “void” materials.
As such the volume fraction varies continuously throughout the structure. This
method is impractical because the optimized microstructures cannot be
fabricated. However, we can make this approach practical by limiting the
morphology of the microstructure, for example by using lattice
microstructures.

For example White et al. [9] designed a bridge at the macroscale level, which
consisted of a lattice of Anisotruss microscale unit cells. A neural network
was used to predict the unit cell’s homogenized elastic response as a function
of four unit cell geometric parameters. White et al. [9] only considered a
single unit-cell topology, whereas Sanders et al. [1] presented a similar
formulation that incorporates multiple unit cell architectures.

Various machine learning or surrogate models have been used to predict the
homogenized elastic response of unit cells as functions of their geometric and
material parameters [9, 10, 11, 12]. For linear elastic material models, the
anisotropic homogenized response reduces (in Voigt notation) to a $6\times 6$
SPD matrix. The database to produce these models is obtained from the
homogenization of unit cells via highly resolved continuum finite element
simulations [13]. In a similar study Watts et al. [10] used polynomial based
surrogate models to predict the homogenized elastic responses of the Isotruss
[14], the Octet truss, and the ORC truss unit cells. Neural networks have also
been used to generate these models [9]. Analogously, Xiao et al. [3] used a
Kriging surrogate model to predict homogenized lattice cell properties in
order to design graded lattice sandwich structures. None of these studies
investigate whether the surrogate model predict SPD tensors. This is important
because indefinite tensors indicate unstable materials which are nonphysical
for our applications.

While our focus is on elasticity, many other physics applications require SPD
matrices (e.g., heat transfer, dynamics). Amsallem et al. [15] developed
surrogate models for mass, damping, and stiffness matrices via interpolation
based on an exponential manifold to preserve SPD qualities. This method
guarantees SPD output by 1) taking the Eigendecomposition of a prediction, 2)
applying the positive valued exponential function to the eigenvalues, and 3)
reassembling the matrix with the new positive eigenvalues and the existing
eigenvectors. Thus, the predicted matrices are SPD by construction. In a
somewhat related approach, Xu et al. [16] developed SPD neural networks that
also always output SPD tensors. Their approach cleverly takes advantage of the
$LL^{\text{T}}$ Cholesky factorization, where predictions are 1) generated in
lower triangular form $L$, 2) the diagonal elements of $L$ are mapped to the
positive reals, and hence the 3) outputted $LL^{\text{T}}$ is SPD by
construction. However, neither [15] nor [16] consider whether these SPD
enforcement methods sacrifice model accuracy. Additionally, they do not
demonstrate what happens to model predictions if no SPD enforcement method is
used.

Other methods have also been used to create SPD models. The simplest approach
may be to consider linear interpolation between two SPD matrices, which can be
easily proven to result in a SPD matrix. However, linear interpolation is not
able to produce SPD matrices via extrapolation [17]. Linear interpolation is
also $C^{0}$ continuous, which may produce undesirable non-continuous
derivatives for multi-scale topology optimization. Amos et al. [18] presented
input convex neural networks which add constraints on the network parameters
to ensure that the output is a convex function. Klein et al. [19] used input
convex neural networks to model a hyperelastic material. Constraints could be
constructed using SPD metrics defined in [20] to restrict neural network
parameters to always output SPD matrices. However, unlike [15] and [16], such
constrained methods do not guarantee that outputs are SPD for all possible
inputs (unit cell geometries and material properties in our application) and
network parameters.

The primary contribution of this work is to compare SPD enforcement methods
with commonly used surrogate models [21]. The specific focus only considers
data points that produce SPD matrices (from specific physics applications).
The paper briefly describes the creation of the SPD datasets. It is then shown
that simply fitting a model to the SPD data does not produce SPD output. To
remedy this, two SPD preservation methods are presented. These methods are
implemented as transformation layers or functions that have no learnable
parameters, hence they are easily incorporated into any surrogate model. We
compare the two SPD generating surrogate models and investigate their affects
on model accuracy.

##  2 Elastic response of Isotruss dataset

The elastic stiffness tensor $\mathbb{C}$ maps infinitesimal strain
$\bm{\epsilon}$ to stress $\bm{\sigma}$ as
$\bm{\sigma}=\mathbb{C}\bm{\epsilon}$. Using Voigt notation this mapping is
expressed in matrix from as

$\begin{bmatrix}\sigma_{11}\\\ \sigma_{22}\\\ \sigma_{33}\\\ \sigma_{23}\\\
\sigma_{13}\\\ \sigma_{12}\\\
\end{bmatrix}=\begin{bmatrix}C_{11}&C_{12}&C_{13}&C_{14}&C_{15}&C_{16}\\\
&C_{22}&C_{23}&C_{24}&C_{25}&C_{26}\\\ &&C_{33}&C_{34}&C_{35}&C_{36}\\\
&&&C_{44}&C_{45}&C_{46}\\\ &\textit{symm}&&&C_{55}&C_{56}\\\ &&&&&C_{66}\\\
\end{bmatrix}\begin{bmatrix}\epsilon_{11}\\\ \epsilon_{22}\\\ \epsilon_{33}\\\
2\epsilon_{23}\\\ 2\epsilon_{13}\\\ 2\epsilon_{12}\\\ \end{bmatrix}.$ (1)

We henceforth treat $\bm{C}$ as a symmetric matrix222Symmetry ensures angular
momentum is balanced. with 21 independent components. Depending upon material
symmetry, the number of independent components can be reduced. For example, an
orthotropic material is simplified to

$\begin{bmatrix}C_{11}&C_{12}&C_{13}&&&\\\ &C_{22}&C_{23}&&&\\\ &&C_{33}&&&\\\
&&&C_{44}&&\\\ &\textit{symm}&&&C_{55}&\\\ &&&&&C_{66}\\\ \end{bmatrix}$ (2)

with just 9 independent components. In all cases we assume $\bm{C}$ is SPD so
that the strain energy $\bm{\epsilon}^{\text{T}}\bm{C}\bm{\epsilon}$ is
positive [20].

To exemplify our methodology, we develop surrogate models of the homogenized
response of the orthotropic Isotruss unit-cell [14]. Two variants of the
Isotruss are investigated, one with solid truss rods, and another with hollow
rods. The constitutive materials in both unit cells are uniform and isotropic.
A homogenization procedure following [13] was used to determine the
orthotropic homogenized $\bm{C}$ matrices from different rod geometries and
constitutive Poisson’s ratios. The homogenization procedure always produces
SPD matrices. Datasets were constructed with the rod geometry and Poisson’s
ratio as input and the homogenized orthotropic elasticity matrix as output.

A 2D dataset was generated for solid rod Isotruss structures. The first input
variable was the outer rod radius, which was sampled over $[0.001a,0.25a)$,
where $a$ is the unit cell size. Thus a 0.25 radius would have a physical
length of $0.25a$. The second input was the Poisson’s ratio, which was sampled
over $[0,0.5)$. One hundred random Latin hypercube samples were performed for
the two variables [22]. The output for each pair of inputs is a $6\times 6$
SPD orthotropic homogenized elasticity matrix $\bm{C}$.

In a similar manner, a 3D dataset was generated for hollow rod Isotruss
structures. The three inputs are the inner radius to outer radius ratio, the
outer radius, and the Poisson’s ratio. The sampling intervals were
$[0.01,0.9)$, $[0.01a,0.25a)$, and $[0,0.5)$ respectively. One thousand random
Latin hypercube samples were performed. Like with the 2D dataset, the output
is a $6\times 6$ SPD orthotropic homogenized elasticity matrix $\bm{C}$.

Models were fit to the 2D dataset and then evaluated to understand whether the
models were SPD on the domain. The models learn the 9 independent components
of $\bm{C}$, thus will always produce a symmetric matrix. However, it is
unclear if the models will always produce SPD matrices. We implement four
surrogate models:

  1. 1.

single layer 100 neuron model with radial basis activation functions from [9]
denoted as NN,

  2. 2.

quadratic polynomial response surface,

  3. 3.

quartic polynomial response surface,

  4. 4.

and linear kernel radial basis function (RBF) [23].

Each of the four models were fit 1,080 times to a random fraction of 80% of
the data, with the intention to asses the probability of producing an
indefinite $\bm{C}$ matrix at some portion in the domain.

The trained models were then evaluated on a $10\times 10$ factorial grid
indeed. All of the 1,0080 quadratic and quartic models had at least one grid
point that produced a symmetric indefinite $\bm{C}$ matrix. Only one of the
1,080 neural network models and three of the 1,080 RBF models produced SPD
$\bm{C}$ matrices on the entire $10\times 10$ grid. The percent of the
indefinite predictions for each grid point from the 1,080 models is shown in
Figure 1. The polynomials and RBF had a single grid point that produce
indefinite matrices nearly 100% of the time. It is clear that these surrogate
models do not produce SPD matrices.

Figure 1: The percent (indicated by the dot color) of producing an indefinite
$\bm{C}$ at the factorial points. The 100 data points in the 2D dataset are
marked with xes.

##  3 SPD transformations layers

Let $\bm{\theta}$ represent the inputs to an arbitrary machine learning model,
and let $\bm{\phi}$ represent the learnable parameters of such model. The goal
of this work is to predict $\bm{C}(\bm{\theta},\bm{\phi})$ where $\bm{C}$ is
SPD for all $\bm{\theta}$ and $\bm{\phi}$. Borrowing the layer abstraction
from neural networks, we propose SPD transformation layers to ensure SPD
predictions. These layers just transform a vector input to a SPD matrix. In
the following we describe the Cholesky factorization method that is based on
[16], and the Eigendecomposition that is inspired by [15]. While [15, 16]
required transformations of the output data prior to training, models with our
SPD layers can be applied to the original data without such transformations.
This simplifies the error evaluation during training, and the inference in
production.

Our SPD layers are mappings of a matrix (or second-order tensor) of form
$\bm{X}_{bm}$ to a 3-D matrix (or third-order tensor) $\bm{C}_{bnn}$. The
batch size $b$ denotes the number of samples to be transformed at a time. The
relationship between number of independent matrix components $m$ and the
$n\times n$ output SPD matrix dimension is expressed as the sum of the first
$n$ natural numbers

$m=\frac{n(n+1)}{2}.$ (3)

Our work focuses on producing $6\times 6$ SPD matrices with $n=6$ and $m=21$,
but these layers may also be used to produce larger SPD matrices.

###  3.1 Cholesky decomposition

The Cholesky layer uses the Cholesky decomposition to enforce SPD output. The
Cholesky decomposition of a real symmetric matrix $\bm{C}$ is expressed as

$\bm{C}=\bm{L}\bm{L}^{\text{T}}.$ (4)

If the lower triangular matrix $\bm{L}$ has real values and positive diagonal
values ($L_{ii}>0$), then $\bm{L}\bm{L}^{\text{T}}$ produces an SPD matrix. We
use this observation from [16] to construct an SPD layer that produces SPD
outputs for any real input.

The Cholesky factorization layer takes in matrix $\bm{X}$ of size $*m$ and
outputs a matrix $\bm{C}$ of size $*nn$, where $*$ represents an arbitrary
batch size. The first step in the layer is to transform each row $\bm{X}_{*}$
of the input $\bm{X}$ to a lower triangle matrix $\bm{L}$ of shape $*nn$ as

$\bm{X}_{*}=[x_{1},x_{2},\cdots,x_{m}]\to\bm{L}_{*}=\begin{bmatrix}L_{11}&&&\\\
L_{21}&L_{22}&&\\\ \vdots&\vdots&\ddots&\\\ L_{n1}&L_{n2}&\cdots&L_{nn}\\\
\end{bmatrix}$ (5)

where notation of $\bm{X}_{*}$ and $\bm{L}_{*}$ indicates that we are looking
at a single sample from an arbitrary sized batch.

The lower triangle matrix is populated as

$\begin{bmatrix}L_{11}&&&\\\ L_{21}&L_{22}&&\\\ \vdots&\vdots&\ddots&\\\
L_{n1}&L_{n2}&\cdots&L_{nn}\\\ \end{bmatrix}=\begin{bmatrix}p(x_{1})&&&&\\\
x_{2}&p(x_{3})&&&\\\ x_{4}&x_{5}&p(x_{6})&&\\\ \vdots&\vdots&\vdots&\ddots&\\\
\cdots&\cdots&\cdots&\cdots&p(x_{m})\\\ \end{bmatrix}$ (6)

where $p:\mathbb{R}\to\mathbb{R}^{+}$ represents any strictly positive valued
ordinary function. Such positive enforcing functions can be based on the
exponential, square, ReLU, etc. Then, with matrix multiplication of the
transpose, the SPD matrix is populated as

$\begin{bmatrix}L_{11}&&&\\\ L_{21}&L_{22}&&\\\ \vdots&\vdots&\ddots&\\\
L_{n1}&L_{n2}&\cdots&L_{nn}\\\
\end{bmatrix}\begin{bmatrix}L_{11}&L_{21}&\cdots&L_{n1}\\\
&L_{22}&\cdots&L_{n2}\\\ &&\ddots&\vdots\\\ &&&L_{nn}\\\
\end{bmatrix}=\begin{bmatrix}C_{11}&C_{12}&\cdots&C_{1n}\\\
&C_{22}&\vdots&C_{2n}\\\ \text{Symm}&&\ddots&\vdots\\\ &&&C_{nn}\\\
\end{bmatrix}$ (7)

which is expressed in batch form as

$\bm{C}_{*}=\bm{L}_{*}\bm{L}_{*}^{\text{T}}.$ (8)

Notably $\bm{C}_{*}$ is SPD by construction.

###  3.2 Eigendecomposition

The Eigendecomposition layer enforces SPD by ensuring all predictions have
positive eigenvalues. Let $\bm{A}$ be a real square symmetric matrix. If
$\bm{A}$ has all positive eigenvalues, then $\bm{A}$ is SPD [20]. This
principle is used to create a transformation layer that takes in real values
and outputs SPD matrices.

Like the Cholesky factorization, the eigenvalue decomposition layer first
takes in a size of $*m$ and outputs a size of $*nn$. The first step is to
transform each row of the input into a symmetric and square matrix
$\bm{A}_{*}$. For example the following mapping

$\bm{X}_{*}=[x_{1},x_{2},\cdots,x_{m}]\to\bm{A}_{*}=\begin{bmatrix}A_{11}&&&&\\\
A_{21}&A_{22}&&\textit{symm}&\\\ A_{31}&A_{23}&A_{33}&&\\\
\vdots&\vdots&\vdots&\ddots&\\\ A_{n1}&A_{n2}&A_{n3}&\cdots&A_{nn}\\\
\end{bmatrix}$ (9)

is obtained for for each candidate $\bm{X}_{*}$ in the batch as

$\bm{A}_{*}=\begin{bmatrix}x_{1}&&&&\\\ x_{2}&x_{3}&&\textit{symm}&\\\
x_{4}&x_{5}&x_{6}&&\\\ \vdots&\vdots&\vdots&\ddots&\\\
\cdots&\cdots&\cdots&\cdots&x_{m}\\\ \end{bmatrix}.$ (10)

Then, an eigenvalue decomposition is performed on $\bm{A}_{*}$ such that

$\bm{A}_{*}=\bm{Q}_{*}\bm{\Lambda}_{*}\bm{Q}_{*}^{-1}$ (11)

where the columns of $\bm{Q}$ are the normalized eigenvectors and the
$\Lambda_{ii}$ of the diagonal $\bm{\Lambda}$ are the corresponding
eigenvalues. A positive function $p:\mathbb{R}\to\mathbb{R}^{+}$ (like the
exponential used in [15]) is applied to the eigenvalues

$p(\bm{\Lambda}_{*})=\begin{bmatrix}p(\lambda_{11})&&&\\\
&p(\lambda_{22})&&\\\ &&\ddots&\\\ &&&p(\lambda_{nn})\\\ \end{bmatrix}$ (12)

to ensure that each eigenvalue is strictly positive. Finally, the output
matrix $\bm{C}_{*}$ is assembled as

$\bm{C}_{*}=\bm{Q}_{*}p(\bm{\Lambda}_{*})\bm{Q}_{*}^{-1}$ (13)

which is SPD by construction.

##  4 Experiments with positive enforcing functions

This section investigates the performance of different positive enforcing
functions $p:\mathbb{R}\to\mathbb{R}^{+}$ for a given neural network model.
Specifically we use the same single layer, 100 neuron model with radial basis
activation functions from [9], and add our SPD layer to the final prediction.
Each model outputs the $6\times 6$ orthotropic matrix (with $m=9$), learning
only the non-zero stiffness values in $\bm{C}$. Mean squared error was used as
the loss function on the entire $6\times 6$ matrix prediction. These models
were implemented in PyTorch333Our implementation of the PyTorch SPD layers as
a Python package is available online at https://github.com/LLNL/spdlayers and
take advantage of automatic differentiation [24].

The positive functions $p:\mathbb{R}\to\mathbb{R}^{+}$ investigated are
denoted in Table 1. A small $\epsilon=$1e-8 was added to each function to
ensure positive values for all $x$. The functions are monotonically increasing
for $x>0$ and satisfy either $\lim_{x\to\infty}f(x)=\lim_{x\to-\infty}f(x)$ or
$\lim_{x\to-\infty}f(x)=0$. Derivatives do not exist at $x=0$ for the absolute
value and ReLU functions, but do exist for the other four functions. We also
note that $\lim_{x\to\infty}f(x)=x$ for the Abs, Softplus and ReLU functions.

Name | $p(x)=$
---|---
Abs | $|x|+\epsilon$
Square | $x^{2}+\epsilon$
Softplus | $\log(1+\exp(x))+\epsilon$
ReLU | $\max(0,x)+\epsilon$
Quartic | $x^{4}+\epsilon$
Exp | $\exp(x)+\epsilon$
Table 1: Strictly positive functions used to map $x\in\mathbb{R}\to
p(x)\in\mathbb{R}^{+}$. A small value of $\epsilon=1$e-8 was used to ensure
$p(x)>0$ for all $x\in\mathbb{R}$. Note that $\epsilon$ is not needed for the
Exponential or Softplus functions, but was included in our experiment for
consistency.

A potentially desirable property of the ReLU function is that $p(x)=x$ when
$x>0$ resulting in minimal modification to eigenvalues through the SPD layer.
Unfortunately, the ReLU function is not differentiable at $x=0$ and has a zero
derivative at $x<0$ which can make training a model more difficult than other
positivity functions. The function that is most similar to ReLU is the
Softplus function, which also has the benefit of being continuously
differentiable. The Exponential function is in a similar category as the ReLU
and Softplus, but will significantly alter the magnitude of eigenvalues
through the SPD layer. While the ReLU, Softplus, and Exponential function
preserve the ordering of eigenvalues through the SPD layer, the remaining
positivity functions may not when $x<0$. Our numerical results do not
necessarily indicate this is a problem our application, however it may become
problematic in other machine learning applications.

The numerical experiment repeatedly trains the same model architecture with
Cholesky or Eigendecomposition to study the different positive functions. Each
model positive function combination was trained with a different set of random
initial weights. Additionally, for each training 20% of the test data and 80%
of the training data were randomly assigned. The training process was repeated
1,080 times with different randomly assigned test and training data to asses
their statistical performance. The results report the average ($\mu$) and
standard deviation ($\sigma$) of the mean squared error computed over the
testing data. Additionally, a non-parametric tolerance interval is used to
report the 95th percentile test error to 90% confidence [25]. The conservative
tolerance interval roughly represents the worst-case test error achieved by
the method. This initial experiment used the (3D input) hollow truss data and
all models were trained using double precision.

Two different optimization algorithms were considered. First, the
deterministic gradient-based L-BFGS algorithm (implementation inspired by
[26]) was used with 100 epochs. The results with L-BFGS are presented in Table
2. Additionally, the stochastic Adam algorithm described in [27] was used with
10,000 epochs and fixed learning rates of 1e-3, 3e-4, and 1e-4. The Adam
results are presented in Tables 3-5. It is obvious that the ReLU function
performs the worst on this dataset, having the highest average test errors and
95th percentiles. The other results are more difficult to interpret.

The L-BFGS Eigendecomposition combination seemed to produce more accurate
models than the L-BFGS Cholesky combination. In fact, the lowest test errors
(both on average and worst-case) occurred with the L-BFGS Eigendecomposition
combination with the absolute value, square, or quartic positive functions.

|  | Test errors
---|---|---
SPD layer | Positivity function $p()$ | $\mu$ | $\sigma$ | $P=0.95$
Chol. Fac. | Abs | 0.00069 | 0.00182 | 0.00564
Chol. Fac. | Square | 0.00135 | 0.00233 | 0.00628
Chol. Fac. | Softplus | 0.00159 | 0.00066 | 0.00244
Chol. Fac. | ReLU | 0.00381 | 0.00228 | 0.00753
Chol. Fac. | Quartic | 0.00222 | 0.00265 | 0.00708
Chol. Fac. | Exp | 0.00140 | 0.00066 | 0.00237
Eig. Decom. | Abs | 0.00003 | 0.00005 | 0.00013
Eig. Decom. | Square | 0.00002 | 0.00004 | 0.00008
Eig. Decom. | Softplus | 0.00080 | 0.00035 | 0.00113
Eig. Decom. | ReLU | 0.00202 | 0.00119 | 0.00322
Eig. Decom. | Quartic | 0.00002 | 0.00003 | 0.00009
Eig. Decom. | Exp | 0.00077 | 0.00052 | 0.00203
Table 2: Comparison of positivity functions with L-BFGS to 100 training
epochs.

The Adam Cholesky combination exhibited lower errors than the L-BFGS Cholesky
combination. The best observed positive functions with the former were the
absolute value, Softplus, and exponential. On the other hand, only the quartic
positive function performed much better with the Adam Eigendecomposition
combination than the Adam Cholesky combination. Although it may not perform
the best in all scenarios, Adam with a learning rate of 1e-3 produces the
lowest average and worst-case errors for the data we investigated.

|  | Test errors
---|---|---
SPD layer | Positivity function $p()$ | $\mu$ | $\sigma$ | $P=0.95$
Chol. Fac. | Abs | 0.00008 | 0.00043 | 0.00016
Chol. Fac. | Square | 0.00049 | 0.00148 | 0.00513
Chol. Fac. | Softplus | 0.00008 | 0.00049 | 0.00012
Chol. Fac. | ReLU | 0.00417 | 0.00230 | 0.00795
Chol. Fac. | Quartic | 0.00152 | 0.00240 | 0.00651
Chol. Fac. | Exp | 0.00004 | 0.00007 | 0.00014
Eig. Decom. | Abs | 0.00009 | 0.00008 | 0.00026
Eig. Decom. | Square | 0.00005 | 0.00005 | 0.00017
Eig. Decom. | Softplus | 0.00009 | 0.00026 | 0.00096
Eig. Decom. | ReLU | 0.00491 | 0.00449 | 0.01536
Eig. Decom. | Quartic | 0.00005 | 0.00005 | 0.00017
Eig. Decom. | Exp | 0.00004 | 0.00012 | 0.00010
Table 3: Comparison of positivity functions with ADAM to 10,000 training epochs and a learning rate of 1e-3. |  | Test errors
---|---|---
SPD layer | Positivity function $p()$ | $\mu$ | $\sigma$ | $P=0.95$
Chol. Fac. | Abs | 0.00027 | 0.00104 | 0.00307
Chol. Fac. | Square | 0.00066 | 0.00172 | 0.00560
Chol. Fac. | Softplus | 0.00011 | 0.00052 | 0.00022
Chol. Fac. | ReLU | 0.00358 | 0.00232 | 0.00746
Chol. Fac. | Quartic | 0.00128 | 0.00227 | 0.00644
Chol. Fac. | Exp | 0.00010 | 0.00024 | 0.00045
Eig. Decom. | Abs | 0.00012 | 0.00013 | 0.00038
Eig. Decom. | Square | 0.00007 | 0.00007 | 0.00021
Eig. Decom. | Softplus | 0.00016 | 0.00034 | 0.00118
Eig. Decom. | ReLU | 0.00211 | 0.00130 | 0.00347
Eig. Decom. | Quartic | 0.00006 | 0.00008 | 0.00020
Eig. Decom. | Exp | 0.00008 | 0.00019 | 0.00036
Table 4: Comparison of positivity functions with Adam to 10,000 training epochs and a learning rate of 3e-4. |  | Test errors
---|---|---
SPD layer | Positivity function $p()$ | $\mu$ | $\sigma$ | $P=0.95$
Chol. Fac. | Abs | 0.00066 | 0.00167 | 0.00552
Chol. Fac. | Square | 0.00048 | 0.00137 | 0.00504
Chol. Fac. | Softplus | 0.00014 | 0.00029 | 0.00091
Chol. Fac. | ReLU | 0.00333 | 0.00249 | 0.00767
Chol. Fac. | Quartic | 0.00092 | 0.00195 | 0.00626
Chol. Fac. | Exp | 0.00029 | 0.00066 | 0.00205
Eig. Decom. | Abs | 0.00028 | 0.00038 | 0.00127
Eig. Decom. | Square | 0.00013 | 0.00017 | 0.00049
Eig. Decom. | Softplus | 0.00035 | 0.00056 | 0.00158
Eig. Decom. | ReLU | 0.00180 | 0.00075 | 0.00327
Eig. Decom. | Quartic | 0.00013 | 0.00032 | 0.00046
Eig. Decom. | Exp | 0.00028 | 0.00066 | 0.00196
Table 5: Comparison of positivity functions with ADAM to 10,000 training
epochs and a learning rate of 1e-4.

##  5 Experiments with SPD layers on common surrogates

Experiments were performed to evaluate whether the inclusion of an SPD layer
had detrimental effects to the model accuracy. The models from Section 2 are
revisited to now include the Cholesky factorization layer, Eigendecomposition
layer, and again evaluated without an SPD layer. The weights are selected by
minimizing the mean squared loss, using Adam with a learning rate of 1e-3 for
10,000 epochs.

Different positive functions were selected for each SPD layer. The Cholesky
factorization layer used the Softplus transformation and the Eigenvalue
Decomposition layer used the Square transformation function. We believe these
choices produced reasonable (but not optimal) models based on our previous
experiment. Indeed, the previous experiment demonstrated that the best
positive function is dependent on the optimization algorithm, the dataset, and
the choice of the SPD layer (i.e. Cholesky factorization or
Eigendecomposition).

The surrogates are investigated on both the 2D solid truss and the 3D hollow
truss datasets. The performance of the models was evaluated similarly to the
previous experiment. Again the models were trained 1,080 times but here we use
different training/testing ratios of 80/20, 50/50, and 10/90 to assess the
models from rich to sparse data. And again, double precision was used for the
data and model parameters.

###  5.1 Solid truss dataset

The results from models fit to the 2D solid truss dataset are presented in
Tables 6-8. With one exception, we observe that the inclusion of either SPD
layer improved accuracy. The quadratic polynomial with Eigendecomposition SPD
layer had higher average test errors and 95th percentiles than a quadratic
polynomial with no SPD enforcement. The SPD layers had the greatest
improvement of accuracy with the 10% training and 90% testing data split.

|  | Test errors
---|---|---
Model | SPD layer | $\mu$ | $\sigma$ | $P=0.95$
NN | Chol. Fac. | 0.00012 | 0.00038 | 0.00052
NN | Eig. Decom. | 0.00021 | 0.00053 | 0.00111
NN | None | 0.00031 | 0.00086 | 0.00139
Quadratic | Chol. Fac. | 0.00142 | 0.00180 | 0.00533
Quadratic | Eig. Decom. | 0.00382 | 0.01013 | 0.02920
Quadratic | None | 0.00326 | 0.00427 | 0.01238
Quartic | Chol. Fac. | 0.00059 | 0.00092 | 0.00307
Quartic | Eig. Decom. | 0.00055 | 0.00090 | 0.00296
Quartic | None | 0.00184 | 0.00268 | 0.00864
RBF | Chol. Fac. | 0.00020 | 0.00054 | 0.00169
RBF | Eig. Decom. | 0.00013 | 0.00043 | 0.00095
RBF | None | 0.00048 | 0.00147 | 0.00476
Table 6: Comparison of models. 80% Training 20% Testing using Solid Isotruss data (100 data points, 2D). |  | Test errors
---|---|---
Model | SPD layer | $\mu$ | $\sigma$ | $P=0.95$
NN | Chol. Fac. | 0.00018 | 0.00034 | 0.00090
NN | Eig. Decom. | 0.00029 | 0.00045 | 0.00132
NN | None | 0.00041 | 0.00061 | 0.00202
Quadratic | Chol. Fac. | 0.00148 | 0.00082 | 0.00308
Quadratic | Eig. Decom. | 0.00353 | 0.00828 | 0.02994
Quadratic | None | 0.00348 | 0.00185 | 0.00655
Quartic | Chol. Fac. | 0.00072 | 0.00055 | 0.00186
Quartic | Eig. Decom. | 0.00065 | 0.00055 | 0.00184
Quartic | None | 0.00207 | 0.00131 | 0.00503
RBF | Chol. Fac. | 0.00042 | 0.00066 | 0.00202
RBF | Eig. Decom. | 0.00027 | 0.00048 | 0.00129
RBF | None | 0.00074 | 0.00122 | 0.00395
Table 7: Comparison of models. 50% Training 50% Testing using Solid Isotruss data (100 data points, 2D). |  | Test errors
---|---|---
Model | SPD layer | $\mu$ | $\sigma$ | $P=0.95$
NN | Chol. Fac. | 0.00156 | 0.00261 | 0.00433
NN | Eig. Decom. | 0.00193 | 0.00297 | 0.00499
NN | None | 0.00391 | 0.00689 | 0.01334
Quadratic | Chol. Fac. | 0.00275 | 0.00243 | 0.00717
Quadratic | Eig. Decom. | 0.00294 | 0.00288 | 0.00666
Quadratic | None | 0.00550 | 0.00373 | 0.01174
Quartic | Chol. Fac. | 0.00214 | 0.00197 | 0.00600
Quartic | Eig. Decom. | 0.00213 | 0.00228 | 0.00476
Quartic | None | 0.00445 | 0.00394 | 0.01071
RBF | Chol. Fac. | 0.00290 | 0.00198 | 0.00689
RBF | Eig. Decom. | 0.00197 | 0.00177 | 0.00525
RBF | None | 0.00449 | 0.00382 | 0.01115
Table 8: Comparison of models. 10% Training 90% Testing using Solid Isotruss
data (100 data points, 2D).

###  5.2 Hollow truss dataset

The results from models fit to the 3D hollow truss dataset are presented in
Tables 9-11. Similar to the solid truss, the results show that the inclusion
of either SPD layer resulted in improved accuracy. In contrast to the solid
truss, there were no observed exceptions to this trend. And surprisingly, the
quadratic polynomial with Eigendecomposition SPD layer exhibited the largest
accuracy improvement. The inclusion of the SPD layers with the neural network
was the least impactful, as the neural network without SPD enforcement was
typically more accurate than the other surrogate models with SPD layers.
Regardless, the accuracy of the neural network was further improved with
either SPD layer. And, we emphasize that the SPD layers ensure the desired SPD
output.

|  | Test errors
---|---|---
Model | SPD layer | $\mu$ | $\sigma$ | $P=0.95$
NN | Chol. Fac. | 0.00004 | 0.00006 | 0.00013
NN | Eig. Decom. | 0.00005 | 0.00006 | 0.00018
NN | None | 0.00009 | 0.00009 | 0.00029
Quadratic | Chol. Fac. | 0.00038 | 0.00020 | 0.00081
Quadratic | Eig. Decom. | 0.00044 | 0.00021 | 0.00090
Quadratic | None | 0.00155 | 0.00059 | 0.00269
Quartic | Chol. Fac. | 0.00012 | 0.00011 | 0.00035
Quartic | Eig. Decom. | 0.00011 | 0.00009 | 0.00030
Quartic | None | 0.00045 | 0.00027 | 0.00103
RBF | Chol. Fac. | 0.00012 | 0.00011 | 0.00036
RBF | Eig. Decom. | 0.00009 | 0.00008 | 0.00027
RBF | None | 0.00023 | 0.00015 | 0.00055
Table 9: Comparison of models. 80% Training, 20% Testing using Hollow Isotruss data (1,000 data points, 3D). |  | Test errors
---|---|---
Model | SPD layer | $\mu$ | $\sigma$ | $P=0.95$
NN | Chol. Fac. | 0.00004 | 0.00003 | 0.00009
NN | Eig. Decom. | 0.00006 | 0.00003 | 0.00012
NN | None | 0.00009 | 0.00005 | 0.00022
Quadratic | Chol. Fac. | 0.00038 | 0.00012 | 0.00066
Quadratic | Eig. Decom. | 0.00044 | 0.00011 | 0.00064
Quadratic | None | 0.00150 | 0.00029 | 0.00205
Quartic | Chol. Fac. | 0.00012 | 0.00009 | 0.00038
Quartic | Eig. Decom. | 0.00011 | 0.00005 | 0.00021
Quartic | None | 0.00044 | 0.00018 | 0.00083
RBF | Chol. Fac. | 0.00015 | 0.00009 | 0.00032
RBF | Eig. Decom. | 0.00011 | 0.00006 | 0.00023
RBF | None | 0.00025 | 0.00011 | 0.00046
Table 10: Comparison of models. 50% Training 50% Testing using Hollow Isotruss data (1,000 data points, 3D). |  | Test errors
---|---|---
Model | SPD layer | $\mu$ | $\sigma$ | $P=0.95$
NN | Chol. Fac. | 0.00014 | 0.00016 | 0.00039
NN | Eig. Decom. | 0.00018 | 0.00025 | 0.00050
NN | None | 0.00026 | 0.00026 | 0.00068
Quadratic | Chol. Fac. | 0.00056 | 0.00021 | 0.00097
Quadratic | Eig. Decom. | 0.00070 | 0.00083 | 0.00108
Quadratic | None | 0.00177 | 0.00024 | 0.00229
Quartic | Chol. Fac. | 0.00025 | 0.00018 | 0.00059
Quartic | Eig. Decom. | 0.00025 | 0.00016 | 0.00054
Quartic | None | 0.00069 | 0.00025 | 0.00119
RBF | Chol. Fac. | 0.00045 | 0.00027 | 0.00102
RBF | Eig. Decom. | 0.00036 | 0.00023 | 0.00079
RBF | None | 0.00064 | 0.00031 | 0.00127
Table 11: Comparison of models. 10% Training 90% Testing using Hollow Isotruss
data (1,000 data points, 3D).

##  6 Conclusion

There are applications where the output of a prediction model is expected to
be a symmetric positive definite (SPD) matrix. An example application is our
prediction of homogenized elasticity tensors from orthotropic Isotruss unit
cells as functions of their geometric and material parameters. We show that
fitting a model to a dataset that is strictly SPD does not ensure that model
predictions will be SPD. To remedy this we incorporated layers based on the
Cholesky factorization and an Eigendecomposition. These methods introduce no
additional learnable parameters, and ensure SPD output. The SPD layers require
a strictly positive function to enforce the desired behavior. We studied
several commonly used positive functions to test their efficacy. The ReLU
function appeared to be the hardest to train, resulting in a lower average
accuracy and larger standard deviation than the other positivity functions.
Some positivity functions performed better than others depending upon the
choice of optimization algorithm and machine learning architecture.

An experiment was also performed to evaluate whether the inclusion of an SPD
layer affected the model accuracy. A single layer neural network, polynomials,
and a linear kernel radial basis functions (RBF) were considered. The
inclusion of the SPD layer increased the accuracy for any particular surrogate
model.

## Acknowledgment

This work was performed under the auspices of the U.S. Department of Energy by
Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 and
was supported by the LLNL-LDRD Program under Project No. 20-ERD-020.

## References

  * Sanders et al. [2021] E. Sanders, A. Pereira, G. Paulino,  Optimal and continuous multilattice embedding,  Science Advances 7 (2021).
  * Liao [2021] H. Liao,  A single variable-based method for concurrent multiscale topology optimization with multiple materials,  Computer Methods in Applied Mechanics and Engineering 378 (2021) 113727\. doi:https://doi.org/10.1016/j.cma.2021.113727.
  * Xiao et al. [2021] M. Xiao, X. Liu, Y. Zhang, L. Gao, J. Gao, S. Chu,  Design of graded lattice sandwich structures by multiscale topology optimization,  Computer Methods in Applied Mechanics and Engineering 384 (2021) 113949\. doi:https://doi.org/10.1016/j.cma.2021.113949.
  * Gao et al. [2019] J. Gao, Z. Luo, H. Li, L. Gao,  Topology optimization for multiscale design of porous composites with multi-domain microstructures,  Computer Methods in Applied Mechanics and Engineering 344 (2019) 451–476. doi:https://doi.org/10.1016/j.cma.2018.10.017.
  * Bendsøe [1989] M. P. Bendsøe,  Optimal shape design as a material distribution problem,  Structural optimization 1 (1989) 193–202.
  * Rozvany [2001] G. I. Rozvany,  Aims, scope, methods, history and unified terminology of computer-aided topology optimization in structural mechanics,  Structural and Multidisciplinary optimization 21 (2001) 90–108.
  * Bendsoe and Sigmund [2013] M. P. Bendsoe, O. Sigmund, Topology optimization: theory, methods, and applications, Springer Science & Business Media, 2013\.
  * Nakshatrala et al. [2013] P. Nakshatrala, D. Tortorelli, K. Nakshatrala,  Nonlinear structural design using multiscale topology optimization. part i: Static formulation,  Computer Methods in Applied Mechanics and Engineering 261-262 (2013) 167–176. doi:https://doi.org/10.1016/j.cma.2012.12.018.
  * White et al. [2019] D. A. White, W. J. Arrighi, J. Kudo, S. E. Watts,  Multiscale topology optimization using neural network surrogate models,  Computer Methods in Applied Mechanics and Engineering 346 (2019) 1118–1135. doi:https://doi.org/10.1016/j.cma.2018.09.007.
  * Watts et al. [2019] S. Watts, W. Arrighi, J. Kudo, D. A. Tortorelli, D. A. White,  Simple, accurate surrogate models of the elastic response of three-dimensional open truss micro-architectures with applications to multiscale topology design,  Structural and Multidisciplinary Optimization 60 (2019) 1887–1920.
  * Wang et al. [2020] L. Wang, S. Tao, P. Zhu, W. Chen,  Data-driven multiscale topology optimization using multi-response latent variable gaussian process,  in: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 84003, American Society of Mechanical Engineers, 2020, p. V11AT11A057.
  * Fernández et al. [2022] M. Fernández, F. Fritzen, O. Weeger,  Material modeling for parametric, anisotropic finite strain hyperelasticity based on machine learning with application in optimization of metamaterials,  International Journal for Numerical Methods in Engineering 123 (2022) 577–609. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/nme.6869. doi:10.1002/nme.6869.
  * Watts [2020] S. Watts,  Elastic response of hollow truss lattice micro-architectures,  International Journal of Solids and Structures 206 (2020) 472–564.
  * Messner [2016] M. C. Messner,  Optimal lattice-structured materials,  Journal of the Mechanics and Physics of Solids 96 (2016) 162–183. doi:https://doi.org/10.1016/j.jmps.2016.07.010.
  * Amsallem et al. [2009] D. Amsallem, J. Cortial, K. Carlberg, C. Farhat,  A method for interpolating on manifolds structural dynamics reduced-order models,  International journal for numerical methods in engineering 80 (2009) 1241–1258.
  * Xu et al. [2021] K. Xu, D. Z. Huang, E. Darve,  Learning constitutive relations using symmetric positive definite neural networks,  Journal of Computational Physics 428 (2021) 110072.
  * Moakher and Batchelor [2006] M. Moakher, P. G. Batchelor,  Symmetric positive-definite matrices: From geometry to applications and visualization,  in: Visualization and processing of tensor fields, Springer, 2006, pp. 285–298.
  * Amos et al. [2017] B. Amos, L. Xu, J. Z. Kolter,  Input convex neural networks,  in: International Conference on Machine Learning, PMLR, 2017, pp. 146–155.
  * Klein et al. [2022] D. K. Klein, M. Fernández, R. J. Martin, P. Neff, O. Weeger,  Polyconvex anisotropic hyperelasticity with neural networks,  Journal of the Mechanics and Physics of Solids 159 (2022) 104703. doi:https://doi.org/10.1016/j.jmps.2021.104703.
  * Ting [1996] T. Ting,  Positive definiteness of anisotropic elastic constants,  Mathematics and Mechanics of Solids 1 (1996) 301–314.
  * Queipo et al. [2005] N. V. Queipo, R. T. Haftka, W. Shyy, T. Goel, R. Vaidyanathan, P. K. Tucker,  Surrogate-based analysis and optimization,  Progress in aerospace sciences 41 (2005) 1–28.
  * Viana [2016] F. A. Viana,  A tutorial on latin hypercube design of experiments,  Quality and reliability engineering international 32 (2016) 1975–1985.
  * Park and Sandberg [1991] J. Park, I. W. Sandberg,  Universal approximation using radial-basis-function networks,  Neural computation 3 (1991) 246–257.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al.,  Pytorch: An imperative style, high-performance deep learning library,  Advances in neural information processing systems 32 (2019) 8026–8037.
  * Hong et al. [2021] L. J. Hong, Z. Huang, H. Lam,  Learning-based robust optimization: Procedures and statistical guarantees,  Management Science 67 (2021) 3447–3467.
  * Schmidt [2005] M. Schmidt,  minfunc: unconstrained differentiable multivariate optimization in matlab,  Software available at http://www.cs.ubc.ca/ schmidtm/Software/minFunc.html (2005).
  * Kingma and Ba [2014] D. P. Kingma, J. Ba,  Adam: A method for stochastic optimization,  arXiv preprint arXiv:1412.6980 (2014).


# Revisiting Kernel Temporal Segmentation as an Adaptive Tokenizer for Long-
form Video Understanding

Mohamed Afham Satya Narayan Shukla Omid Poursaeed Pengchuan Zhang
Ashish Shah Sernam Lim
Meta AI

###### Abstract

While most modern video understanding models operate on short-range clips,
real-world videos are often several minutes long with semantically-consistent
segments of variable length. A common approach to process long videos is
applying a short-form video model over uniformly sampled clips of fixed
temporal length and aggregating the outputs. This approach neglects the
underlying nature of long videos since fixed-length clips are often redundant
or uninformative. In this paper, we aim to provide a generic and adaptive
sampling approach for long-form videos in lieu of the de facto uniform
sampling. Viewing videos as semantically-consistent segments, we formulate a
task-agnostic, unsupervised and scalable approach based on Kernel Temporal
Segmentation (KTS) for sampling and tokenizing long videos. We evaluate our
method on long-form video understanding tasks such as video classification and
temporal action localization, showing consistent gains over existing
approaches and achieving the state-of-the-art performance on long-form video
modeling.

##  1 Introduction

Majority of video understanding models are devised to learn representations of
short-form videos ranging from 5 to 10 seconds [10, 37, 19, 43, 9, 3, 22, 26].
These models usually suffer from computation and memory bottlenecks when
processing videos of longer length. A common approach to overcome this
bottleneck is to uniformly divide long videos into fixed-length clips, process
each clip separately and aggregate the results. This approach is highly
redundant as nearby clips often convey similar information and short clips
that overlap semantically meaningful segments are often uninformative.

Several works [28, 23, 41, 12, 20] have previously investigated adaptive
sampling to learn video representations in an efficient manner. These methods
often devise a learnable adaptive sampler to select more representative frames
of the video based on the reward or penalty provided by the final prediction
score. However, these methods are often limited to the classification task and
are heavily dependent on the specific tasks and datasets on which they are
trained and cannot easily transfer to unseen tasks or datasets.

Most of these adaptive sampling approaches are not scalable to sampling large
number of frames which is required for understanding long-form videos. In
fact, all the recent approaches [18, 38] for long-form video understanding use
the de facto uniform sampling for sampling fixed-length clips from long
videos.

In this work we propose a task-agnostic, adaptive and unsupervised sampling
approach for long videos. Motivated by the intuition that humans perceive
videos as semantically-consistent segments of variable length, we decompose
the video to semantically meaningful segments using Kernel Temporal
Segmentation (KTS) [30]. KTS extracts features from sparsely sampled candidate
frames, computes the matrix of frame-to-frame similarity, and outputs a set of
optimal change points corresponding to the boundaries of temporal segments. We
then sample frames from each segment uniformly which comprises the input to
the video understanding model. Our KTS-based input tokenization achieves the
following desirable attributes: (a) it is agnostic to the downstream task, (b)
it yields semantically-consistent segments without relying on training data,
and (c) it is scalable to arbitrary number of segments and frames for a given
long video.

We validate generalizability of KTS-based adaptive sampling on multiple
downstream tasks and benchmarks. We evaluate KTS-based sampling for video
classification on Breakfast [21] and LVU [38] benchmarks achieving state-of-
the-art performance. We also report results for temporal action localization
on ActivityNet [8], showing effectiveness of KTS-based sampling over standard
uniform sampling. Furthermore, we provide a comparison with existing adaptive
frame sampling methods on ActivityNet video classification and show that our
approach outperforms the baselines.

The main contribution of our work can be summarized as follows:

  * •

We propose an adaptive, unsupervised, and task-agnostic frame sampling
mechanism for long videos based on Kernel Temporal Segmentation (KTS), which
overcomes deficiencies of common sampling approaches.

  * •

We extensively evaluate KTS-based adaptive sampling against existing sampling
techniques on video classification and temporal action localization tasks,
showing consistent improvements and achieving state-of-the-art performance on
long-form video understanding.

##  2 Related Work

Most of the video understanding models are devised to learn the
representations of short-form videos ranging from $5$ to $10$ seconds [10, 37,
19, 43, 9, 3, 22, 26]. While these approaches use various architectures such
as 2D CNNs [19, 6, 43], 3D CNNs [5, 10, 37, 36] and Vision Transformers [3, 9,
22, 26], they often share uniform sampling for input tokenization. These
models usually suffer from computation and memory bottlenecks when processing
videos of longer length.

Recent approaches for long-form video modeling can be broadly divided into two
categories: a) building specialized models for learning from long-form videos,
and b) adaptive sampling approaches for selecting frames from long-form
videos. We discuss the related works in both the areas below:

###  2.1 Long-form Video Understanding

Several works have been introduced to study the capability of video models in
modeling videos of longer length. A movie based question answering dataset was
introduced by Tapaswi _et al_. [35] and Bain _et al_. [4] introduced a text-
to-video retrieval benchmark based on videos from movies. However, those line
of works explore the video-language learning ability of the model hence not
ideal for video-only evaluation.

Recent works [15, 16, 38, 18] improve the ability of learning long-range
dependencies in the temporal domain of the videos in video classification
setting. ViS4mer [18] introduces a state-space sequence layer to model the
extracted the short term clip features in a long video. The object transformer
model in [38] aims to capture the long-range interactions between tracked
objects. Wu _et al_. [38] recently introduced a long-video benchmark (LVU)
comprised of 7 classification tasks and 2 regression tasks based on movie
videos, which has become a standard benchmark for long-form video
understanding. Another line of work focuses on temporal action localization
(TAL) [44, 25] task which requires modeling the long-range dependencies and
evaluated on long-video datasets such as ActivityNet [8] and Thumos [17].

While the proposed approaches in video classification and temporal action
localization show promising performance in the modeling aspect, uniform
sampling is employed as the default input sampling strategy and hence, these
approach require a large number of frames for understanding the long-form
videos. Instead, in this work, we deploy the KTS-based adaptive sampling as
input tokenization for both video classification and temporal action
localization to study effectiveness of it over the standard uniform sampling.

###  2.2 Adaptive sampling

Several adaptive sampling based strategies [40, 20, 47, 23, 41, 46, 12] have
been proposed to overcome the computation issues faced by the standard uniform
sampling in video classification. SCSampler [20] used a light-weight network
to predict the saliency score of short-clips sampled uniformally in the long
video. AdaFrame [40] introduces an LSTM network augmented with a global
memory, to learn how to adaptively select frames conditioned on inputs for
efficient video recognition. FrameExit [12] investigates an early existing
procedure by employing a cascade of gating modules to automatically determine
the earliest point in processing where an inference is sufficiently reliable.
OCSampler [23] effectively samples few frames from a selected number of frame
candidates to dynamically localize and attend to the instance-specific
condensed clip of each video. Zhi _et al_. design an adaptive sampling
strategy named MGSampler [47] aiming to choose a more effective input with a
fixed length in trimmed and short.

While our work is closely related to both OCSampler and MGSampler, in contrast
to them KTS-based sampling is task-agnostic and applicable to various long-
video understanding tasks (e.g., video recognition, temporal action
localization). Unlike the prior adaptive sampling approaches, e.g. OCSampler
[23] where sampler is first trained separately, KTS-based sampling does not
require two-stage training. KTS-based adaptive sampling is scalable and can be
used to sample large number of frames from long-range videos effectively
unlike the prior works which focus on sampling a small number of frames and
are not easily scalable. KTS based sampling is also unsupervised and can be
performed independently to the downstream task as it is based on the change
points of the video features.

##  3 Method

Figure 1: An overview of KTS-based adaptive sampling for Video Classification
and Temporal Action Localization. The input video is initially downsampled and
$m-1$ change points are computed using the KTS algorithm. $k$ frames are then
uniformly sampled from each of the $m$ segments and are processed for the
downstream task.

Conventional long-range video models process uniformly sampled short clips
from the video and aggregate the results. However, relevant information in a
long video is often not evenly distributed. Humans perceive videos as a
sequence of coherent scenes/events and can have a semantic understanding of
the scenes given a few sampled frames. Motivated by this intuition, we propose
a similar approach for sampling and tokenizing long videos. We decompose
videos into semantically consistent segments leveraging Kernel Temporal
Segmentation (KTS) and sample frames uniformly from each segment. We first
give an overview of the KTS algorithm in Sec. 3.1. Then we describe our
sampling strategy for long-form video classification and action localization
in Sec. 3.2. Finally, we elaborate on the effectiveness of KTS over the other
adaptive sampling techniques in the context of long-form video understanding.

###  3.1 Kernel Temporal Segmentation

The initial motivation behind KTS is to detect change points in the input and
decompose the video into semantically consistent segments. KTS is a kernel-
based algorithm that operates independently and in an unsupervised manner,
hence it does not require any additional training to yield meaningful video
segments. KTS has been extensively leveraged by several video summarization
approaches [27, 45, 31, 42, 49] as the segmentation output provided by KTS has
a significant impact on identifying highlights of the video and yielding a
high-quality summarization of the video. Here we briefly describe the KTS
algorithm.

Given a long-form video, we initially downsample it, e.g. to one frame per
second, and extract frame-level features using a pre-trained feature extractor
$f_{\theta}$. Let $(x_{i}){{}^{n}_{i=1}}\in\mathbf{X}$ represent the sampled
frames, $\mathbf{K}:\mathbf{X}\times\mathbf{X}\rightarrow{\mathbb{R}}$
represent a kernel function (Gram matrix) between descriptors
$f_{\theta}(x_{i})$ and $\phi:\mathbf{X}\rightarrow\mathcal{H}$ be the
associated feature map with norm $\|.\|_{\mathcal{H}}$. Suppose we want to
choose $m-1$ change points $x_{t_{1}},\cdots,x_{t_{m-1}}$, which correspond to
$m$ segments
$[x_{t_{0}},x_{t_{1}}],[x_{t_{1}},x_{t_{2}}],\cdots,[x_{t_{m-1}},x_{t_{m}}]$
with $x_{t_{0}}=0$ and $x_{t_{m}}=T$ being length of the video.

The KTS algorithm minimizes the sum of the within-segment variances:

$\min\limits_{m,t_{1},\cdots,t_{m-1}}\sum^{m}_{i=1}var(t_{i-1},t_{i})$ (1)

where:

$var(t_{i-1},t_{i})={\sum}_{t=t_{i-1}}^{t_{i}-1}\|\phi(x_{t})-\mu_{i}\|^{2}$
(2)

and $\mu_{i}$ is the within-segment mean:

$\mu_{i}=\frac{{\sum}_{t=t_{i-1}}^{t_{i}-1}\phi{(x_{t})}}{t_{i}-t_{i-1}}$ (3)

We can also make KTS adaptive to each video by making the number of segments
$m$ variable. To avoid over-segmentation we add a penalty term $g(m,n)$ to the
objective function. A common choice for $g(m,n)$ is $m\log(\frac{m}{n}+1)$. In
this case, our final objective is:

$\min\limits_{m,t_{1},\cdots,t_{m-1}}\sum^{m}_{i=1}var(t_{i-1},t_{i})+g(m,n)$
(4)

In order to solve Equation 1 and 4, we first compute the kernel for each pair
of descriptors. We use a dot-product kernel in practice. Then the segment
variances are computed for each possible starting point and segment duration.
Finally, we use dynamic programming to minimize the objective and find the
change points. Refer to [30] for more details.

###  3.2 Adaptive sampling with KTS

KTS algorithm yields a set of change points $x_{t_{1}},\cdots,x_{t_{m-1}}$
which decompose the video into $m$ segments. Note that unlike shot boundary
detection methods which focus on local differences between consecutive frames,
KTS takes into account the differences between all pairs of frames. Therefore
it provides semantically consistent and general segments. To represent each
segment we uniformly sample $k$ frames from it. Long-form video models often
consist of a backbone to process short-range clips and an aggregation
mechanism (e.g. via a transformer or simple averaging). We feed sampled frames
from each segment to the clip-level model which learns the representation for
each segment/scene. The aggregation mechanism then combines scene-level
information to obtain a global video-level representation. This is in line
with how humans perceive videos. Despite its simplicity, we show that our
sampling approach achieves state-of-the-art performance on long-form video
modeling and outperforms existing samplers on several tasks and benchmarks.

###  3.3 Discussion

As explained in Sec. 2 there are several other adaptive sampling techniques
proposed in the literature. Our approach differs from these samplers in
several ways. KTS-based sampling is generic and can be applied to various
downstream tasks without training on them. However, existing samplers are
task-specific and are often limited to video classification. We show in our
experiments that our approach outperforms existing samplers on the video
classification task. Unlike current approaches which disregard large portions
of the video, KTS-based sampling minimizes loss of information as it samples
from all the segments. This makes our approach well-suited for tasks such as
action localization which need to preserve local information.

##  4 Experiments

In this section, we present our experiments and results. We focus on video
classification and temporal action localization tasks. We perform temporal
action localization and classification on the ActivityNet [8] dataset, and
video classification on the Breakfast dataset [21] and the LVU Benchmark [38].
We also perform ablation experiments to show the impact of the number of
frames used for video classification, number of change points estimated by
KTS, and the backbone used as the feature extractor.

###  4.1 Datasets

Breakfast [21] is a human activity dataset focused on cooking-oriented
actions. It comprises of $10$ categories of cooking breakfast. It contains
$1712$ videos in total with $1357$ for training and $335$ for testing. The
average length of a video is $2.3$ minutes. The cooking actions were performed
by $52$ actors with $44$ for training and 8 for testing. This makes the task
more challenging since the actors performing the actions during test time are
not seen during training.

LVU (long-form video understanding benchmark) [38] is compiled from the
publicly available MovieClips dataset [1] which contains around 30,000 short
movie snippets. The benchmarks consist of $9$ diverse tasks that require long-
form video understanding. These tasks could be mainly categorized into content
understanding (‘relationship’, ‘speaking style’, ‘scene/place’), movie
metadata prediction (‘director’, ‘genre’, ‘writer’, ‘movie release year’) and
user engagement prediction (‘YouTube like ratio’, ‘YouTube popularity’).
Content understanding and movie metadata prediction can be considered as
classification tasks hence are evaluated using the top-1 accuracy metric and
user engagement prediction is a regression task and is evaluated using mean
squared error (MSE). Each video is generally one to three minutes long. For
both Breakfast and LVU datasets, we follow the training configuration
suggested in ViS4mer.

ActivityNet [8] dataset contains around 20,000 untrimmed videos spanning 200
action classes of daily activities. The average length of a video is 117
seconds, and the average length of action segments is 48 seconds. Thus it can
be considered as a long-form video dataset. Following the standard practice
[2, 44], we train on the training split of the dataset and evaluate on the
validation split. We report average $mAP@[0.5:0.05:0.95]$ similar to
Actionformer [44] for fair comparison.

Table 1: Video Classification results on Breakfast. We evaluate KTS-based sampling against uniform sampling with ViS4mer [18] as the baseline. Our approach achieves state-of-the-art performance with significantly less computation. Method | Frames | Accuracy
---|---|---
VideoGraph [15] | $64\times 8$ | 69.50
Timeception [16] | $1024\times 8$ | 71.30
GHRM [48] | $64\times 8$ | 75.49
ViS4mer [18] | $32\times 32$ | $85.63$
ViS4mer [18] | $512\times 32$ | $88.17$
ViS4mer + KTS (Ours) | $\mathbf{32\times 32}$ | $\mathbf{89.86}$
Table 2: Evaluation of KTS-based sampling on the LVU benchmark. Our approach shows consistent improvements over uniform sampling on the majority of video understanding tasks. Method | Content ($\uparrow$) | Metadata ($\uparrow$) | User ($\downarrow$)
---|---|---|---
Relation | Speak | Scene | Director | Genre | Writer | Year | Like | Views
SlowFast + NL [10] | 52.40 | 35.80 | 54.70 | 44.90 | 53.00 | 36.30 | 52.50 | 0.38 | 3.77
VideoBERT [33] | 52.80 | 37.90 | 54.90 | 47.30 | 51.90 | 38.50 | 36.10 | 0.32 | 4.46
Object Transformer [38] | 53.10 | 39.40 | 56.90 | 51.20 | 54.60 | 34.50 | 39.10 | 0.23 | 3.55
ViS4mer [18] | 57.24 | 40.79 | 67.44 | 62.62 | 54.71 | 48.8 | 44.75 | 0.26 | 3.63
ViS4mer + KTS (Ours) | 59.52 | 40.79 | 80.23 | 69.16 | 65.86 | 54.16 | 48.25 | 0.29 | 3.29

###  4.2 Video Classification

We perform video classification experiments on Breakfast [21] and LVU [38]
datasets to study the effectiveness of KTS-based adaptive sampling.

Baseline: We adopt the recently introduced ViS4mer [18] as the baseline model
to evaluate the performance of KTS-based adaptive sampling against the uniform
sampling on video classification tasks. ViS4mer is a long-range video
classification model comprised of a standard Transformer encoder [7, 26] and a
multi-scale temporal S4 [13] decoder. It extracts features from input video
tokens using the Transformer encoder which are then fed to the multi-scale S4
decoder that learns hierarchical spatio-temporal video representations.
ViS4mer uses Vision Transformer [7] to extract features for experiments on the
LVU benchmark and uses Video Swin Transformer [26] to extract features in
experiments on the Breakfast dataset. Despite innovation in the modeling
aspect, ViS4mer leverages uniform sampling to tokenize the input video. We
adopt KTS-based adaptive sampling in both settings owing to its task-agnostic
nature.

Implementation Details: Given a video, we downsample it to one frame per
second and use the downsampled frames as candidates for computing the change
points. We use GoogleNet [34] pre-trained on ImageNet-1K for extracting the
feature descriptors. We sample $m\times k$ frames for each video as described
in Sec. 3.2, and the sampled frames are then fed to the video classification
model.

Results: Table. 1 demonstrates the video classification results on the
Breakfast dataset. We observe that KTS-based adaptive sampling achieves state-
of-the-art results on the Breakfast dataset while utilizing $16\times$ fewer
frames per video compared to the original ViS4mer baseline which uses uniform
sampling. When compared with uniform sampling using the same setting
$[32\times 32]$, we observe a significant gain of $4.23\%$ in terms of
accuracy with KTS-based adaptive sampling, showing its superiority over
uniform sampling.

Table. 2 shows the results on the LVU benchmark. KTS-based adaptive sampling
achieves state-of-the-art performance on $7$ out of $9$ tasks and outperforms
uniform sampling in $8$ out of $9$ tasks. In particular, in the scene
prediction under content understanding task of the benchmark, KTS-based
tokenization yields a performance boost of $12.79\%$ while in the genre
prediction task under movie metadata prediction, KTS-based tokenization
outperforms uniform sampling with a significant margin of $11.15\%$. Similar
performance gains can be observed consistently throughout different tasks on
the benchmark. Note that the LVU benchmark is more challenging than the
Breakfast dataset. The tasks in LVU require long-term dependencies to be
captured carefully, where the current short-range video models are proven to
fail even with strong pre-training mechanisms [38]. KTS-based input
tokenization shows promising performance consistently throughout the benchmark
which states the need to perform an adaptive sampling strategy to process
long-form videos.

###  4.3 Temporal Action Localization

Temporal Action localization (TAL) aims to identify the action instances
present in a video in the temporal domain and recognize the action categories.
Despite the steady progress in TAL performance in the modeling aspects (e.g.,
action proposals [24], pretraining [2], single-stage TAL [44]), uniform
sampling is adopted as the de facto sampling approach in most of the action
localization models. We analyze the impact of the KTS-based adaptive sampling
mechanism on action localization.

Baseline: We investigate the performance of KTS-based sampling on the strong
Actionformer [44] baseline, which achieves the current state-of-the art
performance on TAL for ActivityNet. It comprises a multi-scale transformer
encoder which encodes the sequence of embedded video clip features into a
feature pyramid. The feature pyramid is then followed by a classification and
a regression head to recognize the action instance and estimate the action
boundaries respectively. TSP [2] model pre-trained on ActivityNet video
classification task is used to extract non-overlapping clip-level features.
Refer to [44] for a complete description of Actionformer.

Implementation Details: Given a video, we downsample it to one frame per
second when computing the KTS change points and use ResNet-50 [14] pre-trained
on ImageNet-1K to extract feature descriptors for KTS computation. We adopt a
similar training configuration as the Actionformer to study the impact of KTS-
based adaptive sampling in TAL. Actionformer employs clips of 16 frames at a
frame rate of $15$ fps and a stride of $16$ frames (i.e., non-overlapping
clips) as input to the feature extractor followed by the localization module.
This gives one feature vector per $\frac{16}{15}\approx 1.067$ seconds and
$M=\frac{15}{16}T$ segments where $T$ is the video length. We can also
consider $\frac{M}{2}$, $\frac{M}{4}$, $\cdots$ segments by sampling every
$2^{nd}$, $4^{th}$, $\cdots$ frame. Similarly, we can choose $\frac{M}{2}$,
$\frac{M}{4}$, $\cdots$ segments in our KTS-based sampling strategy. For the
baseline, all the segments have the same length while our adaptive sampling
technique yields variable-length segments. Within each segment, we uniformly
sample $16$ frames in both cases. These frames are then fed to the action
localization model. Fig. 2 provides a comparison of KTS vs. uniform sampling,
showing improved performance, especially for smaller number of segments.

Figure 2: KTS vs Uniform sampling comparison on ActivityNet Action
Localization. We report average mAP when varying the number of segments. $M$
corresponds to the number of segments when each segment length is
$\frac{16}{15}$ seconds as used in the Actionformer baseline.

Results: Fig. 2 shows the empirical analysis of KTS-based sampling on TAL.
Note that the performance gain of using KTS-based adaptive sampling is clearly
observed for smaller number of segments (e.g., $\frac{M}{3}$ and below), and
the gap in performance increases when reducing the number of segments. In
particular, for $\frac{M}{6}$ segments uniform sampling achieves $31.05\%$
average mAP while KTS-based sampling attains $32.58\%$ average mAP on
ActivityNet, yielding $1.53\%$ gain. For larger number of segments, the
performance of KTS is nearly similar to uniform sampling. For $M$ segments,
KTS reduces to uniform sampling as there are $M$ change point candidates when
using one frame per second for sampling candidates. Similarly, for
$\frac{M}{2}$ we select half of the candidates as change points, which makes
it quite similar to uniform sampling.

###  4.4 Comparison with Existing Adaptive Sampling Methods

Table 3: Comparison of our approach with existing adaptive sampling strategies on ActivityNet video classification. Method | Backbone | mAP (%) | GFLOPs
---|---|---|---
NSNet [41] | ResNet-101 | 74.9 | 73.2
AdaFrame [40] | ResNet-101 | 71.5 | 78.7
LiteEval [39] | ResNet-101 | 72.7 | 95.1
KTS (Ours) $\left(84\times 84\right)$ [8 frames] | ResNet-101 | 80.9 | 67.1
Uniform | ResNet-50 | 72.5 | 65.8
Random | ResNet-50 | 71.2 | 65.8
SCSampler [20] | ResNet-50 | 72.9 | 41.9
AdaMML [29] | ResNet-50 | 73.9 | 94.0
AR-Net [28] | ResNet-50 | 73.8 | 33.5
ListenToLook [11] | ResNet-50 | 72.3 | 81.4
OCSampler [23] | ResNet-50 | 79.8 | 67.2
KTS (Ours) $\left(84\times 84\right)$ [6 frames] | ResNet-50 | 74.8 | 29.7
KTS (Ours) $\left(84\times 84\right)$ [8 frames] | ResNet-50 | 80.0 | 32.1
KTS (Ours) $\left(112\times 112\right)$ [8 frames] | ResNet-50 | 80.3 | 37.4

Table. 3 compares KTS-based adaptive tokenization with existing efficient
frame sampling methods for video classification on the ActivityNet dataset. We
use MobileNetv2 [32] pre-trained on ImageNet-1K to extract the features. For a
fair comparison with previous methods in terms of accuracy and computational
cost, we initially uniformly sample 16 frames resized to a smaller resolution
(e.g., 112 $\times$ 112) in a given video as the change point candidates and
estimate change points. We sample one frame within each segment and train the
ResNet50 classifier (pre-trained on Imagenet-1K) for video classification on
ActivityNet. Our results show that KTS-based sampling yields a competitive
performance when compared to existing adaptive sampling approaches. In
particular, KTS-based sampling improves the classification accuracy by
$1.03\%$ over AR-Net [28] while minimizing the computational cost by $3.8$
GFLOPS. KTS algorithm incurs only around $0.004$ GFLOPS in our experiments
which is comparatively negligible to the computational cost incurred by
ResNet50 and MobileNetV2. KTS-based sampling method also outperforms OCSampler
[23] while incurring significantly less computation cost.

Figure 3: KTS vs Uniform sampling comparison on classification tasks of the
LVU benchmark by varying the number of input frames. Consistent performance
gain shows the effectiveness of KTS-based adaptive sampling over standard
uniform sampling.

###  4.5 Ablation and Analysis

In this section, we perform ablation experiments to show the impact of the
number of frames used for video classification, number of change points
estimated by KTS, and the backbone used as the feature extractor.

Table 4: Impact of the number of change points in KTS-based adaptive video tokenization. We analyze the performance of our sampling approach on Breakfast video classification for different configurations of the number of change points. $m$: Number of video segments., $k$: Number of frames selected to process within each segment. Total Frames | $m\times k$ | Uniform | KTS | $\Delta$
---|---|---|---|---
$256$ | $16\times 16$ | $78.87$ | $81.69$ | $+2.82\%$
$32\times 8$ | $81.13$ | $84.51$ | $+3.38\%$
$512$ | $16\times 32$ | $80.56$ | $84.51$ | $+3.95\%$
$32\times 16$ | $83.09$ | $88.17$ | $+5.08\%$
$64\times 8$ | $83.66$ | $87.04$ | $+3.38\%$
$1024$ | $\mathbf{32\times 32}$ | $\mathbf{85.63}$ | $\mathbf{89.86}$ | $\mathbf{+4.23\%}$
$64\times 16$ | $85.63$ | $86.76$ | $+1.13\%$
Figure 4: KTS vs Uniform sampling comparison on Breakfast video classification
with a varying number of frames. Figure 5: KTS vs Uniform sampling comparison
on the view count prediction task of LVU benchmark.

####  4.5.1 Impact of the number of frames

We evaluate KTS-based adaptive sampling against uniform sampling on Breakfast
and LVU dataset. We vary the number of input frames per video used for the
classification using both the sampling approach. Fig. 3 presents the results
on the LVU benchmark on four tasks: scene prediction, genre prediction, writer
prediction, and director prediction. We vary the number of frames sampled for
modeling. KTS-based adaptive sampling yields a consistent performance gain
over standard uniform sampling in all configurations.

Fig. 4 demonstrates the results on Breakfast dataset. KTS-based adaptive
sampling shows consistent performance gain over uniform sampling in all
settings. In particular, KTS-based sampling gives an accuracy of $84.51\%$
when using $256$ frames for each video in the $32\times 8$ setting while
uniformly sampling $512$ frames yields $83.09\%$ in the same setting. This
shows KTS-based tokenization not only improves the performance but also
requires significantly less number of frames to process when compared to
uniform sampling. Table. 1 also validates this claim where KTS-based adaptive
input tokenization achieves state-of-the art performance on Breakfast video
classification with $(1/16)^{th}$ number of frames as that of uniform
sampling.

Table. 4 reports the empirical results on different configurations of change
points for a given number of frames input to the model. We observe that
increasing the number of change points improves the performance of the model
up to a certain point. We also observe that KTS-based sampling consistently
achieves significantly better performance compared to uniform sampling when
compared in the same setting over all configurations. In particular, we obtain
a performance gain of $4.23\%$ in the $32\times 32$ setting where the number
of change points estimated by the KTS algorithm is $31$. Similarly, $5.08\%$
boost is observed in $32\times 16$ configuration over uniform sampling in the
ViS4mer baseline.

Fig. 5 demonstrates the results on the regression task (view count prediction)
of the LVU benchmark. We vary the number of frames sampled for modeling. KTS-
based adaptive sampling consistently achieves lower mean-squared error (MSE)
compared to uniform sampling across different configuration settings.

####  4.5.2 Impact of the feature extractor on KTS

We investigate the choice of feature extractor in the KTS algorithm in Table.
5. We consider three standard image feature extractors namely: ResNet50 [14],
MobileNetv2 [32] and GoogleNet [34]. We use the pre-trained models after
discarding the final classification layer. We observe that in both $32\times
16$ and $64\times 8$ settings, the GoogleNet backbone produces better results
on the Breakfast dataset. Hence, similar to the choice of the number of
frames, the choice of feature extractor plays a significant role in the KTS
algorithm.

Table 5: Impact of the feature extractor of KTS on the Breakfast video classification task $m\times k$ | ResNet50 | MobileNetv2 | GoogleNet
---|---|---|---
$64\times 8$ | 81.12 | 83.38 | 87.04
$32\times 16$ | 81.41 | 83.94 | 88.17

##  5 Limitations and Future Work

We identify two major limitations of KTS based input tokenization in long-form
video understanding: (1) KTS, as a kernel based change point detection
algorithm, is not learnable. The segmentation defects happening in KTS would
directly affect the downstream task performance. However, learning change
points specific to a task (e.g., video classification on Breakfast) can make
transferability limited. We plan to investigate a learnable alternative to our
current approach in the future. (2) While KTS performs superior to uniform
sampling in several training configurations as shown in our analysis, the
optimal choice of number of change points and the choice of feature extractor
for KTS-based sampling still remains to be hand-picked. A possible future
direction could be to investigate a learnable mechanism to select the optimal
number of change points for a given video.

##  6 Conclusion

In this work, we present an adaptive and task-agnostic frame sampling
mechanism for long-form video modeling. Our approach leverages Kernel Temporal
Segmentation (KTS) to generate semantically-consistent segments used for
sampling frames. We perform a comprehensive set of experiments on video
classification and temporal action localization on several long-form video
understanding datasets and benchmarks and show the superiority of KTS-based
adaptive sampling against existing sampling strategies. In spite of its
simplicity, our sampling approach achieves state-of-the-art performance on
long-form video understanding benchmarks while being efficient. We plan to
explore other variants of adaptive sampling based on temporal segmentation,
which could operate in a learnable manner, in the future.

## References

  * [1] Movieclips.  https://www.youtube.com/c/MOVIECLIPS.
  * [2] Humam Alwassel, Silvio Giancola, and Bernard Ghanem.  TSP: Temporally-sensitive pretraining of video encoders for localization tasks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 3173–3183, 2021.
  * [3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021.
  * [4] Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman.  Condensed movies: Story based retrieval with contextual embeddings.  In Proceedings of the Asian Conference on Computer Vision, 2020.
  * [5] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [6] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell.  Long-term recurrent convolutional networks for visual recognition and description.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2625–2634, 2015.
  * [7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [8] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–970, 2015.
  * [9] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6824–6835, 2021.
  * [10] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [11] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani.  Listen to look: Action recognition by previewing audio.  In CVPR, 2020.
  * [12] Amir Ghodrati, Babak Ehteshami Bejnordi, and Amirhossein Habibian.  Frameexit: Conditional early exiting for efficient video recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15608–15618, 2021.
  * [13] Albert Gu, Karan Goel, and Christopher Ré.  Efficiently modeling long sequences with structured state spaces.  arXiv preprint arXiv:2111.00396, 2021.
  * [14] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2015.
  * [15] Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders.  Videograph: Recognizing minutes-long human activities in videos.  In ICCV Workshop on Scene Graph Representation and Learning, 2019.
  * [16] Noureldien Hussein, Efstratios Gavves, and Arnold W. M. Smeulders.  Timeception for complex action recognition.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 254–263, 2018.
  * [17] Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah.  The thumos challenge on action recognition for videos “in the wild”.  Computer Vision and Image Understanding, 155:1–23, 2017.
  * [18] Md Mohaiminul Islam and Gedas Bertasius.  Long movie clip classification with state-space video models.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 87–104, 2022.
  * [19] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.  Large-scale video classification with convolutional neural networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014.
  * [20] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Scsampler: Sampling salient clips from video for efficient action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6232–6242, 2019.
  * [21] H. Kuehne, A. B. Arslan, and T. Serre.  The language of actions: Recovering the syntax and semantics of goal-directed human activities.  In Proceedings of Computer Vision and Pattern Recognition Conference (CVPR), 2014.
  * [22] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4804–4814, 2022.
  * [23] Jintao Lin, Haodong Duan, Kai Chen, Dahua Lin, and Limin Wang.  Ocsampler: Compressing videos to one clip with single-step sampling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13894–13903, 2022.
  * [24] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.  Bmn: Boundary-matching network for temporal action proposal generation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 3889–3898, 2019.
  * [25] Xiaolong Liu, Qimeng Wang, Yao Hu, Xu Tang, Shiwei Zhang, Song Bai, and Xiang Bai.  End-to-end temporal action detection with transformer.  IEEE Transactions on Image Processing (TIP), 2022.
  * [26] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202–3211, 2022.
  * [27] Behrooz Mahasseni, Michael Lam, and Sinisa Todorovic.  Unsupervised video summarization with adversarial lstm networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 202–211, 2017.
  * [28] Yue Meng, Chung-Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris.  Ar-net: Adaptive frame resolution for efficient action recognition.  arXiv preprint arXiv:2007.15796, 2020.
  * [29] Rameswar Panda, Chun-Fu Richard Chen, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude Oliva, and Rogerio Feris.  Adamml: Adaptive multi-modal learning for efficient video recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7576–7585, 2021.
  * [30] Danila Potapov, Matthijs Douze, Zaid Harchaoui, and Cordelia Schmid.  Category-specific video summarization.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 540–555. Springer, 2014.
  * [31] Mrigank Rochan, Linwei Ye, and Yang Wang.  Video summarization using fully convolutional sequence networks.  In Proceedings of the European conference on computer vision (ECCV), pages 347–363, 2018.
  * [32] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [33] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464–7473, 2019.
  * [34] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, 2014.
  * [35] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler.  Movieqa: Understanding stories in movies through question-answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631–4640, 2016.
  * [36] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli.  Video classification with channel-separated convolutional networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5552–5561, 2019.
  * [37] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  * [38] Chao-Yuan Wu and Philipp Krahenbuhl.  Towards long-form video understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1884–1894, 2021.
  * [39] Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, and Larry S Davis.  Liteeval: A coarse-to-fine framework for resource efficient video recognition.  Advances in Neural Information Processing Systems, 32, 2019.
  * [40] Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, and Larry S Davis.  Adaframe: Adaptive frame selection for fast video recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1278–1287, 2019.
  * [41] Boyang Xia, Wenhao Wu, Haoran Wang, Rui Su, Dongliang He, Haosen Yang, Xiaoran Fan, and Wanli Ouyang.  Nsnet: Non-saliency suppression sampler for efficient video recognition.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIV, pages 705–723. Springer, 2022.
  * [42] Ting Yao, Tao Mei, and Yong Rui.  Highlight detection with pairwise deep ranking for first-person video summarization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 982–990, 2016.
  * [43] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici.  Beyond short snippets: Deep networks for video classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  * [44] Chen-Lin Zhang, Jianxin Wu, and Yin Li.  Actionformer: Localizing moments of actions with transformers.  In European Conference on Computer Vision (ECCV), pages 492–510, 2022.
  * [45] Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman.  Video summarization with long short-term memory.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14, pages 766–782. Springer, 2016.
  * [46] Yin-Dong Zheng, Zhaoyang Liu, Tong Lu, and Limin Wang.  Dynamic sampling networks for efficient action recognition in videos.  IEEE transactions on image processing, 29:7970–7983, 2020.
  * [47] Yuan Zhi, Zhan Tong, Limin Wang, and Gangshan Wu.  Mgsampler: An explainable sampling strategy for video action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1513–1522, 2021.
  * [48] Jiaming Zhou, Kun-Yu Lin, Haoxin Li, and Wei-Shi Zheng.  Graph-based high-order relation modeling for long-term action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8984–8993, 2021.
  * [49] Wencheng Zhu, Jiwen Lu, Jiahao Li, and Jie Zhou.  Dsnet: A flexible detect-to-summarize network for video summarization.  IEEE Transactions on Image Processing, 30:948–962, 2020.