# NAS-TC: Neural Architecture Search on Temporal
Convolutions for Complex Action Recognition

Pengzhen Ren, Gang Xiao, Xiaojun Chang, Yun Xiao, Zhihui Li, and Xiaojiang
Chen  Corresponding author: Yun Xiao.P. Ren, Y. Xiao, and X. Chen are with
School of Information Science and Technology, Northwest University. (Email:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>xcwang@ucalgary.ca)G.
Xiao is a senior engineer in Huawei Technologies Co.Ltd, China.zzzX. Chang is
with Faculty of Information Technology, Monash University, Australia, 3168.
(E-mail: cxj273@gmail.com)Z. Li is with Shandong Artificial Intelligence
Institute, Qilu University of Technology. (E-mail<EMAIL_ADDRESS>
###### Abstract

In the field of complex action recognition in videos, the quality of the
designed model plays a crucial role in the final performance. However,
artificially designed network structures often rely heavily on the
researchers’ knowledge and experience. Accordingly, because of the automated
design of its network structure, Neural architecture search (NAS) has achieved
great success in the image processing field and attracted substantial research
attention in recent years. Although some NAS methods have reduced the number
of GPU search days required to single digits in the image field, directly
using 3D convolution to extend NAS to the video field is still likely to
produce a surge in computing volume. To address this challenge, we propose a
new processing framework called Neural Architecture Search-Temporal
Convolutional (NAS-TC). Our proposed framework is divided into two phases. In
the first phase, the classical CNN network is used as the backbone network to
complete the computationally intensive feature extraction task. In the second
stage, a simple stitching search to the cell is used to complete the
relatively lightweight long-range temporal-dependent information extraction.
This ensures our method will have more reasonable parameter assignments and
can handle minute-level videos. Finally, we conduct sufficient experiments on
multiple benchmark datasets and obtain competitive recognition accuracy.

###### Index Terms:

Neural Architecture Search, long video action recognition, convolution
decomposition.

Figure 1: A complex action example CUCBN in a long video in the data set
Charades [1]. The video sample CUCBN has a duration of 20.88s and a total of
502 video frames, which contain 4 overlapping unit action segments (they are
represented by 4 long rectangles of different colors, and the gray long
rectangles are unlabeled meaningless transitional action clips).

##  I Introduction

Deep learning has achieved unprecedented success in the field of image vision
[2, 3, 4, 5, 6, 7]. The task of recognizing complex actions in long untrimmed
videos [8] is far closer to real-world human vision than short trimmed video
recognition. The study of such tasks is consequently of great practical
significance. However, applying deep learning in the video field has two
obvious challenges. One is that artificially designed models are too complex
and tend to rely heavily on the prior knowledge and experience of researchers.
The other is that the direct use of 3D convolution to extract spatiotemporal
features typically leads to a spike in computational load.

Working in the neural architecture context, we know that the complexity of the
network model in deep learning determines the model’s learning potential [9].
More complex models require larger training sets and are thus more difficult
to train; moreover, models that are too simple often fail to achieve the
desired performance due to insufficient learning potential. Designing a good
and moderately complex network architecture therefore requires a lot of effort
from researchers, and is also difficult to judge artificially. The successful
application of NAS in the image vision field has provided researchers with new
possibilities to avoid the manual design of network architectures [10, 11, 12,
13, 14].

Unfortunately, although previous NAS algorithms based on reinforcement
learning (RL) [10] achieve excellent performance, the computational cost
(around 2000 GPU days) is prohibitive for most researchers. The computational
load in one framework search employing evolutionary algorithms reached 3150
GPU days [11]. Therefore, a large number of outstanding works have focused on
how to reduce the amount of calculation and accelerate the architecture search
process [15]. In addition, there are architecture search methods based on MCTS
[16], SMBO [17] or Bayesian optimization [18]. However, these methods treat
architecture search as a black-box optimization problem on discrete domains,
which makes it extremely difficult to optimize the model. Fortunately, a new
differentiable architecture search method was proposed in Differentiable
architecture search (DARTS) [19]. These authors relaxed the discrete search
space to make it continuous, then used gradient descent to optimize the model;
in this way, the amount of calculation required by the architecture search can
be greatly reduced so that ordinary researchers can feel the charm of
architecture search. This research thus rapidly attracted the interest of a
large number of researchers [20, 21].

(a) Search space

(b) SepConv_$k$* and DilConv_$k$*

Figure 2: Schematic diagram of search space and unit operations. (a) search
space. Here, $node_{i}$ and $node_{j}$ are the $i$-th and $j$-th nodes in the
cell. The nine operations in the middle are the search space of the NAS. The
operations in the search space are all one-dimensional operations on
$\mathcal{T}$ (if applicable). (b) The execution details of the unit
operations SepConv_$k$* and DilConv_$k$ in the search space. $C$ is the number
of channels of the current feature.

This paper focuses on the recognition of complex actions in long videos.
Complex actions of this kind often last for a long time and include multiple
unit actions. These unit-actions not only vary greatly in terms of their
temporal order, but also have uneven duration. And these unit actions overlap
each other on the timeline. In Fig.1, we show a complex action example CUCBN
in a long video in the data set Charades [1]. For this reason, many innovative
and complex neural architectures have been proposed and have achieved quite
good performance [22, 23, 24, 25, 26, 27, 28, 29]. However, while neural
architecture design in the video field still faces similar challenges as are
encountered in the image vision field, it is not possible to simply translate
architecture search techniques that were successful in the image field to the
video field with its complex action recognition. That is because the
application of 3D convolution in the video field is more complicated than is
the case for the 2D image vision task. Moreover, the image size of frames in
the video field is also larger than in image vision tasks. In addition, the
large amount of calculation required makes it difficult for the architecture
search to complete the corresponding tasks while also using 3D convolution to
extract the spatiotemporal features. To reduce the computational load,
researchers began to consider decomposing the 3D convolutions. For example,
CoST [30] proposed decomposing the 3D convolutions into three 2D convolutions;
moreover, S3D [31] proposed using 2+1D convolutions rather than traditional 3D
convolutions, while Timeception [22] solves spatiotemporal convolutions into
depthwise-separable temporal convolutions.

It is worth mentioning that Timeception [22] uses CNN as the model’s backbone
network. This backbone network is responsible for extracting the
spatiotemporal features. In this case, it completes the task of extracting the
temporal features of complex actions in long videos by stacking carefully
designed Timeception layers. This approach turns the original complex backbone
network into a preprocessing model for spatiotemporal feature extraction.
Subsequent lightweight temporal feature extraction tasks offer the possibility
of using NAS.

In our proposed method, we also use CNN as the backbone network. The
difference is that, in our case, the design of the Timeception layer is done
through NAS. This makes our model able to get rid of the limitations of manual
design to a certain extent. The contributions of this paper can be briefly
described as follows:

  * •

We propose a two-phase framework specifically designed for complex action
recognition in long video. This structure can allocate the parameters of the
model appropriately. The amount of parameters decreased by 36% (when TC Layers
is equal to 6).

  * •

We propose an automated architecture search method for video action
recognition, which successfully extended DARTS to the video field and form a
new approach, Temp-DARTS.

  * •

We design a special search space for video action recognition, which allows
our method to better learn the temporal patterns in minute-level long videos.

  * •

We conduct sufficient experiments on a benchmark dataset to verify the
effectiveness of the proposed model. The experimental performance is
comparable to that of the most advanced algorithms.

The rest of this article is organized as follows. In Section II, some related
work before introducing the NAS-TC method is introduced. Then, we discuss the
composition details of the NAS-TC overall framework in Section III. In Section
IV, we compare the performance of the proposed NAS-TC with the latest related
methods on three real long video data sets. Finally, the concluding remarks
are placed in Section V.

TABLE I: Search space operation information. $C$ is the number of channels of
the current feature.

$\mathcal{O}$ |  kernel_size ($k$) |  dilation ($d$) |  groups ($g$) |  padding ($p$)
---|---|---|---|---
Identity | - | - | - | -
Zero | - | - | - | -
Avg_pool_2 | (2,1,1) | - | - | (1,0,0)
Max_pool_2 | (2,1,1) | - | - | (1,0,0)
DilConv_$k$3 | (3,1,1) | 2 | $C$ | (2,0,0)
DilConv_$k$5 | (5,1,1) | 2 | $C$ | (4,0,0)
SepConv_$k$3 | (3,1,1) | 1 | - | (1,0,0)
SepConv_$k$5 | (5,1,1) | 1 | - | (2,0,0)
SepConv_$k$7 | (7,1,1) | 1 | - | (3,0,0)

##  II Related Works

###  II-A Neural Architecture Search

Neural architecture search is regarded as a revolution in neural network
architecture design. It is expected to break through the limitations of human
experience and thinking mode, so NAS is given high hopes by researchers [12].
Previous NAS work based on reinforcement learning [32, 10, 33] and
evolutionary learning [34, 11] is often too time-consuming and requires a lot
of computing power, so NAS cannot be widely used.

The emergence of DARTS [19] has greatly improved this situation. It uses the
popular cell-based [10] search space and forms the final network architecture
by searching for two different types of cells (normal cell and reduction cell)
and stacking them alternately. In particular, DARTS serializes candidate
operations that are discretized between pairs of nodes in the cell template.
Therefore, the gradient descent method is used to solve the problem of
selecting candidate operations between node pairs. The continuation strategy
of candidate operations between this pair of nodes can be expressed as
follows:

$x_{j}=\sum_{o\in\mathcal{O}}\frac{exp(\alpha^{(i,j)}_{o})}{\sum_{o^{\prime}\in\mathcal{O}}exp(\alpha^{(i,j)}_{o^{\prime}})}x_{i},$
(1)

where $\mathcal{O}$ is the operation set in Fig. 2a, while
$\alpha_{o}^{(i,j)}$ sents the weight of operation $o$ on directed edge
$e^{(i,j)}$. The discretization of the mixing operation between node pairs can
be expressed as follows:

$o^{(i,j)}=argmax_{o\in\mathcal{O}}\alpha_{o}^{(i,j)}.$ (2)

Subsequently, a large number of improvements related to DARTS were proposed
[35, 36, 37]. Although DARTS greatly reduces the time cost of cell search, and
has achieved success in the field of image vision. However, it is still
difficult to directly extend DARTS to the video field.

###  II-B Complex Action Recognition

The biggest difference between image vision task and video tasks is the
information in the temporal dimension. Therefore, video tasks are more
complicated than image vision tasks. Video tasks require additional modeling
of temporal dimensions. Commonly used modeling methods include: statistical
pooling [38, 39, 40, 41], vector aggregation [42, 43, 43], and neural network-
based methods [44, 45, 46]. Statistical pooling and vector aggregation ignore
the time cues in the video task, and the above-mentioned neural network-based
model is also extremely limited for information mining in the temporal
dimension.

Subsequently, some short-range action recognition models based on neural
networks were proposed. For example, the Two-stream convolutional network [47]
uses OpticalFlow as auxiliary information for RGB images. [48] uses dynamic
graphs for action recognition. C3D [49, 50] and I3D [51] implement short-range
temporal information learning by expanding 2D convolution. These methods are
often computationally expensive and cannot effectively learn temporal
information in long video clips.

To enable the model to learn the complex temporal patterns in long videos, TRN
[52] studies the temporal dependence between different video frames on
multiple temporal scales; TSN [53, 54] divides the long video into video
segments, and integrates the temporal patterns of different video segments to
make video-level predictions; Non-local networks [55] introduces the attention
mechanism to the learning of temporal patterns of long videos. There are many
researches related to long video complex action recognition, such as [56, 57].
However, the above method can successfully model 128 timesteps at max [22].
Timeception [22] successfully modeled 1024 timesteps by carefully designing a
Temporal Conv Module for the temporal dimension. However, the manually
designed model is most likely not optimal. The design of an automated Temporal
Conv Module is a problem worthy of study.

##  III Method

###  III-A Motivation

####  III-A1 The Predicament of NAS in the Video Field

Although DARTS [19] was able to reduce the model search cost from thousands of
GPU days to single digits, it is still very difficult to require DARTS to
learn spatiotemporal information directly from video data using 3D
convolution. In addition to the difficulty of coping with the surge in
computational volume, it is also difficult to fully learn the temporal
information in long videos.

####  III-A2 Individual Convolutions on Temporal Information

For video information, the 3D CNN learns the spatiotemporal kernel through
three orthogonal subspaces, specifically the temporal ($\mathcal{T}$), spatial
($\mathcal{S}$) and semantic channel subspace ($\mathcal{C}$) [22]. In
previous models, 3D convolution was often considered to convolve temporal and
spatial information simultaneously [58]), and has also been considered for
decomposing 3D convolutions into 2+1D convolutions [31]. However, the time
information processing ability of these methods is insufficient, leading to
these methods being unable to handle minute-level long videos. The success of
Timeception [22] illustrates to some extent the effectiveness of the
convolution of temporal information alone. This approach can also alleviate
the problem of the surge in computation to some extent.

####  III-A3 A Simple and Effective Implementation

An intuitive and effective idea would be to extract the spatiotemporal
features using the classic CNN as the backbone network of the model. The NAS
method is then used to complete the model design of lightweight tasks for long
video temporal information extraction. In this way, the original heavy video
task can be broken down into two parts: the computationally expensive
spatiotemporal feature extraction task becomes data preprocessing, while the
lightweight design of the long-term feature extraction task is left to NAS.

Figure 3: A network structure diagram with multiple stacked cells. Each cell
has two inputs and one output. The $cell_{k}$ has two input nodes $I_{k-1}$
and $I_{k-2}$, which are the outputs of the two previous cells. $I_{k}$ is the
output of the $k$-th cell, $cell_{k}$. There are four intermediate nodes
($B_{0},B_{1},B_{2}$ and $B_{3}$) inside each cell, which are used to perform
feature extraction and processing tasks. Finally, the features of the four
intermediate nodes are concatenated to obtain the output $I_{k}$ of
$cell_{k}$.

###  III-B NAS-TC: Neural Architecture Search Temporal Convolutional

In this section, we will briefly introduce the basic components of NAS, then
focus on the construction of NAS-TC. Some settings in our model are similar to
previous NAS work [19].

####  III-B1 The Basic Components of NAS-TC

Search Space. As illustrated in Fig. 2a, $node_{i}$ and $node_{j}$ are the
$i$-th and $j$-th nodes in the cell. These nodes represent a latent
representation, such as a feature map in a convolutional network. The nine
operations in the middle are the search space $\mathcal{O}$ of the NAS. All
operations in the search space are one-dimensional operations on $\mathcal{T}$
(if applicable). The search space $\mathcal{O}$ contains $3\times 1\times 1$,
$5\times 1\times 1$ and $7\times 1\times 1$ separable convolutions, $3\times
1\times 1$ and $5\times 1\times 1$ dilated separable convolutions, $2\times
1\times 1$ max pooling, $2\times 1\times 1$ average pooling, identity, and
$zero$. All operations have $stride=1$ (if applicable). Moreover, proper
padding is used to keep the feature map in its original dimension for the
feature map stitching. The order of the convolution operations used here is
ReLU-Conv-BN, and the specific schematic diagram is shown in Fig 2b.
Information on the operation of the search space is summarized in Table I.

Cell Structure. The cell settings in our model are consistent with those in
DARTS [19]. Fig. 3 presents a network structure diagram using multiple stacked
cells. The cell needs to be obtained using the NAS search. Each cell has two
inputs and one output. The $cell_{k}$ has two input nodes, $I_{k-1}$ and
$I_{k-2}$, which are the outputs of the two previous cells. $I_{k}$ is the
output of the $k$-th cell, $cell_{k}$. There are also four intermediate nodes
($B_{0},B_{1},B_{2}$ and $B_{3}$) inside each cell, which are used to perform
feature extraction and processing tasks. Finally, the features of the four
intermediate nodes are concatenated to obtain the output $I_{k}$ of the
$cell_{k}$.

The directed edges $(i,j)$ between every two nodes $B_{i}$, $B_{j}$ represent
the operations $o_{i,j}$. During the training phase, each intermediate node is
calculated from all previous nodes as follows: $B_{j}=\sum_{i<j}o_{i,j}\times
B_{i}$. The original discretization operation is relaxed into a continuous
state, after which the gradient descent method is used to learn the structure
of the network model cell. During the testing phase, each node retains only
the two operations with the highest weight among the associated operations.
This can reduce the total amount of calculation, as well as the requirements
for GPU memory to a certain extent.

####  III-B2 Temp-DARTS

In our method, we extend DARTS [19] to the temporal subspace. Since this
method mainly involves searching for suitable cells in the temporal subspace,
we refer to it as Temp-DARTS. In order to enable Temp-DARTS to be applied
efficiently and conveniently in the video field, we have specially designed a
new search space for it. The search space, $\mathcal{O}$, is shown in Fig. 2a,
while the inflated details for basic operations in the search space are shown
in Table I. Since the searched cells perform corresponding operations only in
the temporal dimension, we refer to the searched cells as NAS-Temp. The nodes
and edges in NAS-Temp are consistent with the corresponding meanings in
Figures 2a and 3. Moreover, Fig. 4b shows the learned cell structure on the
operation set $\mathcal{O}$ using Temp-DARTS on the Charades [1] dataset.

It should be noted that the DARTS search on the two-dimensional image begins
from the most original image and needs to search for two cell structures: a
normal cell and a reduction cell. Normal cells are primarily used to extract
high-level features, while reduction cells are used to reduce the feature map
size. Unlike two-dimensional DARTS, however, Temp-DARTS only need to search
for a normal cell. This is mainly because the spatial dimension of the feature
map obtained following processing by the backbone network has been reduced to
a comparatively small level; thus, the spatial dimension of the feature map on
the subsequent NAS-TC layer is kept unchanged.

(a) The overall framework of NAS-TC network.

(b) NAS-Temp

Figure 4: NAS-TC network and NAS-Temp. (a) We use I3D as the backbone network
for the model; accordingly, feature extraction is performed on the sampled
video using I3D. Multiple NAS-TC layers are then simply stacked and connected
to the classifier to complete the final classification task. (b) Cell learned
on Charades: NAS-Temp. We let $T=32$, sample on Charades, and extract features
using I3D. The extracted features are then used as the input of Temp-DARTS,
which enables us to find the best cell structure.

####  III-B3 Overall Framework

The overall framework of our model is illustrated in Fig. 4a. The overall
framework of NAS-TC is composed of a backbone network and multiple NAS-TC
layers stacked.

Backbone Network. For a video sample of length $t$, we first divide $t$ into
$T$ segments, then take eight consecutive frames as a superframe for each
segment. We then input the extracted $T$ superframes together into the
backbone network I3D [58] for feature extraction. We use the feature map of
mixed-5c, which is the last spatiotemporal convolution layer of I3D, as the
input to the subsequent network. That is, the original video sample $3\times
8T\times 224\times 224$ uses I3D to perform feature extraction in order to
obtain the feature map of $C\times T\times H\times W$; here, $C=1024,H=W=7$.

NAS-TC Layer. On the dataset Charades, we let $T=32$, and then extract
features through I3D. We use the operation in Fig. 2a as the search space for
Temp-DARTS, while the extracted features become the input for Temp-DARTS. We
next search the structure of the cell on the Charades training set. The
structure of the searched cell NAS-Temp is illustrated in Fig. 4b. It should
be pointed out that although the operations used in the NAS-Temp searched in
Fig. 4b do not contain all single operations in the operation set, this does
not mean that single operations not appearing in the search space can be
directly removed from the operation set. After repeating the cell architecture
search experiment under different initializations, we found that the final
cells contained different unit operations, but had similar performance. This
means that different unit operation combinations in the operation set may have
similar learning capabilities; that is, there may be more than one optimal
network structure, which is in line with what common sense suggests.
Therefore, these unit operations are indispensable in cell structure search
research.

In the NAS-TC layer, the feature map $X$ extracted by I3D is divided into $N$
channel groups. Each group feature $x_{i}\in R^{\frac{C}{N}\times T\times
H\times W}$ is then used as the input of NAS-Temp. To reduce the number of
parameters, the channel shuffle operation [59] is then performed on the
obtained features of the $N$ NAS-Temps, enabling the learning of information
across the channels. $2\times 1\times 1$ max pooling is performed on
$\mathcal{T}$ at the end of the NAS-TC layer. The dimension of the obtained
feature is $\frac{SC}{M}\times\frac{T}{2}\times H\times W$. Here, $S=4$, which
is the number of intermediate nodes in the NAS-Temp cell, while
$M=3;\frac{S}{M}$ is the scaling ratio of the channel. This is in accordance
with the design principle of incremental information extraction and
corresponding feature dimension halving [60]. Subsequently, multiple NAS-TC
layers are simply stacked and connected to the classifier, enabling the final
classification task to be completed.

####  III-B4 Implementation Details

Cells are searched on Charades’ training set by Temp-DARTS, with the number of
epochs set to 50. When using the backbone network for feature extraction, we
apply the same settings as in [22] to perform pre-training on a specific
dataset, which allows us to obtain high-quality features. The pre-training
weights are publicly
available111https://github.com/piergiaj/pytorch-i3d/blob/master/models/
rgb_charades.pt and https://github.com/hassony2/kinetics_i3d_
pytorch/blob/master/model/model_rgb.pth. After obtaining the corresponding
features, we insert multiple NAS-TC and MLP layers on top of the backbone
network to achieve the final classification. Throughout the entire training
process, only the parameters inserted in this part are trained. The model is
trained with a batch size of 18 for 300 epoch. We here use the Adam optimizer,
where $lr=0.01$ and $eps=1e-4$.

(a) Temporal Conv Module.

(b) Expanded view of NAS-Temp in Fig. 4b.

Figure 5: Comparison of NAS-Temp and Temporal Conv Module. (a) The Temporal
Conv Module is a multi-scale convolution operation similar to Inception in
Timeception; the difference is that the Temporal Conv Module only performs
convolution on the temporal dimension. The Temporal Conv Module was reproduced
from (Hussein et al. 2019). (b) An expanded view of the four intermediate
nodes of NAS-Temp in Fig. 4b. The corresponding nodes of the four intermediate
nodes ($B_{0},B_{1},B_{2}$ and $B_{3}$) in the expanded view are
($B_{00},B_{11},B_{22}$ and $B_{33}$). For $B_{ij}$, $i$ represents the $i$-th
intermediate node in NAS-Temp, while $j$ represents that the node is
associated with the generated $j$-th node. The input and output nodes
$I_{k-2},I_{k-1}$ and $I_{k}$ remain the same as in Fig. 4b.

###  III-C Parameter Analysis

In this section, we will focus on the rationality of the parameter allocation
of NAS-TC and Timeception.

We present an expanded view of the NAS-Temp and the structure of the Temporal
Conv Module in Fig. 5. More specifically, Fig. 5b is an expanded view of the
four intermediate nodes generated in NAS-Temp in Fig. 4b. The nodes
corresponding to the four intermediate nodes ($B_{0},B_{1},B_{2}$ and $B_{3}$)
in the expanded view are ($B_{00},B_{11},B_{22}$ and $B_{33}$). For node
$B_{ij}$, $i$ represents the $i$-th intermediate node in NAS-Temp, while $j$
represents that the node is associated with the generated $j$-th node. The
input nodes $I_{k-2}$ and $I_{k-1}$ and the output node $I_{k}$ remain the
same as in Fig. 4b. Fig. 5a illustrates a core component of Timeception,
namely the Temporal Conv Module. This module is a multi-scale convolution
operation similar to Inception [61] in Timeception. The difference is that the
Temporal Conv Module only performs convolution on the temporal dimension. The
Temporal Conv Module is reproduced from [22]. As shown in Fig. 5a, this module
has five branches. More specifically, the fifth branch only performs a
dimensionality reduction operation on the input using $1\times 1$ 2D
convolution in the spatial dimension, while the remaining four branches
perform either multi-scale convolution operations or pooling operations on the
temporal dimension. In order to facilitate the management of the output
dimension, the input is reduced by $M$ times using a $1\times 1$ 2D
convolution. After the channels are reduced, the outputs of the five branches
are concatenated across the channel dimensions as the output of the final
Temporal Conv Module. For more details, please see [22].

In the Temporal Conv Module, although the $1\times 1$ 2D convolution seems to
occupy very little in the way of a parameter budget, this is not actually the
case. In fact, the $1\times 1$ 2D convolution requires a large number of
parameters in each branch. More specifically, in the Temporal Conv Module, the
$1\times 1$ 2D convolution takes up 57% of the parameters in the five
branches. These parameters are mainly used to reduce the number of channels of
the feature map. At the same time, these convolution operations are also used
to increase the model complexity, as this improves its learning potential.
However, in fact, only the convolution operations ($kernel\\_size=3,5,7$) on
the temporal dimension in three of the branches are actually used to learn
advanced features on the temporal dimension. Therefore, after the above
analysis, we have reason to be concerned about the rationality of this
parameter allocation.

By contrast, in NAS-Temp, we perform the necessary $1\times 1$ 2D convolution
on only two inputs, specifically, $I_{k-1}$ and $I_{k-2}$. The number of
channels of the two inputs $I_{k-1}$ and $I_{k-2}$ is compressed to one-third
of $I_{k-1}$ by means of a $1\times 1$ 2D convolution. In the four
intermediate nodes ($B_{0},B_{1},B_{2},B_{3}$), the number of channels is kept
unchanged. From the expanded view of NAS-Temp in Fig. 5b, we can clearly see
how these intermediate nodes are reused. However, this generation method makes
the generation of each intermediate node very complicated. For example, the
number of unit operations performed in order to generate
$B_{00},B_{11},B_{22}$ and $B_{33}$ is 2, 4, 6 and 8, respectively. Compared
with the Temporal Conv Module, this connection method makes the $1\times 1$ 2D
convolution in the entire NAS-Temp layer significantly reduce the proportion
of parameters. Moreover, because NAS-Temp stacks and stitches different
convolution operations, the information it learns on the temporal dimension is
also more complicated. This strategy gives NAS-Temp the possibility to learn
more from a temporal dimension. This is in line with the principle of subspace
efficiency [22], which involves allocating a higher parameter budget to the
main difficulty of the current task; that is, learning information on
$\mathcal{T}$.

Therefore, our model is better at allocating the limited parameters to
learning information on $\mathcal{T}$, thereby achieving better performance.
In addition, this effectively reduces the number of parameters in the NAS-Temp
layer. We provide a more detailed parameter comparison in the Parameters and
Search Time section.

TABLE II: Details of the datasets Charades, MultiTHUMOS and BreakfastActions datasets. Datasets | Total videos |  Total time (hrs) | Average duration |  Actions per video | Classes | Type
---|---|---|---|---|---|---
[1] Charades | 9848 | 82 | 30 sec | 6 | 157 | Daily activities
[62] MultiTHUMOS | 400 | 30 | 4.5 min | 11 | 65 | Sports
[8] BreakfastActions | 1712 | 66 | 2.3 min | 7 | 48 | Cooking

##  IV Experiments

###  IV-A Datasets

In order to verify the model’s recognition performance on complex actions in
long videos, we choose to test the performance of the model on the datasets
Charades222https://prior.allenai.org/projects/charades [1],
MultiTHUMOS333http://ai.stanford.edu/$\sim$syyeung/everymoment.html [62] and
BreakfastAction444https://serre-lab.clps.brown.edu/resource/breakfast-actions-
dataset/ [8] from the perspective of the video action composition, the
temporal extent and the temporal order. Those datasets whose video length is
less than half a minute or even shorter are not included in the scope of this
paper. The details of the Charades, MultiTHUMOS and BreakfastActions datasets
are summarized in Table II.

Charades is a large-scale multi-label video classification dataset. It
contains a large number of human daily activities. This dataset contains 9848
videos with a total of 157 categories, including an 8k training set (67 hrs)
and a 1.8k validation set. Each video is up to 30 seconds in length and
contains 6.8 unit-actions, which meets the requirements of long video complex
action recognition tasks. It is therefore a very challenging benchmark
dataset. For comparison, we use the same evaluation index as [55, 22], namely
mean Average Precision (mAP) [63].

MultiTHUMOS is extended from the original THUMOS-14 dataset [64]. It is also
an untrimmed human activity multi-class dataset, containing 400 videos with 65
action classes for a total of 30 hours. It is often used for temporal
positioning in previous work; the difference is that we use it to complete the
complex action recognition task of a long video. Compared to the Charades
dataset, the actions in each video of this dataset are more complicated; on
average, each video contains 11 unit-actions. As with Charades, we also use
mAP as the evaluation index for MultiTHUMOS.

BreakfastActions is by far one of the largest fully annotated long video
action datasets available. This dataset contains 10 actions related to
breakfast preparation, recorded by 52 people in 18 different kitchens. It
contains 1712 videos in total; the size of the training set and the test set
are 1357 and 335 respectively, while the average video length is 2.3 minutes.
Each video represents an activity consisting of a series of unit-actions.
There are a total of 48 types of unit-actions. Similar to [22], we use single-
label activity annotations to calculate accuracy (Acc), and multi-label unit-
action annotation to calculate mAP.

TABLE III: Comparison results of Timeception and NAS-TC under different timesteps inputs. We use I3D as a common backbone network for Timeception and NAS-TC. We insert multiple layers of TC (Timeception or NAS-TC layers) on top of I3D to form their respective networks. CNN_Steps: input timesteps to the CNN; TC_Steps: input timesteps to the first TC layer; Params: number of parameters used by TC layers, in millions. |  CNN_Steps $\Rightarrow$ TC_Steps | Timeception | NAS-TC
---|---|---|---
Params | mAP(%) | Params | mAP(%)
\+ 3 TC | 256 $\Rightarrow$ 32 | 2.0 | 33.9 | 1.5 | 34.5
\+ 3 TC | 512 $\Rightarrow$ 64 | 2.0 | 35.5 | 1.5 | 37.2
\+ 4 TC | 1024 $\Rightarrow$ 128 | 2.8 | 37.2 | 2.0 | 39.3

###  IV-B Experiments on Charades

####  IV-B1 Comparison of Long-range Temporal Dependencies Learning Ability

A performance comparison is conducted of Timeception and NAS-TC under
different timestep inputs. By changing the number of timesteps inputs, we can
observe the model’s learning ability on long-range temporal dependencies. We
use I3D as a common backbone network for Timeception and NAS-TC. We insert
multiple layers of TC (i.e., Timeception or NAS-TC layers) on top of I3D to
form their respective networks. CNN_Steps is the input timesteps of CNN (i.e.,
I3D), while TC_Steps is the input timesteps of the first TC layer; moreover,
Params is the number of parameters used by TC layers (in millions). The
timesteps input into the backbone network is
$\textit{CNN\\_Steps}\in\\{256,512,1024\\}$; accordingly, the timesteps input
to the first TC layer is $\textit{TC\\_Steps}\in\\{32,64,128\\}$.

From Table III, it can be seen that when CNN_Steps gradually increases, the
performance of Timeception and NAS-TC increases monotonically. This reveals
that our NAS-TC and Timeception methods have been able to learn long-range
temporal information to a certain extent. However, in contrast, our method
uses fewer parameters and achieves better recognition performance. This
performance gain also becomes more obvious with the increase of CNN_Steps.
This is mainly because the NAS-Temp structure learned through Temp-DARTS is
able to handle long-range temporal information more effectively.

####  IV-B2 Performance Comparison

To test the validity of our model, in this section, we compare the performance
of NAT-TC with a number of related state-of-the-art methods. These methods are
as follows: Two-stream [56], Two-stream + LSTM [56], ActionVLAD [65], Temporal
Fields [56], Temporal Relations [52], ResNet-152 [56], ResNet-152 +
Timeception [22], 3D ResNet-101 [55], I3D [58], I3D + ActionVLAD [65], I3D +
Timeception [22], I3D-NL [55], EvaNet [66] and I3D + VideoGraph [67]. The
results of this performance comparison are shown in Table IV. The experimental
results reveal that our method can more effectively use the features learned
by the original backbone network I3D. Moreover, compared with the artificially
designed I3D + Timeception network (which is the most structurally similar),
our model obtains more competitive recognition accuracy.

TABLE IV: Performance comparison of different methods on the Charades dataset. Compared with other methods that use the same input (that is, the backbone network is I3D), I3D + NAS-TC has obvious advantages. Method | Modality | mAP(%)
---|---|---
[56] Two-stream | RGB + Flow | 18.6
[56] Two-stream + LSTM | RGB + Flow | 17.8
[65] ActionVLAD | RGB + iDT | 21.0
[56] Temporal Fields | RGB + Flow | 22.4
[52] Temporal Relations | RGB | 25.2
[56] ResNet-152 | RGB | 22.8
[22] ResNet-152 + Timeception | RGB | 31.6
[55] 3D ResNet-101 | RGB | 35.5
[58] I3D | RGB | 32.9
[65] I3D + ActionVLAD | RGB | 35.4
[22] I3D + Timeception | RGB | 37.2
[55] I3D-NL | RGB | 37.5
[67] I3D + VideoGraph | RGB | 37.8
[66] EvaNet | RGB | 38.1
I3D + NAS-TC | RGB | 39.3

####  IV-B3 Parameters and Search Time

Because the architecture of Timeception [22] is the most similar to that of
our model NAS-TC, in this section, we will mainly compare the changes in
parameters between them as the TC layer increases. Secondly, we also report
the search time of the NAS-Temp cell.

We plot the changes in the parameters of NAS-TC and Timeception with the
increase of TC layer in Fig. 6. When $TC\\_Layers=0$, the model has only the
top fully connected layer, and the number of parameters is 0.61 million.
Clearly, NAS-TC has significantly fewer parameters than Timeception; as the
number of TC layers increases, this advantage becomes more obvious. We
analyzed this in detail in the Parameter Analysis section. In terms of
performance, our model also exhibits obvious advantages, as can be seen from
the experimental results in Table III. In theory, the TC layer can also be
stacked with more layers. When $TC\\_Layers=8$, NAS-TC can process 4096
timesteps on the temporal dimension; this means that NAS-TC can handle longer
videos of up to 2.3 minutes in length. Therefore, the advantages of NAS-TC
(i.e. the need for fewer parameters) will be further manifested. It should be
pointed out here that this does not mean that a larger number of TC layers is
always better. A large TC number means a larger frame sample; since the video
length is fixed, overlarge frame sampling may cause serious information
redundancy problems. To facilitate comparison, including the next two
datasets, we keep the number of TC layers the same for Timeception.

We use Temp-DARTS to search for cells on Charades’ training set. The 50 epochs
take only 2.1 GPU days, which is similar to the search time achieved by DARTS
in the image field. This demonstrates that it is feasible to reduce the amount
of calculation by performing convolution operations only in the temporal
dimension.

Figure 6: As the number of TC layers changes, the parameters of NAS-TC and
Timeception are compared. Obviously, with the same number of TC layers, NAS-TC
has fewer parameters.

###  IV-C Experiments on MultiTHUMOS

To further verify the scalability of our model, we also test its performance
on the MultiTHUMOS dataset. We use a processing method similar to that used on
the Charades dataset , and compare the performance of the original I3D,
Timeception, and other related models with our model. The training method is
kept as consistent as possible with that of Timeception. The final
experimental results are reported in Table V. From the experimental results,
it is clear that multi-scale convolution is significantly better than a fixed
single scale; because it is more flexible, it can more easily capture the
changing temporal information in complex actions. Moreover, the I3D +
Timeception method is also a multi-scale convolution operation; by comparison,
because our method automatically searches for cells through NAS, our model has
more flexible temporal information capture capabilities. Our model is
therefore able to obtain relatively good recognition accuracy.

TABLE V: Performance comparison of different methods on the MultiTHUMOS dataset. Clearly, multi-scale convolution is significantly better than a fixed single scale. NAS-TC and Timeception both use multi-scale kernels; of these two methods, NAS-TC has better accuracy. Method |  Kernel Size $k$ |  Dilation Rate $d$ | mAP(%)
---|---|---|---
[62] Two-stream | - | - | 27.60
[62] Two-stream + LSTM | - | - | 28.10
[62] Multi-LSTM | - | - | 29.60
[68] Predictive-corrective | - | - | 29.70
[69] SSN | - | - | 30.30
[70] I3D + super-events | - | - | 36.40
[71] I3D + super-events + TGMs | - | - | 46.40
[22] I3D | - | - | 72.43
[22] I3D + Timeception | 3 | 1 | 72.83
[22] I3D + Timeception | 3 | 1,2,3 | 74.52
[22] I3D + Timeception | 1,3,5,7 | 1 | 74.79
I3D + NAS-TC | 3,5,7 | 1,2 | 76.83

###  IV-D Experiments on BreakfastAction

We also test the performance of our model on the BreakfastAction dataset. For
ease of comparison, we use the same settings as in [22] of which uniformly
sample 64 segments from each video, each of which contains eight successive
frames, and then use a model of three stacked layers of NAS-TC layers to model
the time dimension. The performance comparison of the models is presented in
Table VI. It can be seen from the experimental results that the performance of
NAS-TC still maintains corresponding advantages.

TABLE VI: Performance comparison between NAS-TC and other related methods on the single-label classification of activities and multi-label classification of unit-actions on the BreakfastAction dataset. Method |  Activities (Acc. %) |  Actions (mAP %)
---|---|---
[56] ResNet-152 | 41.13 | 32.65
[65] ResNet-152 + ActionVLAD | 55.49 | 47.12
[22] ResNet-152 + Timeception | 57.75 | 48.47
[67] ResNet-152 + VideoGraph | 59.12 | 49.38
[58] I3D | 58.61 | 47.05
[65] I3D + ActionVLAD | 65.48 | 60.20
[22] I3D + Timeception | 67.07 | 61.82
[67] I3D + VideoGraph | 69.45 | 63.14
I3D + NAS-TC | 71.89 | 64.64

##  V Conclusion

In this paper, a Neural Architecture Search Temporal Convolutional (NAS-TC)
layer on the temporal dimension is proposed. We design a special search space
for NAS, extend DARTS to Temp-DARTS, and design a two-phase framework for NAS
applications in the long video context. In NAS-Temp, the repeated use of
intermediate nodes gives the model sufficient ability to learn long-range
temporal dependencies. In addition, the parameter assignment of the model
follows the principle of subspace efficiency effectively. This allows NAS-TC
to still achieve good performance using relatively few parameters. NAS-TC also
maintains the flexible automation advantages of NAS. Our method is also very
scalable and provides a good reference for solving similar problems in other
fields.

## Acknowledgment

This work was partially supported by the NSFC under Grant (No.61972315 and No.
62072372), the Shaanxi Science and Technology Innovation Team Support Project
under grant agreement (No.2018TD-026) and the Australian Research Council
Discovery Early Career Researcher Award (No.DE190100626).

## References

  * [1] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” in _ECCV_. Springer, 2016, pp. 510–526.
  * [2] Z. Chen, H. Zhou, J. Lai, L. Yang, and X. Xie, “Contour-aware loss: Boundary-aware learning for salient object segmentation,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 431–443, 2020.
  * [3] X. Chen, X. Zheng, and X. Lu, “Bidirectional interaction network for person re-identification,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 1935–1948, 2021.
  * [4] D. Das and C. G. Lee, “A two-stage approach to few-shot learning for image recognition,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 3336–3350, 2019.
  * [5] S. Zhao and B. Zhang, “Learning complete and discriminative direction pattern for robust palmprint recognition,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 1001–1014, 2020.
  * [6] X. Chang, P.-Y. Huang, Y.-D. Shen, X. Liang, Y. Yang, and A. G. Hauptmann, “Rcaa: Relational context-aware agents for person search,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 84–100.
  * [7] F. Zhu, Y. Zhu, X. Chang, and X. Liang, “Vision-language navigation with self-supervised auxiliary reasoning tasks,” in _CVPR_ , 2020.
  * [8] H. Kuehne, A. Arslan, and T. Serre, “The language of actions: Recovering the syntax and semantics of goal-directed human activities,” in _CVPR_ , 2014, pp. 780–787.
  * [9] X. Chang, Y. Yu, Y. Yang, and E. P. Xing, “They are not equally reliable: Semantic event search using differentiated concept classifiers,” in _CVPR_ , 2016.
  * [10] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in _CVPR_ , 2018, pp. 8697–8710.
  * [11] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” in _Proceedings of the aaai conference on artificial intelligence_. AAAI Press, 2019, pp. 4780–4789.
  * [12] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, X. Chen, and X. Wang, “A comprehensive survey of neural architecture search: challenges and solutions,” _ACM Computing Surveys_ , 2020.
  * [13] J. Liu, S. Zhou, Y. Wu, K. Chen, W. Ouyang, and D. Xu, “Block proposal neural architecture search,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 15–25, 2020.
  * [14] X. Cheng, Y. Zhong, M. Harandi, Y. Dai, X. Chang, H. Li, T. Drummond, and Z. Ge, “Hierarchical neural architecture search for deep stereo matching,” in _NeurIPS_ , 2020.
  * [15] H. Cai, T. Chen, W. Zhang, Y. Yu, and J. Wang, “Efficient architecture search by network transformation,” in _AAAI_ , 2018, pp. 2787–2794.
  * [16] R. Negrinho and G. J. Gordon, “Deeparchitect: Automatically designing and training deep architectures,” _CoRR_ , vol. abs/1704.08792, 2017.
  * [17] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in _ECCV_ , 2018, pp. 19–34.
  * [18] K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing, “Neural architecture search with bayesian optimisation and optimal transport,” in _NIPS_ , 2018, pp. 2016–2025.
  * [19] H. Liu, K. Simonyan, and Y. Yang, “DARTS: differentiable architecture search,” in _ICLR_ , 2019.
  * [20] G. Li, G. Qian, I. C. Delgadillo, M. Muller, A. Thabet, and B. Ghanem, “Sgas: Sequential greedy architecture search,” in _Proceedings of the IEEE/CVF Conference on CVPR_ , 2020, pp. 1620–1630.
  * [21] M. Zhang, H. Li, S. Pan, X. Chang, and S. Su, “Overcoming multi-model forgetting in one-shot nas with diversity maximization,” in _Proceedings of the IEEE/CVF Conference on CVPR_ , 2020, pp. 7809–7818.
  * [22] N. Hussein, E. Gavves, and A. W. Smeulders, “Timeception for complex action recognition,” in _CVPR_ , 2019, pp. 254–263.
  * [23] Q. Wan and Y. Choe, “Action recognition and state change prediction in a recipe understanding task using a lightweight neural network model,” in _AAAI_ , vol. 34, 2020, pp. 13 945–13 946.
  * [24] S. Zhang, S. Guo, L. Wang, W. Huang, and M. Scott, “Knowledge integration networks for action recognition.” in _AAAI_ , 2020, pp. 12 862–12 869.
  * [25] X. Chang, F. Nie, Z. Ma, Y. Yang, and X. Zhou, “A convex formulation for spectral shrunk clustering,” in _AAAI_ , 2015.
  * [26] L. Zhu, Z. Huang, X. Chang, J. Song, and H. T. Shen, “Exploring consistent preferences: Discrete hashing with pair-exemplar for scalable landmark search,” in _ACM MM_ , 2017.
  * [27] D. Zhang, L. Yao, K. Chen, S. Wang, X. Chang, and Y. Liu, “Making sense of spatio-temporal preserving representations for eeg-based human intention recognition,” _IEEE Trans. Cybern._ , vol. 50, no. 7, pp. 3033–3044, 2020\.
  * [28] C. Yan, Q. Zheng, X. Chang, M. Luo, C. Yeh, and A. G. Hauptmann, “Semantics-preserving graph propagation for zero-shot object detection,” _IEEE Trans. Image Process._ , vol. 29, pp. 8163–8176, 2020.
  * [29] D. Cheng, Y. Gong, X. Chang, W. Shi, A. G. Hauptmann, and N. Zheng, “Deep feature learning via structured graph laplacian embedding for person re-identification,” _Pattern Recognit._ , vol. 82, pp. 94–104, 2018.
  * [30] C. Li, Q. Zhong, D. Xie, and S. Pu, “Collaborative spatiotemporal feature learning for video action recognition,” in _CVPR_ , 2019, pp. 7872–7881.
  * [31] S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy, “Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification,” in _ECCV_ , 2018, pp. 305–321.
  * [32] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” _arXiv preprint arXiv:1611.01578_ , 2016.
  * [33] H. Cai, J. Yang, W. Zhang, S. Han, and Y. Yu, “Path-level network transformation for efficient architecture search,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 678–687.
  * [34] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, “Large-scale evolution of image classifiers,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 2902–2911.
  * [35] X. Chen, L. Xie, J. Wu, and Q. Tian, “Progressive differentiable architecture search: Bridging the depth gap between search and evaluation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1294–1303.
  * [36] Y. Xu, L. Xie, X. Zhang, X. Chen, G. Qi, Q. Tian, and H. Xiong, “PC-DARTS: partial channel connections for memory-efficient architecture search,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * [37] A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, T. Brox, and F. Hutter, “Understanding and robustifying differentiable architecture search,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * [38] N. Hussein, E. Gavves, and A. W. M. Smeulders, “Unified embedding and metric learning for zero-exemplar event detection,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017, pp. 2087–2096.
  * [39] A. Habibian, T. Mensink, and C. G. M. Snoek, “Video2vec embeddings recognize events when examples are scarce,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 39, no. 10, pp. 2089–2103, 2017.
  * [40] R. Girdhar and D. Ramanan, “Attentional pooling for action recognition,” in _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 34–45.
  * [41] B. Fernando, E. Gavves, J. O. Mogrovejo, A. Ghodrati, and T. Tuytelaars, “Rank pooling for action recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 39, no. 4, pp. 773–787, 2017.
  * [42] D. Oneata, J. J. Verbeek, and C. Schmid, “Action and event recognition with fisher vectors on a compact feature set,” in _IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013_. IEEE Computer Society, 2013, pp. 1817–1824.
  * [43] I. C. Duta, B. Ionescu, K. Aizawa, and N. Sebe, “Spatio-temporal vector of locally max pooled features for action recognition in videos,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017, pp. 3205–3214.
  * [44] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, T. Darrell, and K. Saenko, “Long-term recurrent convolutional networks for visual recognition and description,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_. IEEE Computer Society, 2015, pp. 2625–2634.
  * [45] A. Ghodrati, E. Gavves, and C. Snoek, “Video time: Properties, encoders and evaluation,” in _British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018_. BMVA Press, 2018, p. 160.
  * [46] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017, pp. 2261–2269.
  * [47] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds., 2014, pp. 568–576.
  * [48] H. Bilen, B. Fernando, E. Gavves, and A. Vedaldi, “Action recognition with dynamic image networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 40, no. 12, pp. 2799–2813, 2018.
  * [49] S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural networks for human action recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 35, no. 1, pp. 221–231, 2013.
  * [50] D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_. IEEE Computer Society, 2015, pp. 4489–4497.
  * [51] J. Carreira and A. Zisserman, “Quo vadis, action recognition? A new model and the kinetics dataset,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017, pp. 4724–4733.
  * [52] B. Zhou, A. Andonian, A. Oliva, and A. Torralba, “Temporal relational reasoning in videos,” in _ECCV_ , 2018, pp. 803–818.
  * [53] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in _European conference on computer vision_. Springer, 2016, pp. 20–36.
  * [54] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. V. Gool, “Temporal segment networks for action recognition in videos,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 41, no. 11, pp. 2740–2755, 2019.
  * [55] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _CVPR_ , 2018, pp. 7794–7803.
  * [56] G. A. Sigurdsson, S. Divvala, A. Farhadi, and A. Gupta, “Asynchronous temporal fields for action recognition,” in _CVPR_ , 2017, pp. 585–594.
  * [57] G. Varol, I. Laptev, and C. Schmid, “Long-term temporal convolutions for action recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 40, no. 6, pp. 1510–1517, 2018.
  * [58] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in _CVPR_ , 2017, pp. 6299–6308.
  * [59] X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” in _CVPR_ , 2018, pp. 6848–6856.
  * [60] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _ICLR_ , 2015.
  * [61] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [62] S. Yeung, O. Russakovsky, N. Jin, M. Andriluka, G. Mori, and L. Fei-Fei, “Every moment counts: Dense detailed labeling of actions in complex videos,” _International Journal of Computer Vision_ , vol. 126, no. 2-4, pp. 375–389, 2018.
  * [63] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg _et al._ , “Scikit-learn: Machine learning in python,” _Journal of Machine Learning Research_ , vol. 12, no. Oct, pp. 2825–2830, 2011.
  * [64] H. Idrees, A. R. Zamir, Y.-G. Jiang, A. Gorban, I. Laptev, R. Sukthankar, and M. Shah, “The thumos challenge on action recognition for videos “in the wild”,” _Computer Vision and Image Understanding_ , vol. 155, pp. 1–23, 2017.
  * [65] R. Girdhar, D. Ramanan, A. Gupta, J. Sivic, and B. Russell, “Actionvlad: Learning spatio-temporal aggregation for action classification,” in _CVPR_ , 2017, pp. 971–980.
  * [66] A. Piergiovanni, A. Angelova, A. Toshev, and M. S. Ryoo, “Evolving space-time neural architectures for videos,” in _Proceedings of the IEEE international conference on computer vision_ , 2019, pp. 1793–1802.
  * [67] N. Hussein, E. Gavves, and A. W. Smeulders, “Videograph: Recognizing minutes-long human activities in videos,” _arXiv preprint arXiv:1905.05143_ , 2019.
  * [68] A. Dave, O. Russakovsky, and D. Ramanan, “Predictive-corrective networks for action detection,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017, pp. 2067–2076.
  * [69] Y. Zhao, Y. Xiong, L. Wang, Z. Wu, X. Tang, and D. Lin, “Temporal action detection with structured segment networks,” in _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_. IEEE Computer Society, 2017, pp. 2933–2942.
  * [70] A. J. Piergiovanni and M. S. Ryoo, “Learning latent super-events to detect multiple activities in videos,” in _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_. IEEE Computer Society, 2018, pp. 5304–5313.
  * [71] A. Piergiovanni and M. Ryoo, “Temporal gaussian mixture layer for videos,” in _International Conference on Machine learning_. PMLR, 2019, pp. 5152–5161.


# Reply to ‘ “Comment on Doubly periodic solutions of the focusing nonlinear
Schrödinger equation: Recurrence, period doubling, and amplification outside
the conventional modulation-instability band” ’

Matteo Conforti1 and Nail Akhmediev2 1University of Lille, CNRS, UMR
8523—PhLAM—Physique des Lasers Atomes et Molécules, Lille, France
2Optical Sciences Group, Department of Theoretical Physics, Research School of
Physics, The Australian National University, Canberra, ACT 2600, Australia

###### Abstract

We reply to the comments on our previous paper Physical Review A, Vol. 101,
023843 (2020), raised by H. Schuermann and V. Serov in arXiv:2204.05846.

The Authors of the Comment [8] raise doubts on the correctness of the
solutions reported in our paper published in Phys. Rev. A [1] and the earlier
paper by one of us [2]. We will show below that the criticisms of the Authors
are unfounded or based on confusion. We insist that every single equation
reported in [1] is correct.

Firstly, all solutions obtained in [1] can be confirmed by direct substitution
into the Nonlinear Schrödinger Equation (NLSE) (Eq. (1) in [1]). This has been
done by us as well as by the authors of many other papers on this issue
published since 1987 (The most recent one is [5]). In addition, we have
extensively checked the correctness of the solutions reported in [1] by
numerical solution of the NLSE for a large choice of parameters. In all cases,
the numerical simulations and the analytic results gave identical outcome.
Apparently, the authors of the Comment did not use multiple ways of checking
their derivations in their calculations. We consider this as a deficiency of
such ‘pure’ mathematical approach.

Moreover, as physicists, we have exploited the solutions derived in [1] to
model real experiments in optics and in hydrodynamics [4, 3]. In all cases
that we have considered, the experimental results were in perfect agreement
with the analytic expressions in [1]. Thus, our formulas are not only correct.
They are of practical interest in describing physical effects in optics and
hydrodynamics.

Below we confute the claims of the Authors of the Comment (copied here for
convenience in _italics_).

_“First, $Q(t,z)$, according to Eq.(6) in [1], is not a solution of the
associated Eq.(15) in [2] (since in [1] only the final form of the solutions
are presented, we refer to [2], if necessary).”_ _“- $Q(t,z)$ according to
Eq.(6) in [1] (Eq.(24) in [2]) is flawed.”_

The Authors of the Comment are ignoring the fact that the NLSEs in [1] and [2]
have different forms. Moreover, definitions of $z$ and $t$ are also different.
Therefore, equations cannot be simply transferred from one publication to
another. The differences in definition must be taken into account.

The following two sentences in the Comment is clearly the result of the same
confusion:

_“ Second, consistency of Eq.(6) in [1] with Eq.(5) in [2] has not been
checked neither in [1] nor in [2]. Third, solution Q(t, z) in [1] (see Fig.1)
does not satisfy Eq.(5) in [2].”_

_“– Consistency of Eq.(5) in system (4), (5) (in [2]) with $Q(t,z$) (in [1])
has not been checked. Equation (6) in [1] does not satisfy Eq.(5) in [2] (or
[28]) above).”_

Equations in Refs. [1] and [2] cannot be mixed. They cannot be directly
transferred from Ref.[1] to Ref.[2] and vice versa because of different forms
of the NLSE and different notations for $x$ and $t$ in these works. The
Authors of the Comment should look carefully into this issue. The same
confusion appears in the following excerpt from the Comment:

_Solutions $\delta(z)$ and $Q(t,z)$ (see Eqs.(4), (6), (19), (20) in [1]) are
expressed in terms of Jacobi elliptic functions (with $\alpha_{i}$ as the
roots of the fourth degree polynomial, Eq.(13) in [2]). Despite the
equivalence of Jacobi and Weierstrass elliptic functions, it is not a matter
of preference to use one of both for representations of h(z) and f(t, z) (see
[4], summary). Varying parameters (e.g., W, H, D in [2]) are leading to
various $\alpha_{i}$ and hence (in general) to different Jacobi functions as
in [1] (Eqs.(4) and (19)). Our representation of $\delta(z)$ and $Q(t,z)$ as
$h(z)$ and $f(t,z)$ according to (7) and (14) makes this discrimination
unnecessary, since (7) and (14) are valid, independently on the sign of
$\Delta_{z}$ . Since $f(t,z)$ is triggered by $h(z)$, variable modulus is the
normal case. Thus, compared with (7), (14), it seems (at least) inexpedient to
use Jacobi functions for evaluation._

About the representation of solutions: The solutions reported by the Authors
of the Comment are expressed in terms of Weierstrass elliptic functions. This
idea is not new and also comes from Ref.[2] (see the end of Section 2). This
alternative approach was used recently by Conte [7]. Mathematically, the two
representations are equivalent. Indeed, Weierstrass elliptic functions can be
written in terms of Jacobi elliptic functions and vice versa. The elusive
‘convenience’ of using the Weierstrass function is that it allows to write a
single expression for both type-A and type-B solutions. However, from physical
point of view, this advantage is deceptive because the type-A and -B solutions
describe qualitatively different orbits in phase space. The two families of
solutions are separated by the homoclinic trajectories known as Akhmediev
breathers. Having two different mathematical expressions for solutions
describing different physical phenomena provides clarity. Unfortunately, the
representation given by the Authors of the Comment does not separate the two
cases and doing so would require additional formal analysis which is not
given.

_Solution (4) in [1] is a particular case of solution (7) ( $h_{0}=0$ and
three positive roots of $R_{1}(h)=0$). Additionally, due to $\delta(0)=0$
according to (4) in [1], function value
$Q(0,0)=Q_{D}=\sqrt{\alpha_{1}}-\sqrt{\alpha_{2}}-\sqrt{\alpha_{3}}$ (see (6)
in [1]) is special compared with the range of $f_{0}$ defined by constraint
$R_{2}(f_{0},z)\geq 0$ in (14). Needless to say that free (or free in a
certain domain) parameters are important for matching with experimental data;
it seems that fixed $\delta(0)$ and $Q(0,0)$ are unnecessarily restrictive_

The Authors of the Comment suggest that their solutions are more general
because they have more parameters. This is an illusion. Additional parameters
can be introduced in our solutions by exploiting the well known symmetries of
the NLSE which are (1) the trivial translations in space and time
$z\rightarrow z+z_{0}$, $t\rightarrow t+t_{0}$, (2) the rescaling of
variables, and (3) the Galilean transformation. Using them leave the NLSE
unchanged. Adding trivial translations as suggested by the Authors of the
Comment does not add any mathematical rigour into the solutions. The solutions
reported in [1] use the minimum number of nontrivial parameters, which is
three. So, fixed $\delta(0)$ and $Q(0,0)$ _are not_ really restrictions.

Talking about experiments, the solutions reported in [1] have been
successfully exploited in optics [4, 3]. Better than this, they permitted to
discover a novel form of modulation instability beyond the conventional
(linear) instability band [3]. The latter is a significant extension of the
well known Benjamin-Feir instability.

_Finally, $f(t,z)$ is different from $Q(t,z)$ because the modulus $k_{q}$ of
$Q(t,z)$ does not depend on $z$. $Q(t,z)$ is the solution of Eq.(15) in [2]
(with disregards to different notations of the variables), which is equation
(5) above. The coefficients in both equations are dependent on $z$ and (on $t$
in [2]). Thus it is not clear why the $z$-dependence drops out from the
modulus of Eq.(6) in [1] (Eq.(24) in [2]). Moreover, the validity of solution
(6) in [1] is not justified: If we assume that $Q(t,z$) according to (6) in
[1] is a solution of the corresponding equation (15) in [2] then the modulus
$k_{q}$ in (6) is not correct. If we assume that $Q(t,z)$ with modulus $k_{q}$
is correct then $Q(t,z)$ is not a solution of equation (15) in [2]._

Again, the Authors of the Comment are confusing the matter trying to mix
equations in [1] and [2]. We have to repeat that the forms of the NLSE in
Refs.[1] and [2] are different and the definition of variables $x$ and $t$ are
different. The notations in [1] are common in optics while the notations in
[2] are common in mathematical literature.

_IV. ON THE ADEQUACY OF ANSATZ (2)
As mentioned in the Introduction, consistency of $Q(t,z)$ (according to (6) in
[1]) with Eq.(5) in [2] has not been checked. Thus it leads to the question
whether f(t, z) according to (14) is consistent with Eq.(3a)._

The title of this section in the Comments is bizarre. It assumes that the
basic conjecture used in [1] (and its equivalent in [2]) is incorrect. In
reality, the fact that the class of solutions derived in [1] and [2] satisfy
the ansatz

$\psi(t,z)=[Q(t,z)+i\delta(z)]e^{i\phi(z)}.$ (1)

can easily be confirmed by simple analysis of these solutions. Namely, one-
soliton solutions, Akhmediev breathers, Kutznetzov-Ma solitons, variety of
cnodial waves and the lowest order rogue wave all belong to this class [6].
This is the matter of simple algebra. As explained in [2], Eq.(1) means that
at any fixed $z$ (or $t$ in [2]), the real and imaginary parts of these
solutions are linearly related. Then, it is unclear what the Authors of the
Comment mean by the ‘adequacy of the ansatz’. Rejecting the existence of the
ansatz is equivalent to claiming that the above class of solutions do not
exist.

Our procedure is straightforward and common in physics: we made a conjecture
[Eq. (1)], we found solutions and we checked them extensively by different
means (direct substitution, numerical simulations of the NLSE, and confirmed
the mathematical results in experiments).

_The representation of $Q(t,z)$ in terms of Jacobi functions is not effective
for numerical evaluation._

This claim is simply wrong. Firstly, the Jacobi elliptic functions today are
as common as the trigonometric functions. They are well familiar to most
students specialised in physics or mathematics. Such presentation demonstrates
clearly the qualitative difference between the type-A and the type-B solutions
imposed by the physics of the problem. Secondly, software packages for
calculating elliptic Jacobi functions are common and widely available. This
cannot be said about the Weierstrass function. For example, MATLAB software
implements elliptic Jacobi functions but not the Weierstrass function.
Thirdly, we addressed the solutions reported in [1] to physicists and experts
in nonlinear dynamics. Thus, we limited ourselves to the bounded, smooth and
non-singular solutions and illustrated them graphically. What is presented in
the Comments are manipulations with equations that can easily go wrong without
additional confirmation using either numerical calculations or a simple
technique such as direct substitution of solutions into the original equation.

To conclude, we have confuted point-by-point the criticisms of the Authors of
the Comment. The correctness of the doubly periodic solutions reported in [1]
is easy to check using independent techniques such as a simple substitution
into the NLSE. This is exactly what has been done. While we are sure about the
correctness of our solutions, we have doubts on the expressions reported in
the Comment. Finding the actual errors in their derivations is a task that
goes well beyond the scope of this Reply.

In our opinion the Comment by Schürmann and Serov contains several false
statements. Moreover, it does not report any novelty: the solutions in terms
of Weierstrass functions have been first suggested in [2] and have been
recently implemented in more details by Conte [7] (ref. [10] of the Comment).
Remarkably, the new results (as claimed) in [7] are singular solutions of the
NLSE that have been omitted completely in [1] and [2] as they are unphysical.
We should also note that although the major part of derivations in [7] have
been copied from [1] and [2], no any errors were detected. This is in striking
contradiction with the claims of the Authors of the Comment.

## References

  * [1] M. Conforti, A. Mussot, A. Kudlinski, S. Trillo, and N. Akhmediev, ”Doubly periodic solutions of the focusing nonlinear Schrödinger equation: Recurrence, period doubling, and amplification outside the conventional modulation-instability band,” Phys. Rev. A 101, 023843 (2020).
  * [2] N. N. Akhmediev, V. M. Eleonskii, and N. E. Kulagin, ”Exact first-order solutions of the nonlinear Schrodinger equation,” Theor. Math. Phys. 72, 809–818 (1987).
  * [3] G. Vanderhaegen, C. Naveau, P. Szriftgiser, A. Kudlinski, M. Conforti, A. Mussot, M. Onorato, S. Trillo, A. Chabchoub, and N. Akhmediev, ”“Extraordinary” modulation instability in optics and hydrodynamics,” Proc. Natl. Acad. Sci. 118, e2019348118 (2021).
  * [4] G. Vanderhaegen, P. Szriftgiser, C. Naveau, A. Kudlinski, M. Conforti, S. Trillo, N. Akhmediev, and A. Mussot, ”Observation of doubly periodic solutions of the nonlinear Schrödinger equation in optical fibers,” Opt. Lett. 45, 3757 (2020).
  * [5] J. Chen, D. E. Pelinovsky, and R. E. White, ”Rogue waves on the double-periodic background in the focusing nonlinear Schrödinger equation,” Phys. Rev. E 100, 052219 (2019).
  * [6] N. Akhmediev, J. M. Soto-Crespo, and A. Ankiewicz, ”Extreme waves that appear from nowhere: On the nature of rogue waves,” Phys. Lett. A 373, 2137–2145 (2009).
  * [7] R. Conte, Theor. Math. Phys., 201, 1366 (2021).
  * [8] H. W. Schuermann and V. Serov, ”Comment on ”Doubly periodic solutions of the focusing nonlinear Schrödinger equation: Recurrence, period doubling, and amplification outside the conventional modulation-instability band”,” arXiv:2204.05846 (2022).