# Language-based Action Concept Spaces Improve
Video Self-Supervised Learning

Kanchana Ranasinghe
Stony Brook University
<EMAIL_ADDRESS>
&Michael Ryoo
Stony Brook University
<EMAIL_ADDRESS>

###### Abstract

Recent contrastive language image pre-training has led to learning highly
transferable and robust image representations. However, adapting these models
to video domain with minimal supervision remains an open problem. We explore a
simple step in that direction, using language tied self-supervised learning to
adapt an image CLIP model to the video domain. A backbone modified for
temporal modeling is trained under self-distillation settings with train
objectives operating in an action concept space. Feature vectors of various
action concepts extracted from a language encoder using relevant textual
prompts construct this space. A large language model aware of actions and
their attributes generates the relevant textual prompts. We introduce two
train objectives, concept distillation and concept alignment, that retain
generality of original representations while enforcing relations between
actions and their attributes. Our approach improves zero-shot and linear
probing performance on three action recognition benchmarks.

##  1 Introduction

Actions in videos are defined by individual objects, their relationships, and
interaction [1, 2]. Video self-supervised learning focuses on discovering
representations aware of such action attributes directly from video content
with no human supervision [3]. Particularly in the case of videos, where
manual human annotation can be both expensive and noisy, such self-supervised
approaches are invaluable.

A recent variant of self-supervision explores learning with loosely paired
image-caption pairs, leading to highly transferable and robust representations
such as CLIP [4]. These approaches obtain zero-shot performance often
comparable to fully-supervised methods. However, their counterparts in the
video domain [5, 6, 7, 8, 9, 10, 11] do not exhibit the same generality. In
fact, some approaches training CLIP on videos [11, 12] perform subpar to
image-CLIP under zero-shot settings (see Table 2). Such behaviour can be
attributed to lesser availability and more noisy nature of labelled (or paired
caption) video datasets [3]. This motivates exploration into self-supervised
learning (SSL) techniques that can learn from videos under less supervision
while utilizing existing image CLIP [4] like representations. Existing state-
of-the-art video SSL approaches [13, 14] learn highly transferable
representations from videos, but combining these with image CLIP
representations is not straightforward. In fact, despite methods like SVT [13]
being able to utilize image SSL representations [15] for weight initialization
to achieve better performance, using image CLIP representations instead for
weight initialization leads to performance subpar to image CLIP (see Table 5).
This raises necessity for alternate video SSL approaches compatible with CLIP
like image representations and is our key motivation.

In this work, we explore self-supervised learning techniques that adapt image
CLIP models [4] to video domain under entirely self-supervised settings,
dependent on no form of video level labels or captions. Under this setting,
natural language can still provide strong cues regarding attributes that
compose an action category [16, 17]. We leverage this idea to propose a novel
language-based self-supervised learning objective. Following a standard self-
distillation and multi-view based SSL formulation [15, 13], we introduce
language aligned feature spaces, action concept spaces, where our SSL
objectives operate. Large-language models (LLMs) [18], given their extensive
world knowledge [19, 20], serve as an ideal tool to generate necessary textual
concepts for these spaces. We also introduce regularization suitable for our
language aligned SSL objective to prevent collapse during training. Our
resulting framework is termed Language-based Self-Supervision, or LSS.

In contrast to existing video self-supervised learning approaches [13, 14],
our proposed LSS retains and improves transferability of image CLIP
representations much better (see Tables 1 and 5). Additionally, our language
aligned learning framework allows direct zero-shot operation on downstream
tasks. Moreover, unlike video CLIP methods with similar zero-shot capabilities
[5, 6, 7, 8, 9, 10, 11] that utilize per-video labels / captions for learning,
our proposed LSS requires only videos for training.

We summarize our key contributions as follows:

  * •

Self-supervised learning paradigm capable of retaining and improving strengths
of CLIP like image representations for video domain operation

  * •

Video specific self-supervised learning objectives, namely concept
distillation and concept alignment, that enforce relations between action
categories and their visual attributes

  * •

Novel language-based video self-supervised learning framework operating zero-
shot on downstream action classification tasks without requiring per-video
labels / captions for training

Experiments on action recognition datasets showcase state-of-the-art
performance for our learned representations under linear-probing, standard
zero-shot, and transductive zero-shot settings.

##  2 Related Work

Self-Supervised Learning in Videos was initially dominated by pretext tasks
specific to the video domain [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32].
Recently a shift to contrastive losses led to [33, 34, 35, 36, 37, 38] with
some variants focused on video specific view generation [39, 40, 41, 42, 13].
An alternate direction has been masked auto-encoders [14]. To the best of our
knowledge, existing video self-supervised learning (SSL) approaches operate
purely within the visual domain. By video SSL, we refer to methods that
utilize only videos with no paired captions (or labels) for each video during
training. In contrast, our proposed LSS learns purely from videos in a self-
supervised manner, integrating pre-trained language-image models to learn
language aligned representations.

Zero-shot Action Recognition began with manual attribute and feature selection
[43, 44, 45, 46, 47] with later works utilizing action word embeddings [16,
17]. The idea of connecting action categories with elaborate descriptions of
those actions, within language embedding spaces [48, 49] has been a next step
and is closely related to our work. This idea is also explored in image domain
to boost zero-shot performance [50]. While our work is inspired by such
approaches, in contrast, we use relations between such actions and
descriptions as self-supervised signals for learning. Recent image CLIP models
[4, 51] are another line of works achieving strong performance on some video
classification tasks, with only single frame processing. Multiple approaches
build on image CLIP [4] to learn video level representations [11, 52, 53, 54,
12, 55] under fully-supervised settings. While achieving strong performance on
the training datasets, their zero-shot improvements over CLIP are minimal or
even subpar (see Table 2). Therein, LSS focuses on zero-shot performance under
self-supervised settings while retaining (and improving) the generality of the
representation space.

Self-training methods leverage pseudo-labels on unlabeled data [56, 57, 58]
for supervised-fashion training. Recently they have been combined with CLIP
models for zero-shot operation [59, 60]. While inspired by such self-training
approaches, our proposed LSS differs in its continuous feature space self-
distillation, language-based relations enforcing, video domain operation, and
cross-dataset transfer for zero-shot operation.

Adapting image-CLIP models to video under fully-supervised settings has
gathered much interest [5, 6, 7, 8, 9, 10]. Expanding backbones for temporal
modeling, multi-modal fusion, secondary training objectives, partial parameter
updates, and scaling-up data are key ideas explored [55, 10]. In contrast, LSS
is a first to operate under self-supervised settings using no video
annotations.

Contemporary work in [61] adapts image CLIP features to video tasks label free
similar to our work. ViFi-CLIP [62] introduces zero-shot action recognition
benchmarks and similarly adapts CLIP to videos retaining generality. Using
LLMs for action recognition is also explored in [63].

##  3 Language-based Self-Supervision (LSS)

In this section, we present our proposal, Language-based Self-Supervision
(LSS). The generality and robustness of shared image-language representation
spaces such as that of CLIP [4] allow interesting manipulations of visual
representations using language. We explore such manipulations under the
setting of visual self-supervised learning focusing on video understanding.
Self-supervised objectives can operate within a latent space constructed with
language, retaining language alignment of learned visual representations. This
allows better interpretability of representations as well as zero-shot
inference. We discuss the four key components of our approach: backbone
architecture, concept distillation objective, modifications to avoid collapse,
and concept alignment objective.

Figure 1: Our overall setup contains three components: visual teacher model
(green), visual student model (red), and language model (blue). We utilize the
text encoder of CLIP as our language model and extract concept vectors
relevant to action labels and descriptions of those actions. A visual encoder
(containing a space-time backbone) is partially initialized with the visual
encoder of CLIP and used to obtain sample specific features. The generated
concept vectors are used to project these features to a concept space where
our proposed concept distillation and concept alignment losses are applied.

###  3.1 Backbone Architecture

Our approach introduces a text classifier to self-distillation based SSL works
[15, 13], in place of the projector network. Given a data sample $x$, let
$x_{1},x_{2}\in\mathbb{R}^{(C,T,H,W)}$ be two augmented views generated using
video specific transformations following [13], where $C=3,T=8,H=W=224$ are
channel, time, and spatial dimensions respectively.

Visual Encoder: A visual encoder, $\theta_{v}$, processes $x_{i}$ to produce
feature $f_{i}\in\mathbb{R}^{768}$. We utilize the pre-trained image encoder
of CLIP [4] expanded for temporal modelling using factorized space-time
attention. The vision transformer variant of CLIP is selected to allow our
factorized space-time attention. In particular, we use ViT-B/16 architecture
for the the image encoder, in which for a given augmented view with $H=W=224$
and $T=8$, each transformer block processes 8 temporal and 196 spatial tokens
separately in sequential order, and the embedding dimension of each token is
$\mathbb{R}^{768}$. In addition to the input tokens from the data sample, one
classification token [64, 65] serves as the final feature vector output by the
network, namely $f_{i}$, which is common to the CLIP image encoder. This
classification token is inflated and processed suitably following [66] to
accommodate our modifications for factorized space-time attention. We follow
[66] to zero-initialize additional time-attention parameters, achieving
outputs identical to the pre-trained CLIP image encoder at start of training.

Text Classifier: Inspired by [67], a set of $n$ language embeddings extracted
from the CLIP text encoder, $\theta_{t}$, are used to construct the weight
parameter of a linear layer (with no bias term), which we call our text
classifier, $\theta_{c}$. The role of this text classifier is to project
visual features $f_{i}$ to a vector space defined by those $n$ embeddings,
producing $\tilde{f}_{i}\in\mathbb{R}^{n}$. Next we discuss these vector
spaces (referred to as action concept spaces) and the text classifier module
in detail.

Figure 2: We illustrate a toy concept space constructed with the three action
concepts: run, swim, and walk. In this example, the text classifier projects
visual feature $f_{i}$ into the 3-dimensional toy concept space to produce
$\tilde{f}_{i}$.

###  3.2 Action Concept Spaces

Self-supervised learning approaches following exponential moving average (EMA)
based self-distillation [68, 15, 13] utilize a projector network (MLP) to
operate in a higher dimensional feature space. This is expected to minimize
train-test domain gaps, handle noisy positive sample pairs, and better
discriminate nuanced feature differences [69]. Focused on these notions, we
propose an alternate concept space composed of a set of basis vectors defined
by language-based action concepts. Our language-based self-supervision
objectives operate within such concept spaces.

Concept Spaces: Building off the assumption that text encoder features capture
subtle differences between distinct actions categories, we hypothesize that
necessary nuanced distinctions between these actions will be better captured
in our proposed concept spaces. The defining parameters of concept spaces are
their basis vectors, $b_{i}$. Normalized embeddings (extracted from text
encoder, $\theta_{t}$) of various natural language captions ($c_{i}$) relevant
to action categories are used as these basic vectors.

$\displaystyle b_{i}$
$\displaystyle={\theta_{t}(c_{i})}\left.\right/{||\theta_{t}(c_{i})||^{2}_{2}}$
(1) $\displaystyle\mathbf{b}$ $\displaystyle=[b_{1},b_{2},...\
b_{n}]^{T}\text{ ; }\mathbf{b}\in\mathbb{R}^{(n,d)}$ (2)

Note that these basis vectors are not necessarily orthogonal. As illustrated
in Fig. 2, a single set of basis vectors, $\mathbf{b}$, defines one action
concept space. We define two sets of basic vectors: action category vectors
and action description vectors. Action category vectors relate to a single
action label which is converted to a caption using textual prompting following
[4]. Action description vectors are averaged embeddings of multiple
descriptions and visual characteristics relevant to individual action
categories. These two distinct sets of basic vectors lead to two distinct
concept spaces which we name category concept space and description concept
space respectively.

Category Concept Space: We explore 3 different strategies to construct the
category concept space. The base setup uses action labels from Kinetics-400
[70], UCF-101 [71], and HMDB-51 [72] datasets, leading to a set of 530 (400 +
101 + 51, ignoring overlaps) basis vectors. Our next goal of connecting LLMs
and their action awareness occurs in the second two strategies. We utilize
LLMs [18] and visual-LLMs [73] to extract large sets of action category
labels. While we explore this idea of expanding the basis vector set with LMM
based additional action labels in Section 4, the base setup containing a
modest 530 categories was sufficient to improve downstream task performance.

Description Concept Space: This space is constructed conditioned on the
previous category concept space. For each action label used in the latter, we
extract 4 distinct descriptions and a set of visual characteristics relevant
to that action label using a large language model (LLM). The role of the LLM
is to inject its world knowledge (i.e. awareness on videos, actions, and their
attributes) into our learned representations during self-supervised learning.
In detail, we prompt GPT-3 [18] to generate such descriptions and
characteristics using procedure outlined in Appendix A. We highlight that
GPT-3 is used here as an intelligent LLM containing world knowledge on videos
and actions, in order to create natural language descriptions for given action
category labels. The textual outputs generated for each action label are
processed by our text encoder to produce multiple embeddings for a single
action label. These embeddings are averaged to produce the corresponding basis
vector for the description concept space. Note how this leads to a common
dimensionality between the two concept spaces as well as one to one
correspondences between the basic vectors of the spaces, which we leverage in
our self-supervision objectives.

###  3.3 Concept Distillation

We now describe our primary self-supervised learning objective, concept
distillation. Standard multi-view based self-supervision enforces a network to
encode the common information between two augmented (distorted) views of a
data sample [69]. This common information can be considered as the
augmentation invariant signal present in the original data sample [69, 74]. In
the case of self-distillation based approaches [15, 13], a higher dimensional
feature space is utilized to enforce the self-supervision objectives. Instead,
we propose to use action concept spaces as an alternative.

Proposed concept distillation depends on an action concept space and visual
video features aligned to the basis vectors of that space. Given our visual
features $f_{i}\in\mathbb{R}^{d}$, we obtain projected
$\tilde{f}_{i}\in\mathbb{R}^{n}$ as,

$\displaystyle\tilde{f}_{i}$ $\displaystyle=\mathbf{b}\
(\left.f_{i}\right/||f_{i}||^{2}_{2})=[b_{1}\cdot f_{i}^{\prime},b_{2}\cdot
f_{i}^{\prime},...\ b_{n}\cdot f_{i}^{\prime}]^{T}$ (3)

Similarity Calculation:  Projecting normalized visual video features to a
concept space corresponds to calculating the dot-product similarity with each
basic vector of the concept space. The projected vector $\tilde{f}_{i}$ can be
viewed as a similarity score distribution across all basis vectors of the
concept space. Inspired by [67], we implement this similarity calculation as a
linear layer with weight matrix $\mathbf{b}$ and bias terms zero. We refer to
this layer as the text classifier. Similar to [67], our text classifier
remains frozen (no parameter updates), but in our case, this is to retain the
original language distribution.

Concept Distillation Objective: Viewing projected features for two augmented
views of a single video as score distributions, we argue that the underlying
signal of the original video would relate to a unique score distribution to
which score distributions of each view should be similar. Therein, following
our EMA teacher based self-distillation setup (see Section 3.1 for details),
we enforce the score distribution to be consistent across views. Given two
views $x_{1},x_{2}$ of a single video, our teacher and student visual encoders
process them respectively to produce $f_{1},f_{2}$. The text classifier
projects these to concept space, producing score distributions
$\tilde{f}_{1},\tilde{f}_{2}$. We obtain our objective,
$\mathcal{L}_{\text{CD}}$ as:

$\displaystyle\hat{f}_{i}[k]$
$\displaystyle=\frac{\operatorname{exp}(\tilde{f}_{i}[k]/\lambda_{i})}{\sum_{j=1}^{n}\operatorname{exp}(\tilde{f}_{i}[j]/\lambda_{i})}$
(4) $\displaystyle w_{s}$ $\displaystyle=\operatorname{max}(\hat{f}_{1})$ (5)
$\displaystyle\mathcal{L}_{\text{CD}}(\tilde{f}_{1},\tilde{f}_{2})$
$\displaystyle=-w_{s}\cdot\sum_{j=1}^{n}\hat{f}_{1}[j]\operatorname{log}\hat{f}_{2}[j]$
(6)

The teacher and student score distributions, $\tilde{f}_{1},\tilde{f}_{2}$,
are softmax normalized in Eq. 4, with temperature terms
$\lambda_{1}=0.1,\lambda_{2}=1$ for sharpening only the teacher score
distribution. A significance score $w_{s}$ is calculated for each sample in
Eq. 5. In the softmax normalized teacher score distribution ($\hat{f}_{1}$),
the maximum value is high when peaked at a single action concept and low when
peaked at multiple action concepts. Considering the noisy nature of multi-peak
teacher score distributions, we utilize $w_{s}$ to minimize their overall
effect during training. Our overall $\mathcal{L}_{\text{CD}}$ is thus
implemented as in Eq. 6.

Distinct Concept Spaces: Given the two distinct action concept spaces defined
in Section 3.2, we utilize two parallel text classifiers to implement each,
and obtain two score distributions, one for each concept space. Defining score
distributions $\tilde{f}_{i}^{C},\tilde{f}_{i}^{D}$ for category and
description concept spaces respectively, we apply our
$\mathcal{L}_{\text{CD}}$ on each pair separately to obtain two losses
$\mathcal{L}_{\text{CD}}^{\text{X}}$ for X$\in${C,D} as:

$\displaystyle\mathcal{L}_{\text{CD}}^{\text{X}}=\mathcal{L}_{\text{CD}}(\tilde{f}_{1}^{\text{X}},\tilde{f}_{2}^{\text{X}})$
(7)

We highlight how our concept spaces implemented as text classifiers are
maintained intact by freezing the text classifier during training. This allows
our approach to perform direct zero-shot inference, making concept
distillation additionally advantageous over standard video SSL techniques.

###  3.4 Uniform Distribution Prior

Avoiding collapse is a key concern in SSL methods [15, 13, 69] and recent
self-distillation based approaches utilize feature sharpening and centering
operations to avoid collapse [15, 13]. While we similarly perform sharpening
operations on the teacher outputs, given the nature of our action concept
space, performing a learned vector mean subtraction based centering operations
can break the meaningful structure of score distributions. Instead, we enforce
a uniform distribution prior on the expected score distribution over the
entire training dataset. The centering operation proposed in [15] acts
similarly pushing representations towards a uniform distribution while the
sharpening operation counters its effect. We approximate expectation over the
dataset as a moving average of mean score distributions at each train
iteration and the uniform prior is enforced as:

$\displaystyle\hat{f}_{\text{MA}}^{\text{X}}$
$\displaystyle=\tau\cdot\hat{f}_{2}^{\text{X}}+(1-\tau)\cdot\hat{f}_{\text{MA}}^{\text{X}}$
(8) $\displaystyle\mathcal{L}_{\text{UP}}^{\text{X}}$
$\displaystyle=-\frac{1}{n}\sum_{j}\operatorname{log}\hat{f}_{\text{MA}}^{\text{X}}[j]$
(9)

where the hyper-parameter $\tau=0.5$ is fixed during training. We highlight
that $\mathcal{L}_{\text{UP}}$ is necessary for convergence with concept
distillation and is added to the concept distillation objective,
$\mathcal{L}_{\text{CD}}^{\text{X}}$.

###  3.5 Concept Alignment

Aligning action category labels and their descriptions or attributes within
some embedding space has been explored in video SSL under multiple settings
[48, 49]. Motivated by these promising results, we explore how such alignment
can be integrated to improve our framework with _concept spaces_. In Section
3.2, we define two distinct action concept spaces constructed from category
labels and detailed category descriptions respectively. We hypothesize that
explicit alignment of video features between these two spaces based on their
one to one relationship can learn additional information. Therein, we
introduce our concept alignment objective, $\mathcal{L}_{\text{CA}}$, as
follows:

$\displaystyle\mathcal{L}_{\text{CA}}=\mathcal{L}_{\text{CD}}(\tilde{f}_{1}^{\text{C}},\tilde{f}_{2}^{\text{D}})+\mathcal{L}_{\text{CD}}(\tilde{f}_{1}^{\text{D}},\tilde{f}_{2}^{\text{C}})$
(10)

Overall SSL Objective: Reusing $\mathcal{L}_{\text{CD}}$ from Eq. 6, we match
score distributions across our two concept spaces instead of within a single
concept space.
$\mathcal{L}_{\text{CD}}(\tilde{f}_{1}^{\text{C}},\tilde{f}_{2}^{\text{D}})$
aligns student description score distribution $\tilde{f}_{2}^{\text{D}}$ to
teacher category score distribution $\tilde{f}_{1}^{\text{C}}$ while
$\mathcal{L}_{\text{CD}}(\tilde{f}_{1}^{\text{C}},\tilde{f}_{2}^{\text{D}})$
aligns student category score distribution $\tilde{f}_{2}^{\text{C}}$ to
teacher description score distribution $\tilde{f}_{1}^{\text{D}}$. Combining
all terms, we obtain:

$\displaystyle\mathcal{L}=(\mathcal{L}_{\text{CD}}^{\text{C}}+\mathcal{L}_{\text{UP}}^{\text{C}})+(\mathcal{L}_{\text{CD}}^{\text{D}}+\mathcal{L}_{\text{UP}}^{\text{D}})+\mathcal{L}_{\text{CA}}$
(11)

###  3.6 Concept Space Variants

Our baseline concept space (described in Section 3.2) utilizes labels from
three standard video datasets (Kinetics-400, UCF-101, HMDB-51). However, we
want to ensure scalability with more data and no label leakage to downstream
evaluation tasks. With this goal, we propose 2 additional variants of action
concept spaces tagged LSS-B and LSS-C. These variants do not use any form of
ground truth textual labels from datasets. Moreover, they leverage the world
awareness (i.e. knowledge on videos and actions) of LLMs to generate extensive
action categories. Our baseline setup is hereafter referred as LSS-A.

For LSS-B, we use GPT-3 [18] to generate a large set of action labels. We
first prompt GPT to categorize all common human actions / activities into 20
groups. For each group, we again ask GPT to generate at least 100 visually
diverse action categories. These are all collected to create a set of 2000
action labels. We then use projections of these labels in CLIP text-encoder
representation space to eliminate labels of high semantic similarity (spectral
clustering in feature space from [75] to identify similar features), achieving
1000 diverse action categories. So our 1000 action categories for LSS-B are
generic, not tied to any of our training datasets, and scalable with more
data.

For LSS-C, we generate a label set using only videos from the training
dataset. We use PCA based clustering to identify 2000 representative videos
from a randomly sampled subset (50,000) of our training dataset and then use
image-captioning models (LLaVa [73]) on video center frames to generate a
diverse set of 2000 action labels. This is further reduced to 500 eliminating
labels that are similar in feature space of the CLIP text encoder. In this
case, our generated labels are tied to the training dataset, but uses no
textually annotated category labels. We use only the videos (and an image-to-
text captioning model) to generate our label set, still resulting in a
scalable framework.

Note that each of these alternate strategies relates to construction of our
category concept space. Given the selected set of textual category labels of
this space, the description concept space is constructed in the same common
way (as described in Section 3.2).

##  4 Experiments

In this section, we first describe our experimental setup followed by
discussion of results for linear probing self-supervised representations and
zero-shot analysis.

Datasets: We use three standard action recognition benchmark datasets in our
experiments: Kinetics-400 [70], UCF-101 [71], and HMBD-51 [72]. Kinetics-400
is a large-scale dataset containing 240,000 training videos and 20,000
validation videos belonging to 400 different action classes. On average, these
videos are of duration around 10 seconds, with 25 frames per second (i.e.,
around 250 frames per video). UCF-101 and HMBD-51 are small-scale datasets
each containing 13k videos (9.5k/3.7k train/test) belonging to 101 classes and
5k (3.5k/1.5k train/test) videos belonging to 51 classes respectively. They
also contain similar duration videos.

Self-supervised Training: Our SSL training phase uses the train split of
Kinetics-400 dataset [70] _without_ using any per-video labels. We train for
15 epochs using a batch size of 32 across 4 NVIDIA-A5000 GPUs using ADAM-W
[76, 77] optimizer on the student model with an initial learning rate of
$1e-5$ following a cosine decay schedule. The EMA teacher is updated from
student weights after each training iteration with a decay ratio of $2e-4$.
Unless otherwise specified, this model is used for all downstream task
evaluations.

Transductive Training: For selected experiments, we additionally perform self-
supervised training directly on the train split of each downstream dataset.
For Kinetics-400, we follow the same setup described above. In the case of
HMDB-51 and UCF-101, we perform self-supervised training for a longer duration
of 100 epochs (smaller train sets) leaving all other hyper-parameters
unchanged.

View Generation: Our self-supervised setup requires two views of a single
video. We sample two clips from a video following global view generation in
[13]. In detail, we select two random intervals from a video, and uniformly
sample (equal time gaps between frames) 8 frames of 224x224 spatial dimensions
from within that interval. Standard video augmentations from [78] are also
applied randomly for each view.

Linear Probing: We follow standard linear probing settings on our two
downstream datasets to evaluate quality of representations learned by our
self-supervised learning phase. We follow the same settings in [13] for fair
comparison. Our visual encoder is frozen and a randomly initialized linear
layer is trained on the train split of the downstream dataset in a fully-
supervised manner. We train for 15 epochs using a batch size of 128 across 4
NVIDIA-A5000 GPUs using ADAM-W [76, 77] optimizer with an initial learning
rate of $1e-3$ following a cosine decay schedule. During inference, we sample
three 224x224 dimensional spatial crops with 8 uniformly spaced frames from
each video following prior work [13, 36].

Zero-Shot Inference: For zero-shot inference, we project class labels of
downstream datasets to our text encoder feature space, and construct an
alternate text classifier. Using this text classifier, we make zero-shot
predictions. This setup is identical to dot-product similarity based inference
in CLIP [4] (explanation in Section 3.3). In line with prior work [13, 36], we
feed three 224x224 dimensional spatial crops with 8 uniformly spaced frames
sampled from each video to the visual encoder and average its output feature
embedding prior to normalized dot-product calculation in the text encoder.

Table 1: Linear Probing on HMDB-51 [72] and UCF-101 [79]: We compare our method against prior work, reporting top-1 (%) accuracy (following evaluation procedure in [13]). ‘ITP’ stands for image-text pre-training. Gray shaded methods use additional optical flow (OF) inputs for training. Nevertheless, our performance is comparable to such methods using per-video OF modality information. In contrast, we use generic language modality information and require no one-to-one language relations with individual videos during training.  Method | Backbone | ITP | TFLOPS | Frames | Epochs | HMDB | UCF
---|---|---|---|---|---|---|---
MemDPC [34] ${}_{\text{(ECCV \textquoteleft 20)}}$ | R2D3D-34 | ✗ | - | 64 | - | 30.5 | 54.1
CoCLR [35] ${}_{\text{(NeurIPS \textquoteleft 20)}}$ | S3D | ✗ | 0.07 | 32 | 100 | 52.4 | 77.8
VideoMoCo [80]${}_{\text{(CVPR \textquoteleft 21)}}$ | R(2+1)D | ✗ | 17.5 | 32 | 200 | 49.2 | 78.7
CVRL [36]${}_{\text{(CVPR \textquoteleft 21)}}$ | R3D-50 | ✗ | 3.19 | 32 | 800 | 57.3 | 89.2
MoDist [81] ${}_{\text{(Arxiv \textquoteright 21)}}$ | R3D-50 | ✗ | 3.19 | 8 | 100 | 63.0 | 91.5
BraVe [38]${}_{\text{(ICCV '21)}}$ | R3D-50 | ✗ | 3.19 | 16 | - | 68.3 | 92.5
Vi2CLR [82] ${}_{\text{(ICCV '21)}}$ | S3D | ✗ | 0.07 | 32 | 300 | 47.3 | 75.4
MCN [83] ${}_{\text{(ICCV '21)}}$ | R3D | ✗ | 3.19 | 32 | 50 | 42.9 | 73.1
CORP [84] ${}_{\text{(ICCV '21)}}$ | R3D-50 | ✗ | 3.19 | 16 | 800 | 58.7 | 90.2
SVT [13]${}_{\text{(CVPR \textquoteleft 22)}}$ | ViT-B | ✗ | 0.59 | 16 | 20 | 57.8 | 90.8
VideoMAE [14]${}_{\text{(NeurIPS \textquoteleft 22)}}$ | ViT-B | ✗ | 0.59 | 16 | 800 | 60.3 | 84.7
MERLOT [85]${}_{\text{(NeurIPS \textquoteleft 21)}}$ | ViT-B | ✓ | - | 16 | - | 55.4 | 80.1
VATT [86]${}_{\text{(NeurIPS \textquoteleft 21)}}$ | ViT-B | ✓ | - | 32 | - | 66.4 | 87.6
TVTS [87]${}_{\text{(CVPR \textquoteleft 23)}}$ | ViT-B | ✓ | 0.59 | 16 | 20 | 58.4 | 83.4
LaViLa [88]${}_{\text{(CVPR \textquoteleft 23)}}$ | ViT-L | ✓ | - | 4 | 5 | 61.5 | 88.1
LSS-A (ours) | ViT-B | ✓ | 0.59 | 8 | 20 | 69.2 | 91.0
LSS-B (ours) | ViT-B | ✓ | 0.59 | 8 | 20 | 69.4 | 91.1
LSS-C (ours) | ViT-B | ✓ | 0.59 | 8 | 20 | 69.1 | 90.8

###  4.1 Linear-Probing Analysis

We first evaluate LSS under linear probing settings on HMDB-51 & UCF-101
datasets. Our results (top-1 accuracy) are reported in Table 1. Our proposed
LSS achieves state-of-the-art results on both datasets, outperforming prior
approaches. Note that MoDist [81] and BraVe [89], both of which additionally
utilize video-level optical flow (OF) for self-supervision, are not directly
comparable. Still, our LSS showcases competitive performance to those, even
without such motion information.

Table 2: Zero-shot Transfer on HMDB-51 [72] and UCF-101 [79]: We compare LSS against prior work, reporting top-1 accuracy (%). Mean across three test splits is reported following [55]. ‘ITP’ stands for image-text pre-training and ‘Video Labels’ refers to using per-video annotations (or paired captions) for supervision during video-based training. We highlight how among directly comparable unsupervised (at video level) approaches as well as over the CLIP [4] baseline, LSS boosts zero-shot performance.  Method | Backbone | ITP | Video Labels | Frames | HMDB | UCF
---|---|---|---|---|---|---
TS-GCN [90]${}_{\text{(AAAI `19)}}$ | GCN | ✗ | ✓ | 16 | 23.2 | 34.2
E2E [91]${}_{\text{(CVPR `20)}}$ | CNN | ✗ | ✓ | 16 | 32.7 | 48.0
ER-ZSAR [48]${}_{\text{(ICCV `21)}}$ | CNN | ✗ | ✓ | 8 | 35.3 | 51.8
ActionCLIP [11] | ViT-B | ✓ | ✓ | 32 | 40.8 | 58.3
X-CLIP [12]${}_{\text{(ECCV `22)}}$ | ViT-B | ✓ | ✓ | 32 | 44.6 | 72.0
VicTR [55] | ViT-B | ✓ | ✓ | 32 | 51.0 | 72.4
ViFi [62]${}_{\text{(CVPR `23)}}$ | ViT-B | ✓ | ✓ | 32 | 51.3 | 76.8
MTE [92]${}_{\text{(ECCV `16)}}$ | - | ✗ | ✗ | - | 19.7 | 15.8
ASR [93]${}_{\text{(ECML `17)}}$ | CNN | ✗ | ✗ | 16 | 21.8 | 24.4
ZSECOC [94]${}_{\text{(CVPR `17)}}$ | - | ✗ | ✗ | - | 22.6 | 15.1
UR [49]${}_{\text{(CVPR `18)}}$ | CNN | ✗ | ✗ | 1 | 24.4 | 17.5
CLIP [4]${}_{\text{(ICML `21)}}$ | ViT-B | ✓ | ✗ | 1 | 46.5 | 69.8
CLIP [4]${}_{\text{(ICML `21)}}$ | ViT-B | ✓ | ✗ | 8 | 47.2 | 70.3
LaViLa [88]${}_{\text{(CVPR \textquoteleft 23)}}$ | ViT-L | ✓ | ✗ | 4 | 16.6 | 18.2
LSS-A (ours) | ViT-B | ✓ | ✗ | 8 | 49.5 | 72.0
LSS-B (ours) | ViT-B | ✓ | ✗ | 8 | 50.2 | 73.8
LSS-C (ours) | ViT-B | ✓ | ✗ | 8 | 51.4 | 74.2

###  4.2 Zero-Shot Analysis

Our LSS provides the additional advantage of zero-shot operation unlike
standard video SSL approaches. To this end, we conduct two forms of zero-shot
experiments. First, we evaluate LSS on standard zero-shot classification,
where our model trained on Kinetics-400 (under SSL settings) is evaluated on
the two downstream datasets, HMDB-51 and UCF-101. We report these results
(top-1 accuracy) in Table 2. Compared to prior work utilizing per-video labels
/ captions for training, we achieve competitive performance. We note that MOV
[7] trained under supervised settings with per-video labels and additional
audio information is not a direct comparison.

In contrast to most prior approaches, LSS uses no video level labels for its
Kinetics-400 training. In particular, LSS has not seen any labelled videos
during its training process. Compared to prior work operating under these
settings, LSS achieves state-of-the-art performance on both downstream
datasets as seen in the bottom half of Table 2.

An alternate setting in prior zero-shot work is transductive training, where
self-supervised learning is perfomed directly on train splits of downstream
datasets. Under this setting, we evaluate on all three datasets, Kinetics-400,
HMDB-51, and UCF-101, reporting results (top-1 accuracy) in Table 3. In the
case of HMBD-51 and Kinetics-400, our method achieves state-of-the-art
performance. For UCF-101, we achieve competitive results, and clear
improvements over a CLIP [4] baseline.

Table 3: Transductive Zero-shot Transfer on HMDB-51 [72], UCF-101 [79], and Kinetics-400 [70]: We report top-1 accuracy (%) following the evaluation procedure in [55]. Similar to prior work, we perform dataset specific unsupervised fine-tuning (using our self-supervised objective) on the train-splits of each downstream dataset (no labels used). ‘ITP’ refers to image-text pretaining, and ‘Video Labels’ refers to video level supervised training. Note that CLIP [4] is not transductive and is included only for comparison purposes.  Method | Backbone | ITP | Video Labels | Frames | HMDB | UCF | K400
---|---|---|---|---|---|---|---
UR [49]${}_{\text{(CVPR `18)}}$ | CNN | ✗ | ✗ | 1 | 28.9 | 20.1 | -
TS-GCN [90]${}_{\text{(AAAI `19)}}$ | GCN | ✗ | ✓ | 16 | 23.2 | 34.2 | -
CLIP [4]${}_{\text{(ICML `21)}}$ | ViT-B | ✓ | ✗ | 8 | 47.2 | 70.3 | 49.7
MUST [59]${}_{\text{(ICLR `23)}}$ | ViT-B | ✓ | ✗ | 1 | 48.9 | 81.1 | 51.2
LSS (ours) | ViT-B | ✓ | ✗ | 8 | 55.0 | 75.6 | 54.3

Table 4: Ablation on SSL objectives: We ablate our proposed concept distillation (CD) applied on category (CD${}^{\text{C}}$) and description (CD${}^{\text{D}}$) concept spaces and concept alignment (CA) using linear probing (LP) & zero-shot transfer (ZS) on HMDB-51 [72] dataset. Since our approach cannot be trained without any objective, we construct two new baselines from CLIP [4] and SVT [13] (details in Section 4.3). Each proposed component obtains clear improvements over the baselines.  |  CD${}^{\text{C}}$ |  CD${}^{\text{D}}$ | CA | LP | ZS
---|---|---|---|---|---
CLIP | ✗ | ✗ | ✗ | 63.9 | 46.5
CLIP† | ✗ | ✗ | ✗ | 67.3 | 47.2
SVT§ | ✗ | ✗ | ✗ | 62.2 | -
Ours | ✓ | ✗ | ✗ | 68.5 | 48.8
Ours | ✓ | ✓ | ✗ | 69.0 | 49.2
Ours | ✓ | ✓ | ✓ | 69.2 | 49.5

Table 5: Concept Space Ablation: We report zero-shot accuracy (%) on HMDB-51
and UCF-101 datasets. ‘K400 only’ shows transfer to unseen downstream classes.
The labels of K400 overlapping with UCF/HMDB are not used here. A CLIP [4]
baseline (modified for video domain without re-training) is reported in row 1
for comparison purposes. For 2000 words & 10,000 sentences, we utilize most
common nouns / verbs and example sentences for action verbs from WordNet [95,
96].

Method | Action Labels | HMDB | UCF
---|---|---|---
CLIP | - | 47.2 | 70.3
Ours | K400 only (w/o U,H) | 48.4 | 71.1
Ours | K400+U+H (A) | 49.5 | 72.0
Ours | (A)+2000 words | 48.6 | 71.8
Ours | (A)+10K sentences | 49.6 | 71.4

###  4.3 Ablations

We next study the contribution of each component within our approach. All
ablative experiments follow the same SSL phase on the Kinetics-400 train set
(as described in Section 4) followed by zero-shot analysis on validation sets
of HMDB-51 and UCF-101. In the case of linear probing results, training is
conducted following same settings (see Section 4) on the train set of HMDB-51
followed by evaluation on its validation set.

SSL Objectives: First we ablate each proposed component in Eq. 11 and report
results in Table 5. In addition to a direct CLIP [4] baseline, we construct
two additional baselines building off CLIP [4] and SVT [13] for better
comparison. CLIP† baseline applies our backbone modifications (for temporal
modeling) with no training, which is identical to averaging per-frame visual
encoder features. SVT§ baseline performs SVT [13] training with CLIP visual
encoder initialization (note that language alignment breaks and zero-shot
operation is not possible for this baseline). In comparison to the CLIP
baselines, each proposed component, concept distillation in category and
description concept spaces as well as concept alignment, leads to
improvements. The comparison against SVT§ highlights how our SSL approach
better preserves language aligned information (contained in CLIP) that is
useful even in linear probing. In contrast, the lower performance of SVT§
compared to CLIP baselines indicates that generic SSL techniques may be losing
useful information contained in CLIP.

Concept Spaces: Our next focus is on construction of concept spaces. We
explore how separately augmenting each concept space affects downstream task
performance measured with zero-shot transfer. These results are reported in
Table 5. First, focused on the category concept space, we construct additional
category labels using 1000 most common nouns and verbs each (total of 2000)
from the WordNet dataset [95, 96]. Next, we augment the description concept
space using 10,000 sentences. We select these from example sentences provided
for action verbs in the WordNet dataset [95, 96]. In these experiments, only
the concept distillation objective is applied on these augmented spaces and
concept alignment operates only on the base category set. This is because
independently augmenting one of the action spaces eliminates their shared and
aligned dimensionality. Results for these two settings are reported in row 2 &
3 respectively in Table 5.

Table 6: Ablation on Regularization and Significance Weight: The effect of
proposed regularization (left) and significance weight (right) is
demonstrated. UDP regularization (see Section 3.4) is particularly essential
to prevent collapse during the SSL training phase. Significance weight
($w_{s}$) also improves performance.

Method | HMDB | UCF
---|---|---
LSS | 48.4 | 71.1
LSS w/o UDP | 33.4 | 54.3

Method | HMDB | UCF
---|---|---
LSS | 48.4 | 71.1
LSS w/o $w_{s}$ | 47.2 | 70.3

Uniform Distribution Prior: We ablate on proposed uniform distribution prior
(UDP) which acts as a regularization to prevent collapse (Section 3.4). Our
results in Table 6 (left) indicate clear necessity of such regularization to
prevent collapse during SSL training.

Significance Weight in Concept Distillation: In Eq. 6, we utilize a
significance weight term, $w_{s}$, which represents the confidence of the
target concept space projection for a given sample. We note how each sample
during training is a clip sampled from a video (which covers a temporal crop
of video). Our intuition for this weight is to act as a way of prioritizing
more important clips over the less important ones. Our ablations in Table 6
(right) indicate usefulness of this weight term.

##  5 Conclusion

We introduce a novel language-based self-supervised learning (SSL) approach
for videos, termed LSS, capable of adapting strong language-aligned image
representations (CLIP [4]) to the video domain. In particular, we propose two
self-distillation based SSL objectives, concept distillation and concept
alignment. Our approach trains with no video level labels or paired captions
similar to prior video SSL works, but retains language alignment from image
CLIP enabling direct zero-shot inference. We demonstrate state-of-the art
performance in terms of linear probing with the learned representations on
downstream tasks. For zero-shot operation, LSS demonstrates strong performance
under both standard and transductive settings, indicating a promising
direction for video SSL.

Limitations, Future Work, & Broader Impact: The language alignment of LSS may
be limited mostly to per-frame static information since the alignment is
derived from image CLIP [4]. LSS cannot distinguish motion based categories
like "moving object left to right". Moreover, while containing highly
discriminative and generic information at image level, CLIP features lack
spatial awareness at an object level [75]. Our proposed model building off
these representations in inherently limited in understanding object level
motion and interaction within videos. However, recent progress in localization
aware CLIP models [75, 97, 98] opens avenues for leveraging their object-
centric or pixel-level representations to better model such video motion
patterns, opening up interesting future directions. In terms of broader
impact, the datasets and pre-trained models we use possibly contain biases,
which may be reflected in LSS. However, our reduced reliance on human
annotations may lower additional biases.

Reproducibility Statement: We build a codebase derived from source code of SVT
[13] & CLIP [4] and use pre-trained CLIP weights from
https://github.com/openai. All experiments use publicly available datasets.
Our action descriptions will be released publicly along with our codebase.

Acknowledgements: We thank Xiang Li for helpful discussions and server setup.
We also thank Kumara Kahatapitiya and Cristina Mata for helpful discussions.

## References

  * [1] Michael S. Ryoo and Jake K. Aggarwal.  Recognition of composite human activities through context-free grammar based representation.  2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), 2:1709–1718, 2006.
  * [2] Jake K. Aggarwal and Michael S. Ryoo.  Human activity analysis.  ACM Computing Surveys (CSUR), 43:1 – 43, 2011.
  * [3] Madeline Chantry Schiappa, Yogesh Singh Rawat, and Mubarak Shah.  Self-supervised learning for videos: A survey.  ACM Computing Surveys, 2022.
  * [4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning Transferable Visual Models From Natural Language Supervision.  In ICML, pages 8748–8763. PMLR, 2021.
  * [5] Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo.  CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment.  arXiv preprint arXiv:2209.06430, 2022.
  * [6] Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu.  Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners.  arXiv preprint arXiv:2212.04979, 2022.
  * [7] Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, and Yin Cui.  Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models.  arXiv preprint arXiv:2207.07646, 2022.
  * [8] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie.  Prompting Visual-Language Models for Efficient Video Understanding.  In ECCV, pages 105–124. Springer, 2022.
  * [9] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Fine-tuned CLIP Models are Efficient Video Learners.  arXiv preprint arXiv:2212.03640, 2022.
  * [10] Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius.  VindLU: A Recipe for Effective Video-and-Language Pretraining.  arXiv preprint arXiv:2212.05051, 2022.
  * [11] Mengmeng Wang, Jiazheng Xing, and Yong Liu.  ActionCLIP: A New Paradigm for Video Action Recognition.  arXiv preprint arXiv:2109.08472, 2021.
  * [12] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling.  Expanding language-image pretrained models for general video recognition.  In European Conference on Computer Vision, 2022.
  * [13] Kanchana Ranasinghe, Muzammal Naseer, Salman Hameed Khan, Fahad Shahbaz Khan, and Michael S. Ryoo.  Self-supervised video transformer.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2864–2874, 2021.
  * [14] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  ArXiv, abs/2203.12602, 2022.
  * [15] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [16] Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka.  Rethinking zero-shot video classification: End-to-end training for realistic applications.  In CVPR, 2020.
  * [17] Xun Xu, Timothy Hospedales, and Shaogang Gong.  Transductive zero-shot action recognition by word-vector embedding.  IJCV, 2017.
  * [18] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * [19] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen.  A survey of large language models.  ArXiv, abs/2303.18223, 2023.
  * [20] Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Nick Barnes, and Ajmal S. Mian.  A comprehensive overview of large language models.  ArXiv, abs/2307.06435, 2023.
  * [21] Michael Mathieu, Camille Couprie, and Yann LeCun.  Deep multi-scale video prediction beyond mean square error.  In ICLR, 2016.
  * [22] Viorica Pătrăucean, Ankur Handa, and Roberto Cipolla.  Spatio-temporal video autoencoder with differentiable memory.  In ICLR (Workshop), 2016.
  * [23] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert.  An uncertain future: Forecasting from static images using variational autoencoders.  In ECCV, 2016.
  * [24] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov.  Unsupervised learning of video representations using lstms.  In ICML, 2015.
  * [25] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.  Generating videos with scene dynamics.  In NeurIPS, 2016.
  * [26] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy.  Tracking emerges by colorizing videos.  In ECCV, 2018.
  * [27] Pulkit Agrawal, Joao Carreira, and Jitendra Malik.  Learning to see by moving.  In ICCV, 2015.
  * [28] Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun.  Unsupervised learning of spatiotemporally coherent metrics.  In ICCV, 2015.
  * [29] Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson.  Learning visual groups from co-occurrences in space and time.  In ICLR, 2016.
  * [30] Ishan Misra, C. Lawrence Zitnick, and Martial Hebert.  Shuffle and learn: Unsupervised learning using temporal order verification.  In ECCV, 2016.
  * [31] Xiaolong Wang and Abhinav Gupta.  Unsupervised learning of visual representations using videos.  In ICCV, 2015.
  * [32] AJ Piergiovanni, Anelia Angelova, and Michael S. Ryoo.  Evolving losses for unsupervised video representation learning.  In CVPR, 2020.
  * [33] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He.  A large-scale study on unsupervised spatiotemporal representation learning.  Arxiv, 2021.
  * [34] Tengda Han, Weidi Xie, and Andrew Zisserman.  Video representation learning by dense predictive coding.  In ICCV, 2019.
  * [35] Tengda Han, Weidi Xie, and Andrew Zisserman.  Self-supervised Co-training for Video Representation Learning.  NeurIPS, 33:5679–5690, 2020.
  * [36] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui.  Spatiotemporal contrastive video representation learning.  CVPR, 2021.
  * [37] R Devon et al.  Representation learning with video deep infomax.  Arxiv, 2020.
  * [38] Adrià Recasens, Pauline Luc, Jean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patraucean, Florent Altché, Michal Valko, et al.  Broaden your views for self-supervised video learning.  ICCV, 2021.
  * [39] Deng Huang, Wenhao Wu, Weiwen Hu, Xu Liu, Dongliang He, Zhihua Wu, Xiangmiao Wu, Mingkui Tan, and Errui Ding.  Ascnet: Self-supervised video representation learning with appearance-speed consistency.  In ICCV, 2021.
  * [40] Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, and Chuang Gan.  Rspnet: Relative speed perception for unsupervised video representation learning.  In AAAI, volume 1, 2021.
  * [41] Nadine Behrmann, Mohsen Fayyaz, Juergen Gall, and Mehdi Noroozi.  Long short view feature decomposition via contrastive video representation learning.  In ICCV, 2021.
  * [42] Ishan Rajendra Dave, Rohit Gupta, Mamshad Nayeem Rizve, and Mubarak Shah.  TCLR: Temporal contrastive learning for video representation.  Arxiv, 2021.
  * [43] Jingen Liu, Benjamin Kuipers, and Silvio Savarese.  Recognizing human actions by attributes.  In CVPR, 2011.
  * [44] Rowan Zellers and Yejin Choi.  Zero-shot activity recognition with verb attribute induction.  arXiv preprint arXiv:1707.09468, 2017.
  * [45] Mihir Jain, Jan C Van Gemert, Thomas Mensink, and Cees GM Snoek.  Objects2action: Classifying and localizing actions without any video example.  In ICCV, 2015.
  * [46] Junyu Gao, Tianzhu Zhang, and Changsheng Xu.  I know the relationships: Zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs.  In AAAI, 2019.
  * [47] Junyu Gao, Tianzhu Zhang, and Changsheng Xu.  Learning to model relationships for zero-shot video classification.  TPAMI, 2020.
  * [48] Shizhe Chen and Dong Huang.  Elaborative Rehearsal for Zero-shot Action Recognition.  In ICCV, pages 13638–13647, 2021.
  * [49] Yi Zhu, Yang Long, Yu Guan, Shawn Newsam, and Ling Shao.  Towards Universal Representation for Unseen Action Recognition.  In CVPR, pages 9436–9445, 2018.
  * [50] Sachit Menon and Carl Vondrick.  Visual Classification via Description from Large Language Models.  arXiv preprint arXiv:2210.07183, 2022.
  * [51] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision.  In ICML, pages 4904–4916. PMLR, 2021.
  * [52] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval.  Neurocomputing, 508:293–304, 2022.
  * [53] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  A CLIP-Hitchhiker’s Guide to Long Video Retrieval.  arXiv preprint arXiv:2205.08508, 2022.
  * [54] Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Frozen CLIP Models are Efficient Video Learners.  arXiv preprint arXiv:2208.03550, 2022.
  * [55] Kumara Kahatapitiya, Anurag Arnab, Arsha Nagrani, and Michael S. Ryoo.  Victr: Video-conditioned text representations for activity recognition.  ArXiv, abs/2304.02560, 2023.
  * [56] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In NIPS, pages 1195–1204, 2017.
  * [57] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In NeurIPS, 2020.
  * [58] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  In ICLR, 2020.
  * [59] Junnan Li, Silvio Savarese, and Steven C. H. Hoi.  Masked unsupervised self-training for zero-shot image classification.  ArXiv, abs/2206.02967, 2022.
  * [60] Jonathan Kahana, Niv Cohen, and Yedid Hoshen.  Improving zero-shot models with label distribution priors.  ArXiv, abs/2212.00784, 2022.
  * [61] Wei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Possegger, Mateusz Koziński, Rameswar Panda, Rogério Schmidt Feris, Hilde Kuehne, and Horst Bischof.  Match, expand and improve: Unsupervised finetuning for zero-shot action recognition with language knowledge.  ArXiv, abs/2303.08914, 2023.
  * [62] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Fine-tuned clip models are efficient video learners.  2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6545–6554, 2022.
  * [63] Laura Hanu, Anita Lilla Vero, and James Thewlis.  Language as the medium: Multimodal video classification through text only.  ArXiv, abs/2309.10783, 2023.
  * [64] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  Arxiv, 2018.
  * [65] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [66] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is Space-Time Attention All You Need for Video Understanding?  In ICML, page 4, 2021.
  * [67] Wenhao Wu, Zhun Sun, and Wanli Ouyang.  Revisiting Classifier: Transferring Vision-Language Models for Video Recognition.  AAAI, 2023.
  * [68] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  In NeurIPS, 2020.
  * [69] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari S. Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Grégoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum.  A cookbook of self-supervised learning.  ArXiv, abs/2304.12210, 2023.
  * [70] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? A new model and the Kinetics dataset.  In CVPR, 2017.
  * [71] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  UCF101: A dataset of 101 human actions classes from videos in the wild.  Arxiv, 2012.
  * [72] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.  HMDB: A large video database for human motion recognition.  In ICCV, pages 2556–2563. IEEE, 2011.
  * [73] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning, 2023.
  * [74] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  ArXiv, abs/2105.04906, 2021.
  * [75] Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens.  Perceptual grouping in contrastive vision-language models.  ICCV, 2023.
  * [76] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [77] Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  ICLR, 2019.
  * [78] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui.  Spatiotemporal Contrastive Video Representation Learning.  In CVPR, pages 6964–6974, 2021.
  * [79] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [80] Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu.  Videomoco: Contrastive video representation learning with temporally adversarial examples.  In CVPR, 2021.
  * [81] Fanyi Xiao, Joseph Tighe, and Davide Modolo.  Modist: Motion distillation for self-supervised video representation learning.  Arxiv, 2021.
  * [82] Ali Diba, Vivek Sharma, Reza Safdari, Dariush Lotfi, Saquib Sarfraz, Rainer Stiefelhagen, and Luc Van Gool.  Vi2clr: Video and image for visual contrastive learning of representation.  In ICCV, 2021.
  * [83] Yuanze Lin, Xun Guo, and Yan Lu.  Self-supervised video representation learning with meta-contrastive network.  In ICCV, 2021.
  * [84] Kai Hu, Jie Shao, Yuan Liu, Bhiksha Raj, Marios Savvides, and Zhiqiang Shen.  Contrast and order representations for video self-supervised learning.  In ICCV, 2021.
  * [85] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  Merlot: Multimodal neural script knowledge models.  ArXiv, abs/2106.02636, 2021.
  * [86] Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  ArXiv, abs/2104.11178, 2021.
  * [87] Ziyun Zeng, Yuying Ge, Xihui Liu, Bin Chen, Ping Luo, Shutao Xia, and Yixiao Ge.  Learning transferable spatiotemporal representations from natural script knowledge.  ArXiv, abs/2209.15280, 2022.
  * [88] Yue Zhao, Ishan Misra, Philipp Krahenbuhl, and Rohit Girdhar.  Learning video representations from large language models.  ArXiv, abs/2212.04501, 2022.
  * [89] Adria Recasens, Pauline Luc, Jean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Pătrăucean, Florent Altché, Michal Valko, et al.  Broaden Your Views for Self-Supervised Video Learning.  In ICCV, pages 1255–1265, 2021.
  * [90] Junyu Gao, Tianzhu Zhang, and Changsheng Xu.  I Know the Relationships: Zero-Shot Action Recognition via Two-Stream Graph Convolutional Networks and Knowledge Graphs.  In AAAI, pages 8303–8311, 2019.
  * [91] Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka.  Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications.  In CVPR, pages 4613–4623, 2020.
  * [92] Xun Xu, Timothy M Hospedales, and Shaogang Gong.  Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation .  In ECCV, pages 343–359. Springer, 2016.
  * [93] Qian Wang and Ke Chen.  Alternative Semantic Representations for Zero-Shot Human Action Recognition.  In ECML-PKDD, pages 87–102. Springer, 2017.
  * [94] Jie Qin, Li Liu, Ling Shao, Fumin Shen, Bingbing Ni, Jiaxin Chen, and Yunhong Wang.  Zero-Shot Action Recognition with Error-Correcting Output Codes.  In CVPR, pages 2833–2842, 2017.
  * [95] George A Miller.  Wordnet: a lexical database for english.  In Communications of the ACM, pages 39–41. ACM, 1995.
  * [96] Christiane Fellbaum.  Wordnet and wordnets.  In Keith Brown et al., editors, Encyclopedia of Language & Linguistics, pages 665–670. Elsevier, 2nd edition, 2005.
  * [97] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie.  Learning open-vocabulary semantic segmentation models from natural language supervision.  ArXiv, 2023.
  * [98] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  GroupViT: Semantic Segmentation Emerges from Text Supervision.  In CVPR, pages 18134–18144, 2022.

Appendix

##  Appendix A Prompting details

Our proposed approach utilizes two sets of language based captions: categories
and descriptions. While categories are obtained directly from the class labels
of datasets (set of unique labels - e.g. 400 classes in Kinetics-400 dataset),
the descriptions are generated automatically utilizing GPT-3 [18]. For each
category caption, we query GPT-3 to provide a set of descriptions and visual
characteristics. In detail, we use the following two prompts to generate
descriptions and visual characteristics:
prompt1 = "Give 4 different descriptions for the phrase: {category}?"
prompt2 = "List visual objects or characteristics usually seen with the
action: {category}?" The resulting two sets of captions are converted to text
embeddings using our text-encoder, and a single average text embedding is
computed. This averaged embedding is used as the description basis vector for
that category. Also, the resulting dataset containing these category-
description pairs is made available publicly.

##  Appendix B Additional Experiments

Linear Probing Evaluation: We present more results for linear probing in Table
7 (left). Our proposed LSS improves over the baseline achieving competitive
performance on Kinetics-400.

Text-to-video retrieval: An important characteristic of CLIP [4] is its
retrieval ability across both language and visual modalities. In order to
verify if proposed LSS retains these strengths, we run experiments on MSR-VTT
text-to-video retrieval benchmark. We demonstrate how LSS improves over our
baseline CLIP, reporting these results in Table 7 (top-right).

Charades Evaluation: We explore an alternate task of zero-shot multi-label
classification on the Charades video dataset. We report mAP results for this
task in Table 7 (bottom-right) as an additional point of comparison.

Table 7:  We report top-1 (%) accuracy on the Kinetics-400 [70] validation set
for linear probing evaluation (left). All models are pre-trained on the
training set of Kinetics-400 dataset. We also report a CLIP baseline for
comparison purposes. Performance of our proposed approach is on-par with prior
state-of-the-art and showcases improvements over our baseline method. We also
report retrieval scores (top-right) for MSR-VTT and classification mAP
(bottom-right) for Charades dataset.

Method | Backbone | Acc (%)
---|---|---
CVRL [36] ${}_{\text{(CVPR\textquoteright 21)}}$ | R3D-101 | 67.6
BraVe [38] ${}_{\text{(ICCV'21)}}$ | R3D-50 | 66.7
Vi2CLR [82] ${}_{\text{(ICCV '21)}}$ | S3D | 63.4
CORP [84] ${}_{\text{(ICCV '21)}}$ | R3D-50 | 66.6
SVT [13] ${}_{\text{(CVPR `22)}}$ | ViT-B | 68.1
VideoMAE [14] ${}_{\text{(NeurIPS `22)}}$ | ViT-B | 61.3
CLIP [4] | ViT-B | 66.4
LSS (ours) | ViT-B | 67.3

Method | R@1 | R@5 | R@10
---|---|---|---
CLIP [4] | 30.6 | 54.4 | 64.3
LSS (ours) | 33.8 | 58.2 | 70.3
Method | Classification mAP
---|---
CLIP [4] | 19.7
LSS (ours) | 23.1


# EXISTENCE OF TRAVELING WAVES in a Nicholson Blowflies Model WITH DELAYED
DIFFUSION TERM

William Barker  Department of Mathematics and Statistics, University of
Arkansas at Little Rock, 2801 S University Ave, Little Rock, AR 72204. USA
<EMAIL_ADDRESS>
###### Abstract.

In this paper we consider traveling waves for a diffusive Nicholson Blowflies
Equation with different discrete time delays in the diffusion term and birth
function. We construct quasi upper and lower solutions via the monotone
iteration method. This also allows for the construction of $C^{2}$ upper and
lower solutions, and then traveling wave solutions. We then provide numerical
results for the kernel for the iteration.

###### Key words and phrases:

Traveling waves; Reaction-diffusion equations; Delay; Nicholson Blowflies
Equation

###### 2000 Mathematics Subject Classification:

Primary: 35C07 ; Secondary: 35K57

##  1\. Introduction

The diffusive Nicholson Blowflies Model with a single delay in the birth term
is of the form

(1.1) $\frac{\partial u(t,x)}{\partial t}=\frac{\partial^{2}u(t,x)}{\partial
x^{2}}-\delta u(t,x)+pu(t-\tau_{1},x)e^{-au(t-\tau_{1},x)}$

for $x\in\mathbb{R},t\geq 0.$ $u(t,x)$ is the blowfly population in space at a
certain time, $t.$ The death rate is $\delta>0,$ impact rate on the immature
population is $p>0$ due to birth, and $\tau_{1}>0$ is the maturation delay.
For more information about the model see Nicholson’s groundbreaking work, [13,
14] and the adaptation to include spatial diffusion, [17, 18, 23]. The above
model has been studied in several celebrated papers.

In 2001, So and Zou, [19] provided an elegant result providing a construction
of upper and lower solutions when there is delay in the birth function. This
extended the seminal results on the existence of traveling wave solutions via
the so called monotone iteration method put forth by Wu and Zou, [22]. In the
standard monotone or quasi monotone iteration methods, it is often important
to construct upper and lower wave front solutions. This idea was extended by
Ma, [10] who developed super and sub solutions, which relaxed the requirements
in Wu and Zou.

The dynamics of traveling waves for the Nicholson Blowflies model with
distributed delay was discussed in 2000 by Gourley and Ruan, [5] by using
energy methods and by a comparison principle for functional differential
equations. In a recent paper, the global attractivity of the positive steady
state was studied for a non-monotone model with distributed delay by Deng and
Wu, [4].

In 2004, Mei et al., [12] discussed the nonlinear stability of traveling
wavefronts of a time-delayed diffusive Nicholson blowflies equation under a
weighted $L^{2}$ norm. This result is very interesting and was extended by Lin
and Mei, [9] who also provided numerical results via a finite difference
method. Furthermore, recent results by Huang and Liu, [7] and Huang and Xu [8]
studied the existence of traveling waves with a birth function given two
different delays.

Boumenir and Nguyen, [3] developed the idea of quasi upper and lower solutions
via a modified Perron Theorem, see Theorem (4.1) in Mallet-Peret, [11]. More
information can also be found in Pruss, [15]. This leads us to the motivation
of this paper. The idea of placing the diffusion term is a relatively new
idea. In fact, it was shown in Barker and Nguyen, [2] traveling waves exist
for reaction diffusion equations with discrete delay in both the diffusion and
reaction term of the form

(1.2) $\frac{\partial u(x,t)}{\partial
t}=D\frac{\partial^{2}u(x,t-\tau_{1})}{\partial x^{2}}+f(u_{t}),$

where $t\in\mathbb{R},\tau_{1}>0,x,u(x,t)\in\mathbb{R},\ D>0,\
f:C\left([-\tau_{2},0],\mathbb{R}\right)\to\mathbb{R}$ is continuous and
$u_{t}(x)\in C\left([-\tau_{2},0],\mathbb{R}\right),$ defined as

$u_{t}(x)=u(x,t+\theta),\ \theta\in[-\tau_{2},0],\ t\geq 0,\ x\in\mathbb{R}.$

Where $f$ is Lipschitz continuous and

$f(0)=f(K)=0,\ \text{and}\ f(u)\neq 0,\ 0<u<K.$

under certain monotone conditions.

This paper will be organized as follows: Section 2 will lay out some
preliminary notation that may be used in the later sections. Our main results
will be stated in Section 3. The existence of traveling waves will be
discussed as well as solutions to exponential type polynomials of second
order. The last part of Section 3 will be the explicit construction of quasi
upper and lower solutions for the Nicholson Blowflies Equation with diffusive
delay. Section 4 will cover some numerical results for the convolution kernel
and the resulting upper and lower solutions.

##  2\. Preliminaries

In this paper we will use some standard notations such as
$\mathbb{R},\mathbb{C}$ standing for the fields of reals and complex numbers.
$\Re z$ and $\Im z$ denote the real part and imaginary part of a complex
number $z$. The space of all bounded and continuous functions from
$\mathbb{R}\to\mathbb{R}^{n}$ is denoted by $BC(\mathbb{R},\mathbb{R}^{n})$
which is equipped with the sup-norm $\|f\|:=\sup_{t\in\mathbb{R}}\|f(t)\|$.
$BC^{k}(\mathbb{R},\mathbb{R}^{n})$ stands for the space of all $k$-time
continuously differentiable functions $\mathbb{R}\to\mathbb{R}^{n}$ such that
all derivatives up to order $k$ are bounded.

If the boundedness is dropped from the above function spaces we will simply
denote them by $C(\mathbb{R},\mathbb{R}^{n})$ and
$C^{k}(\mathbb{R},\mathbb{R}^{n})$. We will use the natural order in
$BC(\mathbb{R},\mathbb{R}^{n})$ that is defined as follows: For $f,g\in
BC(\mathbb{R},\mathbb{R}^{n})$ we say that $f\leq g$ if and only if $f(t)\leq
g(t)$ for all $t\in\mathbb{R}$, and we will say that $f<g$ if $f(t)\leq g(t)$
for all $t\in\mathbb{R}$, and $f(t)\not=g(t)$ for all $t\in\mathbb{R}$.

##  3\. Main Results

We consider the following diffusive delayed Nicholson Blowflies model

(3.1) $\frac{\partial u(t,x)}{\partial
t}=\frac{\partial^{2}u(t-\tau_{1},x)}{\partial x^{2}}-\delta
u(t,x)+pu(t-\tau_{2},x)e^{-au(t-\tau_{2},x)}$

Moreover, we assume that $1<p/\delta\leq e,$ then we can find two equilibria

$U_{0}=0,\ U_{e}=\frac{1}{a}(\ln{(p/\delta)}).$

We are interested in the question: Is there a traveling wave front connecting
the two equilibrium of Eq. (3.1)?

To this end, we assume the wave translation form as $\xi=x+ct$ where $c$ is a
positive wave speed. Applying the transformation $\phi(\xi)=u(t,x)$ to Eq.
(3.1) gives

(3.2)
$\phi^{\prime\prime}(\xi-c\tau_{1})-c\phi^{\prime}(\xi)-\delta\phi(\xi)+p\phi(\xi-c\tau_{2})e^{-a\phi(\xi-c\tau_{2})}=0.$

Moving the delay out of the diffusion term via the transformation $\zeta=\xi-
r_{1},$ where $r_{1}=c\tau_{1},r_{2}=c\tau_{2}$ yields the equivalent model

(3.3)
$\phi^{\prime\prime}(\zeta)-c\phi^{\prime}(\zeta+r_{1})-\delta\phi(\zeta+r_{1})+p\phi(\zeta+(r_{1}-r_{2}))e^{-a\phi(\zeta+(r_{1}-r_{2}))}=0.$

For simplicity let $\zeta=t$ and the model becomes

(3.4)
$\phi^{\prime\prime}(t)-c\phi^{\prime}(t+r_{1})-\delta\phi(t+r_{1})+p\phi(t+(r_{1}-r_{2}))e^{-a\phi(t+(r_{1}-r_{2}))}=0.$

We invoke the following monotone conditions on $f$.

  1. (H1)

$f(\hat{0})=f(\hat{K})=0,$ where $\hat{0}$, ($\hat{U}_{e}$, respectively) is
the constant function $\phi(\theta)=0$ ($\phi(\theta)=U_{e}$, respectively),
for all $\theta\in[-\tau_{2},0]$;

  2. (H2)

There exists a positive constant $\beta$ such that

$f(\varphi)-f(\psi)+\beta(\varphi(0)-\psi(0))\geq 0$

for all $\varphi,\psi\in C([-\tau_{2},0],\mathbb{R})$ with
$0\leq\varphi(s)\leq\phi(s)\leq U_{e}$ for all $s\in[-\tau_{2},0]$;

Using the monotone conditions (H1) and (H2) we can write Equation (3.4) as

(3.5)
$\phi^{\prime\prime}(t)-c\phi^{\prime}(t+r_{1})-\delta\phi(t+r_{1})-\beta\phi(t+r_{1})+H(\phi(t))=0,$

where
$H(\phi(t))=p\phi(t+(r_{1}-r_{2}))e^{-a\phi(t+(r_{1}-r_{2}))}+\beta\phi(t+r_{1}).$
The asymptotic behavior is

$\lim_{t\to-\infty}\phi(t)=U_{0},\ \lim_{t\to\infty}\phi(t)=U_{e}.$

######  Definition 3.1.

A function $\varphi\in BC^{2}(\mathbb{R},\mathbb{R})$ is called an upper
solution (lower solution, respectively) for the wave equation (3.5) if it
satisfies the following

$\displaystyle
D\varphi^{\prime\prime}(t)-c\varphi^{\prime}(t+r_{1})+f^{c}(\varphi_{t+r_{1}})\leq
0,$
$\displaystyle(D\varphi^{\prime\prime}(t)-c\varphi^{\prime}(t+r_{1})+f^{c}(\varphi_{t+r_{1}})\geq
0,\ \mbox{respectively})$

for all $t\in\mathbb{R}$.

Here,
$f_{c}\in\mathbb{X}_{c}:=:C([-c\tau_{2},0],\mathbb{R}^{n})\rightarrow\mathbb{R}$,
defined as

$f_{c}(\psi)=f(\psi^{c}),\quad\psi^{c}(\theta):=\psi(c\theta),\quad\theta\in[-\tau_{2},0].$

######  Definition 3.2.

A function $\varphi\in C^{1}(\mathbb{R},\mathbb{R}),$ where
$\varphi,\varphi^{\prime}$ are bounded on $\mathbb{R}$,
$\varphi^{\prime\prime}$ is locally integrable and essentially bounded on
$\mathbb{R}$ (that is, $\varphi^{\prime\prime}\in L^{\infty}$), is called a
quasi- upper solution (quasi-lower solution, respectively) for the wave
equation (3.5) if it satisfies the following for almost every $t\in\mathbb{R}$

$\displaystyle
D\varphi^{\prime\prime}(t)-c\varphi^{\prime}(t+r_{1})+f^{c}(\varphi_{t+r_{1}})\leq
0,$
$\displaystyle(D\varphi^{\prime\prime}(t)-c\varphi^{\prime}(t+r_{1})+f^{c}(\varphi_{t+r_{1}})\geq
0,\ \mbox{respectively}).$

The quadratic equation $\lambda^{2}-c\lambda-\delta=0,$ which has two distinct
real roots

$\lambda_{1}=\frac{c-\sqrt{c^{2}+4\delta}}{2}<0,\
\lambda_{2}=\frac{c+\sqrt{c^{2}+4\delta}}{2}>0,$

whenever $c>2\sqrt{\delta}.$ This allows for the following lemma.

######  Lemma 3.3.

Let $c>2\sqrt{\delta}$ and consider the characteristic equation for Eq. (3.5)

(3.6) $\lambda^{2}-c\lambda e^{r_{1}\lambda}-\delta e^{r_{1}\lambda}=0.$

Define an open strip $U\subset\mathbb{C}$ by
$\\{z\in\mathbb{C}:\lambda_{1}-\varepsilon\leq\Re{z}\leq 0\\}.$ Then For
sufficiently small $r_{1},\varepsilon>0$

  1. i)

the characteristic equation has no roots on the imaginary axis,

  2. ii)

the characteristic equation has a single root continuously dependent on
$r_{1}$ in $U,$ denoted as $\eta_{1}(r_{1}).$

This is a special case of Proposition (3.1) from [2]. This allows us to see
that we have a unique bounded solution from the modified Perron Theorem, see
[11]. Thus, we have the following results from [2]. The solution for Eq. 3.5
is given by

(3.7) $F\left(\phi\right)=\int^{\infty}_{-\infty}G(t-s,r)H(\phi(s))ds,$

where there exists some positive constants $M_{1},\delta_{1}$ such that for
all $t\in\mathbb{R},$ $|G(t,r)|\leq M_{1}e^{-\delta_{1}|t|}\\\ $

######  Lemma 3.4.

Let $(\phi)$ be a quasi- upper solution (quasi-lower solution, respectively)
of Eq. (3.5). Then, $F(\phi)$ is an upper solution (lower solution,
respectively) of Eq. (3.5).

######  Theorem 3.5.

Assume $(H1)$ and $(H2)$ hold, if there is an upper $\overline{\phi}$ and a
lower solution $\underline{\phi}$ for Eq.(3.5) in $\Gamma$ such that for all
$t\in\mathbb{R}$

$0\leq\underline{\phi}(t)\leq\overline{\phi}(t).$

Then, there exists a monotone traveling wave solution to the system (3.5).

###  3.1. Quasi Upper Solutions

We will now explicitly find an acceptable quasi upper solution. The quadratic
equation $\mu^{2}-c\mu+p=0,$ which has two real roots $\mu_{1}<0<\mu_{2}$
whenever $c>2\sqrt{p}.$ In fact, we need the following lemma.

######  Lemma 3.6.

Let $c>2\sqrt{p}$ and consider the equation

(3.8) $\mu^{2}-c\mu e^{r_{1}\mu}+pe^{r_{1}\mu}=0,$

and define an open strip $V\subset\mathbb{C}$ by
$\\{z\in\mathbb{C}:0<\Re{z}<\mu_{2}+\varepsilon\\}.$ Then for sufficiently
small $r_{1},\varepsilon>0$ such that $V\cap\\{\mu_{2}\\}=\emptyset.$ Then Eq.
(3.8) has a single root continuously dependent on $r_{1}$ in $V,$ denoted as
$\eta_{2}(r_{1}).$ Moreover, $\eta_{2}(r_{1})$ is real and

(3.9) $\lim_{r_{1}\to 0}\eta_{2}(r_{1})=\mu_{2}.$

######  Claim 3.7.

For sufficiently small $r_{1}$ and $c>2\sqrt{p}$, and $\eta_{2}$ to be the
root of Eq. (3.8) in $V$, then the function

$\varphi_{1}(t)=\left\\{\begin{array}[c]{l}\frac{U_{e}}{2}e^{\eta_{2}t},\;\;\;\;\;\;\quad
t\leq t_{0},\\\ U_{e}(1-\frac{1}{2}e^{-\eta_{2}t}),\quad
t>t_{0}\end{array}\right.$

is a quasi-upper solution of (3.5).

###### Proof.

From elementary calculus it is easy to see

$\displaystyle\varphi_{1}^{\prime}(t)$
$\displaystyle=\left\\{\begin{array}[c]{l}\frac{U_{e}\eta_{2}}{2}e^{\eta_{2}t},\quad
t\leq 0,\\\ \frac{U_{e}\eta_{2}}{2}e^{-\eta_{2}t},\quad
t>0\end{array}\right.,\quad\varphi_{1}^{\prime\prime}(t)=\left\\{\begin{array}[c]{l}\frac{U_{e}\eta_{2}^{2}}{2}e^{\eta_{2}t},\quad
t\leq 0,\\\ \frac{-U_{e}\eta_{2}^{2}}{2}e^{-\eta_{2}t},\quad
t>0\end{array}\right.$

Note that $\varphi_{1}^{\prime}$ is continuous and bounded on $\mathbb{R}$ and
$\varphi_{1}^{\prime\prime}$ exist and is continuous everywhere and bounded
except for $t=0$. We will take $U_{e}=1,$ for brevity, since it appears in
every term The proof can now be completed in three cases.
Case 1 $t<-r_{1}$: We have

$\displaystyle\phi^{\prime\prime}(t)-c\phi^{\prime}(t+r_{1})-\delta\phi(t+r_{1})+p\phi(t+(r_{1}-r_{2}))e^{-a\phi(t+(r_{1}-r_{2}))}$
$\displaystyle=\frac{\eta_{2}^{2}}{2}e^{\eta_{2}t}-\frac{c\eta_{2}}{2}e^{\eta_{2}(t+r_{1})}-\frac{\delta}{2}e^{\eta_{2}(t+r_{1})}+\frac{p}{2}e^{\eta_{2}(t+r_{1})}e^{-a\phi(t+(r_{1}-r_{2}))}$
$\displaystyle\leq\left(\frac{\eta_{2}^{2}}{2}-\frac{c\eta_{2}}{2}e^{\eta_{2}r_{1}}+\frac{p}{2}e^{\eta_{2}r_{1}}\right)e^{\eta_{2}t}-\frac{\delta}{2}e^{\eta_{2}(t+r_{1})}$
$\displaystyle=-\frac{\delta}{2}e^{\eta_{2}(t+r_{1})}\leq 0.$

Case 2 $-r_{1}\leq t\leq 0$: We have

$\displaystyle\phi^{\prime\prime}(t)-c\phi^{\prime}(t+r_{1})-\delta\phi(t+r_{1})+p\phi(t+(r_{1}-r_{2}))e^{-a\phi(t+(r_{1}-r_{2}))}$
$\displaystyle=\frac{\eta_{2}^{2}}{2}e^{\eta_{2}t}-\frac{c\eta_{2}}{2}e^{-\eta_{2}(t+r_{1})}-\delta\phi(t+r_{1})+\frac{p}{2}e^{\eta_{2}(t+r_{1}-r_{2})}e^{-a\phi(t+(r_{1}-r_{2}))}$
$\displaystyle\leq\frac{\eta_{2}^{2}}{2}e^{\eta_{2}t}-\frac{c\eta_{2}}{2}e^{-\eta_{2}(t+r_{1})}-\delta\phi(t+r_{1})+\frac{p}{2}e^{\eta_{2}(t+r_{1})}$
$\displaystyle=\frac{\eta_{2}^{2}}{2}e^{\eta_{2}t}-\frac{c\eta_{2}}{2}e^{\eta_{2}(t+r_{1})}+\frac{p}{2}e^{\eta_{2}(t+r_{1})}+\frac{c\eta_{2}}{2}e^{\eta_{2}(t+r_{1})}-\frac{c\eta_{2}}{2}e^{-\eta_{2}(t+r_{1})}-\delta\phi(t+r_{1})$
$\displaystyle\left(\frac{\eta_{2}^{2}}{2}-\frac{c\eta_{2}}{2}e^{\eta_{2}r_{1}}+\frac{p}{2}e^{\eta_{2}r_{1}}\right)e^{\eta_{2}t}+c\eta_{2}\sinh(r_{1}\eta_{2})-\delta\phi(t+r_{1})$
$\displaystyle=c\eta_{2}\sinh(r_{1}\eta_{2})-\delta\phi(t+r_{1}),$

because $\eta_{2}$ is a root of Eq. (3.8). When $r_{1}\to 0,\
c\eta_{2}\sinh(r_{1}\eta_{2})\to 0.$ Thus, there is a small enough $r_{1}$
such that

$c\eta_{2}\sinh(r_{1}\eta_{2})-\delta\phi(t+r_{1})\leq 0.$

Case 3 $0\leq t$: We have

$\displaystyle\phi^{\prime\prime}(t)-c\phi^{\prime}(t+r_{1})-\delta\phi(t+r_{1})+p\phi(t+(r_{1}-r_{2}))e^{-a\phi(t+(r_{1}-r_{2}))}$
$\displaystyle=\frac{-\eta_{2}^{2}}{2}e^{-\eta_{2}t}-\frac{c\eta_{2}}{2}e^{-\eta_{2}(t+r_{1})}+p\left(1-\frac{1}{2}e^{-\eta_{2}(t+(r_{1}-r_{2}))}\right)e^{-a\phi(t+(r_{1}-r_{2}))}-\delta\phi(t+r_{1})$
$\displaystyle\leq\frac{-\eta_{2}^{2}}{2}e^{-\eta_{2}t}-\frac{c\eta_{2}}{2}e^{-\eta_{2}(t+r_{1})}+p\left(1-\frac{1}{2}e^{-\eta_{2}(t+(r_{1}-r_{2}))}\right)-\delta\phi(t+r_{1})$
$\displaystyle=\frac{-\eta_{2}^{2}}{2}e^{-\eta_{2}t}+\frac{c\eta_{2}}{2}e^{-\eta_{2}t+\eta_{2}r_{1}}-\frac{p}{2}e^{-\eta_{2}t+\eta_{2}r_{1}}+\frac{p}{2}e^{-\eta_{2}t+\eta_{2}r_{1}}-\frac{c\eta_{2}}{2}e^{-\eta_{2}t+\eta_{2}r_{1}}-\frac{c\eta_{2}}{2}e^{-\eta_{2}(t+r_{1})}$
$\displaystyle+p\left(1-\frac{1}{2}e^{-\eta_{2}(t+(r_{1}-r_{2}))}\right)-\delta\phi(t+r_{1}).$

∎

Using the fact that $\eta_{2}$ is a root of Eq. (3.8) we see the following
simplification

$\displaystyle=\frac{p}{2}e^{-\eta_{2}t+\eta_{2}r_{1}}-\frac{c\eta_{2}}{2}e^{-\eta_{2}t+\eta_{2}r_{1}}-\frac{c\eta_{2}}{2}e^{-\eta_{2}(t+r_{1})}+p\left(1-\frac{1}{2}e^{-\eta_{2}(t+(r_{1}-r_{2}))}\right)-\delta\phi(t+r_{1})$
$\displaystyle=-c\eta_{2}e^{-\eta_{2}t}\cosh(\eta_{2}r_{1})+\frac{p}{2}e^{-\eta_{2}t}\left(e^{\eta_{2}r_{1}}-e^{\eta_{2}(r_{1}-r_{2})}\right)-\delta\phi(t+r_{1})+p.$

Noticing that when $r_{1}\to 0,$ implies $\cosh(\eta_{2}r_{1})\to 1$ and
$e^{\eta_{2}r_{1}}-e^{\eta_{2}(r_{1}-r_{2})}\to 1-e^{-\eta_{2}r_{2}}$ we can
take $r_{1},r_{2}$ small enough such that
$e^{\eta_{2}r_{1}}-e^{\eta_{2}(r_{1}-r_{2})}\approx o(r_{1}-r_{2})$. This
allows one to see that for sufficiently small $r_{1},r_{2}$

$\displaystyle=-c\eta_{2}e^{-\eta_{2}t}\cosh(\eta_{2}r_{1})+\frac{p}{2}e^{-\eta_{2}t}\left(e^{\eta_{2}r_{1}}-e^{\eta_{2}(r_{1}-r_{2})}\right)-\delta\phi(t+r_{1})+p$
$\displaystyle\approx\left(-c\eta_{2}+\frac{p}{2}o(r_{1}-r_{2})\right)e^{-\eta_{2}t}-\delta\phi(t+r_{1})+p$
$\displaystyle\leq-c\eta_{2}+\frac{p}{2}o(r_{1}-r_{2})-\delta\phi(t+r_{1})+p.$

Moreover,

$\lim_{c\to 2\sqrt{p}}-c\mu_{1}=-p,$

and

$\lim_{r_{1}\to 0}\eta_{2}(r_{1})=\mu_{1}$

we have that

$\displaystyle-c\eta_{2}+\frac{p}{2}o(r_{1}-r_{2})-\delta\phi(t+r_{1})+p$
$\displaystyle\approx\frac{p}{2}o(r_{1}-r_{2})-\delta\phi(t+r_{1}).$

Thus, we can take $r_{1},r_{2}$ small enough such that

$\frac{p}{2}o(r_{1}-r_{2})<\delta\phi(t+r_{1}).$

This proves the result.

In order to construct quasi lower solutions we will look at a function,
defined as

$\displaystyle f(t)$ $\displaystyle=a(t-T)^{3}+b(t-T)^{2}+\frac{1}{2}.$

Furthermore, for some large $T>0$ we have the properties:

  1. i)

This bridges smoothly the function $e^{\eta_{1}t}/4$ and the constant function
$1/2$

  2. ii)

$f(-T)=(1/4)e^{\eta_{1}T}$, $f^{\prime}(-T)=(-\eta_{1}/4)e^{\eta_{1}T}$,
$f^{\prime}(T)=0$, $f(T)=1/2$.

It was shown in Barker and Nguyen, [2] that

$\displaystyle a$
$\displaystyle=\frac{\eta_{1}Te^{-\eta_{1}T}+e^{-\eta_{1}T}-2}{16T^{3}}$
$\displaystyle b$
$\displaystyle=\frac{-\eta_{1}Te^{-\eta_{1}T}+6\left(\frac{e^{-\eta_{1}T}}{4}-\frac{1}{2}\right)}{8T^{2}},$

as well as the following claim.

######  Claim 3.8.

Define $f(t)$ to be the bridge function from above, then

(3.10) $\lim_{T\to\infty}\sup_{-T\leq t\leq
T}\max\\{|f^{\prime}(t)|,|f^{\prime\prime}(t)|\\}=0.$

######  Claim 3.9.

For sufficiently small $r_{1}$ and $c>2\sqrt{\delta}$, and $\eta_{1}$ to be
the positive root of Eq. (3.6), then the function

$\underline{\varphi}(t)=\begin{cases}\frac{e^{\eta_{1}t}}{4},\ t<-T,\\\ f(t),\
-T\leq t\leq T\\\ \frac{1}{2},\ t>T,\end{cases}$

is a quasi-upper solution of (3.5).

###### Proof.

We will do this in cases just as before.

Case 1: $t\leq-T-r_{1}$

$\displaystyle\phi^{\prime\prime}(t)-c\phi^{\prime}(t+r_{1})-\delta\phi(t+r_{1})+p\phi(t+(r_{1}-r_{2}))e^{-a\phi(t+(r_{1}-r_{2}))}$
$\displaystyle=\frac{\eta_{1}^{2}}{2}e^{\eta_{1}t}-\frac{c\eta_{1}}{2}e^{\eta_{1}(t+r_{1})}-\frac{\delta}{2}e^{\eta_{1}(t+r_{1})}+\frac{p}{2}e^{\eta_{1}(t+r_{1})}e^{-a\phi(t+(r_{1}-r_{2}))}$
$\displaystyle=\left(\frac{\eta_{1}^{2}}{2}-\frac{c\eta_{1}}{2}e^{\eta_{1}r_{1}}-\frac{\delta}{2}e^{\eta_{1}r_{1}}\right)e^{\eta_{1}t}+\frac{p}{2}e^{\eta_{1}(t+r_{1})}e^{-a\phi(t+(r_{1}-r_{2}))}$
$\displaystyle=\frac{p}{2}e^{\eta_{1}(t+r_{1})}e^{-a\phi(t+(r_{1}-r_{2}))}\geq
0,$

because $\eta_{1}$ is a root for Eq. (3.6).

Case 2: $-T-r_{1}\leq t\leq T$ This case follows from the fact that on this
interval

$\displaystyle\sup_{-T-r_{1}\leq t\leq
T}|\underline{\varphi}_{1}^{\prime\prime}(t)|\pm
c|\underline{\varphi}_{1}^{\prime}(t+r_{1})|=\sup_{-T\leq t\leq
T}|f^{\prime}(t)|\pm c|f^{\prime\prime}(t)|$

that could be made as small as we like by taking $T$ sufficiently large, and
$p>\delta.$ Thus, we have for some large $T$

$\displaystyle\phi^{\prime\prime}(t)-c\phi^{\prime}(t+r_{1})-\delta\phi(t+r_{1})+p\phi(t+(r_{1}-r_{2}))e^{-a\phi(t+(r_{1}-r_{2}))}$
$\displaystyle=-\delta\phi(t+r_{1})+p\phi(t+(r_{1}-r_{2}))e^{-a\phi(t+(r_{1}-r_{2}))}+o(T)\geq
0.$

This case has been proven.

Case 3: $T\leq t$ This case is trivially due to the fact that the function is
constant. ∎

######  Corollary 3.10.

Assume that $c>2\sqrt{p}$ is given. Then, the Eq. (3.5) has a traveling wave
solution $u(x,t){\phi(x+ct)}$ for sufficiently small delays
$\tau_{1},\tau_{2}$.

##  4\. Numerical Simulations

In this section we will construct, via a specific example numerical upper and
lower solutions for Eq. (3.5). Using the formula found in Theorem (4.1) from
Mallet-Peret, [11] we have

(4.1) $G(t,r)=-\frac{1}{2\pi}\int_{-\infty}^{\infty}\frac{e^{i\xi
t}}{-\xi^{2}-ci\xi e^{i\xi r_{1}}-\delta e^{i\xi r_{1}}}d\xi$

A relatively simple numerical scheme should be appropriate due to the nature
of the quasi upper and lower solutions and the smoother upper and lower
solutions. To this end, we have the following lemma.

######  Lemma 4.1.

Let $G(t,r)$ be the Green’s function found in from Eq. (4.1) , then

  1. i)

for all $N\in\mathbb{N},\ t,\xi\in\mathbb{R}$ there is some positive constant,
$K$ independent of $N$ such that

$\left|G(-N,r)+G(N,r)\right|<K.$

  2. ii)

For any small positive number $\varepsilon$ there is some sufficiently large
$N\in\mathbb{N}$ dependent upon $\varepsilon$ such that for all
$t,\xi\in\mathbb{R}$

$\left|G(-N,r)+G(N,r)\right|<\varepsilon.$

###### Proof.

The proof for both parts are straight forward due to the fact that for all
$t\in\mathbb{R}$ there is some positive constant, $\delta_{1}$ such that
$||G(t,r)||\leq K_{1}e^{-\delta_{1}|t|}.$ In fact, part $i.)$ can be shown via
induction on $N$. Take $N=1$, then

$\displaystyle\left|G(-N,r)+G(N,r)\right|\leq 2K_{1}e^{-\delta_{1}}<2K_{1}.$

Take $K=2K_{1}$ then the result follows. In order to show part $ii.)$ fix
$\varepsilon>0$ then for some $N\in\mathbb{N}$ such that

$N\geq\ln\left({\frac{\varepsilon}{2K_{1}}}^{\frac{-1}{\delta_{1}}}\right)$

we have the following estimate:

$\displaystyle\left|G(-N,r)+G(N,r)\right|\leq
2K_{1}e^{-\delta_{1}N}<\varepsilon.$

∎

This allows us to disregard the tails of the Green’s function and focus on
some finite interval in order to numerically approximate the integral for any
fixed $t\in\mathbb{R}.$ In fact, the interval can be chosen to be relatively
small. We know that the following iteration is convergent

$\phi_{n}=\int^{\infty}_{-\infty}G(t-s,r)H(\phi_{n-1}(s))ds,$

where $H$ We will take $r_{1}=1,r_{2}=1/4,p=2,\delta=1,c=2\sqrt{2},N=50,T=1$.
then we will approximate

$G(t,r)\approx\ -\frac{1}{2\pi}\int_{-50}^{50}\frac{e^{i\xi t}}{-\xi^{2}-ai\xi
e^{i\xi}-be^{i\xi}}d\xi.$

Furthermore, since $\phi_{n}\geq 0,$ we have the approximation

$\phi_{n}=\left|\int^{\infty}_{-\infty}G(t-s,r)H(\phi_{n-1}(s))ds\right|\leq\int^{\infty}_{-\infty}\left|G(t-s,r)H(\phi_{n-1}(s))\right|ds.$

With this in mind, we know that $\left|G(t-s,r)\right|\leq
K_{1}e^{-\delta_{1}|t|}.$ Shifting the line of integration to the parallel
line $z=\xi+i|\lambda_{1}|,$ where $\lambda_{1}$ is the negative root of Eq.
(3.6) gives the following

$\left|G(t-s,r)\right|\leq K_{1}e^{\lambda_{1}|t|}.$

A composite Simpson’s rule will be used to approximate

$K_{1}\approx\frac{1}{2\pi}\int_{-50}^{50}\frac{1}{\left(\xi+i|\lambda_{1}|\right)^{2}+\left[2i\left(\xi+i|\lambda_{1}|\right)+1\right]e^{(\xi+i|\lambda_{1}|)}}d\xi.$

For brevity, we denote

$f(\xi)=\frac{1}{2\pi}\int_{-50}^{50}\frac{1}{\left(\xi+i|\lambda_{1}|\right)^{2}+\left[2i\left(\xi+i|\lambda_{1}|\right)+1\right]e^{(\xi+i|\lambda_{1}|)}},$

then it is well know that the composite Simpson’s rule can be written as

$I_{n}=\int_{-50}^{50}f(\xi)\approx\frac{h}{3}\left[f(-50)+4\sum_{i=1}^{n/2}f(\xi_{2i-1})+2\sum_{i=1}^{n/2-1}f(\xi_{2i})+f(50),\right]$

where $h=100/n,\ n$ is the number of sub intervals. The results for various
step sizes (calculated in MATLAB) can be found in the table below, as well as
plots for the quasi upper and lower solutions below

$n$ | $h$ | $|I_{n}|$
---|---|---
100 | 1 | .2861
1000 | .1 | .3064
10000 | .01 | .3066
100000 | .001 | .3067
1000000 | .0001 | .3067

|
---|---
Quasi Upper Solution |  Quasi Lower Solution

## References

  * [1] J. Al-Omari, S. Gourley, Monotone travelling fronts in an age-structured reaction-diffusion model of a single species. J Math Biol 45, 294–312 (2002)
  * [2] W.K. Barker, N. Van Minh, Traveling waves in reaction-diffusion equations with delay in both diffusion and reaction terms arXiv preprint arXiv:2301.11504 (2023). Submitted for final publication
  * [3] A. Boumenir, N. Van Minh, Perron Theorem in the Monotone Iteration Method for Traveling Waves in Delayed Reaction-Diffusion Equations, J. Differential Equations 7 (2008) 1551-1570.
  * [4] K. Deng, Y. Wu, On the diffusive Nicholson’s blowflies equation with distributed delay. Applied Mathematics Letters 50 (2015) 126-132.
  * [5] S.A. Gourley, S. Ruan, Dynamics of the diffusive Nicholson’s blowflies equation with distributed delay, Proc. Roy. Soc. Edinburgh 130A (2000) 1275-1291.
  * [6] W.S.C. Gurney, S.P. Blythe, R.M. Nisbet, Nicholson’s blowflies revisited, Nature 287 (1980) 17-21.
  * [7] C. Huang and B. Liu, Traveling wave fronts for a diffusive Nicholson’s Blowflies equation accompanying mature delay and feedback delay, Applied Mathematics Letters 134 (2022), Paper No. 108321, 6 pp.
  * [8] L. Huang and Z. Xu, Traveling wave fronts of a diffusive Nicholson’s Blowflies equation with two delays, Applied Mathematics Letters, 139 (2023) 108545, 7 pp.
  * [9] C.K Lin, M. Mei, On travelling wavefronts of Nicholson’s blowflies equation with diffusion, Proceedings of the Royal Society of Edinburgh 140A (2010) 135-152.
  * [10] S. Ma, Traveling Waves for Delayed Reaction-Diffusion Systems Via a Fixed Point Theorem, J. Differential Equations, 171 (2001) 294-314.
  * [11] J. Mallet-Paret. The Fredholm Alternative for Function Differential Equations of Mixed Type. J. Dynamics and Differential Equations, 11 (1999). p. 1-47.
  * [12] M. Mei, J. W.-H. So, M. Y. Li and S. S. P. Shen. Asymptotic stability of travelling waves for Nicholson’s blowflies equation with diffusion. Proc. R. Soc. Edinb. A 134 (2004), 579-594.
  * [13] A. J. Nicholson. Competition for food amongst Lucilia cuprina larvae. In Proc. VIII Int. Congr. of Entomology, Stockholm, (Stockholm: Axel R Elfstroms Boktryckeri AB, 1948) 277-281
  * [14] A. J. Nicholson. An outline of the dynamics of animal populations. Aust. J. Zool. 2 (1954), 9–65.
  * [15] J. Pruss. Bounded Solutions of Volterra Equations. SIAM J. Math Anal., 19 (1988), 133-149.
  * [16] K. Schaaf, Asymptotic Behavior and Traveling Wave Solutions for Parabolic Functional Differential Equations, Transactions of the American Mathematical Society 302 (1987) 587-615.
  * [17] J. W.-H. So and Y. Yang. Dirichlet problem for the diffusive Nicholson’s blowflies equation. J. Differential Equations 150 (1998), 317-348.
  * [18] J. W.-H. So, J. Wu and Y. Yang. Numerical steady state and Hopf bifurcation analysis on the diffusive Nicholson’s blowflies equation. Appl. Math. Computat. 111 (2000), 53-69.
  * [19] J. W.-H. So and X. Zou. Traveling waves for the diffusive Nicholson’s blowflies equation. Appl. Math. Computat. 122 (2001), 385-392
  * [20] H. Smith, Monotone Dynamical Systems: An Introduction to the Theory of Competitive and Cooperative Systems, Am. Math. Soc., Providence, RI, 1995.
  * [21] A. I. Volpert, V. A. Volpert, and V. A. Volpert. Traveling Wave Solutions of Parabolic Systems. Translations of Mathematical Monographs, Vol. 140, Am. Math. Soc., Providence, RI, 1994.
  * [22] J. Wu, X. Zou, Traveling Wave Fronts of Reaction-Diffusion equations with Delay, Journal of Dynamics and Differential Equations 13 (2001) 651-687.
  * [23] Y. Yang and J. W.-H. So. Dynamics for the diffusive Nicholson’s blowflies equation. In Dynamical systems and differential equations, vol. II (ed. W. Chen and S. Hu), (Springfield, MO: Southwest Missouri Sate University Press, 1998). 333-352
  * [24] X. Q. Zhao, W.Wang, Fisher waves in an epidemic model. Discrete Contin. Dyn. Syst. Ser. B 4 (2004) 1117-1128.