(cvpr) Package cvpr Warning: Incorrect font size specified - CVPR requires
10-point fonts. Please load document class ‘article’ with ‘10pt’ option (cvpr)
Package cvpr Warning: Incorrect paper size - CVPR uses paper size ‘letter’.
Please load document class ‘article’ with ‘letterpaper’ option

# Scene Consistency Representation Learning for Video Scene Segmentation

Haoqian Wu1,2,3,4∗, Keyu Chen2∗, Yanan Luo2, Ruizhi Qiao2, Bo Ren2,
Haozhe Liu1,3,4,5, Weicheng Xie${}^{1,3,4}{}^{\dagger}$, Linlin Shen1,3,4
1 Computer Vision Institute, Shenzhen University 2 Tencent YouTu Lab
3 Shenzhen Institute of Artificial Intelligence and Robotics for Society
4 Guangdong Key Laboratory of Intelligent Information Processing 5 KAUST
<EMAIL_ADDRESS>{yolochen, ruizhiqiao<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>{wcxie<EMAIL_ADDRESS>
###### Abstract

A long-term video, such as a movie or TV show, is composed of various scenes,
each of which represents a series of shots sharing the same semantic story.
Spotting the correct scene boundary from the long-term video is a challenging
task, since a model must understand the storyline of the video to figure out
where a scene starts and ends. To this end, we propose an effective Self-
Supervised Learning (SSL) framework to learn better shot representations from
unlabeled long-term videos. More specifically, we present an SSL scheme to
achieve scene consistency, while exploring considerable data augmentation and
shuffling methods to boost the model generalizability. Instead of explicitly
learning the scene boundary features as in the previous methods, we introduce
a vanilla temporal model with less inductive bias to verify the quality of the
shot features. Our method achieves the state-of-the-art performance on the
task of Video Scene Segmentation. Additionally, we suggest a more fair and
reasonable benchmark to evaluate the performance of Video Scene Segmentation
methods. The code is made available.1

Figure 1: An illustration of representation learning methods from the shot-to-
scene perspective. Several continuous shots are shown in Fig. (a), where
existing SSL approaches obtain positive pairs from the adjacent shots (e.g.,
by performing Nearest Neighbor (NN) Selection [1]). While we propose to look
further for scenes that are often crossed over, as Scene A/C and Scene B/E
shown in Fig. (b), where positive samples are explored in a broader region and
the shots are clustered to the same scene in the feature representation space,
i.e., Fig. (c). Best viewed in color. ††footnotetext: ∗Equal Contribution
†Corresponding Author††footnotetext:
1https://github.com/TencentYoutuResearch/SceneSegmentation-SCRL

##  1 Introduction

In the process of video creation, to make the story more compelling, the
editor will use various editing techniques, such as montage, one shot to the
end, etc. Quickly switching between stories and scenes makes the movie plot
tighter, e.g. inserting outdoor battle scenes into indoor dialogue scenes, as
shown in Fig. 1 (b), making the scene transition more intriguing and
unpredictable, thus the task of Video Scene Segmentation turns out to be
rather challenging. Hence, it is essential to understand the high-level
semantic information of each scene in the long-term video.

There has been extensive studies dealing with video understanding tasks on
datasets where the individual video clip is typically short, while requiring a
lot of labor to segment uncurated videos into short videos by category.
Although some studies focus on splitting the long video into smaller segments,
e.g., the methods of Action Spotting [2, 3, 4, 5] aim to locate the positions
of the beginning and ending of the action, however, they are the category-
aware approaches. By contrast, Video Scene Segmentation is a category-agnostic
task that only the scene boundary label is available, and it’s very confusing
to classify a scene fragment taxonomically.

Since a long-term video is inherently structured in a specific way, a sequence
of frames can be divided into shots or scenes in terms of the granularity of
semantics [6] [7]. More specifically, a shot contains only continuous frames
taken by the camera without interruption, and a scene is composed of
successive shots and describes the same short story. For detecting shot
boundaries, [8] [7] split a video into many separate shots using lower-level
visual context. Based on this, many mainstream approaches of Video Scene
Segmentation [9] [10] [6] [1] determine scene boundaries by exploring semantic
correlations among the adjacent shots.

While computer vision tasks suffer from the high cost of manual annotation,
Self-Supervised Learning (SSL) based methods [11, 12, 13, 14, 15, 16, 17, 18]
are proposed to train a general feature extractor using unlabeled data. By
leveraging a small amount of annotated data for training, these SSL methods
can achieve appealing feature representation to even rival some supervised
learning methods. For Video Scene Segmentation, [1] proposes to narrow the
feature representation distance of the most similar shot pair in a local
region, it significantly surpasses the supervised learning method [6] by
employing a mere MLP classifier. However, in current SSL methods on the task
of Video Scene Segmentation, the strategy of positive sample selection,
pretraining protocol, evaluation metric and downstream model are not well
discussed or addressed.

To achieve this goal, we propose a self-supervised learning scheme to learn
better representations, as well as the evaluation metric for the task of Video
Scene Segmentation. The contributions of this paper are summarized as follows:

  * •

A representation learning scheme based on Scene Consistency is proposed to
obtain better shot representations on the unlabeled long-term video.

  * •

A simple yet effective temporal model with less inductive bias is proposed to
assess the quality of the shot representation for the downstream Video Scene
Segmentation task.

  * •

A benchmark that is more fair and reasonable is introduced for both
pretraining and evaluation. More importantly, the proposed method outperforms
the state-of-the-art methods under all the protocols, and can significantly
improve the performance of existing supervised methods without bells and
whistles.

Figure 2: The pipeline of the proposed method. (a) Unsupervised Representation
Learning Stage for learning shot representations, where $Map(i)$ is the
mapping function for selecting positive samples. (b) Supervised Video Scene
Segmentation Stage, where the quality of the shot representations is evaluated
under the protocols of the non-temporal (MLP) and temporal (Bi-LSTM [19])
models.

##  2 Related Work

Self-Supervised Learning in Images and Videos. To address the problems of the
insufficient and expensive manual annotation, many approaches explore the
inherent knowledge in unlabeled data by designing a lot of pretext tasks,
including predicting the transformations of images, e.g., image rotation [20],
inpainting [21], colorizing [22], jigsaw [23], etc. In short, these Self-
Supervised Learning (SSL) methods use the information explored from the data
themselves for the supervision. Recently, [11, 12, 13, 14, 15, 16, 17, 18]
introduce the contrastive similarity metrics to learn invariant feature
representation of various views augmented from the original image, where
strong data augmentations [13] are frequently used in image-level SSL methods
to improve the robustness of the learned representations. From another aspect,
by finetuning the model with a small amount of labeled data, SSL methods can
achieve competitive performance compared with supervised learning methods,
furthermore, the pretrained model can be used in specific downstream tasks.
For video-oriented SSL methods, [24, 25, 26, 27, 28, 29, 30] show the
appealing performance and potential on the task of video classification, while
their positive pairs are selected from the adjacent clips within a same video.
Meanwhile, most of studies are based on short videos and the quality of
learned features is assessed based on video classification. Hence, it is
meaningful to explore a suitable SSL scheme for tasks with long-term videos.

Video Shot Boundary Detection and Scene Segmentation. For Video Scene
Segmentation, shot boundary detection is often conducted in advance, which is
specified as a task of locating the transition positions in videos based on
the similarity of the frames. 3D convolutional networks and color histogram
differencing [8] are used to identify the transition boundaries. Based on the
shot boundaries, [6] learns the local and global shot representations and
utilizes them to split the continuous shots into scenes according to the
transition of the story. More specifically, identification of each shot’s
segmentation point is treated as a binary classification, which is free to the
location of the shot. [1] leverages unlabeled video data to obtain shot
representations, which outperforms many supervised learning methods on the
downstream task of Video Scene Segmentation. However, this method is
pretrained on the entire video data of MovieNet [31] that include the testing
videos, i.e. the training protocol is inconsistent with that of conventional
Self-Supervised Learning methods [24] [25]. For evaluating Video Scene
Segmentation approaches, the datasets of OVSD [32], BBC planet earth [10],
MovieNet [31] and AdCuepoints [1] are frequently employed.

In this work, we propose an unsupervised representation learning method based
on scene consistency and a reasonable evaluation scheme for Video Scene
Segmentation task.

##  3 Methodology

As shown in Fig. 2, we aim to obtain scene consistency representations on
unlabeled long-term videos and design a more reasonable benchmark to verify
the quality of the extracted features on the task of Video Scene Segmentation.
To this end, we (i) propose a Self-Supervised Learning scheme based on a novel
non-temporal selection strategy to achieve scene consistency from various
shots, and (ii) introduce a vanilla temporal model with less inductive bias as
well as the corresponding benchmark for this segmentation task.

###  3.1 Consistency based Representation Learning

Approaches of Self-Supervised Learning (SSL) aim to model representation
consistency to enhance network robustness against various variations, e.g.
spatial or temporal transformations. In this work, we use an SSL framework of
Siamese network to achieve the representation consistency.

More precisely, for a given query shot, the objective is to (i) maximize the
similarity between the representations of query shot and positive samples,
i.e., key shots; (ii) minimize the similarity of the negative sample pairs if
they exist. As shown in Fig. 2 (a), the input samples $X$ are first augmented,
i.e., $Q=Aug_{Q}(X),K=Aug_{K}(X)$, and the $i$-th positive pair
$\\{q,k^{+}\\}$ is formulated as follows:

$\displaystyle\\{q,k^{+}\\}=\\{f\left(Q[i]\mid\theta_{Q}\right),\
f\left(K[MAP(i)]\mid\theta_{K}\right)^{+}\\}$ (1)

where $[\cdot]$ stands for the indexing operation,
$f\left(\cdot\mid\theta_{Q}\right)$ and $f\left(\cdot\mid\theta_{K}\right)$
are the encoders with parameters $\theta_{Q}$ and $\theta_{K}$, respectively,
$MAP(i)$ is the mapping function for selecting positive samples.

For the selection of positive samples in SSL methods based on video data,
three selection strategies are frequently employed, i.e., Self-Augmented [14],
Random [27] and Nearest Neighbor (NN) [1] selections. For clarity, the three
conventional selection strategies for positive samples are represented in Fig.
3 (a)-(c).

Figure 3: The illustration of four different selection strategies for positive
pairs. Best viewed in color.

####  3.1.1 Conventional Positive Sample Selections

Self-Augmented Selection. As image-level SSL approaches, the augmented view of
one shot is frequently used as its positive sample, as shown in Fig. 3 (a),
the mapping function, i.e. the identity mapping, is employed as follows:

$\displaystyle MAP_{SA}(i)=i$ (2)

Random Selection. As some SSL methods [27] [24] for video classification, we
select two adjacent shots of the same video as the positive pair, as shown in
Fig. 3 (b), and the mapping function can be formulated as follows:

$\displaystyle MAP_{RS}(i)=max(i+j,0)$ (3)

where $\ j\in\\{-n,-n+1,\ ...\ ,n-1,n\\}$ and $n$ denotes the size of the
search region around the $i$-th shot.

Nearest Neighborhood (NN) Selection. As shown in Fig. 3 (c), [1] proposed to
select the positive shot with the closest representation distance to the query
shot within a fixed range, and the mapping function is as follows:

$\displaystyle MAP_{NN}(i)=\arg\max_{j\in
I_{M}}f\left(Q[i]\mid\theta_{Q}\right)\cdot
f\left(Q[max(j,0)]\mid\theta_{Q}\right)$ (4)

where $I_{M}=\\{i-m,\ ...\ ,\ i-1,\ i+1,\ ...\ ,i+m\\}$, $I_{M}$ stands for
the indices of candidate samples for NN selection, $m$ is the search region
size of a given shot, and $2m+1$ is the length of the sliding window.

####  3.1.2 Scene Consistency Selection

In this work, we propose the Scene Consistency Selection, while exploring
considerable data augmentation and shuffling methods for the task of Video
Scene Segmentation.

Positive Sample Selection with Scene Consistency. As shown in Fig. 1, for the
video with non-linear narrative, previous selection methods may not work in
the case that the most matching shots are far away. Therefore, we propose to
select positive shot pair based on scene consistency, while the main advantage
over Random/NN Selection is that our method is non-temporal, which is free to
the shots order.

We argue that scene consistency is critical for the training on the unlabeled
long-term videos due to the three reasons: (i) the similar shots in the same
scene may be far away; (ii) the greater feature spacing between scenes is
beneficial to the downstream task of Video Scene Segmentation, and it can be
achieved by maximizing inter-scene distance and minimizing intra-scene
distance; (iii) while the NN selection may result in a trivial objective, due
to the maximization of the similarity of the sample pairs that maybe already
the closest, the scene consistency enables the selection to achieve a more
non-trivial objective.

For the proposed scene consistency-based selection, we perform online
clustering of samples in a batch, and use the cluster center sample as the
positive sample with respect to the query shot, as shown in Fig. 3 (d). The
specified mapping function is formulated as follows:

$\displaystyle MAP_{SC}(i)=\arg\min_{j\in
I_{C}}\left\|f(Q[i]\mid\theta_{Q})-f(Q[j]\mid\theta_{Q})\right\|_{2}$ (5)

where $\ I_{C}=\\{ic_{1},ic_{2},\ ...\ ,ic_{\\#class}\\}$ stands for the
indices of cluster centers, $\\#class$ is the number of cluster centers.

While center sample reflects the cluster-specified common information, we
additionally use the query-specific individual information to generate the
positive sample. Unlike the conventional multiple-instance learning [29],
which treats center and query samples as multiple positive samples, we propose
to construct the soft positive sample, namely Soft-Scene Consistency (SC)
sample as follows:

$\displaystyle k_{Soft-SC}=\gamma k_{SA}+(1-\gamma)k_{SC}$ (6)

where $\gamma$ is a trade-off parameter, $k_{SA}$ and $k_{SC}$ are the key
(positive) samples selected by Self-Augmented Selection and Scene Consistency
Selection.

Scene Consistency Data Augmentation. Since the early stage of training is not
stable, too much color augmentations, e.g. grayscale transformations, color
jitter, etc., misguide the selection of positive samples, namely as Selection
Shift. In this case, the model focuses more on non-semantic information. To
solve this problem, some studies [33] directly omit color augmentations for
better performance. By contrast, we propose Asymmetric Augmentation to
alleviate the influence of Selection Shift and use color augmentation to
further improve the performance. More specifically, augmentations without the
color transformation are used in $Aug_{Q}$ to get more accurate and scene
consistent positive samples, while the color data augmentation operations are
performed in $Aug_{K}$.

Scene Agnostic Clip-Shuffling. For fully leveraging the limited video data, we
propose to construct more pseudo scene cues. In this work, the data
augmentation is based on the basic unit of clip, i.e. $\rho$ continuous shots,
the generated clips are then randomly spliced disorderly for the training.

The process of Scene Agnostic Clip Generation and Shuffling is shown in Fig.
4.

Figure 4: The illustration of Scene Agnostic Clip-Shuffling. Clips are spliced
disorderly for training and each clip contains $\rho$ continuous shots.

####  3.1.3 Negative Sample Selections

The way to choose negative samples varies according to the specific SSL
frameworks. For SimCLR [12], the set of all non-positive samples within a
batch is used as the negative samples, and MoCo [14] leverages a negative
sample queue, which is a memory bank of previous samples output from the key
encoder. However, BYOL [17] and SimSiam [16] do not use negative samples and
instead resort to exploring more non-trivial solutions of SSL.

####  3.1.4 Objective Function

With Negative Samples. By defining $sim(\cdot,\cdot)$ as the cosine
similarity, the contrastive loss function, i.e., InfoNCE [11] is employed and
formulated as follows:

$\displaystyle{L_{con}}=-\log\frac{\sum_{k\in\left\\{k^{+}\right\\}}e^{(\operatorname{sim}(q,k)/\tau)}}{\sum_{k\in\left\\{k^{+},k^{-}\right\\}}e^{(\operatorname{sim}(q,k)/\tau)}}$
(7)

where ${k^{+}}$ and ${k^{-}}$ stand for the positive and negative samples for
the query $q$, and the $\tau$ is the temperature term [34].

Without Negative Samples. By maximizing the similarity between the query and
positive samples, the contrastive loss without negative samples is formulated
as follows:

$\displaystyle{L_{con}}=-2\sum_{k\in\left\\{k^{+}\right\\}}(\operatorname{sim}(\mathcal{P}_{\theta}(q),k_{SG})+\operatorname{sim}(\mathcal{P}_{\theta}(k),q_{SG}))$
(8)

where $\mathcal{P}_{\theta}$ is the predictor $\mathcal{P}$ with parameters
$\theta$ [17, 16], $k_{SG}$ and $q_{SG}$ are the samples with stop-gradient
(SG) [17, 16].

###  3.2 Video Scene Segmentation

After the unsupervised pretraining, two downstream models are used to evaluate
the quality of the extracted features with the frozen query encoder.

Problem Definition. For the Video Scene Segmentation, [6] [1] convert the task
into a binary classification task of shot semantics by modeling the temporal
relationship of adjacent shot features. In this way, we can determine whether
the end of a shot is the end of a scene story.

Figure 5: The illustration of boundary based model (a) and boundary free model
(b) for Video Scene Segmentation.

Boundary free model. While the previous downstream task of Video Scene
Segmentation is concluded to a shot boundary modeling based approach, as shown
in Fig. 5 (a), we introduce a vanilla boundary-free model. As shown in Fig. 5
(b), the proposed model covers the long-term dependence of shot
representations based on sequence-to-sequence learning. Compared with boundary
based model in Fig. 5 (a) that introduces inductive bias for the shot boundary
modeling with the sliding windows, the suggested model (b) takes the shot
features as the basic temporal input unit, enabling the model to explore both
local and global semantic relations.

##  4 Experiments

###  4.1 Experimental Setup

Dataset. MovieNet [31] consists of 1,100 movies with a large amount of multi-
modal data and annotations, and the total duration of all movies is about 3000
hours, it is the largest dataset for movie understanding analysis by far.
Besides, MovieNet [31] is split into a training set with 660 movies, a
validation set with 220 movies and a testing set with 220 movies. Currently,
for the task of Video Scene Segmentation, 190, 64 and 64 videos are labeled
with scene boundaries for the training, validation and test sets,
respectively. More importantly, movies in the MovieScene [6] are all included
in MovieNet[31].

It is worth noting that there are two versions of annotation about Video Scene
Segmentation task associated with MovieNet, one with only 150 annotations
(called MovieScenes in [6, 1], used in earlier methods [6] but it is no longer
available), and one with a total of 318 annotations (abbreviated as
MovieScenes-318 in this work). Since the small scale of of BBC [10] and OVSD
[32] datasets and unavailability of AdCuepoints [1] dataset, we instead adopt
MovieNet [31] dataset to evaluate the related approaches, more details are in
the Supplementary Materials.

Table 1: Results of supervised methods w/o SSL for
the task of Video Scene Segmentation on MovieNet.

Methods | Dataset | AP | F1
---|---|---|---
SCSA [9] | M.S. | 14.7 | -
Story Graph [35] | M.S. | 25.1 | -
Siamese [10] | M.S. | 28.1 | -
ImageNet [36] | M.S. | 41.26 | -
Places [37] | M.S. | 43.23 | -
LGSS [6] | M.S. | 47.1 | -
LGSS w/o DP [6] | M.S. | 44.9 | -
LGSS w/o DP [6] * | M.S-318 | 44.9 | 38.52

  * *

Our implementations based on official public codebase,2 while DP (Dynamic
Programming) isn’t public available.

Table 2: Results of methods w/ SSL for the task of Video Scene Segmentation on
MovieNet.

Methods | Pretrain Data | Eval. | Protocol | AP | F1
---|---|---|---|---|---
ShotCoL [1] | Train.+Test.+Val. | M.S | MLP [1] | 52.83 | -
ShotCoL [1] | Train.+Test.+Val. | M.S-318 | MLP [1] | 53.37 | -
ShotCoL [1] * | Train.+Test.+Val. | M.S-318 | MLP [1] | 52.89 | 49.17
SCRL (ours) | Train.+Test.+Val. | M.S-318 | MLP [1] | 54.82 | 51.43
ShotCoL [1] * | Train. only | M.S-318 | MLP [1] | 46.77 | 45.78
SCRL (ours) | Train. only | M.S-318 | MLP [1] | 53.74 | 50.40
ShotCoL [1] * | Train. only | M.S-318 | Bi-LSTM | 48.21 | 46.52
SCRL (ours) | Train. only | M.S-318 | Bi-LSTM | 54.55 | 51.39

  * *

Our implementations.

Representation Learning Stage. For visual modality, each shot consists of 3
keyframes and ResNet50 [38] is chosen as the default backbone to learn the
shot representations. The audio backbone used in [6] is applied for audio
modality, more details about the backbone encoders can be found in
Supplementary Materials.

For pretraining data, (i) training set (660 movies) in MovieNet [31] is used
to learn the shot representations, while we also conduct experiments with (ii)
all data (1,100 movies) [1] for a fair comparison. In particular, although
test data without the scene boundary labels are used for representation
learning in setting (ii), it is not recommended to use all the data for
pretraining because we usually have no prior access to test data in real
scenarios. Moreover, for the most of Self-Supervised benchmarks [11, 12, 13,
14, 15, 16, 17, 18], representation learning is performed only on the training
set, rather than all of the data.

Video Scene Segmentation Stage. For existing Self-Supervised methods on images
and videos, a simple downstream model is frequently used to evaluate the
representation quality of the frozen encoders. For instance, a linear fully-
connected layer is widely used for evaluation. However, for the Video Scene
Segmentation task, we cannot determine whether the ending position of a single
shot is the scene boundary or not. Consequently, a boundary-based non-temporal
model (MLP-based protocol, followed by [1]) and a boundary-free temporal model
(Bi-LSTM [19]-based protocol, proposed by us) are employed to evaluate the
capability of the encoder for local-to-global modeling.

Metrics. We use the mean of Average Precision (AP) [6] [1] specified to ground
truth scene boundaries of each movie, as well as F1-score for the evaluation.

Implementation Details. During the learning stage of Self-Supervised
representation, the batch size is set to 1,024 (shots), initial learning rate
is set to 0.03 and the training epoch is 100. The parameters of the visual and
audio encoders are randomly initialized. Besides, we perform naive K-Means
algorithm [39, 40] for online clustering and the cluster number $\\#class$ is
set to 24, while the clip length, i.e. $\rho$ of Scene Agnostic Clip Shuffling
is set to 16. MoCov2 [14] with the queue size of 65,536, momentum value of
0.999, temperature of 0.07 and cosine learning rate decay, are used as our SSL
framework setting. For the Video Scene Segmentation task, $num$-$of$-$shot$
[1] is set to 4 and 40 for the MLP [1] and Bi-LSTM protocols, respectively.
Each pretraining trial is conducted on the server with 8 NVIDIA V100 GPUs for
approximate 24 hours in visual modality and 10 hours in audio modality. The
dimensions of visual and audio features used for both pretraining and
evaluation are 2,048 and 512, respectively. More details, e.g., the choice of
hyperparameter, are presented in Supplementary Materials.

††footnotetext: 2https://github.com/AnyiRao/SceneSeg

###  4.2 Comparison with Existing Methods

Tables 4.1 and 4.1 present an overall performance of methods w/ or w/o SSL for
the Video Scene Segmentation task, where M.S. stands for MovieScenes dataset
with 150 annotated movies, and Eval. means the dataset used for supervised
evaluation stage after the pretraining. Besides, Train., Test., and Val.
represent training, testing and validation sets of MovieNet[31].

We have reproduced the performance of ShotCoL [1] on the entire dataset (1,100
movies) for comparison, although it is suggested to conduct the pretraining
stage only on the training set. Compared with ShotCoL [1] that has a decline
of 6.12 in terms of AP, our method can achieve competitive performance with
less training data, with only a decline of 1.08 in terms of AP. The proposed
method outperforms the supervised state-of-the-art method, i.e., LGSS [6] by
margins of 9.65 in terms of AP and 12.87 in terms of F1.

Table 3: Ablation results of Positive Sample
Selection.

Methods | Selection Strategy | AP
---|---|---
MoCo [14] | Self-Augmented | 42.51
- | Random ($n=1$) | 43.24
ShotCol [1] | NN ($m=8$) | 46.77
SC | Scene Consistency | 49.71

Table 4: Ablation results of SSL methods
w/ and w/o Scene Agnostic Clip-Shuffling.

Methods | w/o | w/ | AP
---|---|---|---
NN | ✓ | $\times$ | 46.77
NN | $\times$ | ✓ | 48.63
SC | ✓ | $\times$ | 49.71
SC | $\times$ | ✓ | 52.17

Table 5: Ablation results of Multiple Positive Samples (MPS).

Methods | Positive Sample(s) | AP
---|---|---
SC | Center | 52.17
MPS-SC | Self and Center | 51.20
Soft-SC | Eq. 6 ($\gamma$=0.5) | 53.74

###  4.3 Ablation Study

We perform all the ablation experiments using only the training data of
MovieNet in SSL stage, and evaluate the performance on downstream task based
on MLP protocol for fairness.

Positive Sample Selection. We first conduct ablation experiments on the four
different selection methods of positive pairs, i.e., Self-Augmented Selection,
Random Selection, Nearest Neighborhood (NN) Selection and Scene Consistency
(SC) Selection. Tab. 4.2 shows that Scene Consistency Selection method
achieves better performance than the other selection methods, which
outperforms the state-of-the-art algorithm [1] by a margin of 2.95 in terms of
AP. Meanwhile, the loss evolution curves of above methods are shown in Fig. 6.
We can find that Self-Augmented Selection reaches the lowest loss value, while
obtaining the worst performance on the task of Video Scene Segmentation. Due
to the trivial objective introduced by NN Selection that is discussed in
Section 3.1.2, it achieves the fastest convergence rate during the early
training, while stagnating to a mediocre performance. By contrast, SC
Selection has a relatively moderate convergence rate, and achieves the best
performance among all the selection strategies.

Figure 6: Loss evolution curves and AP results of the training with different
selection strategies.

Scene Agnostic Clip-Shuffling. The results of ablation study specific to the
Scene Agnostic Clip-Shuffling are presented in Tab. 4.2. Tab. 4.2 shows that
the proposed Clip-Shuffling achieves improvements of 1.85 and 2.46 in terms of
AP for NN and SC methods, respectively. These results verify the advantage of
the proposed positive sample selection discussed in Section 3.1.2 that SC is
free to the shot order in a video.

Multiple Positive Samples (MPS). Moreover, we study the performance of
multiple positive samples in Tab. 4.2. As shown in Tab. 4.2, Soft-SC achieves
the best performance of 53.74 in terms of AP. Although single positive sample
is employed in SC, it still achieves better performance than MPS-SC that
employ multiple positive samples.

Figure 7: The visualization results of shot retrieval. Overall, NN tends to
select adjacent shots, Self shows less relevance to the query and ImageNet
retrieves many kinds of boats. Compared with the other methods, the results of
SC are more consistent in the semantic information, i.e., there is a man
staying in the boat, and SC achieves a larger span (i.e., from $641$ to $850$)
than NN according to shot IDs. Meanwhile, SC shows better robustness against
the interference of pink smoke in the $994$-th shot as the results are more
pure.

###  4.4 Analysis of the Proposed Method

Generalizability to the large-scale supervised approach. To study the
generalizability of the proposed method, we equip our trained models with LGSS
[6], where LGSS is a large-scale supervised method and utilizes various
pretrained models with multi-modality. As is Shown in Tab. 6, our trained
model, trained only on the unlabeled training set, and based on the same
backbone, i.e. ResNet-50 [38], achieves an improvement of 4.0 in terms of AP
over the approach without our trained model.

Table 6: Generalizability to the large-scale supervised approach.

Methods | Modalities | AP
---|---|---
LGSS [6] w/o SSL |  | Visual(Place, ResNet50)
---
+Action+Actor+Audio
44.9
LGSS [6] w/ SSL |  | Visual(SSL, ResNet50)
---
+Action+Actor+Audio
48.9

Performance on different Self-Supervised Learning (SSL) frameworks. Four
popular SSL frameworks are used for evaluating our method, i.e., SimCLR [12],
MoCo [14], BYOL [17] and SimSiam [16]. Tab. 7 shows that the SSL framework
with momentum updates and negative samples achieves the best performance for
the Video Scene Segmentation task. Due to the momentum update mechanism, the
proposed method embedded in the framework of BYOL [17] achieves an improvement
of 10.53 over that in SimSiam [16], and a similar conclusion is reached in
[27].

Table 7: Results of the proposed method based on various Self-Supervised
Frameworks.

Methods |  | SSL Fra-
---
meworks
| w/ negative
---
samples
| w/ momen-
---
tum update
AP
SCRL | SimSiam [16] | $\times$ | $\times$ | 39.82
SCRL | SimCLR [12] | ✓ | $\times$ | 45.32
SCRL | BYOL [17] | $\times$ | ✓ | 50.35
ShotCoL [1] | MoCo [14] | ✓ | ✓ | 46.77
SCRL | MoCo [14] | ✓ | ✓ | 53.74

Boundary free model for evaluation. To study the performance of the introduced
boundary free model, the proposed method under MLP and Bi-LSTM protocols for
the scene segmentation task is evaluated in Tab. 8. Since Bi-LSTM protocol has
less inductive bias than sliding window based MLP protocol, it is able to
model representations of longer shot, hence achieves better performance on the
task of Video Scene Segmentation. More specifically, the performance of Bi-
LSTM protocol increases as the length of the shots increases, while the
performance of MLP protocol decreases instead. More details can be found in
Supplementary Materials.

Table 8: Results of Video Scene Segmentation using the proposed method under
MLP and Bi-LSTM protocols.

Protocols | Shot-Len | AP | F1 | #Param
---|---|---|---|---
MLP [1] | 4 | 53.74 | 50.40 | 37.75 M
MLP [1] | 10 | 49.61 $\downarrow$ | 44.04 $\downarrow$ | 88.09 M $\uparrow$
Bi-LSTM | 10 | 43.94 | 42.12 | 15.22 M
Bi-LSTM | 40 |  54.55 $\uparrow$ |  51.39 $\uparrow$ | 15.22 M

Visualization of Shot Retrieval. To get more intuition for the proposed
selection, we conduct retrieval experiments using four selection methods,
i.e., SC, NN, Self-Augmented and ImageNet selections, and present the results
in Fig. 7. More specifically, for a given shot, we calculate the similarities
between it and the other shots in the entire movie, then visualize the TOP-5
most similar shots in Fig. 7.

###  4.5 Limitations

Multi-modal Pretraining. In order to test the performance of the proposed
algorithm generalizing to multi-modal data, we also conduct experiments with
audio and visual modalities in the SSL stage, the joint multi-modal learning
scheme follows [41]. However, we did not achieve any improvement and were
confronted with the same concern that is mentioned in [1], as shown in Table.
9. Possible reasons are that (i) the publicly available audio data of each
shot are incomplete, (ii) the raw audio data are not available yet due to
copyright restrictions [31] and (iii) LGSS [6] utilizes various pretrained
models on the other datasets, while the methods in the comparison are trained
from scratch. Therefore, it is meaningful to shed light on how to pretrain
better multi-modal representations on the MovieNet [31].

Table 9: AP results of the multi-modal experiment on MovieNet. Backbones of
following methods for each modality are the same.

Methods | Visual | Audio | Visual+Audio
---|---|---|---
LGSS [6] | 39.0 | 17.5 | 43.4
ShotCoL [1] | 46.77 | 27.92 | 44.32
SCRL | 53.74 | 29.39 | 50.80

##  5 Conclusion

We present a Self-Supervised Learning (SSL) scheme based on Scene Consistency
to obtain better shot representations for the unlabeled long-term videos. The
proposed method achieves the state-of-the-art performance on the task of Video
Scene Segmentation under various protocols, and significant better
generalization performance when it is equipped with large-scale supervised
approach. Besides, we introduce a fair pretraining protocol and a more
comprehensive evaluation metric for the task of Video Scene Segmentation, to
make the assessment of the SSL more meaningful in practice.

Acknowledgment The work was supported by the National Natural Science
Foundation of China under grants no. 61602315, 91959108, the Science and
Technology Project of Guangdong Province under grant no. 2020A1515010707, the
Science and Technology Innovation Commission of Shenzhen under grant no.
JCYJ20190808165203670.

## References

  * [1] Shixing Chen, Xiaohan Nie, David Fan, Dongqing Zhang, Vimal Bhat, and Raffay Hamid.  Shot contrastive self-supervised learning for scene boundary detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9796–9805, 2021.
  * [2] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  Bsn: Boundary sensitive network for temporal action proposal generation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19, 2018.
  * [3] Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu.  Relaxed transformer decoders for direct action proposal generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13526–13535, 2021.
  * [4] Anthony Cioppa, Adrien Deliege, Silvio Giancola, Bernard Ghanem, Marc Van Droogenbroeck, Rikke Gade, and Thomas B Moeslund.  A context-aware loss function for action spotting in soccer videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13126–13136, 2020.
  * [5] Silvio Giancola, Mohieddine Amine, Tarek Dghaily, and Bernard Ghanem.  Soccernet: A scalable dataset for action spotting in soccer videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1711–1721, 2018.
  * [6] Anyi Rao, Linning Xu, Yu Xiong, Guodong Xu, Qingqiu Huang, Bolei Zhou, and Dahua Lin.  A local-to-global approach to multi-modal movie scene segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10146–10155, 2020.
  * [7] Panagiotis Sidiropoulos, Vasileios Mezaris, Ioannis Kompatsiaris, Hugo Meinedo, Miguel Bugalho, and Isabel Trancoso.  Temporal video segmentation to scenes using high-level audiovisual features.  IEEE Transactions on Circuits and Systems for Video Technology, 21(8):1163–1177, 2011.
  * [8] Jakub Lokoč, Gregor Kovalčik, Tomáš Souček, Jaroslav Moravec, and Přemysl Čech.  A framework for effective known-item search in video.  In Proceedings of the 27th ACM International Conference on Multimedia, pages 1777–1785, 2019.
  * [9] Vasileios T Chasanis, Aristidis C Likas, and Nikolaos P Galatsanos.  Scene detection in videos using shot clustering and sequence alignment.  IEEE transactions on multimedia, 11(1):89–100, 2008.
  * [10] Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara.  A deep siamese network for scene detection in broadcast videos.  In Proceedings of the 23rd ACM international conference on Multimedia, pages 1199–1202, 2015.
  * [11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [13] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton.  Big self-supervised models are strong semi-supervised learners.  Advances in Neural Information Processing Systems, 33:22243–22255, 2020.
  * [14] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [15] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649, 2021.
  * [16] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [17] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in Neural Information Processing Systems, 33:21271–21284, 2020.
  * [18] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [19] Zhiheng Huang, Wei Xu, and Kai Yu.  Bidirectional lstm-crf models for sequence tagging.  arXiv preprint arXiv:1508.01991, 2015.
  * [20] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In ICLR 2018, 2018.
  * [21] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [22] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer, 2016.
  * [23] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European conference on computer vision, pages 69–84. Springer, 2016.
  * [24] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui.  Spatiotemporal contrastive video representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964–6974, 2021.
  * [25] Ali Diba, Vivek Sharma, Reza Safdari, Dariush Lotfi, Saquib Sarfraz, Rainer Stiefelhagen, and Luc Van Gool.  Vi2clr: Video and image for visual contrastive learning of representation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1502–1512, 2021.
  * [26] Haofei Kuang, Yi Zhu, Zhi Zhang, Xinyu Li, Joseph Tighe, Soren Schwertfeger, Cyrill Stachniss, and Mu Li.  Video contrastive learning with global context.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3195–3204, 2021.
  * [27] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He.  A large-scale study on unsupervised spatiotemporal representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3299–3309, 2021.
  * [28] Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Ong, and Chen Change Loy.  Unsupervised object-level representation learning from scene images.  Advances in Neural Information Processing Systems, 34, 2021.
  * [29] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9879–9889, 2020.
  * [30] Adria Recasens, Pauline Luc, Jean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Pătrăucean, Florent Altché, Michal Valko, et al.  Broaden your views for self-supervised video learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1255–1265, 2021.
  * [31] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin.  Movienet: A holistic dataset for movie understanding.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 709–727. Springer, 2020.
  * [32] Daniel Rotman, Dror Porat, and Gal Ashour.  Optimal sequential grouping for robust video scene detection using multiple modalities.  International Journal of Semantic Computing, 11(02):193–208, 2017\.
  * [33] Jiarui Xu and Xiaolong Wang.  Rethinking self-supervised correspondence learning: A video frame-level similarity perspective.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10075–10085, 2021.
  * [34] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733–3742, 2018.
  * [35] Makarand Tapaswi, Martin Bauml, and Rainer Stiefelhagen.  Storygraphs: visualizing character interactions as a timeline.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 827–834, 2014.
  * [36] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [37] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
  * [38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [39] Stuart Lloyd.  Least squares quantization in pcm.  IEEE transactions on information theory, 28(2):129–137, 1982.
  * [40] Haozhe Liu, Haoqian Wu, Weicheng Xie, Feng Liu, and Linlin Shen.  Group-wise inhibition based feature regularization for robust classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 478–486, 2021.
  * [41] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovic, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman.  Self-supervised multimodal versatile networks.  NeurIPS, 2(6):7, 2020.


First we show that $i\in I_{DAT}$. The interpretations we consider are groundings, $G$, for which the network parameters define $G(f)$ and $G(c)$ for each function and constant symbol. Because each grounding is fully defined by its interpretation of function symbols, constant symbols, and predicates, These groundings are in one-to-one correspondence with the set of truth assignments to the ground atoms of the form $R(t_1,...,t_k)$ where $t_i\in O$. We choose a subset of atoms, $A'$, to consist of all grounded atoms, $R(t_1,...,t_k)$, for which $R(\theta_1,...,\theta_k)$ appears in $L$ and $t_i=\theta_i(\bar{x},\bar{c})$ for some variable assignment $\bar{x}$ and the networks constant assignment $\bar{c}$. For simplicity, we assume that this set contains no duplicates, in other words each atom has at most one representation of this form[ Because the transfer function and input weights agree for all neurons representing the same predicate, in the case that there are multiple representations of an atom in a network, both neurons will assign the same truth value to the atom and we can make an arbitrary choice for the functions $h$ and $r$ defining a map in $I_{DAT}$].
We explicitly define the maps, $g,h,r$, required in definition $\ref{def:IDAT}$ to prove that $i\in I_{DAT}$. The map $h$ specifies, for an atom in $A'$, which variable assignment will represent it, in particular, $h$ maps an atom $R(\theta_1(\bar{x},\bar{c}),...,\theta_k(\bar{x},\bar{c}))$ to $(\bar{x})$. $r$ maps $R(\theta_1(\bar{x},\bar{c}),...,\theta_k(\bar{x},\bar{c}))$ to the label of neuron $x_{G\theta_1,...,\theta_k}$ ($h$ and $r$ are well defined by the previous assumption), and $g$ is the identity function on $\mathbb{R}$. Surjectivity of $h$ and $r$ is immediate from the definition, furthermore if $q,q'\in A'$ with $q\neq q'$ then $R(\theta_1(\hat{x}=t),...,\theta_k(\hat{x}=t))\neq R'(\theta'_1(\hat{x}=t'),...,\theta'_k(\hat{x}=t'))$. If $t\neq t'$ then $h(q)\neq h(q')$; otherwise, if $R \neq R'$ or $\theta_i'\neq \theta_i$ then $r(q)\neq r(q')$. This proves the second condition. The final condition is satisfied by definition of $i$, in particular, if a grounding is in $i(x)$ then the value of each atom in $A'$ in the grounding is $G(R(\theta_1(\hat{x}=\bar{x},\bar{c}),...,\theta_k(\hat{x}=\bar{x},\bar{c})))=x_{R\theta_1,...,\theta_k}=x_{R\theta_1,...,\theta_k}=g(x_{r(R(\theta_1(\hat{x}=\bar{x},\bar{c}),...,\theta_k(\hat{x}=\bar{x},\bar{c})))})$.

With our encoding in $I_{DAT}$, we now must look at $X_{inf}$. Define the computation time $t_c$ to be the least integer such that all neurons in the network are updated. Because the LTN is feedforward, it will converge to a fixed point in which $\bar{x}$ is equal to its initial state, $t$, each hidden unit representing a term $\theta$, will be updated to $\theta(\bar{x}=t,\bar{c})$ and $x_{R\theta_1,...,\theta_k}$ will be equal to the truth value obtained by computing $R(\theta_1(\bar{x}=t,\bar{c}),...,\theta_k(\bar{x}=t,\bar{c}))$ using the network parameters for $R$. With $Agg=\cap$, $\mathcal{M}_N$ represents the partial grounding in which the truth value of each atom $R(\theta_1(\bar{x}=t,\bar{c}),...,\theta_k(\bar{x}=t,\bar{c})) \in A'$ is equal to the value of $x_{R\theta_1,...,\theta_k}$ in the fixed state with $\bar{x}=t$.

This shows that an LTN defines a partial grounding, $\hat{G}$, on a knowledge base, $L$. The loss function used to train the LTN is given in [Badreddine et al., 2020] as $\SatAgg\limits_{\phi \in L} \hat{G}(\phi)$ where $\hat{G}(\phi)$ is the fuzzy truth value assigned to $\phi$ by $\hat{G}$ using the chosen fuzzy connectives. This is equivalent to $\SatAgg\limits_{\phi \in L} 1-1- \hat{G}(\phi)=\SatAgg\limits_{\phi \in L} 1-d(\hat{G}(\phi),[1,1])$. Recalling that $\hat{G}$ represents a set of interpretations, $\{G\}$, each of which has the same value of $G(\phi)$, this becomes $\inf\limits_{M \in \mathcal{M}_N}\SatAgg\limits_{\phi \in L} 1-1- \hat{G}(\phi)=\SatAgg\limits_{\phi \in L} 1-d(\hat{G}(\phi),[1,1])$, which is just $Fid_{fuzzy}((N,i,\cap),L)$. Thus LTNs are trained to maximize $Fid_{fuzzy}$.

LTNs are a representative of a family of similar fuzzy techniques, they differ in their choice of fuzzy connectives and the restrictions on the type of sentences in the language [Diligenti et al., 2017][van Krieken et al., 2020]. In our framework, all of these techniques can be succinctly summarized as encodings under $I_{DAT}$ which are trained to maximize $Fid_{fuzzy}$, this opens the door for a general theory regarding these techniques without committing to  a single method. We can go further than this, however, as there exist similar semantic regularization methods that do not minimize $Fid_{fuzzy}$ as we show next, our framework extends to these as well.

[Xu et al., 2018] gives a generic semantic loss function for deep neural networks whose output labels represent propositional variables. Each labeling predicted by the neural network represents an assignment of truth values to the propositional variables. Each neuron in the output layer is assumed to represent the probability that the variable corresponding to that neuron is true. Then the semantic loss is defined by $Loss(L,p)=-\log \sum\limits_{M:M(L)=\top} \prod\limits_{j: M(X_j)=\top} p_j \prod\limits_{j: M(X_j)=\bot} (1-p_j)$. Where $L$ is a propositional knowledge base, $p$ is the vector of probabilities that each variable represented by the output layer is assigned True, and $M$ is an assignment of the $n$ propositional variables represented by the output neurons. We show that minimizing this loss is equivalent to maximizing $Fid_{prob}$.

To see this, first we describe the networks under consideration and their encoding functions. Assume that we have a deep feed-forward neural network with $k$ layers whose first $k-1$ layers are deterministic and whose final layer consists of $n$ binary-valued neurons whose state is determined by a vector of probabilities $p=[p_1,...,p_n]$ computed from the state of the previous layer. Assume, like in most of our feed-forward examples, that the input layer is fixed for a given initial state, $x_0$, denote by $p(x_0)$ the vector of probabilities of the output layer after $k-1$ time steps. Let the output neurons be visible and the rest hidden, define $i\in I_{NAT}$ by $i(x)=\{M|M(Q_j)=\top$ if $x_j=1$ and $M(Q_j)=\bot$ if $x_j=0\}$ where $Q_j$ is a propositional variable corresponding to the $j^{th}$ output neuron. $Loss(L,p(x_0))$ is simply the negative log probability that $i(X^{(k)})$ is a model of $L$. Minimizing this for all input states $x_0$ is thus equivalent to maximizing $Fid_{prob}((N,i,\cup),L)$ where the initial distribution is uniform over the input space.

While the previous method looks at encodings in $I_{NAT}$, [Marra et al., 2020] combines the distributed encodings of LTNs with probabilistic optimization to create a method that maximizes the expected satisfiability of knowledge base, this is an encoding in $I_{DAT}$ that maximizes $Fid_{prob}$. This method, as with all of the methods discussed in this section, weights the information coming from the observations in the dataset with a measure of the distance of a neural network from being a model of a knowledge base $L$. By formalizing this in the language of semantic encodings, a general theory can be developed to guide the training of neural networks in the presence of approximate background knowledge. We will fully explore the implications of this framework in the conclusion.

§.§ Other Encodings

The survey we present here is far from exhaustive, recent interest in neural-symbolic  computing has led to a proliferation of techniques that makes it impossible to cover them all in full detail here. Crucially, not all techniques can be described as semantic encodings. In particular, many methods map states of a neural network directly onto sentences of a logical system rather than interpretations and models (see [Smolensky et al., 2016] and [Rocktäschel and Riedel, 2017] for examples).

Our framework is meant to cover those methods in which states of a neural network are mapped onto interpretations of a logical system. We believe that the proposed definitions capture the vast majority of these techniques. Although formal identification with other specific techniques is left to future work, in most cases the relationship is obvious. For example, Proposition 4.3 in [Richardson and Domingos, 2006] shows that, as the weights of an MLN go to infinity, the probability that a state of the MLN is a model of the desired knowledge base converges to $1$. This is clearly an approximate encoding under with $Fid_{prob}$ according to our definitions. Another recent approach that has become common is to do inductive logic programming with a differentiable structure [Sen et al., 2022][Evans and Grefenstette, 2018]. In these approaches, a vector space is used to represent truth assignments to ground atoms and the weights between atoms are learned from examples. The resulting structure then implements a logic program. Other examples modify a network to have hard constraints on the output layer [Giunchiglia and Lukasiewicz, 2021]. These methods can be seen as similar to semantic regularization but with fidelity equal to $1$.

Although not all methods fall under the category of semantic encodings, enough of them do to allow a general theory of neural-symbolic AI to begin to be developed. We conclude with a discussion on future work in this area.

§ CONCLUSION AND FUTURE WORK

For a long time many have recognized that rule-based symbolic systems and neural networks have complementary strengths and weaknesses. This has led to the development of a subfield of AI looking to address the deficiencies of current AI systems through neural-symbolic integration. The object of neural-symbolic integration is to find ways to take advantage of the strengths of both neural networks and symbolic systems in order to develop a learning and reasoning system which is robust, flexible, generalizable, and more comprehensible than neural networks are on their own.

What has been lacking in neural-symbolic AI, is a robust theory that can be leveraged for the design of new neural-symbolic systems. In this paper, we have made some first steps towards such a theory by providing a formal framework for an important class of neural-symbolic techniques that we call semantic encodings. This is far from the final word on a theory of neural-symbolic AI and should be looked at more as a starting point for future work in this area. We hope that this paper can serve both as a formal introduction to neural-symbolic methods to those who are new to the field as well as an organizational framework that can help place the many different approaches to neural-symbolic computing within a broader context. We conclude by outlining various directions for future work.

The most obvious direction for future work is to continue to expand on the definitions provided so that they can be used to describe more neural-symbolic techniques. Within the context of semantic encoding, task-specific definitions of fidelity, descriptions for more useful sets of candidate maps, and definitions to cover probabilistic networks in general and approaches such as [Manhaeve et al., 2018] are desirable. We have also not discussed a number of techniques which are not semantic in nature. These might be adequately described as syntactic techniques and proper definitions for these should also be a priority of future work. Furthermore, with the definitions in place, a general theory can start to be developed. Some properties of semantic encodings which we have not discussed, but which are simple to prove, include how mappings between neural networks or logical systems that preserve relevant structure should also preserve semantic encodings. For example, if $N_1$ is a semantic encoding of $L$ and there is a mapping from the state space of $N_2$ to $N_1$ that preserves the dynamics of $N_2$ then $N_2$ should likewise be able to be encoded in $L$. Developing theory of this kind has the potential to elucidate an explicit relationship between network architecture, the set of candidate maps, and the ability to encode knowledge in neural networks.

(0,0) rectangle (4,4);

[shape=rectangle,draw=none, anchor=center,minimum width=2.1cm] (x_1) at (-1.5,3.5)Constraints ;

[shape=rectangle,draw=none, anchor=center, minimum width=2cm] (x_2) at (-1.5,0.5)knowledge base;

[shape=rectangle,draw=none, anchor=center, minimum width=2cm] (y_1) at (4.5,3.5)Dataset;

[shape=rectangle,draw=none, anchor=center, minimum width=2cm] (y_2) at (4.5,0.5)Neural Network;
decoration=text along path,
text=Represented by,
text align=left indent=0.1

(x_1) – (x_2);

decoration=text along path,

text align=left indent=0.25

(y_1) – (y_2);

[->] (x_1) – (y_1) node[ below,yshift = 0.5cm,xshift=-3cm]Satisfied by;
[<->] (x_2) –  (y_2)  node[ below,yshift = -0.2cm,xshift=-3cm]Encodings;

[->] (y_1) – (y_2);
[->] (x_1) – (x_2);

Semantic encoding as a hypothetical relationship between a trained neural network on a data set and a logic representing constraints on the data.

Perhaps the most practical potential application of a theory of this kind involves relating the properties of a dataset to encodings of neural networks. What we mean by this is, suppose that a dataset satisfies some constraints that can be encoded in a knowledge base $L$, and suppose that we want to train a neural network, $N$, on this dataset. Given encoding and aggregation functions, a neural network that is a semantic encoding of $L$ with the given encoding and aggregation function should perform better than a neural network that is not (Figure <ref> illustrates this relationship visually). Theory in this direction could be developed for both exact and approximate cases. Results of this type have the potential to overcome many of the limitations of a purely data-driven approach to AI, but as the practical work on this progresses and expands, a robust theory of neural-symbolic computing is needed. Within this paper we have aimed to provide the first steps in this direction.

§ DECLARATIONS

Funding: This research was not funded

Conflicts of interest: None

Ethics approval: not applicable

Consent to participate: not applicable

Consent for publication: not applicable

Availability of data and material: not applicable

Code Availability: not applicable

Authors' Contributions: The primary author is responsible for the definitions and most of the technical examples, proofs and writing, the secondary authors contributions include being responsible for the writing of substation portions of text, reviewing and editing the work of the primary author, and using their expertise to revise and correct examples and proofs used throughout.

[Bader et al., 2006]
Sebastian Bader, Pascal Hitzler, and Steffen Hölldobler.
The core method: Connectionist model generation.
In In Proceedings of the 16th International Conference on
Artificial Neural Networks (ICANN, pages 1–13. Springer, 2006.

[Bader et al., 2007]
Sebastian Bader, Pascal Hitzler, Steffen Hölldobler, and Andreas Witzel.
A fully connectionist model generator for covered first-order logic
In Proceedings of the 20th International Joint Conference on
Artifical Intelligence, IJCAI'07, page 666–671, San Francisco, CA, USA,
2007. Morgan Kaufmann Publishers Inc.

[Badreddine et al., 2020]
Samy Badreddine, Artur S. d'Avila Garcez, Luciano Serafini, and Michael
Logic tensor networks.
CoRR, abs/2012.13635, 2020.
URL <https://arxiv.org/abs/2012.13635>.

[Barceló et al., 2020]
Pablo Barceló, Egor V. Kostylev, Mikaël Monet, Jorge Pérez,
Juan L. Reutter, and Juan Pablo Silva.
The logical expressiveness of graph neural networks.
In 8th International Conference on Learning Representations,
ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL <https://openreview.net/forum?id=r1lZ7AEKvB>.

[Bengio, 2019]
Yoshua Bengio.
From system 1 deep learning to system 2 deep learning, 2019.
posner lecture.

[d'Avila Garcez and Gabbay, 2004]
Artur d'Avila Garcez and Dov Gabbay.
Aaai'04: Proceedings of the 19th national conference on artifical
AAAI Press, 2004.
ISBN 0262511835.

[d'Avila Garcez and Lamb, 2020]
Artur d'Avila Garcez and Luís C. Lamb.
Neurosymbolic AI: the 3rd wave.
CoRR, abs/2012.05876, 2020.
URL <https://arxiv.org/abs/2012.05876>.

[d'Avila Garcez and Zaverucha, 1999]
Artur d'Avila Garcez and Gerson Zaverucha.
The connectionist inductive learning and logic programming system.
Appl. Intell., 11:0 59–77, 1999.

[Diligenti et al., 2017]
Michelangelo Diligenti, Marco Gori, and Claudio Saccà.
Semantic-based regularization for learning and inference.
Artif. Intell., 244:0 143–165, 2017.
URL <https://doi.org/10.1016/j.artint.2015.08.011>.

[Evans and Grefenstette, 2018]
Richard Evans and Edward Grefenstette.
Learning explanatory rules from noisy data.
J. Artif. Intell. Res., 61:0 1–64, 2018.
URL <https://doi.org/10.1613/jair.5714>.

[Fitting, 2002]
Melvin Fitting.
Fixpoint semantics for logic programming a survey.
Theoretical Computer Science, 278:0 25–51, 2002.

[Gabbay, 1996]
Dov M. Gabbay.
Labelled Deductive Systems.
Oxford University Press, New York, oxford logic guides 33 edition,

[Garcez et al., 2008]
Artur d'Avila Garcez, Lus C. Lamb, and Dov M. Gabbay.
Neural-Symbolic Cognitive Reasoning.
Springer, 1st edition, 2008.
ISBN 3540732454, 9783540732457.

[Garcez et al., 2002]
Artur d’Avila Garcez, Dov M. Gabbay, and Krysia B. Broda.
Neural-Symbolic Learning System: Foundations and Applications.
Springer-Verlag, Berlin, Heidelberg, 2002.
ISBN 1852335122.

[Garnelo et al., 2016]
Marta Garnelo, Kai Arulkumaran, and Murray Shanahan.
Towards deep symbolic reinforcement learning, 2016.

[Giunchiglia and Lukasiewicz, 2021]
Eleonora Giunchiglia and Thomas Lukasiewicz.
Multi-label classification neural networks with hard logical
J. Artif. Intell. Res., 72:0 759–818, 2021.
URL <https://doi.org/10.1613/jair.1.12850>.

[Goodfellow et al., 2015]
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining and harnessing adversarial examples.
In Proceedings of the 3rd International Conference on Learning
Representations, 2015.

[Guha, 2014]
Ramanathan V. Guha.
Towards a model theory for distributed representations.
CoRR, abs/1410.5859, 2014.
URL <http://arxiv.org/abs/1410.5859>.

[Hodgkin and Huxley, 1952]
A. L. Hodgkin and A. F. Huxley.
A quantitative description of membrane current and its application to
conduction and excitation in nerve.
The Journal of Physiology, 1170 (4):0
500–544, 1952.

[Hopfield, 1982]
J. J. Hopfield.
Neural networks and physical systems with emergent collective
computational abilities.
Proceedings of the National Academy of Sciences of the United
States of America, 790 (8):0 2554–2558, 1982.
ISSN 0027-8424.
URL <http://view.ncbi.nlm.nih.gov/pubmed/6953413]>.

[Hornik et al., 1989]
Kurt Hornik, Maxwell Stinchcombe, and Halbert White.
Multilayer feedforward networks are universal approximators.
Neural Networks, 20 (5):0 359 – 366, 1989.
ISSN 0893-6080.

[Hölldobler and Kalinke, 1994]
Steffen Hölldobler and Yvonne Kalinke.
Towards a new massively parallel computational model for logic
07 1994.

[Hölldobler et al., 1999]
Steffen Hölldobler, Yvonne Kalinke, and Hans-Peter Störr.
Approximating the semantics of logic programs by recurrent neural
Appl. Intell., 11:0 45–58, 07 1999.

[Komendantskaya, 2009]
Ekaterina Komendantskaya.
Neurons or symbols - why does OR remain exclusive?
In António Dourado Correia, Agostinho C. Rosa, and Kurosh
Madani, editors, IJCCI 2009 - Proceedings of the International Joint
Conference on Computational Intelligence, Funchal, Madeira, Portugal, October
5-7, 2009, pages 502–507. INSTICC Press, 2009.

[Komendantskaya et al., 2007]
Ekaterina Komendantskaya, Máire Lane, and Anthony Seda.
Connectionist Representation of Multi-Valued Logic Programs,
volume 77, pages 283–313.
08 2007.

[Manhaeve et al., 2018]
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and
Luc De Raedt.
Deepproblog: Neural probabilistic logic programming.
In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman,
Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in
Neural Information Processing Systems 31: Annual Conference on Neural
Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018,
Montréal, Canada, pages 3753–3763, 2018.

[Marra et al., 2020]
Giuseppe Marra, Michelangelo Diligenti, Francesco Giannini, Marco Gori, and
Marco Maggini.
Relational neural machines.
In European Conference on Artificial Intelligence, 2020.

[McCarthy, 1988]
John McCarthy.
Epistemological challenges for connectionism.
Behavioural and Brain Sciences, 110 (1):0 44,

[McCulloch and Pitts, 1943]
Warren S. McCulloch and Walter Pitts.
A logical calculus of the ideas immanent in nervous activity.
The bulletin of mathematical biophysics, 50
(4):0 115–133, 1943.
ISSN 1522-9602.
URL <https://doi.org/10.1007/BF02478259>.

[Merrill and Sabharwal, 2022]
William Merrill and Ashish Sabharwal.
Transformers can be translated to first-order logic with majority
quantifiers, 2022.
URL <https://arxiv.org/abs/2210.02671>.

[Pinkas, 1995]
Gadi Pinkas.
Reasoning, nonmonotonicity and learning in connectionist networks
that capture propositional knowledge.
Artificial Intelligence, 770 (2):0 203 – 247,
ISSN 0004-3702.

[Podnieks and Detlovs, 2017]
Karlis Podnieks and Vilnis Detlovs.
Introduction to Mathematical Logic.
05 2017.

[Putnam, 1987]
Hilary Putnam.
Representation and Reality.
MIT Press, 1987.

[Richardson and Domingos, 2006]
Matthew Richardson and Pedro Domingos.
Markov logic networks.
Machine Learning, 620 (1):0 107–136, 2006.
ISSN 1573-0565.
URL <https://doi.org/10.1007/s10994-006-5833-1>.

[Rocktäschel and Riedel, 2017]
Tim Rocktäschel and Sebastian Riedel.
End-to-end differentiable proving.
In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach,
Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances
in Neural Information Processing Systems 30: Annual Conference on Neural
Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA,
USA, pages 3788–3800, 2017.

[Sen et al., 2022]
Prithviraj Sen, Breno W. S. R. de Carvalho, Ryan Riegel, and Alexander G. Gray.
Neuro-symbolic inductive logic programming with logical neural
In Thirty-Sixth AAAI Conference on Artificial Intelligence,
AAAI 2022, Thirty-Fourth Conference on Innovative Applications of
Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational
Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 -
March 1, 2022, pages 8212–8219. AAAI Press, 2022.
URL <https://ojs.aaai.org/index.php/AAAI/article/view/20795>.

[Serafini and d'Avila Garcez, 2016]
Luciano Serafini and Artur d'Avila Garcez.
Logic tensor networks: Deep learning and logical reasoning from data
and knowledge.
CoRR, abs/1606.04422, 2016.
URL <http://arxiv.org/abs/1606.04422>.

[Smolensky, 1988]
Paul Smolensky.
The proper treatment of connectionism.
Behavioral and Brain Sciences, 11, 1988.

[Smolensky, 1990]
Paul Smolensky.
Tensor product variable binding and the representation of symbolic
structures in connectionist systems.
Artificial Intelligence, 460 (1):0 159 – 216,
ISSN 0004-3702.

[Smolensky et al., 2016]
Paul Smolensky, Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, and
Li Deng.
Basic reasoning with tensor product representations.
CoRR, abs/1601.02745, 2016.
URL <http://arxiv.org/abs/1601.02745>.

[Su et al., 2019]
Jiawei Su, Danilo V. Vargas, and Sakurai Kouichi.
One pixel attack for fooling deep neural networks.
IEEE Transactions on Evolutionary Computation, 230
(5):0 828–841, 2019.
ISSN 1941-0026.

[Szegedy et al., 2014]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan,
Ian J. Goodfellow, and Rob Fergus.
Intriguing properties of neural networks.
In 2nd International Conference on Learning Representations,
ICLR 2014,, 2014.
URL <http://arxiv.org/abs/1312.6199>.

[Towell and Shavlik, 1994]
Geoffrey G. Towell and Jude W. Shavlik.
Knowledge-based artificial neural networks.
Artificial Intelligence, 70:0 119–165, 1994.

[van Krieken et al., 2020]
Emile van Krieken, Erman Acara, and Frank van Harmelen.
Analyzing differentiable fuzzy logic operators.
Master's thesis, 2020.
URL <https://arxiv.org/pdf/2002.06100.pdf>.

[Xu et al., 2018]
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck.
A semantic loss function for deep learning with symbolic knowledge.
In Jennifer G. Dy and Andreas Krause, editors, Proceedings of
the 35th International Conference on Machine Learning, ICML 2018,
Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of
Proceedings of Machine Learning Research, pages 5498–5507. PMLR,
URL <http://proceedings.mlr.press/v80/xu18h.html>.

[Yi et al., 2018]
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and
Joshua B. Tenenbaum.
Neural-symbolic vqa: Disentangling reasoning from vision and language
understanding, 2018.
URL <https://arxiv.org/abs/1810.02338>.

§ CONTRIBUTION SHEET

In this paper, we seek to provide a framework for a family of neural-symbolic techniques we call semantic encodings. Nuerosymbolic techniques are those which aim to combine neural networks and symbolic systems in such a way that the resulting system will have the strengths of neural networks and symbolic systems with neither of these weaknesses. One common approach to this is to encode a knowledge base of a logical system into a neural network. The idea being that the neural network will benefit from the background knowledge encoded in the knowledge base while still being flexible enough to learn from new data and handle noisy and incomplete data.

Many different techniques to this end have been proposed, but unfortunately, they all rely on different, generally ad-hoc definitions of what it means for a knowledge base to be encoded into a neural network. This makes the development of a general theory of neural encodings, which could be used to guide the development of new encodings, impossible. In this paper we use the fact that most encodings in the literature share some common traits to define a general framework for a family of techniques we call semantic encodings that generalizes the definitions used by techniques such as penalty logic, knowledge based artificial neural networks, logic tensor networks, and others.

The importance of such a framework is twofold. One, it helps to systematize and organize existing results allowing for easy comparison and for future encodings to build directly off of previous work done without having to redefine the basic properties of a semantic encoding for a particular method. Two, it has the potential to develop a general theory of semantic encodings which could be used to prove results relating the structure of a neural network and the details of a learning algorithm to the ability to encode logical information or learn datasets satisfying constraints described by a knowledge base.

We achieve this goal by developing formal definitions for semantic encodings and motivating them through discussing and the the use of simple examples. We justify our choice of definition by surveying existing semantic encodings and showing how in each case, despite appearing quite different on the surface, they can all be shown to be particular cases of our more general framework. Although not all encoding techniques are covered by our framework, in particular encodings which do not encode semantic information, a broad enough selection of encodings do fall into the category of semantic encodings that we believe formalizing the definitions as we have done is valuable.

To our knowledge, little work on a formal theory of neural-symbolic computing, let alone semantic encodings has been done. The papers that are most closely related to our papers are those of the encodings which we seek to describe, in particular, KBANN [Towell and Shavlik, 1994], CILP [d'Avila Garcez and Zaverucha, 1999], Penalty Logic [Pinkas, 1995], and Logic Tensor Networks [Serafini and d'Avila Garcez, 2016]. Additionally, several position papers have been used to argue for the need of a general theory of neural-symbolic computing, which we claim to have taken the first steps towards with this paper <cit.> [Komendantskaya, 2009]