# Breaking Shortcut: Exploring Fully Convolutional
Cycle-Consistency for Video Correspondence Learning

Yansong Tang1, indicates equal contribution, The work is done when Yansong
Tang, Zhenyu Jiang and Zhenda Xie are interns at Microsoft Research Asia.
Zhenyu Jiang2,∗ Zhenda Xie3,4,∗ Yue Cao4 Zheng Zhang4 Philip H. S. Torr1 Han
Hu4
1 University of Oxford 2 University of Texas 3 Tsinghua University 4 Microsoft
Research Asia
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Previous cycle-consistency correspondence learning methods usually leverage
image patches for training. In this paper, we present a fully convolutional
method, which is simpler and more coherent to the inference process. While
directly applying fully convolutional training results in model collapse, we
study the underline reason behind this collapse phenomenon, indicating that
the absolute positions of pixels provide a shortcut to easily accomplish
cycle-consistence, which hinders the learning of meaningful visual
representations. To break this absolute position shortcut, we propose to apply
different crops for forward and backward frames, and adopt feature warping to
establish correspondence between two crops of a same frame. The former
technique enforces the corresponding pixels at forward and back tracks to have
different absolute positions, and the latter effectively blocks the shortcuts
going between forward and back tracks. In three label propagation benchmarks
for pose tracking, face landmark tracking and video object segmentation, our
method largely improves the results of vanilla fully convolutional cycle-
consistency method, achieving very competitive performance compared with the
self-supervised state-of-the-art approaches. Our trained model and code are
available at https://github.com/Steve-Tod/STFC3.

Figure 1:  Comparison of vanilla fully convolutional cycle-consistency (FC3)
method and our proposed spatial transformation fully convolutional cycle-
consistency (STFC3) approach. During training, the vanilla FC3 converges very
fast, as illustrated in the training loss curve (a). However, when evaluated
on the pose tracking and video object segmentation tasks, this model shows
inferior results (b). The visualization of its feature map in (c) further
indicates that the model collapses into a shortcut solution. In this paper, we
study the underline reason behind this phenomenon and introduce a new STFC3
method to address this issue, which achieves significant improvements on the
vanilla FC3.  Figure 2:  The state-of-the-art self-supervised video
correspondence learning method CRW [17] adopts a patch-based method (a-left)
to perform cycle-consistency learning (a-right), but extracts feature of the
whole image during testing. To bridge the gap between the training and
testing, a more simple and effective way is to use a fully convolutional
method during training (b). However, directly doing this causes the model to
collapse into a shortcut solution as the CNN encodes absolute position
information of each pixel. To address this issue, as shown in (c), we explore
a spatial transformation fully convolutional cycle-consistency (STFC3) method,
which achieves significant improvements over three widely-used label
propagation benchmarks.

##  1 Introduction

Visual correspondence learning is a fundamental problem in computer vision. In
the past few years, a variety of approaches [40, 23, 22, 28] have been
proposed to tackle this topic in a self-supervised learning paradigm, by
leveraging the inherent continuity between adjacent frames from unlabelled
videos. Among these works, a typical strategy is to take the cycle-consistency
in time as free supervisory [42, 26, 41], and train an encoder tracking
forward in time and backward to the initial position to form a cycle. Along
this line, Allan et al. [17] recently propose a method called “contrastive
random walk” (CRW), which achieves state-of-the-art results on various label
propagation tasks. Specifically, CRW uses an encoder to learn representation
of different patches sampled from each frame (see Figure 2(a)). After training
based on the cycle-consistency supervision, the optimized encoder is used to
extract features of the whole image for the downstream tasks. Despite its
elegant framework and promising performance, the inconsistency between its
training and testing process motivates us to consider a natural question: can
we apply a fully convolutional method rather than the patch-based one during
the training process?

Towards this direction, we attempt to directly execute cycle-consistency
training based on a sequence of single feature maps extracted by a fully
convolutional model (Figure 2(b)). As shown in Figure 1(a), during training,
this vanilla fully convolutional cycle-consistency (FC3) converges very fast.
But when evaluated on the pose tracking and video object segmentation tasks,
this model shows inferior results (see Figure 1(b)). Based on this
observation, we further visualize its feature map in Figure 1(c), which
indicates that the model collapses into a shortcut solution during
training111This phenomenon was also observed in the original CRW paper [17].
To better understand this collapse phenomenon, we make a throughout analysis
and reveal that the underline reason behind is that the fully convolutional
network encodes absolute position information of pixels [16, 17], which
provides a shortcut for the correspondence task and hinders the learning of
meaningful visual representations.

To tackle this, we propose a spatial transformation fully convolutional cycle-
consistency (STFC3) approach, which applies different cropping operations when
walking through forward and back tracks, and adopts feature warping other than
probabilistic walking [17] to connect two crops of the same frame. The former
technique lets the corresponding pixels at forward and back tracks have
different absolute positions, and the latter effectively blocks the shortcuts
going between forward and back tracks. Our STFC3 effectively addresses the
collapse issue. On three label propagation benchmarks, it largely improves the
performance of vanilla FC3 from 32.4$\%$ to 62.0$\%$<EMAIL_ADDRESS>on
J-HMDB [19] for pose tracking, from 56.7$\%$ to 18.8$\%$ (RMSE$\downarrow$)
for on 300VW [36] for face landmark tracking, and from 18.0$\%$ to 60.5$\%$
($\mathcal{J}$&$\mathcal{F}_{\textrm{m}}$$\uparrow$) on DAVIS-2017 [33] for
video object segmentation. Compared with the self-supervised state-of-the-art
methods, our STFC3 achieves very competitive results on the unlabelled
Kinetics dataset [4].

##  2 Related Works

Self-supervised Image Representation Learning: Self-supervised visual
representation learning has attracted growing attention in recent years.
Currently, mainstream works [11, 5, 8, 1, 2] usually adopt an instance
discrimination pretext task at image level, which regards each image as a
single class and optimize the network to discriminate each other. More
recently, Xie et al. [43] propose to perform unsupervised representation
learning at pixel level, which effectively transfers performance to downstream
tasks such as semantic segmentation and object detection. While it is a
natural way to borrow the merit of recent progress [43] from image domain to
video domain, the trivial solution caused by the position encoding becomes a
main obstacle for correspondence learning. In this paper, we make a throughout
analysis on this point and propose to break the shortcut with a spatial
transformation strategy.

Self-supervised Video Representation Learning: In the past few years, there
are growing numbers of works dedicated to self-supervision video
representation learning for various downstream tasks, such as action
recognition [10, 9, 34], video retrieval [30], video caption [38, 48] and many
others. In this paper, we focus on the downstream task of label propagation.
According to the type of self-supervision used for training, we divide
existing approaches into colorization-based, cycle-consistency-based and
frame-level similarity learning methods. For the first category, Vondrick et
al. [40] and Lai et al. [24] utilize the pretext task to reconstruct a target
frame from a reference frame. More recently, Lai et al. [22] introduce a
memory component that allows the network to capture more contextual
information of different frames to enhance the existing methods. For the
second category, Wang et al. [42] and [41] proposed to use cycle-consistency
in time as free supervision, to train an encoder tracking forward in time and
backward to form a cycle. More recently, Allan et al. [17] propose a
probabilistic framework that largely improves the performance on various
tasks. We also follow this line but explore a new fully convolutional method
for cycle-consistency learning. Concurrently, Xu et al. [44] and Mathilde et
al. [3] show that using frame-level similarity could also learn the
correspondence in an implicit way, which are complementary methods with ours.

Position Information Encoded by Neural Network: Recent studies [16, 20] have
shown that convolutional neural networks can encode the position information
of the input image, mainly caused by the padding operation. There also have
been a series of works explicitly leverage the position information for
devising different architectures (e.g., Transformer network [39], local
relation network [14] and generative adversarial network [45]) or enhancing
the performance for various computer vision tasks, such as instance
segmentation [31], refer expression segmentation [27], etc. On the contrary,
for cycle-consistency corresponding learning, we find the position information
causes the neural network to collapse into a shortcut solution, and aim to
eliminate this effect during the self-supervised learning stage.

##  3 Proposed Approach

###  3.1 Revisiting the CRW

During training, CRW [17] first unfolds a sequence of video frames into
different patches $\\{\\{P_{t}^{i}\\}_{t=1}^{T}\\}_{i=1}^{N}$, as illustrated
in Figure 2 (a-left). The transition matrix between two adjacent frames is
defined as:

$\displaystyle
A_{t}^{t+1}(i,j)=\frac{\exp({{<\phi(P_{t}^{i}),\phi(P_{t+1}^{j})>}/{\tau}})}{\sum_{l}\exp({<\phi(P_{t}^{i}),\phi(P_{t+1}^{l})>}/{\tau})},$
(1)

where $\phi$ denotes a ResNet18 encoder followed by a pooling and an fully-
connected layer, $<,>$ represents inner product and $\tau$ is a temperature
factor. Then a long-range walking for timestep 1 to timestep $T$ could be
calculated as $\bar{A}_{1}^{T}=\prod_{i=1}^{T-1}{A_{i}^{i+1}}$. To leverage
temporal cycle-consistency as free supervision, CRW concatenates the sequence
with a temporally reversed version of itself. After walking through this
palindrome, the expected terminal of each query node is its initial position,
which formulates the loss function as:

$\displaystyle\mathcal{L}^{k}_{cyc}$
$\displaystyle=\mathcal{L}_{CE}(\bar{A}_{1}^{T}\bar{A}_{T}^{1},I).$ (2)

During testing, the trained model is used to extract the feature map of the
whole video frame. Then the similarity between the feature nodes from adjacent
frames is calculated, which guides the ground-truth label given in the first
frame to propagate to the later frames progressively.

###  3.2 Vanilla FC3 and its Shortcut

In order to eliminate the inconsistency between the training and testing
process of CRW, it is straightforward to explore a fully convolutional cycle-
consistency correspondence learning during training. Towards this direction,
we first make an attempt as shown in Figure 2 (b). Specifically, given a
sequence of video frames $\\{I_{1},I_{2},...,I_{T}\\}$ with the size of
$T\times H\times W\times 3$, we use a ResNet-18 encoder followed by a fully-
connected layer, denoted as $\Psi$, to extract feature of each image directly,
rather than first dividing them into patches. We obtain a set of feature map
$\\{\phi(I_{1}),\phi(I_{2}),...,\phi(I_{T})\\}$ with the size of $T\times
H^{\prime}\times W^{\prime}\times C$. As previously, we consider them as a
space-time graph, where $H^{\prime}\times W^{\prime}$, $T$ and $C$ corresponds
to the spatial dimension, temporal dimension, and channel dimension,
respectively. Different from Eqn. (1), the transition matrix in this case is
defined as:

$\displaystyle
A_{t}^{t+1}(i,j)=\frac{\exp({{<\phi^{i}(I_{t}),\phi^{j}(I_{t+1})>}/{\tau}})}{\sum_{l}\exp({<\phi^{i}(I_{t}),\phi^{l}(I_{t+1})>}/{\tau})}.$
(3)

Here $\phi^{i}(I_{t})$ denotes the $i$-th node in the feature map
$\phi(I_{t})$. Then, we could perform the cycle-consistency learning similar
to Eqn. (2). However, after doing this, we found the performance drops
significantly (See Figure 1). As indicated by previous work [17], this is
probably because the convolutional architecture encodes the position
information of the input image, thus the network can perform correspondence
learning based on the position information of each node, rather than its
appearance information.

To make a deeper understanding of this, we present visualization results in
Figure 3. In the first row, we show the affinity matrix between two
neighboring frames ($I_{t}$ and $I_{t+1}$) from the same video. Both affinity
matrices from vanilla FC3 and STFC3 (will be detailed later) show a pattern of
large diagonal items. This is plausible because the matching target features
of source features are in the neighboring position. In the bottom row, we show
the affinity matrix between two very different frames. In this case, there is
no strong pattern in the affinity matrix of STFC3 since there are no matching
points between the source and target frames. However, vanilla FC3 still gives
an affinity matrix with large diagonal items. This indicates vanilla FC3
learns correspondence by encoding positional information rather than visual
information in the feature map. The reason behind this is that vanilla FC3
learns a shortcut when trained with cycle consistency loss. The cycle
consistency loss enforces a near identity transition matrix of a cycle. The
transition matrix of a cycle is composed of affinity matrices of adjacent
frames. So vanilla FC3 can encode positional information in the features,
which induce near identity affinity matrices and transition matrix of the
cycle. With this shortcut, vanilla FC3 can fit the training target while
ignoring the visual correspondence.

In order to break this shortcut, we explore a variety of attempts. We
introduce our spatial transform FC3 as below, while other strategies could be
found in the experiment part.

Figure 3:  Visualization of affinity matrices between features given by
vanilla FC3 and our STFC3. These matrices are computed following Eqn. (3). A
brighter pixel indicates a higher similarity between feature points. Vanilla
FC3 gives very similar affinity matrices between a pair of alike frames
($I_{t}$ and $I_{t+1}$) and a pair of very different frames ($I_{t}$ and
$I_{s}$), while STFC3 shows plausible correspondence in the affinity matrices
between the two pairs. This indicates vanilla FC3 learns the correspondence
based on the position rather than visual appearance.

###  3.3 Spatial Transformation FC3

Figure 2 (c) illustrates the pipeline of our method. To break the shortcut
learned by vanilla FC3, our key idea is to perform different spatial
transformation operations on the same frame to get different copies of a raw
frame. For a certain pixel in the raw frame, the positions of the
corresponding pixels in these copies are different since they undergo
different operations. The position information can no longer guide the network
to learn correspondence. In this way, the network is enforced to learn
meaningful representations based on the visual appearance rather than the
position information. In this subsection, we will introduce the spatial
transformation operations, as well as how we align the nodes to form a cycle
and calculate a new loss after the spatial transformation.

Spatial Transformation: As previous, suppose we have a sequence of video
frames $\\{I_{1},I_{2},...,I_{T}\\}$, we first perform different spatial
transformation operations to the forward and backward frames respectively:

$\displaystyle
X_{t}^{f}=\Omega_{\theta_{f}}(I_{t}),\;X_{t}^{b}=\Omega_{\theta_{b}}(I_{t}),\;t=1,2,...,T.$
(4)

For different transformations $\Omega_{\theta_{f}}$ and $\Omega_{\theta_{b}}$,
we execute random resized crop random horizontal flip on the original frames,
and resize them to the same size of $[H_{1},W_{1},3]$. This kind of spatial
transformation could actually be determined by an affine matrix $B_{\theta}$
with hyper-parameter $\theta$ [18], which transforms the source coordinates to
the target ones. We leverage $B_{\theta}$ to warp feature, which will be
detailed later. Similar to vanilla FC3, we adopt a feature extractor $\psi$
containing a ResNet-18 encoder and an 1 $\times$ 1 convolutional layer to
extract features of these augmented frames:

$\displaystyle
F_{t}^{f}=\psi(X_{t}^{f}),\;F_{t}^{b}=\psi(X_{t}^{b}),\;t=1,2,...,T.$ (5)

Through the transformation Eqn.(4), the same region of the raw input $I_{i}$
corresponds different nodes of forward features $F_{i}^{f}$ and backward
features $F_{i}^{b}$.

Walking through a Palindrome:  After spatial transformation, we concatenate
the obtained feature to form a new sequence:

$\displaystyle S=[F_{1}^{f},\cdots,F_{T}^{f},F_{T}^{b},\cdots,F_{1}^{b}].$ (6)

The walk between two adjacent frames is slightly different as previous, which
could be formulated as:

$\displaystyle\\{A^{S}_{t}\\}_{i,j}=\frac{\exp({{<S^{i}_{t},S^{j}_{t+1}>}/{\tau}})}{\sum_{l}\exp({<S^{i}_{t},S^{l}_{t+1}>}/{\tau})}.$
(7)

To form a palindrome, a node first walks from timestep 1 to $T$ in the forward
track, then moves to timestep $T$ in the backward track, and finally returns
to timestep 1. The whole process could be written as:

$\displaystyle A_{cyc}^{S}=\prod_{t=1}^{2T-1}A^{S}_{t}.$ (8)

Feature Warping: With spatial transformation, the identity mapping between the
nodes of the beginning and the ending of a palindrome is not preserved
anymore, because the forward frame $I^{f}_{t}$ and the corresponding backward
frame $I^{b}_{t}$ undergo different spatial transformation and are not aligned
anymore. To this end, we introduce a feature warping strategy to find the
mapping between the start nodes $F_{1}^{f}$ and end nodes $F_{1}^{b}$ of a
palindrome.

When transforming the forward and backward frames, we collect the
corresponding affine matrices $B_{\theta_{f}},B_{\theta_{b}}$ Then we compute
the affine matrix that can align the forward frame with the backward frame by
$B_{fb}=B_{\theta_{b}}B_{\theta_{f}}^{-1}$, and apply it to the first feature
map $F_{1}^{f}$ in the forward track as:

$\displaystyle F_{1}^{fb}=\mathcal{T}_{B_{fb}}(F_{1}^{f}),$ (9)

where $\mathcal{T}_{B}(F)$ means applying transformation with affine matrix
$B$ to a 2D feature $F$. After this warping, the nodes of $F_{1}^{fb}$ become
aligned with the backward feature map $F_{1}^{b}$. Notice here, after the
spatial feature warping, the position-feature mapping is changed. In this way,
we can keep the identity mapping between the start and end nodes while
factoring out the position information. The sequence of features after warping
is:

$\displaystyle
S_{w}=[F_{1}^{fb},\cdots,F_{T}^{f},F_{T}^{b},\cdots,F_{1}^{b}].$ (10)

Figure 4: (a) We employ different spatial transformation
$\Omega_{\theta_{f}}$, $\Omega_{\theta_{b}}$ to the frames in the forward
track and backward track, so that the same content in the original input
$I_{t}$ corresponds to different positions in the extracted feature maps
$F_{t}^{f}$ and $F_{t}^{b}$. (b) Illustration of the feature warping.
According to [18], the spatial transformation $\Omega_{\theta_{f}}$ and
$\Omega_{\theta_{b}}$ corresponds to two affine matrices $B_{\theta_{f}}$ and
$B_{\theta_{b}}$. We use $B_{fb}=B_{\theta_{b}}B_{\theta_{f}}^{-1}$ as the
warping matrix, and perform the corresponded transformation
$\mathcal{T}_{B_{fb}}$ on $F_{1}^{f}$. With the obtained feature $F_{1}^{fb}$
and other features, we form a new cycle
$S_{w}=[F_{1}^{fb},\cdots,F_{T}^{f},F_{T}^{b},\cdots,F_{1}^{b}]$ for self-
supervised learning.

Mask Loss: The random crop for forward and backward frames can be different.
When the crop region of backward frames is not contained in the counterpart of
forward frames, the warped forward frame $F_{1}^{fb}$ would have zero margins.
And the nodes in these zero margins do not correspond to any of the end nodes.
To tackle this problem, we compute a mask by applying the same affine
transformation to an all-one 2D tensor:

$\displaystyle M=\mathcal{T}_{B_{fb}}(\mathbbm{1}).$ (11)

Here $\mathbbm{1}$ is an all-one 2D tensor with the same spatial size as
$F_{1}^{f}$. Then we binarize $M$ using 0.5 as threshold to get binary mask
$M_{b}$. We mask out the losses of all walks that start from a node in zero
margin, which are marked as 0 in $M_{b}$. Finally, we optimize the total loss
as follows:

$\displaystyle\mathcal{L}^{S_{w}}_{cyc}$
$\displaystyle=\frac{1}{H_{2}W_{2}}\sum_{i=1}^{H_{2}W_{2}}M_{b}(i)\mathcal{L}_{CE}(A_{cyc}^{S_{w}}(i),i)$
(12)
$\displaystyle=-\frac{1}{H_{2}W_{2}}\sum_{i=1}^{H_{2}W_{2}}M_{b}(i)\log(A_{cyc}^{S_{w}}(i,i)).$
(13)

Here $H_{2}$ and $W_{2}$ are the height and width of the feature map. Given a
video sequence of length T, we train our model on the multiple cycles of the
sequence $S_{w}^{1},...,S_{w}^{T-1}$, where
$S_{w}^{k}=[F_{k}^{fb},\cdots,F_{T}^{f},F_{T}^{b},\cdots,F_{k}^{b}].$ And the
final loss involves cycles of multiple lengths during training
$\mathcal{L}_{cyc}=\sum_{k=1}^{T-1}\mathcal{L}^{S_{w}^{k}}_{cyc}.$

Table 1: Experiment results on different methods to avoid the shortcut solution. Our method achieves significant improvement over the vanilla FC3 method on three label propagation tasks. Experiments are conducted on the J-HMDB [19], 300VW [36] and DAVIS-2017 [33] for pose tracking, face landmark tracking and video object segmentation respectively. Task | Pose Tracking | Face Landmark Tracking | Video Object Segmentation
---|---|---|---
Metric |<EMAIL_ADDRESS>$\uparrow$ |<EMAIL_ADDRESS>$\uparrow$ | RMSE $\downarrow$ |  $\mathcal{J}$&$\mathcal{F}_{\textrm{m}}\uparrow$ | $\mathcal{J}_{\textrm{m}}\uparrow$ | $\mathcal{F}_{\textrm{m}}\uparrow$
Vanilla FC3 (Zero Padding) | 32.4 | 48.4 | 56.7 | 18.0 | 15.7 | 20.2
FC3 (Replicate Padding) | 49.7 | 67.6 | 28.2 | 31.5 | 29.8 | 33.3
FC3 (Reflect Padding) | 45.9 | 63.7 | 26.9 | 28.8 | 26.3 | 32.0
FC3 (No Padding) | 35.1 | 52.7 | 50.2 | 38.8 | 35.6 | 41.9
STFC3 (Ours) | 62.0 | 80.5 | 18.8 | 60.5 | 58.0 | 63.1
Table 2: Evaluation of the pose tracking task with the J-HMDB benchmarks. SM, D and I represent using Sintel Movie [25], DAVIS2017 and ImageNet as the training data respectively. Method | Supervised? | Training Data | Backbone |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---|---|---
Thin-Slicing Network [37] | $\surd$ | J-HMDB+I | Self-Designed | 68.7 | 92.1
PAAP [15] | $\surd$ | J-HMDB+I | VGG-16 | 51.6 | 73.8
ResNet-18 [12] | $\surd$ | ImageNet | ResNet18 | 59.0 | 80.6
MoCo [11] | $\times$ | ImageNet | ResNet18 | 58.1 | 75.6
VINCE [7] | $\times$ | Kinetics | ResNet18 | 58.4 | 75.7
Identity | $\times$ | - | - | 43.1 | 64.5
ColorPointer [40] | $\times$ | Kinetics | ResNet18 | 45.2 | 69.6
TimeCycle [42] | $\times$ | VLOG | ResNet18 | 57.3 | 78.1
mgPFF [21] | $\times$ | SM+D+J-HMDB | ResNet18 | 58.4 | 78.1
UVC [26] | $\times$ | Kinetics | ResNet18 | 58.6 | 79.6
CRW [17] | $\times$ | Kinetics | ResNet18 | 58.8 | 80.2
VFS [44] | $\times$ | Kinetics | ResNet18 | 60.5 | 79.5
STFC3 (Ours) | $\times$ | Kinetics | ResNet18 | 62.0 | 80.5

##  4 Experiments

###  4.1 Datasets and Experiment Settings

We train our model from scratch in the Kinetics dataset [4], without using any
human annotation. The evaluation is performed on three label propagation
tasks. Given the ground-truth labels of the first frames, the goal is to
predict the label of the later frames. We introduce the datasets used for
different tasks as follows.

J-HMDB [19]: We conduct experiments in J-HMDB dataset for pose tracking,
following the protocol adopted in [26, 17]. There are 268 videos for testing,
where each video contains about 40 frames with 15 keypoints of body. We use
frames with 640$\times$640 resolution as the input of the model. We report the
performance of different methods with the PCK@$\alpha$ evaluation metric,
which denotes the Probability of Correct Keypoint at a threshold of $\alpha$.

300VW [36]: The 300VW is a widely used video dataset for face landmark
tracking. We evaluate our model on split-1 which contains 31 videos. This task
is more difficult than the pose tracking task, as there are thousands of
frames in each video and each video frame contains 68 face landmarks. We
follow [47] to use the cropped face as input and resized to the size of 320
$\times$ 320\. We employ the point-to-point Root-Mean-Square Error (RMSE)
defined in [36] as the evaluation metric.

DAVIS-2017 [33]: We study the video object segmentation task on the validation
set of DAVIS-2017, which contains 30 Youtube videos. We use 720p resolution
images as input, and evaluate our method based on the m (mean) of Jacaard
index $\mathcal{J}$ and contour-based accuracy $\mathcal{F}$ metrics [33].

###  4.2 Implementation Details

During the training process, we adopted ResNet-18 encoder followed by 2 fully-
connected (fc) layers to extract the feature map of the input video frames.
The dimensions of the fc layers were 2048 and 512. We used the Adam optimizer
with the initial learning rate of 1e-4 to train our model. The batch size,
temperature factor $\tau$, and the cycle-length was set to be 64, 0.05 and 4,
respectively. A random resized crop with size $~{}U(0.08,1)$ and a random
horizontal flip with probability 0.5 were used as the spatial transformation.
We also applied color augmentations in [8] to all the frames. We followed [17]
to modify the stride of the layer3 and layer4 from 2 to be 1, resulting in
$32\times 32$ nodes for random walking. We trained our model with 8 V100 GPUs
and found it converged at the first epoch on the Kinetics dataset, The
training process only costed about 4.5 hours, largely saving the time compared
with previous self-supervised learning works [17, 26].

During the testing process, we removed the fc layers and the 4-th ResNet18
layer of the trained model to extract features, obtaining a set of source
nodes of the first frame and the target nodes of the later frame. Then we use
Eqn. (3) to calculate the similarity of each node, and consider the top-k
transitions to perform label propagation [17, 42, 26]. We also adopted the
temporal context frames $m$ and restricted the label radius $r$ [17] to
achieve better performance. More details of the hyper-parameters we used could
be found in the supplementary material.

###  4.3 Study of Different Methods to Avoid the Shortcut Solution

Through the visualization results in Figure 3, we find that the position
encoded by the CNN model is the main reason to cause the shortcut solution.
According to the previous work [16], the position information is mainly
introduced by the zero paddings. So a straightforward way to avoid the
shortcut solution is to use different padding methods (e.g., replicate padding
and reflect padding). Another strategy is to give up padding operation and
crop the feature boundary as the network goes in deeper [17]. We first
evaluate these methods and present the results in Table 1. We observe that
these strategies can improve the baseline accuracy and alleviate the shortcut
effect, but the performance is still not promising. This is because changing
padding methods do not eliminate the source of the position information,
although different padding methods make it harder to learn to encode position
information. In comparison, our STFC3 achieves the improvements of 29.6
<EMAIL_ADDRESS>on J-HMDB for pose tracking, -37.4 RMSE($\downarrow$) on 300VW for
face landmark tracking and 42.5 $\mathcal{J}$&$\mathcal{F}_{\textrm{m}}$ on
DAVIS-2017 for video object segmentation. This strongly demonstrates the
effectiveness of our method to address the shortcut issue.

###  4.4 Comparison with the State-of-the-art

Pose Tracking: We present the experimental results of pose tracking in Table
2, where our STFC3 achieves the state-of-the-art of 62.0<EMAIL_ADDRESS>and 80.5
<EMAIL_ADDRESS>under the self-supervised setting. Compared with the most related work
CRW [17] (the results are based on the officially released model) , our STFC3
obtains 3.2 improvements on<EMAIL_ADDRESS>which strongly demonstrates its
superiority of pose tracking task. We display some qualitative results for
pose tracking in Figure 5.

Table 3: Face landmark tracking results on the 300VW dataset [36], where the lower $\downarrow$ is better. Method | RMSE $\downarrow$ | Supervision
---|---|---
STRRN [47] | 5.31 | 300W [35]
SBR [6] | 5.77 | 300W [35] \+ ImageNet
ResNet-18 | 22.8 | ImageNet
MoCo [11] | 23.3 | self-supervision
VINCE [7] | 23.4 | self-supervision
CRW [17] | 21.6 | self-supervision
UVC [26] | 19.9 | self-supervision
STFC3 (Ours) | 18.8 | self-supervision
Figure 5: Visualization of the pose tracking results in the J-HMDB dataset [19], given the ground-truth position of 15 key points of human body in the first frame, our proposed method effectively tracks them in the later frames. Figure 6: Qualitative results of our method on 300VW [36] for face landmark tracking and DAVIS-2017 [33] for video object segmentation. The ground-truth labels in the first frame with a red outline are propagated to the later frames.  Table 4: Evaluation on the DAVIS-2017 dataset for video object segmentation. I, C, D, K, P, M, Y represents ImageNet, COCO, DAVIS2017, Kinetics, PASCAL-VOC, Mapillary and YouTube-VOS. The methods with * are under fully-supervised learning setting. All the self-supervised methods are based on ResNet-18. Method | Train Data |  $\mathcal{J}$&$\mathcal{F}_{\textrm{m}}$ | $\mathcal{J}_{\textrm{m}}$ | $\mathcal{F}_{\textrm{m}}$
---|---|---|---|---
PReMVOS* [29] | I/C/D/P/M | 77.8 | 73.9 | 81.8
STM* [32] | I/D/Y | 81.8 | 79.2 | 84.3
CFBI* [46] | I/C/D | 83.3 | 80.5 | 86.0
ResNet-18* [13] | I | 62.9 | 60.6 | 65.2
MoCo [11] | I | 60.8 | 58.6 | 63.1
VINCE [7] | K | 60.4 | 57.9 | 62.8
Colorization [40] | K | 34.0 | 34.6 | 32.7
TimeCycle [42] | VLOG | 48.7 | 46.4 | 50.0
CorrFlow [23] | OxUvA | 50.3 | 48.4 | 52.2
UVC+track [26] | K | 59.5 | 57.7 | 61.3
MAST [22] | Y | 65.5 | 63.3 | 67.6
VFS[44] | K | 66.6 | 64.0 | 69.4
CRW[17] | K | 67.6 | 64.8 | 70.2
STFC3 (Ours) | K | 60.5 | 58.0 | 63.1
Figure 7: Visualization of the top 3 PCA components of the feature map learned
by different self-supervisedly pretrained models.

Face Landmark Tracking: As shown in Table 3, compared with the methods without
using any supervision or initialized models pretrained on other datasets, our
STFC3 achieves the state-of-the-art result of 18.8 RMSE, which indicates its
advantages for unsupervised face landmark tracking. However, we notice there
is still a performance gap compared with the STRRN [47] and SBR [6]. This is
because these methods used the face detector pretrained on the 300W dataset,
which introduce strong prior knowledge for this challenging task.

Video Object Segmentation: We conduct experiments on the DAVIS-2017 dataset
for video object segmentation (VOS). We present the quantitative results
compared with the current VOS methods in Table 4 and some visualization
results in Figure 8. Our proposed STFC3 method achieves a performance of 60.5
$\mathcal{J}$&$\mathcal{F}_{\textrm{m}}$ with only 1 training epoch on the
Kinetics dataset, while the CRW requires 25 epochs to achieve the state-of-
the-art result.

While we note the CRW approach performs better than our STFC3 method, it turns
out that it learns invariant patch-level region features. On the other hand,
our FCN based method performs pixel-level learning as shown in Figure 7 and
achieves better results in keypoint tracking shown in Table 2 and Table 3,
which is the more important aim of this paper.

###  4.5 Ablation Study

We further study different variants of our STFC3 in the validation set of
DAVIS-2017. Table LABEL:tab:ablation presents the experiments ablation
results. We first find that using data augmentation strategy will lead to 1.3
$\mathcal{J}$&$\mathcal{F}_{\textrm{m}}$ improvement. This is consistent with
the finding obtained by previous image-based self-supervised learning works
[11, 5, 8, 1, 2].

Last but not least, since the crop operation is an important process of our
method, we conduct experiments on different arranged ratios of crop size area
$\gamma$, fixing the upper bound to 1. When the lower bound is relatively
large, the model obtains poor result (e.g., 49.6
$\mathcal{J}$&$\mathcal{F}_{\textrm{m}}$ for $\gamma$ = [0.75, 1.0]). When the
lower bound decreases, the model achieves better accuracy, and reaches the
peak of 60.5 $\mathcal{J}$&$\mathcal{F}_{\textrm{m}}$ when $\gamma$ = [0.08,
1.0]. This is reasonable, because using a small lower bound of $\gamma$
results in a large variance of the crops positions, which is more beneficial
for breaking the shortcut solution caused by the position encoding. Meanwhile,
when the lower bound of $\gamma$ becomes too small, the performance will
slightly drop since the probability of no overlap between two crops (no
correspondence) might be too high.

Table 5: Ablation study on the color augmentation and the arranged ratio of cropsize area $\gamma$. All the model are evaluated on DAVIS-2017 validation set. Method |  $\mathcal{J}$&$\mathcal{F}_{\textrm{m}}$
---|---
STFC3 | 60.5
STFC3 w/o color augmentation | 59.2
STFC3, $\gamma$ = [0.75, 1.0] | 49.6
STFC3, $\gamma$ = [0.50, 1.0] | 54.6
STFC3, $\gamma$ = [0.25, 1.0] | 59.5
STFC3, $\gamma$ = [0.08, 1.0] | 60.5
STFC3, $\gamma$ = [0.01, 1.0] | 60.1

##  5 Conclusion

In this paper, we have explored using fully convolutional cycle-consistency
methods for self-supervised video correspondence learning. While directly
doing this leads to a shortcut solution, we have analysed the underline reason
of this phenomenon, and proposed a spatial transformation approach to address
this issue. Our method has largely improved the performance of the vanilla
model, and achieved very competitive performance on J-HMDB dataset for pose
tracking, 300VW dataset for face landmark tracking and DAVIS-2017 dataset for
video object segmentation. This work has demonstrated the potential of using
the fully convolutional method to perform cycle-consistency learning, and we
hope it will shed light in this direction for future works involving more
dense learning objectives.

##  6 Acknowledgement

This work is supported by the EPSRC grant/Turing AI Fellowship EP/W002981/1,
EPSRC/MURIgrant EP/N019474/1, and the Tsinghua Zijing scholar Fellowship. We
would like to thank the Royal Academy of Engi-neering, Tencent, and FiveAI. We
would also like to thank Allan Jabri, Prof. Hao Liu, Congcong Zhu, Dr. Zheng
Zhu and Xueting Li for helpful discussion.

##  Appendix A Qualitative Comparison to Previous State-of-the-art

Figure 8: Visualization comparison of our method with previous state-of-the-
art CRW [17].

We show two qualitative comparison examples in Figure 8. The two examples are
about pose tracking on J-HMDB and face landmark tracking on 300VW,
respectively. Red circle areas indicate the better localization results by our
method compared with those of CRW. Our approach is more precise in localizing
elbows, and also more accurate at neck area of a face. Please zoom in to have
a better view.

##  Appendix B Color Augmentations

We follow [8] and use two sets of color augmentations for forward and backward
frames. For forward frames, we use the following augmentations:

  * •

Randomly apply color jitter with a probability of 0.8. The parameters of color
jitter are brightness 0.8, contrast 0.8, saturation 0.8, hue 0.2.

  * •

Randomly convert image to grayscale with a probability of 0.2. The sigma of
the Gaussian kernel is uniformly sampled in [0.1, 2.0].

  * •

Randomly apply Gaussian blur to image with a probability of 0.2.

For backward frames, we use the following augmentations:

  * •

Randomly apply color jitter with a probability of 0.8. The parameters of color
jitter are brightness 0.4, contrast 0.4, saturation 0.2, hue 0.1.

  * •

Randomly convert image to grayscale with a probability of 0.2.

  * •

Randomly apply Gaussian blur to image with a probability of 0.2.

  * •

Randomly solarize the image with a probability of 0.2.

We apply the color augmentations to each frames independently.

##  Appendix C Inference Hyper-parameters

We provide some hyper-parameters we used during the inference stages in Table
LABEL:tab:hyper, including the context length of $m$, the spatial radius of
source nodes $r$, and the number of neighours $k$. We chose them mainly
following the prior work [17].

Table 6: Hyper-parameters used for different downstream tasks. Task | $m$ | $r$ | $k$
---|---|---|---
Pose Tracking | 7 | 10 | 10
Face Landmark Tracking | 4 | 5 | 10
Video Object Segmentation | 20 | 18 | 10

## References

  * [1] Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu.  Parametric instance classification for unsupervised visual feature learning.  In Advances in neural information processing systems, 2020.
  * [2] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In NeurIPS, 2020.
  * [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [4] João Carreira and Andrew Zisserman.  Quo vadis, action recognition? A new model and the kinetics dataset.  In CVPR, pages 4724–4733, 2017.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, volume 119, pages 1597–1607.
  * [6] Xuanyi Dong, Shoou-I Yu, Xinshuo Weng, Shih-En Wei, Yi Yang, and Yaser Sheikh.  Supervision-by-registration: An unsupervised approach to improve the precision of facial landmark detectors.  In CVPR, pages 360–368, 2018.
  * [7] Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi.  Watching the world go by: Representation learning from unlabeled videos, 2020.
  * [8] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko.  Bootstrap your own latent - A new approach to self-supervised learning.  In NeurIPS, 2020.
  * [9] Tengda Han, Weidi Xie, and Andrew Zisserman.  Memory-augmented dense predictive coding for video representation learning.  In ECCV, pages 312–329, 2020.
  * [10] Tengda Han, Weidi Xie, and Andrew Zisserman.  Self-supervised co-training for video representation learning.  In NeurIPS, 2020.
  * [11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, pages 9726–9735. IEEE, 2020.
  * [12] Kaiming He and Jian Sun.  Convolutional neural networks at constrained time cost.  In Computer Vision and Pattern Recognition (CVPR), 2015.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Computer Vision and Pattern Recognition (CVPR), 2016.
  * [14] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin.  Local relation networks for image recognition.  In ICCV, pages 3463–3472. IEEE, 2019.
  * [15] Umar Iqbal, Martin Garbade, and Juergen Gall.  Pose for action - action for pose.  In FG, pages 438–445, 2017.
  * [16] Md. Amirul Islam, Sen Jia, and Neil D. B. Bruce.  How much position information do convolutional neural networks encode?  In ICLR, 2020.
  * [17] Allan Jabri, Andrew Owens, and Alexei A Efros.  Space-time correspondence as a contrastive random walk.  In NeurIPS, 2020.
  * [18] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  In NeurIPS, pages 2017–2025, 2015.
  * [19] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black.  Towards understanding action recognition.  In ICCV, 2013.
  * [20] Osman Semih Kayhan and Jan C. van Gemert.  On translation invariance in cnns: Convolutional layers can exploit absolute spatial location.  In CVPR, pages 14262–14273. IEEE, 2020.
  * [21] Shu Kong and Charless Fowlkes.  Multigrid predictive filter flow for unsupervised learning on videos.  arXiv preprint arXiv:1904.01693, 2019.
  * [22] Zihang Lai, Erika Lu, and Weidi Xie.  MAST: A memory-augmented self-supervised tracker.  In CVPR, pages 6478–6487. IEEE, 2020.
  * [23] Zihang Lai and Weidi Xie.  Self-supervised learning for video correspondence flow.  arXiv preprint arXiv:1905.00875, 2019.
  * [24] Zihang Lai and Weidi Xie.  Self-supervised video representation learning for correspondence flow.  In BMVC, page 299, 2019.
  * [25] C. Levy and T. Roosendaal.  Sintel.  In ACM SIGGRAPH ASIA 2010 Computer Animation Festival, 2010\.
  * [26] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang.  Joint-task self-supervised learning for temporal correspondence.  In Advances in Neural Information Processing Systems, pages 317–327, 2019.
  * [27] Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan L. Yuille.  Recurrent multimodal interaction for referring image segmentation.  In ICCV, pages 1280–1289, 2017.
  * [28] Xiankai Lu, Wenguan Wang, Jianbing Shen, Yu-Wing Tai, David J. Crandall, and Steven C. H. Hoi.  Learning video object segmentation from unlabeled videos.  In CVPR, pages 8957–8967, 2020.
  * [29] Jonathon Luiten, Paul Voigtlaender, and Bastian Leibe.  Premvos: Proposal-generation, refinement and merging for video object segmentation.  In Asian Conference on Computer Vision, pages 565–580. Springer, 2018.
  * [30] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  In CVPR, pages 9876–9886, 2020.
  * [31] Davy Neven, Bert De Brabandere, Marc Proesmans, and Luc Van Gool.  Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth.  In CVPR, pages 8837–8845, 2019.
  * [32] Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim.  Video object segmentation using space-time memory networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9226–9235, 2019.
  * [33] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  arXiv:1704.00675, 2017.
  * [34] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge J. Belongie, and Yin Cui.  Spatiotemporal contrastive video representation learning.  In CVPR, 2021.
  * [35] Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic.  300 faces in-the-wild challenge: The first facial landmark localization challenge.  In ICCVW, pages 397–403, 2013.
  * [36] Jie Shen, Stefanos Zafeiriou, Grigoris G. Chrysos, Jean Kossaifi, Georgios Tzimiropoulos, and Maja Pantic.  The first facial landmark tracking in-the-wild challenge: Benchmark and results.  In ICCVW, pages 1003–1011, 2015.
  * [37] Jie Song, Limin Wang, Luc Van Gool, and Otmar Hilliges.  Thin-slicing network: A deep structured model for pose estimation in videos.  In CVPR, pages 5563–5572, 2017.
  * [38] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In ICCV, pages 7463–7472, 2019.
  * [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, pages 5998–6008, 2017.
  * [40] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy.  Tracking emerges by colorizing videos.  In ECCV, 2017.
  * [41] Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, and Houqiang Li.  Unsupervised deep tracking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1308–1317, 2019.
  * [42] Xiaolong Wang, Allan Jabri, and Alexei A Efros.  Learning correspondence from the cycle-consistency of time.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2566–2576, 2019.
  * [43] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu.  Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.  In CVPR, 2021.
  * [44] Jiarui Xu and Xiaolong Wang.  Rethinking self-supervised correspondence learning: A video frame-level similarity perspective.  In ICCV, 2021.
  * [45] Rui Xu, Xintao Wang, Kai Chen, Bolei Zhou, and Chen Change Loy.  Positional encoding as spatial inductive bias in gans.  In CVPR, pages 8743–8752, 2020.
  * [46] Zongxin Yang, Yunchao Wei, and Yi Yang.  Collaborative video object segmentation by foreground-background integration.  In ECCV, pages 332–348, 2020.
  * [47] Congcong Zhu, Hao Liu, Zhenhua Yu, and Xuehong Sun.  Towards omni-supervised face alignment for large scale unlabeled videos.  In AAAI, pages 13090–13097.
  * [48] Linchao Zhu and Yi Yang.  Actbert: Learning global-local video-text representations.  In CVPR, pages 8743–8752, 2020.