#  What and When to Look?:
Temporal Span Proposal Network for Video Visual Relation Detection

Sangmin Woo${}^{\textsuperscript{\href
https://orcid.org/0000-0003-4451-9675}}$, Junhyug
Noh${}^{\textsuperscript{\href https://orcid.org/0000-0003-1239-8178}}$, and
Kangil Kim${}^{\textsuperscript{\href https://orcid.org/0000-0003-3220-6401}}$
This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government (MSIT) (2019R1A2C109107712), and the
Institute of Information & communications Technology Planning & Evaluation
(IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01842,
Artificial Intelligence Graduate School Program (GIST)). (Corresponding
author: Kangil Kim)Sangmin Woo is with the School of Electrical Engineering
and Computer Science, Gwangju Institute of Science and Technology, Gwangju
61005, Korea (email: shmwoo9395@gmail.com)Junhyug Noh is with the
Computational Engineering Division, Lawrence Livermore National Laboratory, CA
94550, United States (email: noh1@llnl.gov).Kangil Kim is with the School of
Electrical Engineering and Computer Science and the AI Graduate School,
Gwangju Institute of Science and Technology, Gwangju 61005, Korea (email:
kangil.kim.01@gmail.com).

###### Abstract

Identifying relations between objects is central to understanding the scene.
While several works have been proposed for relation modeling in the image
domain, there have been many constraints in the video domain due to
challenging dynamics of spatio-temporal interactions (e.g., Between which
objects are there an interaction? When do relations occur and end?). To date,
two representative methods have been proposed to tackle Video Visual Relation
Detection (VidVRD): segment-based and window-based. We first point out the
limitations these two methods have and propose Temporal Span Proposal Network
(TSPN), a novel method with two advantages in terms of efficiency and
effectiveness. 1) TSPN tells what to look: it sparsifies relation search space
by scoring relationness (i.e., confidence score for the existence of a
relation between pair of objects) of object pair. 2) TSPN tells when to look:
it leverages the full video context to simultaneously predict the temporal
span and categories of the entire relations. TSPN demonstrates its
effectiveness by achieving new state-of-the-art by a significant margin on two
VidVRD benchmarks (ImageNet-VidVDR and VidOR) while also showing lower time
complexity than existing methods – in particular, twice as efficient as a
popular segment-based approach.

###### Index Terms:

Video Visual Relationship Detection (VidVRD), Spatio-temporal Video
Understanding, Temporal Span Proposal Network (TSPN).

##  I Introduction

Figure 1:  Who is handing over the bear to whom? The relation within a image
is equivocal. While guessing relations in short-term video segments is still
questionable, the answer becomes clear in full video thanks to the temporal
contexts.

Capturing the semantics of the visual scene has long been a concern in
computer vision. Despite the remarkable progress of computer vision,
understanding visual scenes remains a challenging task. On the way to leap
forward, visual relations serve as the stepping stone for narrowing the gap
between perceptive and cognitive tasks. A number of works leverages the visual
relations including the tasks of image retrieval [1], dynamics prediction [2,
3], image captioning [4, 5], visual question answering [6], image generation
[7, 8] and video understanding [9]. The Visual Relation Detection (VRD) task
requires modeling both visible information about what and where entities are
and underlying information of what interactions are happening between objects
in the scene. Relation reasoning is essential in a high-level understanding of
the scene. Since a pioneer work of VRD [10] was proposed, several interesting
relational reasoning approaches have been studied on the image domain [11, 12,
13]. Recently, a Video Visual Relation Detection (VidVRD) task has been
proposed, but due to the difficulty of spatio-temporal relationship modeling,
it has not yet been received much attention, and just a handful of works [14,
15, 16, 17, 18, 19, 20] have been proposed.

Figure 2:  Conceptual Comparison of three VidVRD approaches (empirical
comparisons are in Table II, III, VII). (a) Segment-based approach: first
chunk video into several segments, predict the short-term relations, and then
greedily associate the relations of adjacent segments into the long-term
relations. (b) Window-based approach: first sample a set of sub-tracklet pairs
via a size-varying sliding window, and then predict all relations with
different temporal span, and (c) TSPN-based approach (Ours): jointly predict
relation categories and its temporal span with a single video-level object
trajectory pair. $O_{i}$ stands for $i$-th object trajectory of all object
trajectories in the video, and $R_{j}$ denotes $j$-th relation category. We
assume that the relations are predicted only for temporal span in which two
object trajectories appear simultaneously in the video. Also note that
illustration of each method may not contain all the detailed procedures.

What makes detecting relations in video difficult? At first glance, we
consider the common difficulties of the task. In the view of the object, new
objects may appear or disappear due to viewpoint variation or occlusion over
time and may also contain motion blur. Therefore, we need to adopt proper
object detection [21] and tracking [22, 23] methods. In the view of the
relation, it requires modeling long-term temporal dependency and complex
dynamics. Relational reasoning in the video requires contextual information
over time, unlike static images. For example, in Fig. 1, the relation between
two men and bear is ambiguous and hard to define with a static image but can
be inferred from temporal contexts. Suppose there are $n$ object and $m$
relation categories, the number of possible combinations is $O(n^{2}\times
m)$. A naive approach to this problem is to learn a distribution over the
$n\times n\times m$ lattice space – the complete combinations of objects and
relations. However, due to the sparsity of existing relation types and data
limits, the model is highly prone to bias. A simple sidestep is to learn low-
rank matrices, factorized by object and relations. To put it more simply, the
strategy is to separate pipelines of predicting objects and relations,
reducing the complexity to $O(n+m)$. To this end, we first detect objects and
then predict relations conditionally. Here we further factorize the relation
prediction process into two subprocesses: relationness scoring and temporal
span proposal. We detail the steps in Sec. III.

To investigate the more complicated issues of the task, we compare existing
VidVRD approaches in Fig. 2. They are divided into two approaches: (a)
segment-based and (b) window-based. Segment-based approaches [14, 15, 16, 17,
18, 19] first split a video into several segments – typically a segment
contains 30 frames, and then predict relations at the segment-level. If the
neighboring segments share the same relation triplets, they are merged into a
single relation with an extended range. This approach predicts segment-wise
relations with only corresponding 30 frames, assuming that the basic relations
can always be found in a short duration and can indirectly build the video-
level predictions via the association method. However, we argue that 30 frames
are not enough to model long-term interactions such as ‘dog-past-person’.
Because the relation ‘past’ can only be inferred based on the overall process
by which a dog is behind a person and overtakes a person, continuous
monitoring of interactions between objects is required during successive
events. More recently, in order to leverage the temporal globality, a window-
based approach [20] has been proposed. It adopts a sliding window scheme over
the whole object trajectories and obtains all possible sizes of object
tracklets. With the object tracklets of various lengths, it can predict both
short-term and long-term interactions. However, we see that this is hardly
scalable for extremely long videos such as movies because it requires an
exhaustive search to sample all combinations of size-varying tracklet pairs to
cover various durations of relations, resulting in a cubic complexity.

This work aims to bridge the gap between segment- and window-based methods
while improving the shortcomings and exploiting the advantages of the methods
with Temporal Span Proposal Network (TSPN) (see Fig. 2(c)). Given a pair of
object trajectories, TSPN first finds what pairs of objects are probable to
have relations in-between111In this paper, we use the term ‘relationness’ to
indicate the probability that a relation exists between a pair of objects.,
then predicts when the pair-wise relations begin and end. We call the first
step relationness scoring and the second step temporal span proposal. Since it
is possible that the object pairs can have multiple relations, we formalize
the second step of TSPN as a multi-label classification in practice. By
leveraging both temporal locality as in the segment-based approach and
temporal globality as in the window-based approach, TSPN can effectively
predict pair-wise object relationships. It is also clear that TSPN is more
efficient than both methods because it only uses each object pair once to
predict all relationships across the video. In particular, it prevents
duplicate use of each object tracklet by dividing it into different window
lengths, such as the window-based method. By design, the proposed TSPN can
coarsen relation search space by capturing the regularity of pair-wise object
interaction. Also, TSPN leverages the global video context features, making it
strong in both the short and long-term relationship modeling.

We validate TSPN against existing approaches on two popular video visual
relation datasets: ImageNet-VidVRD [14] and VidOR [24]. We observe that our
method achieved new state-of-the-art on both benchmarks with a significant
performance gain. Also, we examine the theoretical computation complexity of
three methods (segment, window, and TSPN) and see that TSPN is approximately
twice more efficient than the segment-based method. Comprehensive quantitative
and qualitative analyses demonstrate the efficacy of TSPN.

##  II Related Work

###  II-A Object/Relation Proposals

The Region Proposal Network (RPN) [21] used for the object detection strongly
inspired our TSPN. RPN simultaneously predicts object bounds and ‘objectness’
score (we named the term ‘relationness’ after this) to detect the region of
interest. In line with RPN, we extend the proposal concept from the space
domain to the time domain.

Several recently proposed works also share a similar spirit with TSPN that of
reducing search space [25, 26, 20]. However, those approaches are limited to
proposing relationships between objects and do not address the time domain.

###  II-B Relation Detection

In an effort to detect relations, numerous studies have been explicitly
modeled and adopted neural networks. The challenging and open-ended nature of
the task lends itself to various forms. It includes VRD [10, 12] which is the
generic form of visual relational reasoning tasks, Human-Object Interaction
(HOI) [27, 28, 29] which mainly focuses on human and considers them as the
subject, and Scene Graph Generation (SGG) [30, 26, 31, 32, 33, 34] which is a
structured form of VRD. Most of the works are mainly conducted in the static
image domain.

As a breakthrough, the pioneer work [14] has introduced the first dataset and
the baseline of VRD in the dynamic video domain. Since then, just a few works
have been proposed due to the difficulty of spatio-temporal modeling. A
majority of existing methods follow the segment-based approach [14, 15, 17,
16, 18, 19] which was firstly introduced in [14]. It first breaks a video into
segments, predicts per-segment relations, and finally associates relations in
a greedy manner. More recently, [20] proposed a window-based approach to
handle long-term relations. However, the above-mentioned methods either lack
temporal globality or scale poorly for extremely long videos such as movies.

To move one step forward, we introduce a novel TSPN to directly predict
relations in an end-to-end manner without the need for many heuristics such as
segment length and window size.

##  III Temporal Span Proposal Network (TSPN)

Figure 3:  Overview of TSPN. TSPN is built upon the object trajectory proposal
head. (a) We first extract video visual features via a pre-trained CNN
backbone. (b) With visual features and RoIs, we then obtain RoI-Aligned
subject, object, and union features. (c) The features are concatenated with
corresponding bounding box coordinates and class distribution, then linearly
transformed and fused via a Hadamard product (denoted as $\odot$ in the
figure), resulting in a $H$. The relationness between an object pair is first
calculated by feeding $H$ into a Fully-Connected (FC) layer. After then, a set
of pairs with high relationness scores (colored in red in the figure) is only
considered in the subsequent process. (d) Finally, joint features are
concatenated and fed into another FC layer to predict relation labels along
with temporal spans.

In this section, we present our novel model Temporal Span Proposal Network
(TSPN) for VidVRD. TSPN is constructed on top of the object trajectory
proposal module and shares the video-level convolutional features, enabling
TSPN to make cost-efficient proposals. An overview of TSPN is shown in Fig. 3,
and Table I summarizes the notation about it.

Step | Notation | Description
---|---|---
Object Trajectory Proposal | $\mathcal{V}$ | input video
$f$ | number of frames in input video
$\mathcal{O}$ | object trajectories
$n$ | number of object trajectories
Object-level Feature Extraction | $V$ | visual features
${A}_{s},{A}_{o},{A}_{u}$ | RoI-Aligned features
${B}_{s},{B}_{o},{B}_{u}$ | bounding box coordinates
${C}_{s},{C}_{o},{C}_{u}$ | classification probability distributions
Relationness Scoring | ${J}_{s},{J}_{o},{J}_{u}$ | joint features
$\mathcal{S}$ | relationness scores
$p$ | number of pair of interests
Relation & Temporal Span Prediction | $\mathcal{R}$ | relation categories
$m$ | number of relation categories
$\mathcal{T}$ | temporal spans
$k$ | number of temporal sectors
$l$ | coverage per sector
$Z$ |  outer product of $\mathcal{R}$ and $\mathcal{T}$
TABLE I: Summary of notations.

###  III-A Problem Formulation

We design a video relation detector with the notion of object trajectories,
relationness, temporal span, and relation labels. Formally, let $\mathcal{V}$
denote an input video, $\mathcal{O}$ be a set of object trajectories and
$\mathcal{S}$, $\mathcal{T}$ and $\mathcal{R}$ denote the relationness score,
temporal span, and relation, respectively. The goal is to build a model for
$P({\theta}_{\text{VidVRD}}=(\mathcal{O},\mathcal{S},\mathcal{T},\mathcal{R})|\mathcal{V})$.
The VidVRD can be factorized into three processes:

$\displaystyle\begin{split}P(&{\theta}_{\text{VidVRD}}|\mathcal{V})=\\\
&\underbrace{P(\mathcal{O}|\mathcal{V})}_{\begin{subarray}{c}\text{Object
Trajectory}\\\
\text{Proposal}\end{subarray}}\underbrace{P(\mathcal{S}|\mathcal{O},\mathcal{V})}_{\begin{subarray}{c}\text{Relationness}\\\
\text{Scoring}\end{subarray}}\underbrace{P(\mathcal{T},\mathcal{R}|\mathcal{S},\mathcal{O},\mathcal{V})}_{\begin{subarray}{c}\text{Relation
\&}\\\ \text{Temporal Span Prediction}\end{subarray}}\end{split}$ (1)

The most fundamental yet essential part of relation reasoning is the reliable
object detection, since relationships are established via interactions between
two objects. Thus, we first detect categories and locations of objects
appearing within the video. The rest of the processes are modeled by the TSPN.
Specifically, TSPN can be break down into 1) relationness scoring and 2)
relation & temporal span prediction subprocesses. Given a set of object
trajectory pairs, TSPN first calculates the probability that each pair will
have a relation (i.e., relationness), and sample pairs of object trajectories
with high relationness (i.e., pair-of-interest). TSPN finally predicts when
relations start and end (i.e., temporal span) for all relation categories
within a pair-of-interest.

###  III-B Object Trajectory Proposal

The purpose of this step is to find trajectories $\mathcal{O}$ for all $n$
objects present in the video $\mathcal{V}$ with $f$ frames. We adopt the off-
the-shelf object detector [21] and tracker [23] to obtain the object
trajectory proposals. As all the subsequent processes heavily rely on the
object trajectory proposals, it is essential to assure high-quality object
trajectories. Thus, we pre-trained object detector. Since the object
trajectories have variations in scale, aspect ratio, position, and temporal
span, we use Region-of-Interest (RoI) Align [35] to generate fixed 1D
representations, easing the subsequent relation computations.

###  III-C Object-level Feature Extraction

Given a video $\mathcal{V}$, we first extract a set of visual features $V$ via
a pre-trained CNN backbone [36]. With $V$ and a pair of object trajectories
$(\mathcal{O}_{s},\mathcal{O}_{o})$ – subscript $s$ and $o$ indicate the
semantic identity (subject or object) of object trajectories – we obtain a set
of RoI-Aligned features $A_{s}$, $A_{o}$, $A_{u}$ by applying RoI-Align [35]
operation, where $s,o,u$ denote subject, object, and union RoIs of two object
trajectories, respectively. Here, union RoI is a rectangular bound that
tightly encompasses the subject RoI and object RoI (see Fig. 3 for better
understanding). The pre-trained object detection head allows us to obtain
bounding box coordinates $B_{s}$, $B_{o}$, $B_{u}$. The bounding box
coordinates are defined by two points on the lower left and upper right, i.e.,
(${x}_{min}$, ${y}_{min}$, ${x}_{max}$, ${y}_{max}$). Finally, object class
distribution $C_{s}$, $C_{o}$, $C_{u}$ can be predicted using RoI-Aligned
features, where $C_{u}$ is defined as the sum of $C_{s}$ and $C_{o}$.

###  III-D Relationness Scoring

The computational cost to learn relation distributions for every pair of
objects is prohibitively expensive. Only a handful of objects have meaningful
relations due to the sparse nature of real-world interactions. In order to
model such regularities, we start by scoring relationness between object
trajectory proposals. More formally, from $n$ object trajectory proposals
$\mathcal{O}$, the possible pairs are in ${n}^{2}\choose 2$. Our goal here is
to sample $p$ $\ll{n}^{2}$ pairs.

To this end, we filter out less correlated pairs based on the relationness
score. We first jointly model visual, geometric and semantic relationness by
concatenating RoI-Aligned appearance features $A_{i}$ along with corresponding
bounding box coordinates $B_{i}$ and classification probability distributions
$C_{i}$, resulting in joint features $J_{i}$.

$\displaystyle{J}_{i}={A}_{i}\parallel{B}_{i}\parallel{C}_{i},\quad\text{where,\
\ }i\in\\{\textit{s,o,u}\\}$ (2)

where $\parallel$ is a concatenation operation.

Then, relationness scores $S$ is calculated as follows:

$\displaystyle
H=\textbf{W}_{s}^{\textrm{T}}{J}_{s}\odot\textbf{W}_{o}^{\textrm{T}}{J}_{o}\odot\textbf{W}_{u}^{\textrm{T}}{J}_{u}+\textbf{B}_{h}$
(3) $\displaystyle
S=\sigma\left(\textbf{W}_{s}^{\textrm{T}}H+\textbf{B}_{s}\right)$ (4)

where W and B respectively denotes weight matrix and bias matrix, $\odot$
indicates Hadamard product, and $\sigma$ is a non-linear activation, which
maps any real values into a finite interval (e.g., sigmoid).

After iterating the process over every object trajectory pairs existing in the
video, we sort the output relationness scores in descending order and maintain
only the top-$p$ pair-of-interests per video. Only these are considered in
subsequent processes.

###  III-E Relation & Temporal Span Prediction

Due to the video’s dynamic characteristic, numerous relations can exist within
multiple time span between a single pair of object trajectories. This means
that different relation instances can exist in a different time span as well
as in the same time span. For example, think of racehorses. They first stand
side by side, move ahead, and compete during the race. Even to simplify the
situation, it would be horse1 – stand next to – horse2 in the early timespan,
and horse1 – chase – horse2 in the late timespan. Therefore, we need to
simultaneously predict what relations exist and the range of time they occur,
given the object trajectory pair.

There is an open choice in predicting the temporal span of relations. RPN [21]
suggests using anchors in different scales with a sliding window scheme and
learn to reduce the localization error for region proposal. However, when it
comes to temporal span proposal, it becomes unnecessary since relation
detection does not require fine-grained temporal duration to match the ground-
truth. We thus are not using the anchor or sliding-window scheme in this work.
The brute-force approach to find temporal span is to learn a probability
distribution over the ${m}\times{f}$ lattice space – the number of relation
categories is $m$, and the total number of frames is $f$ – which is highly
costly. To conform with the parsimonious property, we simplify the process by
quantizing the video-level temporal span into several sectors; we then predict
the likelihood of each relation category’s presence within the sectors. This
strategy is distinguished from the segment-based approaches in terms of the
feature level because it is not possible to encode the video-level temporal
contexts by extracting features from segments. In contrast, we use the video-
level features and predict relations with just a single glance at a video. In
other words, there is no need to chunk into overlapping short segments or
repeat computation over the same part. That is, TSPN can directly specify
relationship ranges and categories across the whole video without redundancy.

Let the time span over which two object trajectories intersect (i.e.,
${O}_{s}\cap{O}_{o}$) as $\mathcal{T}$. After $\mathcal{T}$ breaks down into
$k$ temporal sectors, one sector is responsible for the range $l$ of the
entire time span. In other words, each temporal sector per pair-of-interest
covers a length of $l$, which is calculated as:

$\displaystyle
l=\frac{1}{k}\left(len\left(\mathcal{O}_{s}\cap\mathcal{O}_{o}\right)\right)$
(5)

where $\textit{len}(\cdot)$ stands for the length of a given trajectory.

We can now discretely predict the temporal sectors in which the relations
exist instead of predicting the temporal span directly in continuous video
space. We concatenate joint features ${J}_{s},{J}_{o},{J}_{u}$ and feed into a
fully-connected layer followed by a non-linearity.

$\displaystyle
Z=\sigma\left(\textbf{W}_{z}^{\textrm{T}}\left({J}_{s}\parallel{J}_{o}\parallel{J}_{u}\right)+\textbf{B}_{z}\right)$
(6)

where the output matrix $Z\in\mathbb{R}^{m\times k}$ represents the outer
product (denoted as $\otimes$ the equation below) of the probability
distribution of relation categories $\mathcal{R}\in\mathbb{R}^{m}$ and that of
temporal span $\mathcal{T}\in\mathbb{R}^{k}$, enabling multi-label
classification within multiple time spans.

$\displaystyle Z=\mathcal{R}\otimes\mathcal{T}$ (7)

###  III-F Loss Function

The whole network can be trained in an end-to-end manner. Total loss is sum of
relationness ($\mathcal{L}_{R}$) and temporal span ($\mathcal{L}_{T}$) loss.
Both are binary cross entropy loss.

$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{R}+\mathcal{L}_{T}$ (8)

$\mathcal{L}_{R}$ measures the loss between $p$ pair of ground truth binary
values $r^{*}$ (pair-of-interest as 1, otherwise 0) and the predicted
relationness ($r$ in Eq. (4)).

$\displaystyle\mathcal{L}_{R}=-\frac{1}{p}\sum_{i=1}^{p}r^{*}\cdot\log
r+(1-r^{*})\cdot\log(1-r)$ (9)

$\mathcal{L}_{T}$ measures the loss between $m\times k$ pair of ground truth
temporal spans $t^{*}$ and predicted temporal spans $t$.

$\displaystyle\mathcal{L}_{T}=-\frac{1}{m\times k}\sum_{i}^{m\times
k}t^{*}\cdot\log t+(1-t^{*})\cdot\log(1-t)$ (10)

We construct a ground truth temporal span into a set of temporal sectors with
values of $0$ or $1$, each of which is set to $1$ if the relation lasts more
than half within that temporal sector, otherwise set to $0$.

|  | VRDet | VRTag
---|---|---|---
Approach | Model | R@50 | R@100 | mAP | P@1 | P@5 | P@10
Segment | VidVRD [14] | 5.54 | 6.37 | 8.58 | 43.00 | 28.90 | 20.80
MHRA [15] | 6.82 | 7.39 | 13.27 | 41.00 | 28.70 | 20.95
GSTEG [16] | 7.05 | 8.67 | 9.52 | 51.50 | 39.50 | 28.23
VRD-GCN [17] | 8.07 | 9.33 | 16.26 | 57.50 | 41.00 | 28.50
MHA [19] | 9.53 | 10.38 | 19.03 | 57.50 | 41.40 | 29.45
Window | PPN-STGCN [20] | 11.21 | 13.69 | 18.38 | 60.00 | 43.10 | 32.24
| TSPN (Ours) | 11.56 | 14.13 | 18.90 | 60.50 | 43.80 | 33.73
TABLE II: Comparison with recent approaches on ImageNet-VidVRD [14] dataset. |  | VRDet | VRTag
---|---|---|---
Approach | Model | R@50 | R@100 | mAP | P@1 | P@5 | P@10
Segment |  MMFF [18] | 6.89 | 8.83 | 6.56 | 51.20 | 40.73 | -
MHA [19] | 6.35 | 8.05 | 6.59 | 50.72 | 41.56 | 32.53
Window |  PPN-STGCN [20] | 8.21 | 9.90 | 6.85 | 48.92 | 36.78 | -
| TSPN (Ours) | 9.33 | 10.71 | 7.61 | 53.14 | 42.22 | 34.94
TABLE III: Comparison with recent approaches on VidOR [24] dataset.

##  IV Experiments

In this section, we conduct comprehensive studies to validate the
effectiveness and efficiency of the proposed TSPN. Here, we report results on
two challenging dataset: ImageNet-VidVRD [14] and VidOR [24]. To understand
the behavior of TSPN, we provide extensive quantitative analyses. Also, we
present qualitative results to examine how our model can benefit from temporal
globality concretely. Finally, we compare the estimated computation time of
TSPN with two representative VidVRD methods (segment-based and window-based).

###  IV-A Implementation Details

We first generate bounding boxes at each frame using Faster R-CNN [21]
equipped with ResNet-101 [36] backbone. The detector is trained on train/val
images of $35$ object categories – which are used in the ImageNet-VidVRD [14]
dataset – of MS-COCO [37] and ILSVRC2016-DET [38] datasets. To be consistent
with [20], we perform Non-Maximum Suppression (NMS) in the detection phase
with IoU threshold $=0.5$ to reduce the redundant bounding boxes since NMS in
the tracking phase might wrongly remove the overlapping trajectories with
different classes since it is class-agnostic. As an object tracker, we adopt
deep SORT [23], which integrates visual appearance information to gain
robustness against identity switching while associating the objects with the
same identity in neighboring frames. We set the number of pair-of-interests as
$64$ (i.e., $p=64$). For the Eq. (5), the number of sectors is set to $16$
(i.e., $k=16$). We keep top-$100$ relation triplet predictions per video.

###  IV-B Dataset Configuration

In both datasets, objects and relation triplets (with temporal span) are
annotated in the form of $\\{$trajectory id, category, bounding box
coordinates$\\}$ and $\\{$(subject id – predicate category – object id),
(start frame – end frame)$\\}$, respectively. We exclude several unannotated
videos in the experiment.

#### ImageNet-VidVRD

ImageNet-VidVRD [14] is a subset of ILSVRC2016-VID [38] train and validation
set. Videos are selected with the criteria of whether they contain clear
visual relations. It contains $1{,}000$ videos (train:test split is $800:200$)
with the manually labeled object categories, corresponding bounding box
trajectories, and relation triplets. It covers $35$ and $132$ categories for
objects and predicates, respectively. Note that the relations in the training
set are all annotated in the segment-level (e.g., $30$ frames). Thus, we link
them into the video-level relations in order to directly optimize long-term
relations.

#### VidOR

VidOR [24] is a large-scale video dataset which contains $10{,}000$ videos
(train:val:test split is $7{,}000:835:2{,}165$) from YFCC100M [39] collection.
The average video length in train/val set is ${\sim}36$ seconds, totaling
${\sim}84$ hours. It contains $80$ categories of objects with trajectories to
indicate their spatio-temporal location in the videos, $50$ categories of
relation predicates, and carefully annotated the relation triplets. This
results in around $50{,}000$ objects and $380{,}000$ relation instances
annotated. Only the training and validation set were used for the experiment
because the test set is not yet publicly available.

###  IV-C Evaluation Protocol

We follow the standard evaluation task settings and metrics [14] listed below:

#### Tasks

In Visual Relation Detection (VRDet) task, given a video, the objective is to
jointly predict object trajectories, categories, and existing relations with
its duration. The object trajectories are considered matched only if
vIoU222vIoU denotes the volumetric intersection over the union of two
trajectories. with the ground truth trajectory is greater than half.

In Visual Relation Tagging (VRTag) task, given video with a set of ground
truth object trajectories, the objective is to find all object categories and
existing relations. In other words, it needs to detect the relation triplet
$<subject-predicate-object>$ correctly.

In both tasks, detected visual relation instances are treated as correct only
if the trajectories of subject and object that form the relation both have
sufficiently high vIoU (i.e., vIoU $>$ 0.5) with the ground truth.

#### Evaluation metrics

For Visual Relation Detection (VRDet) task, we adopt mean Average Precision
($mAP$) and $Recall@K$ to evaluate the detection performance. Specifically, we
use video-wise $Recall@50$ and $Recall@100$, which measures the fraction of
the top-$K$ predictions among the ground-truth triplets. Following [14], we
also report the $mAP$ metric to evaluate the overall precision performance at
different recall values.

For Visual Relation Tagging (VRTag) task, $Precision@K$ as the evaluation
metric to emphasize the ability to tag accurate visual relations. It measures
the fraction of ground-truth among the top-$K$ triplets. Specifically, we use
video-wise $Precision@1$, $Precision@5$, and $Precision@10$ since the number
of visual relation instances per segment is $9.5$.

###  IV-D Comparison with State-of-the-Art

We compare TSPN with state-of-the-art approaches on the two VidVRD benchmarks:
ImageNet-VidVRD [14] and VidOR [24]. As a comparison, we consider following
methods: VidVRD [14], MHRA [15], GSTEG [16], VRD-GCN [17], MMFF [18], MHA [19]
and PPN-STGCN [20]. VidVRD detects relations in the short-term and greedily
associates them. MHRA generates multiple hypotheses for video relation
instances for more robust long-term relation prediction. GSTEG constructs a
conditional random field on a spatio-temporal graph exploiting the statistical
dependency of relational entities. VRD-GCN passes the message through fully-
connected spatial-temporal graphs and conducts reasoning in the 3D graphs
using Graph Convolution Network (GCN) [40]. MMFF predicts relations by jointly
using spatial-temporal visual feature and language context feature. MHA
maintains multiple possible relation hypotheses during the association process
to handle the inaccuracy of the former steps. PPN-STGCN employs a sliding-
window scheme to predict both short-term and long-term relationships and
utilizes GCN to calculate the compatibility of tracklet proposal pair.
Existing VidVRD approaches can be divided into two families: segment-based
[14, 15, 16, 17, 19] and window-based [20] approaches. Results are presented
in Table II and Table III. We observe that TSPN achieves new state-of-the-art
results on both VidVRD benchmarks, demonstrating its effectiveness among the
competing methods. TSPN leverages the complementariness of multiple
information, and is designed to utilize the spatio-temporal contexts of the
entire video, making it strong not only for the short-term but also for the
long-term relations.

###  IV-E Quantitative Analysis

The experiments are conducted on the ImageNet-VidVRD.

| Ablations | VRDet | VRTag
---|---|---|---
Exp | $\mathcal{R}$ | $\mathcal{T}$ | R@50 | R@100 | mAP | P@1 | P@5 | P@10
1 |  |  | 6.14 | 7.52 | 9.38 | 40.50 | 28.90 | 21.95
2 | ✓ |  | 9.62 | 11.69 | 16.29 | 57.00 | 41.90 | 30.45
3 |  | ✓ | 9.95 | 12.35 | 15.93 | 54.50 | 40.20 | 29.50
4 | ✓ | ✓ | 11.56 | 14.13 | 18.90 | 60.50 | 43.80 | 33.73
TABLE IV:  Model ablations. $\mathcal{R}$ and $\mathcal{T}$ denotes
relationness scoring and temporal span proposal modules, respectively.

#### TSPN Ablations

We consider several ablations on building blocks of TSPN to identify how each
component contributes to the performance and verify its efficacy. Ablation
results are summarized in Table IV. Without both relationness scoring
($\mathcal{R}$) and temporal span proposal ($\mathcal{T}$) (equivalent to Exp
1), we have to optimize ${m}\times{l}$ distribution for ${n}^{2}$ pairs, but
with both $\mathcal{R}$ and $\mathcal{T}$ (equivalent to Exp 4), we only need
to optimize ${m}\times{k}$ distribution for $p$ pairs, where $p\ll{n}^{2}$ and
${k}\ll{l}$. In this experiment, we can see that the $\mathcal{R}$ and
$\mathcal{T}$ are complementary to each other in constructing a strong VidVRD
model.

|  | VRDet | VRTag
---|---|---|---
Exp |  No. of PoIs ($p$) | R@50 | R@100 | mAP | P@1 | P@5 | P@10
1 | 16 | 9.85 | 11.84 | 18.03 | 62.00 | 44.60 | 34.96
2 | 32 | 10.32 | 12.93 | 18.41 | 61.50 | 44.10 | 34.04
3 | 64 | 11.56 | 14.13 | 18.90 | 60.50 | 43.80 | 33.73
4 | 128 | 10.92 | 14.29 | 18.29 | 57.50 | 43.10 | 32.68
TABLE V: Optimal number of Pair-of-Interests (PoIs).

#### Optimal Size of Relation Search Space

We further explore the optimal number of pair-of-interest ($p$) per video.
Considering that VidVRD contains $4{,}835$ video-level relation instances in
200 test set (${\sim}24$ instances per video on average), we set the number of
pair-of-interest as $32$ by default. We examine several variations ($p=16$ or
$64$ or $128$) in Table V. Although 16 shows the highest precision since it
has a low false-positive rate, $64$ holds the best overall results. The
results indicate that balancing positives and negatives labels is crucial
since the TSPN learns from penalizing the false positives. The number of PoIs
should be appropriately set because the resulting sampled object pairs affect
the subsequent processing. If it is too large (e.g., $128$), the probability
of the noisy object pairs being included in the PoIs becomes higher and result
in degraded performance. We empirically found that $64$ works well, and use
this number for the rest of the experiments.

|  | VRDet | VRTag
---|---|---|---
Exp |  No. of sectors (${k}$) | R@50 | R@100 | mAP | P@1 | P@5 | P@10
1 | 4 | 5.19 | 6.25 | 9.79 | 33.50 | 23.10 | 15.44
2 | 8 | 10.02 | 11.49 | 14.74 | 49.00 | 36.50 | 24.04
3 | 16 | 11.56 | 14.13 | 18.90 | 60.50 | 43.80 | 33.73
4 | 32 | 11.32 | 14.50 | 18.69 | 59.50 | 42.60 | 33.14
TABLE VI: Adequate quantization of temporal span.

#### Trade-off in Temporal Span Quantization

We investigate the effect of quantizing the temporal span proposals into
sectors ($k$ in Eq. (5)). Since the length of the videos varies, each video
has a different quantization impact. Sparse quantization narrows the search
space (i.e., easy to optimize) but lowers the matching rate at the same time.
On the other hand, dense quantization increases the matching rate, but it also
widens the search space (i.e., expensive to optimize). We conduct an
experiment in Table VI to assess the optimal degree of quantization. We can
observe the best results when the number of sectors is set to $16$.

Figure 4:  Qualitative examples of visual relation detection results. For
comparison, we contrast the predicted relation triplets (i.e., subject-
relation-object) of VidVRD with those of TSPN for each given video. The same
color means the same object instance. The arrows represents the time axis,
providing an approximation of the temporal span of the predicted relation
triplets. We highlight relations that TSPN correctly predicted, while VidVRD
does not. The predicted relations are considered correct only if the pair of
object trajectories have sufficiently high vIoU (i.e., vIoU $>$ 0.5) with
ground truth trajectories, and only the correct relations of the top-20
predictions are shown in the figure.

###  IV-F Qualitative Analysis

To better see how TSPN understands the dynamics of spatio-temrpoal
interactions, we provide qualitative examples in Fig. 4. Here, we compare TSPN
with VidVRD [14], which first proposed a popular segment-based approach. The
results show that the VidVRD often fails to capture long-term relations since
it relies on temporarily local features. On the other hand, TSPN successfully
identifies them. For example, the relation person-past-bicycle
($105{-}150$)333The numbers in parentheses mean ground truth “begin frame –
end frame”. can only be detected by understanding the long-term relationship
since the model should capture the entire process of person being behind
bicycle and catching up with bicycle as it moves forward. It goes the same for
the relation red panda-walk past-red panda ($0{-}75$). TSPN also detects more
active or sophisticated relations such as elephant-kick-ball ($0{-}60$) and
skateboard-move beneath-dog ($0{-}360$), while VidVRD simply detects trivial
relations (e.g., elephant-taller-person, person-taller-dog), which can also be
predicted in a still image. This also reveals the limitation of segment-based
methods. That is, it cannot leverage the video-level context. The reason is
that VidVRD is unable to directly utilize the temporarily global feature since
it predicts relations in the segment-level and aggregates them into the video-
level relation. In contrast, TSPN can directly predict long-term relations
from the proposed temporal span.

Method | Approximated Time | Upper Bound | Typical Setting ($s=\frac{l}{2}$) | Big-$O$
---|---|---|---|---
Segment-based | $\left\lceil\frac{L-l+s}{s}\right\rceil$ | $\frac{L-l+2s}{s}$ | $\frac{2L}{l}$ | $O(L)$
Window-based | $\sum_{k=1}^{\frac{L}{l}}\left\lceil\frac{L-kl+s}{s}\right\rceil$ | $\left(\frac{L^{2}}{s}-\frac{L^{2}}{2l^{2}s}-\frac{L}{2ls}+2L\right)^{2}$ | $\left(\frac{2L^{2}}{l}-\frac{L^{2}}{l^{3}}-\frac{L}{l^{2}}+2L\right)^{2}$ | $O(L^{4})$
TSPN-based (ours) | $\frac{L}{l}$ | $\frac{L}{l}$ | $\frac{L}{l}$ | $O(L)$
TABLE VII:  Comparison of estimated computation time with existing approaches.
$L$ denotes the intersection of subject trajectory and object trajectory
(i.e., pair-of-interest), $l$ denotes segment length or minimum window size or
coverage of each temporal sector, and $s$ denotes the stride of segment or
window, where $s<l\ll L$.

###  IV-G Comparison of Estimated Computation Time

To observe the efficiency of TSPN, we compare TSPN with two representative
VidVRD approaches (segment-based and window-based) in terms of estimated
computation time. The segment-based approach predicts relations by dividing
the video into several segments and connecting to long relations if adjacent
relations are the same. The window-based approach first obtains multiple
tracklets by sliding multi-size windows over the trajectories, finds tracklet
pairs with high similarity via a correlation embedding module, and finally
predicts the relations for the valid pairs. TSPN first finds out what object
trajectory pairs are highly likely to have relations by calculating
relationness score, then TSPN simultaneously predicts relation categories and
their temporal span with sampled pairs.

Note that direct comparisons are difficult as the greedy association in the
segment-based approach is an offline algorithm, and the source codes of the
window-based approach are not publicly available. We thus provide the
asymptotic computational complexity of each method for relation prediction.
Here, we do not consider additional features or modules (e.g., greedy
association, pair correlation embedding module, and relationness scoring)
other than the main components for simplicity. Also, we consider that the
segment length of the segment-based approach, a minimum window size of the
window-based approach, and coverage per temporal sector in TSPN are all on the
same scale. Now the time complexity can be represented by the number of object
pairs to consider, assuming that predicting relations for each pair of objects
takes the same amount of time.

Let the intersection of subject and object be $L$, segment or minimum-window
or sector length be $l$, and stride be $s$. Intuitively, the computation time
of segment-based approach is proportional to the number of segments. The
number of segments can be calculated as:

$\displaystyle\mathbb{N}_{s}=\left\lceil\frac{L-l+s}{s}\right\rceil$ (11)

Similarly, the window-based method depends on the number of windows. The
window-based method exhaustively uses window of all sizes, from smallest
(i.e., $l$) to largest (i.e., $L$), to capture both the short-term and long-
term object relations. The number of all windows can be calculated as:

$\displaystyle\mathbb{N}_{w}=\sum_{k=1}^{\frac{L}{l}}\left\lceil\frac{L-kl+s}{s}\right\rceil$
(12)

Since relation detection requires two windows each responsible for subject and
object, the overall computation shows a quadratic growth rate with respect to
the number of windows.

In contrast, TSPN directly predicts the relationship categories and their
temporal span using the entire video context only once. Thus, it shows a
linear growth rate with respect to the number of temporal sectors. Since there
is no overlap between sectors, the number of sectors can be simplified as:

$\displaystyle\mathbb{N}_{t}=\frac{L}{l}$ (13)

Under the condition of $s<l$, we can obtain the following conclusions444 Here
is a simple derivation:
1) $\mathbb{N}_{w}$ can be rewritten as
$\mathbb{N}_{s}+\sum_{k=2}^{\frac{L}{l}}\left\lceil\frac{L-kl+s}{s}\right\rceil$;
therefore, $\mathbb{N}_{s}<\mathbb{N}_{w}$.
2) The lower bound of $\mathbb{N}_{s}$ is $\frac{L-l}{s}+1$; since $s<l$, it
satisfies: $\frac{L-l}{s}+1>\frac{L-l}{l}+1=\frac{L}{l}=\mathbb{N}_{t}$;
therefore, $\mathbb{N}_{t}<\frac{L-l}{s}+1\leq\mathbb{N}_{s}$. :

$\displaystyle\mathbb{N}_{t}<\mathbb{N}_{w}<\mathbb{N}_{s}$ (14)

which means that TSPN is always more efficient than the segment and window-
based approaches. In practice, we follow the typical settings of
stride/segment length ratio ($s=15$, $l=30$; thus, $s=l/2$), so the estimated
computation time of segment and TSPN-based approaches are proportional to
$\frac{L}{s}$ and $\frac{L}{l}$, respectively. Therefore, TSPN is
approximately $2{\times}$ more efficient than the segment-based approach. The
overall estimated computation times are summarized in Table VII.

##  V Conclusion

This work introduces a novel Temporal Span Proposal Network (TSPN) for Video
Visual Relation Detection (VidVRD). TSPN effectively reduces the relational
search space by learning which object pairs should be considered based on the
relationness score between object trajectory pairs. TSPN simultaneously
predicts the temporal span and categories of the entire relations with a
single global video feature, which is not only efficient but also effective in
predicting both short and long-term relations. We validate the TSPN through
comprehensive experiments. TSPN establishes a strong baseline for VidVRD by
leveraging the complementariness of the two key modules. In particular, TSPN
achieves new state-of-the-art performance on two VidVRD benchmarks (ImageNet-
VidVRD and VidOR) without many external algorithms or heuristics (e.g., greedy
association, window sliding) commonly seen in previous approaches while
theoretically faster than conventional approaches – $2{\times}$ faster than
segment-based approach.

## References

  * [1] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image Retrieval Using Scene Graphs,” in _CVPR_ , 2015, pp. 3668–3678.
  * [2] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap, “A Simple Neural Network Module for Relational Reasoning,” in _NeurIPS_ , 2017, pp. 4967–4976.
  * [3] P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu, “Interaction Networks for Learning About Objects, Relations and Physics,” _arXiv preprint arXiv:1612.00222_ , 2016.
  * [4] T. Yao, Y. Pan, Y. Li, and T. Mei, “Exploring Visual Relationship for Image Captioning,” in _ECCV_ , 2018, pp. 684–699.
  * [5] X. Yang, K. Tang, H. Zhang, and J. Cai, “Auto-Encoding Scene Graphs for Image Captioning,” in _CVPR_ , 2019, pp. 10 685–10 694.
  * [6] D. Teney, L. Liu, and A. van Den Hengel, “Graph-Structured Representations for Visual Question Answering,” in _CVPR_ , 2017, pp. 1–9.
  * [7] J. Johnson, A. Gupta, and L. Fei-Fei, “Image Generation From Scene Graphs,” in _CVPR_ , 2018, pp. 1219–1228.
  * [8] O. Ashual and L. Wolf, “Specifying Object Attributes and Relations in Interactive Scene Generation,” in _ICCV_ , 2019, pp. 4561–4569.
  * [9] C.-Y. Ma, A. Kadav, I. Melvin, Z. Kira, G. AlRegib, and H. Peter Graf, “Attend and Interact: Higher-Order Object Interactions for Video Understanding,” in _CVPR_ , 2018, pp. 6790–6800.
  * [10] C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei, “Visual Relationship Detection With Language Priors,” in _ECCV_. Springer, 2016, pp. 852–869.
  * [11] B. Dai, Y. Zhang, and D. Lin, “Detecting Visual Relationships With Deep Relational Networks,” in _CVPR_ , 2017, pp. 3076–3086.
  * [12] H. Zhang, Z. Kyaw, S.-F. Chang, and T.-S. Chua, “Visual Translation Embedding Network for Visual Relation Detection,” in _CVPR_ , 2017, pp. 5532–5540.
  * [13] S. Jae Hwang, S. N. Ravi, Z. Tao, H. J. Kim, M. D. Collins, and V. Singh, “Tensorize, Factorize and Regularize: Robust Visual Relationship Learning,” in _CVPR_ , 2018, pp. 1014–1023.
  * [14] X. Shang, T. Ren, J. Guo, H. Zhang, and T.-S. Chua, “Video Visual Relation Detection,” in _ACMMM_ , 2017, pp. 1300–1308.
  * [15] D. Di, X. Shang, W. Zhang, X. Yang, and T.-S. Chua, “Multiple Hypothesis Video Relation Detection,” in _BigMM_. IEEE, 2019, pp. 287–291.
  * [16] Y.-H. H. Tsai, S. Divvala, L.-P. Morency, R. Salakhutdinov, and A. Farhadi, “Video Relationship Reasoning Using Gated Spatio-Temporal Energy Graph,” in _CVPR_ , 2019, pp. 10 424–10 433.
  * [17] X. Qian, Y. Zhuang, Y. Li, S. Xiao, S. Pu, and J. Xiao, “Video Relation Detection With Spatio-Temporal Graph,” in _ACMMM_ , 2019, pp. 84–93.
  * [18] X. Sun, T. Ren, Y. Zi, and G. Wu, “Video Visual Relation Detection via Multi-Modal Feature Fusion,” in _ACMMM_ , 2019, pp. 2657–2661.
  * [19] Z. Su, X. Shang, J. Chen, Y.-G. Jiang, Z. Qiu, and T.-S. Chua, “Video Relation Detection via Multiple Hypothesis Association,” in _ACMMM_ , 2020, pp. 3127–3135.
  * [20] C. Liu, Y. Jin, K. Xu, G. Gong, and Y. Mu, “Beyond Short-Term Snippet: Video Relation Detection with Spatio-Temporal Global Context,” in _CVPR_ , 2020, pp. 10 840–10 849.
  * [21] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection With Region Proposal Networks,” in _NeurIPS_ , 2015, pp. 91–99.
  * [22] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple Online and Realtime Tracking,” in _ICIP_. IEEE, 2016, pp. 3464–3468.
  * [23] N. Wojke, A. Bewley, and D. Paulus, “Simple Online and Realtime Tracking With a Deep Association Metric,” in _ICIP_. IEEE, 2017, pp. 3645–3649.
  * [24] X. Shang, D. Di, J. Xiao, Y. Cao, X. Yang, and T.-S. Chua, “Annotating Objects and Relations in User-Generated Videos,” in _ICMR_ , 2019, pp. 279–287.
  * [25] J. Zhang, M. Elhoseiny, S. Cohen, W. Chang, and A. Elgammal, “Relationship Proposal Networks,” in _CVPR_ , 2017, pp. 5678–5686.
  * [26] J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh, “Graph R-CNN for Scene Graph Generation,” in _ECCV_ , 2018, pp. 670–685.
  * [27] Y.-W. Chao, Y. Liu, X. Liu, H. Zeng, and J. Deng, “Learning to Detect Human-Object Interactions,” in _WACV_. IEEE, 2018, pp. 381–389.
  * [28] G. Gkioxari, R. Girshick, P. Dollár, and K. He, “Detecting and Recognizing Human-Object Interactions,” in _CVPR_ , 2018, pp. 8359–8367.
  * [29] Y.-L. Li, S. Zhou, X. Huang, L. Xu, Z. Ma, H.-S. Fang, Y. Wang, and C. Lu, “Transferable Interactiveness Knowledge for Human-Object Interaction Detection,” in _CVPR_ , 2019, pp. 3585–3594.
  * [30] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei, “Scene Graph Generation by Iterative Message Passing,” in _CVPR_ , 2017, pp. 5410–5419.
  * [31] R. Zellers, M. Yatskar, S. Thomson, and Y. Choi, “Neural Motifs: Scene Graph Parsing With Global Context,” in _CVPR_ , 2018, pp. 5831–5840.
  * [32] K. Tang, H. Zhang, B. Wu, W. Luo, and W. Liu, “Learning to Compose Dynamic Tree Structures for Visual Contexts,” in _CVPR_ , 2019, pp. 6619–6628.
  * [33] K. Tang, Y. Niu, J. Huang, J. Shi, and H. Zhang, “Unbiased Scene Graph Generation From Biased Training,” in _CVPR_ , 2020, pp. 3716–3725.
  * [34] S. Woo, J. Noh, and K. Kim, “Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions,” _arXiv preprint arXiv:2106.08543_ , 2021.
  * [35] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in _CVPR_ , 2017, pp. 2961–2969.
  * [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [37] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _ECCV_. Springer, 2014, pp. 740–755.
  * [38] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _IJCV_ , vol. 115, no. 3, pp. 211–252, 2015\.
  * [39] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, “YFCC100M: The New Data in Multimedia Research,” _Communications of the ACM_ , vol. 59, no. 2, pp. 64–73, 2016.
  * [40] T. N. Kipf and M. Welling, “Semi-Supervised Classification With Graph Convolutional Networks,” _ICLR_ , 2017.

|  Sangmin Woo is currently working toward the M.S. degree in Electrical
Engineering and Computer Science at Gwangju Institute of Science and
Technology (GIST), Gwangju, Korea. He received a B.S. degree in Electrical
Engineering from Kyungpook National University, Daegu, Korea, in 2019. His
research interests include computer vision, visual understanding, and deep
learning. He is a student member of the IEEE.
---|---
|  Junhyug Noh is a postdoctoral researcher at Lawrence Livermore National
Laboratory (LLNL). He received the B.S. in Computer Science and Engineering &
Statistics from Seoul National University in 2013, and the M.S. and Ph.D. in
Computer Science Engineering from Seoul National University in 2015 and 2020,
respectively. His research has focused on artificial intelligence, machine
learning, and computer vision with a particular interest in object detection
and its related high-level vision tasks such as semantic/instance
segmentation, scene understanding, and image captioning.
---|---
|  Kangil Kim received the B.S. degree in computer science from the Korea
Advanced Institute of Science and Technology, Daejeon, South Korea, in 2006,
and the Ph.D. degree from Seoul National University, Seoul, South Korea, in
2012. He was a Senior Researcher with the Natural Language Processing Group,
Electronics and Telecommunications Research Institute, Seoul, until 2016, and
an Assistant Professor with the Computer Science and Engineering Department,
Konkuk University, until 2019. He is currently an Assistant Professor with the
Electronics Engineering and Computer Science Department and Artificial
Intelligence Graduate School, Gwangju Institute of Science and Technology,
Gwangju, South Korea. His research interests include artificial intelligence,
evolutionary computation, machine learning, and natural language processing.
---|---


# Feedback-based Digital Higher-order Terminal Sliding Mode
for 6-DOF Industrial Manipulators

Zhian Kuang1,2, Xiang Zhang2, Liting Sun2, Huijun Gao1 and Masayoshi Tomizuka2
1 Zhian Kuang and Huijun Gao are with the Research Institute of Intelligent
Control and Systems, Harbin Institute of Technology, 150001, Harbin, P.R.
China<EMAIL_ADDRESS><EMAIL_ADDRESS>Xiang Zhang, Liting Sun, and
Masayoshi Tomizuka are with the Department of Mechanical Engineering,
University of California, Berkeley, CA 94720, USA.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Zhian Kuang is also with the Department of Mechanical Engineering, University
of California, Berkeley, CA 94720, USA.

###### Abstract

The precise motion control of a multi-degree of freedom (DOF) robot
manipulator is always challenging due to its nonlinear dynamics, disturbances,
and uncertainties. Because most manipulators are controlled by digital
signals, a novel higher-order sliding mode controller in the discrete-time
form with time delay estimation is proposed in this paper. The dynamic model
of the manipulator used in the design allows proper handling of
nonlinearities, uncertainties and disturbances involved in the problem.
Specifically, parametric uncertainties and disturbances are handled by the
time delay estimation and the nonlinearity of the manipulator is addressed by
the feedback structure of the controller. The combination of terminal sliding
mode surface and higher-order control scheme in the controller guarantees a
fast response with a small chattering amplitude. Moreover, the controller is
designed with a modified sliding mode surface and variable-gain structure, so
that the performance of the controller is further enhanced. We also analyse
the condition to guarantee the stability of the closed-loop system in this
paper. Finally, the simulation and experimental results prove that the
proposed control scheme has a precise performance in a robot manipulator
system.

##  I INTRODUCTION

Manipulators have been extensively used in the industry to manufacture
products. However, with nonlinearities, high couplings, and significant
parametric uncertainties existing in the dynamics, the manipulator’s precise
control is always challenging [1]. Moreover, with more and more industrial
scenarios involving manipulators, different kinds of disturbances and high
accelerations are applied, making it harder to design a precise and robust
controller for a manipulator.

For the past several years, researchers have proposed several kinds of
manipulator controllers, such as feedforward and PID control [2], sliding mode
control [3], adaptive robust control [4]. Among these methods, sliding mode
control is proven to be straightforward to be applied in complex
electromechanical system [5]. More importantly, because of the sliding mode
surface, this control method shows outstanding robustness to disturbances and
uncertainties [6]. Researchers have proposed several kinds of modified SMC to
make up the original drawbacks like the slow convergence on the sliding
surface [7, 8]. For instance, Yong proposed a non-singular terminal sliding
mode control (TSMC) method, which makes the states converge in finite time on
the sliding surface [9].

Despite that TSMC has precise and robust performance in theory, it cannot be
applied directly to the manipulators for two reasons. One reason is that the
digital processor is always used to control the manipulator nowadays, directly
applying the continuous-time controller to the discrete-time environment
results in large chattering amplitude or even instability [10]. To address
this problem, researchers have put forward the discrete-time terminal sliding
mode control (DTSMC) [11] and apply it to motion control systems [12, 13].
Compared with the traditional linear sliding mode surface, the terminal
sliding surface has a faster response benefiting from the finite-time
convergence [14, 15]. Nevertheless, this kind of dynamics also enlarges the
chattering amplitude, which makes the precision of the manipulator decrease. A
proper way to solve this problem is to raise the order of the sliding mode
[16, 17]. For example, Xu proposed a second-order terminal sliding mode
strategy [18]. However, the implementation is on a linear positioning model,
which cannot be applied instantly to the manipulator. Moreover, the order of
the sliding mode controller is fixed, which cannot be changed based on the
performance of the practical application. Raising the order is beneficial to
reduce the chattering, but may also reduce the controller’s robustness. Thus
we need to build a new and flexible structure that can balance the chattering
amplitude and the robustness at the same time.

The other obstacles to improving manipulators’ performance are the
nonlinearity, the significant parametric uncertainties, and disturbances. They
need to be handled appropriately; otherwise, the tracking precision will
deteriorate, especially when the manipulator is in operation with large
acceleration [19]. A popular way is to introduce the feedforward term to
compensate for the nonlinear dynamics and use the controller to suppress the
uncertainties and disturbances [2]. As the feedforward structure is open-loop,
this method is not sufficient enough when there are parametric uncertainties
on the manipulator’s dynamic model. Another method to handle the nonlinearity
is the time-delay estimation (TDE) [20, 21]. By using TDE, the nonlinear
dynamics, the present lumped disturbances, and uncertainties are estimated by
the previous states [1]. When the sampling frequency of the system is high,
the delay time is appropriately chosen, and the disturbances change relatively
slowly, this method has excellent performance in [22] and [23]. However, as
TDE highly rely on the state at a particular time point, extra disturbances
are easy to be introduced. Moreover, when the manipulator operates with fast
speed and massive acceleration, the estimation error will increase due to the
fast change of dynamics. Thus we need a new structure to make the best use of
TDE to achieve better precision.

This paper proposes a novel feedback-based digital higher-order terminal
sliding mode control (DHTSMC) scheme. This controller is designed based on the
manipulator’s nominal dynamics, including the linear and nonlinear parts. The
manipulator’s nonlinear dynamics are directly handled by the equivalent
control part of the controller in a feedback way. As an assistant, TDE is used
to compensate for the uncertainties and disturbances in the system. For the
controller, a new terminal sliding mode surface is designed to improve the
states’ dynamics on the sliding mode surface. We also give out a universal and
flexible form of a higher-order switching controller. A specific order can be
selected to balance the chattering phenomenon and the robustness of the
controller. To address the performance reduction when there are large
accelerations, we mainly use the controller’s variable gain. More importantly,
we implement the proposed controller to a 6-DOF manipulator in the
environments of both simulations and experiments, whose results show that our
proposed method has an advantage over previous ones.

The remainder of this paper is as follows. Section II introduces the model of
manipulators and the proposed controller. In Section III, the stability of the
closed-loop control system is analyzed. Section IV introduces the practical
application of the proposed controller to manipulators and the results of
simulations and experiments.

##  II CONTROLLER DESIGN

The continuous-time Lagrange model of a manipulator with n-DOFs is written as

$\displaystyle\bm{M}(\bm{q})\bm{\ddot{q}}+\bm{C}(\bm{q},\bm{\dot{q}})\bm{\dot{q}}+\bm{G}(\bm{\dot{q}})+\bm{F_{f}}(\bm{q},\bm{\dot{q}})+\bm{d}=\bm{\tau}$
(1)

where $\bm{M}(\bm{q})\in\mathbb{R}^{n\times n}$ is the inertia matrix,
$\bm{C}(\bm{q},\bm{\dot{q}})\in\mathbb{R}^{n\times n}$ denotes the Coriolis
matrix, $\bm{G(\dot{q})}\in\mathbb{R}^{n}$ and
$\bm{F_{f}}(\bm{q},\bm{\dot{q}})\in\mathbb{R}^{n}$ are the gravity vector and
the friction vector respectively, $\bm{\ddot{q}}$, $\bm{\dot{q}}$ and $\bm{q}$
are respectively the angular acceleration, velocity and position of each
joint, $\bm{d}\in\mathbb{R}^{n}$ stands for the disturbances, and
$\bm{\tau}\in\mathbb{R}^{n}$ is the torque vector.

Because the parametric uncertainties extensively exist in practical systems,
we use $\bm{\bar{\bullet}}$ to denote the matrices or vectors calculated from
the nominal values, then (1) is rewritten as

$\displaystyle\bm{\tau}=$
$\displaystyle\bm{\bar{M}}(\bm{q})\bm{\ddot{q}}+\bm{\bar{C}}(\bm{q},\bm{\dot{q}})\bm{\dot{q}}+\bm{\bar{G}}(\bm{\dot{q}})+\bm{\bar{F_{f}}}(\bm{q},\bm{\dot{q}})$
$\displaystyle+\bm{H}(\bm{q},\bm{\dot{q}},\bm{\ddot{q}})$ (2)

with

$\displaystyle\bm{H}(\bm{q},\bm{\dot{q}},\bm{\ddot{q}})=$
$\displaystyle\bm{\tilde{M}}(\bm{q})\bm{\ddot{q}}+\bm{\tilde{C}}(\bm{q},\bm{\dot{q}})\bm{\dot{q}}+\bm{\tilde{G}}(\bm{\dot{q}})$
$\displaystyle+\bm{\tilde{F_{f}}}(\bm{q},\bm{\dot{q}})+\bm{d}$ (3)

where $\tilde{\bullet}=\bullet-\bar{\bullet}$ stands for the uncertain part of
each matrix or vector.

If the manipulator is sampled with a sampling interval $T$, then the discrete-
time model at time $t$ is described as

$\displaystyle\bm{\tau_{k}}=$
$\displaystyle\bm{\bar{M}}(\bm{q_{k}})\bm{\ddot{q_{k}}}+\bm{\bar{C}}(\bm{q_{k}},\bm{\dot{q}_{k}})\bm{\dot{q}_{k}}+\bm{\bar{G}}(\bm{\dot{q}_{k}})+\bm{\bar{F_{f}}}(\bm{q}_{k},\bm{\dot{q}}_{k})$
$\displaystyle+\bm{H}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}})$ (4)

with

$\displaystyle\bm{H}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}})=$
$\displaystyle\bm{\tilde{M}}(\bm{q_{k}})\bm{\ddot{q}_{k}}+\bm{\tilde{C}}(\bm{q_{k}},\bm{\dot{q}_{k}})\bm{\dot{q}_{k}}+\bm{\tilde{G}}(\bm{\dot{q}_{k}})$
$\displaystyle+\bm{\tilde{F_{f}}}(\bm{q_{k}},\bm{\dot{q}_{k}})+\bm{d_{k}}$ (5)

where $t=kT$, $\bm{\dot{q}_{k}}=\frac{\bm{q_{k+1}}-\bm{q_{k}}}{T}$, and
$\bm{\ddot{q}_{k}}=\frac{\bm{\dot{q}_{k+1}}-\bm{\dot{q}_{k}}}{T}$.

We use TDE to estimate
$\bm{H}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}})$, i.e., we calculate
out the uncertain part in the previous sampling point and use this value to
approximate the uncertain part in the present. It is formulated as

$\displaystyle\bm{\hat{H}}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}})=$
$\displaystyle\bm{H}(\bm{q_{k-1}},\bm{\dot{q}_{k-1}},\bm{\ddot{q}_{k-1}})$
$\displaystyle=$
$\displaystyle\bm{\tau_{k-1}}-\bm{\bar{M}}(\bm{q_{k-1}})\bm{\ddot{q}_{k-1}}$
$\displaystyle-\bm{\bar{C}}(\bm{q_{k-1}},\bm{\dot{q}_{k-1}})\bm{\dot{q}_{k-1}}$
$\displaystyle-\bm{\bar{G}}(\bm{\dot{q}_{k-1}})-\bm{\bar{F_{f}}}(\bm{q}_{k-1},\bm{\dot{q}}_{k-1}).$
(6)

The estimation error between
$\bm{H}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}})$ and
$\bm{\hat{H}}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}})$ is presented as
$\bm{\check{H}}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}})=\bm{H}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}})-\bm{H}(\bm{q_{k-1}},\bm{\dot{q}_{k-1}},\bm{\ddot{q}_{k-1}})$.

Define the tracking error of the joints as

$\displaystyle\bm{\check{q}_{k}}=\bm{q_{k}}-\bm{r_{k}}$ (7)

where $\bm{r_{k}}$ is the reference vector of the joints.

To obtain precise, fast and robust performance, we design a novel discrete-
time terminal sliding surface as

$\displaystyle\bm{s_{k}}=\bm{a_{1}}\bm{\check{q}_{k}}+\bm{a_{2}}\textrm{sig}^{\bm{\beta_{k}}}\bm{\check{q}_{k}}+\bm{\dot{\check{q}}_{k}}$
(8)

with
$\textrm{sig}^{\bm{\beta_{k}}}\bm{\check{q}_{k}}=[\textrm{sig}^{\beta_{k,1}}\check{q}_{k,1}~{}\textrm{sig}^{\beta_{k,2}}\check{q}_{k,2}~{}\cdots~{}\textrm{sig}^{\beta_{k,n}}\check{q}_{k,n}]^{\bm{T}}$,
$\beta_{k,i}=\frac{|{\check{q}}_{k,i}|+0.5}{|{\check{q}}_{k,i}|+1},~{}i=1,2,\cdots,n$,
where $\textrm{sig}^{\bullet}*=|*|^{\bullet}\textrm{sgn}(*)$,
$\bm{a_{1}}=\textrm{diag}[a_{1,i}],~{}i=1,2,\cdots,n$,
$\bm{a_{2}}=\textrm{diag}[a_{2,i}],~{}i=1,2,\cdots,n$, $a_{1,i}$ and $a_{2,i}$
are positive constants.

We design the reaching law of r-order variable-gain sliding mode controller as

$\displaystyle s_{k+1,i}=$ $\displaystyle-
b_{0,i}(\ddot{q}_{k,i})Ts_{k,i}^{\eta}-b_{1,i}(\ddot{q}_{k,i})Ts_{k-1,i}^{\eta}-\dots$
$\displaystyle-b_{r,i}(\ddot{q}_{k,i})Ts_{k-r,i}^{\eta}$ (9)

where $0<\eta<1$ is a positive constant,
$b_{j,i}(\ddot{q}_{k,i}),~{}j=1,2,\cdots,r$ is a function of $\ddot{q}_{k,i}$.

Based on the reaching law (II),the discrete-time r-order variable-gain
terminal sliding mode controller with TDE is obtained as

$\displaystyle\bm{\tau_{k}}=$
$\displaystyle\frac{1}{T}\bm{\bar{M}}(\bm{\dot{r}_{k+1}}-a_{1}(\bm{\dot{q}_{k}}T+\bm{q_{k}}-\bm{r_{k+1}})-a_{2}\textrm{sig}^{\bm{\beta_{k}}}(\bm{\dot{q}_{k}}T$
$\displaystyle+\bm{q_{k}}-\bm{r_{k+1}})-\bm{\dot{q}_{k}})+\bm{\bar{C}}(\bm{q_{k}},\bm{\dot{q}_{k}})\bm{\dot{q}_{k}}+\bm{\bar{G}}(\bm{\dot{q}_{k}})$
$\displaystyle+\bm{\bar{F_{f}}}(\bm{q}_{k},\bm{\dot{q}}_{k})+\bm{\hat{H}}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}})$
$\displaystyle+\bm{\bar{M}}((1-\bm{b_{0}}(\bm{\ddot{q}_{k}})T)\bm{s_{k}}-\bm{b_{1}}(\bm{\ddot{q}_{k}})T\bm{s_{k-1}}$
$\displaystyle-\bm{b_{2}}(\bm{\ddot{q}_{k}})T\bm{s_{k-2}}-\dots-\bm{b_{r}}(\bm{\ddot{q}_{k}})T\bm{s_{k-r}}).$
(10)

##  III Stability Analysis

The following theorem states the sufficient condition to guarantee the control
system stable:

###### Theorem 1

For the system (II) under the controller (II), if there are a series positive
constants ${\alpha_{1},~{}\alpha_{2},~{}\cdots,~{}\alpha_{r}}$ satisfying
$1>\alpha_{1}>\alpha_{2}>\cdots>\alpha_{r}>0$ and

$\displaystyle
b_{n,i}(\ddot{q}_{k,i})\leq\frac{1}{T}\sqrt{\frac{\alpha_{n}-\alpha_{n+1}}{r+2}}$
(11)

then $s_{k-m,i}$ will converge to the region

$\displaystyle\Gamma_{i}=\left\\{s_{k-m,i}\big{|}|s_{k-m,i}|<\gamma_{i}\right\\}$
(12)

where

$\displaystyle\gamma_{i}$
$\displaystyle=\frac{(r+2)E_{k,i}^{2}+\sum_{j=0}^{r}(b_{j,i}(\ddot{q}_{k,i})T)^{2}}{\alpha_{m}-\alpha_{m+1}-(r+2)(b_{m,i}(\ddot{q}_{k,i})T)^{2}}$
(13) $\displaystyle|s_{k-m,i}|$
$\displaystyle=\max\left\\{|s_{k-r,i}|,~{}|s_{k-r+1,i}|,~{}\cdots,~{}|s_{k,i}|\right\\}$
(14)

and $E_{k,i}$ is the elements of
$\bm{E_{k}}=\bm{\bar{M}}^{-1}\bm{\check{H}}(\bm{q_{k}},\bm{\dot{q}_{k}},\bm{\ddot{q}_{k}}).$

###### Proof:

Substituting (II) into (II), we obtain the dynamics of $s_{k+1,i}$ as
$s_{k+1,i}=-b_{0,i}(\ddot{q}_{k,i})Ts_{k,i}^{\eta}-b_{1,i}(\ddot{q}_{k,i})Ts_{k-1,i}^{\eta}-\dots-
b_{r,i}(\ddot{q}_{k,i})Ts_{k-r,i}^{\eta}+E_{k,i}.$

Construct a Lyapunov candidate function $U_{k,i}$ as

$\displaystyle
U_{k,i}=s_{k,i}^{2}+\alpha_{1}s_{k-1,i}^{2}+\alpha_{2}s_{k-2,i}^{2}+\dots+\alpha_{r}s_{k-r,i}^{2}.$
(15)

Then the difference between $U_{k+1,i}$ and $U_{k,i}$ is

$\displaystyle\Delta U_{k}=$ $\displaystyle U_{k+1,i}-U_{k,i}$
$\displaystyle=$
$\displaystyle(\alpha_{1}-1)s_{k,i}^{2}+(\alpha_{2}-\alpha_{1})s_{k-1,i}^{2}+\dots$
$\displaystyle+(\alpha_{r}-\alpha_{r-1})s_{k-r+1,i}^{2}+s^{2}_{k+1,i}-\alpha_{r}s_{k-r,i}^{2}.$
(16)

When $s_{i}>1$, we have $s_{i}^{2\eta}<s_{i}^{2}$. When $s_{i}\leq 1$, we have
$s_{i}^{2\eta}\leq 1$. Thus, it holds that

$\displaystyle s_{i}^{2\eta}\leq s_{i}^{2}+1.$ (17)

Based on the fact that $2ab\leq a^{2}+b^{2},~{}a,b\in\mathbb{R}$, we have

$\displaystyle s_{k+1,i}^{2}=$ $\displaystyle
b_{0,i}^{2}(\ddot{q}_{k,i})T^{2}s_{k,i}^{2\eta}+2b_{0,i}(\ddot{q}_{k,i})b_{1,i}(\ddot{q}_{k,i})T^{2}s_{k,i}^{\eta}s_{k-1,i}^{\eta}$
$\displaystyle+(b_{1,i}(\ddot{q}_{k,i})T)^{2}s_{k-1,i}^{2\eta}$
$\displaystyle+2b_{1,i}(\ddot{q}_{k,i})b_{2,i}(\ddot{q}_{k,i})T^{2}s_{k-1,i}^{\eta}s_{k-2,i}^{\eta}$
$\displaystyle+(b_{2,i}T)^{2}s_{k-2,i}^{2\eta}+\dots+(b_{r,i}(\ddot{q}_{k,i})T)^{2}s_{k-r,i}^{2\eta}$
$\displaystyle+(b_{r,i}(\ddot{q}_{k,i})T)s_{k-r,i}^{\eta}E_{i}+E_{i}^{2}$
$\displaystyle\leq$
$\displaystyle(r+2)[(b_{0,i}(\ddot{q}_{k,i})T)^{2}s_{k,i}^{2\eta}+(b_{1,i}(\ddot{q}_{k,i})T)^{2}s_{k-1,i}^{2\eta}$
$\displaystyle+\dots+(b_{r,i}(\ddot{q}_{k,i})T)^{2}s_{k-r,i}^{2\eta}+E_{i}^{2}].$
(18)

According to (17) further, we have

$\displaystyle s_{k+1,i}^{2}<$
$\displaystyle(r+2)\left((b_{0,i}(\ddot{q}_{k,i})T)^{2}s_{k,i}^{2}+(b_{1,i}(\ddot{q}_{k,i})T)^{2}s_{k-1,i}^{2}\right.$
$\displaystyle\left.+\dots+(b_{r,i}(\ddot{q}_{k,i})T)^{2}s_{k-r,i}^{2}\right)$
$\displaystyle+(r+2)\left(E_{i}^{2}+\sum_{j=0}^{r}(b_{j,i}(\ddot{q}_{k,i})T)^{2}\right).$
(19)

If we define $\alpha_{0}=1$ and $\alpha_{r+1}=0$, then there is

$\displaystyle\Delta U_{k,i}=$
$\displaystyle\left(\alpha_{1}-\alpha_{0}+(r+2)(b_{0,i}(\ddot{q}_{k,i})T)^{2}\right)s_{k,i}^{2}$
$\displaystyle+\left(\alpha_{2}-\alpha_{1}+(r+2)(b_{1,i}(\ddot{q}_{k,i})T)^{2}\right)s_{k-1,i}^{2}+\dots$
$\displaystyle+\left(\alpha_{r}-\alpha_{r-1}+(r+2)(b_{r-1,i}(\ddot{q}_{k,i})T)^{2}\right)s_{k-r+1,i}^{2}$
$\displaystyle+(\alpha_{r+1}-\alpha_{r}+(r+2)(b_{r,i}(\ddot{q}_{k,i})T)^{2})s_{k-r,i}^{2}$
$\displaystyle+(r+2)E^{2}.$ (20)

From (11), we obtain that

$\displaystyle\alpha_{n+1}-\alpha_{n}+(r+2)(b_{n,i}(\ddot{q}_{k,i})T)^{2}s_{k,i}^{2}\leq
0.$ (21)

Assume that $s_{k-m,i}=\max\\{s_{k,i},s_{k-1,i},\dots,s_{k-r,i}\\}$, then we
have

$\displaystyle\Delta U_{k,i}$
$\displaystyle\leq\left(\alpha_{m+1,i}-\alpha_{m,i}+(r+2)(b_{m,i}(\ddot{q}_{k,i})T)^{2}\right)s_{k-m,i}^{2}$
$\displaystyle+(r+2)E^{2}+\sum_{j=0}^{r}(b_{j,i}(\ddot{q}_{k,i})T)^{2}.$ (22)

When $|s_{k-m}|$ is not in the region $\Gamma_{i}$, i.e., there is

$\displaystyle s_{k-m,i}^{2}$
$\displaystyle>\frac{(r+2)E^{2}+\sum_{j=0}^{r}(b_{j,i}(\ddot{q}_{k,i})T)^{2}}{\alpha_{m}-\alpha_{m+1}-(r+2)(b_{m,i}(\ddot{q}_{k,i})T)^{2}}$
(23)

it is obtained that

$\displaystyle\Delta U_{k,i}<0.$ (24)

This completes the proof of Theorem 1. ∎

###### Remark 1

From (11), we can infer that when the sampling interval $T$ is smaller,
$b_{n,i}(\ddot{q}_{k,i})$ has a larger range of values. In other words, the
stability of the system determines a maximum value of the sampling interval.

###### Remark 2

Equation (13) gives some clues on the source of errors. Firstly, a larger $T$
results in a larger region $\Gamma_{i}$. Secondly, a larger $E_{k,i}$ also
causes a larger convergence region of $s_{k-m,i}$, and worse tracking
precision further. As $E_{k,i}$ is related to the estimation error of TDE, we
can get that a precise TDE is the precondition of the precise tracking.
Traditional methods such as [20] and [24], only rely on TDE to compensate for
the nonlinearity. They have a larger estimation error than our method under
the situations that the nonlinearity is large. Thus our method has better
precision theoretically.

##  IV SIMULATIONS AND EXPERIMENTS

Figure 1: The simulation setup of the FANUC LR Mate 200iD industrial robot, including (a) the frames and (b) the reference trajectory. Figure 2: Structure of the closed-loop system where DHTSMC is applied to robots ($R_{r}$ stands for the reduction ratios of gears).  TABLE I: D-H Parameters of 6-DOF FANUC LR Mate 200iD Robot $i$ |  $\alpha_{i-1}$ (rad) |  $a_{i-1}$ (mm) |  $d_{i}$ (mm) |  $\theta_{i}$ (rad)
---|---|---|---|---
1 | -$\frac{\pi}{2}$ | 50 | 0 | $q_{1}$
2 | $\pi$ | 440 | 0 | $q_{2}-\frac{\pi}{2}$
3 | -$\frac{\pi}{2}$ | 35 | 0 | $q_{3}$
4 | $\frac{\pi}{2}$ | 0 | -420 | $q_{4}$
5 | -$\frac{\pi}{2}$ | 0 | 0 | $q_{5}$
6 | $\pi$ | 0 | -80 | $q_{6}$
TABLE II: Physical Parameters of 6-DOF FANUC LR Mate 200iD Robot NO. | Gear Ratio | Link Mass | Motor Inertia | Link Coulomb | Link Viscous | Motor Coulomb | Motor Viscous
---|---|---|---|---|---|---|---
(kg) | (kg m2) | Friction (N$\cdot$m) | Coefficient (N$\cdot$s/m) | Friction (N$\cdot$m) | Coefficient (N$\cdot$s/m)
1 | 114.6 | 2.4 | 8.9 $\times$10-5 | 0.045 | 3.1 | 0.052 | 0.23$\times$10-3
2 | 121 | 7.8 | 6.0 $\times$10-5 | 0.095 | 4.1 | 0.052 | 0.28$\times$10-3
3 | 102.1 | 3.0 | 5.2 $\times$10-5 | 0.11 | 1.2 | 0.041 | 0.11$\times$10-3
4 | 73.0 | 4.1 | 7.2 $\times$10-5 | 0.094 | 0.81 | 0.044 | 0.15$\times$10-3
5 | 83.3 | 1.7 | 1.4 $\times$10-5 | 0.10 | 0.37 | 0.012 | 0.053$\times$10-3
6 | 41.4 | 0.17 | 1.7 $\times$10-5 | 0.15 | 0.18 | 0.028 | 0.11$\times$10-3

###  IV-A Simulation

The simulation environment is developed based on MATLAB R2019b and Simulink.
With the help of the tool of Mechanics Explorer, a complete model of FANUC LR
Mate 200iD industrial robot (shown in Fig. 1) is established. Coordinates of
the manipulator are set as shown in Fig. 1 (a), and corresponding D-H
parameters are displayed in Table I. Other related physical parameters of the
simulation model are given in Table II.

The control structure of the proposed method is shown in Fig. 2. First of all,
the reference positions and rotations in Cartesian space are generated. The
initial position is $p_{1}(0.47,0,0.395)$, and the initial rotation angle is
$\psi_{1}(0,0,\pi)$ in the order of $Z_{1}Y_{2}X_{3}$. Then it moves to
$p_{2}(0.2,0.2,0.2)$ with rotation angle $\psi_{2}(\pi/4,\pi/4,\pi/2)$. At
point $p_{3}(0.2,0.3,0.3)$ with rotation angle $\psi_{3}(\pi/2,\pi/2,\pi/2)$,
after that, the manipulator returns to the initial position and rotation via
$p_{4}(0.3,0.1,0.5)$ with rotation $\psi_{4}(\pi/4,\pi/4,\pi/2)$, as shown in
Fig. 1 (b).

Then the motion planning rules are conducted to smooth the trajectories and
realize some additional functions on robots. The simple S-curve motion
planning rule is considered in the simulation to make the acceleration and
deceleration of the robots are performed gently and steadily. After that, we
input the planning results to the inverse kinematics, which is derived based
on the frames and Denavit–Hartenberg (D-H) parameters of the robot
manipulator, to obtain the reference $\bm{q}$. Then the proposed controller is
applied in the joint space.

Figure 3: The reference angle of each joint in the simulation. Figure 4: The
tracking error in the joint space in the simulation.

We set the maximum acceleration as $50~{}m/s^{2}$ from $p_{1}$ to $p_{2}$ and
as $10~{}m/s^{2}$ from $p_{2}$ to $p_{1}$. Before every motion, the
manipulator will wait for $0.1~{}s$, and the sampling time is set as the same
as the controller, $1~{}ms$. After that, we calculate the inverse kinematics
based on the frames shown in Fig. 1 (a) and D-H parameters shown in Table. I.
After the planning and calculation of inverse kinematics, the trajectory of
each joint is shown in Fig. 3. We introduce the band-limited white noise as an
additional disturbance, whose sampling time is set as $0.1$ s and other
parameters remain default. For the controller, parameters are tuned as
$a_{1}=diag[10,100,100,15,100,10]$, $a_{2}=0.01\bm{I}$, $b_{0}=4.5\times
10^{5}+0.005\ddot{q}_{i}$, $b_{1}=2.25\times 10^{5}$, $r=1$ and $\bm{I}$ is
the unit matrix.

Apart from the proposed DHTSMC method, we also implement the traditional
feedforward control scheme with TSMC (noted as FF-TSMC) with TDE proposed in
[1] as a comparison. The results of the simulations are illustrated in Fig. 4.
For each joint, there are four peaks or valleys in the trajectory of tracking
error, corresponding to the processes moving from one point to the other. When
the joint is waiting in a position, tracking errors of both method can
converge to a relatively small value. This phenomenon means that when the
joint is accelerating or decelerating, the joint tends to have a more massive
tracking error. However, with the proposed method, the tracking error is
significantly reduced when the joints change their positions, which is about
two-thirds of the traditional FF-TSMC. This shows that compared with the
feedforward structure, the proposed method can make better use of the
dynamics, to get better performance.

Figure 5: The experimental setup. Figure 6: The tracking error in the joint
space in the experiments. Figure 7: The tracking error in the Cartesian space
in the experiments.

###  IV-B Experiments

Practical experiments are also conducted to verify the effectiveness of the
proposed method further. As shown in Fig. 5, the experimental setup consists
of the host computer, the target computer, the digital servo adapter (DSA),
and the FANUC LR Mate 200iD robots. The control algorithm is programmed on the
host computer and applied to the DSA and robot via Simulink Real-Time running
on the target computer. The parameters of the robot are the same as the model
in the previous simulation. The sampling interval of the robot is set as 0.25
ms, and that of the controller is set as 1 ms.

We also compare the FF-TSMC and the proposed control method in the
experiments. The reference trajectory of the robots is directly given in joint
space. All the joints are set to rotate from $0^{\circ}$ to $20^{\circ}$ and
then return to $0^{\circ}$. Parameters of the controller are set as
$a_{1}=diag[1,20,13,2,15,3]$, $a_{2}=0.015\bm{I}$, $b_{0}=1\times
10^{5}+0.002\ddot{q}_{i}$, $b_{1}=0.25\times 10^{5}$ and $r=1$.

Fig. 6 shows the tracking error of each joint under the two controllers. We
can see that the tracking error of the proposed method is overall smaller in
each joint compared with the previous FF-TSMC. We have also noticed that for
both methods, there is an offset after the manipulator reaches the target
position. We think this is caused by the unmodelled dynamics of the robot. It
is easy to find that for most joints, the proposed method has a smaller offset
than the FF-TSMC.

Further, the performance of the robots in the Cartesian space is shown in Fig.
7. The position and pose of the end-effector are calculated by using the joint
position and the forward kinematics. In particular, the pose is described by
Euler angles, which rotates in the order of $Z_{1}Y_{2}X_{3}$. From Fig. 7, we
can see that the proposed method has a smaller error in both positions and
poses. The position error of the proposed method is around 1 mm, and the Euler
angular error is below 0.2∘, which are half of the previous method.

##  V CONCLUSIONS

A control method of DHTSMC with TDE was proposed and implemented on the
manipulators in this paper. The controller was derived with the nonlinear
dynamics of robots taken into the feedback loop directly, so that the
nonlinearities, the uncertainties, and disturbances were adequately handled.
Specifically, we improved the structure of the sliding surface to achieve a
faster response, and we gave a universal control method of different order
sliding mode controllers. By using the variable gains, the conflict between
the fast response and small chattering amplitude was further settled. We also
analyzed the stability of the closed-loop system. Results of simulations and
experiments showed that the proposed controller was much more effective than
conventional DTSMC methods in both joint space and Cartesian space.

## ACKNOWLEDGMENT

The authors would like to thank Dr. Yu Zhao for his contribution to the MATLAB
simulator of the robot manipulator used in this work.

## References

  * [1] Y. Wang, K. Zhu, B. Chen, and M. Jin, “Model-free continuous nonsingular fast terminal sliding mode control for cable-driven manipulators,” _ISA Trans._ , vol. 98, pp. 483–495, 2020.
  * [2] F. Abdelhedi, Y. Bouteraa, A. Chemori, and N. Derbel, “Nonlinear pid and feedforward control of robotic manipulators,” in _2014 15th Int. Conf. on Sciences and Techniques of Automatic Control and Computer Engineering (STA)_. IEEE, 2014, pp. 349–354.
  * [3] J.-M. Yang and J.-H. Kim, “Sliding mode control for trajectory tracking of nonholonomic wheeled mobile robots,” _IEEE Trans. Robot. Autom._ , vol. 15, no. 3, pp. 578–587, 1999.
  * [4] V. Alakshendra and S. S. Chiddarwar, “A robust adaptive control of mecanum wheel mobile robot: simulation and experimental validation,” in _2016 IEEE/RSJ Int. Conf. on Intelligent Robots and Syst. (IROS)_. IEEE, 2016, pp. 5606–5611.
  * [5] J. H. Park and H. C. Cho, “Sliding mode control of bilateral teleoperation systems with force-reflection on the internet,” in _Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000)(Cat. No. 00CH37113)_ , vol. 2. IEEE, 2000, pp. 1187–1192.
  * [6] Z. Kuang, H. Gao, and M. Tomizuka, “Precise linear-motor synchronization control via cross-coupled second-order discrete-time fractional-order sliding mode,” _IEEE/ASME Trans. Mechatronics_ , 2020.
  * [7] Z. Ma, P. Huang, and Z. Kuang, “Fuzzy approximate learning-based sliding mode control for deploying tethered space robot,” _IEEE Trans. Fuzzy Syst._ , 2020\.
  * [8] Z. Kuang, L. Sun, H. Gao, and M. Tomizuka, “Fractional-order variable-gain super-twisting control with application to wafer stages of photolithography systems,” in _2020 Int. Symp. on Flexible Automation_. American Society of Mechanical Engineers Digital Collection, 2020.
  * [9] Y. Feng, X. Yu, and Z. Man, “Non-singular terminal sliding mode control of rigid manipulators,” _Automatica_ , vol. 38, no. 12, pp. 2159–2167, 2002\.
  * [10] Z. Kuang, G. Sun, and H. Gao, “Simplified newton-based cee and discrete-time fractional-order sliding-mode cec,” _IEEE/ASME Trans. Mechatronics_ , vol. 24, no. 1, pp. 175–185, 2018.
  * [11] S. Li, H. Du, and X. Yu, “Discrete-time terminal sliding mode control systems based on euler’s discretization,” _IEEE Trans. Autom. Control_ , vol. 59, no. 2, pp. 546–552, 2013.
  * [12] K. Abidi, J.-X. Xu, and J.-H. She, “A discrete-time terminal sliding-mode control approach applied to a motion control problem,” _IEEE Trans. Ind. Electron._ , vol. 56, no. 9, pp. 3619–3627, 2008.
  * [13] Z. Kuang, Z. Yu, G. Sun, W. Lin, and H. Gao, “Contouring control of linear motor direct-drive xy table via chattering-free discrete-time sliding mode control,” in _2018 IEEE 27th International Symposium on Industrial Electronics (ISIE)_. IEEE, 2018, pp. 201–205.
  * [14] V. K. Tripathi, A. K. Kamath, N. K. Verma, and L. Behera, “Fast terminal sliding mode super twisting controller for position and altitude tracking of the quadrotor,” in _2019 International Conf. on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 6468–6474.
  * [15] J. Kim, M. Jin, W. Choi, and J. Lee, “Discrete time delay control for hydraulic excavator motion control with terminal sliding mode control,” _Mechatronics_ , vol. 60, pp. 15–25, 2019.
  * [16] J. J. Rath, M. Defoort, H. R. Karimi, and K. C. Veluvolu, “Output feedback active suspension control with higher order terminal sliding mode,” _IEEE Trans. Ind. Electron._ , vol. 64, no. 2, pp. 1392–1403, 2016.
  * [17] S. Ahmed, H. Wang, and Y. Tian, “Adaptive high-order terminal sliding mode control based on time delay estimation for the robotic manipulators with backlash hysteresis,” _IEEE Trans. Syst., Man, Cybern., Syst._ , 2019.
  * [18] Q. Xu, “Piezoelectric nanopositioning control using second-order discrete-time terminal sliding-mode strategy,” _IEEE Trans. Ind. Electron._ , vol. 62, no. 12, pp. 7738–7748, 2015.
  * [19] T. Iwamura and M. Toda, “Motion control of an oscillatory-base manipulator using sliding mode control via rotating sliding surface with variable-gain integral control,” in _2013 American Control Conf._ IEEE, 2013, pp. 5742–5747.
  * [20] J. Lee, P. H. Chang, and M. Jin, “Adaptive integral sliding mode control with time-delay estimation for robot manipulators,” _IEEE Trans. Ind. Electron._ , vol. 64, no. 8, pp. 6796–6804, 2017.
  * [21] B. Della Corte, H. Andreasson, T. Stoyanov, and G. Grisetti, “Unified motion-based calibration of mobile multi-sensor platforms with time delay estimation,” _IEEE Robot. Autom. Lett._ , vol. 4, no. 2, pp. 902–909, 2019\.
  * [22] M. Van, S. S. Ge, and H. Ren, “Finite time fault tolerant control for robot manipulators using time delay estimation and continuous nonsingular fast terminal sliding mode control,” _IEEE Trans. Cybern._ , vol. 47, no. 7, pp. 1681–1693, 2016.
  * [23] Y. Kali, M. Saad, K. Benjelloun, and C. Khairallah, “Super-twisting algorithm with time delay estimation for uncertain robot manipulators,” _Nonlinear Dyn._ , vol. 93, no. 2, pp. 557–569, 2018.
  * [24] Y. Wang, L. Gu, Y. Xu, and X. Cao, “Practical tracking control of robot manipulators with continuous fractional-order nonsingular terminal sliding mode,” _IEEE Trans. Ind. Electron._ , vol. 63, no. 10, pp. 6194–6204, 2016\.