# Learning Visual Affordance Grounding from Demonstration Videos

Hongchen Luo, Wei Zhai, Jing Zhang, , Yang Cao, , and Dacheng Tao H. Luo, W.
Zhai and Y. Cao are with the School of Information Science and Technology, at
the University of Science and Technology of China, Anhui, China. (email:
{lhc12<EMAIL_ADDRESS>forrest@ustc.edu.cn).J. Zhang and D. Tao
are with the School of Computer Science, in the Faculty of Engineering, at the
University of Sydney, Sydney, Australia. (email:
{jing.zhang1,dacheng.tao}@sydney.edu.au).

###### Abstract

Visual affordance grounding aims to segment all possible interaction regions
between people and objects from an image/video, which is beneficial for many
applications, such as robot grasping and action recognition. However, existing
methods mainly rely on the appearance feature of the objects to segment each
region of the image, which face the following two problems: (i) there are
multiple possible regions in an object that people interact with; and (ii)
there are multiple possible human interactions in the same object region. To
address these problems, we propose a Hand-aided Affordance Grounding Network
(HAG-Net) that leverages the aided clues provided by the position and action
of the hand in demonstration videos to eliminate the multiple possibilities
and better locate the interaction regions in the object. Specifically, HAG-Net
has a dual-branch structure to process the demonstration video and object
image. For the video branch, we introduce hand-aided attention to enhance the
region around the hand in each video frame and then use the LSTM network to
aggregate the action features. For the object branch, we introduce a semantic
enhancement module (SEM) to make the network focus on different parts of the
object according to the action classes and utilize a distillation loss to
align the output features of the object branch with that of the video branch
and transfer the knowledge in the video branch to the object branch.
Quantitative and qualitative evaluations on two challenging datasets show that
our method has achieved state-of-the-art results for affordance grounding. The
source code will be made available to the public.

###### Index Terms:

Visual affordance grounding, Learning from demonstrations, Weakly supervised
learning, Deep learning.

##  I Introduction

For an object in the scene, we need to know what it is and what interactions
it may have with people, and the possible interactions are the affordance of
the object. The concept of affordance was introduced by the ecological
psychologist Gibson [1] in 1979 to describe the potential “action
possibilities” that the environment can provide. The “action possibilities”
reflect all possible interactions between the object and the agent. In
particular, perceiving object affordance is a valuable capability and has a
wide range of applications in action recognition, robot grasping, autonomous
driving, scene understanding, _etc_. [2, 3, 4, 5, 6, 7, 8].

This paper focuses on the affordance grounding task, _i.e._ , given an action
label and an object image, our goal is to generate the heatmap where the
action happens. However, in Gibson’s definition of affordance, “it implies the
complementarity of the animal and the environment” [1]. The multiple potential
complementarities between the animal and the environment, leading to the
possibility that affordance has multiple possibilities: (i) there are multiple
possible regions in an object where people interact. As shown in Fig 1 (a),
the same object may imply two different interactions, “pull” and “rotate”, and
the interaction occurs at different locations; and (ii) there are multiple
possible human interactions in the same region. As shown in Fig. 1 (b), two
different interactions, “rotate” and “touch”, can happen at the same position
on the same object. Existing methods [9, 10, 11, 12] typically establish the
mapping relationship between apparent features to affordance labels, which are
not skilled at capturing the affordance-related context and perceiving the
affordance multiple possibilities.

Figure 1: The challenges of affordance grounding. We show some examples of
affordance with multiple possibilities. (a) Different interactions occur at
different locations of the same object. (b) Different interactions occur at a
similar position of the same object.

At the same time, the study [13] found that, in a particular scene, the
affordance of the object is uniquely determined by the intention of the human
action. For example, in a demonstration video, human hands and objects
interact frequently, and human action intention can be inferred from hand
position and action, providing an essential clue to solve the above two
problems. For the first question, we can infer the region where the person
interacts with the object by the hand’s position, such as “touch” and “pull”
in Fig. 1 (a). For the second question, we eliminate the ambiguity of
affordance by inferring human intention through human hand actions, such as
(b) in Fig. 1, according to the hand action in the video, we can infer whether
the object’s affordance is “touch” or “rotate”. To this end, we both consider
the affordance clues provided by the position and the action of the hand and
solve multiple possibilities of affordance grounding by establishing a mapping
relationship between hand action intentions and object parts.

This paper presents a Hand-aided Affordance Grounding Network (HAG-Net) to
learn the affordance of objects from demonstration videos. We use the clues
provided by the position and the action of the hand to solve the two problems
in the affordance grounding. The pipeline of our method is shown in Fig. 2. To
better capture the affordance cues from the demonstration videos, we propose a
hand-related selection network to select the keyframes which primarily reflect
the interaction between the hand and the object. During learning the object’s
affordance from the demonstration videos, we emphasize the features around the
hand to provide valuable context to learn the region where the person
interacts with the object. Instead of simply feeding the hand features
directly into the network, we consider the occluded features and the objects
surrounding the hand. Since the region of human-object interaction is related
to the category of action, we design a semantic enhancement module (SEM) that
considers both the action category and appearance features to jointly adjust
the feature of the object image so that the network can better focus on the
feature regions related to the action category. Finally, we consider both the
position and the action of the hand to better transfer the knowledge of the
video branch to the object branch. The contributions of this paper are as
follows:

  1. 1.

We propose a Hand-aided Affordance Grounding Network (HAG-Net) to use the
clues provided by the hands in the demonstration videos. We leverage both the
position of the hand and the action to eliminate the multiple possibilities in
the affordance grounding task.

  2. 2.

We introduce the Semantic Enhancement Module (SEM), which utilizes the action
category and appearance features to jointly modulate the feature of the object
image to locate the relevant region of the interaction more effectively.

  3. 3.

Experimental results on the two most challenging affordance grounding datasets
have demonstrated the superiority of the proposed model against the state-of-
the-arts.

##  II Related Work

###  II-A Affordance Grounding

Affordance grounding is to detect the regions of the object where interactions
may occur. Early works [9, 10, 11, 14] are mainly on image-based supervised
segmentation tasks, outputting the affordance label of each pixel for a given
image. However, these tasks rely on pixel-level labels, and these methods can
not learn how humans interact with objects.

Another part of the early works represents affordance as a pose in which
humans interact with objects. Yao et al. [15] used a clustering method to find
that all possible object functionalities and represented it in the form of a
pose in which human interaction with musical instruments. Wang et al. [16]
used Variational-Auto Encoders (VAE) to construct their model to predict
affordance poses. Li et al. [17] proposed a 3D generative model to predict
physically plausible and physically feasible human poses in a given 3D scene.
The difference from the above methods is that we focus on the possible
interactions of the various parts of the object instead of the human pose.

A large number of works solve the affordance grounding by learning the
interactions between humans and objects in a video. Koppula et al. [9]
proposed a generative model to ground the affordance into the form of the
object’s spatial position and time trajectory. Fang et al. [18] used the
feature embedding of the demonstration video to predict the interaction region
of the target object and proposed the OPRA (Online Product Review dataset for
Affordance) dataset. Nagarajan et al. [19] learned the interaction between
humans and objects by observing videos. Unlike the above methods, we
explicitly infer human action intentions from the cues provided by the
position and action of a human hand to specify the uniqueness of affordance in
a given scene, thus eliminating ambiguity due to the multiple possibilities of
affordance.

Figure 2: The pipeline of our method. Our method first trains a hand-related
selection network to select keyframes and then sends the pre-processed video
frames to the Hand-aided Affordance Grounding Network (HAG-Net), trained using
only action classes as supervisory signals.
\begin{overpic}[width=433.62pt]{figs/figure_3.pdf} \put(18.0,-1.2){{(a)}}
\put(69.0,-1.2){{(b)}} \put(6.0,20.7){\hbox{\pagecolor{white}\scriptsize{OPRA}
\cite[cite]{[\@@bibref{}{demo2vec2018cvpr}{}{}]}}}
\put(26.0,20.7){\hbox{\pagecolor{white}\scriptsize{EPIC}
\cite[cite]{[\@@bibref{}{Damen2018EPICKITCHENS}{}{}]}}} \end{overpic} Figure
3: Hand-aided mask. (a) Some hand detection results, We use the trained YOLOv3
pruning model [21, 22] to detect the position of the hand on each frame of the
two datasets of OPRA [18] and EPIC [20]. (b) The shadow regions are the
surrounding regions of the hand enhanced by hand-aided attention. We extend
the bounding box of the hand to the surroundings and remove the regions
occluded by the hand. The blue mask on the right side is the hand mask
($Mask_{t}$) that we input into the network.

###  II-B Hand and Affordance

There is a large number of researches in the field of hand-based egocentric
action recognition. Hand detection, segmentation, and tracking technologies
[23, 24, 25] continue to develop and their research results are applied to
model actions and activities [26, 27]. Kjellstrom et al. [4] learned the
affordance of objects from the human demonstration and extracted hand
position, orientation, and articulated pose for embedding action features.
Stark et al. [28] obtained the affordance clues from the interaction between
the human hand and the objects in the trainset and then determined the
functions of the objects according to the affordance cue features. Sun et al.
[29] significantly improved the detection accuracy of the interactive objects
by learning hand movement trajectories and statistical knowledge in training
data. Song et al. [30] predicted the human intention by observing the
human/object interaction. Pieropan et al. [31] directly represent objects as
their interactions with human hands for modeling human activity. In this
paper, we use the clues provided by the hand to learn the affordance of the
objects.

###  II-C Learning from Demonstrations

By learning how humans interact with objects through demonstrations, the robot
can perceive the affordance of the objects. Furthermore, when faces with
objects in different environments, the robot can mimic human actions to
interact with the objects through the knowledge learned from human
demonstrations. In recent years, there has been a great deal of works on
learning from demonstrations [32, 33, 34, 35, 18]. Schulman et al. [32]
proposed a method based on non-rigid trajectory transfer for adapting the
demonstrated trajectory from the training geometry to the test geometry,
enabling the robot to tie different types of knots in the rope automatically.
Chu et al. [33] learned different haptic affordances by demonstrating learning
to provide a robot with an example of successful interaction with a given
object. Aleotti et al. [34] performed the task of robot grasp planning well by
combining the limited and automatic 3D shape segmentation of human
demonstrators for object recognition and semantic modeling. Fang et al. [18]
proposed to learn from online product review videos and then transfer the
knowledge from the videos to the target image to learn the relevant regions of
human-object interaction. We also consider learning object affordance from
demonstrator videos but only use the action labels of the videos as
supervisory signals to learn the mapping relationships from static objects to
the object features interacted in the videos.

##  III Method

###  III-A Problem Description

In this paper, our goal is to learn the affordance of objects through the
demonstration videos while only using the action class as supervision without
pixel-level labels. The complete process is shown in Fig. 2. Before learning
affordance, we first detect the hand in each frame of the video. Some
detection results are shown in Fig. 3 and then introduce a specific selection
network. It combines the clues provided by the position and action of the hand
to select the keyframes where the hand interacts with the objects. After
preprocessing the video datasets, we introduce a Hand-aided Affordance
Grounding Network (HAG-Net), as shown in Fig. 4. In the affordance learning
phase (train phase), we leverage the affordance cues of the object from the
demonstration videos and transferring them to a static object. Finally, in the
affordance grounding phase (test phase), we feed an object image and an action
label into the Hand-aided Affordance Grounding Network (HAG-Net) and output a
heatmap of the region on the object where the action may appear using
visualization techniques.

###  III-B Pre-processing

Since human-object interactions frequently occur in the demonstration videos,
the hand position and actions can offer valuable clues for locating the
affordance region of interest. Both OPRA [18] and EPIC [20] have no hand
annotations, however, the large amount of existing work on object detection
[21, 36, 37] allows us to accurately detect the human hand. We choose the
YOLOv3 pruning model [21, 22] trained on the Oxford hand dataset [38] to
detect the hands in each frame of the two datasets. At most, two hands can be
detected in each frame. If the number of detected hands is greater than two,
we select the two hands with the highest confidence. Some of the detection
results are shown in Fig. 3 (a). As can be seen, the YOLOv3 model can
accurately detect the hand.

Figure 4: The architecture of Hand-aided Affordance Grounding Network (HAG-
Net). (A) Learning object affordance from demonstration videos. (A-1) For
video branches, we first extract features using ResNet50 [39], introduce Hand-
aided attention to enhance the hand-related region features, and finally use
LSTM to aggregate the features of video frames to obtain action classification
results. For the object branch, HAG-Net uses the ResNet50 to extract features,
which together with the action label are fed into the semantic enhancement
module (SEM). Then, a projection layer transfers the affordance information
from the demonstration video to the object image. Finally, we use the
distillation loss to align the features of the object branch and those of the
video branch. (A-2) The Semantic Enhancement Module (SEM) utilizes a
combination of action categories and object apparent features to guide the
network to enhance features in areas relevant to action interactions. (B)
Affordance grounding. Given an object image and an action label, we use the
Grad-CAM to activate the affordance regions of the feature map according to
the action (affordance) label.

Since many video frames do not contain helpful information about the
interaction between the human and the objects, which may cause an undesirable
influence on the affordance grounding, therefore, we use a specific hand-
related selection network to select the keyframes where the hand interacts
with the object and filter out the frames whose intermediate states cannot
provide affordance clues. The framework has a similar structure to the video
branch in Fig. 4 (A-1). Since hand position and action cues play an essential
role in affordance grounding, we also introduce hand-aided attention in the
keyframe selection process to make the network pay attention to hand-aided
information. We send the feature maps after CNN into hand-aided attention to
enhance the relevant features of human hands, thus making the selected frames
more discriminative. We will introduce how hand-aided attention can make use
of hand information better in Section III-C. To capture the action information
of the hands in the demonstration videos, we use the Long Short-Term Memory
(LSTM) [40] to aggregate the temporal features. After training, we process the
videos in both OPRA [18] and EPIC [20] datasets. Specifically, we randomly
choose a starting position and select eight consecutive frames, then feed into
the network, and the frame with the highest confidence is selected. We only
keep the frame when the action prediction is correct or the confidence is
greater than a certain threshold (generally set to 0.3). We repeat this
process and select several keyframes for each video. In this way, the retained
frames are discriminative and can provide affordance-related information.

###  III-C Affordance Learning

After preprocessing the video, we introduce the Hand-aided Affordance
Grounding Network (HAG-Net). It is mainly divided into two parts, as shown in
Fig. 4. The sub-figure (A) describes the process of affordance learning of
objects from demonstration videos, and the sub-figure (B) describes the
process of affordance grounding. This section mainly introduces the process of
affordance learning, and we will describe the affordance grounding in Section
III-D. We divide the network into two branches during the training phase to
learn how human interacts with the objects in the videos and transfer the
knowledge to the object branch.

For the video branch, let a video contain T frames
$\mathcal{V}=\\{f_{1},f_{2},...,f_{T}\\}$ with an afforded action class $c$.
First, we use a ResNet50 [39] backbone to extract the features of each frame,
and the features of the video are represented as
$\mathcal{X}=\\{X_{1},X_{2},...,X_{T}\\}$. We then input $\mathcal{X}$ into
hand-aided attention (in Section III-C) to obtain the enhanced feature
$\mathcal{M}=\\{M_{1},M_{2},...,M_{T}\\}$. After enhancing the relevant
features of the human hand, we input them into the LSTM [40] for aggregating
the hand action features:

$H(\mathcal{V})=LSTM(M_{1},M_{2},...,M_{T}),$ (1)

where $H$ is the feature representation after aggregation. Finally, $H$ is
sent to a classifier to predict the action class.

For the object branch, we first extract the features of the object image using
the same ResNet50 backbone. Then, the semantic enhancement module $SEM(\cdot)$
is introduced to enhance the features. Since there is a large gap between the
feature of the interacting object and the static object, we use a projection
layer $Proj(\cdot)$ to learn the object feature representation $\tilde{X}$
when the interaction occurs:

$\tilde{X}=Proj(SEM(X,c)),$ (2)

$X$ is the feature map of the object image after ResNet50 [39].

Finally, we introduce a distillation loss ($\mathcal{L}_{distill}$) to align
the feature space of static objects to the feature space of human-object
interaction, from the affordance of the objects in the demonstration video to
the static objects. To ensure that the feature representation of the object
after the projection layer has the same classification performance as the
video branch, we send the output of the projection layer to the same LSTM and
action classifier and calculate the classification loss of the object branch.
The total loss ($\mathcal{L}_{total}$) is mainly composed of the following
three parts:

$\mathcal{L}_{total}=\lambda_{vcls}\mathcal{L}_{vcls}+\lambda_{distill}\mathcal{L}_{distill}+\lambda_{ocls}\mathcal{L}_{ocls},$
(3)

where $\lambda_{vcls}$, $\lambda_{distill}$, and $\lambda_{ocls}$ are hyper-
parameters that balance the loss items. $\mathcal{L}_{vcls}$ and
$\mathcal{L}_{ocls}$ represent the cross-entropy loss of video branch and
object branch for action classification, respectively. In the following
sections, we present the details of Hand-Aided Attention, Semantic Enhancement
Module, Projection Layer, and Distillation Loss, respectively.

$\bullet$ Hand-Aided Attention. We use the results of Section III-B to enhance
the features of the hand regions. Since we need the context information of the
hands and the objects, we expand the bounding box of the detected hand and
exclude the hand regions that may occlude the object. The expansion and
exclusion regions are controlled by parameters $\alpha_{1}$ ($0<\alpha_{1}<1$)
and $\alpha_{2}$ ($0<\alpha_{2}<1$). As shown in Fig. 3 (b), $h$ and $w$ are
the height and width of the hand bounding box, respectively. The expansion
region is the bounding box enclosed by $(1+\alpha_{1})\times w$ and
$(1+\alpha_{1})\times h$, while the exclusion region is the bounding box
enclosed by $(1-\alpha_{2})\times w$ and $(1-\alpha_{2})\times h$. The hand
mask ($Mask_{t}$) used in our network is shown in the right part of Fig. 3
(b). We set $\alpha_{2}$ to 1 when calculating the hand-aided attention in the
selection network and do not exclude the hand region. While in the affordance
learning phase, since it will affect the extraction of affordance clues, we
set $\alpha_{2}$ to 0.4 accordingly. After obtaining $Mask_{t}$, we calculate
the masked feature $X_{t}^{M}$ by multiplying it with the feature map $X_{t}$
of the video frame via an position-wise dot product operator $\odot$:

$X_{t}^{M}=X_{t}\odot Mask_{t}.$ (4)

Then the feature representation of each frame and the mask of the hand are
respectively passed through L2 pooling layer ($L2P(\cdot)$) and added to
obtain the enhanced feature $M_{t}$:

$g_{t}=L2P(X_{t}),$ (5) $g_{t}^{M}=L2P(X_{t}^{M}),$ (6)
$M_{t}=g_{t}+g_{t}^{M},$ (7)

where $g_{t}$ and $g_{t}^{M}$ are the feature representations of the video
frame and the masked feature after pooling, respectively.

$\bullet$ Semantic Enhancement Module.

Many previous studies [41, 42, 43] have demonstrated that the features learned
by the grouping mechanism are more compact and have better representational
power. Therefore, we introduce an Semantic Enhancement Module (SEM) to improve
the feature representation capability of object branches by leveraging a
group-wise feature enhancement mechanism. Furthermore, within each grouping,
the feature of the object image is modulated jointly by action categories and
global features, thus enabling the network to focus more on affordance-related
features. Our SEM is shown in Fig. 4 (A-2), the feature map of size $C\times
H\times W$ is divided into G groups along the channel dimension. The feature
of the $i$th position of the $k$th group is represented as $X_{i}^{k}$,
$k\in[1,G],i\in[1,H\times W]$.

Besides, we use a fully connected layer $fc(\cdot)$ to map the one-hot action
label to have $C$ channels and use a Softmax layer to get the action class
weights $X_{class}$:

$X_{class}=\text{Softmax}(fc(\text{one-hot}(c))),$ (8)

where $\text{one-hot}(\cdot)$ is the one-hot representation of action class.
We also divide $X_{C}$ into G groups. Then, we use a global average pooling to
aggregate the global statistical feature from $X^{k}$ and add it with
$X_{class}^{k}$ to obtain the semantic vector $S^{k}$ of the $k$th group:

$S^{k}=\frac{1}{H\times W}\sum_{i=1}^{H\times W}X_{i}^{k}+X_{class}^{k}.$ (9)

Then, we calculate the correlation coefficient $c^{k}$ between $X^{k}$ and
$S^{k}$ and normalize it at each position as follows:

$\begin{split}&c^{k}=S^{k}\otimes X^{k},\\\ \end{split}$ (10)
$\hat{c}_{i}^{k}=\frac{c_{i}^{k}-\mu^{k}}{\sigma^{k}+\epsilon},$ (11)
$\mu^{k}=\frac{1}{H\times W}\sum_{j=1}^{H\times W}c_{j}^{k},$ (12)
$\sigma^{k}=\sqrt{\frac{1}{H\times W}\sum_{j}^{H\times
W}(c_{j}^{k}-\mu^{k})^{2}},$ (13)

where $\otimes$ means the element-wise multiplication operation, $\mu^{k}$ and
$\sigma_{k}$ represent the mean and standard deviation of the group of
coefficients, respectively, and $\epsilon$ is a regularization constant. After
that, we introduce $\gamma$ and $\beta$, which scale and shift the normalized
value $\hat{c}_{k,i}$:

$a_{i}^{k}=\gamma^{k}\hat{c}_{i}^{k}+\beta^{k},$ (14)

where $k$ denotes the $k$th group of parameters. Finally, we input $a_{i}^{k}$
to a sigmoid function gate $\sigma(\cdot)$, and multiply it with the original
feature representation $X_{i}^{k}$ to get the enhanced feature representation
$\hat{X}_{i}^{k}$:

$\hat{X}_{i}^{k}=X_{i}^{k}\odot\sigma(a_{i}^{k}).$ (15)

$\bullet$ Projection Layer and Distillation Loss.

We introduce a projection layer and a distillation loss to align the feature
representations of the object branch and the video branch, reducing the
distance in feature space between the object in the static image and the
object in the scene of human-object interaction in the video. During aligning
the feature representation of interaction between humans and objects to that
of static objects, we mainly focus on two factors: 1) the highest-confidence
frame $X_{t^{*}}$ in the input video contains the affordance clues of the
hand’s position, and 2) the average feature of the input contains the hand
context.

Specifically, in the process of obtaining the features of $X_{t^{*}}$, we
determine $t^{*}$ using the following equation:

$t^{*}=\mathop{\min\arg}\limits_{t\in
1,...,T}\mathcal{L}_{fcls}(LSTM(M_{1},...,M_{T}),c),$ (16)

where $L_{fcls}$ is the cross-entropy loss for classification using the hidden
state of LSTM at time $t$ (each frame of input). The average feature of the
video is represented as $\bar{X}_{\mathcal{V}}$:

$X_{\mathcal{V}}=\frac{1}{T}(\sum_{t=1}^{T}X_{t}).$ (17)

We define a distillation loss
($\mathcal{L}_{distill}(\tilde{X},X_{\mathcal{V}},X_{t^{*}})$) to adjust the
projection layer so that it can align the feature representation of static
object and that of interactive object:

$\begin{split}\mathcal{L}_{distill}&=\lambda_{1}\times||L2P(\tilde{X})-L2P(X_{t^{*}})||_{2}\\\
&+\lambda_{2}\times||L2P(\tilde{X})-L2P(X_{\mathcal{V}})||_{2},\end{split}$
(18)

where $L2P(\cdot)$ is the L2 pooling operation, $\lambda_{1}$ and
$\lambda_{2}$ are the loss weights. $\mathcal{L}_{distill}$ is the key to
ensure that the object image is mapped to the feature space of character
interactions.

###  III-D Affordance Grounding

In Affordance Grounding process, the input is just a static object image, and
our goal is to infer the interaction regions of all possible actions on this
object image. For each action class $c$, the model generates a heatmap of the
position where the human interacts with the object, as shown in Fig. 4 (B). In
this paper, we use the Grad-CAM [44] feature visualization technique to
establish the mapping between action labels to regions related to human-object
interaction. In the ablation study in Section IV-D, we also compare other
advanced visualization methods and prove that Grad-CAM can achieve the best
results. The detailed approach is as follows: 1) for a specific object image
encoding $X$ and action class $c$, the sensitivity of the action relative to
each channel of the last layer of the feature map is calculated, which can be
regarded as the attention mask of the feature map of this layer; 2) unlike
Grad-CAM, we directly combine it with the weighted linear combination of the
last layer of the feature map instead of taking the gradient map to the mean
value; and 3) we input it to the ReLU activation function for further
processing. Using the ReLU function can help our model focus on the pixels
that have a positive impact on a particular category:

$H_{c}(X)=\sum_{k}ReLU(\frac{\partial Y^{c}}{\partial X^{k}}\odot X^{k}),$
(19)

where $\odot$ is the element-wise multiplication operator, $X^{k}$ is the
$k$th channel of the input embedding, and $\frac{\partial Y^{c}}{\partial
X^{k}}$ is a two-dimensional attention mask. $H_{c}(X)$ represents the final
interaction heatmap for the given action class.

##  IV Experiments

In our experiments, we explore the following questions:

  * $\bm{-}$

$\bm{Q1:}$ Does our method outperform other weakly or fully supervised methods
on the affordance grounding task? (In Section IV-C)

  * $\bm{-}$

$\bm{Q2:}$ Does our method have excellent generalization performance on unseen
objects? (In Section IV-C)

  * $\bm{-}$

$\bm{Q3:}$ What is the influence of each module of our model and different
visualization strategies on affordance grounding? (In Section IV-D)

We chose the OPRA [18] dataset for the third-person perspective, and the EPIC
[20] dataset for the first-person perspective to evaluate the effectiveness of
the model in terms of both objective metrics as well as subjective
visualization. In the ablation study, we explore different visualization
strategies and investigate the impact of each module of our model on
affordance grounding.

###  IV-A Experimental Setup

$\bullet$ Datasets. Our main goal is to use human demonstration videos to
learn how people interact with objects. We need datasets that contain a large
number of demonstration videos in which people interact with various objects.
To this end, we conducted our experiments on the following two datasets.

  * $\bm{-}$

Online Product Review dataset for Affordance (OPRA) [18]: Fang et al. [18]
proposed the OPRA dataset, which aims to use demonstration videos for object
affordance inference. Each sample contains a video, an object image,
affordance class, and annotations of the interacting regions on the object
image.

  * $\bm{-}$

EPIC-KITCHENS (EPIC) [20]: The dataset contains a large number of egocentric
videos of activities in kitchens. Each clip contains an action label and an
object. Moreover, each frame has a bounding box of the object that interacts
with the person. In this paper, we use the data annotated in [19].

$\bullet$ Evaluation Metrics. To evaluate the results of different models
comprehensively, we choose four metrics from [45], including KLD [45], SIM
[46], AUC-J [47], NSS [48].

TABLE I: Results of the proposed method and several weakly supervised methods, including Saliency detection methods (Egogaze [49], Mlnet [50], DeepgazeII [51], Salgan [52]) and Hotspot [19], and two affordance detection methods using mask labels as strong supervision during training, including Demo2vec [18] and Affsegnet [19], on the OPRA [18] and EPIC [20] datasets. $K$, $S$, $A$, and $N$ represent the four evaluation metrics KLD [45], SIM [46], AUC-J [47], and NSS [48], respectively. $\uparrow$ /$\downarrow$ indicates higher/lower results are better.  | Affordance Grounding | Generalization to Novel Objects
---|---|---
Dataset |  OPRA [18] |  EPIC [20] |  OPRA [18] |  EPIC [20]
Method | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_ | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_ | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_ | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_
Center bias | $11.132$ | $0.205$ | $0.625$ | $0.323$ | $10.660$ | $0.222$ | $0.634$ | $0.333$ | $6.281$ | $0.244$ | $0.680$ | $0.340$ | $5.910$ | $0.277$ | $0.699$ | $0.344$
Egogaze [49] | $2.428$ | $0.245$ | $0.646$ | $0.247$ | $2.241$ | $0.273$ | $0.614$ | $0.281$ | $2.083$ | $0.278$ | $0.694$ | $0.244$ | $1.974$ | $0.298$ | $0.673$ | $0.280$
Mlnet [50] | $4.022$ | $0.284$ | $0.763$ | $0.607$ | $6.116$ | $0.318$ | $0.746$ | $0.809$ | $2.458$ | $0.316$ | $0.778$ | $0.551$ | $3.221$ | $0.361$ | $0.799$ | $0.831$
DeepgazeII [51] | $1.897$ | $0.296$ | $0.720$ | $0.496$ | $1.352$ | $0.394$ | $0.751$ | $0.888$ | $1.757$ | $0.318$ | $0.742$ | $0.466$ | $1.297$ | $0.400$ | $0.793$ | $0.917$
Salgan [52] | $2.116$ | $0.309$ | $0.769$ | $0.659$ | $1.510$ | $0.395$ | $0.774$ | $0.978$ | $1.698$ | $0.337$ | $0.790$ | $0.622$ | $1.296$ | $0.406$ | $0.808$ | $0.987$
Hotspot [19] | $1.427$ | $0.362$ | $0.806$ | $0.907$ | $1.258$ | $0.404$ | $0.785$ | $0.923$ | $1.381$ | $0.374$ | $0.826$ | $0.912$ | $1.249$ | $0.405$ | $0.817$ | $1.001$
Ours | $1.409$ | $0.365$ | $0.812$ | $0.948$ | $1.209$ | $0.414$ | $0.801$ | $1.045$ | $1.366$ | $0.373$ | $0.817$ | $0.927$ | $1.197$ | $0.412$ | $0.820$ | $1.084$
Img2heatmap [19] | $1.473$ | $0.355$ | $0.821$ | $0.894$ | $1.400$ | $0.359$ | $0.794$ | $0.925$ | $1.431$ | $0.362$ | $0.820$ | $0.850$ | $1.466$ | $0.353$ | $0.770$ | $0.720$
Demo2vec[18] | $1.197$ | $0.482$ | $0.847$ | $1.170$ | - | - | - | - | - | - | - | - | - | - | - | -
\begin{overpic}[width=429.28616pt]{figs/figure_6.pdf}
\put(21.5,59.3){\hbox{\pagecolor{white}{OPRA}
\cite[cite]{[\@@bibref{}{demo2vec2018cvpr}{}{}]}}}
\put(71.0,59.3){\hbox{\pagecolor{white}{EPIC}
\cite[cite]{[\@@bibref{}{Damen2018EPICKITCHENS}{}{}]}}}
\put(-0.5,52.0){\footnotesize{\rotatebox{90.0}{GT}}}
\put(-0.5,41.0){\footnotesize{\rotatebox{90.0}{ Ours}}}
\put(-0.5,31.0){\footnotesize{\rotatebox{90.0}{Hotspot}}}
\put(1.0,31.5){\footnotesize{\rotatebox{90.0}{
\cite[cite]{[\@@bibref{}{interaction-hotspots}{}{}]}}}}
\put(-0.5,22.0){\footnotesize{\rotatebox{90.0}{Saliency}}}
\put(1.0,21.0){\footnotesize{\rotatebox{90.0}{
\cite[cite]{[\@@bibref{}{Pan_2017_SalGAN,DBLP:journals/corr/KummererWB16,DBLP:conf/icpr/CorniaBSC16,DBLP:conf/eccv/HuangCLS18}{}{}]}}}}
\put(-0.5,10.5){\footnotesize{\rotatebox{90.0}{Img2heatmap}}}
\put(1.0,13.5){\footnotesize{\rotatebox{90.0}{
\cite[cite]{[\@@bibref{}{interaction-hotspots}{}{}]}}}}
\put(-0.5,1.0){\footnotesize{\rotatebox{90.0}{Demo2vec}}}
\put(1.0,3.0){\footnotesize{\rotatebox{90.0}{
\cite[cite]{[\@@bibref{}{demo2vec2018cvpr}{}{}]}}}} \end{overpic} Figure 5:
Visualization of the affordance heatmaps generated by different methods,
including Hotspot [19], Saliency (Egogaze [49], Mlnet [50], DeepgazeII [51]
and Salgan [52]), Img2heatmap [19] and Demo2vec [18].

  * $\bm{-}$

KLD [45]: Kullback-Leibler Divergence (KLD) is used to measure the
distribution difference between the prediction map and the target map. Given a
prediction map $P$ and a ground truth map $Q^{D}$, $KL(\cdot)$ is computed as
follows:

$KL(P,Q^{D})=\sum_{i}Q_{i}^{D}log(\epsilon+\frac{Q_{i}^{D}}{\epsilon+P_{i}}),$
(20)

where $\epsilon$ is a regularization constant.

  * $\bm{-}$

SIM [46]: The similarity metric (SIM) measures the similarity between the
prediction map and the ground truth map. Given a prediction map $P$ and a
continuous ground truth map $Q^{D}$, $SIM(\cdot)$ is computed as the sum of
the minimum values at each pixel, after normalizing the input maps:

$\begin{split}&SIM(P,Q^{D})=\sum_{i}min(P_{i},Q_{i}^{D}),\\\
&where\quad\sum_{i}P_{i}=\sum_{i}Q_{i}^{D}=1.\end{split}$ (21)

  * $\bm{-}$

AUC-J [47]: AUC-Judd (AUC-J) is a variant of AUC proposed by Judd et al. [47].
It measures the relative prediction map values at ground truth locations.

  * $\bm{-}$

NSS [48]: The Normalized Scanpath Saliency measures the correspondence between
the prediction map and the ground truth, and it treats false positives and
false negatives symmetrically. Given a prediction map $P$ and a binary ground
truth map $Q^{D}$, $NSS(\cdot)$ computes the average normalized prediction at
ground truth locations:

$\begin{split}&NSS(P,Q^{D})=\frac{1}{N}\sum_{i}\hat{P}\times Q_{i}^{D},\\\
&where\quad N=\sum_{i}Q_{i}^{D}\quad
and\quad\hat{P}=\frac{P-\mu(P)}{\sigma(P)}.\end{split}$ (22)

$\bullet$ Implementation Details. Each video in the OPRA [18] dataset has an
object image associated with it, while the EPIC [20] dataset does not. Thus,
for the EPIC dataset, we use the provided bounding box to crop the object from
the video frame according to [19] and randomly select an image whose class
matches the object class in the video. Due to the background of EPIC is more
complex and the apparent features between the objects in the video and the
objects in the object image are different, we replace L2 loss with triplet
loss when calculating
$\mathcal{L}_{distill}(\tilde{X},\tilde{X^{{}^{\prime}}},X_{\mathcal{V}},X_{t^{*}})$.
The triplet can better utilize positive and negative samples for feature
learning so that the same category is close in the space and different
categories are far away in the space. Closing the gap of features belonging to
the same category can help the network better focus on interacting objects and
ignore irrelevant backgrounds. The $\mathcal{L}_{distill}$ is calculated as
follows:

$\begin{split}\mathcal{L}_{distill}=&\lambda_{1}\times\max[0,d(L2P(X_{t^{*}}),L2P(\tilde{X}))\\\
&-d(L2P(X_{t^{*}}),L2P(\tilde{X^{{}^{\prime}}}))+M]\\\
&+\lambda_{2}\times\max[0,d(L2P(X_{\mathcal{V}}),L2P(\tilde{X}))\\\
&-d(L2P(X_{\mathcal{V}}),L2P(\tilde{X^{{}^{\prime}}}))+M].\end{split}$ (23)

We set $\lambda_{vcls}=\lambda_{ocls}=1$ on the two datasets. For OPRA
dataset, we set $\lambda_{1}=1$, $\lambda_{2}=0.2$, and
$\lambda_{distill}=0.1$. For EPIC dataset, we set $\lambda_{1}=1$,
$\lambda_{2}=0.5$, and $\lambda_{distill}=1$. We carry out all experiments on
1080Ti, and set the batch size and learning rate to 128 and 1e-4,
respectively. During training the hand-related selection network, the video
input is eight frames, while during affordance learning from demonstration
videos, the video input is three frames. Furthermore, we set the stride of the
last two residual stages of ResNet50 to 1 and using dilated convolutions in
the convolutional layers. In this way, the output spatial resolution of
ResNet50 is 1/8 of the input.

###  IV-B Contenders

We compare the performance of our method with several state-of-the-art methods
on OPRA [18] and EPIC [20]. It is worth to notice that we select a series of
saliency detection models as the comparison methods, because the human visual
system has the ability to quickly orient attention to the most informative
parts of visual scenes and the research on salient object detection is derived
from this ability of the human visual system to extract the most attention-
grabbing objects from the image. These saliency detection models include
Egogaze [49], Mlnet [50], DeepgazeII[51], and Salgan[52]. The generated
heatmaps by them represent the parts of the object first noticed by the human
visual system. We directly use the trained saliency detection models for test.
In addition, we also choose the Hotspot [19], Demo2vec [18], and Img2heatmap
[19] models, where our model, Hotspot, and the saliency detection models are
all weakly supervised, while Demo2vec and Img2heatmap are fully supervised.
All these methods are briefly described as follows:

  * $\bm{-}$

Center bias: It generates a gaussian heatmap at the center of an image. Thus,
it is a simple baseline for a dataset with features of central bias.

  * $\bm{-}$

Egogaze[49]: It is a hybrid gaze prediction model that exploits both the
visual saliency of bottom-up and task-dependent attention transition and is
the first work to explore the attention transition model in the egocentric
gaze prediction task and achieves state-of-the-art results in gaze prediction.

  * $\bm{-}$

Mlnet[50]: Unlike previous works that predict saliency maps directly from the
last layer of convolution neural network, the model fuses features extracted
from different layers of the CNN. Their method contains three main blocks:
feature extraction CNN, feature encoding network (weighting of low and high
features), and a prior learning network. The method achieves promising results
in all datasets for saliency detection.

  * $\bm{-}$

DeepgazeII[51]: Unlike other saliency models, DeepGazeII does not perform
additional fine-tuning of the VGG features and only trains some output layers
to predict saliency on top of VGG.

  * $\bm{-}$

Salgan[52]: It introduces the adversarial training mechanism of GAN for
salient object prediction, which consists of two main parts: one predicts
saliency maps based on the input image, and the other discriminates whether
the input is the prediction result or the grounding truth. It explores the
application of GAN to salient object detection and achieves excellent results
on relevant datasets.

  * $\bm{-}$

Hotspot [19]: It is a weakly supervised way to learn the affordance of an
object through video, and affordance grounding is achieved only through action
labels.

  * $\bm{-}$

Demo2vec [18]: It is a fully supervised way to learn the affordance of objects
by learning the coded representation in the demonstration video to predict the
regions in the object image that also interact with each other. The network’s
input is a demonstration video and an object image, and the output is the
action category of the demonstration video and the interacting regions.

  * $\bm{-}$

Img2heatmap [19]: The encoder is a VGG16 [53] pre-trained on Imagenet [54].
The decoder is a mirrored VGG16, where the max-pooling is replaced by the
upsampling operation. The network’s final output is a heatmap of the same size
as the input, and the number of channels is the same as the action class.

###  IV-C Results Analysis

In this section, we compare the ability of different models for affordance
grounding on two datasets and their generalization abilities on unseen
objects, as well as discuss the compatibility of our model for datasets with
first- and third-person view videos. Finally, we discuss the effectiveness of
our model for handling affordance with multiple possibilities.

$\bullet$ Affordance Grounding Results. We compare our method with the state-
of-the-art methods on OPRA [18] and EPIC [20] datasets, the results are
summarized in the left part of Table I. Our method surpasses all other weakly
supervised methods in all metrics and is close to the supervised Demo2vec [18]
and Img2heatmap [19]. It proves that our method utilizes the affordance cues
provided by hand position and action can achieve promising results. We also
visualize the heatmaps generated by different methods in Fig. 5. Our method
generates heatmaps that are closer to ground truth than those of Hotspot and
saliency detection models. And there is no large response on object parts that
are not related to actions. The results on some objects are even better than
those of the supervised Img2heatmap [19] and Demo2vec [18] methods. It shows
that our method transfers the affordance cues from the hand to the static
object, which can make the network pay more attention to the regions related
to the affordance while suppressing the regions unrelated to the action.

$\bullet$ Generalization to Novel Objects. To verify the generalization
ability of our method on new objects, we re-divide the datasets according to
[19]. The results are shown in Table I (right part). On EPIC [20] dataset, our
model outperforms all other methods in all metrics. On OPRA [18] dataset, our
method is superior to other methods in most metrics. It demonstrates that our
method can predict the region of interaction on unseen objects. This is
because the cues provided by our method from hand motion and position
information enable the network to pay more attention to the local details of
the particular affordance class of objects, thus having better generalization
performance.

$\bullet$ Compare the Results of Merging Two Datasets. To verify whether our
method can perform better on a more complex multi-view dataset, we merge the
OPRA [18] and EPIC [20] datasets. The experimental results are shown in Table
II. Our method is superior to other methods in all metrics, suggesting that
our method can also transfer hand position and action clues to static objects
in a complex dataset. This is due to the fact that our method focuses more on
hand-object interactions and is not too sensitive to changes in viewpoint and
can be more robust to complex backgrounds.

We compared the results of “Affordance grounding”, “Generalization to novel
objects”, and “ Merging two datasets” for three task settings with 20 metrics,
ranked the results of various methods on each metric in each task setting, and
summarized the result as a matrix, where each element (i,j) indicates how many
metrics the model has ranked the $j$th. The results are shown in Fig. 6, from
which we can see that our method achieves promising results in different task
settings, with stronger robustness and generalization ability.

TABLE II: The results of different methods on the mixture of OPRA [18] and EPIC [20] datasets, $i.e.$, a dataest with images at different views.  Dataset |  OPRA [18] + EPIC [20]
---|---
Method | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_
Center bias | $10.968$ | $0.211$ | $0.629$ | $0.327$
Egogaze[49] | $2.361$ | $0.255$ | $0.635$ | $0.327$
Mlnet[50] | $4.735$ | $0.297$ | $0.757$ | $0.676$
DeepgazeII[52] | $1.705$ | $0.331$ | $0.730$ | $0.629$
Salgan[51] | $1.902$ | $0.339$ | $0.771$ | $0.768$
Hotspot[19] | $1.435$ | $0.366$ | $0.769$ | $0.825$
Ours | 1.378 | 0.373 | 0.786 | 0.901
Img2heatmap [19] | 1.444 | 0.358 | 0.769 | 0.811
\begin{overpic}[width=433.62pt]{figs/fig9.pdf} \put(29.0,2.0){\hbox{\pagecolor{white}1}} \put(38.0,2.0){\hbox{\pagecolor{white}2}} \put(47.0,2.0){\hbox{\pagecolor{white}3}} \put(56.0,2.0){\hbox{\pagecolor{white}4}} \put(65.0,2.0){\hbox{\pagecolor{white}5}} \put(74.0,2.0){\hbox{\pagecolor{white}6}} \put(83.0,2.0){\hbox{\pagecolor{white}7}} \put(92.0,2.0){\hbox{\pagecolor{white}8}} \par\put(18.0,75.0){\small Ours} \put(17.0,71.0){\small({\color[rgb]{1,0,0}1.20})} \put(7.0,66.0){\small Hotspot \cite[cite]{[\@@bibref{}{interaction-hotspots}{}{}]} } \put(17.0,62.0){\small({\color[rgb]{1,0,0}2.20})} \put(-1.0,57.0){\small Img2heatmap \cite[cite]{[\@@bibref{}{interaction-hotspots}{}{}]}} \put(17.0,53.0){\small({\color[rgb]{1,0,0}3.45})} \put(9.0,48.0){\small Salgan \cite[cite]{[\@@bibref{}{Pan_2017_SalGAN}{}{}]}} \put(17.0,44.0){\small({\color[rgb]{1,0,0}3.65})} \par\put(2.0,39.0){\small DeepgazeII \cite[cite]{[\@@bibref{}{DBLP:journals/corr/KummererWB16}{}{}]}} \put(17.0,35.0){\small({\color[rgb]{1,0,0}4.90})} \put(10.0,30.0){\small Mlnet \cite[cite]{[\@@bibref{}{DBLP:conf/icpr/CorniaBSC16}{}{}]}} \put(17.0,26.0){\small({\color[rgb]{1,0,0}5.75})} \put(6.0,21.0){\small Egogaze\cite[cite]{[\@@bibref{}{DBLP:conf/eccv/HuangCLS18}{}{}]}} \put(17.0,17.0){\small({\color[rgb]{1,0,0}7.05})} \put(9.0,12.0){\small Center bias} \put(17.0,8.0){\small({\color[rgb]{1,0,0}7.65})} \end{overpic} Figure 6: Rank list. We rank the 20 results of different methods on the tasks of “Affordance grounding”, “Generalization to novel object”, and “Merging two datasets” in Table I and Table II, where the element ($i$, $j$) indicates how many metrics that the model $i$ are ranked the $j$th. The red letter denotes the average rank.  TABLE III: Results of different methods on twelve most frequent classes in the EPIC [20] dataset. The best performing method is highlighted in bold.  Dataset |  EPIC [20]
---|---
Method | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_
Center bias | $10.478$ | $0.219$ | $0.632$ | $0.328$
Egogaze[49] | $2.335$ | $0.258$ | $0.614$ | $0.328$
Mlnet[50] | $5.888$ | $0.315$ | $0.744$ | $0.796$
DeepgazeII[52] | $1.434$ | $0.373$ | $0.746$ | $0.862$
Salgan[51] | $1.537$ | $0.381$ | $0.772$ | $0.968$
Hotspot [19] | $1.347$ | $0.376$ | $0.781$ | $0.905$
Ours | 1.292 | 0.387 | 0.798 | 1.030
Img2heatmap [19] | $1.426$ | $0.350$ | $0.791$ | $0.911$
\begin{overpic}[width=433.62pt]{figs/figure_7.pdf}
\put(36.0,46.0){\hbox{\pagecolor{white}{(a) Hotspot}
\cite[cite]{[\@@bibref{}{interaction-hotspots}{}{}]}}}
\put(42.0,1.0){\hbox{\pagecolor{white}{(b) Ours}}} \end{overpic} Figure 7: The
confusion matrices of Hotspot [19] and our method for action prediction on the
EPIC dataset [20]. We select eight most frequent classe.
\begin{overpic}[width=433.62pt]{figs/fig_x.pdf} \put(23.5,-0.5){ {(a)}}
\put(74.0,-0.5){ {(b)}} \end{overpic} Figure 8: Visualization of heatmaps
generated by our method when handling affordance with multiple possibilities.
(a) Different interactions may occur in different regions of the object. (b)
Multiple possible interactions take place at the same location of the object.

$\bullet$ Comparison on 12 Most Frequent Classes of EPIC. Since the data
distribution in the EPIC [20] dataset is highly unbalanced, e.g., the number
of samples for some categories is less than one percent. The experimental
results are more affected by the classes that contain a large number of
samples. To verify the effectiveness of our method in the case of a large
number of samples, we compare our method with others on the twelve most
frequent classes. The experimental results are shown in Table III. Our method
outperforms all other methods, implying that our method can better locate the
region where people interact with objects in more samples.

$\bullet$ Comparison of the Action Classification Accuracy. Since our method
and hotspot [19] only use action class as supervision during affordance
learning, the accuracy of action recognition affects the results of affordance
grounding. We compare the performance of action classification between ours
and hotspot [19] on the EPIC [20] dataset. We only choose the eight most
frequent classes and calculate the confusion matrix, as shown in Fig. 7. Our
method performs better on “open”, “close”, “take”, “put”, and “wash”,
demonstrating that it can better distinguish different actions by combining
hand position and action-related affordance cues to improve the result of
action classification.

TABLE IV: Comparison of different visualization approaches, including Grad-CAM++ [55], XGrad-CAM [56], and Grad-CAM [44] for obtaining the object affordance region.  | Affordance Grounding | Generalization to Novel Objects
---|---|---
Dataset |  OPRA [18] |  EPIC [20] |  OPRA [18] |  EPIC [20]
Method | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_ | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_ | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_ | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_
HAG-Net & Grad-CAM++ [55] | $1.482$ | $0.356$ | $0.770$ | $0.797$ | $1.319$ | $0.400$ | $0.750$ | $0.799$ | $1.436$ | $0.366$ | $0.773$ | $0.797$ | $1.322$ | $0.398$ | $0.758$ | $0.786$
HAG-Net & XGrad-CAM [56] | $1.464$ | $0.359$ | $0.782$ | $0.837$ | $1.274$ | $0.407$ | $0.775$ | $0.912$ | $1.417$ | $0.368$ | $0.785$ | $0.832$ | $1.274$ | $0.404$ | $0.785$ | $0.907$
HAG-Net & Grad-CAM (Ours) | 1.409 | 0.365 | 0.812 | 0.948 | 1.209 | 0.414 | 0.801 | 1.045 | 1.366 | 0.373 | 0.817 | 0.927 | 1.197 | 0.412 | 0.820 | 1.084
\begin{overpic}[width=433.62pt]{figs/fig_10.png}
\put(23.6,44.4){\hbox{\pagecolor{white}{OPRA}
\cite[cite]{[\@@bibref{}{demo2vec2018cvpr}{}{}]}}}
\put(71.3,44.4){\hbox{\pagecolor{white}{EPIC}
\cite[cite]{[\@@bibref{}{Damen2018EPICKITCHENS}{}{}]}}}
\put(2.0,36.0){\footnotesize{\rotatebox{90.0}{GT}}}
\put(0.5,24.0){\footnotesize{\rotatebox{90.0}{ HAG-Net \&}}}
\put(2.0,24.0){\footnotesize{\rotatebox{90.0}{ Grad-CAM}}}
\put(3.5,25.0){\footnotesize{\rotatebox{90.0}{ (Ours)}}}
\put(0.5,14.0){\footnotesize{\rotatebox{90.0}{ HAG-Net \&}}}
\put(2.0,14.0){\footnotesize{\rotatebox{90.0}{XGrad-CAM}}}
\put(3.5,17.0){\footnotesize{\rotatebox{90.0}{
\cite[cite]{[\@@bibref{}{fu2020axiom}{}{}]}}}}
\put(0.5,3.0){\footnotesize{\rotatebox{90.0}{ HAG-Net \&}}}
\put(2.0,3.0){\footnotesize{\rotatebox{90.0}{Grad-CAM++}}}
\put(3.5,6.5){\footnotesize{\rotatebox{90.0}{
\cite[cite]{[\@@bibref{}{chattopadhay2018grad}{}{}]}}}} \end{overpic} Figure
9: Heatmaps by using different visualization approaches including Grad-CAM++
[55], XGrad-CAM [56], and Grad-CAM [44] in our method.

$\bullet$ Affordance with Multiple Possibilities. We also show the visual
heatmap results to demonstrate that our model can address the challenge of
multiple possibilities of affordance well. As shown in Fig. 8 (a), different
actions interact with different regions of same the object, and our model can
localize to different regions of the object based on the action category. Fig.
8 (b) shows that different interactions occur in the same region of the
object, and our model can also accurately localize to the region associated
with the affordance of the interaction. It demonstrates that our model can
reason about human intentions through human actions and positions and well
address the challenge of ambiguity in affordance.

###  IV-D Ablation Studies

$\bullet$ Impact of Visualization Strategy. In this section, we also compare
two more advanced visualization methods, i.e., XGrad-CAM [56] and Grad-CAM++
[55], to verify the effectiveness of Grad-CAM [44] used in our method for
affordance grounding both subjectively and objectively.

  * $\bm{-}$

XGrad-CAM [56]: This paper introduces two axioms: sensitivity [57] and
consistency [58], and makes XGrad-CAM satisfy these two constraints. XGrad-CAM
is a visualization method with class discriminative ability to highlight the
relevant regions belonging to the class, which is calculated as follows:

$w_{k}=\sum_{i}\sum_{j}\frac{X^{k}_{ij}}{\sum_{a}\sum_{b}X^{k}_{ab}}\odot\frac{\partial
Y^{c}}{\partial X^{k}_{ij}},$ (24)
$H^{xgradcam}_{c}(X)=\sum_{k}ReLU(w_{k}\otimes X^{k}).$ (25)

  * $\bm{-}$

Grad-CAM++ [55]: Grad-CAM++ is a more generalized approach based on Grad-CAM
and is capable of predicting better visual interpretation of CNN models and
better visual localization. Furthermore, it provides a way to measure the
importance of each pixel in the feature map to the overall decision of the CNN
by introducing a pixel weighting of the gradients of the output to get better
visualization results, which is calculated as follows:

$\alpha^{k}_{ij}=\frac{\frac{\partial^{2}Y^{c}}{(\partial
X^{k}_{ij})^{2}}}{2\frac{\partial^{2}Y^{c}}{(\partial
X^{k}_{ij})^{2}}+\sum_{a}\sum_{b}X^{k}_{ab}\odot\frac{\partial^{3}Y^{c}}{(\partial
X^{k}_{ij})^{3}}},$ (26) $w_{k}=\sum_{i}\sum_{j}\alpha^{k}_{ij}\odot
ReLU(\frac{\partial Y^{c}}{\partial X^{k}_{ij}}),$ (27)
$H^{gradcam++}_{c}(X)=\sum_{k}ReLU(w_{k}\otimes X^{k}).$ (28)

The experimental results are shown in Table IV. As can be seen, using Grad-CAM
in our method outperforms both of the remaining two more advanced
visualization methods. In the affordance grounding task setting, it
outperforms XGrad-CAM [56] and Grad-CAM++ [55] by 13.3% and 18.9% in terms of
NSS [48] metrics on the OPRA [18] dataset, respectively. On the EPIC [20]
dataset, it exceeds XGrad-CAM by 14.6% and surpasses Grad-CAM++ by 30.8% in
terms of NSS metrics. These results demonstrate that our method can obtain
more accurate region localization. Some of the visualization results are shown
in Fig. 9. The heatmaps generated by XGrad-CAM and Grad-CAM++ are large and
contain a large number of irrelevant interaction regions, while our results
can better focus on affordance-related regions, probably because their
calculations focus more on the object as a whole, whereas our method can
retain local details to better focus on affordance-related local regions of
the object.

TABLE V: Ablation results on both OPRA [18] and EPIC [20] datasets. For details please refer Section IV-D. Dataset |  OPRA [18] |  EPIC [20]
---|---|---
Method | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_ | _$K\downarrow$_ | _$S\uparrow$_ | _$A\uparrow$_ | _$N\uparrow$_
Random (a) | $1.446$ | $0.358$ | $0.798$ | $0.877$ | $1.281$ | $0.403$ | $0.772$ | $0.922$
Random (b) | $1.436$ | $0.360$ | $0.802$ | $0.900$ | $1.273$ | $0.403$ | $0.779$ | $0.944$
w/o hand-aided attention (a) | $1.424$ | $0.363$ | $0.806$ | $0.916$ | $1.253$ | $0.404$ | $0.789$ | $0.960$
w/o hand-aided attention (b) | $1.420$ | $0.363$ | $0.808$ | $0.914$ | $1.248$ | $0.405$ | $0.791$ | $0.974$
w/o hand-aided attention (c) | $1.416$ | $0.364$ | $0.809$ | $0.928$ | $1.235$ | $0.408$ | $0.795$ | $0.996$
w/o select frame | $1.418$ | $0.364$ | $0.806$ | $0.934$ | $1.239$ | $0.408$ | $0.793$ | $0.990$
Max score | $1.437$ | $0.359$ | $0.801$ | $0.890$ | $1.261$ | $0.403$ | $0.787$ | $0.966$
Average | $1.443$ | $0.361$ | $0.799$ | $0.885$ | $1.264$ | $0.404$ | $0.781$ | $0.968$
w/o SEM | $1.419$ | $0.363$ | $0.809$ | $0.917$ | $1.241$ | $0.407$ | $0.797$ | $1.001$
Ours | 1.409 | 0.365 | 0.812 | 0.948 | 1.209 | 0.414 | 0.801 | 1.045

$\bullet$ Effectiveness of Different Modules. In this section, We investigate
the impact of each module in our method on affordance grounding.

  * $\bm{-}$

Random (a): In the preprocessing phase, a single frame is randomly chosen from
the input and the process is iterated three times. We also remove the hand-
aided attention during the affordance learning phase.

  * $\bm{-}$

Random (b): In preprocessing, a single frame is randomly chosen from the input
and this process is iterated three times. We keep the network structure for
affordance learning as in Fig. 4 (A).

  * $\bm{-}$

w/o hand-aided attention (a): We remove the hand-aided attention in the
selection network during the preprocessing process, and also remove the hand-
aided attention in the HAG-Net.

  * $\bm{-}$

w/o hand-aided attention (b): We remove the hand-aided attention in the
selection network during the preprocessing process, but keep it for affordance
learning as shown in Fig. 4.

  * $\bm{-}$

w/o hand-aided attention (c): There is no change in the pre-processing stage.
But during the affordance learning phase, the hand-aided attention of HAG-Net
is removed.

  * $\bm{-}$

w/o select frame: We input 8 video frames into our HAG-Net without selecting
keyframes.

  * $\bm{-}$

Max frame: We use the frame with the highest confidence when calculating the
distillation loss.

  * $\bm{-}$

Average frames: We take the average of the input three frames when calculating
the distillation loss.

  * $\bm{-}$

w/o SEM: We remove the SEM from the object branch of the affordance learning
network.

The ablation study results are shown in Table V. From the top rows we can see
that using hand-aided attention matters for affordance grounding, e.g., our
model achieves relative improvements of up to 8.1%, and 13.3% (NSS) compared
to random (a) on the OPRA [18] and EPIC [20] datasets, respectively. It proves
that the position and the action of the hand provide essential clues for the
affordance of learning objects from the demonstration videos. In the setting
without selecting keyframes, where we input the original video frames into the
network for training, it can still achieve good results but falls behind of
ours. For example, our method achieves a gain of 1.5% NSS on OPRA and 5.6% NSS
on EPIC over the model “w/o select frame”, respectively. These results
demonstrate that the pre-processing step of selecting keyframes enables the
network to learn more critical cues, leading to better results.

Compared to using only the frame with the highest confidence, our method
achieves relative improvements of 6.5% and 8.2% (NSS) on OPRA and EPIC
datasets, respectively. Furthermore, compared to using only the input average
features, our method achieves relative improvements of 7.1% and 8.0% (NSS) on
the two datasets, respectively. These results validate that calculating the
distillation loss of the object and video branches from the highest confidence
frame and the average feature of the input can better transfer the affordance
cues of the video to the static objects. Compared with the model without SEM,
our method achieves 3.4% and 4.4% (NSS) relative improvements on OPRA and
EPIC, respectively. SEM can allow the object branch to focus on different
parts based on appearance and action class information, thereby improving the
results of affordance grounding.

$\bullet$ Effectiveness of Hand-aided Attention in Other Models. We also
incorporate the preprocessing strategy as well as hand-aided attention in the
hotspot [19] model to explore the impact of hand cues for affordance
grounding. The results are shown in Table VI, from which we can see that
levearging the human hand information can also improve the performance of the
existing methods. For example, using the preprocessed frames as input to the
Hotspot brings a relative improvement of 1.7%, while adding hand-aided
attention to the affordance learning process of Hotspot brings a relative 1.9%
improvement. However, the performance is still inferior to ours, which is
mainly because we also leverage affordance labels to modulate the features of
the target image and consider the affordance clues from hand actions and
positions. Consequently, our method can better locate the human-object
interactions and learn affordance-related contextual information efficiently.

##  V Conclusion and Discussion

In this paper, we propose a novel Hand-aided Affordance Grounding Network
(HAG-Net) for learning the affordance of objects from demonstration videos. By
leveraging the clues offered by the position and action of the hand, we
address two challenging problems in affordance grounding: (i) the same object
has multiple possible interaction regions; and (ii) the same region has
multiple possible interactions. Experiments on two public datasets demonstrate
that our method achieves state-of-the-art results for affordance grounding.

$\bullet$ Weakness Although our HAG-Net has achieved good affordance grounding
performance, there are still some limitations that should be addressed in the
future work. Firstly, it is not an end-to-end solution, where a separate pre-
processing stage is needed. In the future, we plan to devise an end-to-end
affordance grounding method, which can optimize the selection of video frames
containing affordance-related information and select hand context features in
the same framework. Secondly, since the training efficiency of LSTM is low, we
hope to explore the latest transformer structure in the future to improve the
training efficiency as well as the performance, which has been proved to be an
efficient model to handle sequential data [59, 60], [61].

TABLE VI: Ablation study results of exploring hand-aided attention and the preprocessing strategy in the hotspot [19] model on the OPRA [18] dataset. Dataset |  OPRA [18]
---|---
Method | $K\downarrow$ | $S\uparrow$ | $A\uparrow$ | $N\downarrow$
Hotspot [19] | $1.427$ | $0.362$ | $0.806$ | $0.907$
Select frame & Hotspot [19] | $1.425$ | $0.364$ | $0.802$ | $0.922$
Hotspot [19] & hand attention | $1.422$ | $0.363$ | $0.806$ | $0.924$
Ours | 1.409 | 0.365 | 0.812 | 0.948

$\bullet$ Potential Applications There may be some potential applications of
our method as listed below.

  * 1)

Our method can provide candidate operation areas for robot grasping. The
problem of vision-based robot grasping has been a hot research topic in the
field of robotics [62, 63, 64]. Our approach can learn human grasping habits
by observing human-object interactions, enabling the robot to mimic human
actions in selecting relevant areas that can be manipulated.

  * 2)

Our method can assist visual perception agent to comprehensively understanding
of the scene [65, 16]. For example, the agent need to know the semantic
category of each part in a scene as well as how it interacts with people and
feedbacks the environment. Moreover, it can deliver a goal-oriented
understanding of the object and the environment.

  * 3)

Visual affordance grounding can also be used in virtual reality [66]
applications, where the object affordance heatmap can highlight the regions
where human interacts with an object, together with some virtually displayed
information, e.g., affordance indicators or warning signs.

  * 4)

In the physical domain, object affordance information can be used to improve
object design and manufacture. For example, objects can be made according to
human-object interaction characteristics and habits to facilitate possible
human-object interactions [67].

$\bullet$ Future Research Directions There are several promising directions
for future research on this task.

  * 1)

Dataset: In the future, we will consider constructing a more extensive and
richer dataset that contains videos from both first-person and third-person
perspectives [68], complex backgrounds, diverse interactions, various object
classes, and human pose annotations [69].

  * 2)

Generalization: It is also worthwhile to further investigate how to improve
the generalization ability [70, 71, 72, 73, 74] of affordance grounding
methods, e.g., locating the affordance-related regions precisely among unseen
objects, which is of practical importance [75].

  * 3)

Compatibility: First-person perspective videos can provide a unique viewpoint
of people’s interactions with objects, attention, and even intentions. In
contrast, third-person videos observe human actions from an objective
perspective. Such a difference poses a significant challenge for one-third-
person video compatibility [76, 77]. Therefore, learning better affordance
grounding ability from a dataset containing both first-person and third-person
videos is worth further study.

  * 4)

Transferability: Although third-person videos are more accessible, in cases
such as robot manipulation where the agent observes the environment and
objects from the first-person perspective, it is necessary to transfer the
knowledge learned from third-person videos to first-person scenarios. However,
transferring affordance grounding knowledge between different perspectives
[78] is still under-explored and of practical meaning.

  * 5)

Multi-Information Assistance: The affordance grounding task is related to many
visual factors and to obtain better results in practice we need to investigate
affordance grounding methods with the aid of other information, such as the
introduction of texture cues [79, 80], depth cues [81], human-related
information [69, 82], _etc_.

## References

  * [1] J. J. Gibson, “The theory of affordances,” _The people, place, and space reader_ , 1979.
  * [2] J. Zhang and D. Tao, “Empowering things with intelligence: A survey of the progress, challenges, and opportunities in artificial intelligence of things,” _IEEE Internet of Things Journal_ , 2020.
  * [3] M. Hassanin, S. Khan, and M. Tahtali, “Visual affordance and function understanding: A survey,” _CoRR_ , vol. abs/1807.06775, 2018. [Online]. Available: http://arxiv.org/abs/1807.06775
  * [4] H. Kjellström, J. Romero, and D. Kragic, “Visual object-action recognition: Inferring object affordances from human demonstration,” _Comput. Vis. Image Underst._ , vol. 115, no. 1, pp. 81–90, 2011.
  * [5] S. Qi, S. Huang, P. Wei, and S.-C. Zhu, “Predicting human activities using stochastic grammar,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1164–1172.
  * [6] T.-H. Vu, C. Olsson, I. Laptev, A. Oliva, and J. Sivic, “Predicting actions from static scenes,” in _European Conference on Computer Vision_. Springer, 2014, pp. 421–436.
  * [7] N. Yamanobe, W. Wan, I. G. Ramirez-Alpizar, D. Petit, T. Tsuji, S. Akizuki, M. Hashimoto, K. Nagata, and K. Harada, “A brief review of affordance in robotic manipulation research,” _Advanced Robotics_ , vol. 31, no. 19-20, pp. 1086–1101, 2017.
  * [8] C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. L. Yuille, and L. Fei-Fei, “Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 82–92.
  * [9] H. S. Koppula and A. Saxena, “Physically grounded spatio-temporal object affordances,” in _ECCV_ , ser. Lecture Notes in Computer Science, vol. 8691\. Springer, 2014, pp. 831–847.
  * [10] A. Nguyen, D. Kanoulas, D. G. Caldwell, and N. G. Tsagarakis, “Object-based affordances detection with convolutional neural networks and dense conditional random fields,” in _International Conference on Intelligent Robots and Systems,(IROS)_. IEEE, 2017, pp. 5908–5915.
  * [11] T. Do, A. Nguyen, and I. Reid, “Affordancenet: An end-to-end deep learning approach for object affordance detection,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 5882–5889.
  * [12] J. Sawatzky, A. Srikantha, and J. Gall, “Weakly supervised affordance detection,” in _Conference on Computer Vision and Pattern Recognition, (CVPR)_. IEEE, 2017, pp. 5197–5206.
  * [13] H. Heft, “Affordances and the body: An intentional analysis of gibson’s ecological approach to visual perception,” _Journal for the theory of social behaviour_ , vol. 19, no. 1, pp. 1–30, 1989.
  * [14] X. Zhao, Y. Cao, and Y. Kang, “Object affordance detection with relationship-aware network,” _Neural Computing and Applications_ , 2019.
  * [15] B. Yao, J. Ma, and F. Li, “Discovering object functionality,” in _International Conference on Computer Vision(ICCV)_. IEEE, 2013, pp. 2512–2519.
  * [16] X. Wang, R. Girdhar, and A. Gupta, “Binge watching: Scaling affordance learning from sitcoms,” in _CVPR_ , 2017.
  * [17] X. Li, S. Liu, K. Kim, X. Wang, M.-H. Yang, and J. Kautz, “Putting humans in a scene: Learning affordance in 3d indoor environments,” in _CVPR_ , 2019.
  * [18] K. Fang, T.-L. Wu, D. Yang, S. Savarese, and J. J. Lim, “Demo2vec: Reasoning object affordances from online videos,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [19] T. Nagarajan, C. Feichtenhofer, and K. Grauman, “Grounded human-object interaction hotspots from video,” in _ICCV_ , 2019.
  * [20] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, “Scaling egocentric vision: The epic-kitchens dataset,” in _European Conference on Computer Vision (ECCV)_ , 2018.
  * [21] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv_ , 2018\.
  * [22] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, “Learning efficient convolutional networks through network slimming,” in _ICCV_ , 2017.
  * [23] C. Li and K. M. Kitani, “Pixel-level hand detection in ego-centric videos,” in _Conference on Computer Vision and Pattern Recognition(CVPR)_. IEEE Computer Society, 2013, pp. 3570–3577.
  * [24] A. Betancourt, “A sequential classifier for hand detection in the framework of egocentric vision,” in _Conference on Computer Vision and Pattern Recognition(CVPR)_. IEEE Computer Society, 2014, pp. 600–605.
  * [25] A. Betancourt, P. Morerio, E. I. Barakova, L. Marcenaro, M. Rauterberg, and C. S. Regazzoni, “Left/right hand segmentation in egocentric videos,” _Comput. Vis. Image Underst._ , vol. 154, pp. 73–81, 2017.
  * [26] S. Bambach, S. Lee, D. J. Crandall, and C. Yu, “Lending A hand: Detecting hands and recognizing activities in complex egocentric interactions,” in _International Conference on Computer Vision, (ICCV)_. IEEE Computer Society, 2015, pp. 1949–1957.
  * [27] G. Kapidis, R. Poppe, E. Van Dam, L. Noldus, and R. Veltkamp, “Egocentric hand track and object-based human action recognition,” in _2019 IEEE SmartWorld, Ubiquitous Intelligence Computing, Advanced Trusted Computing, Scalable Computing Communications, Cloud Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI)_ , 2019, pp. 922–929.
  * [28] M. Stark, P. Lies, M. Zillich, J. L. Wyatt, and B. Schiele, “Functional object class detection based on learned affordance cues,” in _Computer Vision Systems, 6th International Conference(ICVS)_ , 2008, pp. 435–444.
  * [29] Y. Sun, S. Ren, and Y. Lin, “Object-object interaction affordance learning,” _Robotics Auton. Syst._ , vol. 62, no. 4, pp. 487–496, 2014.
  * [30] D. Song, N. Kyriazis, I. Oikonomidis, C. Papazov, A. A. Argyros, D. Burschka, and D. Kragic, “Predicting human intention in visual observations of hand/object interactions,” in _International Conference on Robotics and Automation(ICRA)_. IEEE, 2013, pp. 1608–1615.
  * [31] A. Pieropan, C. H. Ek, and H. Kjellström, “Functional object descriptors for human activity modeling,” in _International Conference on Robotics and Automation(ICRA)_. IEEE, 2013, pp. 1282–1289.
  * [32] J. Schulman, J. Ho, C. Lee, and P. Abbeel, “Learning from demonstrations through the use of non-rigid registration,” in _Robotics Research_. Springer, 2016, pp. 339–354.
  * [33] V. Chu, B. Akgun, and A. L. Thomaz, “Learning haptic affordances from demonstration and human-guided exploration,” in _2016 IEEE haptics symposium (HAPTICS)_. IEEE, 2016, pp. 119–125.
  * [34] J. Aleotti and S. Caselli, “Part-based robot grasp planning from human demonstration,” in _2011 IEEE International Conference on Robotics and Automation_. IEEE, 2011, pp. 4554–4560.
  * [35] Y. Zha, S. Bhambri, and L. Guan, “Contrastively learning visual attention as affordance cues from demonstrations for robotic grasping,” _arXiv preprint arXiv:2104.00878_ , 2021.
  * [36] Z. Chen, J. Zhang, and D. Tao, “Recursive context routing for object detection,” _International Journal of Computer Vision_ , pp. 1–19, 2020\.
  * [37] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: towards real-time object detection with region proposal networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2016\.
  * [38] A. Mittal, A. Zisserman, and P. H. S. Torr, “Hand detection using multiple proposals,” in _British Machine Vision Conference, (BMVC)_ , J. Hoey, S. J. McKenna, and E. Trucco, Eds., 2011, pp. 1–11.
  * [39] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _(IEEE) Conference on Computer Vision and Pattern Recognition, (CVPR)_ , 2016, pp. 770–778.
  * [40] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [41] P. Singh, V. K. Verma, P. Rai, and V. P. Namboodiri, “Hetconv: Heterogeneous kernel-based convolutions for deep cnns,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4835–4844.
  * [42] K. Sun, Y. Zhao, B. Jiang, T. Cheng, B. Xiao, D. Liu, Y. Mu, X. Wang, W. Liu, and J. Wang, “High-resolution representations for labeling pixels and regions,” _arXiv preprint arXiv:1904.04514_ , 2019.
  * [43] Y. Wu and K. He, “Group normalization,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [44] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [45] Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand, “What do different evaluation metrics tell us about saliency models?” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 3, pp. 740–757, 2018.
  * [46] M. J. Swain and D. H. Ballard, “Color indexing,” _International journal of computer vision_ , vol. 7, no. 1, pp. 11–32, 1991.
  * [47] T. Judd, K. A. Ehinger, F. Durand, and A. Torralba, “Learning to predict where humans look,” in _ICCV_. IEEE Computer Society, 2009, pp. 2106–2113.
  * [48] R. J. Peters, A. Iyer, L. Itti, and C. Koch, “Components of bottom-up gaze allocation in natural images,” _Vision research_ , vol. 45, no. 18, pp. 2397–2416, 2005.
  * [49] Y. Huang, M. Cai, Z. Li, and Y. Sato, “Predicting gaze in egocentric video by learning task-dependent attention transition,” in _European Conference on Computer Vision (ECCV)_ , 2018.
  * [50] M. Cornia, L. Baraldi, G. Serra, and R. Cucchiara, “A deep multi-level network for saliency prediction,” in _ICPR_ , 2016.
  * [51] M. Kümmerer, T. S. A. Wallis, and M. Bethge, “Deepgaze II: reading fixations from deep features trained on object recognition,” _CoRR_ , vol. abs/1610.01563, 2016.
  * [52] J. Pan, C. Canton, K. McGuinness, N. E. O’Connor, J. Torres, E. Sayrol, and X. a. Giro-i Nieto, “Salgan: Visual saliency prediction with generative adversarial networks,” in _arXiv_ , January 2017.
  * [53] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _ICRL_ , 2015.
  * [54] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, “Imagenet: A large-scale hierarchical image database,” in _Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE Computer Society, 2009, pp. 248–255.
  * [55] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” in _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2018, pp. 839–847.
  * [56] R. Fu, Q. Hu, X. Dong, Y. Guo, Y. Gao, and B. Li, “Axiom-based grad-cam: Towards accurate visualization and explanation of cnns,” _arXiv preprint arXiv:2008.02312_ , 2020.
  * [57] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 3319–3328.
  * [58] G. Montavon, W. Samek, and K.-R. Müller, “Methods for interpreting and understanding deep neural networks,” _Digital Signal Processing_ , vol. 73, pp. 1–15, 2018.
  * [59] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _arXiv preprint arXiv:1706.03762_ , 2017.
  * [60] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” _arXiv preprint arXiv:2101.01169_ , 2021\.
  * [61] R. Girdhar and K. Grauman, “Anticipative video transformer,” _arXiv preprint arXiv:2106.02036_ , 2021.
  * [62] G. Du, K. Wang, S. Lian, and K. Zhao, “Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review,” _Artificial Intelligence Review_ , vol. 54, no. 3, pp. 1677–1734, 2021.
  * [63] K. Fang, Y. Zhu, A. Garg, A. Kurenkov, V. Mehta, L. Fei-Fei, and S. Savarese, “Learning task-oriented grasping for tool manipulation from simulated self-supervision,” _The International Journal of Robotics Research_ , vol. 39, no. 2-3, pp. 202–216, 2020.
  * [64] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” _arXiv preprint arXiv:1703.09312_ , 2017.
  * [65] Y. Zhu, Y. Zhao, and S. Zhu, “Understanding tools: Task-oriented object modeling, learning and recognition,” in _Conference on Computer Vision and Pattern Recognition, (CVPR)_. IEEE, 2015, pp. 2855–2864.
  * [66] E. Fujinawa, S. Yoshida, Y. Koyama, T. Narumi, T. Tanikawa, and M. Hirose, “Computational design of hand-held vr controllers using haptic shape illusion,” in _Proceedings of the 23rd acm symposium on virtual reality software and technology_ , 2017, pp. 1–10.
  * [67] N. Zhao, Y. Cao, and R. W. Lau, “What characterizes personalities of graphic designs?” _ACM Transactions on Graphics (TOG)_ , vol. 37, no. 4, pp. 1–15, 2018.
  * [68] G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and K. Alahari, “Charades-ego: A large-scale dataset of paired third and first person videos,” _arXiv preprint arXiv:1804.09626_ , 2018.
  * [69] J. Zhang, Z. Chen, and D. Tao, “Towards high performance human keypoint detection,” _International Journal of Computer Vision_ , pp. 1–24, 2021\.
  * [70] K. Zhu, W. Zhai, Z.-J. Zha, and Y. Cao, “One-shot texture retrieval with global context metric,” _IJCAI_ , 2019.
  * [71] K. Zhu, Y. Cao, W. Zhai, and Z.-J. Zha, “One-shot texture retrieval using global grouping metric,” _IEEE Transactions on Multimedia_ , 2020.
  * [72] K. Zhu, W. Zhai, and Y. Cao, “Self-supervised tuning for few-shot segmentation,” in _Twenty-Ninth International Joint Conference on Artificial Intelligence IJCAI-20_ , 2020.
  * [73] K. Zhu, Y. Cao, W. Zhai, J. Cheng, and Z.-J. Zha, “Self-promoted prototype refinement for few-shot class-incremental learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6801–6810.
  * [74] W. Wang, W. Zhai, and Y. Cao, “Deep inhomogeneous regularization for transfer learning,” in _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2020, pp. 221–225.
  * [75] H. Luo, W. Zhai, J. Zhang, Y. Cao, and D. Tao, “One-shot affordance detection,” _IJCAI_ , 2021.
  * [76] H. Yu, M. Cai, Y. Liu, and F. Lu, “First-and third-person video co-analysis by learning spatial-temporal joint attention,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [77] G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and K. Alahari, “Actor and observer: Joint modeling of first and third-person videos,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7396–7404.
  * [78] Y. Li, T. Nagarajan, B. Xiong, and K. Grauman, “Ego-exo: Transferring visual representations from third-person to first-person videos,” _arXiv preprint arXiv:2104.07905_ , 2021.
  * [79] W. Zhai, Y. Cao, J. Zhang, and Z.-J. Zha, “Deep multiple-attribute-perceived network for real-world texture recognition,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3613–3622.
  * [80] W. Zhai, Y. Cao, Z.-J. Zha, H. Xie, and F. Wu, “Deep structure-revealed network for texture recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 010–11 019.
  * [81] C. Zhao, Q. Sun, C. Zhang, Y. Tang, and F. Qian, “Monocular depth estimation based on deep learning: An overview,” _Science China Technological Sciences_ , pp. 1–16, 2020.
  * [82] H. He, J. Zhang, Q. Zhang, and D. Tao, “Grapy-ml: graph pyramid mutual learning for cross-dataset human parsing,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 10 949–10 956.