# A Skeleton-aware Graph Convolutional Network for Human-Object Interaction
Detection ††thanks: Emails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>* Corresponding author

Manli Zhu  Department of Computer and Information Sciences, Northumbria
University, Newcastle upon Tyne, UK Edmond S. L. Ho  Department of Computer
and Information Sciences, Northumbria University, Newcastle upon Tyne, UK
Hubert P. H. Shum

###### Abstract

Detecting human-object interactions is essential for comprehensive
understanding of visual scenes. In particular, spatial connections between
humans and objects are important cues for reasoning interactions. To this end,
we propose a skeleton-aware graph convolutional network for human-object
interaction detection, named SGCN4HOI. Our network exploits the spatial
connections between human keypoints and object keypoints to capture their
fine-grained structural interactions via graph convolutions. It fuses such
geometric features with visual features and spatial configuration features
obtained from human-object pairs. Furthermore, to better preserve the object
structural information and facilitate human-object interaction detection, we
propose a novel skeleton-based object keypoints representation. The
performance of SGCN4HOI is evaluated in the public benchmark V-COCO dataset.
Experimental results show that the proposed approach outperforms the state-of-
the-art pose-based models and achieves competitive performance against other
models.

###### Index Terms:

human-object interaction, graph convolutional network, deep learning, human
pose, object skeleton

##  I Introduction

Human-object interaction (HOI) detection is a core problem for in-depth
understanding of visual scenes. The task requires localizing instances (humans
and objects) and predicting their interactions in the form of
$\left<human,action,object\right>$ from a given image. It plays an important
role in analyzing visual scenes, such as visual question answering [1] and
activity analysis [2].

Recent HOI detection approaches can roughly be divided into transformer-based
methods [3, 4, 5] and non-transformer methods[6, 7, 8]. Particularly, the
transformer-based models have shown superior performance, yet they require a
large amount of memory and are difficult to train under limited resources. As
there has been significant progress in object detection such as Faster R-CNN
[9] and DETR [10], many HOI models [11, 12] utilized them to simplify the HOI
detection task, resulting in a good trade-off between performance and
complexity. However, as a human can interact with multiple objects and vice-
versa, it remains challenging for these models to detect HOIs from visual
image features. Human-object pairs with different interactions may share
indistinguishable appearance or spatial configuration, which confuses
classifiers that use such features.

The advantage of modelling the mutual contexts of human pose and object in
addition to their co-occurrence has been demonstrated in the literature [13].
Most existing pose-based methods [14, 15, 16] simply use convolutional
networks for human poses embedding, this can be inadequate for capturing the
fine-grained human skeleton structure that is useful for HOI detection. For
example, the “foot” and “legs” should be paid more attention than other human
body parts in the scene of “kick football”. To address this, Zheng et al. [17]
propose to use graph convolutional networks (GCN) for modelling the structured
connections in human skeletons and the fine-grained interactions between human
keypoints and object keypoints. They simply use two corner points of the
object bounding box, however, either the 2-d points or 4-d bounding box
representation considers only the rectangular spatial scope of an object and
does not account for the shape and pose of semantically important local areas
[18, 19].

In this paper, we propose a skeleton-aware graph convolutional network that
makes use of both human skeleton keypoints and object skeleton keypoints for
HOI detection, namely SGCN4HOI. Our idea is to utilize the keypoints from
skeletons of humans/objects to capture their structural connections and as a
guide to differentiating HOIs with subtle appearances and different
structures. To this end, the graph convolutional network is exploited by
considering keypoints of instances as nodes to capture their fine-grained
spatial interactions, and we refer to this network as a spatial skeleton
stream. In addition, to make full use of the available features, we add a
visual stream that learns visual features from images and a spatial
configuration stream that learns interaction patterns of human-object pairs to
the framework. The two streams have proven to be effective and are commonly
used in existing works [6, 12, 11]. The three streams form our final multi-
stream framework for HOI detection.

We also propose a novel skeleton-based method for representing object
keypoints due to the limited literature on object skeleton and keypoints
representation in HOI detection. Unlike humans that often have fixed skeleton
structure, different kinds of objects usually have different structures, thus
it is difficult to apply a unified algorithm for representing them. The
dominant method for representing objects in object detection and HOI detection
is object bounding box, which is too coarse to represent object’s structure.
Recent research such as [20] propose to replace the bounding box with a convex
hull consisting of 9 points for object representation, yet the main focus of
their work is to make the object area more accurate and it is still supervised
by the bounding box. Han et al. [21] propose to encode the coarse-grained
orientation information for aerial object representation, while the object
structure is not explored. Here, we propose to represent fine-grained object
structure information in an indirect manner, i.e., we exploit a morphological
skeletonization approach [22] that works on binary masks to obtain objects’
skeletons. We extract a set of keypoints whose distribution can represent the
objects’ structural information using the K-means algorithm.

Figure 1: Overview of the framework. Rounded rectangles represent operations,
sharp rectangles with colors represent features, and $\otimes$ denotes
element-wise multiplication. The model consists of three main streams. The
Spatial configuration stream that extracts coarse-grained spatial features
between humans and objects by utilizing the configuration of their bounding
boxes and visual stream that extracts visual features from given images. The
spatial skeleton stream extracts fine-grained spatial interaction features by
modelling human keypoints and object keypoints as nodes and similarities
between nodes plus learnable weights as edges. HOI probabilities from each
stream are multiplied together as the final prediction.

We evaluate our model in the HOI detection benchmark V-COCO dataset. Our
proposed SGCN4HOI achieves state-of-the-art performance on this dataset
compared to pose-based HOI models and other non-transformer methods.

The contributions of this work are summarized as follows:

  * •

We propose a skeleton-aware graph network that considers the skeletons of both
humans and objects to model their fine-grained spatial interactions.

  * •

We propose a novel skeleton-based object keypoints representation method to
preserve object structural information. To the best of our knowledge, this is
the first work that exploits object structure representation in HOI detection.

  * •

We demonstrate the effectiveness of our proposed model by conducting
experiments in the public HOI detection benchmark V-COCO [6] dataset. Our code
is available at https://github.com/zhumanli/SGCN4HOI.

##  II Related Works

In HOI detection, deep learning methods have been widely used due to its rapid
development and superiority in modelling complicated data. In this section, we
review the highly relevant research which focuses on modelling the skeletal
structure of humans and utilizing graph networks.

###  II-A Skeleton-based HOI Detection

Human pose which provides fine-grained cues for HOI detection has been studied
in existing works. Fang et al. [23] propose a pairwise body-part attention
module to guide the network focusing on important body parts of interaction
detection. Wan et al. [15] propose a zoom-in module to utilize the human
skeleton features for mining interaction patterns between humans and objects.
Generally, the human skeleton has been well studied in these methods, while
the object skeleton and object keypoints representation are far less explored.
Although Zheng et al. [17] apply the graph network to model the interactions
between human joints and object keypoints, they simply use two corner points
of the object bounding box. We argue that this is inadequate for object
representation as object sizes and shapes greatly vary (e.g., a football is
different from a horse in either size or structure). In this paper, we propose
a novel skeleton-based object keypoints representation that benefits HOI
detection by better capturing the underlying structure of the objects. This is
motivated by the success of modelling humans using the skeletal structure in
the literature.

###  II-B Graph Networks for Skeleton-based Action Recognition

Graph neural netoworks have received increasing attention in skeleton-based
action recognition due to the natural graphic structure of human body. It was
firstly adopted in [24], which represented human body joints as graph nodes to
capture their correlations. Shi et al. [25] exploit the adaptive adjacency
matrix with a fully connected graph to effectively learn the joint
correlations. Zhang et al. [26] propose to use the similarity between nodes as
an adjacency matrix and inject semantics into human skeleton representation to
guide network learning. In this paper, we bring in these insights of graph
models in action recognition to HOI detection, that is we model both human
keypoints and object keypoints as graph nodes to learn their fine-grained
spatial correlations.

Figure 2: Illustration of object keypoints extraction.

##  III Overview of SGCN4HOI

In this section, we give an overview of our proposed multi-stream framework
for HOI detection. We start with our model architecture (Section III-A),
followed by a brief introduction of the backbone VSGNet (III-B). A detailed
description of our proposed skeleton-aware graph convolution network will be
presented in Section IV.

###  III-A Model Architecture

Our model for human-object interaction detection is a two-stage method that
consists of two main steps: i) detecting instances and ii) predicting
interactions for human-object pairs. First, human/object instances are
detected by object detectors (e.g., Faster R-CNN [9]). Second, we evaluate all
the human-object pairs through the proposed spatial skeleton stream, the
spatial configuration stream, and visual stream respectively, and the final
HOI prediction is obtained by fusing the three stream’s prediction scores. A
graphical illustration of our framework is shown in Figure 1.

The inputs of our model including bounding boxes
$\mathbf{b}_{h}\in\mathbb{R}^{4}$ for human $h\in[1,H]$ and
$\mathbf{b}_{o}\in\mathbb{R}^{4}$ for object $o\in[1,O]$, keypoints
$\mathbf{v}_{h}\in\mathbb{R}^{17\times 2}$ for human and
$\mathbf{v}_{o}\in\mathbb{R}^{9\times 2}$ for object. $H$ and $O$ denote the
number of humans and objects respectively from a given image. Our main focus
is to capture fine-grained spatial correlations between humans and objects by
considering their keypoints as graph nodes. The bounding boxes are obtained
from VSGNet [11] which has included the instance detection using Faster R-CNN
[9]. Similar to previous works [12, 14] that use ground-truth bounding boxes,
we use the ground-truth keypoints of humans, and apply ground-truth
segmentations of objects to extract their keypoints (details are decribed in
Section IV-A).

###  III-B Backbone VSGNet

In order to take advantage of using available features to facilitate HOI
detection, we adopt VSGNet [11] as our backbone network to generate spatial
configuration features and visual features through two streams:

####  III-B1 Spatial Configuration Stream

This stream aims to learn the coarse-grained spatial interactions between
humans and objects from their spatial configuration patterns. The spatial
configurations encode the spatial relationship of humans and objects that help
visual features make better predictions [11, 12]. Given the human bounding box
$\mathbf{b}_{h}$ and object bounding box $\mathbf{b}_{o}$, two binary maps are
firstly generated by allocating zeros to the outside area of the bounding box
in the entire image. By stacking them, a 2-channel spatial configuration map
$\mathbf{S}_{ho}$ is then generated for each human-object pair.

The spatial configuration features $\mathbf{x}_{ho}^{Spa}$ of this stream are
obtained by:

$\mathbf{x}_{ho}^{Spa}=\operatorname{VSGNet}\left(\mathbf{S}_{ho}\right)$ (1)

The interaction probabilities are then obtained:

$\mathbf{p}_{ho}^{Spa}=\sigma\left(\operatorname{FC}\left(\mathbf{x}_{ho}^{Spa}\right)\right)$
(2)

where $\sigma$ is the sigmoid function, $\operatorname{FC}$ is the fully
connected network, and $\mathbf{p}_{ho}^{Spa}$ is the action class
probabilities with the dimension that equals the number of action categories.

####  III-B2 Visual Stream

This stream extracts visual features for human-object pairs from images. It
learns the human visual feature, object visual feature, and the global context
(i.e., the entire image) feature from the human bounding box, object bounding
box, and human-object union bounding box, respectively. Here, the context
provides scene-specific global cues which are helpful for HOI prediction.

The refined visual features by spatial configuration features are obtained:

$\mathbf{x}_{ho}^{Vis}=\operatorname{VSGNet}\left(\mathbf{b}_{h},\mathbf{b}_{o},\mathbf{x}_{ho}^{Spa}\right)$
(3)

We then obtain the interaction prediction $\mathbf{p}_{ho}^{Vis}$ of this
stream as follows:

$\mathbf{p}_{ho}^{Vis}=\sigma\left(\operatorname{FC}\left(\mathbf{x}_{ho}^{Vis}\right)\right)$
(4)

##  IV Skeleton-aware Graph Convolutional Network

As the core of our framework, we propose to capture the fine-grained spatial
correlations between humans and objects by using graph networks with keypoints
of both of them. We first introduce the object keypoints representation, then
the proposed skeleton-aware graph network will be presented.

###  IV-A Object Keypoints Representation

We now introduce how do we extract object keypoints and it consists of three
main steps. Given an image (Figure 2(a)), we fist obtain the ground-truth mask
of the object which is a dog in this example (Figure 2(b)), we take the
following steps to obtain its keypoints:

  1. (i)

Skeleton Extraction. We exploit a morphological skeletonization algorithm [22]
on segmentation to represent object skeletons, aiming to obtain the object
structure information to facilitate the modelling of fine-grained spatial
relations between the object and human. The idea is to utilize the well-
studied object segmentation and available skeletonization algorithms to obtain
an object’s skeleton in an indirect manner. Concretely, we use the scikit-
image library (in Python) that works on binary masks to generate object
skeleton as shown in Figure 2(c).

  2. (ii)

Key-point Set Generation. The ending points and intersecting points of the
skeleton are considered key points since they represent the topology of the
shape of the object. Same as [27], an ending point is a pixel (point) with
only one neighbor, and an intersecting point is a pixel with two or more
neighbors. Specifically, we generate a $9\times 9$ grid centered at each
skeleton pixel and count the number of its adjacent pixels. By doing so, the
key points are then extracted, which form a key-point set (dense blue points
in Figure 2(d)).

  3. (iii)

Keypoints Extraction. In order to better preserve the object skeleton
structural information, we apply the K-means algorithm on the key-point set to
obtain a certain number (9 in our experiment) of keypoints for an object. The
K-means algorithm aims to partition the input data into k partitions
(clusters) and can be used to get an intuition about the structure of the
data. Here we are trying to represent the object’s skeletal structure using a
spare set of keypoints. The K-means clustering is suitable for our goal of
object structure representation. The reason is that it minimizes the intra-
cluster variance of input skeletal points, the resulting finite number of
cluster centers can well represent the local regions that are part of the
object. As shown in Figure 2(e)), by applying K-means, all the points are
clustered into different groups (represented in different colors) with the
cluster centers colored in red. It can be seen that the distribution of the
cluster centers can represent the underlying structure of the object, and they
are stored as the final keypoints.

###  IV-B Skeleton-aware Graph Convolution Network

We construct a spatial graph $G=(V,E)$ with the keypoints of humans and
objects as graph nodes to learn their fine-grained structural connections.
Similarities/affinities between nodes plus learnable weights are the edge
connectivities during graph learning, such that we can have a fully connected
graph to fully exploit the correlations between all the nodes, which has
proved to be effective in action recognition [25, 26].

Specifically, $V_{h}=\left\\{v_{i}|i=1,2,3,\cdots,J\right\\}$ and
$V_{o}=\left\\{v_{i}|i=J+1,J+2,J+3,\cdots,J+K\right\\}$ are the keypoint sets
for human and object separately, where $J$ and $K$ are the number of keypoints
of human and object respectively. The graph node set $V=\left\\{V_{h}\cup
V_{o}\right\\}$, the edge set $E=\left\\{\varnothing\right\\}$ is the initial
input. For the sake of simplicity and readability, a simplified example is
shown in Figure 3 by focusing on a single object keypoint. The object keypoint
not only connects to its intra-class points (green points of this object,
orange dotted connections) but also connects to all the inter-class points
(light blue points of the human, yellow dotted connections).

Figure 3: Illustration of the graph structure. The connections of a single
object keypoint is shown in this example for the sake of simplicity and
readability.

Given the keypoint locations of a human and an object, we obtain their
embeddings which are the inputs of graph convolutions using FC layers with
ReLU activation function:

$\mathbf{f}_{in}=\operatorname{ReLU}\left(\operatorname{FC_{2}}\left(\operatorname{ReLU}\left(\operatorname{FC_{1}}\left(\mathbf{v}_{h/o}\right)\right)\right)\right)$
(5)

Once the embeddings of keypoints are obtained, we implement a residual graph
convolution layer as follows, and its structure is shown in Figure 4.

$\mathbf{f}_{out}=\operatorname{FC}\left(\mathbf{f}_{in}\right)+\operatorname{GCN}\left(\mathbf{f}_{in},\left(\mathbf{A}+\mathbf{B}\right)\right)$
(6)

where $\operatorname{FC}\left(\mathbf{f}_{in}\right)$ is the residual
connection, $\operatorname{GCN}\left(\right)$ is the graph convolution
operation. $(\mathbf{A+B})\in\mathbb{R}^{(J+K)\times(J+K)}$ is the adjacency
matrix, in which $\mathbf{A}$ is fully learnable and its values are
initialized as ones (i.e., adaptively adjust the values during learning), it
can play the same role of the attention mechanism as it indicates the strength
of the connections between two nodes. $\mathbf{B}$ is a data-dependent
adjacency matrix represented by nodes similarities/affinities to determine
whether there is a connection between two nodes and how strong the connection
is [25]. The addition operation between $\mathbf{A}$ and $\mathbf{B}$ provides
more flexibility than the commonly used dot multiplication on the adjacency
matrix, that is if some elements of $\mathbf{A}$ are zeros, it still can
generate connections by $\mathbf{B}$. The nodes similarities are defined as
follows:

$s(i,j)=\theta\left(\mathbf{v}_{i}\right)^{T}\phi\left(\mathbf{v}_{j}\right)$
(7)

where $\theta$ and $\phi$ are transformation functions, which are implemented
by an FC layer. We then stack three GCN layers to obtain the graph feature
$\mathbf{x}_{ho}^{Graph}$.

Figure 4: Illustration of the graph convolution layer with residual
connection.

Visual image features can be helpful for identifying same/different objects,
for instance, different cellphones often share similar colors and shapes, but
they are different from bikes. Thereby we use the visual features to refine
our graph feature:

$\mathrm{x}_{ho}^{RG}=\mathbf{x}_{ho}^{Graph}\otimes\mathbf{x}_{ho}^{Vis}$ (8)

The interaction prediction $\mathbf{p}_{ho}^{Graph}$ of this stream is then
made based on the refined graph feature:

$\mathbf{p}_{ho}^{Graph}=\sigma\left(\operatorname{FC}\left(\mathbf{x}_{ho}^{RG}\right)\right)$
(9)

Finally, we combine the predictions from each stream by multiplying the
probabilities to make the final HOI prediction $\mathbf{p}_{ho}$ similar to
previous works [12, 28, 14, 11]:

$\mathbf{p}_{ho}=\mathbf{p}_{ho}^{Graph}\times\mathbf{p}_{ho}^{Vis}\times\mathbf{p}_{ho}^{Spa}$
(10)

(a) Keypoints distribution represents object’s underlying structure

(b) Keypoints distribution fails for structure representation due to occlusion

Figure 5: Cases of keypoints distribution. Figures from left to right are:
input image, object skeleton, keypoints with K-means clustering.

##  V Experiments

###  V-A Dataset

To evaluate the performance of our model, we conduct experiments in the HOI
benchmark V-COCO [6] dataset, which is a subset of the COCO dataset [29]. It
has a total of 10,346 images with 16,199 human instances, 2533 images are for
training, 2867 images are for validating, and 4946 images are for testing.
Each human is annotated with binary labels indicating one of 29 different
action categories. As four actions have no object associated with humans, and
one action has limited samples, the results on the rest 24 classes are
reported same as previous works [6, 15, 11].

###  V-B Metrics

We evaluate the performance on the commonly used role mean average precision
(mAP) metrics: Scenario 1 and Scenario 2. The two scenarios indicate different
evaluation methods for the cases when objects are occluded. In Scenario 1, the
object bounding box must be empty, and in Scenario 2 the object bounding box
is ignored. A prediction of a human-object pair is considered a true positive
if both the predicted human and object bounding boxes have IoUs greater than
0.5 with ground-truth annotations and the interaction label is correct.

###  V-C Implementation Details

Our output features of all three streams have the size of 512 dimensions,
followed by an FC layer for the classification. The detected bounding boxes
from Faster R-CNN [9] are filtered by setting 0.6 confidence threshold for
humans and 0.3 for objects.

We implement our network based on VSGNet [11] and use Resnet-152 [30] as the
backbone for visual feature extraction. We train our network on the V-COCO
train and validation sets for 60 epochs with Stochastic Gradient Descent (SGD)
optimizer, a batch size of 16, a learning rate of 0.005, a weight decay of
0.0001, and a momentum of 0.9.

###  V-D Comparisons with the State of the Art

We evaluate the performance of our proposed SGCN4HOI model and compare it with
the state-of-the-art pose-based methods and other models in the HOI benchmark
V-COCO dataset. We report the mean Average Precision (mAP) score in two
settings provided in [6].

Table I shows that our model performs on par with or better than the pose-
based methods in Scenario 1 and achieves state-of-the-art performance in
Scenario 2. Compared to those methods, our model adds object structure
information in addition to the human pose. By utilizing GCN to capture their
fine-grained spatial interactions, the results show an improvement in the
performance of HOI detection.

TABLE I: Comparison of performance with pose-based methods. $AP_{role}^{\\#1}$ and $AP_{role}^{\\#2}$ represent the performance under Scenario1 and Scenario2 in V-COCO respectively. Pose-based Method | Backbone | $AP_{role}^{\\#1}$ | $AP_{role}^{\\#2}$
---|---|---|---
RPNN [31] | ResNet-50 | - | 47.5
TIN∗ [14] | ResNet-50 | 48.7 | -
PastaNet [32] | ResNet-50 | 51.0 | 57.5
PMFNet [15] | ResNet-50 | 52.0 | -
PD-Net [23] | ResNet-152 | 52.0 | -
ACP∗ [33] | ResNet-152 | 53.0 | -
FCMNet [34] | ResNet-50 | 53.1 | -
SIGN [17] | ResNet50-FPN | 53.1 | -
SGCN4HOI | ResNet-152 | 53.1 | 57.9

Table II shows the comparison results with other methods. We observe that
SGCN4HOI achieves an improvement of 1.3% mAP and 0.9% mAP on the backbone
VSGNet [11] in Scenario 1 and Scenario 2 respectively, and it outperforms all
the other non-transformer methods as well. Compared to transformer-based
methods [3, 4], SGCN4HOI has much lower computational complexity, it also
achieves better performance than HOI Transformer [3] in Scenario 1 and there
was no Scenario 2 result reported in [3].

TABLE II: Comparison of performance with other methods. Our model achieves a
good trade-off between performance and computational complexity comparing with
transformer-based models.

Other Method | Backbone | $AP_{role}^{\\#1}$ | $AP_{role}^{\\#2}$
---|---|---|---
VSRL [6] | ResNet-50 | 31.8 | -
InteractNet [28] | ResNet-50 | 40.0 | 48.0
iCAN [12] | ResNet-50 | 45.3 | 52.4
VCL [35] | ResNet-101 | 48.3 | -
UnionDet [36] | ResNet-50 | 47.5 | 56.2
IPNet [7] | Hourglass-104 | 51.0 | -
DRG [37] | ResNet-50 | 51.0 | -
VSGNet [11] | ResNet-152 | 51.8 | 57.0
HOI Transformer [3] | ResNet-101 | 52.9 | -
HOTR [4] (transformer method) | ResNet-50 | 55.2 | 64.4
SGCN4HOI | ResNet-152 | 53.1 | 57.9

In Table III, we report the per-class performance of our model and compare it
with methods that reported per-class AP. We can see that SGCN4HOI outperforms
other methods in the majority of classes, particularly in “skateboard-instr”
and “snowboard-instr” classes. We believe that our proposed skeleton-based
object keypoints representation well preserves such visible objects structures
(shown in Figure 5 (a)) which facilitates HOI detection. Notice that some of
the classes such as “eat-instr” perform badly, and we believe this is mainly
caused by two reasons. First, as claimed in VSGNet [11], the objects of class
“eat-instr” are usually small and occluded in the images, thus object
detectors often fail in these cases. Second, as shown in Figure 5 (b), in
these occluded cases, either the generated skeleton or keypoints cannot
represent the object structure well due to the various occlusion
configurations and the reliance on the quality of object segmentation.

TABLE III: Per class mAP comparisons with the existing methods in Scenario 1.

HOI Class |  InteractNet [28] |  iCAN [12] |  VSGNet [11] | SGCN4HOI
---|---|---|---|---
hold-obj | 26.38 | 29.06 | 48.27 | 51.55
sit-instr | 19.88 | 26.04 | 29.9 | 29.55
ride-instr | 55.23 | 61.9 | 70.84 | 68.9
look-obj | 20.2 | 26.49 | 42.78 | 47.56
hit-instr | 62.32 | 74.11 | 76.08 | 78.63
hit-obj | 43.32 | 46.13 | 48.6 | 50.62
eat-obj | 32.37 | 37.73 | 38.3 | 43.63
eat-instr | 1.97 | 8.26 | 6.3 | 3.18
jump-instr | 45.14 | 51.45 | 52.66 | 55.14
lay-instr | 20.99 | 22.4 | 21.66 | 20.86
talk_on_phone | 31.77 | 52.82 | 62.23 | 63.55
carry-obj | 33.11 | 32.02 | 39.09 | 38.09
throw-obj | 40.44 | 40.62 | 45.12 | 53.09
catch-obj | 42.52 | 47.61 | 44.84 | 46.87
cut-instr | 22.97 | 37.18 | 46.78 | 47.81
cut-obj | 36.4 | 34.76 | 36.58 | 37.64
work_on_comp | 57.26 | 56.29 | 64.6 | 69.11
ski-instr | 36.47 | 41.46 | 50.59 | 47.74
surf-instr | 65.59 | 77.15 | 82.22 | 82.81
skateboard-instr | 75.51 | 79.35 | 87.8 | 89.32
drink-instr | 33.81 | 32.19 | 54.41 | 47.99
kick-obj | 69.44 | 66.89 | 69.85 | 77.37
read-obj | 23.85 | 30.74 | 42.83 | 41.57
snowboard-instr | 63.85 | 74.35 | 79.9 | 81.35
Average | 40.0 | 45.3 | 51.8 | 53.1

###  V-E Ablation Studies

####  V-E1 Analysis of the Skeleton Stream

Our overall framework consists of three main streams. To evaluate how our
proposed skeleton stream is affecting the overall performance, each component
is evaluated. We consider the base model as the framework without skeleton
stream, then the parts of human skeleton keypoints and object skeleton
keypoints are evaluated separately.

Figure 6: Distribution of the number of object keypoints that are under 30.

The results are shown in Table IV. We can see that with the addition of object
skeleton keypoints individually, the performance has decreased, while the
performance of the addition of human skeleton keypoints alone increases, and
adding both can further improve the performance. This demonstrates that the
human skeleton is more important than the object skeleton for HOI detection,
and the object skeleton can boost the performance by working together with the
human skeleton. On the one hand, the human skeleton structure is unified and
its poses are versatile, which provide more pose cues for action recognition,
e.g., the skeleton of the “sit” action is different from the “lay” action. On
the other hand, the object structures vary and the same object can be involved
in multiple interactions, for example, a “bed” can be involved in the “sit”
and “lay” actions, in this case, it is hard for classifiers to differentiate
such HOIs without human pose cues.

TABLE IV: Components analysis of the skeleton stream. Components | $AP_{role}^{\\#1}$ | $AP_{role}^{\\#2}$
---|---|---
Visual + Configuration (Base) | 50.5 | 55.1
Visual + Configuration + Skeleton (object) | 48.6 | 54.1
Visual + Configuration + Skeleton (human) | 52.3 | 57.2
Visual + Configuration + Skeleton (both) | 53.1 | 57.9

####  V-E2 Analysis of Parameter

We analyze the affection of the number of object keypoints on the performance,
that is the value k of K-means, and the results are shown in Table V. We can
see that with the increase of the number of keypoints, the model performance
improves gradually. We conclude that when the number of keypoints (e.g., 2 and
4) is too small, it is not enough to represent object structures, while larger
numbers (e.g., 7 and 9) can represent object structures well.

In order to choose the best K parameter, we visualize the distribution of the
number of object keypoints. For the sake of clarity and readability, we only
show the distribution for the range of 0 to 30 (Figure 6) as the largest
number is a few thousands. It can be seen that most of the objects have a
number of keypoints around 8 or 9, and we choose 9 in this work. We do not
wish to have a very large number on K, because we padding zeros for small
objects that have limited points to align with the K value, thereby a large K
would harm the performance due to too many meaningless zeros.

TABLE V: K value analysis of object keypoints representation. K value | $AP_{role}^{\\#1}$ | $AP_{role}^{\\#2}$
---|---|---
2 | 47.5 | 52.6
4 | 51.1 | 56.2
5 | 51.8 | 56.8
7 | 52.3 | 57.2
9 | 53.1 | 57.9

##  VI Conclusion

In this paper, we have proposed an effective skeleton-aware graph network
SGCN4HOI that models fine-grained spatial correlations between human skeleton
keypoints and object skeleton keypoints for HOI detection. Our proposed
skeleton-based object keypoints representation method is able to preserve
objects’ sizes and structural information, which help differentiate objects
that share subtle appearances or have different structures, resulting in
better HOI detection performance. The presented multi-stream framework made
full use of available features including visual features and spatial
configuration features. Additionally, we found that visual features could be
used as attention features to refine graph features that are learned from the
skeleton stream. Experimental results have shown that SGCN4HOI improves the
performance and outperforms the available pose-based state of the arts.

In the future, more robust object keypoints representation can be explored,
and the adaptive representation in object detection such as RepPoint [18] and
CFA [20] can be considered. As discussed in the experimental results, our
approach to object keypoints representation is limited by the quality of
object segmentation, thereby it performs badly in occluded scenes. We hope our
work can be helpful for future explorations on object keypoints
representation.

## References

  * [1] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6904–6913.
  * [2] F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles, “Activitynet: A large-scale video benchmark for human activity understanding,” in _Proceedings of the ieee conference on computer vision and pattern recognition_ , 2015, pp. 961–970.
  * [3] C. Zou, B. Wang, Y. Hu, J. Liu, Q. Wu, Y. Zhao, B. Li, C. Zhang, C. Zhang, Y. Wei, and J. Sun, “End-to-end human object interaction detection with hoi transformer,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 11 820–11 829.
  * [4] B. Kim, J. Lee, J. Kang, E.-S. Kim, and H. J. Kim, “Hotr: End-to-end human-object interaction detection with transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 74–83.
  * [5] M. Tamura, H. Ohashi, and T. Yoshinaga, “Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 10 405–10 414.
  * [6] S. Gupta and J. Malik, “Visual semantic role labeling,” _arXiv preprint arXiv:1505.04474_ , 2015.
  * [7] T. Wang, T. Yang, M. Danelljan, F. S. Khan, X. Zhang, and J. Sun, “Learning human-object interaction detection using interaction points,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 4115–4124.
  * [8] T. He, L. Gao, J. Song, and Y.-F. Li, “Exploiting scene graphs for human-object interaction detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15 984–15 993.
  * [9] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , ser. NIPS’15. Cambridge, MA, USA: MIT Press, 2015, p. 91–99.
  * [10] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [11] O. Ulutan, A. S. M. Iftekhar, and B. S. Manjunath, “Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 13 614–13 623.
  * [12] C. Gao, Y. Zou, and J.-B. Huang, “ican: Instance-centric attention network for human-object interaction detection,” _arXiv preprint arXiv:1808.10437_ , 2018\.
  * [13] B. Yao and L. Fei-Fei, “Modeling mutual context of object and human pose in human-object interaction activities,” in _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2010, pp. 17–24.
  * [14] Y.-L. Li, S. Zhou, X. Huang, L. Xu, Z. Ma, H.-S. Fang, Y. Wang, and C. Lu, “Transferable interactiveness knowledge for human-object interaction detection,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3580–3589.
  * [15] B. Wan, D. Zhou, Y. Liu, R. Li, and X. He, “Pose-aware multi-level feature network for human object interaction detection,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 9468–9477.
  * [16] H.-B. Zhang, Y.-Z. Zhou, J.-X. Du, J.-L. Huang, Q. Lei, and L. Yang, “Improved human-object interaction detection through skeleton-object relations,” _Journal of Experimental & Theoretical Artificial Intelligence_, vol. 34, no. 1, pp. 41–52, 2022.
  * [17] S. Zheng, S. Chen, and Q. Jin, “Skeleton-based interactive graph network for human object interaction detection,” in _2020 IEEE International Conference on Multimedia and Expo (ICME)_ , 2020, pp. 1–6.
  * [18] Z. Yang, S. Liu, H. Hu, L. Wang, and S. Lin, “Reppoints: Point set representation for object detection,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 9656–9665.
  * [19] A. Monroy and B. Ommer, “Beyond bounding-boxes: Learning object shape by model-driven grouping,” in _European Conference on Computer Vision_. Springer, 2012, pp. 580–593.
  * [20] Z. Guo, C. Liu, X. Zhang, J. Jiao, X. Ji, and Q. Ye, “Beyond bounding-box: Convex-hull feature adaptation for oriented and densely packed object detection,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 8788–8797.
  * [21] J. Han, J. Ding, N. Xue, and G.-S. Xia, “Redet: A rotation-equivariant detector for aerial object detection,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 2785–2794.
  * [22] T. Y. Zhang and C. Y. Suen, “A fast parallel algorithm for thinning digital patterns,” _Commun. ACM_ , vol. 27, no. 3, p. 236–239, mar 1984.
  * [23] X. Zhong, C. Ding, X. Qu, and D. Tao, “Polysemy deciphering network for human-object interaction detection,” in _European Conference on Computer Vision_. Springer, 2020, pp. 69–85.
  * [24] S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in _Thirty-second AAAI conference on artificial intelligence_ , 2018.
  * [25] L. Shi, Y. Zhang, J. Cheng, and H. Lu, “Two-stream adaptive graph convolutional networks for skeleton-based action recognition,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 12 018–12 027.
  * [26] P. Zhang, C. Lan, W. Zeng, J. Xing, J. Xue, and N. Zheng, “Semantics-guided neural networks for efficient skeleton-based human action recognition,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 1109–1118.
  * [27] G. Rojas-Albarracín, C. A. Carbajal, A. Fernández-Caballero, and M. T. López, “Skeleton simplification by key points identification,” in _Mexican Conference on Pattern Recognition_. Springer, 2010, pp. 30–39.
  * [28] G. Gkioxari, R. Girshick, P. Dollár, and K. He, “Detecting and recognizing human-object interactions,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8359–8367.
  * [29] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [31] P. Zhou and M. Chi, “Relation parsing neural network for human-object interaction detection,” in _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_. IEEE, 2019, pp. 843–851.
  * [32] Y.-L. Li, L. Xu, X. Liu, X. Huang, Y. Xu, S. Wang, H.-S. Fang, Z. Ma, M. Chen, and C. Lu, “Pastanet: Toward human activity knowledge engine,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 379–388.
  * [33] D.-J. Kim, X. Sun, J. Choi, S. Lin, and I. S. Kweon, “Detecting human-object interactions with action co-occurrence priors,” in _European Conference on Computer Vision_. Springer, 2020, pp. 718–736.
  * [34] Y. Liu, Q. Chen, and A. Zisserman, “Amplifying key cues for human-object-interaction detection,” in _European Conference on Computer Vision_. Springer, 2020, pp. 248–265.
  * [35] Z. Hou, X. Peng, Y. Qiao, and D. Tao, “Visual compositional learning for human-object interaction detection,” in _European Conference on Computer Vision_. Springer, 2020, pp. 584–600.
  * [36] B. Kim, T. Choi, J. Kang, and H. J. Kim, “Uniondet: Union-level detector towards real-time human-object interaction detection,” in _European Conference on Computer Vision_. Springer, 2020, pp. 498–514.
  * [37] C. Gao, J. Xu, Y. Zou, and J.-B. Huang, “Drg: Dual relation graph for human-object interaction detection,” in _European Conference on Computer Vision_. Springer, 2020, pp. 696–712.


#  non-archimedean Fréchet algebras and the loop space of a hypersurface
complement

Emile Bouaziz

###### Abstract.

We study the space, $L\mathbf{A}^{d}_{f}$, of loops into a hypersurface
complement, and show that the corresponding topological algebra of Laurent
series with coefficients in $\mathcal{O}(L\mathbf{A}^{d}_{f})$ is a
topological localisation of $\mathcal{O}(L\mathbf{A}^{d})$. This requires
introducing a small amount of non-Archimedean functional analysis. In
particular we work with topological algebras whose topology is generated by a
family of sub-multiplicative, non-Archimedean semi-norms.

##  1\. introduction

If $X$ is a variety over a field, $k$, of characteristic $0$, then we write
$LX$ for its space of _algebraic loops_. We have by definition
$LX(R)=X(R((z)))$. $LX$ can be thought of as roughly the mapping space
$\operatorname{Map}(D^{*},X)$, where $D^{*}:=\operatorname{spec}(k((z)))$ is
the _formal punctured disc_ , and as such is an algebraic analogue of the
space $\operatorname{Map}(S^{1},M)$ of smooth loops into a manifold. The space
$LX$ has been studied by numerous authors from a variety of perspectives, we
mention here the works [2], [4] and [3] in particular. A familiar feature is
the presence of topologies on the algebras of functions under consideration,
notably the so-called _Tate_ topological $k$-vector space $k((z))$, cf. [2]
for a discussion of Tate objects in algebraic geometry.

If $X$ is affine then $LX$ is representable by an ind-affine scheme with
transition maps closed embeddings. The algebra $\mathcal{O}(LX)$ is thus
naturally a topological $k$-algebra with a basis of open neighbourhoods at $0$
consisting of ideals. We can form then the algebra $\mathcal{O}(LX)\\{z\\}$
consisting of two way infinite Laurent series with Laurent tails tending to
$0$ topologically. There is a natural morphism
$\operatorname{ev}_{X}:\mathcal{O}(X)\rightarrow\mathcal{O}(LX)\\{z\\}$ which
for $X=\mathbf{A}^{1}$ corresponds to the _universal Laurent series_. We can
think of this, informally, as corresponding geometrically to an evaluation map
$\operatorname{Map}(D^{*},X)\times D^{*}\rightarrow X$.

Our goal in this note is to show that, in a sense that we will make precise,
this mapping correspondence is surprisingly local on $X$, at least when $X$ is
open inside an affine space $\mathbf{A}^{d}$ (although we conjecture it is
true for smooth varieties generally). More precisely we will prove, writing
$\mathbf{A}^{d}_{f}$ for the complement of the hypersurface $f=0$, the
following theorem.

######  Theorem 1.1.

$\mathcal{O}(L\mathbf{A}^{d}_{f})\\{z\\}$ is obtained as the Cauchy completion
of the localisation of $\mathcal{O}(L\mathbf{A}^{d})\\{z\\}$ at the element
$\operatorname{ev}_{\mathbf{A}^{d}}(f).$

In order to do this we need to find a convenient category of topological
$k$-algebras to which $\mathcal{O}(L\mathbf{A}^{d})\\{z\\}$ belongs in which
to form the localisation. This is why we will work with non-Archimedean
Fréchet algebras. Let us note that this result stands in stark contrast to the
case of $\mathcal{O}(LX)$ in place of $\mathcal{O}(LX)\\{z\\}$. Firstly we
lack a candidate element at which to localise, as $LX$ does not naturally live
over $X$ because the punctured disc $D^{*}$ has no $k$-points at which to
evaluate a loop. Secondly, if $U$ is open in $X$, then $LU$ is very far from
open in $LX$. Indeed $L\mathbf{A}^{1}$ is a vector space and $L\mathbf{G}_{m}$
is highly non-reduced. Note that in the work [4] a variant of the loop space
is introduced which has some desirable properties that $LX$ lacks, nonetheless
this variant does not have $k$-points $X(k((z)))$, and we will not deal with
it in this note.

##  2\. Preliminary Notions

###  2.1. Loops.

We will now introduce the geometric objects of study. The initiated reader can
safely skim this section so as to fix notation.

######  Definition 2.1.

If $X$ is a $k$-scheme then the functor
$LX:\mathbf{Alg}_{k}\rightarrow\mathbf{Sets}$ is defined by $LX(R):=X(R((z)))$
and referred to as the _loop space_ of $X$.

We have the following standard lemma, the proof of which we include so as to
fix some notation.

######  Lemma 2.1.

If $X$ is a finite type $k$-scheme then $LX$ is representable by an ind-affine
scheme which is a countable colimit along closed embeddings.

###### Proof.

Let $X$ be given as the zero locus of some polynomials $f_{1},...,f_{c}$ in
variables $x^{1},...,x^{d}$. For each $n\geq 0$ consider variables
$x^{i}_{j}$, for $i=1,...,d$ and $j\geq-n$. Write
$x^{i}(z):=\sum_{j\geq-n}x_{j}z^{j}$ with $z$ a formal variable. Let us write
$L^{n}X$ for the closed subset of the affine space with coordinates
$x^{i}_{j}$ as before, subject to the relations implied by the equations of
Laurent series $f_{l}(x^{1}(z),...,x^{d}(z))=0$, for $l=1,...,c$. $L^{n}X$ is
naturally a closed subscheme of $L^{n+1}X$ and we form the ind-scheme
$LX:=\operatorname{colim}L^{n}X$. It is easily seen that this represents the
desired functor. ∎

######  Remark.

  * •

Note that the schemes $L^{n}X$ do not have intrinsic meaning as functors of
$X$, we had to choose an embedding to make sense of them. Nonetheless the
colimit of course has intrinsic meaning, representing as it does the functor
$LX$.

  * •

In the particular case of $\mathbf{A}^{1}$ we have that
$\mathcal{O}(L\mathbf{A}^{1})\cong\operatorname{lim}_{n}k[x_{i}\,|\,i\geq-n]$,
which we will consider as a topological $k$-algebra in the obvious way.

######  Definition 2.2.

Let $A$ be a topological $k$-algebra which is the limit of an inverse system
$(A_{n})_{n}$ of topologically discrete algebras. The topological $k$-algebra
$A\\{z\\}$ is defined as $\operatorname{lim}_{n}A((z))$, with the limit taken
inside topological $k$-algebras.

######  Remark.

Elements of $A\\{z\\}$ are explicitly represented as two way infinite sums
$\sum_{i}a_{i}z^{i}$, such that $a_{-i}\rightarrow 0$ as
$i\rightarrow+\infty$.

######  Definition 2.3.

  * •

For $X$ an affine $k$-scheme we define the topological $k$-algebra
$\mathcal{A}_{X}$ to be $\mathcal{O}(LX)\\{z\\}$.

  * •

The evaluation morphism
$\operatorname{ev}_{\mathbf{A}^{1}}:\mathcal{O}(\mathbf{A}^{1})\rightarrow\mathcal{A}_{\mathbf{A}^{1}}$
is defined by sending $x\in\mathbf{A}^{1}$ to $x(z):=\sum_{i}x_{i}z^{i}$. An
identical formula defines
$\operatorname{ev}_{X}:\mathcal{O}(X)\rightarrow\mathcal{A}_{X}$ for each
affine $X$, noting that the relations defining the algebras are obviously
preserved.

###  2.2. Fréchet Alegrbas.

We wish to study $\mathcal{A}_{X}$ as a topological algebra. Let us note that
is _non-Archimedean_ , which is to say the underlying topological $k$-vector
space has a basis of open neighbourhoods at $0$ consisting of subspaces.
Further more, the underlying topological $k$-vector space is Cauchy complete.
Equivalently, an infinite sum $\sum_{i}a_{i}$ is convergent iff
$a_{i}\rightarrow 0$. We want to be able to perform some familiar algebraic
operations on the algebra $\mathcal{A}_{X}$, in particular to localize at an
element of the image of $\operatorname{ev}_{X}$. We will see how this can be
done in a somewhat ad-hoc manner, that is nonetheless good enough for our
purposes, using non-Archimedean Fréchet algebras over $k$. Of course the
definition we present is heavily inspired by the corresponding notion in
classical functional analysis, as well as some well known constructions in
non-Archimedean analysis, cf. [1] for example. Note that we claim no real
originality in these definitions, our goal is simply to introduce a context
adequate for our purposes.

######  Definition 2.4.

A non-Archimedean $\mathbf{R}$ semi-norm topological algebra $A$ is a
continuous sub- multiplicative map $|-|:A\rightarrow\mathbf{R}_{\geq 0}$ so
that $|a+b|\leq\operatorname{max}\\{|a|,|b|\\}.$ We refer to these as semi-
norms henceforth, and leave implicit that they are non-Archimedean. A family
$|-|_{i\in I}$ of semi-norms on $A$ is said to _induce_ the topology on $A$ if
a sequence $a_{n}\rightarrow 0$ iff $|a_{n}|_{i}\rightarrow 0$ for all $i\in
I$.

######  Remark.

Note that by _sub-multiplicative_ we simply mean that $|fg|\leq|f||g|$. We
require sub-multipicativity as we wish to deal with topological algebras such
as $A((z))$ where $A$ is possibly not an integral domain, so that the
valuation of the product of two elements can potentially be greater than the
sum of the valuations of these elements.

######  Definition 2.5.

A complete topological $k$ algebra is called a non-Archimedean Fréchet algebra
($\mathbf{nAF}_{k}$ henceforth) if it has a countable family of non-
Archimedean semi-norms inducing its topology.

######  Lemma 2.2.

For $X$ a finite type affine scheme the topological $k$-algebra
$\mathcal{A}_{X}$ is an element of $\mathbf{nAF}_{k}$.

###### Proof.

We write $\mathcal{O}(LX)$ as a countable limit, $\lim_{n}A_{n}$, of
topologically discrete algebras. We have for each $n$, a norm on $A_{n}((z))$
which induces its topology by fixing $0<\varepsilon<1$ and setting
$|a|_{n}:=\varepsilon^{\operatorname{val}_{z}(a)}$, where
$\operatorname{ord}_{z}$ denotes $z$-adic order. This induces a family of
$\mathbf{R}$ norms on $\mathcal{A}_{X}$ which are easily seen to induce its
topology, whence we have exhibited $\mathcal{A}_{X}$ as an element of
$\mathbf{nAF}_{k}$. ∎

######  Definition 2.6.

An element $a\in A$, for $A$ in $\mathbf{nAF}_{k}$, is called _good_ if for
all $i\in I$ we have $|a|_{i}\neq 0$ and if further $|a|_{i}^{n}=|a^{n}|_{i}$
for all $i$ and $n$. If this is the case, each of the semi-norms extends
uniquely to the usual algebraic localisation, which we denote
$A[a^{-1}]^{\operatorname{alg}}$. The completion of this with respect to the
topology induced by the family of extended norms will be denoted
$A[a^{-1}]\in\mathbf{nAF}_{k}$.

######  Remark.

We note that (suppressing the index $i$) we require $|a|\neq 0$ so that we can
define $|ba^{-n}|:=|b||a|^{-n}$ and we require $|a^{n}|=|a|^{n}$ so that the
extended norms remain sub-multiplicative, which ensures for example the
continuity of multiplication.

######  Lemma 2.3.

For any $f\in\mathcal{O}(\mathbf{A}^{d})$, the element
$\operatorname{ev}_{\mathbf{A}^{d}}(f)\in\mathcal{O}(L\mathbf{A}^{d})$ is good
in the sense of definition 2.6.

###### Proof.

$\mathcal{O}(L\mathbf{A}^{d})$ is the pro-limit of discrete integral domains,
so the multiplicativity of norms is immediate. It remains to check that
$\operatorname{ev}_{\mathbf{A}^{d}}(f)$ does not vanish modulo the ideal
generated by elements $x_{i}$ with $i\leq-n$, which is clear. ∎

We now identify a class of morphisms at which localisations are well behaved.

######  Definition 2.7.

A morphism $f:A\rightarrow B$ is called bounded above (resp. below) if for all
$i\in I$ there exists $C_{i}>0$ so that $|f(a)|_{i}$ is at most (resp. at
least) $C_{i}|a|_{i}$. If a morphism is bounded above and below we call it a
quasi-isometric embedding, and if we can take $C_{i}=c_{i}=1$ for all $i$ then
it is called an isometric embedding.

######  Remark.

A quasi-isometric embedding is necessarily injective, justifying the name.
Further note that the induced topology on the image agrees with the topology
on the domain.

######  Lemma 2.4.

If $f:A\rightarrow B$ is a quasi-isometric embedding in $\mathbf{nAF}_{k}$,
$a\in A$ is good and $f(a)$ is a unit of $B$, then there is a unique
continuous extension of $f$ to $A[a^{-1}]\rightarrow B$. Further, $A[a^{-1}]$
agrees with the completion of $A[a^{-1}]^{\operatorname{alg}}$ inside $B$,
with respect to the induced topology.

###### Proof.

First note that $f(a)$ is automatically good, so behaves multiplicatively with
respect to norms. We need to check that $a_{n}a^{-n}$ tends to zero, i.e. for
all $i\in I$, $a_{n}a^{-n}$ tends to zero with respect to the semi-norm
$|-|_{i}$, iff $f(a_{n}a^{-i}):=f(a_{n})f(a)^{-n}$ tends to zero. Again this
can be checked on semi-norms, and then the quasi-isometry assumption
immediately implies the claim.∎

######  Lemma 2.5.

If $A\rightarrow B$ is a map of pro-discrete algebras, which is a limit of
injective maps $A_{j}\rightarrow B_{j}$, for discrete $A_{j}$ and $B_{j}$,
then the map in $\mathbf{nAF}_{k}$, $A\\{z\\}\rightarrow B\\{z\\}$ is a quasi-
isometric embedding.

###### Proof.

Some thought confirms shows that it suffices to prove this for a map
$A((z))\rightarrow B((z))$, induced from an injection of discrete algebras
$A\rightarrow B$. This is now immediate, indeed we can take the constants $C$
and $c$ to be $1$, and we have an isometric embedding. ∎

##  3\. Proof of main theorem

We are now in a position to formulate and prove the main theorem of this note.
We recall here that we write $\mathbf{A}^{d}_{f}$ for the complement of the
hypersurface $f=0$ inside $\mathbf{A}^{d}$. We let $x^{1},...,x^{d}$ denote
coordinates on the affine space $\mathbf{A}^{d}$. We consider
$\mathbf{A}^{d}_{f}$ as emebedded into $\mathbf{A}^{d+1}$, with coordinates
$x^{1},...,x^{d},y$, as the vanishing locus of $f(x^{1},...,x^{d})y=1$.

We begin with the following lemma, which is where we crucially have to work
with $\mathbf{A}^{d}$ in place of a general smooth variety.

######  Lemma 3.1.

The morphism
$\mathcal{A}_{\mathbf{A}^{d}}\rightarrow\mathcal{A}_{\mathbf{A}^{d}_{f}}$ is a
quasi-isometric embedding in $\mathbf{nAF}_{k}$.

###### Proof.

By lemma 2.4. above, it suffices to show that the map
$\mathcal{O}(L\mathbf{A}^{d})\rightarrow\mathcal{O}(L\mathbf{A}^{d}_{f})$ can
be written as the limit of a projective system consisting of injections of
topologically discrete algebras. As usual we consider
$\mathcal{O}(L\mathbf{A}^{d})$ as the limit of discrete algebras
$A_{n}:=k[x^{i}_{j}\,|\,i=1,...,d,\,j\geq-n]$. We write then
$B_{n}:=A_{n}[y_{i}\,|\,i\geq-n]/(\operatorname{relations})$, where the
relations are those implied by the identity of Laurent series
$y(z)f(x^{1}(z),...,x^{d}(z))=1$, where as usual we have written $y(z):=\sum
y_{i}z^{i}$. Note that
$\mathcal{O}(L\mathbf{A}^{d}_{f})\cong\operatorname{lim}B_{n}$.

Now a morphism $B_{n}\rightarrow C$ is equivalent to elements $c^{i}(z)\in
z^{-n}C[[z]]$, for $i=1,...,d$, along with an inverse to
$f(c^{1}(z),...,c^{d}(z))$ which also lies in $z^{-n}C[[z]]$. Let
$f_{\operatorname{min}}\in A_{n}$ be the coefficient of the highest power of
$z^{-1}$ occuring in $f(x^{1}(z),...,x^{d}(z))$, and note that
$f_{\operatorname{min}}\neq 0$. Further let
$C_{n}:=A_{n}[f_{\operatorname{min}}^{-1}]$ and note that the map
$A_{n}\rightarrow C_{n}$ is an injection as $A_{n}$ is obviously an integral
domain. Now we observe that there is a map $B_{n}\rightarrow C_{n}$ which
factors the inclusion $A_{n}\rightarrow C_{n}$, indeed
$f(x^{1}(z),...,x^{d}(z))$ is invertible in $C_{n}((z))$ because its lowest
order term is. We deduce thus that $A_{n}\rightarrow B_{n}$ is an inclusion
and thus the lemma is proven. ∎

The main theorem now follows from a pleasant trick involving differential
operators on the punctured disc $D^{*}$, the essential idea is that
differential operators are dense inside all $k$-linear continuous
endomorphisms of $\mathcal{O}(D^{*})=k((z))$. This fact is presumably well
known, but we essentially give a proof below anyway.

######  Theorem 3.2.

The natural map of non-Archimedean Fréchet algebras,
$\mathcal{A}_{\mathbf{A}^{d}}[\operatorname{ev}_{\mathbf{A}^{d}}(f)^{-1}]\rightarrow\mathcal{A}_{\mathbf{A}^{d}_{f}}$,
is an equivalence.

###### Proof.

We know by lemmas 2.4. and 3.1. that the topology on
$\mathcal{A}_{\mathbf{A}^{d}}[\operatorname{ev}_{\mathbf{A}^{d}}(f)^{-1}]$
agrees with the induced topology for the inclusion into
$\mathcal{A}_{\mathbf{A}^{d}_{f}}$. As such we need only show that the image
is dense in $\mathcal{A}_{\mathbf{A}^{d}_{f}}$.

In order to prove this let us first note that the algebra,
$\operatorname{Diff}_{D^{*}}$, of differential operators on $D^{*}$ acts
continuously on $\mathcal{A}_{X}$, simply by acting on powers of $z$ in the
evident manner. We claim now that the sub-algebra spanned by the
$\operatorname{Diff}_{D^{*}}$ span of the image of $\operatorname{ev}_{X}$ is
dense in $\mathcal{A}_{X}$ for any affine finite type $X$. Indeed it will
follow immediately from the case of $\mathbf{A}^{1}$, which we now prove. It
suffices to show that each topological $x_{i}\in\mathcal{A}_{\mathbf{A}^{1}}$
is in the desired topological closure.

We write $\nu:=z\partial_{z}$ and define, for each $j$, a sequence of
differential operators, $(P^{j}_{n})_{n}$ as follows. We set
$P^{j}_{n}:=\lambda^{j}_{n}\prod_{i\in[-n,n]\setminus\\{j\\}}(\nu-i)$, where
$\lambda^{j}_{n}$ are chosen so that $P^{j}_{n}(z^{j})=z^{j}$. We claim now
that $P^{j}_{n}(x(z))\rightarrow x_{j}z^{j}$, as $n\rightarrow\infty$. Indeed
let us note that the difference $P^{j}_{n}(x(z))-x_{j}z^{j}$ is of the form
$\sum_{i}c_{in}x_{i}z^{i}$ for some scalars $c_{in}$, so that $c_{in}=0$ for
all $i\in[-n,n]$. It follows that this is of the form $I_{\leq
n}+z^{n}\mathcal{O}(L\mathbf{A}^{1})[[z]]$, where $I_{\leq n}$ is the ideal
generated by the coordinates $x_{i}$ with $i\leq n$. This difference of course
tends to $0$, whence we are done.

Now to conclude it suffices to show that the image of
$\mathcal{A}_{\mathbf{A}^{d}}[\operatorname{ev}_{\mathbf{A}^{d}}(f)^{-1}]$
inside $\mathcal{A}_{\mathbf{A}^{d}_{f}}$ contains the image of
$\operatorname{ev}_{\mathbf{A}^{d}_{f}}$ and is stable under
$\operatorname{Diff}_{D^{*}}$. For this it suffices to check that it is stable
under $\partial_{z}$, and it is clear how to extend the derivation to the
localisation, so we are done. ∎

######  Remark.

We expect this result to be true for smooth affine schemes more generally. It
is lemma 3.1. which we have been unable to prove in this context.

## References

  * [1] V. Berkovich, Spectral Theory and Analytic Geometry over Non-Archimedean Fields, Mathematical surveys and monographs, no. 33. (AMS.)
  * [2] V. Drinfeld, Infinite Dimensional Vector Bundles in Algebraic Geometry, The Unity of Mathematics. Progress in Mathematics, vol 244. Birkhauser Boston.
  * [3] D. Gaitsgory, N. Rozenblyum, DG indschemes, Perspectives in Representation Theory (Contemporary Mathematics 610)
  * [4] M. Kapranov, E. Vasserot, Vertex Algebras and Formal Loop Space, Publications Mathématiques de l’IHÉS, Tome 100 (2004), pp. 209-269.