# Exploiting CLIP for Zero-shot HOI Detection Requires
Knowledge Distillation at Multiple Levels

Bo Wan Tinne Tuytelaars
ESAT, KU Leuven
<EMAIL_ADDRESS>
###### Abstract

In this paper, we investigate the task of zero-shot human-object interaction
(HOI) detection, a novel paradigm for identifying HOIs without the need for
task-specific annotations. To address this challenging task, we employ CLIP, a
large-scale pre-trained vision-language model (VLM), for knowledge
distillation on multiple levels. Specifically, we design a multi-branch neural
network that leverages CLIP for learning HOI representations at various
levels, including global images, local union regions encompassing human-object
pairs, and individual instances of humans or objects. To train our model, CLIP
is utilized to generate HOI scores for both global images and local union
regions that serve as supervision signals. The extensive experiments
demonstrate the effectiveness of our novel multi-level CLIP knowledge
integration strategy. Notably, the model achieves strong performance, which is
even comparable with some fully-supervised and weakly-supervised methods on
the public HICO-DET benchmark. Code is available at
https://github.com/bobwan1995/Zeroshot-HOI-with-CLIP.

##  1 Introduction

HOI detection aims to identify triplets of $\langle$human, object,
interaction$\rangle$ within the context of a given image, which requires
localization of human and object regions and recognition of their interactive
behavior, e.g., play-basketball. It enables the intelligent system to
understand and interpret human behavior in real-world scenarios, thus playing
an instrumental role in anomalous behavior detection [34, 40], motion tracking
[52, 39] and visual scene understanding [41, 27].

HOI detection has typically been investigated in a fully-supervised learning
paradigm [5, 10, 13, 54, 63], where aligned HOI annotations (i.e. human-object
locations and interaction types) are provided during the training stage, as
illustrated in Fig. 1(a). Despite the high performance due to such
comprehensive annotations, it suffers from the labor-intensive process of
labeling HOI instances. This limitation has led recent studies to transition
towards a weakly-supervised setup [64, 1, 25, 22, 53], where only image-level
HOI categories (without corresponding bounding boxes) are provided for model
learning, as demonstrated in Fig. 1(b). While this setup significantly reduces
the labeling costs and enhances robustness to label errors, it nonetheless
necessitates image-level annotations on HOI datasets. These are still costly
to obtain, often noisy and incomplete. Moreover, annotators may inadvertently
introduce bias. Drawing inspiration from the great success of zero-shot
learning [43, 21, 60, 66, 26, 65], we introduce the new challenging problem of
zero-shot HOI detection, where NO HOI annotations are required for model
learning, as shown in Fig.1(c). Importantly, note how our setup diverges from
previous zero-shot HOI detection frameworks [45, 14, 2, 36, 38], which
predominantly focus on knowledge transfer from observed HOI concepts to unseen
categories (c.f. Sec. 2 for more details). In contrast, this work takes it a
step further by tackling the extreme scenario where none of the HOI categories
have been annotated during training, although we assume the category space is
known.

Figure 1: Comparison on training annotations for different setups: (a) Fully-
supervised; (b) Weakly-supervised; (c) Zero-shot.

Large-scale pre-trained VLM, such as CLIP [43], have shown substantial promise
in various domains of zero-shot learning, including image classification [21,
60, 46, 28, 57, 37, 7], object detection [26, 66], and instance segmentation
[65]. However, extending these models to zero-shot HOI detection poses a
unique challenge, primarily due to the high-level relational understanding
required in this context. To the best of our knowledge, this paper makes the
first effort to propose and tackle this challenging task. A naive solution
might be directly employing CLIP on union regions (the joint areas of human-
object proposals detected by an external object detector) to produce
corresponding HOI scores. However, this approach proves to be suboptimal in
terms of both inference speed and performance, as illustrated in Sec. 4.4.
Alternatively, we leverage the power of CLIP in two ways: i) In terms of model
design, we build a multi-branch network that extracts CLIP-oriented features
on multiple levels, thus incorporating its robust generalization capabilities
into HOI representations. ii) For model learning, we use the CLIP scores
generated from global images and union regions as supervisory signals to train
our multi-branch network.

Figure 2: Overview of our multi-level knowledge distillation.

Prior works [54, 56, 36] have revealed that HOI relations manifest across
various levels, including global images, relational union regions, and
individual human-object instances. Inspired by this insight, our work employs
a multi-branch neural network to leverage the capabilities of CLIP for multi-
level HOI representation learning. As sketched in Fig. 2, the network
architecture comprises a global branch, a union branch, and a human-object
branch. The process begins with computing an HOI embedding by using the CLIP
text encoder to encode HOI label prompts. This embedding is shared across all
branches. Concurrently, we detect human-object proposals with an off-the-shelf
object detector and generate an image feature map with the CLIP visual
encoder. Each branch within our network adopts a similar design, where the HOI
features of global image, union regions, and human-object pairs are extracted
from the image feature map, and then serve to compute scores corresponding to
the HOI embedding. Finally, we bring these branches together using a late
fusion strategy: rather than focusing on merging semantic features, our
approach concentrates on fusing the multi-level HOI scores, offering a
comprehensive view of HOIs on different scales.

To generate supervision for model training, we leverage CLIP to produce
meaningful HOI scores for both global images and local union regions. For
global images, the entire image is re-scaled and fed into the CLIP model to
derive a global HOI score that captures the entire context of the image. Given
that CLIP is pre-trained to understand a wide range of image-text pairs, it
can provide a holistic understanding of visual relationships. Similarly, for
union regions, we extract regions of interest that include both the human and
the object. These regions, capturing more focused and localized interactions,
are separately fed into CLIP to generate local and region-specific HOI scores.
The local HOI scores can be noisy due to the coexistence of multiple HOIs in
the same region or the presence of HOI irrelevant distractions, but they
complement the global supervision signal. We conduct ablative studies on the
supervision strategy in Tab. 4, which demonstrate the application of global
supervision on the global & human-object branch, and local supervision on the
union branch, yields the most effective results.

By incorporating CLIP knowledge on multiple levels for both model design and
learning, our approach substantially enhances the zero-shot detection
capability in a variety of HOI scenarios. In summary, our main contributions
are three-fold:

  * •

We pioneer the challenging task of zero-shot HOI detection, a new learning
setup where no HOI annotations are used during training. This is a significant
leap forward in the field of HOI detection.

  * •

We propose a multi-level knowledge distillation strategy from CLIP for this
task, where we seamlessly incorporate CLIP into the model design for detecting
HOIs on different scales, and capture both global and local contexts to
provide rich supervision for model training.

  * •

Extensive experiments are conducted to verify the effectiveness of our CLIP
integration strategies. Impressively, our method achieves a strong performance
even on par with some fully-supervised and weakly-supervised methods on HICO-
DET benchmarks.

##  2 Related Works

##### HOI detection

Fully-supervised HOI detection has been the most common setup due to its
superior performance. Research in this area generally falls into two
categories: two-stage and one-stage frameworks. Two-stage methods [11, 54, 29,
15, 10, 63, 49, 67, 31, 68] adopt a hypothesize-and-classify strategy, which
first generates a set of human-object proposals with the off-the-shelf object
detector, and then enumerates all possible HOI pairs to classify their
interactions. One-stage methods predict human & object locations and their
interaction types simultaneously in an end-to-end manner, which are currently
dominated by transformer-based architectures [4, 24, 8, 61, 62].

To decrease the reliance on HOI annotations, weakly-supervised HOI detection
is proposed to learn HOIs with only image-level annotations. Due to the lack
of location annotations, current works in this domain adopts the two-stage
framework, and they focus on recognizing HOIs by developing advanced network
structures to encode context [1, 25] and integrating external knowledge for
representation learning [50, 53]. In this work, we propose a novel zero-shot
setup without the need for any manual annotations in HOI detection.

##### Zero-shot HOI detection

As most HOI classes are distributed in a long-tail manner [45, 14, 53] due to
the inherent compositionality of HOIs [38], previous works on zero-shot HOI
detection [45, 42, 14, 36, 3, 20, 18, 19, 56, 32, 38] aim to distill knowledge
from observed HOI concept to unseen classes. They can be categorized into
three scenarios: unseen object, unseen action, and unseen combination. There
are mainly two streams of research for solving this problem. One stream [45,
14, 3, 20, 18] focuses on factorizing the human and object features by
performing disentangled reasoning on verbs and objects, which allows the
composition of novel HOI triplets for training and inference. Another stream
[36, 56, 32, 38] transfers knowledge from knowledge graphs or pre-trained VLM
to recognize unseen HOI concepts. Despite the substantial success of these
approaches in knowledge transfer, they still rely heavily on the base
knowledge provided by the seen HOI categories. In contrast, our setup does not
require any HOI annotations for learning.

Figure 3: A detailed introduction of our method: We design a multi-branch
neural network that incorporates CLIP components to extract multi-level
information for HOI detection, which is supervised by the CLIP scores derived
from global images and local union regions.

##  3 Method

###  3.1 Problem Setup

Formally, zero-shot HOI detection aims to learn an HOI detector that takes an
image $I$ as input and generates a collection of tuples
$\mathcal{O}=\\{(\mathbf{b}_{h},\mathbf{b}_{o},r_{h,o},s_{h,o}^{r})\\}$. Each
tuple corresponds to a HOI instance, where
$\mathbf{b}_{h},\mathbf{b}_{o}\in\mathbb{R}^{4}$ indicate human and object
bounding boxes, $r_{h,o}\in\\{1,...,N\\}$ represents the interaction type
between $\mathbf{b}_{h}$ and $\mathbf{b}_{o}$, and $s_{h,o}^{r}\in\mathbb{R}$
is the confidence score of the detected interaction.

###  3.2 Method Overview

To address the challenging zero-shot HOI detection task, we leverage CLIP for
multi-level knowledge integration. To this end, we exploit the visual and
textual encoders of CLIP to construct a multi-branch network for HOI
representation learning, and use CLIP to generate global and local supervision
for model training.

##### Model Design

Due to the lack of HOI location annotations, we adopt a typical two-stage
formulation [64, 1, 25] for HOI detection: in the first stage, we generate a
group of human proposals $\\{(\mathbf{b}_{h},s_{h})\\}$ and object proposals
$\\{(\mathbf{b}_{o},c_{o},s_{o})\\}$ with an off-the-shelf object detector
[44], where $s_{h},s_{o}\in\mathbb{R}$ are detection scores and
$c_{o}\in\\{1,...,C\\}$ is the object class. In the second stage, we pair up
all human and object proposals and predict the interaction class for each
combination.

In order to infuse the generalization capability of CLIP into the HOI
representation, we design a multi-branch deep network by incorporating CLIP’s
visual and textual encoders, as sketched in Fig. 3. Specifically, the global
branch performs image-level HOI recognition, utilizing the HOI embedding
produced by CLIP textual encoder as a classifier. In parallel, for each
detected human-object pair $(\mathbf{b}_{h},\mathbf{b}_{o})$, a union branch
extracts the contextual cues in their shared region of interest, providing a
comprehensive view of the surrounding environment and potential interactions.
On top of that, a human-object branch focuses on fine-grained HOI features and
encodes the specific relational attributes of the interactive pairs, which are
used to predict their interaction types. All the branches are integrated with
a late fusion strategy, where the HOI scores from different levels are
combined to obtain the final predictions.

##### Model Learning

To train our model, we first employ CLIP on global image and local union
regions to compute the corresponding HOI scores as supervision. Then we apply
global image supervision on the global branch and human-object branch, and
local union supervision on the union branch. Our training procedure can be
viewed as a multi-level knowledge distillation approach from the pre-trained
CLIP model. The primary objective of this strategy is to ensure that the HOI
scores derived from distinct branches align with the CLIP scores.

###  3.3 Model Design

####  3.3.1 CLIP Backbone

CLIP builds a powerful vision-language model by pretraining on large-scale
image-text pairs. It consists of a visual encoder $\mathcal{F}_{V}$ (e.g., a
ResNet [17] or Vision Transformer [9]), a self-attention module
$\mathcal{F}_{ATT}$ and a textual encoder $\mathcal{F}_{T}$ (e.g., a
Transformer [51]), to map the visual and textual inputs to a shared latent
space.

Specifically, for an input image $I$, the visual encoder $\mathcal{F}_{V}$
produces a feature map $\bm{\Gamma}\in\mathbb{R}^{H\cdot W\cdot D}$, where
$H,W,D$ denote the height, width, and depth of $\bm{\Gamma}$, respectively.
Then a self-attention module $\mathcal{F}_{ATT}$ is adopted to encode
$\bm{\Gamma}$ to a feature vector $v\in\mathbb{R}^{D}$: it takes a linear
projection of the average pooling on the spatial dimensions of $\bm{\Gamma}$
as query $Q\in\mathbb{R}^{D}$, and a linear projection of the reshaped feature
columns as key and value $K,V\in\mathbb{R}^{(HW)\cdot D}$ :

$\displaystyle\vspace{-2mm}Q=\mathcal{F}_{Q}(AvgPool$
$\displaystyle(\bm{\Gamma}));\quad
K,V=\mathcal{F}_{K}(\bm{\Gamma}),\mathcal{F}_{V}(\bm{\Gamma})$ $\displaystyle
v$ $\displaystyle=\mathcal{F}_{MHA}(Q,K,V)\vspace{-2mm}$ (1)

where $\mathcal{F}_{Q,K,V}$ are linear projection layers and
$\mathcal{F}_{MHA}$ is a standard multi-head attention module [51], all
incorporated in $\mathcal{F}_{ATT}$.

To leverage CLIP for HOI detection, we utilize CLIP textual encoder
$\mathcal{F}_{T}$ to generate HOI embedding
$\mathcal{W}_{T}\in\mathbb{R}^{N\cdot D}$. In a prompting strategy akin to
CLIP, we adopt a common template ‘a person is {verb}-ing {object}’ to convert
HOI labels into text prompts. For instance, ‘play basketball’ would be
converted to ‘a person is playing basketball’. These sentences are then
processed by $\mathcal{F}_{T}$ to create the HOI embedding $\mathcal{W}_{T}$,
which is used to classify different levels of visual features into
corresponding HOI scores.

##### Modified Visual Encoder

By default, a CLIP model with a ResNet visual encoder downscales the feature
map by a factor of 32. Consequently, the spatial dimension of the feature map
is insufficient for detailed region-specific feature extraction. To adapt this
visual encoder for HOI detection with a larger feature map, we tweak the
original ResNet structure. This is done by discarding the last average pooling
module and adding an upsampling layer, thereby reducing the feature map size
to only 8 times smaller than the image resolution. To maintain compatibility
with the CLIP’s self-attention module for feature aggregation, we implement an
ROI-Align module[16] to resize the cropped feature map to a $7\times 7$ grid
for global images, union regions, and human-object proposals.

####  3.3.2 Global Branch

The global branch adapts the visual encoder and HOI embedding to perform an
image-wise HOI recognition task, thereby capitalizing on the knowledge
embedded within the CLIP model. Firstly, we adopt an ROI-Align on the image
feature map followed by the self-attention module to compute a global feature
vector $v_{g}\in\mathbb{R}^{D}$. Then the global HOI scores
$s_{g}\in\mathbb{R}^{N}$ are predicted by conducting the inner product between
the global vector $v_{g}$ and the HOI embedding $\mathcal{W}_{T}$:
$s_{g}=\textrm{Softmax}(\mathcal{W}_{T}\times v_{g})$, where $\times$ is
matrix multiplication.

####  3.3.3 Union Branch

The union region encapsulates the contextual relationship between humans and
objects, which is crucial in comprehending their context. To exploit these
context cues, we compute the union region $\mathbf{b}_{u}\in\mathbb{R}^{4}$
for each human proposal $\mathbf{b}_{h}$ and object proposal $\mathbf{b}_{o}$,
and extract the corresponding appearance feature $v_{u}\in\mathbb{R}^{D}$ via
RoI-align over the feature map $\bm{\Gamma}$ and self-attention aggregation.
Similar union scores $s_{u}\in\mathbb{R}^{N}$ are computed with HOI embedding:
$s_{u}=\textrm{Softmax}(\mathcal{W}_{T}\times v_{u})$.

####  3.3.4 Human-object Branch

The human-object branch performs a fine-level classification for interaction
pairs. For each human proposal $\mathbf{b}_{h}$ and object proposal
$\mathbf{b}_{o}$, we crop the feature maps from $\bm{\Gamma}$ using RoI-Align,
followed by a self-attention operation to generate their appearance features
$v_{h},v_{o}\in\mathbb{R}^{D}$. We also compute a spatial feature $v_{sp}$ by
encoding the relative positions of their bounding boxes
$(\mathbf{b}_{h},\mathbf{b}_{o})$111For details c.f. Appendix. The holistic
HOI representation $v_{ho}\in\mathbb{R}^{D}$ is an embedding of the human and
object appearance features and their spatial feature:
$v_{ho}=\mathcal{F}_{ho}([v_{h};v_{o};v_{sp}])$, where $[;]$ is the
concatenation operation and $\mathcal{F}_{ho}$ is a multi-layer perceptron
(MLP). Finally, we use the shared HOI embedding to predict the pairwise
interaction scores $s_{ho}\in\mathbb{R}^{N}$ for each human-object
combination: $s_{ho}=\mathcal{W}_{T}\times v_{ho}$

###  3.4 Model Learning with CLIP Supervision

In this section, we first utilize CLIP to generate two types of HOI
supervision that are based on global images and local union regions, and then
design a multi-task loss on various levels to train our multi-branch network.
The overall loss function $\mathcal{L}$ consists of three terms: i) an image-
wise recognition loss $\mathcal{L}_{g}$ to detect global HOIs; ii) a union
loss $\mathcal{L}_{u}$ for identifying contextual regional HOIs; and iii) a
pairwise interaction classification loss $\mathcal{L}_{ho}$ to guide the
learning of instance-specific HOIs. Formally, the overall loss is written as:
$\mathcal{L}=\mathcal{L}_{g}+\mathcal{L}_{u}+\mathcal{L}_{ho}$.

##### CLIP Supervision Generation

To obtain supervision for model learning, we directly employ CLIP on the image
and union regions to generate the corresponding scores on the training set. As
shown in Fig. 3(b), we crop the image to a square region at its center, with
the side length equal to its shortest edge. This region is subsequently
resized to $224\times 224$ pixels and fed into the pre-trained CLIP model to
generate the global supervision $d_{g}\in\mathbb{R}^{N}$. Similarly, for each
union box $\bm{b}_{u}$, we crop the corresponding region in the raw image and
resize it to $224\times 224$ pixels, and then apply CLIP to generate local
union supervision $d_{u}\in\mathbb{R}^{N}$.

##### Image-wise loss $\mathcal{L}_{g}$:

Given the global image HOI scores $s_{g}$ and CLIP supervision $d_{g}$,
$\mathcal{L}_{g}$ is a standard Kullback-Leibler (KL) divergence defined as:
$\mathcal{L}_{g}=\mathcal{D}_{KL}(s_{g}||d_{g})$. Notably, $d_{g}$ and $s_{g}$
are independent predictions from the original CLIP and the global branch,
respectively. The aim of $\mathcal{L}_{g}$ is to align the up-scaled image
feature map with the original CLIP representation, which plays a crucial role
in extracting regional features for the union and human-object branches.

##### Union loss $\mathcal{L}_{u}$:

Similarly, we take a KL divergence on union HOI scores $s_{u}$ and the local
union supervision $d_{u}$ to formulate the union loss:
$\mathcal{L}_{u}=\frac{1}{M}\sum_{m=1}^{M}\mathcal{D}_{KL}(s_{u}^{m}||d_{u}^{m})$.
Here $M$ is the total number of human-object combinations for a given image.

##### Human-object pairwise loss $\mathcal{L}_{ho}$:

Inspired by [55, 53], we adopt a Multiple Instance Learning (MIL) strategy to
train the human-object branch. In detail, we first string together all the
interaction scores to form a bag, denoted as
$S_{ho}=[s_{ho}^{1};...;s_{ho}^{M}]\in\mathbb{R}^{M\cdot N}$, where
$s_{ho}^{m}$ stands for the score of the $m$-th pair. Then we take
maximization over all pairs to obtain the image-wise interaction scores:
$\hat{s}_{ho}=\mathop{max}\limits_{m}S_{ho}$. This step essentially distills
the most representative HOIs from the pairwise predictions, providing a
consolidated representation of image-wise interaction scores that can be
guided by the global CLIP supervision $d_{g}$. Formally, the human-object loss
$\mathcal{L}_{ho}$ is a KL divergence defined on $\hat{s}_{ho}$ and $d_{g}$:
$\mathcal{L}_{ho}=\mathcal{D}_{KL}(\hat{s}_{ho},d_{g})$.

###  3.5 Inference

During the inference stage, we combine multiple scores to obtain the final
interaction score $s_{h,o}^{r}$ for each human-object pair
$(\mathbf{b}_{h},\mathbf{b}_{o})$. It includes the global HOI scores $s_{g}$,
the union score $s_{u}$, the normalized pairwise interaction scores $p_{ho}$
(rather than $s_{ho}$), and the object detection scores $\langle
s_{h},s_{o}\rangle$ as follows:

$s_{h,o}^{r}=s_{g}\cdot s_{u}\cdot p_{ho}\cdot(s_{h}\cdot s_{o})^{\gamma}$ (2)

where $\gamma$ is a hyper-parameter to balance the HOI scores and the object
detection scores.

It is noteworthy that we avoid using the original pairwise interaction score
$s_{ho}$ as it fails to measure the contribution of each pair when multiple
pairs in an image share the same interaction. Instead, we institute a
competitive environment among the pairs by applying a Softmax operation on
$S_{ho}$: $\bar{S}_{ho}=\mathop{\textrm{Softmax}}\limits_{m}(S_{ho})$.
Following this, we derive the normalized pairwise interaction scores
$p_{ho}=\sigma(\hat{s}_{ho})\cdot\bar{s}_{ho}$, where $\bar{s}_{ho}$ is a row
from $\bar{S}_{ho}$ and $\sigma$ is Sigmoid function.

##  4 Experiments

Table 1: Results comparison of different methods on HICO-DET test set (a full
table of results comparison c.f. Appendix). †means re-implementation in [53].
Here FS, WS, and ZS indicate fully-supervised, weakly-supervised, and zero-
shot HOI detection methods, respectively. The notation (D) means the visual
encoder or the detector is pre-trained on dataset D, D$\in\\{$COCO, HICO-DET,
YFCC-15M$\\}$.

S | Methods | Visual Encoder | Detector | HICO-DET (%)
---|---|---|---|---
Full | Rare | Non-Rare
FS | InteractNet [12] | RN50-FPN (COCO) | FRCNN (COCO) | 9.94 | 7.16 | 10.77
iCAN [11] | RN50 (COCO) | FRCNN (COCO) | 14.84 | 10.45 | 16.15
TIN [30] | RN50-FPN (COCO) | FRCNN (COCO) | 17.22 | 13.51 | 18.32
PMFNet [54] | RN50-FPN (COCO) | FRCNN (COCO) | 17.46 | 15.56 | 18.00
HOTR [23] | RN50+Transformer (COCO) | DETR (HICO-DET) | 25.10 | 17.34 | 27.42
QPIC [47] | RN101+Transformer (COCO) | DETR (COCO) | 29.90 | 23.92 | 31.69
GEN-VLKT [32] | RN50+Transformer (HICO-DET) | DETR (HICO-DET) | 33.75 | 29.25 | 35.10
HOICLIP [38] | RN50+Transformer (HICO-DET) | DETR (HICO-DET) | 34.69 | 31.12 | 35.74
WS | Explanation-HOI† [1] | ResNeXt101 (COCO) | FRCNN (COCO) | 10.63 | 8.71 | 11.20
MX-HOI [25] | RN101 (COCO) | FRCNN (COCO) | 16.14 | 12.06 | 17.50
PPR-FCN† [64] | RN50 (YFCC-15M) | FRCNN (COCO) | 17.55 | 15.69 | 18.41
PGBL [53] | RN50 (YFCC-15M) | FRCNN (COCO) | 22.89 | 22.41 | 23.03
ZS | baseline | RN50 (YFCC-15M) | FRCNN (COCO) | 10.48 | 9.45 | 10.78
ours | RN50 (YFCC-15M) | FRCNN (COCO) | 17.12 | 20.26 | 16.18

###  4.1 Datasets and Metrics

We use the public HOI detection dataset HICO-DET to benchmark our model. The
dataset contains 37,633 training images and 9,546 test images. It includes
$C=80$ common objects (the same as MSCOCO [33]) and 117 unique action
categories, together forming $N=600$ HOI categories.

We adopt the mean average precision (mAP) metric [6] for evaluating HOI
detection results. A human-object pair is deemed positive when the predicted
human and object boxes have an IoU of at least 0.5 with their ground truth
boxes, and the HOI class is correctly classified.

###  4.2 Implementation Details

We use an off-the-shelf Faster R-CNN [44] pre-trained on MSCOCO to generate up
to 100 object candidates for each image. It’s crucial to note that we only
keep the detection results and do not re-use the feature maps. We rather
employ a CLIP with a ResNet-50 visual encoder, which is pre-trained on the
YFCC-15M dataset [48], with an image resolution of $224\times 224$. The CLIP
model we used was implemented by OpenCLIP
222https://github.com/mlfoundations/open_clip, which achieved an mAP of 35.5
on zero-shot image-wise HOI recognition on the test set, suggesting that it is
capable of providing a comprehensive holistic understanding of HOIs.

During training, we use a larger image resolution with a minimum edge length
of 384, while maintaining the original aspect ratio of the input images in our
multi-branch network. We freeze the weights of the pre-trained visual encoder
and optimize the remaining modules by AdamW, with a learning rate of 1e-4 and
batch size of 16. The model is trained for 30K iterations on two NVIDIA V100
GPUs. After 15K iterations, the learning rate is decayed by a factor of 10.
Following previous works [63, 30], we set feature dimension $D$ as 1024 and
the detection score weight $\gamma$ as 2.8.

###  4.3 Quantitative Results

As shown in Tab. 1, with our multi-level knowledge integration strategy from
CLIP, our approach achieves 17.12 mAP with ResNet-50, which is on par or even
surpasses some fully-supervised (FS) [12, 11, 30, 54] and weaky-supervised
(WS) [1, 25, 64] methods. For the FS and WS methods, we observe the mAP on
Non-rare classes (i.e., those with more than 10 HOI instance annotations in
the training set) is always higher than on Rare classes. This skew is to be
expected given that HICO-DET is inherently an imbalanced dataset [45, 14].
Models tend to learn frequently occurring patterns for which they have
training supervision. Despite certain HOIs being simpler to learn, their
performance may lag due to the relative lack of supervision, compared to the
more challenging yet annotated HOIs.

Remarkably, we obtain a higher mAP on Rare classes compared to Non-Rare
classes in our results. This consequence stems from the integration of CLIP
for HOI representation learning and model supervision. Firstly, CLIP is pre-
trained on large-scale image-text pairs and has potentially encountered every
imaginable HOI scenario during its pre-training phase. We build our model on
top of CLIP components, which allows us to exploit its strong generalization
capability for learning a better HOI representation. Secondly, when comparing
our results with PGBL [53], which also exploits CLIP for HOI representation
learning and achieves the best performance in a weakly-supervised setting
(i.e., image-level HOI annotations are available). We experience a small drop
on Rare classes (from $22.41\xrightarrow{}20.26$), but a significant drop on
Non-Rare classes (from $23.03\xrightarrow{}16.18$). This disparity suggests
that the learning of Non-Rare HOIs is more reliant on strong annotations,
whereas Rare HOIs can be effectively learned by distilling the ‘dark
knowledge’ from CLIP scores. Consequently, we sidestep issues associated with
the long-tailed distribution.

###  4.4 Ablation Studies

In this section, we mainly assess the effectiveness of each component with
detailed ablation studies on HICO-DET dataset. We first introduce our
baseline, based on which we will answer some interesting questions regarding
the model design and learning.

##### Baseline:

Our baseline model is constructed on top of the human-object branch, where the
human-object representation $v_{ho}$ is used to predict their normalized
interaction scores $s_{ho}$, which are supervised by $\mathcal{L}g$. During
the inference process, the final interaction scores in 2 are recomputed as
$s_{h,o}^{r}=p_{ho}\cdot(s_{h}\cdot s_{o})^{\gamma}$.

Table 2: Comparison of inference speed and performance between baseline,
training-free (TF) approach, and our method. TF* means $d_{g}$ is added to
predict $s^{r}_{ho}$ on top of TF.

Exp | Speed (fps) | mAP (%)
---|---|---
Full | Rare | Non-Rare
base | 56.19 | 10.48 | 9.45 | 10.78
TF | 6.52 | 11.19 | 13.98 | 10.37
TF* | 6.47 | 12.24 | 15.75 | 11.19
ours | 35.64 | 17.12 | 20.26 | 16.18

Table 3: Ablation study of multi-level incorporation on HICO-DET dataset. The
baseline is the human-object (h-o) branch, and we add other branches on top of
it. We denote ’early’ as the early fusion of union features and ’late’ as the
late fusion of union scores.

Exp | Branch | mAP (%)
---|---|---
h-o | union | global | Full | Rare | Non-Rare
0 | ✓ | - | - | 10.48 | 9.45 | 10.78
1 | ✓ | ✓(late) | - | 14.49 | 16.33 | 13.95
2 | ✓ | - | ✓ | 15.84 | 17.91 | 15.21
3 | ✓ | ✓(early) | ✓ | 14.64 | 16.09 | 14.21
4 | ✓ | ✓(late) | ✓ | 17.12 | 20.26 | 16.18

##### Why not a training-free (TF) approach?

In a TF approach, we directly apply CLIP scores on union regions $d_{u}$ along
with object detection scores $\langle s_{h},s_{o}\rangle$ for inference. This
reformulates Eq. 2 as: $s_{h,o}^{r}=d_{u}\cdot(s_{h}\cdot s_{o})^{\gamma}$.

While this straightforward approach only relies on a pre-trained CLIP and
avoids the need for model training, it has two notable drawbacks compared to
our method: (i) It is time-consuming for inference. For each image, the
approach requires forwarding the CLIP model for the image and all union
regions, resulting in $M+1$ forward passes. As indicated in Tab. 2, given the
detected bounding boxes, the inference speed for the TF approach is only 6.52
frames per second (fps) on a single Nvidia V100 GPU with a batch size of 1,
whereas our method achieves an fps of 35.64. (ii) The performance is low, even
enhanced by $d_{g}$ (i.e., TF*). This issue arises because both $d_{g}$ and
$d_{u}$ correspond to large regions, and as such, they fail to specify the
interacted human and object pair. Consequently, the CLIP scores, when directly
used for inference, tend to be noisy. In contrast, our method opts to distill
knowledge from these scores at different levels, leading to a significant
performance increase, from an mAP of 12.24 to 17.12.

##### Does multi-level CLIP knowledge distillation strategy work?

As demonstrated in Tab. 3, the answer is definitively yes. We added the union
branch and the global branch on top of our baseline model (Exp 0). The results
indicate that the union branch enhances the mAP from 10.48 to 14.49 (Exp 0 vs.
1), while the global branch raises the mAP from 10.48 to 15.84 (Exp 0 vs. 2).
When we combine both branches, the mAP sees a more significant improvement,
from 10.48 to 17.12 (Exp 0 vs. 4). These results underscore the effectiveness
of the multi-level CLIP knowledge distillation strategy.

Table 4: Ablation study of different supervision strategies on HICO-DET
dataset. g means supervision from $d_{g}$ and u means supervision from union
region CLIP scores $d_{u}$. g+u indicates using both supervisions. In this
table, the global branch is added for all the experiments and supervised with
$d_{g}$.

Exp | Branch | mAP (%)
---|---|---
union | h-o | Full | Rare | Non-Rare
0 | g | g | 15.05 | 16.76 | 14.55
1 | u | u | 14.22 | 15.00 | 13.99
2 | u | g | 17.12 | 20.26 | 16.18
3 | u | g+u | 15.83 | 18.20 | 15.13
4 | g+u | g | 16.96 | 19.54 | 16.18

Table 5: Generalization to Unseen Categories. We randomly select $N^{\prime}$
HOI categories for model learning.

$N^{\prime}$ | mAP (%)
---|---
Full | Rare | Non-Rare
100 | 15.25 | 18.97 | 14.15
300 | 16.45 | 19.48 | 15.55
600 | 17.12 | 20.26 | 16.18

Figure 4: Visualization of the HOI detection results. We compare our method
with the baseline model based on the relative ranking of the detection scores,
which is presented in percentile format. The percentiles highlighted in green
signify the model’s confident HOI predictions, whereas those in red indicate
negative HOI predictions that the model treats as background.

##### How to incorporate contextual cues from the union region?

In Tab. 3, we also explore two different designs for integrating the
contextual union branch, including an early fusion strategy and a late fusion
strategy. For the early fusion strategy, we concatenate the union feature
$v_{u}$ with human-object appearance features $\langle v_{h},v_{o}\rangle$ and
their spatial encoding $v_{sp}$ to compose $v_{ho}$. The experimental results
(Exp 3 vs. 4) indicate that the early fusion strategy performs considerably
worse than a late fusion strategy (14.64 vs 17.12), and even underperforms Exp
2 where the union branch is not included.

We hypothesize that this is because the union feature, encompassing a large
region, may include other HOIs or background distractions, making it noisy.
These irrelevant features may not offer valuable contextual information for
HOI representation learning, particularly in a zero-shot setup where the
training signals are also somewhat noisy. Conversely, with the late fusion
strategy, the union branch aims to learn context-aware union HOI scores.
Although these union scores cannot precisely describe the specific human-
object pair, they provide some contextual HOI cues that are compatible with
the HOI predictions from the other branches.

##### What kind of supervision works best for different branches?

In Tab. 4, we compare different supervision strategies for each branch. By
default, we supervise the image branch with global supervision $d_{g}$. Then,
we apply $d_{g}$ on the union branch in the same manner as $\mathcal{L}_{ho}$
in Exp 0. This results in an mAP drop from 17.12 to 15.05, indicating that the
union features, being noisy and non-discriminative, are not suitable for a MIL
strategy. Besides, we apply local supervision $d_{u}$ on the human-object
branch in Exp 1 and observe the drop in mAP to 14.22. This is because of the
noisiness of $d_{u}$, as it does not always correspond accurately to a
specific human-object pair.

Experimental results reveal that the mismatch of the supervision signals can
result in a performance drop, and Combining both types of supervision does not
necessarily lead to better results (as seen in Exp 3 and 4).

##### Can our method generalize to unseen categories?

To answer this question, we randomly select $N^{\prime}$ out of $N=600$ HOI
categories on HICO-DET during training. This implies the class dimensions of
predicted scores $\langle s_{g},s_{u},s_{ho}\rangle$ and CLIP supervisions
$d_{g},d_{u}$ are set to $N^{\prime}$. For inference, we evaluate across all
600 categories. As shown in Tab. 5, even though the training is limited to
just 100 categories, we observed only a slight $1.87$ mAP drop compared to the
final model. This indicates that our approach effectively captures common
knowledge that can be shared across all HOI categories.

###  4.5 Qualitative Results

Figure 4 offers a qualitative evaluation of our method. Due to the way we
incorporate multiple scores into our final prediction $s_{ho}^{r}$ in our
methodology, it’s not practical to directly compare the HOI scores with the
baseline. Instead, our visualization and comparison with the baseline model
are based on the relative ranking of the detection scores rather than their
absolute values. Concretely, for each HOI prediction, we present its ranking
amongst all predictions belonging to the same category across the entire test
set. The ranking is exhibited in the form of a percentile (top $p\%$), wherein
a lower percentile value (smaller $p$) signifies the model’s strong confidence
in the positive prediction (represented by green colored numbers).

As depicted in Fig.4(a), both our method and the baseline successfully
identify some Rare HOI classes. However, when the objects are quite small, as
shown in Fig.4(b), our model tends to offer more confident predictions,
attributing this advantage to its ability to factor in contextual cues. For
example, in the top image, our model infers that a baby is likely licking a
spoon due to the context of sitting in a baby chair with residual sauce
visible. In a similar vein, Fig.4(c) showcases situations where objects are
heavily occluded. Despite this challenge, our model manages to discern some
unapparent relationships by considering the overall environment. As an
illustration, in the image at the bottom, a man in an office, engaged in
computer work on the table, is more likely identified by our model as ’sitting
on a chair’, which would be a challenging task for the baseline method.

##  5 Limitations and Future Works

Although inspiring results have been achieved by our method, the zero-shot HOI
detection is far from satisfactory. As an example of its limitations, the top
image in Fig.4(d) shows a typical failure case where our model incorrectly
associates a person with a computer that is considerably distant. This error
can be attributed to the absence of adequate supervision for pairwise
associations. Furthermore, the bottom image exhibits a scenario where our
method struggles to recognize relations when the object is completely
obscured.

A potential area for further exploration based on this work involves the
detection of ambiguous HOI associations. Previous research has investigated
this issue within fully-supervised or weakly-supervised settings [31, 35, 53].
However, transferring these learnings to a zero-shot setup remains a largely
unexplored area. Besides, this study employs a classic CLIP structure for
zero-shot HOI detection. Nonetheless, it is intriguing to explore various
adaptations of CLIP [60, 58, 59] for enhancing performance in this task.

## Acknowledgement

We acknowledge funding from European Research Council under the European
Union’s Horizon 2020 research and innovation program (Grant Agreement No.
101021347) and Flemish Government under the Onderzoeksprogramma Artificiele
Intelligentie (AI) Vlaanderen programme.

## References

  * [1] Federico Baldassarre, Kevin Smith, Josephine Sullivan, and Hossein Azizpour.  Explanation-based weakly-supervised learning of visual relations with graph networks.  In ECCV, 2020.
  * [2] Ankan Bansal, Sai Saketh Rambhatla, Abhinav Shrivastava, and Rama Chellappa.  Detecting human-object interactions via functional generalization.  In AAAI, 2020.
  * [3] Ankan Bansal, Sai Saketh Rambhatla, Abhinav Shrivastava, and Rama Chellappa.  Detecting human-object interactions via functional generalization.  In AAAI, 2020.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [5] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng.  Learning to detect human-object interactions.  In WACV, 2018.
  * [6] Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng.  HICO: A benchmark for recognizing human-object interactions in images.  In ICCV, 2015.
  * [7] Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu.  Altclip: Altering the language encoder in clip for extended language capabilities.  arXiv preprint arXiv:2211.06679, 2022.
  * [8] Leizhen Dong, Zhimin Li, Kunlun Xu, Zhijun Zhang, Luxin Yan, Sheng Zhong, and Xu Zou.  Category-aware transformer network for better human-object interaction detection.  arXiv preprint arXiv:2204.04911, 2022.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [10] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang.  Drg: Dual relation graph for human-object interaction detection.  In ECCV, 2020.
  * [11] Chen Gao, Yuliang Zou, and Jia-Bin Huang.  ican: Instance-centric attention network for human-object interaction detection.  In BMVC, 2018.
  * [12] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He.  Detecting and recognizing human-object interactions.  In CVPR, 2018.
  * [13] Saurabh Gupta and Jitendra Malik.  Visual semantic role labeling.  arXiv preprint arXiv:1505.04474, 2015.
  * [14] Tanmay Gupta, Alexander Schwing, and Derek Hoiem.  No-frills human-object interaction detection: Factorization, layout encodings, and training techniques.  In ICCV, 2019.
  * [15] Tanmay Gupta, Alexander Schwing, and Derek Hoiem.  No-frills human-object interaction detection: Factorization, layout encodings, and training techniques.  In ICCV, 2019.
  * [16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV2017, 2017.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [18] Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao.  Visual compositional learning for human-object interaction detection.  In ECCV, 2020.
  * [19] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao.  Affordance transfer learning for human-object interaction detection.  In CVPR, 2021.
  * [20] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao.  Detecting human-object interaction via fabricated compositional learning.  In CVPR, 2021.
  * [21] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, 2021.
  * [22] Mert Kilickaya and Arnold Smeulders.  Human-object interaction detection via weak supervision.  arXiv preprint arXiv:2112.00492, 2021.
  * [23] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J. Kim.  Hotr: End-to-end human-object interaction detection with transformers.  In CVPR, 2021.
  * [24] Bumsoo Kim, Jonghwan Mun, Kyoung-Woon On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim.  Mstr: Multi-scale transformer for end-to-end human-object interaction detection.  arXiv preprint arXiv:2203.14709, 2022.
  * [25] Suresh Kirthi Kumaraswamy, Miaojing Shi, and Ewa Kijak.  Detecting human-object interaction with mixed supervision.  In WACV, 2021.
  * [26] Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded language-image pre-training.  In CVPR, 2022.
  * [27] Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He.  Bipartite graph network with adaptive message passing for unbiased scene graph generation.  In CVPR, 2021.
  * [28] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  In International Conference on Learning Representations, 2022.
  * [29] Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu.  Detailed 2d-3d joint representation for human-object interaction.  In CVPR, 2020.
  * [30] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu.  Transferable interactiveness knowledge for human-object interaction detection.  In CVPR, 2019.
  * [31] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yan-Feng Wang, and Cewu Lu.  Transferable interactiveness prior for human-object interaction detection.  In CVPR, 2019.
  * [32] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu.  Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection.  arXiv preprint arXiv:2203.13954, 2022.
  * [33] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [34] Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao Gao.  Future frame prediction for anomaly detection – a new baseline.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [35] Xinpeng Liu, Yong-Lu Li, Xiaoqian Wu, Yu-Wing Tai, Cewu Lu, and Chi-Keung Tang.  Interactiveness field in human-object interactions.  arXiv preprint arXiv:2204.07718, 2022.
  * [36] Ye Liu, Junsong Yuan, and Chang Wen Chen.  Consnet: Learning consistency graph for zero-shot human-object interaction detection.  In ACMMM, 2020.
  * [37] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  arXiv preprint arXiv:2112.12750, 2021.
  * [38] Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He.  Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models.  In CVPR, 2023.
  * [39] Hitoshi Nishimura, Satoshi Komorita, Yasutomo Kawanishi, and Hiroshi Murase.  Sdof-tracker: Fast and accurate multiple human tracking by skipped-detection and optical-flow.  arXiv preprint arXiv:2106.14259, 2021.
  * [40] Guansong Pang, Cheng Yan, Chunhua Shen, van den Hengel Anton, and Xiao Bai.  Self-trained deep ordinal regression for end-to-end video anomaly detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [41] Julia Peyre, Ivan Laptev, Cordelia Schmid, and Josef Sivic.  Detecting unseen visual relations using analogies.  In ICCV, 2019.
  * [42] Julia Peyre, Ivan Laptev, Cordelia Schmid, and Josef Sivic.  Detecting unseen visual relations using analogies.  In ICCV, 2019.
  * [43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [44] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  arXiv preprint arXiv:1506.01497, 2015.
  * [45] Liyue Shen, Serena Yeung, Judy Hoffman, Greg Mori, and Li Fei-Fei.  Scaling human-object interaction recognition through zero-shot learning.  In WACV, 2018.
  * [46] Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Anna Rohrbach, Zhe Gan, Lijuan Wang, Lu Yuan, et al.  K-lite: Learning transferable visual models with external knowledge.  In NeurIPS, 2022.
  * [47] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga.  Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information.  In CVPR, 2021.
  * [48] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  arXiv preprint arXiv:1503.01817, 2015.
  * [49] Oytun Ulutan, A S M Iftekhar, and B. S. Manjunath.  Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions.  In CVPR, 2020.
  * [50] Mesut Erhan Unal and Adriana Kovashka.  Weakly-supervised hoi detection from interaction labels only and language/vision-language priors.  arXiv preprint arXiv:2303.05546, 2023.
  * [51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [52] Mrabti Wafae, Baibai Kaoutar, Bellach Benaissa, Oulad Haj Thami Rachid, and Tairi Hamid.  Human motion tracking: A comparative study.  Procedia Computer Science, 148:145–153, 2019.
  * [53] Bo Wan, Yongfei Liu, Desen Zhou, Tinne Tuytelaars, and Xuming He.  Weakly-supervised hoi detection via prior-guided bi-level representation learning.  In ICLR, 2023.
  * [54] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He.  Pose-aware multi-level feature network for human object interaction detection.  In ICCV, 2019.
  * [55] Qinxin Wang, Hao Tan, Sheng Shen, Michael Mahoney, and Zhewei Yao.  MAF: Multimodal alignment framework for weakly-supervised phrase grounding.  In EMNLP, 2020.
  * [56] Mingrui Wu, Jiaxin Gu Gu, Yunhang Shen, Mingbao Lin, Chao Chen, Xiaoshuai Sun, and Rongrong Ji.  End-to-end zero-shot hoi detection via vision and language knowledge distillation.  arXiv preprint arXiv:2204.03541, 2022.
  * [57] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao.  Unified contrastive learning in image-text-label space.  In CVPR, 2022.
  * [58] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  FILIP: Fine-grained interactive language-image pre-training.  In ICLR, 2022.
  * [59] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.  Sigmoid loss for language image pre-training.  arXiv preprint arXiv:2303.15343, 2023.
  * [60] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In CVPR, 2022.
  * [61] Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li.  Mining the benefits of two-stage and one-stage hoi detection.  NeuIPS, 2021.
  * [62] Frederic Z Zhang, Dylan Campbell, and Stephen Gould.  Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer.  arXiv preprint arXiv:2112.01838, 2021.
  * [63] Frederic Z Zhang, Dylan Campbell, and Stephen Gould.  Spatially conditioned graphs for detecting human-object interactions.  In ICCV, 2021.
  * [64] Hanwang Zhang, Zawlin Kyaw, Jinyang Yu, and Shih-Fu Chang.  Ppr-fcn: Weakly supervised visual relation detection via parallel pairwise r-fcn.  In ICCV, 2017.
  * [65] Haotian* Zhang, Pengchuan* Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao.  Glipv2: Unifying localization and vision-language understanding.  arXiv preprint arXiv:2206.05836, 2022.
  * [66] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In CVPR, 2022.
  * [67] Penghao Zhou and Mingmin Chi.  Relation parsing neural network for human-object interaction detection.  In ICCV, 2019.
  * [68] Tianfei Zhou, Wenguan Wang, Siyuan Qi, Haibin Ling, and Jianbing Shen.  Cascaded human-object interaction recognition.  In CVPR, 2020.


# Redefining Relationships in Music

Christian Detweiler
The Hague University of Applied Sciences
Beth Coleman
University of Toronto
Fernando Diaz
Google
Lieke Dom
Google
Chris Donahue
Google
Jesse Engel
Google
Cheng-Zhi Anna Huang
Google
Larry James
Google
Ethan Manilow
Google
Amanda McCroskery
Google
Kyle Pedersen
Google
Pamela Peter-Agbia
Google
Negar Rostamzadeh
Google
Robert Thomas
Marco Zamarato
Google
Bendert Zevenbergen
Google
<EMAIL_ADDRESS>
##  1 Introduction

AI tools increasingly shape how we discover, make and experience music. While
these tools can have the potential to empower creativity, they may
fundamentally redefine relationships between stakeholders, to the benefit of
some and the detriment of others. In this position paper, we argue that these
tools will fundamentally reshape our music culture, with profound effects (for
better and for worse) on creators, consumers and the commercial enterprises
that often connect them. By paying careful attention to emerging Music AI
technologies and developments in other creative domains and understanding the
implications, people working in this space could decrease the possible
negative impacts on the practice, consumption and meaning of music. Given that
many of these technologies are already available, there is some urgency in
conducting analyses of these technologies now. It is important that people
developing and working with these tools address these issues now to help guide
their evolution to be equitable and empower creativity. We identify some
potential risks and opportunities associated with existing and forthcoming AI
tools for music, though more work is needed to identify concrete actions which
leverage the opportunities while mitigating risks.

###  1.1 Positionality statement

This position paper is based on discussions that emerged during a two-day
workshop on sociotechnical considerations of AI in music technology. Given the
scope of the problem considered, we would like to acknowledge our technical
affiliations and academic backgrounds are relatively narrow, with most members
working in a large tech corporation and sharing affiliation with several
select academic institutions in North America and Western Europe.

##  2 Technologies considered

In the workshop we selected a variety of music tools that make use of some
form of AI as a starting point for discussions about broader implications of
this technology. The idea was not to "call out" any shortcomings of these
particular tools, but to ground our discussions in experiences with concrete
tools. The discussions surfaced a number of common attributes. The selected
tools differ in the amount of control or agency they offer users. Whereas some
of the tools generate a complete audio file within a few clicks, others allow
users to modify and run code to influence how the tool generates music. Tools
differ with regard to the visibility of model provenance. For example, in some
tools, users can upload MIDI files to influence music generation or create
models using their own audio. Other tools do not explicitly reference models
or source material in the functions they offer users.

##  3 Actors affected

The tools we examined target a variety of users. They provide non-musicians
with novice-friendly ways of making music. Professional creatives (e.g., video
producers) can also use some of these tools to cheaply and easily create
(royalty-free) music to fit their content. New "AI artists" can use tools to
create new forms of art and in doing so often push tools’ limits and help
improve them as can be seen, for example, in the work of Portrait XO. In some
cases, large corporations create these tools and use them to provide music as
a service. Although consumers might not encounter these tools directly, they
could hear their output as the (generated) soundtrack to a video or podcast,
or new types of music created by new AI artists. In some cases, consumers
might become creators themselves, as can be seen on the Jukebox Discord server
(Jukebox, n.d.). These tools rely on training data. This could come from
living musicians (as "style" data), but also from non-living recorded
musicians. This raises questions about consent for being included in a
dataset, which projects such as Spawning are beginning to address by building
tools to allow artists to opt in or opt out of their data being included in
the training of large AI models and set permissions on how their style and
likeness is used (Dryhurst et al., 2022).

##  4 Opportunities and risks

AI tools are changing how musicians, cultural producers, and listeners relate
to each other and to musical expression more broadly. In our discussions, we
identified a number of opportunities and risks associated with existing and
emerging tools. AI tools create possibilities for new types of roles and
relationships in music. Configurations of performer-instrument-audience or
composer-score-performer are shifting or breaking down with the use of AI in
music (Magnusson, 2019). Such changes in musical practices can alter
established relationship dynamics by expanding the notion of authorship,
potentially diluting it, and correspondingly the connection between the
creator and the listener, and trust in the shared intention of using this
music for a shared societal purpose. There are already examples of artists
pushing the limits of these tools to create new forms of musical expression
and art more broadly, such as Holly Herndon’s Holly+ (Herndon, 2019). Music
making could take on new meanings if these tools incentivize creation of input
for new models.

### Credit, compensation or exposure in or from these tools

That said, AI tools in music redefine the relationships between certain
stakeholders and value captured by certain (other) stakeholders. There are
potential risks to how these technologies will both borrow from and impact
existing musicians, and transfer captured value to other stakeholders (Drott,
2021). Most of the tools we analyzed rely on large amounts of recorded audio
to create the models used to generate new audio. Once these systems are
trained, determining how much the system relies on a given piece of source
material is currently an underexplored topic. Therefore, what "influences" the
system uses when it generates a new piece is difficult to know. As a result,
few tools are transparent about what source material they rely on for specific
outputs. The musicians who "bring in the sounds" rarely receive credit,
compensation or exposure in or from these tools, though exceptions exist. For
example, CoSo by Splice combines bits of existing music, making attribution
clear and allowing musicians to receive money and credit for their work
(Splice, 2022).

### Homogenization of the landscape of music

A further risk is that these tools homogenize the landscape of music. The
tools we examined encapsulate specific ideas about what music is (e.g., music
as a product to be consumed rather than as “musicking” (Small, 1998), specific
economic models, musical aesthetics, values and music theories of specific
cultural contexts. These affect the way people conceive, produce and master
musical work (Magnusson, 2019). “Automated aesthetic judgments” can be used to
shape the way media sound (Sterne and Razlogova, 2021). The volume of music
made with AI tools will potentially outpace that of music made without such
tools. If AI tools lack diverse outputs, the landscape of music might
homogenize quickly, both through more overt influence, such as autotune, and
more subtle constraints on what is generated.

### Changing notions of musicianship, mastery and skill

These tools can also change notions of musicianship, mastery and skill
(Tahiroğlu, 2021). Creating music traditionally requires honing a skill over
time (Gurevich, 2014). Some AI tools have a lower barrier to entry, which can
increase access to novices, but also potentially diminish the perceived value
of musical practice, reducing the public conception of music to a commodity
and musicianship to an automatable task (Morreale, 2021). This could impact
musicians’ ability to economically sustain themselves while they develop and
practice their craft. Alternatively, as more consumers engage with music
creation, it could drive newfound levels of appreciation for the process of
creation and those who dedicate their life to it, similar to how widespread
access to digital audio workstations has increased the musical literacy of the
public and popular video games such as Guitar Hero have created new waves of
fans for technically virtuosic musicians featured in the games.

##  5 Open questions for the community

Based on these issues, we identify six themes that merit further attention.

  * •

Centering artists in the process of developing these tools, and in the tools
themselves

  * •

Creating systems that empower more creativity than they borrow

  * •

Improving interpretability of these systems and their (intermediate) outputs,
and giving credit, exposure and compensation to artists where it is due

  * •

Questioning what parts of the process are being automated and to whose benefit

  * •

Centering new music technologies in ways that are responsive to non-Western
cultures

  * •

Studying the broader and long term impact of these tools

##  6 Conclusion

To leverage opportunities and mitigate risks of AI tools in music, we urgently
need to study these tools and their potential impact. In this paper, we have
consolidated a number of issues that need further work. The discussions
captured here really only scratch the surface and more work is needed to come
up with frameworks and specific recommendations for how to proceed in light of
the risks and opportunities identified here.

## Acknowledgments and Disclosure of Funding

This work has been supported by Google through an unrestricted grant.

## References

Drott, E. (2021). Copyright, compensation, and commons in the music AI
industry. Creative Industries Journal, 14(2), 190-207

Dryhurst, M., Herndon, H., Hoepner, P. and Meyer, J. (2022). Spawning.
https://spawning.ai/

Gurevich, M. (2014). Skill in Interactive Digital Music Systems. In K.
Collins, B. Kapralos, and H. Tessler (Eds.), The Oxford Handbook of
Interactive Audio. Oxford: Oxford University Press.

Herndon, H. (2021). Holly+. https://holly.plus/

Jukebox. (n.d.). Jukebox [Discord server]. Discord.
https://discord.me/jukeboxmusic/

Magnusson, T. (2019). Sonic writing: technologies of material, symbolic, and
signal inscriptions. Bloomsbury Academic.

Morreale, F. (2021). Where Does the Buck Stop? Ethical and Political Issues
with AI in Music Creation. Transactions of the International Society for Music
Information Retrieval, 4(1), 105–113.

Small, C. (1998). Musicking: The meanings of performing and listening.
Wesleyan University Press.

Splice. (2022). CoSo by Splice. (Version 1.3.7) [Mobile App]. Google Play.
https://play.google.com/store/apps/details?id=com.splice.stacksmobile&hl=en&gl=US1

Sterne, J. and Razlogova, E. (2021) Tuning sound for infrastructures:
artificial intelligence, automation, and the cultural politics of audio
mastering, Cultural Studies, 35(4-5), 750-770

Tahiroğlu, K. (2021). Ever-shifting roles in building, composing and
performing with digital musical instruments, Journal of New Music Research,
50(2), 155-164