###### Abstract

The Lagrange formalism is developed for Bateman oscillators, which include
both damped and amplified systems, and a novel method to derive the Caldirola-
Kanai and null Lagrangians is presented. For the null Lagrangians,
corresponding gauge functions are obtained. It is shown that the gauge
functions can be used to convert the undriven Bateman oscillators into the
driven ones. Applications of the obtained results to quantizatation of the
Bateman oscillators are briefly discussed.

Bateman Oscillators: Caldirola-Kanai and Null Lagrangians and Gauge Functions

L.C. Vestal and Z. E. Musielak

##  1 Introduction

The Bateman model consists of two uncoupled oscillators, damped or time-
forward and amplified or time-reversed [1], which are called here the Bateman
oscillators. The equations of motion for these oscillators are derived from
one Lagrangian that is known as the Bateman Lagrangian [1,2] and used in
studies of the damped harmonic oscillator and its quantization [3,4]. General
solutions of the equations of motion for the Bateman model are well-known and
given in terms of elementary functions [5].

We solve the inverse problem of the calculus of variations [6] for the Bateman
oscillators and develop a novel method to derive the standard and null
Lagrangians. The standard Lagrangians (SLs) contain the square of the first
order derivative of the dependent variable (the kinetic energy like term) and
the square of dependent variable (the potential energy like term), and among
the derived SLs, the Caldirola-Kanai (CK) Lagrangian is obtained [7,8] and its
validity to describe the Bateman oscillators is discussed. The null (or
trivial) Lagrangians (NLs), when substituted into the Euler-Lagrange (EL)
equation, make this equation vanish identically, which means that no equation
of motion is obtained from the NLs. The NLs are also required to be expressed
as the total derivative of a scalar function [9-13], which is called a gauge
function [14,15].

Our objective is to derive the SLs, NLs and their gauge functions (GFs) for
the Bateman oscillators. Since these oscillators are non-conservative systems,
the derived Lagrangians are not consistent with the Helmholtz conditions
[16,2], which guarantee the existence of Lagrangians for the conservative
systems. We present a different view on these conditions and their validity
for damped (amplified) oscillators. One obtained SL is the CK Lagrangian,
which is well-known, and all other NLs and GFs that are derived simultaneously
with the CK Lagrangian are new. The CK Lagrangian is modified by taking into
account the GFs and it is shown that this modification allows for the
conversion of the undriven Bateman oscillators into driven ones, which is our
main result.

The outline of the paper is as follows: the equations of motion for the
Bateman oscillators, their standard and null Lagrangians, and the gauge
functions are derived in Section 2; derivation of the modified Caldirola-Kanai
Lagrangian and the conversion of undriven oscillatory systems and driven ones
are presented and discussed in Section 3; our conclusions are given in Section
4.

##  2 Lagrangians and gauge functions for Bateman oscillators

###  2.1 Equations of motion

The original paper by Bateman [1] considers a non-conservative system for
which the following Lagrangian is proposed

$L_{B}[\dot{x}(t),\dot{y}(t),x(t),y(t)]=m\dot{x}(t)\dot{y}(t)+{\gamma\over
2}\left[x(t)\dot{y}(t)-\dot{x}(t)y(t)\right]$ $\hskip 122.85876pt-kx(t)y(t)\
,$ (1)

where $x(t)$ and $y(t)$ are coordinate variables and $\dot{x}(t)$ and
$\dot{y}(t)$ are their derivatives with respect to time $t$. In addition, $m$
is mass and $\gamma$ and $k$ are damping and spring constants, respectively.
This Lagrangian is now known as the Bateman Lagrangian [2-4] and its
substitution into the EL equation for $y(t)$ and $x(t)$ gives respectively the
resulting equation of motion for the damped oscillator

$m\ddot{x}(t)+\gamma\dot{x}(t)+kx(t)=0\ ,$ (2)

and the equation of motion for the amplified oscillator

$m\ddot{y}(t)-\gamma\dot{y}(t)+ky(t)=0\ .$ (3)

The equations are uncoupled but they are related to each other by the
transformation $[x(t),y(t),\gamma]\rightarrow[y(t),x(t),-\gamma]$, which
allows replacing Eq. (2) by Eq. (3) and vice versa.

The equations of motion for the damped and amplified oscillators are also
considered in this paper but as two unrelated dynamical systems for which the
SLs, NLs and GFs are independently derived; once the SLs are obtained for both
oscillators, they will be compared to the Bateman Lagrangian given by Eq. (1).

Let us define $b=\pm\gamma/m$ and $c=k/m=\omega^{2}_{o}$, where $\omega_{o}$
is the characteristic frequency of the oscillators, and write Eqs (2) and (3)
as one equation of motion

$\ddot{x}(t)+b\dot{x}(t)+\omega_{o}^{2}x(t)=0\ ,$ (4)

with the understanding that the damped and amplified oscillators require $b>0$
and $b<0$, respectively, and that the variable $x(t)$ describes either damped
or amplified oscillator. Let $\hat{D}=d^{2}/dt^{2}+bd/dt+\omega_{o}^{2}$ be a
linear operator, then Eq. (4) can be written in the compact form
$\hat{D}x(t)=0$.

In the following, we derive the SLs, NLs and GFs for the Bateman oscillators
whose equations of motion are $\hat{D}x(t)=0$.

###  2.2 Novel method to derive standard and null Lagrangians

The first-derivative term in Eq. (4) can be removed by using the standard
transformation of the dependent variable [17]. The transformation is

$x(t)=x_{1}(t)e^{-bt/2}\ ,$ (5)

where $x_{1}(t)$ is the transformed dependent variable, and it gives

$\ddot{x}_{1}(t)+\left(\omega_{o}^{2}-{1\over 4}b^{2}\right)x_{1}(t)=0\ .$ (6)

Despite the fact that the first derivative term is removed, the coefficient
$b$ is still present in the transformed equation of motion. However, if $b=0$,
then $x_{1}(t)=x(t)$ and Eq. (6) becomes the equation of motion for a undamped
harmonic oscillator [18,19].

The standard Lagrangian for this equation is

$L_{s}[\dot{x}_{1}(t),x_{1}(t)]={1\over
2}\left[\left(\dot{x}_{1}(t)\right)^{2}-\left(\omega_{o}^{2}-{1\over
4}b^{2}\right)x_{1}^{2}(t)\right]\ ,$ (7)

and its substitution into the EL equation gives Eq. (6). Let us now follow
[20,21] and consider the following Lagrangian

$L_{n}[\dot{x}_{1}(t),x_{1}(t),t]=C_{1}\dot{x}_{1}(t)x_{1}(t)+C_{2}\left[\dot{x}_{1}(t)t+x_{1}(t)\right]+C_{4}\dot{x}_{1}(t)+C_{6}\
,$ (8)

where $C_{1}$, $C_{2}$, $C_{4}$ and $C_{6}$ are constants. It is easy to
verify that $L_{n}[\dot{x}_{1}(t),x_{1}(t),t]$ is the null Lagrangian with the
constants being arbitrary, and that this NL is constructed to the lowest order
of its dynamical variables [20,21]. The NL can be added to
$L_{s}[\dot{x}_{1}(t),x_{1}(t)]$ without changing the form of the equation of
motion (see Eq. 6) resulting from it.

Since the original equations of motion for the Bateman oscillators depend on
the dynamical variable $x(t)$ not $x_{1}(t)$ (see Eqs LABEL:eq62 and 3), we
now use the inverse transform given by Eq. (5) to convert the variable
$x_{1}(t)$ into $x(t)$ in both $L_{s}[\dot{x}_{1}(t),x_{1}(t)]$ and
$L_{n}[\dot{x}_{1}(t),x_{1}(t),t]$. The resulting Lagrangian is

$L[\dot{x}(t),x(t),t]=L_{CK}[\dot{x}(t),x(t),t]+L_{n}[\dot{x}(t),x(t),t]\ ,$
(9)

where

$L_{CK}[\dot{x}(t),x(t),t]={1\over
2}\left[\left(\dot{x}(t)\right)^{2}-\omega_{o}^{2}x^{2}(t)\right]e^{bt}\ ,$
(10)

is the Caldirola-Kanai (CK) Lagrangian [7,8], derived here independently;
comparison of the CK and Bateman Lagrangians (Eqs 10 and 1) shows significant
differences between them. The presented method gives the following null
Lagrangian

$L_{n}[\dot{x}(t),x(t),t]=\sum_{i=1}^{3}L_{ni}[\dot{x}(t),x(t),t]\ ,$ (11)

where the partial null Lagrangians are

$L_{n1}[\dot{x}(t),x(t),t]=\left(C_{1}+{1\over
2}b\right)\left[\dot{x}(t)+{1\over 2}bx(t)\right]x(t)e^{bt}\ ,$ (12)
$L_{n2}[\dot{x}(t),x(t),t]=C_{2}\left[\left(\dot{x}(t)+{1\over
2}bx(t)\right)t+x(t)\right]e^{bt/2}\ ,$ (13)

and

$L_{n3}[\dot{x}(t),x(t),t]=C_{4}\left[\dot{x}(t)+{1\over
2}bx(t)\right]e^{bt/2}+C_{6}\ ,$ (14)

where $C_{1}$, $C_{2}$, $C_{4}$ and $C_{6}$ are arbitrary but their physical
units are different to make all partial null Lagrangians having the same units
of energy. These are new null Lagrangians for the Bateman oscillators. The
fact that $L_{n}[\dot{x}(t),x(t),t]$ and its partial Lagrangians are the NLs
can be shown by defining $\hat{EL}$ to be the differential operator of the EL
equation and verifying that $\hat{EL}(L_{n})=0$ as well as
$\hat{EL}(L_{ni})=0$. It must be also noted that $b=0$ reduces
$L_{n}[\dot{x}(t),x(t),t]$ to the previously obtained null Lagrangian [15].

###  2.3 Gauge functions and general null Lagrangians

For each partial null Lagrangian, its corresponding partial gauge function can
be obtained and the results are

$\phi_{n1}[x(t),t]={1\over 2}\left(C_{1}+{1\over 2}b\right)x^{2}(t)e^{bt}\ ,$
(15) $\phi_{n2}[x(t),t]=C_{2}x(t)te^{bt/2}\ ,$ (16)

and

$\phi_{n3}[x(t),t]=C_{4}x(t)e^{bt/2}+C_{6}t\ .$ (17)

These partial gauge functions can be added together to form the gauge function

$\phi_{n}[x(t),t]=\sum_{i=1}^{3}\phi_{ni}[x(t),t]\ .$ (18)

The derived gauge function and partial gauge functions reduce to those
previously obtained [15] when $b=0$ is assumed.

The above partial gauge functions are obtained to the lowest order of the
dynamical variables for the Bateman oscillators. Therefore, the only way to
generalize the gauge functions given by Eqs. (15), (16) and (17), without
changing the order of their dynamical variables, is to replace the constants
$C_{1}$, $C_{2}$, $C_{4}$ and $C_{6}$ by the corresponding functions
$f_{1}(t)$, $f_{2}(t)$, $f_{4}(t)$ and $f_{6}(t)$, which must be continuous
and at least twice differentiable and depend only on the independent variable
$t$. In order to obey the assumption that our method of constructing null
Lagrangians is limited to the lowest order of the dynamical variables, the
functions cannot depend on the space coordinates. An interesting result is
that the functions give additional degrees of freedom in constructing the null
Lagrangians and their gauge functions.

We follow [20,21] and call these GFs the general GFs and write them here as

$\phi_{gn1}[x(t),t]={1\over 2}\left[f_{1}(t)+{1\over 2}b\right]x^{2}(t)e^{bt}\
,$ (19) $\phi_{gn2}[x(t),t]=f_{2}(t)x(t)te^{bt/2}\ ,$ (20)

and

$\phi_{gn3}[x(t),t]=f_{4}(t)x(t)e^{bt/2}+f_{6}(t)t\ .$ (21)

The general gauge function is obtained by adding these partial gauge functions

$\phi_{gn}[x(t),t]=\sum_{i=1}^{3}\phi_{gni}[x(t),t]\ .$ (22)

The results given by Eqs. (19) through (22) are new general gauge functions
for the Bateman oscillators and they generalize those previously obtained for
Newton’s law of inertia [20] and for linear undamped oscillators [21] for
which $b=0$.

Using the general gauge function $\phi_{gn}[x(t),t]$, the resulting general
null Lagrangian can be calculated and the result is

$L_{gn}[\dot{x}(t),x(t),t]=\sum_{i=1}^{3}L_{gni}[\dot{x}(t),x(t),t]\ ,$ (23)

where the partial null Lagrangians are given by

$L_{gn1}[\dot{x}(t),x(t),t]=\left[f_{1}(t)+{1\over
2}b\right]\left[\dot{x}(t)+{1\over 2}bx(t)\right]x(t)e^{bt}$ $\hskip
90.3375pt+{1\over 2}\dot{f}_{1}(t)x^{2}(t)e^{bt}\ ,$ (24)
$L_{gn2}[\dot{x}(t),x(t),t]=\left[f_{2}(t)\dot{x}(t)+\dot{f}_{2}(t)x(t)\right]te^{bt/2}$
$\hskip 90.3375pt+\left(1+{1\over 2}bt\right)f_{2}(t)x(t)e^{bt/2}\ ,$ (25)

and

$L_{gn3}[\dot{x}(t),x(t),t]=\left[f_{4}(t)\dot{x}(t)+\dot{f}_{4}(t)x(t)\right]e^{bt/2}+{1\over
2}bf_{4}(t)x(t)e^{bt/2}$ $\hskip
90.3375pt+\left[\dot{f}_{6}(t)t+f_{6}(t)\right]\ .$ (26)

The obtained results show that the replacement of the constants $C_{1}$,
$C_{2}$, $C_{4}$ and $C_{6}$ by the functions $f_{1}(t)$, $f_{2}(t)$,
$f_{4}(t)$ and $f_{6}(t)$ gives the Lagrangian $L_{gn}[\dot{x}(t),x(t),t]$,
which is the null Lagrangian that significantly generalizes
$L_{n}[\dot{x}(t),x(t),t]$ given by Eq. (11). From a physical point of view,
it is required that the functions have correct physical units, so each partial
general Lagrangian has the units of energy. The functions may also be further
constrained by postulating the invariance of action for the null Lagrangians
and introducing the so-called exact gauge functions [20], which allows
specifying the end points of some of these functions.

The general null Lagrangian $L_{gn}[\dot{x}(t),x(t),t]$ and its partial null
Lagrangians reduce to those previously found [20,21] when $b=0$. These
Lagrangians depend on four functions, which can be constrainted by the initial
conditions if they are specified. The initial conditions are used to make the
action invariant, which requires that the gauge functions become zero at the
initial conditions. This contrains the values of the four functions at the
initial conditions, and the gauge functions that make the action invariant are
called the exact gauge functions [20].

Since the null Lagrangians do not affect the derivation of the equations of
motion, their existence is not restricted by the Helmholtz conditions [6,16].
However, the conditions seem to imply that the Caldirola-Kanai Lagrangian
cannot be constructed for the Bateman oscillators. In the following, we
consider and discuss this problem in detail.

###  2.4 Role of the Caldirola-Kanai Lagrangian

The Caldirola-Kanai Lagrangian, $L_{CK}[\dot{x}(t),x(t),t]$, when substituted
into the EL equation, yields $[\hat{D}x(t)]e^{bt}=0$, which is consitstent
with all Helmholtz conditions that are valid for any system of ordinary
differential equations [16]. However, with $e^{bt}\neq 0$, the resulting
$\hat{D}x(t)=0$ does obey the first and second Helmholtz conditions but fails
to satisfy the third condition. This shows that after the division by $e^{bt}$
the equation of motion fails to satisfy the third condition [22]; see further
discussion below Eq. (30).

Since the CK Lagrangian depends explicitly on time, the energy function
[18,19] must be calculated. Let $E_{CK}[\dot{x}(t),x(t),t]$ be the energy
function for $L_{CK}[\dot{x}(t),x(t),t]$ given by

$E_{CK}[\dot{x}(t),x(t),t]=\dot{x}(t)p_{c}(t)-L_{CK}[\dot{x}(t),x(t),t]\ ,$
(27)

where the canonical momentum $p_{c}(t)$ is

$p(t)={{\partial
L_{CK}[\dot{x}(t),x(t),t]}\over{\partial\dot{x}(t)}}=\dot{x}(t)e^{bt}\ ,$ (28)

and is different than the linear momentum $p(t)=\dot{x}(t)$.

Then, the energy function $E_{CK}[\dot{x}(t),x(t),t]$ can be written as

$E_{CK}[\dot{x}(t),x(t),t]=E_{tot}[\dot{x}(t),x(t)]e^{bt}\ .$ (29)

where the total energy is

$E_{tot}[\dot{x}(t),x(t)]={1\over
2}\left[\left(\dot{x}(t)\right)^{2}+\omega_{o}^{2}x^{2}(t)\right]\ .$ (30)

According to Eqs. (29) and (30), the energy function depends explicitly on
time through its exponential term, moreover, both the energy function and
total energy do depend on time through $x(t)$.

Having obtained $E_{CK}[\dot{x}(t),x(t),t]$, we use

${{dE_{CK}[\dot{x}(t),x(t),t]}\over{dt}}=-{{\partial
L_{CK}[\dot{x}(t),x(t),t]}\over{\partial t}}\ ,$ (31)

and find the following equation of motion

$[\hat{D}x(t)]\dot{x}(t)e^{bt}=0\ .$ (32)

Since $\dot{x}(t)e^{bt}\neq 0$, Eq. (32) becomes $\hat{D}x(t)=0$, which is the
equation of motion for the Bateman oscillators (see Eq. 4) with the well-known
solutions for $x(t)$ [18] that decrease (increase) with time as $e^{-bt/2}$
for the damped (amplified) oscillators.

The exponentially decreasing (increasing) in time solutions to $\hat{D}x(t)=0$
make $E_{CK}[\dot{x}(t),x(t),t]$ = const, when substituted into Eq. (29)
because of the cancellation of $e^{-bt}$ and $e^{bt}$, and show that
$E_{tot}[\dot{x}(t),x(t)]$ decreases (increases) exponentially in time for the
damped (amplified) oscillators. The fact that $E_{tot}[\dot{x}(t),x(t)]\neq$
const but $E_{CK}[\dot{x}(t),x(t),t]=E_{tot}[\dot{x}(t),x(t)]e^{bt}$ = const
guarantees that the equation of motion $[\hat{D}x(t)]e^{bt}=0$ resulting from
the CK Lagrangian satisfies the third Helmholtz condition; however, the
equation $[\hat{D}x(t)]=0$ does not satisfy it as is already pointed out at
the begining of this section.

The role of the CK Lagrangian in deriving the equation of motion for the
Bateman oscillators has been questioned in the literature [23] based on the
previous work [24]. The main conclusion of that previous research was that the
CK Lagrangian does not describe the Bateman oscillators but instead a
different oscillatory system with its mass increasing ($b>0$) or decreasing
($b<0$) exponentially in time. However, as pointed out recently in [25], the
previous work has some conceptual errors that led to incorrect conclusions.

The results presented in this paper are consistent with those given in [25] as
we demonstrated that the energy function (see Eq. 27) that depends on the
canonical momentum $p_{c}(t)$ is constant in time, and that the total energy
(see Eq. 30) that depends on the linear momentum $p(t)$ is not constant in
time. This shows that the total energy and the linear momentum decrease
(increase) in time for the damped (amplified) Bateman oscillators, which is
consistent with the physical picture of these dynamical systems. The increase
(decrease) of the canonical momentum in time does not contradict this picture
but instead guarantees that the CK Lagrangian can be used to derive the
equations of motion for the Bateman oscillators.

##  3 From undriven to driven Bateman oscillators

###  3.1 Total energy function

Having demonstrated the validity of the CK Lagrangian for the Bateman
oscillators, we now investigate effects of the gauge functions on the CK
Lagrangian. Let us combine the CK Lagrangian, given by Eq. (10), and the
general null Lagrangian, given by Eqs. (23) through (26), together to obtain
the following Lagrangian
$L[\dot{x}(t),x(t),t]=L_{CK}[\dot{x}(t),x(t),t]+L_{gn}[\dot{x}(t),x(t),t]$. By
using this new Lagrangian, we find the resulting energy function to be

$E[\dot{x}(t),x(t),t]={1\over
2}\left[\left(\dot{x}(t)\right)^{2}+\left(\omega_{o}^{2}-\bar{\omega}_{o}^{2}(t)\right)x^{2}(t)\right]e^{bt}$
$\hskip 90.3375pt-F(t)x(t)e^{bt/2}-G(t)\ ,$ (33)

where

$\bar{\omega}_{o}^{2}(t)=\left[\left(f_{1}(t)+{1\over
2}b\right)b+\dot{f}_{1}(t)\right]\ ,$ (34) $F(t)=\left[\left(1+{1\over
2}bt\right)f_{2}(t)+\dot{f}_{2}(t)t+\dot{f}_{4}(t)+{1\over 2}bf_{4}(t)\right]\
,$ (35)

and

$G(t)=\dot{f}_{6}(t)t+f_{6}(t)\ .$ (36)

We make the frequency shift $\bar{\omega}_{o}^{2}$ time-independent by taking
$f_{1}(t)=C_{1}$ and obtaining $\bar{\omega}_{o}^{2}=(C_{1}+b/2)b$. We also
define the total energy

$E_{tot}[\dot{x}(t),x(t)]={1\over
2}\left[\left(\dot{x}(t)\right)^{2}+\omega_{s}^{2}x^{2}(t)\right]\ ,$ (37)

where $\omega_{s}^{2}=\omega_{o}^{2}-\bar{\omega}_{o}^{2}$ = const. It must be
noted that $E_{tot}[\dot{x}(t),x(t)]$ is not constant because $x(t)$ and
$\dot{x}(t)$ decrease exponentially in time if $b>0$ or increase exponentially
in time if $b<0$.

Using Eq. (37), we write Eq. (33) as

$E[\dot{x}(t),x(t),t]=\left[E_{tot}[\dot{x}(t),x(t)]-F(t)x(t)e^{-bt/2}-G(t)e^{-bt}\right]e^{bt}\
,$ (38)

which shows that the total energy of the system is modified by the presence of
$F(t)$ and $G(t)$ that depend on the functions $f_{2}(t)$, $f_{4}(t)$ and
$f_{6}(t)$, which are arbitrary. Therefore, these arbitrary functions can be
chosen so that the difference between $E_{tot}[\dot{x}(t),x(t)]$ and the two
energy terms $F(t)x(t)e^{-bt/2}$ and $G(t)e^{-bt}$ approach zero as
$t\rightarrow\infty$.

It must be also noted that in the special case of $b=0$, Eq. (38) reduces to
Eq. (40) derived in [20]. However, if $b=0$ and $f_{2}(t)=C_{2}$,
$f_{4}(t)=C_{4}$ and $f_{6}(t)=C_{6}$, then the total energy function is equal
to that obtained in [20].

###  3.2 Gauge functions and forces

Despite the fact that the partial null Lagrangians were used in deriving the
total energy functions, these null Lagrangians were derived from the partial
gauge functions given by Eqs. (19) through (21). Based on definitions given by
Eqs. (34) through (36), it can be seen that the partial gauge function
$\phi_{gn1}[x(t),t]$ contributes only to $\bar{\omega}_{o}^{2}(t)$, and that
$F(t)$ is determined by $\phi_{gn2}[x(t),t]$ and also by $\phi_{gn3}[x(t),t]$,
which contributes partially to both $F(t)$ and $G(t)$.

Since the energy $E[\dot{x}(t),x(t),t]$ is modified by the presence of the
extra energies $F(t)x(t)$ and $G(t)$, a new Lagrangian corresponding to
$E[\dot{x}(t),x(t),t]$ must also be modified, which is achieved by adding
these extra terms to $L_{CK}[\dot{x}(t),x(t),t]$ given by Eq. (10). Then, the
modified CK Lagrangian $L_{CK,mod}[\dot{x}(t),x(t),t]$ becomes

$L_{CK,mod}[\dot{x}(t),x(t),t]={1\over
2}\left[\left(\dot{x}(t)\right)^{2}-\omega_{s}^{2}x^{2}(t)\right]e^{bt}$
$\hskip 90.3375pt-F(t)x(t)e^{bt/2}-G(t)\ ,$ (39)

which is the well-known Lagrangian for forced and damped oscillators [18,19].
Our results demonstrate that the gauge function is responsible for introducing
two extra energy terms to the original CK Lagrangian.

Using $L_{CK,mod}[\dot{x}(t),x(t),t]$, the following equation of motion is
obtained

$[\ddot{x}(t)+b\dot{x}(t)+(\omega_{o}^{2}-\bar{\omega}_{o}^{2}(t))x(t)]e^{bt}=F(t)e^{bt/2}\
,$ (40)

which can also be written as

$\ddot{x}(t)+b\dot{x}(t)+\omega_{s}^{2}x(t)=F(t)e^{-bt/2}\ .$ (41)

An interesting result is that the solutions to the homogeneous equation of Eq.
(41) depend on $e^{-bt/2}$, which is the exponential factor as in the forcing
function; see below for the full contemplemantary and particular solutions.

The definition of $F(t)$ shows that it is determined by the gauge function
$\phi_{gn2}[x(t),t]$, which contributes through its function $f_{2}(t)$ and
its derivative, and also partially by $\phi_{gn3}[x(t),t]$, which contributes
through its function $f_{4}(t)$ and its derivative. Moreover, $f_{2}(t)$ and
$f_{4}(t)$ can be any functions of $t$ as long as they are differentiable.
More constraints on these functions can be imposed after invariance of the
action is considered [20] or the initial conditions are specified.

Let us define $\beta=b/2$ and write Eq. (40) as

$\ddot{x}(t)+2\beta\dot{x}(t)+\omega_{s}^{2}x(t)=F(t)e^{-\beta t}\ .$ (42)

Since $\beta$ = const and $\omega_{s}$ = const, the solution to the
homogeneous part of Eq. (41) is well-known [18,19] and given by

$x_{h}(t)=x_{o}e^{(-\beta+i\omega)t}\ ,$ (43)

where $x_{o}$ is an integration constant and
$\omega=\sqrt{\omega_{s}^{2}-\beta^{2}}$ is the natural frequency of
oscillations.

To find the particular solution $x_{p}(t)$, the force function $F(t)$ must be
specified. Let us take

$F(t)=F_{o}e^{(i\Omega)t}\ ,$ (44)

where $F_{o}$ and $\Omega$ are the constant amplitude and frequency of the
driving force, respectively. Then, we seek the particular solution in the
following form

$x_{p}(t)=\Gamma e^{(-\beta+i\Omega)t}\ ,$ (45)

with $\Gamma$ is a constant to be determined. Substituting Eq. (45) into Eq.
(42) and using Eq. (44), we obtain

$\Gamma={{F_{o}}\over{\omega^{2}-\Omega^{2}}}\ .$ (46)

Thus, the solution to Eq. (41) is

$x(t)=x_{o}e^{(-\beta+i\omega)t}+{{F_{o}}\over{\omega^{2}-\Omega^{2}}}e^{(-\beta+i\Omega)t}\
.$ (47)

The main difference between our solution for $x(t)$ and that found in
textbooks of Classical Mechanics [18,24] is that both $x_{h}(t)$ and
$x_{p}(t)$ decay exponentially in time, which results in a different $\Gamma$
that shows a resonance if $\omega=\Omega$.

To derive the standard solution given in the textbooks [18,24], it is required
that

$F(t)=F_{o}e^{(\beta+i\Omega)t}\ .$ (48)

This is allowed as $F(t)$ is arbitrary as long as it is differentiable.
However, $F(t)$ is expressed in terms of the functions $f_{2}(t)$ and
$f_{4}(t)$ and their derivatives, these two functions are still to be
determined once $F(t)$ is specified. Since $f_{2}(t)$ and $f_{4}(t)$ are
arbitrary, we may take either $f_{2}(t)=C_{2}$ = const and solve a first-order
ordinary differential equation (ODE) for $f_{4}(t)$ or assume that
$f_{4}(t)=C_{4}$ = const and solve another ODE for $f_{2}(t)$.

The important points of the procedure described above are: (i) Any null
Lagrangian has no effect on the equation of motion; (ii) Null Lagrangians
depend explicitly on time, so when added to the standard Lagrangian they make
the total Lagrangian to depend on time; (iii) The time-dependent Lagrangian
requires that the energy function is calculated; (iv) The resulting energy
function has extra energy-like terms that are not null Lagrangians; (v) By
adding these energy-like terms to the standard Lagrangian, the derived
equation of motion becomes inhomoheneous (driven). This shows an interesting
connection between the null Lagrnagians and their gauge functions, and
classical forces [20,21].

The presented results demonstrate that the CK Lagrangian modified by the gauge
functions gives the equations of motion for the driven Bateman oscillators,
which shows that the gauge functions can be used to introduce forces to
undriven dynamical systems [21]. The result justifies searches for null
Lagrangians and their gauge functions. Our results concern only the Bateman
oscillators but the same method can be applied to other dynamical systems,
whose existing standard Lagrangians may give hints how to construct null
Lagrangians for these systems. The role of null Lagrangians in classical
physics has not yet been established, with [26,27] being the exceptions.

##  4 Conclusions

We developed the Lagrangian formalism for Bateman oscillators and used it to
derive the Caldirola-Kanai and null Lagrangians. For each null Lagrangian the
corresponding gauge function was obtained. Our results confirm the previous
findings [7,8,25] that the CK Lagrangian can be used to derive the equations
of motion for the Bateman oscillators. We also considered the Helmholtz
conditions that guarantee the existence of Lagrangians for given ODEs, and
demonstrated that the energy function obtained from the CK Lagrangian is the
sufficient condition for the resulting equation of motion to satisfy the
conditions.

The results obtained in this paper contradict the previous claims [23,24] that
the CK Lagrangian is not valid for the Bateman oscillators but instead it
represents a different dynamical system in which mass increases or decreases
exponentially in time. As a result of this contradiction, derivations of the
Kanai-Caldirola propagator in the de Broglie-Bohm theory [28], which are based
on the CK Lagrangian with its mass being time-dependent [29] must be taken
with caution.

Our main result is that the CK Lagrangian modified by the presence of extra
energy terms generated by the partial gauge functions can be used to describe
the driven Bateman oscillators. The modified CK Lagrangian given by Eq. (39)
and the resulting equation of motion for the driven Bateman oscillators (see
Eq. 41) clearly show that the Lagrangian formulation is possible, and that the
undriven Bateman oscillators can be converted into the driven ones if the
gauge functions are taken into account.

The Pais-Uhlenbeck [30] and Feshbach-Tikochinsky [31] methods to quantize the
Bateman oscillators are based on the Bateman Lagrangian given by Eq. (1).
Using these methods, it was shown that quantization of the damped quantum
harmonic oscillator can be done in terms of ladder operators [32], which is
partially incorrect due to a no-go theorem [33]. Moreover, there seems to be
also a quantum vacuum problem in [32] as demonstrated recently [34].
Quantization of the damped harmonic oscillator can also be performed using the
CK Lagrangian given by Eq. (10), whose symmetries were investigated and
constant-of-motion operators that generate the Heisenberg-Weyl algebra were
introduced [35]. Based of the results obtained in this paper, it can also be
possible to quantize the driven Bateman oscillators by using the modified CK
Lagrangian given by Eq. (39), which is suggested for future work.

Data Availability The data that supports the findings of this study are
available within the article.

Acknowledgments We are indebted to three anonymous referees for their valuable
suggestions that allow us to improve significantly the first version of this
manuscript.

## References

  * [1] H. Bateman, On dissipative systems and related variational principles, Phys. Rev. 38 (1931) 815
  * [2] B.D. Vujanovic and S.E. Jones, Eds., Variational Methods in Nonconservative Phenomena, Vol. 182, 1-371, Academic Press, New York, 1989
  * [3] U. Weiss, Quantum Dissipative Systems, World Scientific, Singapore, 2008
  * [4] M. Razavy, Classical and Quantum Dissipative Systems, World Scientific, Singapore, 2017
  * [5] G.M. Murphy, Ordinary Differential Equations and Their Solutions, Dover Publication, Inc., New York, 2011
  * [6] J. Lopuszanski, The Inverse Variational Problems in Mechanics, World Scientific, 1999
  * [7] P. Caldirola, Forze non conservative nella meccanica quantista, Nuovo Cim. 18 (1941) 393
  * [8] E. Kanai, On the quantization of the dissipative systems, Prog. Theor. Phys. 3 (1948) 44
  * [9] P.J. Olver, Applications of Lie Groups to Differential Equations, Springer-Verlag, New York, 1993
  * [10] M. Crampin and D.J. Saunders, On null Lagrangians, Diff. Geom. and its Appl., 22, 131, 2005
  * [11] R. Vitolo, On different geometric formulations of Lagrange formalism, Diff. Geom. and its Appl. 10 (1999) 293
  * [12] D. Krupka and J. Musilova, Trivial Lagrangians in field theory, Diff. Geom. and its Appl. 9 (1998) 225
  * [13] D. Krupka. O. Krupkova, and D. Saunders, The Cartan form and its generalizations in the calculus of variations, Int. J. Geom. Meth. Mod. Phys. 7 (2010) 631
  * [14] J.-M. Levy-Leblond, Group theoretical foundations of classical mechanics: teh Lagrange gauge problem, Comm. Math. Phys. 12 (1969) 64
  * [15] Z. E. Musielak and T. B. Watson, Gauge functions and Galilean invariance of action, Phys. Let. A 384 (2020) 126642
  * [16] H. Helmholtz, On physical meaning of the principle of least action, J. f. d. Reine u. Angew Math., 100 (1887) 213
  * [17] P.B. Kahn, Mathematical Methods for Scientists and Engineers, Wiley, New York, 1990
  * [18] H. Goldstein, C.P. Poole, J.L. Safko, Classical Mechanics (3rd Edition), Addison-Wesley, San Francisco, CA, 2002
  * [19] J.V. José, E.J. Saletan, Classical Dynamics, A Contemporary Approach, Cambridge Univ. Press, Cambridge, 2002
  * [20] Z.E. Musielak, T.B. Watson, General null Lagrangians, exact gauge functions and forces in Newtonian mechanics, Phys. Let. A, 384 (2020) 126838
  * [21] Z.E. Musielak, L.C. Vestal, B.D. Tran, T.B. Watson, Gauge functions in classical mechanics: From undriven to driven dynamical systems, Physics, 2 (2020) 425
  * [22] Z.E. Musielak, N. Davachi, M. Rosario-Franco, Special Functions of Mathematical Physics: A Unified Lagrangian Formalism, Mathematics, 8 (2020) 379
  * [23] F. Segovia-Chaves, The one-dimensional harmonic oscillator damped with Caldirola-Kania Hamiltonian, Rev. Mex. de Fisica E, 64 (2019) 47
  * [24] J.R. Ray, Am. J. Phys., 47 (1979) 626
  * [25] G.F. Torres del Castillo, Comment on ”The one-dimensional harmonic oscillator damped with Caldirola-Kania Hamiltonian”, Rev. Mex. de Fisica E, 65 (2019) 103
  * [26] D.R. Anderson, D.E. Carlson and E. Fried, A continuum-mechanical theory for nematic elastomers, J. Elasticity 56 (1999) 35
  * [27] G. Saccomandi and R. Vitolo, R., Null Lagrangians for nematic elastomers, J. Math. Sci. 136 (2006) 4470
  * [28] D. Bohm, Phys. Rev., 85 (1952) 166
  * [29] A. Tilbi, T. Boudjedaa, M. Merad, L. Chetouani, The Kanai-Caldirola propagator in the de Broglie-Bohm tehory, Phys. Scr., 75 (2007) 474
  * [30] A. Pais, G.E. Uhlenbeck, On field theories with non-localized action, Phys. Rev., 79 (1950) 145
  * [31] H. Feshbach, Y. Tikochinsky, Quantization fo the damped harmonic oscillator, N.Y. Acad. Sci., 38 Ser. II (1977) 44
  * [32] S. Deguchi, Y. Fujiwara, K. Nakano, Two quantization approaches to the Bateman oscillator model, Ann. Phys., 403 (2019) 2836
  * [33] F. Bagarello, F. Gargano, F. Roccati, A no-go result for the quantum damped harmonic oscillator, Phys. Let. A, 383 (2019) 2836
  * [34] F. Bagarello, F. Gargano, F. Roccati, Some remarks on few recent results on the damped quantum harmonic oscillator, Ann. Phys., 414 (2020) 168091
  * [35] J. Guerrero, F.F. López-Ruiz, V. Aldaya, F. Cossio, Symmetries of the quantum damped harmonic oscillator, J. Phys. A: Math. Theor., 45 (2012) 475303


# Weakly-supervised HOI Detection via Prior-guided Bi-level Representation
Learning

Bo Wan 1, Yongfei Liu 2∗, Desen Zhou 2, Tinne Tuytelaars 1, Xuming He 2,3
1 KU Leuven, Leuven, Belgium; 2 ShanghaiTech University, Shanghai, China
3 Shanghai Engineering Research Center of Intelligent Vision and Imaging
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Authors contributed equally and are
listed in alphabetical order.

###### Abstract

Human object interaction (HOI) detection plays a crucial role in human-centric
scene understanding and serves as a fundamental building-block for many vision
tasks. One generalizable and scalable strategy for HOI detection is to use
weak supervision, learning from image-level annotations only. This is
inherently challenging due to ambiguous human-object associations, large
search space of detecting HOIs and highly noisy training signal. A promising
strategy to address those challenges is to exploit knowledge from large-scale
pretrained models (e.g., CLIP), but a direct knowledge distillation strategy
(Liao et al., 2022) does not perform well on the weakly-supervised setting. In
contrast, we develop a CLIP-guided HOI representation capable of incorporating
the prior knowledge at both image level and HOI instance level, and adopt a
self-taught mechanism to prune incorrect human-object associations.
Experimental results on HICO-DET and V-COCO show that our method outperforms
the previous works by a sizable margin, showing the efficacy of our HOI
representation. Code is available at https://github.com/bobwan1995/Weakly-HOI.

##  1 Introduction

Human object interaction detection aims to simultaneously localize the human-
object regions in an image and to classify their interactions, which serves as
a fundamental building-block in a wide range of tasks in human-centric
artificial intelligence, such as human activity recognition (Heilbron et al.,
2015; Tina et al., 2021), human motion tracking (Wafae et al., 2019; Nishimura
et al., 2021) and anomalous behavior detection (Liu et al., 2018; Pang et al.,
2020).

Usually, HOI detection adopts a supervised learning paradigm (Gupta & Malik,
2015; Chao et al., 2018; Wan et al., 2019; Gao et al., 2020; Zhang et al.,
2021c). This requires detailed annotations (i.e. human and object bounding
boxes and their interaction types) in the training stage. However, such HOI
annotations are expensive to collect and prone to labeling errors. In
contrast, it is much easier to acquire image-level descriptions of target
scenes. Consequently, a more scalable strategy for HOI detection is to learn
from weak annotations at the image level, known as weakly-supervised HOI
detection (Zhang et al., 2017). Learning under such weak supervision is
particularly challenging mainly due to the lack of accurate visual-semantic
associations, large search space of detecting HOIs and highly noisy training
signal from only image level supervision.

Most existing works (Zhang et al., 2017; Baldassarre et al., 2020; Kumaraswamy
et al., 2021) attempt to tackle the weakly-supervised HOI detection in a
Multiple Instance Learning (MIL) framework (Ilse et al., 2018). They first
utilize an object detector to generate human-object proposals and then train
an interaction classifier with image-level labels as supervision. Despite
promising results, these methods suffer from several weaknesses when coping
with diverse and fine-grained HOIs. Firstly, they usually rely on visual
representations derived from the external object detector, which mainly focus
on the semantic concepts of the objects in the scene and hence are
insufficient for capturing the concept of fine-grained interactions. Secondly,
as the image-level supervision tends to ignore the imbalance in HOI classes,
their representation learning is more susceptible to the dataset bias and
dominated by frequent interaction classes. Finally, these methods learn the
HOI concepts from a candidate set generated by pairing up all the human and
object proposals, which is highly noisy and often leads to erroneous human-
object associations for many interaction classes.

To address the aforementioned limitations, we introduce a new weakly-
supervised HOI detection strategy. It aims to incorporate the prior knowledge
from pretrained foundation models to facilitate the HOI learning. In
particular, we propose to integrate CLIP (Radford et al., 2021b), a large-
scale vision-language pretrained model. This allows us to exploit the strong
generalization capability of the CLIP representation for learning a better HOI
representation under weak supervision. Compared to the representations learned
by the object detector, the CLIP representations are inherently less object-
centric, hence more likely to incorporate also aspects about the human-object
interaction, as evidenced by Appendix A. Although a few works have
successfully exploited CLIP for supervised HOI detection in the past,
experimentally we find they do not perform well in the more challenging
weakly-supervised setting (c.f. Appendix.B). We hypothesize this is because
they only transfer knowledge at image level, and fail without supervision at
the level of human-object pairs.

To this end, we develop a CLIP-guided HOI representation capable of
incorporating the prior knowledge of HOIs at two different levels. First, at
the image level, we utilize the visual and linguistic embeddings of the CLIP
model to build a global HOI knowledge bank and generate image-level HOI
predictions. In addition, for each human-object pair, we enrich the region-
based HOI features by the HOI representations in the knowledge bank via a
novel attention mechanism. Such a bi-level framework enables us to exploit the
image-level supervision more effectively through the shared HOI knowledge
bank, and to enhance the interaction feature learning by introducing the
visual and text representations of the CLIP model.

We instantiate our bi-level knowledge integration strategy as a modular deep
neural network with a global and local branch. Given the human-object
proposals generated by an off-the-shelf object detector, the global branch
starts with a backbone network to compute image feature maps, which are used
by a subsequent HOI recognition network to predict the image-wise HOI scores.
The local branch builds a knowledge transfer network to extract the human-
object features and augment them with the CLIP-guided knowledge bank, followed
by a pairwise classification network to compute their relatedness and
interaction scores 111Relatedness indicates whether a human-object pair has a
relation, and interaction scores are multi-label scores on the interaction
space.. The relatedness scores are used to prune incorrect human-object
associations, which mitigates the issue of noisy proposals. Finally, the
outputs of the two branches are fused to generate the final HOI scores.

To train our HOI detection network with image-level annotations, we first
initialize the backbone network and the HOI knowledge bank from the CLIP
encoders, and then train the entire model in an end-to-end manner. In
particular, we devise a novel multi-task weak supervision loss consisting of
three terms: 1) an image-level HOI classification loss for the global branch;
2) an MIL-like loss for the interaction scores predicted by the local branch,
which is defined on the aggregate of all the human-object pair predictions; 3)
a self-taught classification loss for the relatedness of each human-object
pair, which uses the interaction scores from the model itself as supervision.

We validate our methods on two public benchmarks: HICO-DET (Chao et al., 2018)
and V-COCO (Gupta & Malik, 2015). The empirical results and ablative studies
show our method consistently achieves state-of-the-art performance on all
benchmarks. In summary, our contributions are three-fold: (i) We exploit the
CLIP knowledge to build a prior-enriched HOI representation, which is more
robust for detecting fine-grained interaction types and under imbalanced data
distributions. (ii) We develop a self-taught relatedness classification loss
to alleviate the problem of mis-association between human-object pairs. (iii)
Our approach achieves state-of-the-art performance on the weakly-supervised
HOI detection task on both benchmarks.

##  2 Related works

##### HOI detection:

Most works on supervised HOI detection can be categorized in two groups: two-
stage and one-stage HOI detection. Two-stage methods first generate a set of
human-object proposals with an external object detector, then classify their
interactions. They mainly focus on exploring additional human pose information
(Wan et al., 2019; Li et al., 2020a; Gupta et al., 2019), pairwise relatedness
(Li et al., 2019a; Zhou et al., 2020) or modeling relations between object and
human (Gao et al., 2020; Zhang et al., 2021c; Ulutan et al., 2020; Zhou & Chi,
2019), to enhance the HOI representations. One-stage methods predict human &
object locations and their interaction types simultaneously in an end-to-end
manner, which are currently dominated by transformer-based architectures
(Carion et al., 2020; Kim et al., 2022; Dong et al., 2022; Zhang et al.,
2021a; b). Supervised methods show superior performance, but require labor-
intensive HOI annotations that are infeasible to obtain in many scenarios.
Thus, in this work we focus on HOI detection under weak supervision.

##### Weakly-supervised HOI detection:

Weakly-supervised HOI detection aims to learn instance-level HOIs with only
image-level annotations.  (Prest et al., 2011) learns a set of binary action
classifiers based on detected human-object pairs, where human proposal is
obtained from a part-based human detector and object is derived from the
relative position with respect to the human. PPR-FCN (Zhang et al., 2017)
employs a parallel FCN to perform pair selection and classification.
Explainable-HOI (Baldassarre et al., 2020) adopts graph nets to capture
relations for better image-level HOI recognition, and uses backward
explanation for instance-level HOI detection. MX-HOI (Kumaraswamy et al.,
2021) proposes a momentum-independent learning strategy to utilize strong &
weak labels simultaneously. AlignFormer (Kilickaya & Smeulders, 2021) proposes
an align layer in transformer framework, which utilizes geometric & visual
priors to generate pseudo alignments for training. Those methods focus on
learning HOIs with advanced network structures or better pseudo alignments.
However, they still suffer from noisy human-object associations and ambiguous
interaction types. To address those challenges, we exploit prior knowledge
from CLIP to build a discriminative HOI representations.

##### Knowledge exploitation of pretrained V&L models:

Recently, CLIP (Radford et al., 2021a) model has demonstrated strong
generalization to various downstream tasks (Ghiasi et al., 2021; Du et al.,
2022; Gu et al., 2021). Some works also explore CLIP knowledge in supervised
HOI detection, e.g., CATN (Dong et al., 2022) initializes the object query
with category-aware semantic information from CLIP text encoder, and GEN-VLTK
(Liao et al., 2022) employs image feature distillation and classifier
initialization with HOI prompts. However, they only exploit CLIP knowledge at
a coarse level and require detailed annotations of human-object pairs. It is
non-trivial to extend such strategies to the weak supervision paradigm due to
highly noisy training signals. In our work, we build a deep connection between
CLIP and HOI representation by incorporating the prior knowledge of HOIs at
both image and HOI instance levels.

##  3 Method

Figure 1: Model Overview: There are four modules in our network: a backbone
Network, an HOI recognition network, a knowledge transfer network and a
pairwise classification network.

###  3.1 Problem setup and method overview

##### Problem setup

Given an input image $I$, the task of weakly-supervised HOI detection aims to
localize and recognize the human-object interactions, while only the
corresponding image-level HOI categories are available for training. Formally,
we aim to learn a HOI detector $\mathcal{M}$, which takes an image $I$ as
input and generates a set of tuples
$\mathcal{O}=\\{(\mathbf{x}_{h},\mathbf{x}_{o},c_{o},a_{h,o},R_{h,o}^{a})\\}$,
i.e., $\mathcal{O}=\mathcal{M}(I)$. Here each tuple indicates a HOI instance,
in which $\mathbf{x}_{h},\mathbf{x}_{o}\in\mathbb{R}^{4}$ represent human and
object bounding boxes, $c_{o}\in\\{1,...,C\\}$ is the object category,
$a_{h,o}\in\\{1,...,A\\}$ denotes the interaction class associated with
$\mathbf{x}_{h}$ and $\mathbf{x}_{o}$, and $R_{h,o}^{a}\in\mathbb{R}$ is the
HOI score. For the weakly-supervised setting, each training image is annotated
with a set of HOI categories $\mathcal{R}=\\{r^{*}\\}$ at the image level
only, where $r^{*}\in\\{1,...,N\\}$ is an index to a combination of ground-
truth object category $c^{*}$ and interaction category $a^{*}$, and $N$
denotes the number of all possible HOI combinations defined on the dataset.

##### Method Overview

As we lack supervision for the HOI locations, we adopt a typical hypothesize-
and-recognize strategy (Zhang et al., 2017; Baldassarre et al., 2020;
Kumaraswamy et al., 2021) for HOI detection: first we generate a set of human
and object proposals with an off-the-shelf object detector (Ren et al., 2015)
and then predict the interaction class for all human-object combinations.

Unlike other methods, we do not re-use the feature maps of the object or human
detector - we only keep the bounding boxes. Instead, we learn a new
representation optimized for the HOI task. This is challenging under the weak
setting as the model learning is noisy,

but feasible by leveraging the rich semantic knowledge from a pretrained
large-scale multimodal model, like CLIP. However, the naive knowledge
integration strategies for supervised setting fail when directly applied in
the weak setting, as evidenced by our experiments in Appendix.B

Our framework adopts two philosophies to address the challenges in the weakly-
supervised HOI task: the first is to integrate the prior knowledge into
discriminative representation learning, and the second is to suppress noise in
learning. For the first philosophy, we utilize the prior knowledge from CLIP
to guide the representation learning in both global image-level and fine-
grained human-object pairs, which is instantiated by a bi-level knowledge
integration strategy. For the second philosophy, we adopt an effective self-
taught learning mechanism to suppress the irrelevant pairs.

We instantiate the bi-level knowledge integration strategy with a two-branch
deep network. Our detection pipeline starts with a set of human proposals with
detection scores $\\{(\mathbf{x}_{h},s_{h})\\}$, and object proposals with
their categories and detection scores $\\{(\mathbf{x}_{o},c_{o},s_{o})\\}$.
Then, the global branch performs image-level HOI recognition by utilizing a
CLIP-initialized HOI knowledge bank as a classifier. This allows us to exploit
both visual and text encoders from CLIP to generate better HOI
representations. In parallel, for each human-object pair
$(\mathbf{x}_{h},\mathbf{x}_{o})$, the local branch explicitly augments the
pairwise HOI features with the HOI knowledge bank to then identify their
relatedness and interaction classes.

To train our model, we use a multi-task loss, which incorporates a HOI
recognition loss defined on image-wise HOIs for the visual encoder and
knowledge bank finetuning, and a self-taught relatedness classification for
suppressing the background human-object associations, on top of the standard
MIL-based loss. We first present model details in Sec.3.2, followed by the
training strategy in Sec.3.3.

###  3.2 Model Design

Now we introduce our bi-level knowledge integration strategy, where the aim is
to exploit CLIP textual embeddings of HOI labels as a HOI knowledge bank for
the HOI representation learning, and to transfer such knowledge both at image
level as well as at the level of human-object pairs for interaction
predictions. Specifically, as shown in Fig. 1, our network consists of a
global branch and a local branch. The global branch includes a backbone
network (Sec.3.2.1) that extracts image features, and a HOI recognition
network (Sec.3.2.2) that uses a HOI knowledge bank based on CLIP to predict
image-level HOI scores. For each human-object proposal generated by an off-
the-shelf object detector, the local branch employs a knowledge transfer
network (Sec.3.2.3) to compute its feature representation with enhancement
from the HOI knowledge bank, and a pairwise classification network (Sec.3.2.4)
to compute their relatedness and interaction scores. Finally, we generate the
final HOI detection scores by combining global HOI scores with local
predictions (Sec. 3.2.5).

##### HOI Knowledge Bank Generation

CLIP builds a powerful vision-language model by pretraining on large-scale
image-text pairs. It consists of a visual encoder $\mathcal{F}_{V}$ and
textual encoder $\mathcal{F}_{T}$, mapping both visual and textual inputs to a
shared latent space. Here, we exploit CLIP to generate a HOI knowledge bank.
We take a similar prompt strategy as in CLIP, adopting a common template ‘a
person {verb} a/an {object}’ to convert HOI labels into text prompts (e.g.,
converting ‘drive car’ to ‘a person driving a car’). Then we input the
sentences into the CLIP textual encoder $\mathcal{F}_{T}$ to initialize the
HOI knowledge bank $\mathcal{W}_{T}\in\mathbb{R}^{N\cdot D}$, with $D$
denoting the feature dimension. One can think of $\mathcal{W}_{T}$ as a set of
‘prototypes’ in feature space, one for each HOI in the dataset.

####  3.2.1 Global branch: Backbone Network

To incorporate CLIP for feature extraction, we initialize the backbone network
(e.g., a ResNet-101 (He et al., 2016)) with CLIP’s visual encoder
$\mathcal{F}_{V}$ to generate a feature map $\bm{\Gamma}$ for the input image
$I$. We further compute a global feature vector $v_{g}\in\mathbb{R}^{D}$ with
self-attention operation (Radford et al., 2021b).

####  3.2.2 Global branch: HOI Recognition Network

We perform an image-wise HOI recognition task with the HOI knowledge bank
$\mathcal{W}_{T}$. We obtain global HOI scores $s_{g}\in\mathbb{R}^{N}$ by
computing the inner product between the image feature $v_{g}$ and the
knowledge bank $\mathcal{W}_{T}$: $s_{g}=\mathcal{W}_{T}\times v_{g}$, where
$\times$ is matrix multiplication. This has the effect of adapting the visual
encoder and knowledge bank parameters to the HOI recognition task, fully
taking advantage of the knowledge from CLIP.

Figure 2: The knowledge transfer network explicitly transfers the
discriminative relation-level semantic knowledge derived from CLIP to the
pairwise HOI representations. Pseudo relatedness label generation uses the
pairwise interaction scores to generate the pseudo association labels for
self-taught relatedness classification

####  3.2.3 Local branch: Knowledge Transfer Network

Given the CLIP-initialized visual encoder, a standard HOI representation can
be formed by concatenating the human and object appearance features along with
their spatial encoding. However, even after the finetuning as described above,
such a representation still mainly focuses on object-level semantic cues
rather than relation-level concepts. In this module, we explicitly exploit the
HOI knowledge bank $\mathcal{W}_{T}$ to learn a local relation-specific HOI
representation. To achieve this, we propose an attention-based architecture as
shown in Fig.2(a).

Specifically, for each human proposal $\mathbf{x}_{h}$ and object proposal
$\mathbf{x}_{o}$, we use RoI-Align (He et al., 2017) to crop the feature maps
from $\bm{\Gamma}$ followed by a self-attention operation to compute their
appearance features $v_{h},v_{o}\in\mathbb{R}^{D}$. Then we compute a spatial
feature $v_{sp}$ by encoding the relative positions of their bounding boxes
$(\mathbf{x}_{h},\mathbf{x}_{o})$ 222For details c.f. the appendix C. The
holistic HOI representation $v_{p}\in\mathbb{R}^{D}$ is an embedding of the
human and object appearance features and their spatial feature, i.e.,
$v_{p}=\mathcal{F}_{E}([v_{h};v_{o};v_{sp}])$, where $[;]$ is the
concatenation operation and $\mathcal{F}_{E}$ is a multi-layer perceptron
(MLP).

To enhance relation-level concepts, we further compute its union region
$\mathbf{x}_{u}\in\mathbb{R}^{4}$ (see Fig. 2a) and extract the corresponding
appearance feature $v_{u}\in\mathbb{R}^{D}$ via RoI-align over the feature map
$\bm{\Gamma}$. The union region is important as it encodes relational context
cues, but it potentially also contains a large amount of background that is
noisy for model learning. We thus devise an attention module that is similar
in design to the HOI recognition network, but uses the union feature $v_{u}$
as query to extract a meta-embedding $v_{meta}\in\mathbb{R}^{D}$ from the HOI
knowledge bank $\mathcal{W}_{T}$. The final HOI representation
$\hat{v}_{p}\in\mathbb{R}^{D}$ is built by fusing the holistic representation
$v_{p}$ and $v_{meta}$ with a MLP $\mathcal{F}_{K}$.

$\alpha=Softmax(\mathcal{W}_{T}\times v_{u});\quad
v_{meta}=\alpha^{\intercal}\times\mathcal{W}_{T};\quad\hat{v}_{p}=\mathcal{F}_{K}(v_{p}+v_{meta}).$
(1)

Here $\alpha\in\mathbb{R}^{N}$ is the normalized attention weight and
$\intercal$ is the transpose operation. $v_{meta}$ encodes a discriminative
representation from CLIP and facilitates feature sharing between HOI classes.

####  3.2.4 Local branch: Pairwise Classification Network

Given the relation-aware HOI representation $\hat{v}_{p}$ , our final module
performs a coarse-level classification on human-object association and a fine-
level classification for interaction recognition. Specifically, we use two
MLPs $\mathcal{F}_{P}$ and $\mathcal{F}_{B}$ to predict the interaction scores
$s_{p}\in\mathbb{R}^{A}$ and the relatedness score $s_{b}\in\mathbb{R}$ for
each human-object pair:

$\vspace{-1mm}s_{p}=\mathcal{F}_{P}(\hat{v}_{p});\quad
s_{b}=\mathcal{F}_{B}(\hat{v}_{p})\vspace{-1mm}$ (2)

To train the model under weak supervision (see Sec. 3.3), we further aggregate
the pairwise interaction scores into image-level interaction scores . Assume
we have $M$ pairs of human-object proposals for a given image, and denote the
interaction scores for the $m$-th pair as $s_{p}^{m}$. We first concatenate
all the interaction scores to compose a bag
$S=[s_{p}^{1};...;s_{p}^{M}]\in\mathbb{R}^{M\cdot A}$, then we maximize over
all pairs to obtain the image-wise interaction scores:
$\tilde{s}_{p}=\mathop{max}\limits_{m}S$.

####  3.2.5 Model Inference

During model inference, we do not use the local interaction scores $s_{p}$
directly. Instead, we normalize $S$ with a $Softmax$ operation defined on all
pairs: $\bar{S}=\mathop{Softmax}\limits_{m}(S)$, and then compute the
normalized pairwise interaction scores
$e_{p}=\sigma(\tilde{s}_{p})\cdot\bar{s}_{p}$, where $\bar{s}_{p}$ is a row
from $\bar{S}$ and $\sigma$ is $Sigmoid$ function. This has the effect of
measuring the contribution of a given pair, in case of multiple pairs in an
image share the same interaction.

The final interaction score $s_{h,o}^{a}$ for human-object pair
$(\mathbf{x}_{h},\mathbf{x}_{o})$ combines multiple scores, including the
global HOI scores $s_{g}$, the normalized pairwise interaction scores $e_{p}$,
and the relatedness score $s_{b}$. The overall HOI score $R_{h,o}^{a}$ is a
combination of the interaction score and the object detection scores.

$\vspace{-1mm}s_{h,o}^{a}=\sigma(s_{g}^{a,c_{o}})\cdot
e_{p}^{a}\cdot\sigma(s_{b});\quad R_{h,o}^{a}=(s_{h}\cdot s_{o})^{\gamma}\cdot
s_{h,o}^{a}$ (3)

where $s_{g}^{a,c_{o}}$ is the HOI score corresponding to $a$-th interaction
and $c_{o}$-th object category in $s_{g}$, $e_{p}^{a}$ is the score of $a$-th
interaction in $e_{p}$, and $\gamma$ is a hyper-parameter to balance the
scores (Zhang et al., 2021c; Li et al., 2019b).

###  3.3 Learning with Weak Supervision

To train our deep network in a weakly supervised setting, we use a multi-task
loss defined on three different levels. Specifically, our overall loss
function $\mathcal{L}$ consists of three terms: i) an image-wise HOI
recognition loss $\mathcal{L}_{g}$ to adapt CLIP features to the task of
human-object interaction detection; ii) a pairwise interaction classification
loss $\mathcal{L}_{p}$ to guide the knowledge transfer towards fine-grained
relation-aware representations; and iii) a self-taught relatedness
classification loss $\mathcal{L}_{b}$ to prune non-interacting human-object
combinations. Formally, the overall loss is written as:

$\vspace{-1mm}\mathcal{L}=\mathcal{L}_{g}+\mathcal{L}_{p}+\mathcal{L}_{b}\vspace{-1mm}$
(4)

##### Image-wise HOI recognition loss $\mathcal{L}_{g}$:

Given the HOI scores $s_{g}$ and ground-truth HOI categories $\mathcal{R}$,
$\mathcal{L}_{g}$ is a standard binary cross-entropy loss for multi-label
classification: $\mathcal{L}_{g}=L_{BCE}(s_{g},\mathcal{R})$.

##### Pairwise interaction classification loss $\mathcal{L}_{p}$:

We adopt a MIL strategy that first aggregates the pairwise interaction scores
and supervises this with image-level interaction labels as
$\mathcal{A}=\\{a^{*}\\}$. Given the image-wise interaction scores
$\tilde{s}_{p}$, $\mathcal{L}_{p}$ is a standard binary cross-entropy loss for
multi-label classification as:
$\mathcal{L}_{p}=L_{BCE}(\tilde{s}_{p},\mathcal{A})$.

##### Self-taught relatedness classification loss $\mathcal{L}_{b}$:

As human-object associations are not annotated, we devise a novel pseudo
relatedness label generation mechanism for training a self-taught binary
classifier to identify valid human-object associations. Specifically, we
observe that the human-object pairs with confident interaction scores are
often associated after a short period of initial training without self-taught
classification loss. Motivated by this, we use the interaction scores $s_{p}$
from the model under training to supervise the relatedness classification.

Concretely, we generate pseudo labels $\mathcal{B}=\\{b_{1},...,b_{M}\\}$ for
all human-object pairs in an image, where $b_{m}\in\\{0,1\\}$ indicates the
relatedness for the $m$-th combination. To this end, as illustrated in
Fig.2(b), we first propose a binary mask $Z\in\\{0,1\\}^{M\cdot A}$ for all
interaction scores $S$ with respect to the ground-truth object categories
$\mathcal{C}=\\{c^{*}\\}$. For each human-object pair where the object label
$c_{o}$ is included in $\mathcal{C}$, we consider it as a potential
interactive combination and thus assign the corresponding row in $Z$ as 1, and
other rows as 0. For the latter, we also immediately set $b_{m}=0$. Then we
generate pairwise scores $t^{a}\in\mathbb{R}^{M}$ for each ground-truth
interaction $a^{*}$ by selecting the corresponding row from $S\odot Z$. The
pseudo label for the pair with the highest score is assigned as 1, i.e.,
$m_{a}=\mathop{\arg\max}\limits_{m}t^{a}$ and $b_{m_{a}}=1$. We only select
one positive pair333We also explore top-K selection in Appendix F for each
$a^{*}$. Finally, $\mathcal{L}_{b}$ is defined as a binary cross-entropy loss:
$\mathcal{L}_{b}=\sum\limits_{m}L_{BCE}(s_{b}^{m},b_{m})$, where $s_{b}^{m}$
is the relatedness score for the $m$-th pair.

##  4 Experiments

###  4.1 Experimental setup

Datasets: We benchmark our model on two public datasets: HICO-DET and V-COCO.
HICO-DET consists of 47776 images (38118 for training and 9658 for test). It
has $N=600$ HOI categories, which are composed of $C=80$ common objects (the
same as MSCOCO (Lin et al., 2014)) and $A=117$ unique interaction categories.
V-COCO is a subset of MSCOCO, consisting of 2533 images for training, 2867 for
validation and 4946 for test. It has 16199 human instances, each annotated
with binary labels for $A=26$ interaction categories.

Evaluation Metric: Following (Chao et al., 2015), we use mean average
precision (mAP) to evaluate HOI detection performance. A human-object pair is
considered as positive when both predicted human and object boxes have at
least 0.5 IoU with their ground-truth boxes, and the HOI class is classified
correctly.

###  4.2 Implementation Details

We use an off-the-shelf Faster R-CNN (Ren et al., 2015) pretrained on MSCOCO
to generate at most 100 object candidates for each image. For V-COCO, it is
worth noting that we train the object detector by removing the images in
MSCOCO that overlap with V-COCO to prevent information leakage. The backbone
network is initialized with the visual encoder from CLIP-RN101 model and the
feature dimension $D=1024$.

For model learning, we set the detection score weight $\gamma=2.8$ as default
by following previous works (Zhang et al., 2021c; Li et al., 2019b), then
optimize the entire network with AdamW and an initial learning rate of 1e-5
for backbone parameters and 1e-4 for others. We detach the parameters of the
knowledge bank on the local branch for better model learning. We train up to
60K iterations with batch-size 24 in each on 4 NVIDIA 2080TI GPUs, and decay
the learning rate by 10 times in 12K and 24K iteration.

Table 1: mAP comparison on HICO-DET and V-COCO test set. - denotes the results
are not available. * stands for the method we re-evaluate with the correct
evaluation protocol (see Appendix.I for details) and †means our re-
implementation. For V-COCO, all object detectors are pretrained on MSCOCO
dataset by default, and details about the evaluation metrics APS1&2 c.f.
Appendix H. IN-1K denotes ImageNet with 1000 classes.

Methods | Backbone | Detector | HICO-DET (%) | V-COCO (%)
---|---|---|---|---
Full | Rare | Non-Rare | AP${}_{role}^{S1}$ | AP${}_{role}^{S2}$
supervised
iCAN (Gao et al., 2018) | RN50 (IN-1K&COCO) | FRCNN (COCO) | 14.84 | 10.45 | 16.15 | 45.30 | 52.40
PMFNet (Wan et al., 2019) | RN50-FPN (IN-1K&COCO) | FRCNN (COCO) | 17.46 | 15.56 | 18.00 | 52.00 | -
TIN (Li et al., 2019b) | RN50-FPN (IN-1K&COCO) | FRCNN (COCO) | 17.22 | 13.51 | 18.32 | 47.80 | 54.20
DJ-RN (Li et al., 2020a) | RN50 (IN-1K&COCO) | FRCNN (COCO) | 21.34 | 18.53 | 21.18 | 53.30 | 60.30
IDN (Li et al., 2020b) | RN50 (IN-1K&COCO) | FRCNN (HICO-DET) | 26.29 | 22.61 | 27.39 | 53.30 | 60.30
SCG (Zhang et al., 2021c) | RN50-FPN (IN-1K&HICO-DET) | FRCNN (HICO-DET) | 31.33 | 24.72 | 33.31 | 54.20 | 60.90
HOTR (Kim et al., 2021) | RN50+Transformer (IN-1K&COCO) | DETR (HICO-DET) | 25.10 | 17.34 | 27.42 | 55.20 | 64.40
QPIC (Tamura et al., 2021) | RN101+Transformer (IN-1K&COCO) | DETR (COCO) | 29.90 | 23.92 | 31.69 | 58.30 | 60.70
CATN (Dong et al., 2022) | RN50+Transformer (IN-1K&HICO-DET&COCO) | DETR (HICO-DET) | 31.86 | 25.15 | 33.84 | 60.10 | -
MSTR (Kim et al., 2022) | RN50 + Transformer (IN-1K&COCO) | DETR(HICO-DET) | 31.17 | 25.31 | 33.92 | 62.00 | 65.20
DisTr (Zhou et al., 2022) | RN50+Transformer (IN-1K&COCO) | DETR (HICO-DET) | 31.75 | 27.45 | 33.03 | 66.20 | 68.50
SSRT (Iftekhar et al., 2022) | R101+Transformer (IN-1K&COCO) | DETR (COCO) | 31.34 | 24.31 | 33.32 | 65.00 | 67.10
GEN-VLKT (Liao et al., 2022) | RN101+Transformer (IN-1K&HICO-DET) | DETR (HICO-DET) | 34.95 | 31.18 | 36.08 | 63.58 | 65.93
between supervised & weakly-supervised setting, learning with image-level HOIs
and box annotations
AlignFormer (Kilickaya & Smeulders, 2021) | RN101+Transformer (IN-1K&HICO-DET) | DETR (HICO-DET) | 20.85 | 18.23 | 21.64 | 15.82 | 16.34
weakly-supervised
Explanation-HOI* (Baldassarre et al., 2020) | ResNeXt101 (IN-1K&COCO) | FRCNN (COCO) | 10.63 | 8.71 | 11.20 | - | -
MX-HOI (Kumaraswamy et al., 2021) | RN101 (IN-1K&COCO) | FRCNN (COCO) | 16.14 | 12.06 | 17.50 | - | -
PPR-FCN† (Zhang et al., 2017) | RN50 (CLIP dataset) | FRCNN (COCO) | 17.55 | 15.69 | 18.41 | - | -
ours | RN50 (CLIP dataset) | FRCNN (COCO) | 22.89 | 22.41 | 23.03 | 42.97 | 48.06
ours | RN101 (CLIP dataset) | FRCNN (COCO) | 25.70 | 24.52 | 26.05 | 44.74 | 49.97

###  4.3 Quantitative Results

For HICO-DET (Tab.1), our approach outperforms the previous state of the arts
on the weakly supervised setting by a clear margin, achieving 22.89 mAP with
ResNet-50 and 25.70 mAP with ResNet-101 as backbone. For a fair comparison, we
also re-implement PPR-FCN with CLIP visual encoder. The results show that we
still outperform PPR-FCN by a sizeable margin, which validates the superiority
of our framework. Besides, we even perform comparably with HOTR and IDN under
an inferior experimental setting where HOTR adopts a more advanced transformer
encoder-decoder architecture, and both methods are trained with strong
supervision. Furthermore, the mAP gap between Rare (training annotations < 10)
and Non-rare HOI classes in our results is much smaller than other methods,
demonstrating the superior generalization capability of our HOI representation
for solving the long-tailed distribution issue. In detail, we achieve a 0.62
mAP gap with ResNet-50 and 1.53 with ResNet-101 backbone, which is much
smaller than AlignFormer (3.14) and PPR-FCN (2.64), and supervised methods
SSRT (9.01) and GEN-VLKT (4.9).

For V-COCO dataset, we report the performance of AProle in both scenario1 and
scenario2 for a complete comparison, which are 42.97 / 48.06 AProle with
ResNet-50 and 44.74 / 49.97 AProle with ResNet-101 as backbone. As shown in
Tab.1, our model achieves significant improvement compared with AlignFormer,
and even is comparable with supervised methods TIN and iCAN.

Table 2: Ablation study on HICO-DET dataset. “RN50-FPN(COCO)” denotes the
backbone initialized with Faster R-CNN parameters pretrained on MSCOCO dataset
while “CLIP RN50” stands for the backbone initialized with CLIP visual
encoder. Besides, we construct the knowledge bank $\mathcal{W}_{T}$ with
random initialization, or computing HOI prompts by RoBERTa or CLIP text
transformer.

Methods | Parameter initialization | CLIP Knowledge | SRC | mAP (%)
---|---|---|---|---
Backbone | knowledge bank | HOI recognition | KTN | score fusion | Full | Rare | Non-Rare
baseline | CLIP RN50 | - | - | - | - | - | 19.52 | 16.58 | 20.40
Exp 1 | CLIP RN50 | CLIP Text | ✓ | - | - | - | 20.31 | 18.34 | 20.90
Exp 2 | CLIP RN50 | CLIP Text | ✓(freeze $\mathcal{W}_{T}$) | - | - | - | 20.09 | 18.23 | 20.64
Exp 3 | CLIP RN50 | CLIP Text | ✓ | ✓ | - | - | 20.86 | 18.40 | 21.60
Exp 4 | CLIP RN50 | CLIP Text | ✓ | ✓ | ✓ | - | 22.40 | 20.70 | 22.90
Exp 5 | CLIP RN50 | - | - | - | - | ✓ | 19.88 | 17.45 | 20.61
Exp 6 | CLIP RN50 | CLIP Text | ✓ | - | - | ✓ | 20.75 | 19.38 | 21.16
Exp 7 | CLIP RN50 | CLIP Text | ✓ | ✓ | - | ✓ | 21.53 | 20.05 | 21.97
ours | CLIP RN50 | CLIP Text | ✓ | ✓ | ✓ | ✓ | 22.89 | 22.41 | 23.03
Exp 8 | RN50-FPN (COCO) | - | - | - | - | - | 19.44 | 16.20 | 20.41
Exp 9 | RN50-FPN (COCO) | random | ✓ | ✓ | ✓ | ✓ | 19.61 | 15.57 | 20.82
Exp 10 | RN50-FPN (COCO) | RoBERTa | ✓ | ✓ | ✓ | ✓ | 20.45 | 16.46 | 21.65

###  4.4 Ablation Study

In this section, we mainly validate the effectiveness of each component with
detailed ablation studies on HICO-DET dataset. We use ResNet-50 as the
backbone network to reduce experimental costs.

##### Baseline:

The baseline adopts the visual encoder from CLIP-RN50 to generate the vanilla
HOI representation $v_{p}$, which is directly used to predict the interaction
scores $s_{p}$. Only pairwise interaction classification loss
$\mathcal{L}_{p}$ is used for model learning.

##### HOI recognition:

We augment the baseline with a HOI recognition network and observe the full
mAP improves from 19.52 to 20.41, as reported in Exp 1 of Tab. 2. It suggests
that the learnable knowledge bank $\mathcal{W}_{T}$ serves as a powerful
classifier to perform image-level HOI recognition and update the visual
encoder for better HOI representation. We visualize the learned parameters of
knowledge bank in Appendix D to demonstrate its effectiveness. Furthermore, as
in Exp 2, the performance slightly decreases from 20.31 to 20.09 when we
freeze the training of the knowledge bank, indicating that joint learning of
visual features and the knowledge bank is more appropriate for HOI detection.

##### Knowledge Transfer Network (KTN):

KTN explicitly transfers the CLIP meta-knowledge to pairwise HOI features. As
a result, it contributes 0.55 Full mAP improvement (Exp 3 v.s. Exp 1) and most
of the performance gains come from Non-rare classes. This result shows KTN is
capable of extracting discriminative features from the relational knowledge
bank to our HOI representation. We also study the effectiveness of the
attention mechanism of KTN in Appendix E.

##### Score fusion:

In Tab. 2, we largely improve the Full mAP from 20.86 (Exp 3) to 22.40 (Exp 4)
by fusing the global HOI scores $s_{g}$ to pairwise interaction score $s_{p}$.
As the HOI recognition network seamlessly inherits the visual-linguistic
features from CLIP and directly adopts image labels as supervision, the global
interaction scores are pretty accurate and largely enhance the pairwise
scores, demonstrating its strong capabilities to cope with long-tailed and
fine-grained HOI recognition.

##### Self-taught Relatedness Classification (SRC):

Self-taught classification aims to identify the relatedness between human and
objects. The improvements from Exp 4 to ours show the effectiveness of our
self-taught strategy, which is capable of figuring out the irrelevant human-
object pairs and suppressing their interaction scores during inference.

##### Combining KTN & SRC:

The ablation results of Exp 5-7 in Tab. 2 show the KTN and SRC are able to
facilitate each other. In detail, the SRC obtains 0.49 Full mAP improvement
when the KTN is introduced (ours v.s. Exp 4), which is only 0.36 without KTN
(Exp 5 v.s. baseline). Similarly, the KTN contributes 0.78 Full mAP
improvement with SRC (Exp 7 v.s. Exp 6), which is only 0.55 without SRC (Exp 3
v.s. Exp 1).

##### Parameter initialization:

Our visual encoder and knowledge bank are both initialized from CLIP. We also
explore different parameter initialization strategy in Exp 8-10. Specifically,
we initialize the visual encoder with a ResNet50-FPN pretrained on COCO
detection task for the baseline (Exp 8), and the knowledge bank with random
parameters (Exp 9) or embeddings of HOI labels from RoBERTa model (Exp 10) for
the final model. We observe severe drops with all these initialization methods
compared with ours, demonstrating the effectiveness and generalization ability
of CLIP model. It is worth noting that the mAP of Rare classes decreases from
16.20 in Exp 8 to 15.57 in Exp 9, which suggests the randomly initialized
knowledge bank even aggravates the imbalance issue in final model.

Figure 3: Visualization of HOI detection results on HICO-DET test set. Red
scores denote the negative HOI predictions. We mainly demonstrate the model’s
capabilities on four aspects: (a) coping with imbalanced HOI distribution; (b)
distinguishing subtle differences among interaction types; (c) suppressing
background HOI classes, and (d) pruning irrelevant human-object associations.
The numbers reported are normalized pairwise interaction score, global HOI
score and relatedness score.

###  4.5 Qualitative Results

We show some qualitative results of our method in Fig.3. For each HOI
prediction, we report (i) normalized pairwise interaction score, (ii) global
HOI score and (iii) relatedness score for ours, and only pairwise interaction
score for baseline. In Fig.3(a), ours interaction scores are more confident
than baseline in Rare HOI classes, demonstrating the generalization ability of
our CLIP-guided HOI representation. Besides, when incorporating relational
knowledge bank into pairwise HOI representation, our method is capable of
distinguishing the subtle differences among similar HOIs in Fig.3(b) (e.g.,
repair_truck:0.23 v.s. inspect_truck:0.48 in the bottom figure). Moreover, in
Fig.3(c), the global branch suppresses background HOIs by predicting low
global scores for them (e.g., the global HOI score is 0.033 for
sit_on_motorcycle while the ground-truth is sit_on_bicycle). Finally, in
Fig.3(d), our self-taught relatedness classification strategy shows strong
capability at recognizing the ambiguous human-object associations (e.g., 0.079
v.s. 0.994 in the upper figure).

##  5 Conclusion

In this paper, we propose a bi-level knowledge integration strategy that
incorporates the prior knowledge from CLIP for weakly-supervised HOI
detection. Specifically, we exploit CLIP textual embeddings of HOI labels as a
relational knowledge bank, which is adopted to enhance the HOI representation
with an image-wise HOI recognition network and a pairwise knowledge transfer
network. We further propose the addition of a self-taught binary pairwise
relatedness classification loss to overcome ambiguous human-object
association. Finally, our approach achieves the new state of the art on both
HICO-DET and V-COCO benchmarks under the weakly supervised setting.

## Acknowledgement

We acknowledge funding from Flemish Government under the Onderzoeksprogramma
Artificiele Intelligentie (AI) Vlaanderen programme, Shanghai Science and
Technology Program 21010502700 and Shanghai Frontiers Science Center of Human-
centered Artificial Intelligence.

## Ethics Statement

Hereby, we consciously assure that our study is original work which has not
been previously published elsewhere, and is not currently being considered for
publication elsewhere. We do not have ethics risks as mentioned in the author
guidelines.

## Reproducibility Statement

We use publicly available benchmarks, HICO-DET and V-COCO, to validate our
method. Code is available at https://github.com/bobwan1995/Weakly-HOI.

## References

  * Baldassarre et al. (2020) Federico Baldassarre, Kevin Smith, Josephine Sullivan, and Hossein Azizpour.  Explanation-based weakly-supervised learning of visual relations with graph networks.  In _ECCV_ , 2020.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _ECCV_ , 2020.
  * Chao et al. (2015) Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng.  HICO: A benchmark for recognizing human-object interactions in images.  In _ICCV_ , 2015.
  * Chao et al. (2018) Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng.  Learning to detect human-object interactions.  In _WACV_ , 2018.
  * Dong et al. (2022) Leizhen Dong, Zhimin Li, Kunlun Xu, Zhijun Zhang, Luxin Yan, Sheng Zhong, and Xu Zou.  Category-aware transformer network for better human-object interaction detection.  _arXiv preprint arXiv:2204.04911_ , 2022.
  * Du et al. (2022) Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  _arXiv preprint arXiv:2203.14940_ , 2022.
  * Gao et al. (2018) Chen Gao, Yuliang Zou, and Jia-Bin Huang.  ican: Instance-centric attention network for human-object interaction detection.  In _BMVC_ , 2018.
  * Gao et al. (2020) Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang.  Drg: Dual relation graph for human-object interaction detection.  In _ECCV_ , 2020.
  * Ghiasi et al. (2021) Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Open-vocabulary image segmentation.  _arXiv preprint arXiv:2112.12143_ , 2021.
  * Gu et al. (2021) Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  In _ICLR_ , 2021.
  * Gupta & Malik (2015) Saurabh Gupta and Jitendra Malik.  Visual semantic role labeling.  _arXiv preprint arXiv:1505.04474_ , 2015.
  * Gupta et al. (2019) Tanmay Gupta, Alexander Schwing, and Derek Hoiem.  No-frills human-object interaction detection: Factorization, layout encodings, and training techniques.  In _ICCV_ , 2019.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _ICCV2017_ , 2017.
  * Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 961–970, 2015.
  * Iftekhar et al. (2022) ASM Iftekhar, Hao Chen, Kaustav Kundu, Xinyu Li, Joseph Tighe, and Davide Modolo.  What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions.  _arXiv preprint arXiv:2204.00746_ , 2022.
  * Ilse et al. (2018) Maximilian Ilse, Jakub Tomczak, and Max Welling.  Attention-based deep multiple instance learning.  In _ICML_ , pp. 2127–2136, 2018.
  * Kilickaya & Smeulders (2021) Mert Kilickaya and Arnold Smeulders.  Human-object interaction detection via weak supervision.  _arXiv preprint arXiv:2112.00492_ , 2021.
  * Kim et al. (2021) Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J. Kim.  Hotr: End-to-end human-object interaction detection with transformers.  In _CVPR_ , 2021.
  * Kim et al. (2022) Bumsoo Kim, Jonghwan Mun, Kyoung-Woon On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim.  Mstr: Multi-scale transformer for end-to-end human-object interaction detection.  _arXiv preprint arXiv:2203.14709_ , 2022.
  * Kumaraswamy et al. (2021) Suresh Kirthi Kumaraswamy, Miaojing Shi, and Ewa Kijak.  Detecting human-object interaction with mixed supervision.  In _WACV_ , 2021.
  * Li et al. (2019a) Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yan-Feng Wang, and Cewu Lu.  Transferable interactiveness prior for human-object interaction detection.  In _CVPR_ , 2019a.
  * Li et al. (2019b) Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu.  Transferable interactiveness knowledge for human-object interaction detection.  In _CVPR_ , 2019b.
  * Li et al. (2020a) Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu.  Detailed 2d-3d joint representation for human-object interaction.  In _CVPR_ , 2020a.
  * Li et al. (2020b) Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu.  Hoi analysis: Integrating and decomposing human-object interaction.  In _NeurIPS_ , 2020b.
  * Liao et al. (2022) Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu.  Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection.  _arXiv preprint arXiv:2203.13954_ , 2022.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Liu et al. (2018) Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao Gao.  Future frame prediction for anomaly detection – a new baseline.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Nishimura et al. (2021) Hitoshi Nishimura, Satoshi Komorita, Yasutomo Kawanishi, and Hiroshi Murase.  Sdof-tracker: Fast and accurate multiple human tracking by skipped-detection and optical-flow.  _arXiv preprint arXiv:2106.14259_ , 2021.
  * Pang et al. (2020) Guansong Pang, Cheng Yan, Chunhua Shen, van den Hengel Anton, and Xiao Bai.  Self-trained deep ordinal regression for end-to-end video anomaly detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Prest et al. (2011) Alessandro Prest, Cordelia Schmid, and Vittorio Ferrari.  Weakly supervised learning of interactions between humans and objects.  _IEEE TPAMI_ , 2011.
  * Radford et al. (2021a) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021a.
  * Radford et al. (2021b) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021b.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _arXiv preprint arXiv:1506.01497_ , 2015.
  * Tamura et al. (2021) Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga.  Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information.  In _CVPR_ , 2021.
  * Tina et al. (2021) Tina, Anmol Kumar Sharma, Siddharth Tomar, and Kapil Gupta.  Various approaches of human activity recognition: A review.  In _International Conference on Computing Methodologies and Communication(ICCMC)_ , 2021.
  * Ulutan et al. (2020) Oytun Ulutan, A S M Iftekhar, and B. S. Manjunath.  Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions.  In _CVPR_ , 2020.
  * van der Maaten & Hinton (2008) Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _JMLR_ , 2008.  URL http://jmlr.org/papers/v9/vandermaaten08a.html.
  * Wafae et al. (2019) Mrabti Wafae, Baibai Kaoutar, Bellach Benaissa, Oulad Haj Thami Rachid, and Tairi Hamid.  Human motion tracking: A comparative study.  _Procedia Computer Science_ , 148:145–153, 2019.
  * Wan et al. (2019) Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He.  Pose-aware multi-level feature network for human object interaction detection.  In _ICCV_ , 2019.
  * Zhang et al. (2021a) Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li.  Mining the benefits of two-stage and one-stage hoi detection.  _NeuIPS_ , 2021a.
  * Zhang et al. (2021b) Frederic Z Zhang, Dylan Campbell, and Stephen Gould.  Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer.  _arXiv preprint arXiv:2112.01838_ , 2021b.
  * Zhang et al. (2021c) Frederic Z Zhang, Dylan Campbell, and Stephen Gould.  Spatially conditioned graphs for detecting human-object interactions.  In _ICCV_ , 2021c.
  * Zhang et al. (2017) Hanwang Zhang, Zawlin Kyaw, Jinyang Yu, and Shih-Fu Chang.  Ppr-fcn: Weakly supervised visual relation detection via parallel pairwise r-fcn.  In _ICCV_ , 2017.
  * Zhou et al. (2022) Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang.  Human-object interaction detection via disentangled transformer.  _arXiv preprint arXiv:2204.09290_ , 2022.
  * Zhou & Chi (2019) Penghao Zhou and Mingmin Chi.  Relation parsing neural network for human-object interaction detection.  In _ICCV_ , 2019.
  * Zhou et al. (2020) Tianfei Zhou, Wenguan Wang, Siyuan Qi, Haibin Ling, and Jianbing Shen.  Cascaded human-object interaction recognition.  In _CVPR_ , 2020.

## Appendix

In this appendix, we first describe the spatial feature generation, and then
supplement more experimental results of different CLIP knowledge integration
strategies for weakly-supervised HOI detection. For Explanation-HOI
(Baldassarre et al., 2020), we further clarify the difference between their
mAP evaluation protocol and the standard one. Finally, we demonstrate the
limitations, potential negative societal impacts as well as the result error
bars of our method.

##  Appendix A The Advantage of Our HOI Representation

To verify the improvement obtained with our CLIP-based HOI representation, we
visualize the HOI representation $\hat{v}_{p}$ in feature space with t-SNE(van
der Maaten & Hinton, 2008). For clarity, we randomly sample 80 HOI categories,
and collect 50 samples for each category. For comparison, we also demonstrate
the object-based HOI representation derived from ‘Exp 9’ in Tab.2 (i.e., the
model without CLIP knowledge and using a random knowledge bank). As shown in
Fig.4, we observe that CLIP-based HOI representations for different HOI
categories are diverse and well separated in feature space, which is better
for HOI detection. In contrast, the object-based representations are not well
separated in feature space (see the red box region in Fig.4b). Besides, the
experimental results in the ablation study (ours v.s. ‘Exp 9’) also validate
the advantage of CLIP-based HOI representation, improving full mAP from 19.61
to 22.89.

##  Appendix B Ablation on CLIP Knowledge Integration

To further demonstrate the superiority of our CLIP knowledge integration
strategy, we study several proven techniques for CLIP knowledge transfer in
Tab. 3. In Abl 1, for each human-object pair, we directly infer the HOI scores
with CLIP by computing the cross-modal similarities between their visual union
region and the HOI prompts. Without introducing any HOI priors, the promising
results indicate the powerful generalization ability of CLIP and motivate the
design of incorporating CLIP knowledge for weakly-supervised HOI detection. In
Abl 2, we duplicate the experiment setting and results from Exp 8 in Tab. 2 of
the main paper. It is a simplified baseline model but initializes the visual
encoder with a ResNet50-FPN pretrained on COCO detection task. Then we
introduce three different CLIP knowledge transfer strategies (Abl 3-4 and
ours) based on Abl 2.

In Abl 3, we directly enhance baseline scores in Abl 2 with the CLIP
similarity scores in Abl 1 on the inference stage. Without bells and whistles,
we obtain 1.12 gain in Full mAP.

Furthermore, in Abl 4, we adopt a similar knowledge transfer strategy as GEN-
VLKT (Liao et al., 2022), where we initialize the HOI classifier
$\mathcal{F}_{P}$ with HOI prompt and regularize the global HOI representation
with CLIP image feature $v_{g}$. In detail, we first compute the global HOI
representation $v_{mean}$ with mean pooling on all pairwise HOI
representations, i.e.,
$v_{mean}=\mathop{MeanPool}\limits_{m}(\\{v_{p}^{m}\\}_{m=1}^{M})$. Here
$v_{p}^{m}$ is the holistic HOI representation (c.f. Sec. 3.2.3 in the main
paper) for $m$-th human-object pair. Then we develop an additional $L2$ loss
$\mathcal{L}_{reg}$ to transfer the knowledge from CLIP to HOI
representations: $\mathcal{L}_{reg}=L2(v_{mean},v_{g})$. The performance even
decreases slightly from 19.44 to 19.39, which might be caused by the
incompatibility of parameters between backbone network (ResNet50-FPN
pretrained on COCO) and $\mathcal{F}_{P}$ (HOI prompt embeddings from CLIP).
When directly applying the knowledge transfer strategy of GEN-VLKT to a
weakly-supervised setting, it is difficult to map the unmatched HOI
representation and classification weights to a joint space as the supervisory
signals are noisy.

Finally, our approach achieves the best performance compared with other
strategies, demonstrating the effectiveness of our bi-level knowledge
integration strategy.

Figure 4: The t-SNE visualization of CLIP-based HOI representation and object-
based HOI representation. Table 3: Ablation of different CLIP knowledge
integration strategies on HICO-DET dataset.

Methods | Experimental setting | mAP (%)
---|---|---
Full | Rare | Non-Rare
Abl 1 | CLIP inference score | 11.84 | 13.72 | 11.27
Abl 2 | RN50-FPN (COCO) + $\mathcal{F}_{P}$ random init. | 19.44 | 16.20 | 20.41
Abl 3 | RN50-FPN (COCO) + $\mathcal{F}_{P}$ random init. + CLIP inference score | 20.56 | 18.19 | 21.27
Abl 4 | RN50-FPN (COCO) + $\mathcal{F}_{P}$ HOI prompt init. + CLIP visual regularization | 19.39 | 15.12 | 20.66
ours | CLIP RN50 + HOI recognition + KTN + self-taught relatedness cls. | 22.89 | 22.41 | 23.03

##  Appendix C Spatial Feature Generation

Following (Zhang et al., 2021c), we generate the spatial feature
$v_{sp}\in\mathbb{R}^{D}$ for each pair of human-object proposals
$(\mathbf{x}_{h},\mathbf{x}_{o})$. Specifically, we first compute the bounding
boxes information for $\mathbf{x}_{h}$ and $\mathbf{x}_{o}$ separately,
including their center coordinates, widths, heights, aspect ratios and areas,
all normalized by the corresponding dimension of the image. We also encode
their relative spatial relations by estimating the intersection over union
(IoU), a ratio of the area of $\mathbf{x}_{h}$ and $\mathbf{x}_{o}$, a
directional encoding and the distance between center coordinates of
$\mathbf{x}_{h}$ and $\mathbf{x}_{o}$. We concatenate all the above-mentioned
preliminary spatial cues and obtain a spatial encoding
$\mathbf{p}\in\mathbb{R}_{+}^{18}$. To encode the second and higher order
combinations of different terms, the spatial encoding is concatenated with its
logarithm and then embedded to $v_{sp}$:
$v_{sp}=\mathcal{F}_{sp}([\mathbf{p};log(\mathbf{p}+\epsilon)])$. Where
$\epsilon>0$ is a small constant to guarantee the numerical stability, and
$\mathcal{F}_{sp}$ is a multi-layer fully connected network.

##  Appendix D Visualization of HOI Knowledge Bank $\mathcal{W}_{T}$

To further understand $\mathcal{W}_{T}$, we visualize the knowledge bank
features initialized by CLIP (Fig.5(a)) and learned from scratch (Fig.5(b)) in
feature space by t-SNE. It is worth noting that the knowledge bank learned
from scratch is derived from ‘Exp 9’ in Tab.2. As shown in Fig.5, we observe
that the knowledge features of HOI classes initialized with CLIP are more
discriminative than random initialized, and show a better clustering result
(e.g. the HOI classes in red box regions).

Figure 5: The t-SNE visualization of knowledge bank $\mathcal{W}_{T}$. (a) is
the knowledge bank distribution in feature space based on our CLIP-based HOI
representation while (b) is the knowledge bank learned from scratch (the model
in Tab.2-Exp 9) based on object-based HOI representation.

##  Appendix E Different designs of KTN

To further validate the effectiveness of our attention mechanism in KTN, we
compare our design with some variants in Tab. 4. First of all, we directly
encode the relation-level features within the union region to enhance the
pairwise representation rather than the external knowledge bank. As a result,
the mAP even decreases a little bit from 20.75 (Exp 6) to 20.69 (Exp 11). The
potential reason is that the union region contains more ambiguous visual
relations and background clutters, which are difficult to learn in a weak
setting. Besides, we also explore different normalization strategies in KTN.
The results in Tab. 4 demonstrate that $Softmax$ operation (ours) performs
better than uniform attention (Exp 12) or $Sigmoid$ operation (Exp 13),
indicating our attention mechanism is non-trivial and more effective on
aggregating the relational cues from HOI knowledge bank.

Table 4: Different network design of Knowledge Transfer Network (KTN).

Methods | Parameter initialization | CLIP Knowledge | SRC | mAP (%)
---|---|---|---|---
Backbone | knowledge bank | HOI recognition | KTN | score fusion | Full | Rare | Non-Rare
Exp 11 | CLIP RN50 | CLIP Text | ✓ | ✓(union) | - | ✓ | 20.69 | 19.55 | 21.04
Exp 12 | CLIP RN50 | CLIP Text | ✓ | ✓(uniform) | - | ✓ | 21.14 | 19.82 | 21.53
Exp 13 | CLIP RN50 | CLIP Text | ✓ | ✓(sigmoid) | - | ✓ | 21.28 | 19.27 | 21.88
ours | CLIP RN50 | CLIP Text | ✓ | ✓ | - | ✓ | 21.53 | 20.05 | 21.97

##  Appendix F Top-K positive pair selection for SRC

In this section we show the results of selecting top-2 and top-5 pairs as
positive in Tab. 5. We notice that there is a small performance drop, which is
likely to be caused by mislabeling more negative pairs as positive, resulting
in model learning with more noise.

Table 5: Ablation of top-K positive pair selection for SRC on HICO-DET
dataset.

Methods | mAP (%)
---|---
Full | Rare | Non-Rare
Top-5 | 22.45 | 21.61 | 22.70
Top-2 | 22.49 | 21.83 | 22.69
ours (Top-1) | 22.89 | 22.41 | 23.03

##  Appendix G The Prompt Generation for V-COCO

For the V-COCO dataset, each action has two different semantic roles
(’instrument’ and ’object’) for different objects, like ‘cut cake’ and ‘cut
with knife’. We use two different prompt templates to convert a HOI label to a
language sentence. For the former one, we take template “a person verb a/an
object”, and use “a person verb with object” for the latter.

##  Appendix H Evaluation Metric for V-COCO

V-COCO dataset has two scenarios for role AP evaluation. In Tab. 1, APS1&2
refer to ‘Average Precision in scenario 1&2’. V-COCO dataset has two different
annotations for HOIs: the first is a full label of (human location,
interaction type, object location, object type), and the second misses target
object (also denoted as ‘role’ in the original paper (Gupta & Malik, 2015))
annotations, and the label only includes (human location, interaction type).
For the second case, there are two different evaluation protocols (scenarios)
when taking a prediction as correct 444https://github.com/s-gupta/v-coco: In
scenario 1, it requires the interaction is correct & the overlap between the
human boxes is $>0.5$ & the corresponding role is empty, which is more
restricted; in scenario 2, it only requires the interaction is correct & the
overlap between the person boxes is $>0.5$.

Figure 6: The screenshot of the evaluation code in Explanation-HOI. (a) is the
original code while (b) is the correct one based on the standard evaluation
code. We use red rectangle boxes to highlight the most important differences

##  Appendix I Evaluation of Explanation-HOI

The Explanation-HOI (Baldassarre et al., 2020) has a misunderstanding of mAP
evaluation protocol. As shown in Fig.6(a) L200-L205, the Explanation-HOI only
takes some specific predicted HOIs into the evaluation process, which has the
same HOI labels as groundtruth HOIs. Thus, they ignore lots of false-positive
HOI predictions when calculating mAP, leading to an untrustable high mAP score
(reported in their original paper). In Fig.6(b) L204-L208, we evaluate all
predicted HOIs, which is the same as the standard evaluation protocol proposed
in HICO-DET (Chao et al., 2015). The correct results have already been
reported in Tab.1 in the main paper.

##  Appendix J Limitations

As described in Sec. 3.1, we adopt an external object detector to generate
human-object proposals and then recognize their interactions. Consequently,
our method is faced with two limitations brought by erroneous object detection
results. Firstly, the positive human-object pairs are not recalled if the
human or object proposals are not detected. Secondly, the proposals are kept
fixed during learning, which leads to the problem of inaccurate localization
and object types.

##  Appendix K Risk of using CLIP

For all the methods that adopt CLIP in their model design, there is a
potential risk of data leakage as CLIP has seen quite a lot of data during
pretraining. For HOI detection task, we cannot get access to CLIP dataset and
do not know the exact overlap between CLIP and HOI benchmarks (i.e., HICO-DET
and V-COCO), we carefully read Sec. 5 (Data Overlap Analysis) of the CLIP
paper (Radford et al., 2021b), including an analysis of the overlap between
its dataset with 35 popular datasets (HICO-DET and V-COCO are not included).
It shows the overlap is small (median is 2.2% and average is 3.2%) and the
influence is limited (“overall accuracy is rarely shifted by more than 0.1%
with only 7 datasets above this threshold”). Besides, the training text
accompanying an image in the CLIP dataset is often not related to the HOI
annotations. Thus, we think the risk is limited.

##  Appendix L License

The licenses of the assets used in our work are listed below, including open-
sourced CLIP model, HICO-DET dataset, and V-COCO dataset. As for HICO-DET, we
cannot find its license in the paper and the official project page. Thus we
provide the official project page instead here for clarity.

  1. 1.

CLIP: https://github.com/openai/CLIP MIT License

  2. 2.

VCOCO: https://github.com/s-gupta/v-coco/ MIT License

  3. 3.

HICO-DET: http://www-personal.umich.edu/ ywchao/hico/ No license