# Locality in collider tests of Quantum Mechanics with top quark pairs

Regina Demina  Gabriel Landi  Department of Physics and Astronomy, University
of Rochester, 206 Bausch and Lomb Hall, Rochester, NY
<EMAIL_ADDRESS>
###### Abstract

Tests of quantum properties of fundamental particles in high energy colliders
are starting to appear. However, such experiments may suffer from the locality
loophole. We argue for criteria that take into account the space-like
separation between measurements for the case of spin correlations in top quark
pairs produced at the LHC. We derive bounds considering three different
definitions of what constitutes the quantum measurement - the decay of top
quarks, the decay of $W$ bosons, and the stable decay products contacting a
macroscopic device.

##  I Introduction

The ATLAS and CMS Collaborations at the LHC recently reported [1, 2]
observations of entanglement between the spin degrees of freedom of top and
anti-top ($t$ and $\bar{t}$) quarks. The experiments found a level of spin
correlations in the $t\bar{t}$ pair that exceeded the Peres-Horodecki bound
[3, 4, 5] which all non-entangled systems must respect. These results prove
that the foundations of Quantum Mechanics (QM) can be successfully
investigated at the TeV scale, and also in systems with unstable particles.

In addition to observing entanglement, LHC experiments will also seek to
observe Bell Inequality violations (BIV) [6, 7, 8], in $t\bar{t}$ pairs and
other high-energy two-particle systems [9, 10, 11, 12, 13, 14, 15]. BIVs
provide proof of the non-classicality of quantum theory, by ruling out its
compatibility with local hidden variable theories. Since not all entangled
states violate a Bell inequality, BIVs are considered a stronger form of
correlations.

A number of loopholes has been identified in connection with an experimental
demonstration of BIVs [16, 17, 18]. In this work we investigate the
possibility of closing the so-called “locality loophole” [19, 20]. It has been
proven that a system of two parties, obeying the laws of classical physics,
can exactly reproduce the outcome of a quantum measurement on a maximally
entangled state with the exchange of just a few bits of classical information
[21]. Therefore, we should ask ourselves what constitutes a convincing
observation of non-classical properties in a collider setting, where the
absence of communication is not guaranteed.

Ideally, experimenters should restrict their study to events, where the QM
measurements are space-like separated, to enforce that no classical
information can be exchanged between the particles under consideration.
However, in a collider setting this cannot be accomplished on an event-by-
event basis. In this work we argue for criteria to enforce such condition on
average in an ensemble of $t\bar{t}$ events considered for a particular
analysis. Of course, we have no reason to believe that top quarks, or any
other pair of fundamental particles, actually possess the ability to exchange
information during their lifetime, nor the ability to conspire with their
partner to fake the presence of entanglement or Bell violations, when none is
actually present; our argument simply attempts to close a loophole that is
present in principle.

Figure 1:  Schematics of $t\bar{t}$ production with subsequent decay to $W$
bosons and $b$ quarks. Each $W$ boson then decays to a charged lepton and a
neutrino. The distance between the tops’ decay is denoted $\Delta
r(\text{top})$, while the distance between $W$’s decay is denoted $\Delta
r(\text{W})$.

The central question in this paradigm is: what constitutes a measurement? The
schematics of the $t\bar{t}$ production and decay is shown in Fig 1. Which
interaction should be considered as the act of measurement? Spin correlations,
and therefore entanglement and BIV, are measured via angular correlations in
the top quarks’ decay products [10, 22, 23]. Are these angles determined in
the moment top quarks decay? Is it, instead, the moment $W^{\pm}$ bosons decay
that constitutes the quantum measurement, since the particles that are
actually observed, leptons and light jets, are produced at that stage? Is it
the moment of the first contact of these final decay products with the
macroscopic apparatus [24]? The argument can continue, from the moment, when
the pixel detector integrates the charge of the first hit, to the moment, when
the first person looks at a plot. Without getting into a philosophical
argument, here we will derive criteria based on the requirements of space-like
separation of the following moments: the decays of the top and antitop quarks,
the decays of $W$ bosons, and the first contacts of leptons with the
macroscopic apparatus. There is also a possibility that space-like separated
top quark decays produce $W$ bosons that become time-like separate at the
moment of their decay. To exclude this possibility we also consider a
requirement that both types of decays are space-like separated. Out of these
criteria we will recommend the most stringent one.

A couple of remarks on the assumptions made in the presented analysis are
prudent. First, we treat short-lived top quarks and $W$ bosons as particles,
not as fields. Second, we assume that the location of the decay can be
inferred from the kinematics of the particle, while strictly speaking, the
Heisenberg uncertainty principle should be applied. It should be noted though
that both counter arguments are based on a quantum mechanical treatment of the
system, while we are trying to refute the classical explanation of
entanglement. Third, the fact that the particles are space-like separated at
the decay point does not exclude a possibility that their light cones
overlapped in the past. This is always true for particles moving at
subluminous speeds, which were in interaction at some point. Despite being in
contact the system remains in a mixed state at least until one of the top
quarks decays, hence the points of particles decays must be space-like
separated to exclude causal contact.

This work is organized as follows. After a brief introduction to top spin
correlations and entanglement, in Section II we propose criteria to close the
locality loophole based on $f$, the fraction of events where the measurements
on the top and anti-top quarks are space-like separated. Next, in Section III
we evaluate $f$, in the four definitions described above. For the ”top” and
”W” definitions we use Monte Carlo simulation of the top pair production and
decay kinematics. Since this is a straightforward task, we propose that
analyzers use their own best MC simulations to determine the relevant space-
like fractions. This way detector efficiency and resolution will also be taken
into account. For the ”top” and ”lepton” definitions we will also show
analytical calculations of $f$. We conclude in Section IV.

##  II Top spin correlations, entanglement, and Bell violations

Let us give a quick overview of the strategy used to measure spin
correlations, and the corresponding conditions for observing entanglement and
Bell inequality violations. The correlation between the spin orientations of
top and anti-top quarks is described by the spin correlation matrix $C_{ij}$,
which directly enters the top/anti-top spin density matrix,

$\rho=\frac{\mathbf{1}\otimes\mathbf{1}+B_{1i}\,\sigma_{i}\otimes\mathbf{1}+B_{2j}\,\mathbf{1}\otimes\sigma_{j}+C_{ij}\,\sigma_{i}\otimes\sigma_{j}}{4},$
(1)

where $i,j=1,2,3$ and $\sigma$ are the Pauli matrices. The vectors $B_{1i}$
and $B_{2j}$ represent the spin polarization of the top and anti-top quarks,
while the $C_{ij}$ matrix parameterizes their spin correlations. The angular
distribution of the top and anti-top quark decay products is used to evaluate
$C_{ij}$ experimentally, and therefore extract spin correlations. In some
regions of phase space, correlations are so strong that they can only be
explained by entanglement. According to the Peres-Horodecki criterion the
system is entangled if [25]

$\Delta_{E}=C_{nn}+|C_{rr}+C_{kk}|>1,\ $ (2)

where $C_{nn},C_{rr}$ and $C_{kk}$ are the diagonals of $C_{ij}$ in the
helicity basis $\\{k,r,n\\}$. An even stronger quantum connection is
manifested in the violation of Bell inequalities. It is convenient to consider
the CHSH formulation, which in our case reduces to [10]:

$B_{\pm}=|{C_{rr}\pm C_{nn}|>\sqrt{2}}\ ,$ (3)

where either condition is sufficient to establish a violation. Let’s assume we
know how to calculate $f$, the fraction of $t\bar{t}$ events where the quantum
measurements of spin take place at space-like separation. In [26], a proposal
was made to extend the condition in Eq. 2 to ensure its value cannot be
explained via classical communication. Here we continue the same argument to
$B_{\pm}$, the Bell markers of Eq. 3, proceeding as follows: the time-like
separated events are assumed to have maximum correlations, and therefore the
maximum mathematically allowed values of $\Delta_{E}^{\text{max}}=3$ and
$B_{\pm}^{\text{max}}=2$; the space-like separated events are assumed to have
the maximum allowed ”classical” values, i.e. those that do not require
entanglement or BIV, $\Delta_{E}^{\text{class}}=1$ and
$B_{\pm}^{\text{class}}=\sqrt{2}$. Then, with $f$ being the fraction of events
where the measuremens are space-like separated, the largest values that could
be explained by classical communication are:

$\Delta_{E}^{\star}=f\Delta_{E}^{\text{class}}+(1-f)\Delta_{E}^{\text{max}}=3-2f\
$ (4)

for entanglement, and

$B_{\pm}^{\star}=fB_{\pm}^{\text{class}}+(1-f)B_{\pm}^{\text{max}}=\sqrt{2}f+2(1-f)\
$ (5)

for Bell inequality violation.

##  III Space-like fraction

All that is left, at this point, is evaluating the probability $f$ that an
event is space-like. Let $\tau_{1}=(ct_{1},x_{1},y_{1},z_{1})$ and
$\tau_{2}=(ct_{2},x_{2},y_{2},z_{2})$ be the four dimensional coordinates of
the measurements of the top/anti-top quark spin orientation. To guarantee that
the connection between the top and anti-top quarks is of quantum nature, we
need to make sure that these points are separated by a space-like interval

$(x_{1}-x_{2})^{2}+(y_{1}-y_{2})^{2}+(z_{1}-z_{2})^{2}-(ct_{1}-ct_{2})^{2}>0\
.$ (6)

The values of $\tau_{1}$ and $\tau_{2}$ depend on the chosen definition of
quantum measurement, and we will investigate each one in detail next. It is
important to remember that experimentally we do not measure the top quark or
the $W$ boson decay length. Thus, in the cases of top quark and $W$ boson
decays, we can only determine $f$ on statistical basis. This is not an issue,
however, since the entanglement and BIV markers are also determined as
averages on an ensemble of events. In the lepton contact definition the
directions of leptons and the location of the their production vertex are
known event-by-event, hence the interval between first contacts with the beam
pipe can in principle be extracted for each event. (We will show that in
practice this is not necessary.)

###  III.1 The top quark decay

Figure 2: Top row: top quark decays, bottom row: $W$ boson decays. Left: time
from production to decay of particle vs that of anti-particle. Lines show the
limits of space-like interval for $M_{t\bar{t}}=400$ GeV (solid) and
$M_{t\bar{t}}=800$ GeV (dashed). Center and right: distance vs time difference
between particle decays for $M_{t\bar{t}}<400$ GeV (center) and
$M_{t\bar{t}}>800$ GeV (right). Events below the diagonal are time-like
separated, while the ones above the diagonal are space-like separated.

Let us consider the center of mass of the top and anti-top quark system. In
this reference frame the velocities of the two particles are equal in
magnitude and opposite in direction $\vec{\beta_{t}}=-\vec{\beta}_{\bar{t}}$.
For brevity we shall refer to the magnitude of these vectors as $\beta$, which
is related to the invariant mass of the $t\bar{t}$ system, $M_{t\bar{t}}$,
through the simple relation

$\beta=\sqrt{1-(2m_{t}/M_{t\bar{t}})^{2}}\ ,$ (7)

where $m_{t}$ is the top quark mass. The corresponding relativistic factor is
$\gamma=1/\sqrt{1-\beta^{2}}$.

The top quark is an unstable particle, and its lifetime is exponentially
distributed with a time constant corresponding to the top width
$\Gamma_{t}=1.42$ GeV. We can direct the $z$ axis along the direction of
flight of the top quark, so that the 4D coordinates of the top/anti-top decay
points are $(\gamma ct_{1},0,0,\beta\gamma ct_{1})$ and $(\gamma
ct_{2},0,0,-\beta\gamma ct_{2})$. To guarantee that the connection between the
top and anti-top quarks is of quantum nature we need to make sure that their
decays are separated by a space-like interval,

$(\beta\gamma ct_{1}+\beta\gamma ct_{2})^{2}-(\gamma ct_{1}-\gamma
ct_{2})^{2}>0\ .$ (8)

Thus, the condition for space-like separation is

$\frac{1-\beta}{1+\beta}t_{1}<t_{2}<\frac{1+\beta}{1-\beta}t_{1}\ .$ (9)

This is shown by the solid lines corresponding to different values of
$M_{t\bar{t}}$ in Fig. 2 (top left), obtained via a Monte Carlo simulation
accurate at NLO in QCD using POWHEG [27]. The events are distributed according
to the survival probability of each top that decays exponentially but is
boosted by the time dilation factor $\gamma$. Even though it is impossible to
say on an event-by-event basis when the decays are time-like or space-like
separated, we can approach the problem statistically. Integrating the
probability density function over the area between the solid lines we find a
simple relation,

$f=\beta\ .$ (10)

Note, that this fraction does not depend on the actual decay times. Since
$\beta$ depends on $M_{t\bar{t}}$ so does $f$, and indeed for $M_{t\bar{t}}$
going to infinity we find that $f$ approaches 1. To appreciate the dependence
of $\beta$ on $M_{t\bar{t}}$, in Fig. 2 we also split $t\bar{t}$ phase space
in the regions $M_{t\bar{t}}<400$ GeV (top center) and $M_{t\bar{t}}>800$ GeV
(top right). The fraction of the space-like separated events, above the
diagonal, is significantly lower for the events near production threshold
compared to those at high invariant mass.

###  III.2 The $W$ boson decay

Next we consider the definition of the quantum measurement based on the $W$
boson decays. The lifetime of the $W$ boson is exponentially distributed with
a time constant corresponding to its width, $\Gamma_{W}=2.085$ GeV. The two
measurements are then separated by the time and distance of the top quark
decays plus that of the $W$ boson decays, as shown schematically in Fig. 1.
The time elapsed from the $t\bar{t}$ production to decays of the $W^{+}$ and
$W^{-}$ bosons, also obtained via a Monte Carlo simulation, is shown in Fig. 2
(bottom left). The distance separation vs time separation between $W$ boson
decays is also shown for events with $M_{t\bar{t}}<400$ GeV (bottom center)
and $M_{t\bar{t}}>800$ GeV (bottom right). Similar to the top quark decays we
observe that the probability of space-like separation is significantly lower
for events near $t\bar{t}$ production threshold compared to those at high
invariant mass.

It is important to note that in this case more events are above the diagonal,
that is, space-like separated, compared to what we had found for the top quark
decay definition. We will come back to this point.

###  III.3 Leptons hitting the beam pipe

While top quark and $W$ boson decays happen on the scale of a femtometer, $1$
fm $=10^{-15}$ m, contact of leptons (or jets produced by light quarks) with
the macroscopic apparatus, presumably the beam pipe, happens on the scale of 1
cm, so leptons from $t\bar{t}$ decay can be considered originating from the
same point. Since leptons (and hadrons) have negligible mass compared to the
LHC collision energy, they can be considered as moving at the speed of light.
Let $l_{1}$ / $l_{2}$ be the distance from the production vertex to the
positions of first contact of the leptons with the beam pipe, and $t_{1}$ /
$t_{2}$ be the corresponding times. Let $\chi$ be the angle between the
direction of flight of the two particles. Then the distance between their
first contacts is

$\Delta l=\sqrt{l_{1}^{2}+l_{2}^{2}-2l_{1}l_{2}\cos\chi}\ ,$ (11)

and the time between the two events is

$\Delta t=|l_{1}/c-l_{2}/c|\ .$ (12)

Thus, the interval between the two events is

$\Delta l^{2}-(c\Delta t)^{2}=2l_{1}l_{2}(1-\cos\chi).$ (13)

This quantity is clearly non-negative. Hence, if we take the perspective that
the quantum measurements take place when leptons or jets contact the
macroscopic apparatus around the interaction region, the locality loophole is
automatically closed. This argument is similar to the one illustrated in Fig.
3 of [28] except the speed of electron or muon from top quark decay is much
closer to the speed of light than the speed of $\tau$ lepton produced at
$\sqrt{s}=10$ GeV, making the fraction of time-like events negligible.

Figure 3:  Legend inline. Left: Fraction of events with space-like separated
measurements using four different definitions. The solid line shows the result
of the analytical calculation, $f=\beta$, for the top quark decays. Center and
right: Critical values of $\Delta_{E}$ (center) and $B_{\pm}$ (right) as a
function of $M_{t\bar{t}}$ based on the considered definitions. The dotted
lines show the critical values in the absence of our proposed corrections and
coincide with the definition based on lepton contact.

###  III.4 Summary

In Fig. 3 (left) we show the space-like fraction $f$ as a function of
$M_{t\bar{t}}$ based on the top quark and $W$ boson decays and leptons hitting
the beam pipe. We also include a more stringent requirement that both types of
decays happen in a space-like interval. For low $M_{t\bar{t}}$ this
requirement reduces the fraction of space-like events, while for
$M_{t\bar{t}}>800$ GeV it essentially coincides with the requirement based on
top quark decay. For comparison we also show the result of the analytical
calculation for top quark decays, which is in excellent agreement with the
simulation.

Of course, we advocate for selecting the most stringent requirement of top
quarks and $W$ bosons both being in the space-like interval, when evaluating
$f$. Therefore, the critical values for the entanglement marker $\Delta_{E}$
and for the Bell markers $B_{\pm}$ become a function of $M_{t\bar{t}}$, and
are shown in Fig. 3 (center and right). If the observed value of $\Delta_{E}$
or $B_{\pm}$ exceeds these critical values with enough significance, we can
say that the detection of entanglement or of Bell violations cannot be argued
away by assuming classical communication.

##  IV Conclusion

In light of the observation of entanglement in $t\bar{t}$ pairs, of concrete
prospects for detecting Bell violations, and in general for a future Quantum
Information program at the LHC, it is crucial to carefully analyze our
strategies, to ensure that such counter-intuitive physical phenomena are
detected experimentally in a convincing manner. Part of this endeavour
consists of making sure ”alternative” explanations for quantum phenomena,
however unlikely, are ruled out. We analyzed the locality loophole, and
proposed criteria to establish that the observed values of $\Delta_{E}$ or
$B_{\pm}$ cannot be explained by classical communication. We considered three
different definitions for when the ”quantum measurement” of spin takes place:
at the top quark decay, at the $W$ boson decay, and at the lepton/jet contact
with the macroscopic apparatus. We showed that the space-like fraction of
events is the smallest, when requiring that both top quarks and $W$ bosons
decay within space-like interval. For high invariant masses, typically
required for the Bell violation this is almost identical to just the top quark
decay requirement. We provided numerical values for our proposed criteria. We
also show that if the measurement happens when particles hit the detector
there is no locality issue at all.

##  V Post Scriptum

In the argument presented above one critical assumption is that the top quark
decay time is not correlated with that of the antitop quark. Should that not
be the case, e.g. the events are clustered in the upper and lower wedges in
Fig. 2 (top left), while still individually following the exponential
distributions, the locality loophole cannot be closed. Since measuring top
quark decay length is out of the question, a system consisting of a pair of
$B$-mesons or $\tau$ leptons might be better suited for addressing this
potential issue.

## Acknowledgements

The authors thank Marcel Vos, Claudio Severi, Dorival Goncalves and Alan Barr
for stimulating discussions. RD acknowledges support from the U.S. Department
of Energy under the grant DE-SC0008475.

## References

  * Collaboration [2023] A. Collaboration, Observation of quantum entanglement in top-quark pairs using the atlas detector (2023), arXiv:2311.07288 [hep-ex] .
  * Collaboration [2024] C. Collaboration, Observation of quantum entanglement in top quark pair production in proton-proton collisions at $\sqrt{s}$ = 13 tev (2024), arXiv:2406.03976 [hep-ex] .
  * Peres [1996] A. Peres, Separability criterion for density matrices, Phys. Rev. Lett. 77, 1413 (1996).
  * Horodecki  _et al._ [1996] M. Horodecki, P. Horodecki, and R. Horodecki, Separability of mixed states: necessary and sufficient conditions, Physics Letters A 223, 1–8 (1996).
  * Afik and de Nova [2021] Y. Afik and J. R. M. de Nova, Entanglement and quantum tomography with top quarks at the lhc, The European Physical Journal Plus 136, 10.1140/epjp/s13360-021-01902-1 (2021).
  * Bell [1964] J. S. Bell, On the einstein podolsky rosen paradox, Physics Physique Fizika 1, 195 (1964).
  * Clauser  _et al._ [1969] J. F. Clauser, M. A. Horne, A. Shimony, and R. A. Holt, Proposed experiment to test local hidden-variable theories, Phys. Rev. Lett. 23, 880 (1969).
  * Brunner  _et al._ [2014] N. Brunner, D. Cavalcanti, S. Pironio, V. Scarani, and S. Wehner, Bell nonlocality, Rev. Mod. Phys. 86, 419 (2014).
  * Fabbrichesi  _et al._ [2021] M. Fabbrichesi, R. Floreanini, and G. Panizzo, Testing Bell Inequalities at the LHC with Top-Quark Pairs, Phys. Rev. Lett. 127, 161801 (2021), arXiv:2102.11883 [hep-ph] .
  * Severi  _et al._ [2022] C. Severi, C. D. E. Boschi, F. Maltoni, and M. Sioli, Quantum tops at the LHC: from entanglement to Bell inequalities, Eur. Phys. J. C 82, 285 (2022), arXiv:2110.10112 [hep-ph] .
  * Han  _et al._ [2023] T. Han, M. Low, and T. A. Wu, Quantum entanglement and bell inequality violation in semi-leptonic top decays (2023), arXiv:2310.17696 [hep-ph] .
  * Fabbri  _et al._ [2024] F. Fabbri, J. Howarth, and T. Maurin, Isolating semi-leptonic $H\rightarrow WW^{*}$decays for Bell inequality tests, Eur. Phys. J. C 84, 20 (2024), arXiv:2307.13783 [hep-ph] .
  * Barr [2022] A. J. Barr, Testing Bell inequalities in Higgs boson decays, Phys. Lett. B 825, 136866 (2022), arXiv:2106.01377 [hep-ph] .
  * Fabbrichesi  _et al._ [2023] M. Fabbrichesi, R. Floreanini, E. Gabrielli, and L. Marzola, Bell inequalities and quantum entanglement in weak gauge boson production at the LHC and future colliders, Eur. Phys. J. C 83, 823 (2023), arXiv:2302.00683 [hep-ph] .
  * Bi  _et al._ [2024] Q. Bi, Q.-H. Cao, K. Cheng, and H. Zhang, New observables for testing Bell inequalities in W boson pair production, Phys. Rev. D 109, 036022 (2024), arXiv:2307.14895 [hep-ph] .
  * Aspect  _et al._ [1982] A. Aspect, P. Grangier, and G. Roger, Experimental realization of einstein-podolsky-rosen-bohm gedankenexperiment: A new violation of bell’s inequalities, Phys. Rev. Lett. 49, 91 (1982).
  * Weihs  _et al._ [1998] G. Weihs, T. Jennewein, C. Simon, H. Weinfurter, and A. Zeilinger, Violation of bell’s inequality under strict einstein locality conditions, Phys. Rev. Lett. 81, 5039 (1998).
  * Scheidl  _et al._ [2010] T. Scheidl, R. Ursin, J. Kofler, S. Ramelow, X.-S. Ma, T. Herbst, L. Ratschbacher, A. Fedrizzi, N. K. Langford, T. Jennewein, and A. Zeilinger, Violation of local realism with freedom of choice, Proceedings of the National Academy of Sciences 107, 19708–19713 (2010).
  * Aspect [1975] A. Aspect, Proposed experiment to test separable hidden-variable theories, Physics Letters A 54, 117 (1975).
  * Aspect [1976] A. Aspect, Proposed experiment to test the nonseparability of quantum mechanics, Phys. Rev. D 14, 1944 (1976).
  * Brassard  _et al._ [1999] G. Brassard, R. Cleve, and A. Tapp, The Cost of exactly simulating quantum entanglement with classical communication, Phys. Rev. Lett. 83, 1874 (1999), arXiv:quant-ph/9901035 .
  * Dong  _et al._ [2024] Z. Dong, D. Gonçalves, K. Kong, and A. Navarro, Entanglement and bell inequalities with boosted ¡mml:math xmlns:mml=”http://www.w3.org/1998/math/mathml” display=”inline”¿¡mml:mi¿t¡/mml:mi¿¡mml:mover accent=”true”¿¡mml:mi¿t¡/mml:mi¿¡mml:mo stretchy=”false”¿¯¡/mml:mo¿¡/mml:mover¿¡/mml:math¿, Physical Review D 109, 10.1103/physrevd.109.115023 (2024).
  * Barr  _et al._ [2024] A. J. Barr, M. Fabbrichesi, R. Floreanini, E. Gabrielli, and L. Marzola, Quantum entanglement and bell inequality violation at colliders (2024), arXiv:2402.07972 [hep-ph] .
  * Afik  _et al._ [2024] Y. Afik, Y. Kats, J. R. M. de Nova, A. Soffer, and D. Uzan, Entanglement and bell nonlocality with bottom-quark pairs at hadron colliders (2024), arXiv:2406.04402 [hep-ph] .
  * Bernreuther and Si [2013] W. Bernreuther and Z.-G. Si, Top quark spin correlations and polarization at the LHC: standard model predictions and effects of anomalous top chromo moments, Phys. Lett. B 725, 115 (2013), [Erratum: Phys.Lett.B 744, 413–413 (2015)], arXiv:1305.2066 [hep-ph] .
  * Hayrapetyan  _et al._ [2024] A. Hayrapetyan _et al._ (CMS),  _Measurements of polarization, spin correlations, and entanglement in top quark pairs using lepton+jets events from pp collisions at $\sqrt{s}=13~{}\mathrm{TeV}$_, Tech. Rep. (CERN, Geneva, 2024).
  * Campbell  _et al._ [2015] J. M. Campbell, R. K. Ellis, P. Nason, and E. Re, Top-pair production and decay at nlo matched with parton showers, Journal of High Energy Physics 2015, 10.1007/jhep04(2015)114 (2015).
  * Ehatäht  _et al._ [2024] K. Ehatäht, M. Fabbrichesi, L. Marzola, and C. Veelken, Probing entanglement and testing Bell inequality violation with e+e-→$\tau$+$\tau$\- at Belle II, Phys. Rev. D 109, 032005 (2024), arXiv:2311.17555 [hep-ph] .


# One-Shot Affordance Detection

Hongchen Luo1 Wei Zhai1,3 Wei Zhai is an intern at JD Explore Academy. Jing
Zhang2 Corresponding Author Yang Cao122footnotemark: 2 Dacheng Tao3 1
University of Science and Technology of China, China
2 The University of Sydney, Australia
3 JD Explore Academy, JD.com, China
{lhc12<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Affordance detection refers to identifying the potential action possibilities
of objects in an image, which is an important ability for robot perception and
manipulation. To empower robots with this ability in unseen scenarios, we
consider the challenging one-shot affordance detection problem in this paper,
i.e., given a support image that depicts the action purpose, all objects in a
scene with the common affordance should be detected. To this end, we devise a
One-Shot Affordance Detection (OS-AD) network that firstly estimates the
purpose and then transfers it to help detect the common affordance from all
candidate images. Through collaboration learning, OS-AD can capture the common
characteristics between objects having the same underlying affordance and
learn a good adaptation capability for perceiving unseen affordances. Besides,
we build a Purpose-driven Affordance Dataset (PAD) by collecting and labeling
4k images from 31 affordance and 72 object categories. Experimental results
demonstrate the superiority of our model over previous representative ones in
terms of both objective metrics and visual quality. The benchmark suite is at
ProjectPage.

Figure 1: Illustration of perceiving affordance. Given a support image that
depicts the action purpose, all objects in a scene with the common affordance
could be detected.

##  1 Introduction

The concept of affordance was proposed by the ecological psychologist Gibson
Gibson (1977). It describes how the inherent “value” and “meanings” of objects
in an environment are directly perceived, and explains how this information
can be linked to the action possibilities offered to an organism by the
environment Hassanin et al. (2018). After Gibson put forward the definition of
affordance, a lot of cognitive psychologists made profound studies on the
relationship between object affordance and perceiver intention in recent
decades Heft (1989). In particular, perceiving object affordance in unseen
scenarios is a valuable capability and has a wide range of applications in
scene understanding, action recognition, robot manipulation, and Human-
Computer Interaction Zhang and Tao (2020).

To learn such capability of perceiving affordance, we consider the challenging
one-shot affordance detection 111“Detection” follows the term pixel-wise
detection task, which has also been used in the area of salient object
detection. problem in this paper, i.e., given a support image that depicts the
action purpose, all objects in a scene with the common affordance should be
detected (see Figure 1). Unlike the object detection/segmentation problem
Shaban et al. (2017), affordance and semantic categories of objects are highly
inter-correlated but do not imply each other. An object may have multiple
affordances (see Figure 2), e.g., the sofa can be used to sit or lie down.
Actually, the possible affordance depends on the person’s purpose in real-
world application scenarios. Directly learning the affordance from a single
image without the guidance of purpose makes the model tend to focus on the
statistically dominant affordances while ignoring other visual affordances
that are possibly suitable for completing the task.

Figure 2: The part (A) shows that objects usually have multiple affordance.
The part (B) shows that the objects with different semantic categories may
have the same affordance.

To address this problem: 1) We try to find clear hints about the action
purpose (i.e., via the subject and object locations Chen et al. (2020)) from a
single support image, which implicitly defines the object affordance and is a
reasonable setting in real-world unseen scenarios. 2) We adopt collaboration
learning to capture the inherent relationship between different objects to
counteract the interference caused by visual appearance differences and to
improve generalization. Specifically, we devise a novel One-Shot Affordance
Detection (OS-AD) network to solve the problem. Taking an image as support and
a set of images (5 images in this paper) as a query, the network first
captures the human-object interactions from the support image using a purpose
learning module (PLM) to encode the action purpose. Then, a purpose transfer
module (PTM) is devised to use the encoding of the action purpose to activate
the features in query images that have the common affordance. Finally, a
collaboration enhancement module (CEM) is devised to capture the intrinsic
relationships between objects with the same affordance and suppress
backgrounds that are irrelevant to the action purpose. In this way, OS-AD can
learn a good adaptation capability for perceiving unseen affordances.

Moreover, the existing datasets still have gaps relative to real application
scenarios, due to the limitation of its diversity. The affordance detection
for scene understanding and general applications should be able to learn from
the human-object interaction when the robot arrives at a new environment and
retrieves the objects in the environment, rather than just finding objects
with the same categories or similar appearance. To address the limitations of
the datasets, we propose the Purpose-driven Affordance Dataset (PAD), which
contains 4,002 diverse images covering 31 affordance categories as well as 72
object categories from different scenes. Moreover, we trained several
representative models to address the problem and compare them with our OS-AD
model comprehensively in terms of both objective evaluation metrics and visual
quality.

Contributions  (1) We introduce a new one-shot affordance detection problem
along with a benchmark to facilitate the research for empowering robots with
the ability to perceive unseen affordances in real-world scenarios. (2) We
propose a novel OS-AD network that can efficiently learn the action purpose
and use it to detect the common affordance of all objects in a scene via
collaboration learning, resulting in a good adaptation capability that can
deal with unseen affordances. (3) Experiments on the proposed PAD benchmark
demonstrate that OS-AD outperforms state-of-the-art models and can serve as a
strong baseline for future research.

Figure 3: Our One-Shot Affordance Detection (OS-AD) network. OS-AD consists of
three key modules: Purpose Learning Module (PLM), Purpose Transfer Module
(PTM), and Collaboration Enhancement Module (CEM), which are detailed in
Figure 4.

##  2 Related Work

###  2.1 Affordance Detection

The problem of visual affordance detection has been investigated for decades
Hassanin et al. (2018), it is a widely used strategy in AI community to
perceive action intentions and thereby to infer visual affordance from the
image/video of human and objects. Early works mainly attempted to establish
and learn an association between the apparent characteristics of objects and
their affordance for perceiving affordance. Myers et al. (2015) proposed a
framework for jointly locating and identifying the affordance of object parts
and presented the RGB-D Part Affordance dataset, which is the first pixel-wise
labeled affordance dataset. However, the affordances of objects do not simply
correspond to representational characteristics, which are shifted in response
to the state of interactions between objects and humans. Therefore, Chuang et
al. (2018) considered the problem of affordance reasoning in the real world by
taking into account both the physical world and the social norms imposed by
the society, and constructed the ADE-Affordance dataset based on ADE20k Zhou
et al. (2017). Since the change of an object’s affordance state is usually due
to the interaction between the human and the object, further research has
begun to consider human action as a cue for learning affordance. Fang et al.
(2018) used human-object interactions in demonstration videos to predict the
affordance regions of static objects via linking human actions to object
affordance, and proposed the OPRA dataset for affordance reasoning. Different
from the above existing works, our proposed method aims to explore the
multiplicity of affordance by a collaborative learning strategy. To a certain
extent, our work conforms to Gibson’s definition of affordance that “it
implies the complementarity of the animal and the environment”. Since there
exist multiple potential complementarities between animal and environment, it
leads to multiple possibilities of particular affordance. To address this
issue, we present a novel task of one-shot affordance detection, in which
action intension is introduced through support image to alleviate the
multiplicity of affordance.

###  2.2 One-Shot Learning

Recently, one-shot learning has received a lot of attention and substantial
progress has been made based on metric learning using the siamese neural
network He et al. (2021); Koch et al. (2015); Snell et al. (2017). Besides,
some works build upon meta-learning and generation models to achieve one-shot
learning. Specifically, Michaelis et al. (2018) proposed the problem of one-
shot segmentation in clutter, finding and segmenting a previously unseen
object in a cluttered scene based on a single instruction example. Zhu et al.
(2019) proposed a one-shot texture retrieval, given an example of a new
reference texture, detecting and segmenting all the pixels of the same texture
category within an arbitrary image. Different from these studies, we aim to
identify the potential action possibilities (affordance) of objects in unseen
scenarios, given a single support image that implicitly defines the action
purpose without an explicit mask to denote the object affordance, which is
very challenging but of practical value.

##  3 Method

Our One-Shot Affordance Detection (OS-AD) network is shown in Figure 3. During
training, it receives a single support image that implicitly defines the
affordance via human-object interactions, e.g., in the form of bounding boxes
of the person and the object interacting with the person, which are weaker
signals and easier to acquire than a pixel-wise affordance mask. Meanwhile, a
set of query images (5 in this paper) containing objects with the same
affordance. Our goal is to estimate the action purpose from the support image,
transfer it to the query images, and learn to segment all the objects having
the same affordance. To this end, three modules named PLM, PTM, and CEM are
devised. During testing, OS-AD can recognize the common affordance from any
number of query images given a single support image.

Figure 4: (a) PLM aims to estimate action purpose from the human-object
interaction in the support image. (b) PTM transfers the action purpose to the
query images via an attention mechanism to enhance the relevant features. (c)
CEM captures the intrinsic characteristics between objects having the common
affordance to learn a better affordance perceiving ability.

###  3.1 Framework

Given a set of query images $\mathcal{I}=\\{I_{1},...,I_{n}\\}$ and a support
image $I_{\text{sup}}$ containing human-object interactions, we first extract
the features of $\mathcal{I}$ and $I_{\text{sup}}$ using resnet50 He et al.
(2016) to obtain their feature representations
$\mathcal{X}=\\{X_{1},...,X_{n}\\}$ and $X_{\text{sup}}$, respectively. We
then feed $X_{\text{sup}}$ and the bounding boxes of the human and object into
PLM to extract information about the human-object interaction and encode the
action purpose. As shown in Figure 3, our network needs to estimate the
purpose that the person wants to swing. Subsequently, we feed the feature
representation of the action purpose and $\mathcal{X}$ into PTM, which
transfers the action purpose to $\mathcal{X}$, enabling the network to learn
to attend those objects with that affordance. Finally, the encoded features
are fed into CEM, which captures the intrinsic connections between objects
with the same affordance and suppresses irrelevant object regions, predicting
the common affordance masks.

###  3.2 Purpose Learning Module

As shown in Figure 4 (a), we feed the $X_{\text{sup}}$ and the bounding boxes
of the person and object into the purpose learning module to estimate the
action purpose of the person. We first use the bounding boxes to extract the
features of the person and object from $X_{\text{sup}}$ as $X_{\text{H}}$ and
$X_{\text{O}}$. Subsequently, inspired by human-object interaction and visual
relationship detection Zhan et al. (2019, 2020), the features of the instance
(person or object) can provide guidance information about where the network
should focus, we use the features of the person and object to activate
$X_{\text{sup}}$ respectively to obtain $M_{\text{H}}$ and $M_{\text{O}}$:

$M_{\text{O}}=\text{Softmax}(f_{\text{O}}\otimes X_{\text{sup}})\otimes
X_{\text{sup}},$ (1) $M_{H}=\text{Softmax}(f_{\text{H}}\otimes
X_{\text{sup}})\otimes X_{\text{sup}},$ (2)

where $f_{\text{O}}$ and $f_{\text{H}}$ are the representations of
$X_{\text{O}}$ and $X_{\text{H}}$ respectively after global maximum pooling
(GMP). $\otimes$ denotes element-wise product. We then use $f_{\text{O}}$ to
guide the network to focus on the area of human-object interaction
$M_{\text{HO}}$:

$M_{\text{HO}}=\text{Conv}(f_{\text{O}}\otimes X_{\text{H}}).$ (3)

Then, $M_{\text{HO}}$ is multiplied with $M_{\text{H}}$ and $M_{\text{O}}$
respectively to activate the relevant features of human-object interaction on
the global features, which are added together and go through a GMP layer to
get the encoding of the action purpose $F_{\text{sup}}$:

$F_{\text{sup}}=\text{MaxPooling}((M_{\text{HO}}\odot
M_{\text{H}})+(M_{\text{HO}}\odot M_{\text{O}})),$ (4)

where $\odot$ denotes position-wise dot product.

###  3.3 Purpose Transfer Module

After obtaining $F_{\text{sup}}$, we transfer it to each query image to
segment the objects that can fulfill that purpose, i.e., having the common
affordance. As shown in Figure 4 (b), we perform correlation calculations
between $F_{\text{sup}}$ and each position of the query image. After
normalization, we get the attention probability map that may contain objects
with the common affordance. It is used to enhance the feature that may contain
the affordance object and suppress the area that does not contain the
affordance object. Finally, the attended feature is added back to the original
feature to get $\mathcal{X}_{\text{T}}$:

$X_{\text{T}_{i}}=X_{i}+\text{Softmax}(X_{i}\otimes F_{\text{sup}})\otimes
X_{i},i\in[1,n].$ (5)

###  3.4 Collaboration Enhancement Module

There are some inherent characteristics between objects with the same
affordance. For example, both cups and bowls have concave areas in the middle
so they can be used to hold water. Thus, by discovering intrinsic features
from the query image collection Ma et al. (2020), objects with the same
affordance can be activated and unrelated regions can be suppressed, resulting
in better segmentation results (see Figure 4 (c)).

Inspired by the Expectation-Maximization (E-M) Dempster et al. (1977)
algorithm and EMANet Li et al. (2019), we run the “E-step” and “M-step”
alternately to obtain a compact set of bases, which are used to reconstruct
the feature map of query images. During the iterative process, this set of
bases can learn the common properties between the input set of features and
thus can segment the common feature attributes. For a set of input feature
maps $\mathcal{X}_{\text{T}}=\\{X_{\text{T}_{1}},...,X_{\text{T}_{n}}\\}$
($X_{T_{i}}\in R^{N\times C}$, $N=W\times H$), they firstly go through a
convolution layer to get $\mathcal{F}=\\{F_{1},...,F_{n}\\}$. Then, a set of
base $\mu\in R^{K\times C}$ is initialized. The E-step estimates the latent
variable $\mathcal{Z}=\\{Z_{1},...,Z_{n}\\}$, $\mathcal{Z}\in R^{N\times K}$.
The weight of the $k$-$th$ basis for the $j$-$th$ pixel of the $i$-$th$ image
is calculated as:

$Z_{ijk}=\frac{\kappa(f_{ij},\mu_{k})}{\sum_{l=1}^{K}\kappa(f_{ij},\mu_{l})},$
(6)

where $f_{ij}$ is the feature of the $j$-$th$ position of the $i$-$th$ image,
$\kappa$ denotes the exponential kernel function, i.e., $exp(,)$. Thus, we
have $Z_{i}=Softmax(F_{i}\mu^{T})$. The M-step updates the bases and
calculates $\mu$ as the weighted average of $\mathcal{F}$:

$\mu_{k}=\frac{\sum_{i=1}^{n}\sum_{j=1}^{L}z_{ijk}f_{ij}}{\sum_{i=1}^{n}\sum_{j=1}^{L}z_{ijk}}.$
(7)

After convergence of the E-M iterations, we use $\mu$ and $\mathcal{Z}$ to
reconstruct $\mathcal{X}$ to obtain $\tilde{\mathcal{F}}$:
$\tilde{F}_{i}=Z_{i}\mu$. Finally, $\tilde{\mathcal{F}}$ is mapped to the
residue space of $\mathcal{X}$ using convolution and added to $\mathcal{X}$ to
obtain $\hat{\mathcal{X}}$, i.e.,
$\hat{X}_{i}=X_{i}+\text{Conv}(\tilde{F}_{i}$),
$\hat{\mathcal{X}}=\\{\hat{X}_{1},...,\hat{X}_{n}\\}$.

###  3.5 Decoder

For the $i$-$th$ image, the output of the $m-th$ layer is
$P_{i}^{m}=\text{Conv}(\text{Unsample}(\text{Conv}(X_{i}^{m})+P_{i}^{m+1})$,
$m\in[1,4]$, where $P_{i}^{5}=\text{Conv}(\hat{X_{i}^{5}})$. A convolutional
prediction layer is used to get the final output, i.e.,
$D_{i}^{m}=\text{Conv}(P_{i}^{m})$, $m\in[1,5]$.

The cross-entropy loss is used as the training objective of our network. For
the prediction $D_{i}^{m}$ from the $m$-$th$ layer of the $i$-$th$ image, we
calculate the loss $\mathcal{L}_{i}^{m}$ as:

$\begin{split}\mathcal{L}_{i}^{m}=-\frac{1}{N}&(\sum_{j\in
Y_{+}}logPr(y_{ij}=1|D_{ij}^{m})\\\ &+\sum_{j\in
Y_{-}}logPr(y_{ij}=0|D_{ij}^{m})),\end{split}$ (8)

where $Pr(y_{ij}=1|D_{ij}^{m})$ is the prediction map in which each pixel
denotes the affordance confidence. $Y_{+}$ and $Y_{-}$ denote the affordance
region pixels set and non-affordance pixels set, $N=H\times W$. The final
training objective is defined as:

$\mathcal{L}=\sum_{i=1}^{N}\sum_{m=1}^{5}\mathcal{L}_{i}^{m}.$ (9)

##  4 Experiments

###  4.1 Dataset

Data collection. We construct the Purpose-driven Affordance Dataset (PAD) with
images mainly from ILSVRC Russakovsky et al. (2015), COCO Lin et al. (2014),
etc. The affordance categories of the dataset is shown in Figure 5. Different
images are collected according to the class keywords of the objects to make
sure that the images in these datasets cover different scenes and appearance
features of the objects, which constitute a challenging benchmark.
Additionally, to make the dataset richer in categories, we obtain a part of
the images from the Internet to expand the dataset. The description of each
affordance and the object categories it contains are provided in the
supplementary material. To benchmark different models comprehensively, we
follow the k-fold evaluation protocol, where k is 3 in this paper. To this
end, the dataset is divided into three parts with non-overlapped categories,
where any two of them are used for training while the left part is used for
testing. See the supplementary material for more details about the setting.
Data annotation. 1) Category labeling: We create a hierarchy for PAD dataset
by selecting 72 common object categories (e.g. “cups”, “bowls”, “basketballs”,
etc.) and assigning affordance labels to each object category. An affordance
category may contain multiple subcategories, e.g. objects with “Swing”
affordance include “tennis rackets”, “table tennis rackets”, “golf clubs”,
“baseball bats”, etc., and the appearance feature of these objects varies
considerably. An object may contain more than one affordance category, e.g.
“sofa” and “bench” have both “Sit” and “Lie” affordance labels. 2) Mask
labeling: For the images from COCO etc., we select a portion of the mask
labels from the original dataset. Since the above dataset may not annotate all
the objects belonging to the same affordance, we select these images and
annotate them manually. The images downloaded from the web are also manually
annotated with the objects containing the affordance category. 3) Purpose
image labeling: 656 purpose images are downloaded from the Internet which
contains diverse human-object interactions. We annotate the bounding boxes of
human and object accordingly.

Figure 5: The classification system of the Purpose-driven Affordance Dataset (PAD), which contains 4,002 images covering 72 object classes and 31 affordance classes. See the supplementary material for more details of each affordance category. | Metrices |  $i$ = 1 |  $i$ = 2 |  $i$ = 3 | Mean $\pm$ Std
---|---|---|---|---|---
UNet |  IoU $\uparrow$ | $.186$ | $.215$ | $.226$ | $.209_{\pm.018}$
$E_{\phi}$ $\uparrow$ | $.574$ | $.558$ | $.578$ | $.570_{\pm.009}$
CC $\uparrow$ | $.338$ | $.377$ | $.344$ | $.353_{\pm.017}$
MAE $\downarrow$ | $.162$ | $.163$ | $.169$ | $.165_{\pm.003}$
PSPNet |  IoU $\uparrow$ | $.261$ | $.244$ | $.295$ | $.267_{\pm.021}$
$E_{\phi}$ $\uparrow$ | $.640$ | $.601$ | $.636$ | $.626_{\pm.018}$
CC $\uparrow$ | $.427$ | $.409$ | $.402$ | $.413_{\pm.011}$
MAE $\downarrow$ | $.144$ | $.142$ | $.137$ | $.141_{\pm.003}$
CPD |  IoU $\uparrow$ | $.258$ | $.256$ | $.317$ | $.277_{\pm.028}$
$E_{\phi}$ $\uparrow$ | $.615$ | $.601$ | $.630$ | $.615_{\pm.012}$
CC $\uparrow$ | $.413$ | $.386$ | $.433$ | $.411_{\pm.019}$
MAE $\downarrow$ | $.123$ | $.106$ | $.132$ | $.120_{\pm.011}$
BASNet |  IoU $\uparrow$ | $.239$ | $.263$ | $.281$ | $.261_{\pm.017}$
$E_{\phi}$ $\uparrow$ | $.604$ | $.598$ | $.628$ | $.610_{\pm.013}$
CC $\uparrow$ | $.310$ | $.318$ | $.339$ | $.322_{\pm.012}$
MAE $\downarrow$ | $.130$ | $.124$ | $.146$ | $.133_{\pm.009}$
CSNet |  IoU$\uparrow$ | $.173$ | $.210$ | $.238$ | $.207_{\pm.027}$
$E_{\phi}$ $\uparrow$ | $.557$ | $.555$ | $.557$ | $.556_{\pm.001}$
CC $\uparrow$ | $.394$ | $.392$ | $.386$ | $.391_{\pm.003}$
MAE $\downarrow$ | $.184$ | $.162$ | $.184$ | $.177_{\pm.010}$
CoEGNet |  IoU $\uparrow$ | $.281$ | $.262$ | $.289$ | $.277_{\pm.017}$
$E_{\phi}$ $\uparrow$ | $.674$ | $.637$ | $.645$ | $.652_{\pm.016}$
CC $\uparrow$ | $.389$ | $.350$ | $.362$ | $.367_{\pm.016}$
MAE $\downarrow$ | $.116$ | $.110$ | $.134$ | $.120_{\pm.010}$
Ours |  IoU $\uparrow$ | $.401$ | $.375$ | $.407$ | $\bm{.394}_{\pm.011}$
$E_{\phi}$ $\uparrow$ | $.732$ | $.653$ | $.687$ | $\bm{.691}_{\pm.032}$
CC $\uparrow$ | $.540$ | $.507$ | $.501$ | $\bm{.519}_{\pm.017}$
MAE $\downarrow$ | $.103$ | $.116$ | $.122$ | $\bm{.114}_{\pm.008}$
Table 1: Results of different methods on the PAD dataset under the 3-fold test setting. The best results are in bold. | Kick | Play-4 | Contain-2 | Brush | Jump
---|---|---|---|---|---
UNet | $.256$ | $.332$ | $.253$ | $.082$ | $.065$
PSPNet | $.450$ | $.452$ | $.283$ | $.108$ | $.122$
CPD | $.500$ | $.505$ | $.347$ | $.140$ | $.162$
BASNet | $.358$ | $.408$ | $.328$ | $.114$ | $.109$
CSNet | $.260$ | $.393$ | $.243$ | $.084$ | $.043$
CoEGNet | $.458$ | $.453$ | $.321$ | $.107$ | $.132$
Ours | $\bm{.615}$ | $\bm{.593}$ | $\bm{.414}$ | $\bm{.246}$ | $\bm{.206}$
Table 2: Comparison of part results on different affordance categories, using
IoU as a metric. Where “Container-2” refers to the object with the
availability of liquid and “Play-4” refers to the objects that a person blows
through his mouth to make a sound. See the supplementary material for more
details. Figure 6: Visual results of different segmentation, saliency
detection, co-saliency detection models and our OS-AD on the PAD dataset. OS-
AD can learn a better capability to perceive the affordance of objects, i.e.,
segmenting all objects that complete this purpose and suppressing object
regions that are not related to affordance. “Container-2” refers to the
affordance category that objects can fill liquid.

###  4.2 Benchmark Setting

To provide a comprehensive evaluation, four widely used metrics are used to
evaluate the performance of affordance segmentation. The IoU metric for
segmentation task Long et al. (2015) is adopted in our task. Mean Absolute
Error (MAE) Perazzi et al. (2012) is used to measure the absolute error
between the prediction and ground truth (GT). E-measure ($\bm{E_{\phi}}$) Fan
et al. (2018) is a metric that combines local pixels and image-level average
values to jointly capture image-level statistics and local pixel matching
information. Pearson Correlation Coefficient (CC) Le Meur et al. (2007) is
used to evaluate the correlation between the prediction and GT. We report the
average metric score for all test images.

Our method is implemented in Pytorch and trained with the Adam optimizer
Kingma and Ba (2014). The backbone is resnet50 He et al. (2016). The input is
randomly clipped from $360\times 360$ to $320\times 320$ with random
horizontal flipping. We train the model for $40$ epochs on a single NVIDIA
$1080$ti GPU with an initial learning rate $1e$-$4$. The number of bases in
the collaboration enhancement module is set to $K$=$256$. The number of E-M
iteration steps is 3. Besides, two segmentation models (UNet Ronneberger et
al. (2015), PSPNet Zhao et al. (2017)), three saliency detection models (CPD
Wu et al. (2019), BASNet Qin et al. (2019), CSNet Gao et al. (2020)) and one
co-saliency detection models (CoEGNet Fan et al. (2021)) are chosen for
comparison.

###  4.3 Quantitative and Qualitative Comparisons

The results are shown in Table 1, our results outperform all methods on all
metrics in the three one-shot learning settings. Measured by the mean IoU
value of the three-fold test, our model improves by 42.2% compared to the co-
saliency detection approach. Compared to the best saliency detection model and
the best segmentation model, our method improves by 42.2% and 47.6%
respectively. This indicates that our method can effectively use the action
purpose extracted from the support image to guide the segmentation of query
images, rather than simply constructing a link between apparent features and
affordance. The prediction results generated by each model are shown in Figure
6. It can be seen that our method can detect all the objects with common
affordance according to the human purpose. In the 1st and 3rd rows of “Kick”,
the soccer ball and the punching bag are completely different in terms of
apparent features, but both are kickable objects. This demonstrates that our
method can detect object affordance in the unseen scenarios by extracting the
action purpose and collaborative learning strategy. Meanwhile, we calculate
the IoU for each affordance class, and Table 2 shows the results for several
of them. The results from the first three columns show that our method can
detect object affordance with different apparent features but belonging to the
same affordance, indicating that our model can capture the common relation of
the objects in the unseen scenarios well and detect them more completely. The
two categories with the worst results out of all the results are “Jump” and
“Brush”, respectively. The possible reasons for the poor performance are as
follows: while in the “Brush” category, its accompanying actions are not
obvious and the size of the objects are relatively small, making it more
difficult for the network to extract the features of the objects. In summary,
our approach can learn object affordances well by transferring human purposes
to new objects and capturing common features between objects using
collaborative learning strategies, with good generalization ability on the
task of detecting object affordance in unseen scenarios.

##  5 Conclusion

In this paper, we make the first attempt to deal with a challenging task named
one-shot affordance detection, which has practical meaning for real-world
applications, such as empowering robots with the ability to perceive unseen
affordance. Specifically, we devise a novel one-shot affordance detection (OS-
AD) network and construct a benchmark named purpose-driven affordance dataset
(PAD). OS-AD exceeds representative models adapted from related areas such as
segmentation and saliency detection, which can serve as a strong baseline for
this task. PAD contains 4k images with diverse affordance and object
categories from different scenes, which can serve as a pioneer test bed for
this task. In the future, we plan to deploy the model in a real-world robot to
complete some well-defined tasks and evaluate its performance. Going a step
further, we attempt to make such a test suite available to the community via a
web-based interface.

## Acknowledgments

This work was supported by the National Key R&D Program of China under Grant
2020AAA0105701, the National Natural Science Foundation of China (NSFC) under
Grants 61872327, the Fundamental Research Funds for the Central Universities
under Grant WK2380000001.

## References

  * Chen et al. [2020] Zhe Chen, Jing Zhang, and Dacheng Tao.  Recursive context routing for object detection.  IJCV, 2020.
  * Chuang et al. [2018] Ching-Yao Chuang, Jiaman Li, Antonio Torralba, and Sanja Fidler.  Learning to act properly: Predicting and explaining affordances from images.  In CVPR, 2018.
  * Dempster et al. [1977] Arthur P Dempster, Nan M Laird, and Donald B Rubin.  Maximum likelihood from incomplete data via the em algorithm.  Journal of the Royal Statistical Society: Series B (Methodological), 1977.
  * Fan et al. [2018] Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji.  Enhanced-alignment measure for binary foreground map evaluation.  In IJCAI, 2018.
  * Fan et al. [2021] Deng-Ping Fan, Tengpeng Li, Zheng Lin, Ge-Peng Ji, Dingwen Zhang, Ming-Ming Cheng, Huazhu Fu, and Jianbing Shen.  Re-thinking co-salient object detection.  TPAMI, 2021.
  * Fang et al. [2018] Kuan Fang, Te-Lin Wu, Daniel Yang, Silvio Savarese, and Joseph J. Lim.  Demo2vec: Reasoning object affordances from online videos.  In CVPR, 2018.
  * Gao et al. [2020] Shang-Hua Gao, Yong-Qiang Tan, Ming-Ming Cheng, Chengze Lu, Yunpeng Chen, and Shuicheng Yan.  Highly efficient salient object detection with 100k parameters.  In ECCV, 2020.
  * Gibson [1977] James J Gibson.  The theory of affordances.  Hilldale, 1977.
  * Hassanin et al. [2018] Mohammed Hassanin, Salman Khan, and Murat Tahtali.  Visual affordance and function understanding: A survey.  arXiv, 2018.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * He et al. [2021] Haoyu He, Jing Zhang, Bhavani Thuraisingham, and Dacheng Tao.  Progressive one-shot human parsing.  AAAI, 2021.
  * Heft [1989] Harry Heft.  Affordances and the body: An intentional analysis of gibson’s ecological approach to visual perception.  J Theory Soc Behav, 1989.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv, 2014.
  * Koch et al. [2015] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov.  Siamese neural networks for one-shot image recognition.  In ICML workshop, 2015.
  * Le Meur et al. [2007] Olivier Le Meur, Patrick Le Callet, and Dominique Barba.  Predicting visual fixations on video based on low-level visual features.  Vision research, 2007.
  * Li et al. [2019] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu.  Expectation-maximization attention networks for semantic segmentation.  In ICCV, 2019.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * Long et al. [2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In CVPR, 2015.
  * Ma et al. [2020] Benteng Ma, Jing Zhang, Yong Xia, and Dacheng Tao.  Auto learning attention.  NeurIPS, 2020.
  * Michaelis et al. [2018] Claudio Michaelis, Matthias Bethge, and Alexander S Ecker.  One-shot segmentation in clutter.  arXiv, 2018.
  * Myers et al. [2015] Austin Myers, Ching L Teo, Cornelia Fermüller, and Yiannis Aloimonos.  Affordance detection of tool parts from geometric features.  In ICRA, 2015.
  * Perazzi et al. [2012] Federico Perazzi, Philipp Krähenbühl, Yael Pritch, and Alexander Hornung.  Saliency filters: Contrast based filtering for salient region detection.  In CVPR, 2012.
  * Qin et al. [2019] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand.  Basnet: Boundary-aware salient object detection.  In CVPR, 2019.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, 2015.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  IJCV, 2015.
  * Shaban et al. [2017] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots.  One-shot learning for semantic segmentation.  arXiv, 2017.
  * Snell et al. [2017] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In NeurIPS, 2017.
  * Wu et al. [2019] Zhe Wu, Li Su, and Qingming Huang.  Cascaded partial decoder for fast and accurate salient object detection.  In CVPR, 2019.
  * Zhan et al. [2019] Yibing Zhan, Jun Yu, Ting Yu, and Dacheng Tao.  On exploring undetermined relationships for visual relationship detection.  In CVPR, 2019.
  * Zhan et al. [2020] Yibing Zhan, Jun Yu, Ting Yu, and Dacheng Tao.  Multi-task compositional network for visual relationship detection.  IJCV, 2020.
  * Zhang and Tao [2020] Jing Zhang and Dacheng Tao.  Empowering things with intelligence: A survey of the progress, challenges, and opportunities in artificial intelligence of things.  IEEE Internet of Things Journal, 2020.
  * Zhao et al. [2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In CVPR, 2017.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In CVPR, 2017.
  * Zhu et al. [2019] Kai Zhu, Wei Zhai, Zheng-Jun Zha, and Yang Cao.  One-shot texture retrieval with global context metric.  IJCAI, 2019.