# Reducing $g^{(2)}(0)$ of a parametric down-conversion source via photon-
number resolution with superconducting nanowire detectors

S. Sempere-Llagostera  1, * G. S. Thekkadath  1, 2 R. B. Patel  1, 3 W. S.
Kolthammer  1 and I. A. Walmsley1  1Department of Physics, Imperial College
London, Prince Consort Rd., London SW7 2AZ, UK
2National Research Council of Canada, 100 Sussex Drive, Ottawa, Ontario K1A
0R6, Canada
3Clarendon Laboratory, University of Oxford, Parks Road, Oxford, OX1 3PU, UK
<EMAIL_ADDRESS>
###### Abstract

Multiphoton contributions pose a significant challenge for the realisation of
heralded single-photon sources (HSPS) based on nonlinear processes. In this
work, we improve the quality of single photons generated in this way by
harnessing the photon-number resolving (PNR) capabilities of commercial
superconducting nanowire single-photon detectors (SNSPDs). We report a $13\pm
0.4\%$ reduction in the intensity correlation function $g^{(2)}(0)$ even with
a collection efficiency in the photon source of only $29.6\%$. Our work
demonstrates the first application of the PNR capabilities of SNSPDs and shows
improvement in the quality of an HSPS with widely available technology.

††journal: osajournal††articletype: Research Article

##  1 Introduction

Single-photon sources based on nonlinear processes, also referred to as HSPS
[1], and single-photon detectors have played a prominent role in the fast-
paced progress of quantum optics and quantum information experiments in the
last two decades [2, 3]. Their applications span from quantum communications
[4, 5] and quantum metrology [6, 7, 8, 9] to quantum computing [10, 11, 12].
In an ideal HSPS, spontaneous scattering through nonlinear wave mixing
probabilitistically creates a photon in each of two optical modes. One mode is
measured, and the detection of a photon confirms the presence of the other,
which is taken as the output. A shortcoming of this method, however, is
exhibited when the nonlinear interaction instead generates twin beams that
each contain more or less than one photon. For a given mean photon-number per
pulse, $\langle n\rangle$, the probability to generate $k$ photon pairs
follows a thermal distribution, $\langle n\rangle^{k}/(1+\langle
n\rangle)^{k+1}$. For multiple applications [13], it is required to employ
true single photons and these high photon-number contributions can be
problematic. To work around this problem, researchers tend to work in the so-
called low-squeezing regime, i.e. $\langle n\rangle\rightarrow 0$, to minimize
the contribution of such higher order-terms [14, 15, 16]. The increasing
demand for higher photon rates to conduct experiments involving a growing
number of photons [17, 18, 19] restricts the possibility of working in this
regime. This presents a clear trade-off between photon statistics free from
multiphoton noise and detection rates. Another option to reduce the noise from
multiphoton emission is to harness the strong photon-number correlations of
two-mode squeezed vacuum sources, and perform a photon-number resolving
measurement in the herald arm to filter out the unwanted multiphoton events.
This technique is commonly used in the field [20, 21, 22, 23], and, in this
sense, the role played by the detection system is crucial.

Detectors that operate at the single-photon level may be broadly categorized
when considering their photon-number resolution capabilities as:  (i) photon-
number resolving detectors;  (ii) non-photon-number resolving (NPNR) detectors
, and,  (iii) pseudo-photon-number resolving (PPNR) detectors.  Photon-number
resolving detectors, e.g. transition-edge sensors (TESs) [24], can resolve up
to 20 photons with efficiencies close to 100 %, but their long recovery time
($\sim\mu$s) severely limits their ability to prepare heralded single-photon
states with a high repetition rate. Also, the temperature at which they
operate, $\sim 100$ mK, require advanced cryogenic systems. On the other hand,
non-photon-number resolving detectors, such as avalanche photo-diodes (APD)
[25], can only discriminate between states with “no photons” or “some
photons”. These are also referred to as “click” detectors and are ubiquous in
the quantum optics community and widely available. Signatures of photon-number
resolution capabilities have been shown in the past using APDs by measuring
very weak avalanches and analysing the analog output signal [26, 27]. And,
lastly, pseudo-photon-number resolving detectors, such as spatially-
multiplexed [28, 29] or time-multiplexed [30, 31, 32, 33] NPNR detectors,
split the incoming state into many NPNR detectors and look for coincident
detections. They are the usual approach to resolve the number of photons in
the herald arm, implying the need to consume many resources, either in the
form of physical detectors or repetition rate, to have accurate photon-number
resolution and sufficient dynamic range [34].

Superconducting nanowire single-photon detectors, which are widely used in the
community [35, 36], are usually treated as NPNR detectors. These exhibit high
detection efficiencies [37], low timing jitter [38] and low dark count rates
[39], all of which are essential for producing HSPS. Recently, Cahall et al.
[40] showed that the electronic trace outputted by a conventional single-pixel
SNSPD in conjunction with a cryogenic low-noise amplifier does in fact contain
photon-number information. Due to a time- and photon-number dependent
resistance, by looking at the slope of the rising edge of the waveforms one
can extract information on the photon-number of the detected state. The PNR
ability of SNSPDs was further improved in Refs. [41] and [42], which employed
a conventional single-pixel SNSPD system with a low-noise cryogenic amplifier,
and a superconducting nanowire with a taper for impedance-matching,
respectively. These works show that SNSPDs are not merely NPNR detectors, but
rather fast and highly efficient photon-number resolving detectors, posing
them as the ideal tool to realise HSPS.

In this work, we employ the PNR capabilities of a commercial SNSPD system to
discard multiphoton contributions of a HSPS and show an experimental reduction
of the second-order correlation function $g^{(2)}(0)$. We report a very good
agreement with the theoretical model and show that further $g^{(2)}(0)$
reduction can be obtained by improving the collection efficiency of the photon
source. In this regard, to the best of the authors’ knowledge, this is the
first application of the PNR capabilities of conventional SNSPDs to a
practical problem. Moreover, we show that this system of SNSPDs can be
employed for this purpose without any further modification, such as a
specialized cryogenic low-noise amplifier. We believe this technique provides
a practical tool that can readily benefit research labs employing similar
detectors.

##  2 Using a conventional SNSPDs to reduce the second-order correlation
function

Single-photon sources based on nonlinear processes do not generate single
photons but, in the ideal case, two-mode squeezed vacuum states (TMSV). These
are described by the following pure state

$\ket{\psi}=\sqrt{1-\lambda^{2}}\sum_{n=0}\lambda^{n}|n,n\rangle,$ (1)

where $\lambda$ is the squeezing parameter. This description is a
simplification used for clarity and can be extended to multiple modes to more
precisely describe real sources. By measuring one of the modes, i.e. the
herald, we can approximate the state on the other mode, the signal, to be a
single photon based on a detection event if $\lambda\lll 1$. A usual metric to
characterise single-photon sources and assess their quality is the second-
order correlation function, $g^{(2)}(0)$, [43] namely

$g^{(2)}(0)=\frac{\langle\hat{n}(\hat{n}-1)\rangle}{\langle\hat{n}\rangle^{2}}=\frac{\sum
n(n-1)P(n)}{(\sum nP(n))^{2}},$ (2)

where $P(n)$ is the photon-number probability distribution of the output state
and the sums run over all possible values of photon-number $n$. Ideally, a
single-photon state should have $g^{(2)}(0)=0$, since $P(n>1)=0$. For heralded
single-photon sources, however, due to the limited collection and detection
efficiencies and the higher-order photon-number contributions, we obtain
$g^{(2)}(0)>0$ and, to a first-order approximation, proportional to
$\lambda^{2}$ [44, 45].

Formally, the measurement process can be described by a positive operator-
valued measure (POVM),

$\hat{\Pi}(\mathbf{c})=\sum_{n=0}^{\infty}c_{n}\outerproduct{n}{n},$ (3)

where $\mathbf{c}$ is a vector that depends on the detector parameters and the
measurement outcome. Table 1 shows the different POVMs corresponding to the
different types of detectors, where we only consider the efficiency as their
main imperfection whilst ignoring other negligible contributions such as dark
counts and background light.

Detector | POVM
---|---
Click | $\hat{\Pi}_{0}^{\mathrm{click}}=\sum_{n=0}^{\infty}(1-\eta)^{n}\outerproduct{n}{n},\quad\hat{\Pi}_{1}^{\mathrm{click}}=\sum_{n=0}^{\infty}\left[1-(1-\eta)^{n}\right]\outerproduct{n}{n}$
Pseudo-PNR | $\hat{\Pi}_{m}^{\mathrm{PPNR}}=\sum_{n=m}^{\infty}\binom{M_{\mathrm{det}}}{m}\sum_{j=0}^{m}(-1)^{j}\binom{m}{j}\left((1-\eta)+\frac{\eta(m-j)}{M_{\mathrm{det}}}\right)^{n}\outerproduct{n}{n}$
PNR | $\hat{\Pi}_{m}^{\mathrm{PNR}}=\sum_{n=m}^{\infty}\binom{n}{m}(1-\eta)^{n-m}\eta^{m}\outerproduct{n}{n}$
Table 1: POVM elements for the different types of detectors, where $m$ denotes
the number of photons (or clicks) detected, $\eta$ is the detection efficiency
and $M_{\mathrm{det}}$ are the number of detectors used in the pseudo-PNR
scheme. These will determine the photon-number probability distributions and,
therefore, will influence the photon statistics of the signal arm. Note that
in the limit $\eta\rightarrow 1$, the PNR detector converges to
$\hat{\Pi}_{m}^{\mathrm{PNR}}=\outerproduct{m}{m}$.

We can use these to calculate the probability to trigger the heralding arm,
either if there is a detection in a click detector or a single count in a PNR
detector, as

$P_{\mathrm{h}}(\lambda,\mathbf{c})=\mathrm{Tr}\left(\hat{\Pi}(\mathbf{c})\outerproduct{\psi}{\psi}\right)=(1-\lambda^{2})\sum_{n=0}c_{n}|\lambda|^{2n},$
(4)

and to compute the resulting state of the signal arm given that the heralding
arm triggered as

$\rho_{\mathrm{s}}(\lambda,\mathbf{c})=\frac{\mathrm{Tr_{\mathrm{h}}}(\hat{\Pi}(\mathbf{c})\outerproduct{\psi}{\psi})}{P_{\mathrm{h}}(\lambda,\mathbf{c})}=\frac{\sum_{n=0}c_{n}|\lambda|^{2n}\outerproduct{n}{n}}{\sum_{n=0}c_{n}|\lambda|^{2n}}.$
(5)

From these output states we can then calculate $g^{(2)}(0)$, namely

$g^{(2)}(0)=\sum_{n=0}c_{n}|\lambda|^{2n}\frac{\sum_{n=0}n(n-1)c_{n}|\lambda|^{2n}}{(\sum_{n=0}nc_{n}|\lambda|^{2n})^{2}}.$
(6)

Ideally, for a PNR detector with unit efficiency, i.e. $\eta=1$, we find
$g^{(2)}(0)=0$ for all values of $\lambda$, since the heralding detection
event only occurs when precisely one pair of photons is generated. Otherwise,
as shown in Fig. 3, as soon as $\eta<1$, we see that the $g^{(2)}(0)$
increases with the probability of triggering the heralding arm.

The experimental setup is shown in Fig. 1. A type-II collinear ppKTP waveguide
source is pumped by a frequency-doubled mode-locked laser at 1550 nm [46]. The
length of the photon wavepacket is $1$ ps, orders of magnitude smaller than
the rise-time of the electrical signal from the SNSPD system ($400$ ps). This
ensures that if a pulse contains multiple photons, all will impinge the
detector at the same time relative to the detector state [47]. Similarly, the
repetition rate of the laser is 10 MHz, allowing enough time for the SNSPDs to
return to their detection-ready state after a detection event, following a
reset time of $<100$ ns as shown in Fig. 2(a).

The herald and signal arms are split using a Wollaston prism and collected
into single-mode fibers. The herald is directed straight to the detector,
$\mathrm{D}_{\mathrm{h}}$, and on the signal arm, we employ a Hanbury-Brown
and Twiss (HBT) setup using a 50:50 fiber beam splitter to analyse the photon
statistics on detectors $\mathrm{D}_{1}$ and $\mathrm{D}_{2}$. The SNSPDs are
a Single Quantum Eos CS system with detection efficiencies of 80-86%, $250$ Hz
dark counts and 11-17 ps timing jitter. The electronic signals from the SNSPDs
are recorded by a high-bandwidth (8 GHz) and high-sampling rate (25 GS/s)
oscilloscope (Tektronix DPO70804C) and saved for analysis. We use the herald
arm as the trigger for the oscilloscope and record a total of $3\times 10^{5}$
traces per pump power. In total, we record data for 16 different pump powers,
8 for each herald efficiency.

Figure 1: Schematic of the setup used for the experiment. We have excluded
most of the optics used for the source for clarity. See details in the main
text. WP: Wollaston prism, FBS: fibre beam splitter, SNSPDs: superconducting
nanowire single-photon detectors.

Fig. 2(a) shows an example of a collection of traces obtained. In order to
resolve the number of photons contained in each measurement we follow a
similar procedure as in Refs. [40, 41]. In particular, the resistance induced
in the nanowire due to the absorption of an optical wavepacket exhibits a
photon-number dependence that can be resolved by examining the slope of the
rising edge of the output signal. Here, we aim to see if this still holds for
our unmodified system, i.e. without the addition of a low-noise cryogenic
amplifier. In contrast to [40], we do no additional analog signal processing
prior to recording the raw signal on the oscilloscope straight from the
detector. For each waveform we fit a line, $y=ax+b$, to part of the rising
edge and extract its slope, $a$. We find that good results are achieved when
the linear fit is applied to data between the 10% to 60% level of the rising
edge of each waveform. This assures we only capture information contained in
the linear region of the rising edge. The inset zooms into the relevant region
and the colouring distinguishes the photon-number associated with each of the
traces, obtained from the histogram of the slopes, as explained below.

Fig. 2(b) displays the histogram of slopes obtained in the herald arm for
different input states and shows oscillations due to the different photon-
number detection events. We determine the photon-number bins by fitting a sum
of Gaussian functions to the distribution and using the intersection between
the Gaussians as the bin edges. For example, all events whose slope lie within
[0, 0.58] mV/ns are assigned the photon-number $n=1$. To confirm these
oscillations correspond to actual photon-number detections, we check that the
resulting photon-number distribution, $P_{\mathrm{exp}}(n)$, follows
approximately the expected thermal distribution, $P_{\mathrm{th}}(n)$,
obtained from the inferred values for the squeezing, $\lambda$, and the herald
efficiencies, $\eta_{\mathrm{h}}$. The distance,

$D(p,q)=\frac{1}{2}\sum_{i}|p_{i}-q_{i}|,$ (7)

gives an indication of how close two probability distributions are to one
another. It varies from 0 to 1 for identical to completely non-overlapping
distributions, respectively. We find
$D(P_{\mathrm{exp}}(n),P_{\mathrm{th}}(n))<4\times 10^{-3}$ for all the
squeezing parameters used. For a click detector, the distance would have been
approximately $3\times 10^{-2}$, suggesting that this binning procedure is
assigning the correct photon-number to each detection event.

To quantify the resolving capabilities of the detector, we approximate its
response using a Gaussian model, similar to the bin assignment we described
earlier. In this sense, the probability to measure one photon when there were
two photons in the state, $P(1|2)$, or vice versa, $P(2|1)$, can be computed
from the area of the normalised Gaussians for each photon-number detection
events’ that fall under the other photon-number region. These probabilities
will ultimately limit the resolution of the detector. The conditional
probability $P(1|2)$ will limit the value of $g^{(2)}(0)$, whereas $P(2|1)$
will affect the output rate since these detections will not be considered a
herald. The value found for these probabilities are $P(1|2)=4.47\%$ and
$P(2|1)=0.2\%$. Higher-order conditional probabilities, i.e. $P(1|n>2)$, are
negligible. Likewise, the probability of obtaining a background detection in
the high squeezing regime, i.e. high $P_{\mathrm{h}}$, is less than $10^{-2}$
times the probability than obtaining a detection from the signal, and,
therefore, the effect of dark counts on the POVMs can be neglected.

Figure 2: (a) Example of 100 traces recorded (random colouring) with the inset
showing a zoom into the rising edge (pink box) where the traces have been
coloured according to the assigned photon-number with them depending on the
slope of the rising edge. (b) Histogram of the rising edges slope for three
different input states with $\eta_{\mathrm{h}}=16.22\%$. The shaded areas
correspond to different photon-number allocations, as indicated. The black
dashed line corresponds to the sum of Gaussians fitting.

From the recorded traces, we can compute the number of herald detections in
$\mathrm{D}_{\mathrm{h}}$, $S_{\mathrm{h}}$, the coincidences with detector
$\mathrm{D}_{1}$, $C_{1,\mathrm{h}}$, with detector $\mathrm{D}_{2}$,
$C_{2,\mathrm{h}}$, and the threefold coincidences, $C_{1,2,\mathrm{h}}$. From
this data, we can approximate the second-order correlation function as [43]

$g^{(2)}(0)\approx\frac{S_{\mathrm{h}}C_{1,2,\mathrm{h}}}{C_{1,\mathrm{h}}C_{2,\mathrm{h}}}.$
(8)

In Fig. 3(a), we show the experimentally obtained $g^{(2)}(0)$ as a function
of the heralding probability, $P_{\mathrm{h}}$, calculated by  (i) discarding
events for which the herald arm detected more than one photon [squares] and
(ii) keeping all detection events [triangles]  . We perform this measurement
for two herald efficiencies, $\eta_{\mathrm{h}}=16.22\pm 0.13\%$ and
$\eta_{\mathrm{h}}=29.61\pm 0.31\%$, obtained from a Klyshko measurement [48]
as $\eta_{\mathrm{h}}=(C_{1,\mathrm{h}}+C_{2,\mathrm{h}})/(S_{1}+S_{2})$,
where $S_{1}$ and $S_{2}$ are the single counts in detectors $\mathrm{D}_{1}$
and $\mathrm{D}_{2}$, respectively. We vary the herald efficiency by changing
the coupling efficiency into the single-mode fiber in the herald arm. In order
to remove the dependence of the herald efficiency with the squeezing
parameter, we fit a line to the experimental values obtained as a function of
the pump power and take the y-axis intercept value as the herald efficiency.
We find up to a $13\pm 0.4$% reduction in the $g^{(2)}(0)$ when discarding
multiphoton events. Our results agree very well with the expected behaviour
for PNR detectors and show a significant improvement over both pseudo-PNR
(with $M_{\mathrm{det}}=2$) and click detectors [see Eq. (6)]. For high
heralding probabilities the experimental data deviates from the theory due to
the approximation in Eq. (8) [49], since $P(1)\gg P(2)$ no longer holds. Fig.
3(b) shows how by applying a tighter filtering, i.e. setting a lower
discrimination voltage between $n=1$ and $n=2$, we can lower $g^{(2)}(0)$ at
the expense of fewer detection events. If the discrimination point is too high
such that we also include the $n>1$ events, the second-order correlation
function converges to the unfiltered one, as expected.

Figure 3:  (a) Second-order correlation $g^{(2)}(0)$ as a function of the
probability to have a detection on the trigger for two herald efficiencies
$\eta_{\mathrm{h}}$. The click, pseudo-PNR and PNR model curves correspond to
the dashed, dotted and dash-dotted lines, respectively. The triangles
correspond to the unfiltered data, i.e. click detection, and the squares
correspond to the data after filtering the multiphoton events in the herald
arm. The error bars (smaller than marker size) correspond to Poissonian
statistics, i.e. $\sigma=\sqrt{N}$, and show one standard deviation. (b)
Second-order correlation $g^{(2)}(0)$ for $\lambda^{2}=0.08$ and
$\eta_{\mathrm{h}}=0.162$ as a function of the discrimination point between
$n=1$ and $n=2$. It shows that we can further reduce $g^{(2)}(0)$ (left axis)
by sacrificing more detection events (right axis). The dashed and solid lines
correspond to the unfiltered and filtered $g^{(2)}(0)$, respectively. The
dashed black line indicates the intersection point between the two Gaussians.
This is the discrimination point utilised for the data shown in the top
figure.

Fig. 4 shows a 3D plot of the ratio between the $g^{(2)}(0)$ when using PNR
detectors as opposed to click detectors, namely

$r=\frac{g^{(2)}(0)_{\mathrm{click}}}{g^{(2)}(0)_{\mathrm{PNR}}}.$ (9)

as a function of the squeezing parameter, $\lambda$, and the herald
efficiency, $\eta_{\mathrm{h}}$. We observe that the improvement ratio is
independent of the squeezing applied and it is more sensitive to the herald
efficiency of the source. As exhibited, for higher herald efficiencies as the
one achieved in Ref. [15], of approximately 80 %, the improvement ratio would
be close to $r=3$.

Figure 4:  Theoretical curve for the improvement ratio $r$ as a function of
the squeezing and herald efficiency. The red arrows and lines indicate the
herald efficiencies studied experimentally and emphasize how much improvement
these detectors can add when using more efficient HSPS. The colormap
qualitatively reflects the value of $r$.

##  3 Discussion and conclusions

We have demonstrated a reduction in the second-order correlation function of
an HSPS of $13\pm 0.4\%$ by discriminating the photon-number in the herald arm
using commercial SNSPDs. On the detection side, we use the information
contained in the slope of the rising edge of the electronic traces to extract
the photon-number information, obtaining conditional probabilities $P(1|2)$
and $P(2|1)$ of $4.47\%$ and $0.2\%$, respectively. These values reflect that
this type of system can be reliably employed for the purpose of discriminating
the detection of one photon from the detection of many photons. Note that in
this application, the most important feature needed from the detector is to be
able to discern the detection of $n=1$ from $n>1$, while achieving high
fidelities in resolving higher order photon-numbers, e.g. two-photon versus
three-photon absorptions, is less relevant. The low value for $P(1|2)$
indicates that the reduction of $g^{(2)}(0)$ is primarily limited by the
herald efficiency of the source rather than by the PNR capabilities of the
detector. We envisage this technique can be straightforwardly applied in other
high-efficiency photon sources such as [50, 15, 20]. The procedure shown in
this work will either allow for higher detection rates with the same quality
in terms of photon statistics or photon statistics with reduced multiphoton
noise. Moreover, the substitution of pseudo-PNR schemes by a single SNSPD for
multiphoton detection would entail a relevant reduction in hardware
requirements.

We foresee these detection concepts to lead to widespread tools for quantum
optics and quantum information protocols when moderate photon-number
resolution is desirable. For instance, these could be employed for state
engineering based on measurement post-selection [51, 52] or to increase the
computational cost of simulating Gaussian Boson Sampling experiments such as
[53] as shown in [54]. An important next step is the development of fast and
affordable electronics that can analyze detection signals on the fly, perhaps
using a high speed field-programmable gate array (FPGA). These techniques will
benefit from further advances in SNSPDs, such as potential improvements to
dynamic range that allow resolution of larger numbers of photons [42]. Funding
This work was supported by: Engineering and Physical Sciences Research Council
via the Quantum Systems Engineering Skills Hub and the Quantum Computing and
Simulation Hub (P510257, T001062). National Research Council of Canada.

Acknowledgments We thank Single Quantum for loaning us the superconducting
nanowire detector system employed in this work. We also thank Jamie Francis-
Jones for his contributions at the early stage of this project.

Disclosures The authors declare no conflicts of interest.

Data availability Data underlying the results presented in this paper are not
publicly available at this time but may be obtained from the authors upon
reasonable request.

## References

  * [1] C. K. Hong and L. Mandel, “Experimental realization of a localized one-photon state,” Physical Review Letters 56, 58 (1986).
  * [2] J. W. Pan, Z. B. Chen, C. Y. Lu, H. Weinfurter, A. Zeilinger, and M. Zukowski, “Multiphoton entanglement and interferometry,” Reviews of Modern Physics 84, 777 (2012).
  * [3] M. D. Eisaman, J. Fan, A. Migdall, and S. V. Polyakov, “Invited review article: Single-photon sources and detectors,” Review of Scientific Instruments 82, 071101 (2011).
  * [4] N. Gisin and R. Thew, “Quantum communication,” Nature Photonics 1, 165–171 (2007).
  * [5] M. Krenn, M. Malik, T. Scheidl, R. Ursin, and A. Zeilinger, “Quantum communication with photons,” in _Optics in Our Time,_ (Springer International Publishing, 2016), pp. 455–482.
  * [6] V. Giovannetti, S. Lloyd, and L. Maccone, “Advances in quantum metrology,” Nature Photonics 5, 222–229 (2011).
  * [7] K. R. Motes, J. P. Olson, E. J. Rabeaux, J. P. Dowling, S. J. Olson, and P. P. Rohde, “Linear Optical Quantum Metrology with Single Photons: Exploiting Spontaneously Generated Entanglement to Beat the Shot-Noise Limit,” Physical Review Letters 114 (2015).
  * [8] S. Slussarenko, M. M. Weston, H. M. Chrzanowski, L. K. Shalm, V. B. Verma, S. W. Nam, and G. J. Pryde, “Unconditional violation of the shot-noise limit in photonic quantum metrology,” Nature Photonics 11, 700–703 (2017).
  * [9] G. S. Thekkadath, M. E. Mycroft, B. A. Bell, C. G. Wade, A. Eckstein, D. S. Phillips, R. B. Patel, A. Buraczewski, A. E. Lita, T. Gerrits, S. W. Nam, M. Stobińska, A. I. Lvovsky, and I. A. Walmsley, “Quantum-enhanced interferometry with large heralded photon-number states,” npj Quantum Inf 6, 1–6 (2020).
  * [10] P. Kok, W. J. Munro, K. Nemoto, T. C. Ralph, J. P. Dowling, and G. J. Milburn, “Linear optical quantum computing with photonic qubits,” Reviews of Modern Physics 79, 135 (2007).
  * [11] J. B. Spring, B. J. Metcalf, P. C. Humphreys, W. S. Kolthammer, X. M. Jin, M. Barbieri, A. Datta, N. Thomas-Peter, N. K. Langford, D. Kundys, J. C. Gates, B. J. Smith, P. G. Smith, and I. A. Walmsley, “Boson sampling on a photonic chip,” Science 339, 798–801 (2013).
  * [12] S. Bartolucci, P. Birchall, H. Bombin, H. Cable, C. Dawson, M. Gimeno-Segovia, E. Johnston, K. Kieling, N. Nickerson, M. Pant, F. Pastawski, T. Rudolph, and C. Sparrow, “Fusion-based quantum computation,” arXiv:2101.09310 (2021).
  * [13] S. Bartolucci, P. M. Birchall, M. Gimeno-Segovia, E. Johnston, K. Kieling, M. Pant, T. Rudolph, J. Smith, C. Sparrow, and M. D. Vidrighin, “Creation of Entangled Photonic States Using Linear Optics,” arXiv:2106.13825 (2021).
  * [14] A. J. Menssen, A. E. Jones, B. J. Metcalf, M. C. Tichy, S. Barz, W. S. Kolthammer, and I. A. Walmsley, “Distinguishability and Many-Particle Interference,” Physical Review Letters 118 (2017).
  * [15] H. S. Zhong, Y. Li, W. Li, L. C. Peng, Z. E. Su, Y. Hu, Y. M. He, X. Ding, W. Zhang, H. Li, L. Zhang, Z. Wang, L. You, X. L. Wang, X. Jiang, L. Li, Y. A. Chen, N. L. Liu, C. Y. Lu, and J. W. Pan, “12-Photon Entanglement and Scalable Scattershot Boson Sampling with Optimal Entangled-Photon Pairs from Parametric Down-Conversion,” Physical Review Letters 121, 250505 (2018).
  * [16] A. E. Jones, A. J. Menssen, H. M. Chrzanowski, T. A. Wolterink, V. S. Shchesnovich, and I. A. Walmsley, “Multiparticle interference of pairwise distinguishable photons,” Physical Review Letters 125 (2020).
  * [17] N. Spagnolo, C. Vitelli, M. Bentivegna, D. J. Brod, A. Crespi, F. Flamini, S. Giacomini, G. Milani, R. Ramponi, P. Mataloni, R. Osellame, E. F. Galvão, and F. Sciarrino, “Experimental validation of photonic boson sampling,” Nature Photonics 8, 615–620 (2014).
  * [18] B. A. Bell, G. S. Thekkadath, R. Ge, X. Cai, and I. A. Walmsley, “Testing multi-photon interference on a silicon chip,” Opt. Express 27, 35646–35658 (2019).
  * [19] H. Wang, J. Qin, X. Ding, M.-C. Chen, S. Chen, X. You, Y.-M. He, X. Jiang, L. You, Z. Wang, C. Schneider, J. J. Renema, S. Höfling, C.-Y. Lu, and J.-W. Pan, “Boson sampling with 20 input photons and a 60-mode interferometer in a $1{0}^{14}$-dimensional hilbert space,” Phys. Rev. Lett. 123, 250503 (2019).
  * [20] F. Kaneda and P. G. Kwiat, “High-efficiency single-photon generation via large-scale active time multiplexing,” Science Advances 5 (2019).
  * [21] R. Okamoto, S. Takeuchi, and T. Kiyohara, “Realization of multiplexing of heralded single photon sources using photon number resolving detectors,” Optics Express 24, 27288–27297 (2016).
  * [22] T. Horikiri, Y. Takeno, A. Yabushita, and T. Kobayashi, “Photon-number-resolved heralded-photon source for improved quantum key distribution,” Physical Review A 76, 012306 (2007).
  * [23] K. Tsujino, S. Takeuchi, and K. Sasaki, “Detailed analysis of the fidelity of quantum teleportation using photons: Considering real experimental parameters,” Physical Review A 66, 042314 (2002).
  * [24] A. E. Lita, A. J. Miller, and S. W. Nam, “Counting near-infrared single-photons with 95% efficiency,” Optics Express 16, 3032–3040 (2008).
  * [25] F. Ceccarelli, G. Acconcia, A. Gulinatti, M. Ghioni, I. Rech, and R. Osellame, “Recent Advances and Future Perspectives of Single-Photon Avalanche Diodes for Quantum Photonics Applications,” Advanced Quantum Technologies 4, 2000102 (2021).
  * [26] B. E. Kardynał, Z. L. Yuan, and A. J. Shields, “An avalanchephotodiode-based photon-number-resolving detector,” Nature Photonics 2, 425–428 (2008).
  * [27] G. Wu, Y. Jian, E. Wu, and H. Zeng, “Photon-number-resolving detection based on InGaAs/InP avalanche photodiode in the sub-saturated mode,” Optics Express 17, 18782–18787 (2009).
  * [28] L. A. Jiang, E. A. Dauler, and J. T. Chang, “Photon-number-resolving detector with 10 bits of resolution,” Physical Review A 75, 062325 (2007).
  * [29] R. Heilmann, J. Sperling, A. Perez-Leija, M. Gräfe, M. Heinrich, S. Nolte, W. Vogel, and A. Szameit, “Harnessing click detectors for the genuine characterization of light states,” Scientific Reports 6, 1–6 (2016).
  * [30] M. J. Fitch, B. C. Jacobs, T. B. Pittman, and J. D. Franson, “Photon-number resolution using time-multiplexed single-photon detectors,” Physical Review A 68, 6 (2003).
  * [31] D. Achilles, C. Silberhorn, C. Śliwa, K. Banaszek, and I. A. Walmsley, “Fiber-assisted detection with photon number resolution,” Optics Letters 28, 2387–2389 (2003).
  * [32] J. Řeháček, Z. Hradil, O. Haderka, J. Peřina, and M. Hamar, “Multiple-photon resolving fiber-loop detector,” Physical Review A 67, 4 (2003).
  * [33] C. M. Natarajan, L. Zhang, H. Coldenstrodt-Ronge, G. Donati, S. N. Dorenbos, V. Zwiller, I. A. Walmsley, and R. H. Hadfield, “Quantum detector tomography of a time-multiplexed superconducting nanowire single-photon detector at telecom wavelengths,” Optics Express 21, 893–902 (2013).
  * [34] R. Kruse, J. Tiedau, T. J. Bartley, S. Barkhofen, and C. Silberhorn, “Limits of the time-multiplexed photon-counting method,” Physical Review A 95, 23815 (2017).
  * [35] C. M. Natarajan, M. G. Tanner, and R. H. Hadfield, “Superconducting nanowire single-photon detectors: physics and applications,” Superconductor Science and Technology 25, 063001 (2012).
  * [36] I. Holzman and Y. Ivry, “Superconducting Nanowires for Single-Photon Detection: Progress, Challenges, and Opportunities,” Advanced Quantum Technologies 2 (2019).
  * [37] D. V. Reddy, R. R. Nerem, S. W. Nam, R. P. Mirin, and V. B. Verma, “Superconducting nanowire single-photon detectors with 98% system detection efficiency at 1550 nm,” Optica 7, 1649–1653 (2020).
  * [38] B. Korzh, Q. Y. Zhao, J. P. Allmaras, S. Frasca, T. M. Autry, E. A. Bersin, A. D. Beyer, R. M. Briggs, B. Bumble, M. Colangelo, G. M. Crouch, A. E. Dane, T. Gerrits, A. E. Lita, F. Marsili, G. Moody, C. Peña, E. Ramirez, J. D. Rezac, N. Sinclair, M. J. Stevens, A. E. Velasco, V. B. Verma, E. E. Wollman, S. Xie, D. Zhu, P. D. Hale, M. Spiropulu, K. L. Silverman, R. P. Mirin, S. W. Nam, A. G. Kozorezov, M. D. Shaw, and K. K. Berggren, “Demonstration of sub-3 ps temporal resolution with a superconducting nanowire single-photon detector,” Nature Photonics 14, 250–255 (2020).
  * [39] H. Shibata, K. Fukao, N. Kirigane, S. Karimoto, and H. Yamamoto, “SNSPD with Ultimate Low System Dark Count Rate Using Various Cold Filters,” IEEE Transactions on Applied Superconductivity 27 (2017).
  * [40] C. Cahall, K. L. Nicolich, N. T. Islam, G. P. Lafyatis, A. J. Miller, D. J. Gauthier, and J. Kim, “Multi-photon detection using a conventional superconducting nanowire single-photon detector,” Optica 4, 1534–1535 (2017).
  * [41] M. Endo, T. Sonoyama, M. Matsuyama, F. Okamoto, S. Miki, M. Yabuno, F. China, H. Terai, and A. Furusawa, “Quantum detector tomography of a superconducting nanostrip photon-number-resolving detector,” Optics Express 29, 11728–11738 (2021).
  * [42] D. Zhu, M. Colangelo, C. Chen, B. A. Korzh, F. N. Wong, M. D. Shaw, and K. K. Berggren, “Resolving photon numbers using a superconducting nanowire with impedance-matching taper,” Nano Letters 20, 3858–3863 (2020).
  * [43] R. Loudon and T. von Foerster, _The Quantum Theory of Light_ , vol. 42 (Oxford University Press, 1974), 3rd ed.
  * [44] A. Christ and C. Silberhorn, “Limits on the deterministic creation of pure single-photon states using parametric down-conversion,” Physical Review A 85, 023829 (2012).
  * [45] E. Meyer-Scott, C. Silberhorn, and A. Migdall, “Single-photon sources: Approaching the ideal through multiplexing,” Review of Scientific Instruments 91, 041101 (2020).
  * [46] G. S. Thekkadath, B. A. Bell, R. B. Patel, M. S. Kim, and I. A. Walmsley, “Measuring the joint spectral mode of photon pairs using intensity interferometry,” arXiv:2107.06244 (2021).
  * [47] C. T. Cahall, “Enabling Technologies for High-Rate , Free-Space Quantum Communication,” Ph.D. thesis (2019).
  * [48] D. N. Klyshko, “Use of two-photon light for absolute calibration of photoelectric detectors,” Sov. Phys. Tech. Phys 11, 797 (1975).
  * [49] M. J. Stevens, “Chapter 2 - Photon Statistics, Measurements, and Measurements Tools,” in _Single-Photon Generation and Detection,_ vol. 45 of _Experimental Methods in the Physical Sciences_ A. Migdall, S. V. Polyakov, J. Fan, and J. C. Bienfang, eds. (Academic Press, 2013), pp. 25–68.
  * [50] M. M. Weston, H. M. Chrzanowski, S. Wollmann, A. Boston, J. Ho, L. K. Shalm, V. B. Verma, M. S. Allman, S. W. Nam, R. B. Patel, S. Slussarenko, and G. J. Pryde, “Efficient and pure femtosecond-pulse-length source of polarization-entangled photons,” Optics Express 24, 10869–10879 (2016).
  * [51] I. Tzitrin, J. E. Bourassa, N. C. Menicucci, and K. K. Sabapathy, “Progress towards practical qubit computation using approximate Gottesman-Kitaev-Preskill codes,” Physical Review A 101, 032315 (2020).
  * [52] M. Eaton, R. Nehra, and O. Pfister, “Non-Gaussian and Gottesman-Kitaev-Preskill state preparation by photon catalysis,” New Journal of Physics 21, 113034 (2019).
  * [53] H. S. Zhong, H. Wang, Y. H. Deng, M. C. Chen, L. C. Peng, Y. H. Luo, J. Qin, D. Wu, X. Ding, Y. Hu, P. Hu, X. Y. Yang, W. J. Zhang, H. Li, Y. Li, X. Jiang, L. Gan, G. Yang, L. You, Z. Wang, L. Li, N. L. Liu, C. Y. Lu, and J. W. Pan, “Quantum computational advantage using photons,” Science 370, 1460–1463 (2021).
  * [54] J. F. F. Bulmer, B. A. Bell, R. S. Chadwick, A. E. Jones, D. Moise, A. Rigazzi, J. Thorbecke, U.-U. Haus, T. Van Vaerenbergh, R. B. Patel, I. A. Walmsley, and A. Laing, “The Boundary for Quantum Advantage in Gaussian Boson Sampling,” arXiv:2108.01622 (2021).


# ContactArt: Learning 3D Interaction Priors for
Category-level Articulated Object and Hand Poses Estimation

Zehao Zhu1∗, Jiashun Wang2∗, Yuzhe Qin4, Deqing Sun3, Varun Jampani3, Xiaolong
Wang4
1 Zhejiang University 2Carnegie Mellon University 3Google Research 4UC San
Diego
Zehao Zhu1* Jiashun Wang2* Yuzhe Qin4 Deqing Sun3 Varun Jampani3 Xiaolong
Wang4
1Zhejiang University 2Carnegie Mellon University 3Google Research 4UC San
Diego

###### Abstract

We propose a new dataset and a novel approach to learning hand-object
interaction priors for hand and articulated object pose estimation. We first
collect a dataset using visual teleoperation, where the human operator can
directly play within a physical simulator to manipulate the articulated
objects. We record the data and obtain free and accurate annotations on object
poses and contact information from the simulator. Our system only requires an
iPhone to record human hand motion, which can be easily scaled up and largely
lower the costs of data and annotation collection. With this data, we learn 3D
interaction priors including a discriminator (in a GAN) capturing the
distribution of how object parts are arranged, and a diffusion model which
generates the contact regions on articulated objects, guiding the hand pose
estimation. Such structural and contact priors can easily transfer to real-
world data with barely any domain gap. By using our data and learned priors,
our method significantly improves the performance on joint hand and
articulated object poses estimation over the existing state-of-the-art
methods. The project is available at https://zehaozhu.github.io/ContactArt/.

Figure 1: Overview. We collect a dataset named ContactArt, which is created by
human interacting with the articulated objects in a simulator, using
teleoperation. Two interaction priors are learned from ContactArt: (i) a
contact prior predicted by a diffusion model to improve 3D hand pose
estimation; (ii) an articulation prior with a discriminator to improve
category-level articulated object pose estimation. We visualize the pose
estimation results in real-world data, leveraging the learned priors.  ††*:
equal contributions

##  1 Introduction

The understanding of the 3D articulated structure has caught a lot of
attention recently: Studies have been conducted on estimating the articulated
object poses [45, 79, 47]. Beyond studying the single object in isolation,
understanding the interactions between human hands and articulated objects
play an important role in a wide range of applications such as robotics and
Augmented Reality. However, there remain several challenges for hand and
category-level articulated object pose estimation given the high Degree of
Freedom on poses and mutual occlusions.

Most current research focusing on articulated object pose estimation has been
limited by the high cost of annotations on real-world objects [49]. To
alleviate this issue, approaches to using synthetic data with cheaper
annotations have been proposed [34, 45, 47, 79]. However, this inevitably
introduces a sim2real gap when transferring pose estimation to images in the
wild. The joint estimation of the human hand and articulated object poses
makes the problem even more challenging given their mutual occlusions. Recent
efforts on collecting the real-world category-level human-object 3D pose
annotations [48] have largely advanced this field. However, the expensive
labeling process still makes it hard to scale and it is very difficult to
obtain accurate contact labels between hands and objects from observing the
images. Is there a cheaper and more scalable way to obtain the hand-object
interaction annotations?

Our answer is affirmative and our key insight is that, while there is a large
sim2real appearance gap, the geometric contacts between hand and objects are
actually consistent across simulation and the real world [58]. In this paper,
we collect the hand-object interaction data and accurate annotations by asking
humans to directly play within a physical simulator using visual teleoperation
(Fig. ContactArt: Learning 3D Interaction Priors for  Category-level
Articulated Object and Hand Poses Estimation 1st column). We name this dataset
ContactArt: Contact with Articulation. Specifically, we deploy an off-the-
shelf visual teleoperation system [58] for data collection. It only requires a
single camera on a mobile device to record the human hand, which is mapped to
a MANO hand [61], to manipulate the articulated objects in a physical
simulator. Within each object category, we collect interaction data across
diverse articulated object instances. We can obtain the accurate hand-object
poses and their contact points for free by reading from the simulator. This
largely reduces the labeling cost from previous approaches [12, 48].

The ContactArt dataset enables us to train real-world pose estimators with
free annotations. To minimize the sim2real gap, we learn 3D interaction priors
from ContactArt and use them to improve the real-world hand and object pose
estimation. We train a generalizable model for each object category and
evaluate the model on unseen instances. We propose to learn two types of 3D
hand-object interaction priors, which capture how object parts are generally
articulated and where humans generally touch the object for manipulation. The
first prior is to learn the discriminator network, modeling the joint
distribution of object part arrangement inside each object category (Fig.
ContactArt: Learning 3D Interaction Priors for  Category-level Articulated
Object and Hand Poses Estimation 3rd column). Following a GAN framework [24],
we consider the pose estimators as the generators, and we train the
discriminator by using the estimated hand and object poses as the fake data
inputs, and the ground-truth ContactArt poses as the real data. The
discriminator then learns how object parts “naturally” connect together, and
we use this discriminator via back-prop to optimize the estimated object pose.
The second prior is to learn a contact map diffusion generator [68] for
modeling where the hand can touch the object (Fig. ContactArt: Learning 3D
Interaction Priors for  Category-level Articulated Object and Hand Poses
Estimation 2nd column). Given the input articulated object, this model
predicts the plausible regions where the human hand operates (object
affordance regions), using a diffusion process. With an initial hand pose
estimation, we optimize the hand pose to match the estimated hand-object
contact information. The two priors are complementary to each other and are
used jointly to optimize both estimated hand and articulated object poses.

We perform our experiments on three in-the-wild articulated object datasets,
HOI4D [48], BMVC [50] and RBO [49] including five categories in total. We find
that with our ContactArt dataset and the proposed articulation and the contact
prior, we can not only achieve large improvements over previous state-of-the-
art methods of estimating articulated object poses, but also observe
significant improvements in the hand pose estimation. Further, we find
training on ContactArt first as a warm start then finetuning on HOI4D can
bring better performance while requiring less data compared with training from
scratch on HOI4D.

Our contributions include: (i) A new dataset with contact-rich hand-
articulated object interaction; (ii) A contact diffusion model used to
estimate the contact map of interaction; (iii) An articulation discriminator
which learns articulation prior and boosts articulated object pose estimation;
(iv) Substantial performance improvement on the articulated object and hand
pose estimation.

##  2 Related Work

Articulated Object Pose Estimation. Beyond understanding single rigid objects,
more attention has been put on articulated object modeling and pose estimation
recently [50, 45, 79, 76, 47, 52, 78, 83]. For example, Li _et al_. [45]
propose to perform category-level articulated object pose estimation and
evaluate their approach on unseen instances during training. Weng _et al_.
[79] adopt the ANCSH [76] to handle category-level pose tracking for both
rigid and articulated objects by leveraging the RotationNet and CoordinateNet.
Liu _et al_. [47] reform the articulated object pose estimation setting for
real-world environments and build an articulated object dataset ReArt-48.
However, these approaches mainly focus on modeling the articulated object
itself without considering how hands and objects contact and interact. In this
paper, we study the joint pose estimation problem with hand and object
together using different priors learned from our ContactArt dataset.

Figure 2: To collect ContactArt, the hardware requirement is an iPhone and a
laptop. The system allows us to easily scale up the dataset without human
annotation effort. We can collect manipulation sequences and render images
from different camera views.

Conditional Diffusion Probabilistic Models. Recent progress on diffusion
probabilistic models [69, 35, 71, 70] has shown to be very effective in
generating high-quality images. Inspired by these results, the conditional
diffusion model has been widely applied in text-to-image generation [68, 17,
53, 72, 60, 65], image super resolution [66, 39, 60] and image-to-image
translation tasks [15, 64, 88, 6]. Different from the above diffusion models
which are conditioned on input image or prompt, our proposed contact diffusion
model is conditioned on the point-wise feature from the point cloud. There
have been several work [6, 3, 80] performing semantic segmentation with a
conditional diffusion model. For example, Wolleb _et al_. [80] uses the
stochastic sampling process to implicitly group the segmentation masks of
medical images. Inspired by these works, our contact diffusion model adopts
the diffusion process to predict the contact map indicating where the hand
should touch the articulated object.

Hand-Object Interaction. Estimating hand-object interaction has been a long-
standing problem in computer vision [30, 55, 5, 56, 73]. More recently, a line
of studies [34, 42, 54, 75, 84, 31, 18, 13, 11, 33, 86] use data-driven and
deep-learning based methods to jointly estimate or reconstruct the hand and
object. Another line of research studies synthesizing plausible hand-object
interactions [16, 38, 36, 9, 25, 85, 90, 14]. For example, Jiang et al. [36]
propose to generate the hand grasp pose and contact map at the same time and
optimize the consistency between the hand and object during test time. Our
method is also related to the approach proposed by Grady et al. [25] which
estimates contact given a complete mesh. In contrast, our estimation is based
on partial point clouds from the depth sensor which is much more challenging.

The success of these recent works is inseparable from the hand-object
interaction datasets [34, 31, 8, 10, 74, 12, 59, 48, 21, 27, 82, 81], which
are playing crucial roles in both estimation, synthesis and robot manipulation
tasks. For example, DexYCB [12] and HOI4D [48] are two recent hand-object pose
datasets annotated by humans, which are much more expensive compared to 2D
labels. ContactDB [8] is proposed to capture the hand-object contact map with
a thermal camera. But this is difficult to scale given the equipment
requirement. To remove the constraints from annotation cost and hardware
setup, we propose to collect the human and articulated object interaction
using visual teleoperation in a physical simulator. We provide a scalable
solution with free annotations from the simulator. Such geometric priors and
contacts are transferable to the real world.

Vision-based manipulation teleoperation [41, 19, 20, 44, 4, 67, 58, 32] is a
commonly applied technique in robotics. To reduce the device cost for scalable
collection, we build our system upon [58], using only a single camera to
record the human hand to manipulate the articulated objects inside the Sapien
[81] simulator. While using visual teleoperation itself is not new, applying
it to record hand-object poses and contacts for learning 3D priors has not
been done before. We believe this can provide a new perspective on how
interaction dataset can be collected for computer vision tasks.

Adversarial Learning for Priors While adversarial learning is initially
proposed for image generation [23], the discriminator trained with adversarial
learning is also utilized in multiple tasks such as 3D human pose estimation
[37, 40, 77, 7, 28, 1] and 2D human trajectory prediction [29, 43, 63, 2]. Our
articulation prior is inspired by [37, 40], which are focusing on a specific
articulation category: human. These approaches try to jointly learn a prior
for what is a natural pose for humans, and how each articulated part is
combined with the others. Similarly, in articulated objects, we have the upper
drawer and the lower drawer are always parallel for example. Thus we propose
to utilize the discriminator from adversarial learning to capture the
articulation priors.

##  3 ContactArt Dataset

To the best of our knowledge, there is only one large-scale dataset [48]
including 3D hand-articulated object interaction. However, it still holds the
following limitations. (i) HOI4D dataset is only captured in the egocentric
view and can not generalize to the third view. (ii) The annotation of the
HOI4D dataset is not accurate enough to provide contact information. Therefore
we build a large hand-articulated object interaction dataset with no
annotation effort and more accurate pose and contact information.

We deploy the teleoperation system introduced in [58] to manipulate
articulated objects in the Sapien [81] simulation. This simulation system
allows us to get accurate pose annotation and contact information using only a
mobile phone and a laptop (Fig. 2 (a)). Instead of using professional
equipments like Kinect or Real-Sense cameras, the utilization of a mobile
phone makes the data collection more accessible and easier to scale up. Since
the teleoperation is in the simulation, the annotations can be automatically
recorded, which will make it easy to scale up the size of the dataset. We
train our models with this collected dataset. While the use of the
teleoperation system itself is not new, its application in collecting 3D
interaction prior data for pose estimation is novel, compared to previous
literature on collecting human demonstrations.

Dataset collection. We use the front camera of a mobile phone to stream the
RGB-D video at 15 fps. The setup and collection interface are shown in Fig. 2
(a). We adopt the teleportation system [58] which allows one to control the
customized robot hand with the captured hand motion as control signals in the
simulation. One can easily manipulate the articulation object, such as opening
the drawer. We provide a sequence of ContactArt collection process in Fig. 2
(b). We render the RGB, depth, and segmentation images respectively. We give
examples in Fig. 2 (c). We record the object pose, bounding boxes and hand
poses, and the hand-object contact regions. Note that for rendering the depth
image, we apply the active stereovision depth sensor simulation proposed in
[87], which renders realistic depth images close to the depth camera captured
in the real world.

Dataset statistics. We select five common articulated object categories in our
daily life including laptops, drawers, safes, microwaves and trashcans, 80
instances in total. All the object models are from Partnet dataset [51] so it
is convenient to scale up. Tab. 1 summarizes the statistics of ContactArt
comparing previous datasets. ContactArt can provide accurate annotation, rich
hand-object interaction, and contact information. One can also easily render
more frames by using different camera views. Please see the supplementary
materials for more details about dataset statistics.

Dataset | HOI | Hand GT | Multi Views | Contact Label | Frames
---|---|---|---|---|---
BMVC [50] | ✕ | ✕ | ✕ | ✕ | 8K
RBO [49] | ✓ | ✕ | ✕ | ✕ | 12K
ReArtMix [47] | ✕ | ✕ | ✓ | ✕ | 100K
ReArtVal [47] | ✕ | ✕ | ✕ | ✕ | 6K
HOI4D [48] | ✓ | ✓ | ✕ | ✕ | 1.44M
ContactArt | ✓ | ✓ | ✓ | ✓ | 332K
Table 1: Comparison with other articulated object datasets. HOI refers to
hand-object interaction and Hand GT refers to the annotation of ground truth
hand pose. ContactArt allows rendering in multi-view and has accurate contact
information. Statistics are performed on articulated objects.

##  4 Method

Figure 3: Training and Testing framework. Left: During training, we first
extract a point-wise feature and regress the part segmentation, NOCS map, and
part-level rotation. We then compute the 3D bounding box of each part and feed
it to a discriminator. We utilize a contact diffusion model conditioned on the
point-wise feature to estimate the contact map. We visualize the contact
points in green. $\bigoplus$ denotes concatenation. Right: At test time, we
utilize the discriminator with fixed parameters to calculate adversarial loss
and backpropagate the gradients to update the object estimator. Then we
optimize hand pose by minimizing the distance between hand vertices and the
contact points at the predicted contact map.

In this paper, we target at the problem of hand and articulated object pose
estimation from known categories with interaction. Compared to [79, 47, 47,
45] which only focus on the pose estimation of articulated object, our method
pays attention to both hand and articulated object since they influence a lot
each other during interaction. Our method takes a RGB-D image as input and
output part-level 6D object pose (rotation and translation) and the hand pose
parameterized by the MANO model [61].

We propose a NOCS-based [76] category-level pose estimator for articulated
objects, together with two 3D interaction priors. In our framework, we train
the articulated object pose estimator using both the reconstruction loss, and
adversarial training with a discriminator. This Articulation Discriminator
will serve as a prior of how object parts should be arranged together within a
category. During test time, given an initial estimation from the pose
estimator, we can use the discriminator to provide the gradients through back-
propagation to optimize the pose of each object part. Meanwhile, to model the
hand-object interaction, we also propose a diffusion-based contact map
generator, which estimates the regions where the hand will touch the object,
namely Contact Diffusion Model. We will use it as an optimization constraint
to encourage the hand to reach the generated contact region. The architecture
of our model is shown in Fig. 3.

###  4.1 Object Pose Estimator

We design a multi-branch pose estimator $\mathcal{E}$ to predict the
articulated object pose (Fig. 3 dotted blue box). We first detect the hand and
object and get the 2D bounding box of them with off-the-shelf method [22] and
back project the patch to point cloud $v\in\mathbb{R}^{N\times 3}$. Then we
utilize PointNet++ [57] to extract the point features, which include both
object-aware and hand-aware information. Building on the points features, we
use three separate MLPs to predict (i) the part segmentation, (ii) part-level
NOCS map [76, 45], and (iii) rotation of each part. We adopt the 6D continuous
rotation representation [89] for rotation. The part-level NOCS map is defined
as a 3D space contained within a unit cube of each articulated part and
consistently aligns to a category-level canonical orientation.

In a forward pass, given the predictions of per-part rotation and dense
correspondence between the NOCS map and point cloud, we can analytically
compute translation and scale via the Umeyama algorithm. Then we leverage the
computed pose (rotation, translation, scale) to transform the canonical 3D
bounding box into the camera space. Different from [47, 45], by predicting the
rotation of each part using a neural network, we make the prediction of pose
and bounding box fully differentiable, allowing end-to-end training. This also
provides opportunities for optimization using the discriminator from
adversarial learning.

Specifically, we use the cross-entropy loss ($CE$) for part segmentation. For
rotation loss, we calculate the L2 distance between our prediction and ground
truth in this 6D space in the form of continuous rotation representation. We
use L2 distance for NOCS map loss. The object pose estimation loss can be
written as,

$\displaystyle L_{pose}=$
$\displaystyle\lambda_{seg}\sum_{i}^{N}{CE(s_{i},s_{i}^{*})}+\lambda_{rot}\|r-r^{*}\|_{2}$
$\displaystyle+\lambda_{nocs}\sum_{i}^{N}{\mathds{1}(s_{i}^{*}>0)\|n_{i}-n_{i}^{*}\|_{2}}$
(1)

where $N$ is the number of sampled points, $s^{*},s$ are the ground truth and
predicted part segmentation, $n^{*},n$ are the ground truth and predicted
part-level NOCS maps and $r^{*},r$ are the ground truth and predicted
rotation. $\lambda_{nocs}$, $\lambda_{seg}$ and $\lambda_{rot}$ are
hyperparameters balancing the weights. In addition to $L_{pose}$, we introduce
an adversarial loss with the Articulation Discriminator as described in the
following section.

###  4.2 Articulation Discriminator

Our model jointly learns an Articulation Discriminator $\mathcal{D}$ (Fig. 3
green box) as the articulation structure prior during training the estimator
$\mathcal{E}$. This discriminator will improve the naturalness on how parts
are arranged together. The discriminator takes inputs as the 3D bounding
boxes, which are represented by 8 vertices for each part and fully reflect the
part placement rules. Furthermore, there is only a small sim2real domain gap
in the 3D bounding box space. We can calculate each part’s bounding box
$\hat{b}$ with the outputs from the estimator. During training, we use the
estimated boxes $\hat{b}\sim p_{\mathcal{E}}$ as negative samples. We use the
accurate bounding boxes $b$ from simulation data $p_{S}$ as positive samples.
We define the loss function for the discriminator as,

$L_{\mathcal{D}}=\mathbb{E}_{b\sim
p_{S}}[(\mathcal{D}(b)-1)^{2}]+\mathbb{E}_{\hat{b}\sim
p_{\mathcal{E}}}[(\mathcal{D}(\hat{b}))^{2}].$ (2)

And the adversarial loss term for the estimator is,

$L_{adv}=\mathbb{E}_{\hat{b}\sim
p_{\mathcal{E}}}[(\mathcal{D}(\hat{b})-1)^{2}].$ (3)

###  4.3 Contact Diffusion Model

Diffusion models have shown state-of-the-art performance in generation tasks.
In our work, we extend it to generate a realistic 3D contact map between the
object and hand point clouds (Fig. 3 bottom). Once we get such contact
information, we use it to guide the optimization of the 3D hand.

We first define the contact map. For an input point cloud set
$v\in\mathbb{R}^{N\times 3}$, the contact map $x\in\mathbb{R}^{N\times 1}$ is
defined as a binary vector indicates whether each point belongs to the contact
region or not. We calculate the L2 distance between the points from the object
and its nearest points from the hand. If this distance is smaller than a
threshold, we take this point as contacted.

We formulate the details of our contact diffusion model as follows. Let
$X_{0}=(x_{0},z)$ denote the input, where $x_{0}\in\mathbb{R}^{N\times 1}$ is
the contact map and $z\in\mathbb{R}^{N\times 3}$ is the PointNet++ local
feature. $T$ is the number of steps in the diffusion model and the
intermediate results can be denoted as $X_{t}=(x_{t},z)$, where $0\leq t\leq
T$. Diffusion models are composed of forward and backward processes. The
forward process gradually injects random noise into the distribution, while
the generative process learns to remove noise to obtain realistic samples by
mimicking the reverse process. The forward process converts the original
contact map distribution into a noise distribution, which can be described by
the formulation,

$q(x_{1:T}|x_{0})=q(x_{0})\prod\limits_{t=1}^{T}q(x_{t}|x_{t-1}).$ (4)

The reverse process $p_{\theta}$ is learned by the model parameters $\theta$.
Different from the forward process which simply adds noise to the contact map,
the reverse process recovers the desired contact map from the input noise,
encoded by the Pointnet++ feature $z$. The reverse diffusion process is,

$\displaystyle
p_{\theta}(x_{0:T}|z)=p(x_{T})\prod\limits_{t=1}^{T}p_{\theta}(x_{t-1}|x_{t},z).$
(5)

A parameterization trick [35] is used to simplify the training objective. The
simplified training objective becomes,

$\displaystyle E_{X_{0}\sim q(x_{0}),x_{1:T}\sim
q(x_{1:T}|x_{0},z_{0})}[\sum_{t=1}^{T}\log{p_{\theta}(x_{t-1}|x_{t},z)}].$ (6)

Since posterior $q(x_{t-1}|x_{t},x_{0},z)$ is known and its derivation is
similar to the unconditional generative model, we define the $L_{diff}$ as,

$\displaystyle L_{diff}=\|\epsilon-\epsilon_{\theta}(x_{t},z,t)\|^{2}.$ (7)

Please note that the whole diffusion model is trained with the estimator in an
end-to-end fashion, which takes the Pointnet++ feature predicted by the
estimator as input. Specifically, in our diffusion model, $\epsilon_{\theta}$
is implemented in MLP. We first concatenate Pointnet++ feature $z$, contact
map $x_{t}$, and the time embedding in the current step t. After that, we pass
the concatenated feature into MLP. Then we use the predicted noise to compute
the affordance map in the next step. Same to [3], we also employ multiple
generations to boost performance. Finally, the total loss of the whole
pipeline can be written as,

$\displaystyle L=L_{pose}+\lambda_{adv}L_{adv}+\lambda_{diff}L_{diff},$ (8)

where $\lambda_{diff}$ and $\lambda_{adv}$ are hyperparameters balancing the
contact diffusion model loss and the adversarial training loss.

###  4.4 Test Time Adaptation

Once we learn the Articulation Discriminator and the Contact Diffusion Model,
we can improve the initial object and hand pose estimation with test time
adaption. For optimizing object pose during test time, we fix the parameters
of the discriminator $\mathcal{D}$ and use it to calculate the adversarial
loss and backpropagate the gradients to object pose estimator $\mathcal{E}$ to
boost object pose estimation. For optimizing hand pose, we employ the contact
diffusion model to estimate the contact region and obtain the contact point
set $C\in\mathbb{R}^{K\times 3}$ where $K$ is the number of contact points. We
then optimize the MANO [61] parameters of the hand which is initialized by the
FrankMocap [62] hand pose estimator. Specifically, we minimize the chamfer
distance between the hand vertices $V\in\mathbb{R}^{N\times 3}$ where $N$ is
the number of vertices and the contact points,

$L_{CD}=\frac{1}{N}\sum_{v\in V}\min_{c\in C}\lVert
v-c{\rVert}_{2}+\frac{1}{K}\sum_{c\in C}\min_{v\in V}\lVert c-v{\rVert}_{2}.$
(9)

##  5 Experiments

###  5.1 Datasets

We train our model on ContactArt and test on HOI4D [48], RBO [49], BMVC [50]
respectively. HOI4D [48] is a large-scale hand-object interaction dataset
where we can evaluate both object and hand pose estimation. We use 4
categories for evaluation: safe, trashcan, laptop, and drawer. We use 6000
frames for each category as the test set. We also perform experiments with
finetuning on it. We find the model trained on ContactArt then finetuned on
HOI4D will benefit from ConatactArt and achieve better performance than
training from scratch. RBO [49] is a collection of RGB-D video sequences.
There is no annotation of hand pose in RBO and we perform object pose
estimation comparisons. We evaluate on 3 categories: laptop, microwave and
drawer. BMVC [50] includes video sequences recording articulated object with a
moving camera. There is no human manipulating the object, we evaluate object
pose estimation on laptop following CAPTRA [79].

###  5.2 Metrics and Methods for Comparison

Metrics for comparison. For category-level articulated object pose estimation,
we evaluate the following metrics: 5∘5cm: percentage of results with rotation
error smaller than 5∘ and translation error smaller than 5cm, mIoU: the
average 3D intersection over the union of ground-truth and predicted bounding
boxes, $R_{err}$: rotation error in degrees, $T_{err}$: translation error in
centimeters. For hand pose estimation, we report mean per vertex position
error (MPVPE) and mean per joint position error (MPJPE).

Methods for comparison. We compare our method with two state-of-the-art image-
based pose estimation works ANCSH [45] and ReArtNocs [47], and a tracking
method CAPTRA [79], we provide the initial pose estimated by our method for
fair comparisons. All the methods are trained on ContactArt. We also compare
first training on ContactArt then finetuning on HOI4D (named Finetune) with
training on HOI4D from scratch (named HOI4D*).

Category | Metric | ANCSH | ReArtNocs | CAPTRA | Ours | HOI4D* | Finetune
---|---|---|---|---|---|---|---
Laptop |  5∘5cm $\uparrow$ | 10.54 | 10.60 | 16.35 | 18.65 | 61.95 | 62.50
mIoU$\uparrow$ | 47.8 | 49.52 | 51.57 | 52.45 | 65.21 | 66.45
$R_{err}$ $\downarrow$ | 24.06 | 23.40 | 18.52 | 17.73 | 6.24 | 5.20
$T_{err}$ $\downarrow$ | 23.35 | 22.41 | 19.75 | 18.91 | 7.68 | 7.17
Trashcan |  5∘5cm$\uparrow$ | 0 | 0 | 3.05 | 2.70 | 22.8 | 24.2
mIoU$\uparrow$ | 38.38 | 39.30 | 41.50 | 41.95 | 63.65 | 64.95
$R_{err}$ $\downarrow$ | 26.93 | 25.57 | 21.97 | 21.43 | 7.05 | 5.98
$T_{err}$ $\downarrow$ | 36.72 | 36.65 | 30.70 | 30.75 | 7.75 | 7.22
Safe |  5∘5cm$\uparrow$ | 1.66 | 5.30 | 4.65 | 8.43 | 32.05 | 33.40
mIoU$\uparrow$ | 46.20 | 47.05 | 46.83 | 47.96 | 62.31 | 63.50
$R_{err}$ $\downarrow$ | 18.63 | 16.91 | 16.43 | 16.24 | 5.74 | 5.11
$T_{err}$ $\downarrow$ | 17.52 | 16.51 | 16.55 | 15.66 | 6.85 | 6.50
Cabinet |  5∘5cm$\uparrow$ | 0.50 | 0 | 0.16 | 1.00 | 22.07 | 23.87
mIoU$\uparrow$ | 49.83 | 49.90 | 49.76 | 50.40 | 64.43 | 66.71
$R_{err}$ $\downarrow$ | 18.94 | 19.52 | 19.87 | 17.84 | 6.46 | 5.75
$T_{err}\downarrow$ | 23.33 | 23.28 | 24.23 | 22.60 | 6.03 | 5.83
Average |  5∘5cm$\uparrow$ | 3.17 | 3.97 | 6.05 | 7.69 | 34.72 | 35.99
mIoU$\uparrow$ | 45.55 | 46.44 | 47.41 | 48.19 | 63.90 | 65.40
$R_{err}$ $\downarrow$ | 22.14 | 21.35 | 19.20 | 18.31 | 6.37 | 5.51
$T_{err}$ $\downarrow$ | 25.23 | 24.71 | 22.81 | 21.98 | 7.08 | 6.68
Table 2: Quantitative comparison of articulated object pose estimation on
HOI4D [48]. Our method and dataset could both improve the estimation
performance on different categories and metrics.

Category | Metric | Ansch | ReArtNocs | CAPTRA | Ours
---|---|---|---|---|---
|  5∘5cm$\uparrow$ | 1.45 | 0.75 | 4.02 | 4.70
BMVC |  mIoU$\uparrow$ | 54.32 | 54.93 | 60.25 | 61.22
Laptop |  $R_{err}$ $\downarrow$ | 26.72 | 24.04 | 19.08 | 17.30
|  $T_{err}$ $\downarrow$ | 18.58 | 18.05 | 12.34 | 11.45
|  5∘5cm$\uparrow$ | 23.01 | 29.12 | 33.12 | 33.83
RBO |  mIoU$\uparrow$ | 49.85 | 51.11 | 51.77 | 52.95
Laptop |  $R_{err}$ $\downarrow$ | 11.39 | 11.56 | 10.89 | 10.76
|  $T_{err}$ $\downarrow$ | 9.60 | 8.95 | 7.12 | 6.65
|  5∘5cm$\uparrow$ | 43.02 | 47.51 | 55.31 | 57.66
RBO |  mIoU$\uparrow$ | 69.21 | 70.65 | 71.45 | 73.05
Microwave |  $R_{err}$ $\downarrow$ | 7.28 | 6.56 | 10.89 | 4.69
|  $T_{err}$ $\downarrow$ | 5.20 | 4.87 | 4.91 | 4.70
|  5∘5cm$\uparrow$ | 22.02 | 26.98 | 33.79 | 28.23
RBO |  mIoU$\uparrow$ | 53.93 | 54.83 | 55.01 | 56.34
Drawer |  $R_{err}$ $\downarrow$ | 8.27 | 8.34 | 8.01 | 7.85
|  $T_{err}$ $\downarrow$ | 14.93 | 13.45 | 13.10 | 12.60
Table 3: Quantitative comparison of articulated object pose estimation on BMVC
[50] and RBO [49] datasets. Given to the articulation prior the model learnt,
our method can achieve the best performance on most categories.

Figure 4: Qualitative comparison of object pose estimation. We use red box to indicate error larger than 10∘ or 10 cm. Image-based baselines fail to get an accurate pose. And our method also performs better than the tracking-based method [79].  Figure 5: Optimization process. The hand is reaching the predicted contact map and getting to the correct pose. Metric |  Frankmocap [62] |  MG [46] | Affine | Regression | Closest | Ours
---|---|---|---|---|---|---
MPVPE$\downarrow$ | 71.6 | 67.4 | 54.1 | 52.4 | 60.5 | 49.9
MPJPE$\downarrow$ | 64.3 | 65.3 | 45.9 | 44.8 | 56.1 | 41.9
Table 4: Quantitative comparison of hand pose estimation on HOI4D. Utilizing
contact map can largely reduce the estimation error compared with Frankmocap.
And among all the ablative baselines, our MLP-based contact diffusion model
achieves the best performance. Figure 6: Ablation on the amount of training
data compared between Finetuning (Finetune) and training from scratch
(HOI4D*). Finetuning can always achieve better performance with much fewer
data.

###  5.3 Object Pose Estimation Comparison

We summarize the quantitative articulated object pose estimation results on
HOI4D in Tab. 3. Compared with the other methods, ours has the lowest average
rotation and translation error, the highest mIoU and 5∘5cm. Although CAPTRA
leverages the temporal information, our method still outperforms it. The last
two columns in Tab. 3 shows results of training from scratch and finetuning
respectively. We observe finetuning performs better than training from scratch
for all the metrics, which demonstrated ContactArt can serve as a “prior” for
pose estimation and could be used as a warm start for other datasets with
smaller sizes.

We visualize the qualitative comparisons of articulated object pose estimation
in Fig. 4. Following [45], we utilize 10∘10cm as a threshold and use red color
to indicate the results larger than this threshold and green to indicate the
one within it. We observe that our method could achieve the best performance
compared with the other methods. Two image-based estimation baselines fail to
estimate right 3D poses when testing on challenging layouts or camera views.
Our method is the most accurate and robust one even compared with the
tracking-based method.

We summarize the quantitative comparisons on BMVC and RBO in Tab. 3. Our
method outperforms the baselines on all the categories across all the metrics.
Though our model is trained on ContactArt with rich hand-object interaction
but it still works well on BMVC.

Metric | CAPTRA | Ours | HOI4D* | Finetune
---|---|---|---|---
w/o tta | w/ tta | w/o tta | w/ tta | w/o tta | w/ tta | w/o tta | w/ tta
5∘5cm $\uparrow$ | 16.35 | 16.70 | 18.00 | 18.65 | 62.05 | 62.50 | 61.55 | 61.95
mIoU$\uparrow$ | 51.50 | 51.65 | 52.15 | 52.45 | 66.25 | 66.45 | 65.25 | 65.20
$R_{err}$ $\downarrow$ | 18.22 | 18.3 | 18.26 | 17.73 | 5.54 | 5.20 | 6.42 | 6.24
$T_{err}$ $\downarrow$ | 19.75 | 19.60 | 19.10 | 18.90 | 7.35 | 7.20 | 7.75 | 7.70
Table 5: Evaluation for test time adaption(TTA) on HOI4D. TTA can almost
benefit all methods. HOI4D* denotes training on HOI4D from scratch.

###  5.4 Hand Pose Estimation Comparison

For hand pose estimation, we design an effective post-process specifically
designed for hand-object interaction. We take off-the-shelf motion capturer
Frankmocap [62] and our test time adaption to improve the hand estimation
results based on Frankmocap. We use variants of the backbone for our contact
diffusion model as baselines: MG, a state-of-the-art hand pose estimation
method Mesh Graphormer [46]; Regression, a non-diffusion baseline where we use
1D Convolution network to decode the Pointnet++ feature and directly regress
the contact map; Affine, where we change the MLP architecture of the diffusion
model to ConcatSquash layers [26]; Closest, where we simply optimize the hand
attracted to the closest object vertices. We report MPJPE and MPVPE in Tab. 4.
All three methods leveraging contact map to optimize hand (Ours, Affine,
Regression) outperform methods without contact map(Frankmocap, MG, Closest).
Therefore, estimating a contact map is necessary for accurate hand pose
estimation. Among the three contact map estimator, our method which employs
MLP-based diffusion model performs best. Specifically, for MPVPE, our method
largely decreases from 71.6 to 49.9 and decreases from 64.3 to 41.9 for MPJPE.
We also visualize the optimization process in Fig. 5. Our contact diffusion
model can successfully learn the contact prior during the interaction and
anchor the hand to a more reasonable pose in space.

###  5.5 Ablation Study

Ablation on the amount of training data. Our ContactArt could serve as a warm
start before training on other datasets. To prove this, we perform ablation
study training with different amounts of training data on HOI4D. We compare
Finetune and HOI4D* on training with 300, 500, 1k, 3k, 5k, 7k and 10k images.
We compare the mIoU, rotation error, and translation error in Fig. 6. In
general, finetuning is always better than training from scratch with the same
amount of data and has much lower standard deviation. The fewer data we give,
the larger improvements our model can achieve. We can also observe that
finetuning with 300, 1k, and 3k images are better than training with 1k, 3k,
and 10k images respectively. We only need one-third of the data to achieve
comparable performance if using ContactArt as a warm start. This is of great
importance when we only have a small amount of in-the-wild data. One can first
train on ContactArt and quickly adapt to new test scenarios.

Ablation on Test time adaption (TTA). Our articulation discriminator is highly
scalable and can be plugged into any pose estimation method, such as CAPTRA.
We select four methods, CAPTRA; our method trained on ContactArt; HOI4D*; and
Finetune. We add the discriminator to each method and apply the TTA to each.
We compare the one with and without TTA on laptop of HOI4D. We report the
results in Tab. 5, and observe that the articulation discriminator and TTA
improve all the methods. The test time adaption mechanism enables the
estimator to adapt to various test scenes since the discriminator can learn an
invariant prior of the articulation structure, specifically the layout of the
bounding boxes. In Fig. 7, we visualize the results of our method with and
without TTA, our articulation could learn a good prior and improve the layout
naturalness of each articulated part. After TTA, the drawers are parallel and
the laptop’s screen and keyboard are at similar sizes.

Figure 7: Comparison between our method with and without TTA. TTA can help get
a more natural layout.

##  6 Conclusion

In this paper, we present ContactArt, an interaction- and contact-rich and
easily scalable dataset. We further propose to use a discriminator as an
articulation prior to improve the articulated object pose estimation. We
introduce a contact diffusion model to estimate the contact map between the
hand and articulated objects, which can be utilized to optimize the hand pose
estimation. Extensive experiments demonstrate the effectiveness of our
ContactArt dataset, the articulation prior and the contact prior.

## References

  * [1] Emre Aksan, Manuel Kaufmann, and Otmar Hilliges.  Structured prediction helps 3d human motion modelling.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [2] Javad Amirian, Jean-Bernard Hayet, and Julien Pettré.  Social ways: Learning multi-modal distributions of pedestrian trajectories with gans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [3] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf.  Segdiff: Image segmentation with diffusion probabilistic models, 2021\.
  * [4] Dafni Antotsiou, Guillermo Garcia-Hernando, and Tae-Kyun Kim.  Task-oriented hand motion retargeting for dexterous manipulation imitation.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  * [5] Luca Ballan, Aparna Taneja, Jürgen Gall, Luc Van Gool, and Marc Pollefeys.  Motion capture of hands in action using discriminative salient points.  In European Conference on Computer Vision, pages 640–653. Springer, 2012.
  * [6] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko.  Label-efficient semantic segmentation with diffusion models, 2021.
  * [7] Emad Barsoum, John Kender, and Zicheng Liu.  HP-GAN: probabilistic 3d human motion prediction via GAN.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 1418–1427. Computer Vision Foundation / IEEE Computer Society, 2018.
  * [8] Samarth Brahmbhatt, Cusuh Ham, Charles C Kemp, and James Hays.  Contactdb: Analyzing and predicting grasp contact via thermal imaging.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8709–8719, 2019.
  * [9] Samarth Brahmbhatt, Ankur Handa, James Hays, and Dieter Fox.  Contactgrasp: Functional multi-finger grasp synthesis from contact.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2386–2393. IEEE, 2019.
  * [10] Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays.  Contactpose: A dataset of grasps with object contact and hand pose.  In European Conference on Computer Vision, pages 361–378. Springer, 2020.
  * [11] Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik.  Reconstructing hand-object interactions in the wild.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12417–12426, 2021.
  * [12] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al.  Dexycb: A benchmark for capturing hand grasping of objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044–9053, 2021.
  * [13] Yujin Chen, Zhigang Tu, Di Kang, Ruizhi Chen, Linchao Bao, Zhengyou Zhang, and Junsong Yuan.  Joint hand-object 3d reconstruction from a single image with cross-branch feature fusion.  IEEE Transactions on Image Processing, 30:4008–4021, 2021.
  * [14] Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges.  D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20577–20586, 2022.
  * [15] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye.  Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction, 2021.
  * [16] Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez.  Ganhand: Predicting human grasp affordances in multi-object scenes.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5031–5041, 2020.
  * [17] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis, 2021.
  * [18] Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David J Crandall.  Hope-net: A graph-based model for hand-object pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6608–6617, 2020.
  * [19] Guanglong Du, Ping Zhang, Jianhua Mai, and Zeling Li.  Markerless kinect-based hand tracking for robot teleoperation.  International Journal of Advanced Robotic Systems, 9(2):36, 2012\.
  * [20] Guang-Long Du, Ping Zhang, Li-Ying Yang, and Yan-Bin Su.  Robot teleoperation using a vision-based manipulation method.  In 2010 International Conference on Audio, Language and Image Processing, pages 945–949. IEEE, 2010.
  * [21] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges.  Articulated objects in free-form hand interaction.  arXiv preprint arXiv:2204.13662, 2022.
  * [22] Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He.  Detectron.  https://github.com/facebookresearch/detectron, 2018.
  * [23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [24] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014.
  * [25] Patrick Grady, Chengcheng Tang, Christopher D Twigg, Minh Vo, Samarth Brahmbhatt, and Charles C Kemp.  Contactopt: Optimizing contact to improve grasps.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1471–1481, 2021.
  * [26] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud.  Ffjord: Free-form continuous dynamics for scalable reversible generative models.  International Conference on Learning Representations, 2019.
  * [27] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022.
  * [28] Liang-Yan Gui, Yu-Xiong Wang, Xiaodan Liang, and José M. F. Moura.  Adversarial geometry-aware human motion prediction.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV, volume 11208 of Lecture Notes in Computer Science, pages 823–842. Springer, 2018.
  * [29] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi.  Social gan: Socially acceptable trajectories with generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2255–2264, 2018.
  * [30] Henning Hamer, Juergen Gall, Thibaut Weise, and Luc Van Gool.  An object-dependent hand pose prior from sparse training data.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 671–678. IEEE, 2010.
  * [31] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit.  Honnotate: A method for 3d annotation of hand and object poses.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196–3206, 2020.
  * [32] Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, and Dieter Fox.  Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164–9170. IEEE, 2020.
  * [33] Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, and Cordelia Schmid.  Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 571–580, 2020.
  * [34] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid.  Learning joint reconstruction of hands and manipulated objects.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11807–11816, 2019.
  * [35] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  arXiv preprint arxiv:2006.11239, 2020.
  * [36] Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang.  Hand-object contact consistency reasoning for human grasps generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11107–11116, 2021.
  * [37] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7122–7131, 2018.
  * [38] Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang.  Grasping field: Learning implicit representations for human grasps.  In 2020 International Conference on 3D Vision (3DV), pages 333–344. IEEE, 2020.
  * [39] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising diffusion restoration models.  In Advances in Neural Information Processing Systems, 2022.
  * [40] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  Vibe: Video inference for human body pose and shape estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5253–5263, 2020.
  * [41] Jonathan Kofman, Siddharth Verma, and Xianghai Wu.  Robot-manipulator teleoperation by markerless vision-based hand-arm tracking.  International Journal of Optomechatronics, 1(3):331–357, 2007.
  * [42] Mia Kokic, Danica Kragic, and Jeannette Bohg.  Learning to estimate pose and shape of hand-held objects from rgb images.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3980–3987. IEEE, 2019.
  * [43] Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian Reid, Hamid Rezatofighi, and Silvio Savarese.  Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks.  Advances in Neural Information Processing Systems, 32, 2019.
  * [44] Shuang Li, Xiaojian Ma, Hongzhuo Liang, Michael Görner, Philipp Ruppel, Bin Fang, Fuchun Sun, and Jianwei Zhang.  Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network.  In 2019 International Conference on Robotics and Automation (ICRA), pages 416–422. IEEE, 2019.
  * [45] Xiaolong Li, He Wang, Li Yi, Leonidas Guibas, A Lynn Abbott, and Shuran Song.  Category-level articulated object pose estimation.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [46] Kevin Lin, Lijuan Wang, and Zicheng Liu.  Mesh graphormer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 12939–12948, 2021.
  * [47] Liu Liu, Han Xue, Wenqiang Xu, Haoyuan Fu, and Cewu Lu.  Toward real-world category-level articulation pose estimation.  IEEE Transactions on Image Processing, 31:1072–1083, 2022.
  * [48] Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi.  Hoi4d: A 4d egocentric dataset for category-level human-object interaction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013–21022, 2022.
  * [49] Roberto Martín-Martín, Clemens Eppner, and Oliver Brock.  The rbo dataset of articulated objects and interactions, 2018.
  * [50] Frank Michel, Alexander Krull, Eric Brachmann, Michael Ying Yang, Stefan Gumhold, and Carsten Rother.  Pose estimation of kinematic chain instances via object coordinate regression.  In Proceedings of the British Machine Vision Conference (BMVC), 2015\.
  * [51] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su.  Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding, 2018.
  * [52] Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang.  A-sdf: Learning disentangled signed distance functions for articulated shape representation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13001–13011, 2021.
  * [53] Alex Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models, 2021.
  * [54] Markus Oberweger, Paul Wohlhart, and Vincent Lepetit.  Generalized feedback loop for joint hand-object pose estimation.  IEEE transactions on pattern analysis and machine intelligence, 42(8):1898–1912, 2019.
  * [55] Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros.  Full dof tracking of a hand interacting with an object by modeling occlusions and physical constraints.  In 2011 International Conference on Computer Vision, pages 2088–2095. IEEE, 2011.
  * [56] Paschalis Panteleris, Nikolaos Kyriazis, and Antonis A Argyros.  3d tracking of human hands in interaction with unknown objects.  In BMVC, pages 123–1, 2015.
  * [57] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  Advances in neural information processing systems, 30, 2017.
  * [58] Yuzhe Qin, Hao Su, and Xiaolong Wang.  From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation.  arXiv preprint arXiv:2204.12490, 2022.
  * [59] Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang.  Dexmv: Imitation learning for dexterous manipulation from human videos.  In European Conference on Computer Vision, pages 570–587. Springer, 2022.
  * [60] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [61] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), Nov. 2017.
  * [62] Yu Rong, Takaaki Shiratori, and Hanbyul Joo.  Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration.  arXiv preprint arXiv:2008.08324, 2020.
  * [63] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese.  Sophie: An attentive gan for predicting paths compliant to social and physical constraints.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1349–1358, 2019.
  * [64] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models, 2021.
  * [65] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding, 2022.
  * [66] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement, 2021.
  * [67] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak.  Robotic telekinesis: learning a robotic hand imitator by watching humans on youtube.  arXiv preprint arXiv:2202.10448, 2022.
  * [68] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256–2265, Lille, France, 07–09 Jul 2015\. PMLR.
  * [69] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [70] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv:2010.02502, October 2020.
  * [71] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  arXiv preprint arXiv:2011.13456, 2020.
  * [72] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [73] Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, and Christian Theobalt.  Real-time joint tracking of a hand manipulating an object from rgb-d input.  In European Conference on Computer Vision, pages 294–310. Springer, 2016.
  * [74] Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas.  Grab: A dataset of whole-body human grasping of objects.  In European conference on computer vision, pages 581–600. Springer, 2020.
  * [75] Bugra Tekin, Federica Bogo, and Marc Pollefeys.  H+ o: Unified egocentric recognition of 3d hand-object poses and interactions.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4511–4520, 2019.
  * [76] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [77] Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang.  Multi-person 3d motion prediction with multi-range transformers.  Advances in Neural Information Processing Systems, 34:6036–6049, 2021.
  * [78] Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu.  Shape2motion: Joint analysis of motion parts and attributes from 3d shapes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8876–8884, 2019.
  * [79] Yijia Weng, He Wang, Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, and Leonidas J. Guibas.  Captra: Category-level pose tracking for rigid and articulated objects from point clouds.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 13209–13218, October 2021.
  * [80] Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C. Cattin.  Diffusion models for implicit image segmentation ensembles, 2021.
  * [81] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al.  Sapien: A simulated part-based interactive environment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11097–11107, 2020.
  * [82] Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu.  Oakink: A large-scale knowledge repository for understanding hand-object interaction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20953–20962, 2022.
  * [83] Vicky Zeng, Tabitha Edith Lee, Jacky Liang, and Oliver Kroemer.  Visual identification of articulated object parts.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2443–2450. IEEE, 2021.
  * [84] Hao Zhang, Zi-Hao Bo, Jun-Hai Yong, and Feng Xu.  Interactionfusion: real-time reconstruction of hand poses and deformable objects in hand-object interactions.  ACM Transactions on Graphics (TOG), 38(4):1–11, 2019.
  * [85] He Zhang, Yuting Ye, Takaaki Shiratori, and Taku Komura.  Manipnet: Neural manipulation synthesis with a hand-object spatial representation.  ACM Transactions on Graphics (ToG), 40(4):1–14, 2021.
  * [86] Hao Zhang, Yuxiao Zhou, Yifei Tian, Jun-Hai Yong, and Feng Xu.  Single depth view based real-time reconstruction of hand-object interactions.  ACM Transactions on Graphics (TOG), 40(3):1–12, 2021.
  * [87] Xiaoshuai Zhang, Rui Chen, Fanbo Xiang, Yuzhe Qin, Jiayuan Gu, Zhan Ling, Minghua Liu, Peiyu Zeng, Songfang Han, Zhiao Huang, et al.  Close the visual domain gap by physics-grounded active stereovision depth sensor simulation.  arXiv preprint arXiv:2201.11924, 2022.
  * [88] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu.  Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations.  arXiv preprint arXiv:2207.06635, 2022.
  * [89] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks, 2018\.
  * [90] Tianqiang Zhu, Rina Wu, Xiangbo Lin, and Yi Sun.  Toward human-like grasp: Dexterous grasping via semantic representation of object-hand.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15741–15751, 2021.