# Observation of Fermi-Pasta-Ulam Recurrence Induced by Breather Solitons in
an Optical Microresonator

Chengying Bao<EMAIL_ADDRESS>School of Electrical and Computer Engineering,
Purdue University, 465 Northwestern Avenue, West Lafayette, IN 47907-2035, USA
Jose A. Jaramillo-Villegas  School of Electrical and Computer Engineering,
Purdue University, 465 Northwestern Avenue, West Lafayette, IN 47907-2035, USA
Facultad de Ingenierías, Universidad Tecnológica de Pereira, Pereira, RI
66003, Colombia  Yi Xuan  School of Electrical and Computer Engineering,
Purdue University, 465 Northwestern Avenue, West Lafayette, IN 47907-2035, USA
Birck Nanotechnology Center, Purdue University, 1205 West State Street, West
Lafayette, Indiana 47907, USA  Daniel E. Leaird  School of Electrical and
Computer Engineering, Purdue University, 465 Northwestern Avenue, West
Lafayette, IN 47907-2035, USA  Minghao Qi  School of Electrical and Computer
Engineering, Purdue University, 465 Northwestern Avenue, West Lafayette, IN
47907-2035, USA  Birck Nanotechnology Center, Purdue University, 1205 West
State Street, West Lafayette, Indiana 47907, USA  Andrew M. Weiner
<EMAIL_ADDRESS>School of Electrical and Computer Engineering, Purdue
University, 465 Northwestern Avenue, West Lafayette, IN 47907-2035, USA  Birck
Nanotechnology Center, Purdue University, 1205 West State Street, West
Lafayette, Indiana 47907, USA  Purdue Quantum Center, Purdue University, 1205
West State Street, West Lafayette, IN 47907, USA

###### Abstract

We present, experimentally and numerically, the observation of Fermi-Pasta-
Ulam recurrence induced by breather solitons in a high-Q SiN microresonator.
Breather solitons can be excited by increasing the pump power at a relatively
small pump phase detuning in microresonators. Out of phase power evolution is
observed for groups of comb lines around the center of the spectrum compared
to groups of lines in the spectral wings. The evolution of the power spectrum
is not symmetric with respect to the spectrum center. Numerical simulations
based on the generalized Lugiato-Lefever equation are in good agreement with
the experimental results and unveil the role of stimulated Raman scattering in
the symmetry breaking of the power spectrum evolution. Our results shows that
optical microresonators can be exploited as a powerful platform for the
exploration of soliton dynamics.

###### pacs:

The Fermi-Pasta-Ulam (FPU) recurrence was first raised by Fermi and his
colleagues in the 1950s Fermi  _et al._ (1955). In a numerical simulation of
string oscillation with nonlinear coupling between different modes to test
thermalization theory, they found at a certain point the energy will return to
the fundamentally excited mode, rather than distributing homogenously among
different modes. This discovery triggered the rigorous investigation on plasma
physics by Zubusky and Kruskal Zabusky and Kruskal (1965), which led to the
discovery of solitons. Solitons and their related theory have revolutionized
the research in diverse arenas, including fluid dynamics Chabchoub  _et al._
(2011), optics Mollenauer  _et al._ (1980); Dudley and Taylor (2009), Bose-
Eistein condensation Denschlag  _et al._ (2000); Burger  _et al._ (1999).

In optics, the FPU recurrence was first demonstrated based on the modulation
instability (MI) in optical fibers Van Simaeys  _et al._ (2001). As a feature
of the FPU recurrence, the powers of the pump mode and the signal mode in MI
evolves periodically with a phase delay of $\pi$. The collision between
solitons in fibers also facilitated the observation of FPU recurrence in an
active cavity Wu and Patton (2007). Furthermore, optical breathers, e.g., the
Akhmediev breather (AB) in the nonlinear schrödinger equation (NLSE) Akhmediev
and Korneev (1986); Devine  _et al._ (2011); Kibler  _et al._ (2010), are an
important manifestation of FPU recurrence. Since collisions between breathers
and solitons can result in optical rogue waves Solli  _et al._ (2007); Dudley
_et al._ (2014); Soto-Crespo  _et al._ (2016), studying FPU recurrence and the
control of the transition between solitons and breathers may contribute to the
understanding of optical rogue waves.

Recently, maturity in the fabrication of high-Q microresonators Vahala (2003)
has fueled rapid progress on Kerr frequency comb generation Del’Haye  _et al._
(2007); Levy  _et al._ (2010); Razzari  _et al._ (2010); Kippenberg  _et al._
(2011); Herr  _et al._ (2014a); Xue  _et al._ (2015). In the frequency domain,
microresonators based frequency comb synthesis has promising applications in
optical clock Papp  _et al._ (2014), optical arbitrary waveform generation
Ferdous  _et al._ (2011), and microwave photonics Xue  _et al._ (2014); Liang
_et al._ (2015) etc. In the time domain, microresonators provide a new and
important approach to realize optical solitons Herr  _et al._ (2014a); Brasch
_et al._ (2016); Yi  _et al._ (2015); Wang  _et al._ (2016); Joshi  _et al._
(2016). Different from mode-locked lasers, passive microresonators have no
active gain or saturable absorber, making them free from the influence of the
complex gain dynamics. Hence, soliton generation in microresonators can
exhibit excellent predictability. Moreover, the bandwidth and peak power of
the soliton can be controlled by varying the pump phase detuning Herr  _et
al._ (2014a); Yi  _et al._ (2015); Bao and Yang (2015). The ability to
accurately predict and control soliton dynamics in microresonators accurately
will make microresonators a versatile test bed for the study of fundamental
soliton physics, including the FPU recurrence. Moreover, breather solitons,
which can exhibit FPU recurrence, have been widely predicted in
microresonators Matsko  _et al._ (2012); Leo  _et al._ (2013); Bao and Yang
(2014); Godey  _et al._ (2014); Bao  _et al._ (2015); Milián  _et al._ (2015),
but still lack rigorous experimental investigation, to our knowledge.

Figure 1: (color online). (a) The experimental setup and operating regimes of
Kerr frequency combs. Breather solitons are generated at relatively small
detuning and high pump power and we use three steps illustrated by I, II, III
to generate the breather soliton. MI: modulation instability, PD: photodiode,
LNA: low noise amplifier, osc: oscillation. (b) The optical spectrum of the
stable soliton (blue) and the simulated spectrum from the generalized LLE (red
line). (c) The RF spectrum of the soliton state. (d) The averaged spectrum of
the breather soliton spectrum from the optical spectrum analyzer (blue) and
from simulation (red line). (e) The RF spectrum of the breather soliton. The
pumped resonance has a linewidth of 100 MHz.

In this Letter, we present the observation of FPU recurrence induced by
breather solitons in a siicon-nitride (SiN) microresonator. By controlling the
pump condition, we can excite the breather soliton in the microresonator and
show the power evolution for the comb lines at the center and the wing is out
of phase. We also observe the spectral breathing is not symmetric with respect
to the spectrum center and identify that stimulated Raman scattering (SRS) is
responsible for the symmetry breaking. Both the recurrence and the symmetry
breaking are well described by numerical simulations based on the generalized
Lugitato-Lefever equation (LLE), including the Raman effect Lugiato and
Lefever (1987); Bao  _et al._ (2015); Milián  _et al._ (2015); Karpov  _et
al._ (2016a); Bao  _et al._ (2014). The observation of FPU recurrence in
microresonators will improve our understanding of nonlinear systems and
breather theory. Furthermore, it also gives more insight into the different
operating regimes of Kerr frequency combs and adds to the understanding of
soliton mode-locking in microresonators.

The experimental scheme is shown in Fig. 1(a). An on-chip SiN microresonator
is used for the generation of the Kerr frequency comb and the breather
soliton. When the microresonaor is pumped by a continuous-wave laser, the
power enhancement inside the cavity initiates parametric oscillation and
frequency comb generation under moderate pump power. In experiments, we use an
anomalous dispersion microresonator, whose dimensions are 800$\times$2000 nm
with a radius of 100 $\mu$m, to generate the Kerr frequency comb. A notch
filter (1550 nm, 4 nm bandwidth) is used to suppress the strong pump line by
over 35 dB. Figure 1(a) also illustrates the operating regimes of Kerr
frequency combs and the experimental method to excite the breather soliton.
The breather soliton regime is close to the soliton regime and breather
solitons are generated at relatively small pump phase detuning and high pump
power Leo  _et al._ (2013); Bao and Yang (2015). Therefore, we have three
steps to generate the breather soliton: I) tune the laser to generate the
soliton, II) tune the pump laser backward several picometers to lower the
excitation threshold of breather solitons (see Fig. S1 in Supplementary
Information SI ; Weiner (2011); Horowitz  _et al._ (1997)), III) increase the
pump power to excite the breather soliton.

For phase I, stable solitons can be generated by scanning the laser across the
cavity resonance from blue-side to red-side Herr  _et al._ (2014a). To
overcome the transient instability in soliton generation in SiN
microresonators, the laser is tuned backward after crossing the cavity
resonance Wang  _et al._ (2016); Karpov  _et al._ (2016b). This backwards
tuning also gives access to single soliton. At the pump power of $\sim$ 300 mW
(in the bus waveguide), when pumped around 1551.28 nm, a stable single soliton
is generated in the microresonator. The comb has a well-defined sech2 spectrum
(Fig. 1(b)) and low intensity noise (Fig. 1(c)). There are some spectral jumps
on the optical spectrum, which can be attributed to the mode-interaction,
however they will not change the soliton property significantly Yi  _et al._
(2015); Wang  _et al._ (2016); Herr  _et al._ (2014b), qualitatively different
from the multi-phase-step combs in Ref. Del’Haye  _et al._ (2015).

Soliton behavior in microresonators is governed by the LLE Lugiato and Lefever
(1987); Coen  _et al._ (2013). For SiN microresonators, SRS is important in
determining the property of solitons Wang  _et al._ (2016); Karpov  _et al._
(2016a); Bao  _et al._ (2015, 2014). Hence, we use the generalized LLE, with
SRS included, to describe the soliton generation,

$\displaystyle\left({{\tau}_{0}}\frac{\partial}{\partial
t}+\frac{{{\alpha}}+\theta}{2}+i{{\delta}_{0}}+i\frac{{{\beta}_{2}}L}{2}\frac{{{\partial}^{2}}}{\partial{{\tau}^{2}}}\right)E-i(1-f_{R})\gamma
L{{\left|E\right|}^{2}}E$ (1) $\displaystyle-
if_{R}{{\gamma}}L\left(E\int_{-\infty}^{\tau}{{{h}_{R}}\left(\tau-\tau^{\prime}\right){{\left|E\right|}^{2}}d\tau^{\prime}}\right)-\sqrt{\theta}{{E}_{in}}=0,$

where $E$ is the envelope of the intracavity field, $\tau_{0}$ is the round-
trip time (4.5 ps), $L$ is the length of the cavity, $t$ and $\tau$ are the
slow and fast time respectively, $\alpha$ and $\theta$ are the intrinsic loss
and the external coupling coefficient respectively, $\beta_{2}$ is the group
velocity dispersion, $\gamma$ is the nonlinear coefficient, $\delta_{0}$ is
the pump phase detuning, $\left|E_{in}\right|^{2}$ is the pump power, and
$h_{R}(\tau)$ is the Raman response function. In simulations, the Raman effect
is calculated in the frequency domain, with a Lorentzian gain spectrum Lin
_et al._ (2007), whose peak is centered at $-$14.3 THz and bandwidth is 2.12
THz. When choosing $\alpha$=0.0024, $\theta$=0.0004, $\beta_{2}=-$81 ps2/km,
$\gamma$=0.9 W/m, $f_{R}$=0.13 and setting the pump condition as 220 mW, and
the $\delta_{0}$=0.022, a stable soliton is generated, whose spectrum is shown
by the red-line in Fig. 1(b), in close agreement with experiments. The
difference in the amplitude of the pump line results from the strong directly
transmitted pump, superimposed on the output comb. A closer agreement can be
reached, if we include a small third order dispersion (TOD). However, to rule
out the role of TOD in the symmetry breaking of the spectral breathing,
discussed below, we exclude it in the simulation (see sections 2 and 5 in
Supplementary Information).

Figure 2: (color online). (a) The recorded fast evolution of the comb lines
around the center and in the wings of the spectrum. The inset is the RF
spectrum of the power evolution measured over 4 $\mu$s for comb lines in the
wings. A 20 MHz bandpass filter is used to select the strongest RF tone. (b)
Reconstructed comb line power evolution after numerical filtering. The blue
(orange) shaded regions illustrate the time slots where the power flows toward
the center (wing). Center (blue lines): comb lines within 1553 nm$\sim$1569
nm, wing (orange lines): comb lines within 1530 nm$\sim$1550 nm and 1569
nm$\sim$1600 nm. Figure 3: (color online). The recorded power evolution for
different comb line groups when the pulse-shaper is used to select spectral
regions with finer resolution. The red dashed line illustrates the peak of
different groups and shows different groups experience modulations with
different relative phases.

After the generation of stable solitons, we follow the road-map shown in Fig.
1(a) to generate the breather soliton. By tuning the laser backwards several
picometers further and increasing the pump power to $\sim$430 mW, the breather
soliton can be excited (see section 3 of Supplemental Information for the
transition dynamics from stable solitons to breather solitons). The breather
soliton state is identified by the sharp peak in the RF spectrum (Fig. 1(e)).
Note that the breathing frequency is nearly 4 times the linewidth of the
pumped resonance (100 MHz). From our measurements, the modulation depth of the
converted comb lines, defined as
${\left({{P}_{\max}}-{{P}_{\min}}\right)}/{\left({{P}_{\max}}+{{P}_{\min}}\right)}$
with $P_{\max(\min)}$ being the maximum (minimum) average power, is
$\sim$50$\%$. Similar narrow RF peaks were observed in normal dispersion
microresonators and interpreted as dark breather pulse Xue  _et al._ (2015).
Narrow RF peaks have also been reported recently in anomalous dispersion Si
and SiN microresonators Yu  _et al._ (2016). The spectrum of the breather
solitons becomes sharper at the top of the spectrum, compared to the soliton
spectrum. In simulations, breather solitons can be generated by decreasing
$\delta_{0}$=0.014 and increasing the pump power to 360 mW. The averaged
spectrum (averaging over slow time $t$) of the simulated breather soliton is
shown by the red line in Fig. 1(d), which reproduces the sharp top of the
experimental spectrum. The breather soliton retains a Raman induced frequency
shift, implying the breather soliton remains as a pulse, as chaotic waveforms
do not exhibit the frequency shift Karpov  _et al._ (2016a). The
autocorrelation trace also provides evidence of pulse-like behavior (see
section 4 of Supplementary Information for further temporal details and
discussion of the temporal breathing). Furthermore, the excitation process is
reversible, i.e., we can return to the soliton state from the breather soliton
state by manually decreasing the pump power.

Figure 4: (color online). (a) The numerically filtered spectral evolution of
the breather soliton in simulation, with SRS included. The various spectral
groups far away from the center have approximately the same phase but their
phase is clearly different from that of the group of comb lines around the
center ($\sim$1560 nm). (b) Simulation of the FPU recurrence for comb lines
around the center (blue line, 1552 nm$\sim$1567 nm) and comb lines in the
wings (orange line, 1521 nm$\sim$1550 nm and 1567 nm$\sim$1592 nm, with the
pump at 1551 nm excluded). The shaded regions illustrate the power exchange
between the center and the wing. (c) The experimental traces at long (above
1600 nm) and short wavelengths (below 1530 nm), showing the same phase between
them. (d) The numerically filtered spectral breathing in simulation, without
SRS. The phase for different slices is symmetric with respect to the center of
the spectrum (1551 nm). The red dashed lines in (a), (d) illustrate the peak
of different slices.

To test the FPU recurrence induced by breather solitons, we use a pulse-shaper
Weiner (2000) to select out some specific comb lines to record the fast
evolution of the breathing spectrum. The pulse-shaper has a transmission
bandwidth spanning from 1530 nm to 1600 nm, with the ability to programmably
select out specific comb lines. For synchronization between different spectral
slices, a portion of the output comb is used as reference signal to trigger
the oscilloscope (PD1 in Fig. 1(a)). We use the pulse-shaper to select out
either 9 comb lines around the center of the spectrum ($\sim$ 1560 nm, 9 nm to
the red of the pump） or the remaining 31 comb lines (but not the pump line)
within the passband of the pulse-shaper in the short and long wavelength
wings.

The recorded traces for the comb lines around the center and comb lines in the
wings are depicted in Fig. 2(a). The power change of the comb lines around the
center is nearly out of phase of the comb lines in the wings, a signature of
FPU recurrence. Due to the low power of a single comb line after the pulse-
shaper and its fast breathing rate ($<$1 $\mu$W, $>$350 MHz), the recorded
trace shows some distortions. However, the breathing exhibits good periodicity
(see Fig. 1(e)), allowing numerical filtering of the signal. Here, we record
the traces over 4 $\mu$s, corresponding to more than 1000 breathing cycles;
the RF spectrum computed from the recorded trace for the wing is shown in the
inset of Fig. 2(a). The strongest RF tone is selected by a 20 MHz numerical
filter and inverse Fourier transformed to yield the reconstructed traces in
Fig. 2(b), providing a clearer view of the recurrence.

To gain more insights into the breathing of the spectrum, the pulse-shaper is
programmed to select different spectral regions continuously across the
spectrum. The recorded map of the spectral breathing is presented in Fig. 3.
The dashed line in Fig. 3 clearly illustrates the phase delay between
different slices when the spectrum is breathing. Furthermore, the phase delay
is not symmetric with respect to the center of the spectrum ($\sim$1560 nm).

Breather solitons in microresonators arise from the Hopf bifurcation in LLE
Matsko  _et al._ (2012); Leo  _et al._ (2013); Godey  _et al._ (2014); Milián
_et al._ (2015) and can also be attributed to the mismatch between the
carrier-envelope phase slip of the soliton and the pump phase detuning Bao and
Yang (2015). To understand the FPU recurrence and the symmetry breaking in
breather solitons, we use simulation to look into the breathing dynamics. To
explicitly show the phase delay between different slices, the breather soliton
evolution is numerically filtered in a similar way to that used in Fig. 2. The
phase delay between different slices observed in experiment is also seen in
the simulated breathing dynamics with SRS included, shown in Fig. 4(a) (this
effect is also highlighted by the normalized spectral breathing dynamics in
Fig. S5 in the Supplementary Information, which also shows TOD alone is
insufficient to cause asymmetric breathing). Moreover, the symmetry breaking
with respect to the spectrum center is also in agreement with Fig. 3. As shown
in Fig. 4(b), the comb lines at the center and in the wing show out of phase
evolution and FPU recurrence, similar to the experiment. However, the comb
lines far away from the center (beyond the bandwidth of our commercial pulse-
shaper) breathes with the phase delay of 2$\pi$, i.e., the same phase. This is
verified by recording and numerical filtering the evolution recorded for comb
lines at the long and short wavelengths selected using a home-built pulse-
shaper, see Fig. 4(c). Furthermore, simulation also shows the mode-interaction
induced spectral jumps in Fig. 1(d) have negligible influence on the breathing
dynamics (see section 6 in the Supplementary Information).

To further unveil what causes the asymmetric breathing for the breather
solitons, we turn off the SRS term in simulation. The breather soliton can
still be excited under the same pump condition. However, the breathing
dynamics in Fig. 4(d) are symmetric with respect to the center of the spectrum
(coincide with the 1551 nm pump, as there is no SRS induced soliton frequency
shift). The comparison with Fig. 4(a) reveals that SRS is responsible for the
symmetry breaking in the breathing dynamics observed in Fig. 3. SRS is
generally significant for SiN microresonators Karpov  _et al._ (2016a); Wang
_et al._ (2016). However, in fluoride microresonators, SRS is much weaker Herr
_et al._ (2014a); hence, symmetric breathing of solitons can be expected.

For the breather in the microresonator, one significant difference from ABs is
the energy in the wing of the spectrum can return to a group of several modes
around the center (see Fig. 2 and Fig. 4(b)) while energy returns to the
single pump for ABs. This is because the breather in microresonators remains
to be a pulse during evolution, while ABs fully recover to be a continuous
wave in FPU recurrence. This difference shows how the soliton dynamics in the
framework of the LLE are distinct from those in the NLSE and illustrates how
the dissipative effects and SRS break the integrability of the system and
affect the breather behavior.

In conclusion, we have observed breather solitons and FPU recurrence in an on-
chip SiN microresonator. Breather solitons can be excited at high pump power
and small detuning. By selecting out two groups of comb lines around the
center and in the wing, we find the energy returns to the center and flows out
from the center periodically. Furthermore, we show SRS breaks the symmetry of
the spectral breathing. Our results shows how the dissipative effects and SRS
affect the breather properties and can contribute to the understanding of
breathers and the operation of Kerr frequency combs. Furthermore, the
observation of FPU recurrence in microresonators shows on-chip microresonators
can be used as a powerful platform to explore soliton physics.

###### Acknowledgements.

This work was supported in part by the Air Force Office of Scientific Research
(AFOSR) (FA9550-15-1-0211), by the DARPA PULSE program (W31P40-13-1-0018) from
AMRDEC, and by the National Science Foundation (NSF) (ECCS-1509578).

## References

  * Fermi  _et al._ (1955) E. Fermi, J. Pasta, and S. Ulam, Los Alamos Report LA-1940 978 (1955).
  * Zabusky and Kruskal (1965) N. J. Zabusky and M. D. Kruskal, Phys. Rev. Lett. 15, 240 (1965).
  * Chabchoub  _et al._ (2011) A. Chabchoub, N. Hoffmann, and N. Akhmediev, Phys. Rev. Lett. 106, 204502 (2011).
  * Mollenauer  _et al._ (1980) L. F. Mollenauer, R. H. Stolen, and J. P. Gordon, Phys. Rev. Lett. 45, 1095 (1980).
  * Dudley and Taylor (2009) J. M. Dudley and J. R. Taylor, Nat. Photonics 3, 85 (2009).
  * Denschlag  _et al._ (2000) J. Denschlag, J. Simsarian, D. Feder, C. W. Clark, L. Collins, J. Cubizolles, L. Deng, E. Hagley, K. Helmerson, W. P. Reinhardt, S. Rolston, I. Schneider, and W. Phillips, Science 287, 97 (2000).
  * Burger  _et al._ (1999) S. Burger, K. Bongs, S. Dettmer, W. Ertmer, K. Sengstock, A. Sanpera, G. V. Shlyapnikov, and M. Lewenstein, Phys. Rev. Lett. 83, 5198 (1999).
  * Van Simaeys  _et al._ (2001) G. Van Simaeys, P. Emplit, and M. Haelterman, Phys. Rev. Lett. 87, 033902 (2001).
  * Wu and Patton (2007) M. Wu and C. E. Patton, Phys. Rev. Lett. 98, 047202 (2007).
  * Akhmediev and Korneev (1986) N. Akhmediev and V. Korneev, Theor. Math. Phys. 69, 1089 (1986).
  * Devine  _et al._ (2011) N. Devine, A. Ankiewicz, G. Genty, J. M. Dudley, and N. Akhmediev, Phys. Lett. A 375, 4158 (2011).
  * Kibler  _et al._ (2010) B. Kibler, J. Fatome, C. Finot, G. Millot, F. Dias, G. Genty, N. Akhmediev, and J. M. Dudley, Nat. Physics 6, 790 (2010).
  * Solli  _et al._ (2007) D. Solli, C. Ropers, P. Koonath, and B. Jalali, Nature 450, 1054 (2007).
  * Dudley  _et al._ (2014) J. M. Dudley, F. Dias, M. Erkintalo, and G. Genty, Nature Photonics 8, 755 (2014).
  * Soto-Crespo  _et al._ (2016) J. Soto-Crespo, N. Devine, and N. Akhmediev, Phys. Rev. Lett. 116, 103901 (2016).
  * Vahala (2003) K. J. Vahala, Nature 424, 839 (2003).
  * Del’Haye  _et al._ (2007) P. Del’Haye, A. Schliesser, O. Arcizet, T. Wilken, R. Holzwarth, and T. Kippenberg, Nature 450, 1214 (2007).
  * Levy  _et al._ (2010) J. S. Levy, A. Gondarenko, M. A. Foster, A. C. Turner-Foster, A. L. Gaeta, and M. Lipson, Nature Photonics 4, 37 (2010).
  * Razzari  _et al._ (2010) L. Razzari, D. Duchesne, M. Ferrera, R. Morandotti, S. Chu, B. Little, and D. Moss, Nature Photonics 4, 41 (2010).
  * Kippenberg  _et al._ (2011) T. J. Kippenberg, R. Holzwarth, and S. Diddams, Science 332, 555 (2011).
  * Herr  _et al._ (2014a) T. Herr, V. Brasch, J. Jost, C. Wang, N. Kondratiev, M. Gorodetsky, and T. Kippenberg, Nat. Photonics 8, 145 (2014a).
  * Xue  _et al._ (2015) X. Xue, Y. Xuan, Y. Liu, P.-H. Wang, S. Chen, J. Wang, D. E. Leaird, M. Qi, and A. M. Weiner, Nat. Photonics 9, 594 (2015).
  * Papp  _et al._ (2014) S. B. Papp, K. Beha, P. Del’Haye, F. Quinlan, H. Lee, K. J. Vahala, and S. A. Diddams, Optica 1, 10 (2014).
  * Ferdous  _et al._ (2011) F. Ferdous, H. Miao, D. E. Leaird, K. Srinivasan, J. Wang, L. Chen, L. T. Varghese, and A. M. Weiner, Nat. Photonics 5, 770 (2011).
  * Xue  _et al._ (2014) X. Xue, Y. Xuan, H.-J. Kim, J. Wang, D. E. Leaird, M. Qi, and A. M. Weiner, J. Lightwave Tech. 32, 3557 (2014).
  * Liang  _et al._ (2015) W. Liang, D. Eliyahu, V. Ilchenko, A. Savchenkov, A. Matsko, D. Seidel, and L. Maleki, Nat. Communications 6, 7057 (2015).
  * Brasch  _et al._ (2016) V. Brasch, M. Geiselmann, T. Herr, G. Lihachev, M. Pfeiffer, M. Gorodetsky, and T. Kippenberg, Science 351, 357 (2016).
  * Yi  _et al._ (2015) X. Yi, Q.-F. Yang, K. Y. Yang, M.-G. Suh, and K. Vahala, Optica 2, 1078 (2015).
  * Wang  _et al._ (2016) P.-H. Wang, J. A. Jaramillo-Villegas, Y. Xuan, X. Xue, C. Bao, D. E. Leaird, M. Qi, and A. M. Weiner, Opt. Express 24, 10890 (2016).
  * Joshi  _et al._ (2016) C. Joshi, J. K. Jang, K. Luke, X. Ji, S. A. Miller, A. Klenner, Y. Okawachi, M. Lipson, and A. L. Gaeta, Opt. Lett. 41, 2565 (2016).
  * Bao and Yang (2015) C. Bao and C. Yang, Phys. Rev. A 92, 053831 (2015).
  * Matsko  _et al._ (2012) A. Matsko, A. Savchenkov, and L. Maleki, Opt. Lett. 37, 4856 (2012).
  * Leo  _et al._ (2013) F. Leo, L. Gelens, P. Emplit, M. Haelterman, and S. Coen, Opt. Express 21, 9180 (2013).
  * Bao and Yang (2014) C. Bao and C. Yang, J. Opt. Soc. Am. B 31, 3074 (2014).
  * Godey  _et al._ (2014) C. Godey, I. V. Balakireva, A. Coillet, and Y. K. Chembo, Phys. Rev. A 89, 063814 (2014).
  * Bao  _et al._ (2015) C. Bao, L. Zhang, L. C. Kimerling, J. Michel, and C. Yang, Opt. Express 23, 18665 (2015).
  * Milián  _et al._ (2015) C. Milián, A. V. Gorbach, M. Taki, A. V. Yulin, and D. V. Skryabin, Phys. Rev. A 92, 033851 (2015).
  * Lugiato and Lefever (1987) L. A. Lugiato and R. Lefever, Phys. Rev. Lett. 58, 2209 (1987).
  * Karpov  _et al._ (2016a) M. Karpov, H. Guo, A. Kordts, V. Brasch, M. H. Pfeiffer, M. Zervas, M. Geiselmann, and T. J. Kippenberg, Phys. Rev. Lett. 116, 103902 (2016a).
  * Bao  _et al._ (2014) C. Bao, L. Zhang, A. Matsko, Y. Yan, Z. Zhao, G. Xie, A. M. Agarwal, L. C. Kimerling, J. Michel, L. Maleki, and A. Willner, Opt. Lett. 39, 6126 (2014).
  * (41) See the Supplementary Information for more experiment and simulation details, which includes the Refs.[42, 43] .
  * Weiner (2011) A. Weiner,  _Ultrafast optics_ , Vol. 72 (John Wiley & Sons, 2011).
  * Horowitz  _et al._ (1997) M. Horowitz, Y. Barad, and Y. Silberberg, Opt. Lett. 22, 799 (1997).
  * Karpov  _et al._ (2016b) M. Karpov, H. Guo, E. Lucas, A. Kordts, M. Pfeiffer, G. Lichachev, V. Lobanov, M. Gorodetsky, and T. Kippenberg, arXiv preprint arXiv:1601.05036 (2016b).
  * Herr  _et al._ (2014b) T. Herr, V. Brasch, J. Jost, I. Mirgorodskiy, G. Lihachev, M. Gorodetsky, and T. Kippenberg, Phys. Rev. Lett. 113, 123901 (2014b).
  * Del’Haye  _et al._ (2015) P. Del’Haye, A. Coillet, W. Loh, K. Beha, S. B. Papp, and S. A. Diddams, Nat. Commun. 6, 5668 (2015).
  * Coen  _et al._ (2013) S. Coen, H. G. Randle, T. Sylvestre, and M. Erkintalo, Opt. Lett. 38, 37 (2013).
  * Lin  _et al._ (2007) Q. Lin, O. J. Painter, and G. P. Agrawal, Opt. Express 15, 16604 (2007).
  * Yu  _et al._ (2016) M. Yu, J. K. Jang, Y. Okawachi, A. Griffith, K. Luke, S. Miller, X. Ji, M. Lipson, and A. L. Gaeta, in  _CLEO: QELS_Fundamental Science_ (Optical Society of America, 2016) pp. FM2A–8.
  * Weiner (2000) A. M. Weiner, Rev. Sci. Instr. 71, 1929 (2000).


11institutetext: The University of Tokyo, Tokyo, Japan 22institutetext:
Carnegie Mellon University, PA, USA
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>
Project: https://tkhkaeio.github.io/projects/22-hand-ps-da/

# Domain Adaptive Hand Keypoint and Pixel Localization in the Wild

Takehiko Ohkawa 1122 Yu-Jhe Li 22 Qichen Fu 22 Ryosuke Furuta 11 Kris M.
Kitani 22 and Yoichi Sato 11

###### Abstract

We aim to improve the performance of regressing hand keypoints and segmenting
pixel-level hand masks under new imaging conditions (_e.g_., outdoors) when we
only have labeled images taken under very different conditions (_e.g_.,
indoors). In the real world, it is important that the model trained for both
tasks works under various imaging conditions. However, their variation covered
by existing labeled hand datasets is limited. Thus, it is necessary to adapt
the model trained on the labeled images (source) to unlabeled images (target)
with unseen imaging conditions. While self-training domain adaptation methods
(_i.e_., learning from the unlabeled target images in a self-supervised
manner) have been developed for both tasks, their training may degrade
performance when the predictions on the target images are noisy. To avoid
this, it is crucial to assign a low importance (confidence) weight to the
noisy predictions during self-training. In this paper, we propose to utilize
the divergence of two predictions to estimate the confidence of the target
image for both tasks. These predictions are given from two separate networks,
and their divergence helps identify the noisy predictions. To integrate our
proposed confidence estimation into self-training, we propose a teacher-
student framework where the two networks (teachers) provide supervision to a
network (student) for self-training, and the teachers are learned from the
student by knowledge distillation. Our experiments show its superiority over
state-of-the-art methods in adaptation settings with different lighting,
grasping objects, backgrounds, and camera viewpoints. Our method improves by
$4\%$ the multi-task score on HO3D compared to the latest adversarial
adaptation method. We also validate our method on Ego4D, egocentric videos
with rapid changes in imaging conditions outdoors.

##  1 Introduction

Figure 1:  We aim to adapt the model of localizing hand keypoints and pixel-
level hand masks to new imaging conditions without annotation.

In the real world, hand keypoint regression and hand segmentation are
considered important to work under broad imaging conditions for various
computer vision applications, such as egocentric video understanding [29, 17],
hand-object interaction analysis [12, 21], AR/VR [40, 72], and assistive
technology [41, 38]. For building models for both tasks, several labeled hand
datasets have been proposed in laboratory settings, such as multi-camera
studios [35, 47, 13, 82] and attaching sensors to hands [24, 77, 27]. However,
their imaging conditions do not adequately cover real-world imaging conditions
[52], consisting of various lighting, hand-held objects, backgrounds, and
camera viewpoints. In addition, the annotation of keypoints and pixel-level
masks are not always available in real-world environments because they are
labor-intensive to acquire. As shown in Fig. 1, when localizing hand keypoints
and pixels in real-world egocentric videos [29] (_e.g_., outdoors), we may
only have access to a hand dataset [13] taken under completely different
imaging conditions (_e.g_., indoors). Given these limitations, we need methods
that can robustly adapt the models trained on the available labeled images
(source) to unlabeled images (target) with new imaging conditions.

To enable such adaptation, the approach of self-training domain adaptation has
been developed for both tasks. This approach aims to learn unlabeled target
images by optimizing a self-supervised task, which exhibits effectiveness in
various domain adaptation tasks [15, 18, 69, 79, 7]. For keypoint estimation,
consistency training, a method that regularizes keypoint predictions to be
consistent under geometric transformations, has been proposed [75, 68, 79]. As
for hand segmentation, prior studies use pseudo-labeling [54, 7], which
produces hard labels by thresholding a predicted class probability for
updating a network. However, these self-training methods for both tasks
perform well only when the predictions are reasonably correct. When the
predictions become noisy due to the gap in imaging conditions, the trained
network will cause over-fitting to the noisy predictions, resulting in poor
performance in the target domain.

To avoid this, it is crucial to assign a low importance (confidence) weight to
the loss of self-training with noisy predictions. This confidence weighting
can mitigate the distractions from the noisy predictions. To this end, we
propose self-training domain adaptation with confidence estimation for hand
keypoint regression and hand segmentation. Our proposed method consists of (i)
confidence estimation based on the divergence of two networks’ predictions and
(ii) an update rule that integrates a training network for self-training and
the two networks for confidence estimation.

To (i) estimate confidence, we utilize the predictions of two different
networks. While class probability can be used as the confidence in
classification tasks, it is not trivial to obtain such a measure in keypoint
regression. Thus, we newly focus on the divergence of the two networks’
predictions for each target image. We design their networks to have an
identical architecture but have different learning parameters. We observe that
when the divergence measure is high, the predictions of both networks are
noisy and should be avoided in self-training.

To (ii) integrate the estimated confidence into self-training, inspired by the
single-teacher-single-student update [66, 55], we develop mutual training with
self-training based on consistency training for a training network (student)
and distillation-based update for the two networks (teachers). For training
the student network, we build a unified self-training framework that can work
favorably for the two tasks. Motivated by supervised or weakly-supervised
learning for jointly estimating both tasks [70, 78, 28, 50, 16], we expect
that jointly adapting both tasks will allow one task to provide useful cues to
the other task even in the unlabeled target domain. Specifically, we enforce
the student network to generate consistent predictions for both tasks under
geometric augmentation. We weight the loss of the consistency training using
the confidence estimated from the divergence of the teachers’ predictions.
This can reduce the weight of the noisy predictions during the consistency
training. To learn the two teacher networks differently, we train the teachers
independently from different mini-batches by knowledge distillation, which
matches the teacher-student predictions in the output level. This framework
enables the teachers to update more carefully than the student and prevent
over-fitting to the noisy predictions. Such stable teachers provide reliable
confidence estimation for the student’s training.

In our experiments, we validate our proposed method in adaptation settings
where lighting, grasping objects, backgrounds, camera viewpoints, etc., vary
between labeled source images and unlabeled target images. We use a large-
scale hand dataset captured in a multi-camera system [13] as the source
dataset (see Fig. 1). For the target dataset, we use HO3D [30] with different
environments, HanCo [80] with multiple viewpoints and diverse backgrounds, and
FPHA [24] with a novel first-person camera viewpoint. We also apply our method
to in-the-wild egocentric video Ego4D [29] (see Fig. 1), including diverse
indoor and outdoor activities worldwide. Our method improves the average score
of the two tasks by $14.4\%$, $14.9\%$, and $18.0\%$ on HO3D, HanCo, and FPHA,
respectively, compared to a unadapted baseline. Our method further exhibits
distinct improvements compared to the latest adversarial adaption method [34]
and consistency training baselines with uncertainty estimation [7], confident
instance selection [54], and the teacher-student scheme [66]. We finally
confirm that our method also performs qualitatively well on the Ego4D videos.

Our contributions are summarized as follows:

  * •

We propose a novel confidence estimation method based on the divergence of the
predictions from two teacher networks for self-training domain adaptation of
hand keypoint regression and hand segmentation.

  * •

To integrate our proposed confidence estimation into self-training, we propose
mutual training using knowledge distillation with a student network for self-
training and two teacher networks for confidence estimation.

  * •

Our proposed framework outperforms state-of-the-art methods under three
adaptation settings across different imaging conditions. It also shows
improved qualitative performance on in-the-wild egocentric videos.

##  2 Related Work

Hand keypoint regression is the task of regressing the positions of hand joint
keypoints from a cropped hand image. 2D hand keypoint regression is trained by
optimizing keypoint heatmaps [71, 51, 81] or directly predicting keypoint
coordinates [61]. The 2D keypoints are informative for estimating 3D hand
poses [63, 76, 48, 5]. To build an accurate keypoint regressor, collecting
massive hand keypoint annotations is required but laborious. While early works
annotate the keypoints manually from a single view [57, 64, 49], recent
studies have collected the annotation more densely and efficiently using
synthetic hand models [31, 48, 81, 49], hand sensors [24, 77, 65, 27], or
multi-camera setups [35, 47, 13, 6, 82, 30, 44]. However, these methods suffer
the gap in imaging conditions with real-world images in deployment [52]. For
instance, the synthetic hand models and hand sensors induce different lighting
conditions from actual human hands. The multi-camera setup lacks a variety of
lighting, grasping objects, and backgrounds. To tackle these problems, domain
adaptation is a promising solution that can transfer the knowledge of the
network trained on source data to unlabeled target data. Jiang  _et al_.
proposed an adversarial domain adaptation for human and hand keypoint
regression, optimizing the discrepancy between regressors [34]. Additionally,
self-training adaptation methods have been studied in the keypoint regression
of animals [11], humans [68], and objects [79]. Unlike these prior works, we
incorporate confidence estimation into a self-training method based on
consistency training for keypoint regression.

Hand segmentation is the task of segmenting pixel-level hand masks in a given
image. CNN-based segmentation networks [67, 3, 36] are popularly used. The
task can be jointly trained with hand keypoint regression because detecting
hand regions guides to improve keypoint localization [70, 78, 28, 50, 16].
Since hand mask annotation is laborious as hand keypoint regression, a few
domain adaptation methods with pseudo-labeling have been explored [7, 54]. To
reduce the effect of highly noisy pseudo-labels in the target domain, Cai  _et
al_. incorporate the uncertainty of pseudo-labels in model adaptation [7], and
Ohkawa  _et al_. select confident pseudo-labels by the overlap of two
predicted hand masks [54]. Unlike [7], we estimate the target confidence using
two networks. Instead of using the estimated confidence for instance selection
[54], we assign the confidence to weight the loss of consistency training.

Domain adaptation via self-training aims to learn unlabeled target data in a
self-supervised learning manner. This approach can be divided into three
categories. (i) Pseudo-labeling [15, 60, 83, 54, 7] learns unlabeled data with
hard labels assigned by confidence thresholding from the output of a network.
(ii) Entropy minimization [43, 69, 56] regularizes the conditional entropy of
unlabeled data and increases the confidence of class probability. (iii)
Consistency regularization [73, 14, 20] enforces regularization so that the
prediction on unlabeled data is invariant under data perturbation. We choose
to leverage this consistency-based method for our task because it works for
various tasks [46, 42, 53] and the first two approaches cannot be directly
applied. Similar to our work, Yang  _et al_. [75] enforce the consistency for
two different views and modalities in hand keypoint regression. Mean teacher
[66] provides teacher-student training with consistency regularization, which
regularizes a teacher network by a student’s weights and avoids over-fitting
to incorrect predictions. Unlike [75], we propose to integrate confidence
estimation into the consistency training and adopt the teacher-student scheme
with two networks. To encourage the two networks to have different
representations, we propose a distillation-based update rule instead of
updating the teacher with the exponential moving average [66].

##  3 Proposed Method

In this section, we present our proposed self-training domain adaptation with
confidence estimation for adapting hand keypoint regression and hand
segmentation. We first present our problem formulation and network
initialization with supervised learning from source data. We then introduce
our proposed modules: (1) geometric augmentation consistency, (2) confidence
weighting by using two networks, and (3) teacher-student update via knowledge
distillation. As shown in Fig. 2, our adaptation is done with two different
networks (teachers) for confidence estimation and another network (student)
for self-training of both tasks.

Problem formulation. Given labeled images from one source domain and unlabeled
images from another target domain, we aim to jointly estimate hand keypoint
coordinates and pixel-level hand masks on the target domain. We have a source
image $\bm{x}_{\mathrm{s}}$ drawn from a set
$X_{\mathrm{s}}\subset\mathbb{R}^{H\times W\times 3}$, its corresponding
labels $(\bm{y}_{\mathrm{s}}^{\mathrm{p}},\bm{y}_{\mathrm{s}}^{\mathrm{m}})$,
and a target image $\bm{x}_{\mathrm{t}}$ drawn from a set
$X_{\mathrm{t}}\subset\mathbb{R}^{H\times W\times 3}$. The pose label
$\bm{y}_{\mathrm{s}}^{\mathrm{p}}$ consists of the 2D keypoint coordinates of
21 hand joints obtained from a set
$Y_{\mathrm{s}}^{\mathrm{p}}\subset\mathbb{R}^{21\times 2}$, while the mask
label $\bm{y}_{\mathrm{s}}^{\mathrm{m}}$ denotes a binary mask obtained from
$Y_{\mathrm{s}}^{\mathrm{m}}\subset{(0,1)}^{H\times W}$. A network
parameterized by $\bm{\theta}$ learns the mappings $f^{k}(x;\bm{\theta}):X\to
Y^{k}$ where $k\in\\{\mathrm{p},\mathrm{m}\\}$ represents the indicator for
both tasks.

Initialization with supervised learning. To initialize networks used in our
adaptation, we train the network $f$ on the labeled source data following
multi-task learning. Given the labeled dataset
$(X_{\mathrm{s}},Y_{\mathrm{s}})$ and the network $\bm{\theta}$, a supervised
loss function is defined as

$\displaystyle\mathcal{L}_{\mathrm{task}}\left(\bm{\theta},X_{\mathrm{s}},Y_{\mathrm{s}}\right)=\sum_{k}\lambda^{k}\mathbb{E}_{(\bm{x}_{\mathrm{s}},\bm{y}_{\mathrm{s}}^{k})\sim(X_{\mathrm{s}},Y_{\mathrm{s}}^{k})}\left[\mathcal{L}^{k}(\bm{p}_{s}^{k},\bm{y}_{\mathrm{s}}^{k})\right],$
(1)

where
$Y_{\mathrm{s}}=\\{Y_{\mathrm{s}}^{\mathrm{p}},Y_{\mathrm{s}}^{\mathrm{m}}\\}$
and $\bm{p}_{\mathrm{s}}^{k}=f^{k}(\bm{x}_{\mathrm{s}};\bm{\theta})$.
$\mathcal{L}^{k}(\cdot,\cdot):Y^{k}\times Y^{k}\to\mathbb{R}^{+}$ is a loss
function in each task and $\lambda^{k}$ is a hyperparameter to balance the two
tasks. We use a smooth L1 loss [33, 59] as $\mathcal{L}^{\mathrm{p}}$ and a
binary cross-entropy loss as $\mathcal{L}^{\mathrm{m}}$.

###  3.1 Geometric Augmentation Consistency

Inspired by semi-supervised learning using hand keypoint consistency [75], we
advance a unified training with consistency for both hand keypoint regression
and hand segmentation. We expect that joint adaption of both tasks will allow
one task to provide useful cues to the other task in consistency training, as
studied in supervised or weakly-supervised learning setups [70, 78, 28, 50,
16]. We design consistency training by predicting the location of hand
keypoints and hand pixels in a given geometrically transformed image,
including rotation and transition. This consistency under geometric
augmentation encourages the network to learn against positional bias in the
target domain, which helps capture the hand structure related to poses and
regions. Specifically, given a paired augmentation function
$(T_{\mathrm{x}},T_{\mathrm{y}}^{k})\sim\mathcal{T}$ for an image and an
label, we generate the prediction on the target images
$\bm{p}_{\mathrm{t}}^{k}=f^{k}\left(\bm{x}_{\mathrm{t}};\bm{\theta}\right)$
and the augmented target images
$\bm{p}_{\mathrm{t,aug}}^{k}=f^{k}\left(T_{\mathrm{x}}(\bm{x}_{\mathrm{t}});\bm{\theta}\right)$.
We define the loss function of geometric augmentation consistency (GAC)
$\mathcal{L}_{\mathrm{gac}}$ between $\bm{p}_{\mathrm{t,aug}}^{k}$ and
$T_{\mathrm{y}}^{k}(\bm{p}_{\mathrm{t}})$ as

$\displaystyle\mathcal{L}_{\mathrm{gac}}\left(\bm{\theta},{X}_{t},\mathcal{T}\right)=\mathbb{E}_{\bm{x}_{\mathrm{t}},(T_{\mathrm{x}},T_{\mathrm{y}}^{\mathrm{p}},T_{\mathrm{y}}^{\mathrm{m}})}\left[\sum_{k\in\\{\mathrm{p},\mathrm{m}\\}}\tilde{\lambda}^{k}\mathcal{\tilde{L}}^{k}\left(\bm{p}_{\mathrm{t,aug}}^{k},T_{\mathrm{y}}^{k}(\bm{p}_{\mathrm{t}}^{k})\right)\right].$
(2)

To correct the augmented prediction $\bm{p}_{\mathrm{t,aug}}^{k}$ by
$T_{\mathrm{y}}^{k}(\bm{p}_{\mathrm{t}})$, we stop the gradient update for
$\bm{p}_{\mathrm{t}}^{k}$, which can be viewed as the supervision to
$\bm{p}_{\mathrm{t,aug}}^{k}$. We use the smooth L1 loss (see Equation 1) as
$\mathcal{\tilde{L}}^{\mathrm{p}}$ and a mean squared error as
$\mathcal{\tilde{L}}^{\mathrm{m}}$. We introduce $\tilde{\lambda}^{k}$ as a
hyperparameter to control the balance of the two tasks. The augmentation set
$\mathcal{T}$ contains the geometric augmentation and photometric
augmentation, such as color jitter and blurring. We set
$T_{\mathrm{y}}(\cdot)$ to align geometric information to the augmented input
$T_{\mathrm{x}}(\bm{x}_{\mathrm{t}})$. For example, we apply rotation
$T_{\mathrm{y}}(\cdot)$ to the outputs $\bm{p}_{\mathrm{t}}^{k}$ with the same
degree of rotation $T_{\mathrm{x}}(\cdot)$ to the input $\bm{x}_{\mathrm{t}}$.

Figure 2: Method overview. Left: Student training with confidence-aware
geometric augmentation consistency. The student learns from the consistency
between its prediction and the two teachers’ predictions. The training is
weighted by the target confidence computed by the divergence of both teachers.
Right: Teacher training with knowledge distillation. Each teacher
independently learns to match the student’s predictions. The task index $k$ is
omitted for simplicity.

###  3.2 Confidence Estimation by Two Separate Networks

Since the target predictions are not always reliable, we aim to incorporate
the estimated confidence weight for each target instance into the consistency
training. In Equation 2, the generated outputs $\bm{p}_{\mathrm{t}}^{k}$ that
is the supervision to $\bm{p}_{\mathrm{t,aug}}^{k}$ may be unstable and noisy
due to the domain gap between source and target domains. Due to that, the
network trained with the consistency readily overfits to the incorrect
supervision $\bm{p}_{\mathrm{t}}^{k}$, which is known as confirmation bias [2,
66]. To reduce the bias, it is crucial to assign a low importance (confidence)
weight to the consistency training with the incorrect supervision. This
enables the network to learn primarily from reliable supervision while
avoiding being biased to such erroneous predictions. In classification tasks,
predicted class probability can serve as the confidence, while these measures
are not trivially defined and available in regression tasks. To estimate the
confidence of keypoint predictions, Yang  _et al_. [75] measure the confidence
of 3D hand keypoints by the distance to the fitted 3D hand template, but the
hand template fitting is an ill-posed problem for 2D hands and is not
applicable to hand segmentation. Dropout [22, 7, 8] is a generic way of
estimating uncertainty (confidence), calculated by the variance of multiple
stochastic forwards. However, the estimated confidence is biased to the
current state of the training network because the training and confidence
estimation are done by a single network. When the training network works
poorly, the confidence estimation becomes readily unreliable.

To perform reliable confidence estimation for both tasks, we propose a
confidence measure by computing the divergence of two predictions.
Specifically, we introduce two networks (_a.k.a_., teachers) for the
confidence estimation and the estimated confidence is used to train another
network (_a.k.a_., student) for the consistency training. The architecture of
the teachers is identical, yet they have different learning parameters. We
observe that when the divergence of the two predictions from the teachers for
a target instance is high, the predictions of both networks become unstable.
In contrast, a lower divergence indicates that the two teacher networks
predict stably and agree on their predictions. Thus, we use the divergence for
representing the target confidence. Given the teachers
$\bm{\theta}^{\mathrm{tch1}},\bm{\theta}^{\mathrm{tch2}}$, we define a
disagreement measure $\ell_{\mathrm{disagree}}$ to compute the divergence as

$\displaystyle\ell_{\mathrm{disagree}}\left(\bm{\theta}^{\mathrm{tch1}},\bm{\theta}^{\mathrm{tch2}},\bm{x}_{\mathrm{t}}\right)=\sum_{k\in\\{\mathrm{p},\mathrm{m}\\}}\tilde{\lambda}^{k}\mathcal{\tilde{L}}^{k}(\bm{p}_{\mathrm{t1}}^{k},\bm{p}_{\mathrm{t2}}^{k}),$
(3)

where
$\bm{p}_{\mathrm{t1}}^{k}=f^{k}(\bm{x}_{\mathrm{t}};\bm{\theta}^{\mathrm{tch1}})$
and
$\bm{p}_{\mathrm{t2}}^{k}=f^{k}(\bm{x}_{\mathrm{t}};\bm{\theta}^{\mathrm{tch2}})$.

As a proof of concept, we visualize the correlation between the disagreement
measure and a validation score averaged over evaluation metrics of the two
tasks (PCK and IoU) in Fig. 3. We compute the score between the ensemble of
the teachers’ predictions
$\bm{p}_{\mathrm{ens}}^{k}=\left(\bm{p}_{\mathrm{t1}}^{k}+\bm{p}_{\mathrm{t2}}^{k}\right)/2$
and its ground truth in the validation set on HO3D [30]. The instances with a
small disagreement measure tend to have high validation scores. In contrast,
the instances with a high disagreement measure entail false predictions,
_e.g_., detecting the hand-held object as a hand joint and hand class. When
the disagreement measure was high at the bottom of Fig. 3, we found that both
predictions were particularly unstable on the keypoints of the ring finger
(yellow). This study shows that the disagreement measure can represent the
correctness of the target predictions.

Figure 3: The correlation between a disagreement measure and task scores.
Target instances with smaller disagreement values between the two teacher
networks tend to have higher task scores.

With the disagreement measure $\ell_{\mathrm{disagree}}$, we define a
confidence weight $w_{\mathrm{t}}\in[0,1]$ for assigning importance to the
consistency training. We compute the weight $w_{\mathrm{t}}$ as
$w_{\mathrm{t}}=2\left(1-\mathrm{sigm}\left(\lambda_{\mathrm{d}}\ell_{\mathrm{disagree}}\left(\bm{\theta}^{\mathrm{tch1}},\bm{\theta}^{\mathrm{tch2}},\bm{x}_{\mathrm{t}}\right)\right)\right)$
where $w_{\mathrm{t}}$ is a normalized disagreement measure with sign
inversion, $\mathrm{sigm}(\cdot)$ denotes a sigmoid function, and
$\lambda_{\mathrm{d}}$ controls the scale of the measure. With the confidence
weight $w_{\mathrm{t}}$, we enforce the consistency training between the
student’s prediction on the augmented target images
$\bm{p}_{\mathrm{s,aug}}^{k}$ and the ensemble of the two teachers’
predictions $\bm{p}_{\mathrm{ens}}^{k}$. Our proposed loss function of
confidence-aware geometric augmentation consistency (C-GAC)
$\mathcal{L}_{\mathrm{cgac}}$ for the student $\bm{\theta}^{\mathrm{stu}}$ is
formulated as

$\displaystyle\mathcal{L}_{\mathrm{cgac}}\left(\bm{\theta}^{\mathrm{stu}},\bm{\theta}^{\mathrm{tch1}},\bm{\theta}^{\mathrm{tch2}},{X}_{\mathrm{t}},\mathcal{T}\right)$
$\displaystyle=\mathbb{E}_{\bm{x}_{\mathrm{t}},(T_{\mathrm{x}},T_{\mathrm{y}}^{\mathrm{p}},T_{\mathrm{y}}^{\mathrm{m}})}\left[w_{\mathrm{t}}\sum_{k\in\\{\mathrm{p},\mathrm{m}\\}}\tilde{\lambda}^{k}\mathcal{\tilde{L}}^{k}\left(\bm{p}_{\mathrm{s,aug}}^{k},T_{\mathrm{y}}^{k}(\bm{p}_{\mathrm{ens}}^{k})\right)\right],$
(4)

where
$\bm{p}_{\mathrm{s,aug}}^{k}=f^{k}\left(T_{\mathrm{x}}(\bm{x}_{\mathrm{t}});\bm{\theta}^{\mathrm{stu}}\right)$.
Following [66, 55], we design the student prediction
$\bm{p}_{\mathrm{s,aug}}^{k}$ to be supervised by the teachers. We generate
the teachers’ prediction by doing ensemble $\bm{p}_{\mathrm{ens}}^{k}$, which
is better than the prediction of either teacher.

###  3.3 Teacher-Student Update by Knowledge Distillation

In addition to the student’s training, we formulate an update rule for the two
teacher networks by using knowledge distillation. Since
$\ell_{\mathrm{disagree}}$ would not work if the two teachers had the same
output values, we aim to learn two teachers that have different
representations yet keep high task performance as co-training works [4, 58,
15, 60]. In a prior teacher-student update, Tarvainen  _et al_. [66] found
that the teacher’s update by an exponential moving average (EMA), which
averages the student’s weights iteratively, makes the teacher’s learning more
slowly and mitigates the confirmation bias as discussed in Section 3.2. While
this EMA-based teacher-student framework is widely used in various domain
adaptation tasks [19, 9, 39, 74, 26], naively applying the EMA rule to the two
teachers would produce exactly the same weights for both networks.

To prevent this, we propose independent knowledge distillation for building
two different teachers. The distillation matches the teacher-student
predictions in the output level. To let both networks have different
parameters, we train the teachers from different mini-batches and using
stochastic augmentation as

$\displaystyle\mathcal{L}_{\mathrm{distill}}\left(\bm{\theta},\bm{\theta}^{\mathrm{stu}},X_{\mathrm{t}},\mathcal{T}\right)=\mathbb{E}_{\bm{x}_{\mathrm{t}},T_{\mathrm{x}}}\left[\sum_{k\in\\{\mathrm{p},\mathrm{m}\\}}\tilde{\lambda}^{k}\mathcal{\tilde{L}}^{k}(\bm{p}_{\mathrm{t,aug}}^{k},\bm{p}_{\mathrm{s,aug}}^{k})\right],$
(5)

where
$~{}\bm{\theta}\in\\{\bm{\theta}^{\mathrm{tch1}},\bm{\theta}^{\mathrm{tch2}}\\}$,
$\bm{p}_{\mathrm{t,aug}}^{k}=f^{k}\left(T_{\mathrm{x}}(\bm{x}_{\mathrm{t}});\bm{\theta}\right)$,
and
$\bm{p}_{\mathrm{s,aug}}^{k}=f^{k}(T_{\mathrm{x}}\left(\bm{x}_{\mathrm{t}}\right);\bm{\theta}^{\mathrm{stu}})$.
The distillation loss $\mathcal{L}_{\mathrm{distill}}$ is used for updating
the teacher networks only. This helps the teachers to adapt to the target
domain more carefully than the student and avoid falling into exactly the same
predictions on a target instance.

###  3.4 Overall Objectives

Overall, the objective of the student’s training consists of the supervised
loss (Equation 1) from the source domain and the self-training with
confidence-aware geometric augmentation consistency (Equation 4) in the target
domain as

$\displaystyle\min_{\bm{\theta}^{\mathrm{stu}}}\mathcal{L}_{\mathrm{task}}\left(\bm{\theta}^{\mathrm{stu}},X_{\mathrm{s}},Y_{\mathrm{s}}\right)+\mathcal{L}_{\mathrm{cgac}}\left(\bm{\theta}^{\mathrm{stu}},\bm{\theta}^{\mathrm{tch1}},\bm{\theta}^{\mathrm{tch2}},{X}_{t},\mathcal{T}\right).$
(6)

The two teachers are asynchronously trained with the distillation loss
(Equation 5) in the target domain, which is formulated as

$\displaystyle\min_{\bm{\theta}}\mathcal{L}_{\mathrm{distill}}\left(\bm{\theta},\bm{\theta}^{\mathrm{stu}},X_{\mathrm{t}},\mathcal{T}\right),$
(7)

where
$\bm{\theta}\in\\{\bm{\theta}^{\mathrm{tch1}},\bm{\theta}^{\mathrm{tch2}}\\}$.
Since the teachers are updated carefully and can perform better than the
student, we use the ensemble of the two teachers’ predictions for a final
output in inference.

##  4 Experiments

In this section, we first present our experimental datasets and implementation
details and then provide quantitative and qualitative results along with the
ablation studies. We analyze our proposed method by comparing it with several
existing methods in three different domain adaptation settings. We also show
qualitative results by applying our method to in-the-wild egocentric videos.

###  4.1 Experiment Setup

Datasets. We experimented with several hand datasets including a variety of
hand-object interactions, the annotation of 2D hand keypoints, and hand masks
as follows. We adopted DexYCB [13] dataset as our source dataset since it
contains a large amount of training images, their corresponding labels, and
natural hand-object interactions. We chose to use the following datasets as
our target datasets: HO3D [30] captured in different environments with the
same YCB objects [10] as the source dataset, HanCo [80] captured in a multi-
camera studio and generated with synthesized backgrounds, and FPHA [24]
captured by a first-person view. We also used Ego4D [29] to verify the
effectiveness of our method in real-world scenarios. During training, we used
cropped images of the hand regions from the original images as input.

Implementation details. Our teacher-student networks share an identical
network architecture, which consists of a unified feature extractor and task-
specific branches for hand keypoint regression and hand segmentation. For
training our student network, we used the Adam optimizer [37] with a learning
rate of $10^{-5}$, while the learning rate of the teacher networks was set to
$5\times 10^{-6}$. We set the hyperparameters
($\lambda^{\mathrm{p}}(=\tilde{\lambda}^{\mathrm{p}})$,
$\lambda^{\mathrm{m}}$, $\tilde{\lambda}^{\mathrm{m}}$,
$\lambda_{\mathrm{d}}$) to $(10^{7},10^{2},5,0.5)$. Since both task-specific
branches have different training speeds, we began our adaptation with the
backbone and keypoint regression branch. We then trained all sub-networks,
including the hand segmentation branch. We report the percentage of correct
keypoints (PCK) and the mean joint position error (MPE) for hand keypoint
regression, and the intersection over union (IoU) for hand segmentation.

Baseline methods. We compared quantitative performance with the following
methods. Source only denotes the network trained on the source dataset without
any adaptation. To compare with another adaptation approach with adversarial
training, we trained DANN [23] that aligns marginal feature distributions
between domains, and RegDA [34] with an adversarial regressor that optimizes
domain disparity. In addition, we implemented several self-training adaptation
methods by replacing pseudo-labeling with the consistency training. GAC is a
simple baseline with the consistency training updated by Equation 2. GAC + UMA
[7] is a GAC method with confidence estimation by Dropout [22]. GAC + CPL [54]
is a GAC method with confident instance selection using the agreement with
another network. GAC + MT [66] is a GAC method with the single-teacher-single-
student architecture using EMA for the teacher update. Target only indicates
the network trained on the target dataset with labels, which shows an
empirical performance upper bound.

Our method. We denote our full method as C-GAC introduced in Section 3.4. As
an ablation study, we present a variant of the proposed method as GAC-Distill
with a teacher-student pair, which is updated by the consistency training
(Equation 2) and the distillation loss (Equation 5). GAC-Distill is different
from GAC + MT only in the way of the teacher update.

###  4.2 Quantitative Results

We show the results of three adaptation settings: DexYCB → {HO3D, HanCo, FPHA}
in Tables 1 and 4.2. We then provide detailed comparisons of our method.

Table 1: DexYCB [13] → HO3D [30]. We report PCK (%) and MPE (px) for hand
keypoint regression and IoU (%) for hand segmentation. Each score format of
$val~{}/~{}test$ indicates the validation and test scores. Red and blue
letters indicate the best and second best values.

Method | 2D Pose | Seg | 2D Pose + Seg
---|---|---|---
PCK ↑ (%) | MPE ↓ (px) | IoU ↑ (%) | Avg. ↑ (%)
Source only | $42.8/33.5$ | $15.39/19.32$ | $57.9/49.1$ | $50.3/41.3$
[2pt/2pt] DANN [23] | $49.0/46.8$ | $12.39/13.39$ | $52.8/54.7$ | $50.9/50.8$
RegDA [34] | $48.8/48.2$ | $12.50/12.64$ | $55.7/55.3$ | $52.2/51.7$
GAC | $47.6/47.4$ | $12.47/12.54$ | $58.0/56.9$ | $52.8/52.2$
GAC + UMA [7] | $47.1/45.3$ | $12.97/13.51$ | $58.0/55.0$ | $52.5/50.2$
GAC + CPL [54] | $48.1/48.1$ | $12.74/12.61$ | $57.2/55.6$ | $52.7/51.8$
GAC + MT [66] | $45.5/44.4$ | $13.65/14.05$ | $54.8/52.3$ | $50.2/48.3$
[2pt/2pt] GAC-Distill (Ours) | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}49.9}/{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}50.4}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}11.98}/{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}11.51}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}60.7}/{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}60.6}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}55.3}/{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}55.5}$
C-GAC (Ours-Full) | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}50.3}/{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}51.1}$ | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}11.89}/{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}11.22}$ | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}60.9}/{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}60.3}$ | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}55.6}/{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}55.7}$
Target only | $55.1/58.6$ | $11.00/9.29$ | $68.2/66.1$ | $61.7/62.4$

DexYCB → HO3D. Table 1 shows the results of the adaptation from DexYCB to HO3D
where the grasping objects are overlapped. The baseline of the consistency
training (GAC) was effective in learning target images in both tasks. Our
proposed method (C-GAC) improved by $5.3/14.4$ in the average task score from
the source-only performance. The method also outperformed all comparison
methods and achieved close performance to the upper bound.

Table 2: DexYCB [13] → {HanCo [80], FPHA [24]}. We report PCK (%) and MPE (px)
for hand keypoint regression and IoU (%) for hand segmentation. We show the
validation and test results on HanCo and the validation results on FPHA. Red
and blue letters indicate the best and second best values.

Method | DexYCB → HanCo
---|---
2-5 | 2D Pose | Seg |
| PCK ↑ (%) | MPE ↓ (px) | IoU ↑ (%) | Avg. ↑ (%)
Source only | $26.0/27.3$ | $21.82/21.48$ | $41.8/41.4$ | $33.9/34.3$
[2pt/2pt] DANN [23] | $32.3/33.0$ | $19.99/19.82$ | $56.3/56.9$ | $44.3/45.0$
RegDA [34] | $33.0/33.6$ | $19.51/19.44$ | $57.8/58.4$ | $45.4/46.0$
GAC | $36.6/37.1$ | $16.63/16.59$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}58.1}/{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}58.8}$ | $47.4/47.9$
GAC + UMA [7] | $35.1/35.6$ | $17.51/17.48$ | $57.1/57.7$ | $46.1/46.6$
GAC + CPL [54] | $32.7/33.5$ | $19.85/19.62$ | $55.8/56.4$ | $44.2/45.0$
GAC + MT [66] | $33.2/33.8$ | $18.93/18.83$ | $54.3/55.1$ | $43.8/44.4$
[2pt/2pt] GAC-Distill (Ours) | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}38.8}/{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}39.5}$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}16.06}/{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}15.97}$ | $57.5/57.7$ | ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}48.1}/{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}48.6}$
C-GAC (Ours-Full) | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}39.2}/{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}39.9}$ | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}15.83}/{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}15.74}$ | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}58.2}/{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}58.6}$ | ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}48.7}/{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}49.2}$
Target only | $76.8/77.3$ | $4.91/4.80$ | $75.9/76.1$ | $76.3/76.7$

DexYCB → FPHA
---
1-4 2D Pose | Seg |
PCK | MPE | IoU | Avg.
14.0 | 31.32 | 24.8 | 19.4
[2pt/2pt] 24.4 | 25.79 | 28.4 | 26.4
23.7 | 24.27 | 41.7 | 32.7
37.2 | 17.02 | 33.3 | 35.3
36.8 | 17.29 | 39.2 | 38.0
25.7 | 24.99 | 32.7 | 29.2
31.3 | 20.81 | 38.4 | 34.9
[2pt/2pt] 36.8 | 15.99 | 35.5 | 36.1
37.2 | 15.36 | 37.7 | 37.4
63.3 | 8.11 | - | -

DexYCB → HanCo. Table 4.2 shows the results of the adaptation from DexYCB to
HanCo across laboratory setups. The source-only network less generalized to
the target domain because the HanCo has diverse backgrounds, while GAC
succeeded in adapting up to $47.4/47.9$ in the average score. Our method C-GAC
showed further improved results in hand keypoint regression.

DexYCB → FPHA. Table 4.2 also shows the results of the adaptation from DexYCB
to FPHA, which captures egocentric users’ activities. Since hand markers and
in-the-wild target environments cause large appearance gaps, the source-only
performance performed the most poorly among the three adaption settings. In
this challenging setting, RegDA and GAC + UMA performed well for hand
segmentation, while their performance on hand keypoint regression was inferior
to the GAC baseline. Our method C-GAC further improved than the GAC method in
the MPE and IoU metrics and exhibited stability in adaptation training among
the comparison methods.

Comparison to different confidence estimation methods. We compare the results
with existing confidence estimation methods. GAC + UMA and GAC + CPL estimate
the confidence of target predictions by computing the variance of multiple
stochastic forwards and the task scores between a training network and an
auxiliary network, respectively. GAC + UMA performed effectively on DexYCB →
FPHA, whereas the performance gain was thin in the other settings compared to
GAC. GAC + CPL worked well for keypoint regression on DexYCB → HO3D, but it
cannot address the other settings with a large domain gap well since the
prediction of the auxiliary network became unstable. Although these prior
methods had different disadvantages depending on the settings, our method
C-GAC using the divergence of the two teachers for confidence estimation
performed stably in the three settings.

Comparison to standard teacher-student update. We compare our teacher update
with the update with an exponential moving average (EMA) [66]. The EMA-based
update (GAC-MT) degraded the performance from the source only in hand
segmentation in Table 1. This suggests that the EMA update can be sensitive to
the task. In contrast, our method GAC-Distill matching the teacher-student
predictions in the output level did not produce such performance degeneration
and worked more stably.

Comparison to adversarial adaptation methods. We compared our method with
another major adaptation approach with adversarial training. In Tables 1 and
4.2, the performance of DANN and RegDA was mostly worse than the consistency-
based baseline GAC. We found that instead of matching features between both
domains [23, 34], directly learning target images by the consistency training
was critical in the adaptation of our tasks.

Comparison to an off-the-shelf hand pose estimator. We tested the
generalization ability of an open-source library for pose estimation: OpenPose
[32]. It resulted in $15.75/12.72$, $18.31/18.42$, and $29.02$ in the MPE on
HO3D, HanCo, and FPHA, respectively. Since it is built on multiple source
datasets [35, 1, 45], the baseline showed higher generalization than the
source-only network. However, the performance did not exceed our proposed
method in the MPE. This shows that generalizing hand keypoint regression to
other datasets is still challenging, and our adaptation framework supports
improving target performance.

Figure 4: Qualitative results. We show qualitative examples of the source-only
network (top), the Ours-Full method (middle), and ground truth (bottom) on
HO3D [30], HanCo [80], FPHA [24], and Ego4D [29] without ground truth.

###  4.3 Qualitative Results

We show the qualitative results of hand keypoint regression and hand
segmentation in Fig. 4. When hands are occluded in HO3D and FPHA or the
backgrounds are diverse in HanCo, the keypoint prediction of the source only
(top) represented infeasible hand poses and hand segmentation was too noisy or
missing. However, our method C-GAC (middle) corrected the hand keypoint errors
and improved to localize hand regions. Hand segmentation in FPHA was still
noisy because visible white markers obstructed hand appearance. We can also
see distinct improvements in the Ego4D dataset. We provide additional
qualitative analysis in adaptation to the Ego4D beyond countries, cultures,
ages, indoors/outdoors, and performing tasks with hands in our supplementary
material.

###  4.4 Ablation Studies

Effect of confidence estimation. To confirm the effect of our proposed
confidence estimation, we compare our full method C-GAC and our ablation model
GAC-Distill without the confidence weighting. In Tables 1 and 4.2, while GAC-
Distill mostly surpassed the comparison methods in most cases, C-GAC showed
further performance gain in all three adaptation settings.

Multi-task _vs_. single-task adaptation. We studied the effect of our multi-
task adaptation compared with single-task adaptation on DexYCB → HO3D. The
single-task adaptation results are $50.1/51.0$ in the PCK and $58.2/57.7$ in
the IoU. Compared to Table 1, our method in the multi-task setting improved by
$2.7/2.6$ over the single-task adaption in hand segmentation while it provided
marginal gain in hand keypoint regression. This shows that the adaptation of
hand keypoint regression helps to localize hand regions in the target domain.

Figure 5: Visualization of bone length distributions. We show the
distributions of the bone length between hand joints, namely, Wrist,
metacarpophalangeal (MCP), proximal interphalangeal (PIP), distal
interphalangeal (DIP), and fingertip (TIP). Using kernel density estimation,
we plotted the density of the bone length for the predictions of the source
only, the Ours-Full method, and ground truth on test data of HO3D [30].

Bone length distributions. To study our adaptation results in each hand joint,
we show the distributions of bone length between hand joints in Fig. 5. In
Wrist-MCP, PIP-DIP, and DIP-TIP, the distribution of the source-only
prediction on target images (blue) was far from that of the target ground
truth (green), whereas our method (orange) improved to approximate the target
distribution (green). In MCP-PIP, we could not observe such clear differences
because the source-only model already represented the target distribution
well. This indicates that our method improved to learn hand structure near the
palm and fingertips.

##  5 Conclusion

In this work, we tackled the problem of joint domain adaptation of hand
keypoint regression and hand segmentation. Our proposed method consists of the
self-training with geometric augmentation consistency, confidence weighting by
the two teacher networks, and the teacher-student update by knowledge
distillation. The consistency training under geometric augmentation served to
learn the unlabeled target images for both tasks. The divergence of the
predictions from two teacher networks could represent the confidence of each
target instance, which enables the student network to learn from reliable
target predictions. The distillation-based teacher-student update guided the
teachers to learn from the student carefully and mitigated over-fitting to the
noisy predictions. Our method delivered state-of-the-art performance on the
three adaptation setups. It also showed improved qualitative results in the
real-world egocentric videos.

### Acknowledgments

This work was supported by JST ACT-X Grant Number JPMJAX2007, JSPS Research
Fellowships for Young Scientists, JST AIP Acceleration Research Grant Number
JPMJCR20U1, and JSPS KAKENHI Grant Number JP20H04205, Japan. This work was
also supported in part by a hardware donation from Yu Darvish.

##  Appendix 0.A Appendix

###  0.A.1 Dataset Details

  * •

DexYCB [13] contains 582K RGB-D frames captured by 10 subjects interacting 20
different YCB objects [10] from eight different views. In our experiment, we
split the dataset by the subject IDs to create train, validation, and test
sets with 212K, 71K, and 80K images, respectively.

  * •

HO3D [30] contains 103K RGB-D frames captured by 10 subjects interacting 10
different YCB objects [10] from a single third-person view. In our experiment,
we randomly split the video sequences to train, validation, and test sets with
51K, 12K, and 8K images, respectively.

  * •

HanCo [80] is an extended FreiHAND [82] dataset captured in a multi-view
camera setup with eight cameras, which consists of 518K, 106K, and 104K RGB
images for training, validation, and testing, respectively. The backgrounds
are randomly synthesized using diverse scenery images.

  * •

FPHA [24] is an egocentric video dataset capturing users’ actions in daily
indoor environments from a first-person perspective, and their hand poses are
tracked by hand magnetic sensors. It contains 69K training images and 16K
validation images. Due to lacking hand mask annotation, we annotated 50 hand
masks in the validation set.

  * •

Ego4D [29] is a collection of daily-life egocentric activity videos lasting
over 3,000 hours and gathered across the world. Due to the lack of annotation
for the two tasks, we show qualitative examples in our experiments. We treated
each video sequence as the domain to adapt.

###  0.A.2 Preprocessing and Augmentation

For creating an input of a training network, we assumed to have hand center
positions, cropped hand regions of the original images, and resized them to
$128\times 128$ pixels. To extract hand centers and regions in Ego4D videos
without ground truth, we used an off-the-shelf hand detector [62]. Inspired by
[42, 18, 39], we used two different augmentation sets: strong augmentation for
the student’s learning (Equation 4) and weak augmentation for the teacher’s
learning (Equation 5). We used horizontal flip, rotation, transition, gaussian
blur, brightness/contrast jitter, hue/saturation/input value jitter, and
cutout as the strong augmentation. In contrast, we adopted horizontal flip,
rotation, transition, and gaussian blur as the weak augmentation.

###  0.A.3 Network Architecture and Evaluation

For the design of our multi-task baseline model, we employed an hourglass
network [51] as the backbone and the keypoint regression branch. We added
1d-convolution to its intermediate features to predict hand pixel labels.
Following hand keypoint regression methods [71, 51, 25], we optimized 2D joint
heatmaps for each 2D ground-truth joint location instead of joint coordinates.

We also provide the details of out evaluation, namely, MPE, PCK, and IoU. MPE
(px) indicates the euclidean error per joint in the image coordinate. PCK (%)
represents the percentage of joints whose MPE is smaller than a given joint
error threshold, which is calculated by the area under the curve (AUC) over
the joint error range [0, 20 px]. IoU (%) measures the overlap over two masks.
We report the average score (Avg.) over PCK and IoU to evaluate multi-task
performance.

###  0.A.4 Qualitative Analysis

In Figs. 6, 7, 8, and 9, we show additional qualitative results of our
proposed method. As shown in Fig. 6, our method performed well when complex
hand-object interactions occur on HO3D and FPHA and when the backgrounds are
diverse on HanCo. In Fig. 7, we show qualitative comparison between GAC and
C-GAC (Ours-Full). Our full method particularly improved keypoint regression
compared to the simple consistency baseline, GAC. Our method (right) corrected
the keypoint prediction of the GAC (left), which contains incorrect
predictions on the position of the thumb (red).

Our method also demonstrated improved performance on Ego4D, an egocentric
video dataset collected across various countries, cultures, ages,
indoors/outdoors, and performing tasks with hands. In particular, we observed
that our method successfully adapted to various imaging conditions, such as
outdoor environments (rows 1 and 2 in Fig. 8), extremely dark environments
(rows 3 to 6 in Fig. 8), the second person’s hands in social interactions (row
7 in Fig. 8), _e.g_., playing board games, and indoor environments (Fig. 9),
_e.g_., where people perform cooking, cleaning, fitness, DIY, painting, and
crafting.

Figure 6: Additional qualitative results on HO3D [30], HanCo [80], and FPHA
[24]. Figure 7:  Comparison between GAC and C-GAC (Ours-Full). Left: GAC,
Right: C-GAC (Ours-Full).  Figure 8: Additional qualitative results on Ego4D
[29]. Figure 9: Additional qualitative results on Ego4D [29].

## References

  * [1] M. Andriluka, L. Pishchulin, P. V. Gehler, and B. Schiele.  2D human pose estimation: New benchmark and state of the art analysis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3686–3693, 2014.
  * [2] E. Arazo, D. Ortego, P. Albert, N. E. O’Connor, and K. McGuinness.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In IEEE International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2020.
  * [3] G. Benitez-Garcia, L. Prudente-Tixteco, L. C. Castro-Madrid, R. Toscano-Medina, J. Olivares-Mercado, G. Sanchez-Perez, and L. J. G. Villalba.  Improving real-time hand gesture recognition with semantic segmentation.  Sensors, 21(2), 2021.
  * [4] A. Blum and T. M. Mitchell.  Combining labeled and unlabeled data with co-training.  In Proceedings of theACM Annual Conference on Computational Learning Theory (COLT), pages 92–100, 1998.
  * [5] A. Boukhayma, R. A. Bem, and P. H. S. Torr.  3d hand shape and pose from images in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10843–10852, 2019.
  * [6] S. Brahmbhatt, C. Tang, C. D. Twigg, C. C. Kemp, and J. Hays.  ContactPose: A dataset of grasps with object contact and hand pose.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 361–378, 2020.
  * [7] M. Cai, E. Lu, and Y. Sato.  Generalizing hand segmentation in egocentric videos with uncertainty-guided model adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14380–14389, 2020.
  * [8] M. Cai, M. Luo, X. Zhong, and H. Chen.  Uncertainty-aware model adaptation for unsupervised cross-domain object detection.  CoRR, abs/2108.12612, 2021.
  * [9] Q. Cai, Y. Pan, C.-W. Ngo, X. Tian, L. Duan, and T. Yao.  Exploring object relation in mean teacher for cross-domain detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11457–11466, 2019.
  * [10] B. Çalli, A. Walsman, A. Singh, S. S. Srinivasa, P. Abbeel, and A. M. Dollar.  Benchmarking in manipulation research: Using the yale-cmu-berkeley object and model set.  IEEE Robotics Automation Magazine, 22(3):36–52, 2015.
  * [11] J. Cao, H. Tang, H. Fang, X. Shen, Y.-W. Tai, and C. Lu.  Cross-domain adaptation for animal pose estimation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 9497–9506, 2019.
  * [12] Z. Cao, I. Radosavovic, A. Kanazawa, and J. Malik.  Reconstructing hand-object interactions in the wild.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 12417–12426, 2021.
  * [13] Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, J. Kautz, and D. Fox.  DexYCB: A benchmark for capturing hand grasping of objects.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9044–9053, 2021.
  * [14] C.-H. Chen, A. Tyagi, A. Agrawal, D. Drover, R. MV, S. Stojanov, and J. M. Rehg.  Unsupervised 3d pose estimation with geometric self-supervision.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5714–5724, 2019.
  * [15] M. Chen, K. Q. Weinberger, and J. Blitzer.  Co-training for domain adaptation.  In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 2456–2464, 2011.
  * [16] X. Chen, G. Wang, C. Zhang, T.-K. Kim, and X. Ji.  Shpr-net: Deep semantic hand pose regression from point clouds.  IEEE Access, 6:43425–43439, 2018.
  * [17] D. Damen, H. Doughty, G. M. Farinella, A. Furnari, J. Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray.  Rescaling egocentric vision.  International Journal of Computer Vision (IJCV), early access, 2021\.
  * [18] J. Deng, W. Li, Y. Chen, and L. Duan.  Unbiased mean teacher for cross-domain object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4091–4101, 2021.
  * [19] G. French, M. Mackiewicz, and M. H. Fisher.  Self-ensembling for visual domain adaptation.  In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  * [20] H. Fu, M. Gong, C. Wang, K. Batmanghelich, K. Zhang, and D. Tao.  Geometry-consistent generative adversarial networks for one-sided unsupervised domain mapping.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2427–2436, 2019.
  * [21] Q. Fu, X. Liu, and K. M. Kitani.  Sequential decision-making for active object detection from hand.  CoRR, abs/2110.11524, 2021.
  * [22] Y. Gal and Z. Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In Proceedings of the International Conference on Machine Learning (ICML), pages 1050––1059, 2016.
  * [23] Y. Ganin and V. Lempitsky.  Unsupervised domain adaptation by backpropagation.  In Proceedings of the International Conference on Machine Learning (ICML), pages 1180–1189, 2015.
  * [24] G. Garcia-Hernando, S. Yuan, S. Baek, and T.-K. Kim.  First-person hand action benchmark with RGB-D videos and 3D hand pose annotations.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 409–419, 2018.
  * [25] L. Ge, Z. Ren, Y. Li, Z. Xue, Y. Wang, J. Cai, and J. Yuan.  3D hand shape and pose estimation from a single RGB image.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10833–10842, 2019.
  * [26] Y. Ge, D. Chen, and H. Li.  Mutual mean-teaching: Pseudo label refinery for unsupervised domain adaptation on person re-identification.  In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
  * [27] O. Glauser, S. Wu, D. Panozzo, O. Hilliges, and O. Sorkine-Hornung.  Interactive hand pose estimation using a stretch-sensing soft glove.  ACM Transactions on Graphics, 38(4):41:1–41:15, 2019.
  * [28] D. Goudie and A. Galata.  3D hand-object pose estimation from depth with convolutional neural networks.  In Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition (FG), pages 406–413, 2017.
  * [29] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M.g Xu, E. Zhongcong Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, C. Fuegen, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. Soo Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, Lo Torresani, M.i Yan, and J. Malik.  Ego4D: Around the world in 3, 000 hours of egocentric video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 18995–19012, 2022.
  * [30] S. Hampali, M. Rad, M. Oberweger, and V. Lepetit.  Honnotate: A method for 3D annotation of hand and object poses.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3196–3206, 2020.
  * [31] Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, I. Laptev, and C. Schmid.  Learning joint reconstruction of hands and manipulated objects.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11807–11816, 2019.
  * [32] G. Hidalgo, Z. Cao, T. Simon, S.-E. Wei, Y. Raaj, H. Joo, and Y. Sheikh.  OpenPose.  https://github.com/CMU-Perceptual-Computing-Lab/openpose.
  * [33] W. Huang, P. Ren, J. Wang, Q. Qi, and H. Sun.  AWR: Adaptive weighting regression for 3D hand pose estimation.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 11061–11068, 2020.
  * [34] J. Jiang, Y. Ji, X. Wang, Y. Liu, J. Wang, and M. Long.  Regressive domain adaptation for unsupervised keypoint detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6780–6789, 2021.
  * [35] H. Joo, H. Liu, L. Tan, L. Gui, B. C. Nabbe, I. A. Matthews, T. Kanade, S. Nobuhara, and Y. Sheikh.  Panoptic studio: A massively multiview system for social motion capture.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3334–3342, 2015.
  * [36] S. Kim, H. -G. Chi, X. Hu, A. Vegesana, and K. Ramani.  First-person view hand segmentation of multi-modal hand activity video dataset.  In Proceedings of the British Machine Vision Conference (BMVC), 2020.
  * [37] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
  * [38] K. Lee, A. Shrivastava, and H. Kacorri.  Hand-priming in object localization for assistive egocentric vision.  In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 3422–3432, 2020.
  * [39] Y.-J. Li, X. Dai, C.-Y. Ma, Y.-C. Liu, K. Chen, B. Wu, Z. He, K. Kitani, and P. Vajda.  Cross-domain object detection via adaptive self-training.  CoRR, abs/2111.13216, 2021.
  * [40] H. Liang, J.G. Yuan, D. Thalmann, and N. Magnenat-Thalmann.  AR in hand: Egocentric palm pose tracking and gesture recognition for augmented reality applications.  In Proceedings of the ACM International Conference on Multimedia (MM), pages 743–744, 2015.
  * [41] J. Likitlersuang, E. R. Sumitro, T. Cao, R. J. Visée, S. Kalsi-Ryan, and J. Zariffa.  Egocentric video: A new tool for capturing hand use of individuals with spinal cord injury at home.  Journal of Neuroengineering and Rehabilitation (JNER), 16(1):83, 2019.
  * [42] Y.-C. Liu, C.-Y. Ma, Z. He, C.-W. Kuo, K. Chen, P. Zhang, B. Wu, Z. Kira, and P. Vajda.  Unbiased teacher for semi-supervised object detection.  In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [43] M. Long, H. Zhu, J. Wang, and M. I. Jorda.  Unsupervised domain adaptation with residual transfer networks.  In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 136–144, 2016.
  * [44] Y. Lu and W. W. Mayol-Cuevas.  Understanding egocentric hand-object interactions from hand pose estimation.  CoRR, abs/2109.14657, 2021.
  * [45] R. McKee, D. McKee, D. Alexander, and E. Paillat.  NZ sign language exercises.  Deaf Studies Department of Victoria University of Wellington, http://www.victoria.ac.nz/llc/llc_resources/nzsl.
  * [46] L. Melas-Kyriazi and A. K. Manrai.  Pixmatch: Unsupervised domain adaptation via pixelwise consistency training.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12435–12445, 2021.
  * [47] G. Moon, S.-I. Yu, H. Wen, T. Shiratori, and K. M. Lee.  InterHand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 548–564, 2020.
  * [48] F. Mueller, F. Bernard, O. Sotnychenko, D. Mehta, S. Sridhar, D. Casas, and C. Theobalt.  GANerated Hands for real-time 3D hand tracking from monocular RGB.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 49–59, 2018.
  * [49] F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, and C. Theobalt.  Real-time hand tracking under occlusion from an egocentric RGB-D sensor.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1163–1172, 2017.
  * [50] N. Neverova, C. Wolf, F. Nebout, and G. W. Taylor.  Hand pose estimation through semi-supervised and weakly-supervised learning.  Computer Vision and Image Understanding, 164:56–67, 2017.
  * [51] A. Newell, K. Yang, and J. Deng.  Stacked hourglass networks for human pose estimation.  In Proceedings of the European Conference on Computer Vision (ECCV), volume 9912, pages 483–499, 2016.
  * [52] T. Ohkawa, R. Furuta, and Y. Sato.  Efficient annotation and learning for 3d hand pose estimation: A survey.  CoRR, abs/2206.02257, 2022.
  * [53] T. Ohkawa, N. Inoue, H. Kataoka, and N. Inoue.  Augmented cyclic consistency regularization for unpaired image-to-image translation.  In Proceedings of the International Conference on Pattern Recognition (ICPR), pages 362–369, 2020.
  * [54] T. Ohkawa, T. Yagi, A. Hashimoto, Y. Ushiku, and Y. Sato.  Foreground-aware stylization and consensus pseudo-labeling for domain adaptation of first-person hand segmentation.  IEEE Access, 9:94644–94655, 2021.
  * [55] H. Pham, Z. Dai, Q. Xie, and Q. V. Le.  Meta pseudo labels.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11557–11568, 2021.
  * [56] V. Prabhu, S. Khare, D. Kartik, and J. Hoffman.  SENTRY: Selective entropy optimization via committee consistency for unsupervised domain adaptation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 8558–8567, 2021.
  * [57] C. Qian, X. Sun, Y. Wei, X. Tang, and J. Sun.  Realtime and robust hand tracking from depth.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1106–1113, 2014.
  * [58] S. Qiao, W. Shen, Z. Zhang, B. Wang, and A. L. Yuille.  Deep co-training for semi-supervised image recognition.  In Proceedings of the European Conference on Computer Vision (ECCV), volume 11219, pages 142–159, 2018.
  * [59] P. Ren, H. Sun, Q. Qi, J. Wang, and W. Huang.  SRN: stacked regression network for real-time 3D hand pose estimation.  In Proceedings of the British Machine Vision Conference (BMVC), 2019.
  * [60] K. Saito, Y. Ushiku, and T. Harada.  Asymmetric tri-training for unsupervised domain adaptation.  In Proceedings of the International Conference on Machine Learning (ICML), pages 2988–2997, 2017.
  * [61] N. Santavas, I. Kansizoglou, L. Bampis, E. G. Karakasis, and A. Gasteratos.  Attention! A lightweight 2d hand pose estimation approach.  CoRR, abs/2001.08047, 2020.
  * [62] D. Shan, J. Geng, M. Shu, and D. Fouhey.  Understanding human hands in contact at internet scale.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9866–9875, 2020.
  * [63] T. Simon, H. Joo, I. Matthews, and Y. Sheikh.  Hand keypoint detection in single images using multiview bootstrapping.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4645–4653, 2017.
  * [64] S. Sridhar, F. Mueller, M. Zollhoefer, D. Casas, A. Oulasvirta, and C. Theobalt.  Real-time joint tracking of a hand manipulating an object from RGB-D input.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 294–310, 2016.
  * [65] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas.  GRAB: A dataset of whole-body human grasping of objects.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 581–600, 2020.
  * [66] A. Tarvainen and H. Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
  * [67] A. Urooj and A. Borji.  Analysis of hand segmentation in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4710–4719, 2018.
  * [68] L. O. Vasconcelos, M. Mancini, D. Boscaini, S. R. Bulò, B. Caputo, and E. Ricci.  Shape consistent 2d keypoint estimation under domain shift.  In Proceedings of the International Conference on Pattern Recognition (ICPR), pages 8037–8044, 2020.
  * [69] T. H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pere.  Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2512–2521, 2019.
  * [70] Y. Wang, C. Peng, and Y. Liu.  Mask-pose cascaded CNN for 2d hand pose estimation from single color image.  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 29(11):3258–3268, 2019.
  * [71] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh.  Convolutional pose machines.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4732, 2016.
  * [72] M.-Y. Wu, P.-W. Ting, Y.-H. Tang, E. T. Chou, and L.-C. Fu.  Hand pose estimation in object-interaction based on deep learning for virtual reality applications.  Journal of Visual Communication and Image Representation, 70:102802, 04 2020.
  * [73] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le.  Unsupervised data augmentation for consistency training.  In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2020.
  * [74] L. Yan, B. Fan, S. Xiang, and C. Pan.  CMT: cross mean teacher unsupervised domain adaptation for VHR image semantic segmentation.  IEEE Geoscience and Remote Sensing Letters, 19:1–5, 2022.
  * [75] L. Yang, S. Chen, and A. Yao.  Semihand: Semi-supervised hand pose estimation with consistency.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 11364–11373, 2021.
  * [76] L. Yang, J. Li, W. Xu, Y. Diao, and C. Lu.  Bihand: Recovering hand mesh with multi-stage bisected hourglass networks.  In Proceedings of the British Machine Vision Conference (BMVC), 2020.
  * [77] S. Yuan, Q. Ye, B. Stenger, S. Jain, and T.-K. Kim.  BigHand2.2M benchmark: Hand pose dataset and state of the art analysis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2605–2613, 2017.
  * [78] C. Zhang, G. Wang, X. Chen, P. Xie, and T. Yamasaki.  Weakly supervised segmentation guided hand pose estimation during interaction with unknown objects.  In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, (ICASSP), pages 2673–2677, 2020.
  * [79] X. Zhou, A. Karpur, C. Gan, L. Luo, and Q. Huang.  Unsupervised domain adaptation for 3d keypoint estimation via view consistency.  In Proceedings of the European Conference on Computer Vision (ECCV), volume 11216, pages 141–157, 2018.
  * [80] C. Zimmermann, M. Argus, and T. Brox.  Contrastive representation learning for hand shape estimation.  CoRR, abs/2106.04324, 2021.
  * [81] C. Zimmermann and T. Brox.  Learning to estimate 3D hand pose from single RGB images.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4913–4921, 2017.
  * [82] C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. J. Argus, and T. Brox.  FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 813–822, 2019.
  * [83] Y. Zou, Z. Yu, B. V. Kumar, and J. Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 289–305, 2018.