# Efficient, ever-ready quantum memory at room temperature for single photons

Anthony C. Leung,1,2† W. Y. Sarah Lau,1,2,3,4† Aaron D. Tranter,1,2 Karun V.
Paul,1,2 Markus Rambach,3,4 Ben C. Buchler,1,2 Ping Koy Lam,1,2 Andrew G.
White,3,4 Till J. Weinhold3,4∗
1ARC Centre of Excellence for Quantum Computation and Communication Technology
2Department of Quantum Science and Technology, Research School of Physics, The
Australian National University, Acton, ACT 2601, Australia
3ARC Centre of Excellence for Engineered Quantum Systems
4School of Maths and Physics, University of Queensland, Brisbane, QLD 4072
Australia
†These authors contributed equally to this work.
∗Corresponding author; E-mail<EMAIL_ADDRESS>
###### Abstract

Efficient quantum memories will be an essential building block of large scale
networked quantum systems and provide a link between flying photonic qubits
and atomic or quasi-atomic local quantum processors. To provide a path to
scalability avoidance of bulky, difficult to maintain systems such as high
vacuum and low temperature cryogenics is imperative. Memory efficiencies above
50% are required to be operating above the quantum no-cloning limit. Such high
efficiencies have only been achieved in systems with photon sources tailored
to the memory bandwidth. In this paper we explore the combination of an
ultralow spectral bandwidth source of single photons from cavity-enhanced
spontaneous parametric down-conversion with a gas-ensemble atomic memory. Our
rubidium vapour gradient echo memory achieves 84$\pm$3 % recall efficiency of
single photons: a record for an always-ready, hot, and vacuum system free
optical memory.

## Introduction

Harnessing the full alluring potential of quantum technologies will require
linking quantum systems locally and globally [1, 2, 3] to share resources and
enhance capabilities: similar to the multi-processor CPUs of local computers,
or the vast computing power that comes from connecting data centers to form
the internet. Quantum interconnects will be an essential element in linking
the nodes of a quantum network consisting of sensors [4], distributed quantum
computers [5, 6], and quantum clock signals [7]. To ensure information
fidelity and practical transmission rates over long distances, the quantum
network channels need to be optical [8], like their classical counterparts.
Synchronisation between different systems will require the availability of
quantum memories [9] to store and relay the qubits and to build quantum
repeaters [10] that enable long distance communications.

A wide variety of quantum memory protocols have been demonstrated each with
their own set of advantages and disadvantages in terms of performance metrics
in efficiency, storage time, frequency bandwidth and other metrics. Solid-
state systems excel in storage times of the order of one minute [11] and the
potential for many hours [12]. Recall efficiencies around 90% were
demonstrated using both warm [13] and laser-cooled gaseous alkali atoms [14,
15]. Memories using off-resonant Raman transitions have demonstrated THz
bandwidths [16] and time-bandwidth products around 100 [17].

Efficient storage of the photonic qubit in these atom-based memories requires
matching memory and photon source features. Due to the complexity of
engineering a quantum light source to optimally integrate with the quantum
memory’s temporal, spectral, and spatial acceptance bandwidth, most
demonstrations have relied on attenuated classical light sources rather than
true quantum light. But weak coherent states cannot replace single photons for
applications such as photonic quantum computation [18], and physics at the
quantum limit may exhibit fundamentally different interactions [19, 20]. Hence
operation with tailored quantum light sources is vital for real-world relevant
benchmarking of quantum memories [21, 22, 23]. To date the most efficient
realisations employed electromagnetically induced transparency in a rubidium
magneto-optical trap (MOT) for the memory and four-wave mixing or spontaneous
Raman scattering in a separate cold atomic ensemble as the single photon
source [24, 25]. These produced an outstanding recall efficiency of $85\%$,
but the duty-cycle — the fractional time period where the memory is ready for
read/write operations— was limited to $3\%$ by the loading and cooling steps
of the MOTs [24].

If quantum networks are to scale then highly efficient quantum memories with
high duty-cycles are a necessity. Any efficiency below unity ultimately limits
the range and depth of the network, adding to this low duty-cycles massively
increases operational overheads. Utilisation of memories based on hot or room-
temperature atomic vapors can alleviate the loading and cooling steps, but
typically suffer from added noise [26, 27], low storage times [28], or both.

Here we report microsecond storage of single photons, with recall efficiencies
up to 84 $\pm$ 3%, in an always ready memory system. We achieve this by
integrating a hot vapor rubidium gradient echo memory (GEM) [29, 13, 30, 15]
with a tailored single-photon source based on cavity-enhanced spontaneous
parametric down-conversion (SPDC) [31]. Our system is scalable: the first
quantum memory and photon source combination that is vacuum-system free, non-
cryogenic, free of an operational duty-cycle and surpasses the no-cloning [32]
threshold.

## Results

Typical storage and recall of single photons for a 4 $\mu$s duration is
presented in Figure 1. Dark/light pink shaded regions indicate the times when
the memory is in the read/write modes respectively as indicated by the
magnetic field gradient state and presence of the control field shown in
Figures 1A and 1B respectively. The red histogram of Figure 1C indicates
single photons arriving at the memory during the “no memory” stage while the
control field is turned off: they are detected as coincidences with a herald
detection on the idler photon produced by the down-conversion source (see
Methods for details). The blue histogram in Figure 1D indicates photon
coincidences with the memory active and set to delay the signal by 4 $\mu$s.
We exploit the temporal correlation between the twin photons emitted from the
SPDC source to provide a high quality single photon signal through the
detection in the coincidence basis [33, 34]. This powerful technique enables
us to conclusively identify the single photon signal in front of a background
of leaked control field photons of $\sim$ 5-10k counts/s (See Methods for
details). Such strong background would have rendered other, more sensitive
single photon characterisation measurements like cross and auto-correlation
[35] fruitless.

Figure 1: Memory operation and storage results. The timing for (A) the
magnetic field gradient, and (B) control field switching. These determine the
input phase (light pink), the storage phase (white) and the recall phase (dark
pink) indicated in (D and E). (C) Experimentally collected coincidence data
displayed using histograms of the input single photon state (red) overlaid
with the corresponding input coherent state (black). (D) Experimental
coincidence data of single photon state stored and recalled after 4 $\mu$s
(light blue) overlaid with simulated results (green) showing the sharp onset
of the recall signal due to the larger spectral bandwidth of the single
photons causing bandwidth mismatch with the memory. (E) Experimental data of
corresponding coherent state storage (blue) overlaid with simulated results
(purple) showing bandwidth matched storage.

We compare the performance of the memory acting on single photons to the
storage of a coherent state. We temporally shape attenuated pump laser light
using an acousto-optic modulator (AOM) so that it possesses the same temporal
profile as the single photons. The input pulse without storage is shown as the
black solid line in Figure 1C, which agrees well with the measured not-stored
single photons. Recall of the coherent state from memory is shown in Figure
1E: it is strikingly similar to the recalled photon profile. The main
difference being how sharply the single photon recall appears as the control
field was turned back on. This phenomena arises from different spectral
bandwidths of the two states; single photons generated by SPDC have a wider
spectral bandwidth than the coherent state generated directly by the narrow
linewidth laser. This is elucidated from a simulation for the same input
temporal profile but with different relative bandwidths shown in Figures 1D
and 1E. The simulated single photon output was obtained by increasing the
input to memory bandwidth ratio by 25% from the simulated coherent output and
caused the recall to appear earlier, and flattened as observed in the single
photon experimental results. This slight spectral mismatch delivers a
substantial difference in the recall efficiency: the coherent state recall
reaches a maximal value of 55$\pm$1% compared to the 84% for the single
photon.

Storage and recall experiments were completed at varying storage times shown
in Figure 2. We classified as recalled, any coincident photons detected after
the control field was switched back on and before the magnetic field gradient
was flipped back to write mode. The data clearly shows the centre of the
recalled signal shifting with the extended storage time for both the coherent
state (solid lines) and the single photons (red and light blue histograms),
which retain their scattered signal preceding the main recall associated with
the larger bandwidth.

Figure 2: Single photon storage over a range of storage times. Coincidence
histograms demonstrating storage of single photons at a range of storage times
along with their matching results from coherent state storage.

Figure 3: Memory recall efficiency at different storage times. The red shaded
area shows storage times shorter than our minimum storage duration. Memory
efficiencies for the different times are shown for both the single photon
states (Red markers) and the coherent state (Purple markers).

Recall efficiency decreases with an increasing storage time, in part due to
addressed atoms moving out of the interaction region via atomic motion in the
warm vapour and because of atom-atom collisions. The reduction in recall
efficiency is clearly visible as reduced area of the recalled signal in Figure
2 and is plotted in Figure 3. The efficiency drops from 84 $\pm$ 3% at 4 $\mu
s$ to 57 $\pm$ 3% at 13 $\mu$s before falling below the no-cloning limit [32]
at 17 $\mu$s when we recall only 40 $\pm$ 3% of the stored signal. Storage
times shorter than 4 $\mu s$ are inaccessible due to limitations of our setup.
Error bars are derived from the propagated statistical noise in each
measurement, directly related to the total number of coincidence counts
measured. Our protocol was optimised to store the single photons, the success
of this optimisation is evident as the recall efficiency for the single
photons outperforms the recall of the coherent state at all times. From the
relative scaling between these, it is evident that the single photon storage
does not suffer from any additional degradation for extended storage times
compared to the coherent state storage. A detailed discussion on how the
recall efficiency was calculated can be found in the Supplementary Materials.

Clearly GEM operates as well in the single photon regime as it does for weak
coherent states when appropriately optimised[13] and we demonstrate a peak
recall efficiency of 84%, only one percent less than the demonstration by Wang
_et al._ [24], but free of the complexity of vacuum systems and cold atom
traps and we operate near continuously (only no storage during switching
times). We note that during the switching phase of the field gradient
indicated as the white background in Figures 1 & 2, some additional signal is
detected (not included in the recall efficiency). Possible explanations are
noise from the change in currents in the coils or optical leakage of fields
stored in the edges of the memory. See Supplementary material for more
details.

Performance of our source and memory system can be further improved:
introducing a non-poled crystal into the SPDC cavity to utilise the clustering
effect for intracavity filtering [36, 37, 38], enhances the relevant single
mode brightness of the source; optimising and improving the filtering systems
to reduce the level of control field leakage; and optimising the memory
geometry and magnetic field coil alignment, to improve coupling of the single
photon wave function to the memory. All these steps will bring GEM even closer
to its theoretical limit of 100% recall efficiency.

## Discussion

Our demonstration highlights the advantages and viability of tailoring cavity-
enhanced SPDC to produce memory-compatible single photons; together with the
vacuum-free room temperature memory this provides a significant step towards a
scalable quantum memory approach for networked quantum resources. We present
the first storage and recall of single photon states using the gradient echo
memory protocol. Our achieved 84% recall efficiency with a warm-vapour,
vacuum-free, high duty-cycle, quantum memory is comparable with the best
demonstrated quantum memory efficiency, $85\%$, which suffers intrinsically
low duty-cycle, employing a high vacuum photon source and memory based on cold
atoms. Our system is the first warm-vapor quantum memory to surpass the no-
cloning limit with single photons. Through its always ready nature and absence
of vacuum and cryo-technology our quantum memory system is suitable for
integration with large scale networked quantum technology applications.

## Materials and Methods

Figure 4: Single photon storage setup. (A) Single photon pairs are generated
using cavity-enhanced spontaneous parametric down conversion. A dichroic
mirror filters out the 397.5 nm pump light and an etalon is used to suppress
some of the unwanted single photon frequency modes. The single photon pairs
are separated using a polarizing beamsplitter (PBS) and fibre-coupled (not
shown) with one sent for herald detection (B) and other sent to the quantum
memory (C). (B) The single photon is further filtered down to a single
frequency mode with an optical cavity before being coupled to a single mode
fibre (SMF) to be detected by a single photon avalanche photodiode (APD). Both
the cavity throughput and the reflected input are further filtered to remove
scattering from the locking beam and detected to measure the single photon
generation rate. This rate is actively fed back to optimize the lock point of
the SPDC cavity. (C) The single photon and the control beam are combined on an
output port of an optical cavity and sent into a hot vapor Rb cell. A coil
wrapped around the cell with varying pitch forms the magnetic field gradient
to operate GEM. (D) The control beam is mostly filtered using an iris, Rb
filter cell, and optical cavity while the single photon is eventually detected
using an APD. (E) Level scheme of Rb87 showing the utilised Raman transition
for the GEM memory. The single photon together with the strong coherent
control beam links the $5S_{1/2}F{=}1$ and $F{=}2$ ground states via off-
resonant scattering of the $5P_{1/2}F=2$ state. During the storage phase the
excitation is stored as a spinwave in the $5S_{1/2}F{=}2$ state.

### Gradient Echo Memory

In the warm, rubidium-vapour based Gradient Echo Memory (GEM) [13] memory
scheme an optical lambda transition in rubidium 87 links the $F{=}1$ and
$F{=}2$ states of the $5S_{1/2}$ ground state, see Figure 4E. The photon from
the single-photon source addresses the probe transition (Rb87 D1
$5S_{1/2}F{=}1\rightarrow 5P_{1/2}F{=}2$) and the bright ($50$mW) control beam
(Rb87 D1 $5P_{1/2}F{=}2\rightarrow 5S_{1/2}F{=}2$) transfers the excitation
into the long lived ground state of the memory. Both optical fields are
detuned by 804 MHz from resonance with the excited $5P_{1/2}F{=}2$ state. The
detailed laser setup to facilitate this can be found in the Supplementary
Materials. To gain control of the read and write times [29] and avoid
reabsorption of recalled signal in the memory, we apply a linear magnetic
field gradient to the memory cell with coils of varying pitch. This creates a
spatially varying shift of the atomic resonance causing the memory to store
different frequency components of the optical field at different spatial
regions of the memory. Reversal of the magnetic field gradient and presence of
the optical control beam initiate the recall of the stored signal. As the GEM
protocol eliminates unwanted reabsorption its theoretical efficiency can reach
100%, and experimental results confirm that no noise is added to stored states
making GEM highly suitable for single photon storage [30]. To maximise our
experimentally achievable memory efficiency we use a Rb87 enriched cell with
0.5 Torr of Kr buffer gas, heated to about $80^{o}$C with the coils and the
cell contained inside a Mu-metal tube to shield from stray magnetic fields.

We keep the memory in write mode—magnetic field gradient and control field
on—at all times until an idler photon detection event heralds a single photon
storage event in the memory, see Figure 4B. This triggers a set of relays to
block the control field, ending the write phase, and flip the magnetic field
gradients as shown in Figures 1A and 1B. Once the gradient is fully reversed
and the control field is turned back on, recall of the single photon
commences. We switch the field gradient again after a predetermined time
period depending on the storage time to reset the memory back into write mode
(Figure 1A), ready to store for the next single photon from our source.

### Single photon generation

To match the strict spectral requirements of GEM, a cavity-enhanced SPDC
source was designed to produce sub-MHz bandwidth biphoton pairs suitable for
integration with the memory protocol [31]. A periodically poled potassium
titanyl phosphate (ppKTP) crystal, cut for type-II quasi-phase matching, is
used to produce orthogonally polarised photon pairs from a bow-tie cavity, as
depicted in Figure 4A. Residual pump is removed with a dichroic mirror and
ultra-narrow bandpass filter at 795 nm (FWHM 1.0 nm) after the source cavity.
We use a polarising beam splitter to separate the idler and signal modes of
the SPDC, which are fibre-coupled and sent to the heralding and memory arms
respectively, as shown in Figure 4.

When placed in a cavity, the otherwise broadband process of SPDC is modified
so that emission into resonant modes are enhanced and, in the case where both
downconverted modes are simultaneously resonant, further narrowed [39, 40].
Our source produces roughly 1000 frequency modes with a bandwidth of 429 kHz
separated by the cavity’s free spectral range (120.8 MHz) over the entire
phase matching bandwidth of ${\approx}100$ GHz. GEM can only store one of
these frequency modes at a time, thus a concatenation of external filtering
systems shown in Figure 4 is required to isolate the single central frequency
mode relevant for GEM. The first stage of this system is an air-spaced flat
surface etalon (SLS Optics, 75 GHz FSR, Finesse $\mathcal{F}{=}15$) placed
immediately after the SPDC cavity before the idler and signal modes are
separated.

The herald is then further filtered by a mode-cleaning cavity ($764$ MHz FSR,
$1$ MHz linewidth), locked with a counterpropagating beam resonant to a Rb85
transition and orthogonally polarized with respect to the herald photon (more
on how this cavity was set up in the Supplementary Material). The filtered
herald photons are then passed through a Rb85 cell removing backscattered
locking signal, before being detected by a fibre-coupled single photon
counting module (SPCM). Similarly the stored signal photon is filtered after
the memory with a different mode-cleaning cavity ($877$ MHz FSR, $1.6$ MHz
linewidth) locked in a similar fashion. The convoluted transmission spectrum
of these two mode-cleaning cavities, etalon and source cavity near perfectly
isolate the central frequency mode of the SPDC (see Supplementary material,
Figure S3C), ensuring that coincident events detected between the herald and
the stored photons will be of photons resonant with the memory transition.

To achieve a memory system ready to operate at 100% duty-cycle, the photon
source also must always be operational, without down-time due to load or re-
locking sequences. The source cavity is locked to the pump beam using the
Pound-Drever-Hall method [41, 42], ensuring the SPDC process to be pumped at
all times. Birefringence between the signal and idler modes is compensated
with the half-waveplate ‘flip-trick’ [31] which reduces double resonance of
both modes to one resonance condition. Fine temperature control of the SPDC
crystal then achieves triple resonance by overlapping the resonance of the two
down-converted modes at 795 nm (red) with that of the pump at 397.5 nm (blue).
Slight temperature drifts allowed the resonances between the pump and down-
converted photons to drift apart with time. To compensate we established a
feedback-lock based on the discarded single photon counts of the herald’s
filter cavity, see Figure 4B. This signal comprises the majority of frequency
modes and thus total number of produced single photons, enabling automated
tracking of the triple resonance condition and continuous user-intervention
free data collection for 8+ hours at a time. More details can be found in the
Supplementary Material.

Efficient storage and recall requires a strong control beam (50 mW) to co-
propagate with the single photons through the memory cell. Isolating the
single photon signal after storage necessitates filtering of the control field
as shown in Figure 4D. First an iris removed the part of the control beam that
was larger than the signal beam. By tuning the control field to a Rb85
resonance (see Supplementary Material Figure S2) we gain 60dB of isolation
from a Rb85 filter cell. However, atomic filter cells emit broadband
fluorescence when absorbing resonant light [43], which we removed with the
aforementioned signal beam filter cavity. A second Rb85 filter cell suppresses
leaking lock signal from this mode-cleaning cavity and adds further isolation
against the control beam. Finally an edge filter removes the remaining
fluorescence noise. The combined filtering chain measured 133dB suppression of
the control beam.

Figure 5: Data taking sequence. The data taking time sequence has a period of
2 s. The memory is turned off for 0.1 s four times by turning the control
field off (orange) measuring the rate of single photons injected into the
memory. The memory is operated for 1.2 s of the cycle (blue), and the final
0.4 s (green) operate with the control field on but with the incoming single
photons blocked to measure the background noise.

### Single photon detection

Single photons are detected using avalanche photodiodes (Perkin Elmer SPCM-
AQR-14-FC) and recorded using a time-tagging module (Roithner Lasertechnik,
100.1 ps time resolution). While both memory and source operated continuously,
data taking was sequenced to compensate for long term drifts. The 2 second
long sequence is broken into three different stages as shown in Figure 5. In
the first stage, labelled “no memory”, we measure the injected photon rate; in
the second stage, labelled ”memory”, we store, then recall single photons; in
the third stage, labelled ”no input” we measure background counts, allowing
for extraction of the single photon recall efficiency of the memory
independent of long term drifts or variations in the experiment. In the memory
stage, the recalled single photon signal is overlayed by leaked photons from
the coherent control field contributing $\sim 5-10$k counts/s. Without
detection in the coincidence basis between the signal and idler photon from
the SPDC source recovery of the stored single photon signal would have been
prohibited by such noise. Through our measurement sequencing we can subtract
the noise using the scaled background data from the ”no input” sequence,
yielding the recall data shown in Figures 1 and 2 the data set without
background subtraction is shown in the Supplementary Material as Figure S5
together with more details on the subtraction procedure. We emphasis that this
noise is not caused by our memory protocol, GEM being a proven noiseless
memory [30], but leaked field that can be suppressed through improved spatial
and spectral filtering.

## References

  * [1] H. J. Kimble, Nature 453, 1023 (2008).
  * [2] S. Wehner, D. Elkouss, R. Hanson, Science 362, eaam9288 (2018).
  * [3] S.-H. Wei, et al., Laser & Photonics Reviews p. 2100219 (2022).
  * [4] D. Gottesman, T. Jennewein, S. Croke, Physical Review Letters 109 (2012).
  * [5] L. Jiang, J. M. Taylor, A. S. Sørensen, M. D. Lukin, Physical Review A \- Atomic, Molecular, and Optical Physics 76 (2007).
  * [6] N. H. Nickerson, J. F. Fitzsimons, S. C. Benjamin, Physical Review X 4 (2014).
  * [7] P. Kómár, et al., Nature Physics 10, 582 (2014).
  * [8] S. W. Lee, T. C. Ralph, H. Jeong, Phys. Rev. A 100, 052303 (2019).
  * [9] K. Heshami, et al., Journal of Modern Optics 63, 2005 (2016). PMID: 27695198.
  * [10] H. J. Briegel, W. Dür, J. I. Cirac, P. Zoller, Phys. Rev. Lett. 81, 5932 (1998).
  * [11] G. Heinze, C. Hubrich, T. Halfmann, Phys. Rev. Lett. 111, 033601 (2013).
  * [12] M. Zhong, et al., Nature 517, 177 (2015).
  * [13] M. Hosseini, B. Sparkes, G. Campbell, P. Lam, B. Buchler, Nature Communications 2 (2011).
  * [14] Y.-F. Hsiao, et al., Phys. Rev. Lett. 120, 183602 (2018).
  * [15] Y.-W. Cho, et al., Optica 3, 100 (2016).
  * [16] D. G. England, et al., Phys. Rev. Lett. 114, 053602 (2015).
  * [17] J. Guo, et al., Nature Communications 10, 148 (2019).
  * [18] E. Knill, R. Laflamme, G. J. Milburn, Nature 409, 46–52 (2001).
  * [19] B. Fan, et al., Phys. Rev. Lett. 110, 053601 (2013).
  * [20] H. Vural, et al., Optica 5, 367 (2018).
  * [21] P.-J. Tsai, Y.-F. Hsiao, Y.-C. Chen, Phys. Rev. Research 2, 033155 (2020).
  * [22] L. Heller, J. Lowinski, K. Theophilo, Padrón-Brito, and H. de Riedmatten, arXiv:2111.08598 (2021).
  * [23] Y. Yu, et al., Phys. Rev. Lett. 127, 160502 (2021).
  * [24] Y. Wang, et al., Nature Photonics 13, 346-351 (2019).
  * [25] M. Cao, F. Hoffet, S. Qiu, A. S. Sheremet, J. Laurat, Optica. 7, 1440 (2020).
  * [26] K. Heshami, et al., Journal of Modern Optics 63, 2005 (2016).
  * [27] P. S. Michelberger, et al., New Journal of Physics 17, 043006 (2015).
  * [28] K. T. Kaczmarek, et al., Phys. Rev. A 97, 042316 (2018).
  * [29] M. Hosseini, et al., Nature 461, 241–245 (2009).
  * [30] M. Hosseini, G. Campbell, B. M. Sparkes, P. K. Lam, B. C. Buchler, Nature Physics 7, 794–798 (2011).
  * [31] M. Rambach, A. Nikolova, T. J. Weinhold, A. G. White, APL Photonics 1, 096101 (2016).
  * [32] W. K. Wootters, W. H. Zurek, Nature 299, 802 (1982).
  * [33] C. K. Hong, L. Mandel, Phys. Rev. Lett. 56, 58 (1986).
  * [34] T. B. Pittman, B. C. Jacobs, J. D. Franson, Optics Communications 246, 545 (2005).
  * [35] H. J. Kimble, M. Dagenais, L. Mandel, Phys. Rev. Lett. 39, 691 (1977).
  * [36] P.-J. Tsai, Y.-C. Chen, Quantum Science and Technology 3, 034005 (2018).
  * [37] X.-H. Bao, et al., Phys. Rev. Lett. 101, 190501 (2008).
  * [38] D. Rieländer, A. Lenhard, M. Mazzera, H. d. Riedmatten, New Journal of Physics 18, 123013 (2016).
  * [39] U. Herzog, M. Scholz, O. Benson, Phys. Rev. A 77, 023826 (2008).
  * [40] O. Slattery, L. Ma, K. Zong, X. Tang, Journal of Research of the National Institute of Standards and Technology 124, 124019 (2019).
  * [41] R. W. P. Drever, et al., Appl. Phys. B 31, 97 (1983).
  * [42] E. D. Black, American Journal of Physics 69, 79 (2001).
  * [43] A. Heifetz, et al., Optics Communications 232, 289–293 (2004).

## Acknowledgments

We thank Geoff T. Campbell and Daniel B. Higginbottom for valuable discussions
during the initial stages of planning and prototyping the experiment.

#### Funding

This research was supported by the Australian Research Council Centres of
Excellence for Engineered Quantum Systems (EQUS, CE170100009) and Quantum
Computation and Communication Technology (CE170100012).

#### Author contributions

Building single photon source: WYSL, MR, TJW. Building quantum memory: ACL.
Measurements and analysis: ACL and WYSL. Combining source with memory: ACL,
WYSL, ADT, KVP. Supervision: BCB, PKL, AGW, TJW. Writing - original draft:
ACL, WYSL. Writing - review and editing: ACL, WYSL, BCB, PKL, AGW, TJW.

## Supplementary Materials

### Laser Setup and Locking chains

To operate GEM efficiently requires the frequencies of the probe light and the
control field to match in their detuning from the $5P_{1/2}F{=}2$ state of
Rb87. By design, this detuning was chosen to have the control field resonant
with the Rb85 D1 transition which allowed the use of filter cells that heavily
attenuated control frequency light while having high transmission at the probe
frequency. The conceptual diagram of how all optical frequencies required were
generated and referenced, including those for locking filter cavities is shown
in Fig. S1. The 795 nm Ti:Sapphire laser is frequency locked to a rubidium
reference cell on a Rb85 D1 transition which is 804 MHz blue detuned from the
Rb87 D1 ($F{=}2\rightarrow F{=}2$) control field transition shown in Fig. 4E.
We detect the beatnote of the Ti:Sapphire laser and the diode laser (MOGLabs
795 MSA), by mixing the beams of both lasers on a high speed photodiode. The
signal is sent to a phase-frequency detector to phase lock the diode laser at
a detuning of 6.8 GHz. This stabilizes the diode laser 804 MHz away from the
Rb87 D1 transition ($F{=}1\rightarrow F{=}2$). Using acoustic-optic modulators
(AOMs) for finer frequency tuning, light from the Ti:Sapphire laser is also
used to generate the two locking beams required to actively lock the filter
cavities shown in Fig. 4 of the main text. The second harmonic generation
(SHG) cavity is stabilised to the diode laser via Pound-Drever-Hall (PDH)
locking [41, 42] for generating 397.5 nm light. This is then used to pump the
cavity-enhanced SPDC process in Fig. 4A of the main text, which is in turn
also locked via PDH locking to the pump light generated by the SHG cavity.
Electro-optical modulators are used to create the sidebands for PDH locking.
It is clearly essential for the GEM protocol that the central frequency mode
of the SPDC cavity spectrum is correctly referenced to the desired detuned
probe frequency of the Raman transition; this was achieved by referencing the
pump light of the SHG cavity to the rubidium resonance via the beatnote,
acting as an absolute reference for the down-conversion process.

Fig. S1: Setup for generating all optical frequencies. The 795 nm Ti:Sapphire
laser is frequency locked to a rubidium reference via saturated absorption.
The memory control field and cavity locking beams are derived from it and
controlled with acoustic-optic modulators (AOM). Some of the Ti:Sapphire light
is combined on a 50:50 beamsplitter (BS) with light from a diode laser to form
a beatnote that is detected with a phase frequency detector (PFD) and used to
phase lock the diode laser 6.8 GHz away from the Ti:Sapphire. The diode laser
pumps a second harmonic generation (SHG) cavity which produces 397.5 nm light
used to pump the SPDC cavity to generate single photons at the probe
frequency.

In Fig. S2, the different frequencies for the locking beams, control field for
the memory and the probe frequency of the single photons are plotted relative
to the spectrum of Rb85 and Rb87. Clearly the control field and locking beams
fall within the broad absorption band of the hot rubidium atoms shown as the
orange line, while the single photons at the probe beam frequency see a
virtually transparent path in the absence of the lambda coupling facilitated
by the control field.

Fig. S2: Filter cell absorption spectra. Absorption of Rb filter cells and the
frequency of the probe (single photons), control and locking beams

### Monitoring the SPDC Cavity Lock Position

Triply resonant parametric down conversion requires that the pump and both
down converted fields are all simultaneously resonant in the enhancement
cavity. As we are using type-II down-conversion, we have light fields of two
wavelengths (397.5 nm, blue; and 795 nm, red) and with horizontal and vertical
polarizations at 795 nm simultaneously present in the cavity. Ensuring triple
resonance naively requires three free parameters. Using the flip-trick [31],
we reduce the problem to only two free parameters. We use a piezo-mounted
cavity mirror (stabilising with respect to the pump light), and the
temperature of the down-conversion crystal, controlling the relative
birefringence between the pump field and the down-converted modes. When the
lock point on the blue resonance is exactly overlapped with the peak of the
red resonance the generation rate of the photon pairs is maximised for a given
pump power. Environmental temperature drifts cause relative shifts between the
red and blue resonances, hence shifting the set lock point away from the peak
of the single photon resonance decreasing the generation rate. The cavity
remains locked as the shifts are narrower than the PDH feature of the
resonance used to lock the cavity. By design, the blue resonance
($\mathcal{F}\approx 8.5$) is much broader than the red ($\mathcal{F}\approx
181$) so that small adjustments to the offset of the PDH lock point can be
made, allowing the lock point to follow the peak of the efficiency for
generating down-converted photons. To do this we use the reflected single
photons from the mode-cleaning cavity in the herald arm. This signal is
proportional to the generation rate of the SPDC cavity as the number of modes
rejected by the mode-cleaner is effectively constant and the most direct
method of measuring the SPDC cavity drift alone. Any other available single
photon signal (ie. at the end of the heralding arm, and after the memory) are
subject to other cavity drifts masking the actual generation rate directly out
of the SPDC cavity. To isolate these reflected photons from any leaked locking
signal, another polarising beam splitter and filter cell is placed in this
path, as depicted in Fig. 4B of the main paper.
As however we are effectively locking to the top of the peak, we implemented a
dither lock that we dubbed PhD-lock (in honour of replacing the PhD student
that used to perform this optimisation manually), that took a random step on
the offset point of the PDC-cavity lock to try to increase the rejected single
photon count rate. If count rate increased the PhD-lock would follow with a
step in the same direction until the rate declined. Once a decline was
detected the PhD-lock reversed polarity and stepped in the opposite direction
until the count rate again declined leading to an effective dither around the
maximal count rate. Our PhD-lock used a dynamic step size, where each
adjustment was proportional to the difference in single photon counts averaged
over n clock cycles from the previous n cycles. Changing the number of
integration cycles allowed trading the PhD-lock’s robustness for speed and
vice-versa. On occasion the drift of the optimal lock position exceeded the
capture range of the PDH lock, which could then be regained by a small
temperature adjustment of the crystal oven to re-overlap them again. The
stability of the entire lock chain was such that continuous unsupervised data
acquisition over approximately half a day was possible.

### Single Photon Filtering Spectrum

The biphoton pairs produced by the cavity-enhanced SPDC source exist as a
frequency comb spanning approximately 100 GHz with a 120.8 MHz free spectral
range and a linewidth of $429\pm 10$ kHz for each mode. The spectrum is formed
as the convolution of the crystal’s phase matching bandwidth, and the cavity’s
spectrum. To reduce the frequency comb to a single frequency mode, external
filtering was required. We combined an etalon (SLS Optics, FSR $=75$ GHz,
$\mathcal{F}=15$) and a mode-cleaning cavity (FSR = $761\pm 14$ MHz, high
finesse mode with a linewidth of $1.1\pm 0.4$ MHz). A simulation of the
resulting spectra of each filtering component is presented in Fig. S3(A and
B), and the final spectrum from the concatenated chain in Fig. S3C. The most
conservative measure of the power in this central frequency mode is $76\%$.
For memory recalled events we are detecting the events in coincidence. So
while the herald has undergone the filtering as described above, the signal
photon also passed through the same etalon, but was filtered by different
mode-cleaning cavity in the memory arm (with a FSR $=881$ MHz). The mismatch
in FSR between the two mode-cleaning cavities thus ensures that all coincident
events result from only the central frequency mode as all other modes are
suppressed in either the heralds or the memory arm. Note that while the etalon
spectrum in Fig. S3B contain 3 main transmission sections, within each section
the frequency modes separated by the FSR of the SPDC cavity still exists, as
seen in the inset of the figure.

Fig. S3: Cavity only, etalon only and combined cavity and etalon filtering of
SPDC. The simulated spectrum for the single photons after filtering with just
(A) the herald arm mode-cleaner cavity, (B) the etalon after the SPDC cavity
or (C) combined cavity and etalon filtering. The pink shading is the original
unfiltered output spectrum of the SPDC. The insets in each are the zoomed in
spectrum centered on the central frequency mode.

### Filtering cavity setup

Both filtering cavities in the herald and memory arms were set up the same way
with a locking beam generated by an AOM from the Ti:Sapphire combined with a
PBS to lock the cavities in the backward direction. The locking beam had an
orthogonal polarization to the single photons and locked the cavities in the
low finesse mode while the single photons were being filtered in the high
finesse mode. To optimize the throughput of the single photons, the locking
beam frequency was fine-tuned to maximize the single photon count rate
measured by the single photon detectors. Backscattering of the locking beam
added to the noise counts measured by the single photon detectors so a Rb85
filter cell was placed after the cavity which absorbed any backscattered
light. These cavities, in addition to the etalon, was sufficient for filtering
the SPDC output to a single frequency mode. The mode-cleaning cavity after the
memory also served for control filtering; it is used to suppress both control
frequency light that passed through the first filter cell and also any wide-
band fluorescence caused by the filter cell absorbing the control beam. The
second filter cell after the cavity was also used to provide further
suppression of control light. Finally, a long pass wavelength filter (cutoff
at 790 nm) removed the rest of the fluorescence before the single photons were
coupled into a single mode fibre for spatial filtering and sent to a single
photon detector.

### Quantifying memory efficiency

The recall efficiencies measured from GEM depend directly on the integration
time over which the counts were considered. The upper bound to the efficiency
is found by starting the integration at the time when the gradient of the
applied magnetic field is reversed, thus switching the memory from write mode
to read mode. This was calculated as a check to ensure that the total amount
of recalled and transmitted light did not exceed what was initially stored.
The gradient flip is the required condition for a recall, but the recall is
not initiated until also the control beam is applied. The value for the phase
where both of these conditions are met, is thus the reported efficiency in the
main text. A lower bound was determined based on the time window during which
the recalled coherent state was measured. As noted earlier, there were key
differences in the temporal shape between the single photons and coherent
state, thus the coherent state defined time window does not extend to all
times when recalled single photons are expected; the efficiencies calculated
this way serves as a lower bound to the recall efficiency. These upper and
lower bounds to recall efficiency are shown together with the recall
efficiency in Figure S4. All three forms of efficiency were generated with the
scaled coincidences as explained in the single photon detection section of the
main text.

Fig. S4: Memory recall efficiency at different storage times. The red shaded
area identifies storage times shorter than our minimum storage duration. We
show here the results of our upper bound, the reported recall efficency and
the lower bound. As upper bound serves the gradient defined efficiency (blue
dots), which considers all photons from the memory while the magnetic field
gradient is negative (read mode). The reported recall efficiency (red crosses)
considers photons when the gradient is reversed _and_ the control field is on.
Finally, the lower bound is set by the coincidences integrated over the times
at which the coherent state recall was measured (green stars).

### Leakage during magnetic field switching

The small level of recall observed coming from the memory in the storage phase
during which the memory’s magnetic fields are switching could potentially have
been due to: a combination of the arrival of single photons from other
frequency modes in the SPDC acting as control photons completing a Raman or
EIT transition inducing some recall; and the memory being transparent to
signal photons that have arrived during the magnetic field switching time due
to the sub-unity heralding efficiency. Another possible explanation for this
observation is electronic noise in the detection system induced by the
magnetic fields switching. Further investigation is required to fully
understand this. The method in which this data was collected and processed
means that the final results, like those shown in Fig. 1 of the main text,
show only the effect of sending single photons to the quantum memory. This
rules out the possible effect of additional photons caused via four wave
mixing due to the control field, as well as other noise caused by the input
single photons as they are too weak and far detuned from any atomic transition
to cause similar effects. Such conclusions are in agreement to what was
demonstrated in [30] which showed the GEM scheme in warm vapor does not
introduce additional noise to its recall.

### Background subtraction methodology for memory data

Fig. S5: Example data set for the coincidence collection for 4 $\mu$s storage.
The purple area are coincidences collected during the “No input” data
acquisition phase. The blue area labeled Raw recall is data collected during
the “Memory” phase and the green area is the difference between the “Memory”
and “No input” data sets that was then used to provide the recalled signal.

While our filtering chains used for the control field and the locking light
beams for the filter cavities provided excellent suppression of these fields
on the single photon counter for the signal single photons, some small amount
of light from these sources still reached this detector. The total filtering
chain provided 133 dB isolation as mentioned in the main text, but despite
this significant reduction, we still observed 5-10k counts/s as noise
background. As our scheme attributes any single photons detected during the
recall phase before the magnetic field gradient is flipped back as a
coincidence detection with the herald that triggered the recall procedure, we
need to remove these noise events from the real recalls of single photons.
Figure S5 shows the collected data for the 4 $\mu$s storage from the “Memory”
and “No input” phases of the data collection cycle. During the “No memory”
phase used to collect the input single photon signal shown in Fig. 1C (of the
main paper) such subtraction was not required as no control field was present.
The success of this method is evident from our data presented in the main
paper. As the control field leakage amount drifted with alignment and
temperature changes in the filter cells, this motivated the sequenced data
acquisition method ensuring that “Memory” and “No input” data was taken at
effectively equal conditions.


# DualMotion: Global-to-Local Casual Motion Design for Character Animations

Yichen Peng*
JAIST
<EMAIL_ADDRESS>Chunqi Zhao*
The University of Tokyo
<EMAIL_ADDRESS>Haoran Xie
JAIST
<EMAIL_ADDRESS>Tsukasa Fukusato
The University of Tokyo
<EMAIL_ADDRESS>Kazunori Miyata
JAIST
<EMAIL_ADDRESS>Takeo Igarashi
The University of Tokyo
<EMAIL_ADDRESS>
###### Abstract

Animating 3D characters using motion capture data requires basic expertise and
manual labor. To support the creativity of animation design and make it easier
for common users, we present a sketch-based interface DualMotion, with rough
sketches as input for designing daily-life animations of characters, such as
walking and jumping. Our approach enables to combine global motions of lower
limbs and the local motion of the upper limbs in a database by utilizing a
two-stage design strategy. Users are allowed to design a motion by starting
with drawing a rough trajectory of a body/lower limb movement in the global
design stage. The upper limb motions are then designed by drawing several more
relative motion trajectories in the local design stage. We conduct a user
study and verify the effectiveness and convenience of the proposed system in
creative activities.

##  1 Introduction

Creativity support systems have been extensively explored for both
professional and common users in computer graphics and human-computer
interaction fields. In particular, the production of character animation is a
common creative process in both entertainment use and industries like game and
film developments, sports, and medical applications. Creating natural
character animation requires expertise and manual labor, which prohibits
novice users from even casually producing some simple character animations. In
practice, motion captures are widely used in the industry to create natural
character animations. In the motion capture process, one or more motion actors
who are equipped with sensors in their body parts act the desired motions. The
word “natural” means to convey the credibility in the animation that there is
really a virtual human there. At the same time, the cameras in the environment
sample and record the locations of the equipped sensors many times per second.
In this sense, the cost of motion capture process, which includes labor, site,
time, and equipment, can be expensive and time consuming. A big volume of
motion captures have been recorded and many motion capture databases have been
released for open access, such as [1] and [15]. However, the reuse of existing
motion capture database is still difficult for novices: for specific
applications and stylish motion details, animators still need to record motion
captures from scratch. Although existing motion capture databases usually
provide keyword-based motion search, the subjectively defined keywords can not
represent all details in each motion sequence and thus prohibit animators from
retrieving desired motion sub-sequences from a large scale database.

Figure 1: DualMotion enables users to input rough sketches and create casual
character animations.

As another option for motion retrieval, sketch-based interfaces have been
developed and researched by a previous work [23]. The sketch-based interfaces
are for computer graphics applications, such as interactive animation design
[12, 17, 27], 3D modeling [28] and mesh editing [22], and have been proven to
be intuitive to users especially novice users. Nevertheless, applying sketch-
based interfaces in motion retrieval and editing faces some challenges. Motion
capture takes a complicated format by its spatio-temporal nature. A motion
capture sequence consists of a static part and a dynamic part. The static part
defines the character’s skeletal structure along with node offsets. The
dynamic part defines the character’s changing pose in each frame, represented
by the relative node rotations w.r.t. (with reference to) each node’s parent
node. In a typical human character animation, the skeletal structure may
contain over ten nodes, and the dynamic sequence can consist of thousands of
frames. On the contrary, 2D sketching space is much simpler than the motion
capture data in terms of dimension and is not expressive enough for tasks such
as motion retrieval and composition. Particularly, the spread of touch screen
devices and short video applications proposes a potential situation to create
character animations, where the operation range and accuracy are both limited.
In such a situation, the fact that users can be “casual” and can only input
with rough sketches makes the task “sketch to animate characters” even more
challenging.

To solve these issues, we propose DualMotion, a multi-stage sketching scheme
for character animation retrieval and composition considering novices as
target users in a casual animating situation (the operation range and accuracy
are limited, while the naturalness of the animation can be sacrificed somewhat
for casual creation). The key idea is to decompose the challenging and
complicated design task into several stages. In each stage, a novice user only
needs to accomplish a sub-task and pursue a sub-objective. For the sketch-
based motion retrieval and composition task, we decompose the entire design
process into two stages: The first one is the global stage, where the
reference point is the virtual ground and the user draws a rough trajectory to
represent the entire movement of the character; the second one is the local
stage, where the reference point is the character’s center of mass and the
user draws relative trajectories to represent the detailed movement of
character limbs. We adopt a data-driven approach to assist users in retrieving
and combining desired motions from databases. In addition, in either of the
two stages, we show the retrieved motion candidates in dashed polygons as
shadow guidance. We implement the proposed method and conducted a user study
to compare the two-stage sketching interface with an one-stage baseline. Our
qualitative and quantitative evaluation results showed that the proposed
method can significantly (p ¡ 0.01) assist novice users in searching and
editing their desired character animations.

We summarize our main contributions as below:

  1. 1.

A creativity support system with two-stage character animation retrieval and
composition scheme that decomposes the challenging task into stages. In each
stage, novice users just need to concentrate on a sub-goal of the design
process.

  2. 2.

The implementation called DualMotion,which utilizes the proposed two-stage
design scheme. DualMotion supports novice users in designing usual motion in a
data-driven manner.

  3. 3.

A user study involving 14 participants to validate the effectiveness and
usability of the proposed scheme and the DualMotion implementation.

##  2 Related Work

This section reviews prior work on frameworks for (1) character animations
authoring and (2) motion retrieval and composition.

###  2.1 Authoring Character Animations

Some systems enable users to design character animations using physical-based
simulation [27, 18, 26] and users’ demonstration [7, 8, 9]. Although these
methods can be suitable for simple object motions, such as graffiti, they are
unsuitable for skeletal character motions. The reason is that skeletal data
consists of $N$ joints ($N>20$), that is, to make natural-looking motions,
users must repeatedly set all joint angles at each frame, which might be very
time consuming and tedious. To address this issue, animators often use motion
capture data (Mocap), which are records of the actual movement of objects or
people. Dontcheva et al. [6] enabled the animator to author character
animations by imitating the movement under a customized Mocap system. Besides,
the large scale Mocap databases have been constructed by many research
institutions and companies [1, 15], and many works to generate motion
sequences using the Mocap databases have been proposed [11, 16]. Then, we take
a “data-driven” approach but consider (i) how to intuitively find motion data
from the database and (ii) how to edit motion data for our needs.

###  2.2 Retrieving and Editing Character Motion Data

Against this background, retrieving motion data from large motion databases
has been thoroughly investigated, for example, typing keywords. Kruger et al.
[19] introduced a similarity searches algorithm for motion retrieval in a
large-scale Mocap databases. Peng et al. [23] introduced a sketching interface
to retrieve motion data by drawing motion trajectories on a screen. Unlike
static pose drawings [5], motion trajectories are suitable for imagining the
change of character motions. However, we need to edit the retrieved data to
meet the user’s needs, rather than use the data as is.

Motion Doodle [25] segments the input sketch of motion trajectory into several
primitives (e.g., walk and jump) and generates simple motion sequences (e.g.,
walk $\rightarrow$ jump $\rightarrow$ jump). However, this system focuses only
on the root motions of the character (located at the hip), called “global”
motion, so it remains difficult to locally edit character motions, such as
hands and legs. By contrast, Choi et al. [4] proposed SketchiMo, which allows
users to directly edit trajectories of character motions. These approaches
enable users to directly edit local motions one by one on the screen, but they
require local coordinates of the character joints at each frame to be
carefully tuned. MotionMaster [21] built a database of Kungfu motion which
allowed users to sketch the start and end poses of the character and draw the
trajectory of a user-specified joint movement after labeling those joint
pairs. However, this system required a high level of drawing skill with the
time-consuming labelling tasks.

Building on these approaches, we simply separate the sketch process of motion
trajectories into two stages: (i) global stage (i.e., root motion retrieval)
and (ii) local motion stage (i.e., limb motion retrieval), and make motion
sequences by combining the retrieved global/local motions.

##  3 Global-to-Local Motion Design

The proposed character animation retrieval and composition scheme was inspired
from the following observations: character animation design is challenging for
novice users because of the complicated data format of motion data. In
Addition, limited by their experience and skills, novice users may find the
design task difficult as a single process and suffer from the blank page
syndrome. To address this issue, we propose a two-stage design scheme to
assist novice users in designing motions of character animations with both
global and local stages. We decompose the whole motion design process into two
stages. In each stage, users can just concentrate on one sub-goal of the
design process. Users begin from the global stage and sketch on the rough
movement of the character’s lower limb. After users are satisfied with the
work, they enter the local stage and draw the detailed movements of the
character’s upper limbs. In this section, we go through the proposed motion
design scheme and describe the global and local stages in details.

Figure 2: The proposed Global-to-Local design scheme for data-driven character
animation design. In the global stage, users retrieve the rough motion of the
whole character by inputting a query stroke; in the local stage, users further
draw the strokes to edit the upper limb motion.

###  3.1 Overview

In the proposed character animation authoring scheme, the character is shown
on a virtual ground. The users draw query strokes that represent motion
trajectories of the character nodes. To fully utilize and reuse existing Mocap
databases, we conduct a trajectory similarity comparison between the user-
input query strokes and the node trajectories in the database by projecting
them into the same 2D camera coordinate system. Because the main purpose of
the current prototype is the locomotion retrieval, we only adopted the motions
of walking, running, jumping and punch and kicking for simplification. The
overall pipeline is shown in Figure 2.

###  3.2 Global Stage (Lower Limb)

Global stage is the first stage and the default state of the two-stage design
scheme. When users enter the stage by opening an initial scene, a blank
virtual ground is displayed. To start the query and further composition, the
users draw a trajectory on the screen, to represents the ground-projected
moving trajectory of the character’s center of mass. The system then starts to
match the user-input query stroke with the projected trajectories in the
database, and apply the best-matched animation on the character in the
interface. The users can choose the desired rough movement of the whole
character and enter the local stage to specify more details of the character’s
upper limbs, or just repeat the retrieval in the global stage. In the global
stage, we define the virtual ground as a reference point, which allows the
users to freely adjust the position and angle of the camera.

###  3.3 Local Stage (Upper Limb)

After the users are satisfied with the retrieved global movement of the
character, they can continue the design process and choose a specific upper
limb for further detailed retrieval. The reference point of the local stage is
the character’s center of mass, which enables a surveillance camera that
always follows the character and stares at the character’s center of mass. The
camera locates on a sphere whose radius is adjustable and center is also the
character’s center of mass. Similarly, the users draw trajectories of the
selected limb node with respect to the center of mass. When the users finish
the query stroke input, the system translates all trajectories of the selected
limb node from the database by projecting them to the camera coordinate system
and starts similarity matching. The users then pick the desired limb motion
from the retrieved results, and the system applies the motion to the selected
limb based on the rough motion retrieved in the global stage to make a
combined new animation. With iterations between global and local stages on
different limbs, the users finally make a new animation from existing motion
captures and save it to end the design process. Although legs should be
considered as upper limbs, we only enable head and hands motion retrieval and
composition in the local stage because leg motion is highly entangled with the
overall movements of the character.

##  4 System Framework

In this section, we explain how we implement the motion editing system
prototype called DualMotion in the proposed framework.

###  4.1 Dataset Construction

To build the data-driven motion editing system, we construct the motion
dataset by selecting the locomotion data from the CMU Mocap data library [1],
which consist of several motion category, such as locomotion, physical
activities & Sports, etc. (Note that our data is mainly chosen from the
locomotion.) We empirically choose motion data with similar sizes and skeletal
proportions. We then manually trim the motion data into 100 frames to allow
them to share the same duration, and normalize all the motion sequence into a
same initial position that is the origin of the virtual spatial coordinate
system. In our case, the root node (hip) movement of each motion data is
considered the global movement, and each limb movement is considered the
relative local movements of the root node. Lastly, each motion data point is
split into limbs and hip, which are stored.

###  4.2 Trajectory Representations

The users directly sketch on the widget that visualizes the character
animation for both global and local stages. That is, the sketching canvas
coordinate system shares with the camera coordinate system of the
visualization widget. Note that as stated in Section 3, the editing tool
maintains a camera with different reference points in the global and local
stages.

In each stage, motion trajectories in the motion dataset are projected into
the canvas coordinate system, as follows:

$\mathbf{V}_{canvas}^{j}(i,t)=M_{proj}\mathbf{V}_{orig}^{j}(i,t)$ (1)

where $\mathbf{V}_{orig}^{j}(i,t)\in\mathbb{R}^{3}$ is the 3D coordinate of
$i$-th node at time $t\in\\{1,T\\}$ in the $j$-th motion data in the database,
$\mathbf{V}_{canvas}^{j}(i,t)\in\mathbb{R}^{2}$ is the corresponding projected
2D coordinates of the motion trajectory, and
$M_{proj}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$ is a 3D to 2D projection
matrix of the maintained camera. In this paper, we represent the motion
trajectory in the database as follows:

$V_{orig}^{j}(i)=\\{\mathbf{V}_{orig}^{j}(i,1),\mathbf{V}_{orig}^{j}(i,2),\cdots,\mathbf{V}_{orig}^{j}(i,T)\\}$
(2)

where $T$ is the number of the motion frame. When the users pan or zoom to
relocate the camera so that the projection matrix $M_{proj}$ changes, the
proposed tool updates the projected trajectories $V_{canvas}^{j}(i)$
$=\\{\mathbf{V}_{canvas}^{j}(i,1),\cdots,\mathbf{V}_{canvas}^{j}(i,T)\\}$ for
comparing the user query sketch with the motion in database.

Figure 3: Shadow-like guidance that displays joint movement projected
trajectory in global (left) and local (right) design stages.

###  4.3 Trajectory-Based Retrieval

Similarly, we represent a user-input stroke $S_{user}$ for both two stages as
follows:

$S_{user}=\\{\mathbf{S}_{user}(1),\mathbf{S}_{user}(2),\cdots,\mathbf{S}_{user}(T)\\}$
(3)

where ${S}_{user}(t)\in\mathbb{R}^{2}$ is the coordinate of $t$-th sampling
point of the user-input stroke, and $T$ is the number of total sampling points
of the stroke.

We then compute a similarity between the query stroke $S_{user}$ and the
projected trajectory of the $j$-th motion data in the database
$V_{canvas}^{j}$, as follows:

$Sim^{\>j}(i)=F(S_{user},V_{canvas}^{j}(i))$ (4)

where $F(a,b)$ is the Fréchet distance between two strokes $a$ and $b$. Note
that $i$ represents the root node in the global stage, and the limb nodes in
the local stage.

After computing all the similarities, we pick the top $N$ relevant retrieval
results in the database ($N=5$ in our implementation) and merge them as a
shadow-like guidance to display their projected trajectories in real-time,
inspired by ShadowDraw [20]. As shown in Figure. 3, the user can click on the
most desirable motion candidate for selecting global/local motion data.
Benefiting from the guidance, users understand the locations and shapes of
each stroke drawing.

###  4.4 Motion Sequence Synthesis

Given the retrieved global and local motions, DualMotion generates a final
motion sequence. In our prototype, we utilized a BVH data format, which
consists of the position of the root node and all joints’ rotation at each
frame, and simply assigned the rotational matrices of the associated nodes in
the retrieved local motion to the retrieved global motion. For instance, the
rotational matrices of left shoulder node and its child nodes would be
modified when the left hand movement is authored. Similarly, right-shoulder
node, lower-neck node and their child nodes are allowed to be re-assigned when
the related movement is modified. The similar approach is applied by Iwamoto
et al. [16] which they segmented the hands movement and body motion to
synthesize the dance motion sequence. Although this approach seems to work
well, other methods can also be used in our framework.

Figure 4: The design process of DualMotion. Figure 5: Interface of DualMotion.
It consists of: (1) a tool panel contains buttons like undo, load, save, draw,
select, camera pan, and camera zooming; (2) a stage panel to switch between
global and local stages; (3) a control panel for timeline visualization and
(4) a graphical interface for drawing and character animation visualization.

###  4.5 User Interface

We implement the two-stage character animation design scheme as a prototype
called DualMotion using openGL and Qt. Figure 5 shows the user interface of
DualMotion. It consists of a graphical widget and a control panel. The
graphical widget is main region for users to interact (visualization and
drawing). At the top of the graphical widget, the control panel provides
buttons to enable basic functions (undo, load, save, etc.), mode selection
between drawing and result selecting, camera controlling, global and local
stage switching, and timeline visualization. We have introduced the the basic
design scheme in 3 and 4. As an alternative for showing the character
animation, we implemented a timeline visualization to help users check the
entire animation in a static way rather than waiting for the animation
playback. The character animation is shown in the visualization widget by
sampling the frames in the motion sequence, and the character skeleton of each
frame is statically overlaid in the graphical widget. Only one frame of the
character skeleton is highlighted, whereas the others are semi-transparent. A
slider is used for adjusting the position of the highlighted frame, and
another slider is used for manipulating the frame interval.

###  4.6 Design Process

The design process of DualMotion is shown in Figure 4. The user starts from
the global stage with a rough objective about the character animation in mind
and then goes forward into the local stage and decides more details in upper
limbs. The user can repeat the two design stages until satisfied with the
final output. Every time the user inputs query strokes, the DualMotion
interface feeds forward the user inputs, compares them with the database, and
render the retrieved and composited motion sequences.

##  5 User Study

To evaluate the validation of DualMotion, we conducted a user study that
consists of a comparison study and a post-experiment questionnaire. In this
section, we describe the experimental details and discuss the evaluation
result.

###  5.1 Comparison Study

To evaluate the validation of the proposed user interface, we conduct a user
study to compare DualMotion with a one-stage motion retrieval UI, which has
the same function as DualMotion, except that it only allows users to edit and
synthesize motion to obtain the results but only in the global stage editing
(i.e. hip movement). For example, users are allowed to edit a hand waved
movement while the character is stepping forward. It is difficult for users to
define such a hand movement on a moving character in this case. Similar to
DualMotion, the implemented one-stage UI provides the shadow guidance to aid
users in finding similar results (trajectories of motion) after each time they
finish drawing a stroke. However, in contrast to DualMotion, it only allows
users to retrieve motion by drawing the trajectories in a global manner.

The study consists of two sections: 1) motion design with a reference motion,
2) free motion design. In the first section, we provided five skeleton motions
that consist of different identities of joint movements from the prepared
database. By using DualMotion and the one-stage edit UI respectively, the
participants were required to retrieve and edit the motion following one of
these references randomly.

On the one hand, to compare the efficiency, we recorded the time cost and the
number of operations (mouse clicks). On the other hand, to compare the quality
of the design result, we also calculated the mean absolute error between the
synthetic motion and the reference motion to compare the similarity, which
confirms the accuracy of the user design result in the two different UIs. In
the second section, to evaluate whether our interface meets user intent, the
participants are asked to do a free motion design using DualMotion in 5
minutes. They are asked to answer their editorial intent or the reason why
they would like to design such kind of the character motion in the post-
experiment questionnaire.

###  5.2 System Evaluation

Except to answer the editorial intent in free design task, the participants
are required to evaluate the overall system by answering the questions rely on
System Usability Scale (SUS) [3] metrics after the experiment. To rate the
perceived workload of design motion using DualMotion, we requested the
participants to fill in a post-experiment questionnaire designed following
NASA Task Load Index [10] (Considering the length of the questionnaire, we
utilized the raw version of NASA-TLX in our evaluation study where weighing
each of the evaluation item is not necessary).

###  5.3 Experiment Process

We invited 14 graduate students at the age of approximately 25 years to
participate in our experiment. We choose a total of 55 skeleton motion data
from the CMU [1] motion database for retrieval in the user study, which are
gait data in different directions with different styles, including fast walk,
slow walk, duck walk, zombie walk, and so on. They had approximately 5 minutes
to get familiar with each tool before the task. In addition, a player window
with a modifiable virtual camera is provided so that reference motion clips
can be viewed at any time during design. The order of finishing tasks using
one-stage UI and DualMotion are randomly shuffled to make sure that we equally
evaluated each UI. The free motion design experiment is the final task that we
believed that the participants are familiar enough to user DualMotion to
create some motion following their intent. Lastly, they are required to fill
the post-experiment questionnaire mentioned above.

Figure 6: Examples of participant design results by using each interface for
the five motion references. The design results using DualMotion are more
similar to the reference compared to the one-stage interface, especially the
hands movements .

##  6 Results

###  6.1 Evaluation Results

Figure 6 (right) shows various reference motions which used in our user study.
For the validation of DualMotion, we compared the Euler angle of $i$-th node
at time $t\in\\{1,T\\}$ in the user-designed results $\mathbf{x}_{i}(t)$ to
the reference $\mathbf{x}^{\prime}_{i}(t)$ and computed the MSE loss as
follows:

${\color[rgb]{0,0,0}MSE=\frac{1}{TN}\sum_{t=1}^{T}\sum_{i=1}^{N}\|\mathbf{x}_{i}(t)-\mathbf{x}^{\prime}_{i}(t)\|^{2}}$
(5)

where $N$ is the number of joints in the character skeleton. The edited motion
results by using DualMotion have a higher similarity compared with the one-
stage design approach. In addition, we analyzed the overall MSE result of
DualMotion by running a one-tailed $t$-test. The $t$-value was 2.48, and the
$p$-value was 0.01 ($p<0.05$), which revealed significant differences in the
MSE loss.

For the evaluation of the efficiency, Figure 8 shows the average time cost and
operation times (in this case, we recorded the number of mouse clicking during
the task). It illustrates that DualMotion allowed users to design motions more
efficiently. Notably, the time cost and the operation times of the free motion
design task (Figure 8 (green)) included the time for envisioning their target
motion for each participant. Therefore, the participants are able to manage
DualMotion and create the motion following their intent in a short time.

Figure 7: MSE loss results between the user design results and the references.
(R1, R2, etc are indexes of the reference motion.) Figure 8: Average time cost
(left) and operation times (right) of each task. Noted that the free design
task was only done by using DualMotion. It shows that the participant were
able to design a expected motion within few minutes.

We show the post-experiment questionnaire results below. The SUS metrics
evaluation result is shown in Table 1. All of the participants claimed that
they felt satisfaction of the overall interface, and they would imagine that
most of the people would learn to use this system very quickly. Furthermore,
86% (12/14) of the participants said that they would like to use DualMotion
frequently to design the prototype of character animations. Moreover, 79%
(11/14) and 71% (10/14) of the participants said that they were confident when
using the system and that the functions were well-integrated, respectively.
DualMotion scored 75.6 out of 100 through the SUS metrics, which implies the
overall good usability.

Lastly, the work load evaluation result is shown in Figure 9. The high score
of ‘Overall Performance’ illustrates that the participants were satisfied with
their own animations. By contrast, the low level of ’physical demand’ and
’frustration level’ implies that the large loads of labor unnecessary when
utilizing DualMotion to design a character animation.

Table 1: Results of the post-experiment SUS metrics questionnaire. $\big{\Uparrow}$ indicates higher scores are better. $\big{\Downarrow}$ for the other case. The total score is 75.6 out of 100. # |  Questions | Mean | SD
---|---|---|---
1 |  I think that I would like to use DualMotion frequently. $\big{\Uparrow}$ | 4.00 | 0.48
2 |  I found DualMotion unnecessarily complex. $\big{\Downarrow}$ | 2.07 | 0.92
3 |  I thought DualMotion was easy to use. $\big{\Uparrow}$ | 3.93 | 0.99
4 |  I think that I would need the support of a technical person to be able to use this DualMotion. $\big{\Downarrow}$ | 2.64 | 1.28
5 |  I found the various functions in this DualMotion were well integrated. $\big{\Uparrow}$ | 3.86 | 0.66
6 |  I thought there was too much inconsistency in DualMotion. $\big{\Downarrow}$ | 1.92 | 0.91
7 |  I would imagine that most people would learn to use DualMotion very quickly. $\big{\Uparrow}$ | 4.50 | 0.52
8 |  I found DualMotion very cumbersome to use. $\big{\Downarrow}$ | 2.07 | 0.99
9 |  I felt very confident using DualMotion. $\big{\Uparrow}$ | 4.07 | 0.73
10 |  I needed to learn a lot of things before I could get going with DualMotion. $\big{\Downarrow}$ | 1.42 | 0.51

###  6.2 Animation Prototype

To evaluate whether the design results are qualified enough to make a
animation prototype, we retargetted the users’ free designed skeletal motion
on to several character models with various body sizes (Michielle, Maynord,
Ninja, and Ortiz from Mixamo [2]). We observed that the animation prototype
was consistent with the user’s editorial intent. As shown in Figure 10, the
user editorial intents were a crash (a) a zombie walk (b), a standing long
jump (c), and a run with joy (d). More results are shown in the supplemental
video. We also conduct oral interviews with users after finishing their free
designed animation. Most of the users said that although they only had a rough
idea at the very first and not clearly know exactly how to implement movement
(e.g., someone wanted to design a zombie walk but did not exactly know how the
hands or head move.), they can achieved their goal and were satisfied with the
results. The results verify that DualMotion can not only supports high-
precision motion design, but can also help users successfully complete their
animations with only vague ideas and achieve satisfaction with their own work.

Figure 9: The result of raw NASA-TLX evaluation.

##  7 Discussion and Limitation

###  7.1 Applications on Mobile Platforms

In our user study, we found that DualMotion supports novices to create
character animations in two usage scenarios: 1) to create a character
animation with a clear target (given a reference motion sequence), and 2) to
create a character animation without a target. Particularly, DualMotion
reduces the operations in casual character animating. As an extreme case,
animating on mobile devices with touch screen strictly limits on the count and
accuracy of user operation. Actually verifying DualMotion’s extendibility on
tablets and smartphones is one of our future work.

###  7.2 Trade-off between Dataset Size and Computation Performance

The current prototype of DualMotion and the user study are based on a light-
weight motion database. blue(55 motion data in total) Our system updates the
projection matrix every time the camera is relocated by users and then applies
on all motion sequences in the database. The average retrieval time is around
0.84 second for each query. As the size of the database grows, the computation
performance may not be able to meet the demands of real-time interaction
sensitively. To be mentioned that the retrieval task is implemented by a
single node loop on CPU, a parallel processing pipeline on GPU is required to
increase the efficiency. Besides, we will extend the current database and
accelerate the algorithm to improve the diversity of design results.

###  7.3 User Sketches in Design Process

The current user-input stroke is made up by uniformly distributed points,
which means that the input omits the velocity of user drawing. We only enable
head and hand nodes authoring in the prototype for simplicity of the design
process; however it limits the diversity of design results. Exploration on
inputs that contain more information is yet another future work for more
professional users.

###  7.4 Creativity Support with Two-Stage Scheme

DualMotion adopted a two-stage scheme with both global and local stages [13,
24]. Along with the perception capabilities of human, designing the target
with a bottom-up scheme is difficult. For example, we usually have a gradually
clear design intention during the design process, and only an ambiguous target
in our mind at the initial step. We thought the two-stage scheme is an
approach of human-centered design thinking and could be useful in various
creativity support systems, such as illustration and story designs.

As technical limitations, the current prototype of DualMotion combined the
limbs’ motion data to the torso’s of the character model. The current
implementation may produce the artifacts such as the unnatural body balance. A
lightweight algorithm for automatically correcting the full body balance can
be considered for further development, such as a double inverted pendulum
model [14].

Figure 10: Motion retargeting results on Mixamo’s character models. The query
sketch is showed above each result which are global editorial strokes (left)
and local editorial strokes (right). The different colors in local design
stage represent the design history of various joints which are head (cyan),
left hand (violet), and right hand (pink) respectively.

##  8 Conclusion

In this work, we proposed DualMotion, a casual motion design UI for character
animations. This helps common users create character motion animation, as they
are not always able to conduct accurate and a large amounts of operations. In
this case, editing motion by using professional software, which has many
complex functions when the operation count and range are limited, poses
further challenges. The proposed system utilizes a two-stage design scheme
with global and local motion retrieval and composition to tackle the
challenging task of decomposing full-body motions into lower limb and upper
limb movements. Users are allowed to query the database with simple and rough
sketch input and retrieve desirable results by a data-driven manner. Lastly, a
user study is conducted, which verified that DualMotion is available to
supports users to create character animations in low labor demand, even with
only vague ideas.

## References

  * [1] CMU graphics lab motion capture database.  http://mocap.cs.cmu.edu./, 2022.
  * [2] Mixamo.  https://www.mixamo.com/, 2022.
  * [3] John Brooke.  SUS: A ’Quick and Dirty’ Usability Scale.  In Usability Evaluation In Industry, pages 207–212. CRC Press, 1996.
  * [4] Byungkuk Choi, Roger Blanco i Ribera, J. P. Lewis, Yeongho Seol, Seokpyo Hong, Haegwang Eom, Sunjin Jung, and Junyong Noh.  SketchiMo: Sketch-based motion editing for articulated characters.  ACM Transactions on Graphics (ToG), 35(4):146:1–146:12, 2016.
  * [5] M. G. Choi, K. Yang, T. Igarashi, J. Mitani, and J. Lee.  Retrieval and visualization of human motion data via stick figures.  Computer Graphics Forum, 31(7):2057–2065, 2012.
  * [6] Mira Dontcheva, Gary Yngve, and Zoran Popović.  Layered acting for character animation.  ACM Transactions on Graphics (ToG), 22(3):409–416, jul 2003.
  * [7] Marek Dvorožňák, Daniel Sýkora, Cassidy Curtis, Brian Curless, Olga Sorkine-Hornung, and David Salesin.  Monster mash: A single-view approach to casual 3D modeling and animation.  ACM Transactions on Graphics (ToG), 39(6):214:1–214:12, 2020.
  * [8] Martin Guay, Rémi Ronfard, Michael Gleicher, and Marie-Paule Cani.  Adding dynamics to sketch-based character animations.  In Proceedings of the Workshop on Sketch-Based Interfaces and Modeling (SBIM’15), pages 27–34, Goslar, DEU, 2015. Eurographics Association.
  * [9] Martin Guay, Rémi Ronfard, Michael Gleicher, and Marie-Paule Cani.  Space-time sketching of character animation.  ACM Transactions on Graphics (ToG), 34(4):118:1–118:10, 2015.
  * [10] S. G. Hart and Lowell E. Staveland.  Development of NASA-TLX (task load index): Results of empirical and theoretical research.  Advances in Psychology, 52:139–183, 1988.
  * [11] Edmond S. L. Ho, Hubert P. H. Shum, Yiu-ming Cheung, and P. C. Yuen.  Topology aware data-driven inverse kinematics.  Computer Graphics Forum, 32(7):61–70, Oct 2013.
  * [12] Zhongyuan Hu, Haoran Xie, Tsukasa Fukusato, Takahiro Sato, and Takeo Igarashi.  Sketch2vf: Sketch-based flow design with conditional generative adversarial network.  Computer Animation and Virtual Worlds (CAVW), 30(3–4):e1889:1–e1889:11, 2019.
  * [13] Zhengyu Huang, Yichen Peng, Tomohiro Hibino, Chunqi Zhao Zhao, Haoran Xie, Tsukasa Fukusato, and Kazunori Miyata.  dualface: Two-stage drawing guidance for freehand portrait sketching.  Computational Visual Media (CVMJ), 8:64–77, 2022.
  * [14] Jaepyung Hwang, Il Hong Suh, Gangrae Park, and Taesoo Kwon.  Human character balancing motion generation based on a double inverted pendulum model.  In Proceedings of the Tenth International Conference on Motion in Games (MIG’17), pages 11:1–11:6, New York, NY, USA, 2017. ACM.
  * [15] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3D human sensing in natural environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 36(7):1325–1339, 2014.
  * [16] Naoya Iwamoto, Hubert P. H. Shum, Wakana Asahina, and Shigeo Morishima.  Automatic sign dance synthesis from gesture-based sign language.  In Proceedings of the 2019 International Conference on Motion, Interaction and Games (MIG’19), pages 18:1–18:9, New York, NY, USA, 2019. ACM.
  * [17] Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, Shengdong Zhao, and George Fitzmaurice.  Draco: Bringing life to illustrations with kinetic textures.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’14), pages 351–360, New York, NY, USA, 2014. ACM.
  * [18] Rubaiat Habib Kazi, Tovi Grossman, Nobuyuki Umetani, and George Fitzmaurice.  Motion Amplifiers: sketching dynamic illustrations using the principles of 2D animation.  In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16), pages 4599–4609, New York, NY, USA, 2016. ACM.
  * [19] Björn Krüger, Jochen Tautges, Andreas Weber, and Arno Zinke.  Fast local and global similarity searches in large motion capture databases.  In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA’10), pages 1–10, Goslar, DEU, 2010. Eurographics Association.
  * [20] Yong Jae Lee, C. Lawrence Zitnick, and Michael F. Cohen.  Shadowdraw: Real-time user guidance for freehand drawing.  ACM Transactions on Graphics (ToG), 30(4):27:1–27:10, 2011.
  * [21] Q. L. Li, W. D. Geng, T. Yu, X. J. Shen, N. Lau, and G. Yu.  Motionmaster: Authoring and choreographing kung-fu motions by sketch drawings.  In Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA’06), pages 233–241, Goslar, DEU, 2006. Eurographics Association.
  * [22] Mengqi Peng, Jun Xing, and Li-Yi Wei.  Autocomplete 3D sculpting.  ACM Transactions on Graphics (ToG), 37(4):132:1–132:15, 2018.
  * [23] Yichen Peng, Zhengyu Huang, Chunqi Zhao, Haoran Xie, Tsukasa Fukusato, and Kazunori Miyata.  Sketch-based human motion retrieval via shadow guidance.  In Proceedings of Nicograph International (NicoInt’21), pages 42–45. IEEE, 2021.
  * [24] Yichen Peng, Chunqi Zhao, Zhengyu Huang, Tsukasa Fukusato, Haoran Xie, and Kazunori Miyata.  Two-stage motion editing interface for character animation.  In The ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA’21), pages 3:1–3:2, New York, NY, USA, 2021. ACM.
  * [25] Matthew Thorne, David Burke, and Michiel van de Panne.  Motion doodles: An interface for sketching character motion.  ACM Transactions on Graphics (ToG), 23(3):424–431, 2004.
  * [26] Nora S Willett, Wilmot Li, Jovan Popovic, Floraine Berthouzoz, and Adam Finkelstein.  Secondary motion for performed 2D animation.  In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST’17), pages 97–108, New York, NY, USA, 2017. ACM.
  * [27] Jun Xing, Rubaiat Habib Kazi, Tovi Grossman, Li-Yi Wei, Jos Stam, and George Fitzmaurice.  Energy-brushes: Interactive tools for illustrating stylized elemental dynamics.  In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST’16), pages 755–766, New York, NY, USA, 2016. ACM.
  * [28] Song-Hai Zhang, Yuanchen Guo, and Qing-Wen Gu.  Sketch2Model: View-aware 3D modeling from single free-hand sketches.  In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21), pages 6012–6021, Nashville, TN, USA, 2021. IEEE.