1182 Research  (a) We present RemoteTouch for emulating hand touch between two
remote users in an immersive 3D video communication environment as if they
were sitting on different sides of a glass window. (b) The multiple RGBD
cameras installed around a large display provide good rendering for user’s
body for immersive video communication but fail to capture and render the hand
as it closes to screen. (c) We propose dual hand representation for rendering
realistic hand as it touches the screen. The remote touch between two remote
users is tested in our prototype system.

# RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

Yizhong Zhang ∗
Microsoft Research Asia  e-mail<EMAIL_ADDRESS>Zhiqi Li ∗
Zhejiang University
Microsoft Research Asia e-mail<EMAIL_ADDRESS>work done during
internship at MSRA Sicheng Xu
Microsoft Research Asia  e-mail<EMAIL_ADDRESS>Chong Li
Microsoft Research Asia  e-mail<EMAIL_ADDRESS>Jiaolong Yang
Microsoft Research Asia  e-mail<EMAIL_ADDRESS>Xin Tong
Microsoft Research Asia  e-mail<EMAIL_ADDRESS>Baining Guo
Microsoft Research Asia  e-mail<EMAIL_ADDRESS>
∗joint first authors

###### Abstract

Recent research advance has significantly improved the visual realism of
immersive 3D video communication. In this work we present a method to further
enhance this immersive experience by adding the hand touch capability (“remote
hand clapping”). In our system, each meeting participant sits in front of a
large screen with haptic feedback. The local participant can reach his hand
out to the screen and perform hand clapping with the remote participant as if
the two participants were only separated by a virtual glass. A key challenge
in emulating the remote hand touch is the realistic rendering of the
participant’s hand and arm as the hand touches the screen. When the hand is
very close to the screen, the RGBD data required for realistic rendering is no
longer available. To tackle this challenge, we present a dual representation
of the user’s hand. Our dual representation not only preserves the high-
quality rendering usually found in recent image-based rendering systems but
also allows the hand to reach to the screen. This is possible because the dual
representation includes both an image-based model and a 3D geometry-based
model, with the latter driven by a hand skeleton tracked by a side view
camera. In addition, the dual representation provides a distance-based fusion
of the image-based and 3D geometry-based models as the hand moves closer to
the screen. The result is that the image-based and 3D geometry-based models
mutually enhance each other, leading to realistic and seamless rendering. Our
experiments demonstrate that our method provides consistent hand contact
experience between remote users and improves the immersive experience of 3D
video communication.

Human-centered computingCollaborative and social computing; Computing
methodologiesComputer graphicsGraphics systems and interfacesVirtual reality

##  1 Introduction

The ultimate goal of a teleportation system is to enable people in different
locations to meet with each other as if they were in the same room. Studies
[11, 49] have shown that to achieve this goal, the system should faithfully
capture facial expressions and body gestures and support natural eye contact
between remote meeting participants. Recent advances [39, 82] in 3D
telepresence and 3D video communication make it possible to create convincing
visual illusion that remote meeting participants are in the same room. While
these advances are encouraging, much work remains. In particular, existing
methods still cannot support physical touch between remote participants, which
is an important form of nonverbal communication for real-world social
interactions [64, 24, 19].

In this paper, we present RemotelTouch, a method for emulating hand touch
between two remote participants in an immersive 3D video communication
environment. As shown in Fig.RemoteTouch: Enhancing Immersive 3D Video
Communication with Hand Touch, our method renders the life-size remote
participant on a large display according to the local user’s view as if the
two users were sitting on two different sides of a virtual glass window. As
the remote user raises her hand to touch the screen, the local user can
perceive her action and respond by raising his own hand accordingly. The
virtual hand touch happens when both users’ hands contact the screen and their
hands overlap on the virtual glass window. The resulting touch sensation is
rendered as a screen vibration similar to the haptic rendering of iPhone
touch, although other renderings of the touch feedback are certainly possible.

A key challenge in emulating the hand touch between remote participants is the
realistic rendering of the participants’ hands and upper bodies as the hands
approach and touch the screen. For a believable touch experience, the
rendering needs to be both realistic and seamless as the hand touch the
screen. This rendering task is highly non-trivial. Existing image-based
rendering solutions such as [39, 82] can realistically render the participants
hands when they are positioned away from the screen (above a distance
threshold) and thus visible by the RGBD cameras that provide the color and
depth video feeds to the rendering algorithms. However, these rendering
solutions stop working when the hand is too close to the screen (below the
distance threshold and thus invisible to the RGBD cameras). Although 3D model
based solutions such as [78] can successfully track the hand motion as the
hand reaches the screen, the rendering quality is unsatisfactory and inferior
compared to image-based rendering solutions.

To tackle this challenge we introduce a dual representation of the hand
consisting of three components: an image-based model, a 3D geometry-based
model, and a distance-based model fusion scheme that seamlessly combines the
image- and geometry-based models. The first component, the image-based model,
is a Lumigraph of the hand constructed using image and geometry data obtained
by a set of RGBD cameras. Similar to [82], we use a sparse set of RGBD cameras
installed around the screen to capture the image and geometry data needed by
the Lumigraph construction. The second component of the our dual
representation, the 3D geometry-based model, is obtained by tracking the
movement of the hand skeleton (including the arm and fingers) with a sideview
camera and rigging the resulting skeleton. Finally, the third component of the
dual representation, the distance-based model fusion, ensures the seamless
fusion of the image- and geometry-based models. Specifically, when the hand is
away from the screen, the image-based model is used for rendering. As the hand
moves closer to the screen, the geometry-based model is given increasingly
more weights in rendering and when the hand is completely invisible from the
RGBD cameras, the geometry model gains the full weight. The model fusion
scheme is also responsible for fitting the geometry and texture of the
geometry-based model to the image-based model in the initial phase of the hand
touch operation, when the hand is still away from the screen and hence fully
visible from the RGBD cameras.

We implement our method and test the virtual hand touch between participants
sitting at two locations. Our studies demonstrate that our design provide
immersive communication experience. The virtual hand touch enabled by our
method efficiency enhances the nonverbal communication of the users.

##  2 Related Work

Our work is related to multiple research fields such as 3D video
communication, VR/AR, and human-computer interaction. In this section, we
briefly review the techniques and systems that are closely related to our
work. For comprehensive surveys of the advances in related fields, readers are
referred to [8, 6, 60, 18].

###  2.1 Immersive 3D Video Communication

Many techniques and systems have been proposed for immersive teleconferencing
between the participants at different locations. Early methods [4, 21, 59]
enable 3D video communication by capturing the geometry and appearance of the
participants and rendering them into one shared virtual environments. Later, a
number of approaches have been introduced to enhance the gaze contacts between
the participants in various meeting setups [51, 37, 32, 56, 38, 22, 77, 73,
34, 81] or enhance the rendering quality of human characters [16, 47, 85, 54,
17, 23]. Most recently, Lawrence et al. presented Project Starline [39] for
high fidelity 3D communication between two participants at different locations
with the help of dedicated hardware setup for 3D video capturing and light
field display. Zhang et al. introduced VirtualCube [82] that supports
immersive 3D video communication between remote participants in various
meeting setups based on off-the-shelf devices. Although these techniques and
systems efficiently improve the nonverbal communications between the
participants with natural gaze contact and high fidelity facial expression,
none of them supports social touch between the users. Besides, their setups
cannot capture human hands as they move close to screen.

Different from existing 3D video communication solutions, our method focuses
on emulating hand clapping between remote users in immersive 3D video
communication. We present a dual representation for modeling and rendering
user’s hand to support visually and physically consistent hand touch
experience.

###  2.2 Hand Rendering in Remote Collaboration

Existing methods of remote collaboration with hands can be categorized into
two categories: those designed for the shared tabletop scenario, and those for
the shared whiteboard scenario. For the shared tabletop scenario, many
approaches [28, 70, 15, 30, 33, 20, 41, 53, 29] have been proposed for
capturing and visualizing the hand gestures, in which the hand embodiment
captured by the cameras placed above the table is rendered over the remote
table via different display devices (e.g., projector, screen, or other devices
[53, 41]). Le et al. [40] developed a mobile setup by capturing the user’s
hand interactions above a tablet screen via the built-in front facing camera
and a mirror. Several methods [28, 15, 30, 41, 29] apply another vertical
screens to display the remote participants’ body and face to enhance the user
experience. Leithinger et al. [41] developed a physical 3D display for
rendering 3D arm embodiment and objects captured from the remote side. Iwai et
al. [29] further improved the user experience by rendering the upper limb
image of the remote participants according to its relative position to the
shared document on the local desktop and extending the rendered limb image
from the vertical image of the remote participant. All these methods assume
that all participants’ views are on the same side of their upper limbs. As a
result, they cannot be used in our scenario where the local participant’s view
is on the opposite side of the remote participant.

In the shared whiteboard scenario, a large display at each site is typically
used as a semi-transparent whiteboard for local users to input the contents
and for rendering the remote collaborators standing at the opposite side of
the whiteboard. Early methods [27, 71, 69] developed for this scenario place
the camera behind a semi-transparent screen to directly capture the hand
gesture, face and body of the remote participant from local user’s view.
Although these methods demonstrate convincing 3D remote communication and
collaboration capabilities, their device setup requires larger spaces and
specialized screens. Later, a few methods place the cameras in front of the
screen for capturing the remote participants. For example, the 3D-Board system
[84] uses two RGBD cameras installed on the top corners of the large display
to capture the 3D geometry and texture of the remote participant and render it
from local user’s view. However, the cameras fail to capture the user’s hand
when it is close to or touches the display. Higuchi et al.[25] use a single
RGBD camera mounted on the side of a large display for capturing the
participant. Unfortunately, the noisy and incomplete 3D model captured by the
RGBD camera fails to realistically reproduce the appearance of the remote
participant from local user’s view. Wood et al. [78] tracked the 3D hand
motion of a remote user via a depth camera and then render a non-
photorealistic 3D hand model driven by the tracked motion over the shared
workspace.

Similar to the shared whiteboard scenario, our work aims to emulate the hand
clapping of two participants sitting at the opposite side of a virtual glass.
However, different from all these methods that model the hand either by image-
based representation or geometry-based 3D model and are designed for offering
accurate pointing position or hand pose, our method tries to deliver a
visually and physically consistent touch experience. A dual hand
representation and rendering scheme is proposed for realistic hand rendering.

###  2.3 Social Touch in VR and AR

A number of works [3, 62, 74, 57, 68] investigate the impact of social touch
in VR and AR and validate that the social touch can enhance the user
experience in remote communication and collaborations. Other works develop
techniques for emulating social touch between remote users [50, 7, 52]. Nakani
et al. [50] used a robot hand to emulate hand shaking between remote users and
proved that hand touch improves the experience of video communication. Bevan
et al. [7] set a humanoid robot as the avatar of a remote user for emulating
the hand shaking between the remote users. Oh et al. [52] presented avatar
animation techniques to support handshaking between remote users in a shared
VR environment. Gallace et al. [18] presented a survey of the devices and
studies of social touch in VR applications. Different from these methods that
only focus on emulating tactile feedback of hand touch between remote users,
our work aims to provide hand touch with consistent visual and haptic
experience in immersive 3D video communication.

###  2.4 3D Hand Tracking

As a critical technique for animating and rendering the synthetic hand in
virtual reality and remote collaboration, vision-based 3D hand tracking has
been extensively studied in computer vision field in the past decades [26]. A
set of real-time methods [76, 72] have been developed for tracking 3D hand
poses from monocular video at the cost of pose accuracy. Other methods [35,
44] improve the robustness and accuracy of real-time 3D hand tracking with the
help of multiple-view video input. However, the specialized multiple view
setup limits their usage in many applications. Smith et al. [65] introduced a
multi-view method for high accurate 3D hand tracking. However, their method
cannot be performed in real time.

In our work, we selected an off-the-shelf solution provided by LeapMotion [1]
to track 3D hand pose of the user due to the simplicity of the device setup
and software development. Since 3D hand tracking is not the contribution of
our work, any other real-time 3D hand tracking methods can be used in our
method.

###  2.5 Haptic Rendering

Various force feedback devices have been developed for rendering haptic in
human-computer interaction tasks, including special-purpose haptic controllers
[10, 48, 45, 5], in-air haptic [66, 12], multi-sensory wristband [55], and
physical geometry primitives [14]. The goal of our work is not to develop new
force feedback devices or haptic rendering methods. We thus leverage existing
vibration solutions to provide the haptic feedback in our solution.

Several approaches have been developed for detecting the tactile interactions
between the user’s hand and physical surfaces [36, 79] or capacitive
touchscreens [67], which are then used as the input for different applications
[79, 2, 61]. In this work, we apply off-the-shelf hardware (e.g., a Leap
Motion camera) to track 3D hand motion and detect the contact between hand and
screen.

##  3 Dual Representation for Emulating Touch

Our work in this paper extends existing immersive video communication systems
by providing emulated hand touch. For immersive experience, our system equips
with a large display in front of each user, as shown in Fig. RemoteTouch:
Enhancing Immersive 3D Video Communication with Hand Touch (c). The display
has dual purposes: it not only displays the life-size 3D portrait video of the
remote user with respect to local user’s viewpoint but also serves as the
physical media for hand contact. A set of RGBD cameras are installed around
the display to capture the appearance and rough geometry of the user for
rendering the 3D video of the user, and a side camera is used to track hand
motion. Although our method simplifies the surrounding three-display setup in
[82] to one frontal display, the 3D rendering of the remote user displayed on
the screen will follow the local user’s viewpoint and still provides immersive
experience for tele-communication.

Figure 1:  The body and hand are modeled by an unstructured lumigraph model
that consists of a geometry proxy and multiple RGB images (left). The geometry
proxy is aligned with the target view and refined from the input RGBD images.
A neural lumigraph-based rendering algorithm (middle) is used to render the
lumigraph to RGB$\alpha$ image at the target view (right).

A key challenge for hand touch emulation is high quality rendering of the
users’ hands and upper body. A realistic and temporally-smooth hand rendering
throughout the virtual touch process is of paramount importance to maintain
the immersive experience. To achieve this goal, we propose a dual hand
representation which consists of an image-based model that uses a set of
multiview images for rendering and a 3D geometry model that provides mesh
rigging and rendering. A distance based fusion algorithm is applied to achieve
seamless fusion of the renderings from the two models of our dual
representation.

###  3.1 Dual Representation of Hand

A touch process can be roughly divided into three phases: the starting phase
where the users begin to raise their hands, the approaching phase where the
users move the hands towards the display, and the ending phase where the palms
touch the screens and physical feedback is triggered (if real and rendered
hands coincide).

Our dual hand representation has two rendering models, one image-based and
another 3D geometry-based. The former takes the multiview RGBD images captured
by the cameras installed around the display as input and synthesize the
target-view hand image. It provides high-quality, photorealistic results when
the hand is well within the shared field of view of all RGBD cameras (e.g., in
the starting phase). The latter uses a predefined hand mesh model driven by
the real hand’s motion for rendering, which is a realistic substitute when the
hand is completely invisible to the cameras (e.g., in the ending phase). To
seamlessly fuse these two models and their renderings, we adapt the texture of
the mesh model to the real hand’s appearance and develop a distance-based
fusion algorithm to smoothly transit the former to the latter when the hands
get closer to the displays (e.g., in the approaching phase).

####  3.1.1 Image-based Model

When hands and other parts of the human body are visible to the RGBD cameras
(we assume the latter to be always visible), an image-based model produces the
best rendering result due to the model’s capability to support photorealistic
image synthesis and handle view-dependent texture appearance commonly seen in
non-Lambertian surfaces including the human skin. Image-based model [42, 9,
63] has been extensively studied in the literature. In particular, the recent
image-based rendering solutions [39, 82] leveraging advanced depth sensing and
deep learning techniques have shown high-quality, real-time rendering of human
in the 3D video communication scenario.

As shown in Fig. 1, our image-based model is essentially a Lumigraph
consisting of a geometry proxy and associated image data. The geometry proxy
serves the vital function of determining how to compute the color of each
pixel of the rendered image. The geometry proxy we built is a view-dependent
one obtained as follows. We start with image and depth data from the RGBD
cameras with the background pixels removed. For each view, we computes an
initial depth map by fusing the multiview depth maps from the RGBD cameras and
then refine it with a multiview-stereo (MVS) neural network [80, 46, 82] using
only the RGB images from these cameras. The image data of the Lumigraph are
simply the RGB images of the multiple cameras that we place around the
display. This Lumigraph is called an Unstructured Lumigraph [9] in the
literature.

Figure 2:  Comparison of the original rendering method in [82] and ours. (a)
and (c) are rendering results generated by our method and (b) and (d) are
images rendered by the original method. It is clear that our method generates
better rendering results when the user’s hand is in front of his body.

Rendering this Lumigraph for a target view can be done in two steps. First we
use the geometry proxy to warp the RGB images of the RGBD cameras to this
view. Thus for each pixel in the target view, we have multiple pixel colors
from the warped RGB images. The final pixel color is a weighted average of
these pixel colors, computed by using the geometry proxy as a guide. This
weighted average is usually computed using the conventional Unstructured
Lumigraph algorithm [9]. In this work, we compute the weighted average using
the neural Lumigraph based rendering method described in [82], which uses a
neural network called Lumi-Net to compute the weighted average.

This neural Lumigraph based rendering method generally works well for human
body. However, under severe self-occlusion, e.g., when the user waves hand in
front of the body or reaches out hand towards the display, the rendering
quality of both the hand and the occluded body parts are not satisfactory. We
make two algorithmic modifications to improve the quality. The first one is a
new edge-preserving depth initialization strategy. We identified that initial
depth maps contain blurry edges along depth discontinuity due to the
misalignment between the projected multiview depth maps. We replace the
averaging operation for depth fusion with a min-filter, which brings obvious
hand rendering improvements in practice. Second, we add a new loss function
dedicated to the hand region to train the neural networks. Specifically, we
detect hand bounding boxes on the target-view images in the training data, and
apply a perceptual loss [31] between the final network output and the ground
truth. Fig. 2 shows a visual comparison between the original method and ours,
which shows the improved hand rendering quality in our system.

Conceptually, one can certainly use other image-based models for our proposed
dual representation.

Figure 3:  Shape adaption for model fusion. (a) With the scales computed from
the captured skeleton of the real hand, we scale the hand template (in red)
with linear blend skinning to obtain the adapted hand model (in blue) (b)
Before shape adaption, the shape of the hand template does not match to the
user’s hand. (c) After shape adaption, the hand mesh fits to the user’s hand.

####  3.1.2 3D Geometry-based Model

As the user’s hand moves close to the display, the palm region becomes
invisible to all the RGBD cameras. To render the hand palm under such
situations, we model the hand with a rigged 3D mesh template.

The 3D hand mesh has a predefined topology and texture map representing the
skin surface of a hand at rest pose. It is also associated with a set of 3D
skeleton and per-vertex skinning weights with respect to the bones of the
skeleton.

To obtain hand rendering using this mesh model, we track the 3D skeleton of
user’s real hand by a hand tracking camera, which can successfully reconstruct
the position and pose of the real hand as it moves close to the screen and
touches the display. The hand mesh and tracked 3D skeleton share the same bone
structure so that the former can be easily driven by the latter. Given the
local hand motion (bone transformations) reconstructed by the 3D skeleton
tracked in each frame, we first rig the hand template mesh using standard
linear blending skinning [43]. Let $V_{i}$ be a vertex on the rest mesh, its
transformed potion after skinning is calculated as

$V_{i}^{\prime}=\sum_{j}\omega_{ji}\mathbf{T}_{j}V_{i}$ (1)

where $\mathbf{T}_{j}$ is the tracked rigid transformation of bone $j$ and
$\omega_{ji}$ is the skinning weight of vertex $v_{i}$ with respect to bone
$j$. This way, the skin deforms in a smooth manner. After rigging, we
transform the mesh to the real hand’s 3D position in the global virtual
environment and render it from the target viewpoint of the remote user.

In our current implementation, we captured the skeleton of the medium-sized
hands of a user via LeapMotion and asked an artist to manually modeled the
hand mesh and associated skinning weights based on the captured skeleton.

The texture map and hand shape of the mesh model are not fixed, but are
adjusted at runtime to fit the user’s real hand. This process will be
described the next section.

####  3.1.3 Model Fusion

Given two hand representation models, we seamlessly combine them to form our
full model and get the final rendering.

##### Shape Adaption

Prior to touching actions, we first adjust the shape of the hand template to
fit the shape of the user’s real hand. Given the skeleton of the real hand
that share the same topology of the skeleton of the hand template at the same
pose, we first scale each bone of the template skeleton so that the length of
each bone matches the target skeleton. We then update the transformation
matrix $B_{j}$ that transform the local coordinate system of each bone to the
global coordinate system due to the changed bone lengths. After that, we scale
the hand mesh template by transforming each vertex $V_{i}$ to its new position
$V_{i}^{\prime}$:

$V_{i}^{\prime}=\sum_{j}\omega_{ji}\mathbf{B}_{j}^{\prime}\mathbf{S}_{j}\mathbf{B}_{j}^{-1}V_{i},$
(2)

where $\omega_{ji}$ is the skinning weight of the vertex $V_{i}$ with respect
to the bone $j$. $\mathbf{S}_{j}$ is a uniform scaling matrix determined by
the scaling factor of the bone $j$. Fig. 3 shows the hand mesh before and
after shape adaption.

Figure 4:  Appearance adaption for model fusion. (a) The image generated by
image-based body and hand rendering. The mask is the projection of 3D
geometry-based hand model. (b) The rendering of the geometry-based hand model
without appearance adaption. The color of the hand differs a lot from the real
hand. (c) With appearance adaption, the rendering of hand is much closer to
the real hand.

##### Appearance Adaption

Prior to touching actions, we first personalize the mesh model by adjusting
its appearance to fit the renderings of the image-based model. This is an
automatic process done online whenever the 3D skeleton of the hand is tracked
and the hand palm is rendered by the image-based method while it is well
within the field of views of the RGBD cameras (_e.g._ , in the touch starting
phase). To fit the appearance to the rendered hand palm, we adjust the overall
color of the hand template texture according to the hand palm image rendered
from the image-based model. Specifically, let the color and depth images
rendered by the image-based model be $\mathbf{I}_{i}$ and $\mathbf{D}_{i}$. We
can also get the color and depth images of the 3D geometry-based model
$\mathbf{I}_{g}$ and $\mathbf{D}_{g}$ by rendering the mesh template under the
tracked 3D pose using the same camera projection parameters of the image-based
model. The region with consistent depth values on $\mathbf{D}_{i}$ and
$\mathbf{D}_{g}$ (depth difference $<5cm$) is the hand region on the color
images, labeled as $\mathbf{I}_{i}^{H}$ and $\mathbf{I}_{g}^{H}$. This will
help to filter out small misalignment pixels. We minimize the color difference
between $\mathbf{I}_{i}^{H}$ and $\mathbf{I}_{g}^{H}$ by calculating a color
transformation inspired by [58], then apply this color transformation to
refine the texture $\mathbf{T}$ of the hand mesh model. In this way, the 3D
geometry-based model could have similar visual experience with the image-based
model.

Specifically, we first convert $\mathbf{I}_{i}^{H}$ and $\mathbf{I}_{g}^{H}$
to the $L\alpha\beta$ color space. For each channel of $\mathbf{I}_{g}^{H}$,
we can transform the color $c_{g}$ of each pixel to match $\mathbf{I}_{i}^{H}$
via

$c_{g}^{\prime}=\frac{\sigma_{i}}{\sigma_{g}}(c_{g}-m_{g})+m_{i}$ (3)

where $m_{i}$, $m_{g}$ and $\sigma_{i}$, $\sigma_{g}$ are the mean and
standard deviation values of $\mathbf{I}_{i}^{H}$ and $\mathbf{I}_{g}^{H}$ on
the channel. After this color transformation, the resultant image
$\mathbf{I}_{g}^{H\prime}$ has the same mean and standard deviation with
$\mathbf{I}_{i}^{H}$, making them look similar when transformed back to the
RGB color space. As $\mathbf{I}_{g}^{H}$ is produced by rendering the 3D hand
mesh with texture $\mathbf{T}$ without lighting, the same color transformation
can be applied to $\mathbf{T}$ as well, which gives rise to the adjusted
texture $\mathbf{T^{\prime}}$. With this texture, the rendered hand using the
3D hand model will have similar visual appearance with the hand of the image-
based model. Fig. 4 illustrates the rendered hand mesh before and after our
shape and appearance fitting. Visually inspected, the fitted one well matches
the real hand.

Figure 5:  The process of distance-based hand rendering fusion. As the hand
approaches the screen, the hand skeleton will be detected and rendered and get
more weight in the fused image. When the hand is very close to the screen and
no longer visible in RGBD cameras, hand is not visible in image-based
rendering and mesh-based rendering will be selected.

##### Distance-based Rendering Fusion

Given two renderings $\mathbf{I}_{i}$ and $\mathbf{I}_{g}$ from the two
models, we fuse them to get the final rendering $\mathbf{I}_{f}$ according to
the distance of hand to screen, denoted as $d$. Note that $\mathbf{I}_{i}$ is
an RGB$\alpha$ image predicted by the neural network in our image-based model.
For $\mathbf{I}_{g}$ which is the rendering of the 3D hand mesh, we also
append it with a distance-based alpha channel to blending it with
$\mathbf{I}_{i}$. Specifically, we calculate the alpha value $\alpha_{g}$ of a
pixel on $\mathbf{I}_{g}$ as

$\mathbf{\alpha}_{g}=\begin{cases}(d_{max}-\hat{d})/(d_{max}-d_{min})&\text{for
a hand pixel}\\\ 0&\text{for a non-hand pixel}\end{cases}$ (4)

where $\hat{d}=clip(d,[d_{min},d_{max}])$ is the hand distance clipped to a
predefined range $[d_{min},d_{max}]$.

To blend the two RGB$\alpha$ images, we set $\mathbf{I}_{i}$ as the base layer
and blend $\mathbf{I}_{g}$ onto it, as the hand 3D model is in front of the
portrait image. The overlay blending mode [75] is used here. Specifically, the
alpha and color value ($\mathbf{\alpha}_{f}$, $\mathbf{c}_{f}$) of a pixel on
the final image $\mathbf{I}_{f}$ is calculated using the following equations:

$\begin{split}&\mathbf{\alpha}_{f}=\mathbf{\alpha}_{g}+\mathbf{\alpha}_{i}(1-\mathbf{\alpha}_{g}),\\\
&\mathbf{c}_{f}=\frac{\mathbf{c}_{g}\mathbf{\alpha}_{g}+\mathbf{c}_{i}\mathbf{\alpha}_{i}(1-\mathbf{\alpha}_{g})}{\mathbf{\alpha}_{f}}\end{split}$
(5)

where ($\mathbf{\alpha}_{g}$, $\mathbf{c}_{g}$) and ($\mathbf{\alpha}_{i}$,
$\mathbf{c}_{i}$) are alpha and color values of the corresponding pixels on
$\mathbf{I}_{g}$ and $\mathbf{I}_{i}$, respectively.

As we can see from Eq. (4) and (5), when $d\geq d_{max}$, we have
$\alpha_{g}=0$, $\alpha_{f}=\alpha_{i}$ and $\mathbf{c}_{f}=\mathbf{c}_{i}$,
_i.e._ , the final rendering simply takes the output of the image-based model.
When $d\leq d_{min}$, we have $\alpha_{g}=\alpha_{f}=1$ and
$\mathbf{c}_{f}=\mathbf{c}_{g}$ for the hand region, _i.e._ , the mesh-based
rendering is selected. For $d_{min}<d<d_{max}$, which predominately happens in
the approaching phase of a touch process, a smooth transition takes place. The
fusion process is visualized in Fig. 5.

In practice, we obtain $d$ for each frame based on the averaged joint
positions obtained by the 3D hand skeleton tracker, and we empirically set
$d_{max}=0.4m$ and $d_{min}=0.2m$ in our system. As the final rendering result
$\mathbf{I}_{f}$ contains alpha channel as well, we can compose the hybrid
human portrait image with the rendering of a 3D background scene very easily.

###  3.2 Touch Emulation

Beyond visual interaction, our system provides a physical experience of hand
touching. We aim to mimic real-world situations where people’s hands can
receive physical feedback when they touch each other. While the types of
touching are various and the corresponding tactile sense are different, we
emulate a simple yet commonly-seen scenario, _i.e._ , hand clap, where the
participants feel an impact force and vibration when two hands slap each
other.

##### Physical and Virtual Design

To emulate physical touch, we use the screen as the physical media for hand
contact. During communication, the two users can reach out their hands to the
screens and gently clap each other’s rendered hands. Apart from the natural
sense of touch when the hand reaches the screen, we provide additional
physical feedback when mutual touch is detected.

To realize such a process, we found that maintaining visual-physical
consistency is crucial. When the users reach out their hands towards each
other, the real and rendered hands on each site should finally coincide and
mutual touch should be triggered. This requires a seamless integration of the
virtual and physical spaces. To achieve this, we transform the physical spaces
of two sites into a unified global virtual environment where the surfaces of
the two screens overlap, as shown in Fig. 6. Under this setup, the content
shown on the screen of each site should be a projection of the virtual space
behind the screen, as if the users were seeing and performing hand clapping
with each other through one transparent glass between them.

Figure 6:  The transformation between the physical space and the global
virtual space. The position of TV screen from two sites should be aligned in
the virtual environment.

It should be noted that since the screen displays are involved in the touch
emulation process and their positions matter, our physical and virtual space
design described above is different from existing 3D video communication
systems such as [82, 39] which have no considerations regarding visual-
physical consistency. In these systems, the displays are simply visualization
portals and they do not necessarily align with each other in the virtual space
(the screen positions of [82] are determined by other factors such as distance
between users; the system of [39] places the display on each site at the
remote user’s position).

##### Mutual Touch Detection and Feedback

We define mutual touch as the situation where the users’ hand palms reach the
screen and overlap each other’s rendered hand. Note that if the real and
rendered hands do not overlap for some reason, we do not call it mutual touch
even if both palms are on the screen. Our system detects mutual touch by
calculating the overlap of two hands. Specifically, each site keeps
transferring the tracked 3D hand skeleton to the remote site. For each site,
we identify the joints of the local and remote hand skeletons that are within
a distance threshold to the screen, then project them onto the screen plane to
get two bounding boxes, and finally compute the area of bounding box overlap.
If the overlapping area is larger than a threshold, a mutual touch is
detected. In our implementation, the joint-to-screen distance threshold is set
as 2$cm$ and overlapping area threshold is $50cm^{2}$. In practice, this
mutual touch detection method can be replaced by other simple alternatives,
such as hardware-assisted touch area identification.

Once mutual touch is detected, the system immediately provides additional
physical feedback to mimic the feeling of hand clapping. Specifically, a
vibration motor tightly attached to the display (Fig. 7) is triggered which
incurs screen vibration that can be felt by hand. Despite of the gap between
our simulated vibration and the real sense of impact force and vibration in
reality, we found this simple design has already improved the immersive
experience and successfully delivers a sense of touch.

###  3.3 Implementation Details

In this section, we present the details of our system including hardware setup
and calibration as well as software implementation.

##### Hardware Setup

Figure 7:  The hardware setup of our system. (a) and (b) illustrates the
device layout in our design. (c) is an implementation of our design. (d) shows
the vibration motor attached to the corner of the screen.

As shown in Fig. 7, our system on each site is equipped with a 65-inch LCD
display and six Azure Kinect RGBD cameras. The cameras are installed on a
stand behind the display and their positions are adjusted to be near the
corners and the center of top and bottom boundaries of the display. A
vibration motor controlled by Arduino is rigidly attached to the top-left
corner of the screen for offering touch feedback. During remote communication
and hand touch, a user sits on a seat that is about $0.7m$ away from the
display center. A Leap Motion camera is placed besides the hand for 3D hand
tracking. The camera is about $0.5m$ away from the screen and faced to the
center of the screen region to which the user’s hand can easily touch. In this
way, the Leap Motion camera can accurately track the user’s hand pose as it
approaches and touches the screen. All these devices are connected to a PC and
controlled by our system.

##### Device Calibration

We calibrate the devices at each site after installation so that all devices
and the captured data (_i.e._ , the RGBD frames and 3D hand skeleton of the
participant) are all aligned in a unified local coordinate system, which is
illustrated in Fig. 7 (c).

We calibrate Azure Kinect cameras using the method proposed in [83]. After
that, we manually measure the distance of each Kinect camera to the screen
corners and compute a transformation to align the calibrated six Kinect
cameras into the local coordinate system from the measured distances. For the
Leap Motion camera, we follow the method in [2] for relative pose calibration.
Specifically, we render a regular grid of 2D points on the screen with known
3D positions $p^{i}$ in the local coordinate system and then use the index
finger of a hand to touch each point in a predefined order and record the 3D
positions $q^{i}$ of the finger tip defined in the Leap Motion camera space.
The transformation $T$ from the Leap Motion camera space to the local
coordinate system can then be computed by
$T=\arg\min_{T}{\Sigma_{i}||p^{i}-T*q^{i}||}$.

##### Viewpoint Computation

To compute the user’s viewpoint defined in the local coordinate system, we
detect the eye positions in the RGB frames captured by multiple Kinect cameras
using the method of [13]. The 3D positions can then be computed from their 2D
eye positions in the multiple views by triangulation. After that, our method
takes the middle point of two eyes as the user’s 3D viewpoint. To obtain the
target viewpoints for rendering in the local coordinate systems, we transform
the detected local user viewpoints of the two sites to the global virtual
environment and then transform them to each other’s local coordinate system.

##### Software Workflow

The workflow of our system in action is shown in Fig. 8. In our system, each
site has a sender and a receiver. During online communication, the sender
first computes the local user’s viewpoint and transmit it to the remote site.
It then captures the RGBD frames and 3D hand skeleton of the user and then
renders the portrait (_i.e._ , body and hand) image at the received target
viewpoint based on image-based representation described in Section 3.1.1.
After that, the sender transmits the rendered portrait images and 3D hand
skeleton to the receiver at the remote site. The receiver animates the
geometry-based 3D hand model with the received 3D hand skeleton and then
renders its image. Finally, the receiver performs the distance-based fusion to
blend the hand images rendered from two representations (Section 3.1) and
composes the rendering results with the image of the 3D background scene to
generate the final image. Meanwhile, the receiver detects mutual hand touch of
two users with the algorithm described in Section 3.2 and triggers the
vibration motor once mutual hand touch happens. Our system executes all
software modules (_i.e._ image-based rendering, hand tracking, geometry-based
hand rendering, and touch detection) in different threads in parallel on each
site.

Figure 8:  The workflow of our RemoteTouch system, which consists of a sender
and a receiver at each site. The colorful blocks refer to input and output
devices and the grey blocks are software components in our system.

##  4 Experiments

We implemented two instances of our system at two sites, each with a PC that
has an Intel Core i9-10980XE CPU, 64GB memory and two Nvidia GeForce RTX 3090
GPUs. The two PCs are connected to the LAN network with 1Gbps bandwidth.
During communication, our method takes RGBD frames of four Azure Kinects that
are on the opposite side of touching hand for rendering (_e.g._ , two central
and two right ones for the left hand). This camera configuration can minimize
the occlusion region caused by the raised hand in the captured frames and thus
generates better rendering results than other choices. For viewpoint
computation, we assume the user’s head does not move a large distance during
the communication and thus compute the user’s viewpoint only once at the
beginning of the communication.

##### System Performance.

For image-based body and hand representation, the Azure Kinect cameras capture
synchronized RGBD frames at 30fps, with RGB and depth at $2560\times 1440$ and
$512\times 512$ resolutions respectively. The image-based modeling and
rendering algorithm takes nearly 150ms to process one frame (from raw data
captured by Azure Kinect to synthesized portrait image sending to network).
Since each step in the algorithm can be executed in parallel, the system
renders the portrait and hand image at 30FPS. The end-to-end delay of portrait
image rendering and transmission is around 400ms.

For geometry-based hand representation, the Leap Motion tracks hand skeleton
around 50Hz and the 3D hand rigging and rendering can be performed on GPU in
real time. The total delay of the geometry-based hand rendering and network
transmission is around 250ms. We synchronize the rendering and display of
these two representations with a timestamp on each frame and fuses the images
of two representations with the closest timestamp in the receiver at 30FPS.

For touch emulation, the touch detection is performed at 50Hz. The delay from
the vibration triggering to physical motor vibration is around 60ms, which is
hardly noticed by users.

##### Evaluation of Dual Hand Representation.

Fig. 9 (a-c) illustrates the rendering frames of the participants sampled in
different hand touch phases. Note that our dual representation and model
fusion algorithm generates convincing rendering results as the hand is at
different distance to the screen. When the hand touch the screen, the palm
region is missing in the image-based rendering result, as the hand is
invisible to source view cameras, as shown in Fig. 9 (d). In this case, our
dual representation of hand successfully render the final portrait image by
fusing the 3D geometry-based model into the image-based rendering, as shown in
Fig. 9 (c). Fig. 9 (e) show the hand without appearance adaption to the
geometry-based model, which looks unnatural compared with other parts of the
body. Our appearance fitting scheme is robust to different skin tones.

Figure 9:  The rendering of the subjects with different skin tones, where the
hand are rendered by image-based model (a), fused dual representation (b), and
geometry-based model (c) where the hand touched the screen. Our dual
representation and fusion scheme generate convincing results for different
poses. When hand touch the screen, the hand is not correctly rendered by the
image-based model (d) as the hand is invisible in source view cameras. If
appearance adaption is not applied to geometry-based model (e), the original
color of the hand differs a lot from the hand.

##### User Experience.

We invited 26 participants to try and evaluate our touch-enhanced 3D video
communication system. All of them have experience on commercial video
conferencing software on PCs or cellphones but are the first-time user of our
system. During the experiment, our developer sits in one site and the user
sits in the other site. After a short introduction, the two users can chat
with each other and try the hand clapping via our system. After the
experiment, we ask each user to provide comments on our system and rank the
impact of touch experience to remote video communication. Among all 26 users,
24 of 26 ($92\%$) participants agreed or strongly agreed that the remote touch
enhances the experience and decreases mental distance between the remote
users.

To further evaluate the contribution of our dual hand representation to the
user experience, we let all 26 participants to try the hand clapping rendered
with image-based representation only and with our dual-hand representation.
Our investigation demonstrated that all users agreed that our dual hand
representation enhances the touch experience compared with image-based
representation alone, even if they noticed the artifacts of hand misalignment
when their hand approached to the screen. Several users with computer graphics
background mentioned that they expected better portrait image quality, which
may significantly improve their experience. They also expected better hand
tracking accuracy and less system latency. Despite these shortcomings, all
users still liked the feeling that they can visualize the hand and arm in the
full hand clap process. They never have such experience in other remote
communication applications.

Finally, we asked the user to evaluate the impact of vibration for touch
experience by testing the hand clapping with vibration and without vibration
feedback. 25 of 26 ($96\%$) participants commented that the vibration makes
the hand clapping more attractive. One user mentioned that the vibration made
her believe that the touch did happened. 7 users mentioned that they are
excited about the idea haptic feedback when touching and expect more kinds of
physical feedback, such as the sound of clapping hands. 5 users expect more
kinds hand touching gestures to be supported, such as a fist bump.

##  5 Conclusion and Future Work

We present a method for emulating hand touch (“the high five”) between two
remote participants. The key of our method is a dual representation of hand
that combines the advantages of the image-based and 3D geometry-based
representation and a distance-based fusion scheme for rendering. We also
design a touch emulation scheme that map the two remote site into one shared
environment. Our solution offers a visual-physical consistent touch experience
for immersive 3D video communication. We validate our dual representation for
hand, and our experiments demonstrate that our method reduces the mental
distance between the remote users and enhances the user experience in
immersive 3D video communication.

Currently, our method only supports hand touch between two remote users.
Extending our method to hand touch between three or more remote users would be
an interesting topic worth further exploration.

The misalignment between geometry-based hand and real hand in our prototype is
mainly due to the inaccurate 3D hand tracking provided by the LeapMotion. A
more accurate real-time hand tracking method will resolve this issue. For
model fusion, our method adapts the hand shape according to the bone length
and fails to compensate the detailed shape difference between the real hand
and hand mesh template. Also we use the same texture map to approximate the
detailed appearance of different hands. It is interesting to explore how to
capture and reconstruct the detailed geometry and texture of the real hands on
the fly.

We also would like to further improve the hardware and software design of our
method to facilitate better touch experience in 3D video communication, which
includes simplifying the device setup for data capturing, integrating other
force feedback devices, and reducing the system delay for rendering and
network transmission. Besides these improvements, a possible application of
our method is the collaborative whiteboard system. Another possible research
direction is to enable more social touch behaviors in immersive 3D video
communication.

### Acknowledgment

We would like to thank Yuxiao Guo, Guojun Chen for neural network conversion
with NNFusion, Yangyu Huang for optimizing the training and run-time code of
neural lumigraph algorithm, Haowen Xu for his support in camera calibration,
all colleagues for their help in data capturing and useful discussions, and
our users for their valuable feedback of our RemoteTouch system. We also thank
the anonymous reviewers for their helpful suggestions for our paper.

## References

  * [1] Ultraleap, how hand tracking works.  https://www.ultraleap.com/company/news/blog/how-hand-tracking-works/.
  * [2] K. Ahuja, P. Streli, and C. Holz.  Touchpose: Hand pose prediction, depth estimation, and touch classification from capacitive images.  In UIST, p. 997–1009. ACM, 2021.
  * [3] A. A. A. Alsamarei.  Remote social touch: a framework to communicate physical interaction across long distances.  PhD thesis, Middle East Technical University, 2021.
  * [4] H. Baker, D. Tanguay, I. Sobel, D. Gelb, M. E. Goss, W. B. Culbertson, and T. Malzbender.  The coliseum immersive teleconferencing system.  In International Workshop on Immersive Telepresence, vol. 6, 2002\.
  * [5] H. Benko, E. Ishak, and S. Feiner.  Cross-dimensional gestural interaction techniques for hybrid immersive environments.  In IEEE Proc. VR, pp. 209–216, 2005.
  * [6] C. Bermejo and P. Hui.  A survey on haptic technologies for mobile augmented reality.  ACM Comput. Surv., 54(9), oct 2021.
  * [7] C. Bevan and D. Stanton Fraser.  Shaking hands and cooperation in tele-present human-robot negotiation.  In Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction, pp. 247–254, 2015.
  * [8] E. Bouzbib, G. Bailly, S. Haliyo, and P. Frey.  “can i touch this?”: Survey of virtual reality interactions via haptic solutions: Revue de littérature des interactions en réalité virtuelle par le biais de solutions haptiques.  In 32e Conférence Francophone Sur l’Interaction Homme-Machine. ACM, 2021.
  * [9] C. Buehler, M. Bosse, L. McMillan, S. Gortler, and M. Cohen.  Unstructured lumigraph rendering.  In SIGGRAPH, pp. 425–432, 2001.
  * [10] G. Burdea, J. Zhuang, E. Roskos, D. Silver, and N. Langrana.  A portable dextrous master with force feedback.  Presence: Teleoper. Virtual Environ., 1(1):18–28, 1992.
  * [11] W. Buxton, A. Sellen, and M. Sheasby.  Interfaces for multiparty videoconferences.  1997\.
  * [12] T. Carter, S. A. Seah, B. Long, B. Drinkwater, and S. Subramanian.  Ultrahaptics: Multi-point mid-air haptic feedback for touch surfaces.  In UIST, p. 505–514. ACM, 2013.
  * [13] D. Chen, S. Ren, Y. Wei, X. Cao, and J. Sun.  Joint cascade face detection and alignment.  In ECCV, pp. 109–122. Springer, 2014.
  * [14] L.-P. Cheng, E. Ofek, C. Holz, H. Benko, and A. D. Wilson.  Sparse haptic proxy: Touch feedback in virtual environments using a general passive prop.  In CHI, p. 3718–3728. ACM, 2017.
  * [15] F. Coldefy and S. Louis-dit Picard.  Digitable: an interactive multiuser table for collocated and remote collaboration enabling remote gesture visualization.  In CVPR, pp. 1–8, 2007.
  * [16] A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan.  High-quality streamable free-viewpoint video.  ACM TOG, 34(4):1–13, 2015.
  * [17] M. Dou, S. Khamis, Y. Degtyarev, P. Davidson, S. R. Fanello, A. Kowdle, S. O. Escolano, C. Rhemann, D. Kim, J. Taylor, et al.  Fusion4d: Real-time performance capture of challenging scenes.  ACM TOG, 35(4):1–13, 2016.
  * [18] A. Gallace and M. Girondini.  Social touch in virtual reality.  Current Opinion in Behavioral Sciences, 43:249–254, 2022.
  * [19] A. Gallace and C. Spence.  The science of interpersonal touch: An overview.  Neuroscience & Biobehavioral Reviews, 34(2):246–259, 2010.  Touch, Temperature, Pain/Itch and Pleasure.
  * [20] A. M. Genest, C. Gutwin, A. Tang, M. Kalyn, and Z. Ivkovic.  Kinectarms: A toolkit for capturing and displaying arm embodiments in distributed tabletop groupware.  In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, p. 157–166. ACM, 2013.
  * [21] S. J. Gibbs, C. Arapis, and C. J. Breiteneder.  Teleport–towards immersive copresence.  Multimedia Systems, 7(3):214–221, 1999.
  * [22] D. Gotsch, X. Zhang, T. Merritt, and R. Vertegaal.  Telehuman2: A cylindrical light field teleconferencing system for life-size 3d human telepresence.  In CHI, vol. 18, p. 552, 2018.
  * [23] K. Guo, F. Xu, T. Yu, X. Liu, Q. Dai, and Y. Liu.  Real-time geometry, albedo, and motion reconstruction using a single rgb-d camera.  ACM TOG, 36(4), 2017.
  * [24] A. Haans and W. IJsselsteijn.  Mediated social touch: a review of current research and future directions.  Virtual Reality, 9(2):149–159, 2006.
  * [25] K. Higuchi, Y. Chen, P. A. Chou, Z. Zhang, and Z. Liu.  Immerseboard: Immersive telepresence experience using a digital whiteboard.  In CHI, p. 2383–2392. ACM, 2015.
  * [26] L. Huang, B. Zhang, Z. Guo, Y. Xiao, Z. Cao, and J. Yuan.  Survey on depth and rgb image-based 3d hand shape and pose estimation.  Virtual Reality & Intelligent Hardware, 3:207–234, 2021.
  * [27] H. Ishii and M. Kobayashi.  Clearboard: A seamless medium for shared drawing and conversation with eye contact.  In CHI, p. 525–532. ACM, 1992.
  * [28] H. Ishii and N. Miyake.  Toward an open shared workspace: Computer and video fusion approach of teamworkstation.  Commun. ACM, 34(12):37–50, dec 1991.
  * [29] D. Iwai, R. Matsukage, S. Aoyama, T. Kikukawa, and K. Sato.  Geometrically consistent projection-based tabletop sharing for remote collaboration.  IEEE Access, 6:6293–6302, 2018.
  * [30] S. Izadi, A. Agarwal, A. Criminisi, J. Winn, A. Blake, and A. Fitzgibbon.  C-slate: A multi-touch and object recognition system for remote collaboration using horizontal surfaces.  In Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems, pp. 3–10, 2007.
  * [31] J. Johnson, A. Alahi, and L. Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, pp. 694–711, 2016.
  * [32] A. Jones, M. Lang, G. Fyffe, X. Yu, J. Busch, I. McDowall, M. Bolas, and P. Debevec.  Achieving eye contact in a one-to-many 3d video teleconferencing system.  ACM TOG, 28(3), 2009.
  * [33] S. Junuzovic, K. Inkpen, T. Blank, and A. Gupta.  Illumishare: Sharing any surface.  In CHI, p. 1919–1928. ACM, 2012.
  * [34] P. Kauff and O. Schreer.  An immersive 3d video-conferencing system using shared virtual team user environments.  In International Conference on Collaborative Virtual Environments, pp. 105–112, 2002.
  * [35] L. Khaleghi, A. Sepas-Moghaddam, J. Marshall, and A. Etemad.  Multi-view video-based 3d hand pose estimation.  IEEE Transactions on Artificial Intelligence, 2022.
  * [36] D. Kim, S. Izadi, J. Dostal, C. Rhemann, C. Keskin, C. Zach, J. Shotton, T. A. Large, S. Bathiche, M. Nießner, A. Butler, S. Fanello, and V. Pradeep.  Retrodepth: 3d silhouette sensing for high-precision input on and above physical surfaces.  CHI, 2014.
  * [37] M. Kuechler and A. Kunz.  Holoport-a device for simultaneous video and data conferencing featuring gaze awareness.  In IEEE VR, pp. 81–88, 2006.
  * [38] C. Kuster, N. Ranieri, H. Zimmer, J.-C. Bazin, C. Sun, T. Popa, M. Gross, et al.  Towards next generation 3d teleconferencing systems.  In 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video, pp. 1–4, 2012.
  * [39] J. Lawrence, D. B. Goldman, S. Achar, G. M. Blascovich, J. G. Desloge, T. Fortes, E. M. Gomez, S. Häberling, H. Hoppe, A. Huibers, C. Knaus, B. Kuschak, R. Martin-Brualla, H. Nover, A. I. Russell, S. M. Seitz, and K. Tong.  Project starline: A high-fidelity telepresence system.  SIGGRAPH Asia, 40(6), 2021.
  * [40] K.-D. Le, K. Zhu, and M. Fjeld.  Mirrortablet: Exploring a low-cost mobile system for capturing unmediated hand gestures in remote collaboration.  In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia, p. 79–89. ACM, 2017.
  * [41] D. Leithinger, S. Follmer, A. Olwal, and H. Ishii.  Physical telepresence: Shape capture and display for embodied, computer-mediated remote collaboration.  In UIST, p. 461–470. ACM, 2014.
  * [42] M. Levoy and P. Hanrahan.  Light field rendering.  In SIGGRAPH, pp. 31–42, 1996.
  * [43] J. P. Lewis, M. Cordner, and N. Fong.  Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation.  In SIGGRAPH, pp. 165–172, 2000.
  * [44] G. M. Lim, P. Jatesiktat, C. W. K. Kuah, and W. T. Ang.  Camera-based hand tracking using a mirror-based multi-view setup.  In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 5789–5793. IEEE, 2020\.
  * [45] R. W. Lindeman, R. Page, Y. Yanagida, and J. L. Sibert.  Towards full-body haptic feedback: The design and deployment of a spatialized vibrotactile feedback system.  In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, p. 146–149. ACM, 2004.
  * [46] K. Luo, T. Guan, L. Ju, H. Huang, and Y. Luo.  P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo.  In CVPR, pp. 10452–10461, 2019.
  * [47] A. Maimone and H. Fuchs.  Encumbrance-free telepresence system with real-time 3d capture and display using commodity depth cameras.  In ISMAR, pp. 137–146, 2011.
  * [48] T. Massie.  The phantom haptic interface: A device for probing virtual objects.  1994\.
  * [49] A. Mehrabian.  Nonverbal communication.  Routledge, 2017.
  * [50] H. Nakani shi, K. Tanaka, and Y. Wada.  Remote handshaking: touch enhances video-mediated social telepresence.  In CHI, pp. 2143–2152, 2014.
  * [51] D. Nguyen and J. Canny.  Multiview: spatially faithful group video conferencing.  In CHI, pp. 799–808, 2005.
  * [52] J. Oh, Y. Lee, Y. Kim, T. Jin, S. Lee, and S.-H. Lee.  Hand contact between remote users through virtual avatars.  In Proceedings of the 29th International Conference on Computer Animation and Social Agents, pp. 97–100, 2016.
  * [53] Y. Onishi, K. Tanaka, and H. Nakanishi.  Spatial continuity and robot-embodied pointing behavior in videoconferencing.  In CRIWG, 2017.
  * [54] S. Orts-Escolano, C. Rhemann, S. Fanello, W. Chang, A. Kowdle, Y. Degtyarev, D. Kim, P. L. Davidson, S. Khamis, M. Dou, V. Tankovich, C. Loop, Q. Cai, P. A. Chou, S. Mennicken, J. Valentin, V. Pradeep, S. Wang, S. B. Kang, P. Kohli, Y. Lutchyn, C. Keskin, and S. Izadi.  Holoportation: Virtual 3d teleportation in real-time.  In UIST, p. 741–754. ACM, 2016.
  * [55] E. Pezent, A. Israr, M. Samad, S. Robinson, P. Agarwal, H. Benko, and N. Colonnese.  Tasbi: Multisensory squeeze and vibrotactile wrist haptics for augmented and virtual reality.  In 2019 IEEE World Haptics Conference, pp. 1–6, 2019.
  * [56] C. Plüss, N. Ranieri, J.-C. Bazin, T. Martin, P.-Y. Laffont, T. Popa, and M. Gross.  An immersive bidirectional system for life-size 3d communication.  In International Conference on Computer Animation and Social Agents, p. 89–96. ACM, 2016.
  * [57] S. Price, C. Jewitt, and N. Yiannoutsou.  Conceptualising touch in vr.  Virtual Real., 25(3):863–877, sep 2021.
  * [58] E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley.  Color transfer between images.  IEEE Computer graphics and applications, 21(5):34–41, 2001.
  * [59] A. Sadagic, H. Towles, L. Holden, K. Daniilidis, and B. Zeleznik.  Tele-immersion portal: Towards an ultimate synthesis of computer graphics and computer vision systems.  In Annual International Workshop on Presence, pp. 21–23, 2001.
  * [60] A. Schäfer, G. Reis, and D. Stricker.  A survey on synchronous augmented, virtual and mixed reality remote collaboration systems.  CoRR, abs/2102.05998, 2021.
  * [61] M. Schmitz, F. Müller, M. Mühlhäuser, J. Riemann, and H. V. V. Le.  Itsy-bits: Fabrication and recognition of 3d-printed tangibles with small footprints on capacitive touchscreens.  In CHI. ACM, 2021.
  * [62] S. Seinfeld, I. Schmidt, and J. Müller.  Evoking realistic affective touch experiences in virtual reality.  arXiv preprint arXiv:2202.13389, 2022.
  * [63] H. Shum and S. B. Kang.  Review of image-based rendering techniques.  In Visual Communications and Image Processing, vol. 4067, pp. 2–13, 2000.
  * [64] M. T. A. Sin, S. L. Koole, et al.  That human touch that means so much: exploring the tactile dimension of social life.  Mind Magazine, 2:17, 2013.
  * [65] B. Smith, C. Wu, H. Wen, P. Peluse, Y. Sheikh, J. K. Hodgins, and T. Shiratori.  Constraining dense hand surface tracking with elasticity.  TOG, 39(6), 2020.
  * [66] R. Sodhi, I. Poupyrev, M. Glisson, and A. Israr.  Aireal: Interactive tactile experiences in free air.  ACM TOG, 32(4), jul 2013.
  * [67] P. Streli and C. Holz.  Capcontact: Super-resolution contact areas from capacitive touchscreens.  In CHI. ACM, 2021.
  * [68] P. Sykownik and M. Masuch.  The experience of social touch in multi-user virtual reality.  In 26th ACM Symposium on Virtual Reality Software and Technology. ACM, 2020.
  * [69] K.-H. Tan, I. Robinson, R. Samadani, B. Lee, D. Gelb, A. Vorbau, B. Culbertson, and J. Apostolopoulos.  Connectboard: A remote collaboration system that supports gaze-aware interaction and sharing.  In 2009 IEEE International Workshop on Multimedia Signal Processing, pp. 1–6, 2009.
  * [70] A. Tang, C. Neustaedter, and S. Greenberg.  Videoarms: Embodiments for mixed presence groupware.  In N. Bryan-Kinns, A. Blanford, P. Curzon, and L. Nigay, eds., People and Computers XX — Engage, pp. 85–102. Springer London, 2007.
  * [71] J. C. Tang and S. Minneman.  Videowhiteboard: Video shadows to support remote collaboration.  In CHI, p. 315–322. ACM, 1991.
  * [72] X. Tang, T. Wang, and C.-W. Fu.  Towards accurate alignment in real-time 3d hand-mesh reconstruction.  In ICCV, pp. 11698–11707, 2021.
  * [73] H. Towles, W. chao Chen, R. Yang, S. uok Kum, H. F. N. Kelshikar, J. Mulligan, K. Daniilidis, H. Fuchs, C. C. Hill, N. K. J. Mulligan, L. Holden, B. Zeleznik, A. Sadagic, and J. Lanier.  3d tele-collaboration over internet2.  In International Workshop on Immersive Telepresence, 2002.
  * [74] J. B. F. van Erp and A. Toet.  Social touch in human–computer interaction.  Frontiers in Digital Humanities, 2, 2015.
  * [75] B. A. Wallace.  Merging and transformation of raster images for cartoon animation.  In SIGGRAPH, pp. 253–262, 1981.
  * [76] J. Wang, F. Mueller, F. Bernard, S. Sorli, O. Sotnychenko, N. Qian, M. A. Otaduy, D. Casas, and C. Theobalt.  Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video.  TOG, 39(6):1–16, 2020.
  * [77] W.-C. Wen, H. Towles, L. Nyland, G. Welch, and H. Fuchs.  Toward a compelling sensation of telepresence: demonstrating a portal to a distant (static) office.  In IEEE Conference on Visualization, pp. 327–333, 2000.
  * [78] E. Wood, J. Taylor, J. Fogarty, A. Fitzgibbon, and J. Shotton.  Shadowhands: High-fidelity remote hand gesture visualization using a hand tracker.  In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces, p. 77–84. ACM, 2016.
  * [79] R. Xiao, J. Schwarz, N. Throm, A. D. Wilson, and H. Benko.  Mrtouch: Adding touch input to head-mounted mixed reality.  IEEE TVCG, 24(4):1653–1660, apr 2018.
  * [80] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In ECCV, pp. 767–783, 2018.
  * [81] C. Zhang, Q. Cai, P. A. Chou, Z. Zhang, and R. Martin-Brualla.  Viewport: A distributed, immersive teleconferencing system with infrared dot pattern.  IEEE MultiMedia, 20(1):17–27, 2013.
  * [82] Y. Zhang, J. Yang, Z. Liu, R. Wang, G. Chen, X. Tong, and B. Guo.  Virtualcube: An immersive 3d video communication system.  IEEE TVCG, 28(5):2146–2156, 2022.
  * [83] Z. Zhang.  A flexible new technique for camera calibration.  IEEE TPAMI, 22(11):1330–1334, 2000.
  * [84] J. Zillner, C. Rhemann, S. Izadi, and M. Haller.  3d-board: A whole-body remote collaborative whiteboard.  In UIST, p. 471–479. ACM, 2014.
  * [85] M. Zollhöfer, M. Nießner, S. Izadi, C. Rehmann, C. Zach, M. Fisher, C. Wu, A. Fitzgibbon, C. Loop, C. Theobalt, et al.  Real-time non-rigid reconstruction using an rgb-d camera.  ACM TOG, 33(4):1–12, 2014.


# Scissors Modes of a Bose-Einstein Condensate in a Synthetic Magnetic Field

Chunlei Qu  Corresponding author<EMAIL_ADDRESS>Department of Physics,
Stevens Institute of Technology, Hoboken, NJ 07030, USA  Center for Quantum
Science and Engineering, Stevens Institute of Technology, Hoboken, NJ 07030,
USA  Chuan-Hsun Li  Department of Physics and Astronomy, Purdue University,
West Lafayette, Indiana 47907, USA  Yong P. Chen  Department of Physics and
Astronomy, Purdue University, West Lafayette, Indiana 47907, USA  School of
Electrical and Computer Engineering, Purdue University, West Lafayette,
Indiana 47907, USA  Purdue Quantum Science and Engineering Institute, Purdue
University, West Lafayette, Indiana 47907, USA  Sandro Stringari  Pitaevskii
BEC Center and Dipartimento di Fisica, Università degli Studi di Trento, 38123
Povo, Italy

###### Abstract

We study the scissors modes of a harmonically trapped Bose-Einstein condensate
under the influence of a synthetic magnetic field, which induces rigid
rotational components in the velocity field. Our investigation reveals that
the scissors mode, excited in the plane perpendicular to the synthetic
magnetic field, becomes coupled to the quadrupole modes of the condensate,
giving rise to typical beating effects. Moreover, the two scissors modes
excited in the vertical planes are also coupled together by the synthetic
magnetic field, resulting in intriguing gyroscope dynamics. Our analytical
results, derived from a spinor hydrodynamic theory, are further validated
through numerical simulations of the three-dimensional Gross-Pitaevskii
equation. These predictions for the condensates subject to a synthetic
magnetic field are experimentally accessible with current cold-atom setups and
hold promise for potential applications in quantum sensing.

Introduction.— Superfluidity is one of the most extraordinary consequences of
Bose-Einstein condensation (BEC). In usual condensates, superfluidity is
manifested in various remarkable rotational phenomena such as a reduced moment
of inertia relative to the rigid-body value and the vanishing of angular
momentum in absence of quantized vortices and in presence of isotropic
trapping [1, 2]. Both effects are directly linked to the constraint of
irrotationality, a peculiar feature of usual superfluids. In vortex-free BECs,
superfluidity is often probed by exciting the so-called scissors mode which
corresponds to an angular rotation around the symmetry axis of an anisotropic
trap [3]. This technique has proved invaluable in discerning the superfluid
nature of diverse systems, including Bose [4] and Fermi gases [5] as well as
elongated dipolar atomic gases [6, 7].

Figure 1: (a) Phase diagram of the condensate in a synthetic magnetic field.
The two spin components can be overlapped, exhibit vortices, or be separated,
depending on the value of the relevant parameter $\eta$ fixing the position
dependence of the detuning (see text) and on the particle number $N$. (b) A
typical distribution of the condensate for $\eta=0.005,0.01,0.013$ at
$N=10^{5}$. Note that there are two vortices at $y=0$ for $\eta=0.01$.

The experimental realization of synthetic spin-orbit coupling (SOC) has
offered novel opportunities for exploring topological condensed matter physics
with ultracold neutral atoms [8, 9, 10]. Synthetic SOC substantially impacts
the superfluid properties of condensates due to the breaking of Galilean
invariance and irrotationality [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
22, 23]. For instance, in BECs with an equal superposition of Rashba and
Dresselhaus SOC, the superfluid fraction associated with the flow along the
SOC direction vanishes near the transition point between the plane-wave phase
and the zero-momentum phase [21]. Furthermore, introducing a position-
dependent detuning along the direction orthogonal to the SOC direction
generates new intriguing features in the phase diagram [15]. Figure 1 shows
that, as a function of the position-dependent detuning, the two spin
components of spin-orbit-coupled BEC can overlap, exhibit vortices, or be
fully separated. Vortices have been actually observed experimentally for
values of the detuning gradient larger than a critical value [11]. Despite
these advances, the effect of the synthetic magnetic field on the superfluid
dynamics of the spin-orbit-coupled condensates has been vastly unexplored.
Recently it was found that a vortex-free BEC, in the presence of position-
dependent detuning, exhibits a rigid-like rotational velocity field and
possesses a finite angular momentum even when the system is confined in an
isotropic harmonic trap [22]. Due to spin-orbit coupling the position-
dependent detuning actually brings the system into an effective non-inertial
rotational frame, leading to the Foucault-like precession of two dipole modes
in the rotating plane [23].

In this paper, we extend the spinor hydrodynamic theory developed in [22] and
[23], accounting for position-dependent tuning effects, in order to
investigate the precession of the scissors modes of a BEC subject to a
rotational velocity field induced by a synthetic magnetic field. Our findings
indicate that the synthetic magnetic field induces a coupling between the
scissors modes and the other collective modes of quadrupole nature.
Specifically, the scissors mode in the horizontal plane perpendicular to the
synthetic magnetic field becomes coupled to the three quadrupole modes related
to the shape oscillation of the condensate. Moreover, the two scissors modes
in the vertical planes are coupled together, giving rise to intriguing
gyroscope dynamics.

Spinor hydrodynamic theory.—We begin by considering a BEC with a Raman
transition induced equal-Rashba-Dresselhaus synthetic SOC along the
$x$-direction. In the spin-rotated frame, the single-particle Hamiltonian is
given by

$H_{0}=\frac{1}{2m}(\hat{\mathbf{p}}-\hbar
k_{0}\sigma_{z}\hat{e}_{x})^{2}+V_{trap}-\frac{\hbar\Omega}{2}\sigma_{x}-\eta
k_{0}y\sigma_{z},$ (1)

where $m$ is the atomic mass, $\hbar k_{0}$ is the recoil momentum, $\Omega$
represents the Raman coupling strength,
$V_{trap}=m(\omega_{x}^{2}x^{2}+\omega_{y}^{2}y^{2}+\omega_{z}^{2}z^{2})/2$
denotes the harmonic trapping potential with angular frequencies
$\omega_{x,y,z}$, and $\eta$ is the coefficient of the $y$-position-dependent
detuning. The mean-field interaction energy of the condensate is characterized
by $V_{int}=\int
d\mathbf{r}\sum_{\alpha\beta}g_{\alpha\beta}n_{\alpha}n_{\beta}/2$ where
$n_{\alpha}$ is the particle density of the $\alpha$-th component, and
$g_{\alpha\beta}=4\pi\hbar^{2}a_{\alpha\beta}/m$ are the interaction constants
in different spin channels with $a_{\alpha\beta}$ the corresponding $s$-wave
scattering lengths. In this work, we focus on isotropic interactions by
assuming $g_{\alpha\beta}=g$. The last term in Eq. (1) induces a synthetic
magnetic field $\mathbf{B}_{syn}$ along the vertical $z$-direction (Fig.
2(a)). When $\eta$ is larger than a critical value $\eta_{c}$, vortices will
appear which has been demonstrated in the pioneering experiment [11].

Figure 2: (a) Illustration of a spin-orbit-coupled BEC in the presence of a
position-dependent detuning $\delta(y)=\eta k_{0}y$ which induces a synthetic
magnetic field $\mathbf{B}_{syn}$ along the $z$-direction and a rotational
velocity field in the horizontal plane. (b) Dependence of the quadrupole modes
eigenfrequencies on the Raman coupling strength for $\eta=0$. The three red
curves are the eigenfrequencies corresponding to the width oscillation of the
condensate (Eq. 5) and the blue curve corresponds to the oscillation frequency
$\omega_{xy}/2\pi$ of the scissors mode in the horizontal plane. The trapping
frequencies are
$(\omega_{x},\omega_{y},\omega_{z})/2\pi=(50\sqrt{3},50,35)$Hz. At the two
points indicated by the green star and cyan dot, the scissors mode frequency
$\omega_{xy}$ and one of the three hybrid quadrupole modes are degenerate. In
this work, we shall focus on the green star point at $\Omega=6E_{r}$ where the
degeneracy condition Eq. (6) is satisfied.

For simplicity, we shall focus on the zero-momentum phase where
$\hbar\Omega>\hbar\Omega_{c}\equiv 4E_{r}$, with $E_{r}=\hbar^{2}k_{0}^{2}/2m$
representing the recoil energy. When the detuning gradient $\eta$ is small,
the total density of the condensate can be approximated using the Thomas-Fermi
(TF) distribution. Additionally, due to the large Raman energy gap, the
relative phase between the two spin components is locked in the low-frequency
oscillations. In this regime the hydrodynamic equations for the two spin
components can be simply reduced to the following equations

$\displaystyle\frac{\partial n}{\partial
t}+\mathbf{\nabla}\cdot(n\mathbf{v})=0$ (2)
$\displaystyle\frac{\partial\phi}{\partial
t}+\frac{1}{2}m^{*}v_{x}^{2}+\frac{1}{2}mv_{y}^{2}+\frac{1}{2}mv_{z}^{2}+\mu-\frac{\Omega}{2}=0$
(3)

for the total density $n$, and the the absolute phase $\phi$, with

$\mathbf{v}=\left(\frac{\hbar}{m^{*}}\nabla_{x}\phi-\frac{\eta}{\hbar}\frac{\Omega_{c}}{\Omega}y,\frac{\hbar}{m}\nabla_{y}\phi,\frac{\hbar}{m}\nabla_{z}\phi\right),$
(4)

denoting the physical superfluid velocity and
$m^{*}=m(1-\Omega_{c}/\Omega)^{-1}$ the effective mass in the zero-momentum
phase. It is worth pointing out that the second term of $v_{x}$ arises from
the spin density $s_{z}/n$ which becomes nonzero for finite $\eta$. This term
plays an important role in generating the rotational velocity field, the
finite angular momentum, and the coupling among different collective modes. In
Eq. (3), we have introduced the chemical potential
$\mu=ng+V_{trap}-\frac{1}{2}m^{*}\eta^{2}\Omega_{c}^{2}y^{2}/\hbar^{2}\Omega^{2}$
where the last term results from the anti-trap effect of the synthetic
magnetic field and renormalizes the trapping frequency along the
$y$-direction. Since it is proportional to $\eta^{2}$, this term is neglected
in the following analysis.

At equilibrium, the total density and absolute phase are determined as
$n_{0}=(\mu-V_{trap})/g$ and $\phi_{0}=\alpha xy$, respectively, with
$\alpha=2\eta k_{0}^{2}\omega_{x}^{2}/\Omega\omega_{xy}^{2}$ where
$\omega_{xy}=\sqrt{\omega_{x}^{2}m/m^{*}+\omega_{y}^{2}}$ is the frequency of
the scissors mode in the horizontal plane in the absence of detuning gradient.
Consequently, the velocity field becomes
$\mathbf{v}_{0}=(-\omega_{eff}y,\omega_{eff}^{\prime}x,0)$ where
$\omega_{eff}=\eta\Omega_{c}/\hbar\Omega-\hbar\alpha/m^{*}$,
$\omega_{eff}^{\prime}=\hbar\alpha/m$. Such a rigid-like rotational velocity
field causes a coupling of various collective modes, including the two dipole
modes in the horizontal plane [23] and the quadrupole modes discussed in this
work.

Without synthetic magnetic field ($\eta=0$).— In this case the quadrupole
modes of a trapped BEC include three scissors modes excited by the operators
$xy$, $xz$, and $yz$ and corresponding, respectively, to the angular rotation
of the condensate around the $z$, $y$, and $x$ axis, and other three modes
excited by the operators $x^{2}$, $y^{2}$, and $z^{2}$, which correspond to
the shape oscillations of the condensate. The oscillation frequencies of the
three scissors modes are given by
$\omega_{xy}=\sqrt{\omega_{x}^{2}(m/m^{*})+\omega_{y}^{2}}$,
$\omega_{xz}=\sqrt{\omega_{x}^{2}(m/m^{*})+\omega_{z}^{2}}$,
$\omega_{yz}=\sqrt{\omega_{y}^{2}+\omega_{z}^{2}}$, respectively.

The frequencies of the quadrupole modes related to the shape oscillations are
instead given by the solution of the equation [2]

$\displaystyle\omega^{6}-3\omega^{4}(\frac{m}{m^{*}}\omega_{x}^{2}+\omega_{y}^{2}+\omega_{z}^{2})+8\omega^{2}(\frac{m}{m^{*}}\omega_{x}^{2}\omega_{y}^{2}+\frac{m}{m^{*}}\omega_{x}^{2}\omega_{z}^{2}$
$\displaystyle+\omega_{y}^{2}\omega_{z}^{2})-20\frac{m}{m^{*}}\omega_{x}^{2}\omega_{y}^{2}\omega_{z}^{2}=0.$
(5)

Figure 2(b) illustrates how the three eigenfrequencies related to the shape
oscillations and the scissors mode oscillation frequency $\omega_{xy}$ depend
on the Raman coupling strength. It shows that the degeneracy between the
scissors mode $xy$ and one of the shape oscillation quadrupole modes takes
place for two values of the Raman coupling. In the following we will focus on
the case indicated with the green star point at $\Omega=6E_{r}$, corresponding
to the degeneracy condition [24] (for the more general case, and detailed
information on the derivation of the equations of motion for the collective
modes, see SM [25])

$\displaystyle\sqrt{\frac{m}{m^{*}}}\omega_{x}=\omega_{y}$ (6)

which actually coincides with the condition that the two dipole modes in the
horizontal $x-y$ plane be degenerate.

With synthetic magnetic field ($\eta\neq 0$).—In the presence of a position-
dependent detuning, the six quadrupole modes discussed above are further
coupled together by the synthetic magnetic field, leading to interesting
beating phenomena that can be induced through distinct excitation methods:

Figure 3: (a,b) Time evolution of the scissors mode $xy$ in the horizontal
plane and the quadrupole mode characterized by the operator
$Q=(\omega_{x}^{2}/\omega_{y}^{2})x^{2}-y^{2}$ in the presence of a detuning
gradient $\eta=0.001E_{r}$. The trapping frequencies are
$(f_{x},f_{y},f_{z})=(50\sqrt{3},50,35)$Hz and the Raman coupling strength is
$\Omega=6E_{r}$ to meet the degeneracy condition Eq. (6). The dynamics is
excited by a sudden rotation of the harmonic trap by a small angle
$\varphi_{0}=3^{\circ}$ in the $x-y$ plane.

i) Sudden rotation of the harmonic trap by a small angle $\varphi_{0}$ in the
x-y plane. This excites the scissors mode in the horizontal plane,
characterized by the operator $xy$. Due to the influence of the synthetic
magnetic field, this scissors mode becomes coupled to the three quadrupole
modes $x^{2}$, $y^{2}$, and $z^{2}$, related to the shape oscillations of the
condensate. The ansatz for the fluctuation of the total density and absolute
phase can be written as

$\displaystyle\delta n$ $\displaystyle\sim$
$\displaystyle\epsilon_{1}xy+\epsilon_{2}x^{2}+\epsilon_{3}y^{2}+\epsilon_{4}z^{2}$
(7) $\displaystyle\delta\phi$ $\displaystyle\sim$
$\displaystyle\alpha_{1}xy+\alpha_{2}x^{2}+\alpha_{3}y^{2}+\alpha_{4}z^{2}$
(8)

Substituting them into the linearized spinor hydrodynamic Eqs. (2) and (3), we
obtain the coupled differential equations for the variables $\epsilon_{j}$ and
$\alpha_{j}$ ($j=1,\ldots,4$) (see SM [25]). By seeking solutions of the form
$\epsilon_{j}(t)\sim e^{-i\omega t}$ and $\alpha_{j}(t)\sim e^{-i\omega t}$,
the collective frequencies are found to satisfy an equation of the form
$\omega^{8}+c_{3}\omega^{6}+c_{2}\omega^{4}+c_{1}\omega^{2}+c_{0}=0$.

Figure 4: Time evolution of the scissors modes in the vertical planes $\langle
xz\rangle$ and $\langle yz\rangle$. The trapping frequencies are
$(f_{x},f_{y},f_{z})=(50\sqrt{3},50,35)Hz$, $\Omega=6E_{r}$. Thus,
$\sqrt{m/m^{*}}\omega_{x}=\omega_{y}$, a beat appears for the two coupled
modes. The detuning gradient $\eta=0.001E_{r}$. The dynamics is excited by a
sudden rotation of the harmonic trap by a small angle $\theta_{0}=3^{\circ}$
in the $x-z$ plane.

Under the degeneracy condition (Eq. (6)), these coupled differential equations
can be further simplified as the scissors mode $xy$ is only coupled to a mode
associated with condensate deformation in the horizontal plane, excited by the
operator
$Q=\frac{\omega_{x}}{\omega_{y}}x^{2}-\frac{\omega_{y}}{\omega_{x}}y^{2}$.
Introducing the convenient variables
$\sqrt{\frac{m}{m^{*}}}\omega_{x}=\omega_{y}\equiv\omega_{0}$ and
$\omega_{eff}\frac{\omega_{x}}{\omega_{y}}=\omega_{eff}^{\prime}\frac{\omega_{y}}{\omega_{x}}\equiv\omega_{\eta}$,
a straightforward calculation, which includes only effects linear in $\eta$,
yields (see SM [25])

$\displaystyle\frac{d^{2}}{dt^{2}}\langle
xy\rangle+(2\omega_{0}^{2}-4\omega_{\eta}^{2})\langle
xy\rangle-2\omega_{\eta}\frac{d}{dt}\langle Q\rangle=0$ (9)
$\displaystyle\frac{d^{2}}{dt^{2}}\langle
Q\rangle+(2\omega_{0}^{2}-4\omega_{\eta}^{2})\langle
Q\rangle+8\omega_{\eta}\frac{d}{dt}\langle xy\rangle=0.$ (10)

Consequently, the two modes excited by the operators $xy$ and $Q$ exhibit a
beating effect with a frequency splitting $\Delta\omega=4\omega_{\eta}$. The
other two modes, excited by the operators
$\frac{\omega_{x}}{\omega_{y}}x^{2}+\frac{\omega_{y}}{\omega_{x}}y^{2}$ and
$z^{2}$, remain coupled but they do not depend on the detuning gradient (see
SM [25]). A representative numerical solution of the beating effect exhibited
by the $xy$ and $Q$ modes is shown in Fig. 3(a, b).

To validate our theoretical hydrodynamic analysis, we have performed a
numerical simulation of the time-dependent three-dimensional Gross-Pitaevskii
(GP) equation. For our numerics, we have chosen a total particle number of
$N=10^{5}$, and a recoil momentum $k_{0}=2\pi/\lambda$ with $\lambda=782nm$.
Our results reveal excellent agreement between the hydrodynamic predictions
and numerical GP results.

ii) Sudden rotation of the harmonic trap by a small angle $\theta_{0}$ in the
x-z plane. This excites the scissors mode in the vertical plane, associated
with the operator $xz$. Due to the presence of the synthetic magnetic field,
this mode couples to the other scissors mode $yz$. In this case the ansatz for
the variations of the total density and absolute phase are

$\displaystyle\delta n\sim\epsilon_{5}xz+\epsilon_{6}yz$ (11)
$\displaystyle\delta\phi\sim\alpha_{5}xz+\alpha_{6}yz$ (12)

Under the same degeneracy condition Eq. (6), the two scissors modes share the
same oscillation frequency when $\eta=0$. Substituting Eqs. (11-12) into the
linearized spinor hydrodynamic Eqs. (2-3), we obtain the following coupled
second-order differential equations

$\displaystyle\frac{d^{2}\langle
xz\rangle}{dt^{2}}+(\omega_{0}^{2}+\omega_{z}^{2}-\omega_{\eta}^{2})\langle
xz\rangle+2\omega_{\eta}\frac{\omega_{y}}{\omega_{x}}\frac{d\langle
yz\rangle}{dt}=0$ $\displaystyle\frac{d^{2}\langle
yz\rangle}{dt^{2}}+(\omega_{0}^{2}+\omega_{z}^{2}-\omega_{\eta}^{2})\langle
yz\rangle-2\omega_{\eta}\frac{\omega_{x}}{\omega_{y}}\frac{d\langle
xz\rangle}{dt}=0$

which clearly reveals that the two scissors modes exhibit a beating effect,
with frequency splitting $\Delta\omega=2\omega_{\eta}$ (Fig. 4).

Figure 5: Illustration of a BEC gyroscope realized via a synthetic magnetic
field. The polar angle $\theta(t)$ corresponds to a fast scissors mode
oscillation in the vertical plane and the azimuthal angle $\varphi(t)$
corresponds to a slower precession around the symmetry axis $z$ due to the
synthetic magnetic field.

The coupling between the two scissors modes in the vertical planes gives rise
to a typical gyroscopic effect associated with the precession of angular
momentum. A sudden rotation of the trap in the $x-z$ plane in fact causes the
angular momentum to deviate from the vertical axis. As shown in Fig. 5, this
results in two simultaneous dynamical effects: a rapid oscillation of angular
momentum around the vertical axis due to the restoring force of the trapping
potential (the scissors mode), and a slower precession of angular momentum
around the vertical axis due to the synthetic magnetic field.

In the linear regime, it is easy to find the relation between the azimuthal
angle $\varphi$, that characterizes the gyroscope precession and the polar
angle $\theta$, that characterizes the scissors modes in the vertical planes:

$\displaystyle\langle xz\rangle$ $\displaystyle=$
$\displaystyle\frac{R_{x}R_{z}}{7}\frac{\omega_{x}^{2}-\omega_{z}^{2}}{\omega_{x}\omega_{z}}\theta(t)\cos\varphi(t)$
(13) $\displaystyle\langle yz\rangle$ $\displaystyle=$
$\displaystyle\frac{R_{y}R_{z}}{7}\frac{\omega_{y}^{2}-\omega_{z}^{2}}{\omega_{y}\omega_{z}}\theta(t)\sin\varphi(t)$
(14)

where we have assumed that the polar angle $\theta(t)$ is small. A similar
gyroscopic effect has been previously investigated in the case of a single
component Bose-Einstein condensate in the presence of a quantized vortex [26,
27]. In the present case the effect is caused by the synthetic magnetic field
which gives rise to angular momentum even in the absence of quantized
vortices, and to a precession rate $d\varphi/dt$ of the azimuthal angle
proportional to $\omega_{\eta}$.

For larger values of the position-dependent detuning gradient $\eta$, and
hence larger synthetic magnetic fields, additional effects take place, like
bifurcation dynamics [28] and the occurrence of quantized vortices [11], whose
consequences on the dynamic behavior of the scissors mode, are however beyond
the purpose of the present work.

In conclusion, we have presented a comprehensive investigation of the scissors
modes and other quadrupole modes of a spin-orbit-coupled BEC under the
influence of a position-dependent detuning. Due to the existence of a
synthetic magnetic field, these collective modes are coupled together, giving
rise to interesting beating effects and gyroscope dynamics. Our framework can
be naturally extended to the case of the plane-wave phase or the application
of a position-dependent Raman coupling. Our results offer valuable insights
pertinent to the understanding of a recent experimental work carried out at
Purdue University where the scissors mode of a spin-orbit-coupled BEC with a
position-dependent Raman coupling was directly observed [29].

###### Acknowledgements.

C.Q is supported by the ACC-New Jersey under Contract No. W15QKN-18-D-0040 and
the Stevens Startup Fund. C.H.L and Y.P.C acknowledge support from Quantum
Science Center (QSC), a Department of Energy (DOE) quantum information science
(QIS) research center.

## References

  * Stringari [1996] S. Stringari, Moment of inertia and superfluidity of a trapped bose gas, Phys. Rev. Lett. 76, 1405 (1996).
  * Pitaevskii and Stringari [2016] L. P. Pitaevskii and S. Stringari, Bose-einstein condensation and superfluidity, 1st ed., International Series of Monographs on Physics Vol. 164 (Oxford University Press, Oxford, United Kingdom (2016).
  * Guéry-Odelin and Stringari [1999] D. Guéry-Odelin and S. Stringari, Scissors mode and superfluidity of a trapped bose-einstein condensed gas, Phys. Rev. Lett. 83, 4452 (1999).
  * Maragò  _et al._ [2000] O. M. Maragò, S. A. Hopkins, J. Arlt, E. Hodby, G. Hechenblaikner, and C. J. Foot, Observation of the scissors mode and evidence for superfluidity of a trapped bose-einstein condensed gas, Phys. Rev. Lett. 84, 2056 (2000).
  * Wright  _et al._ [2007] M. J. Wright, S. Riedl, A. Altmeyer, C. Kohstall, E. R. S. Guajardo, J. H. Denschlag, and R. Grimm, Finite-temperature collective dynamics of a fermi gas in the bec-bcs crossover, Phys. Rev. Lett. 99, 150403 (2007).
  * Ferrier-Barbut  _et al._ [2018] I. Ferrier-Barbut, M. Wenzel, F. Böttcher, T. Langen, M. Isoard, S. Stringari, and T. Pfau, Scissors mode of dipolar quantum droplets of dysprosium atoms, Phys. Rev. Lett. 120, 160402 (2018).
  * Tanzi  _et al._ [2021] L. Tanzi, J. G. Maloberti, G. Biagioni, A. Fioretti, C. Gabbanini, and G. Modugno, Evidence of superfluidity in a dipolar supersolid from nonclassical rotational inertia, Science 371, 1162 (2021).
  * Dalibard  _et al._ [2011] J. Dalibard, F. Gerbier, G. Juzeliūnas, and P. Öhberg, Colloquium: Artificial gauge potentials for neutral atoms, Rev. Mod. Phys. 83, 1523 (2011).
  * Galitski and Spielman [2013] V. Galitski and I. B. Spielman, Spin–orbit coupling in quantum gases, Nature 494, 49 (2013).
  * Cooper  _et al._ [2019] N. R. Cooper, J. Dalibard, and I. B. Spielman, Topological bands for ultracold atoms, Rev. Mod. Phys. 91, 015005 (2019).
  * Lin  _et al._ [2009] Y.-J. Lin, R. L. Compton, K. Jiménez-García, J. V. Porto, and I. B. Spielman, Synthetic magnetic fields for ultracold neutral atoms, Nature 462, 628 (2009).
  * Wang  _et al._ [2010] C. Wang, C. Gao, C.-M. Jian, and H. Zhai, Spin-orbit coupled spinor bose-einstein condensates, Phys. Rev. Lett. 105, 160403 (2010).
  * Ho and Zhang [2011] T.-L. Ho and S. Zhang, Bose-einstein condensates with spin-orbit interaction, Phys. Rev. Lett. 107, 150403 (2011).
  * Wu  _et al._ [2011] C.-J. Wu, I. Mondragon-Shem, and X.-F. Zhou, Unconventional bose—einstein condensations from spin-orbit coupling, Chinese Physics Letters 28, 097102 (2011).
  * Radić  _et al._ [2011] J. Radić, T. A. Sedrakyan, I. B. Spielman, and V. Galitski, Vortices in spin-orbit-coupled bose-einstein condensates, Phys. Rev. A 84, 063604 (2011).
  * Zhu  _et al._ [2012] Q. Zhu, C. Zhang, and B. Wu, Exotic superfluidity in spin-orbit coupled bose-einstein condensates, Europhysics Letters 100, 50003 (2012).
  * Li  _et al._ [2012] Y. Li, L. P. Pitaevskii, and S. Stringari, Quantum tricriticality and phase transitions in spin-orbit coupled bose-einstein condensates, Phys. Rev. Lett. 108, 225301 (2012).
  * Hu  _et al._ [2012] H. Hu, B. Ramachandhran, H. Pu, and X.-J. Liu, Spin-orbit coupled weakly interacting bose-einstein condensates in harmonic traps, Phys. Rev. Lett. 108, 010402 (2012).
  * Zhang  _et al._ [2012] Y. Zhang, L. Mao, and C. Zhang, Mean-field dynamics of spin-orbit coupled bose-einstein condensates, Phys. Rev. Lett. 108, 035302 (2012).
  * Ozawa  _et al._ [2013] T. Ozawa, L. P. Pitaevskii, and S. Stringari, Supercurrent and dynamical instability of spin-orbit-coupled ultracold bose gases, Phys. Rev. A 87, 063610 (2013).
  * Zhang  _et al._ [2016] Y.-C. Zhang, Z.-Q. Yu, T. K. Ng, S. Zhang, L. Pitaevskii, and S. Stringari, Superfluid density of a spin-orbit-coupled bose gas, Phys. Rev. A 94, 033635 (2016).
  * Stringari [2017] S. Stringari, Diffused vorticity and moment of inertia of a spin-orbit coupled bose-einstein condensate, Phys. Rev. Lett. 118, 145302 (2017).
  * Qu and Stringari [2018] C. Qu and S. Stringari, Angular momentum of a bose-einstein condensate in a synthetic rotational field, Phys. Rev. Lett. 120, 183202 (2018).
  * [24] The other degeneracy condition can be analytically obtained as $\omega_{z}=\sqrt{2(m\omega_{x}^{2}/m^{*}+\omega_{y}^{2})/5}$, .
  * [25] See the supplemental material for detailed information on the derivation of the equations of motion for the collective modes, .
  * Stringari [2001] S. Stringari, Superfluid gyroscope with cold atomic gases, Phys. Rev. Lett. 86, 4725 (2001).
  * Hodby  _et al._ [2003] E. Hodby, S. A. Hopkins, G. Hechenblaikner, N. L. Smith, and C. J. Foot, Experimental observation of a superfluid gyroscope in a dilute bose-einstein condensate, Phys. Rev. Lett. 91, 090403 (2003).
  * Recati  _et al._ [2001] A. Recati, F. Zambelli, and S. Stringari, Overcritical rotation of a trapped bose-einstein condensate, Phys. Rev. Lett. 86, 377 (2001).
  * [29] C.-H. Li, et al., unpublished .