# ShapeGraFormer: GraFormer-Based Network for Hand-Object Reconstruction from
a Single Depth Map

Ahmed Tawfik Aboukhadra<EMAIL_ADDRESS>Jameel Malik  Nadia
Robertini  Ahmed Elhayek  Didier Stricker

###### Abstract

3D reconstruction of hand-object manipulations is important for emulating
human actions. Most methods dealing with challenging object manipulation
scenarios, focus on hands reconstruction in isolation, ignoring physical and
kinematic constraints due to object contact. Some approaches produce more
realistic results by jointly reconstructing 3D hand-object interactions.
However, they focus on coarse pose estimation or rely upon known hand and
object shapes. We propose the first approach for realistic 3D hand-object
shape and pose reconstruction from a single depth map. Unlike previous work,
our voxel-based reconstruction network regresses the vertex coordinates of a
hand and an object and reconstructs more realistic interaction. Our pipeline
additionally predicts voxelized hand-object shapes, having a one-to-one
mapping to the input voxelized depth. Thereafter, we exploit the graph nature
of the hand and object shapes, by utilizing the recent GraFormer network with
positional embedding to reconstruct shapes from template meshes. In addition,
we show the impact of adding another GraFormer component that refines the
reconstructed shapes based on the hand-object interactions and its ability to
reconstruct more accurate object shapes. We perform an extensive evaluation on
the HO-3D and DexYCB datasets and show that our method outperforms existing
approaches in hand reconstruction and produces plausible reconstructions for
the objects.

###### keywords:

Computer Vision , Deep Learning , Graph Convolutional Network , Transformers ,
Hand-Object 3D Reconstruction , Pose Estimation

††journal: Pattern Recognition

[dfki]organization=Augmented Vision, DFKI,addressline=Trippstadter Str. 122,
city=Kaiserslautern, postcode=67663, state=Rhineland-Palatinate,
country=Germany

[rptu]organization=University of Kaiserslautern-Landau
(RPTU),addressline=Erwin-Schrödinger-Straße 52, city=Kaiserslautern,
postcode=67663, state=Rhineland-Palatinate, country=Germany

[nust]organization=NUST-SEECS,city=Islamabad, country=Pakistan

[upm]organization=University of Prince Mugrin (UPM),city=Madinah,
postcode=42241, country=Saudi Arabia

##  1 Introduction

Understanding and reconstructing hand and object interactions in 3D is
important for analyzing and imitating human behavior. Modeling hand-object
interactions realistically has applications in a number of fields, including
robotics, virtual reality, and augmented reality among others. The last decade
has witnessed rapid advance in 3D hand pose [1, 2, 3, 4, 5, 6, 7, 8, 9] and
object [10, 11, 12, 13, 14] estimation in isolation. In contrast,
reconstructing a hand and an object simultaneously from a monocular image has
received lesser attention. Besides the common issues from complex pose
variation, clutter, and self-occlusion, methods for reconstructing hand-object
in close contact have to additionally cope with mutual occlusions. Existing
methods, dealing with challenging object manipulation scenarios, tend to focus
on hand reconstruction alone [15, 16]. Recent approaches to jointly
reconstruct hand and object, often neglect the intrinsic kinematic and
physical correlation, which exists among the two [17, 18, 19, 20]. Approaches
exploiting that mutual relation typically focus on coarse pose estimation or
assume known hand and object shapes [21, 22, 23, 24, 25, 26, 27].

In this paper, we propose one of the first approaches to jointly reconstruct
physically valid hand and object shapes from a single depth map. In contrast
to most methods, ours can generalize to different hand models and unknown
object shapes, by directly regressing mesh vertices, rather than model
parameters. We avoid perspective distortion and scale ambiguities, typical for
RGB image-based methods by working exclusively in the 3D domain. The input of
our deep network is a 3D voxelized grid of a given depth map, centered around
the hand-object interaction. The output consists in: (i) 3D heatmaps that
describe the location of hand-object pose keypoints (ii) hand-object shape
predictions in voxelized form, and (iii) the corresponding 3D hand-object mesh
vertex coordinates.

To effectively tackle the problem of simultaneous hand-object pose and shape
reconstruction, we propose a novel architecture based on Graph Convolutional
network and Multi-headed Attention layers. Specifically, we introduce the
following novel modules:

  1. 1.

$PoseNet$ & $VoxelNet$: Two 3D-to-3D voxel-based networks for hand-object pose
and shape estimation, respectively;

  2. 2.

$ShapeGraFormer$: State-of-the-art GraFormer (Transformers with Graph
Convolutional layers) for Hand-Object shape reconstruction;

  3. 3.

Positional Embedding layer based on the template meshes for the hand and the
object;

  4. 4.

Topologically consistent object mesh registration for optimal object modeling
and shape prediction;

We validate our design choices and evaluate our approach both quantitatively
and qualitatively. Our approach outperforms previous work on popular datasets
[28, 29], as also reported on the challenge website
111https://codalab.lisn.upsaclay.fr/competitions/4393#results, with a minimal
shape reconstruction improvement of $0.43$ cm over the state-of-the-art.

##  2 Related Work

In this section, we discuss the existing methods for joint hand-object
reconstruction from challenging monocular object manipulation scenarios. For a
survey of works focusing on the reconstruction of hands and objects in
isolation, please refer respectively to [9] and [11].

Most methods that jointly reconstruct 3D hand-object from monocular take
single RGB [24, 23, 26, 27, 30, 31, 20] or RGB+D input [21, 22, 17]. Very few
recent approaches consider single depth input [32, 33, 34], due to its
intrinsic challenges. Nevertheless, the availability of the depth information
is a key factor to allow proper, in-scale and scene-dependent 3D shape
reconstruction, required for e.g. virtual and augmented reality applications,
especially for single frame, one-shot approaches. While RGB-based approaches
can rely on large labeled data for training their models [35, 18, 23],
typically based on the MANO hand model [36], methods that rely on the depth
channel have access to only a limited amount of data, because labeling of real
scenes in the 3D domain are impractical. For this reason, most depth as well
as some RGB+D approaches build their own synthetic datasets as an attempt to
improve their results [32, 23, 33, 34, 17]. Recently, some datasets have been
introduced to bridge the gap between RGB and depth data availability, i.e.
HO3D and DexYCB [28, 29]. Still, they only provide limited sample variation,
especially in terms of number of objects considered. As a result, modeling and
reconstructing the object of interaction remains an underconstrained,
challenging problem. Many methods restrict themselves on objects with known
shape [21, 22] or reconstruct them on the fly, under certain object shape and
visibility assumptions [34, 37, 38]. The remaining approaches only output
coarse object pose, e.g. represented with bounding boxes [24, 30, 19, 31].
Some more sophisticated methods, define a deformable object model, typically
based on a 3D sphere, capable to coarsely adapt to virtually any convex shape
[23, 37, 20]. The resulting object reconstruction typically lacks of surface
details, due to over-smoothing. Nevertheless, we believe it to be a strong
base with sufficient surface information to reliably reconstruct interactions
with the hand surface. The remaining hand reconstruction as a standalone
problem has been largely studied in the past [39]. However, modeling and
reconstructing 3D hand-object interactions is still very challenging,
especially on a monocular setting due to the large mutual occlusions. To
simplify the task, many approaches focus on fewer model parameter estimation
[23, 26]. In contrast, we directly regress hand-object vertices, which makes
our method capable to generalize to different models or geometries. While many
approaches independently reconstruct hand and object before putting them in
context [23, 26], our approach is designed to simultaneously regress hand-
object geometries, thus allowing to implicitly study the underlying physical
and kinematic correlation existing among the two. Another advantage of direct
vertex regression is the possibility to reconstruct extra, more realistic
deformation, which cannot be synthesized via model-parameter tuning. In this
case, to avoid distortions in the 3D reconstructions, though, particular care
has to be given to the algorithm design, especially to the shape estimation
components. To avoid perspective distortion from the start, we convert the
input depth map to 3D pointcloud and base the remaining algorithmic steps to
the corresponding voxelized domain, following the approach from Malik et al.
[9] (HandVoxNet). The core of our pipeline is based on Graph Convolutional
neural network (GCN), which has been shown to effectively tackle shape
reconstruction problems on graph-structured data, such as mesh topology [40].
In contrast to the RGB-based approach by Aboukhadra et al. [20] (THOR-Net),
our depth-based method reconstructs hand-object geometries in one-shot, thus
significantly reducing computational costs and training time. We qualitatively
and quantitatively demonstrate the effectiveness of our $ShapeGraFormer$
component, comprising a combination of GCN and Multi-headed Attention layers,
as in [41], in the simultaneous reconstruction of hand and object interaction
as well as in shape refinement. For the latter, we demonstrate its
effectiveness in improving physical hand-object interactions, without the
explicit need for expensive physical simulation [21] or penetration and
contact loss, as required in previous work [26, 23].

##  3 Method

We design a voxel-based 3D CNN along with a $ShapeGraFormer$ network to
reconstruct plausible 3D hand-object shapes in a single forward pass from an
input depth image. Our pipeline is depicted in Figure 1. In a preprocessing
step, we convert the input depth map to its voxelized form $V_{D}$ by
projecting the raw depth image pixels into a cubic binary 3D grid around the
hand-object interaction space, similarly to [9]. Given $V_{D}$, the first
network component in the pipeline, $PoseNet$, predicts 3D hand and object pose
in the form of 3D heatmaps, resp. $\hat{P}^{H}$ and $\hat{P}^{O}$, see Section
3.2. The resulting heatmaps concatenated with $V_{D}$ are forwarded to the
second network component, $VoxelNet$, which produces a voxelized shape
representation of the hand and the object, resp. $\hat{V}^{H}$ and
$\hat{V}^{O}$, see Section 3.3. The voxelized depth $V_{D}$ along with the
intermediate voxelized representation and the features of the $VoxelNet$ serve
as input to the next network component, $ShapeGraFormer$, which regresses
topologically-consistent hand and object vertices, see Section 3.4. We
describe our hand and object models in the next Section.

Figure 1: Overview of our pipeline. Our method takes as input a single depth
image, converted to a 3D voxelized representation, and outputs 3D realistic
hand-object interactions. The input depth is first forwarded through a
sequence of three components, namely $PoseNet$, $VoxelNet$, and
$ShapeGraFormer$, predicting respectively hand-object pose heatmaps, voxelized
shape representations, and topologically-consistent shapes.

###  3.1 Hand and Object Models

##### Hand Model

To represent hands, we use MANO parametric hand model [36], which maps joint
angles and shape parameters to a triangulated mesh representation
$S^{H}:=\\{v^{H},f^{H}\\}$, consisting of $|v_{H}|=778$ 3D vertices, and a set
of 3D hand-skeleton joints $\mathcal{J}$, where $|\mathcal{J}|=21$,
representing the hand pose.

##### Object Model

Since objects differ greatly in terms of shape and size, direct regression of
object vertices from a set of topologically inconsistent meshes results in
strong noise in the output. In order to bring all the known object geometries
into the same topologically consistent representation
$S^{O}:=\\{v^{O},f^{O}\\}$, we deform a source mesh via a set of vertex
displacements $D^{j}:=\\{d^{j}_{i},\forall v^{O}_{i}\\}$, scale $c_{j}$ and
translation $t_{j}$ to approximate all target objects $S^{j}$ as:

$S^{j}\sim c^{j}\cdot(S^{O}+D^{j})+t^{j},\forall j$ (1)

Our source mesh is a sphere obtained performing $4$ subdivisions of a unit,
i.e.radius equals to 1, regular icosahedron centered at the origin, each
subdivision generating four new faces per face, resulting in a total of
$|v^{O}|=2562$ vertices and $|f^{O}|=5120$ faces. The object pose is defined
as the 3D bounding box, consisting of eight 3D corner coordinates.

##### Object Approximation Procedure

In order to approximate differently shaped objects, we first scale ($c^{j}$)
and translate ($t^{j}$) the target mesh $S^{j}$ to fit inside the sphere.
Then, to learn the set of displacements $D^{j}$, we minimize the Chamfer
distance between the predicted and target mesh computed on a total of $5,000$
surface samples. We additionally enforce surface smoothness by adding the
following shape regularizers to the objective: (i) surface Laplacian
smoothness, (ii) normals consistency across neighboring faces, and (iii) edge
length consistency across the entire deformed mesh. We minimize the weighted
summation of all mentioned terms using stochastic gradient descent (SGD).

Figure 2: The complete collection of our sphere-based object approximations
used as ground-truth.

###  3.2 PoseNet: 3D Pose Estimation

The first component of our network pipeline, $PoseNet$, simultaneously
estimates 3D hand joint location and 3D bounding box corners of the object
from an input voxelized depth map. We modify the V2V-PoseNet architecture of
[42] by introducing a new $1\times 1\times 1$ volumetric convolutional back-
layer, targeted to predict 3D object pose in parallel to the 3D hand pose. The
input depth map is converted to a 3D binary voxelized representation $V_{D}$
of size $88\times 88\times 88$. Each voxel value is set to $1$ when occupied
and $0$ otherwise, i.e.$V_{D}\in\\{0,1\\}$. The cubic size is fixed to the
empirically found value $200$ mm. The output of the network consists in a set
of 3D heatmaps: (i) One set for each $j$ hand joints coordinates
$\hat{P}^{H}_{j}(v)$, and (ii) one for each $b$ object bounding box corners
coordinates $\hat{P}^{O}_{b}(v)$. All heatmaps are discretized on a grid of
size $44\times 44\times 44$. Ground-truth heatmaps $P_{j}^{H}(v)$ and
$P_{b}^{O}(v)$ are generated by applying a 3D Gaussian centered on the ground-
truth locations with fixed standard deviation. The training loss between the
predicted and targeted heatmaps is calculated by mean-squared error (MSE):

$\mathcal{L}_{pose}=\sum_{v}\sum_{j=1}^{|J|}||\hat{P}^{H}_{j}-P^{H}_{j}||+\sum_{b=1}^{|B|}||\hat{P}^{O}_{b}-P^{O}_{b}||$
(2)

where $J=21$ is the number of hand joints and $B=8$ is the number of object
corners In the above formula, we omitted all dependencies on the voxels $v$
for brevity.

###  3.3 VoxelNet: 3D Voxelized Shape

Given $V_{D}$, $\hat{P}^{H}$ and $\hat{P}^{O}$, the second component of our
pipeline, $VoxelNet$, predicts hand-object voxelized shapes $\hat{V}^{H}(v)$
and $\hat{V}^{O}(v)$ for all existing voxels $v$ in the grid. The 3D CNN-based
architecture of $VoxelNet$ is inspired by the work of Malik et al. [9]. We
introduce an additional 3D convolutional layer to predict the voxelized object
shape. Our voxelized hand-object shapes are defined in the range $[0,1]$, as
done in the previous work. The predicted voxelized shapes represent complete
surface representation for both the hand and the object, including both
visible and occluded surface information, as learned from the dataset. Thus,
entailing richer information for the next algorithmic steps.

To train $VoxelNet$, we use the per-voxel combined sigmoid activation function
with binary cross-entropy loss for the voxelized hand shape (similarly for the
voxelized object shape):

$\mathcal{L}_{voxel}^{H}(v)=-(V^{H}log(\hat{V}^{H})+(1-V^{H})log(1-\hat{V}^{H}))$
(3)

where $V^{H}$ and $\hat{V}^{H}$ are the ground truth and the estimated
voxelized hand and object shapes, respectively.

###  3.4 ShapeGraFormer

To obtain topologically-coherent hand-object mesh representation, from the
voxelized hand and object predictions obtained from $VoxelNet$, we train a new
$ShapeGraFormer$ [41]. The $ShapeGraFormer$ includes Graph Convolutional
layers and Multi-headed Attention layers in order to convert depth features
into a pose and a shape for both the hand and the object. Although it was
originally designed to lift 2D poses to 3D, GraFormer’s design which combines
the advantages of Graph Convolutional Networks and Transformers making it
capable of effectively solving any problem that can be represented as a graph.
Both the hand and the spherical objects have consistent topology and hence can
be represented as a graph. Therefore, in our method, we utilize three separate
GraFormers, namely: hand GraFormer, object GraFormer, and refinement
GraFormer, with two feature extractors: a hand feature extractor and an object
feature extractor. as described in the following subsection.

Specifically, the network outputs hand-object vertex coordinates $v^{H}$ and
$v^{O}$, respectively from the MANO hand model $S^{H}$ and the sphere-based
object approximated shape $S^{O}$. The hand and object vertices loss for
training is defined using MSE:

$\mathcal{L}_{shape}=\frac{\sum_{i}(v^{H}_{i}-\hat{v}^{H}_{i})^{2}}{|v^{H}|}+\frac{\sum_{j}(v^{O}_{j}-\hat{v}^{O}_{j})^{2}}{|v^{O}|}$
(4)

where $\hat{v}^{H}_{i}$ and $v^{H}_{i}$ are respectively the predicted and
ground-truth $i$-th hand vertex coordinates, and $\hat{v}^{O}_{j}$ and
$v^{O}_{j}$ are predicted and ground-truth object vertex coordinates.

####  3.4.1 Feature Extractor and Graph Initialization

For the graph initialization, we use the outputs of the $VoxelNet$ and the raw
voxelized depth as shown in Figure 1. Namely, we extract featuremaps
$\mathcal{F}_{V}$ from the $VoxelNet$ and pass them through an MLP that
converts them into a feature vector of size $256$. Additionally, we utilize a
simple 3D CNN to extract features from the voxelized shapes and reduce them to
$128$ features. Furthermore, a 3D Max Pool layer reduces the size of the raw
voxelized depth from $44\times 44\times 44$ to $11\times 11\times 11$ creating
a $1331$ feature vector. These features are then combined to create a hand
feature vector $\mathcal{F}^{H}_{1715}$ of size $1715$ to initialize the graph
vertices as shown in Figure 1. The same operation is repeated for the object’s
vertices resulting in $\mathcal{F}^{O}_{1715}$.

####  3.4.2 Positional Embedding

In order to generate distinct features for each vertex in the graph, we
propose a positional embedding layer that converts the vertices of the
template meshes for both the hand and object into positional vectors
$\mathcal{E}_{i}^{p}$ of the same size as the feature vector where i is the
index of the ith vertex in the combined hand-object graph. The positional
vectors $\mathcal{E}_{i}^{p}$ are then accumulated with the shared feature
vectors $\mathcal{F}^{H}_{1715}$ and $\mathcal{F}^{O}_{1715}$ depending on
whether the ith vertex belongs to the hand or the object in order to create a
unique representation for each vertex. For the hand mesh, we adopt the default
MANO hand as the template, while for the object mesh, we use the sphere that
is utilized for deforming objects, see Section 3.1.

####  3.4.3 GraFormer Details

Each GraFormer consists of five consecutive GraAttention components followed
by Chebyshev Graph Convolutional layers as described in [41] and shown in
Figure 3. A GraAttention component consists of a Multi-headed Attention layer
where the hidden dimension is $128$ and the number of heads is $4$, following
the ablation study mentioned in [41] where they studied the impact of hidden
dimension size and number of layers in the GraFormer on pose estimation.
Compared to normal Transformers, the last fully-connected layer of the Multi-
headed Attention in the GraFormer is a graph convolutional layer, not a feed-
forward layer. The GraFormer also contains an input layer that maps the
feature vector to the hidden dimension size and an output layer that maps the
hidden dimension into the corresponding 3D coordinate value for each vertex in
the graph. To initialize the adjacency matrix of the graph layers, we use the
mesh faces of the MANO model along with the faces of the spherical mesh.

Figure 3: Illustration of the GraFormer architecture.

####  3.4.4 Refinement GraFormer

To enhance the realistic appearance and correct minor shape-related artifacts
in the predicted object shapes, we add an additional GraFormer for refinement.
We utilize the initial shape produced by the Hand and Object GraFormer as
input to another positional embedding layer while we use
$\mathcal{F}^{H}_{1715}$ and $\mathcal{F}^{O}_{1715}$ to initialize the new
graph. In the case of using a refinement GraFormer during training, we get $2$
separate shapes from the $ShapeGraFormer$ and apply the same loss function
mentioned in 3.4 on both of them.

###  3.5 Training and Implementation Details

We train all the components of our network on fully annotated public hand-
object pose and shape datasets, namely, HO-3D [28] and DexYCB [29]. For
training, we use the Adam [43] optimizer with a learning rate set to $.001$.
For improved convergence, we train $PoseNet$ separately, then fix the obtained
weights to train the remaining network components. All learning and inference
are implemented in PyTorch and are conducted on an NVIDIA A100 GPU.

##  4 Experiments

In this section, we quantitatively and qualitatively evaluate the
effectiveness of our 3D hand-object reconstruction pipeline on two popular
datasets, namely HO-3D dataset (version v2 and v3) [28] and DexYCB [29]. We
additionally compare our pipeline with state-of-the-art approaches in Section
4.2. For quantitative evaluation and comparisons, we use the following two
metrics: (i) the average 3D joint location error and (ii) the mean vertex
location error over all test frames.

###  4.1 Datasets

##### HO-3D

The HO-3D dataset [28], is a publicly available dataset with 3D pose
annotations for hands interacting with objects captured from third-person
views. The dataset has multiple versions and we report the results on v2 and
v3 of the dataset. For HO-3D (v3), the training set
$\mathcal{D}:=\\{\mathcal{D}^{train},\mathcal{D}^{valid}\\}$ contains
annotations for $|\mathcal{D}^{train}|=71,662$ and
$|\mathcal{D}^{valid}|=10,927$ images and a total of $55$ sequences and $10$
different objects from the YCB-Video dataset [44], $9$ for the training set
and $1$ (unseen) for the evaluation set. The evaluation set
$\mathcal{D}^{eval}$ comprises $13$ sequences with a total of
$|\mathcal{D}^{eval}|=20,137$ frames addressing challenging scenarios: namely,
(i) $3$ sequences with ($2$) seen objects and seen hands, (ii) $5$ sequences
with ($1$) seen object but unseen hands, and (iii) $5$ sequences with seen
hands but $1$ unseen object. Hands in the evaluation set are just annotated
with the wrist coordinates, while the full hand is not annotated. Object pose
and shape are annotated over all available sets.

##### DexYCB

To extend our evaluation, we also train our network on the DexYCB dataset
[29]. DexYCB contains hand pose and shape and 6D object pose annotations for
$582k$ frames recorded on $10$ different subjects, using $20$ different
objects and $8$ views. We use the S1 evaluation setup as specified by the
authors where $\mathcal{D}^{valid}$ contains $1$ unseen subject and
$\mathcal{D}^{eval}$ contains $2$ unseen subjects and $\mathcal{D}^{train}$
contains 7 subjects and all different objects. The exact split sizes are:
$|\mathcal{D}^{train}|=407,088$, $|\mathcal{D}^{valid}|=58,592$, and
$|\mathcal{D}^{eval}|=116,288$.

In the next section, we show results on $\mathcal{D}^{eval}$ for both
datasets.

###  4.2 Evaluation

In this section, we evaluate hand-object shape and pose reconstruction and
compare it to state-of-the-art approaches in challenging scenarios.

##### Methods for Comparison

We compare our work on HO-3D quantitatively with six different methods. The
work of Hasson et al. [23] and THOR-Net [20] are the most related to ours in
terms of the goals. However, as they focus on RGB inputs, comparisons are made
up to scale. Malik et al. [9] on the other hand, is based on depth inputs and
has a comparable voxel-based network pipeline to our design. However, they
ignore the presence of an object and reconstruct hands in isolation. We
include the representative RGB-based hand reconstruction approach of Hampali
et al. [28], and ArtiBoost [31] for completeness. For a fair comparison, all
methods have been (re-)trained on the HO-3D dataset. The first two methods
provide publicly available results (hand-only), which we report in Table 1. We
re-implemented HandVoxNet following the authors’ instructions and trained all
the network components on HO-3D. In addition to that, we also report the root-
relative pose estimation error in Table 2 and compare it to two benchmark
methods mentioned by the DexYCB authors. We also show qualitative samples for
hand reconstruction from DexYCB in Figure 6.

##### Hand Reconstruction

Quantitative evaluation on hand pose and shape reconstruction, shows that our
method outperforms existing methods, see Table 1 and Figures 4, 5.

Method | Joint err. (cm) | Mesh err. (cm)
---|---|---
Hampali et al. [28] | 8.42 | 8.34
Hasson et al. [23] | 4.97 | 4.96
HandVoxNet [9] | 2.77 | 2.67
HandVoxNet++ [40] | 2.46 | 2.70
THOR-Net [20] | 2.63 | 2.63
Ours (hand-object, w/ contact loss) | 2.27 | 2.20
Ours (hand-only, w/o refinement) | 2.13 | 2.07
ArtiBoost [31] | 2.26 | 10.95
THOR-Net [20] | 2.56 | 2.37
Ours (hand-object, w/ refinement) | 2.15 | 2.09
Ours (hand-only, w/ refinement) | 2.00 | 1.94
Ours (hand-only, w/o refinement) | 1.99 | 1.94
Table 1: Quantitative results on the evaluation set $\mathcal{D}^{eval}$ of
HO-3D [28] dataset. Upper table: HO-3D (v2), lower table: HO-3D (v3).

We achieve a minimum pose and shape prediction improvement respectively of
$0.27$ cm and $0.43$ cm over the state-of-the-art. The experiments show that
training the network for hand reconstruction alone without object
reconstruction improves hand reconstruction and that the refinement stage has
no impact on hand reconstruction as shown in Table 1. However, joint training
for the hand and the object outperforms other methods. Figure 5 shows our
results on a few frames of the challenging evaluation sequence compared to
THOR-Net. With respect to THOR-Net, our approach reconstructs more accurate
hand shapes, as it better exploits hand-object kinematic correlation and depth
information. These are implicitly learned while predicting both interacting
shapes simultaneously. In Table 2, we show an improvement of $0.53$ cm in hand
pose estimation compared to benchmark results on the DexYCB dataset. Figure 6
also shows the qualitative reconstruction results on DexYCB without the
additional refinement network.

Figure 4: Reconstruction results at different steps. Each row shows a different sample from $\mathcal{D}^{valid}$ or $\mathcal{D}^{eval}$ from HO-3D (v3). Figure 5: Comparison of our hand-object reconstruction with the THOR-Net pipeline on the evaluation set $\mathcal{D}^{eval}$. Our hand-object reconstruction approach is shown to generate more accurate hand and object shapes. Figure 6: Reconstruction results at different steps. Each row shows a different sample from $\mathcal{D}^{eval}$ from DexYCB.  Method | Joint err. (cm) | Mesh err. (cm)
---|---|---
A2J [45] | 2.55 | -
Spurr et al. [46] | 2.27 | -
Ours | 1.74 | 2.65
Table 2: Root-relative hand pose estimation results on $\mathcal{D}^{eval}$ of
DexYCB [29] dataset (S1) in comparison to benchmark results.

##### Object Reconstruction

We found that topological consistency is the key factor allowing
$ShapeGraFormer$ to predict smooth vertices point clouds across all sequences,
without the need for additional smoothness constraints, see Figure 4. After
sphere-based registration, all objects share the same topology and number of
vertices. We believe our choice for the sphere resolution to be a good trade-
off between approximation quality and the number of vertices. Sphere-based
approximation implicitly repairs irregularities in the target object shapes,
making it best suited for hand-object prediction, at the cost of over-smoothed
sharp edges and tiny surface details, see Figure 2.

We evaluate object reconstruction on $\mathcal{D}^{valid}$ and
$\mathcal{D}^{eval}$ of HO-3D dataset, see Figure 7 and Table 3, as well as on
the DexYCB dataset, see Figure 6. The additional hand-object refinement step
improved object reconstruction error on all objects. This suggests that the
refinement network utilizes hand information in order to improve object
reconstruction. We notice that even in presence of inaccurate pose prediction
as input, our approach recovers smooth object shapes, see Figure 4. Compared
to THOR-Net our approach tend to oversmooth object edges, see Figure 5,
possibly due to the simplified reconstruction approach.

Figure 7: Object reconstruction error (in mm) on different objects from $\mathcal{D}^{valid}$ and $\mathcal{D}^{eval}$ of HO-3D. The results show that adding a refinement GraFormer improves object reconstruction. | Bottle | Box | Can | Marker | Wood
---|---|---|---|---|---
MPVPE (cm) | 5.9 | 6.4 | 7.0 | 9.7 | 11.5
Table 3: Object Reconstruction MPVPE on a selected set of objects from HO-3D
and DexYCB.

###  4.3 Ablation Study

To validate our design choices for the $ShapeGraFormer$, we conduct an
ablation study to study the modality of $\mathcal{F}_{1715}^{H}$ and the
choice of layers within the GraFormer and study the impact on hand
reconstruction. The ablation study in Table 4 shows that combining voxelized
depth $V_{D}$, $\mathcal{F}_{V}$ and $\hat{V}^{H}$ as input to the
$ShapeGraFormer$ yields the best results. Furthermore, the mixture of graph
layers with transformers in the design of the GraFormer is critical to achieve
best performance.

Experiment | Joint err. (cm) | Mesh err. (cm)
---|---|---
ShapeGraFormer (w/o $V_{D}$) | 2.01 | 2.06
ShapeGraFormer (w/o $\mathcal{F}_{V}$) | 2.05 | 2.01
ShapeGraFormer (w/o $\hat{V}^{H}$) | 2.00 | 1.95
ShapeGraFormer (w/ $V_{D}\oplus\mathcal{F}_{V}\oplus\hat{V}^{H}$) | 1.99 | 1.94
ShapeGraFormer (w/o GCN) | 19.32 | 19.36
ShapeGraFormer (w/o Transformer) | 2.06 | 2.01
ShapeGraFormer (w/ GCN + Transformer) | 1.99 | 1.94
Table 4: Top: An ablation study on the modality of $\mathcal{F}_{1715}^{H}$.
Bottom: An ablation study on the GraFormer design choice.

##### Contact Loss: Penetration Avoidance and Contact Enforcement

We test the impact of differentiable contact loss, consisting of an attraction
$\mathcal{L}_{attraction}$ and a repulsion $\mathcal{L}_{repulsion}$ term,
similar to Hasson et al. [23]. $\mathcal{L}_{attraction}$ is aimed at
enforcing hand-object contact, by penalizing the distance between the object
and the fingertips, while $\mathcal{L}_{repulsion}$ penalizes mesh
interpenetration. As shown in Table 1, the additional loss term does not
introduce a tangible increase in performance. Thus, suggesting our independent
Refinement GraFormer component implicitly and successfully learns valid hand-
object interactions.

##  5 Conclusion

In this paper, we propose one of the first methods for realistic hand-object
pose and shape reconstruction from a single depth map. We introduce a novel 3D
voxel-based GraFormer network pipeline, which reconstructs detailed 3D shapes
via direct regression of mesh vertices. We conduct an ablation study to show
the effectiveness of our design choices and the impact of utilizing the power
of GCNs along with Transformers for hand-object shape estimation and
refinement. We perform quantitative and qualitative analysis on the HO-3D
dataset [28] and DexYCB dataset [29] and show outstanding comparative results
with the state-of-the-art. In future work, we plan to address limitations such
as inaccurate annotations. In addition, we will study RGB+D methods to utilize
the extra features found in RGB frames.

##### Acknowledgments

This work was partially funded by the Federal Ministry of Education and
Research of the Federal Republic of Germany (BMBF), under grant agreements:
DECODE [Grant Nr 01IW21001], GreifbAR [Grant Nr 16SV8732], and by the EU
project FLUENTLY [Grant Nr 101058680].

## References

  * [1] J. Malik, A. Elhayek, F. Nunnari, K. Varanasi, K. Tamaddon, A. Heloir, D. Stricker, Deephps: End-to-end estimation of 3d hand pose and shape by learning from synthetic depth, in: International Conference on 3D Vision (3DV), 2018.
  * [2] J. Malik, A. Elhayek, D. Stricker, Structure-aware 3d hand pose regression from a single depth image, in: International Conference on Virtual Reality and Augmented Reality (EuroVR), 2018.
  * [3] L. Ge, Z. Ren, Y. Li, Z. Xue, Y. Wang, J. Cai, J. Yuan, 3d hand shape and pose estimation from a single rgb image, in: Computer Vision and Pattern Recognition (CVPR), 2019.
  * [4] D. Kulon, R. A. Guler, I. Kokkinos, M. M. Bronstein, S. Zafeiriou, Weakly-supervised mesh-convolutional hand reconstruction in the wild, in: Computer Vision and Pattern Recognition (CVPR), 2020.
  * [5] X. Zhang, Q. Li, H. Mo, W. Zhang, W. Zheng, End-to-end hand mesh recovery from a monocular rgb image, in: International Conference on Computer Vision (ICCV), 2019.
  * [6] J. Yang, H. J. Chang, S. Lee, N. Kwak, Seqhand: Rgb-sequence-based 3d hand pose and shape estimation, in: European Conference on Computer Vision (ECCV), 2020\.
  * [7] J. Malik, A. Elhayek, D. Stricker, Whsp-net: A weakly-supervised approach for 3d hand shape and pose recovery from a single depth image, Sensors 19 (17) (2019) 3784.
  * [8] Y. Zhou, M. Habermann, W. Xu, I. Habibie, C. Theobalt, F. Xu, Monocular real-time hand shape and motion capture using multi-modal data, in: Computer Vision and Pattern Recognition (CVPR), 2020.
  * [9] J. Malik, I. Abdelaziz, A. Elhayek, S. Shimada, S. A. Ali, V. Golyanik, C. Theobalt, D. Stricker, Handvoxnet: Deep voxel-based network for 3d hand shape and pose estimation from a single depth map, in: Computer Vision and Pattern Recognition (CVPR), 2020.
  * [10] D. Tzionas, J. Gall, 3d object reconstruction from hand-object interactions, International Conference on Computer Vision (ICCV) (2015).
  * [11] B. Tekin, S. N. Sinha, P. Fua, Real-time seamless single shot 6d object pose prediction, Conference on Computer Vision and Pattern Recognition (CVPR) (2018).
  * [12] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, L. Guibas, Normalized object coordinate space for category-level 6d object pose and size estimation, in: Computer Vision and Pattern Recognition (CVPR), 2019.
  * [13] S. Peng, Y. Liu, Q. Huang, H. Bao, X. Zhou, Pvnet: Pixel-wise voting network for 6dof pose estimation, in: Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [14] C. Wang, R. Martín-Martín, D. Xu, J. Lv, C. Lu, L. Fei-Fei, S. Savarese, Y. Zhu, 6-pack: Category-level 6d pose tracker with anchor-based keypoints, in: International Conference on Robotics and Automation (ICRA), 2020\.
  * [15] F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, C. Theobalt, Real-time hand tracking under occlusion from an egocentric rgb-d sensor, in: International Conference on Computer Vision (ICCV), 2017.
  * [16] S. Baek, K. I. Kim, T.-K. Kim, Weakly-supervised domain adaptation via gan and mesh model for estimating 3d hand poses interacting objects, in: Computer Vision and Pattern Recognition (CVPR), 2020.
  * [17] S. Sridhar, F. Mueller, M. Zollhöfer, D. Casas, A. Oulasvirta, C. Theobalt, Real-time joint tracking of a hand manipulating an object from rgb-d input, in: European Conference on Computer Vision (ECCV), 2016.
  * [18] F. Mueller, F. Bernard, O. Sotnychenko, D. Mehta, S. Sridhar, D. Casas, C. Theobalt, Ganerated hands for real-time 3d hand tracking from monocular rgb, in: Computer Vision and Pattern Recognition (CVPR), 2018.
  * [19] B. Doosti, S. Naha, M. Mirbagheri, D. Crandall, Hope-net: A graph-based model for hand-object pose estimation, in: Computer Vision and Pattern Recognition (CVPR), 2020.
  * [20] A. T. Aboukhadra, J. Malik, A. Elhayek, N. Robertini, D. Stricker, Thor-net: End-to-end graformer-based realistic two hands and object reconstruction with self-supervision, in: Winter Conference on Applications of Computer Vision (WACV), 2023, pp. 1001–1010.
  * [21] D. Tzionas, L. Ballan, A. Srikantha, P. Aponte, M. Pollefeys, J. Gall, Capturing hands in action using discriminative salient points and physics simulation, International Journal of Computer Vision (IJCV) 118 (2) (2016) 172–193.
  * [22] A. Tsoli, A. A. Argyros, Joint 3d tracking of a deformable object in interaction with a hand, in: European Conference on Computer Vision (ECCV), 2018\.
  * [23] Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, I. Laptev, C. Schmid, Learning joint reconstruction of hands and manipulated objects, in: Computer Vision and Pattern Recognition (CVPR), 2020.
  * [24] B. Tekin, F. Bogo, M. Pollefeys, H+ o: Unified egocentric recognition of 3d hand-object poses and interactions, in: Computer Vision and Pattern Recognition (CVPR), 2019.
  * [25] H. Zhang, Z.-H. Bo, J.-H. Yong, F. Xu, Interactionfusion: real-time reconstruction of hand poses and deformable objects in hand-object interactions, ACM Transactions on Graphics (TOG) 38 (4) (2019) 1–11.
  * [26] Z. Cao, I. Radosavovic, A. Kanazawa, J. Malik, Reconstructing hand-object interactions in the wild, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12417–12426.
  * [27] Y. Hasson, B. Tekin, F. Bogo, I. Laptev, M. Pollefeys, C. Schmid, Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction, in: Computer Vision and Pattern Recognition (CVPR), 2020.
  * [28] S. Hampali, M. Rad, M. Oberweger, V. Lepetit, Honnotate: A method for 3d annotation of hand and object poses, in: Computer Vision and Pattern Recognition (CVPR), 2020.
  * [29] Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, J. Kautz, D. Fox, DexYCB: A benchmark for capturing hand grasping of objects, in: Computer Vision and Pattern Recognition (CVPR), 2021.
  * [30] S. Hampali, S. D. Sarkar, M. Rad, V. Lepetit, Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation, in: Computer Vision and Pattern Recognition (CVPR), 2022, pp. 11090–11100.
  * [31] L. Yang, K. Li, X. Zhan, J. Lv, W. Xu, J. Li, C. Lu, Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis, in: Computer Vision and Pattern Recognition (CVPR), 2022, pp. 2750–2760.
  * [32] M. Oberweger, P. Wohlhart, V. Lepetit, Generalized feedback loop for joint hand-object pose estimation, CoRR (2019).
  * [33] J. Chen, M. Yan, J. Zhang, Y. Xu, X. Li, Y. Weng, L. Yi, S. Song, H. Wang, Tracking and reconstructing hand object interactions from point cloud sequences in the wild, arXiv preprint arXiv:2209.12009 (2022).
  * [34] H. Zhang, Y. Zhou, Y. Tian, J.-H. Yong, F. Xu, Single depth view based real-time reconstruction of hand-object interactions, ACM Trans. Graph. (2021).
  * [35] G. Garcia-Hernando, S. Yuan, S. Baek, T.-K. Kim, First-person hand action benchmark with rgb-d videos and 3d hand pose annotations, in: Computer Vision and Pattern Recognition (CVPR), 2018.
  * [36] J. Romero, D. Tzionas, M. J. Black, Embodied hands: modeling and capturing hands and bodies together, ACM Transactions on Graphics (TOG) 36 (6) (2017) 245\.
  * [37] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, M. Aubry, A papier-mache approach to learning 3d surface generation, Conference on Computer Vision and Pattern Recognition (CVPR) (2018).
  * [38] H. Kato, Y. Ushiku, T. Harada, Neural 3d mesh renderer, in: Computer Vision and Pattern Recognition (CVPR), 2018.
  * [39] L. Huang, B. Zhang, Z. Guo, Y. Xiao, Z.-G. Cao, J. Yuan, Survey on depth and rgb image-based 3d hand shape and pose estimation, Virtual Reality & Intelligent Hardware (2021).
  * [40] J. Malik, S. Shimada, A. Elhayek, S. A. Ali, C. Theobalt, V. Golyanik, D. Stricker, Handvoxnet++: 3d hand shape and pose estimation using voxel-based neural networks, IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (12) (2022) 8962–8974.
  * [41] W. Zhao, W. Wang, Y. Tian, Graformer: Graph-oriented transformer for 3d pose estimation, in: Computer Vision and Pattern Recognition (CVPR), 2022, pp. 20438–20447.
  * [42] G. Moon, J. Yong Chang, K. Mu Lee, V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from a single depth map, in: Computer Vision and Pattern Recognition (CVPR), 2018.
  * [43] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: International Conference on Learning Representations (ICLR), 2015.
  * [44] Y. Xiang, T. Schmidt, V. Narayanan, D. Fox, Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes, in: Robotics: Science and Systems (RSS), 2018.
  * [45] F. Xiong, B. Zhang, Y. Xiao, Z. Cao, T. Yu, J. T. Zhou, J. Yuan, A2j: Anchor-to-joint regression network for 3d articulated pose estimation from a single depth image, in: International Conference on Computer Vision (ICCV), 2019\.
  * [46] A. Spurr, U. Iqbal, P. Molchanov, O. Hilliges, J. Kautz, Weakly supervised 3d hand pose estimation via biomechanical constraints, in: European Conference on Computer Vision (ECCV), 2020.


# Entanglement resonance in the asymmetric quantum Rabi model

Yu-Qing Shi School of Physical Science and Technology, Lanzhou University,
Lanzhou 730000, China College of Electrical Engineering, Northwest Minzu
University, Lanzhou 730030, China Lei Cong<EMAIL_ADDRESS>International
Center of Quantum Artificial Intelligence for Science and Technology
(QuArtist)
and Department of Physics, Shanghai University, 200444 Shanghai, China
Helmholtz-Institut, GSI Helmholtzzentrum fur Schwerionenforschung, Mainz
55128, Germany Department of Applied Physics, Nanjing Tech University, Nanjing
210009, China Hans-Peter Eckle<EMAIL_ADDRESS>Humboldt Study
Centre, Ulm University, Ulm D-89069, Germany

###### Abstract

We investigate the entanglement features in the interacting system of a
quantized optical field and a two-level system which is statically driven,
known as the asymmetric quantum Rabi model (AsymQRM). Intriguing entanglement
resonance valleys with the increase of the photon-atom coupling strength and
peaks with the increase of the driving amplitude are found. It is revealed
that both of these two kinds of entanglement resonance are caused by the
avoided level crossing of the associated eigenenergies. In sharp contrast to
the quantum Rabi model, the entanglement of the AsymQRM collapses to zero in
the strong coupling regime except when the driving amplitude equals to
$m\omega/2$, with $m$ being an integer and $\omega$ being the photon
frequency. Our analysis demonstrates that such entanglement reappearance is
induced by the hidden symmetry of the AsymQRM. Supplying an insightful
understanding on the AsymQRM, our results will be helpful in exploring the
hidden symmetry and in preparing photon-atom entanglement in light-matter
coupled systems.

##  I Introduction

Light-matter interaction is described by coupling Bosonic and Fermionic
subsystems. Modelling the interaction between Bosonic modes and the electrons
of atomic levels plays therefore a fundamental role in the physics of strongly
interacting quantum systems HarocheRaimond2006 , especially, but not
exclusively, in models related to Quantum Optics GarrisonChiao2008 .
For instance, resonance phenomena of optical systems where the frequency of a
single-mode (Bosonic) radiation field is such that it couples approximately
only to two relevant atomic levels, i.e. a two-level system (TLS) or qubit
(see below for this latter designation), have attracted considerable interest
since many years AllenEberly1987 and continue to do so, as evidenced in many
recent publications, a pertinent selection of which we are going to reference
in the course of this publication, especially its introduction. The two-level
system can be described also by spin degrees of freedom.
While the semi-classical treatment of such systems Rabi by I. I. Rabi
initiated the study of the eponymous model, the quantum version, introduced by
Jaynes and Cummings JC (for more details on the quantum Rabi model (QRM), see
e.g. Book8.10 ), has attracted considerable recent interest. A theoretical
breakthrough was achieved by the exact solution of the QRM DB2011 ;
ChenQingHu2012 using two different methods, which, however, both crucially
used the $\mathbb{Z}_{2}$ symmetry of the QRM and a Bargmann space analysis
(for these and further recent theoretical developments, see YuEtAl ; DB2013
and, DB2015 , and for recent reviews sym11101259 ; XieQiongtao2017 ;
AlexandreReview ; LarsonBook ; ArrazolaThesis ).
The QRM has been realized experimentally in solid state devices, physical
systems including cavity quantum electrodynamic (cavity-QED)
HarocheRaimond2006 , circuit quantum electrodynamic (circuit-QED)
circuitQEDReview2021 systems, and also in trapped ion systems
PhysRevX.8.021027 ; CaiEtAl2021 .
In recent years, it became possible, triggered by the technological
possibilities these and further experimental systems have opened up, to tune
the parameters of the QRM, in particular to reach new regimes where the
interaction between the Bosonic modes and the TLS is strong, i.e. to reach the
so-called ultra-strong (where the ratio of the field-TLS coupling strength $g$
and the field frequency $\omega$ is between $0.1$ and $1$) Gnter2009 ;
Niemczyk2010 ; PhysRevLett.105.237001 ; Yoshihara and even the so-called deep
strong ($g/\omega>1$) coupling regimes, predicted theoretically
PhysRevLett.105.263603 and realized experimentally in photonic systems
PhysRevLett.108.163601 . Through these developments the necessity arose to
consider fully the QRM (for a recent review, see SCReview2019 ) instead of the
simpler quantum Jaynes-Cummings model which is obtained from the QRM by
applying the rotating wave approximation (RWA) JC .
Further intriguing developments include the theoretical prediction of a few-
body quantum phase transition in the QRM, HPP ; PHCP which has subsequently
been observed experimentally in a single trapped ion CaiEtAl2021 , exited
state quantum phase transitions PHP and quantum phase transitions in
extensions of the QRM that include symmetry breaking terms, the focal point of
the present study, and non-linear interaction terms PhysRevA.103.063701 and
the application of the QRM in quantum metrology QMetro .
Recently systems related to the QRM have been studied especially in connection
with quantum computing NakaharaOhmi2008 ; SuterStolze2004 ; Nielsen2010 where
the fundamental building units (qubits) are two-level systems. Moreover, many
physically interesting generalizations of the QRM have been examined (for a
recent review of the theoretical developments in this area, see
XieQiongtao2017 ).
In this paper, we shall especially consider the asymmetric extension of the
QRM (AsymQRM) DB2011 , where a $\mathbb{Z}_{2}$ symmetry breaking term, a
driving term, $\epsilon\sigma_{x}$, is added to the Hamiltonian of the QRM.
Other than in cavity QED systems, such a term arises naturally in the solid
state devices mentioned earlier Gnter2009 ; Niemczyk2010 ;
PhysRevLett.105.237001 ; Yoshihara . The AsymQRM has earlier been proposed for
experimental realization in a micromechanical resonator coupled to a Cooper-
pair box Armour2002 ; PhysRevB.68.155311 .
The AsymQRM is of great theoretical importance, especially because the broken
$\mathbb{Z}_{2}$ symmetry is restored for particular values of the diving
amplitude $\epsilon$ DB2011 and is currently positioned at the centre of the
fundamental quest for a characterisation of integrability in the quantum
regime Larson2013 .
Before we return to the focus of this paper, the asymmetric QRM, we mention in
passing one of the numerous other extensions of the QRM that are currently
under intense investigation and which originated in a suggestion for an
experimental arrangement creating a non-linear interaction between the Bosonic
modes and the TLS GP . This so-called Stark-term,
$\gamma\sigma_{z}a^{\dagger}a$, preserves the $\mathbb{Z}_{2}$ symmetry of the
QRM and has also been rigorously solved using a Bargmann representation Poles
; EJ2017 ; XieDuanChenStark and its physical properties been further
investigated cong2020selective . This model is also discussed as a promising
candidate for another foray to shed light on the fundamental notion of quantum
integrability EckleUnpub . The reason for this expectation is that, within the
rotating wave approximation, the quantum Jaynes-Cummings model admits an exact
solution using the infinite Lie algebra approach of the Richardson-Gaudin type
Bethe ansatz RGBAReview ; GaudinCaux ; Book12 while adding a non-linear Stark
term $\gamma a^{\dagger}a\sigma_{z}$ renders the quantum Jaynes-Cummings-Stark
model amenable to an algebraic Bethe ansatz and the model is thus a Yang-
Baxter integrable model Book12 ; NMB .
Another intriguing development, with promising applications to other physical
systems, the anisotropic QRM, interpolating between the QJCM and the full QRM
PhysRevX.4.021046 ,Tomka, can be extended to the QRSM QHC ; LeiPeter .
Furthermore, asymmetric models have been discussed where the asymmetry term
does not break the $\mathbb{Z}_{2}$ symmetry Scala . Lastly, we mention the
polaron picture PhysRevA.92.053823 ; PhysRevA.95.063803 which has been
successfully applied to investigate the two-photon PhysRevA.99.013815 and two-
qubit SunEtAl QRM.
Currently, the QRM, and its many variants are also discussed in connection
with the fundamental question of quantum entanglement QE ; QEP which reflects
the non-local nature of quantum physics and is, thus, a basic resource of
quantum technology. Quantum entanglement, on the other hand, is again at the
root of such promising technological developments as quantum computation,
quantum information Nielsen2010 ; NakaharaOhmi2008 ; SuterStolze2004 , which
we have already mentioned above, as well as quantum communication QComm .
The present study adds to these developments, especially their theoretical
side, by studying numerically the phenomenon of quantum entanglement resonance
in the asymmetric quantum Rabi model (AsymQRM). We address in particular how
the physical quantity of entanglement entropy can be used to distinguish level
crossings from (narrowly) avoided level crossings. This distinction is of
central importance in the study of the hidden symmetry of the asymmetric QRM
DB2011 ; Wakayama2017 ; SempleKollar2017 ; Ashhab2020 ; Mangazeev2021 ;
LiBatchelor2021 ; Reyes_Bustos_2021 ; LFTB ; PhysRevResearch.3.033057 where
the $\mathbb{Z}_{2}$ symmetry, and hence level crossings, are restored for
half-integer values of the amplitude of the driving term $\epsilon$ in units
of the field strength $\omega$. In order to address this distinction, the
numerical accuracy with which spectra can be calculated is often insufficient
to decide clearly between these two cases, true level crossings and narrowly
avoided level crossings. The entanglement entropy offers a more sensitive way
to distinguish level crossings from avoided level crossings because it uses
not only the eigenvalues but also the (low-lying) eigenstates of the
Schrödinger equation of the AsymQRM. Studying the von Neumann entanglement
entropy (for other entanglement entropy notions, see e.g. BBJS2015 ) of the
eigenstates of the asymmetric QRM for different coupling strengths $g$ and
driving amplitudes $\epsilon$, we find that the entropy is sensitive to the
spectral structure, exhibiting distinctive resonance valleys when the coupling
strength $g$ and resonance peaks when the driving amplitude $\epsilon$ is
increased. These resonances occur in both cases at the loci of the avoided
level crossings of the energy spectra.
We note that the entanglement entropy has already been used in the study the
level crossings in the anisotropic QRM PhysRevX.4.021046 , and the spectral
classification of coupling regimes Rossatto , as well as the QRM in the
polaron picture LiuEtAl . Moreover, similar entanglement resonance behaviour
has been discussed earlier in quantum spin chains Karthik and periodically
driven multi-partite quantum systems Sauer .

In Sec II of this paper, we give the Hamiltonian and the entanglement
characterization of the AsymQRM. The entanglement resonance with the increase
of the photon-atom coupling strength is also revealed. In Sec. III, we study
the entanglement resonance with the increase of the driving amplitude. The
entanglement preservation caused by the hidden symmetry is also uncovered.
Finally, we give a summary in Sec. IV. In Appendix A, we provide some physical
intuition for the entanglement entropy of the simpler asymmetric quantum
Jaynes-Cummings model, where analytical calculations are feasible to a much
greater extent than in the asymmetric quantum Rabi model.

##  II Entanglement Resonance

The AsymQRM describes the interaction between a quantized Bosonic field and a
two-level system and is subject to a static-field driven two-level atom with
Hamiltonian

$\hat{H}=\omega_{0}\hat{\sigma}_{+}\hat{\sigma}_{-}+\omega\hat{a}^{\dagger}\hat{a}+[g(\hat{a}^{\dagger}+\hat{a})+\epsilon](\hat{\sigma}_{+}+\hat{\sigma}_{-}),$
(1)

acting in the tensor Hilbert space ${\mathcal{H}}_{a}\otimes{\mathcal{H}}_{f}$
where $\hat{a}$ and $\hat{a}^{\dagger}$ are the Bosonic annihilation and
creation operators acting in the Hilbert space ${\mathcal{H}}_{f}$ with
frequency $\omega$,
$\hat{\sigma}_{+}=\hat{\sigma}_{-}^{\dagger}=|e\rangle\langle g|$ are the
transition operators between the ground state $|g\rangle$ and the excited
state $|e\rangle$ acting in the two-dimensional Hilbert pace
${\mathcal{H}}_{a}$ of the two-level atom with frequency $\omega_{0}$, $g$ is
the atom-field coupling strength, and $\epsilon$ is the amplitude of static
driving.
The atom-field entanglement of any eigenstate $|\Psi\rangle$ of the AsymQRM
can be quantified by the entanglement entropy which we choose here as the von
Neumann entropy of the reduced density matrix for any one of the subsystems
QEP , e.g. the field ($f$) and atomic ($a$) subsystems considered here,

$S=-\text{Tr}{(\rho_{\text{a}}\mathrm{log}_{2}\rho_{\text{a}})}=-\text{Tr}{(\rho_{\text{f}}\mathrm{log}_{2}\rho_{\text{f}})},$
(2)

where $\rho_{\text{a}}=\text{Tr}_{\text{f}}(|\Psi\rangle\langle\Psi|)$ and
$\rho_{\text{f}}=\text{Tr}_{\text{a}}(|\Psi\rangle\langle\Psi|)$. The
entanglement entropy vanishes for a separable state and equals one for a
maximally entangled state.

Figure 1: Entanglement entropy of different eigenstates $|\Psi_{n}\rangle$ of
the AsymQRM with $\epsilon=0.01\omega$ denoted by thick lines when $n=1,2$ in
(a), $3,4$ in (b), $5,6$ in (c), and $7,8$ in (d) as a function of atom-field
coupling strength $g$. The corresponding thin lines denote the results of QRM
with $\epsilon=0$. Different colors from left to right denote the pUSC, the
npUSC-npDSC, and the pDSC regimes, respectively. We use $\omega_{0}=\omega$.
The truncation number in the numerical calculation is 400.

The Hamiltonian (1) has no obvious symmetry. Therefore the eigenstates
$|\Psi_{n}\rangle$ can only be labeled by the energy eigenvalues $E_{n}$ of
the Schrödinger equation

$H|\Psi_{n}\rangle=E_{n}|\Psi_{n}\rangle.$ (3)

We numerically evaluate the entropy expressions (2) by expanding the
Hamiltonian (1) in the complete basis
$|m,p\rangle=|m\rangle|p\rangle,m=0,1,2,\ldots,p=\pm$ of the combined system
of the atom and the field

$|\Psi_{n}\rangle=\sum_{m,p}c^{(n)}_{m,p}|m,p\rangle$ (4)

to obtain the matrix representation of the Hamiltonian (1) which we use to
numerically obtain eigenvalues and eigenstates in a truncated Hilbert space at
a photon number $n=n_{\mathrm{trunc}}$ such that the obtained magnitudes of
the eigenenergies converge.
We present in Fig. 1 the entanglement entropy of different eigenstates
$|\Psi_{n}\rangle$ of the AsymQRM and the QRM Hamiltonian, respectively, as a
function of the atom-field coupling strength $g$. According to the
classification rule proposed in Ref. PhysRevA.96.013849 for $\epsilon=0$, the
coupling regimes of both, the QRM and the AsymQRM, can be divided into three
regions, the perturbative ultrastrong coupling (pUSC) regime, the perturbative
deep strong coupling (pDSC) regime, and the intermediate region, designated as
the nonperturbative ultrastrong-deep strong coupling (npUSC-npDSC) regime. The
regimes pUSC and npUSC-npDSC are separated by the first energy-level crossing
point and the npUSC-npDSC and pDSC regimes are separated by the energy-level
coalescence point where the adjacent eigenenergies become quasidegenerate.

Figure 2: Eigenenergies corresponding to the AsymQRM in Fig. 1 relative to the
ground state energy $E_{1}$. The gray solid lines in each inset are the
eigenenergies of the QRM with $\epsilon=0$. The coupling values of $g$ of
entanglement resonances in the npUSC-npDSC regime in Fig. 1 exactly match the
ones at which the associated eigenenergies show avoided level crossings. The
truncation number in the numerical calculation is again 400.

This spectral classification is based on the validity of perturbative criteria
of the quantum Rabi model, which allows the use of exactly solvable effective
Hamiltonians. When the counter-rotating terms can still be treated
perturbatively, the system is in the pUSC regime. The system is in the pDSC
regime, when the interaction term cannot any more be treated as a perturbation
but is the main driver of the dynamics. However, in the pDSC regime, the qubit
term $\omega_{0}$ can be treated perturbatively. As emphasized in
PhysRevA.96.013849 , this spectral classification is a qualitative
classification and does not imply an abrupt but rather a gradual change in the
physical properties of the QRM and the AsymQRM.
From Fig. 1 we infer that, in sharp contrast to the case of the QRM with
$\epsilon=0$, the entanglement entropy in the AsymQRM shows a number of
resonance valleys in the npUSC-npDSC regime. Furthermore the number of the
resonance valleys in this weak driving case equals to $\lfloor(n-1)/2\rfloor$
(where $\lfloor x\rfloor\equiv\\{m\in\mathbb{Z}|m\leq x\\}$) matching the
number of energy-level crossing points in the original QRM DB2011 EJ2017 .
Another remarkable difference of the AsymQRM is that the entanglement entropy
in the pDSC regime decays to zero with the increase of the coupling strength
$g$, while it remains one in the QRM.
The entanglement resonance signifies the efficient coupling of the relevant
quantum states, which is essentially determined by the energy spectrum of the
system. To understand the physical reason for the presence of the entanglement
resonance in the npUSC-npDSC regime, we plot in Fig. 2 the corresponding
energy spectrum of the AsymQRM. We can see that all the energy-level crossings
in the original QRM are opened by the static driving in the AsymQRM. It is
interesting to find that the places of the avoided level crossings in Fig. 2
exactly match the ones of the entanglement resonances in Fig. 1. This can be
physically understood as follows. The application of the static driving breaks
the $\mathbb{Z}_{2}$ symmetry of the original QRM which results in the opening
of the energy-level crossings of the QRM. At the points of avoided level
crossings, the high mixing of the two associated levels with different
parities causes an abrupt change to the entanglement of the two involved
quantum states.

Figure 3: Entanglement resonance of the eigenstates $|\Psi_{n}\rangle$ of the
AsymQRM in the parameter plane of $\omega_{0}$ and $g$ when $n=1$ in (a),
$n=3$ in (b), $n=5$ in (c), and $n=7$ in (d). We use $\epsilon=0.1\omega$.

To give a global picture of the entanglement resonance induced by the static
driving, we plot in Fig. 3 the entanglement entropy in the plane $\omega_{0}$
versus $g$ for a chosen driving amplitude $\epsilon$. It can be seen that the
entanglement in the weak-coupling limit is almost zero except for the
resonance case $\omega_{0}=\omega$. Then it changes to one with a tiny
increase of $g$. With the further increase of $g$ to the npUSC-npDSC regime,
$\lfloor(n-1)/2\rfloor$ entanglement resonance valleys appear. The
entanglement in the small-$\omega_{0}$ limit equals zero. It abruptly jumps to
one with the increase of $\omega_{0}$ except for the entanglement resonance
position. Such resonance valleys become sharper and sharper with the increase
of $\omega_{0}$. Confirming the entanglement resonance induced by the avoided
level crossing, the result suggests a useful way to understand the features of
the energy spectra of the family of quantum-Rabi models by monitoring the
entanglement between the atom and the quantized field.

Figure 4: Entanglement preservation of the eigenstates $|\Psi_{n}\rangle$ of
the AsymQRM when $\epsilon=m\omega/2$ with $m\in\mathbb{Z}$. Here $n=2$ in
(a), $n=4$ in (b), $n=6$ in (c), and $n=8$ in (d).

##  III Entanglement preservation in the pDSC regime

We have seen from Fig. 1 that the entanglement is not preserved in the pDSC
regime when a static field is applied. Is this valid for arbitrary $\epsilon$?
To answer this question, we explore the entanglement property of the
eigenstates $|\Psi_{n}\rangle$ of the AsymQRM for varying $\epsilon$. In Fig.
4, we present the entanglement entropy of $|\Psi_{n}\rangle$ as a function of
driving amplitude $\epsilon$ versus coupling strength $g$ for $n=2$, $4$, $6$,
and $8$ as examples (the result for other, even very large values of n, show
the same physics). It is revealed that, besides the repeated resonance valleys
in the npUSC-npDSC regime, which have been analyzed in the last section, the
entanglement in the pDSC regime also shows periodic resonance with increasing
$\epsilon$. Unlike the resonance valleys with increasing $g$ in the npUSC-
npDSC regime, the entanglement resonance with increasing $\epsilon$ in the
pDSC regime shows periodic peaks. A maximal entanglement is observed in the
pDSC regime at discrete values $\epsilon=m\omega/2$, with $m$ being an
integer. As $\epsilon$ further increases, the entanglement disappears again.

Figure 5: Eigenenergies relative to the ground state energy $E_{1}$ of the
AsymQRM as a function of the driving amplitude $\epsilon$ in the pDSC regime
when $g=3\omega$. We use $\omega_{0}=\omega$.

The appearance of the entanglement preservation in the pDSC regime when
$\epsilon=m\omega/2$ is also caused by the avoided energy-level crossings.
Taking $n=8$ as an example, we plot in Fig. 5 the eighth eigenenergy and its
nearest-neighbour energies as a function of $\epsilon$ when $g=3\omega$. It is
interesting to observe that the eighth energy level has eight avoided level
crossings with its nearest-neighbour levels, which all occur at
$\epsilon=m\omega/2$. These avoided level crossings corresponds exactly with
the entanglement preservation in Fig. 4(d). The result confirms that the
entanglement preservation in the pDSC regime when $\epsilon=m\omega/2$ is
essentially determined by the avoided level crossings.

It was unexpected to observe the restoration, in the pDSC regime of the
$\mathbb{Z}_{2}$-symmetry broken AsymQRM, the avoided level crossings, also
called energy quasi-degeneracies, which are governed by the $\mathbb{Z}_{2}$
symmetry and are present in the QRM. Actually, the reappearance of the energy
quasi-degeneracies in the AsymQRM is caused by a hidden symmetry of the
AsymQRM Wakayama2017 ; SempleKollar2017 ; Ashhab2020 ; Mangazeev2021 ;
LiBatchelor2021 occurring when $\epsilon=m\omega/2$. Very recently, its
symmetry operators for $m=1$ and $m=2$ were rigorously derived Mangazeev2021 ;
Reyes_Bustos_2021 and a general scheme to obtain the symmetry operators has
been proposed xie2021general . Thus, in addition to the energy-spectrum
features which have been investigated in previous work, see e.g. Ashhab2020 ,
the entanglement preservation in the pDSC regime revealed in the present work
may serve as another evidence of the hidden symmetry of the AsymQRM.

Figure 6: Schematic diagram of the two harmonic potentials $V_{\pm}$
associated to the two $\hat{\sigma}_{x}$-eigenstates of $\hat{H}_{0}$. The
static driving $\epsilon$ leads to the upward and downward shifts of $V_{+}$
and $V_{-}$, respectively. Here $\epsilon=\omega/2$ is shown as an example.

Another interesting behavior is that the entanglement preservation occurring
at $\epsilon=m\omega/2$ only happens for a finite number of integers $m$. In
order to obtain a physical understanding of this behavior, we apply the
polaron picture PhysRevA.92.053823 ; PhysRevA.95.063803 ; PhysRevA.99.013815 ;
SunEtAl . This picture has been applied to a number of variants of the QRM.
Rotating the Hamiltonian of Eq. (1) by the operator $e^{(i\pi/4)\sigma_{y}}$
to

$\hat{H}=\epsilon\hat{\sigma}_{z}+\frac{\omega_{0}}{2}\hat{\sigma}_{x}+\omega\hat{a}^{\dagger}\hat{a}+g(\hat{a}^{\dagger}+\hat{a})\hat{\sigma}_{z}+\frac{\omega_{0}}{2}$
(5)

and then expanding the Hamitonian in the complete basis
$\sum_{s_{x}=\pm}|{s_{x}}\rangle\langle{s_{x}}|=1$ of
$\hat{\sigma}_{x}\equiv\hat{\sigma}_{+}+\hat{\sigma}_{-}$ and introducing the
unit-mass coordinate $\hat{x}=(\hat{a}+\hat{a}^{\dagger})/\sqrt{2\omega}$ and
momentum operators $\hat{p}=i\sqrt{\omega/2}(\hat{a}^{\dagger}-\hat{a})$ of
the quantized optical field PhysRevA.92.053823 ; PhysRevA.95.063803 , we can
rewrite Eq. (1) as $\hat{H}=\hat{H}_{0}+\hat{H}_{1}$ with

$\displaystyle\hat{H}_{0}$ $\displaystyle=$
$\displaystyle\sum_{s_{x}=\pm}\hat{h}_{s_{x}}|{s_{x}}\rangle\langle{s_{x}}|+\varepsilon_{0},$
(6) $\displaystyle\hat{H}_{1}$ $\displaystyle=$
$\displaystyle\sum_{s_{x}=\pm}{\omega_{0}\over
2}|{s_{x}}\rangle\langle{\bar{s}_{x}}|,$ (7)

where $\bar{s}_{x}$ means the flipped spin of $s_{x}$,
$\varepsilon_{0}=(\omega_{0}-\omega)/2-g^{2}/\omega$ is a constant energy, and
$\hat{h}_{s_{x}}={\hat{p}^{2}/2}+\hat{V}_{s_{x}}$ with

$\hat{V}_{s_{x}}={\omega^{2}\over 2}(\hat{x}+x_{s_{x}})^{2}+s_{x}\epsilon$ (8)

and $x_{s_{x}}={\sqrt{2\omega}s_{x}g/\omega^{2}}$. Here $\hat{V}_{s_{x}}$ are
harmonic potentials with $s_{x}=\pm$ labeling the two $\hat{\sigma}_{x}$
eigenstates. In the pDSC regime, taking $\hat{H}_{1}$ as a perturbation, we
obtain to leading order the eigenenergies of $\hat{H}$ as

$E^{(0)}_{n,\pm}=n\omega\pm{\epsilon}.$ (9)

As illustrated in Fig. 6, one can readily see how the driving $\epsilon$
affects the entanglement between the atom and the field. The dashed lines
represent the case of $\epsilon=0$, where $V_{\pm}$ are degenerate. With
increasing $\epsilon$, $V_{+}$ shifts upward and $V_{-}$ shifts downward with
the difference of their valleys being $2\epsilon$. The eigenenergies
$E^{(0)}_{n,\pm}$ increase or decrease correspondingly. The second energy
level $|2_{-}\rangle$ (take reference to the notation in Scala ) in $V_{-}$
crosses with the second energy level $|2_{+}\rangle$ and the first
one$|1_{+}\rangle$ in $V_{+}$ when $\epsilon=0$ and $\omega/2$, respectively.
Except for these two values of $\epsilon$, $|2_{-}\rangle$ has no chance to
cross with the energy levels in $V_{-}$ anymore. Due to the perturbation of
$\hat{H}_{1}$, such energy-level crossings are reopened, which causes a
sufficient coupling of $|2_{-}\rangle$ with $|2_{+}\rangle$ and
$|1_{+}\rangle$, respectively. This generates a large entanglement between the
atom and the photon. It well explains the result in Fig. 4(a) that a finite
entanglement for the second energy level is preserved in the pDSC regime only
when $\epsilon=0$ and $\omega/2$. In the same picture, the results in Figs.
4(b), 4(c), and 4(d) that the entanglement preservation occurs at
$\epsilon=m\omega/2$ only for $m=0,\cdots,n-1$ can be understood. Thus, such a
simple picture provides an intuitive explanation of the avoided level crossing
and entanglement preservation in the pDSC regime.

##  IV Conclusions

In summary, we have investigated the entanglement features in the eigenstates
$|\Psi_{n}\rangle$ of the coupled system of a quantized optical field with a
TLS subject to a statically driven two-level atom, i.e., the AsymQRM. The
entanglement exhibits interesting features depending on the light-matter
coupling strength and the driving amplitude, which are intrinsically related
to the structure of the energy spectrum of the AsymQRM. We find that the
entanglement shows $\lfloor(n-1)/2\rfloor$ resonance valleys with the change
of the light-matter coupling strength in the npUSC-npDSC regime. Our results
indicate that this is caused by the avoided level crossings induced by the
static field. In the stronger pDSC regime, the entanglement exhibits resonance
peaks at $\epsilon=m\omega/2$, with $m=0,\cdots,n-1$. Our analysis
demonstrates that such entanglement preservation is induced by the avoided
level crossings due to the hidden symmetry of the AsymQRM. Our result is
expected to be helpful to identify the features of the energy spectrum, such
as the avoided energy level crossings, and to further explore the related
hidden symmetry properties of more theoretical models for light-matter
interaction xie2021double ; lu2021hidden ; lu2021hidden2 . In addition, our
method is promising to be applied to study quantum dot(s) with broken
inversion symmetry in a cavity PhysRevLett.102.023601 ; PhysRevA.85.053818 .

## Acknowledgements

We thank Prof. Jun-Hong An for helpful discussions. HPE thanks the School of
Physical Science and Technology of Lanzhou University for their friendly and
supportive hospitality. This work is supported by the International
Postdoctoral Exchange Fellowship Program (Grant No. ZD202116), and the
National Natural Science Foundation of Gansu Province, China (Grant No.
20JR5RA509).

##  Appendix A Entanglement entropy for the Asymmetric Jaynes-Cummings model

In order to provide a physical intuition for the appearance of the
entanglement features in the AsymQRM, we apply a perturbation method to
analytically study the entanglement entropy in the asymmetric quantum Jaynes-
Cummings (AsymQJC) model which is a model obtained from the AsymQRM after
application of the rotating wave approximation JC with the Hamiltonian

$\displaystyle\begin{split}\hat{H}_{\mathrm{AsymQJC}}=\frac{\omega_{0}}{2}\hat{\sigma}_{z}+\omega\hat{a}^{\dagger}\hat{a}+g(\hat{a}^{\dagger}\hat{\sigma}_{-}+\hat{a}\hat{\sigma}_{+})+\epsilon\hat{\sigma}_{x}.\end{split}$
(10)

The AsymQJC model shows similar entanglement resonance behavior as the
AsymQRM. For instance, the energy-level crossings in the original quantum JC
model are opened by the static driving in the AsymQJC model. The quantum JC
model is exactly solvable, on the basis of which, by applying the degenerate
perturbation method, we obtain the accurate value of the entropy for the peak
of the entanglement resonance.

In Fig. 7, the opened energy gap is presented in panel (a) and the numerical
and analytical result of the entropy is presented in panel (b). In Fig. 7 (b),
one can see that the analytical solution obtained by the degenerate
perturbation method matches the numerical result perfectly. The fact that the
entanglement is reduced (or increased) when there is a perturbation that
removes a level crossing, can be explained as a consequence of such a simple
observation, i.e. if the unperturbed eigenstates are close to being maximally
entangled, it is likely that their sum and difference are less entangled. In
the same way, if the unperturbed eigenstates are close to being separable,
their sum and difference are likely to be highly entangled. The detailed
derivation is presented in the following section.

Figure 7: (a) Avoided level crossing in the energy spectra of JC model with
external perturbation $\epsilon$. (b) Entanglement entropy obtained by
numerical (solid lines) and degenerate (yellow star) pertubative method. The
entropy of the JC model is also provided as a bench mark. $g/\omega=0.2749$ is
the point where the energies of the states $|1,+\rangle$ and $|2,-\rangle$ for
JC model becomes degenerate. $\omega_{0}=1.5\omega$ and $\epsilon=0.05\omega$.

###  A.1 Analytical solution of the quantum Jaynes-Cummings model

In this section we summarize the analytical results for the quantum Jaynes-
Cummings model Book8.10 . The quantum Jaynes-Cummings model with Hamiltonian

$\displaystyle\begin{split}\hat{H}_{\mathrm{JC}}=\frac{\omega_{0}}{2}\hat{\sigma}_{z}+\omega\hat{a}^{\dagger}\hat{a}+g(\hat{a}^{\dagger}\hat{\sigma}_{-}+\hat{a}\hat{\sigma}_{+}).\end{split}$
(11)

is known to be exactly solvable by elementary means. It conserves the total
number operator
$\hat{N}=\hat{a}^{\dagger}\hat{a}+\frac{1}{2}(1+\hat{\sigma}_{z})$.

For $N=0$, the ground state energy is $E_{0g}=-\frac{\omega_{0}}{2}$ and the
eigenstate is denoted by $|g,0\rangle$. The excitation energies and the
excited states are given by

$E_{n,\pm}=(n+\frac{1}{2})\omega\pm\frac{\Omega_{n,\Delta}}{2},\quad
n=0,1,2,\ldots,$ (12)

where $\Omega_{n,\Delta}=\sqrt{\Delta^{2}+4g^{2}(n+1)}$ and
$\Delta=\omega_{0}-\omega$, and

$\displaystyle\begin{split}|n,+\rangle&=C_{n}|n,e\rangle+D_{n}|n+1,g\rangle,\\\
|n,-\rangle&=D_{n}|n,e\rangle-C_{n}|n+1,g\rangle,\end{split}$ (13)

where $C_{n}=\cos(\frac{\alpha_{n}}{2})$ and
$D_{n}=\sin(\frac{\alpha_{n}}{2})$ with
$\alpha_{n}=\tan^{-1}(\frac{2g\sqrt{n+1}}{\Delta})$.

###  A.2 Degenerate perturbation for the asymmetric quantum Jaynes-Cummings
model

The avoided level crossing points in the AsymQJC model correspond to the
doubly degenerate, i.e. the crossing, points of the quantum JC model. This is
the reason we need to apply the degenerate perturbation method.

For the Hamiltonian of the quantum JC model at the degenerate point, i.e. for
a particular value of $g$ to be determined, the eigenvalue satisfies

$\mathcal{E}_{n}^{(0)}=E_{n,+}=E_{n+1,-},$ (14)

corresponding to two independent and orthogonal eigenfunctions
$\phi_{n1}=|n,+\rangle$ and $\phi_{n2}=|n+1,-\rangle$,

$\hat{H}_{\mathrm{JC}}\phi_{ni}=\mathcal{E}_{n}^{(0)}\phi_{ni},\quad i=1,2.$
(15)

This eigenvalue, $\mathcal{E}_{n}^{(0)}$, which is exact for the quantum JC
model, now plays the role of the zeroth order approximation eigenvalue for the
asymmetric quantum JC model.

The corresponding zero-order wave function

$\varphi_{n}^{(0)}=\sum_{i=1}^{2}c_{i}^{(0)}\phi_{ni}$ (16)

where $c_{i}^{(0)}$ and the first-order energy eigenvalue
$\mathcal{E}_{n}^{(1)}$ can be obtained by solving the eigenvalue equation

$\sum_{j=1}^{2}(H_{ij}^{{}^{\prime}}-\mathcal{E}_{n}^{(1)}\delta_{ij})c_{j}^{(0)}=0,\quad
i=1,2$ (17)

where $H_{ij}^{\prime}=\langle\phi_{ni}|\hat{H}^{\prime}|\phi_{nj}\rangle$
represents the coupling between eigenstates $\phi_{ni}$ and $\phi_{nj}$ due to
the operation of $\hat{H}^{\prime}=\epsilon\sigma_{x}$. We obtain the first-
order modifications of the degenerate eigenenergy

$\mathcal{E}_{n,\pm}^{(1)}=\pm\epsilon D_{n}D_{n+1}$ (18)

as well as $c_{1}^{(0)}=\pm c_{2}^{(0)}=\frac{1}{\sqrt{2}}$ for the zeroth-
order coefficients of the eigenstates.

Thus, the total energy at the degenerate point is

$\displaystyle\begin{split}E_{n,\pm}=\mathcal{E}_{n}^{(0)}+\mathcal{E}_{n,\pm}^{(1)}.\end{split}$
(19)

The modified wave functions $\varphi_{n1}^{(0)}$ and $\varphi_{n2}^{(0)}$ are

$\displaystyle\begin{split}\varphi_{n1}^{(0)}=\frac{1}{\sqrt{2}}(\phi_{n1}+\phi_{n2}),\\\
\varphi_{n2}^{(0)}=\frac{1}{\sqrt{2}}(\phi_{n1}-\phi_{n2}).\end{split}$ (20)

Note, that the degeneracy of the quantum JC model is now lifted due to the
asymmetry term $\epsilon\sigma_{x}$, the level crossing at the degenerate
point of the quantum JC model is now avoided for the asymmetric model.

Finally, we take the wave function $\varphi_{n1}^{(0)}$ as an example to
calculate its corresponding entanglement entropy. It is known that the total
density matrix is
$\rho_{AB}=|\varphi_{n1}^{(0)}\rangle\langle\varphi_{n1}^{(0)}|$. One can get
its reduced density matrix $\rho_{A}\equiv\text{Tr}_{B}(\rho_{AB})$

$\rho_{A}=\left(\begin{array}[]{ccccc}D^{2}_{n+1}+C^{2}_{n}&D_{n}D_{n+1}\\\
D_{n}D_{n+1}&D^{2}_{n}+C^{2}_{n+1}\\\ \end{array}\right).$ (21)

Now the von Neumann entropy $S_{\rho A}$ of the zero-level modified wave
functions can be obtained

$S_{\rho_{A}}=-\text{Tr}{[\rho_{A}\mathrm{log}_{2}(\rho_{A})]}.$ (22)

As an example, we calculate the avoided level crossing corresponding to the
intersection of states $|1,+\rangle$ and $|2,-\rangle$ in the JC model. The
result is shown in Fig. 7. In the case of $\Delta=\omega_{0}-\omega=0.5\omega$
and $\epsilon=0.05\omega$, the crossing point is at $g=0.2749\omega$ Then,
with the definition of $\alpha_{n}$, $C_{n}$ and $D_{n}$ (given below Eq. 13),
one can calculate the reduced density matrix $\rho_{A}$ and then the entropy
$S_{\rho A}$. Note, that the condition for which the perturbative method
works, $\epsilon\ll\omega,\omega_{0},g$, should be satisfied.

## References

  * (1) S. Haroche and J.-M. Raimond Exploring the Quantum — Atoms, Cavities, and Photons (Oxford: Oxford University Press, 2006).
  * (2) J. C. Garrison and R. Y. Chiao Quantum Optics (Oxford University Press, 2008).
  * (3) L. Allen and J. H. Eberly Optical Resonance and Two-Level Atoms (Dover, 1987).
  * (4) I. I. Rabi1936 On the process of space quantization Phys. Rev. 49 324; 1937 Space quantization in a gyrating magnetic field Phys. Rev. 51 652\.
  * (5) E. T. Jaynes and F. W. Cummings, Comparison of quantum and semiclassical radiation theories with application to the beam maser, Proceedings of the IEEE, 51, 89–109, (1963).
  * (6) H.-P. Eckle, Models of Quantum Matter — A First Course on Integrability and the Bethe Ansatz, (Oxford University Press, 2019), section 8.10.
  * (7) D. Braak, Integrability of the Rabi Model, Phys. Rev. Lett. 107 100401 (2011).
  * (8) Q.-H. Chen, C. Wang, S. He, T. Liu, and K.-L. Wang, Exact solvability of the quantum Rabi model using Bogoliubov operators, Phys. Rev. A 86, 023822 (2012).
  * (9) L. Yu, S. Zhu, Q. Liang, G. Chen, and S. Jia, Analytical solutions for the Rabi model, Phys. Rev. A, 86 015803, 2012.
  * (10) D. Braak, A generalized G-function for the Quantum Rabi Model, Ann. Phys. (Berlin) 525, L23–L28 (2013).
  * (11) D. Braak, Analytical solutions of basic models in quantum optics, Proceedings of the Forum of Mathematics for Industry ed R S Anderssen (Springer, 2015)
  * (12) D. Braak, Symmetries in the Quantum Rabi Model, Symmetry, 11, 1259 (2019).
  * (13) Q. Xie, H. Zhong, M. T. Batchelor, and C. Lee, The quantum Rabi model: solution and dynamics, J. Phys. A: Math. Theor., 50, 113001 (2017).
  * (14) A. Le Boité, Theoretical Methods for Ultrastrong Light-Matter Interactions, Advanced Quantum Technologies, 3, 1900140 (2020).
  * (15) J. Larson and T. Mavrogordatos, The Jaynes–Cummings Model and Its Descendants, (IOP Publishing, 2021)
  * (16) I. Arrazola, Design of light-matter interactions for quantum technologies, (2021), arXiv:2101.11695 [quant-ph] .
  * (17) A. Blais, A. L. Grimsmo, S. M. Girvin andAndreas Wallraff, Circuit quantum electrodynamics, Rev. Mod. Phys. 93, 025005 (2021).
  * (18) D. Lv, S. An, Z. Liu, J.-N. Zhang, J. S. Pedernales, L. Lamata, E. Solano, and K. Kim, Quantum Simulation of the Quantum Rabi Model in a Trapped Ion, Phys. Rev. X 8, 021027 (2018).
  * (19) M.-L. Cai, Z.-D. Liu, W.-D. Zhao, Y.-K. Wu, Q.-X. Mei, Y. Jiang, L. He, X. Zhang, Z.-C. Zhou, and L.-M. Duan, “Observation of a quantum phase transition in the quantum Rabi model with a single trapped ion,” Nat. Commun. 12, 1126 (2021).
  * (20) G. Günter, A. A. Anappara, J. Hees, A. Sell, G. Biasiol, L. Sorba, S. De Liberato, C. Ciuti, A. Tredicucci, A. Leitenstorfer, and R. Huber, Sub-cycle switch-on of ultrastrong light-matter interaction, Nature 458, 178 (2009).
  * (21) T. Niemczyk, F. Deppe, H. Huebl, E. P. Menzel, F. Hocke, M. J. Schwarz, J. J. Garcia-Ripoll, D. Zueco, T. Hümmer, E. Solano, A. Marx, and R. Gross, Circuit quantum electrodynamics in the ultrastrong-coupling regime, Nat. Phys. 6, 772 (2010).
  * (22) P. Forn-Diaz, J. Lisenfeld, D. Marcos, J. J. Garcia-Ripoll, E. Solano, C. J. P. M. Harmans, and J. E. Mooij, Observation of the Bloch-Siegert Shift in a Qubit-Oscillator System in the Ultrastrong Coupling Regime, Phys. Rev. Lett. 105, 237001 (2010).
  * (23) F. Yoshihara, T. Fuse, S. Ashhab, K. Kakuyanagi, S. Saito, and K. Semba, Superconducting qubit–oscillator circuit beyond the ultrastrong-coupling regime, Nat. Phys. 13, 44 (2016).
  * (24) J. Casanova, G. Romero, I. Lizuain, J. J. Garcia-Ripoll, and E. Solano, Deep Strong Coupling Regime of the Jaynes-Cummings Model, Phys. Rev. Lett. 105, 263603 (2010).
  * (25) A. Crespi, S. Longhi, and R. Osellame, Photonic Realization of the Quantum Rabi Model, Phys. Rev. Lett. 108, 163601 (2012).
  * (26) P. Forn-Díaz, L. Lamata, E. Rico, J. Kono and E. Solano, Ultrastrong coupling regimes of light-matter interaction Rev. Mod. Phys. 91, 025005 (2019).
  * (27) M.-J. Hwang, R. Puebla, and M. B. Plenio, Quantum Phase Transition and Universal Dynamics in the Rabi Model, Phys. Rev. Lett. 115, 180404 (2015).
  * (28) R. Puebla, M.-J. Hwang, J. Casanova, and M. B. Plenio, Probing the Dynamics of a Superradiant Quantum Phase Transition with a Single Trapped Ion, Phys. Rev. Lett. 118, 073001 (2017).
  * (29) R. Puebla, M.-J. Hwang, and M. B. Plenio M B, Excited-state quantum phase transition in the Rabi model, Phys. Rev. A 94 023835 (2016).
  * (30) Z.-J. Ying, Symmetry-breaking patterns, tricriticalities, and quadruple points in the quantum Rabi model with bias and nonlinear interaction, Phys. Rev. A 103, 063701 (2021); From Quantum Rabi Model to Janes-Cummings Model: Symmetry-Breaking Quantum Phase Transitions, Symmetry-Protected Topological Transitions and Multicriticality, Adv. Quantum Technol. 5, 2100088 (2021).
  * (31) L. Garbe, M. Bina, A. Keller, M. G. A. Paris, and S. Felicetti, Critical Quantum Metrology with a Finite-Component Quantum Phase Transition, Phys. Rev. Lett. 124, 120504 (2020).
  * (32) M. Nakahara and T. Ohmi, Quantum Computing — From Linear Algebra to Physical Realizations (CRC Press, 2008).
  * (33) J. Solze and D. Suter, Quantum Computing — A Short Course from Theory to Experiment, (Wiley-VCH, 2nd edn, 2008).
  * (34) M. A. Nielsen and I. L. Chuang, Quantum computation and quantum information, (10th anniversary edn), (Cambridge University Press, 2010).
  * (35) A. D. Armour, M. P. Blencowe, and K. C. Schwab, Entanglement and decoherence of a micromechanical resonator via coupling to a cooper-pair box, Phys. Rev. Lett. 88, 148301 (2002).
  * (36) E. K. Irish and K. Schwab, Quantum measurement of a coupled nanomechanical resonator–cooper-pair box system. Phys. Rev. B, 68:155311, Oct 2003.
  * (37) Jonas Larson, Integrability versus quantum thermalization, J. Phys. B 46 224016 (2013)
  * (38) A. L. Grimsmo and S. Parkins, Cavity-QED simulation of qubit–oscillator dynamics in the ultrastrong-coupling regime, Phys. Rev. A 87 033814 (2013); Open Rabi model with ultrastrong coupling plus large dispersivetype nonlinearity: Nonclassical light via a tailored degeneracy Phys. Rev. A 89 033802 (2014).
  * (39) Maciejewski, Andrzej J, Przybylska, Maria, and Stachowiak, Tomasz, Analytical method of spectra calculations in the Bargmann representation, Phys Lett A, 378, 3445–51 (2014); An exactly solvable system from quantum optics. Phys Lett A, 379, 1505–9 (2015).
  * (40) H.-P. Eckle and H. Johannesson, A generalization of the quantum Rabi model: Exact solution and spectral structure, J. Phys. A: Math. Theor. 50, 294004 (2017).
  * (41) Y.-F. Xie, L. Duan, and Q.-H. Chen, Quantum Rabi–Stark model: solutions and exotic energy spectra, J. Phys. A: Math. Theo. 52 245304 (2019).
  * (42) L. Cong, S. Felicetti, J. Casanova, L. Lamata, E. Solano, and I. Arrazola, Selective interactions in the quantum Rabi model, Phys. Rev. A 101, 032350 (2020).
  * (43) H.-P. Eckle, unpublished.
  * (44) J. Dukelsky, S. Pittel, and G. Sierra, Exactly solvable Richardson-Gaudin models for many-body quantum systems, Rev. Mod. Phys. 76, 643 (2004)
  * (45) M. Gaudin and Jean-Sébastien Caux, The Bethe Wavefunction, (Cambridge University Press, 2014), chapter 13.
  * (46) H.-P. Eckle, Models of Quantum Matter — A First Course on Integrability and the Bethe Ansatz, (Oxford University Press, 2019), chapter 12.
  * (47) N. M. Bogoliubov, R. K. Bullough, and J. Timonen, Exact solution of generalized Tavis-Cummings models in quantum optics, J. Phys. A, 29, 6305 (1996).
  * (48) Q.-T. Xie, S. Cui, J.-P. Cao, L. Amico, and H. Fan, Anisotropic Rabi model, Phys. Rev. X 4, 021046 (2014).
  * (49) M. Tomka, O. El Araby, M Pletyukhov, and M. Gritsev, Exceptional and regular spectra of a generalized Rabi model, Phys. Rev. A 90 063839, (2014)
  * (50) Y.-F. Xie, X.-Y. Chen, X.-F. Dong, and Q.-H. Chen, First-order and continuous quantum phase transitions in the anisotropic quantum Rabi-Stark model, Phys. Rev. A 101, 053803 (2020)
  * (51) L. Cong and H.-P. Eckle, in preparation.
  * (52) G. Scala, K. Słowik, P. Facchi, S. Pascazio, and F. V. Pepe, Beyond the Rabi model: Light interactions with polar atomic systems in a cavity, Phys. Rev. A 104 013722, (2021)
  * (53) Z.-J. Ying, M. Liu, H.-G. Luo, H.-Q. Lin, and J. Q. You, Ground-state phase diagram of the quantum Rabi model, Phys. Rev. A, 92 053823, (2015).
  * (54) L. Cong, X.-M. Sun, M. Liu, Z.-J. Ying, and H.-G. Luo, Frequency-renormalized multipolaron expansion for the quantum Rabi model, Phys. Rev. A, 95 063803, (2017).
  * (55) L. Cong, X.-M. Sun, M. Liu, Z.-J. Ying, and H.-G. Luo, Polaron picture of the two-photon quantum Rabi model, Phys. Rev. A 99, 013815 (2019).
  * (56) X. M. Sun, L. Cong, H.-P. Eckle, Z.-J. Ying, and H.-G. Luo, Application of the polaron picture in the two-qubit quantum Rabi model Phys. Rev. A 101, 063832 (2020).
  * (57) J. Audretsch. Entangled systems— New directions in quantum physics, (Wiley–VCH), (2007).
  * (58) M. B. Plenio and S. S. Virami, An Introduction to Entanglement Theory, Chapter 8 in ”Quantum Information and Coherence”, Erika Andersson and P. Öhberg (Eds.) (Springer, 2014).
  * (59) G. Cariolaro, Quantum Communications, (Spinger, 2015).
  * (60) M. Wakayama, Symmetry of asymmetric quantum Rabi models J. Phys. A: Math. Theor. 50, 174001 (2017).
  * (61) J. Semple and M. Kollar, Asymptotic behaviour of observables in the asymmetric quantum Rabi model, J. Phys. A: Math. Theor. 51, 044002 (2017).
  * (62) S. Ashhab, Attempt to find the hidden symmetry in the asymmetric quantum Rabi model, Phys. Rev. A 101, 023808 (2020).
  * (63) V. V. Mangazeev, M. T. Batchelor, and V. V. Bazhanov, J. Phys. A: Math. Theor. 54, The hidden symmetry of the asymmetric quantum Rabi model, 12LT01 (2021).
  * (64) Z.-M. Li and M. T. Batchelor, Hidden symmetry and tunneling dynamics in the asymmetric quantum Rabi model, Phys. Rev. A 103, 023719 (2021).
  * (65) K. Kimoto, C. Reyes-Bustos, and M. Wakayama, Determinant Expressions of Constraint Polynomials and the Spectrum of the Asymmetric Quantum Rabi Model, International Mathematics Research Notices, 2021, 9458 (2021).
  * (66) Z.-M. Li, D. Ferri, D. Tilbrook, and M. T. Batchelor, Generalized adiabatic approximation to the asymmetric quantum Rabi model: conical intersections and geometric phases J. Phys. A: Math. Theor. 54 405201 (2021).
  * (67) Y. F. Xie, and Q. H. Chen, Double degeneracy associated with hidden symmetries in the asymmetric two-photon Rabi model, Physical Review Research 3 033057 (2021).
  * (68) S. Bose, A. Bayat, H. Johannesson, and P. Sodano, Entanglement Content of Many-Body States via Concurrence, Negativity and Schmidt Gap, in Strongly Coupled Field Theories for Condensed Matter and Quantum Information Theory, A. Ferraz, K. S. Gupta, G. W. Semenoff, and P. Sodano (Eds.), (2015).
  * (69) D. Z. Rossatto, C. J. Villas-Bôas, M. Sanz, and E. Solano, Spectral classification of coupling regimes in the quantum Rabi model, Phys. Rev. A 96, 013849 (2017).
  * (70) M. Liu, Z.-J. Ying, J.-H. An, H.-G. Luo, and H.-Q. Lin, The asymmetric quantum Rabi model in the polaron picture, J. Phys. A: Math. Theor. 50, 084003 (2017).
  * (71) J. Karthik, A. Sharma, and A. Lakshminarayan, Entanglement, avoided crossings, and quantum chaos in the Ising model with a tilted magnetic field, Phys. Rev. A 75, 022304 (2007).
  * (72) S. Sauer, F. Mintert, C. Gneiting, and A. Buchleitner, Entanglement resonances of driven multi-partite quantum systems, Journal of Physics B: Atomic, Molecular and Optical Physics 45, 154011 (2012).
  * (73) D. Z. Rossatto, C. J. Villas-Bâs, M. Sanz, and E. Solano, Spectral classification of coupling regimes in the quantum Rabi model. Phys. Rev. A 96, 013849 (2017).
  * (74) Y.-F. Xie and Q.-H. Chen, General symmetry operators of the asymmetric quantum Rabi model within Bogoliubov operator approach, arXiv:2107.08937v1 [quant-ph] (2021).
  * (75) Y.-F. Xie and Q.-H. Chen, Double degeneracy associated with hidden symmetries in the asymmetric two-photon Rabi model, Phys. Rev. Res. 3 033057 (2021).
  * (76) X. Lu, Z.-M. Li, V. V. Mangazeev, and M. T. Batchelor, Hidden symmetry operators for asymmetric generalised quantum Rabi models, Chinese Physics B 31 014210 (2021).
  * (77) X. Lu, Z.-M. Li, V. V. Mangazeev, and M. T. Batchelor, Hidden symmetry in the biased Dicke model, J. Phys. A: Math. Theor. 54 325202 (2021).
  * (78) O. V. Kibis, G. Ya. Slepyan, S. A. Maksimenko, and A. Hoffmann, Matter coupling to strong electromagnetic fields in two-level quantum systems with broken inversion symmetry, Phys. Rev. Lett. 102 023601 (2009).
  * (79) I. G. Savenko, O. V. Kibis, and I. A. Shelykh, Asymmetric quantum dot in a microcavity as a nonlinear optical element, Phys. Rev. A 85 053818 (2012).