# Tracking People with 3D Representations

Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, Jitendra Malik
UC Berkeley

###### Abstract

We present a novel approach for tracking multiple people in video. Unlike past
approaches which employ 2D representations, we focus on using 3D
representations of people, located in three-dimensional space. To this end, we
develop a method, Human Mesh and Appearance Recovery (HMAR) which in addition
to extracting the 3D geometry of the person as a SMPL mesh, also extracts
appearance as a texture map on the triangles of the mesh. This serves as a 3D
representation for appearance that is robust to viewpoint and pose changes.
Given a video clip, we first detect bounding boxes corresponding to people,
and for each one, we extract 3D appearance, pose, and location information
using HMAR. These embedding vectors are then sent to a transformer, which
performs spatio-temporal aggregation of the representations over the duration
of the sequence. The similarity of the resulting representations is used to
solve for associations that assigns each person to a tracklet. We evaluate our
approach on the Posetrack, MuPoTs and AVA datasets. We find that 3D
representations are more effective than 2D representations for tracking in
these settings, and we obtain state-of-the-art performance. Code and results
are available at: https://brjathu.github.io/T3DP

##  1 Introduction

Humans are three-dimensional beings, who live and move in a three-dimensional
space. However, monocular people tracking algorithms from the computer vision
community, e.g., [4, 7, 12, 28, 36, 41, 46] are typically based on 2D
representations of people in 2D space. This made a lot of sense in the 1990s
when people tracking first came to the fore as (1) the application context was
often video surveillance, where people are at low resolutions (2) there were
not reliable algorithms for lifting people from 2D to 3D. But today neither of
these limitations hold. Applications of video analysis can use HD video (1080p
or better), and we have excellent technology e.g., [17] for 3Dfying people.

Figure 1 illustrates why tracking people might be easier in 3D than in 2D.
Even when two people overlap and occlude each other in 2D, their spatial
location is typically separable in 3D. While 2D appearance features are
sensitive to viewpoint and body pose, 3D appearance features are less affected
by viewpoint and pose changes. These properties enable robust tracking through
people-to-people occlusion and even shot changes, a common phenomena in edited
media and online videos.

Tracking people in 3D also opens up many downstream tasks such as predicting
3D human motion from video [18, 21], predicting their behavior [11, 48], and
imitating human behavior from video [31]. These tasks involve videos of people
other than pedestrians, such as videos of people performing sports, dance,
daily activities [10], or even edited media such as movies or TV shows [13,
30].

Our input is a monocular video with detected bounding boxes. On each bounding
box, we develop an approach that extracts both 3D geometry and appearance
information of the person. Specifically, we extend HMR [17], a method that
predicts 3D human mesh from monocular images, to also extract the appearance
information of the person by predicting a flow field that estimates the
texture map, which is a viewpoint and pose invariant space. We refer to this
model as Human Mesh and Appearance Recovery (HMAR). From HMAR output, we
extract 3D appearance, 3D pose, joints, and the estimate of their 3D location
as separate features. We then aggregate this per-bounding box 3D information
across time and space via a transformer [38] that takes these separate 3D cues
as input. The transformer acts as a spatio-temporal diffusion mechanism that
can propagate information across similar features by means of attention.
Finally, we associate the diffused features across time to assign identities
to each bounding box, which produces the final tracklets. The overview of our
approach is shown in Figure 2.

We evaluate our algorithm on three different datasets: PoseTrack [1], MuPots
[27] and AVA [13]. AVA is a particularly interesting dataset as it is derived
from movies, which include shot changes. These exhibit much greater variety of
behavior than videos in the traditional tracking challenges such as MOT. In
addition, people are larger (in terms of number of image pixels), which is
essential for the 3D approach to work. We find the following: 1. 3D
representations are more effective than 2D representations, 2. Ablation
studies show that using 3D information in both appearance and location is
helpful, 3. We outperform current state of the art on these datasets.

Figure 1: _The benefits of tracking people in 3D._ Left: Using the 3D location
of people makes it easier to separate them than using their 2D location or 2D
bounding boxes. Right: The appearance of people, when computed in a 3D
representation (texture map) is less affected by changes in viewpoint and
pose.

##  2 Related work

##### Tracking

The tracking literature is vast and we refer readers to [7] for a
comprehensive summary. Tracking may be designed for generic categories such as
any bounding boxes [52], here we focus on methods that track humans. These
approaches mostly depend on 2D location or keypoint features [12, 36, 41] and
sometimes 2D appearance [4, 28, 43, 46]. Most modern-day tracking approaches
focus on evaluating their performance on the MOT benchmark [7], which mainly
focus on high-density pedestrian scenes observed in outdoor or indoor
settings. Under this setting, detection plays a key role and many methods
jointly solve for both detection and association [4, 28]. In this work, we are
interested in the effectiveness of 3D representations for tracking and thus
assume that detection bounding boxes are provided, which we associate through
our representations. Furthermore, we are interested in tracking people in
every day Internet videos where people occupy larger portion of the scene,
performing activities such as sports or dancing, and even edited media such as
movies.

There are methods that incorporate 3D information in tracking, however these
approaches assume multiple input cameras [23, 49] or 3D point cloud
observation from lidar data [40]. Here we focus on the setting where the input
is a monocular video. This setting has also been addressed in the past, e.g.,
[5], however, here we leverage much more expressive and fine-grained 3D
information, including detailed representations for appearance and 3D joint
locations, which help to improve the overall tracking performance.

##### Monocular 3D human reconstruction

There is long history of methods that predict 3D human structure or motion
from monocular images or videos [6, 14]. Recently, there has been rapid
progress in this area due to the emergence of statistical models of human
bodies such as SMPL [25] that provide a low dimensional parameterization of a
deformable 3D mesh of human bodies. Human Mesh Recovery (HMR) method [17] and
follow-up works, e.g., [3, 22], employ convolutional neural networks to
predict the parameters of an existing body model from a single image.
Extending this result in the temporal dimension, HMMR [19] and VIBE [21]
reconstruct mesh output for a single person given a series of tracked frames
or tracklets as the input. Extensions to movie data have also been
investigated in [30]. Other works focus on multiple people [16, 37, 47] and
regress the parameters corresponding to multiple people on the same input
frame. All of these methods rely on detected bounding boxes or tracklets in
the case of temporal domain. As such, these approaches rely heavily on
existing tracking algorithms to extract long tracklets associated to the same
individual. In this work we explore how the predicted 3D human information may
be used to empower tracking in exchange. There are two recent works that
jointly solve tracking and 3D pose estimation of multiple people from
monocular video [26, 33]. While these approaches focus on the task of 3D pose
estimation, we focus on the problem of identity association and study the
effectiveness of 3D vs 2D representation and compare extensively against
state-of-the-art 2D tracking methods.

Related to our approach are recent methods that extract the texture of a
person along with their geometry [29, 39, 45]. The first recovers the texture
only for the visible areas of the body, and uses it to improve the 3D
reconstruction, while the other two use identity labels to learn an encoder
that predicts complete 3D appearance expressed as UV images. In this work we
explore how UV images can be used for associating people across time for
tracking.

Figure 2: _Overview of our framework._ The input to our pipeline consists of a
video sequence with bounding box detections. For each bounding box, we use our
HMAR model we extract 3D representations corresponding to the person’s 3D
appearance, pose, and location. The 3D representations for all bounding boxes
are then processed jointly by a transformer which aggregates information
across space and time. Finally, an online association step assigns identity to
each detected human, which constitute the tracklets.

##  3 Method

In order to assess the effectiveness of 3D information in tracking, we propose
a simple tracking framework that assumes as input a monocular video and a set
of detected bounding boxes. We then explore how to associate these bounding
boxes across time using various cues.

Our tracking algorithm consists of two main modules: our proposed HMAR model,
which encodes humans into a rich embedding space, and a transformer model for
learning associations between detected humans across multiple frames. The HMAR
model learns to predict appearance, pose, and keypoints. These are essential
cues for tracking, and each of them carries complementary information. For
example, when a person is temporarily occluded, the appearance is important to
establish its identity after re-appearance, while when many people share
similar clothing in a video, pose and location become the primary cues for
tracking. Given this rich embedding of a person, we need to learn associations
between different human identities so that each person can be matched in the
upcoming frames. We train a modified version of a transformer [38] to learn
these associations across multiple human identities.

###  3.1 Human Mesh and Appearance Recovery (HMAR)

Our starting point for 3D feature extraction is the popular HMR architecture
[17]. Given a bounding box of a person, HMR regresses the 3D pose and shape
parameters of the person. While the 3D pose provides rich geometric
information about the person, it throws away the appearance information, which
is essential for tracking. To augment this, we extend HMR such that it can
also recover the 3D appearance of the person by means of a texture image,
which is a space that is viewpoint and pose invariant. We call this method
HMAR, which stands for Human Mesh and Appearance Recovery.

In order to recover the texture image, we create skip connections from the
spatial features of the Resnet-50 backbone used in HMR and connect them to a
new appearance head that maintains the spatial alignment of features through a
series of convolution and bilinear upsampling layers (Figure 3). Please see
the supplemental for details of the architecture. The appearance head takes
these spatial features over multiple scales as input and is trained to predict
the texture of the person, using the canonical SMPL texture mapping [25]. This
is done in the form of appearance or texture flow in the spirit of [18, 51].
Specifically, we regress a texture flow $\mathcal{F}$ that has the same
dimensions as the texture map. Each value of this flow indicates the
coordinates $(x,y)$ of the input image that the corresponding texel is sampled
from. Eventually, the final texture map $I_{\textrm{tex}}=G(I,\mathcal{F})$ is
produced by bilinear sampling $G$ of the input image $I$. The predicted
texture map is then used to render the textured 3D mesh. The model is trained
to minimize the reconstruction loss after rendering the texture and geometry
on the foreground pixels using L1 and perceptual loss.

###  3.2 3D Representations for tracking

The tracking performance is greatly affected by the cues and the type of
representation used for association. Our embeddings are computed as natural
outputs of the HMAR model as follows:

Appearance: HMAR recovers the appearance of the person in the form of the
texture image $I_{\textrm{tex}}$. For tracking, we need to create a compact
embedding for appearance, so we employ an auto-encoder to further encode this
appearance. The auto-encoder takes the texture image as input and is trained
to reconstruct it by going through a low dimensional bottleneck. This
bottleneck is represented by a 512-dimensional vector $\mathbf{\bar{a}}$,
which is used as the representation of the appearance in our pipeline.

3D Pose: For the pose representation, we follow previous work [19, 48] and use
the embedding $\mathbf{\bar{p}}\in\mathbb{R}^{2048}$ of HMR as a
representation of the pose of the person.

3D Location: Given an image with dimensions $[W,H]$, HMAR is applied on a
square bounding box with center $[c_{x},c_{y}]$ and dimensions $[b,b]$. HMAR
also predicts the local camera parameters $[s,t_{x},t_{y}]$ (scale and 2D
translation) for the input bounding box. By assuming a constant focal length
$f$ for the image, we can recover the approximate 3D global translation of the
person, which localizes the mesh in the view coordinate 3D space corresponding
to the full image. This translation is expressed as:

$T=\left[\begin{array}[]{ccc}t_{x}+\frac{2c_{x}-W}{sb},&t_{y}+\frac{2c_{y}-H}{sb},&\frac{2f}{sb}\end{array}\right].$
(1)

Since we do not know the actual focal length, this is only an approximation of
the metric location, but we can still recover consistent relative 3D positions
of the people over a single video up to scale.

For a more detailed description of the 3D location, we also extract the 3D
keypoints of the person that are directly regressed from the recovered mesh,
and we position them in the view coordinate 3D space by adding the translation
$T$ to them. This representation is strictly more informative than the 2D
bounding box information, both because it is in 3D, and because it provides a
localized information about the pose of the person than a single bounding box.
We also concatenate the temporal information to the 3D keypoints in the spirit
of [38], which leads to space-time representation
$\mathbf{\bar{s}}\in\mathbb{R}^{90}$.

Figure 3: _Human Mesh and Appearance Recovery (HMAR):_ Our HMAR model learns
to predict 3D pose, a UV-image and a set of keypoints relative to the whole
image from a single person center cropped image. For the pose, we use the HMR
model [17], which gives us a 2048-dimensional pose vector
($\mathbf{\bar{p}}$). We add an additional appearance head to HMR, which
learns to predict a flow-field. When the input image is sampled using the
flow-field, it produces a UV-image corresponding to the appearance of the
person in the image. The UV-image is then encoded into a appearance vector
($\mathbf{\bar{a}}$). Finally, we extract the location of 15 3D keypoints
($\mathbf{s}$) from the 3D mesh. These keypoints represent the position of the
human in the 3D space. Therefore, using our HMAR model we can get pose,
appearance, and location information of the person. Figure 4: _HMAR results._
For each example we show from left to right: (a) Input image, (b) Predicted
mesh, (c) predicted texture map, (d) and (e) textured mesh from frontal and
side view.

###  3.3 Spatio-temporal Feature Aggregation

Our HMAR model encodes the 3D appearance, pose, and location information of
each bounding box. While this gives us a rich 3D embedding, this
representation only captures information from a single-frame. In order to
incorporate the spatio-temporal information of the surrounding bounding boxes,
we employ a modified transformer model to aggregate global information across
space and time. This transformer essentially performs anisotropic diffusion
[32] of the representation across all bounding boxes, where the similarity is
captured by attention.

Specifically, the input to transformer is the combination of the three
different representations obtained from HMAR corresponding to 3D appearance,
pose and location,
$\mathbf{h}=[\mathbf{\bar{a}}^{T},\mathbf{\bar{p}}^{T},\mathbf{\bar{s}}^{T}]^{T}\in\mathbb{R}^{2650\times
1}$. The transformer architecture consists of 3 blocks of self-attention and
feed-forward layers. We use a single head attention inside the self-attention
layer. During training, we feed $T$ number of frames, each with a maximum
number of $P$ people on it. Therefore, for a single clip, the transformer
could see $T\times P$ number of humans.

We use a modified transformer that treats each of the three attributes of the
$\mathbf{h}^{t,i}$ separately. Specifically, inside the self-attention layer,
appearance, pose and keypoints have their own value, query, and key vectors.

$\displaystyle\mathbf{v}_{app}^{t,i}=W^{v}_{app}\mathbf{\bar{a}},\ \ \ \ \ \
\mathbf{v}_{pose}^{t,i}=W^{v}_{pose}\mathbf{\bar{p}},\ \ \ \ \
\mathbf{v}_{loc}^{t,i}=W^{v}_{loc}\mathbf{\bar{s}}$ (2)

Here, $W^{v}_{app}\in\mathbb{R}^{512\times
512},W^{v}_{pose}\in\mathbb{R}^{2048\times 2048}$ and
$W^{v}_{loc}\in\mathbb{R}^{90\times 90}$ are the transformation matrices for
the value vectors of each attribute in $\mathbf{h}^{t,i}$ vector. Similarly,
we can get
$[\mathbf{q}_{app}^{t,i},\mathbf{q}_{pose}^{t,i},\mathbf{q}_{loc}^{t,i}]$ and
$[\mathbf{k}_{app}^{t,i},\mathbf{k}_{pose}^{t,i},\mathbf{k}_{loc}^{t,i}]$
vectors by learning projections $W^{q}_{att},W^{k}_{att},\ \ \text{for}\ \
att\in[pose,loc]$. With this setting, we can find attentions for each
attribute separately. Finally, the total attention
$\mathtt{A}\in\mathbb{R}^{TP\times TP}$ is taken as a weighted sum over
various attentions (see also Fig 5):

$\displaystyle\mathtt{A}^{(t,i),(t^{\prime},i^{\prime})}$
$\displaystyle=\sum_{att\in[app,pose,loc]}\beta_{att}\cdot\texttt{softmax}\left(\frac{\mathbf{q}_{att}^{t,i}(\mathbf{k}_{att}^{t^{\prime},i^{\prime}})^{T}}{\sqrt{dim_{att}}}\right)$
(3)

Here, $\beta_{app},\beta_{pose}$ and $\beta_{loc}$ are a set of hyper-
parameters. While we use equal values for all of our experiments for
simplicity, this design allows for selective attention to different
attributes. For example, if we use $[0,1,0]$ for $\beta_{app},\beta_{pose}$
and $\beta_{loc}$ respectively, the appearance and location embeddings are
accumulated based only on the attention of the pose embedding. This could be
useful for extreme cases, such as when where all people wear the same uniform.
Finally, the output of the self-attention layer $\mathbf{h}^{t,i}_{sa}$ is
calculated as the weighted sum of the total attention.

$\displaystyle\mathbf{h}^{t,i}_{sa}$
$\displaystyle=\mathbf{h}^{t,i}+\sum_{t^{\prime},i^{\prime}}\mathtt{A}^{(t,i),(t^{\prime},i^{\prime})}\cdot[\mathbf{v}_{app}^{t^{\prime},i^{\prime}},\mathbf{v}_{pose}^{t^{\prime},i^{\prime}},\mathbf{v}_{loc}^{t^{\prime},i^{\prime}}]^{T}$
(4)

We use the same strategy for following feed-forward layers and normalization
layers, by treating each attribute separately as in Eq 2. After these human
vectors $\mathbf{h}^{t,i}$ have been processed by the transformer, the output
vector $\mathbf{\widehat{h}}^{t,i}$ will accumulate information across
multiple people over space-time. However, ideally we want
$\mathbf{\widehat{h}}^{t,i}$ to attend only the persons with same identities.
For example, $\mathbf{\widehat{h}}^{t,i}$ should attend
$\mathbf{\widehat{h}}^{t^{\prime},i^{\prime}}$ if and only if $i,i^{\prime}$
are the same person in different frames. If this happens,
$\mathbf{\widehat{h}}^{t,i}$ and $\mathbf{\widehat{h}}^{t^{\prime},i;}$ will
be the same or at least close to each other in some similarity metric
$d(\mathbf{\widehat{h}}^{t,i},\mathbf{\widehat{h}}^{t^{\prime},i;})$. We
enforce this during training by minimizing the ReID (Re-Identification) loss.

$\displaystyle
L_{\text{ReID}}=\sum_{t,t^{\prime}\sim[1,2,...,T]}d(\mathbf{\widehat{h}}^{t,i},\mathbf{\widehat{h}}^{t^{\prime},i^{\prime}})+\max\\{0,m-d(\mathbf{\widehat{h}}^{t,i},\mathbf{\widehat{h}}^{t^{\prime},j^{\prime}})\\}$
(5)

Here, $\mathbf{\widehat{h}}^{t,i}$ and
$\mathbf{\widehat{h}}^{t^{\prime},i^{\prime}}$ are the embedding vectors for
the same person in frame $t,t^{\prime}$ respectively, while
$\mathbf{\widehat{h}}^{t,i}$ and
$\mathbf{\widehat{h}}^{t^{\prime},j^{\prime}}$ are the embedding vectors for
two different people in frames $t,t^{\prime}$. $\mathbf{\widehat{h}}^{t,i}$
acts as an anchor for the contrastive loss. We use margin $m=10$ for the
negative samples and $L_{2}$ distance for the distance metric $d$.

Figure 5: _Spatio-Temporal Aggregation with Transformer:_ For $T$ frames in a
video, we obtain a set of bounding boxes from an off-the-shelf human detector
[34]. We use HMAR to extract representations of 3D appearance, pose, and
location from each bounding box, whose concatenation serves as a token for
each bounding box. Since this information is only extracted from a single
image, we send the tokens through a transformer, which acts as a diffusion
mechanism to propagate information across space and time. Our modified
transformer computes attention on each embedding separately, which then gets
merged together for a final attention map.

###  3.4 Association

Given the 3D representations that have been processed over space-time, the
remaining task is to assign identities to each bounding box. We employ a
simple association scheme to demonstrate the effectiveness of these
representations. Specifically, the output vectors from the transformer
$\mathbf{\widehat{h}}^{t,i}$ for all $i\in[P]$ at frame $t+1$ need to be
associated to the existing tracklets up to $t$ or to a new tracklet if no
match exists. We use a simple tracking algorithm that takes one frame at a
time and assigns track-ids based on the similarities between the past
tracklets and the new detections.

First, we start with frame 1, and assign unique identities for each person.
This initiates a set of tracklets $\mathcal{T}=\\{\mathbf{Tr}_{p}\\}$ for all
people $p$ in the first frame. Every tracklet is also associated with the
corresponding human embeddings $\mathbf{\widehat{h}}^{1,k}$. Then we move on
to frame 2 and calculate the affinity matrix between the detected humans
$\mathcal{D}=\\{\mathbf{\widehat{h}}^{2,i}\\}$ in frame 2, and all active
tracklets $\mathcal{T}=\\{\mathbf{Tr}_{j}\\}$. We use the Hungarian algorithm
to assign detected humans to the tracklets, using the minimum distance between
the last 20 embeddings vectors associated with the tracklet and the embeddings
vector of the detected human. This process continues for all frames
sequentially. In this process, some of the tracklets might be missed and some
detections might not be assigned. In some cases, there will be new humans
added to the scene, and some people will be occluded or leave the scene. For
the first case, we assign a new tracklet to the new person if there are no
matches between old tracklets. For the latter case, we keep the old tracklets
alive, even if there are no matches for few number of frames. We remove the
tracklet if no new person is assigned to the tracklet continuously for 20
frames. A sketch of this tracking algorithm is shown in Algorithm 1.

Algorithm 1 Tracking Algorithm

1:procedure Online Tracking

2:Require: $\mathbf{\widehat{h}}^{t,i}$ for all $t\in[T],i\in[P]$, total
number of frames $T$, maximum number of people per frame $P$.

3: $\mathcal{T}\leftarrow\\{\mathbf{Tr}_{i}(\mathbf{\widehat{h}}^{1,i}),\ \
\forall i\in[1,2,...,P]\\}$ $\triangleright$ Initiate tracklets with
detections from the first frame.

4: for $t\in[2,3,...,T]$ do

5: $\mathcal{D}\leftarrow\\{\mathbf{\widehat{h}}^{t,i},\ \ \forall
i\in[1,2,...,P]\\}$ $\triangleright$ Create a detected human set.

6: Calculate Affinity
$\mathbb{A}_{i,j}\leftarrow\texttt{min}\\{\tau,d(D_{i},\mathbf{Tr}_{j})\\}$
for $D_{i}\in\mathcal{D}$ and $\mathbf{Tr}_{j}\in\mathcal{T}$

7:
$\mathcal{M},\mathcal{T}_{u},\mathcal{D}_{u}\leftarrow\texttt{Assignment}(\mathbb{A})$
$\triangleright$ Hungarian algorithm to assign detections to tracklets.

8: $\mathcal{T}\leftarrow\\{\mathbf{Tr}_{j}(\mathbf{\widehat{h}}^{t,i}),\
\mathbf{Tr}_{j}(age)=0,\ \ \forall(i,j)\in\mathcal{M}\\}$ $\triangleright$
assign detections to tracks.

9: $\mathcal{T}\leftarrow\\{\mathbf{Tr}_{j}(age)+=1,\ \
\forall(j)\in\mathcal{T}_{u}\\}$ $\triangleright$ Increase the age of
unmatched tracks by one.

10: $\mathcal{T}\leftarrow\\{\mathbf{Tr}_{j}(\mathbf{\widehat{h}}^{t,i}),\ \
\forall(i)\in\mathcal{D}_{u},\ j=|\mathcal{T}+1|\\}$ $\triangleright$ Add new
tracks for new detections.

11: Kill the tracks with age $\geq$ 24\. return Tracklets $\mathcal{T}$

##  4 Experiments

To evaluate the proposed approach, we first evaluate the efficacy of 3D
representations over 2D representations, then provide ablations for the
different components of our system and finally compare against state-of-the-
art tracking methods. We experiment on three datasets: PoseTrack [1], MuPoTS
[27], and AVA [13], which span a large variety of challenging sequences,
including sports, daily activities and storytelling in movies. Since our
approach requires mid-to-high resolution images of people, we do not evaluate
on the MOTChallenge datasets [7], because of their focus on low resolution
humanss. However, we expect that as the technology for 3D human reconstruction
is improving, e.g., [44], our approach will be applicable in these settings as
well. In all cases, we rely on detections from an off-the-shelf detection
system [34], and we solve for the association, assigning identity labels to
each bounding box using the proposed approach. As a result, the metrics we
employ also focus on identity tracking on the bounding box level.
Specifically, we report Identity switches (IDs), or the number of times a
recovered trajectory switches from one ground truth tracklet to another. We
also report the standard Multi-Object Tracking Accuracy (MOTA) [20] which
summarizes errors from false positives, false negatives and ID switches.
Finally, we also report the ID F1 score (IDF1) [35] which emphasizes the
preservation of the track identity over the entire sequence.

Tracking people in edited media requires robustness against shot changes. For
this evaluation we use AVA, which is a dataset of movies with actors annotated
with identities. We use the test split from [30], which consists of sequences
that contain shot changes. In this setting we _only_ evaluate on tracklets
that are persistent across shot changes, so to avoid being influenced by
detection, we only report IDs and IDF1 on this dataset.

###  4.1 Implementation details

To train HMAR, we use a pretrained HMR model as the starting point. On top of
the Resnet-50 backbone that HMR uses, we add additional layers from which we
predict the texture flow to predict the texture map. We train this method
using images from COCO [24], MPII [2] and Human3.6M [15]. We train the
appearance head for roughly 500k iterations with a learning rate of 0.0001 and
a batch size of 16 images while keeping the pose head frozen. Please see
supplemental for more details. After training HMAR, we train an auto-encoder
on the predicted texture map, which results in an encoded appearance embedding
$\mathbf{\bar{a}}$. We train this using the same datasets as HMAR, and
training lasts for about 100k iterations, with a learning rate of 0.0001.
Finally, for training the transformer, we use the training set of PoseTrack
[1] as this process requires videos with multiple people with labeled IDs for
training. Training lasts for 10k iterations with a learning rate of 0.001. All
experiments are conducted on a single RTX 2080 Ti.

###  4.2 Quantitative evaluation

####  4.2.1 3D vs 2D

We first evaluate the effectiveness of 3D representations over 2D
representations. To this end, we train a simpler version of our system that
only uses one cue and compare with 2D and 3D versions of these cues. The full
results of these experiments are presented in Table 1. We experiment with two
popular 2D appearance features used in AlphaPose [9, 50] and Tracktor [4],
which are convolutional features trained with Re-ID losses. We compare this
against the proposed 3D appearance i.e., the encoded texture map. As we can
see at the first three rows of Table 1, our 3D-based representation
outperforms the popular 2D-based representations, except on AVA where we
perform on par with Tracktor. Regarding the performance of pose/location cues,
we compare 3D keypoints with the corresponding 2D keypoints (coming from the
projection of the 3D keypoints on the image plane). Again, the 3D-based cue
outperform the 2D-based counterpart across the board.

| Embedding | 2D/3D | Posetrack | MuPoTs | AVA
---|---|---|---|---|---
| IDs$\downarrow$ | MOTA$\uparrow$ | IDF1$\uparrow$ | IDs$\downarrow$ | MOTA$\uparrow$ | IDF1$\uparrow$ | IDs$\downarrow$ | IDF1$\uparrow$
App. | Re-ID (OSNet) [50] | 2D | 892 | 63.1 | 77.2 | 88 | 62.7 | 81.2 | 454 | 59.2
Re-ID (Tracktor) [4] | 2D | 848 | 63.2 | 78.0 | 67 | 62.8 | 79.8 | 242 | 64.6
Texture map (Ours) | 3D | 783 | 63.4 | 77.3 | 44 | 62.9 | 82.2 | 246 | 64.2
P&L | 2D pose & location | 2D | 1872 | 59.8 | 74.2 | 194 | 62.2 | 80.8 | 575 | 56.6
3D pose & location | 3D | 1563 | 61.6 | 75.4 | 52 | 62.8 | 80.1 | 280 | 64.2

Table 1: _Effect of 3D Information:_ To assess the effect of 3D information,
we train our model using only one embedding that corresponds to either
appearance or position/location and compare between the 2D and 3D
representations. Using representations based on 3D information performs
consistently better than the corresponding 2D-based representations.

####  4.2.2 Relative importance of different components

Having established the effectiveness of the 3D representation for tracking, we
continue with a careful ablation of the components of our system. We report
the performance of the pipeline by individually removing each of the 3D cues
(i.e., appearance, pose and 3D location). We also remove the spatio-temporal
aggregation of the transformer for ablation. The results for the different
versions, as well as the full system are presented in Table 2.

These ablations show that 3D appearance is one of the most salient cue across
the board. On PoseTrack and MuPoTs 3D location is more important than 3D pose
(which is partially subsumed by location since that is based on keypoints).
However on AVA, because of shot changes, the two most important cues are
appearance and 3D pose, because across a shot boundary 3D pose is preserved
while location is not. The transformer consistently adds to the overall
robustness of the method. Note that PoseTrack training data is limited
(roughly 800 videos each less than a minute), particularly compared to the
usual scale of the datasets that transformers are usually trained on, e.g.
300M images for JFT [8]. Potentially using larger training data could improve
the performance even more.

Method | PoseTrack | MuPoTs | AVA
---|---|---|---
IDs$\downarrow$ | MOTA$\uparrow$ | IDF1$\uparrow$ | IDs$\downarrow$ | MOTA$\uparrow$ | IDF1$\uparrow$ | IDs$\downarrow$ | IDF1$\uparrow$
w/o appearance | 1783 | 55.2 | 73.7 | 151 | 64.5 | 78.1 | 739 | 48.7
w/o 3D pose | 711 | 55.6 | 72.2 | 43 | 65.3 | 82.1 | 355 | 65.1
w/o 3D location | 1315 | 54.3 | 69.7 | 59 | 65.3 | 82.8 | 273 | 61.9
w/o transformer | 685 | 49.6 | 64.5 | 41 | 59.1 | 73.7 | 260 | 63.4
Full system | 655 | 55.8 | 73.4 | 38 | 62.1 | 79.1 | 240 | 61.3

Table 2: _Ablation of the components of our system:_ We train our system by
removing each cue at a time as well as the transformer for the ablative study.
We find that appearance is a salient cue across the board, 3D location is more
important than 3D pose in PoseTrack and MuPoTs, but in AVA due to shot changes
3D pose (which is view invariant) is more useful than location. Transformer
consistently improves the performance.

####  4.2.3 Comparison with the state-of-the-art

Lastly, we compare our approach with the state-of-the-art tracking methods.
For this comparison, we use four recent popular methods for tracking:
Trackformer [28], Tracktor [4], Alphapose [9] and PoseFlow [42]. We run these
methods out-of-the-box on all benchmarks and report their performance on
identity tracking. For a more clear comparison, we provide results with two
versions of our approach, one without using the transformer (Ours-v1), and one
with the spatio-temporal aggregation of the transformer (Ours-v2). We do this,
because our transformer is trained on PoseTrack, but some of the above
baselines are not using PoseTrack for training [4, 28], so the Ours-v1
baseline is a more fair choice for a direct comparison. The full results are
presented in Table 3. The two versions of our approach perform consistently
better than previous work across the board. In Figure 6, we can also see
qualitatively the benefits of our approach compared to these baselines. More
specifically, the design choices of our approach make it particularly suitable
for challenging cases involving person-person occlusion, or shot changes which
are common in edited media (AVA dataset), because our 3D representations
improve identity association, when it comes to linking IDs after the person-
person occlusions or shot changes.

Method | Posetrack | MuPoTs | AVA
---|---|---|---
IDs$\downarrow$ |  MOTA$\uparrow$ |  IDF1$\uparrow$ |  IDs$\downarrow$ |  MOTA$\uparrow$ |  IDF1$\uparrow$ |  IDs$\downarrow$ |  IDF1$\uparrow$
Trackformer [28] | 1263 | 33.7 | 64.0 | 43 | 24.9 | 62.7 | 716 | 40.9
Tracktor [4] | 702 | 42.4 | 65.2 | 53 | 51.5 | 70.9 | 289 | 46.8
Ours-v1 | 685 | 49.6 | 64.5 | 41 | 59.1 | 73.7 | 260 | 63.4
AlphaPose [9] | 2220 | 36.9 | 66.9 | 117 | 37.8 | 67.6 | 939 | 41.9
FlowPose [42] | 1047 | 15.4 | 64.2 | 49 | 21.4 | 67.1 | 452 | 52.9
Ours-v2 | 655 | 55.8 | 73.4 | 38 | 62.1 | 79.1 | 240 | 61.3
Table 3: _Comparison with state-of-the-art tracking methods:_ We provide
results for two versions of our approach. Ours-v1 does not include the
transformer, while Ours-v2 applies the transformer for the spatio-temporal
aggregation. Since our transformer is trained on PoseTrack, we provide these
two versions for a more fair comparison to methods that do not train on this
dataset (i.e., [4, 28]). Figure 6: _Qualitative results._ We show samples from
the three datasets we experiment on, PoseTrack [1], MuPoTs [27] and AVA [13].
We provide results for our method, AlphaPose [9] and Trackformer [28]. More
video results are available in: https://brjathu.github.io/T3DP.

##  5 Concluding Remarks

We have demonstrated the value of using 3D representations for tracking people
in a wide variety of videos. The proposed approach is simple and modular, and
can easily incorporate extra representations such as face embeddings, which
may be useful for movies. Our technique relies on having enough resolution on
each person to enable lifting to 3D from 2D. Our approach is also susceptible
to bad detections, since 3D features computed on non-human regions lead to
spurious 3D representations. There are many applications of tracking people in
video that have positive societal impact, such as home monitoring for elder
care, design of architectural spaces, scientific analysis of child development
and more. However, such technology can also be misused for mass-surveillance.

Acknowledgements: This work was supported by ONR MURI (N00014-14-1-0671) as
well as BAIR and BDD sponsors.

## References

  * [1] Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele.  PoseTrack: A benchmark for human pose estimation and tracking.  In CVPR, 2018.
  * [2] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2D human pose estimation: New benchmark and state of the art analysis.  In CVPR, 2014.
  * [3] Anurag Arnab, Carl Doersch, and Andrew Zisserman.  Exploiting temporal context for 3D human pose estimation in the wild.  In CVPR, 2019.
  * [4] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe.  Tracking without bells and whistles.  In ICCV, 2019.
  * [5] Ernesto Brau, Jinyan Guan, Kyle Simek, Luca Del Pero, Colin Reimer Dawson, and Kobus Barnard.  Bayesian 3D tracking from monocular video.  In CVPR, 2013.
  * [6] Christoph Bregler and Jitendra Malik.  Tracking people with twists and exponential maps.  In CVPR, 1998.
  * [7] Patrick Dendorfer, Aljosa Osep, Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leal-Taixé.  MOTChallenge: A benchmark for single-camera multiple target tracking.  IJCV, 129(4):845–881, 2021.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [9] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu.  RMPE: Regional multi-person pose estimation.  In ICCV, 2017.
  * [10] David F Fouhey, Wei-cheng Kuo, Alexei A Efros, and Jitendra Malik.  From lifestyle VLOGs to everyday interactions.  In CVPR, 2018.
  * [11] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik.  Recurrent network models for human dynamics.  In ICCV, 2015.
  * [12] Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri, and Du Tran.  Detect-and-track: Efficient pose estimation in videos.  In CVPR, 2018.
  * [13] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik.  AVA: A video dataset of spatio-temporally localized atomic visual actions.  In CVPR, 2018.
  * [14] Peng Guan, Alexander Weiss, Alexandru O Balan, and Michael J Black.  Estimating human shape and pose from a single image.  In ICCV, 2009.
  * [15] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments.  PAMI, 36(7):1325–1339, 2013.
  * [16] Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis.  Coherent reconstruction of multiple humans from a single image.  In CVPR, 2020.
  * [17] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In CVPR, 2018.
  * [18] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In ECCV, 2018.
  * [19] Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik.  Learning 3D human dynamics from video.  In CVPR, 2019.
  * [20] Rangachar Kasturi, Dmitry Goldgof, Padmanabhan Soundararajan, Vasant Manohar, John Garofolo, Rachel Bowers, Matthew Boonstra, Valentina Korzhova, and Jing Zhang.  Framework for performance evaluation of face, text, and vehicle detection and tracking in video: Data, metrics, and protocol.  PAMI, 31(2):319–336, 2008.
  * [21] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  VIBE: Video inference for human body pose and shape estimation.  In CVPR, 2020.
  * [22] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis.  Learning to reconstruct 3D human pose and shape via model-fitting in the loop.  In ICCV, 2019.
  * [23] Oh-Hun Kwon, Julian Tanke, and Juergen Gall.  Recursive bayesian filtering for multiple human pose tracking from multiple cameras.  In ACCV, 2020.
  * [24] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In ECCV, 2014.
  * [25] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  SMPL: A skinned multi-person linear model.  ACM Transactions on Graphics (TOG), 34(6):1–16, 2015.
  * [26] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, and Christian Theobalt.  XNect: Real-time multi-person 3D motion capture with a single RGB camera.  ACM Transactions on Graphics (TOG), 39(4):82–1, 2020.
  * [27] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, and Christian Theobalt.  Single-shot multi-person 3D pose estimation from monocular RGB.  In 3DV, 2018.
  * [28] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer.  TrackFormer: Multi-object tracking with transformers.  arXiv preprint arXiv:2101.02702, 2021.
  * [29] Georgios Pavlakos, Nikos Kolotouros, and Kostas Daniilidis.  TexturePose: Supervising human mesh estimation with texture consistency.  In ICCV, 2019.
  * [30] Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa.  Human mesh recovery from multiple shots.  arXiv preprint arXiv:2012.09843, 2020.
  * [31] Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine.  Sfv: Reinforcement learning of physical skills from videos.  ACM Transactions on Graphics (TOG), 37(6):1–14, 2018.
  * [32] Pietro Perona and Jitendra Malik.  Scale-space and edge detection using anisotropic diffusion.  PAMI, 12(7):629–639, 1990.
  * [33] N Dinesh Reddy, Laurent Guigues, Leonid Pischulin, Jayan Eledath, and Srinivasa Narasimhan.  TesseTrack: End-to-end learnable multi-person articulated 3D pose tracking.  In CVPR, 2021.
  * [34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  PAMI, 39(6):1137–1149, 2016.
  * [35] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi.  Performance measures and a data set for multi-target, multi-camera tracking.  In ECCV, 2016.
  * [36] Michael Snower, Asim Kadav, Farley Lai, and Hans Peter Graf.  15 keypoints is all you need.  In CVPR, 2020.
  * [37] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael J Black, and Tao Mei.  Monocular, one-stage, regression of multiple 3D people.  In ICCV, 2021.
  * [38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, 2017.
  * [39] Jian Wang, Yunshan Zhong, Yachun Li, Chi Zhang, and Yichen Wei.  Re-identification supervised texture generation.  In CVPR, 2019.
  * [40] Xinshuo Weng, Yongxin Wang, Yunze Man, and Kris M Kitani.  GNN3DMOT: Graph neural network for 3D multi-object tracking with 2D-3D multi-feature learning.  In CVPR, 2020.
  * [41] Bin Xiao, Haiping Wu, and Yichen Wei.  Simple baselines for human pose estimation and tracking.  In ECCV, 2018.
  * [42] Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, and Cewu Lu.  Pose Flow: Efficient online pose tracking.  In BMVC, 2018.
  * [43] Jiarui Xu, Yue Cao, Zheng Zhang, and Han Hu.  Spatial-temporal relation networks for multi-object tracking.  In ICCV, 2019.
  * [44] Xiangyu Xu, Hao Chen, Francesc Moreno-Noguer, László A Jeni, and Fernando De la Torre.  3D human shape and pose from a single low-resolution image with self-supervised learning.  In ECCV, 2020.
  * [45] Xiangyu Xu, Hao Chen, Francesc Moreno-Noguer, Laszlo A Jeni, and Fernando De la Torre.  3D human pose, shape and texture from low-resolution images and videos.  PAMI, 2021.
  * [46] Yihong Xu, Aljosa Osep, Yutong Ban, Radu Horaud, Laura Leal-Taixé, and Xavier Alameda-Pineda.  How to train your deep multi-object tracker.  In CVPR, 2020.
  * [47] Andrei Zanfir, Elisabeta Marinoiu, Mihai Zanfir, Alin-Ionut Popa, and Cristian Sminchisescu.  Deep network for the integrated 3D sensing of multiple people in natural images.  NeurIPS, 2018.
  * [48] Jason Y Zhang, Panna Felsen, Angjoo Kanazawa, and Jitendra Malik.  Predicting 3D human dynamics from video.  In ICCV, 2019.
  * [49] Yuxiang Zhang, Liang An, Tao Yu, Xiu Li, Kun Li, and Yebin Liu.  4D association graph for realtime multi-person motion capture using multiple video cameras.  In CVPR, 2020.
  * [50] Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang.  Omni-scale feature learning for person re-identification.  In ICCV, 2019.
  * [51] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros.  View synthesis by appearance flow.  In ECCV, 2016.
  * [52] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Tracking objects as points.  In ECCV, 2020.


# Topological edge states in photonic Floquet insulator with unpaired Dirac
cones

Hua Zhong  Key Laboratory for Physical Electronics and Devices, Ministry of
Education, School of Electronic Science and Engineering, Xi’an Jiaotong
University, Xi’an 710049, China  Yaroslav V. Kartashov  Institute of
Spectroscopy, Russian Academy of Sciences, Troitsk, Moscow, 108840, Russia
Yongdong Li  Key Laboratory for Physical Electronics and Devices, Ministry of
Education, School of Electronic Science and Engineering, Xi’an Jiaotong
University, Xi’an 710049, China  Ming Li  State Key Laboratory of Transient
Optics and Photonics, Xi’an Institute of Optics and Precision Mechanics of
Chinese Academy of Sciences, 710119 Xi’an, China  Yiqi Zhang  Key Laboratory
for Physical Electronics and Devices, Ministry of Education, School of
Electronic Science and Engineering, Xi’an Jiaotong University, Xi’an 710049,
China<EMAIL_ADDRESS>
###### Abstract

Topological insulators are most frequently constructed using lattices with
specific degeneracies in their linear spectra, such as Dirac points. For a
broad class of lattices, such as honeycomb ones, these points and associated
Dirac cones generally appear in non-equivalent pairs. Simultaneous breakup of
the time-reversal and inversion symmetry in systems based on such lattices may
result in the formation of the unpaired Dirac cones in bulk spectrum, but the
existence of topologically protected edge states in such structures remains an
open problem. Here photonic Floquet insulator on honeycomb lattice with
unpaired Dirac cones in its spectrum is introduced that can support
unidirectional edge states appearing at the edge between two regions with
opposite sublattice detuning. Topological properties of this system are
characterized by the nonzero valley Chern number. Remarkably, edge states in
this system can circumvent sharp corners without inter-valley scattering even
though there is no total forbidden gap in the spectrum. Our results reveal
unusual interplay between two different physical mechanisms of creation of
topological edge states based on simultaneous breakup of different symmetries
of the system.

††journal: opticajournal

##  1 Introduction

Dirac points are specific degeneracies, where two bands touch each other,
representative for linear spectra of many periodic systems, including
honeycomb, triangular, kagome, and Lieb lattices. Dispersion relation is
linear around Dirac points that leads to unusual dispersion dynamics of the
wavepackets exciting these points [1]. A variety of physical effects
associated with the presence of Dirac points are reported in material
sciences, acoustics, optics and physics of matter waves. In particular,
honeycomb lattices with Dirac degeneracies [2, 3, 4, 5], representing photonic
analogues of graphene [6], are widely used for construction of topological
insulators – novel materials, where excitations can propagate along the edges
of the structure, but not in its bulk [7, 8, 9, 10]. In usual honeycomb
lattices Dirac points emerge in six corners of the Brillouin zone, in pairs
corresponding to non-equivalent K and $\textbf{K}^{\prime}$ points. However,
in [11] a system with “parity anomaly” was suggested, which may possess
unpaired Dirac cones if the time-reversal symmetry and the inversion symmetry
of the system are broken simultaneously. The existence of the unpaired Dirac
cones implies that massless chiral fermions in such a system loose anomaly-
cancelling partners with opposite chirality, and thus their appearance breaks
the “fermion doubling” theorem [11]. However, due to unique nature of the
unpaired Dirac points and specific symmetry properties that the system
possessing them should manifest, the search of such materials is a challenging
and open problem.

Different approaches can be used to break time-reversal symmetry in photonic
or optoelectronic systems. Among them is the utilization of gyromagnetic
optical materials in external magnetic fields [12], or polariton condensates
with pronounced spin-orbit coupling in magnetic fields [13, 14, 15], see also
review [8]. This effect can also be achieved by using longitudinal modulations
in waveguide-based systems, such as photonic Floquet insulators on helical
waveguide arrays [16], where a number of topological phenomena have been
predicted and observed in linear and nonlinear regimes [17, 18, 19, 20]. In
these latter systems the breakup of time-reversal symmetry is achieved due to
the effective gauge field arising from waveguide modulations [21, 22, 23]. By
the way, the Floquet waveguides arrays always provide promising results in
light manipulations [24, 25, 26]. Unpaired Dirac points can, in principle, be
observed in helical square waveguide arrays, as suggested in [27, 28], where
introduction of difference into phases of helical waveguide rotation in two
sublattices also breaks inversion symmetry of the structure [29, 30, 31].
Experimentally, unpaired Dirac points were observed for microwaves in
gyromagnetic materials [32], while transitions between regimes with broken
inversion and time-reversal symmetries in gyromagnetic structures were
reported in [33]. Very recently, unpaired Dirac points were also proposed and
observed in a photonic crystal consisting of Y-shaped gyromagnetic rods [34].
Nevertheless, unpaired Dirac points remain elusive at optical frequencies in
waveguiding systems. Topological edge states, that should have unique
properties in such materials connected with unusual structure of their linear
spectra lacking total gap, have never been reported, to the best of our
knowledge.

In this work, we show that such topological edge states can be observed when
one simultaneously introduces into a honeycomb waveguide array the helicity of
waveguides, resulting in breakup of its time-reversal symmetry, and also the
detuning between its two sublattices, resulting in breakup of the inversion
symmetry of the structure. Notice that the approach utilizing sublattice
detuning [35, 36, 37, 38, 39, 40, 41] is also quite involved in the
investigations of the valley Hall effect [42] where, however, the system
possesses the total gap, in contrast to system with unpaired Dirac points
considered here. We show that the bulk spectrum of our system has unique
structure with gap opened between, say, three $\textbf{K}^{\prime}$ points,
and preserved Dirac cones in non-equivalent to them three K points.
Remarkably, we find that the domain wall between two arrays with different
signs of sublattice detuning supports topologically protected edge states,
which, in complete contrast to valley Hall systems, feature asymmetric
projected dependence of quasi-propagation constant on Bloch momentum and
connect two different bands. Moreover, such edge states are topologically
protected and can circumvent sharp corners of the domain wall without
backscattering, despite the absence of complete gap in the spectrum of this
system. Our results suggest a new platform for observation of the unpaired
Dirac cones and open new prospects for investigation of nonlinear effects in
topological systems lacking complete gap.

##  2 Theoretical model

###  2.1 Helical waveguide array and its band structure

The propagation dynamics of a laser beam in helical shallow waveguide array
can be described by the Schrödinger-like equation for normalized amplitude
$\psi$ of the light field:

$i\frac{\partial\psi}{\partial
z}=-\frac{1}{2}\left(\frac{\partial^{2}}{\partial
x^{2}}+\frac{\partial^{2}}{\partial y^{2}}\right)\psi-\mathcal{R}(x,y,z)\psi,$
(1)

where $x,\,y$ are the transverse coordinates which are to the characteristic
scale ${r_{0}=10\,\mu\textrm{m}}$, $z$ is the propagation distance that is
normalized to the diffraction length ${kr_{0}^{2}\approx 1.14\,\textrm{mm}}$,
${k=2\pi n/\lambda}$ is the wavenumber in the medium with the background
refractive index $n$ (for fused silica ${n\approx 1.45}$), and
${\lambda=800\leavevmode\nobreak\ \textrm{nm}}$ is the working wavelength.
$\mathcal{R}(x,y,z)$ describes waveguide array composed from helical Gaussian-
shaped waveguides

$\mathcal{R}=\sum_{m,n}p_{m,n}\exp\left(-\frac{[x-x_{m,n}(z)]^{2}+[y-y_{m,n}(z)]^{2}}{d^{2}}\right).$
(2)

Here, $d$ is the waveguide width, $(x_{m,n},y_{m,n})$ are the $z$-dependent
coordinates of waveguide centers in honeycomb grid,
${p_{m,n}=k^{2}r_{0}^{2}\delta n/n}$ is the waveguide depth that is
proportional to the refractive index contrast $\delta n$ in the array (for
instance, $p=1.0$ corresponds to ${\delta n\sim 1.1\times 10^{-4}}$), spacing
between neighboring waveguides in the grid is equal to $a$. In helical
waveguide array coordinates of waveguides change with $z$ as

$\left\\{\begin{split}&{x_{m,n}(z)=x_{m,n}(0)+r\sin(\Omega z)},\\\
&{y_{m,n}(z)=y_{m,n}(0)+r\cos(\Omega z)-r},\end{split}\right.$

where $r$ is the radius of helix and ${\Omega=2\pi/Z}$ is the rotation
frequency defined by period $Z$. We further introduce detuning between two
sublattices of the array, by setting depth as ${p_{m,n}=p+\delta}$ or
${p_{m,n}=p-\delta}$, where $\delta$ is the depth detuning that is connected
with different refractive index contrast in two sublattices and that
corresponds to different on-site “energies” in the language of condensed-
matter physics [35, 43, 38], as shown in schematic array illustration in Fig.
1(a). Further we set the mean waveguide depth ${p=8.9}$, typical value of
detuning ${\delta=0.1}$, waveguide spacing ${a=1.6}$ (corresponding to
$16\,\mu\rm m$), width ${d=0.4}$ (corresponding to $4\,\mu\rm m$), helix
period ${Z=6}$ (corresponding to $6.8\,\rm mm$), and radius ${r=0.4}$
(corresponding to $4\,\mu\rm m$). These parameters are typical for waveguide
arrays inscribed in fused silica using focused pulses from femtosecond laser
[16, 44, 45, 46, 47, 48].

Figure 1: (a) Schematic illustration of helical waveguide array with detuning between two sublattices, shown with red and blue colors. Green hexagons highlight honeycomb structure of the array. Band structure of bulk waveguide array with ${r=0}$ and ${\delta=0.1}$ (b), ${r=0.4}$ and ${\delta=0}$ (c), and ${r=0.4}$ and ${\delta=0.1}$ (d). Table 1: Sign of the Berry curvature $\mathcal{B}$ of valleys for different signs of detuning and different directions of waveguide rotation (helicity). ${X}$ means that the Dirac cone may appear. The rotation direction (clockwise $\circlearrowright$ or anti-clockwise $\circlearrowleft$) of the waveguide array is defined along negative $z$ direction (viz. counter to the light propagation direction).  |  |  $\mathcal{B}({\bf K})$ |  $\mathcal{B}({\bf K}^{\prime})$
---|---|---|---
① |  ${\delta>0}$ |  $-$ |  $+$
$\circlearrowright$ |  $+$ |  $+$
total |  $X$ |  $+$
② |  ${\delta>0}$ |  $-$ |  $+$
$\circlearrowleft$ |  $-$ |  $-$
total |  $-$ |  $X$
③ |  ${\delta<0}$ |  $+$ |  $-$
$\circlearrowright$ |  $+$ |  $+$
total |  $+$ |  $X$
④ |  ${\delta<0}$ |  $+$ |  $-$
$\circlearrowleft$ |  $-$ |  $-$
total |  $X$ |  $-$

To analyse the impact of helicity and detuning on band structure, we first
consider Floquet-Bloch modes of bulk array that can be found as
${\psi(\bm{r},z)=\phi(\bm{r},z)\exp(ibz+i\bm{k}\cdot\bm{r})}$, where the
Floquet-Bloch function $\phi(\bm{r},z)$ is transversely and $Z$-periodic, $b$
is the quasi-propagation constant, $\bm{k}=(k_{x},k_{y})$ is the Bloch
momentum. The function $\phi$ and the dependence $b({\bm{k}})$ can be found by
solving the equation

$b\phi=\frac{1}{2}\left(\frac{\partial^{2}}{\partial
x^{2}}+\frac{\partial^{2}}{\partial
y^{2}}\right)\phi+\mathcal{R}(x,y,z)\phi+i\frac{\partial\phi}{\partial z}$ (3)

using the “propagation and projection” method [27, 49, 50], which combines
plane-wave expansion for calculation of eigenmodes of static array and their
subsequent propagation in helical structure for calculation of quasi-
propagation constants and Floquet-Bloch modes. The spectrum of static
honeycomb array at $\delta,r=0$ is known to posses six Dirac points, belonging
to two groups K and $\textbf{K}^{\prime}$, where top and bottom bands touch.
If only the detuning $\delta$ is introduced into structure to break its
inversion symmetry, while the waveguides are straight $(r=0)$, all Dirac cones
split and become valleys, and complete gap opens in the spectrum of the system
[Fig. 1(b)]. Similar transformation of the spectrum is observed for zero
detuning $\delta=0$, when waveguides are made helical, that leads to the
breakup of the time-reversal symmetry of the system [see spectrum in Fig.
1(c)]. It should be stressed, however, that even though Dirac cones split in
both Figs. 1(b) and 1(c), the Berry curvature (see definition in Section 2.2)
in valleys behave differently: For upper band in Fig. 1(c) the curvature
becomes positive around all K and $\textbf{K}^{\prime}$ points, while in Fig.
1(b) it acquires opposite signs in K and $\textbf{K}^{\prime}$ points (Berry
curvature in the lower band in a given valley is usually opposite to that in
the upper band). Therefore, simultaneous breakup of time-reversal symmetry and
inversion symmetry in system with $\delta,r\neq 0$ leads to different
variations of Berry curvature in K and $\textbf{K}^{\prime}$ points, that may
translate into competing impact of helicity and detuning on gap width around
these points (as originally predicted in Haldane model [11] for electronic
system). Thus, we find that at ${r=0.4}$ the increase of sublattice detuning
up to ${\delta\approx 0.10}$ (for ${r_{0}=0.3}$ up to ${\delta\approx 0.07}$),
leads to restoration of the Dirac cones in three K points, while gap remains
open in three $\textbf{K}^{\prime}$ points, as shown in Fig. 1(d), i.e.
unpaired Dirac cones appear in the bulk spectrum of this system. By changing
sign of detuning $\delta$, one achieves opening of the gap in K points
instead, while Dirac points are observed in $\textbf{K}^{\prime}$ points in
this case.

Possible scenarios of the band structure transformation are summarized in
Table 1, where we use the fact that breakup of the inversion symmetry tends to
create valleys, where the Berry curvature in each band is an odd function of
$\bm{k}$, while breakup of time-reversal symmetry tends to create valleys,
where the Berry curvature is an even function of $\bm{k}$ [51]. Thus, when
time-reversal and inversion symmetries are broken simultaneously, the total
Berry curvature at valleys may be enhanced (gap width increases) or reduced
(up to restoration of Dirac point) in comparison with the case, when only one
of the symmetries is broken. One can therefore distinguish four different
situations, highlighted in Table 1, where the symbol $X$ implies the
possibility of the Dirac point restoration. For instance, one can see that
changing of only rotation direction or inversion of detuning shifts Dirac
points from $\bf{K}$ to ${\bf K}^{\prime}$ points (or vice versa), while
simultaneous change of rotation direction and inversion of detuning will have
no effect on the location of the Dirac points. We would like to note that the
rotation of the waveguide array in Fig. 1(a) is clockwise $\circlearrowright$,
and the results correspond to the case ① in Table 1.

Figure 2: Berry curvature $\mathcal{B}$ of the bottom band of the lattice with
unpaired Dirac cones. Parameters in (a) are: ${a=1.6}$, $Z=6$, $r=0.3$,
${\wp=0.1}$, ${t=0.18}$, and ${{\bm{A}}(z)=r\Omega[-\cos(\Omega z),\sin(\Omega
z)]}$. Parameters in (b) are: $a=1.6$, $Z=6$, $r=0.3$, ${\wp=-0.1}$,
${t=0.18}$, and ${{\bm{A}}(z)=r\Omega[-\cos(\Omega z),-\sin(\Omega z)]}$. The
dashed hexagon represents the first Brillouin zone. Figure 3: (a) Schematic
representation of composite helical waveguide array with two domain walls
which are indicated by the vertical dashed green lines. Only waveguides of one
type (blue or red) fall onto domain wall. The array is periodic in the $y$
direction. Both detuning and direction of waveguide rotation are opposite in A
and B arrays forming the domain wall. (b) A-B-A array configuration with two
domain walls. (c) Quasi-propagation constant vs Bloch momentum $k_{y}$ for
B-A-B configuration in (a). Two longitudinal Brillouin zones are shown to
stress vertical periodicity of the spectrum. Red and blue dots correspond to
the edge states appearing on red and blue domain walls, respectively. Gray
dots correspond to bulk modes. (d) Field modulus distribution ${|\psi|}$ of
the edge state (three $y$-periods are shown) with ${k_{y}/{\rm K}=0.3}$
residing on the red domain wall in (c) at different propagation distances
within one period $Z$. (e) Edge states with ${k_{y}/{\rm K}=0.35}$ from blue
domain wall at different distances. In all cases $Z=6$, $r_{0}=0.3$ and
$\delta=0.07$.

###  2.2 Topological characterization of the system

To characterize topological properties of helical waveguide array with
detuning between two sublattices, we utilize tight-binding approximation and
the approach summarized in [52]. The evolution of the excitations in our
system over one $Z$-period (in reciprocal space) can be described by the
Floquet evolution operator $U({\bm{k}},Z)$ that allows to formally introduce
also the effective Hamiltonian $H_{\rm eff}$:

$\displaystyle U({\bm{k}},Z)$
$\displaystyle=\mathcal{Z}\exp\left[-i\int_{0}^{Z}H({\bm{k}},z)dz\right]$
$\displaystyle=\exp\left[-iH_{\rm eff}({\bm{k}})Z\right],$ (4)

where $\cal Z$ is the time-ordering operator, and $H({\bm{k}},z)$ is the
“instantaneous” Hamiltonian of the system

$H({\bm{k}},z)=t\begin{bmatrix}\wp&P\\\ P^{\dagger}&-\wp\end{bmatrix}.$ (5)

Here

$P=\sum_{i}\exp[-i({\bm{A}}+{\bm{k}})\cdot{\bm{e}}_{i}]$

and

${\bm{A}}(z)=r\Omega[-\cos(\Omega z),\sin(\Omega z)]$

with ${i=1,2,3}$, ${{\bm{e}}_{1}=[1,0]a}$,
${{\bm{e}}_{2}=[-1/2,-\sqrt{3}/2]a}$, ${{\bm{e}}_{3}=[-1/2,\sqrt{3}/2]a}$ are
the characteristic vectors of the honeycomb lattice, and ${\bm{A}}(z)$ is the
gauge field arising due to helicity of waveguides, $\wp$ is the on-site
“energy” detuning [53, 54] of two sublattices in tight-binding model, which is
different from $\delta$ in Eq. (2) but proportional to it, $t$ is the hopping
strength. Berry curvature [42] of the band with index $n$ can be introduced as

$\displaystyle{\mathcal{B}}_{n}(\bm{k})=i\sum_{n^{\prime}\neq n}$
$\displaystyle\left(\frac{\langle u_{n}|\partial_{k_{x}}H_{\rm
eff}|u_{n^{\prime}}\rangle\langle u_{n^{\prime}}|\partial_{k_{y}}H_{\rm
eff}|u_{n}\rangle}{(b_{n}-b_{n^{\prime}})^{2}}-\right.$
$\displaystyle\left.\frac{\langle u_{n}|\partial_{k_{y}}H_{\rm
eff}|u_{n^{\prime}}\rangle\langle u_{n^{\prime}}|\partial_{k_{x}}H_{\rm
eff}|u_{n}\rangle}{(b_{n}-b_{n^{\prime}})^{2}}\right),$ (6)

where $|u_{n}\rangle$ and $b$ are the eigenstates and eigenvalues of $H_{\rm
eff}$, and $\langle\cdot|\cdot\rangle$ is the inner product. Typical
distribution $\mathcal{B}_{n}(\bm{k})$ in the bottom band ${(n=2)}$ is
presented in Fig. 2(a). It is calculated for a set of parameters ${a=1.6}$,
${Z=6}$, ${r=0.3}$, ${\wp=0.1}$, and ${t=0.18}$ at which Dirac cones emerge in
$\bf K$ points, while gap is opened between ${\bf K}^{\prime}$ valleys, where
Berry curvature is positive. If one simultaneously changes the sign of
detuning $\wp$ and waveguide rotation direction (helicity), the gap again
appears between ${\bf K}^{\prime}$ valleys, but Berry curvature around them
becomes negative in the bottom band, as shown in Fig. 2(b). Topological index
characterizing this system is given by the valley Chern number:

${\mathcal{C}}_{v,n}=\frac{1}{2\pi}\int_{v}{\mathcal{B}}_{v,n}(\bm{k})d\bm{k},$
(7)

where $v$ indicates that the integration is carried over close proximity of
the valleys. Thus, valley Chern number of each valley in Fig. 2(a) is $1/2$
while in Fig. 2(b) it is given by $-1/2$. Therefore, according to the bulk-
edge correspondence principle [7, 8], if the interface is created between two
such arrays in real space, the difference ${1/2-(-1/2)=1}$ of valley Chern
numbers predicts the appearance of the topological edge states connecting the
upper and bottom bands around the valley in the band structure. Importantly,
this situation sharply contrasts with usual valley Hall system, where for
nonzero detuning $\wp$ the gap simultaneously opens between all valleys in
such a way that Berry curvature has opposite sign around $\bf K$ and ${\bf
K}^{\prime}$ valleys, so that the integral (7) over the entire Brillouin zone
yields zero $\mathcal{C}_{n}$. Instead, in our case such an integral would be
nonzero and opposite in sign for both first and second bands.

Figure 4: Zigzag-shaped domain walls with sharp corners that are adopted for
demonstration of the topological protection of the edge states from the red
branch (a) and from the blue branch (b) of Fig. 3(c). The arrows indicate the
direction of motion of the edge states with initial broad envelope at
different stages of propagation. Corresponding field modulus distributions at
different distances are shown too. The two edge states correspond to
${k_{y}/{\rm K}=0.3}$ (a) and ${k_{y}/{\rm K}=0.35}$ (b) respectively.

##  3 Results and discussion

###  3.1 Domain walls and edge states

To illustrate the possibility of formation of topological edge states in this
system with unpaired Dirac points, we constructed the domain wall between
helical arrays of types A and B with opposite detuning of two constituent
sublattices and with opposite waveguide rotation directions, as illustrated in
schematic Figs. 3(a) and 3(b). As discussed previously, the bands of such
arrays with the same index $n$ are characterized by the opposite valley Chern
numbers, while Dirac cones in both A and B arrays are located in K points. The
arrays can be arranged either into B-A-B [Fig. 3(a)] or into A-B-A [Fig. 3(b)]
configurations, with waveguides of only one type (red or blue) falling onto
domain walls, the latter are highlighted by the green dashed lines. Notice
that the waveguides on the domain wall have alternative helix directions since
they belong to different arrays A and B. According to the bulk-edge
correspondence principle the domain walls between such arrays should support
edge states of topological origin. In Fig. 3(c) we show the dependence of the
quasi-propagation constant of all modes in B-A-B configuration on Bloch
momentum $k_{y}$ (along the edge) within first transverse Brillouin zone of
width ${{\rm K}=2\pi/\sqrt{3}a}$. Since the array is periodic in $z$, the
spectrum is also periodic along the vertical $b$ axis with the period
${\Omega=2\pi/}Z$ (defined by the waveguide rotation frequency). In the band
structure in Fig. 3(c) gray dots correspond to delocalized bulk modes, while
red and blue dots correspond to the edge states localized on red and blue
domain walls, respectively. Upon calculation of this spectrum we “glued” B
arrays far away from the domain walls and wide A layer between them by
assuming periodic boundary conditions along the $x$-axis. The remarkable
feature of the “projected" spectrum in Fig. 3(c) is its asymmetry with
coexistence of the Dirac cone at ${k_{y}=-\textrm{K}/3}$ (indicated by the
green arrow) and of a gap at ${k_{y}=+\textrm{K}/3}$, around which
unidirectional edge states form that are located at blue and red domain walls,
in accordance with line colors. The appearance of such edge states is a
remarkable fact, taking into account the absence of the complete gap in the
spectrum. We show selected profiles of the edge states from red and blue
domain walls in Figs. 3(d) and 3(e), respectively. Since our Floquet system is
$Z$-periodic, such states show exactly periodic evolution on one longitudinal
period, with substantial variations of the field modulus distribution in the
internal $z$-points. Notice that localization of such edge states is strongest
in the center of the “local” gap, while when their quasi-propagation constant
approaches the band, such states gradually become extended.

Figure 5: First-order derivative $b^{\prime}$ of the quasi-propagation
constant of the valley Hall edge states at $Z=6$ and $r_{0}=0.3$. Figure 6:
(a) A zigzag domain wall with a defect in the form of missing waveguide
indicated by the green circle. (b-e) Spatial field modulus distributions in
the edge state at different propagation distances. Cyan dashed circle
highlights theoretical radius of the conical diffraction pattern. (f) Total
spatial spectrum of the edge state at ${z=300}$. (g) Part of the spectrum at
${z=300}$ where only contribution from regions around K points is left (see
white dashed circles), while spectral intensity around ${\bf K}^{\prime}$
points is set to zero. (h) Field modulus distribution in the form of conical
wave produced by spectrum in (g). (i) Part of the spectrum at ${z=300}$ where
only contribution from regions around ${\bf K}^{\prime}$ points is left (see
dashed circles), while spectral intensity around ${\bf K}$ points is set to
zero. (j) Field modulus distribution of the edge state produced by spectrum in
(i). White dashed line represent the zigzag domain wall. The parameters are
the same as in Fig. 4.

The edge states reported here require for their existence the interface
between two lattices with broken inversion symmetry and on this reason they do
belong to the class of valley Hall states. However, our structure also
combines in a unique fashion the properties typical for valley Hall and
Floquet systems. Because time-reversal symmetry in our system is broken due to
helicity of waveguides, edge states become truly unidirectional and this
allows to avoid the problem typical for usual valley Hall systems that support
a pair of counterpropagating edge states that can be coupled by sufficiently
narrow edge defects. Our system is free from this limitation and this is
reflected in completely different scenario of interaction of edge states with
narrow defects that is now accompanied not by reflection of the edge state,
but by emission of fraction of power in the form of conical wave (please see
Fig. 6 below) and passage of considerable fraction of power through the
defect. In this sense, our system offers stronger topological protection than
conventional valley Hall systems.

Figure 7: Dynamics of excitation of the valley Hall edge state with a titled
elliptical Gaussian beam. (a) shows input field modulus distribution at
${z=0}$ and field modulus distributions at progressively increasing
propagation distances, indicating that the excited state bypasses zigzag bend
of the domain wall, indicated with white dashed line. (b) shows spectra in
momentum space corresponding to spatial distributions in (a). Circles in (b)
highlight the valleys in six corners of the first Brillouin zone. Parameters
of the array are the same as in Fig. 4. The input Gaussian beam has width
${w_{x}=10}$, ${w_{y}=20\sqrt{5}}$ and initial momentum ${k_{x}=0}$,
${k_{y}=-4\pi/3\sqrt{3}a}$.

###  3.2 Topological protection of the edge states

To confirm the topological nature of the Floquet edge states suggested by
associated nonzero valley Chern numbers discussed in Section 2.2, we designed
two types of the domain walls with a zigzag shape that possess two sharp
corners (the angle is $60^{\circ}$). The domain wall in Fig. 4(a) is similar
to red domain wall in Fig. 3(a), as highlighted by the corresponding line,
while the domain wall in Fig. 4(b) is similar to the blue domain wall in Fig.
3(a), and it highlighted by the blue line. These domain walls support edge
states corresponding to red and blue branches of Fig. 3(c) that have opposite
group velocities ${v_{g}=-b^{\prime}=-db/dk_{y}}$. Since ${b^{\prime}>0}$ for
the red edge state (see red curve in Fig. 5), it moves in the negative
$y$-direction, while its blue counterpart with ${b^{\prime}<0}$ moves in the
positive direction of $y$-axis (see blue curve in Fig. 5). The propagation
dynamics of corresponding edge states with initial broad envelopes is
illustrated in Figs. 4(a) and 4(b), the direction of motion of corresponding
states is highlighted by the black arrows on schematics with zigzag domain
walls. Field modulus distributions at different distances clearly show that
both red and blue edge states can circumvent sharp corners during propagation
without any noticeable backscattering, that illustrates their topological
protection, at least for such type of edge deformations. Notice that small
tails visible in field modulus distributions at ${z=510}$ are due to
dispersion of the wavepacket, as they move in the same direction as wavepacket
and with the same velocity. This is intriguing result, taking into account the
absence of complete gap in the spectrum of the system. We would like to note
that in dynamics illustrated in Fig. 4 the edge state shows neither inter-
valley scattering nor it excites bulk states because this state is taken quite
close to the $\textbf{K}^{\prime}$ point in momentum space, and its quasi-
propagation constant is located practically in the center of the gap. However,
if the input edge state is taken further from the $\textbf{K}^{\prime}$ point,
for example with ${k_{y}=0.2\rm K}$, such that its quasi-propagation constant
can overlap with quasi-propagation constants of the bulk modes, a fraction of
power can be radiated into the bulk when the state bypasses the sharp corner
(see Supplemental Information for corresponding results).

It is a well-established fact that in valley Hall systems the inter-valley
scattering can be induced by strongly localized defects, while broad slowly
changing defects cannot induce such a scattering. To test the behaviour of the
edge states in our system with respect to strongly localized perturbations, we
removed one waveguide located on the domain wall, as illustrated by the green
circle in Fig. 6. The incident edge state at ${z=0}$ in Fig. 6(b) is the same
as in Fig. 4(a) and it encounters the defect around ${z=110}$ [Fig. 6(c)]
while propagating along the domain wall. The interaction between the defect
and the edge state shows that the edge state cannot bypass the defect without
radiation [see Figs. 6(d) and 6(e)]. Remarkably, because edge states in our
system are unidirectional, no backward reflection at the defect occurs.
Instead, inter-valley scattering leads to the excitation of waves in close
proximity of the Dirac cone at the $\bf K$ point, and further propagation of
such waves is accompanied by their conical diffraction resulting in gradually
expanding ring visible in field modulus distributions at ${z=230}$ and
${z=300}$. The dashed cyan circle in Figs. 6(d) and 6(e) [and also in Fig.
6(h)] illustrates theoretically evaluated radius of the conical diffraction
pattern ${\sim|b^{\prime}_{\bf K}(z-110)|}$ with ${b^{\prime}_{\bf K}\sim
0.45}$ being the first-order derivative of the quasi-propagation constant
around Dirac cone at the $\bf K$ point. Remarkably, only a fraction of input
power is emitted in the form of conical wave after interaction width defect,
while the rest of the power is organized into edge state that bypassed the
defect and keeps moving along the domain wall - a consequence of the
unidirectional nature of the edge states in our system. To illustrate this, we
take the exemplary field distribution at $z=300$ and display its total spatial
spectrum in Fig. 6(f). Clearly, there is light around both ${\bf K}$ and ${\bf
K}^{\prime}$ points. If we isolate contributions from ${\bf K}$ points by
applying the filter of the form

$F_{\bf K}({\bm{k}})=\left\\{\begin{split}&1,\leavevmode\nobreak\
|{\bm{k}}-{\bf K}|\leq r_{k},\\\ &0,\leavevmode\nobreak\ {\rm
other\leavevmode\nobreak\ places},\end{split}\right.$ (8)

to total spectrum, where $r_{k}=4\sqrt{3}\pi/27a$ is the radius of the
circular filter, we will obtain the spectrum depicted in Fig. 6(g) (white
dashed circles show the regions outside which the spectral intensity was set
to zero), that corresponds to field distribution shown in Fig. 6(h) in the
form of clear conical diffraction pattern. In contrast, if the filter $F_{\bf
K^{\prime}}({\bm{k}})$ is applied that isolates contributions from ${\bf
K}^{\prime}$ points, see the resulting spectrum in Fig. 6(i), in spatial
domain one obtains spatial distribution depicted in Fig. 6(j) that clearly
shows that edge state partially bypassed the defect and keeps propagating
along the domain wall after emission of a fraction of its initial power in the
form of conical wave. Because the edge states reported here bypass strongly
localized defects with emission of fraction of power in the form of conical
waves, but at the same time they can still clearly bypass sharp corners
practically without radiation, akin to edge states in valley Hall systems
reported previously [38, 39], we attribute them to this last class of edge
states.

###  3.3 Excitation of the valley Hall edge states

We would like to note that the valley Hall edge states reported here can be
efficiently excited by a Gaussian beam with proper initial tilt (i.e. we
multiply the input field distribution with $e^{ik_{x}x+ik_{y}y}$ term). In
Fig. 7, we display an example of such excitation dynamics. The elliptical
Gaussian beam is launched at the left branch of the zigzag domain wall [see
field modulus distribution in Fig. 7(a) and corresponding spatial spectrum in
Fig. 7(b)]. This input beam quickly reshapes and after emission of some
radiation efficiently excites moving valley Hall edge state with Bloch
momentum $k_{y}$ corresponding to the initial momentum of the Gaussian beam.
Excited moving edge state bypasses zigzag bend of the domain wall without
reflections. The spectra at different distances presented in Fig. 7(b)
demonstrate that the beam is always concentrated around the ${\bf K}^{\prime}$
points. The efficiency of this excitation process strongly depends on the
momentum of initial Gaussian beam. For example, if its spectrum of the input
beam is shifted away from ${\bf K}^{\prime}$ valleys, one observes
considerable diffractive broadening in real space without obvious excitation
of localized edge states. For instance, if the input beam excites the ${\bf
K}$ valley for a given type of the domain wall, one observes conical
diffraction in spatial domain [2].

##  4 Conclusion and outlook

Summarizing, we have shown that helical waveguide arrays with detuning between
two sublattices, where time-reversal and inversion symmetries are
simultaneously broken, can feature unpaired Dirac points in their linear
spectra. The formation of the unpaired Dirac points in $\bf K$ or ${\bf
K}^{\prime}$ points of the Brillouin zone can be controlled by the sign of
detuning and waveguide helicity. Even though the location of Dirac cones does
not change upon simultaneous change of the sign of detuning and waveguide
helicity, the sign of Berry curvature changes that allows to construct the
interface between two arrays with opposite valley Chern numbers, hosting
unidirectional topological edge states. These states are topologically
protected despite the absence of the complete gap in the spectrum of the
system. These results suggest a platform and pave the way to exploration of
various nonlinear phenomena in Floquet insulators with unpaired Dirac points.
For instance, we would like to point out that in the past decade a variety of
hybrid topological edge solitons have been reported in topological systems
based on helical waveguide arrays [18, 17, 55, 56, 57, 58, 59, 60], in valley
Hall arrays with straight waveguides [61, 37, 39, 38, 41, 62, 40] and other
systems [14, 63, 64, 65, 66], but all these structures are characterized by
the presence of complete topological gaps and, as a rule, are developed on
lattices possessing paired Dirac points. Similarly, generation of higher
harmonics and topological lasers may acquire very unusual features in systems
with unpaired Dirac points. Last but not least, the unpaired Dirac cones
ensure that the inter-valley scattering will not happen in the system
developed in the work which may bring a different view on the conclusions
reported very recently [67, 68]. We believe that the system with unpaired
Dirac cones may provide a novel platform for investigating higher-order
topological states [69, 70, 71] and may help to observe the unpaired Weyl
points [72].

Funding This work was supported by the Natural Science Basic Research Program
of Shaanxi Province (2024JC-JCQN-06), the National Natural Science Foundation
of China (12074308, 12304370), and the Fundamental Research Funds for the
Central Universities (sxzy012024146). The work of Y.V.K. was supported by the
research project FFUU-2024-0003 of the Institute of Spectroscopy of the
Russian Academy of Sciences.

Acknowledgments The authors appreciate the three anonymous reviewers for their
insightful and helpful comments which improve the work greatly.

Disclosures The authors declare no conflicts of interest.

Data availability Data underlying the results presented in this paper are not
publicly available at this time but may be obtained from the authors upon
reasonable request.

## References

  * [1] D. Leykam and A. S. Desyatnikov, “Conical intersections for light and matter waves,” Adv. Phys. X 1, 101–113 (2016).
  * [2] O. Peleg, G. Bartal, B. Freedman, _et al._ , “Conical diffraction and gap solitons in honeycomb photonic lattices,” Phys. Rev. Lett. 98, 103901 (2007).
  * [3] D. Song, V. Paltoglou, S. Liu, _et al._ , “Unveiling pseudospin and angular momentum in photonic graphene.” Nat. Commun. 6, 6272 (2015).
  * [4] Z. Y. Zhang, R. Wang, Y. Q. Zhang, _et al._ , “Observation of edge solitons in photonic graphene,” Nat. Commun. 11, 1902 (2020).
  * [5] W. Yan, H. Zhong, D. Song, _et al._ , “Flatband line states in photonic super-honeycomb lattices,” Adv. Opt. Mater. 8, 1902174 (2020).
  * [6] A. H. Castro Neto, F. Guinea, N. M. R. Peres, _et al._ , “The electronic properties of graphene,” Rev. Mod. Phys. 81, 109–162 (2009).
  * [7] L. Lu, J. D. Joannopoulos, and M. Soljačić, “Topological photonics,” Nat. Photon. 8, 821–829 (2014).
  * [8] T. Ozawa, H. M. Price, A. Amo, _et al._ , “Topological photonics,” Rev. Mod. Phys. 91, 015006 (2019).
  * [9] D. Smirnova, D. Leykam, Y. Chong, and Y. Kivshar, “Nonlinear topological photonics,” Appl. Phys. Rev. 7, 021306 (2020).
  * [10] Q. Yan, X. Hu, Y. Fu, _et al._ , “Quantum topological photonics,” Adv. Opt. Mater. 9, 2001739 (2021).
  * [11] F. D. M. Haldane, “Model for a quantum Hall effect without Landau levels: Condensed-matter realization of the “parity anomaly”,” Phys. Rev. Lett. 61, 2015–2018 (1988).
  * [12] Z. Wang, Y. Chong, J. D. Joannopoulos, and M. Soljačić, “Observation of unidirectional backscattering-immune topological electromagnetic states,” Nature 461, 772–775 (2009).
  * [13] A. V. Nalitov, D. D. Solnyshkov, and G. Malpuech, “Polariton $\mathbb{Z}$ topological insulator,” Phys. Rev. Lett. 114, 116401 (2015).
  * [14] Y. V. Kartashov and D. V. Skryabin, “Modulational instability and solitary waves in polariton topological insulators,” Optica 3, 1228–1236 (2016).
  * [15] S. Klembt, T. H. Harder, O. A. Egorov, _et al._ , “Exciton-polariton topological insulator,” Nature 562, 552–556 (2018).
  * [16] M. C. Rechtsman, J. M. Zeuner, Y. Plotnik, _et al._ , “Photonic Floquet topological insulators,” Nature 496, 196–200 (2013).
  * [17] Y. Lumer, Y. Plotnik, M. C. Rechtsman, and M. Segev, “Self-localized states in photonic topological insulators,” Phys. Rev. Lett. 111, 243905 (2013).
  * [18] M. J. Ablowitz and J. T. Cole, “Tight-binding methods for general longitudinally driven photonic lattices: Edge states and solitons,” Phys. Rev. A 96, 043868 (2017).
  * [19] S. Stützer, Y. Plotnik, Y. Lumer, _et al._ , “Photonic topological Anderson insulators,” Nature 560, 461–465 (2018).
  * [20] E. Lustig, S. Weimann, Y. Plotnik, _et al._ , “Photonic topological insulator in synthetic dimensions,” Nature 567, 356–360 (2019).
  * [21] M. S. Rudner, N. H. Lindner, E. Berg, and M. Levin, “Anomalous edge states and the bulk-edge correspondence for periodically driven two-dimensional systems,” Phys. Rev. X 3, 031005 (2013).
  * [22] M. S. Rudner and N. H. Lindner, “Band structure engineering and non-equilibrium dynamics in Floquet topological insulators,” Nat. Rev. Phys. 2, 229–244 (2020).
  * [23] S. Yin, E. Galiffi, and A. Alù, “Floquet metamaterials,” eLight 2, 8 (2022).
  * [24] W. Song, H. Li, S. Gao, _et al._ , “Subwavelength self-imaging in cascaded waveguide arrays,” Adv. Photon. 2, 036001 (2020).
  * [25] B.-C. Xu, B.-Y. Xie, L.-H. Xu, _et al._ , “Topological Landau–Zener nanophotonic circuits,” Adv. Photon. 5, 036005 (2023).
  * [26] Z. Zhang, Y. Li, X. Sun, and X. Shu, “Visual observation of photonic Floquet-Bloch oscillations,” Light Sci. Appl. 13, 99 (2024).
  * [27] D. Leykam, M. C. Rechtsman, and Y. D. Chong, “Anomalous topological phases and unpaired Dirac cones in photonic Floquet topological insulators,” Phys. Rev. Lett. 117, 013902 (2016).
  * [28] J. Noh, S. Huang, D. Leykam, _et al._ , “Experimental observation of optical Weyl points and Fermi arc-like surface states,” Nat. Phys. 13, 611–617 (2017).
  * [29] G. Q. Liang and Y. D. Chong, “Optical resonator analog of a two-dimensional topological insulator,” Phys. Rev. Lett. 110, 203904 (2013).
  * [30] M. Pasek and Y. D. Chong, “Network models of photonic Floquet topological insulators,” Phys. Rev. B 89, 075113 (2014).
  * [31] Y. D. Chong and M. C. Rechtsman, “Tachyonic dispersion in coherent networks,” J. Opt. 18, 014001 (2016).
  * [32] G.-G. Liu, P. Zhou, Y. Yang, _et al._ , “Observation of an unpaired photonic Dirac point,” Nat. Commun. 11, 1873 (2020).
  * [33] H. Wang, B. Xie, and W. Ren, “Coexistence of chiral and antichiral edge states in photonic crystals,” Laser Photon. Rev. 18, 2300764 (2024).
  * [34] Y. Wang, H.-X. Wang, L. Liang, _et al._ , “Hybrid topological photonic crystals,” Nat. Commun. 14, 4457 (2023).
  * [35] J. Noh, S. Huang, K. P. Chen, and M. C. Rechtsman, “Observation of photonic topological valley Hall edge states,” Phys. Rev. Lett. 120, 063902 (2018).
  * [36] X. Wu, Y. Meng, J. Tian, _et al._ , “Direct observation of valley-polarized topological edge states in designer surface plasmon crystals,” Nat. Commun. 8, 1304 (2017).
  * [37] H. Zhong, S. Xia, Y. Zhang, _et al._ , “Nonlinear topological valley Hall edge states arising from type-II Dirac cones,” Adv. Photon. 3, 056001 (2021).
  * [38] Q. Tang, B. Ren, V. O. Kompanets, _et al._ , “Valley Hall edge solitons in a photonic graphene,” Opt. Express 29, 39755–39765 (2021).
  * [39] B. Ren, H. Wang, V. O. Kompanets, _et al._ , “Dark topological valley Hall edge solitons,” Nanophoton. 10, 3559–3566 (2021).
  * [40] Y. Tian, Y. Zhang, Y. Li, and M. R. Belić, “Vector valley Hall edge solitons in the photonic lattice with type-II Dirac cones,” Front. Phys. (Beijing) 17, 53503 (2022).
  * [41] Q. Tang, Y. Zhang, Y. V. Kartashov, _et al._ , “Vector valley Hall edge solitons in superhoneycomb lattices,” Chaos Solitons Fract. 161, 112364 (2022).
  * [42] D. Xiao, M.-C. Chang, and Q. Niu, “Berry phase effects on electronic properties,” Rev. Mod. Phys. 82, 1959–2007 (2010).
  * [43] H. Zhong, Y. Q. Zhang, Y. Zhu, _et al._ , “Transport properties in the photonic super-honeycomb lattice – a hybrid fermionic and bosonic system,” Ann. Phys. (Berlin) 529, 1600258 (2017).
  * [44] M. S. Kirsch, Y. Zhang, M. Kremer, _et al._ , “Nonlinear second-order photonic topological insulators,” Nat. Phys. 17, 995–1000 (2021).
  * [45] B. Ren, A. A. Arkhipova, Y. Zhang, _et al._ , “Observation of nonlinear disclination states,” Light Sci. Appl. 12, 194 (2023).
  * [46] A. A. Arkhipova, Y. Zhang, Y. V. Kartashov, _et al._ , “Observation of $\pi$ solitons in oscillating waveguide arrays,” Sci. Bull. 68, 2017–2024 (2023).
  * [47] D. Tan, Z. Wang, B. Xu, and J. Qiu, “Photonic circuits written by femtosecond laser in glass: improved fabrication and recent progress in photonic devices,” Adv. Photon. 3, 024002 (2021).
  * [48] L. Li, W. Kong, and F. Chen, “Femtosecond laser-inscribed optical waveguides in dielectric crystals: a concise review and recent advances,” Adv. Photon. 4, 024002 (2022).
  * [49] B. Ren, Y. V. Kartashov, H. Wang, _et al._ , “Floquet topological insulators with hybrid edges,” Chaos Solitons Fractals 166, 113010 (2023).
  * [50] S. Shen, Y. V. Kartashov, Y. Li, and Y. Zhang, “Floquet edge solitons in modulated trimer waveguide arrays,” Phys. Rev. Appl. 20, 014012 (2023).
  * [51] S. Raghu and F. D. M. Haldane, “Analogs of quantum-Hall-effect edge states in photonic crystals,” Phys. Rev. A 78, 033834 (2008).
  * [52] H. Zhong, D. Mihalache, S. Shen, and Y. Zhang, “The band structure of helical waveguide arrays in topological photonics: A tutorial,” Rom. Rep. Phys. 76, 903 (2024).
  * [53] D. Xiao, W. Yao, and Q. Niu, “Valley-contrasting physics in graphene: Magnetic moment and topological transport,” Phys. Rev. Lett. 99, 236809 (2007).
  * [54] H. Xue, Y. Yang, and B. Zhang, “Topological valley photonics: Physics and device applications,” Adv. Photon. Research 2, 2100013 (2021).
  * [55] S. K. Ivanov, Y. V. Kartashov, A. Szameit, _et al._ , “Vector topological edge solitons in Floquet insulators,” ACS Photon. 7, 735–745 (2020).
  * [56] S. K. Ivanov, Y. V. Kartashov, L. J. Maczewsky, _et al._ , “Edge solitons in Lieb topological Floquet insulator,” Opt. Lett. 45, 1459–1462 (2020).
  * [57] S. K. Ivanov, Y. V. Kartashov, L. J. Maczewsky, _et al._ , “Bragg solitons in topological Floquet insulators,” Opt. Lett. 45, 2271–2274 (2020).
  * [58] S. K. Ivanov, Y. V. Kartashov, M. Heinrich, _et al._ , “Topological dipole Floquet solitons,” Phys. Rev. A 103, 053507 (2021).
  * [59] S. K. Ivanov, Y. V. Kartashov, A. Szameit, _et al._ , “Floquet edge multicolor solitons,” Laser Photon. Rev. 16, 2100398 (2022).
  * [60] S. Mukherjee and M. C. Rechtsman, “Observation of Floquet solitons in a topological bandgap,” Science 368, 856–859 (2020).
  * [61] D. A. Smirnova, L. A. Smirnov, D. Leykam, and Y. S. Kivshar, “Topological edge states and gap solitons in the nonlinear Dirac model,” Laser Photon. Rev. 13, 1900223 (2019).
  * [62] Q. Tang, B. Ren, M. R. Belić, _et al._ , “Valley Hall edge solitons in the kagome photonic lattice,” Rom. Rep. Phys. 74, 504 (2022).
  * [63] D. R. Gulevich, D. Yudin, D. V. Skryabin, _et al._ , “Exploring nonlinear topological states of matter with exciton-polaritons: Edge solitons in kagome lattice,” Sci. Rep. 7, 1780 (2017).
  * [64] C. Li, F. Ye, X. Chen, _et al._ , “Lieb polariton topological insulators,” Phys. Rev. B 97, 081103 (2018).
  * [65] Y. Q. Zhang, Y. V. Kartashov, and A. Ferrando, “Interface states in polariton topological insulators,” Phys. Rev. A 99, 053836 (2019).
  * [66] W. Zhang, X. Chen, Y. V. Kartashov, _et al._ , “Coupling of edge states and topological Bragg solitons,” Phys. Rev. Lett. 123, 254103 (2019).
  * [67] C. A. Rosiek, G. Arregui, A. Vladimirova, _et al._ , “Observation of strong backscattering in valley-Hall photonic topological interface modes,” Nat. Photon. 17, 386–392 (2023).
  * [68] M. C. Rechtsman, “Reciprocal topological photonic crystals allow backscattering,” Nat. Photon. 17, 383–384 (2023).
  * [69] B. Xie, H.-X. Wang, X. Zhang, _et al._ , “Higher-order band topology,” Nat. Rev. Phys. 3, 520–532 (2021).
  * [70] Z.-K. Lin, Q. Wang, Y. Liu, _et al._ , “Topological phenomena at defects in acoustic, photonic and solid-state lattices,” Nat. Rev. Phys. 5, 483–495 (2023).
  * [71] Y. Zhang, D. Bongiovanni, Z. Wang, _et al._ , “Realization of photonic $p$-orbital higher-order topological insulators,” eLight 3, 5 (2023).
  * [72] C. Guo, V. S. Asadchy, B. Zhao, and S. Fan, “Light control with Weyl semimetals,” eLight 3, 2 (2023).