# A sensor fusion approach for improving implementation speed and accuracy of
RTAB-Map algorithm based indoor 3D mapping
††thanks: Hoang-Anh Phan, Phuc Vinh Nguyen, Thu Hang Thi Khuat, Hieu Dang Van,
Dong Huu Quoc Tran, Tung Thanh Bui, Van Nguyen Thi Thanh, and Trinh Chu Duc
are with the Faculty of Electronics and Telecommunications, VNU University of
Engineering and Technology, Hanoi, Vietnam<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>††thanks: Bao Lam Dang is with Hanoi University of Science
and Technology, Hanoi, Vietnam<EMAIL_ADDRESS>
Hoang-Anh Phan, Phuc Vinh Nguyen, Thu Hang Thi Khuat, Hieu Dang Van,
Dong Huu Quoc Tran, Bao Lam Dang, Tung Thanh Bui, Van Nguyen Thi Thanh and
Trinh Chu Duc

###### Abstract

In recent years, 3D mapping for indoor environments has undergone considerable
research and improvement because of its effective applications in various
fields, including robotics, autonomous navigation, and virtual reality.
Building an accurate 3D map for indoor environment is challenging due to the
complex nature of the indoor space, the problem of real-time embedding and
positioning errors of the robot system. This study proposes a method to
improve the accuracy, speed, and quality of 3D indoor mapping by fusing data
from the Inertial Measurement System (IMU) of the Intel Realsense D435i
camera, the Ultrasonic-based Indoor Positioning System (IPS), and the encoder
of the robot’s wheel using the extended Kalman filter (EKF) algorithm. The
merged data is processed using a Real-time Image Based Mapping algorithm
(RTAB-Map), with the processing frequency updated in synch with the position
frequency of the IPS device. The results suggest that fusing IMU and IPS data
significantly improves the accuracy, mapping time, and quality of 3D maps. Our
study highlights the proposed method’s potential to improve indoor mapping in
various fields, indicating that the fusion of multiple data sources can be a
valuable tool in creating high-quality 3D indoor maps.

###### Index Terms:

3D mapping, sensor fusion, Visual SLAM

##  I Introduction

Autonomous robots are becoming increasingly used for a wide range of indoor
applications, including autonomous navigation, inspection, and surveillance.
One of the critical requirements for these applications is fast and accurate
3D mapping of the indoor environment. One of the methods to build 3D maps is
Visual Simultaneous Localization And Mapping (Visual SLAM), which refers to
SLAM [1] systems that use the camera as the primary input sensor to receive
visual information only [2]. Several methods can be mentioned as Mono-SLAM -
Visual Simultaneous Localization And Mapping technology for real-time
applications developed [3], PTAM (Parallel Tracking and Mapping) [4], ORB-SLAM
(Oriented FAST and Rotated BRIEF - SLAM) [5] or ORB-SLAM2 [6]. This feature-
based SLAM technology is the foundation of modern SLAMs for real-time
applications. Visual SLAM still has difficulties in experimenting with indoor
environments such as lighting problems affecting camera quality, and bumpy
terrain affecting robot accuracy [7]. One of the main challenges in Visual
SLAM is the accumulation of errors over time [8], causing drift in the
estimated camera pose and resulting in inaccuracies in the generated map.
Drift can be caused by various factors and requires correction methods such as
loop closure detection and sensor fusion techniques to maintain a consistent
reference frame.

To address the challenge of drift in Visual SLAM, researchers have proposed
several methods, including Loop closure detection [9], Sensor fusion [10],
Keyframe-based SLAM [11, 12], Robust feature detection, and Matching [13]. In
recent years, Sensor fusion has become a popular method to improve the
accuracy of Visual SLAM. Some fusion techniques are the extended Kalman filter
(EKF) [14], the unscented Kalman filter (UKF) [15], and Particle filters [16],
to estimate the position and orientation of the camera by fusing data from
multiple sensors [17]. While these methods are widely used in Visual SLAM,
they present certain limitations such as reliance on accurate sensor data,
linearization of the system in EKF, high computational complexity in particle
filters, and increased computation in UKF.

This study presents a method that utilizes a fusion of data acquired from
multiple sensors to serve as an alternative measurement source to the Visual
Odometry (VO) [18] of the RTAB-Map method. Specifically, we combine data from
an ultrasonic-based Indoor Positioning System (IPS) with wheel encoders, and
an Inertial Measurement System (IMU) obtained from a D435i camera by
implementing the EKF algorithm. The proposed method aims to address two
critical issues in the current system: the drift error in camera pose
estimation and the noise from separate sensors, thereby enhancing the
accuracy, speed, and quality of indoor 3D mapping. The external positioning
source has high accuracy due to ultrasonic technology, making it a valuable
tool for improving indoor mapping. Additionally, the high sampling rate of
IPS, in contrast to the low sampling rate of VO, which has to be estimated
from keyframe images, results in improved 3D mapping quality and time. The
experimental results indicate that the proposed method effectively mitigates
errors in position and trajectory during the robot’s mapping process at high
speeds, thereby improving the accuracy of the resulting map. Moreover, the
proposed method achieves a faster map update frequency and a more optimal time
for constructing the 3D map, resulting in a more informative 3D map compared
to the base method - is the RTAB-Map method where the odometry known as VO is
calculated from the image information and the IMU of the D435i camera device.

##  II Propose method

(a)

(b)

Figure 1: The proposed modified RTAB-map algorithm. (a) Algorithm flowchart
(b) A robot platform for implementing and examining the proposed algorithm.

In this study, a differential two-wheel indoor mobile robot platform was
utilized for the experimental setup (Fig. 1b), which ensures stability during
robot movement. The system was deployed on a Jetson AGX embedded computer,
with the input data obtained from the depth camera D435i for Visual-SLAM. The
Ultrasonic-based IPS employed for the proposed method consisted of five
beacons, including four static beacons fixed to the walls and one dynamic
beacon mounted on the robot to connect and transfer position data. Fig. 1a
illustrates the flowchart of the proposed RTAB-Map algorithm, which includes
the fusion of data from multiple sensor sources such as “IPS”, “Wheel
Encoders” and “IMU” using an extended Kalman filter.

The study aimed to provide an alternative source of measurement with higher
accuracy and reliability than visual odometry by incorporating an Ultrasonic-
based IPS into the system. Moreover, to enhance the quality of 3D maps, the
“Update Detection Rate” block was added to the main algorithm to update the
algorithm with high frequency synchronized with the IPS device. The extended
Kalman filter was selected as a commonly used method for sensor fusion in this
study. The input format includes coordinates $(x,y)$ from “IPS”, velocities
$(v_{x},v_{y})$ from “Wheel Encoders”, orientation $(yaw)$; angular velocities
$(v_{yaw})$ and accelerations $(a_{x},a_{y})$ measured by “IMU” of camera
D435i. The EKF algorithm consists of two main steps: the prediction step and
the update step, the algorithm is implemented as follows:

• The prediction step:

The equation that predicts the robot’s state at the current time step is based
on its previous state estimate and the control input.

$x_{\\{k\mid k-1\\}}=f\left(x_{\\{k-1\\}},u_{k}\right)$ (1)

$P_{\\{k\mid k-1\\}}=F_{k}*P_{\\{k-1\mid k-1\\}}*F_{k}^{T}+Q_{k}$ (2)

Where ${x_{\\{k-1\\}}}$ is the previous state estimate ${x_{\\{k\mid k-1\\}}}$
is the predicted state estimate, ${u_{k}}$ is the control input at time step
$k$, ${F_{k}}$ is the Jacobian matrix of the motion model with respect to the
state vector at time step $k$, $P_{\\{k-1\mid k-1\\}}$ is the covariance
matrix of the previous state estimate, $P_{\\{k\mid k-1\\}}$ is predicted
covariance matrix and $Q_{k}$ is a manually constructed matrix that represents
the noise that occurs throughout the estimate process.

• The update step:

The equation that predicts the sensor’s measurements at time step $k$:

$z_{k}=h\left(x_{\\{k\mid k-1\\}}\right)+v_{k}$ (3)

Kalman gain calculation:

$K_{k}=P_{\\{k\mid k-1\\}}*H_{k}^{T}*\left(H_{k}*P_{\\{k\mid
k-1\\}}*H_{k}^{T}+R_{k}\right)^{\\{-1\\}}$ (4)

Then update the state estimate:

$x_{\\{k\mid k\\}}=x_{\\{k\mid k-1\\}}+K_{k}*\left(z_{k}-h\left(x_{\\{k\mid
k-1\\}}\right)\right)$ (5)

Update the state estimate covariance matrix:

$P_{\\{k\mid k\\}}=\left(I-K_{k}*H_{k}\right)*P_{\\{k\mid k-1\\}}$ (6)

Where $z_{k}$ is sensor measurement at time step $k$, $h\left(x_{\\{k\mid
k-1\\}}\right)$ is expected sensor measurement based on the predicted state
estimate, $v_{k}$ is measurement noise, $H_{k}$ is the Jacobian matrix of the
measurement model with respect to the state vector at time step $k$, $R_{k}$
is measurement noise covariance matrix, $K_{k}$ is Kalman gain matrix,
$x_{\\{k\mid k\\}}$ is updated state estimate and $P_{\\{k\mid k\\}}$ is
updated covariance matrix.

Finally, the output provides filtered odometry data then the data is sent
along with D435i camera image information to the “RTAB-Map” algorithm block.

The results show that the proposed method provides a more accurate estimate of
the robot’s position, which will be presented in Sec III.

The EKF generates filtered odometry data as its output, which is subsequently
transmitted to the “RTAB-Map” algorithm block together with image information
from the D435i camera. In the Working Memory (WM) of the RTAB- Map (Fig. 1a),
the Short-term Memory (STM) block has the function of storing odometry data
and raw sensor data. When a new node is created in STM the local occupancy
grid is now calculated from the depth image [19]. The nodes in the STM will
send data at a rate set in the RTAB-Map algorithm. Two parameters affect the
updating of data from STM and updating of map data including:

• “Rtabmap/MapUpdateDetectionRate”: This parameter relates to how often the
map is updated based on new sensor data and represents the maximum rate (in
Hz).

• “Rtabmap/DetectionRate”: This parameter relates to the rate at which RTAB-
Map performs feature detection on incoming D435i camera data and represents
the maximum rate (in Hz).

By default, the odometry input of the “Loop Closure and Proximity Detection”
block only uses the depth camera, resulting in a low update frequency of 1 Hz
due to delays in visual odometry estimation. This causes real-time delays in
the system, leading to errors in the map and slow updating of point cloud
data. To improve the system’s performance, a new node in the STM is created
that communicates solely with the “Loop Closure and Proximity Detection” block
and receives odometry data fused from the IPS and IMU, increasing the update
rate up to 10 Hz. The proposed method adjusted the
“Rtabmap/MapUpdateDetectionRate” and “Rtabmap/DetectionRate” parameters to
match the filtered odometry source’s update rate, ensuring accurate and timely
map updating.

##  III Experiments and Results

###  III-A Estimated Position Error

In this case, the comparison was performed on the position error of the robot
while stationary for 60 seconds between “Visual Odom” (Blue line) -
representing the base method, and “Fusion Odom” (Red line) - representing the
proposed method, as shown in Fig. 2a and Fig. 2b, respectively.

(a)

(b)

Figure 2: Estimated position error. a) Pose error X-axis b) Pose error Y-axis

The comparison in the selected 60-second interval can show the drift of
”Visual Odom”. It could be observed from the line graph that the position
estimate in the base method exhibits continuous jumps over time during the 3D
map construction process. However, after sensor fusion, the position estimate
becomes more accurate and stable during a specific time period.

###  III-B Estimated Trajectory Error

A comparative analysis of the trajectories obtained from both methods was
conducted, and the results were highly favorable. The mapping process was
carried out at different speeds of the robot with a linear velocity of
$v=0.15m/s$, $0.3m/s$ and rotational velocity of $\omega=0.39rad/s$,
$0.78rad/s$. The trajectory error was estimated with the ground truth taken
from the 3D map after the mapping process was completed. The results of the
$v=0.3m/s$ and $\omega=0.78rad/s$ experiments are shown in Fig. 3. It was
observed that there were significant differences between the trajectory of
“Visual Odom” and the ground truth (Fig. 3a), which was caused by pose drift
when the robot turned. However, “Fusion Odom” performed outstandingly (Fig.
3b) in reducing the trajectory error.

(a)

(b)

(c)

Figure 3: Estimated trajectory error in $v=0.3m/s$ and $\omega=0.78rad/s$. a)
Trajectory error case in the base method. b) Trajectory error case in the
proposed method. c) Estimated absolute trajectory error.

In order to evaluate the accuracy of the proposed method, the Absolute
Trajectory Error (ATE) [20] was used in this study. The ATE was calculated as
the average of the Euclidean distance difference between the ground truth and
the supplied odometry input over the total number of time steps. Formula to
calculate ATE:

$ATE=\frac{1}{N}\sum_{i=1}^{N}\left\|P_{i}-Q_{i}\right\|$ (7)

Where $N$ is the total number of time steps, $\left\|P_{i}-Q_{i}\right\|$ is
the error of the Euclidean distance of the two methods.

The results were shown in Fig. 3c, where the “Fusion Odom” column represented
the ATE coefficient of the proposed method, and the “Visual Odom” column
represented the base method. The ATE coefficient of the proposed method was
significantly lower than that of the base method, ranging from 9-36 times
lower. These findings suggested that the absolute orbital error of the
proposed method had been greatly reduced compared to the base method.
Additionally, as the linear and angular velocities increased, the orbital
error increased significantly. At $(v=0.3m/s$, $\omega=0.78rad/s)$, the error
of the base method was approximately $0.1$m, whereas the error of the proposed
method was only approximately $0.01m$. These results demonstrate the efficacy
of the proposed method in reducing errors and improving accuracy.

###  III-C 3D map quality and processing time

A comparative study (Fig. 4) was conducted to investigate the effectiveness of
synchronizing the frequency of updating the map processing of the RTAB-Map
method with that of location updates from the IPS. The experiment involved
scanning an area of the environment with a robot over a fixed time interval
($\delta t=2s$) and a fixed angular velocity ($\omega=0.2rad/s$). The number
of point clouds on the 3D map generated using the base method and the proposed
method was compared. The results showed that the proposed method generated a
higher number of point clouds (188190 points) in Fig. 4b compared to the base
method (83902 points) in Fig. 4a, indicating an improvement in the quality of
the 3D map. The findings demonstrated the clear benefits of increasing the
frequency of updating the map processing of RTAB-Map to be synchronized with
the frequency of location updates from the “Fusion Odom”.

(a)

(b)

Figure 4: 3D mapping quality a) Base method scans 83902 cloud points. b)
Proposed method scans 188190 cloud points.  Figure 5: Time of 3D map building

During testing for the robot’s construction of 3D maps using two different
methods on the same trajectory with identical linear and angular velocities,
the interference and loss of odometry in the control process led to the
repositioning of the robot by the operator and adjustment of the map
construction trajectory by the RTAB-Map system. This interference
significantly affected the time required to build the map. A test was
conducted to assess the time taken by the robot to build a 3D map with the
trajectory obtained after 5 test runs, present in Fig. 5. The results
indicated that the proposed method had an average mapping time of $110.4$
seconds, while the base method took an average of $158$ seconds to execute.

##  IV Conclusion

In this study, we proposed an approach to enhance the accuracy, speed, and
quality of indoor 3D mapping by integrating data from an external ultrasonic-
based Indoor Positioning System (IPS), Inertial Measurement System (IMU) from
a D435i camera, and wheel encoders. The integration is achieved using the
extended Kalman filter, which serves as an effective means to fuse the data
obtained from these sensors. The proposed method aims to reduce the drift
error of the camera pose and diminish noise from separate sensors. The study
evaluates and compares the base method and the proposed method based on
criteria such as position and trajectory estimation errors, map accuracy,
robot mapping time, and the amount of information in the 3D map. The results
indicate that the proposed method reduces errors in position and trajectory
estimation while mapping at high speeds, leading to improved map accuracy. The
proposed method also optimizes the time required to build the 3D map,
increases the map update frequency, and enhances the amount of information
contained in the 3D map compared to the base method. The obtained results show
the potential of using the proposed method to enhance indoor 3D mapping in
robotics, signifying that the fusion of multiple data sources can serve as a
valuable tool in generating accurate and high-quality 3D maps.

## References

  * [1] H. Durrant-Whyte and T. Bailey, “Simultaneous localization and mapping: part i,” _IEEE Robot. Autom. Mag._ , vol. 13, no. 2, pp. 99–110, 2006.
  * [2] T. Taketomi, H. Uchiyama, and S. Ikeda, “Visual slam algorithms: A survey from 2010 to 2016,” _IPSJ Transactions on Computer Vision and Applications_ , vol. 9, no. 1, pp. 1–11, 2017.
  * [3] A. J. Davison, “Real-time simultaneous localisation and mapping with a single camera,” in _Computer Vision, IEEE International Conference on_ , vol. 3. IEEE Computer Society, 2003, pp. 1403–1403.
  * [4] G. Klein and D. Murray, “Parallel tracking and mapping for small ar workspaces,” in _2007 6th IEEE and ACM international symposium on mixed and augmented reality_. IEEE, 2007, pp. 225–234.
  * [5] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “Orb-slam: a versatile and accurate monocular slam system,” _IEEE Trans. Robot._ , vol. 31, no. 5, pp. 1147–1163, 2015.
  * [6] R. Mur-Artal and J. D. Tardós, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” _IEEE transactions on robotics_ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * [7] M. O. Aqel, M. H. Marhaban, M. I. Saripan, and N. B. Ismail, “Review of visual odometry: types, approaches, challenges, and applications,” _SpringerPlus_ , vol. 5, pp. 1–26, 2016.
  * [8] B. Clipp, J. Lim, J.-M. Frahm, and M. Pollefeys, “Parallel, real-time visual slam,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 3961–3968.
  * [9] X. Zhang, Y. Su, and X. Zhu, “Loop closure detection for visual slam systems using convolutional neural network,” in _2017 23rd International Conference on Automation and Computing (ICAC)_. IEEE, 2017, pp. 1–6.
  * [10] S. García, M. E. López, R. Barea, L. M. Bergasa, A. Gómez, and E. J. Molinos, “Indoor slam for micro aerial vehicles control using monocular camera and sensor fusion,” in _2016 international conference on autonomous robot systems and competitions (ICARSC)_. IEEE, 2016, pp. 205–210.
  * [11] R. Mur-Artal and J. D. Tardós, “Fast relocalisation and loop closing in keyframe-based slam,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2014, pp. 846–853.
  * [12] G. Klein and D. Murray, “Improving the agility of keyframe-based slam,” in _Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part II 10_. Springer, 2008, pp. 802–815.
  * [13] H. A. Kadir and M. R. Arshad, “Features detection and matching for visual simultaneous localization and mapping (vslam),” in _2013 IEEE International Conference on Control System, Computing and Engineering_. IEEE, 2013, pp. 40–45.
  * [14] M. I. Ribeiro, “Kalman and extended kalman filters: Concept, derivation and properties,” _Institute for Systems and Robotics_ , vol. 43, no. 46, pp. 3736–3741, 2004.
  * [15] E. A. Wan and R. Van Der Merwe, “The unscented kalman filter,” _Kalman filtering and neural networks_ , pp. 221–280, 2001.
  * [16] S. Thrun, “Particle filters in robotics.” in _UAI_ , vol. 2. Citeseer, 2002, pp. 511–518.
  * [17] G. Nützi, S. Weiss, D. Scaramuzza, and R. Siegwart, “Fusion of imu and vision for absolute scale estimation in monocular slam,” _Journal of intelligent & robotic systems_, vol. 61, no. 1-4, pp. 287–299, 2011.
  * [18] J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 40, no. 3, pp. 611–625, 2017.
  * [19] M. Labbé and F. Michaud, “Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation,” _Journal of Field Robotics_ , vol. 36, no. 2, pp. 416–446, 2019.
  * [20] Z. Zhang and D. Scaramuzza, “A tutorial on quantitative trajectory evaluation for visual (-inertial) odometry,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 7244–7251.


# Discovery of an Unconventional Quantum Echo by Interference of Higgs
Coherence

C. Huang1,2, M. Mootz2, L. Luo2, D. Cheng2, J. M. Park2, R. H. J. Kim2, Y.
Qiang1,2,
V. L. Quito2, Yongxin Yao 2, P. P. Orth1,2, I. E. Perakis3 and J. Wang1,2†
1Department of Physics and Astronomy, Iowa State University, Ames, IA 50011
USA
2Ames National Laboratory, Ames, IA 50011 USA
3Department of Physics, University of Alabama at Birmingham,
Birmingham, AL 35294-1170, USA
†To whom correspondence should be addressed<EMAIL_ADDRESS><EMAIL_ADDRESS>
> Nonlinearities in quantum systems are fundamentally characterized by the
> interplay of phase coherences, their interference, and state transition
> amplitudes. Yet the question of how quantum coherence and interference
> manifest in transient, massive Higgs excitations, prevalent within both the
> quantum vacuum and superconductors, remains elusive. One hallmark example is
> photon echo, enabled by the generation, preservation, and retrieval of phase
> coherences amid multiple excitations. Here we reveal an unconventional
> quantum echo arising from the Higgs coherence in superconductors, and
> identify distinctive signatures attributed to Higgs anharmonicity. A
> terahertz pulse-pair modulation of the superconducting gap generates a “time
> grating” of coherent Higgs population, which scatters echo signals distinct
> from conventional spin- and photon-echoes in atoms and semiconductors. These
> manifestations appear as Higgs echo spectral peaks occurring at frequencies
> forbidden by equilibrium particle-hole symmetry, an “asymmetric” delay in
> the echo formation from the dynamics of the “reactive” superconducting
> state, and negative time signals arising from Higgs-quasiparticle anharmonic
> coupling. The Higgs interference and anharmonicity control the decoherence
> of driven superconductivity and may enable applications in quantum memory
> and entanglement.

When exposed to terahertz (THz) coherent driving, superconductors (SCs)
manifest reactive behaviors rather than adiabatic responses [1, 2, 3, 4, 5, 6,
7, 8, 9, 10, 11, 12, 13]. The 2$\Delta_{\mathbf{SC}}$ energy gap determined by
the order parameter is not rigid, leading to the emergence of Higgs collective
modes at $\omega_{\mathbf{Higgs}}\simeq$2$\Delta_{\mathbf{SC}}$. The
anharmonicity within the SC state and the evolution of its “soft” low-energy
gaps pose fundamental questions that diverge from the conventional framework
applied to comprehend quantum echo phenomena. First, in the observed
spin/photon echo signals in semiconductors [14, 15, 16, 17] and atoms [18],
correlations within many-electron ground states generally have minimal impact,
as the high-energy gap due to bandstructure maintains a rigid state through
rapid adjustments to the comparatively slower photoexcitation. In contrast,
the energy gap 2$\Delta_{\mathbf{SC}}$, determined by the SC order parameter,
changes during THz oscillation cycles with
$\omega_{\mathbf{0}}\simeq\Delta_{\mathbf{SC}}$. This evolution makes the SC
state evolve into a stationary non-equilibrium state distinct from any thermal
ground state. Second, superconductivity is distinct as it presents both Higgs
excitations featuring broken symmetry and quasi-particle (QP) excitations
arising from pair breaking. However, unlike conventional bound states,
discerning between the $\omega_{\mathbf{Higgs}}$ mode and the QP gap
2$\Delta_{\mathbf{SC}}$ is challenging since their excitation energies are in
close proximity. Observing quantum echoes will reveal the robust phase
coherence and interference effects, allowing for disentangling between Higgs
and QPs. Additionally, the observation of echoes at $\omega_{\mathbf{Higgs}}$
is absent so far, and QP fluctuations mostly overshadow nonlinear spectra in
the absence of inversion symmetry breaking or disorder [19, 20, 21, 4, 5, 22,
10].

Recently, notable progress has been made in coherent control schemes,
particularly with THz-driven supercurrent by pulse-pair excitation that
activates exotic collective excitations, including the hybrid-Higgs mode [7]
and phase-amplitude mode [8]. Our work below explores the quantum echo process
driven by the THz pulse-pair scheme. Figure 1A illustrates a three-step
process that involves both Higgs and QP excitations. First (#1), the nearly
resonant coupling of the light field of pulse A (red wiggled arrows) via
vector potential $A$ to the SC state, where
$\omega_{\mathbf{0}}\simeq$2$\Delta_{\mathbf{SC}}$, induces a Higgs field
excitation $\mathbf{H}(t)$ at time $t=0$ (blue dashed line). This excitation
is facilitated via inversion-symmetry breaking [23, 4, 5, 22] or disorder [1].
We model the former process using the light-induced superfluid momentum
$\mathbf{p}_{\mathrm{S}}(t)$ (red straight arrow) discussed later. This Higgs
excitation subsequently dephases due to destructive interference with the QP
continuum (Landau damping). After the inter-pulse delay time $\tau>0$, the
second light field, B, excites a second Higgs coherence, stops the above
dephasing and stores the coherence in the order parameter amplitude (#2) by
creating a “temporal grating” of coherent Higgs population along the time
$\tau$ axis. This difference-frequency Raman process is controlled by the
relative phase between A and B, which is tuned via the pulse time delay. A
third field B induces a QP coherence also at time $t=\tau$, which scatters off
the coherent Higgs grating and creates the quantum echo signal. The ABB echo
described above manifests as nonlinear current $\mathbf{J}_{NL}$ with a
delayed formation at time $t_{\mathrm{c}}$ from constructive interference.
Moreover, a negative time signal, referred to as BBA pulse sequence, is not
expected for harmonic Higgs excitations. As discussed in supplementary Section
4, in this case, pulse B comes first and creates both a Higgs and a QP
coherence. These evolve for inter-pulse delay $\tau$, at which time pulse A
can sense them. Therefore, the negative time signals observed show storage of
the 2-excitation coherence within the time-evolving quantum system due to
anharmonicity arising from Higgs-QP couplings.

Figure 1. Higgs echo spectroscopy in a niobium superconductor through
excitation by THz pulse pairs. (A) Schematics of the quantum echo process as a
three-step process that involves quantum interference of both Higgs (#1 and
#2) and coherent QP (#3) excitations (see main text). The response of SC
condensate during cycles of THz electric fields (red wiggle line) gives rise
to momentum $\mathbf{p}_{\mathrm{S}}(t)$ (red arrow) that breaks the inversion
symmetry and enables the dipole-forbidden, Higgs excitations. (B) Schematics
of THz multidimensional coherent spectroscopy of a 40 nm Nb film. Two delay
lines are employed to control THz-pulse-pair excitation, both the inter-pulse
delay $\tau$ and the sampling delay $t$. The nonlinear signals are collected
using a double modulation scheme (Supplementary Section 3). (C) Temporal
waveforms of the nearly single-cycle THz pulse–pair following transmission
through the Nb film (red and blue lines), and (D), spectra of the used pulses
centered at $\omega_{0}\sim$4 meV (vertical dashed line) and with broadband
frequency width $\Delta\omega\sim$6 meV. (E) Two-dimensional (2D) false-colour
plot of the measured coherent nonlinear transmission $E_{\mathrm{NL}}(t,\tau)$
in our sample induced by THz pump electric fields of
$E_{\mathrm{THz,A,B}}\sim$50 kV cm-1 at 4K. (F) The Fourier transform is
applied to the $E_{\mathrm{NL}}(t,\tau)$ trace depicted in panel (E). (G) and
(H) 2D Fourier transform of $E_{\mathrm{NL}}(t,\tau)$ employing the identical
THz pump field conditions as (E) for temperatures of 8 K and 15 K. Note that
the subtle yet discernible spectral difference between Figs. 1G and 1H is
depicted in Fig. 2B.

In this report, we demonstrate quantum echoes involving transient, yet
enduring, Higgs coherence. We address two significant questions: (1) How to
induce the wave-mixing process sketched in Fig. 1A given that nonlinear
currents $\mathbf{J}_{NL}$ from Higgs excitations is precluded by the symmetry
in the ground state? (2) What are the unique characteristics of Higgs echoes
in a “reactive” SC state that distinguish them from the conventional photon
echoes? We implement THz Higgs echo spectroscopy in a niobium (Nb)
superconductor as illustrated in Fig. 1B. We follow a phase-resolved and
collinear, pulse-pair excitation protocol within the framework of multi-
dimensional coherent spectroscopy (MDCS) [25, 10, 8, 17, 26, 27, 28, 16, 29,
30, 31, 32, 8, 33]. Unlike for the conventional pump-probe protocol, which has
been used to access a single Higgs or QP excitation pathway [2, 1, 34, 7, 35],
the MDCS scheme employed here enables the observation of photon echo signals
arising from the interference of multiple phase-coherent excitations. Taking
advantage of both the real time $t$ and the relative phase delay time $\tau$
between the two THz fields, we can separate Higgs echo peaks from other
nonlinear pathways in a 2D frequency plane defined by the excitation
$\omega_{t}$ and detection $\omega_{\tau}$ frequencies. In this manner, we
discern distinct and sharp Higgs echo peaks in MDCS spectra at low
temperatures and low fluences. We observe a delayed echo formation influenced
by the time-dependent SC bands and supercurrent flow, accompanied by a
pronounced negative time signal associated with Higgs anharmonicity. These
features are well reproduced by our quantum kinetic simulations, pinpointing
the salient roles of the coherent Higgs grating and Higgs-QP coherent
coupling.

We study a Nb film of 40nm thickness on a 1mm Si substrate (Supplementary
section 1). The sample exhibits a SC transition at T${}_{\mathrm{C}}\sim$9 K
and displays a SC gap of 2$\Delta_{\mathbf{SC}}\sim$ 3.1 meV as characterized
by equilibrium THz conductivity measurements (Supplementary Section 2). Our
THz-MDCS (Supplementary section 3) measures all detected coherent nonlinear
responses to two phase-locked, nearly single-cycle THz pulses, A and B, of
similar field strengths (Figs. 1C). The two pulses have central frequency
$\omega_{0}\sim$4 meV above the SC energy gap (black dashed line, Fig. 1D).
The broadband pulse frequency width, $\Delta\omega_{0}\sim$ 6 meV, excites a
wide range of the QP continuum, in addition to the Higgs mode. The measured
phase-resolved, nonlinear supercurrent differential emission,
$E_{\mathrm{NL}}(t,\tau)=E_{\mathrm{AB}}(t,\tau)-E_{\mathrm{A}}(t)-E_{\mathrm{B}}(t,\tau)$,
is recorded as a function of both the gate time $t$ and the delay time
$\tau=t_{\mathrm{A}}-t_{\mathrm{B}}$ between the two pulses A and B (red
double-arrow, Fig. 1B). The $\tau$ axis facilitates the generation of a
temporal grating, which scatters $N$-wave mixing signals into different
$\omega_{\tau}$ components. The process is analogous to the traditional
spatial grating which scatters wave-mixing signals along various directions
[24, 25].

A representative 2D temporal profile of the THz coherent nonlinear dynamics of
$E_{\mathrm{NL}}(t,\tau)$ driven by low fields $E_{\mathrm{THz,A,B}}\sim$50 kV
cm-1 is shown in Fig. 1E. This figure reveals pronounced coherent temporal
$\tau$-oscillations that last much longer than the temporal overlap between
the two driving pulses (Fig. 1C vs Fig. 1E), which indicates the presence of
long-lived coherent excitations. We underpin the above coherent responses by
performing Fourier transformations of $E_{\mathrm{NL}}(t,\tau)$ with respect
to both $t$ (frequency $\omega_{t}$) and $\tau$ (frequency $\omega_{\tau}$).
Figures 1F-1H compare these experimentally-measured 2D spectra (unnormalized)
at 4 K, 8 K, and 15 K across the superconducting transition temperature of 9
K. We observe multiple sharp peaks with the same
$\omega_{t}=\omega_{\mathrm{H}}\sim$2$\Delta_{\mathrm{SC}}\simeq$3 meV (red
dashed line) that split along the vertical $\omega_{\tau}$-axis. These peaks
change drastically as the temperature increases above $T_{\mathrm{C}}$.
Additional peaks are observed along $\omega_{t}\simeq
2\omega_{\mathbf{0}}\sim$8 meV, indicated by the green arrows in Figs. 1F-1H.
The latter peaks persist both below and above $T_{\mathrm{C}}$, which aligns
with our expectations for QP excitations.

We highlight two key points. First, the observed peaks (red arrows in Fig. 1F
for $T=4$ K) along the
$\omega_{t}\simeq$2$\Delta_{\mathrm{SC}}=\omega_{\mathrm{H}}$ red dashed
vertical line have frequency widths $\sim$0.5 meV, i. e., more than one order
of magnitude smaller than the pulse bandwidth $\Delta\omega_{0}$ (Fig. 1D).
This narrow linewidth indicates that, below $T_{\mathrm{C}}$, the coherent
time evolution indicated by $E_{\mathrm{NL}}(t,\tau)$ is governed by long-
lived oscillations. The observed MDCS peaks at
$\omega_{t}=\omega_{\mathrm{H}}$ exhibit a magnitude surpassing that of
conventional nonlinear wave-mixing signals at $\omega_{t}\simeq
n\omega_{\mathbf{0}}$ ($n=1,2...$), showcasing a unique feature consistent
with our anticipated coherent response from a “reactive” ground state.
Remarkably, MDCS peaks with $\omega_{t}=\omega_{\mathrm{H}}$ are not allowed
by the equilibrium SC state symmetry [36, 4]. Instead, they can arise from the
Higgs collective modes of a non-equilibrium SC state with light-induced
inversion symmetry breaking and/or disorder. The state corresponds to finite-
momentum Cooper pairing that evolves during the pulse but lives well after it
[36, 8]. Furthermore, the sharp $\omega_{t}=\omega_{\mathrm{Higgs}}$ peaks
exhibit significant broadening, redshift, and reduction as the temperature
approaches $T_{\mathrm{C}}$. They vanish above $T_{\mathrm{C}}$. This becomes
apparent when examining the spectra at 4 K (Fig. 1F), denoted by the red
dashed lines, in contrast to those at 8 K (Fig. 1G) and 15 K (Fig. 1H).

Figure 2. Variations of Higgs echo and other collective excitation peaks in
MDCS spectra by changing the THz driving field and temperature. The normalized
THz-MDCS spectra within superconducting states are obtained through the
subtraction of normal state data for both low and high driving fields. A:
$E_{\mathrm{THz,A,B}}\sim$50 kV cm-1 and 4 K; B: $\sim$50 kV cm-1 and 8 K; C:
$\sim$160 kV cm-1 and 4 K; D: $\sim$160 kV cm-1 and 8 K. E The Higgs frequency
vectors, denoted as $\omega_{\mathrm{A}}=(\omega_{\mathrm{H}},0)$ and
$\omega_{\mathrm{B}}=(\omega_{\mathrm{H}},\omega_{\mathrm{H}})$, illustrate
pseudo-spin coherent oscillations induced by THz-pulse pair excitations. F
Schematics of the quantum dynamical pathways for generating two Higgs echo
peaks employing ABB (HE1) and BAA (HE2) pulse sequences. Other collective
excitation peaks in MDCS spectra are depicted as follows: G, 2-quantum Higgs
peaks; H, Higgs (H2) peak by pump-probe process; I: Higgs (H1) peak by non-
rephasing process.

Second, the THz-MDCS spectrum at 4 K (Fig 1F) exhibits four distinct peaks at
$\omega_{t}=\omega_{\mathrm{H}}$. In addition to the two main Higgs peaks (the
red arrows) discussed above, two additional peaks are seen along the vertical
$\omega_{\tau}$ axis (the blue arrows). The suppression of these additional
$\omega_{\tau}$ components as we approach $T_{\mathrm{C}}$ is evident when
comparing the spectra at 4 K (Fig. 1F) and 8 K (Fig. 1G). They come from the
Higgs echo process illustrated in Fig. 1A as elaborated below. To better
visualize these subtle, yet prominent, peaks, we subtract the spectrum
acquired at 15 K from the experimental
$E_{\mathrm{NL}}(\omega_{t},\omega_{\tau})$ spectrum recorded below
$T_{\mathrm{C}}$ to remove any temperature-independent spectral artifacts. We
present the normalized spectra obtained after this subtraction in Figs. 2A-2D
for both low and high driving fields, at $T=4$ K and $T=8$ K. We focus on Fig.
2A obtained at $E_{0}=50$ kV cm-1 and low temperature of 4 K. The
$E_{\mathrm{NL}}(\omega_{t},\omega_{\tau})$ spectrum clearly reveals the
presence of the four prominent peaks along the vertical dashed line positioned
at $\omega_{t}\simeq 2\Delta_{\mathrm{SC}}$. Most intriguingly, apart from the
stronger peaks (black circles) at roughly $(\omega_{\mathrm{H}},0)$ (H2) and
$(\omega_{\mathrm{H}},\omega_{\mathrm{H}})$ (H1), the two Higgs echo peaks
distinctly emerge at $(\omega_{\mathrm{H}},2\omega_{\mathrm{H}})$ (HE1) and
$(\omega_{\mathrm{H}},-\omega_{\mathrm{H}})$ (HE2) (red circles), as confirmed
by our analysis and simulations below.

The Higgs echo features become prominent only when Higgs collective
excitations are long-lived, with sharp Higgs peaks dominating the MDCS
spectrum, indicating that dissipation is suppressed. When either the pump
field is increased to $E_{0}=160$ kV cm-1 (Fig. 2C) or the temperature is
raised to 8 K (Fig. 2B), a substantial reduction and broadening in the
intensity of the HE1 and HE2 Higgs echo peaks are seen, as indicated within
the two red circles in Fig. 2C. This observation coincides with the broadening
and red-shift of the two main Higgs peaks, H1 and H2, consistent with either
excitation or temperature-induced dephasing, discussed later. For strong pump
field $E_{0}=160$ kV cm-1 at 8K (Fig. 2D), the primary features in the
$E_{\mathrm{NL}}(\omega_{t},\omega_{\tau})$ spectrum are the diagonal QP
excitation peaks located roughly at
$(2\omega_{\mathrm{0}},2\omega_{\mathrm{0}})$ (Figs. 2C and 2D).

The prevalence of the peaks H1 and H2, centered around frequencies
$\omega_{\mathrm{H}}<\omega_{0}$ and $2\omega_{0}$, suggests analyzing the
MDCS spectra by introducing Higgs frequency vectors that describe collective
modes of the light-induced non-equilibrium SC state:
$\omega_{\mathrm{A}}=(\omega_{\mathrm{H}},0)$ and
$\omega_{\mathrm{B}}=(\omega_{\mathrm{H}},\omega_{\mathrm{H}})$, respectively,
as depicted in Fig. 2E. Analogous to the formation of a spatial grating, the
$\omega_{\mathrm{A,B}}$ frequency vectors allow us to describe the formation
of a temporal, coherent Higgs grating along the $\omega_{\tau}$ axis via the
difference-frequency Raman processes
$\pm(\omega_{\mathrm{A}}-\omega_{\mathrm{B}})$. Figure 2F illustrates the
nonlinear quantum excitation processes
$2\omega_{\mathrm{B}}-\omega_{\mathrm{A}}$ and
$2\omega_{\mathrm{A}}-\omega_{\mathrm{B}}$ that give rise to Higgs echo peaks
HE1 and HE2. In particular,
$(-\omega_{\mathrm{A}}+\omega_{\mathrm{B}})+\omega_{\mathrm{B}}$ (ABB echo,
black arrows) and
$(-\omega_{\mathrm{B}}+\omega_{\mathrm{A}})+\omega_{\mathrm{A}}$ (BAA echo,
blue arrows) describe the scattering of a SC excitation $\omega_{\mathrm{B}}$
($\omega_{\mathrm{A}}$) off the temporal grating created by
$\omega_{\mathrm{B}}-\omega_{\mathrm{A}}$
($\omega_{\mathrm{A}}-\omega_{\mathrm{B}}$), leading to the HE1 and HE2 peaks
at positions $(\omega_{\mathrm{H}},2\omega_{\mathrm{H}})$ and
$(\omega_{\mathrm{H}},-\omega_{\mathrm{H}})$. The negative time signal for the
ABB (BAA) echo manifest as a 2-quantum Higgs peak at the same position as HE1
(HE2), as illustrated in Fig. 2G (2Q peaks), which as we discuss later can
arise from Higgs-QP coherent coupling. In contrast, the H1(H2) Higgs peak at
$\omega_{\mathrm{B}}=(\omega_{\mathrm{H}},\omega_{\mathrm{H}})$
($(\omega_{\mathrm{H}},0)$) is generated by the pump-probe and non-rephasing
nonlinear process
$(\omega_{\mathrm{A}}-\omega_{\mathrm{A}})+\omega_{\mathrm{B}}$
($(-\omega_{\mathrm{B}}+\omega_{\mathrm{B}})+\omega_{\mathrm{A}}$) in Figs. 2H
and 2I.

To differentiate the temporal evolution and quantum pathway in more detail, we
apply a 2D-Gaussian spectral filter to isolate the HE1 and HE2 peaks indicated
by the red circles in Fig. 2A, and then transform the echo signals back into
the time domain. The signals extracted from the ABB and BAA Higgs-echo peaks
in this way are presented in Figs. 3A and 3B, respectively. The order in which
the pulses interact with the superconductor determines the well-defined phase
fronts of these Higgs-echo signals. Specifically, for the ABB pulse sequence,
the phase fronts are oriented parallel to the blue dashed line in Fig. 3A. On
the other hand, employing the BAA pulse sequence results in phase fronts that
are nearly perpendicular to the blue dashed line in Fig. 3B. Importantly, the
ABB (BAA) signal is nonzero even when pulse B (A) arrives before pulse A (B).
This time ordering, opposite to the one shown in Fig. 1A, gives the negative
time signals that corresponds to the BBA (AAB) sequence marked in Fig. 3A
(Fig. 3B). The remarkable negative time signals observed here are exceptional
and suggest a 2-quantum excitation and anharmonicity. Our simulations identify
the Higgs and QP coupling as the origin.

Figure 3. Identification and origin of the Higgs echo and 2-Higgs quantum
peaks in THz-MDCS signals. (A and B) Higgs-echo signals ABB and BAA in time
domain resulting from the inverse Fourier transformation of the signals
indicated by red circles in Fig.2A, consistent with the HE1 and HE2 echo
signals at positive times from the processes illustrated in Fig. 2F. In
addition, the ABB (BAA) pulse sequences also show a negative time signal (red
arrows) when pulse B (A) arrives before pulse A (B), originating from the
2-Higgs quantum peaks illustrated in Fig. 2G. The positive (negative) time
directions are indicated by black arrows (red arrows). (C) Temporal positions
of the ABB and BAA echoes, $t_{\mathrm{C}}$ (blue and black lines), are time-
delayed compared to both driving pulses (dashed red lines). Remarkably, the
echo formation time exhibits a pronounced departure from symmetric temporal
positions in relation to the pulse-pair excitations (depicted by gray lines),
deviating notably from conventional photon echoes. (D and E) An illustration
of the ABB echo formation process (Fig. 1E) from the dynamical response of the
“reactive” superconducting state and supercurrent, as presented in Fig. 1D.
$E_{\mathbf{k}}^{\mathrm{QP}}(t)$ denotes a time-dependent QP energy
dispersion. (F) The Fourier spectra of two-pulse THz coherent signals
$E_{\mathrm{NL}}$ at various temperatures from 4 K to 10 K for a peak THz pump
electric field of $E_{\mathrm{THz,A,B}}\sim$160 kV/cm and $\tau=$3 ps. (G) The
Fourier spectra of $E_{\mathrm{NL}}$ at various pump fields at temperature of
4 K and $\tau=$3 ps.

To further quantify the unconventional nature of the quantum echo signals
observed, we define the temporal position of any time-dependent signal $E(t)$
as $t_{\mathrm{c}}=\int\mathrm{d}t\,tE^{2}(t)/\int\mathrm{d}t\,E^{2}(t)$,
i.e., the center of gravity [37]. Figure 3C shows this $t_{\mathrm{c}}$ for
the Higgs echo signals HE1 and HE2 (black and blue solid lines) and compares
it to that of the two individual laser pulses A and B (dashed red lines). The
ABB echo signal primarily contributes at $\tau>0$ (blue line), while
$t_{\mathrm{c}}$ at $\tau<0$ (black line) is dominated by the BAA echo signal.
Note that the Higgs echo is time-delayed as compared to both pulses. Such a
behavior is the hallmark for the photon echoes expected from inhomogeneously
broadened systems, as, with purely homogeneous broadening, $t_{\mathrm{c}}$
should coincide with the $t_{\mathrm{c}}$ of the last pulse (red dashed lines,
Fig. 3C) due to a free-induction decay. Most notably, the delayed echo
formation time markedly contrasts to the symmetric time $t_{\mathrm{c}}=\tau$
(gray lines, Fig. 3C) after the last pulse observed in conventional echoes
[15, 38, 39].

The unique characteristics of Higgs echoes originate from the combined
influence of the “reactive” oscillating gap amplitude $|\Delta(t,\tau)|$ and
the damping of the light-induced superfluid momentum
$\mathbf{p}_{\mathrm{S}}(t)$. This interplay results in a time-dependent QP
energy dispersion $E_{\mathbf{k}}^{\mathrm{QP}}(t)$ as illustrated in Fig. 3D.
Excitation by a broadband pulse (Fig. 1D) leads to both Higgs modes and a
broad inhomogeneous QP distribution with energy
$E_{\mathbf{k}}^{\mathrm{QP}}(t)$. The latter results in quantum echo rather
than free induction decay. Figure 3E illustrates the interference processes
for the ABB echo. The phase accumulation at QP momentum $\mathbf{k}$ is
$\omega_{\mathrm{H}}\tau-\int_{0}^{t}E_{\mathbf{k}}^{\mathrm{QP}}(t^{\prime})\cdot
dt^{\prime}$. The first term accounts for the initial Higgs time evolution up
to time $t=0$ (red wiggle lines). The second term describes the subsequent,
re-phasing time evolution (blue wiggle lines) by coherent QP excitation. The
time-dependence of $E_{\mathbf{k}}^{\mathrm{QP}}(t)$ involving both Higgs and
QP coherent excitations results in the observed distinct “asymmetric” delay in
echo formation in Fig. 3C.

Figures 3F and 3G reveal pronounced temperature- and fluence-induced Higgs
dephasing process. Figure 3F shows the temperature-dependent Fourier transform
of the MDCS temporal ($t$) dynamics for 4 K–10 K at fixed $\tau=$3 ps. As we
approach $T_{\mathrm{C}}$ from below, this signal rapidly decreases.
Simultaneously, the frequency of the main peak below $\omega_{0}$ undergoes a
redshift, and eventually gives way to a QP peak centered around
$2\omega_{0}\sim$8 meV. This is seen by comparing the 4 K (black line) and 8 K
(blue line) versus 10 K (red line) traces (black dashed lines), consistent
with our experiment (Figs 1F–1H). Figure 3G shows three representative Fourier
spectra of the MDCS temporal dynamics at $T$=4 K, obtained at
$E_{\mathrm{pump}}=$10 kV/cm (red), 50 kV/cm (blue) and 160 kV/cm (black). We
observe a redshift and broadening of the Higgs mode, accompanied by the
generation of QPs (black dashed lines). Thermal QP excitations suppress both
Higgs and Higgs echo signals consistent with our data in Figs. 2A-2D.

Figure 4. Quantum kinetic simulation of Higgs echo signals from THz-MDCS
spectra. (A) Simulated nonlinear spectrum
$E_{\mathrm{NL}}(\omega_{t},\omega_{\tau})$ , which accurately captures all of
the dominant peaks observed in the measured nonlinear spectrum in Fig. 2A. The
ABB and BAA Higgs echo signals are indicated by red circles. (B)
$E_{\mathrm{NL}}(\omega_{t},\omega_{\tau})$ for a calculation where
$\delta|\Delta|\rho_{2}(k)=0$ in the simulations (Supplementary Section 5).
The quantum echo signals BAA and ABB (red circles) are strongly suppressed in
this case. (C and D) Higgs-echo signals ABB and BAA in time domain resulting
from the inverse Fourier transformation of the signals indicated by circles in
Fig. 4A. The ABB (BAA) pulse sequences shows a negative time signal when pulse
B (A) arrives before pulse A (B), consistent with the experimental
observations in Figs. 3A-3B. The positive (negative) time directions are
indicated by black arrows (red arrows). (E) Calculated temporal position of
the photon echo, $t_{\mathrm{c}}$ (black line), which is time-delayed compared
to both driving pulses (dashed red lines). (F) Two-dimensional spectrum of the
order parameter modulation $\delta|\Delta|$ that creates the Higgs temporal
grating. Horizontal dashed lines indicate
$\omega_{\tau}=\pm\omega_{\mathrm{H}}$. The spectrum shows two strong peaks at
$(0,\pm\omega_{\mathrm{H}})$ (red circles) which are generated by the
difference frequency processes $\pm(\omega_{\mathrm{A}}-\omega_{\mathrm{B}})$
that require two time-delayed pulses with well-defined relative phase.

To solidify the above physical depiction of unconventional Higgs quantum
echoes, we conducted quantum kinetic simulations using the gauge-invariant
superconductor Bloch equation theory [40, 36, 41]. We directly simulated the
nonlinear temporal dynamics measured in the experiment for a one-band $s$-wave
BCS superconductor driven by two intense phase-locked terahertz electric-field
pulses (Supplementary section 5). Figure 4A presents the simulated nonlinear
response $E_{\mathrm{NL}}(t,\tau)$, which accurately captures all of the
dominant peaks observed in the measured nonlinear spectrum in Fig. 2A. Similar
to the experiment, the calculated 2D spectrum shows Higgs peaks at
$(\omega_{\mathbf{H}},\omega_{\mathbf{H}})$ and $(\omega_{\mathbf{H}},0)$,
while Higgs echo signals emerge at
$(\omega_{\mathbf{H}},-\omega_{\mathbf{H}})$ and
$(\omega_{\mathbf{H}},2\omega_{\mathbf{H}})$. Analogous to the photon echo
analysis presented in Fig. 3, we filter out the BAA and ABB Higgs-echo peaks
in Fig. 4A and transform them back to the time domain. The resulting signals
in the time domain are shown in Figs. 4C and 4D. The ABB (BAA) pulse sequences
give a strong negative time signal even when pulse B (A) arrives before pulse
A (B), and the phase fronts of both echoes are nearly perpendicular to each
other. These results are fully consistent with our experimental observations
in Figs. 3A-3B. The calculated temporal position of the photon echo,
$t_{\mathrm{c}}$, is plotted in Fig. 4E (black line). It is time-delayed when
compared to both driving pulses (dashed red lines) and deviates from the
conventional symmetric echo formation time, in agreement with the experimental
result in Fig. 3C.

To clarify the origin of the echo signals in our calculations, we present in
Fig. 4B the result of the simulation when setting $\delta|\Delta|=0$, where
$\delta|\Delta|=(|\Delta_{\mathrm{AB}}(t,\tau)|-|\Delta_{0}|)-(|\Delta_{\mathrm{A}}(t)|-|\Delta_{0}|)–(|\Delta_{\mathrm{B}}(t,\tau)|-|\Delta_{0}|)$
and $|\Delta_{0}|$ denotes the equilibrium order parameter amplitude.
$|\Delta_{\mathrm{AB}}(t,\tau)|$ describes the order parameter amplitude
dynamics driven by coherent modulation of the order parameter amplitude by two
THz pulses with well-defined relative phase controlled by $\tau$. Single pulse
A (B) excitation results in $|\Delta_{\mathrm{A}}(t)|$
($|\Delta_{\mathrm{B}}(t,\tau)|$). If pulses A and B act independently to
create fluctuations of the order parameter from equilibrium, we have
uncorrelated excitation with $\delta|\Delta|=0$. Clearly, the quantum echo
signals HE1 and HE2 are strongly suppressed when we set $\delta|\Delta|=0$ in
Fig. 4B. Such an independent SC excitation by two THz fields is distinctly
different from the pulse-pair driving result of $\delta|\Delta|\neq 0$ in Fig.
4A.

We elaborate below that finite $\delta|\Delta|$ generates a dynamical Higgs
grating and quantum echo signals. Fig. 4A unravels the role of time-delayed
coherent excitations with well defined relative phases and distinguishes it
from the contributions of individual Higgs or QP excitations to the THz-MDCS
spectrum. As shown in the Supplement Section 5, we formulate the gauge
invariant density matrix equations of motion in terms of four-component
Anderson pseudo-spin oscillators $\rho_{i}$, $i=0,1,2,3$. The photon echo
signals in our simulations are driven by the correlated contribution by the
pulse-pair,
$\delta\rho_{2}(t,\tau)=\Delta\rho^{\mathrm{AB}}_{2}(t,\tau)-\Delta\rho^{\mathrm{A}}_{2}(t)-\Delta\rho^{\mathrm{B}}_{2}(t,\tau)$.
Here, $\Delta\rho^{\mathrm{AB}}_{2}(t,\tau)$ describes pseudo-spin canting
away from the equilibrium $x-z$ plane by both phase-coherent pulses, while
$\Delta\rho^{\mathrm{A}}_{2}(t)$ ($\Delta\rho^{\mathrm{B}}_{2}(t,\tau)$)
describes pseudo-spin canting induced by pulse A (B) alone. By comparing the
results of the full calculation in Fig. 4A, with the simulation in Fig. 4B,
where $\delta|\Delta|=0$, we show that the dominant contribution to the echo
peaks originates from parametric driving by the $\delta|\Delta|$ oscillations
in time $\tau$, which leads to a nonlinear driving force of the form
$\delta|\Delta(t,\tau)|\Delta\rho^{\mathrm{A,B}}_{2}(t)$ that is additional to
the conventional Raman processes (discussed further in the Supplement Section
5). In particular, the 2D spectrum of $\delta|\Delta|$ in Fig. 4F shows two
different peaks at $\omega_{t}=0$, which are displaced along the vertical
$\omega_{\tau}$ axis that measures the phase coherence. These peaks of
$\delta|\Delta|$, located at $(0,\pm\omega_{\mathrm{H}})$, are generated by
difference-frequency processes $\pm(\omega_{\mathrm{A}}-\omega_{\mathrm{B}})$.
$\delta|\Delta(t,\tau)|$ thus describes a temporal grating that scatters a QP
coherence $|\Delta\rho^{\mathrm{A,B}}_{2}(t)|$, to be contrasted with order
parameter quench with $\tau\sim 0$. The grating is created by Higgs
excitations A and B with well-defined relative phase. At the same time,
$\Delta\rho^{\mathrm{A,B}}_{2}(t)$ describes the continuum of photoexcited QP
coherences with different ${\bf k}$. The time-delayed quantum echo signal can
be understood in terms of refraction of such QP excitations from the dynamic
Higgs grating. This process is comparable to the photon echo generation
process in semiconductors, where the refracted signal scatters off a coherent
population grating generated by the interference of two polarizations created
by a pulse-pair. Here, the anharmonic nonlinear source term
$\propto\delta|\Delta|\Delta\rho^{\mathrm{A,B}}_{2}$ generates rephasing
photon echo signals at
$(-\omega_{\mathrm{B}}+\omega_{\mathrm{A}})+\omega_{\mathrm{A}}=(\omega_{\mathrm{H}},-\omega_{\mathrm{H}})$
and
$(-\omega_{\mathrm{A}}+\omega_{\mathrm{B}})+\omega_{\mathrm{B}}=(\omega_{\mathrm{H}},2\omega_{\mathrm{H}})$
as well as pump-probe (Fig. 2H) and non-rephasing signals (Fig. 2I) at
$(\omega_{\mathrm{H}},\omega_{\mathrm{H}})$ and $(\omega_{\mathrm{H}},0)$.

In summary, we discover quantum echo signals arising from transient, yet
sufficient long-lived, Higgs coherence in superconductivity. The Higgs echo
signals are characterized by a delayed build-up time deviating from symmetric
formation and a negative time signal from anharmonicity. These features set it
apart from any other known echo phenomena. Importantly, our observations
cannot be solely explained by either Higgs or QP excitation alone. Quantum
interference and anharmonicity between two phase-locked Higgs excitations, and
their interactions with QP coherence, however, comprehensively explain all of
our observations.

## References

  * [1] R. Matsunaga, et al., Science 345, 1145 (2014).
  * [2] R. Matsunaga, et al., Phys. Rev. Lett. 111, 057002 (2013).
  * [3] X. Yang, et al., Nat. Mater. 17, 586 (2018).
  * [4] X. Yang, et al., Nat. Photon. 13, 707 (2019).
  * [5] C. Vaswani, et al., Phys. Rev. Lett. 124, 207003 (2020).
  * [6] F. Giorgianni, et al., Nat. Phys. 15, 341 (2019).
  * [7] C. Vaswani, et al., Nat. Commun. 12, 258 (2021).
  * [8] L. Luo, et al., Nat. Phys. 19, 201 (2023).
  * [9] A. Liu, et al., Probing inhomogeneous cuprate superconductivity by terahertz josephson echo spectroscopy (2023).
  * [10] K. Katsumi, et al., Revealing novel aspects of light-matter coupling in terahertz two-dimensional coherent spectroscopy: the case of the amplitude mode in superconductors (2023).
  * [11] S. Zhang, et al., National Science Review 10, nwad163 (2023).
  * [12] M. Buzzi, et al., Phys. Rev. X 11, 011055 (2021).
  * [13] B. Cheng, et al., Evidence for highly damped Higgs mode in infinite-layer nickelates (2023).
  * [14] D. S. Chemla, J. Shah, Nature 411, 549 (2001).
  * [15] K. Leo, et al., Phys. Rev. Lett. 65, 1340 (1990).
  * [16] J. Lu, et al., Phys. Rev. Lett. 118, 207204 (2017).
  * [17] W. Kuehn, K. Reimann, M. Woerner, T. Elsaesser, R. Hey, The Journal of Physical Chemistry B 115, 5448 (2011).
  * [18] P. Tian, D. Keusters, Y. Suzaki, W. S. Warren, Science 300, 1553 (2003).
  * [19] R. Sooryakumar, M. V. Klein, Phys. Rev. Lett. 45, 660 (1980).
  * [20] P. B. Littlewood, C. M. Varma, Phys. Rev. Lett. 47, 811 (1981).
  * [21] T. Cea, C. Castellani, and L. Benfatto, Phys. Rev. B 93, 180507(R) (2016).
  * [22] S. Nakamura, et al., Phys. Rev. Lett. 122, 257001 (2019).
  * [23] Moor A, Volkov AF, Efetov KB., Phys. Rev. Lett. 118, 047001 (2017).
  * [24] Cundiff S. and Mukamel S., Phys. Today. 66, 7, 44 (2013).
  * [25] S. Mukamel, Principles of nonlinear optical spectroscopy, no. 6 (Oxford University Press on Demand, 1999).
  * [26] C. Somma, K. Reimann, C. Flytzanis, T. Elsaesser, M. Woerner, Phys. Rev. Lett. 112, 146602 (2014).
  * [27] F. Junginger, et al., Phys. Rev. Lett. 109, 147403 (2012).
  * [28] T. Maag, et al., Nat. Phys. 12, 119 (2016).
  * [29] C. L. Johnson, B. E. Knighton, J. A. Johnson, Phys. Rev. Lett. 122, 073901 (2019).
  * [30] S. Pal, et al., Phys. Rev. X 11, 021023 (2021).
  * [31] A. T. Tarekegne, et al., ACS Photonics 7, 221 (2020).
  * [32] F. Mahmood, et al., Nature Physics 17, 627 (2021).
  * [33] T. G. H. Blank, et al., Phys. Rev. Lett. 131, 026902 (2023).
  * [34] R. Matsunaga, et al., Phys. Rev. B 96, 020505(R) (2017).
  * [35] H. Chu, et al., Nat. Commun. 11, 1793 (2020).
  * [36] M. Mootz, L. Luo, J. Wang, l. E. Perakis, Commun. Phys. 5, 47 (2022).
  * [37] W. Kuehn, K. Reimann, M. Woerner, T. Elsaesser, The Journal of Chemical Physics 130, 164503 (2009).
  * [38] K. Leo, et al., Phys. Rev. Lett. 65, 1340 (1990).
  * [39] M. Lindberg, R. Binder, S. W. Koch, Phys. Rev. A 45, 1865 (1992).
  * [40] M. Mootz, J. Wang, I. E. Perakis, Phys. Rev. B 102, 054517 (2020).
  * [41] M. Mootz, L. Luo, C. Huang, J. Wang, llias E. Perakis, Multi-dimensional coherent spectroscopy of light-driven states and their collective modes in multi-band superconductors (2023).

Funding: THz quantum dynamics study was supported by the U.S. Department of
Energy, Office of Basic Energy Science, Division of Materials Sciences and
Engineering (Ames National Laboratory is operated for the U.S. Department of
Energy by Iowa State University under Contract No. DE-AC02-07CH11358).
Equilibrium characterization of superconductors and modeling were supported by
the U.S. Department of Energy, Office of Science, National Quantum Information
Science Research Centers, Superconducting Quantum Materials and Systems Center
(SQMS) under the contract No. DE-AC02-07CH11359


# STGIN: Spatial-Temporal Graph Interaction Network for Large-scale POI
Recommendation

Shaohua Liu  0009-0000-4933-1802 MeituanShanghaiChina<EMAIL_ADDRESS>,  Yu Qi  0009-0001-1135-0597 MeituanShanghaiChina<EMAIL_ADDRESS>,  Gen
Li  0009-0007-0908-182X MeituanShanghaiChina<EMAIL_ADDRESS>,  Mingjian
Chen  0009-0004-9365-8684 MeituanShanghaiChina<EMAIL_ADDRESS>,
Teng Zhang  0009-0009-4199-2935 MeituanShanghaiChina<EMAIL_ADDRESS>,
Jia Cheng  0000-0003-1702-4263 MeituanShanghaiChina<EMAIL_ADDRESS>and  Jun Lei  0000-0002-4015-8668 MeituanShanghaiChina<EMAIL_ADDRESS>
(2023)

###### Abstract.

In Location-Based Services, Point-Of-Interest(POI) recommendation plays a
crucial role in both user experience and business opportunities. Graph neural
networks have been proven effective in providing personalized POI
recommendation services. However, there are still two critical challenges.
First, existing graph models attempt to capture users’ diversified interests
through a unified graph, which limits their ability to express interests in
various spatial-temporal contexts. Second, the efficiency limitations of graph
construction and graph sampling in large-scale systems make it difficult to
adapt quickly to new real-time interests. To tackle the above challenges, we
propose a novel Spatial-Temporal Graph Interaction Network. Specifically, we
construct subgraphs of spatial, temporal, spatial-temporal, and global views
respectively to precisely characterize the user’s interests in various
contexts. In addition, we design an industry-friendly framework to track the
user’s latest interests. Extensive experiments on the real-world dataset show
that our method outperforms state-of-the-art models. This work has been
successfully deployed in a large e-commerce platform, delivering a 1.1% CTR
and 6.3% RPM improvement.

POI recommendation, spatial-temporal, graph neural network

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
32nd ACM International Conference on Information and Knowledge Management;
October 21–25, 2023; Birmingham, United Kingdom††booktitle: Proceedings of the
32nd ACM International Conference on Information and Knowledge Management
(CIKM ’23), October 21–25, 2023, Birmingham, United Kingdom††price:
15.00††doi: 10.1145/xxxxxxx.xxxxxxx††isbn: 979-8-4007-0124-5/23/10††ccs:
Information systems Retrieval models and ranking

##  1\. Introduction

In recent years, Location-Based Service providers including Facebook,
Foursquare, and UberEats have become more and more popular. As one of the key
services of LBS providers, the POI recommendation utilizes past user behaviors
and contextual POI information to make a personalized recommendation. Unlike
the traditional recommendation system, the quality of the POI recommendation
is intrinsically linked to three dimensions of data: personal, spatial, and
temporal, as well as their mutual interactions(Islam et al., 2022). For
example, a user may order coffee in the morning on business days while looking
for a gym on Saturday afternoons.

Many approaches based on sequential user behavior data have been proposed to
recommend POI candidates. LSTPM(Sun et al., 2020) explores the temporal and
spatial correlations from the long-term behavior sequence and captures
geographical influence from short-term sequences. STPIL(Cui et al., 2021)
constructs various sequences to acquire the spatial-temporal periodic
interests of different granularities, then applies integration for multiple
interests. CatDM(Yu et al., 2020) divides a user’s check-in history into
several time windows and applies a personalized attention mechanism for each
time window. However, these works only take into account the user’s own
behaviors which could cause data sparsity issues in certain spatial-temporal
contexts.

Inspired by the idea of collaborative filtering(Wang et al., 2019) that
similar users tend to make similar choices, Graph Neural Network(Kipf and
Welling, 2017; Hamilton et al., 2017) has proven to be an effective way to
mitigate data sparsity. GE(Xie et al., 2016) and JLGE(Christoforidis et al.,
2018) jointly learn the embeddings of multiple bipartite graphs into the same
latent space. STGCN(Han et al., 2020) considers both user-region periodic
pattern and user-POI periodic pattern and fuses all the context information
into a unified graph. STPUDGAT(Lim et al., 2020) leverages spatial, temporal,
and preference factors from both local and global views to learn POI-POI
relationships. While these methods take advantage of graph structure to
improve representation learning, they fail to explicitly model the user’s
changing interests across different spatial-temporal contexts. Moreover, due
to the heavy cost of graph construction and sampling in large-scale industrial
systems, these methods have great difficulty in integrating graph
representation with the user’s real-time behaviors, which can lead to
performance degradation in online services.

Motivated by the above analysis, we propose a novel Spatial-Temporal Graph
Interaction Network for large-scale POI recommendation. Specifically, a
subgraph representation framework has been proposed to learn the user’s
interests from the spatial view, temporal view, spatial-temporal view, and
global view respectively. In addition, to capture the user’s latest interests
in a real-time manner, we devise a flexible mechanism to combine the interests
of multiple views with real-time behaviors in an industrial-friendly way. Our
main contributions can be summarized as follows:

  * •

We propose a spatial-temporal graph interaction method to capture the user’s
diverse interests under different spatial-temporal contexts. To the best of
our knowledge, it is the first work to use multi-view spatial-temporal
subgraphs for POI recommendation.

  * •

We propose an industrial-friendly framework that combines the spatial-temporal
graph learning and the user’s real-time behaviors in an end-to-end manner and
is able to track the user’s latest preferences.

  * •

We have successfully deployed this work in a large location-based e-commerce
platform and achieved encouraging results in both online and offline
experiments.

##  2\. Problem definition

Generally speaking, a recommender system consists of two stages: matching and
ranking(Covington et al., 2016). In this paper, we implement our method in the
matching stage by learning the embedding representations of queries and POIs
respectively. With very few modifications, our model is also available in the
ranking stage.

Let $\mathcal{U}$ denote the set of users, $\mathcal{P}$ denote the set of
candidate POIs, $\mathcal{S}$ denote the set of locations, and $\mathcal{T}$
denote the set of time slots that are divided by a specific pattern(like an
hour, day, etc). ${p^{u}}=\\{p_{1}^{u},p_{2}^{u},...,p_{L}^{u}\\}$ denotes the
behavior sequence of the user $u$. It is a list of POIs that are ordered by
the corresponding behavior timestamps. Here $L$ denotes the length of the user
$u$’s behavior sequence. In this work, we consider clicking as the behavior
type.

Since users are consistently interacting with new POIs, we divide the behavior
sequence $p^{u}$ into two parts, one is the real-time sequence ($p_{r}^{u}$)
which grows with new click behaviors, the other behaviors are considered as
the history sequence ($p_{h}^{u}$).

For a query $q=(u,s,t)$ requested by a user $u$ under current location $s$ and
time $t$, the goal of the POI recommendation is to select the top $K$ POIs
($\mathcal{P}^{\prime}$) that the user would be interested in. It can be
formulated as

(1)
$\mathop{\arg\max}_{\mathcal{P}^{\prime}\subset\mathcal{P},|\mathcal{P}^{\prime}|=K}\sum_{p\in\mathcal{P}^{\prime}}sim(\boldsymbol{e_{q}},\boldsymbol{e_{p}}),$

where $\boldsymbol{e_{q}}$ is the vector representation of the query $q$, and
$\boldsymbol{e_{p}}$ is the vector representation of the POI $p$. The function
$sim(\textperiodcentered)$ calculates the similarity of two vectors.

Figure 1. Overall network structure and construction of multi-view subgraphs.

Model Structure

##  3\. Methodology

In this section, we elaborate on the proposed Spatial-Temporal Graph
Interaction Network. Model structure is shown in Figure 1(a).

###  3.1. Spatial-Temporal Graph Learning

Users tend to show different preferences in different spatial-temporal
contexts. Since a unified graph mixes information from various contexts, it is
difficult to accurately express the user’s interest in a specific spatial-
temporal context. Therefore, we use multi-view subgraphs for context-specific
interest learning.
Graph Construction. Based on the user’s history sequence $p_{h}^{u}$, we
construct subgraphs from the four types of views: global view, spatial view,
temporal view, and spatial-temporal view. Each subgraph has two types of
nodes, representing users and POIs in our task, connected by two types of
heterogeneous edges. Clicking edges represent users’ explicit interests. As
shown in Figure 1(b), we extract POIs from behavior sequences that occurred in
the corresponding spatial-temporal contexts and build an edge between the user
and each POI. Note that, clicking edges in the global view are constructed
with POIs in all contexts. Co-clicking edges represent the implicit relation
between POIs. An edge is constructed between two POIs that have been clicked
by the same user within a session(Feng et al., 2019). It should be emphasized
that co-clicking edges are shared among all subgraphs.
Graph Sampling & Aggregation. To precisely capture the user’s interest in each
view, we devise a spatial-temporal oriented mechanism for node aggregation.
Specifically, for each user $u\in\mathcal{U}$, we adopt meta-path(Dong et al.,
2017) “user $\xrightarrow[]{click}$ POI $\xrightarrow[]{co-click}$ POI” to
generate neighbors. Such a sampling strategy not only preserves the uniqueness
of each view but also enhances the representation through shared co-clicking
edges. Taking the temporal view $t_{i}\in\mathcal{T}$ as an example, neighbor
set $B_{t_{i}}^{u}$ contains both POIs clicked by the user $u$ at time $t_{i}$
and POIs co-clicked by others within a short period. Then we apply a two-layer
GAT(Veličković et al., 2018) mechanism to get the user’s interest in this
temporal view as follows

(2) $\boldsymbol{u_{t_{i}}}=GAT\\_AGG(\\{p|p\in B_{t_{i}}^{u}\\}).$

Similarly, we can get the user’s interest from the spatial view
$\boldsymbol{u_{s_{j}}}$, the spatial-temporal view
$\boldsymbol{u_{s_{j}t_{i}}}$, and the global view $\boldsymbol{u_{g}}$. The
neighbors for each POI are generated from the meta-path “POI
$\xrightarrow[]{co-click}$ POI $\xrightarrow[]{co-click}$ POI”, note that we
skip the user node when generating POI’s neighbors, since a user may have
different types of behaviors in the sequence, which may introduce noise for
learning the POI. Again, we use a two-layer GAT mechanism to get the
representation of the POI as $\boldsymbol{e_{p}}$, which is shared across all
views.
Multi-view Interests Interaction. Different contexts play different roles in
determining each user’s interests. Therefore, we design a user-dependent
attention module to learn the relations among different views. For all spatial
views, since POIs located far away are hardly reachable, we fetch the user’s
spatial view that exactly matches the current location. For all temporal
views, we apply an attention aggregation process to learn their mutual
influence guided by the user’s global view. The process is defined as

(3)
$\boldsymbol{u_{t}}=\sum_{i=1}^{T}\alpha_{t_{i}}\cdot\boldsymbol{u_{t_{i}}}.$

Note that $\alpha_{t_{i}}$ is the attention weight and is defined as

(4)
$\alpha_{t_{i}}=\frac{exp(f(\boldsymbol{W_{t}^{T}}[\boldsymbol{e}_{|t_{i}-t_{q}|}||\boldsymbol{u_{t_{i}}}||\boldsymbol{u_{g}}]))}{\sum_{k=1}^{T}exp(f(\boldsymbol{W_{t}^{T}}[\boldsymbol{e}_{|t_{k}-t_{q}|}||\boldsymbol{u_{t_{k}}}||\boldsymbol{u_{g}}]))},$

where $f(.)$ is the activation function, —— indicates concatenation,
$\boldsymbol{e}_{|t_{i}-t_{q}|}$ denotes the embedding of the absolute
difference between query time $t_{q}$ and the time of the corresponding
temporal view $t_{i}$, $T$ denotes the number of time slots,
$\boldsymbol{W_{t}^{T}}$ is the weighting matrix.

The same process can be applied for spatial-temporal views and get
$\boldsymbol{u_{st}}$ as the desired output.

Finally, we get the user’s multi-view interest through all views

(5)
$\boldsymbol{u_{h}}=f(\boldsymbol{W_{h}^{T}}[\boldsymbol{u_{g}}||\boldsymbol{u_{s}}||\boldsymbol{u_{t}}||\boldsymbol{u_{st}}]).$

###  3.2. Real-time Interests Perception

Users are constantly interacting with new POIs, and these real-time behaviors
are essential for predicting users’ subsequent interests. Therefore, we need
to develop a flexible real-time interest tracking mechanism compatible with
spatial-temporal graph learning.
Real-time Behaviors Modeling. Users’ real-time behaviors ($p_{r}^{u}$)
indicate their latest needs, but historical habits and current context should
not be overlooked. First, historical habits such as brands and lifestyles
often play an important role in current decisions. Second, given the context
of the current time and location, users’ next visit is likely to be related to
a subset of real-time behaviors. Consequently, we model the user’s real-time
interest evolution based on the above factors

(6)
$\boldsymbol{u_{r}}=\sum_{i=1}^{M}\alpha_{r_{i}}\cdot\boldsymbol{e_{p_{r_{i}}}}.$

$\boldsymbol{e_{p_{r_{i}}}}$ denotes the representation of a recently clicked
POI. $M$ is the length of real-time behaviors. $\alpha_{r_{i}}$ is the
attention weight, which is formalized as

(7)
$\alpha_{r_{i}}=\frac{exp(f(\boldsymbol{W_{r}^{T}}[\boldsymbol{e}_{|t_{r_{i}}-t_{q}|}||\boldsymbol{e}_{|s_{r_{i}}-s_{q}|}||\boldsymbol{e_{p_{r_{i}}}}||\boldsymbol{u_{h}}]))}{\sum_{k=1}^{M}exp(f(\boldsymbol{W_{r}^{T}}[\boldsymbol{e}_{|t_{r_{k}}-t_{q}|}||\boldsymbol{e}_{|s_{r_{k}}-s_{q}|}||\boldsymbol{e_{p_{r_{k}}}}||\boldsymbol{u_{h}}]))},$

where —— indicates concatenation, $\boldsymbol{e}_{|s_{r_{i}}-s_{q}|}$ denotes
the embedding of the distance between query location $s_{q}$ and the location
of clicked POI $s_{r_{i}}$, $\boldsymbol{W_{r}^{T}}$ is the weighting matrix.
Adaptive Fusion Mechanism. we conduct a gating-based fusion to generate the
final query representation $\boldsymbol{e_{q}}$

(8)
$\boldsymbol{e_{q}}=\alpha_{1}\cdot\boldsymbol{u_{h}}+\alpha_{2}\cdot\boldsymbol{u_{r}}+\alpha_{3}\cdot[\boldsymbol{e_{t_{q}}}||\boldsymbol{e_{s_{q}}}].$

$\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$ are generated by the following
softmax function

(9)
$[\alpha_{1},\alpha_{2},\alpha_{3}]=Softmax([\boldsymbol{w_{h}^{T}}\boldsymbol{u_{h}},\boldsymbol{w_{r}^{T}}\boldsymbol{u_{r}},\boldsymbol{w_{c}^{T}}[\boldsymbol{e_{t_{q}}}||\boldsymbol{e_{s_{q}}}]]),$

where $\boldsymbol{w_{h}^{T}}$, $\boldsymbol{w_{r}^{T}}$, and
$\boldsymbol{w_{c}^{T}}$ are weighting vectors, $\boldsymbol{e_{t_{q}}}$ and
$\boldsymbol{e_{s_{q}}}$ are the embeddings of query time $t_{q}$ and query
location $s_{q}$.

###  3.3. Optimization

After obtaining the representations $\boldsymbol{e_{q}}$ and
$\boldsymbol{e_{p}}$ , we adopt the pairwise training method(Burges et al.,
2005) and minimize the hinge loss as follows

(10) $Loss=\sum_{i=1}^{N}\sum_{j=1}^{K}max(0,margin-
sim(\boldsymbol{e}_{q}^{i},\boldsymbol{e}_{p_{i}}^{+})+sim(\boldsymbol{e}_{q}^{i},\boldsymbol{e}_{p_{j}}^{-})),$

where $N$ is the length of the training data and $K$ is the number of negative
samples for each positive sample.

##  4\. Implementation

Training. Graph building and learning are the most time-consuming parts in
`STGIN`. Though Cartesian combinations between spatial and temporal
information may be numerous, the actual volume of graph data is acceptable
since most users only visited a small number of locations. Thus the graph can
be trained simultaneously with other parts. In practice, we divide timestamps
into four slots (morning, noon, dinnertime, and night) and use
geohash111https://en.wikipedia.org/wiki/Geohash of length 5 to divide a city
into multiple locations. For our production dataset with over 300 million
users, 20 million POIs, and 600 million clicking behaviors, it takes about
four hours to train on 50 virtual machines, each providing 6 CPU cores and
40GB RAM.
Serving. Benefiting from the real-time interests perception module, the graph
structure could be updated at a given frequency without real-time
requirements. The learned embeddings are stored in a key-value table, thus we
can skip the real-time inference of graphs, which may become a bottleneck in
online service. It’s worth noting that having a new clicking POI will change a
user’s real-time sequence as well as the representation vector
$\boldsymbol{e_{q}}$. On average, it takes just 5.6 milliseconds to make an
online recommendation.

##  5\. Experiments

We design experiments to answer the following research questions: (RQ1): How
does STGIN perform compared to other state-of-the-art models? (RQ2): What are
the effects of different components? (RQ3): How effective are the spatial-
temporal views interacting with each other? (RQ4): How does STGIN perform in
online A/B tests?

###  5.1. Experimental Setup

Table 1. Statistics of dataset Data | Users | POIs | Records | BehaviorLen | Locations
---|---|---|---|---|---
Train | 637563 | 108551 | 2023304 | 149.6 | 320
Test | 217561 | 60982 | 532194 | 149.9 | 305
Table 2. Comparisons of different methods Methods | HitRate@200 | Recall@200
---|---|---
`CatDM` | 21.37% | 15.42%
`STPIL` | 23.80% | 17.45%
`JLGE` | 19.96% | 14.45%
`STGCN` | 18.52% | 13.14%
`STPUDGAT` | 22.21% | 16.04%
`STGIN` | $\boldsymbol{25.87}\%$ | $\boldsymbol{19.12}\%$
`STGIN-RT` | 19.71% | 14.10%
`STGIN-Temporal` | 22.62% | 16.35%
`STGIN-Spatial` | 25.12% | 18.57%
`STGIN-Interaction` | 22.56% | 16.26%

Dataset and Settings. We conduct offline experiments on industrial production
data of a large e-commerce platform. It spans one month(2022-08-09 to
2022-09-08) and is sampled in a few cities. The statistics of dataset are
shown in Table 1. We take behaviors over the last 24 hours as the real-time
sequence. All experiments are trained on Tensorflow with graph learning engine
Euler222https://github.com/alibaba/euler. We adopt Adam(Kipf and Welling,
2015) as the optimizer and the dimension of each trainable feature embedding
is 16. The batch size and learning rate are set to 1024 and 0.001. We apply
LeakyReLU(Nair and Hinton, 2010) as the activation function. The number of
negative samples is 6.
Competitors. We implement several competitive baselines for evaluation.
`JLGE`(Christoforidis et al., 2018) expresses user, POI, location, and time in
the same latent space through multiple bipartite graphs. `STGCN`(Han et al.,
2020) fuses all spatial-temporal information into a unified graph and applies
a time-based neighborhood sampling algorithm. `STPUDGAT`(Lim et al., 2020)
uses both spatial and temporal factors to model the POI-POI relationship.
Besides, we also implement two popular sequential models. `CatDM`(Yu et al.,
2020) and `STPIL`(Cui et al., 2021) both explore user interests from spatial-
temporal behavior subsequences and integrate them through attention
mechanisms. To further understand the contribution of each component, we
design four variants of our model. `STGIN-RT` drops real-time behavior
sequence. `STGIN-Temporal` drops subgraphs of the temporal view. `STGIN-
Spatial` drops subgraphs of the spatial view. `STGIN-Interaction` merely
concatenates graph representation and real-time sequence without their mutual
interaction.
Metrics. We apply widely used HitRate@K and Recall@K(Huang et al., 2020; Lv et
al., 2019; Hidasi et al., 2015) to evaluate the performance of all methods.

###  5.2. Offline Evaluation

Exp-RQ1. As shown in Table 2, `STGIN` outperforms other competitors in all
metrics. Due to the heavy cost of graph construction and sampling, `STGCN`
faces a major challenge in addressing users’ real-time behaviors. `JLGE` and
`STPUDGAT` try to solve this challenge with the POI-POI graph but pay little
attention to the interest interaction. In addition, a unified graph cannot
express fine-grained interests in different spatial-temporal contexts. `CatDM`
and `STPIL` are good at dealing with users’ real-time behaviors, but they
cannot address the issues of data sparsity when users have limited behaviors.
Exp-RQ2. From table 2, all the components significantly benefit our task.
Precisely, `STGIN-RT` demonstrates real-time behaviors are essential for POI
recommendation. `STGIN-Temporal` and `STGIN-Spatial` show that it is useful to
capture the interests of users from spatial-temporal views. `STGIN-
Interaction` shows that the potential of real-time behaviors can be enhanced
by interaction.
Exp-RQ3. To validate the effects of interaction among spatial-temporal views,
we design temporal-related experiments from coarse to fine-grained.
`STGIN_Only_Temporal` only retrieves a temporal subgraph that matches the
current time. `STGIN_Sum_Temporal` retrieves temporal subgraphs of all time
slots and applies sum pooling operation. In Figure 2, `STGIN-Temporal` gets
the lowest score because it omits temporal information. `STGIN_Only_Temporal`
performs slightly better than `STGIN_Sum_Temporal`, for the latter method does
not differentiate the user’s attention on different temporal views.
Consequently, `STGIN` achieves the best performance.

Figure 2. Comparisons of temporal-related subgraphs. Table 3. Result for Online A/B Test Metric | CTR | RPM
---|---|---
Relative Improvement | +1.1% | +6.3%

###  5.3. Online A/B Test (RQ4)

Based on the optimizations described in the section 4, we have successfully
deployed the `STGIN` model in our production environment to handle the real
traffic of a large e-commerce APP. Online evaluation metrics are CTR and RPM
(Revenue Per Mille). As noted in Table 3, both metrics increased compared to
the base model, demonstrating the effectiveness of `STGIN` in practical
environments.

##  6\. Conclusion

In this paper, we propose a novel approach for POI recommendation called
`STGIN`. It can characterize a user’s diverse interests from spatial-temporal
subgraphs and capture a user’s latest interests through real-time behaviors
modeling. Both online and offline experiments demonstrate the effectiveness of
our method.

## References

  * (1)
  * Burges et al. (2005) Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005.  Learning to rank using gradient descent. In _Proceedings of the 22nd international conference on Machine learning_. 89–96.
  * Christoforidis et al. (2018) Giannis Christoforidis, Pavlos Kefalas, Apostolos Papadopoulos, and Yannis Manolopoulos. 2018\.  Recommendation of points-of-interest using graph embeddings. In _2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA)_. IEEE, 31–40.
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep neural networks for youtube recommendations. In _Proceedings of the 10th ACM conference on recommender systems_. 191–198.
  * Cui et al. (2021) Qiang Cui, Chenrui Zhang, Yafeng Zhang, Jinpeng Wang, and Mingchen Cai. 2021.  ST-PIL: Spatial-Temporal Periodic Interest Learning for Next Point-of-Interest Recommendation. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 2960–2964.
  * Dong et al. (2017) Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. 2017\.  metapath2vec: Scalable representation learning for heterogeneous networks. In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_. 135–144.
  * Feng et al. (2019) Yufei Feng, Fuyu Lv, Weichen Shen, Menghan Wang, Fei Sun, Yu Zhu, and Keping Yang. 2019.  Deep session interest network for click-through rate prediction. In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_. 2301–2307.
  * Hamilton et al. (2017) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017\.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ 30 (2017).
  * Han et al. (2020) Haoyu Han, Mengdi Zhang, Min Hou, Fuzheng Zhang, Zhongyuan Wang, Enhong Chen, Hongwei Wang, Jianhui Ma, and Qi Liu. 2020.  STGCN: A Spatial-Temporal Aware Graph Learning Method for POI Recommendation. In _2020 IEEE International Conference on Data Mining (ICDM)_. IEEE, 1052–1057.
  * Hidasi et al. (2015) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015.  Session-based recommendations with recurrent neural networks.  _arXiv preprint arXiv:1511.06939_ (2015).
  * Huang et al. (2020) Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020.  Embedding-based retrieval in facebook search. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2553–2561.
  * Islam et al. (2022) Md Ashraful Islam, Mir Mahathir Mohammad, Sarkar Snigdha Sarathi Das, and Mohammed Eunus Ali. 2022\.  A survey on deep learning based Point-of-Interest (POI) recommendations.  _Neurocomputing_ 472 (2022), 306–325.
  * Kipf and Welling (2015) Thomas N Kipf and Max Welling. 2015.  Semi-Supervised Classification with Graph Convolutional Networks. In _International Conference on Learning Representations_.
  * Kipf and Welling (2017) Thomas N Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _International Conference on Learning Representations_.
  * Lim et al. (2020) Nicholas Lim, Bryan Hooi, See-Kiong Ng, Xueou Wang, Yong Liang Goh, Renrong Weng, and Jagannadan Varadarajan. 2020.  STP-UDGAT: Spatial-temporal-preference user dimensional graph attention network for next POI recommendation. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 845–854.
  * Lv et al. (2019) Fuyu Lv, Taiwei Jin, Changlong Yu, Fei Sun, Quan Lin, Keping Yang, and Wilfred Ng. 2019.  SDM: Sequential deep matching model for online large-scale recommender system. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 2635–2643.
  * Nair and Hinton (2010) Vinod Nair and Geoffrey E Hinton. 2010.  Rectified linear units improve restricted boltzmann machines. In _Proceedings of the 27th international conference on machine learning (ICML-10)_. 807–814.
  * Sun et al. (2020) Ke Sun, Tieyun Qian, Tong Chen, Yile Liang, Quoc Viet Hung Nguyen, and Hongzhi Yin. 2020.  Where to go next: Modeling long-and short-term user preferences for point-of-interest recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 214–221.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018.  Graph attention networks. In _Proceedings of International Conference on Learning Representations_.
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019.  Neural graph collaborative filtering. In _Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval_. 165–174.
  * Xie et al. (2016) Min Xie, Hongzhi Yin, Hao Wang, Fanjiang Xu, Weitong Chen, and Sen Wang. 2016\.  Learning graph-based poi embedding for location-based recommendation. In _Proceedings of the 25th ACM international on conference on information and knowledge management_. 15–24.
  * Yu et al. (2020) Fuqiang Yu, Lizhen Cui, Wei Guo, Xudong Lu, Qingzhong Li, and Hua Lu. 2020\.  A category-aware deep model for successive POI recommendation on sparse check-in data. In _Proceedings of the web conference 2020_. 1264–1274.